Dynamically updating a model on an embedded device

I was wondering about updating the neural network model already deployed on device without compiling and uploading the whole application.

Since we will have better performing models we would want to replace the older models with newer ones. Right now, the way to do it is by using the newer model in the project and compiling and uploading the whole application to the device via the USB.

What is interesting is that the raw bytes of the model get written on the device which are fetched and used by the interpreter. Can it be that updating a model on device is as simple as rewriting these bytes? What if I want to dynamically update the model by receiving the new model over say, BLE?

Are there examples for BLE in Arduino Nano 33 BLE Sense where bytes are sent to the device from some smarphone app etc?

3 Likes

This is a really good question, and one that is quite relevant and important.

Yes, you can update the models. If you have a BLE, then you can stream down the TFLite flat buffer file format and load that in via GetModel() function for TFLite Micro. But you will of course have to re-setup the interpreter to use the new model. Are you familiar with TFLite Micro already or are you still dabbling with it? In Course 3 of the edX TinyML series we cover a lot of this material in depth and introduce you the API functions.

To the best of my knowledge, there are no explicit BLE examples like that but always a good place a start would be to share one :wink:

4 Likes

I am new to TinyML and am enrolled in course 2. I want to create an Arduino application which has dynamic changing of the model at runtime. I think I need to learn about the API functions so I will be enrolling in March 2nd batch of course 3.

Sounds good @abu. In course 3, we are getting at exactly what you are interested in learning ā€“ the API functions. We are almost there, just another few days folks. Hang in there! :upside_down_face:

1 Like

Hello @vjreddi ! I remember about dynamic loading of the model, you said there were some restrictions like redeclaring the operators. I am now enrolled in course 3 and saw Peteā€™s video on OpResolver. I assume you were talking about the same. I tried sending part of the ā€˜Hello Worldā€™ model over serial (since I couldnā€™t send the whole model without errors), combining with the rest of the stored bytes and dynamically loading the model and it worked fine. Is it safe to say that dynamic loading is easy if we use AllOpsResolver?
Another thing was that I did this in setup(). I assume, having this in loop() is bad design/programming because of heap fragmentation. Is that correct?
I also noticed that there are no forums for those modules of the course. I think many interesting discussions can start if a forum exists. :slightly_smiling_face:

1 Like

Hi!

I know you are totally getting the material based on the questions you are asking, thatā€™s awesome! :clap::clap::clap:

Yeap, exactly. If you are going to update the same model and not change any architecture then you can certainly keep the same fixed set of operators. However, thatā€™s rarely going to be the case. Youā€™ll almost want to change the operators at some point, in which case you want to support AllOps.

Even if you use AllOps, keep in mind that the ops supported by TF will change over time, so the interpreter thatā€™s loaded on the device itself might get outdated a bit over time. But this is generally less of an issue compared to the previous case.

Well, you really have to do this in the loop() function because the loop is the one that is going to be updating every time. So you want to check if any new updates are available and if so then you have to pull the new version. For instance, I have a version that works over BLE and I push the model over BLE so Iā€™m cable free :wink:

One of the issues here, with both our solutions, is that we arenā€™t updating the Flash and only update the model in SRAM so when the power goes out ā€¦ we are back to the original baseline model :frowning:

2 Likes

Thanks again for the answer, that cleared a lot of things. I remember Pete used the word ā€˜difficultā€™ when talking about replacing the model in the flash. Maybe you guys are already thinking about how this should be done for microcontrollers.

I have one more question. This was the screenshot you showed


I tried this and created the hexdump but I did not find my model bytes in there. Do I have to take care of something else, like big-endian vs little-endian arrangement or some special text/hex editor?

Thatā€™s odd.

Andā€¦ you followed the exact instructions as I described here: Authn | edX

Right?

Did you use the same exact process?

If not, we may need to figure out OS differences etc.

I found it!
I used www.onlinehexeditor.com/ I donā€™t know why notepad and notepad++ were not able to search in the hexdump. This time, I uploaded the file and used their search tool and it spotted the byte sequence. :grin:

Hi @vjreddi You posted a reply on Course 3 forum, I thought continuing the thread here would probably be better. You wrote, ā€œThere is a way to update the weights on the fly over BLEā€¦ā€ I have updated the entire model over BLE but canā€™t seem to figure out how to only update the weights. Can you give some hints?
I also see your point regarding making permanent changes. I saw some mbed OS libraries, like ones for file system. Should be a good practice and fun to try those outā€¦

1 Like