A story of a DIY project that combines deep learning and an Arduino Pro board.

It finally happened. I permanently fell into the world of machine learning on embedded systems. The first signs of my new crush began to appear timidly. In the beginning, I was fascinated by utilizing Raspberry Pi. A combination of Linux and Python guaranteed fast results. It was a straightforward translation from a bigger machine for a machine learning engineer. Moreover, the feeling of doing something tangible and arming it with my code was a nice change from creating code strictly intended for the backend. 

Then it was time for Nvidia Jetson. This device is powerful and tailor-made for deep learning and computer vision from the first boot. It was a pleasure to set up and work with the environmental shield. Despite its small size, I was still plagued by the feeling that the device was too big, and I couldn’t apply it in all scenarios. To be precise, it was too heavy for my Mavic to carry.

That's how I came across the Arduino Nano. I have to admit that I was impressed and disappointed at the same time. The platform offers great opportunities. We are discussing great expansion possibilities, easy access to components, and a chance to run small-sized machine learning models. Every accessory applicable to the commonly used Arduino Uno is also accessible for Nano. On the other hand, working on ML models is not a pleasant experience. This is due to strict memory constraints.

However, I didn’t give up. Arduino has my infinite trust. At the same time, working with on-edge machine learning was becoming more and more interesting, and the number of ideas for applications was increasing. I needed a more powerful device with similar dimensions. That's how I came across the Arduino Portenta.

This blog post will describe my first DIY project utilizing deep learning on an Arduino Pro board. Grab a cup of coffee. This will be a long one 🙂

Problem

We will start by finding a reasonable problem to solve. I love environmental sciences. That’s why I always look for side-project ideas associated with Earth Sciences, remote sensing, and environmental protection. The winter is not terribly harsh, but even I'm reluctant to leave my warm apartment. It wasn't easy for me to come up with something entertaining.

Fortunately, I didn't have to look far. The idea came from my kids, who, due to their age, do not yet understand the need to introduce waste selection and recycling. Therefore, trash lands in our bins randomly, and there should be a way to prevent that.

The plan is simple. Design and program waste bins in such a way they will help develop correct recycling habits.

Additionally, it has to be a fun project to develop. For me, this means a total overkill in terms of technology 😅. Let’s do it using computer vision, deep learning, embedded ML, and a dose of the Internet of Things.

Looking for an interesting project topic made me think more deeply about embedded ML in industry, agriculture, city management, environmental monitoring, and others. Possibilities are great, but there are also plenty of disadvantages and limitations. This is a topic for a complex discussion. I feel that I need to find an external source of knowledge. If you know of a good conference related to IoT and ML, I would be grateful for providing me with such information.

Unboxing

The main components arrived exactly one day after ordering, which was a pleasant surprise. I decided to work with Arduino Portenta H7 on this project. The board was enhanced with a vision shield. A 320x320 pixels camera sensor is enough to classify basic waste types. The shield is LoRa compatible which will be handy in future projects. There is also an Ethernet version of it.

Preparing the board

Combining the board and the shield is straightforward. It looks sturdy enough to work with additional protection, but I wanted to avoid breaking it during solution prototyping. This is still not a popular board among hobbyists, so no cases are available. Fortunately, the component is tiny, and there are no problems in 3d printing a case. I frequently use a great project called “customizable everything box.” It has tons of remixes, and it is easy to find something that will fit the needs of a DIY project.

Now, the tricky part. To work with the board, we needed to flash it. I am going to use Portenta in two scenarios.

The first one is related to data collection and model preparation. I plan to utilize Edge Impulse for that. Fortunately, they have a great tutorial on integrating the board with the platform. Remember to use your browser to connect to the device. It makes the process smooth.

The second scenario combines inference using an ML model and controlling some additional hardware. I am a fan of MicroPython and CircuitPython. That’s why I used OpenMV, which needs different installation steps. After downloading the IDE and connecting the board, you will be notified about the possibility of flashing the device. 

Please note that the scenarios are not interoperable. 

There is also one obstacle you must tackle if you cannot connect your device to a development machine. Uninstalling modemmanager and adding several udev rules should do the trick. Read more in Porenta setup troubleshooting.

Assemble

Preparing the colorful wastebaskets was the first task on my list. The setup is composed of three bins. Each color represents different types of waste: blue (paper), yellow (plastic and metal), and green (glass). The controller bin has been placed in the middle. On the top, I cut out a rectangular hole where I placed the Arduino board with a camera. Each bin has a pair of LED strips mounted to the internal walls. A small power bank powers everything. Of course, the internal wiring looks like spaghetti 😅.

Data preparation

There are multiple approaches to data preparation. I could work with an existing dataset or prepare it on my own. Although there are publicly available labeled datasets that I could adjust to the problem, I decided to create my own small sample set. It was related to the specificity of the Arduino camera sensor and the desire to check the capabilities of the Edge Impulse platform. This was a good choice. Edge Impulse's data acquisition feature is intuitive, allowing me to deal with the task efficiently. Enabling the connection between the platform and the device was a great idea.

‍

Preparing the model

I decided to continue the flow of Edge Impulse and trained the machine learning model utilizing the TensorflowLite framework. Firstly, I created an Impulse which is a blueprint of the training pipeline. The next step was related to data preprocessing. Finally, I’ve chosen a predefined model architecture and initialized the process. The model converged rapidly with the transfer learning option available and yielded a good result. I am still curious how much customization is possible utilizing Edge Impulse, but I will check it soon on a more complex use case.

‍Deployment

There is a variety of options available in terms of deployment. The preferable one is to install the model as a device firmware. I wanted to add additional customizations and eventually deployed the model as an OpenMV package. This step produces a package containing the model, a text file containing labels, and a template inference MicroPython script. All that was left was to edit the file to add the lines that support the LEDs.

Results

After putting everything together, it was time for user acceptance tests. The kids enjoyed the first 15 minutes of the job, but it was worth it anyway. 

Of course, the desired result was not achieved immediately. It is common to work iteratively on machine learning models. I went through the entire process, from data collection to model training and customization, several times until I managed to bring the solution to a usable state.

Summary

To sum up, it was time well spent. I learned a lot and had a great time assembling and coding the solution. I am convinced that machine learning on edge will be an important topic in the future. There are multiple areas where such an approach is applicable. At the same time, I think there is still much to be done. Fortunately, Edge Impulse and OpenMV make things a lot easier.

To learn more about automated waste classification, you can read an article by Kamil Rzechowski, our Senior Machine Learning Engineer.

Adam Wawrzyński, Senior Machine Learning Engineer at ReasonField Lab:

I've always wondered what areas would benefit from on-edge processing rather than the traditional approach where we have lightweight nodes that send data for processing to a centralized server with more computing power. From the point of view of maintaining a network of devices, it is easier to make much simpler devices that can run for up to several years on battery power than to make more complex and power-hungry devices with advanced processing (in this case, machine learning model) that will last a short time on battery power or even need to be powered from the grid. My experience with IoT systems shows that data collection is the most common use case. In this case, the processing happens in a centralized manner. I'm curious if embedded ML gains an advantage over this approach.