End-to-End transportation mode recognition software

A story about a multilayer machine learning project developed at a glorious velocity

Introduction

The Codos Foundation success story is one of motivation, expertise, and dedication being rewarded. There are many different projects out there all with various problems to solve using creative approaches and technologies. There are also multiple skill sets that can help to accomplish the task and to do so in the desired timeline. We already have a lot of experience in software engineering, we are good at sharing knowledge among team members, and to be honest, in most cases, it’s hard to surprise us.

However, sometimes we are faced with a unique task. You probably know this feeling when you check the initial assumptions of a project and meet with the client, and from this moment, you know that you are going on an adventure.

The best catalyst for a successful project is the team's motivation, and the best way to stimulate it is a super exciting and demanding engineering task. We had this and more for the Codos project.

In this blog post, you will read about our journey from a concept of an advanced, machine-learning-driven transport mode recognition solution to a robust proof-of-concept (PoC) and preparations for a minimum viable product (MVP).

Grab a coffee. This will not be a short story 🙂

Task

A well-defined problem is a prerequisite for a good IT project. It is not just because the entire architecture of the project is based around it, but also because it sets the team in motion and keeps them motivated and focused. Understanding the problem and its implications is perhaps the most critical aspect of any project.

In this regard, Codos did not disappoint. They approached us with the challenge of detecting transportation modes based on data from mobile phones. Our role would be to create an end-to-end solution that utilizes human activity recognition to calculate CO₂ emission avoidance. 

The Codos Foundation's primary objective is to reward users for commuting sustainably. By automatically detecting the mode of transportation, the Codos app calculates CO₂ avoidance compared to a combustion engine car and rewards users accordingly – a noble idea and a viable product.

Want to know more? Visit the Codos Foundation website.  

The task is demanding from a technical perspective. It involves working with continuous incoming mobile sensor data and requires high machine-learning model precision. Moreover, understanding the spatial context is critical since user motion takes place in cities. It places the task at the intersection of science and engineering, thus, making it ideal for our company.

Team

Somebody wise said that you must gather your party before venturing forth. ReasonField Lab comprises of engineers focused on machine learning. Although a deep neural network lies at the base of the solution, it cannot operate independently. A mobile application must be developed to interface the sensor data and as a way to interact with the user. With multiple microservices utilizing the model inference results and working together to calculate the CO₂ avoidance emission score, a platform capable of orchestrating other parts of the system is required.

That’s why we teamed up with other companies: SoftwareMill - a software development company with great Scala developers capable of working with big data and blockchain solutions, and Qodeca - experts in building management and user-facing applications.

This is how we started working on the project: with an engineering team composed of Machine Learning engineers, Scala software developers, iOS developers, DevOps engineers, Quality Assurance Engineers, and Geographic Information System experts. Needless to say, it was a very interesting mix. Simultaneously, other team members worked on coordinating our activities by taking on the roles of Product Owners / Managers, designing the User experience / Interface, and conducting Blockchain workshops. It took about two weeks to establish the team coordination between the three companies' activities.

Importantly, the client team was also involved from day one in the engineering project. They were always available to answer the millions of questions we asked daily. The cooperation between the client and the engineers turned out to be crucial. Luca and Luka, kudos for the approach and support!

Planning before coding

We were given a lot of freedom and got to design the solution's architecture from scratch. We were drawn to coding, but first, we had to ensure that we understood the client's expectations to make a good plan. The time to complete the task was specified. We wanted to allow Codos to present their solution and share their idea with the world as soon as possible.

We wanted to be able to track the progress and stages of the PoC at all times. We wanted the Codos to have control over what is happening in their project despite the considerable speed of development. It was also crucial from the point of view of machine learning, which needs a large volume of high-quality data. This data must be collected and properly processed. In our case, this meant implementing the first version of the entire platform within two months of starting work. The timing was crucial to fit into the requirements imposed by field research (collecting data).

Want to know why we started with a PoC? Read our website to learn about our approach.

We decided to start a series of workshops with Codos so that all elements could be completed on time. We worked out a detailed outline of the entire system, starting from machine learning through the requirements of cloud platforms, the mobile application, and finally, blockchain.

The result of the first round of the workshops was the transport mode detection (TMD) research document in which we described the steps needed to construct a flexible machine learning solution. This led to the development of platform architecture and a mobile application research (annotation) mode.

Research mode activated

Machine learning projects need a lot of data. But not just any data. The model's accuracy depends directly on the quality of the data obtained. In novel projects, the data set required by the problem rarely exists and so has to be created.

One can't just go into the field and download the data. Firstly, a set of rules had to be created to be followed when collecting data. The guarantee of performing this step with due diligence is treating the entire project as scientific research. This approach has not failed us yet, as it works perfectly for machine-learning projects.

‍

The rules have been materialized as a mobile application and an annotation manual. Then we formed a team of volunteers who agreed to help us collect data. Each participant was provided with an iPhone and an Apple Watch to take measurements during their trips between home and work.

Our research collected over 150 million records describing motion patterns.

It should be emphasized that when the first team member went to the field to collect data, we already had a working cloud platform processing data obtained from a mobile phone and preparing it for the machine learning process. This happened at the end of the second month of the project. The project velocity was amazing.

Feedback-driven development

The system wouldn't exist if we didn't talk to each other all the time. Well-designed architecture helps to set the framework in which the engineering team operates. However, success depends on how efficiently we can communicate information about the work being done on the mobile application, cloud platform, or machine learning. Slack, and Google Meet helped a lot. 

Did I mention that the team was remote, and everyone worked from home?

Look how remote teams are built in SoftwareMill.

Wrapping it all up

After six months of development, we were ready for user acceptance tests (UAT) of the system PoC. The cloud platform was ready and capable of serving requests from the mobile application. The machine learning transport mode recognition service was in place and waiting to run inference against acquired sensor data.

The best way to test a machine-learning-driven solution is to utilize it in real conditions. Therefore, our research team, backed by QA and our clients, started their journeys. We checked different scenarios, from walking to public transport commutes, and took a lot of notes to adjust the system.

The results surpassed our expectations. All the platform pieces fit, and the solution is operational.

Follow the Codos Foundation's social media if you want to try it. The foundation is active and attends multiple conferences in Europe and the US. There are also considerable trials being prepared, and you can sign up for their beta version.

The culmination of our efforts is a comprehensive report for the Codos Foundation. We described each part of the Codos Platform in detail. Such a step is important to retain knowledge regarding the project. It gives the possibility to rapidly initialize the next steps and extend the team on demand. Good documentation is also important in the freedom to operate (FTO) verification process or even gaining a patent. 

Summary

This was a great adventure. In about half a year, we designed and implemented an advanced proof-of-concept. The solution can produce and utilize a deep-learning model capable of recognizing transport modes. We combined multiple skill sets: machine learning, backend development, mobile development, blockchain, GIS, and scientific research, to provide Codos with the desired functionalities.

Without the team members' extensive experience and the trust that our clients placed in us, this success would not be possible.

The project's biggest challenge was putting all tasks together in time to start designing and training machine learning models early. We wanted to have as much time as possible for experiments, trying different neural network architectures or data processing methods. This required a lot of discipline from the members of the entire team and the awareness that every element of work on the cloud platform or mobile application directly affects the result of machine learning.

What awaits us now? 

We are currently on the road to an MVP. It's time to plan the next tasks and speed up the machine. Stay tuned for more information. Every commute counts 🌿