IoT Device Management – OTA Software Updates, Config, Monitoring & TroubleshootingAdvice
Mender.io CTO Eystein Måløy Stenberg shares his expert insights on the challenges faced and opportunities to improve by better addressing the needs across the life cycle of IoT device management including OTA software updates, Configuration, Monitoring and Troubleshooting.
This article anticipates the upcoming release of Mender. Mender 2.7 is a landmark release for the Mender.io engineering team. The release sees the addition of device configuration management and troubleshooting features as add on packs to the already best-of-breed Mender.io OTA software updates solution. To mark, this release I want to explore the lifecycle of IoT device management, the challenges and key considerations for planning and execution.
Stages in the IoT device management lifecycle
There are 4 key stages in the IoT device management lifecycle.
- Provisioning – when the customer uses the IoT connected device for the first time
- Rapid improvement – this involves fixing the initial the “Version 1.0” issues with the device to get ahead of the competition
- Maintenance and support – this involves fixing security issues and improving software stability on the device
- Decommissioning – this involves taking the device out of the field at the end of its lifecycle which is typically 5 to 10 years.
These 4 stages map to 4 key processes that should happen at each stage:
- A one-time update over-the-air software update to the latest to provision the initial Configuration
- Monitor anomalies which should entail frequent software over the air (OTA) software application updates to the device. Issues should also be handled and triaged through Troubleshooting support via remote access to the device.
- A routine of quarterly OTA system updates should be established. Monitoring and troubleshooting of the device should be ongoing all the while. The device configuration should also be tweaked so as to optimize it.
- Potentially remove access with Configure or reset with OTA system update. Remove from Device Management.
Next, I examine the challenges faced by those who are managing IoT devices. Often, too much time is spent on integrating, setting up, or building homegrown device management solutions and this means that less time is spent on core product development.
Never underestimate what it takes to do proper device management
Some enterprises may underestimate what it takes to build a best-of-breed device management solution and the total cost of ownership becomes higher over time because it will naturally evolve. These needs can include more advanced device management use cases or changing hardware, operating system workflow needs. You will want to spend as little time as possible on device management, and as much as possible on delighting your customers and differentiating your product to increase market share and revenue for your product.
There are typically two areas of strategic challenge:
- Cloud and platform-based device management solutions cause lock-in. This gives your business less agility to select the best solutions for your team. There is also the future risk of vendor price hikes and future changes of your roadmap is restricted. You need a solution based on open source that you can easily integrate into changing needs of your business, be it development tool-set, cloud vendors or device hardware/OS.
- Robust and secure device management is a necessary cornerstone for an IoT product and therefore you need to find a high quality solution. It needs to not only support your use cases, but also to be built with security and robustness in mind. The last thing you would want to occur is a deluge of customer support queries coming in because of unrecoverable software update failures. Or worse still, to make the front-page industry media magazines for all wrong reasons on how your device management solutions were easily hacked and facilitated malicious actors and thieves with access to your customers’ data.
Opportunities for innovation in IoT device management
Next, I address the really big opportunities for innovation in device management. Innovation is a continuously occurring process and this brings the benefit of allowing you to be able to change continuously and at low risk.
- Therefore, the cornerstone for innovation in IoT is to have a solution for OTA software updates, supporting updating all the software of the device, be it system, application, firmware for the modem and the like. You don’t know in advance what needs to change.
- In order to lower risk and to safely be able to innovate, your OTA software updates process must be robust and secure. If it is robust, it is also resilient against an unreliable environment such as power loss in the middle of the update process, and even from bugs in the software itself. You need to be able to release an early build to some of your customers, knowing that you can safely roll back if there are any issues with the software.
- To innovate and gain a competitive advantage from day one of your IoT project, you must have a robust and secure OTA software updates process in place. Tesla serves as an example of this from the automotive industry.
Missed opportunities in IoT device management
Next, I assess the real failings or missed opportunities in the ways in which IoT device management is done by many organisations today. Device management is too often an afterthought. It becomes a “last minute” addition to the device design, right before the production release. Typically, the engineering team discovers that some form of device management will be needed just as they rush to meet the release deadline. Only then do they realise that the product must be able to be updated and supported after the release. This often results in some kind of quickly developed “backdoor” fix being put in place. This is a band aid solution that has serious consequences for the robustness and security of the devices in the project.
Furthermore, some organisations try to adapt solutions that were built for the cloud to IoT environments. This does not work well as the IoT environment is very different from the cloud environment. This is generally down to the fact that as IoT devices have unreliable power and connectivity, you can’t easily “duplicate” or “spin up a new” IoT device if it fails. A device management solution for IoT needs to take this into account. Let’s look at an example of this. In terms of software updates, Kubernetes is often used in cloud environments. You do not care if a single container works or not, you simply discard it automatically if it fails and start a new one. This can be automated. However, if a container or OS service fails on an IoT device, that device will fail to operate properly as a whole. Again, robustness is key in IoT.
Cloud solutions also often assume that they will be placed in restrictive environments, such as behind VPCs or firewalls. In contrast, an IoT device is out there “in the wild”. It’s not only insecure but may also be directly exposed through 3rd party wireless networks. It’s also prone to physical attacks, where someone may try to physically steal the storage device and extract your proprietary algorithms or customer data. This is why security is so different in IoT environments.
Hardware and software selection is an interesting and sometimes complex journey. There is a lot to choose from and tradeoffs with price, performance and security.
Sources of support for IoT device management
Today, organisations usually solicit help from a number from 3 main sources:
- Consulting or design companies focused on finding the best solutions
- Hardware vendors
- Cloud vendors
The problem with source 2 and 3 is that they are incentivised to sell their own solution and will naturally narrow the selection based on what their solution supports best. They may also have auxiliary solutions they might want to sell: For example, a hardware vendor may also provide some custom IoT management software they want to sell on top of the hardware. But it’s not the core competence of the hardware vendor. This leads to inferior solutions with limited adoption and increased complexity. Number 1 will likely provide the most independent and best solution.
Increasing standardisation will come to the rescue
In the end, rest assured that standardisation in terms of hardware and software will continue over time and make life easier for IoT product companies. It is a slow trend, but it is happening. In the 90s, it was common that product companies built their own operating systems. However, new IoT products today are based on Linux or a more standard RTOS such as FreeRTOS or Zephyr. The same will happen in all areas of the stack with the effect of lowering cost where there is no differentiation.
In summary, do your planning for thorough IoT device management when designing your IoT product. Get a deep understanding of the 4 key stages of device lifecycle management from inception through to decommissioning. Robust and secure OTA software updates must be planned for immediately as it has implications for the long term security proofing and extensibility and future value of your product. You will not want to do physical updates once your devices are in the field especially when your device fleet is large. Plan for a one time initial software update and then a quarterly schedule of OTA software updates or more frequent depending on the requirements. The initial OTA software update provides the initial configuration for the device, monitoring, remote access and troubleshooting will be inexorably integrated with the OTA software process over the next 3 stages of the life cycle.