The Device Chronicle interviews Burkhard Stubert, an expert embedded consultant in Human Machine Interface development.
Burkhard Stubert is an expert embedded consultant at Embedded Use specializing in human machine interface development for industrial machinery. Burkhard has worked on some fascinating projects over the last 10 years including the development of a HMI terminal for Ropa sugar beet harvesters and Krone maize harvesters. He has also served as the chief architect and lead developer for the HMI in an infotainment system for a leading automotive manufacturer, and another in optimizing their infotainment centers. He has helped develop an ebike as they have a CAN communication network. This is the 5th project he has completed which involved connecting a HMI terminal to a CAN network in a vehicle.
How does the human machine interface work?
The human machine interface (HMI) translates between the thinking of the human and the doing of the machine. The HMI puts in the “intelligence” to simplify and automate the tasks of controlling the machine. Ideally, operators supervise machines running on autopilot – very much like pilots fly planes. They only take over control when something out of the ordinary happens.
Currently, machine operators must do this translation more often than not. Burkhard elaborates: “For example, the driver of a harvester spots that the sugar beets have head injuries, which lead to loss in sugar and hence loss of revenue. The driver pushes a button on the touch display to lift the knives and to make them cut higher. With advancing automation, the HMI automatically detects the head injury with image recognition and tells the responsible ECU to lift the knives. Drivers are relieved from routine tasks. Sugar yields are higher.”
AI in the human machine interface
The bigger manufacturers make major technology investments. For example, John Deere bought an AI company for USD $200m. The Claas combine harvesters support automatic threshing. The driver sets a cleanliness and throughput level and the harvester does the rest with image recognition. Krone forage harvesters detect the ripeness of maize from the colour. The yellower the maize is the shorter they cut it and the greener the longer.”
He continues: “Although they are often world champions in their niches, the smaller manufacturers embrace new technologies at a slower pace. They must carefully weigh when to invest and how much.”
HMI use case at Ropa Sugar Beet Harvesters
Burkhard explains that he was brought on board when Ropa wanted to learn how to build their HMI terminal themselves. They needed to take the HMI terminal development inhouse and within 2 years, Burkhard built a team to make this happen: “The driver does the translation between what he sees and what is programmed in the machine. They have not moved to complete automation yet, but they are prepared to go there eventually.”
Embedded hardware based on iMX 6
The Ropa terminal is powered by a quad-core iMX6 processor and sports a 12-inch touch display. It runs embedded Linux custom-built with Yocto. The HMI is written with QML, Qt and C++.
The HMI must cope with 1200 messages per second from 20 ECUs on the machine. With some touches on the HMI, drivers lift the shovels, lower the knives or accelerate the cleaning turbines.
The HMI has a customer database that stores accounting data like miles done, diesel consumed and acres harvested. Agencies, which rent out harvesters with drivers, need this data to invoice the farmers and pay the drivers. Farmers need this data to have an overview over their costs.
The harvester uses a telematics unit with an LTE modem to send 100 data points of accounting and location data to an IoT cloud. Farmers and agencies know exactly when the harvester’s bunker needs emptying on tractors with trailers. They can send the tractors to the right location just in time.
Software updates
The terminals of the Ropa harvesters receive an update of their root file system and bootloader during the routine service just before the harvest. The update is done with a USB drive.
Burkhard clarifies: “The Ropa harvesters didn’t have OTA updates when released in 2019. They certainly need OTA updates to fix bugs during the harvest quickly. The machines must run 24/7 during the harvest. Otherwise, farmers lose money on already thin profit margins. The bigger manufacturers have OTA update for bug fixes and security patches.”
Importance of Yocto in embedded applications
In the last 10 years, Burkhard was in contact with approximately 40 companies about embedded Linux systems. 39 companies used Yocto to build their embedded Linux systems. Only one used Buildroot. Burkhard goes on: “Despite its high learning curve and the need for expensive experts, Yocto is nearly ubiquitous. The manufacturers of construction and agricultural machines have no choice, because the terminal manufacturers choose Yocto. The terminal manufacturers have no choice, because the SoC manufacturers like NXP and TI provide a Yocto BSP.”
We wish Burkhard well with his projects in HMI and embedded development.