Client

Canonical is a UK-based, privately held computer software company founded to market commercial support and related services for Ubuntu and related projects. Principally, these are free and open-source software (FOSS) or tools designed to improve collaboration between free software developers and contributors.

DataArt shares the values and the vision Canonical brings to the industry. We partnered with Canonical to showcase how existing markets can be transformed with technologies and a holistic engineering approach.

Business Challenge

There are lots of existing devices and legacy equipment that work perfectly well. Their performance meets modern standards, they have enough computing power, all designed features function excellently. The only problem is that they were not designed to meet, for example, the Software-Defined Everything / Internet of Things age requirements. These devices are not connected to the Internet and they do not support the addition of new services or features.

The main challenge of this project is to modernize this kind of equipment and to integrate it with modern technologies and infrastructures.

Meeting the Challenge

An elevator was selected as the target for the demo as a simple example of equipment that people use every day. The designed architecture allows adding the following new features to the elevator:

  • Security system based on Machine Learning algorithms;
  • Voice controlled elevator management using Amazon Alexa and DeviceHive;
  • Salesforce IoT Cloud integration;
  • App-Enablement — adding and integrating with new services from the store.

Security System / Ski-Mask Recognition

Improving security is an obvious step for any company / smart building / smart city manager. Implementing a machine learning approach allows for the automated control of the environment. The algorithms can track aggressive behavior, weapons, clothes, emotions, health conditions, etc.

DataArt selected ski-mask recognition for the demo. If a person enters an elevator or a building wearing a mask in the metropolitan area, it is obvious that this person is up to no good.

The machine learning algorithm is trained to recognize masks only if people wear them. It doesn’t react to someone just holding a mask to avoid mistakes.

Once a ski-mask is detected, the system triggers an alarm, sends text messages, and calls the security officers that are pre-registered in the system. Also, theoretically it is able to lock the doors of the elevator to capture the person wearing the mask until the police arrive.

The architecture of the solution is provided below:

The video stream from the camera is analyzed using a TensorFlow model trained to recognize ski masks. In the case of a positive recognition result, the alert actions are taken.

  • An API call to the RestComm telephony server is invoked – this triggers the pre-configured actions (like sending text messages or making calls); upon calling, it can also speak some information.
  • The siren light is invoked – to make it possible we flashed the ESP8266 chip with custom DeviceHive firmware.

Alexa Integration

DataArt developed a solution that allows turning any Linux-based device into an Amazon Echo device.

Amazon provides the API for using its voice service Alexa. Integrating custom devices with it allows bringing the full Amazon Echo functionality to those devices.

To demonstrate the solution in action, we integrated Alexa with DataArt’s open source IoT data management platform DeviceHive.

DeviceHive is an Internet of Thing platform supported commercially by DataArt. It allows connecting various devices and controlling them. We used the DeviceHive cloud solution to control a LED strip for the visual demonstration of voice commands.

The main challenge was to develop a voice activated solution that would run on a wide variety of boards. Ubuntu provides support these boards, which includes x64 and ARM boards such as the Raspberry Pi.

Each request to Alexa has to be triggered by human speech. Typically, this requirement can be solved by adding a button to activate the listening mode, but due to the wide variety of hardware we would need to support, it was impossible to use this approach. The DataArt team added a local voice recognition algorithm to detect the ‘Alexa’ keyword. Once it hears this keyword, it records the sound until the speech stops. And after that it sends this recording to the Amazon Alexa cloud service.

The DataArt team decided to develop a package with one of cross platform language from scratch based on the Amazon Alexa API for wide hardware support. We packed this application into Canonical’s snap package to provide a simple setup and transactional update.

Additional voice commands can be added to Alexa with Alexa Skills — services hosted on Amazon. This service can use the whole Amazon Web Services infrastructure. We created a simple Alexa Skill that makes REST calls to the CRM system and to the DeviceHive server.

We used the WS2812B LED Strip as a visual device, which was connected to the ESP8266 chip with our DeviceHive firmware. This firmware provides connectivity to the DeviceHive server and also supports LED strip devices and provides commands to control them. So we could totally control the LED strip just by sending different commands from the Alexa Skills application.

The Alexa integration architecture diagram is provided below.

As a part of the demo, we demonstrated a voice controlled elevator system that was able to be controlled by a human voice. The LED strip was used to demonstrate the current floor.

In addition, the system was programmed to play advertisements about the places that could be found on the selected floors. It also could answer questions like ‘Where is the best restaurant in the building?’

Salesforce IoT Cloud Integration

DataArt integrated the demo with the Salesforce IoT Cloud via its RESTful API to log usage statistics and analyze the gathered data. Once an event is triggered or a command sent, it all goes straight to the system.

All gathered data can be used to build a predictive / preventive maintenance solution.

App-Enabled Devices

All integrated services were packed into Canonical’s snap packages (https://snapcraft.io). Its architecture allows any developer to create their own branded App Store and uploading snaps there to distribute them.

Once a device is app-enabled, it becomes a part of a snap ecosystem with a huge amount of solutions available for the platform. This technology allows adding new services to equipment without changing its main features. The whole demo was designed to show how easily it can be done.

Business Benefits

The DataArt team developed a sophisticated solution that allows adding new services to an elevator as an example of updating legacy equipment. The solution is integrated with modern services and technologies, adding new value for users and customers:

  • Speech recognition that makes voice commands a part of the IoT ecosystem. The solution uses web based services that allows to scale it to much bigger systems and easily migrate from one hardware platform to another.
  • Leveraging a Machine Learning approach allows increasing security.
  • Cloud technologies enable companies to be more innovative and develop business models faster and use well known high quality products in custom solutions.
  • Salesforce IoT Cloud integration to log and analyze usage data.
  • Ability to add new features and functionality as applications.

Technology

The technology stack includes Amazon Web Services, Amazon Alexa, DeviceHive cloud service, DeviceHive firmware, Linux, Python, C, Ubuntu Snaps, Snapcraft, Java, JavaScript, TensorFlow, Restcomm.