DeviceHive Predictive Maintenance Application with SAP

The DeviceHive project development team has explored the possibility of integrating the DeviceHive data platform with SAP Hana DB. DeviceHive allows third party IoT device developers to concentrate on business tasks and the distinctive features of a project by shifting the data and device management onto the platform. SAP Hana DB provides analytical tools integrated with data storage.

This approach can be used as a reference architecture solution based on devices that use Ubuntu Core. In our example, the devices collect data from the sensors and transmit it to the cloud for further analysis.

The DeviceHive platform allows us to collect data sent by devices in various ways. One of the most convenient ways that is available immediately after server installation is to use the data stream in an Apache Kafka server. The data flow available from Kafka topics contains device notifications. Aggregation and data analysis received from the sensors on the fly makes it possible to create a real-time monitoring system.

We discovered several possible ways to integrate DeviceHive with SAP:

1. Use SAP HANA Smart Data Streaming and implement a custom data adaptor reading from Kafka;
2. Use SPARK as a bridge between Kafka and SAP Hana;
3. Write a custom application for the SAP Cloud platform, which reads directly from the remote Kafka queue and stores data in the local SAP Hana DB.

SAP SDS

In the case of SAP Hana, the solution can use one of several available data-ingestion mechanisms. The SDS package is native to the SAP platform and can be used to develop its own Data Adapter. This solution makes it possible to create a comprehensive real-time analytics system using the rich set of features provided by the Smart Data Streaming solution.

This approach seemed to be the best since it requires only one integration point (class) in this case. The adaptor implemented will be configurable and reusable and the processing of thresholds will be performed inside Smart Data Streaming.

However, this approach has its negative aspects as well: a license is required to use Smart Data Streaming. If the user already have a license, then this approach is definitely the best one.

Spark Streaming

An alternative solution would be to use Spark Streaming, which in turn is similar to SDS in terms of functionality. Spark Streaming makes it possible to use general-purpose programming languages, as well as built-in libraries. Since built-in libraries can be uses, a Spark-based solution allows complex analytical computations and integration of the data flow with other systems.

Architecturally, HANA’s Smart Data Access Component connects to Spark SQL through the ODBC driver and issues SQL queries for retrieving data. The resulting data set is treated as a remote table in HANA, and it can be used as the input for all advanced analytics functionalities in HANA, which includes modeling views, SQL Script procedures, predictive text analytics, and so on.

Custom Application

This is the simplest approach that does not require any additional software access/changes in the SAP cloud platform. In this case, a Kafka connector will be a part of a web application that visualizes device notifications and alert data.

In our work we use the simplest approach as an example of integration, leaving further research and development a possibility by highlighting alternative connectivity options.

Implementation

For our R&D we chose the custom application approach since it’s based on open source solutions and gives more integration freedom. We didn’t have any specific requirements limiting us, so in this case this option appeared to be more universal and faster. However, we encourage our users to consider the other options as well, according to the project requirements.

@Entity
@Table(name = "Notification")
@NamedQueries({
      @NamedQuery(name = "AllNotifications", query = "select n from Notification n"),
      @NamedQuery(name = "LatestNotifications",
              query = "select n from Notification n " +
                      " where n.timestamp > :fromTime " +
                      "   and n.deviceGuid = :deviceId " +
                      " order by n.timestamp")
})
public class Notification {
  @Id
  @GeneratedValue
  private Long id;

  @Basic
  private String deviceGuid;
  @Basic
  private String notification;
  @Basic
  private Timestamp timestamp;
  @Basic
  private String mac;
  @Basic
  private String uuid;
  @Basic
  private Double value;
  ...
}

@Entity
@Table(name = "Alert")
@NamedQueries({
      @NamedQuery(name = "AllAlerts", query = "select a from Alert a"),
      @NamedQuery(name = "LatestAlerts",
              query = "select a from Alert a " +
                      " where a.timestamp > :fromTime " +
                      "   and a.deviceGuid = :deviceId " +
                      " order by a.timestamp")
})
public class Alert {
  @Id
  @GeneratedValue
  private Long id;

  @Basic
  private String deviceGuid;

  @Basic
  private Timestamp timestamp;
  ...
}

public class LocalEntityManagerFactory {
  private static EntityManagerFactory emf;

  public static void init() {
      if (emf == null) {
          try {
              InitialContext ctx = new InitialContext();
              DataSource ds = (DataSource) ctx.lookup("java:comp/env/jdbc/DefaultDB");

              Map properties = new HashMap();
              properties.put(PersistenceUnitProperties.NON_JTA_DATASOURCE, ds);
              emf = Persistence.createEntityManagerFactory("kafka-consumer", properties);
              System.out.println("EMF created");
          } catch (NamingException e) {
              e.printStackTrace();
          }
      }
  }

public class ConsumerGroup {

  ...

  public ConsumerGroup(String zookeeper, String groupId, String notificationTopic,
                       String alertTopic) {
      notificationConsumer = Consumer.createJavaConsumerConnector(
              createConsumerConfig(zookeeper, groupId));
      alertConsumer = Consumer.createJavaConsumerConnector(
              createConsumerConfig(zookeeper, groupId));
      this.notificationTopic = notificationTopic;
      this.alertTopic = alertTopic;
  }

  ...

  private static ConsumerConfig createConsumerConfig(String zookeeper, String groupId) {
      Properties props = new Properties();
      props.put("zookeeper.connect", zookeeper);
      props.put("group.id", groupId);
      props.put("key.deserializer", StringDeserializer.class.getName());
      props.put("value.deserializer", StringDeserializer.class.getName());
      return new ConsumerConfig(props);
  }
}