Introduction

Amazon Kinesis makes it possible to build custom applications that process and analyze streaming data. With Amazon Kinesis, the user can ingest real-time data such as video, audio, application logs, website clickstreams, and IoT telemetry data for machine learning, analytics, and other applications. The user can access data in real-time, and Kinesis is secure and easy to use. Users can also have multiple Kinesis applications processing the same stream, the throughput of which can scale from megabytes to terabytes per hour.

Amazon Kinesis can be used to process streaming data from IoT devices, perform real-time analytics in data science and machine learning, and securely stream video from smart-home devices.

The DeviceHive plugin management system makes it possible to integrate with custom business components. Let’s take a look at how to integrate with AWS Kinesis using DeviceHive Kinesis Plugin.

About the DeviceHive Kinesis Plugin

The DeviceHive Kinesis plugin allows the user to stream data flowed through DeviceHive to AWS Kinesis streams or the AWS Firehose delivery stream. The plugin is easy to configure and use and in addition provides users with the ability to decide where a particular type of data must be streamed. In this article we will configure, run and test the DeviceHive Kinesis plugin.

Before Running the Plugin

Before running the Kinesis Plugin, users need:

  1. DeviceHive platform (consider using Playground)
  2. Configured Kinesis and Firehose streams at AWS

In this guide we will use DeviceHive Playground, so the first step is already completed.  The only thing we need to do is proceed with the plugin launch using AWS. Here are instructions for configuration.

The plugin can write to both AWS Kinesis streams and AWS Firehose delivery streams. We will demonstrate how to write to AWS Firehose Delivery streams; configuring the plugin for Kinesis streams is just a matter of a few config properties. We will create two Firehose streams with S3 buckets as a destination: one is for commands and another for notifications.

Steps to set up Firehose streams for this demo:

  1. Create commands-bucketS3 bucket with default configuration
  2. Create notifications-bucketS3 bucket with default configuration
  3. Create plugin_demo_commandsstream with default configuration but with Direct PUTas a source and S3 commands-bucketas a destination
  4. Create plugin_demo_notificationsstream with default configuration but with Direct PUTas a source and S3 notifications-bucketas a destination

That is basically it. We are ready to configure and run the Kinesis plugin.

Plugin Initial Setup

First, we need to create the demo device. Go to Playground admin consoleand click Devices in the navigation bar, then just create the device:

Second, we need to create the plugin using the plugin management system. For this, click Pluginsin the navigation bar, then click Add new pluginbutton:

 

After successfully creating the plugin, you will see the plugin’s tokens, topic and proxy endpoint You will need this later.

 

Plugin Configuration

There are several ways to configure the plugin and we will use configuration via environment variables Please follow this link to learn more.

Generally in order to configure the plugin you need to specify at least these properties:

  • PLUGIN_TOPIC
  • PLUGIN_REFRESH_TOKEN
  • DEVICE_HIVE_PLUGIN_WS_ENDPOINT (wss://playground.devicehive.com/plugin/proxy)
  • DEVICE_HIVE_AUTH_SERVICE_API_URL (https://playground.devicehive.com/auth/rest)

Detailed plugin configuration described here.

Then create an.env file in the root of the plugin, and insert these following lines:

 

ENVSEPARATOR=_

DEBUG=kinesismodule

plugin_plugin_refresh_token=plugin.JWT.refreshToken

plugin_device_hive_plugin_ws_endpoint=wss://playground.devicehive.com/plugin/proxy

plugin_device_hive_auth_service_api_url=https://playground.devicehive.com/auth/rest

plugin_plugin_topic=your_plugin_topic

 

You can take as an example the example.env file, which is located in the Kinesis plugin repository. Note that configs related to the plugin part are prefixed with plugin_

AWS Configuration

In order to configure the AWS part, we will use the previously created .envfile. Along with AWS configs, we also need to set data destination (Kinesis stream or Direct PUT to Firehose delivery stream) and assign streams to groups (notifications, commands, command updates). Your environment variable-style should look like this:

 

kinesis_aws_region=us-east-2

kinesis_aws_accessKeyId=YOURAWSKEY

kinesis_aws_secretAccessKey=your/awssecret/key

kinesis_dataDestination=direct_put

kinesis_custom_commandStreams=plugin_demo_commands

kinesis_custom_notificationStreams=plugin_demo_notifications

kinesis_custom_commandUpdatesStreams=plugin_demo_commands

 

In kinesis_custom_commandStreams,kinesis_custom_notificationStreams, kinesis_custom_commandUpdatesStreams,the recently created stream must be used. All configs are described in detail here. Note that configs related to the AWS Kinesis part are prefixed with kinesis_.

Plugin Run

The DeviceHive Kinesis plugin is easy to run using Docker; just issue the docker-compose up command in the root of the plugin. You should see logging messages regarding the  plugin launch.

Now go to the Playground admin console, select created in the beginning device, and send the command:

After that you will see the log in terminal related to the new command insert, and after some delay (due to AWS Kinesis buffering) the command will appear in the commands-bucket at S3. Do the same with notifications and observe that they are put into the notifications-bucket.

Summary

That’s it for now. Don’t forget to view the detailed descriptions related to plugin configuration and launch: