Payload converters

Introduction

When connecting external IoT platforms to Waylay via Channels or connecting to MQTT brokers, there may be a need to transform the data before entering the Waylay environment (via Waylay Broker).

A typical use case might be the following: an industrial gateway is connected to Waylay via the MQTT channel. The industrial gateway is sending JSON messages that contain multiple metrics for multiple connected hardware sensors. In Waylay you want to model each hardware sensor as an individual resource. By using a Payload Converter you can disaggregate the gateway message and send the metrics for individual resources to the Waylay Broker.

To allow this transformation with minimal configuration, Waylay provides a ‘converter service’ that uses Velocity templating. As presented in this figure, converter is an optional service that is positioned between channels and Broker:

payload_converter

Velocity templates

The converter service uses velocity templates to convert the payloads received from a channel (which are the same payloads that comes from external IoT hubs or MQTT Broker etc.) After processing the input payload, the output format of the velocity template must be as defined below:

{ 
  "metrics": [
    {
      "resource": "<resourceId1>",
      "<metricName1.1>": "<metricValue1.1>"
    },
    {
      "resource": "<resourceId2>",
      "<metricName2.1>": "<metricValue2.1>"
    }],
  "metadata": [
    {
      "resource": "<resourceId1>",
      "<metadataName1.1>": "<metadataValue1.1>"
    },
    {
      "resource": "<resourceId2>",
      "<metadataName2.1>": "<metadataValue2.1>"
    }]
}

The metrics key is the required argument and values associated with it will be stored in the timeseries database. In addition, metadata key is an optional argument which can update metadata for a given resource, should that be necessary. When no metadata is derived from the original payload, the metadata key-value pair can be excluded from the output.

Accessing original payload in template

When creating the velocity template the original payload is accessible within the template context with the $payload property name. When accessing properties within the context it’s also possible to call methods and properties of primitive values in NodeJs. For example, you can call toLocaleUpperCase() on $payload.name within the template and this will fill in the template with the output of name.toLocaleUpperCase()

Utility functions

Apart from the properties and methods available on the primitive types we also provide utility functions within the template context. These utility functions can be accessed from the property $util. Currently, there is one utility function renderValue.

renderValue

renderValue(value,type)

This function allows you to render a value correctly in the velocity template so it can then be parsed to JSON. Parameters:

  • value: The given value you wan’t to render
  • type: The type you want to render the value as. Can be ‘string’, ‘number’, ‘boolean’ and ‘auto’

When a type is given the function will render the value for the given type.

Other context

It is also possible to access some other context within the template. In some cases not all information required for payload transformation is available in the payload itself. For instance, in case of the MQTT, the device identity could be for instance derived from the subtopic name. For the MQTT pubsub connector the topic is available in the headers context by default.

Currently, one other context is available: headers. When the headers context is set, it can be accessed as $headers in the template.

Example

Consider this Payload (accessible under $payload):

    [
      {
        "ep": "10010008e",
        "tmp": 10,
        "hum": 50,
        "time": 1584690708000,
        "empty": true,
        "water": 20,
        "name": "testing"
      },
      {
        "ep": "10020009c",
        "tmp": 11,
        "hum": 45,
        "time": 1584690708000,
        "empty": false,
        "water": 10,
        "name": "testing2"
      }
    ]

This is an example of a velocity template that uses the above concepts and produces a correct output format for the given payload.

{
"metrics": [
#foreach( $entry in $payload )
{
"temperature": $util.renderValue($entry.tmp,"number"),
"movement": $util.renderValue($entry.empty,"boolean"),
"humidity": $util.renderValue($entry.hum,"number"),
"waterLevel": $util.renderValue($entry.water,"number"),
"resource": $util.renderValue($entry.ep),
"timestamp": $util.renderValue($entry.time)
}#if($foreach.hasNext),#end
#end
],
"metadata": [
#foreach( $entry in $payload )
{
"name": $util.renderValue($entry.name),
"resource": $util.renderValue($entry.ep),
},
#end
]
}

Configuring Converter service

Before the converter service can be used by an IoT platform connector the velocity template needs to be added to the service. This is done using a REST api.

Create/update a converter

curl --user apiKey:apiSecret --request PUT -H 'Content-Type: application/json' 'https://converter.waylay.io/converter/:converter' \
--data-raw '{
               "template": "<escaped velocity template as string>",
                "type": "velocity"
            }'

Test a converter

Before adding a configuration to the converter service it is possible to test and debug it. This is advised to make sure the conversions won’t fail and the output will be as expected. The following endpoint is used for this:

curl --user apiKey:apiSecret -i -X POST -H "Content-Type: multipart/form-data" \
--form 'config=@<path-to-folder>/config.json;type=application/json' \
--form 'payload=@<path-to-folder>/payload.json;type=application/json' 'https://converter.waylay.io/converter/test'

This endpoint accepts two JSON files:

  • payload.json: Contains the json payload you want to test against.
  • config.json: Contains the json body of the Create/update request.

Using the converter service with an IoT platform channel

For this example we will be using the MQTT broker IoT connector, but the converter service can also be used with: Google PubSub, Azure IoT Hub and AWS IoT Core.

First of all you should fill in the fields required for the IoT connector to work as described here. Then to configure the converter service there are two extra fields:

  • Converter
  • Forward by converter

payload_converter

The converter field allows you to select a converter that is configured on the service. If the REST API in the previous section was called correctly, the converter should show up in the dropdown.

payload_converter

The Forward by converter field allows the user to select if the converter itself forwards the payload to the broker. By default, the forwarding is done by the pubsub channel. This options should be used when high load is expected.

Billing impact

Each invocation of the converter velocity template will be counted. The ‘Stats’ tile on the Waylay admin console gives you an overview of number of invocations in the current month (so far) and the previous month. At the end of each month, the number of Converter invocations are added to your Waylay Engine API call count (like webscripts).