Creating and storing derived metrics

Often customers have IoT devices that produce raw measurements (we call them metrics) of the environment (e.g. noise, dust, …) or the asset (e.g. pump flow, pump energy consumption) that they are monitoring. In Waylay, we store these metrics in a time series database to be able to analyse their behavior over time: e.g. is the metric increasing, decreasing, flat, are there any outliers etc.

In a lot of cases though, these metrics mean nothing to the business owner. They want to know if the pump is operating at its best efficiency point and if it is drifting, how long does it take before it goes out of the acceptable range (e.g. to avoid damage to the asset or to any human). These types of business metrics need to be tracked over time and are often derived from the raw measurements using standardized mathematical formulas.

In this section, we show two examples how you can create and store these derived metrics on the Waylay platform:

  • On the fly as data is streamed into the Waylay platform: this is typically used when you have 1 single resource sending the raw data as separate metrics in the resource’s messages
  • As a periodic task: this is typically used when you have multiple resources (e.g. multiple physical sensors), whose data may not come at the exact same timestamp and whose raw data you want to combine in a derived metric.

Derived metrics often require ‘clean’ raw data. If outliers or raw data values that are physically impossible (e.g. due to a software error of the sensor) are used in the formulas for the derived metrics, they deliver nonsensical results to the business owner. Such situations can have significant monetary impact:

  • Loss of customer trust in the solution when they see these nonsensical derived metrics in the asset dashboard
  • Business process flows using the nonsensical derived metrics as input may take automated decisions that are plain wrong and costly: e.g. dispatch field engineers, shut down devices etc

Therefore, the technical experts that are using Waylay to calculate these derived metrics often expand their rules with data cleansing logic too.

Calculating derived metrics from the raw metrics of a single resource

image

In the above rule we assume that we have one single resource (templated as ‘$’) that sends two different metrics. On the left hand side we see the streamBridgeData node called ‘input1’ which is receiving the messages from our resource. The raw data messages will always be forwarded as JSON objects to the scriptSensor.

In the center we have the black scriptSensor. This sensor will execute (on the condition that the node ‘input1’ has finished) the javascript code to combine two raw metrics from the raw data message into a derived metric called ‘bep’.

The scriptSensor is taking the metric values for flow and pressure out of the streamBridgeData sensor ‘input1’ using the waylayUtil library:

var a = waylayUtil.getRawData(options, "input1").streamData.flow
var b = waylayUtil.getRawData(options, "input1").streamData.pressure

var bep = //some magic formula here combining a and b

send(null, 
{
    observedState : "done", 
    rawData : { 
        BEP:bep
    }
})

In the script code we calculate the derived metric ‘bep’ using some mathematical formula combining a and b. The javascript Math library is available out of the box to create complex formulas: JS Math

Once the derived metric is calculated, we create a message that will be consumed by the next nodes in the flow: the rawData JSON object contains the value of the derived metric.

Next in line is the waylayCloudCacheStore node. This node will store the derived metric in the time series database as a new metric for our resource $. Each time this rule executes, a new value for the derived metric will be added to the time series database with the timestamp equal to the time when it was calculated. The properties of this node contain

  • Forward = false (no need to forward it back to the Waylay broker if no other rule is listening for this message)
  • Store = true (we want to store it in the time series database)
  • Payload =

    {
    "BEP" : {{bepscript.BEP}}
    }
    
    

This code will take the value of the derived metric from the scriptSensor

Calculating derived metrics from raw metrics from separate resources

image

In this rule we have multiple resources, templated as $water_sensor and $pump. These will be assigned to actual resources when we instantiate this rule as a periodic task, executing for example every minute.

On the left hand side, there are two getLastValueFromTS sensors for each resource. Each of them will pull the relevant metric (e.g. flow and pressure) from the time series storage for each resource. The reason why we use a periodic task, instead of a streaming task like in the first example, is because in practice data from multiple resources almost never arrive at the same timestamp. It is much more robust to create a periodic rule that pulls the last value as received and stored in the time series database.

In the middle of the rule we have again a scriptSensor. The logic is a bit different now to be able to deal with the case when data suddenly stopped coming in and we don’t want to repeatedly calculate and store our derived metric on stale data. Therefore we check the last timestamp when metric a and b were stored in the time series database and compare that to a timestamp ‘lt’ that we cache (in memory) in the scriptSensor. Each time the scriptSensor is executed the timestamp ‘lt’ is updated with the server time.

var a = waylayUtil.getRawData(options, "input1").lastvalue
var b = waylayUtil.getRawData(options, "input2").lastvalue

var at = waylayUtil.getRawData(options, "input1").lasttimestamp
var bt = waylayUtil.getRawData(options, "input2").lasttimestamp

var lt = waylayUtil.getCacheData(options, "timestamp") || 0

if ((at <= lt) || (bt <= lt)) {
    //we don’t want to take stale data for which we already calculated
    //and stored the derived metric. We do need to update the cached timestamp.
    lt = new Date().getTime()
    send(null, 
    {
        observedState : "error", 
        rawData : { 
            timestamp : lt
        }
    })
}
else {
    var bep = //some magic formula here combining a and b
    lt = new Date().getTime()
    send(null, 
    {
        observedState : "done", 
        rawData : { 
            BEP: bep,
            timestamp : lt
        }
    })
}

Please note that the scriptSensor and the two getLastValueFromTS sensors must have the advanced settings ‘execute on tick’ enabled. This means that these sensors will be executed each time the scheduler (re-launches) the task.

Also note that the scriptSensor has been given a sequence number of ‘2’. This is because we only want to execute the scriptSensor when the sensors for getLastValueFromTS have finished (they both have by default sequence number of ‘1’).

Once the derived metric is calculated, the scriptSensor will send a message with state ‘done’ to the next node in the rule, which is the waylayCloudCacheStore node like in the previous example.