NAV
cURL JavaScript

Waylay Broker

Waylay Broker is at this URL: https://data.waylay.io

For testing you can use labs.

Authentication

When connecting to the broker, you need to provide your credentials (apiKey, apiSecret) by http basic authentication or where not possible in the url (websockets)

For more sensitive environments we also have a device gateway where you can get per-device credentials. These credentials only allow sending or receiving data on the device’s channel.

Resources

All data handled by the broker is linked to a resource. A resource is a generic identifier and can represent devices, persons, cars, customers, anything…

Events and commands

For each resource we provide two channels:

Posting messages towards Broker

The waylay data endpoint lets you store and distribute messages. This can be performed over different protocols: http or websockets.

As soon as data is send to the Broker, data is stored in two different databases, time series database and document database. In the document storage, data is stored without any pre-processing, with original JSON object as it was received. When JSON object (or array of JSON objects) comes to the Broker, Broker also tries to save data in the time series database. In order to achieve that, broker will inspect incoming JSON object and store every metric that is found in the JSON object.

To keep your data private you use your waylay api key and secret. This will also enable the forwarding of your data to your tasks or buckets.

A submitted message is defined by 2 things

For instance, the resource can be the phone and the parameters something like temperature, humidity, acceleration etc.

Important Note : Data will be stored with the timestamp when the object arrived at the Broker. Should you wish to insert data with other timestamp, you must in the JSON object provide a timestamp with a value that is in epoch in milliseconds. For instance: {"temp":21, "humidity": 0.35, "timestamp" : 1475139600000}

HTTP REST API

The REST API is mainly intended for devices that have an HTTP stack available and don’t need to send huge amounts of data. It also allows you to fetch the current and last 100 items from the store.

First you will need to fetch your API key from the profile page.

Available urls

Available urls

GET           /resources/:resourceId/events              // last n / sse / ws publish
GET           /resources/:resourceId/events/subscribe    // ws
GET           /resources/:resourceId/events/publish      // ws
POST          /resources/:resourceId/events

GET           /resources/:resourceId/commands            // last n / sse / ws subscribe
GET           /resources/:resourceId/commands/subscribe  // ws
GET           /resources/:resourceId/commands/publish    // ws
POST          /resources/:resourceId/commands

Posting data (object) to the storage and rule engine

When you post a message, by default, the message is both forwarded to the rule engine and stored in the different databases (document store, time series database).

Example of posting data using resource events endoint:

curl -i --user apiKey:apiSecret  -H "Content-Type: application/json" -X POST \
    -d '{
          "foo":123,
          "bar":"hello"
      }' https://data.waylay.io/resources/testresource/events

Example of posting data with resource defined in the payload:

curl -i  --user apiKey:apiSecret -H "Content-Type: application/json" -X POST \
    -d '{
        "foo":123,
        "bar":"hello",
        "resource":"testresource"
        }' https://data.waylay.io/messages

There are multiple ways to post the same message towards the Broker.

Posting array of objects

You can post objects as an array.

BULK import

curl -i  --user apiKey:apiSecret -H "Content-Type: application/json" -X POST \
    -d '[
      {   
          "foo": 12,
          "bar":"hello",
          "resource":"testresource1"
      },  {   
          "foo": 33,
          "bar":"world",
          "resource":"testresource2"
      }]' https://data.waylay.io/messages

If the array contains only 1 object, or all objects in the array are for different resources, the data is stored in the timeseries database, message cache and forwarded to the engine.

If there are multiple messages for the same resource in the array, these are only stored in the timeseries database (and NOT stored in message cache, nor are they forwarded to the engine)

Forwarding data to the rule engine, without the storage

Example of posting data to the engine only, using resource events endpoint:

curl -i --user apiKey:apiSecret  -H "Content-Type: application/json" -X POST \
    -d '{
          "foo":123,
          "bar":"hello"
      }' https://data.waylay.io/resources/testresource/events?store=false

Forwarding data to the storage, without forward to the engine

Example of posting data to the engine only, using resource events endoint:

curl -i --user apiKey:apiSecret  -H "Content-Type: application/json" -X POST \
    -d '{
          "foo":123,
          "bar":"hello"
      }' https://data.waylay.io/resources/testresource/events?forward=false

Bulk forwarding CSV data to the storage, without forward to the engine

We provide a handy upload script to upload CSV data to the time series database. Make sure you have Python3 and Pandas installed. See the README file.

https://github.com/waylayio/service-scripts

Posting data with a specified time-to-live

Example of posting data which will only be available for 3600s:

curl -i --user apiKey:apiSecret  -H "Content-Type: application/json" -X POST \
    -d '{
          "foo":123,
          "bar":"hello"
      }' https://data.waylay.io/resources/testresource/events?ttl=3600

When posting data, you can specify how long the data needs to be available in the system (both in the timeseries database and the document database). After this time the data is automatically removed.

You specify this by the query parameter ttl, either specified as a number of seconds, or with a period in the format #[w,d,h,m,s] ( as in # weeks/days/hours/minutes/seconds )

Example of posting data which will be available for 52 weeks:

curl -i --user apiKey:apiSecret  -H "Content-Type: application/json" -X POST \
    -d '{
          "foo":123,
          "bar":"hello"
      }' https://data.waylay.io/resources/testresource/events?ttl=52w

Websockets

You can setup a websocket for posting data in a streaming fashion (multiple resources)

Endpoint to post data in a streaming fashion

wss://data.waylay.io/socket

Messages posted to this endpoint must contain a resource key to identify the thing the data is coming from and optionally can have a timestamp message with the time(stamp) (in epoch milliseconds) of the data (if timestamp is not present the data will be stored with the timestamp it arrived at the broker)

Per resource

You can also set up web sockets for a specific resource

Endpoints to connect to using secure web sockets:

wss://data.waylay.io/resources/:resourceId/commands/subscribe (listening to commands)
wss://data.waylay.io/resources/:resourceId/commands/publish (publishing commands to an other device)
wss://data.waylay.io/resources/:resourceId/events/subscribe (listening to events from a device)
wss://data.waylay.io/resources/:resourceId/events/publish (publishing events)

The labs lets you play with websocket support. Remark that the subscribe connections are kept alive by empty {} json messages if no traffic has passed in 1 minute.

When submitting invalid data you will get a response back with an error message.

{
  "error": "Json parse error",
  "details": [
    "/resource <- String value expected"
  ]
}

Message retrieval

Getting latest message from storage

curl -i --user apiKey:apiSecret \
    'https://data.waylay.io/resources/testresource/current'

Getting last X messages from storage

curl -i --user apiKey:apiSecret \
    'https://data.waylay.io/resources/testresource/messages'

Below call has been deprecated. Please use the above call to retrieve the last X messages

curl -i --user apiKey:apiSecret \
    'https://data.waylay.io/resources/testresource/series'

You can always retrieve up to the last 100 messages for every resource over the REST calls. Coming back to the example where resource is the phone, waylay platform would store 100 messages, where each message would hold information about underlying parameters (temperature, humidity etc.)

Querying messages for multiple resources

You can retrieve messages for multiple resources from storage in one REST api call. The messages will always be grouped per resource. They will be sorted by descending timestamp.

The resources parameter in a POST request body is a list of resource id’s .
The limit parameter - number of messages per resource to be return. If that parameter is set to 1 then it means: return the latest message per requested resource.

Optionally you can query for messages in a time range by providing from,until and window parameters.

The from and until parameters can be supplied as either numerical timestamps (i.e. number of milliseconds since the epoch) or as ISO-8601 date times.

The window parameter is a duration that will be used in conjuction with from or until parameters. Duration can be expressed by specifying one of the following formats:

window can also be specified with an expression consisting of a value and a time unit. For example, 42m to represent 42 minutes. The supported time units are

Finally, window can also be expressed in following format based on (a subset of) ISO-8601 duration: PnDTnHnMn.nS where D stands for days (considered to be exactly 24hours), H for hours, M minutes and S seconds and n must be an integer. For example PT42M represents 42 minutes, P2DT3H4M 2 days, 3 hours and 4 minutes

To define the period for the request, you either specify

Getting latest message for multiple resources

curl -i --user apiKey:apiSecret -H "Content-Type: application/json" \
     -X POST \
     -d '{
          "resources": [ "testresource1","testresource2","testresource3" ],
          "limit": 1
         }
     '
    'https://data.waylay.io/messages/query'

HTTP/1.1 200 OK

{
    "query": {
        "resources": ["testresource1","testresource2","testresource3"],
        "limit": 1
    },
    "results": [
        {
            "resource": "testresource2",
            "messages": [
                {
                    "sensor2": "xx",
                    "timestamp": 1589543662212
                }
            ]
        },
        {
            "resource": "testresource3",
            "messages": [
                {
                    "sensor3": "ff",
                    "timestamp": 1589792658905
                }
            ]
        }
    ]
}

Getting messages for a time period

curl -i --user apiKey:apiSecret -H "Content-Type: application/json" \
     -X POST \
     -d '{
          "resources": [ "04:00:00:23:f2:10","thomas@waylay.io@netatmosimulator.00001"],
          "from":1572271520198, 
          "until":1590745888140,
          "limit": 5
         }
     '
    'https://data.waylay.io/messages/query'

HTTP/1.1 200 OK

{
    "query": {
        "resources": [
            "04:00:00:23:f2:10",
            "thomas@waylay.io@netatmosimulator.00001"
        ],
        "from": 1572271520198,
        "until": 1590745888140,
        "limit": 5
    },
    "results": [
        {
            "resource": "04:00:00:23:f2:10",
            "messages": [
                {
                    "temperature": 31.611053382667503,
                    "timestamp": 1572271523159
                },
                {
                    "temperature": 22.358401393033727,
                    "timestamp": 1572271522201
                },
                {
                    "temperature": 30.236288718749655,
                    "timestamp": 1572271521171
                },
                {
                    "temperature": 41.30275341539359,
                    "timestamp": 1572271520199
                }
            ]
        }
    ]
}

Streaming all messages using NDJSON

This is a firehose stream of all messages that are sent to the broker. If you can not keep up with reading the stream messages will be dropped! The connection is kept alive by sending empty {} json messages if no traffic has passed in 1 minute.

curl -i --user apiKey:apiSecret https://data.waylay.io/messages

Streaming using WebSockets

Streaming using WebSockets

wss://data.waylay.io/resources/testresource/socket?apiKey=...&apiSecret=...

You can stream data for a specific resource by setting up a WebSocket to the following url. Remark that the connection is kept alive by empty {} json messages if no traffic has passed in 1 minute.

Time series data retrieval

Each of the metric values of a message is added to a timeseries for the metric. You can retrieve these timeseries with, or without grouping them in time. When you retrieve them, you need to specify which period (from, until) you want to retrieve, if and how you want the datapoints to grouped and if you want to group them, how the datapoints get aggregated.

Standard timeseries metric endpoint returns a maximum of 2000 data points. There is a special timeseries endpoint that can also return an indication if there is still data available for requested time range. It makes use of HAL (Hypertext Application Language) specification. The HTTP request in that case should provide “Accepts: application/hal+json” HTTP header. In that case if still some data exists after query execution then response will contain a HAL link to API call that should return next chunk of data.

Grouping

You can aggregate data by specifying one of the following groupings:

Grouping can also be specified with a grouping expression consisting of a value and a time unit. For example, 42m to represent 42 minutes. The supported time units are

Finally, grouping can also be expressed in following format based on (a subset of) ISO-8601 duration: PnDTnHnMn.nS where D stands for days (considered to be exactly 24hours), H for hours, M minutes and S seconds and n must be an integer. For example PT42M represents 42 minutes, P2DT3H4M 2 days, 3 hours and 4 minutes

Group alignment

Groups are aligned with the from timestamp.

For example, if a from date of 2018-11-01T15:30:00Z is specified along with a grouping of 10s (10 seconds), then the first sample point returned will be labelled with timestamp 2018-11-01T15:30:00Z and include aggregated data from 2018-11-01T15:30:00Z (inclusive) to 2018-11-01T15:30:10Z (exclusive).

The second sample point will be labelled with timestamp 2018-11-01T15:30:10Z and include aggregated data from 2018-11-01T15:30:10Z (inclusive) to 2018-11-01T15:30:20Z (exclusive), and so on.

All sample points from the from timestamp to the until timestamp will be included in the time series aggregation response. If there is no data for a given sample point, the sample point will be populated with a null value.

Aggregates

You can specify how grouped datapoints get aggregated with the following aggregation operations:

Getting time series data

Getting raw time series data (no aggregation)

curl -i --user apiKey:apiSecret
    'https://data.waylay.io/resources/testresource/series/temperature?from=1472947200000&until=1474588800000'

Getting raw time series data with HAL links (no aggregation)

curl -i --user apiKey:apiSecret -H 'Accept: application/hal+json'
    'https://data.waylay.io/resources/04:00:00:23:f2:10/series/temperature/raw?limit=5&from=1572271017423&until=1572271033621&order=descending'

{
    "_links": {
        "next": {
            "href": "/resources/04:00:00:23:f2:10/series/temperature/raw?limit=5&from=1572271017423&until=1572271027623&order=descending",
            "name": "Next series"
        }
    },
    "query": {
        "from": 1572271017423,
        "until": 1572271033621,
        "metric": "temperature",
        "aggregates": null,
        "grouping": "auto",
        "resources": [
            "04:00:00:23:f2:10"
        ],
        "limit": 5
    },
    "series": [
        [
            1572271032632,
            44.18519429826034
        ],
        [
            1572271030864,
            22.224902273954008
        ],
        [
            1572271029628,
            44.42197409553575
        ],
        [
            1572271028630,
            39.715333233763936
        ],
        [
            1572271027624,
            7.467454341199085
        ]
    ]
}

Getting time series data grouped per hour with mean aggregation

curl -i --user apiKey:apiSecret
    'https://data.waylay.io/resources/testresource/series/temperature?from=1472947200000&until=1474588800000&grouping=hour&aggregate=mean'

Multiple aggregates example

curl -i \
     --user apiKey:apiSecret \
     'https://data.waylay.io/resources/testresource/series/temperature?from=1472947200000&until=1474588800000&grouping=hour&aggregates=mean,min,max'

You can specify from and until timestamps. Both are supplied as epoch time in milliseconds (i.e. the number of milliseconds since 1970-01-01 00:00:00.000 GMT) or as ISO-8601 date times.

If until is not specified, the current time will be used. If from is not specified, it will be set to seven days before the until value.

Both from and until are inclusive, meaning that data points which fall on these exact timestamps will be included in the response.

You can specify how datapoints get grouped by specifying the grouping parameter using one of the formats specified above

When you specify a grouping, you also need to specify how all datapoints in a group are aggregated. You do that by setting the aggregate parameter.

Multiple aggregates can be specified as a comma-separated list in the aggregates parameter and the returned response will include a value at each sample point for the requested aggregation.

POST timeseries query

Aggregate via POST over multiple resources by example:

curl -i --user apiKey:apiSecret -H "Content-Type: application/json" \
     -X POST \
     -d '{
        "from": "2018-11-01T15:30:00Z",
        "window": "PT8H",
        "filter": {
          "operator": "between",
          "lowerBound" : 10.0,
          "upperBound" : 37.3
        },        
        "resources": ["testresource-1", "testresource-2"],                                          
        "aggregates": ["min", "max", "mean"],
        "metric": "temperature",
        "grouping": "PT10M"
     }' https://data.waylay.io/series/query

Timeseries data can also be requested by posting a timeseries query.

In such request, the from and until parameters can be supplied as either numerical timestamps (i.e. number of milliseconds since the epoch) or as ISO-8601 date times.

To define the period for the request, you either specify

The window parameter is the same format as the grouping

The resource ids and aggregation operations to be used in the query are supplied as JSON arrays. The resources and aggregates fields in the body must each contain at least one element.

You can also optionally specify a filter, which is an object with following fields:

The filter will be applied to the requested data before the data is aggregated.

Get timestamps where device was OFF during last 7 days aggregated per 15 minutes

curl -i --user apiKey:apiSecret -H "Content-Type: application/json" \     
     -X POST \                                                            
     -d '{                                                                
        "from": "2018-11-01T15:30:00Z",                                   
        "window": "P7D",                                                 
        "filter": {                                                       
          "operator": "eq",                                          
          "value": "OFF"                                             
        },                                                                
        "resources": ["testresource-1"],                
        "aggregates": ["first"],                             
        "metric": "status",                                          
        "grouping": "PT15M"                                               
     }' https://data.waylay.io/series/query                               

The POST body can be constructed with multiple timeseries queries in JSON array. That allows querying of multiple metrics in one HTTP call. For example:

curl -i --user apiKey:apiSecret -H "Content-Type: application/json" \     
     -X POST \                                                            
     -d '[{                                                                
        "from": "2018-11-01T15:30:00Z",                                   
        "window": "P7D",                                                 
        "filter": {                                                       
          "operator": "eq",                                          
          "value": "OFF"                                             
        },                                                                
        "resources": ["testresource-1"],                
        "aggregates": ["first"],                             
        "metric": "status",                                          
        "grouping": "PT15M"                                               
     },{                                                                
               "from": "2018-12-01T15:30:00Z",                                   
               "window": "P3D",                                                 
               "filter": {                                                       
                 "operator": "eq",                                          
                 "value": "OFF"                                             
               },                                                                
               "resources": ["testresource-1"],                
               "aggregates": ["first"],                             
               "metric": "temperature",                                          
               "grouping": "PT15M"                                               
            }
]' https://data.waylay.io/series/query    

Latest value for a series

Getting the latest value for a series

curl -i --user apiKey:apiSecret
    'https://data.waylay.io/resources/testresource/series/temperature/latest'

Metadata

Retrieving list of existing timeseries on resource testresource

curl -i --user apiKey:apiSecret \
   https://data.waylay.io/resources/testresource/series?metadata

You can request the list of timeseries that exist for a certain resource by asking for the series metadata. The response will be an array of objects with following data:

Deleting data

Messages

Removing all messages

curl -i --user apiKey:apiSecret -X DELETE \
    https://data.waylay.io/resources/testresource/messages

You can remove all latest messages for a resource. This will not delete timeseries data for the properties of those messages.

All data for a resource

Removing all data

curl -i --user apiKey:apiSecret -X DELETE \
     https://data.waylay.io/resources/testresource

Removing all data before some date

curl -i --user apiKey:apiSecret -X DELETE \
     https://data.waylay.io/resources/testresource?until=1501538400000

You can delete all data (both messages and timeseries data) for a resource.

Specifying the query parameter until will only delete the data until (and including) the provided timestamp (milliseconds since epoch). The parameter also allows to specify a relative period in the format #[w,d,h,m,s,ms]. This period will be substracted from the current time to become the until timestamp

You can also use query parameter from. If that parameter provided then only data starting from that time (inclusive) will be deleted. The format of that parameter is the same as for the until parameter. With combining both from and until parameters you can specify time range for data which will be deleted.

If you want to delete only a timeseries data for a given resource you can add onlytimeseries parameter with value true. If that parameter is not provided it’s value will be defaulted to false, meaning that both timeseries and messages will be deleted.

Remove all data older then 7 days

curl -i --user apiKey:apiSecret -X DELETE \
     https://data.waylay.io/resources/testresource?until=7d

Remove only timeseries data all data which is between 7 and 10 days old

curl -i --user apiKey:apiSecret -X DELETE \
     https://data.waylay.io/resources/testresource?from=10d&until=7d&onlytimeseries=true