Waylay database ETL

How to extract data from, and ingest data into the Waylay system.

ETL-export service

Waylay supports the export of time series data and metadata to Waylay Storage under Settings > Export. The export can be done on a daily, weekly or monthly basis: ETL conf

For the time series data, data can be exported as:

  • CSV file which has the data in the format “resource”,”metric”,”timestamp”,”value”
  • AVRO file which has the data per “resource” and “metric”

Metadata is exported in newline-delimited json files: resourcetypes, resources and change-logs for resources are exported.

Once configured, data will be available under Storage, in the etl-export bucket: ETL data

ETL-import service

If you place a data file in the upload/ folder of Buckets > etl-import , it will be automatically loaded by Waylay in the time series database.

These csv files require a specific format, identical to the etl export files.

The etl-import tool of the Python SDK helps you to create these import files from your own input sources, upload them to the etl_import bucket, and follow up the import process.

Once an import process has been started, the import file and any result information will be moved to other subfolders (busy/, done/, failed/, ignored/) that reflect the processing state.

buckets

Waylay ETL Database Pipeline

Professional services to integrate waylay data into to your ETL pipeline