How to extract data from, and ingest data into the Waylay system.
Waylay supports the export of time series data and metadata to Waylay Storage under Settings > Export. The export can be done on a daily, weekly or monthly basis:
For the time series data, data can be exported as:
- CSV file which has the data in the format “resource”,”metric”,”timestamp”,”value”
- AVRO file which has the data per “resource” and “metric”
Metadata is exported in newline-delimited json files: resourcetypes, resources and change-logs for resources are exported.
Once configured, data will be available under Storage, in the etl-export bucket:
If you place a data file in the
upload/ folder of Buckets > etl-import , it will be automatically loaded by Waylay in the time series database.
These csv files require a specific format, identical to the etl export files.
The etl-import tool of the Python SDK helps you to create these import files from your own input sources, upload them to the etl_import bucket, and follow up the import process.
Once an import process has been started, the import file and any result information will be moved to other subfolders (
ignored/) that reflect the processing state.