NAV
cURL JavaScript

Storage API

This page provides a detailed description of the REST API of the waylay storage service.

See Waylay Storage Feature for a general introduction into this service, and how it is exposed on the Waylay Console.

This API is supported in the Waylay Python SDK.

Concepts

bucket

A bucket is a container that manages access and notification policies for a collection of storage objects, private to each tenant.

Currently Waylay preconfigures the following buckets for each tenant (the storage feature description has more details):

storage object

A storage object represents a file that has been uploaded to bucket. Within the bucket, the object has a path name that can be split into segments (with /) to simulate the folders of a file system.

The storage API only indirectly lets you upload or download the actual content of a file.

When requested, it will create a signed upload or download link to the content in the actual backend store. This design

subscription

A notification subscription is a rule, associated with a bucket, that defines:

If a change or deletion in a bucket matches any event filter in any of its subscriptions, the corresponding webscript will be invoked, with a payload message that contains a reference to the changed file.

Notification subscriptions are managed by the subscription API.

API Overview

storage API

Manages buckets and storage objects.

endpoint description
GET /bucket
GET /bucket/
Lists buckets
GET /bucket/{name} Retrieves the representation for the bucket with the given name or alias
GET /bucket/{name}/{path} Lists objects that have the given path prefix
GET /bucket/{name}/{path}?stat=true Retrieves a representation for the storage object at this path
GET /bucket/{name}/{path}?sign={method} Creates a signed url for accessing the object store at this path with given http method.
DELETE /bucket/{name}/{path} Deletes the storage object at this path
PUT /bucket/{name}/{path}/ Creates a (virtual) folder at path

subscription API

User operations on subscriptions.

endpoint description
GET /subscription Queries all subscription for this tenant.
GET ​/subscription​/{bucket_name} Queries all subscriptions on this bucket
POST ​/subscription​/{bucket_name} Creates a new subscription
DELETE ​/subscription​/{bucket_name} Deletes all subscriptions for this bucket
GET ​/subscription​/{bucket_name}​/{id} Retrieves a subscription
PUT ​/subscription​/{bucket_name}​/{id} Creates or replace a subscription
DELETE ​/subscription​/{bucket_name}​/{id} Deletes a subscription

status API

Version and status information.

endpoint description
GET / Returns the current software version of the server.

S3 object store

To actually access or modify the content of storage objects, signed URLs must be used, which have a limited validity in time. These URLs point to the backend store, using S3 authentication concepts.

When a public policy is active (e.g. on the public bucket), unsigned urls are made available that do not expire (see get or put HAL link in a get object result).

endpoint description
GET {presigned GET url} Retrieves the storage object content
PUT {presigned PUT url} Uploads content to a new or existing storage object
POST {presigned POST url} uploads content to a new or existing storage object, using a multipart/form-data request

General aspects

Endpoints

On the production SaaS platform, all endpoints documented above are prefixed with

https://storage.waylay.io

Authentication

API calls are authenticated with the regular Waylay credentials.

Waylay Bearer Token

curl -H "Authentication: Bearer $token" https://storage.waylay.io/bucket

API Keys

curl -u $apiKey:$apiSecret https://storage.waylay.io/bucket

Errors

Error conditions will normally lead to a json response with an error key.

HTTP status codes in the 4XX range are part of the normal API specification and should be treated by the caller as such.

Clients should only act on the HTTP status code that is associated with the response. The error message is a technical message that need not be relevant for an end user.

Example

> curl -u $apiKey:$apiSecret 'https://storage.waylay.io/bucket/public/no_such_object.txt?stat=true'
{
  "error": "The specified key does not exist."
}

In this case a 404 Not Found error code is returned

Representations

The waylay storage REST API uses JSON representations for all endpoints (except the version page). Most returned representations contain HAL links to related endpoints.

These HAL links will reflect the authorisation level of the user. As a general rule, an application should only traverse links on behalf of a user, if they were present as a HAL link in a previous response.

The object storage endpoints require you to state the Content-Type when uploading content. Any content type is allowed, as long as a signed URL does not restrict it.

storage API

list bucket

GET /bucket/
GET /bucket/?store={store}

Lists the buckets that are visible by the current user.

parameters

example

curl -u $apiKey:$apiSecret https://storage.waylay.io/bucket?store=gcp
{
  "_links": {
    "self": {
      "href": "/bucket?store=gcp"
    }
  },
  "buckets": [
    {
      "_links": {
        "self": {
          "href": "/bucket/assets?store=gcp"
        },
        "list": {
          "href": "/bucket/assets/?store=gcp"
        },
        "sign_post": {
          "href": "/bucket/assets/?sign=POST&expiry_seconds=300&store=gcp"
        },
        "subscriptions":{
          "href": "/subscription/assets" 
        },
        "store": {
          "href": "gs://ws-assets-6ccc8843-d78d-49e8-84c4-3734a4af9929"
        }
      },
      "alias": "assets",
      "name": "ws-assets-6ccc8843-d78d-49e8-84c4-3734a4af9929",
      "store": {
        "type": "minio_gs",
        "name": "gcp",
        "url": "https://object-storage.waylay.io"
      }
    },
  ...
  ]
}

get bucket

GET /bucket/{name}
GET /bucket/{name}?store={store}

Fetches the details of a given bucket.

parameters

result

The response contains

The HAL links include a self link and

example

curl -u $apiKey:$apiSecret https://storage.waylay.io/bucket/etl-export
{
  "_links": {
    "self": {
      "href": "/bucket/etl-export"
    },
    "list": {
      "href": "/bucket/etl-export/"
    },
    "sign_post": {
      "href": "/bucket/etl-export?expiry_seconds=300&sign=POST"
    },
    "subscriptions": {
      "href": "/subscription/etl-export" 
    },
    "store": {
      "href": "gs://ws-etl-export-6ccc8843-d78d-49e8-84c4-3734a4af9929"
    }
  },
  "alias": "etl-export",
  "name": "ws-etl-export-6ccc8843-d78d-49e8-84c4-3734a4af9929",
  "store": "gcp"
}

list object

GET /bucket/{name}/{path}?recursive={recursive}

Lists all objects in the given bucket with given path prefix.

Note that this always returns a listing, even when the path matches exactly the name of an object or folder, or matches nothing.

To list the content of a (virtual) folder, use a path that includes a final /.

parameters

result

The response contains an object containing HAL Links and a single objects array containing the listed objects.

example

curl -u $apiKey:$apiSecret https://storage.waylay.io/bucket/public/?max_keys=2
{
  "_links": {
    "self": {
      "href": "/bucket/public/thomas_path/?max_keys=2"
    },
    "sign_post": {
      "href": "/bucket/public/thomas_path/?sign=POST&expiry_seconds=300"
    },
    "next": {
      "href": "/bucket/public/thomas_path/?max_keys=2&start_after=thomas_path%2Fiu.jpeg"
    }
  },
  "objects": [
    {
      "_links": {
        "self": {
          "href": "/bucket/public/thomas_path/bottles.png?stat=True"
        },
        "sign_get": {
          "href": "/bucket/public/thomas_path/bottles.png?sign=GET&expiry_seconds=300"
        },
        "sign_put": {
          "href": "/bucket/public/thomas_path/bottles.png?sign=PUT&expiry_seconds=300"
        },
        "get": {
          "href": "https://object-storage-staging.waylay.io/ws-public-6ccc8843-d78d-49e8-84c4-3734a4af9929/thomas_path/bottles.png"
        },
        "remove": {
          "href": "/bucket/public/thomas_path/bottles.png",
          "method": "DELETE"
        }
      },
      "bucket": {
        "_links": {
          "self": {
            "href": "/bucket/public"
          }
        },
        "alias": "public",
        "name": "ws-public-6ccc8843-d78d-49e8-84c4-3734a4af9929"
      },
      "name": "thomas_path/bottles.png",
      "last_modified": "2020-11-26T18:11:58.059000+00:00",
      "etag": "809689ed39bb1b43d19753f1e15c84c3",
      "size": 696592,
      "is_dir": false
    },
    ...
  ]
}

get object

GET /bucket/{name}/{path}?stat=true

Returns a detailed representation of a storage object or virtual folder (for paths ending in /).

parameters

result

An object representation containing the storage object metadata. The HAL links include a self link and (according to authorization levels):

Virtual folders are only listed in non-recursive requests, and have a list HAL link to list the contained objects.

example

curl -u $apiKey:$apiSecret 'https://storage.waylay.io/bucket/public/img/mouse.jpg?stat=true'
{
  "_links": {
    "self": {
      "href": "/bucket/public/img/mouse.jpg?stat=True"
    },
    "sign_get": {
      "href": "/bucket/public/img/mouse.jpg?sign=GET&expiry_seconds=300"
    },
    "sign_put": {
      "href": "/bucket/public/img/mouse.jpg?sign=PUT&expiry_seconds=300"
    },
    "get": {
      "href": "https://object-storage-staging.waylay.io/ws-public-6ccc8843-d78d-49e8-84c4-3734a4af9929/img/mouse.jpg"
    },
    "remove": {
      "href": "/bucket/public/img/mouse.jpg",
      "method": "DELETE"
    }
  },
  "bucket": {
    "_links": {
      "self": {
        "href": "/bucket/public"
      }
    },
    "alias": "public",
    "name": "ws-public-6ccc8843-d78d-49e8-84c4-3734a4af9929"
  },
  "name": "img/mouse.jpg",
  "last_modified": "2020-12-31T09:56:02",
  "etag": "8b8a4428e8dc1ffd7e9ef73d6896a2bc",
  "size": 91651,
  "content_type": "image/jpeg",
  "is_dir": false
}

sign url

GET /bucket/{name}/{path}?sign=GET
GET /bucket/{name}/{path}?sign=PUT
GET /bucket/{name}/{path}?sign=POST

Creates a signed URL to the object storage.

parameters

parameter GET PUT POST value description
expiry_seconds
expiry_hours
expiry_days
integer interval in which the link will expire.
All specified amounts are added together, up to a maximum amount of 7 days.
get_as_attachment boolean Manages the content-disposition response header implied by the signed url.
If set, downloads will be treated as attachments that are saved to a file.
If not, the browser (or other http agent) decides how to render the content.
content_type string Sets the required content type of the upload.
content_length_min integer Sets the minimum number of bytes uploaded (default 0)
content_length_max integer Sets the maximal number of bytes uploaded

result

The response contains a hal link containing

examples

curl -u $apiKey:$apiSecret 'https://staging.waylay.io/bucket/public/img/mouse.jpg?sign=GET'
{
  "_links": {
    "get_object": {
      "href": "https://object-storage.waylay.io/ws-public-6ccc8843-d78d-49e8-84c4-3734a4af9929/img/mouse.jpg?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential= ... 36183cbd2e462273e691c8110"
    }
  }
}
curl -u $apiKey:$apiSecret 'https://staging.waylay.io/bucket/public/img/mouse.jpg?sign=PUT' 
{
  "_links": {
    "put_object": {
      "href": "https://object-storage.waylay.io/ws-public-6ccc8843-d78d-49e8-84c4-3734a4af9929/img/mouse.jpg?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential= ... 0789ab66d",
      "method": "PUT"
    }
  }
}
curl -u $apiKey:$apiSecret 'https://staging.waylay.io/bucket/public/img/?sign=POST' 
{
  "_links": {
    "post_object": {
      "href": "https://object-storage.waylay.io/ws-public-6ccc8843-d78d-49e8-84c4-3734a4af9929/",
      "method": "POST",
      "form_data": {
        "key": "img/",
        "bucket": "ws-public-6ccc8843-d78d-49e8-84c4-3734a4af9929",
        "policy": "eyJleHB ... 3QiXV19",
        "x-amz-algorithm": "AWS4-HMAC-SHA256",
        "x-amz-credential": "6ccc8843-d78d-49e ... ws4_request",
        "x-amz-date": "20201104T135215Z",
        "x-amz-signature": "e257465 ... 8e36cbf4"
      }
    }
  }
}

put folder

PUT /bucket/{name}/{path}/

Creates or asserts the existence of a (virtual) folder at the given location. Virtual folders are created by writing an empty, hidden object at /{path}/.folder. These hidden objects will only be visible when using the all parameter to list objects, e.g. as in GET /bucket/{name}/{path}/?all=true. Without this hidden object, only paths that contain objects will be listed as (virtual) folder.

Note that the special .folder hidden object will not block the deletion of a folder (as other hidden objects will do).

parameters

delete object or folder

DELETE /bucket/{name}/{path}?recursive={recursive}

Removes the object or an empty folder (for paths ending in /).

If recursive=true, recursively deletes all objects beneath the {path} prefix.

parameters

Deleting a virtual folder that has still child objects, will result in a 409 Conflict error message.

Error conditions

The following error conditions can occur (in this table, a folder path ends a / a prefix path not).

path case ? ?recursive=true ?recursive=true&all=true
exact object path (removed) (removed) (removed)
empty folder (removed) (removed) (removed)
non existing folder or prefix 404 Not Found 200 OK, nothing removed 200 OK, nothing removed
prefix to (one or more) objects,
no hidden objects
404 Not Found (removed) (removed)
a non-empty folder,
no hidden objects
409 Conflict (removed) (removed)
prefix to at least one
hidden object
404 Not Found 409 Conflict (removed)
non-empty folder,
with hidden objects
409 Conflict 409 Conflict (removed)

Note that deletion is not prevented for a {path} that contains hidden segments in its name, only hidden child objects will prevent a deletion.

subscription API

The subscriptions api of waylay storage lets you register callbacks that get triggered when objects get created, updated or deleted.

subscription json representation

A subscription is represented by a json document with the following properties (see create subscription for a concrete example):

Identifying properties

property type description
id string The unique identifier for the subscription, optional when creating a subscriptions (see update create).
title optional string A display title for this subscription.
description optional string A description for this subscription

Channel properties

property type description
channel object The specification of the target to which notifications must be forwarded
channel.type webscript or system The channel type. Users can only specify the webscript type, for calling Waylay Webscripts. Other types are for internal usage.
channel.name string The name of the webscript
channel.version optional string The webscript version
channel.method one of GET PUT POST The http method used to forward the notification event
channel.description optional string User description of the channel

Channel expiry properties

property type description
channel.expiry optional object Specification of the period in which forwarded messages should remain usable.
This influences the expiry of signed urls and authentication tokens that are present in the forwarded message.
Note that the total expiry period can not be larger than 7 days.
If not specified, an expiry of 300 seconds holds.
channel.expiry.seconds optional non-negative number seconds part of the expiry period
channel.expiry.hours optional non-negative number hours part of the expiry period
channel.expiry.days optional non-negative number days part of the expiry period

Channel message payload properties

property type description
channel.payload object Specification the content of the forwarded message.
channel.payload.signed_links array of GET, PUT, POST constants Specification the signed links that must be present to the message. This enables the receiver of the message (the webscript) to retrieve or update the content of the object without additional authentication.
channel.payload.reference optional any json value Fixed json value that will be included in the message. This can be used to identify the subscription (version) or parametrise the execution of a webscript that is the target of multiple subscriptions.

Channel authentication properties

property type description
channel.authentication object Specification of the authentication used to call the notification target.
channel.authentication.method optional one of NONE, API_KEY, TOKEN, WAYLAY_TOKEN, WEBSCRIPT (default NONE)
NONE
no authentication
API_KEY
basic authentication, e.g. with waylay apiKey/apiSecret credentials.
WAYLAY_TOKEN
Requires an apiKey/apiSecret. The notification mechanism will resolve this to a Waylay Token and uses that token to authenticate the notication call.
TOKEN
The notification will use the secret as a fixed Authorization: Bearer {secret} http header.
WEBSCRIPT
The notifier will use the secret as a webscript HMAC token.
channel.authentication.key string The authentication key, only relevant for API_KEY and WAYLAY_TOKEN method
channel.authentication.secret string The authentication secret, relevant for all but the NONE authentication method.
Note that this value gets encrypted to a read-only value with an enc: prefix when retrieved from the server. The value should be unencrypted in the request body of a subscription create or update request.

Event filter properties

property type description
filters array of objects The event filters that specify what changes will trigger the notification.
A change event must satisfy the prefix, suffix and events conditions to trigger this notification subscription.
filters[*].prefix optional string If present, the object path of the changed object must have this prefix. (e.g. images/)
filters[*].prefix optional string If present, the object path of the changed object must have this suffix. (e.g. .jpg)
filters[*].events optional array of put delete constants The event type, either put (any creation or update) or delete (removal of the object) or both.
When not specified , ["put"] is assumed.
filters[*].description optional string A description of this filter.
filters[*].queue readonly string This is a property that is set by waylay storage to indicate the notification queue that is handling the subscription.
When present, it is a proof that notification is correctly registered.
It should be left empty when creating or updating a subscription.

Notification Event Message

When a subscription succesfully connects object change events to e.g. a waylay webscript, it will notifiy the webscript according to the configured http method. When this is a POST or PUT, the request body will contain a json object that represents the notification event.

This message has the following json properties:

property type description
event one of put delete The type of the event
object object The representation of the changed object, as if it was retrieved via the get object API call.
Additionally, if channel.payload.signed_links is configured, extra HAL _links will be present with the corresponding signed links to the object content.
event_time datetime The event time (ISO8601 string) of the object change.
reference optional any json The json value configured in the channel.payload.reference of the subscription.
tenant string The id of the Waylay tenant that owns this object
store string The name of the object store that originated this change event

Example

This is the message payload that a webscript received when an object synced-resources/synced-AAAAA on the assets bucket was updated. In this cases "signed_links":["GET"] was configured, hence a signed url is present in object._links.get_object.href.

{
  "reference": {
    "message": "This is the first subscription."
  },
  "object": {
    "is_dir": false,
    "size": 123,
    "metadata": {
      "content-type": "application/octet-stream"
    },
    "name": "synced-resources/synced-AAAAA",
    "bucket": {
      "alias": "assets",
      "name": "ws-6ccc8843-d78d-49e8-84c4-3734a4af9929-assets",
      "_links": {
        "self": {
          "href": "/bucket/assets"
        }
      }
    },
    "etag": "81b87d26826330a3495ff4ee82866235",
    "_links": {
      "get_object": {
        "href": "https://object-02-storage-staging.waylay.io/ws-6ccc8843-d78d-49e8-84c4-3734a4af9929-assets/synced-resources/synced-AAAAA?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=6ccc8..."
      },
      "sign_get": {
        "href": "/bucket/assets/synced-resources/synced-AAAAA?sign=GET&expiry_seconds=300"
      },
      "self": {
        "href": "/bucket/assets/synced-resources/synced-AAAAA?stat=True"
      }
    },
    "delete_marker": false,
    "content_type": "application/octet-stream"
  },
  "store": "min",
  "tenant": "6ccc8843-d78d-49e8-84c4-3734a4af9929",
  "event_time": "2020-12-11T10:18:45.235000+00:00",
  "event": "put"
}

list subscriptions

GET /subscription

Lists buckets that support subscriptions, including their current subscriptions.

parameters

result

A list of objects containing the bucket that support subscriptions, and their subscriptions

example

GET /subscription
{
  "_links": {
    "self": {
      "href": "/subscription"
    }
  },
  "bucket_subscriptions": [
    {
      "_links": {
        "self": {
          "href": "/subscription/assets"
        }
      },
      "bucket": {
        // ... bucket representation
      },
      "subscriptions": [
        // ... subscription representation
      ]
    },
    // ... other buckets and their subscriptions
  ]
}

list bucket subscriptions

GET /subscription/{bucket_name}

Lists all subscriptions for this bucket

parameters

result

An objects containing the bucket and its subscriptions

example

GET /subscription/assets
{
    "_links": {
        "self": { "href": "/subscription/assets" }
    },
    "bucket": { 
        // ... bucket representation
    },
    "subscriptions": [
        // ... see subscription representation
    ]
}

create subscription

POST /subscription/{bucket_name}

Creates a new subscription

parameters

result

The representation of the subscription. This can change from the request body in the following ways:

example

POST /subscription/assets
{
    "id": "jpg_image_2_local-example-3",
    "title": "Handle jpg images",
    "description": "test subscription (integration test)",
    "channel": {
        "description": "POST notification to a webscript",
        "type": "webscript",
        "name": "local-example",
        "method": "POST",
        "authentication": {
            "method": "WEBSCRIPT",
            "secret": "a_secret_that_will_be_encrypted"
        },
        "expiry": {
            "seconds": 120
        },
        "payload": {
            "signed_links": [
                "GET"
            ],
            "reference": {
                "demo_message": "demoing this on friday 4 december"
            }
        }
    },
    "filters": [
        {
            "prefix": "img",
            "suffix": ".jpg",
            "events": [
                "put",
                "delete"
            ]
        },
        {
            "prefix": "img",
            "suffix": ".jpeg"
        }
    ]
}

get subscription

GET /subscription/{bucket_name}/{id}

Retrieve the representation of a subscription by id

parameters

result

The representation of the subscription

example

GET /subscription/assets/jpg_image_2_local-example-3
{
    "_links": {
        "self": {
            "href": "/subscription/assets/jpg_image_2_local-example-3"
        }
    },
    "id": "jpg_image_2_local-example-3",
    "title": "Handle jpg images",
    "description": "test subscription (integration test)",
    "channel": {
        "type": "webscript",
        "description": "POST notification to a webscript",
        "payload": {
            "signed_links": [
                "GET"
            ],
            "reference": {
                "demo_message": "demoing this on friday 4 december"
            }
        },
        "authentication": {
            "method": "WEBSCRIPT",
            "secret": "enc:gAAAAABf04Z_qQyM3spPs_-6F4NgWhK7VKJkTU6U6_SC5a8xBgmfCaSJdbbXlyB4jMCjACHt1bIIYJh-0h7NPUOJnmI_0130JveT58JZiU_U5NX-eyM07JY="
        },
        "expiry": {
            "seconds": 120
        },
        "name": "local-example",
        "method": "POST"
    },
    "filters": [
        {
            "prefix": "img",
            "suffix": ".jpg",
            "events": [
                "put",
                "delete"
            ],
            "queue": "arn:minio:sqs::6ccc8843-d78d-49e8-84c4-3734a4af9929:webhook"
        },
        {
            "prefix": "img",
            "suffix": ".jpeg",
            "events": [
                "put"
            ],
            "description": "this filter illustrates that 'events'=['put'] is default when not specified",
            "queue": "arn:minio:sqs::6ccc8843-d78d-49e8-84c4-3734a4af9929:webhook"
        }
    ]
}

update subscription

PUT /subscription/{bucket_name}/{id}

Creates or updates a subscription

parameters

result

The representation of the subscription. This can change from the request body in the following ways:

example

PUT /subscription/assets/jpg_image_2_local-example-3
{
    "id": "jpg_image_2_local-example-3", // must be absent or identical to url parameter
    ...
}

delete subscriptions

DELETE /subscription/{bucket_name}

Deletes zero or more subscriptions of the given bucket, optionally filtered on the channel or filter properties.

parameters

result

An object containing a HAL link removed containing references to the removed subscriptions

example

DELETE /subscription/assets?prefix=img
{
    "_links": {
        "removed": {
            "href": "/subscription/assets/jpg_image_2_local-example-3"
        }
    }
}

delete subscription

DELETE /subscription/{bucket_name}/{id}

Deletes a single subscription by id.

parameters

result

An object containing a HAL link removed containing a reference to the removed subscription.

example

DELETE /subscription/assets/jpg_image_2_local-example-3
{
    "_links": {
        "removed": {
            "href": "/subscription/assets/jpg_image_2_local-example-3"
        }
    }
}

status API

version

GET /

Returns the current version of the server. This endpoint is NOT authenticated.

See also