http api

This page lists every endpoint of the server. kayak generates it from kayak_core::api_docs::endpoints(). The router is built from the same table. Thus the page describes exactly the routes that exist. The same table is also an OpenAPI 3.1 document. A server renders it at /api/reference, with a panel that sends requests.

Access is the badge on each endpoint. The middleware applies the access from the same table entry. On a server with no accounts, kayak checks nothing. See authentication.

Each body links to its schema. The schemas come from the Rust types that the handlers deserialize.

pipelines

The running graph. A create or a delete has an immediate effect. It writes nothing to disk.

GET /api/pipelines

List the running pipelines

read listPipelines — Any signed-in user.

Returns the config of each running pipeline, with the id that it runs under. If the config has no id, kayak generates one.

This is the state of the server, not of the config file. A pipeline that you created after startup is in this list, but not in the file. GET /api/settings tells you if the two are different.

responses

status	body	description
200	[PipelineDto]	The running pipelines, in no specified order.
500	ApiError	An error occurred on the server. The body describes the error.

POST /api/pipelines

Build and start a pipeline

admin createPipeline — Signed-in users with the admin role.

The body is the config of one pipeline, as in the pipelines array of a config file. If you leave out id, kayak generates a random id that people can read. The response contains the id.

kayak builds and starts the pipeline before it sends the response. Thus, a 201 means that the pipeline runs. If a component does not build, the response is a 422 and nothing starts. For example, an unknown connection or a missing secret gives a 422.

This endpoint writes nothing to disk. To write the config file, use POST /api/config/save.

request body — Config The pipeline to build.

responses

status	body	description
201	PipelineDto	The pipeline is built and runs. The body contains its id.
409	ApiError	A pipeline with this id already runs.
422	ApiError	The JSON is valid, but the pipeline does not build. For example, a connection is unknown, a secret is missing, or an upstream pipeline does not exist.
500	ApiError	An error occurred on the server. The body describes the error.

DELETE /api/pipelines/{pipeline_id}

Stop and remove a pipeline

admin deletePipeline — Signed-in users with the admin role.

Stops the run loop of the pipeline and removes the pipeline from the graph. Downstream pipelines continue to run. They receive nothing more from this pipeline.

This endpoint writes nothing to disk.

path parameters

name	description
pipeline_id	The id of the pipeline.

responses

status	body	description
204	—	The pipeline is stopped and removed.
404	ApiError	No pipeline runs with that id.
500	ApiError	An error occurred on the server. The body describes the error.

POST /api/pipelines/{pipeline_id}/messages

Post messages into a pipeline

public ingestMessages — No credentials are necessary, also when authentication is on.

The endpoint of the http input of a pipeline. The path comes from the id of the pipeline. The endpoint exists while the pipeline runs. A system can use it to send data to kayak without a broker.

The body is one JSON message or an array of messages. An array becomes one batch, so ten messages go through the transforms in one pass. The transforms get the posted JSON with no change.

A 202 means that the pipeline put the batch in its queue. kayak sends the response before the outputs write the batch. Thus, a 202 does not tell you that the data arrived at an output.

This endpoint does not use the sign-in of the server. To protect it, set auth on the http input of the pipeline. The sender then puts a token in a header. Without auth, the endpoint accepts all requests. This is the default.

path parameters

name	description
pipeline_id	The id of the pipeline to post to.

request body — IngestRequest One message, or an array of messages to send as one batch.

responses

status	body	description
202	IngestResponse	The pipeline accepted the messages. The body contains the number of messages.
404	ApiError	No pipeline runs with that id, or the pipeline has no http input.
401	ApiError	The input has auth, and the request did not satisfy it. This is the credential of the input, not the sign-in of the server. A server account does not give access to this endpoint. The token gives access to nothing else.
503	ApiError	The queue of the pipeline is full, because the pipeline reads slower than the requests arrive. The pipeline accepted nothing. Send the request again.
500	ApiError	An error occurred on the server. The body describes the error.

GET /api/pipelines/{pipeline_id}/history

Get the history of a pipeline

read getPipelineHistory — Any signed-in user.

Returns the throughput and the failures of a pipeline over time. The server keeps this data in memory.

The history is not a copy of /events. The event stream is a live sample. It runs only while a client listens, and it drops passes under load. The history comes from counters that the run loop always updates. Thus, its counts are complete. It contains no message payloads. It contains counts, and one failure record for each different error message, with the first time, the last time and a count.

The buckets are oldest first, with no gaps, and empty buckets are included. A sequence of zero counts means that the pipeline stopped.

For an unknown pipeline or a new pipeline, the response is an empty history, not a 404. history.retention_secs in the server config sets how long the server keeps the history. When it is zero, the server records nothing and the response is always empty.

path parameters

name	description
pipeline_id	The id of the pipeline.

query parameters — all optional, so a bare request to this path is a working request.

name	description
resolution	coarse (the default): one bucket for each minute, over the configured retention. fine: one bucket for each 5 s, over the last 30 minutes. An unknown value gives coarse.

responses

status	body	description
200	PipelineHistory	The history of the pipeline at the requested resolution.

GET /api/pipelines/{pipeline_id}/transforms/{index}/script

Get the script of a running transform

read getPipelineScript — Any signed-in user.

Returns the rhai source of one script transform in a running pipeline, and each module that it imports. Use it to read the script of a file source over HTTP.

This is the text that kayak used to build the pipeline. kayak reads a file source and its imports one time, when it builds the pipeline. A running script does not read the filesystem again. If a file changed or was removed after the build, changed_on_disk is true on that script or module. To use the change, reload the config from disk.

path parameters

name	description
pipeline_id	The id of the pipeline.
index	The position of the transform in the chain of the pipeline, from zero.

responses

status	body	description
200	LoadedScript	The script, as built.
404	ApiError	No pipeline runs with that id, or the transform at that position is not a script, or there is no transform at that position.

GET /api/pipelines/{pipeline_id}/config

Get the config of one pipeline as text

read getPipelineConfig — Any signed-in user.

Returns the config of a running pipeline as YAML or JSON text. kayak uses the same code as a save to the config file. Thus, the text is the same as the entry of the pipeline in a saved file, with the id included.

This is the config that the pipeline runs. It can be different from the file on disk, because you can change the graph without a save. Credentials appear as their ${NAME} references. They are on the connections that the config names, not in the config.

path parameters

name	description
pipeline_id	The id of the pipeline.

query parameters — all optional, so a bare request to this path is a working request.

name	description
format	yaml or json. Without a value, or with a different value, the response uses the format of the config file. On a server with no config file, it uses JSON. The response tells you the format.

responses

status	body	description
200	PipelineSource	The config of the pipeline as text.
404	ApiError	No pipeline runs with that id.

POST /api/scripts/dry-run

Run a script over messages, without a pipeline

admin dryRunScript — Signed-in users with the admin role.

Compiles a script and runs it over the messages in the body. Use it to test a script before you put it in a pipeline. The dry run uses the same runner, the same operation limit and the same sandbox as a running transform.

A script with an error gives a 200, not a 400. The response is a tagged union on outcome. With emitted, it contains the batches. With failed, it contains the error message, with a line and a column. A 400 means that the request is not valid: the JSON is malformed, or a file source cannot be read.

The dry run never uses live state. It gets a private bucket, filled from state in the body. The response contains the contents of the bucket at the end. Then kayak discards the bucket.

request body — DryRunRequest The script, and the messages to run it over.

responses

status	body	description
200	DryRunResponse	The script ran, or it did not compile. The outcome field tells you which.
400	ApiError	The request is not valid: the JSON is malformed, or a file source cannot be read.
500	ApiError	An error occurred on the server. The body describes the error.

POST /api/inputs/sample

Read a few messages from an input, without a pipeline

admin sampleInput — Signed-in users with the admin role.

Builds the input in the body as a pipeline does. Then it reads up to max_messages messages in timeout_ms, and stops the input. Use it to see the fields of a stream before you configure the transforms and the outputs.

The sample uses the real input, with its envelope. Thus, the sample contains the metadata fields. The sample ignores the buffer of the input. notes in the response lists each change that the sample made.

Some inputs change their behavior for a sample, and notes tells you. A kafka sample uses a temporary consumer group. Thus, it does not rebalance the group of the pipeline and does not commit offsets. An mqtt sample uses its own client id, because a broker disconnects the older client with the same id. An http input gives a 400, because clients post to it and there is nothing to read.

No messages gives a 200 with an empty list. These inputs cannot read messages that were published before the sample started.

request body — SampleRequest The input to read from, and the limits of the read.

responses

status	body	description
200	SampleResponse	kayak took the sample. outcome tells you if it contains messages or if it failed.
400	ApiError	The request is not valid: the JSON is malformed, the input type is unknown, or the input cannot be sampled.
500	ApiError	An error occurred on the server. The body describes the error.

POST /api/pipelines/dry-run

Run transforms over messages, without a pipeline

admin dryRunPipeline — Signed-in users with the admin role.

Builds the transforms in the body as a pipeline does, and puts the messages through the chain. The response tells you what each stage sent on. Use it to see what a map writes, what a filter drops or what a reduce makes from a batch.

The response gives a list of batches for each stage. For example, a splitter sends many batches, and a filter that dropped all messages sends none. A buffer sends nothing while it holds the messages. At the end, kayak drains the chain. The response gives what a transform releases then as on_flush. A transform that still holds messages after the drain sends nothing, and the response shows that. A dry run has no clock tick, so a window with 30 s left does not close.

A dry run has no outputs. It changes no external system.

The dry run never uses live state, as with a script dry run. The buckets are private to the request. buckets in the body fills them, the response returns them, and then kayak discards them.

A transform that does not build, or that fails on a message, gives a 200 with outcome set to failed. The response also contains the stages that completed before the failure.

request body — PipelineDryRunRequest The messages, and the transforms to put them through.

responses

status	body	description
200	PipelineDryRunResponse	The chain ran, or it failed. The outcome field tells you which.
400	ApiError	The request is not valid: the JSON is malformed, or a transform type is unknown.
500	ApiError	An error occurred on the server. The body describes the error.

connections

The systems that pipelines connect to. You declare each system one time, with a name, and components refer to the name.

GET /api/connections

List the connections

read listConnections — Any signed-in user.

Returns the connections, as an object from name to connection. This is the same shape as the connections file.

Credentials appear as their ${NAME} references. The response never contains the secret values.

responses

status	body	description
200	Connections	All connections, in alphabetical order of name.

POST /api/connections

Add a connection

admin createConnection — Signed-in users with the admin role.

Adds a connection. A component reads its connection one time, when kayak builds the component. Thus, a new or changed connection has an effect only on new and rebuilt pipelines.

This endpoint writes nothing to disk. A save writes the config file and the connections file together.

request body — CreateConnectionRequest The connection, and its name.

responses

status	body	description
201	CreateConnectionRequest	The connection is added. The body is the connection as stored.
409	ApiError	A connection with that name already exists.
500	ApiError	An error occurred on the server. The body describes the error.

DELETE /api/connections/{connection_id}

Remove a connection

admin deleteConnection — Signed-in users with the admin role.

Removes a connection. If a running pipeline uses the connection, kayak does not remove it. The response is then a 409, and the body lists the pipelines.

path parameters

name	description
connection_id	The name of the connection.

responses

status	body	description
204	—	The connection is removed.
404	ApiError	No connection with that name exists.
409	ApiError	Running pipelines use the connection. The body lists them.
500	ApiError	An error occurred on the server. The body describes the error.

state

The contents of the state buckets. These endpoints are read-only. You declare buckets in the config, and remember transforms write to them.

GET /api/state

List the state buckets

read listStateBuckets — Any signed-in user.

Returns one entry for each bucket under state in the config, in alphabetical order. Each entry has the number of keys in the bucket and the limits of the bucket.

The API cannot make or remove buckets. You declare buckets in the config file. These endpoints are read-only.

responses

status	body	description
200	BucketSummary	All declared buckets, in alphabetical order.

GET /api/state/{bucket}

Get the contents of a bucket

read getStateBucket — Any signed-in user.

Returns the keys and the values for each key. The key with the most recent write is first.

The response has a limit. A bucket can hold thousands of keys, and the response returns one page of them. truncated tells you that the response is not complete, and keys gives the total. kayak takes the snapshot under the lock of the bucket. Thus, the snapshot is consistent, but it can be out of date when it arrives.

path parameters

name	description
bucket	The name of the bucket, as declared in the config.

responses

status	body	description
200	BucketContents	The contents of the bucket.
404	ApiError	No bucket with that name is declared.

config

The config file: how the server started, a save of the running graph to the file, and a reload of the graph from the file.

GET /api/settings

Get the config state of the server

read getSettings — Any signed-in user.

Returns the config file of the server, the location where a save writes, and whether the running graph is different from the last load or save.

If the server has no config file, you can still save. The save makes the file.

responses

status	body	description
200	SettingsDto	The config state of the server.

POST /api/config/save

Write the running graph to a config file

admin saveConfig — Signed-in users with the admin role.

Writes the running pipelines to a config file, in a deterministic order: topological, then by id. kayak writes a temporary file and then renames it. The same save writes the connections file and the layout file beside the config file. A config file without its connections does not start.

name must be a file name with no path. kayak writes the file only to the save directory of the server. To overwrite the loaded file, use its name.

format selects JSON or YAML. Without format, the extension of the name sets the format. On a server started without --config, a save makes the config file. After that save, revert reloads this file.

The default of overwrite is true. With false, the save only makes new files. If the file, or one of the two files beside it, exists, the response is a 409 and kayak writes nothing.

request body — SaveConfigRequest The file name, the format (optional), and whether to replace an existing file.

responses

status	body	description
200	SaveConfigResponse	The file is written. The body contains its path.
409	ApiError	overwrite is false, and the file or one of the two files beside it exists. kayak wrote nothing. The message names the files.
422	ApiError	name is not a file name with no path.
500	ApiError	An error occurred on the server. The body describes the error.

POST /api/config/revert

Stop the running graph and reload the config file

admin revertConfig — Signed-in users with the admin role.

Stops all running pipelines and builds the graph again from the config file. Use it to discard the changes of a session. You cannot undo it.

kayak parses the file before it stops the pipelines. If the file has an error, the running graph does not change. kayak reloads the connections first, because the pipelines use them. kayak waits for the old pipelines to stop before it builds the new ones. Thus, when the response arrives, only the new graph runs.

responses

status	body	description
204	—	The graph is reloaded and is the same as the file.
500	ApiError	There is no config file, or kayak cannot read or parse it. The running graph does not change.

layout

The positions of the pipelines in the web UI. kayak writes the layout to its own file immediately. It is not configuration.

GET /api/layout

Get the layout of the web UI

read getLayout — Any signed-in user.

Returns the positions of the pipelines in the web UI. The layout is separate from /api/pipelines, because it does not change what the server runs. A client that ignores the layout gets an automatic layout.

The layout contains only the pipelines that a user moved.

responses

status	body	description
200	LayoutFile	The stored layout.

PUT /api/layout

Replace the layout and write it to disk

admin replaceLayout — Signed-in users with the admin role.

Replaces the full layout. This is not a patch. To reset all positions to automatic, send {}.

kayak writes the layout to disk immediately. A layout change is never an unsaved change, because it does not change what the server runs. Without a config file, kayak keeps the layout in memory until a save makes the file.

request body — LayoutFile The complete layout.

responses

status	body	description
204	—	The layout is stored, and written to disk if there is a config file.
500	ApiError	An error occurred on the server. The body describes the error.

events

A live feed of the activity of the pipelines.

GET /events

Stream the activity of the pipelines

read streamEvents — Any signed-in user.

A text/event-stream of UiEvents. An event is a batch that arrives at a stage, or a failure. Each SSE data: field is one event as JSON.

The stream is a broadcast. A slow client misses events, and the pipelines do not wait for it. A gap in seq shows the missed events. The run loops publish events only while a client listens.

The stream is a tool for development. It is not a durable feed, and it can change.

responses

status	body	description
200	UiEvent	An event stream that stays open.

auth

Sign in and sign out. These endpoints exist on every server. On a server with no accounts, they report that there is nothing to sign in to.

GET /api/auth/me

Get the caller, and whether the server asks for credentials

public whoAmI — No credentials are necessary, also when authentication is on.

authentication_required tells you whether the server checks credentials. It is false on a server started without --server-config, or with auth: {type: none}. Such a server permits all operations to all callers.

username and role describe the caller. Both are null for a caller with no credentials. A null role is different from read: a read user can see the graph, and a caller with no credentials cannot.

This endpoint needs no credentials. A client uses it to decide whether to show a login page.

responses

status	body	description
200	AuthDto	The caller. This is not an error, also when the caller has no credentials.

POST /api/auth/login

Change credentials into a session

public login — No credentials are necessary, also when authentication is on.

Checks a username and a password against the accounts in the server settings file. If they are correct, the response sets an HttpOnly session cookie.

This endpoint is for browsers. Other clients send Authorization: Basic with each request and do not use this endpoint. The cookie exists because EventSource cannot send headers, and the web UI reads /events with EventSource.

A wrong password and an unknown username give the same 401. Thus, the endpoint does not tell a caller which accounts exist. On a server with no accounts, the response is a 200 with authentication_required set to false.

request body — LoginRequest The credentials to check.

responses

status	body	description
200	AuthDto	Signed in. The Set-Cookie header contains the session cookie.
401	ApiError	The username or the password is wrong. The body does not tell you which.
500	ApiError	An error occurred on the server. The body describes the error.

POST /api/auth/token

Change a JWT from an identity provider into a session

public tokenLogin — No credentials are necessary, also when authentication is on.

Use this endpoint on a server with the jwt auth scheme, to embed kayak in a host application. The host has a token from an identity provider, for example Cognito or Keycloak. It puts the token on the iframe URL as ?auth_token=, and the web UI posts it here one time.

kayak checks the token against the published keys of the issuer. If the token is valid, the response sets the same HttpOnly session cookie as a password login. Thus, the token is in one request only, and not in later access logs.

The session ends at the exp of the token, or earlier.

API clients do not need this endpoint. On a jwt server, Authorization: Bearer <token> works on all endpoints.

All refusals give the same 401. For example, an expired token, a wrong issuer and a server that does not accept tokens all give a 401.

request body — TokenLoginRequest The token to check.

responses

status	body	description
200	AuthDto	Signed in. The Set-Cookie header contains the session cookie.
401	ApiError	kayak did not accept the token, or this server does not accept tokens.
500	ApiError	An error occurred on the server. The body describes the error.

POST /api/auth/logout

End the session of this request

read logout — Any signed-in user.

Clears the cookie in the browser and removes the session on the server. Thus, a copy of the cookie also stops working.

The response is a 204, also when there was no session.

responses

status	body	description
204	—	Signed out.

reference

The description of the API.

GET /api/docs

Get the component reference as data

public listComponents — No credentials are necessary, also when authentication is on.

Returns all inputs, transforms, outputs and connections that kayak can build, with their fields, types and documentation. kayak generates the data from the config schemas. Thus, the data agrees with what the server accepts.

The /docs page generates the same data in the browser. Use this endpoint for other tools, for example a config linter, editor completion or a test.

responses

status	body	description
200	[ComponentDoc]	All components. family gives the family of each component.

GET /api/openapi.json

Get this API as an OpenAPI 3.1 document

public getOpenApi — No credentials are necessary, also when authentication is on.

kayak generates the document from the same table that registers the routes. The schemas come from the Rust types. Thus, the document describes the server that serves it.

Use it with a renderer, a client generator or a contract test. GET /api/reference is a renderer on the same server.

responses

status	body	description
200	OpenApiDocument	The OpenAPI document.

GET /api/reference

Get the rendered API reference

public apiReference — No credentials are necessary, also when authentication is on.

An HTML page that renders /api/openapi.json. It has a request panel to try the endpoints on this server.

The /docs page in the web UI shows the same endpoints. This page is the full reference, with the schemas.

responses

status	body	description
200	html	The reference page.