connections

A connection is a system that you declare one time, under a name, in a file beside the config. Components refer to it by that name. The connection holds what the system is: brokers, urls, credentials. The component holds what the pipeline wants from the system: a topic, a group, a subject, a table. There is no inline form. A component names a connection, or it does not build.

One kind serves both directions. For example, a kafka connection serves a kafka input and a kafka output. kayak checks the kind as well as the name. kayak refuses to delete a connection that a pipeline uses.

Credentials are secrets. A connection holds the ${NAME} template, never the value. See secrets. Connections tells how kayak finds the file, what an edit does to a pipeline that runs, and why file is a connection.

kafka

A kafka cluster.

field	type		description
brokers	string	required	The brokers, as a list with commas, for example localhost:9092. You can use ${NAME} secret references.

nats

A nats server, or a cluster of nats servers.

field	type		description
urls	string	required	The url of the server, for example nats://localhost:4222. You can use ${NAME} secret references.

postgres

A postgres database and the role that kayak connects as. The output or the input sets the table.

field	type		description
host	string	required	The hostname of the server, for example localhost.
database	string	required	The database to connect to.
user	string	required	The role to connect as.
password	string	required	The password of the role. Use a ${NAME} secret reference for this value.
port	integer	optional	The port of the server. The default is 5432.

clickhouse

A ClickHouse server and the user that kayak connects as. kayak uses the HTTP interface of the server. The output or the input sets the table.

field	type		description
url	string	required	The url of the HTTP interface, for example http://localhost:8123 for the server in docker-compose.yaml, or https://<host>:8443 for ClickHouse Cloud.
database	string	required	The database to use. The database must exist. The output makes tables. It does not make databases.
user	string	required	The user to connect as.
password	string	required	The password of the user. Use a ${NAME} secret reference for this value.
allow_http	boolean	optional	Permit an http:// url with no TLS. The default is false. The credentials go with every request, as plain text. Use it only for a local server, for example the server in docker-compose.yaml.

file

A directory on the filesystem of the server. A file output writes under it, to a path relative to this directory.

The directory must be inside the --data-dir of the server. kayak checks this when it builds the output. A server started without --data-dir has no file output.

field	type		description
root	string	required	The directory that file outputs write under, for example ./out/events. It must be inside the --data-dir of the server. If the directory does not exist, kayak makes it.

s3

A bucket on an S3-compatible object store, and its credentials. An s3 output writes under a prefix in the bucket.

There is no limit like --data-dir for a bucket. The credentials set what kayak can write. Give kayak a key that can write only to this bucket.

Set endpoint for rustfs, minio or another S3-compatible server. Without endpoint, kayak uses AWS S3 in region.

field	type		description
bucket	string	required	The bucket to write to. The bucket must exist. The output makes objects. It does not make buckets.
access_key_id	string	required	The access key id. Use a ${NAME} secret reference for this value.
secret_access_key	string	required	The secret access key. Use a ${NAME} secret reference for this value.
allow_http	boolean	optional	Permit an http:// endpoint with no TLS. The default is false. The credentials then go over the network as plain text. Use it only for a local server, for example the rustfs in docker-compose.yaml.
endpoint	string	optional	The url of an S3-compatible server, for example http://localhost:9000 for the rustfs in docker-compose.yaml. Leave it out to use AWS S3 in region.
region	string	optional	The region of the bucket. The default is us-east-1. S3-compatible servers with no regions accept this value.

mqtt

An mqtt broker.

The connection uses plain TCP. TLS is not available.

field	type		description
host	string	required	The hostname of the broker, for example localhost.
password	string	optional	The password of the user. Use a ${NAME} secret reference for this value.
port	integer	optional	The port of the broker. The default is 1883.
username	string	optional	The username, if the broker requires one. Set username and password together, or set neither.

redis

A redis server, or a server that uses the same protocol.

kayak uses the pub/sub commands SUBSCRIBE and PUBLISH. It does not use the key-value store. Thus, a redis input has the same delivery guarantees as a nats input.

field	type		description
url	string	required	The url of the server, for example redis://localhost:6379 or redis://:${REDIS_PASSWORD}@localhost:6379/0. You can use ${NAME} secret references.

opcua

An OPC UA server. The connection holds the endpoint. The opcua input selects the nodes to read.

The session is not signed and not encrypted. kayak connects with the security policy None, as an anonymous user or with a username and password. The credentials go over the network as plain text. Use this connection only on a network that you trust.

When a session opens, the OPC UA client writes two errors about a missing application instance certificate to the log. This is not a fault. A session with no encryption needs no certificate.

field	type		description
endpoint	string	required	The url of the endpoint, for example opc.tcp://localhost:50000. You can use ${NAME} secret references. kayak connects directly to this url. It does not ask the server for its list of endpoints first. Thus, a server behind docker, NAT or a load balancer works when this url is correct.
password	string	optional	The password of the user. Use a ${NAME} secret reference for this value.
username	string	optional	The username, if the server requires one. Set username and password together, or set neither. Without them, the session is anonymous.

indu

An Indu Cloud deployment: the API and ingest endpoints, and the API key.

The indu output and the indu input use the same connection. The output writes streams through /ingest/v1/streams. The input reads sensors and streams through /api/v1. Make the key in Indu, on the /keys page or with indud apps register --kind kayak. Give the key a role in Indu.

field	type		description
url	string	required	The origin of the deployment, for example https://app.acme.indu.cloud. The ingest endpoint is /ingest/v1/… under this url, unless you set ingest_url.
api_key	string	required	The API key (indu.ak.…), as a ${NAME} secret reference.
ingest_url	string	optional	The origin of /ingest/v1/… when it is not under url, for example https://ingest.acme.indu.cloud. A single-server installation serves ingest on its own host.

tidepool

A Tidepool server and its ingest token. The tidepool output sets the table. You declare the tables in the Tidepool project. kayak does not make them.

field	type		description
url	string	required	The url of the server, for example http://localhost:7070.
allow_http	boolean	optional	Permit an http:// url with no TLS when token is set. The default is false. The token goes with every batch, as plain text. Without a token, an http:// url is always permitted.
token	string	optional	The ingest token, as a ${NAME} secret reference. Use the TIDEPOOL_INGEST_TOKEN of the server, or its admin token. Leave it out for a server with open ingest.