connections

A connection is a system, declared once under a name in a file beside the config and referred to by the components that use it. The split is what the system is (brokers, urls, credentials) against what this pipeline wants from it (topic, group, subject, table) — there is no inline form: a component names a connection or it does not build.

One kind serves both directions, so a kafka connection feeds a kafka input and a kafka output. The kind is checked as well as the name, and deleting one a running pipeline names is refused rather than breaking it later.

Credentials are typed as secrets and hold the unresolved ${NAME} template, never the value — see secrets. How the file is found, what happens when you edit one under a running pipeline, and why file is a connection at all are covered in connections.

kafka

A kafka cluster: the brokers, and eventually whatever it takes to authenticate against them.

field	type		description
brokers	string	required	comma-separated broker list, e.g. localhost:9092. May reference secrets as ${NAME} — see "secrets" in the readme.

nats

A nats server, or a cluster of them.

field	type		description
urls	string	required	connection url, e.g. nats://localhost:4222. May reference secrets as ${NAME} — see "secrets" in the readme.

postgres

A postgres database, as one role connects to it.

The database and the role are part of the connection; the table is not — that is what a particular output writes into, so it stays on the output.

field	type		description
host	string	required	server hostname, e.g. localhost
database	string	required	the database to connect to
user	string	required	the role to connect as
password	string	required	that role's password. May reference secrets as ${NAME} — see "secrets" in the readme, and prefer a reference to a literal here.
port	integer	optional	server port. Defaults to 5432.

clickhouse

A ClickHouse server, as one user connects to it over its HTTP interface.

The same split [PostgresConnection] makes: the server, the database and the user are the connection's; the table belongs to the output that writes it.

The HTTP interface rather than the native protocol because it is what every ClickHouse deployment exposes — including ClickHouse Cloud, where 8443 is the only port there is — and because it takes an insert as a body in a named format, which is exactly the shape a batch of messages already has.

field	type		description
url	string	required	url of the HTTP interface, e.g. http://localhost:8123 for the server in docker-compose.yaml, or https://<host>:8443 for ClickHouse Cloud.
database	string	required	the database to write into. It has to exist already — an output creates tables, never databases.
user	string	required	the user to connect as
password	string	required	that user's password. May reference secrets as ${NAME} — see "secrets" in the readme, and prefer a reference to a literal here.
allow_http	boolean	optional	allow a plaintext http:// url. Defaults to false: the credentials above go with every insert, so sending them in the clear is a decision worth writing down. The local server in docker-compose.yaml is the case that legitimately wants it.

file

A directory on the server's filesystem that file outputs write under.

The odd one out among the kinds: there is no host, no credentials, nothing to authenticate against. It earns its place as a connection anyway because it holds the same thing the others do — what the system is, as against what one pipeline wants from it. A file output names a path relative to this root exactly as a kafka output names a topic on those brokers, and the object-store connection that replaces it later swaps the root for a bucket without any component changing.

The root is not a boundary on its own. It arrives from POST /api/connections like any other connection, so a browser could name / here; what actually confines writes is the server's --data-dir, which this root has to resolve under. See Root::resolve in the root crate.

field	type		description
root	string	required	directory that file outputs write under, e.g. ./out/events. Created if it does not exist, and it must resolve inside the server's --data-dir — a server started without that flag has file output turned off.

s3

A bucket on an S3-compatible object store, and the credentials that reach it.

The bucket is where [FileConnection]'s root is: the thing the system gives you, against which an output names a prefix of its own. What is not here is any equivalent of --data-dir. There cannot be one — the server has no view of a remote namespace to confine writes within, so the boundary is the credentials, and giving a deployment a key that can only write one bucket is the thing that does what the sandbox does locally.

endpoint is what makes this work against rustfs, minio or any other S3-compatible server; left out, it is real AWS S3 in region.

field	type		description
bucket	string	required	the bucket to write into. It has to exist already — an output creates objects, never buckets.
access_key_id	string	required	access key id. May reference secrets as ${NAME} — see "secrets" in the readme, and prefer a reference to a literal here.
secret_access_key	string	required	secret access key. May reference secrets as ${NAME} — see "secrets" in the readme, and prefer a reference to a literal here.
allow_http	boolean	optional	allow a plaintext http:// endpoint. Defaults to false: credentials over http is a mistake worth having to write down, and the local rustfs is the case that legitimately wants it.
endpoint	string	optional	url of an S3-compatible server, e.g. http://localhost:9000 for the rustfs in docker-compose.yaml. Leave it out for real AWS S3, which is then addressed through region.
region	string	optional	the bucket's region. Defaults to us-east-1, which is also what an S3-compatible server that does not care about regions will accept.

mqtt

An mqtt broker.

Plaintext TCP only for now — there is no TLS field here yet, and that is a deliberate gap (see docs/roadmap.md) rather than an oversight: a CA certificate needs somewhere to live (a Secret? a file path resolved against --data-dir?) and that question deserves its own pass rather than a field bolted on to get this connection working.

field	type		description
host	string	required	broker hostname, e.g. localhost
password	string	optional	that username's password. May reference secrets as ${NAME} — see "secrets" in the readme, and prefer a reference to a literal here.
port	integer	optional	broker port. Defaults to 1883, mqtt's conventional plaintext port.
username	string	optional	username to connect with, if the broker requires one. Must be set together with password or not at all.

redis

A redis server, or a cluster front-end that speaks the same protocol.

Used through its pub/sub commands (SUBSCRIBE/PUBLISH), the same shape [NatsConnection] is — one url, which may already carry a password — rather than the key-value store: there is no queue to consume from here, so a redis input has exactly the delivery guarantees a nats one does (see RedisConfig's doc comment).

field	type		description
url	string	required	connection url, e.g. redis://localhost:6379 or redis://:${REDIS_PASSWORD}@localhost:6379/0. May reference secrets as ${NAME} — see "secrets" in the readme.

opcua

An OPC UA server, as one client session connects to it.

The endpoint is the whole of "what the system is" here — an OPC UA server exposes one address space at one url, and which nodes a pipeline reads out of it is the component's business, exactly as a topic is on a kafka connection.

Plaintext and anonymous or username/password only. There is no security policy field and no certificate: an OPC UA session can be signed and encrypted, and that is worth having, but it needs a client certificate, somewhere for it to live and a server trust list — the same question [MqttConnection]'s missing TLS raises, one size larger. It gets its own pass (see docs/roadmap.md) rather than a field bolted on here, and until then this refuses to pretend: the session is SecurityPolicy::None, so credentials cross the wire in the clear and belong on a network you trust.

One consequence is visible in the log and is not a fault: the OPC UA client prints two errors about a missing application instance certificate when a session is opened. kayak has none by design, and an unencrypted session needs none — a pipeline that logs those and then reports readings is working.

field	type		description
endpoint	string	required	endpoint url, e.g. opc.tcp://localhost:50000. May reference secrets as ${NAME} — see "secrets" in the readme. This is connected to directly: kayak does not ask the server for its endpoint list first. Discovery is the usual way, and it is the usual way to fail — a server behind docker, NAT or a load balancer advertises the hostname it knows itself by, which is regularly not one the client can resolve. What is written here is what is dialled.
password	string	optional	that username's password. May reference secrets as ${NAME} — see "secrets" in the readme, and prefer a reference to a literal here.
username	string	optional	username to sign in with, if the server requires one. Must be set together with password or not at all; without either, the session is anonymous.