state buckets

You declare buckets at the top of the config file. The pipelines that use a bucket refer to it by name, as with connections. Buckets are global. Thus one pipeline can remember the current recipe per machine, and six other pipelines can add it to their output.

the rule that kayak does not enforce

Do not share a bucket between pipelines for data where order matters. Two pipelines are two run loops with no order between them. Put correlation that depends on order in one pipeline. Share a bucket only for state that changes slowly compared with the message rate.

Every bucket has a limit, and there is no bucket without a limit. kayak applies expiry when a pipeline touches a bucket. There is no background task. The contents stay after a config reload, unless the declaration of that bucket changed. State tells what remember and recall do with buckets.

state bucket

state.<name>

A named state bucket and its limits.

Both limits have a default. You cannot remove a limit, so the memory that a bucket uses is always limited.

field	type		description
idle_timeout_secs	integer	optional	Remove a key when this number of seconds went by after its last write. A read does not reset the time. The value must be more than zero. Without it, a key stays until the bucket is full. For example, a machine that you remove from service keeps its key.
max_keys	integer	optional	The maximum number of keys in the bucket. When the bucket is full, kayak removes the key with the oldest write. The default is 10000. The value must be more than zero.

pipeline state

pipelines[].state

The state bucket of a pipeline, and the field that gives the key of each message.

Each pipeline sets its own key, because the same value can have different field names in two streams. For example, one stream has _meta.machine_id and another stream has machine_id.

Make sure that all pipelines that share a bucket use keys with the same values. kayak does not check this.

field	type		description
bucket	string	required	The name of the bucket that this pipeline reads and writes. Declare the bucket under state at the top of the config. If the bucket is not declared, the pipeline does not build.
key	string	optional	The field that gives the key, as a dotted path. For example, _meta.machine_id. Leave it out for one value for the full bucket. Use that only for an item that has one value. For a value for each device, set key.