transforms

Transforms run in the sequence of the list. Each transform takes one batch and returns zero or more batches. Thus splitter and reducer can change the number of messages in a stream. All messages are untyped JSON. A transform addresses a field by its field path, so _meta.subject works in every place where value works.

remember and recall share a state bucket. They are two transforms because their position in the chain sets what they do. remember passes its batch on unchanged. recall writes the remembered values onto the messages that come after it.

Eight transforms keep state per group_by key in the state bucket of the pipeline: deadband, throttle, pivot, derive, rolling, smooth, detect and resample. They do not build in a pipeline without a state. See streaming statistics.

kayak refuses a contradictory transform when it builds the pipeline. Examples are a reducer with no aggregations, an as that overwrites a group field, and a map that writes a path through a scalar.

buffer

Keeps messages and sends them on when a trigger fires.

There are three triggers: a message count, a time, and a condition on a state bucket. You can use them together. The first trigger that fires releases the messages. A buffer with no trigger fails to build.

size sends batches of exactly that number of messages. seconds and until send all messages that the buffer keeps, as one batch.

The buffer setting on an input is a different thing. It makes batches before the transforms. The buffer transform makes batches at its position in the chain, for example after a filter or a recall.

field	type		description
max_messages	integer	optional	The maximum number of messages to keep. At this number, the buffer sends all kept messages and writes one warning to the log. Required if size is not set. Without a limit, a condition that is never true makes the buffer use more and more memory.
seconds	integer	optional	Send all kept messages this number of seconds after the first kept message. The time starts when the buffer keeps a message. An empty buffer sends nothing.
size	integer	optional	Send batches of exactly this number of messages, when each batch is full.
until	object	optional	Send all kept messages when a condition on a state bucket is true. Buckets are global, so a different pipeline can write the value that opens the gate.

until

field	type		description
conditions	list of numeric | string | one_of | none_of	required	The conditions that must all be true to release the buffer. At least one condition is required.
bucket	string	optional	The bucket to watch. The default is the bucket in the state of this pipeline. A pipeline with no state must set it.
key	string	optional	The key in the bucket to read. This is a literal key, not a field path. If you do not set it, the gate reads the value for the full bucket. remember writes that value when the state of its pipeline has no key.

until.conditions — each entry — type: "numeric"

field	type		description
field	string	required	The field to test, as a dotted path.
operator	greater_than | less_than | equal_to | not_equal_to	required	How to compare the field.
value	number	required	The number to compare to.

until.conditions — each entry — type: "string"

field	type		description
field	string	required	The field to test, as a dotted path.
operator	equal_to | not_equal_to | contains	required	How to compare the field.
value	string	required	The string to compare to.

until.conditions — each entry — type: "one_of"

field	type		description
field	string	required	The field to test, as a dotted path.
values	list of string	required	The strings that match.

until.conditions — each entry — type: "none_of"

field	type		description
field	string	required	The field to test, as a dotted path.
values	list of string	required	The strings that do not match.

http

Sends the batch to an http endpoint and continues with the reply. Use it to call a model or another service.

body sets the content of one request: the full batch as a JSON array, or one message. wrap puts the body under a key, for example {"instances": …}. response sets what the transform does with the reply. replace makes the reply the new batch. merge writes the reply onto the message under as. unwrap reads the reply from under a key first.

A status other than 2xx fails the batch. The error contains the reply of the endpoint. A network failure, a 5xx or a 429 is tried again retries times with backoff.

For a model, put a buffer and a features transform before this transform. Use response: merge to keep the identifiers of the message.

field	type		description
url	string	required	The endpoint to send to.
verb	GET | POST | PUT | PATCH | DELETE	required	The http method. GET and DELETE fail to build, because a request with no body cannot send the messages.
as	string	optional	The field to write the reply under. Applies to response: merge.
auth	bearer | header	optional	The credential that the transform sends. If you do not set it, the transform sends no credential.
body	batch | message	optional	The content of one request. The default is batch.
response	replace | merge	optional	What to do with the reply. The default is replace.
retries	integer	optional	The number of times to send a request again before the batch fails. Applies to a network failure, a 5xx and a 429. The default is 0. Each try waits longer than the previous try, and the pipeline waits too.
timeout_seconds	integer	optional	The maximum time for one request, in s. The default is 30.
unwrap	string	optional	A key to read the reply from, for an API that replies with {"predictions": …}.
wrap	string	optional	A key to put the body under, for an API that expects {"key": …}.

auth — type: "bearer"

field	type		description
token	string	required	The token. Use a ${NAME} reference, so that the config file keeps only the name and the secret store keeps the value.

auth — type: "header"

field	type		description
name	string	required	The name of the header. The http input compares the name without case. On an http input, the name must not be a header that an envelope copies into the messages.
value	string	required	The exact value of the header. Use a ${NAME} reference.

splitter

Divides one batch into smaller batches.

The last batch contains the messages that remain and can be smaller. For example, 4 messages with out_size: 3 give a batch of 3 and a batch of 1.

field	type		description
out_size	integer	required	The number of messages in each sent batch.

reducer

Reduces a batch to one message for each group. Put a buffer before it, or it gets only one message at a time.

Without group_by, the full batch is one group and the reducer sends one message. With group_by, it sends one message for each different combination of those fields. The messages are in the sequence in which the groups first occur. Each message contains the group_by fields and the results of the aggregations.

Each aggregation has a function, the field to use and the name as for the result, for example {"function": "avg", "field": "value", "as": "mean"}.

field	type		description
aggregations	list of aggregation	required	The values to calculate. At least one aggregation is required. Each one must have a different as.
group_by	list of string	optional	The fields whose combination defines a group. If you do not set it, the full batch is one group.
on_missing	error | skip	optional	What to do with a message that does not have one of the fields. The default is error.
time	string	optional	The field that contains the time of each message, as an RFC 3339 string or as ms since the epoch. slope requires it. A message without this field fails the batch. If you do not set it, the time of each message is its arrival time.

aggregations — each entry

field	type		description
function	sum | avg | min | max | count | count_distinct | first | last | collect | median | stddev | slope	required	How to combine the values.
as	string	required	The field that contains the result in the sent message. Each aggregation must have a different as. It must not be the same as a group_by field.
field	string	optional	The field to aggregate. Required for all functions except count. Without a field, count counts the messages.

filter

Keeps the messages that match all conditions and drops the other messages. With invert, it drops the messages that match and keeps the other messages.

The transform drops a batch that has no messages left. A message that does not have the field of a condition does not match that condition. A message with a field of the wrong type also does not match. Thus invert keeps these messages.

field	type		description
conditions	list of numeric | string | one_of | none_of	required	The conditions that a message must match. At least one condition is required.
invert	boolean	optional	Drop the messages that match, and keep the other messages. The default is false.

conditions — each entry — type: "numeric"

field	type		description
field	string	required	The field to test, as a dotted path.
operator	greater_than | less_than | equal_to | not_equal_to	required	How to compare the field.
value	number	required	The number to compare to.

conditions — each entry — type: "string"

field	type		description
field	string	required	The field to test, as a dotted path.
operator	equal_to | not_equal_to | contains	required	How to compare the field.
value	string	required	The string to compare to.

conditions — each entry — type: "one_of"

field	type		description
field	string	required	The field to test, as a dotted path.
values	list of string	required	The strings that match.

conditions — each entry — type: "none_of"

field	type		description
field	string	required	The field to test, as a dotted path.
values	list of string	required	The strings that do not match.

remember

Writes values from the messages that match into the state bucket of the pipeline. The key is the field that state.key of the pipeline names.

The transform sends each message on without changes. It does not filter.

The pipeline must have a state, or the transform fails to build.

field	type		description
remember	list of object	required	The values to take from a message that matches. At least one entry is required. Each entry must have a different as.
when	list of numeric | string | one_of | none_of	optional	The conditions that a message must match to be remembered. All of them must match. If you do not set it, the transform remembers values from each message. Set it when the stream contains more than one type of message.

remember — each entry

field	type		description
field	string	required	The field to take the value from.
as	string	required	The name for the value in the bucket. recall reads the value by this name. Each entry must have a different name.

when — each entry — type: "numeric"

field	type		description
field	string	required	The field to test, as a dotted path.
operator	greater_than | less_than | equal_to | not_equal_to	required	How to compare the field.
value	number	required	The number to compare to.

when — each entry — type: "string"

field	type		description
field	string	required	The field to test, as a dotted path.
operator	equal_to | not_equal_to | contains	required	How to compare the field.
value	string	required	The string to compare to.

when — each entry — type: "one_of"

field	type		description
field	string	required	The field to test, as a dotted path.
values	list of string	required	The strings that match.

when — each entry — type: "none_of"

field	type		description
field	string	required	The field to test, as a dotted path.
values	list of string	required	The strings that do not match.

recall

Writes values from the state bucket of the pipeline onto each message, with the names from remember.

Use it to add a slow fact to a fast stream, for example the current recipe of a machine. The values become top-level fields, so a reducer after it can group by them.

The pipeline must have a state, or the transform fails to build.

field	type		description
recall	list of string	required	The names to read from the bucket, as remember wrote them. The transform writes each value onto the message with the same name.
on_missing	skip | null | error	optional	What to do with a message when the bucket has no value for its key. The default is skip.

map

Changes the shape of every message. A mapping can rename, move, cast or remove a field, or write a constant. The transform applies the mappings in order.

Each entry in mappings reads fields from the message and writes one field. A mapping can read the fields that earlier mappings wrote. Use this for intermediate values. With keep: all, a drop can remove them again.

Reads and writes use dotted paths. For example, an as of sensor.id writes the value inside a sensor object. If the object does not exist, the transform makes it.

By default, the message passes through with the mappings applied to it. keep can change this. One message goes in and one message comes out. To drop a message, use filter. To make many messages, use splitter.

field	type		description
mappings	list of copy | constant | coalesce | cast | concat | arithmetic | time_bucket | drop	required	The fields to write, in order. Give one mapping or more. Two mappings must not write the same field.
keep	all | mapped	optional	Which fields of the input message the output keeps. The default is all.
on_missing	error | omit | null	optional	What to do with a message that does not have a field that a mapping reads. The default is error. A default on the mapping applies first. Use a default when you expect one field to be absent.

mappings — each entry — type: "copy"

field	type		description
from	string	required	The field to read, as a dotted path.
as	string	optional	The field to write. The default is the last segment of from.
default	text | number | boolean | null	optional	The value to write when the message does not have from. With it, on_missing does not apply.

mappings — each entry.default — type: "text"

field	type		description
value	string	required	The text.

mappings — each entry.default — type: "number"

field	type		description
value	number	required	The number.

mappings — each entry.default — type: "boolean"

field	type		description
value	boolean	required	The boolean value.

mappings — each entry.default — type: "null"

This component takes no configuration.

mappings — each entry — type: "constant"

field	type		description
value	text | number | boolean | null	required	The value to write.
as	string	required	The field to write the value to.

mappings — each entry.value — type: "text"

field	type		description
value	string	required	The text.

mappings — each entry.value — type: "number"

field	type		description
value	number	required	The number.

mappings — each entry.value — type: "boolean"

field	type		description
value	boolean	required	The boolean value.

mappings — each entry.value — type: "null"

This component takes no configuration.

mappings — each entry — type: "coalesce"

field	type		description
from	list of string	required	The fields to try, in order. Give two fields or more.
as	string	required	The field to write the first value to.
default	text | number | boolean | null	optional	The value to write when the message has none of the fields.

mappings — each entry.default — type: "text"

field	type		description
value	string	required	The text.

mappings — each entry.default — type: "number"

field	type		description
value	number	required	The number.

mappings — each entry.default — type: "boolean"

field	type		description
value	boolean	required	The boolean value.

mappings — each entry.default — type: "null"

This component takes no configuration.

mappings — each entry — type: "cast"

field	type		description
from	string	required	The field to read.
to	text | integer | float | boolean | timestamp | date | uuid | json	required	The type to convert the value to.
as	string	optional	The field to write. The default is the last segment of from. For example, {"from": "value", "to": "float"} converts value in place.
default	text | number | boolean | null	optional	The value to write when the message does not have from. A value that is present and does not convert is always an error.

mappings — each entry.default — type: "text"

field	type		description
value	string	required	The text.

mappings — each entry.default — type: "number"

field	type		description
value	number	required	The number.

mappings — each entry.default — type: "boolean"

field	type		description
value	boolean	required	The boolean value.

mappings — each entry.default — type: "null"

This component takes no configuration.

mappings — each entry — type: "concat"

field	type		description
parts	list of field | value	required	The parts, in order. Give one part or more.
as	string	required	The field to write the string to.

mappings — each entry.parts — each entry — type: "field"

field	type		description
field	string	required	The field to read.

mappings — each entry.parts — each entry — type: "value"

field	type		description
value	string	required	The text.

mappings — each entry — type: "arithmetic"

field	type		description
left	field | value	required	The left operand.
operator	add | subtract | multiply | divide | min | max	required	The operation to do.
right	field | value	required	The right operand.
as	string	required	The field to write the result to.
on_zero	error | null | value	optional	For divide: what to do when the right field holds zero. The default is error, which fails the batch.

mappings — each entry.left — type: "field"

field	type		description
field	string	required	The field to read. It must hold a number.

mappings — each entry.left — type: "value"

field	type		description
value	number	required	The number.

mappings — each entry.right — type: "field"

field	type		description
field	string	required	The field to read. It must hold a number.

mappings — each entry.right — type: "value"

field	type		description
value	number	required	The number.

mappings — each entry.on_zero — type: "error"

This component takes no configuration.

mappings — each entry.on_zero — type: "null"

This component takes no configuration.

mappings — each entry.on_zero — type: "value"

field	type		description
value	number	required	The number to write.

mappings — each entry — type: "time_bucket"

field	type		description
from	string	required	The field that holds the time, as an RFC 3339 string or as milliseconds since the epoch.
every_seconds	integer	required	The length of a period, in seconds.
as	string	required	The field to write the start time to.
format	rfc3339 | millis	optional	The format of the start time. The default is rfc3339.
offset_seconds	integer	optional	The offset of the start of the periods, in seconds. It must be less than every_seconds.
timezone	string	optional	The IANA time zone of the periods, for example Europe/Stockholm. The default is UTC.

mappings — each entry — type: "drop"

field	type		description
from	list of string	required	The fields to remove. Give one field or more.

script

Runs a rhai script over each message, or over the full batch, and sends the values that the script emits.

The script gets the message as msg, and sends values with emit(value). Call emit zero times to drop the message, one time to replace it, or more times to split it. Use a script when filter, map and splitter are not sufficient.

kayak compiles the script when it builds the pipeline. Thus, a syntax error stops the pipeline from starting. An error that occurs only with a message fails that batch. For example, a missing field or a value that does not convert fails the batch.

Each run of the script has an operation limit (max_operations). When the script reaches the limit, kayak stops it and fails the batch. Thus, a script with an endless loop cannot block the pipeline.

A script can import other rhai files. Give a literal path relative to the directory of the config file. The path must stay in that directory. You can leave out the .rhai extension. kayak reads the imports when it builds the pipeline. Thus, a bad import stops the pipeline from starting, and a running script does not read the filesystem.

field	type		description
source	inline | file	required	The script: inline in the config, or in a file beside the config.
max_operations	integer	optional	The maximum number of rhai operations in one run of the script. At the limit, kayak stops the script and fails the batch. The default is 100000. Increase it for a script that walks a large array.
scope	message | batch	optional	Whether the script gets one message at a time or the full batch. The default is message.

source — type: "inline"

field	type		description
code	rhai script	required	The rhai source.

source — type: "file"

field	type		description
path	string	required	The path, relative to the directory of the config file. The path must stay in that directory.

deadband

Drops a message when its field did not change sufficiently since the last message that passed. This is a filter that keeps state per key.

The first message for each key always passes. After that, a message passes in two cases:

• field differs from the last value that passed by more than delta. - max_seconds went by since the last message passed.

With flatline_seconds, the transform also finds a stuck sensor. When the value does not change for that time, the next message passes with stuck: true. This occurs one time for each flat period.

field	type		description
field	string	required	The numeric field to compare.
delta	number	required	The change that a message needs to pass. The unit is set by mode.
flatline_seconds	number	optional	After this number of seconds with no change, pass the next message with stuck: true. This occurs one time for each flat period.
group_by	list of string	optional	The fields that identify a series, as in reduce. Leave it out for one series.
max_seconds	number	optional	Pass a message when this number of seconds went by since the last message passed, also if the value did not change.
mode	absolute | percent	optional	The unit of delta. The default is absolute.
on_missing	error | skip	optional	What to do with a message that does not have the field or a group field. The default is error.
reset_when	list of numeric | string | one_of | none_of	optional	The conditions that clear the state of the message's key. All of them must match. The series then starts again. The transform checks these before when.
time	string	optional	The field that holds the time of each message, as an RFC 3339 string or as milliseconds since the epoch. Leave it out to use the arrival time.
when	list of numeric | string | one_of | none_of	optional	The conditions that a message must match to be applied. All of them must match. Other messages pass through unchanged. Leave it out to apply every message.

reset_when — each entry — type: "numeric"

field	type		description
field	string	required	The field to test, as a dotted path.
operator	greater_than | less_than | equal_to | not_equal_to	required	How to compare the field.
value	number	required	The number to compare to.

reset_when — each entry — type: "string"

field	type		description
field	string	required	The field to test, as a dotted path.
operator	equal_to | not_equal_to | contains	required	How to compare the field.
value	string	required	The string to compare to.

reset_when — each entry — type: "one_of"

field	type		description
field	string	required	The field to test, as a dotted path.
values	list of string	required	The strings that match.

reset_when — each entry — type: "none_of"

field	type		description
field	string	required	The field to test, as a dotted path.
values	list of string	required	The strings that do not match.

when — each entry — type: "numeric"

field	type		description
field	string	required	The field to test, as a dotted path.
operator	greater_than | less_than | equal_to | not_equal_to	required	How to compare the field.
value	number	required	The number to compare to.

when — each entry — type: "string"

field	type		description
field	string	required	The field to test, as a dotted path.
operator	equal_to | not_equal_to | contains	required	How to compare the field.
value	string	required	The string to compare to.

when — each entry — type: "one_of"

field	type		description
field	string	required	The field to test, as a dotted path.
values	list of string	required	The strings that match.

when — each entry — type: "none_of"

field	type		description
field	string	required	The field to test, as a dotted path.
values	list of string	required	The strings that do not match.

throttle

Passes a maximum of one message per key in each period of seconds, and drops the other messages.

The first message for each key passes. The next message that passes is the first one that arrives seconds or more after it. The transform drops all messages between them. The period starts at the message that passed. It is not aligned to the clock.

The transform does not keep messages to send later. A key that becomes quiet sends nothing until its next message. Use resample when you need the last value of each period, or a value from a quiet key.

throttle does not read a value. It passes complete messages with all their fields.

field	type		description
seconds	number	required	The minimum time between two messages that pass for one key, in seconds.
group_by	list of string	optional	The fields that identify a series, as in reduce. Leave it out for one series.
on_missing	error | skip	optional	What to do with a message that does not have a group field. The default is error.
reset_when	list of numeric | string | one_of | none_of	optional	The conditions that clear the state of the message's key. All of them must match. The series then starts again. The transform checks these before when.
time	string	optional	The field that holds the time of each message, as an RFC 3339 string or as milliseconds since the epoch. Leave it out to use the arrival time.
when	list of numeric | string | one_of | none_of	optional	The conditions that a message must match to be applied. All of them must match. Other messages pass through unchanged. Leave it out to apply every message.

reset_when — each entry — type: "numeric"

field	type		description
field	string	required	The field to test, as a dotted path.
operator	greater_than | less_than | equal_to | not_equal_to	required	How to compare the field.
value	number	required	The number to compare to.

reset_when — each entry — type: "string"

field	type		description
field	string	required	The field to test, as a dotted path.
operator	equal_to | not_equal_to | contains	required	How to compare the field.
value	string	required	The string to compare to.

reset_when — each entry — type: "one_of"

field	type		description
field	string	required	The field to test, as a dotted path.
values	list of string	required	The strings that match.

reset_when — each entry — type: "none_of"

field	type		description
field	string	required	The field to test, as a dotted path.
values	list of string	required	The strings that do not match.

when — each entry — type: "numeric"

field	type		description
field	string	required	The field to test, as a dotted path.
operator	greater_than | less_than | equal_to | not_equal_to	required	How to compare the field.
value	number	required	The number to compare to.

when — each entry — type: "string"

field	type		description
field	string	required	The field to test, as a dotted path.
operator	equal_to | not_equal_to | contains	required	How to compare the field.
value	string	required	The string to compare to.

when — each entry — type: "one_of"

field	type		description
field	string	required	The field to test, as a dotted path.
values	list of string	required	The strings that match.

when — each entry — type: "none_of"

field	type		description
field	string	required	The field to test, as a dotted path.
values	list of string	required	The strings that do not match.

pivot

Changes a stream with one reading per message into rows. The transform keeps the latest value of each name in names for each key. It writes all of these values onto every message.

For example, the input is {"sensor": "state", "value": "RUNNING"} and then {"sensor": "fault", "value": "NONE"}. The output row is {"state": "RUNNING", "fault": "NONE", ...}.

When the name field of a message holds one of names, the transform first records that reading. Then it writes every value that it keeps for the key onto the message. It writes them at the top level, or under into. A name with no reading yet for the key is not written. One message goes in and one message comes out.

names is required because it limits the state. A message with a name that is not in names adds nothing, but it gets the row.

field	type		description
name	string	required	The field that holds the name of the reading. kayak compares its value with names.
value	string	required	The field that holds the reading. The value can be any JSON value, for example a string or a number.
names	list of string	required	The names to keep and write. Each name becomes a field. Give one name or more.
group_by	list of string	optional	The fields that identify a row, as in reduce. Leave it out for one row.
into	string	optional	An object field to write the values under. Leave it out to write them at the top level.
on_missing	error | skip	optional	What to do with a message that does not have a group field, or that has a name from names and no value. The default is error.
reset_when	list of numeric | string | one_of | none_of	optional	The conditions that clear the state of the message's key. All of them must match. The series then starts again. The transform checks these before when.
when	list of numeric | string | one_of | none_of	optional	The conditions that a message must match to be applied. All of them must match. Other messages pass through unchanged. Leave it out to apply every message.

reset_when — each entry — type: "numeric"

field	type		description
field	string	required	The field to test, as a dotted path.
operator	greater_than | less_than | equal_to | not_equal_to	required	How to compare the field.
value	number	required	The number to compare to.

reset_when — each entry — type: "string"

field	type		description
field	string	required	The field to test, as a dotted path.
operator	equal_to | not_equal_to | contains	required	How to compare the field.
value	string	required	The string to compare to.

reset_when — each entry — type: "one_of"

field	type		description
field	string	required	The field to test, as a dotted path.
values	list of string	required	The strings that match.

reset_when — each entry — type: "none_of"

field	type		description
field	string	required	The field to test, as a dotted path.
values	list of string	required	The strings that do not match.

when — each entry — type: "numeric"

field	type		description
field	string	required	The field to test, as a dotted path.
operator	greater_than | less_than | equal_to | not_equal_to	required	How to compare the field.
value	number	required	The number to compare to.

when — each entry — type: "string"

field	type		description
field	string	required	The field to test, as a dotted path.
operator	equal_to | not_equal_to | contains	required	How to compare the field.
value	string	required	The string to compare to.

when — each entry — type: "one_of"

field	type		description
field	string	required	The field to test, as a dotted path.
values	list of string	required	The strings that match.

when — each entry — type: "none_of"

field	type		description
field	string	required	The field to test, as a dotted path.
values	list of string	required	The strings that do not match.

derive

Writes a value onto each message that needs the previous message of the same key. The value can be a rate of change, a delta, a running total or a counter that wraps.

You can give many derivations. The transform writes each one to its own as field. The first message for each key has no previous message. For that message, rate and delta write null.

field	type		description
derive	list of derivation	required	The values to calculate. Give one or more, each with a different as.
group_by	list of string	optional	The fields that identify a series, as in reduce. Leave it out for one series.
on_missing	error | skip	optional	What to do with a message that does not have a derived field or a group field. The default is error.
reset_when	list of numeric | string | one_of | none_of	optional	The conditions that clear the state of the message's key. All of them must match. The series then starts again. The transform checks these before when.
time	string	optional	The field that holds the time of each message, as an RFC 3339 string or as milliseconds since the epoch. Leave it out to use the arrival time.
when	list of numeric | string | one_of | none_of	optional	The conditions that a message must match to be applied. All of them must match. Other messages pass through unchanged. Leave it out to apply every message.

derive — each entry

field	type		description
function	rate | delta | cumsum | counter	required	The calculation to do.
field	string	required	The numeric field to read.
as	string	required	The field to write the result to.
wrap_at	number	optional	For counter: the value at which the counter goes back to zero. With it, the transform reads a decrease as a wrap. Without it, the transform reads a decrease as a reset.

reset_when — each entry — type: "numeric"

field	type		description
field	string	required	The field to test, as a dotted path.
operator	greater_than | less_than | equal_to | not_equal_to	required	How to compare the field.
value	number	required	The number to compare to.

reset_when — each entry — type: "string"

field	type		description
field	string	required	The field to test, as a dotted path.
operator	equal_to | not_equal_to | contains	required	How to compare the field.
value	string	required	The string to compare to.

reset_when — each entry — type: "one_of"

field	type		description
field	string	required	The field to test, as a dotted path.
values	list of string	required	The strings that match.

reset_when — each entry — type: "none_of"

field	type		description
field	string	required	The field to test, as a dotted path.
values	list of string	required	The strings that do not match.

when — each entry — type: "numeric"

field	type		description
field	string	required	The field to test, as a dotted path.
operator	greater_than | less_than | equal_to | not_equal_to	required	How to compare the field.
value	number	required	The number to compare to.

when — each entry — type: "string"

field	type		description
field	string	required	The field to test, as a dotted path.
operator	equal_to | not_equal_to | contains	required	How to compare the field.
value	string	required	The string to compare to.

when — each entry — type: "one_of"

field	type		description
field	string	required	The field to test, as a dotted path.
values	list of string	required	The strings that match.

when — each entry — type: "none_of"

field	type		description
field	string	required	The field to test, as a dotted path.
values	list of string	required	The strings that do not match.

rolling

Writes aggregations over the last messages of the series onto each message. The aggregations use the same {function, field, as} list and functions as reduce. One message goes in and one message comes out.

The window holds a maximum of size messages. size is always required. With seconds, the window also drops messages that are older than that time, from the time field.

Here, count needs a field. It counts the messages in the window that have the field. When the window is full, the count is size.

field	type		description
aggregations	list of aggregation	required	The aggregations to calculate over the window. Give one or more, each with a different as.
size	integer	required	The maximum number of messages in the window.
group_by	list of string	optional	The fields that identify a series, as in reduce. Leave it out for one series.
on_missing	error | skip	optional	What to do with a message that does not have an aggregated field or a group field. The default is error.
reset_when	list of numeric | string | one_of | none_of	optional	The conditions that clear the state of the message's key. All of them must match. The series then starts again. The transform checks these before when.
seconds	number	optional	Also drop the messages that are older than this number of seconds, from the time field.
time	string	optional	The field that holds the time of each message, as an RFC 3339 string or as milliseconds since the epoch. Leave it out to use the arrival time.
when	list of numeric | string | one_of | none_of	optional	The conditions that a message must match to be applied. All of them must match. Other messages pass through unchanged. Leave it out to apply every message.

aggregations — each entry

field	type		description
function	sum | avg | min | max | count | count_distinct | first | last | collect | median | stddev | slope	required	How to combine the values.
as	string	required	The field that contains the result in the sent message. Each aggregation must have a different as. It must not be the same as a group_by field.
field	string	optional	The field to aggregate. Required for all functions except count. Without a field, count counts the messages.

reset_when — each entry — type: "numeric"

field	type		description
field	string	required	The field to test, as a dotted path.
operator	greater_than | less_than | equal_to | not_equal_to	required	How to compare the field.
value	number	required	The number to compare to.

reset_when — each entry — type: "string"

field	type		description
field	string	required	The field to test, as a dotted path.
operator	equal_to | not_equal_to | contains	required	How to compare the field.
value	string	required	The string to compare to.

reset_when — each entry — type: "one_of"

field	type		description
field	string	required	The field to test, as a dotted path.
values	list of string	required	The strings that match.

reset_when — each entry — type: "none_of"

field	type		description
field	string	required	The field to test, as a dotted path.
values	list of string	required	The strings that do not match.

when — each entry — type: "numeric"

field	type		description
field	string	required	The field to test, as a dotted path.
operator	greater_than | less_than | equal_to | not_equal_to	required	How to compare the field.
value	number	required	The number to compare to.

when — each entry — type: "string"

field	type		description
field	string	required	The field to test, as a dotted path.
operator	equal_to | not_equal_to | contains	required	How to compare the field.
value	string	required	The string to compare to.

when — each entry — type: "one_of"

field	type		description
field	string	required	The field to test, as a dotted path.
values	list of string	required	The strings that match.

when — each entry — type: "none_of"

field	type		description
field	string	required	The field to test, as a dotted path.
values	list of string	required	The strings that do not match.

smooth

Smooths a numeric field against the earlier values of its series. The transform writes the result into the field, or under as.

All methods except ewma with tau_seconds use the order of the values, and not their time. Thus, time is permitted only with ewma and tau_seconds. With other methods, time is an error when the pipeline builds.

field	type		description
field	string	required	The numeric field to smooth.
method	ewma | median | hampel | savitzky_golay	required	The method to smooth the value with.
as	string	optional	The field to write the smoothed value to. Leave it out to replace the value of field.
group_by	list of string	optional	The fields that identify a series, as in reduce. Leave it out for one series.
on_missing	error | skip	optional	What to do with a message that does not have the field or a group field. The default is error.
reset_when	list of numeric | string | one_of | none_of	optional	The conditions that clear the state of the message's key. All of them must match. The series then starts again. The transform checks these before when.
time	string	optional	For ewma with tau_seconds only: the field that holds the time of each message, as an RFC 3339 string or as milliseconds since the epoch. Leave it out to use the arrival time.
when	list of numeric | string | one_of | none_of	optional	The conditions that a message must match to be applied. All of them must match. Other messages pass through unchanged. Leave it out to apply every message.

method — type: "ewma"

field	type		description
alpha	number	optional	The weight of the newest value, from 0 to 1.
half_life	number	optional	The number of messages after which the weight of a value is half.
tau_seconds	number	optional	The time constant in seconds. After this time, the weight of an old value is about 37%. It reads the time field of the transform.

method — type: "median"

field	type		description
size	integer	required	The number of values in the window.

method — type: "hampel"

field	type		description
size	integer	required	The number of values in the window, with this value included.
threshold	number	optional	The number of scaled MADs from the median that makes an outlier. The default is 3.

method — type: "savitzky_golay"

field	type		description
size	integer	required	The number of values in the window, with this value included.
order	integer	optional	The degree of the polynomial. It must be less than size. The default is 2.

reset_when — each entry — type: "numeric"

field	type		description
field	string	required	The field to test, as a dotted path.
operator	greater_than | less_than | equal_to | not_equal_to	required	How to compare the field.
value	number	required	The number to compare to.

reset_when — each entry — type: "string"

field	type		description
field	string	required	The field to test, as a dotted path.
operator	equal_to | not_equal_to | contains	required	How to compare the field.
value	string	required	The string to compare to.

reset_when — each entry — type: "one_of"

field	type		description
field	string	required	The field to test, as a dotted path.
values	list of string	required	The strings that match.

reset_when — each entry — type: "none_of"

field	type		description
field	string	required	The field to test, as a dotted path.
values	list of string	required	The strings that do not match.

when — each entry — type: "numeric"

field	type		description
field	string	required	The field to test, as a dotted path.
operator	greater_than | less_than | equal_to | not_equal_to	required	How to compare the field.
value	number	required	The number to compare to.

when — each entry — type: "string"

field	type		description
field	string	required	The field to test, as a dotted path.
operator	equal_to | not_equal_to | contains	required	How to compare the field.
value	string	required	The string to compare to.

when — each entry — type: "one_of"

field	type		description
field	string	required	The field to test, as a dotted path.
values	list of string	required	The strings that match.

when — each entry — type: "none_of"

field	type		description
field	string	required	The field to test, as a dotted path.
values	list of string	required	The strings that do not match.

detect

Flags anomalies in a numeric field against its own series. method selects how the transform finds an anomaly.

The transform writes a boolean under as. The default is anomaly. It also writes a score under <as>_score. The score is the distance from normal, in the units of the method. A filter after the transform can use a stricter limit on the score.

The warm-up is min_samples messages for each key. The transform flags nothing during the warm-up.

field	type		description
field	string	required	The numeric field to examine.
method	zscore | mad | cusum | ewma_chart | western_electric | flatline | ewma	required	How the transform finds an anomaly.
as	string	optional	The field to write the flag to. The score goes to <as>_score. The default is anomaly.
group_by	list of string	optional	The fields that identify a series, as in reduce. Leave it out for one series.
learn	all | normal_only	optional	For zscore, mad and ewma: the readings that the baseline learns from. The default is all.
min_samples	integer	optional	The number of messages for each key before the transform flags anything. The default is the size of the method, or 30 for a method with no size.
mode	annotate | only_anomalies	optional	Which messages to send. The default is annotate.
on_missing	error | skip	optional	What to do with a message that does not have the field or a group field. The default is error.
readapt_after_seconds	number	optional	With learn: normal_only: when readings are anomalies for this number of seconds without a break, the baseline learns from them. A change that continues then becomes the new normal.
reset_when	list of numeric | string | one_of | none_of	optional	The conditions that clear the state of the message's key. All of them must match. The series then starts again. The transform checks these before when.
time	string	optional	The field that holds the time of each message, as an RFC 3339 string or as milliseconds since the epoch. The ewma method and readapt_after_seconds use it. Leave it out to use the arrival time.
when	list of numeric | string | one_of | none_of	optional	The conditions that a message must match to be applied. All of them must match. Other messages pass through unchanged. Leave it out to apply every message.
with_baseline	boolean	optional	Also write the baseline of the reading, in the units of the field. The normal value goes to <as>_expected. The permitted distance from it goes to <as>_band. The value is null when the method has no baseline.

method — type: "zscore"

field	type		description
size	integer	required	The number of earlier values in the baseline.
threshold	number	optional	The number of standard deviations that makes an anomaly. The default is 3.

method — type: "mad"

field	type		description
size	integer	required	The number of earlier values in the baseline.
threshold	number	optional	The number of scaled MADs that makes an anomaly. The default is 3.5.

method — type: "cusum"

field	type		description
drift	number	required	The change per message that does not count as drift, in the units of the field.
threshold	number	required	The sum of drift that makes an anomaly.
target	number	optional	The expected value of the series. Leave it out to use the mean of the warm-up.

method — type: "ewma_chart"

field	type		description
alpha	number	optional	The weight of the newest value, from 0 to 1. The default is 0.2.
threshold	number	optional	The width of the band, in standard deviations. The default is 3.

method — type: "western_electric"

This component takes no configuration.

method — type: "flatline"

field	type		description
size	integer	required	The number of identical values in a sequence that makes an anomaly.

method — type: "ewma"

field	type		description
mean_tau_seconds	number	required	The time constant of the mean, in seconds. A smaller value makes the mean follow the series more quickly.
spread_tau_seconds	number	required	The time constant of the spread, in seconds. Make it longer than mean_tau_seconds, so that a short period of noise does not make the band wider immediately.
min_spread	number	optional	The minimum deviation, in the units of the field. Without it, a signal that was very quiet flags its first small change. The default is 0.
threshold	number	optional	The number of deviations that makes an anomaly. The default is 3.

reset_when — each entry — type: "numeric"

field	type		description
field	string	required	The field to test, as a dotted path.
operator	greater_than | less_than | equal_to | not_equal_to	required	How to compare the field.
value	number	required	The number to compare to.

reset_when — each entry — type: "string"

field	type		description
field	string	required	The field to test, as a dotted path.
operator	equal_to | not_equal_to | contains	required	How to compare the field.
value	string	required	The string to compare to.

reset_when — each entry — type: "one_of"

field	type		description
field	string	required	The field to test, as a dotted path.
values	list of string	required	The strings that match.

reset_when — each entry — type: "none_of"

field	type		description
field	string	required	The field to test, as a dotted path.
values	list of string	required	The strings that do not match.

when — each entry — type: "numeric"

field	type		description
field	string	required	The field to test, as a dotted path.
operator	greater_than | less_than | equal_to | not_equal_to	required	How to compare the field.
value	number	required	The number to compare to.

when — each entry — type: "string"

field	type		description
field	string	required	The field to test, as a dotted path.
operator	equal_to | not_equal_to | contains	required	How to compare the field.
value	string	required	The string to compare to.

when — each entry — type: "one_of"

field	type		description
field	string	required	The field to test, as a dotted path.
values	list of string	required	The strings that match.

when — each entry — type: "none_of"

field	type		description
field	string	required	The field to test, as a dotted path.
values	list of string	required	The strings that do not match.

resample

Puts a series onto a regular grid. The transform sends one message for each key in each interval_seconds, at times that are multiples of the interval. The rate of the readings has no effect on the grid.

The message that the transform sends contains:

• the group fields, under their leaf names, - the grid time as an RFC 3339 string, under the name of the time field (or time when the transform uses the arrival time), - the value, under as (the leaf of field when you leave it out).

The transform sends a grid point when a later reading arrives. With forward_fill and the arrival time, it also sends a grid point when the clock passes it.

field	type		description
field	string	required	The numeric field to resample.
interval_seconds	number	required	The distance between two grid points, in seconds.
method	last | mean | linear | forward_fill	required	How the transform calculates the value of an interval.
as	string	optional	The field to write the value to. The default is the leaf of field.
group_by	list of string	optional	The fields that identify a series, as in reduce. Leave it out for one series.
max_gap_seconds	number	optional	For forward_fill: the maximum time to repeat a value into empty intervals, in seconds. Leave it out to repeat the value with no limit.
on_missing	error | skip	optional	What to do with a message that does not have the field or a group field. The default is error.
reset_when	list of numeric | string | one_of | none_of	optional	The conditions that clear the state of the message's key. All of them must match. The series then starts again. The transform checks these before when.
time	string	optional	The field that holds the time of each message, as an RFC 3339 string or as milliseconds since the epoch. Leave it out to use the arrival time. With the arrival time, the clock finds the empty intervals.
when	list of numeric | string | one_of | none_of	optional	The conditions that a message must match to be applied. All of them must match. Other messages pass through unchanged. Leave it out to apply every message.

reset_when — each entry — type: "numeric"

field	type		description
field	string	required	The field to test, as a dotted path.
operator	greater_than | less_than | equal_to | not_equal_to	required	How to compare the field.
value	number	required	The number to compare to.

reset_when — each entry — type: "string"

field	type		description
field	string	required	The field to test, as a dotted path.
operator	equal_to | not_equal_to | contains	required	How to compare the field.
value	string	required	The string to compare to.

reset_when — each entry — type: "one_of"

field	type		description
field	string	required	The field to test, as a dotted path.
values	list of string	required	The strings that match.

reset_when — each entry — type: "none_of"

field	type		description
field	string	required	The field to test, as a dotted path.
values	list of string	required	The strings that do not match.

when — each entry — type: "numeric"

field	type		description
field	string	required	The field to test, as a dotted path.
operator	greater_than | less_than | equal_to | not_equal_to	required	How to compare the field.
value	number	required	The number to compare to.

when — each entry — type: "string"

field	type		description
field	string	required	The field to test, as a dotted path.
operator	equal_to | not_equal_to | contains	required	How to compare the field.
value	string	required	The string to compare to.

when — each entry — type: "one_of"

field	type		description
field	string	required	The field to test, as a dotted path.
values	list of string	required	The strings that match.

when — each entry — type: "none_of"

field	type		description
field	string	required	The field to test, as a dotted path.
values	list of string	required	The strings that do not match.

features

Changes a batch of readings into one message of features for each group. Use it to send a small set of numbers to a model, in place of the raw readings. Put a buffer on the input. Without it, each batch has only one reading.

The transform writes:

• each feature in include under its own name, for example mean, rms or crest_factor, - each entry in bands under its as, - the group_by fields under their leaf names, as reduce does.

A feature with no value for the window is null. For example, a slope of one point is null. The spectral features (dominant_frequency, bands) need a sample rate. kayak uses sample_rate_hz, or calculates the rate from the time field. Without one of the two, the pipeline does not build. This transform keeps no state and needs no state bucket.

field	type		description
field	string	required	The numeric field to read.
bands	list of band	optional	The frequency bands to calculate the power of. The transform writes each one under its as. Give include, bands or both.
group_by	list of string	optional	The fields that identify a series, as in reduce. Leave it out to use the full batch as one window.
include	list of mean | std | min | max | range | slope | skew | kurtosis | rms | crest_factor | zero_crossings | n_peaks | autocorr1 | dominant_frequency | count | duration	optional	The features to calculate. The transform writes each one under its own name.
on_missing	error | skip	optional	What to do with a reading that does not have the field or a group field. The default is error.
sample_rate_hz	number	optional	The sample rate of the readings in hertz, for the spectral features. It must be more than zero. kayak uses it in place of the rate from time. Use it when the source has timestamps with low resolution.
time	string	optional	The field that holds the time of each reading, as an RFC 3339 string or as milliseconds since the epoch. slope and duration use it for seconds. The spectral features use it for the sample rate.

bands — each entry

field	type		description
low_hz	number	required	The lower limit of the band, in hertz. The band includes this value.
high_hz	number	required	The upper limit of the band, in hertz. The band does not include this value. It must be more than low_hz.
as	string	required	The field to write the power of the band to.