Skip to content
05 Building a Workflow Engine

The data model between steps

The decision that shapes everything downstream, and the one place a major engine is quietly leaky.

Lesson 03 of 06 · 11 min

One engine's edges carry an array of items, each with a JSON payload, optional binary, optional error, and a lineage pointer. Three things about that are load-bearing.

The payload/binary envelope. Files travel with the data. Elegant for authoring, expensive at runtime: binary is base64 in the envelope, roughly 33% inflation, in memory by default. And the mitigation — filesystem mode — is not compatible with queue mode. So scaling horizontally pushes your large files into Postgres.

Predict

The docs say nodes process multiple items automatically, each item individually. What does the engine actually do?

The rest of this lesson unlocks when you commit above.