How a flow executes
The engine executes a dependency graph: ready independent blocks run in parallel, required inputs are awaited and results pass to subsequent steps. Destinations first prepare and validate data; writes run as a separate stage. Dependent steps continue after the preceding operation returns its result. Parallel execution reduces waiting between independent actions while preserving graph dependencies.
Parallelism without changing dependencies
The coordinator starts only blocks whose inputs are ready. A combining block waits for every input, including notification of an inactive route. Several destinations can also be prepared in parallel. If external actions require strict sequencing, define it with a link from the preceding destination's result output.
Parameter GRAPH_PARALLELISM defaults to 4. This is the upper limit of concurrent graph work, not a promise of a fourfold speedup. The external API, memory capacity and graph shape determine the actual effect.
Preparation, snapshot and write
One execution stage: two independent branches are prepared in parallel. If later blocks depend on the write result, the engine continues the graph after writing that stage. There is no global transaction across external systems.
- Sources are read and transformed.
- Each destination checks schemas, coordinates and keys and prepares an immutable operation set.
- Conflicts between destinations targeting the same object are checked.
- Permissions, licensing and revision validity are rechecked before write batches.
- Counts of confirmed and unconfirmed operations are stored.
Resource limits
| Limit | Default value | User impact |
|---|---|---|
| Table cells | 2 000 000 | An excessively wide or large block result is rejected. |
| Graph snapshot budget | 512 MiB | Preview results cannot fill storage without limits. |
| Graph | Up to 256 nodes and 512 edges | Split oversized processes into manageable flows. |
| Interactive waiting | 12 hours | A user-input step does not wait indefinitely. |
Improving performance
- Restrict ranges, column lists and row counts as close to the source as possible.
- Filter unnecessary records before expensive joins and grouping.
- Check Join key uniqueness to avoid unexpected row multiplication.
- Align write batch sizes with destination capabilities.
- Measure reading, transformation and writing separately from the log; do not generalize one dataset's results to all scenarios.
Concurrent runs and write ordering
A flow may have several concurrent runs. The default per-flow concurrency limit is 4; worker-thread capacity is configured separately. Writes targeting the same cells, table keys or file are ordered, while independent objects may be written concurrently. Clearing a cube reserves the entire object. “Waiting for the write queue…” means an earlier conflicting write must finish.