Skip to main content

Coordinator

1. Coordination lifecycle (events)​

EventInitiatorWhen
ALL_SERVICES_READYservices-managerOnce at bootstrap, after all services attach (services.js)
BEFORE/AFTER_COMPONENTS_*dataspacePer app load / unload
START/FINISH_BATCH_UNREGISTERdataspaceInside unload() during app load swap and full teardown
COMPONENT_REGISTEREach component with WithCoordinationHelperone emit per instance on loadComponents
COMPONENT_UNREGISTEREach componentunregisterStreams / before('teardown')
ALL_COMPONENTS_STREAMS_PLUGGEDCoordinatorAfter component plug task opens the activation gate
sequenceDiagram
participant SM as services-manager
participant DS as dataspace
participant CO as Coordinator
participant S as Service
participant C as Component

Note over SM,S: Services graph (once, before any app load)
SM->>CO: ALL_SERVICES_READY
CO->>CO: startBatchOperation
CO->>CO: servicesReady plug task
CO->>CO: ingressController.pause()
Note over CO,S: plug task — per target, per batch
CO->>S: RUNTIME_CHANNELS_INFO
opt removed edges only
CO->>S: UNBIND_CHANNEL / UNBIND_GROUP_CHANNEL
end
Note over CO: commit unplug, create edge links (events enqueue globally)
CO->>S: BIND_CHANNEL / BIND_GROUP_CHANNEL
Note over S: bindChannel (handler)
CO->>S: ALL_SERVICES_STREAMS_CONNECTED
Note over S: servicesStreamsConnected
CO->>S: ALL_SERVICES_STREAMS_ACTIVATED
Note over S: servicesStreamsActivated
CO->>S: OLD_LIFECYCLE
Note over S: disconnectStreams → streamsConnected → streamsActivated
CO->>CO: ingressController.release()
CO->>CO: completeBatchOperation

Note over DS,C: Component graph (per app load)
DS->>CO: BEFORE_COMPONENTS_LOAD
DS->>CO: BEFORE_COMPONENTS_UNLOAD
DS->>CO: START_BATCH_UNREGISTER
CO->>CO: startBatchOperation
par each torn-down instance
C->>CO: COMPONENT_UNREGISTER
CO->>C: UNBIND_CHANNEL (edges for that node)
end
DS->>CO: FINISH_BATCH_UNREGISTER
Note over CO: completeBatchOperation → physical unplug
CO->>CO: completeBatchOperation
DS->>CO: AFTER_COMPONENTS_UNLOAD
Note over DS,C: loadComponents — each new instance
par each new instance
C->>C: initialize
C->>C: registerStreams (+ after registerStreams)
C->>CO: COMPONENT_REGISTER
end
DS->>CO: AFTER_COMPONENTS_LOAD
Note over CO: loadingTracker = 0
CO->>CO: startBatchOperation
CO->>CO: componentsReady plug task
CO->>CO: ingressController.pause()
Note over CO,C: plug task — per target, per batch
CO->>C: RUNTIME_CHANNELS_INFO
opt removed edges only (map diff)
CO->>C: UNBIND_CHANNEL / UNBIND_GROUP_CHANNEL
end
Note over CO: commit unplug (diff), create edge links (events enqueue globally)
CO->>C: BIND_CHANNEL / BIND_GROUP_CHANNEL
Note over C: bindChannel (handler)
CO->>C: OLD_LIFECYCLE
Note over C: disconnectStreams → streamsConnected → streamsActivated
CO->>CO: ingressController.release()
CO->>CO: ALL_COMPONENTS_STREAMS_PLUGGED
CO->>CO: completeBatchOperation

Note over DS,C: App unload
DS->>CO: BEFORE_COMPONENTS_UNLOAD
DS->>CO: START_BATCH_UNREGISTER
par each instance
C->>CO: COMPONENT_UNREGISTER
CO->>C: UNBIND_CHANNEL (edges for that node)
end
DS->>CO: FINISH_BATCH_UNREGISTER
Note over CO: completeBatchOperation → physical unplug
DS->>CO: AFTER_COMPONENTS_UNLOAD

Wire buffering: events produced during BIND / activation hooks are enqueued into one plug-task activation buffer and released only after ingressController.release() (See 5).

2. Streaming lifecycles (component API)​

Coordinator actions (BIND, OLD_LIFECYCLE, …) are distinct from Dive hooks (registerStreams, streamsConnected, …). Hooks run inside the component when coordination messages arrive.

Component Sequence (one instance)​

PhaseWhereWhat runs
InitializeloadComponentsinitialize → registerStreams() (declares ports / channel IDs; after('registerStreams') on the component) → COMPONENT_REGISTER
UnregisterCOMPONENT_UNREGISTERunregisterStreams → COMPONENT_UNREGISTER; coordinator UNBIND_*
Plug batchCoordinator → ComponentruntimeChannelsInfo → unbindChannel (if any) → wire commit → bindChannel → v1: OLD_LIFECYCLE hooks or v2-only logic in bindChannel
ServicesServices graph plugSame through bindChannel, then servicesStreamsConnected ← ALL_SERVICES_STREAMS_CONNECTED, servicesStreamsActivated ← ALL_SERVICES_STREAMS_ACTIVATED, then OLD_LIFECYCLE v1 chain
Data emitAfter ingressController.release()Buffered wire events drain globally in FIFO order (See 5); safe to emit downstream on v1 after streamsActivated

NOTE: OLD_LIFECYCLE handler (with_coordination_helper.js) always calls three V1 methods in one synchronous step when the component defines them.

Old vs New API​

Old API (WithStreams)New API (WithStreamsV2 only)
MixinsWithStreamsV2 + WithStreamsV2toV1WithStreamsV2 only
registerStreamsBase map + v2toV1 before creates local bus per portRegisters channel IDs only
Wire attachCoordinator bindChannel → plug link into local busbindChannel plugs into link directly
ActivationstreamsConnected / streamsActivated via OLD_LIFECYCLEOften only bindChannel / runtimeChannelsInfo
Typical emitafter('streamsActivated') or handler bodybindChannel / constant on link
sequenceDiagram
participant CO as Coordinator
participant C as Component

Note over C: instance init (before plug)
C->>C: initialize
C->>C: registerStreams
Note over C: registerInputStream / registerOutputStream

Note over CO,C: plug batch (See 1)
CO->>C: RUNTIME_CHANNELS_INFO
Note over C: runtimeChannelsInfo
CO->>C: BIND_CHANNEL
Note over C: bindChannel
CO->>C: OLD_LIFECYCLE
Note over C: disconnectStreams
Note over C: streamsConnected
Note over C: streamsActivated
Note over CO: ingressController.release() — drain queued events
flowchart LR
subgraph init["Instance init"]
I[initialize]
RS[registerStreams]
I --> RS
end

subgraph plug["Plug batch (v1 WithStreams)"]
RT[runtimeChannelsInfo]
BC[bindChannel]
SC[streamsConnected]
SA[streamsActivated]
RT --> BC --> SC --> SA
end

init --> plug

NOTE: Graph delivery is gated by coordinator buffering (See 5), not by blocking component handlers.

3. Mixins​

WithStreamsV2 — Channel API: registerInputStream / registerOutputStream, bindChannel / unbindChannel, runtimeChannelsInfo. Coordinator owns wire streams (aka links).

WithStreamsV2toV1 — Bridge for legacy components: creates a local bus per port, maps V1-style getInputStream / getOutputStream onto those buses, plugs coordinator link in bindChannel. NOT a separate lifecycle, but adapter on top of V2.

WithStreams — WithStreamsV2 + WithStreamsV2toV1. Default for most data components.

WithCoordinationHelper — Subscribes to coordination stream; dispatches BIND_CHANNEL, OLD_LIFECYCLE (→ v1 hooks if defined), ALL_SERVICES_STREAMS_*, etc.

WithStreamStatus — Wire error/status types on streams.

4. Component examples​

ComponentAPILifecycle events usedPorts / emit
constant (array)New (WithStreamsV2)runtimeChannelsInfo, bindChannelOn bindChannel output: Kefir.constant([1,2]) if no trigger; ignores streamsActivated
combine-byOld (WithStreams)streamsActivated arms combinerInputs via graph after flush; peek reads live p1 buffer
crossfilter-aggregationMixed (V2 + V2toV1)streamsActivated / streamsConnected emitV2 channels + v1 buses; emit after activation hooks

5. Stream plug sequence and global event ordering​

The coordinator has two ordering concerns during componentsReady / servicesReady:

  1. Coordination actions must make components see a consistent topology: runtime info and removed-edge unbinds first, physical unplug, new-edge binds, activation hooks.
  2. Data events produced by those binds and hooks must not reach downstream components until every updated component has received its activation actions.

Plug task order​

Each plug task pauses the coordinator's shared GlobalIngressController before changing edges. The task then:

  1. Builds runtime-channel info and diffs the previous coordination map against the new map.
  2. Sends runtime info plus target-side unbinds, then source-side unbinds, then other unbind actions.
  3. Calls _commitUnplug() so old physical Kefir plugs are removed after components were told to detach.
  4. Creates new edge links with _plugSimpleStream / _plugGroupStream; each direction gets a source-side target bus or pool, a target-side source pool, a bufferedTarget emitter, optional transformations, and BIND_* actions.
  5. Waits for transformation attachment promises, appends service activation actions or component OLD_LIFECYCLE, updates component status, and sends the bind/activation batch.
  6. Calls ingressController.release(). Component plug tasks then broadcast ALL_COMPONENTS_STREAMS_PLUGGED.

orderedEdgeIds still controls edge creation order for component graphs, but the activation buffer means startup data delivery no longer depends on that order.

Event pipeline​

_getBufferedStream subscribes directly to the upstream link and queues delivery callbacks, not intermediate streams:

flowchart LR
OUT[component/service output link] --> TI[onAny ingress callback]
TI -->|assign global index| G[GlobalIngressController]
G --> D[edge delivery callback]
D --> BS[bufferedTarget emitter: status + metadata]
BS --> TR[wire guard and optional transformations]
TR --> IN[target input link]

While paused, the controller queues { target, event } envelopes for connected edges. GLOBAL_EVENT_NUMBER is assigned before enqueue, preserving production order in debugger metadata. Delivery rechecks pause/safe-mode and whether the wire is still alive. Ending bufferedTarget removes its upstream listener, and already queued deliveries to that ended wire are ignored.

release() drains the queued envelopes in FIFO order. New events emitted synchronously during that drain are delivered immediately, before the next queued envelope; this reentrant behavior is intentional.

The wire guard handles debugger injection, loop/burst protection, activity reporting, and pre-transformation publishing. For zero transformations it also publishes the post-transformation event, avoiding a separate observable. Real transformation chains retain a post-transformation wrapper. A simple edge without transformations creates five observables, including the bus's internal pool.

RPC groups filter responses with one stream backed by two listeners and a set of active multi-query IDs. Query initialization adds an ID, query end removes it, and response values/errors pass only for active IDs, broadcasts, or one-shot responses. Unsubscribing releases both listeners and clears the set. An RPC group without transformations creates nine observables.

Legacy note: in original version of activation, each edge had its own bufferWhileBy(syncProp). Opening the gate flushed each edge independently, so cross-edge ordering could become depth-first by wire rather than FIFO by production time. The integration spec covers the reproduced array → flatten → fork → combine-by graph in both exported edge order and combine-edge-first order; both must produce combine-by log values [1, 2].