Durable Execution Model
Rhythm introduces a "Resumable VM" model for durable execution. Unlike traditional frameworks that rely on event sourcing (replaying a history of events to rebuild local state), Rhythm captures the actual execution state of the workflow engine and persists it directly to PostgreSQL.
The Resumable State Model
In Rhythm, a workflow is not just a function—it is a managed process running inside an embedded scripting engine. When a workflow reaches an await point (such as a Task execution, a Sleep timer, or a Signal), the engine:
- Suspends the current execution.
- Serializes the virtual machine's stack, local variables, and instruction pointer.
- Persists this state to the
executionstable in the database. - Resumes execution exactly where it left off once the dependency (e.g., the Task result) is available.
Advantages over Replay
By persisting the VM state instead of replaying events, Rhythm eliminates several common "durable execution" pitfalls:
- No Determinism Requirements: You don't have to worry about using non-deterministic functions (like
Math.random()or date lookups) within your workflow logic. Because the state is saved, the engine never needs to "re-run" the same code to get back to the current point. - No Event Limits: High-iteration loops that would crash a replay-based system (due to event history size limits) are handled natively in Rhythm.
- Cognitive Simplicity: The code you write is the code that runs. There is no hidden "replay" phase where side effects must be carefully wrapped.
Workflow Lifecycle
Workflows transition through several states defined by the ExecutionStatus enum:
| Status | Description |
| :--- | :--- |
| Pending | The workflow is created but not yet picked up by a worker. |
| Running | An active worker is currently executing the workflow instructions. |
| Suspended | The workflow is waiting for an external event (Task completion, Signal, or Timer). The state is safely stored in the DB. |
| Completed | The workflow has reached a return statement or the end of the script. |
| Failed | The workflow encountered an unhandled error or a Task failure. |
Script Sandboxing and Versioning
Rhythm executes workflows using a custom scripting language (a subset of JavaScript) defined in .flow files. This separation provides a clean boundary between Orchestration (the workflow) and Execution (the tasks).
Content-Hash Versioning
Workflow definitions are immutable and self-versioning. When a workflow is registered, Rhythm generates a version hash based on the script content.
- In-flight safety: If you deploy a new version of a workflow, existing executions continue running on the specific version hash they started with.
- Zero-downtime migrations: You can safely modify workflow logic without worrying about breaking currently running "long-lived" processes.
Example: Suspension Points
In the following example, the VM state is persisted to the database at every await keyword.
// workflows/shipment_flow.flow
// 1. VM State is saved here while waiting for the task
let label = await Task.run("generate-shipping-label", { orderId: Inputs.orderId })
// 2. State is saved again for a 24-hour duration
await Timer.sleep("24h")
// 3. State is saved until an external signal is received via SignalService
let confirmation = await Signal.when("carrier-pickup-confirmed")
return { trackingId: label.id, confirmed: true }
Execution Components
The model relies on three primary components within the core engine:
1. The Execution Service
Manages the lifecycle of an execution. It handles the initial creation of the workflow entry and provides methods to query status.
2. The Scheduler Service
Handles delayed executions. If a workflow is suspended via Timer.sleep(), the Scheduler Service is responsible for moving the execution back into the work queue once the run_at timestamp has passed.
3. The Signal Service
Allows external systems to interact with a suspended workflow. Sending a signal triggers the engine to resolve the Signal.when() call and resume the VM.
# Example of interacting with the Durable Model via the Python Client
from rhythm import Client
client = Client("postgresql://...")
# Signals the engine to resume a specific suspended workflow
await client.send_signal(
workflow_id="order_123",
signal_name="carrier-pickup-confirmed",
payload={"pickup_time": "2023-10-27T10:00:00Z"}
)