FrootAI.dev AgentSpace
On this page 5
Operate Worker + D1

Execution Flight Recorder & replay

Observe every remote work-linked execution phase, enforce secret-free telemetry, cancel in-flight jobs safely, and replay failed runs.

Status: Available Architecture: O2 · Operations Read: 6 min Verified: 2026-11-16

Work execution admission invariants

When a work item triggers a remote connector or model execution via POST /api/v1/tenants/:tenantId/work/:workId/executions, D1 enforces strict admission invariants inside a single transaction before any message reaches Cloudflare Queues:

  • Ready connector check: The target connector must belong to the tenant, be in ready state, and target an exact allowlisted HTTPS origin.
  • Single active attempt per work item: At most one queued, running, or retrying job can exist per (tenant_id, work_id) pair.
  • Tenant concurrency cap: Active executions across a tenant cannot exceed its execution policy cap (default 4 concurrent runs).
  • Transactional outbox: Job state and outbox payload commit atomically in D1 before Queue dispatch, preventing orphaned queue messages.

Ordered Flight Recorder timeline

Every admitted job emits ordered, trace-correlated events retrievable via GET /api/v1/tenants/:tenantId/jobs/:jobId/events:

{
  "jobId": "job_01jcva89x2",
  "traceId": "trc_7b31e9a0",
  "events": [
    { "seq": 1, "phase": "admission", "status": "queued", "attempt": 0 },
    { "seq": 2, "phase": "outbox_dispatch", "status": "published", "attempt": 0 },
    { "seq": 3, "phase": "worker_claim", "status": "running", "attempt": 1 },
    { "seq": 4, "phase": "connector_call", "status": "succeeded", "attempt": 1, "httpStatus": 200, "durationMs": 412 },
    { "seq": 5, "phase": "completion", "status": "succeeded", "attempt": 1 }
  ]
}

Zero-secret telemetry boundary

Operational metadata only

Flight Recorder events and structured Worker logs record tenant ID, job ID, trace/span IDs, attempt counts, HTTP status codes, latencies, and non-secret error codes. They never persist raw prompts, secret values, Authorization headers, queue payload bodies, or connector response bodies.

Audited cancellation and late-result discard

Operators can cancel an active execution at any time with POST /api/v1/tenants/:tenantId/jobs/:jobId/cancel:

  • Queued jobs: Immediately marked cancelled; queue consumers refuse to claim them when the message arrives.
  • Running jobs: Cancellation intent is persisted atomically. If an in-flight outbound connector request returns late, its result is discarded and the job transitions cleanly to terminal cancelled state with a Flight Recorder event.

Dead-letter recovery and immutable replay

Transient failures retry automatically on the same job with exponential backoff until the attempt ceiling is reached, at which point the job enters dead_letter.

To recover after fixing an upstream connector or quota issue, call POST /api/v1/tenants/:tenantId/jobs/:jobId/replay:

  • Replay is permitted only for terminal cancelled or dead_letter work-linked jobs—replaying a succeeded job is rejected with 409.
  • The original failed job remains immutable in audit history; replay creates a new job ID linked to the source via replay_of_job_id.
Verified against repository suites
tests/backend/jobs.test.tstests/backend/audit.test.ts
Markdown specification

Execution Flight Recorder & replay