Async API billing: integrator contract + Railway worker

High-volume pilot path: the HTTP accept loop never waits on Groth16.

PostgreSQL is the durable source of truth before HTTP 202 (vdi_billing_replay_guard reserved + vdi_billing_proof_jobs accepted). Redis/BullMQ is only the prove work queue.

Recovery loop (sweeper + leases)

Failure	Behavior
Crash after DB insert, before enqueue	Job stays `accepted`; sweeper claims expired/null lease and enqueues
Enqueue OK, client never got 202	Retry same `event_id` → `idempotent_replay: true`, same `job_id` / `temporary_receipt_hash`
Worker crash mid-proof	`proving` lease expires; sweeper re-drives; replay commit is idempotent
Redis flushed	Jobs remain in Postgres; sweeper reconstructs BullMQ
Two sweepers	`FOR UPDATE SKIP LOCKED` → one owner per job
Webhook twice	Same `job_id` + `temporary_receipt_hash` (+ `x-quantzk-job-id` / `x-quantzk-receipt-hash` headers)

Worker runs the sweeper on an interval (VDI_BILLING_PROOF_SWEEP_MS, default 5s). Lease TTL: VDI_BILLING_PROOF_LEASE_SEC (default 120). After a successful sweeper enqueue, the short-lived sweep claim is cleared so a prove worker can acquire immediately.

bash

cd verifier-api
DATABASE_URL="postgresql://zkcaptcha:development@localhost:5432/zkcaptcha" \
npm run test:billing-async-recovery

Chaos evidence (worker kill + Redis obliterate)

Production-shaped harness: accept N jobs → SIGKILL worker → obliterate BullMQ → restart worker + sweeper → assert every Postgres-accepted job reaches exactly one final receipt.

bash

cd verifier-api
DATABASE_URL="postgresql://zkcaptcha:development@localhost:5432/zkcaptcha" \
REDIS_URL=redis://127.0.0.1:6379 REDIS_ENABLED=true REDIS_DISABLED=false \
npm run chaos:billing-async

Latest PASS write-up: Async billing chaos evidence.

When to use this vs sync attest

Mode	Use when	Latency
Async (`async: true`)	High-volume API metering, enterprise pilots	Accept ~10–30ms; prove seconds later
Sync (default body)	Demos, low volume, need attestation in one response	Warm hotpath ~300–700ms

Async still runs hotpath prove (VDI_BILLING_PROOF_MODE=hotpath) in the worker unless you set full for dispute-grade causal proofs.

Integrator contract

1. Accept (sync)

POST /api/vdi/billing/attest

Enable async with any of:

Body: "async": true or "mode": "async"
Query: ?async=true
Header: Prefer: respond-async
Env default: VDI_BILLING_ATTEST_ASYNC=true

Optional: "webhook_url": "https://your-app/hooks/billing-proof"

bash

curl -sS -X POST "https://YOUR_API/api/vdi/billing/attest" \
  -H "Content-Type: application/json" \
  -H "X-VDI-Attest-Secret: $VDI_ATTEST_SECRET" \
  -d '{
    "billing_version": "v2",
    "async": true,
    "webhook_url": "https://your-app/hooks/billing-proof",
    "profileId": "VDI_VERIFY_STANDARD_V1",
    "meter_event": {
      "event_id": "evt_pilot_async_001",
      "event_timestamp": "2026-05-12T12:00:00.000Z",
      "tenant_id": "tenant_acme",
      "api_key_id": "api_live_001",
      "endpoint": "/v1/completions",
      "usage_units": 1842,
      "contract_version": "contract-v1",
      "billing_window_id": "2026-05"
    },
    "tariff_snapshot": {
      "tariff_id": "tariff_default",
      "tariff_version": "1.0.0",
      "tariff_class": "usage_linear",
      "unit_price_micros": 120,
      "min_charge_micros": 50000,
      "max_charge_micros": 5000000000,
      "rounding_mode": "floor",
      "currency": "USD"
    }
  }'

HTTP status: 202 Accepted

Response (minimum fields):

Field	Meaning
`status`	`"accepted"`
`job_id`	Opaque id, e.g. `vdi:job:0x…`
`status_url`	Relative poll path
`temporary_receipt_hash`	`0x` + sha256 hex of the pending receipt body
`temporary_receipt`	Pending receipt object (schema `api-billing-v2-pending-receipt-v1`)
`queue_transport`	Must be `"bullmq"` in production (not `"inline"`)
`proof_mode`	Usually `"hotpath"`
`accept_time_ms`	Server-side accept duration
`billing.expected_charge_micros`	Deterministic charge already computed
`billing.public_commitments_digest`	Commitment digest pinned at accept

Replay is reserved at accept. Duplicate event_id for the same tenant/window fails closed.

2. Poll job status

GET /api/vdi/billing/attest/jobs/:jobId

States: accepted → proving → completed | failed

bash

curl -sS "https://YOUR_API/api/vdi/billing/attest/jobs/$JOB_ID" | jq .

On completed:

result contains the full sync-style attest payload (attestation, verification, receipt, billing, stage timings)
webhook.delivered / webhook.status reflect delivery attempt

On failed:

error_detail explains prove/verify failure
Replay reservation is released by the worker path on failure after accept-time reserve (treat failed jobs as needing a new event_id or operator recovery)

3. Webhook (optional)

When webhook_url is set, the worker POSTs JSON after prove completes or fails.

Headers:

Header	Value
`Content-Type`	`application/json`
`x-quantzk-event`	`billing.proof.completed` or `billing.proof.failed`
`x-quantzk-signature`	`hex(HMAC-SHA256(VDI_BILLING_WEBHOOK_SECRET, raw_body))`

Completed body (shape):

json

{
  "event": "billing.proof.completed",
  "job_id": "vdi:job:0x…",
  "temporary_receipt_hash": "0x…",
  "status": "completed",
  "attestation_id": "vdi:att:0x…",
  "generation_time_ms": 412,
  "proof_mode": "hotpath",
  "billing": {
    "expected_charge_micros": 221040,
    "public_commitments_digest": "0x…"
  }
}

Verify signature before trusting the payload. Then either:

Poll status_url and take result for full attestation/receipt, or
Call POST /api/vdi/billing/verify with the completed attestation/receipt (same as sync pilot)

4. Failure modes (integrator-facing)

Symptom	Likely cause	Action
`400` on accept, replay message	Duplicate `event_id`	New event id or wait for window
`503` / enqueue error mentioning Redis	Redis down or `REDIS_ENABLED=false` without inline allow	Fix Redis; never enable inline in prod
Job stuck `accepted`	No worker process	Start `worker:billing-proof`
Job `failed`	Prove/artifact/signing error	Read `error_detail`; check circuit artifacts + keys
Webhook missing, job `completed`	Bad URL, TLS, or secret mismatch	Check `webhook` fields on job; fix endpoint

Railway deploy checklist (API + worker)

Use two Railway services from the same verifier-api root so prove never shares the API event loop.

A. Prerequisites

[ ] PostgreSQL linked (DATABASE_URL)
[ ] Redis linked (REDIS_URL)
[ ] Migrations applied in order: 030, 031, 032, 033 (jobs + leases/sweeper)
[ ] Circuit artifacts available to the worker (apiBillingV2 under circuits/ or VDI_BILLING_INCIRCUIT_DIR)

bash

psql "$DATABASE_URL" -f verifier-api/src/infrastructure/database/migrations/030_vdi_billing_v2.sql
psql "$DATABASE_URL" -f verifier-api/src/infrastructure/database/migrations/031_vdi_billing_replay_crash_safe.sql
psql "$DATABASE_URL" -f verifier-api/src/infrastructure/database/migrations/032_vdi_billing_proof_jobs.sql
psql "$DATABASE_URL" -f verifier-api/src/infrastructure/database/migrations/033_vdi_billing_proof_job_leases.sql

B. Shared secrets (both services)

Variable	Notes
`DATABASE_URL`	Same Postgres
`REDIS_URL`	Same Redis (Railway: link Redis service → `REDIS_URL`)
`REDIS_ENABLED`	`true`
`REDIS_DISABLED`	`false`
`NODE_ENV`	`production`
`VDI_ATTEST_ENABLED`	`true`
`VDI_ATTEST_SECRET`	Required on API attest
`VDI_SIGNING_KEY`	64+ hex, stable
`VDI_BILLING_METER_PRIVATE_KEY_PEM` / `VDI_BILLING_METER_KID`	Or trust JSON for external envelopes
`VDI_BILLING_LOG_OPERATOR_PRIVATE_KEY_PEM` / `VDI_BILLING_LOG_OPERATOR_KID`	Transparency
`VDI_BILLING_PROOF_MODE`	`hotpath` for pilots
`VDI_BILLING_WEBHOOK_SECRET`	HMAC for webhooks
`VDI_BILLING_ASYNC_ALLOW_INLINE`	`false`
`VDI_BILLING_ASYNC_WORKER_IN_API`	`false`
`VDI_BILLING_ASYNC_CONCURRENCY`	Start at `1`; raise carefully

Optional: VDI_BILLING_ATTEST_ASYNC=true to default all v2 attests to async.

C. Service 1: API gateway

Root directory: verifier-api
Start command: npm start (or node src/app.js)
Config reference: verifier-api/railway.json
Do not run the proof worker here

D. Service 2: billing proof worker

Root directory: verifier-api (same repo)
Start command: npm run worker:billing-proof
Config reference: verifier-api/railway.billing-worker.json
Scale replicas horizontally; they share the BullMQ queue on Redis
No public HTTP port required

Railway UI steps:

Project → New → GitHub Repo (or duplicate service) → same repo, root verifier-api
Settings → Custom Start Command → npm run worker:billing-proof
Variables → copy/link the shared set above (especially REDIS_URL, DATABASE_URL, signing/meter keys)
Deploy; logs should show: [billing-proof-worker] listening on queue vdi-billing-proof

E. Live smoke (post-deploy)

Async attest with async: true → expect 202, queue_transport: "bullmq", accept_time_ms typically under 50ms
Poll job until completed
Confirm webhook received (if configured) and HMAC verifies
Kill/restart the API mid-queue: job must still complete on the worker
Optional: POST /api/vdi/billing/verify on the completed attestation

F. SLO starters

Signal	Pilot target	Prometheus metric
Accept p95	< 50ms	`quantzk_billing_async_accept_seconds`
Queue depth	Alert if growing unbound	`quantzk_billing_async_queue_depth{state=waiting\|active\|delayed\|failed}`
Prove lag p95	Budget for your tariff (seconds, not minutes)	`quantzk_billing_async_prove_lag_seconds`
Webhook success rate	> 99% after retries	`quantzk_billing_async_webhook_total{result=success\|failure}`
Job lifecycle	Accepted / completed / failed / idempotent	`quantzk_billing_async_jobs_total`
Replay rejects	Expected for duplicates; spike may mean client bugs	app logs + replay_guard

Scrape the existing metrics endpoint (see Monitoring dashboards).

G. Live smoke (script)

bash

API=https://api.quantzk.com VDI_ATTEST_SECRET=... \
  bash scripts/smoke-billing-async.sh

Local chaos (worker kill + Redis obliterate): cd verifier-api && npm run chaos:billing-async — evidence.

Local two-process smoke

bash

# Terminal A: API (enqueues only)
cd verifier-api
REDIS_URL=redis://127.0.0.1:6379 REDIS_ENABLED=true REDIS_DISABLED=false \
VDI_BILLING_ASYNC_WORKER_IN_API=false VDI_BILLING_ASYNC_ALLOW_INLINE=false \
VDI_BILLING_PROOF_MODE=hotpath \
node src/app.js

# Terminal B: worker
cd verifier-api
REDIS_URL=redis://127.0.0.1:6379 REDIS_ENABLED=true REDIS_DISABLED=false \
VDI_BILLING_PROOF_MODE=hotpath \
npm run worker:billing-proof

Async API billing: integrator contract + Railway worker ​

Recovery loop (sweeper + leases) ​

Chaos evidence (worker kill + Redis obliterate) ​

When to use this vs sync attest ​

Integrator contract ​

1. Accept (sync) ​

2. Poll job status ​

3. Webhook (optional) ​

4. Failure modes (integrator-facing) ​

Railway deploy checklist (API + worker) ​

A. Prerequisites ​

B. Shared secrets (both services) ​

C. Service 1: API gateway ​

D. Service 2: billing proof worker ​

E. Live smoke (post-deploy) ​

F. SLO starters ​

G. Live smoke (script) ​

Local two-process smoke ​

Related docs ​

Async API billing: integrator contract + Railway worker

Recovery loop (sweeper + leases)

Chaos evidence (worker kill + Redis obliterate)

When to use this vs sync attest

Integrator contract

1. Accept (sync)

2. Poll job status

3. Webhook (optional)

4. Failure modes (integrator-facing)

Railway deploy checklist (API + worker)

A. Prerequisites

B. Shared secrets (both services)

C. Service 1: API gateway

D. Service 2: billing proof worker

E. Live smoke (post-deploy)

F. SLO starters

G. Live smoke (script)

Local two-process smoke

Related docs