Automation Engine Audit

Audit date: 2026-08-09. Method: static call-site census across apps/storefront, cross-referenced against prisma/schema.prisma, vercel.json cron schedules, and a targeted Vitest run.

Status: all five phases complete (2026-08-10). 52 of 52 live automations function today and 51 carry a passing test; the one that does not is manual by design. The sections below are kept in the order they were written — the audit as found, then each phase as it shipped — so the corrections stay visible next to the claims they correct. The last of those corrections is the largest: the payment fact was missing on four settlement paths.

One decision is still open: the money gate.

Headline

As found on 2026-08-09. The starting assumption for this audit was that the automation layer was designed but never built. That is not what the code showed. Roughly 87% of the automation surface had code behind it and ~72% was wired to a real trigger and worked — order confirmations fired on three of four payment paths, shipped/delivered emails fired off the EasyPost tracking webhook, and the operations sweep genuinely found stale orders, aging returns and stuck webhooks.

The real problem is narrower and more interesting than "nothing exists":

lib/domain-events has 13 producers and zero consumers. Sixteen business facts are recorded to domain_events on every order, payment, refund, shipment and inventory movement — and nothing reads them. getEntityTimeline() has no callers; prisma.domainEvent appears exactly once outside the emitter.

Every automation in the repo is instead hand-wired into the individual route that causes it. That is precisely why some went dead without anyone noticing, and why the same automation is implemented three times on three payment paths and forgotten on the fourth.

The second finding follows from the first:

enrollInAutomation() has zero callers. The EmailAutomation / AutomationStep / AutomationEnrollment schema, the 17-value AutomationTrigger enum, the admin builder at app/admin/email-marketing/automations, the CRUD API, and the 5-minute email-automation cron that drains due steps — all of it exists and is correct. Nobody is ever enrolled, so the cron drains an empty queue forever.

One change — a consumer layer over emitDomainEvent that routes events into enrollInAutomation — resurrects the entire marketing-automation product and fixes the structural cause of the dead spots.

Two things checked first, because either would have changed the report

1. Are the sub-daily crons actually running in production? Yes. Commit ec71426d (2026-08-07) restored */5 schedules "now that the project is on Vercel Pro." Pro allows per-minute crons; the earlier Hobby-era daily downgrade (e8ae9749) no longer binds. All 8 scheduled crons are valid.
2. Do the tables exist in the deployed databases? Yes. domain_events ships in migration 20260807120000_add_fulfillment_channel_domain_events; notifications and its notifications_userId_dedupeKey_key unique index ship in 20260317000000_baseline and 20260807190000_notification_centre. This mattered because emitDomainEvent swallows every error by design — a missing table would have been silently invisible.

State in production

Merged and deployed 2026-08-10 (f297f02a). All seven migrations were applied to the production database ahead of the deploy and verified by querying it directly; the five new columns are present, email_webhooks is gone, and no migration is unfinished. The site returns 200, and an unauthenticated GET to a cron route returns 401 — the fail-open guard fix, confirmed on the deployed system rather than asserted.

One honest limit on all of this. Production has taken 8 orders in its entire history, the most recent on 2026-05-12. The real business still runs on the legacy BigCommerce storefront while this platform is built toward cutover. So domain_events in production is empty, and will stay empty until a real order arrives — which is correct, not broken: no payment, no fact, no event.

That means every automation here is deployed and armed, not observed working in production. The confidence behind the percentages below came, at the time of that check, from 1618 passing tests across the repo — the automation suite alone has since gone from 137 to 530 — from exercising the drain and the lifecycle sweep against the dev database, and from black-box probes of the deployed routes. It does not come from watching a live order flow through, because there has not been one. Stating that plainly matters more than a cleaner-sounding claim: the first real order after cutover is the true test, and the thing to watch is whether domain_events starts filling and draining.

Rubric

Five states, not two. "Built" and "works" are different questions, and the gap between them is the report.

Inventory

A. Customer order communications

B. Operator notifications

C-0. Infrastructure

C. Email marketing automation engine

D. Cart, inventory & purchasing

E. Fulfillment, shipping, financial, social, identity

The three percentages

Across 52 live automation candidates. The list began at 53; Phases 1 and 5 added two (the consumer layer, item 54, and the registration fact, item 55) and three were removed rather than built (items 29, 31 and 43), which is where 52 comes from. Item 21a is a correction note rather than a separately scored candidate.

The one row not counted as proven is item 37, shipping label purchase, which is manual by design — there is no automation to prove. Its route is covered by tests all the same.

A third correction: the VERIFIED column was undercounting

Phase 5 added tests for two automations (items 11 and 30) but the figure moves by ten, because a re-audit found eight more that were already covered and never credited — the operations sweep (aging.test.ts, 31 cases), stock decrement and alert creation (inventory-manager.test.ts, 44), purchase-order receiving (31), QuickBooks sync (21) and processor fees. The original audit counted only tests it had traced by hand, so 15% understated the real starting position. The honest reading is that this codebase was better tested than the first pass credited — the same direction as the headline finding.

The denominator moved twice. Item 43 was a dead stub, deleted at Phase 4 rather than scored, and items 29 (EmailWebhook) and 31 (the duplicate abandoned-cart sender) were removed with sign-off once the audit had recorded them — none of the three was a missing automation.

A fourth correction: the fact itself was missing on four settlement paths

This is the most consequential correction in the audit, because it invalidates the evidence behind several rows that were scored as working.

Every consumer built in Phases 1–4 hangs off payment.completed. Four of the eight routes that mark an order paid never emitted it. checkout/complete (the main storefront path), checkout/paypal/capture-order, checkout/square/process-payment and gift-certificates/complete each set paymentStatus: PAID and recorded nothing. The webhooks that do emit then return early on "already paid" — so whenever the checkout route won the race, which is the ordinary case because the browser calls it the moment payment confirms, the fact was lost for that sale.

The consequences were not subtle. On those paths nobody was enrolled in an automation, the shop raised no new-order notification, no high-value alert fired, participant milestones did not count the sale, and order rules did not evaluate. On checkout/complete the customer also received no order confirmation at all, because that route never sent one and the webhook that would have was skipping the order as already paid.

So items 20, 21, 21a, 22, 24 and 47 were scored VERIFIED against an event that, on the busiest paths, was never emitted. Their handlers were correct and their tests passed; they were simply never called. All four routes now emit inside the transaction that marks the order paid, and tests/lib/domain-events/payment-fact-parity.test.ts enumerates every settlement route and fails if one appears without the emit.

Exactly one event is emitted per sale in either race order: whichever side commits first emits, and the other returns early on isPaid. Where a duplicate is still conceivable, the consumers absorb it — the confirmation is idempotent through confirmationEmailSentAt, notifications through their dedupe keys, and milestones through the marker written when one fires.

Correction to the original audit

The audit recorded the admin new-order email as wired on the Stripe path. It was not wired at all. Its only caller is lib/stripe/webhooks.ts, and nothing imports that module — it is dead code. So a new order notified nobody, by any mechanism, on any payment path: notifyAdminsOfNewOrder had no callers and would have returned early anyway (it reads OrderNotificationSetting, a table with no rows, no seed and no UI), and the email sender was unreachable. Phase 2 fixes all three at once.

The gap between 87% built and 72% working is the 8 orphaned automations — finished code with no caller. The gap between 72% working and 15% verified is the real risk: most of what works is working unwitnessed, and would break silently.

Test baseline: tests/lib/domain-events, tests/lib/notifications, tests/lib/orders/fulfillment, tests/lib/inventory, tests/lib/cron, tests/unit/tracking → 9 files, 137 tests, all passing.

Design decision that needs a signature: the money gate

"Automate everything" has to carve out actions that spend money or are hard to reverse: buying postage, issuing refunds, sending mass marketing email, publishing to social.

The proposed rule is automate the preparation, gate the commit:

• Auto-prepare, one-click commit — rate-shop the parcel, pre-fill the cheapest service, and surface a "Buy label" button. Do not auto-purchase postage.
• Fully automatic — anything that only writes to our own database or sends a transactional email the customer is expecting (confirmation, shipped, delivered, receipt).
• Threshold-gated — refunds under an operator-set ceiling can auto-issue; above it, queue for approval.

Item #37 (shipping label purchase) is currently manual, and under this rule it stays manual. That is a recommendation, not a finding — it needs an explicit decision.

Plan

Each phase is one commit, ships with a Vitest regression test, and is gated on an explicit "proceed".

Phase 1 — The consumer layer ✅ shipped

lib/domain-events/subscribe.ts holds a registry mapping DomainEventType → handlers, and lib/domain-events/handlers/automation-enrollment.ts registers the first consumer: domain event → enrollInAutomation. Items 24, 25 and 26 are live — the marketing-automation product, its admin UI and the 5-minute cron all now do work.

Design changed during implementation, and the reason matters. The plan said handlers would be invoked from emitDomainEvent after the write. They are not: emitters pass their own transaction client — app/api/webhooks/stripe marks a payment failed and emits inside one $transaction — so invoking a handler at emit time would act on work that can still roll back, and would hold a database transaction open across email sends. Instead domain_events became a transactional outbox: a consumedAt column, and a poller that only ever sees committed rows. The drain runs at the head of the existing email-automation cron, so no new schedule was added.

Delivery is at-least-once and handlers must be idempotent — an event is marked consumed only after its handlers finish, so a crash mid-batch replays rather than silently dropping an enrollment. enrollInAutomation already skips anyone on an ACTIVE enrollment, so a replay is a no-op.

Before this runs anywhere

Migration 20260809120000_domain_event_outbox adds domain_events.consumedAt and must be applied before the drain does any work. It is additive — ADD COLUMN, a backfill, CREATE INDEX — and existing rows are backfilled to consumed so the first poll does not replay the entire order history.

The drain degrades to a no-op with a warning if the column is absent (isMissingColumnError, P2022 — the sibling of the existing P2021 handling). That is deliberate: vercel-build wraps prisma migrate deploy in a warning rather than a failure, so code can land ahead of its migration on a green deploy. Without the guard, an unapplied migration would throw at the head of the cron and take the step processor down with it. Apply the migration first regardless; the guard is a seatbelt, not a plan.

Triggers mapped

Triggers not yet mapped, and why

Two mappings from the original plan were deliberately not made, because each would have registered a handler that reads as working while never firing:

• ORDER_PLACED keys off payment.completed, not order.created. The checkout, PayPal and Square routes all create the order before taking payment, so order.created also fires for checkouts abandoned at the payment step — a post-purchase series keyed off it would email people who never bought anything. (Note: the comment at lib/orders/order-timeline.ts:138 claiming order.created is never emitted is stale — emitOrderCreated is called from 6 routes.)
• ORDER_REFUNDED is unmapped. Nothing emits payment.refunded or refund.completed today. The producer is Phase 3 work; mapping the trigger first would be a silent no-op.
• LOW_STOCK is unmapped. enrollInAutomation takes a customer email, and a low-stock fact has no customer — it is an operator concern, handled by notifyOperators (items 13–14). Routing it through the marketing engine would need an operator-recipient concept that does not exist yet.

Phase 2 — Close the orphans ✅ complete

Done — new-order notifications (items 20, 21, 21a). handlers/order-notifications.ts subscribes to payment.completed and raises the in-app notification, the high-value flag above $100, and the admin email. One handler covers Stripe, PayPal and Square — and any payment path added later — instead of a fourth hand-wired copy in each route, which is the pattern that produced the problem.

OrderNotificationSetting was not seeded as originally planned. That table has no rows, no seed and no UI to create one, so reading it would reintroduce the silent no-op this was meant to fix. The threshold is a documented constant matching the column's default, and notifications go through notifyOperators — the same dispatcher the operations sweep and inventory alerts already use, which needs no per-user row. notifyAdminsOfNewOrder, createOrderNotification and OrderNotificationSetting are consequently legacy; they are reported here, not deleted.

Done — ready-for-pickup (item 10). A local-pickup order got nothing: the shipped email comes from the EasyPost tracking webhook and a pickup order never has a label. handlers/pickup-ready.ts subscribes to order.fulfilled and filters out shipped orders in the same handler, so no customer is told both to expect a parcel and to come and collect it.

Done — participant milestone (item 47). Passing 5/10/25/50/100 sales sends the congratulation that had no caller. A new lastMilestoneNotified marker makes it replay-safe and collapses a burst of sales into one message; existing participants are backfilled so switching it on does not congratulate a whole roster retroactively.

Done — campaign launch and summary (items 48, 49). A coordinator set a fundraiser up and heard nothing when it went live, and nothing when it finished — including the totals, the one thing they need in order to hand money to a school. cron/fundraiser-lifecycle (daily) announces campaigns that have opened, closes campaigns whose end date has passed, and sends the closing summary.

This one is a sweep rather than an event consumer, and deliberately so: a campaign ending is a date passing. No route runs, no request is made, and nothing would ever emit an event for it. Since the sweep must exist for that, the launch announcement rides along instead of being wired into the admin route. Both are gated on sent-markers, and existing campaigns are backfilled so switching it on does not announce a fundraiser that started in March or summarise every campaign the shop has ever run.

Phase 2 complete. The one remaining orphan, item 31, is a duplicate rather than a gap: lib/email/automation.ts:384 is shadowed by the abandoned-cart cron's own local copy, so the working automation is unaffected. Per CLAUDE.md §3 it is reported here rather than deleted — removing it is a separate, explicitly-approved change.

Phase 3 — Close the parity gaps ✅ complete

✅ Complete. All three, including the one that was easy to drop:

1. Order confirmation on the POS / manual path (item 4) — a consumer on payment.completed and order.created, gated on confirmationEmailSentAt so it cannot double-send alongside the six hand-wired senders. The POS terminal also now emits payment.completed, which it never did; that single missing fact was why a counter sale raised no notification, enrolled nobody, and counted toward no milestone either.
2. Refund email (item 8) — reports what was actually refunded rather than the order total, and shares Order.refundEmailSentAt with the admin status-change sender so one refund cannot produce two emails.
3. The payment.refunded producer — emitted from lib/payments/refund.ts inside the refund transaction, on SUCCEEDED only. ORDER_REFUNDED is now mapped. Shipping the email without this would have left the trigger dead while looking finished. (The admin new-order gap originally listed here was closed in Phase 2; see the correction above.) These are the "same automation, forgotten on the fourth path" bugs that Phase 1 makes structurally impossible going forward, since a consumer fires for the fact rather than for one route.

Phase 4 — Build the missing engines ✅ complete

Order notification rules (item 22). The model had zero code references anywhere — a table describing which events go to which addresses and Slack channels, that nothing read. Now evaluated against the domain events. Deliberately separate from notifyOperators, which is the built-in operational floor every operator sees; these are rules an admin writes, to recipients who may not be operators at all. ORDER_CANCELLED is deliberately unmapped because nothing emits order.cancelled — a rule using it would sit in the table looking configured and never fire, the same mistake as mapping ORDER_REFUNDED before its producer existed.

Restock emails (item 34). createRestockNotification already computed urgency, days of stock remaining and a recommended order quantity — and had no callers, so a low stock level produced only an in-app badge saying a product was low, not what to do about it. Now hangs off inventory.low and inventory.out_of_stock with a seven-day per-product cooldown, because stock stays below threshold until someone restocks and the event fires on every sale that keeps it there.

Two corrections to the original audit here. Item 34 was recorded as a customer-facing "notify me when back in stock" and as SCHEMA-ONLY. It is neither: it is a staff restock recommendation, and the code existed with no callers, making it ORPHANED.

dashboard-analysis (item 43) — removed. Five lines returning {success: true}, referenced by nothing, not scheduled in vercel.json, and the only cron route with no authorisation check.

Phase 5 — Raise the verified floor ✅ complete

Every remaining WIRED row now carries a test. The count went 8 → 51 of 52, and the only row without one is item 37, which is manual by design.

It did not stay a testing exercise. Writing the payment-path tests turned up the fourth correction above: four of the eight routes that mark an order paid never emitted payment.completed, including checkout/complete — the busiest path in the application, and one the hand-traced census had missed entirely. The structural test found it, which is the argument for that style: a behavioural test only ever covers the route somebody remembered to write one for.

What the rest of the phase pinned, in rough order of what would hurt most if it broke:

• The confirmation stamps confirmationEmailSentAt only on success. Every caller and the domain event handler read that column as "this customer has been told", so stamping a failed send would suppress the confirmation permanently, silently, and only for orders that already hit an email problem.
• The automation engine's bookkeeping. Nothing called enrollInAutomation before Phase 1, so the engine had never run against a real enrollment. An enrollment that advances on a failed send skips a step nobody notices was skipped; one that fails to advance re-sends the same email every five minutes forever. Both directions are asserted, along with not rewinding a customer who is already mid-series and surviving a step whose template was deleted.
• Hard versus soft bounces. A permanent bounce suppresses the address; a full mailbox must not, or a real customer is dropped from every future email over a transient failure.
• Cron due-windows. scheduledAt <= now for campaigns and social posts, and the stalled-recipient cutoff that separates "a live run is working on it" from "the run died". Inverted or missing, each looks exactly like "nothing to do".
• Templates that ship placeholders. Substitution failures do not throw; they send {{TOKEN}} to the customer. The transactional senders now assert that nothing of the sort survives into anything a customer reads.
• Announcing what did not happen. The label route emits shipment.created only once postage is really bought, and nothing on any path that refused, failed, or was already labelled.

Working process

• One automation per commit, <Prefix>: <imperative summary> per CLAUDE.md Part 15.
• Every commit ships a regression test. Verification = targeted vitest run on the touched modules, plus lint and type-check. It is not full-suite green: npm run test has 8–9 pre-existing failures in tests/integration/* caused by remote-Neon latency and shared state, unrelated to this work.
• This table was the work queue. Rows moved ORPHANED → WIRED → VERIFIED and this page was updated in the same commit, so progress stayed visible and the work resumed cleanly across sessions. The queue is now empty.
• Dead code is reported, not deleted (CLAUDE.md Part 1 §3). OrderNotificationRule, EmailWebhook, and the duplicate abandoned-cart sender are listed above as findings; removing them is a separate, explicitly-approved change.