Skip to Content
API (apps/api)Background Workers

Background workers

Nine of the files in apps/api/src/entries/ are not HTTP Lambdas at all: they are SQS consumers or EventBridge-scheduled workers. They live in the same app because they share the persistence layer, the failure protocol, and the RDS pools with the HTTP entries — but they are invoked by event-source mappings or cron rules wired in AppApiStack (see /infra), never by API Gateway, because webhook drains, reminder sweeps, and cleanup reapers must run on their own retry/DLQ semantics instead of a request/response timeout.

The nine workers

EntryTriggerWhat it does
plaid-sync-worker.tsSQS (opsdna-plaid-webhook)Drains verified Plaid webhooks; syncs transactions; promotes versions to candidate operating-source-events
collection-acceptance-worker.tsqueueCommits accepted portco share-up grants into the GP tenant, via SET ROLE into the least-privilege opsdna_collection_acceptance_worker role
collection-reminder-worker.tsEventBridge cronMaterializes each cycle’s reminder_policy into due reminders in sharing.notification_outbox, using the same computeReminderSchedule fn as the portal preview (“preview == scheduler equality”)
kpi-stale-watchlist-worker.tsEventBridge cronMirrors overdue expected KPI periods into sharing.inbox_item as stale_metric actions
notification-dispatch-worker.tsEventBridge cronClaims due outbox rows tenant-by-tenant under normal tenant RLS and hands them to a transport; prod is dry-run by default, staging sends via SES with a recipient override
recognition-sweep-worker.tsEventBridge cron (daily)Drafts deferred-recognition journal entries into the maker-checker inbox — never posts silently
source-document-ingestion-drain-worker.tsEventBridge cronClaims due post-clean ingestion jobs for the stages it has handlers for (e.g. mime_sniff)
source-document-ingestion-reaper-worker.tsEventBridge cronRe-queues/dead-letters stale running jobs; self-heals scan_pending documents from the object’s durable GuardDuty scan tag
source-document-upload-cleanup-worker.tsEventBridge cronExpires abandoned upload items and deletes uncommitted candidate objects

The three source-document workers and the upload cleanup are cross-tenant sweeps — they connect via the dedicated maintenance role pool, not opsdna_app (see Database access and /source-pipeline).

SQS pattern: partial batch response

From entries/plaid-sync-worker.ts — each record is processed independently, and only the failed message IDs are returned, so one bad message is retried/DLQ’d without re-delivering the whole batch:

export const handler: SQSHandler = async (event: SQSEvent): Promise<SQSBatchResponse> => { const deps = await buildDeps(); // memoized per warm container (async secret fetch) const batchItemFailures: { itemIdentifier: string }[] = []; for (const record of event.Records) { try { const message = JSON.parse(record.body) as PlaidWebhookMessage; await processPlaidWebhook(message, deps); } catch { batchItemFailures.push({ itemIdentifier: record.messageId }); } } return { batchItemFailures }; };

Dependencies are built once per warm container and memoized as a promise (depsPromise), because the Plaid secret comes from an SSM SecureString at runtime (PLAID_SECRET_PARAM).

Pitfalls

  • Config throws at cold start. readConfig() throws if PLAID_CLIENT_ID / PLAID_SECRET_PARAM are missing or PLAID_ENV isn’t sandbox/production — the worker then fails every batch until env is fixed.
  • Don’t swallow-and-succeed. Catching an error without adding the record to batchItemFailures acknowledges the message — it is gone from the queue forever. Terminal observation belongs to the worker runtime (ADR-0041), not ad-hoc logs.
  • Wrong role, empty sweep. A cross-tenant worker on the RLS-scoped opsdna_app pool sees (and writes) nothing — no error, just a silently useless sweep. Use the maintenance pool for cross-tenant work.
Last updated on