AI Layer & Evals

Model inference in OpsDNA is deliberately thin and deliberately measured. packages/ai provides one typed inference service; packages/evals proves, on every change, that the LLM-facing surfaces still behave. Together they close the loop on the engineering principle: LLM outputs stay constrained behind Zod schemas, registries, validators, and evals.

The inference layer (`packages/ai`)

Built on Effect AI with the OpenAI provider (@effect/ai-openai). The package exports an Effect service tag, typed errors, a model-purpose registry (model-purpose.ts — call sites declare why they need a model, not which model), structured-output schema helpers (structured-schema.ts), and a fixture layer so any consumer can run fully deterministic in tests.

One structural quirk worth knowing, documented in src/index.ts itself:


// packages/ai/src/index.ts
// The live OpenAI layer (`makeOpenAIEffectAIGenerationLayer`) and
// its options live behind the `./live` subpath because importing
// them eagerly evaluates `@effect/platform-node`'s NodeHttpClient,
// which initializes undici. [...]
// Production code that wants the live layer:
//   import { makeOpenAIEffectAIGenerationLayer } from "@opsdna/ai/live"

So: fixture/test code imports @opsdna/ai; live production layers import @opsdna/ai/live. The src/audited/ subtree adds an audited inference contract with a test provider and failure injection — inference calls whose inputs/outputs are traceable, and whose failure modes are exercisable in tests.

Where Bedrock fits: packages/ai is OpenAI-only today. AWS Bedrock inference (@ai-sdk/amazon-bedrock) lives in apps/chat-agent — the Cloudflare Workers + Durable Objects conversational agent runtime — not in the shared package. Keep that distinction in mind before “adding a provider”: the shared typed layer and the chat agent have separate inference stacks.

Consumers that matter for the runtime: packages/query-planner’s AI candidate extractor (opt-in via OPSDNA_ENABLE_LIVE_QUERY_AI=1 + OPENAI_API_KEY, otherwise a typed configuration gap) and packages/document-extraction. Either way, planner output is only ever a candidate — it must survive validation against the registries and the query runtime’s plan validator before anything executes.

The eval harness (`packages/evals`)

Six suites, one runner (src/runner.ts):

Suite	Root script	Exercises
extraction	`bun run evals:extraction`	document extraction quality
kpi-extraction	`bun run evals:kpi-extraction`	portfolio KPI extraction
economic-terms	`bun run evals:economic-terms`	LPA/side-letter economic term parsing
workflow-policy	`bun run evals:workflow-policy`	DomainPolicy behavior
scenarios	`bun run evals:scenarios`	end-to-end scenario runs
unknown-documents	`bun run evals:unknown-documents`	fail-closed handling of unrecognized docs

How to run

Everything


bun run evals:all

This dispatches to runEvalSuite("all"), which runs all six suites and folds their results into a single graded EvalReport (src/grading.ts).

One suite


bun run evals:workflow-policy

Adjacent harnesses

The planner and retrieval layers have their own deterministic harnesses outside packages/evals:


bun run planner:evaluate -- --planner heuristic   # packages/planner-eval
bun run retrieval:evaluate                        # recall@k over the seeded corpus

Pitfalls. (1) packages/planner-eval is deterministic by design — it no longer owns direct LLM calls; live model-backed planning belongs in query-planner, whose output is still untrusted until validated. Don’t reintroduce API calls into the eval harness. (2) Don’t import @opsdna/ai/live from anything that runs in tests or non-Node environments — the undici import-time crash described above is real. (3) Legacy evals under packages/query-execution are fixtures worth keeping, but new eval coverage for runtime behavior goes in packages/evals or planner-eval.

For what these evals are actually protecting, read the runtime overview — every suite maps to a trust boundary in the plan → gate → execute → seal path.