Challenger v3 — Self-Improving Credit Compliance Agent

0 · What's changed since the first draft

The first draft of this spec described Phase 0 (knowledge layer) and Phase 1 (citations, gaps, maker/checker, manifest) as the MVP. Both shipped. Since then we've added two operator-facing features and tightened the agent runtime. Read this first — the rest of the spec still describes the original target architecture, with the additions noted in their respective sections.

5b · Clarification Flow (operator-in-the-loop)

A junior analyst asks senior staff when stuck. The Approver and Rejector now do the same. Each agent's structured output gained an optional fourth field:

{
  argument: string,
  known: boolean,
  confidence: number (0..1),
  citations: Citation[],
  clarification_request: { question: string } | null   // NEW
}

The agent populates clarification_request when it would otherwise have to guess — specifically, when there's a focused, answerable thing a senior credit/risk officer could tell it that would change the argument. The system prompts include examples of good vs bad questions ("Does our policy treat 6-month gig income as stable?" vs "Is the credit score good?").

Mid-debate pause-and-resume

When the orchestrator sees confidence < CLARIFICATION_THRESHOLD (default 0.65, runtime-tunable) AND a non-null clarification_request, it:

1. Emits two SSE events: factor_turn with speaker: 'system' + the question (timeline persistence), and clarification_request with round metadata (which the analyst view's useDebateStream hook stores keyed by ${factor}::${speaker}).
2. Calls awaitClarification(), which registers a Promise resolver in an in-memory map keyed by (sessionId, factor, speaker), with a 5-minute timeout.
3. The analyst view sees the new pending entry and mounts an ClarificationInlineForm directly underneath the system-question turn — no modal, no backdrop. The form lives inline in the blue mini-thread. Human types an answer or hits Skip.
4. Form POSTs to /decision/:id/clarify with { factor, speaker, answer, reason } (JSON) or, when the operator attaches evidence, the same fields plus an attachment file part (multipart). The route handler calls resolveClarification() which fires the Promise.
5. Orchestrator unblocks, emits a clarification_resolved event (the hook removes the pending entry, which unmounts the inline form) plus a factor_turn with speaker: 'human' + the answer (timeline persistence), then re-runs the same agent with the Q&A appended to the prompt.
6. Loop until: agent reaches confidence ≥ threshold, OR doesn't ask a new question, OR the per-factor cap (MAX_CLARIFICATIONS_PER_FACTOR, default 2) is hit.
7. If the cap fires, an amber ⚑ Needs review pill appears on that factor in the analyst view.

Audit trail (two layers)

Per-decision timeline: question and answer are first-class turns in factor.turns[], with speaker: 'system' | 'human' and a clarificationKind: 'request' | 'response' tag. They render as a blue-bordered mini-thread in the analyst view ("✋ System asked" / "🗣 Operator answered" + reason badge); while a request is still pending, the ClarificationInlineForm mounts inside that same mini-thread directly under the question, so the operator answers in-context. Turns persist into judge_output.factorDebates so a refresh shows the same chronological story. The version manifest is unaffected — clarifications are operator interventions, not corpus material.

Cross-decision dataset: every clarification also writes a row to clarification_events (migration 015) capturing session_id, factor, speaker, round, question_text, answer_text, answer_status, confidence_before, confidence_after (backfilled when the agent re-runs), confidence_delta (generated column), and time_to_answer_ms. The admin Clarifications tab reads from this table; the next phase of the learning loop uses it for analyst feedback marks and per-factor "recent guidance" prompt injection. Writes are best-effort — failure logs but never blocks the debate.

Operator-attached evidence (PDFs and images)

The clarification form also accepts an optional file alongside the typed answer — used when the borrower has supplied evidence (payslip, bank statement, ID photo, screenshot of HMRC tax record). One file per answer, capped at 10 MB, MIME restricted to application/pdf, image/jpeg, image/png, and image/webp. Both client and server enforce the cap; oversized uploads return 413, wrong types return 415.

Files land on the backend's persistent volume (CLARIFICATION_ATTACHMENTS_DIR, e.g. /data/clarification-attachments/{sessionId}/{slug}), and migration 020_clarification_attachment.sql adds four nullable columns to clarification_events: attachment_path, attachment_mime, attachment_size_bytes, attachment_original_name. A CHECK constraint enforces all-or-nothing — a half-populated row can't slip through.

How the LLM consumes the file: when the orchestrator re-prompts the agent on the next round, an ATTACHED EVIDENCE block is synthesised into the operator's answer. For PDFs the file is run through pdf-parse and the extracted text is inlined (capped at ~12k chars, with a "…[truncated]" marker), so the agent reads the actual document content. For images the agent sees only a labelled note (filename + size + "the operator attached visual evidence; rely on the typed answer") — vision-aware re-prompts are a future upgrade. PDF parse failures fall back to the image/note path so a corrupted upload never kills the debate.

Audit surfaces: the human-response timeline turn gains an attachment field ({eventId, mimeType, sizeBytes, originalName}) so the analyst page renders an inline link to the file alongside the typed answer. The admin Clarifications tab renders the same link on each event card. The backend exposes GET /decision/:id/clarifications/:eventId/attachment which streams the file with Content-Disposition: inline so PDFs and images open in a new tab. The agent prompt + the auditor's view stay decoupled — the LLM never sees image bytes; humans always see the original file.

State management

In-memory only. Works because Fly's request affinity keeps the SSE stream and the /clarify POST on the same machine. If we ever shard runDebate across machines, this graduates to Redis pub/sub or a DB-backed coordination primitive. For two-operator testing on a single Fly machine, the in-memory map is fine. Attachment storage carries the same single-machine assumption — when we shard, the volume mount becomes S3/R2.

6b · Operator Console (Admin Panel)

Lives at /admin on the frontend. Five tabs (Corpus, Debates, Clarifications, Evals, Settings), shared bcrypt-hashed password, signed httpOnly cookie session. Replaces the prior workflow where every corpus tweak required SSH + CLI + restart.

Auth

• Operator runs npm run admin:hash -- 'password' locally → bcrypt hash.
• Hash goes into the ADMIN_PASSWORD_HASH env on Fly. Plaintext never touches the server.
• Cookie signing uses COOKIE_SECRET (generated by openssl rand -hex 32).
• Every protected route is guarded by a requireAdmin Fastify preHandler.
• Rotating COOKIE_SECRET invalidates every existing session.

Corpus tab

Table of every knowledge_bundle row joined with chunk metadata (representative tier / jurisdiction / source_id / source_url / version pulled via MIN(c.<col>) across the bundle's chunks). Source ID is linked to the external source_url when present. Inline confirm-delete cascades to chunks via FK.

Upload form: PDF picker, tier dropdown with inline explainer ("Tier 1 = regulator, Tier 4 = academic"), jurisdiction text, source-id text, optional version + source-url. Submits multipart to POST /knowledge/upload, which streams the file to /tmp, runs the existing ingestDocument() pipeline (loader → chunker → embedder → DB writer), and returns the standard IngestReport. Authenticated callers only — an attacker can't fill our DB or burn our embedding tokens via curl.

Settings tab

The tab now ships in three sections, all reading from the same runtime_settings table:

• Tunables — original numbers / booleans / jurisdiction. Widget chosen by the row's type: toggle for boolean, slider for known-bounded numbers, text input otherwise, jurisdiction dropdown for the namesake key.
• Model & provider — LLM_MODEL renders a whitelist dropdown of pricing-table-known Claude models (Haiku 4.5, Sonnet 4.x, Opus 4.x) with per-token cost displayed inline so the operator can compare. Empty value means "use the env-derived default." llmClient.ts caches the built ChatAnthropic / ChatOpenAI client by (provider, model) tuple and rebuilds when the operator's edit lands — no redeploy.
• System prompts — four collapsible textareas (APPROVER_SYSTEM_PROMPT etc., seeded by migration 023). Each shows a default / custom badge, character count, and a hidden-by-default disclosure showing the in-code default text (fetched from GET /admin/prompts/defaults). "Copy default into editor" forks the canonical text; "Clear override" empties the textarea so saving re-engages the default. The orchestrator's computePromptSetHash() reads from the same source so every edit bumps the manifest's prompt_set_hash.

Every row shows the plain-English description seeded by migration 013 / 014. Save button activates only when the value differs from the saved one. Updates POST to PUT /admin/settings/:key; the response writes the new updated_at + updated_by fields, displayed under the widget.

Currently exposed knobs: RETRIEVAL_ENABLED, RETRIEVAL_TOP_K, RETRIEVAL_MIN_SIMILARITY, DEFAULT_JURISDICTION, CLARIFICATION_THRESHOLD, MAX_CLARIFICATIONS_PER_FACTOR, EARLY_EXIT_NEGATIVE_THRESHOLD, GUIDANCE_INJECTION_ENABLED, GUIDANCE_LOOKBACK_DAYS, GUIDANCE_MAX_ITEMS_PER_FACTOR, LOG_SQL.

Debates tab

Newest-first listing of every debate session, joined with decision_reviews for the four-eyes status. Filters: by session status (Completed / Failed / Running) and by final decision (APPROVE / REJECT / REVIEW). An "Include eval runs" checkbox surfaces sessions produced by eval batches; off by default so live operator activity isn't drowned by them. Backed by GET /admin/debates?status=&decision=&includeEval=&limit=.

Roll-up strip at the top: total / approved / rejected / review / fallback-used / cost-loaded counts over the loaded set. Each row shows session UUID prefix, application id, status pill, decision pill (with a fallback tag if the rule engine had to step in, plus a from eval tag when surfaced via the toggle), the first few decision tags, four-eyes review status, duration, USD cost (with input/output token breakdown on hover), and created-at. View action is an eye icon in the rightmost column — opens replay (or live, for in-flight debates). Eval-vs-live is tracked via the debate_sessions.eval_run_id column added in migration 021 — eval runner sets it; the live /decision route leaves it NULL. Cost columns are populated by migration 022 + withUsageTracking in the orchestrator: each LLM call records token usage to an AsyncLocalStorage scope, and the finally block sums + persists at debate end (success, fallback, or cancellation) using the per-model pricing table in agents/usage.ts.

Failed debates render a translated banner instead of the raw ERROR_TRACE when one of the well-known LLM provider errors fires (out of credit, rate limit, invalid API key). The friendlyLlmFailure() helper in the orchestrator tags these into the fallback reasoning; the analyst page's FinalDecisionCard renders a coloured alert with the operator-readable message and tucks any unrecognised stack trace behind a "Show debug trace" disclosure.

Click-through navigates to the existing analyst route: /analyst/decision/:id?replay=true for completed/failed debates (loads from persisted judge_output), or the live streaming view for in-flight ones. No new view code — replay was already built; this tab just makes it discoverable.

Clarifications tab

Read-only feed of every clarification event written to clarification_events, newest first. Filters: factor (dropdown of the five debate factors), status (answered / skipped / timeout / capped), and an "Include eval runs" toggle (off by default). Eval batches run with maxRounds=0 and produce a flood of capped rows when an agent reports low confidence; hiding them by default keeps the operator's own clarification history readable. The toggle is implemented as a LEFT JOIN debate_sessions ON s.id = e.session_id in listEvents with a default WHERE s.eval_run_id IS NULL. Cards from eval runs render a from eval badge when surfaced. Backed by GET /admin/clarifications?factor=&status=&includeEval=&limit=.

Header strip shows four roll-up stats over the loaded set: total events, answered count + percentage, skipped+capped count, and average confidence Δ. The avg-Δ figure is the closest thing to "are these questions actually useful?" you can read at a glance — green if ≥+5%, red if negative.

Each event renders as a card: status pill + factor + speaker + round, the question and answer text in a blue mini-thread, and a footer row with confidence_before, confidence_after, the delta, and the operator's response latency. Future enhancements (analyst feedback marks, similarity-based prompt injection) build on this surface.

Evals tab

Three-block layout. Datasets at the top: list of uploaded CSVs (one row per eval_datasets entry), each showing processed_count / total_rows as a progress bar so you can see at a glance how much of the dataset is left. Inline upload button — multipart POST to /admin/eval/datasets, files written to EVAL_DATA_DIR (a Fly volume in prod, OS tmp in dev), SHA-256 deduped so re-uploads no-op.

Run launcher: dataset dropdown + row-limit input + a single Start button. Submits POST /admin/eval/runs with { datasetId, rowLimit }. The backend creates an eval_runs row in queued status, fires the worker on setImmediate, and returns the row id. The runner skips cursor_start CSV rows (= dataset.processed_count at POST time), then collects up to row_limit mappable rows and runs each through runDebate(...,{ noClarifications: true }) sequentially. Per-row results land in eval_run_rows as they complete; aggregate metrics (TP/TN/FP/FN, accuracy, expected loss, net €) are computed and written to eval_runs at the end. The cursor advances by the number of CSV rows consumed (not just successful) so failed rows aren't retried on the next run.

Runs history: newest-first table of all runs (optionally filtered by clicking a dataset above). Each row shows started timestamp, status pill, slice indices, the four confusion-matrix counts, accuracy, and net €. Clicking a run opens the run detail view: confusion-matrix grid up top (TP/TN/FP/FN + REVIEW + failed + accuracy + net €) and a per-row table below with the agent's decision, the realised outcome, the bucket, and a one-click link to the live debate replay (/analyst/decision/:session_id?replay=true) so you can drill into why any individual call went the way it did. While the run is queued/running the UI polls every 4 s.

What's NOT in the MVP: agent-written interpretation of the results (deferred — we want analysts to look at the per-row data first), file attachments on clarifications, and any kind of automatic dataset selection. The cursor is per-dataset only — there is no "split into train/eval" mode.

Caching policy

Reads use a 60s in-memory map. Writes invalidate the map immediately on the writer's Fly machine; other machines pick up changes on next cache miss. So a setting saved at T+0 takes effect on the writer's debates immediately, on other machines within ≤60s. This was the right trade-off for two-person testing — no Redis dependency, negligible read cost.

1 · Overview & Product Thesis

Challenger v2 is a static multi-agent debate system (Approver → Rejector → Judge) that produces explainable credit decisions — but the intelligence lives entirely in the LLM's pretraining. There is no regulatory grounding, no memory of prior decisions, no way to improve from feedback, and no audit trail a European bank regulator would accept.

v3 keeps the debate harness and wraps it in three new layers: a hierarchical knowledge store (EU → BG → practice), a grounded reasoning layer where every argument carries citations and a self-reported confidence, and a self-improvement layer where the agent logs what it did not know, humans resolve gaps, and the system is version-pinned so every historical decision can be reproduced.

Success criteria for v3 MVP

• Every agent argument ships with at least one citation ID resolvable to a specific regulation section.
• Every decision is pinned to an immutable version_manifest tuple (model, prompt_set, knowledge_bundle, guardrail_set).
• When retrieval or agent confidence is below threshold, the decision surfaces a structured KNOWLEDGE_GAP event that a human can resolve in the UI.
• A resolved gap becomes an ingested chunk and a few-shot correction — next similar decision benefits automatically.
• A 200-case golden set runs as a regression on every version bump; the diff report is visible to the team.

2 · High-Level Architecture

Layer boundaries

3 · Knowledge Layer

3.1 Ingestion pipeline

3.2 Chunking strategy

• Regulatory text is section-structured. Chunker must respect section boundaries (§, Article, Paragraph) — never fixed-window.
• Target ~512 tokens per chunk, with a 64-token overlap preserving section header context.
• Each chunk stores parent section title, article number, and preceding headings as breadcrumbs — crucial for citation rendering.
• Multilingual: start English (most source docs) + Bulgarian (BG-specific). Use bge-m3 embeddings which handle both; avoid per-language pipelines.

3.3 Knowledge bundle (immutable artefact)

Every ingestion batch produces a knowledge_bundle row with a content hash. Decisions reference bundle_id. Bundles are append-only; a "new version" of EBA guidelines creates a new bundle, never mutates the old one.

knowledge_bundles
├── id                UUID, primary key
├── label             VARCHAR  — e.g. "kb-2026-04-20-eu+bg"
├── content_hash      VARCHAR  — SHA256 of sorted chunk_ids
├── source_manifest   JSONB    — [{source_id, version, effective_from, sha256}]
├── chunk_count       INTEGER
├── created_at        TIMESTAMPTZ
└── created_by        VARCHAR

3.4 Chunks table

knowledge_chunks
├── id                UUID, primary key
├── bundle_id         UUID  → knowledge_bundles.id
├── source_id         VARCHAR  — e.g. "EBA-GL-2020-06"
├── source_url        TEXT
├── tier              SMALLINT  — 1=EU, 2=national, 3=practice, 4=literature
├── jurisdiction      VARCHAR  — "EU" | "BG" | "DE" | ...
├── version           VARCHAR  — "2020/06"
├── effective_from    DATE
├── section           VARCHAR  — "Article 4 §2"
├── breadcrumb        TEXT     — "Title II > Chapter 3 > Article 4 > §2"
├── language          VARCHAR  — "en" | "bg"
├── text              TEXT
├── embedding         VECTOR(1024)  — pgvector
├── token_count       INTEGER
└── chunk_hash        VARCHAR   — SHA256 of text

INDEX ivfflat (embedding vector_cosine_ops)
INDEX (bundle_id, tier, jurisdiction)

3.5 Retrieval

3.6 Retrieval contract

// packages/shared/src/retrieval.ts

export type RetrievalQuery = {
  text: string;
  factor: FactorName;                 // "Credit Score" | ...
  jurisdiction: "EU" | "BG";          // application's jurisdiction
  tiers?: Array<1 | 2 | 3 | 4>;       // default: [1,2,3,4]
  topK?: number;                      // default: 5 (after reranking)
  minSimilarity?: number;             // default: 0.72
};

export type RetrievedPassage = {
  chunkId: string;
  sourceId: string;
  sourceUrl: string;
  breadcrumb: string;
  tier: 1 | 2 | 3 | 4;
  jurisdiction: string;
  version: string;
  similarity: number;
  rerankScore: number;
  text: string;
};

export type RetrievalResult = {
  passages: RetrievedPassage[];
  maxSimilarity: number;
  tierCoverage: Record<string, number>;  // tier → count above threshold
  gaps: Array<{
    expectedTier: 1 | 2 | 3 | 4;
    expectedJurisdiction: string;
    reason: "no_match_above_threshold" | "jurisdiction_missing";
  }>;
};

4 · Reasoning Layer (Grounded Debate)

4.1 What changes from v2

4.2 Orchestration (updated)

4.3 Agent contracts

// packages/shared/src/agents-v3.ts

export const MakerOutputSchema = z.object({
  claim: z.string(),
  citation_ids: z.array(z.string()).min(1),   // must cite ≥1 passage
  confidence: z.number().min(0).max(1),
  known: z.boolean(),
  missing_knowledge: z.string().optional(),   // populated when known=false
});

export const CheckerOutputSchema = z.object({
  status: z.enum(["OK", "CONFLICT", "INSUFFICIENT_EVIDENCE"]),
  conflicting_citation_id: z.string().optional(),
  conflict_explanation: z.string().optional(),
});

export const JudgeFactorOutputSchema = z.object({
  verdict: z.enum(["positive", "negative", "neutral"]),
  summary: z.string(),
  cited_passages: z.array(z.string()),
  maker_confidence: z.number(),
  checker_status: z.enum(["OK", "CONFLICT", "INSUFFICIENT_EVIDENCE"]),
});

4.4 Prompt contract — Maker (excerpt)

SYSTEM
You are a credit origination analyst for a European bank.

You MUST:
- Base your claim on the RETRIEVED PASSAGES only.
- Cite every claim with passage IDs from the list below.
- Set `known: false` and fill `missing_knowledge` when no passage supports
  the claim at the given jurisdiction and tier.
- Never invent regulation names, article numbers, or dates.

HUMAN
FACTOR: {factor_name}
JURISDICTION: {jurisdiction}
APPLICATION: {application_json}
RETRIEVED PASSAGES:
{passages_with_ids}

FORMAT:
{format_instructions}

4.5 Gap detector

5 · Self-Improvement Layer

5.1 Closing the loop

5.2 Feedback taxonomy

5.3 Version manifest

version_manifests
├── id                UUID, primary key
├── label             VARCHAR  — "v3.4.1"
├── model             VARCHAR  — "claude-sonnet-4-6"
├── prompt_set_id     VARCHAR  — "ps-2026-04-17"
├── knowledge_bundle  UUID  → knowledge_bundles.id
├── guardrail_set_id  VARCHAR  — "gr-v2"
├── eval_score        NUMERIC  — 0.87
├── eval_run_id       UUID
├── status            ENUM: DRAFT | ACTIVE | DEPRECATED
├── released_at       TIMESTAMPTZ
└── released_by       VARCHAR

-- debate_sessions gains a column
ALTER TABLE debate_sessions
  ADD COLUMN version_manifest_id UUID NOT NULL
  REFERENCES version_manifests(id);

5.4 Eval harness

• Golden set: 200 hand-labelled cases — approve-obvious, reject-obvious, edge cases per factor, guardrail triggers, regulatory edge cases (AMLD6, PSD2).
• Regression CLI: npm run eval -- --version v3.4 --against v3.3 → emits a diff report (decision flips, citation precision, gap-rate delta, latency p95 / p50).
• Metrics: decision-accuracy · citation-precision · guardrail-respect · gap-detection-rate · p95-latency · cost/decision.
• Adversarial set: expert-disagreement cases — the agent should return REVIEW, not pick a side. Tracked separately.

Bondora ground-truth eval (shipped)

A first-pass eval harness is in the codebase: npm run eval:bondora -- --csv path/to/LoanData_Bondora.csv --limit 50. It reads the public Bondora P2P lending dataset, maps each historical loan to a CreditApplication, runs it through runDebate(), and scores the agent's APPROVE / REJECT / REVIEW against the realised outcome (Repaid vs Default).

Output is a confusion-matrix scoreboard:

• True positives (APPROVE on a loan that paid back)
• True negatives (REJECT on a loan that defaulted)
• False positives — the dangerous bucket: APPROVE on a defaulter
• False negatives — lost good loans
• REVIEW count (no decisive call)
• Expected loss / €1000 lent under the agent's policy + a notional 8% margin "net €" on TPs
• Average debate latency

Mapping is documented inline in the script: rating letter → synthetic FICO, employment status code → enum, missed-payments proxy from NewCreditCustomer + PreviousEarlyRepaymentsCountBeforeLoan. Rows with no terminal outcome (Status=Current) or missing required fields are skipped.

Eval calls bypass the clarification flow. The script passes { noClarifications: true } to runDebate(), which forces maxClarifications = 0 for that call only — the cap fires immediately on any clarification request, no awaitClarification pause, no 5-minute timeout per row. This is the only honest way to measure the system's autonomous decision-making against ground truth: feeding placeholder answers would measure how the agents react to fake operator input, not what they decide on their own. Production debates running concurrently with the eval are unaffected.

Domain mismatch caveat: Bondora is unsecured P2P consumer lending in EE/ES/FI, mostly pre-2020. Findings are a useful sanity check, not a precise predictor of EU regulated bank lending performance. The script is designed for "is the direction right" answers, not "is the FP rate exactly 4.7%" answers.

6 · Data Model

6.1 knowledge_gaps

knowledge_gaps
├── id                UUID, primary key
├── session_id        UUID  → debate_sessions.id
├── factor            VARCHAR
├── trigger_signal    ENUM: low_retrieval | maker_unknown | low_confidence |
│                           checker_insufficient | jurisdiction_missing
├── missing_topic     TEXT        — agent's own description
├── suggested_sources TEXT[]      — agent-proposed resources
├── tier_needed       SMALLINT
├── jurisdiction_needed VARCHAR
├── topic_hash        VARCHAR     — for deduplication
├── status            ENUM: OPEN | ANSWERED | INGESTED | DISMISSED
├── resolution_note   TEXT, nullable
├── resolved_by       VARCHAR, nullable
├── resolved_at       TIMESTAMPTZ, nullable
├── resulting_chunk_id UUID, nullable  → knowledge_chunks.id
├── created_at        TIMESTAMPTZ

UNIQUE (topic_hash, status) WHERE status = 'OPEN'

6.2 feedback_corrections

feedback_corrections
├── id                UUID, primary key
├── session_id        UUID  → debate_sessions.id
├── field             VARCHAR     — "final_decision" | "factor.verdict.Credit Score" | ...
├── agent_value       JSONB
├── human_value       JSONB
├── reason            TEXT
├── embedding         VECTOR(1024)  — embedded(application + reason) for few-shot retrieval
├── created_by        VARCHAR
├── created_at        TIMESTAMPTZ

7 · API Surface

7.1 New SSE event types

8 · Local Dev & AWS Infrastructure

One codebase, two deployment targets. Adapters behind interfaces let us swap infra without touching business logic. local = laptop dev, aws = production.

8.1 Component mapping

8.2 AWS deployment topology

8.3 Local dev stack (docker-compose)

# docker-compose.dev.yml — run: docker compose up

services:
  postgres:
    image: pgvector/pgvector:pg16
    environment:
      POSTGRES_PASSWORD: postgres
      POSTGRES_DB: challenger_dev
    ports: ["5432:5432"]
    volumes: ["pgdata:/var/lib/postgresql/data"]

  minio:            # S3-compatible local storage
    image: minio/minio
    command: server /data --console-address ":9001"
    ports: ["9000:9000", "9001:9001"]
    environment:
      MINIO_ROOT_USER: dev
      MINIO_ROOT_PASSWORD: devpassword

  # Reranker served via HuggingFace's TEI for local parity with SageMaker
  reranker:
    image: ghcr.io/huggingface/text-embeddings-inference:latest
    command: ["--model-id", "BAAI/bge-reranker-v2-m3"]
    ports: ["8080:80"]

volumes: { pgdata: {} }

8.4 Environment variables

# Shared
NODE_ENV=development
LLM_PROVIDER=anthropic              # anthropic | openai
LLM_MODEL=claude-haiku-4-5-20251001
LLM_API_KEY=sk-ant-...

# Embeddings — Voyage is Anthropic-recommended; voyage-3-lite has a
# generous free tier and produces 512-dim vectors.
EMBED_PROVIDER=voyage               # voyage | openai | bedrock
EMBED_MODEL=voyage-3-lite
EMBED_API_KEY=pa-...
EMBED_DIMENSIONS=512

# Database — local Postgres + pgvector for dev, Neon for Fly deploy.
DATABASE_URL=postgresql://postgres:postgres@localhost:5433/debate_db

# Static fallbacks for the runtime_settings rows (used only if a row
# is missing from the table — normally the table wins).
RETRIEVAL_ENABLED=true
RETRIEVAL_TOP_K=5
RETRIEVAL_MIN_SIMILARITY=0.55
DEFAULT_JURISDICTION=EU

# Admin panel
ADMIN_PASSWORD_HASH=$2b$12$...      # bcrypt; generate with: npm run admin:hash -- 'pwd'
COOKIE_SECRET=...                   # 32+ random hex chars; openssl rand -hex 32

# CORS — comma-separated allowlist. Wildcards in dev only.
CORS_ORIGIN=https://your-frontend.fly.dev

# Observability
LOG_SQL=1                           # SQL preview + timing in stdout
LOG_COLOR=1                         # ANSI category tags in stdout
FORCE_COLOR=1                       # force color when piped (e.g. fly logs)

# AWS-only (when we eventually flip from Fly to AWS)
AWS_REGION=eu-central-1
BEDROCK_LLM_MODEL=anthropic.claude-sonnet-4-v1:0
BEDROCK_EMBED_MODEL=amazon.titan-embed-text-v2:0
COGNITO_USER_POOL_ID=...
COGNITO_CLIENT_ID=...
S3_BUCKET_RAW=challenger-raw-docs

9 · Directory Structure

challenger/
├── backend/
│   ├── src/
│   │   ├── agents/
│   │   │   ├── maker.ts              # NEW
│   │   │   ├── checker.ts            # NEW (replaces rejector in compliance mode)
│   │   │   ├── rejector.ts           # kept for UX explainability
│   │   │   ├── judge.ts
│   │   │   ├── prompts/
│   │   │   │   ├── maker.ts
│   │   │   │   ├── checker.ts
│   │   │   │   ├── judge.ts
│   │   │   │   └── index.ts          # exports prompt_set_id
│   │   │   └── llmClient.ts          # provider switch (Anthropic | Bedrock)
│   │   ├── knowledge/
│   │   │   ├── ingest/
│   │   │   │   ├── loader.ts         # PDF | HTML
│   │   │   │   ├── chunker.ts        # section-aware
│   │   │   │   ├── embedder.ts       # provider switch
│   │   │   │   └── worker.ts         # SQS consumer in prod
│   │   │   ├── retriever.ts          # hybrid + rerank
│   │   │   ├── bundles.ts
│   │   │   └── index.ts
│   │   ├── engines/
│   │   │   ├── guardrail.ts
│   │   │   ├── fallback.ts
│   │   │   └── gapDetector.ts        # NEW
│   │   ├── orchestrator/
│   │   │   ├── index.ts              # v3 orchestrator
│   │   │   └── factorLoop.ts
│   │   ├── feedback/
│   │   │   ├── corrections.ts
│   │   │   ├── outcomes.ts
│   │   │   └── fewShot.ts            # retrieves top-k corrections
│   │   ├── versioning/
│   │   │   ├── manifest.ts
│   │   │   └── activate.ts
│   │   ├── eval/
│   │   │   ├── goldenSet/            # JSON cases
│   │   │   ├── runner.ts
│   │   │   └── diff.ts               # CLI-callable
│   │   ├── platform/
│   │   │   ├── blobStore.ts          # local FS | S3 adapter
│   │   │   ├── queue.ts              # in-proc | SQS adapter
│   │   │   ├── auth.ts
│   │   │   ├── config.ts
│   │   │   └── logger.ts
│   │   ├── routes/
│   │   │   ├── decision.ts
│   │   │   ├── knowledge.ts
│   │   │   ├── gaps.ts
│   │   │   ├── feedback.ts
│   │   │   ├── versions.ts
│   │   │   └── eval.ts
│   │   ├── db/
│   │   │   ├── migrations/
│   │   │   │   ├── 001_init.sql
│   │   │   │   ├── 010_knowledge.sql
│   │   │   │   ├── 011_gaps.sql
│   │   │   │   ├── 012_feedback.sql
│   │   │   │   └── 013_versioning.sql
│   │   │   ├── pool.ts
│   │   │   ├── sessions.ts
│   │   │   ├── chunks.ts
│   │   │   ├── gaps.ts
│   │   │   └── versions.ts
│   │   ├── buffer.ts                 # SSE buffer (unchanged)
│   │   └── index.ts
│   └── package.json
├── frontend/
│   └── src/
│       ├── app/
│       │   ├── decision/[id]/
│       │   ├── analyst/decision/[id]/
│       │   ├── operator/
│       │   │   ├── gaps/              # NEW — gap queue
│       │   │   ├── knowledge/         # NEW — ingestion + bundles
│       │   │   └── versions/          # NEW — manifest admin
│       ├── components/
│       │   ├── FactorDebate/
│       │   ├── CitationChip/          # NEW
│       │   ├── GapBanner/             # NEW
│       │   ├── ResolveGapDialog/      # NEW
│       │   └── VersionBadge/          # NEW
│       └── hooks/
│           ├── useDebateStream.ts
│           └── useGapStream.ts        # NEW
├── packages/
│   └── shared/
│       └── src/
│           ├── types.ts
│           ├── schemas/
│           │   ├── maker.ts
│           │   ├── checker.ts
│           │   ├── judge.ts
│           │   ├── gap.ts
│           │   └── feedback.ts
│           └── retrieval.ts
├── infra/
│   ├── docker-compose.dev.yml
│   └── aws/
│       ├── cdk/                        # CDK app (TS) — one stack per concern
│       │   ├── bin/challenger.ts
│       │   ├── lib/
│       │   │   ├── network-stack.ts     # VPC + subnets
│       │   │   ├── data-stack.ts        # RDS · S3 · SQS
│       │   │   ├── compute-stack.ts     # ECS services · ALB
│       │   │   ├── ai-stack.ts          # Bedrock IAM · SageMaker endpoints
│       │   │   └── observability-stack.ts
│       │   └── cdk.json
│       └── README.md
├── docs/
│   ├── Architecture_Recommendation_v3_Self_Improving_Agent.md
│   ├── Core_Flows_—_Factor-Based_Debate_(v2).md
│   └── Tech_Plan_—_Multi-Agent_Credit_Decision_System.md
├── challenger-spec-v3.html              # this file
└── README.md

10 · MVP Build Order & Milestones

10.1 Phase 0 definition of done

10.2 Phase 2 success metric

11 · Observability, Evals, Security

11.1 Observability

• Logs: structured JSON (pino). Every LLM call logs {session_id, agent, version_manifest_id, model, prompt_hash, input_tokens, output_tokens, latency_ms, cost_usd}.
• Traces: OpenTelemetry spans on orchestrator · retriever · per-agent · reranker. Propagate trace_id into SSE so UI can link to backend trace.
• Metrics (Prometheus / CloudWatch): p50/p95 debate latency, retrieval coverage, gap-creation rate, correction rate, cost per decision.
• PII: never log full application payload; hash applicant_id; redact loan amount and income ranges by default in non-prod logs.

11.2 Eval cadence

• On every PR touching prompts/* or agents/* → CI runs a 20-case smoke eval.
• Nightly: full 200-case golden set against ACTIVE manifest.
• Pre-release: full eval + adversarial set; diff must be reviewed and signed off before manifest activation.

11.3 Security baseline

• All data at rest encrypted (RDS · S3 · Secrets Manager — KMS-backed).
• All traffic TLS 1.2+. Internal ECS ↔ RDS via private subnets only.
• LLM calls to Bedrock stay in-region (eu-central-1) — no data egress to US endpoints.
• Cognito for analyst / operator auth. Applicant-facing SSE session is signed-URL gated with 60s TTL.
• Audit log: any change to a version manifest, bundle activation, or feedback correction is recorded in an append-only audit_log table.

12 · Key Decisions & Trade-offs

13 · Week-1 Concrete Actions