Paul Bergeron.
Menu

Spec Review / System design

Evidence is a system boundary.

Why the queue, parser, model and reviewer have separate responsibilities.

Architecture

Product decision

An engineer reviewing a specification revision needs an inventory of changes and the evidence behind each one. Asking a model to summarize two whole PDFs makes omissions hard to notice. Spec Review first computes the textual change inventory and only then asks a model to interpret it. Human review is a separate durable step.

  1. Submit & store
  2. Lease worker
  3. Extract & diff
  4. Validate & review

The local UI polls persisted state; it does not own job execution. Closing a tab does not discard submitted work. Each job stores input SHA-256 identities, processing mode, requested model, pipeline version, prompt version and events. Model runs additionally record returned model identity, elapsed time and provider-reported token counts when available. Unknown token counts remain null, not invented costs.

Durable processing

SQLite uses WAL and short BEGIN IMMEDIATE transactions. A claim increments the fencing token and attempt count and grants a 120-second lease. Every completion compares current status, fence and lease expiry. A worker whose lease was reclaimed cannot overwrite the new owner's result. Processing is at-least-once; a crash after a provider response may cause a duplicate provider request. Submission idempotency is separate: the same request key and payload returns the same job; a reused key with different hashes, mode, model or pipeline version conflicts.

Transient provider failure gets one retry. If the retry fails, the textual evidence becomes available with annotation marked unavailable. Invalid model output is rejected immediately. Worker crashes may be recovered up to three claims, after which the job fails. There is no exactly-once provider-call claim and no distributed queue.

Citation and interpretation contract

Each source line has a stable reference within that extraction: revision side, page and line. Change IDs hash the paired evidence. The diff is independent of the model. An annotation must identify an existing change and cite existing evidence from each nonempty side. Every detected change needs one annotation; duplicate and missing annotations reject the set. Quotes displayed to users come from extraction, not generated text.

The schema checks do not establish semantic entailment, detect every prompt injection, or guarantee the model's category and explanation. Generated interpretation remains labeled unverified. Reviewer dispositions—acknowledged, needs action, dismissed—are independent of the model. Optimistic version checks prevent a stale tab silently replacing a newer decision.

Deliberate tradeoffs

  1. SQLite instead of a broker and separate metadata service. Appropriate for a bounded single-host tool; simpler transaction and backup boundary. Not suitable for multi-host workers or large document collections.
  2. Textual alignment instead of embeddings. Preserves an inspectable inventory and handles repagination of unchanged lines. Wrapping, reordered sections and extraction artifacts can introduce noise. Embeddings would not supply a completeness guarantee.
  3. Fail closed on unsupported PDFs. Image-only or encrypted documents require another ingestion path. No “no changes” result is emitted because extraction failed.
  4. Model annotation without tools. Useful for triage, but has no ability to fetch URLs, execute code or approve a review. A more complex agent loop adds no demonstrated value here.
  5. Source PDFs downloaded as attachments. Avoid embedding arbitrary active documents in the application origin. Original visual inspection remains the reviewer's responsibility.
  6. Append-only application events. Events reconstruct workflow and decisions. They are not a cryptographically tamper-evident audit log; a database administrator can alter them.

Versions and public evidence

Pipeline pipeline-v1; prompt change-significance-v1. The checked-in fixture PDFs are newly authored fictional material. The exported bundled run is evidence-only and includes no model interpretation. Regression results test extraction and workflow contracts, not a production document distribution.

Operational readiness

Assets: uploaded PDF bytes, extracted content, reviewer notes, model credentials, evidence integrity and local compute capacity. Trust boundaries: browser → local API; API → database; worker → PDF parser process; worker → external model provider; model output → human review.

Implemented controls:

  • Loopback binding, Host allowlist, required mutation header, same-origin checks, no CORS grants, no arbitrary upstream URL or filesystem path supplied by a client.
  • Bounded request body before multipart parsing, per-file bytes and extraction limits; extraction in a separate process with wall/CPU bounds. Linux additionally sets an address-space limit.
  • Generated temporary paths, database-managed input identity, no credentials in API responses or traces, no raw parser/provider exception text in UI errors.
  • No model tools; document text explicitly treated as untrusted in the prompt. Output schema, coverage and citation references validated in code. All displayed text escaped.
  • Fenced result publication, optimistic reviewer writes and persisted event records. These protect against routine races, not a malicious database administrator.
  • Download attachments for PDFs, restrictive browser CSP, no-store responses, no tracking dependencies.

Required before calling this a production public-upload service:

  1. Authentication and authorization. The reference app is single-user loopback. Add authenticated identities, project/document ownership, authorization on every route and attributable reviewer identities before multi-user use.
  2. Stronger extraction isolation. A child process is not a security sandbox. Apply a dedicated low-privilege OS/container boundary, network denial, strict memory/CPU/filesystem quotas and dependency patching. macOS currently lacks the Linux address-space cap. Test decompression bombs and hostile parser cases in that boundary.
  3. Representative evaluation. Collect permitted real specification pairs with independently reviewed change inventories. Measure false omissions, wrapping/reordering noise, citation quality and semantic usefulness separately. Include scans, tables, figures, negation and prompt-injection attempts. Current fixtures do not establish those rates.
  4. Provider operations. Test the selected model live, set enforceable spend/concurrency limits, measure timeout/retry behavior, and establish data-retention terms. A request timeout does not recall provider work.
  5. Lifecycle and observability. Add monitored worker liveness, retention policy, tested backups/restores and deployment/rollback acceptance. Replace single-host SQLite only when requirements justify it.
  6. Review/export governance. Define authorized reviewer identities and whether evidence needs tamper-evident storage, digital signatures or stronger retention guarantees. Existing events are ordinary database records.

No claim is made that this list alone certifies production readiness. The implemented scope is a tested local engineering reference with a concrete route toward a hardened service.