CareOps

What it is

CareOps is a working demo app where an LLM answers staff questions and proposes actions, while server code controls access, grounding checks and execution. Every answer shows sources, every action waits for a human Confirm click, and every turn leaves an audit trail. All data is synthetic.

What it does

• Authorization runs before retrieval, so a request a role cannot make is denied before any record or document is fetched.
• The model can only propose two narrow tools; code validates arguments and executes only after a user clicks Confirm.
• Model Lab: 14 questions x 3 repetitions on four models; Qwen3 235B and Claude Sonnet 5.5 each passed 42/42, at $0.45 vs $10.09 per 1,000 answers (README results, 2026-10-05 run).
• Attack Lab: 18 attacks against a hardened prompt, with 0 leaks and 0 boundary moves recorded (README, latest run on Qwen3).
• README documents measurement bugs found in its own evals and the regression tests added for each.

The problem

Staff in a healthcare office want to ask an AI about benefits, PTO and denied claims. A plain chatbot can show data the person should not see, make up numbers, or act on its own. CareOps lets the model reason while the software decides what it may see, say and do.

Design decisions

• Check permission before any record or document is fetched, so restricted data never reaches the model. The tradeoff is that some questions are refused outright instead of answered in part.
• The model can only propose two narrow actions, and code runs them only after a human clicks Confirm. The tradeoff is an extra click for every change.
• Answer checks are plain rules (citations, claim IDs, numbers with units, dates, statuses) instead of a second model. They are cheap and easy to audit, but a wording no rule expects can slip through, and the README says so.
• Use one provider interface and a Model Lab that scores several models pass or fail. The default is the cheapest model that cleared the bar, which ties the choice to measured cost on a synthetic task set, not to general rankings.

How it works

1. Identity and policyA server session identifies the user, then a role and intent policy allows or denies the request before any data is read.
2. State and retrievalFacts come from the database with the minimum needed, and documents come only from the knowledge-base collections that role may read.
3. ReasoningA model reached through one provider interface answers from the supplied facts and documents, which are fenced as untrusted data.
4. ValidationCode checks the JSON shape, citations, record IDs, numbers, dates and tool names, and withholds an answer that fails.
5. ExecutionA proposed action runs only after the user clicks Confirm, and it is idempotent so retries do not duplicate work.
6. AuditEach turn stores an eight-step trace and a plain-English decision summary that the Why did this happen button opens.

Security: enforced, simulated and known gaps

• Enforced in code: role permissions, authorization before retrieval, CSRF tokens, a claim state machine, confirm-before-act, and a daily AI spend cap.
• HIPAA-style safeguards on synthetic data: a second sign-in step, an idle logoff after 15 minutes, lockout after 5 failures, and a private workplace per login. They are not a compliance certification.
• Simulated: all people, claims and documents are synthetic. In the demo the emailed code goes to a labelled Demo mailbox, so it shows the flow and does not prove access to a real inbox.
• Known gap: the answer checks are rules, so free-form advice is constrained by the prompt and the Model Lab, not proven correct.

What broke and how it was fixed

What broke

A hostile technical review found that an answer with valid citations could still state a false number, a claim ID that was not the user's, or a dropped PTO-hours value.

How it was found

A separate reviewer acted as a skeptical interviewer and ran the live site. It raised 3 higher-priority and 6 lower-priority weaknesses. Retests found narrower gaps, such as one shared unit for all durations and money checked only with a dollar sign.

The fix

Numbers, claim IDs and dates are now checked against the user's own records and documents, with the unit and the topic bound to each number. Each finding was reproduced first, then fixed test-first.

What guards it now

Unit tests for each finding, plus scripts/replay-answers.js, which re-runs stored real model answers through the current checks.

Reference

e342098

Where it runs

Docker Compose runs the app and its knowledge base on an internal-only network behind Traefik with Let's Encrypt. It is served at shawndemos.com/careops/.

Who did what

The owner conceived the product, architecture, specs and acceptance tests and directed the work. Coding agents (Claude, Codex, Gemini) did much of the implementation and review. Per the README, Claude Code built the backend and tests, Codex built the front end, and Grok and Codex ran the browser test plan.

Limits and honest status

• Synthetic data only. Not built for real patient data, not clinical advice, and not connected to any real EHR, payer or clearinghouse.
• Lab samples are small and task-specific; neither lab proves general safety or model quality.
• Answer checks are deterministic rules, not semantic understanding; shared demo accounts and a reset button are for demonstration, not tenant isolation.

Connected to

Knowledge Base Server (Memstalker)