Entitlement Assistant · Built and Measured

This part is the evidence: what the model does well and less well, measured against the deployed system; how the project was made; what it leaves out; and the perspectives — availability, performance, evolution — that a production system would weigh.

1. Evaluation — the model measured live, every run recorded
2. How it was built — with an AI coding assistant, by written rules
3. Not built — changing a subscription, and the rest
4. Other perspectives — availability, resilience, and more

• References

1. Evaluation

The project evaluates its model live, against the deployment, and records every run. The last complete evaluation sent five messages five times each: the expected tools were called in 22 of 25 turns, the answers were in words in all 25, and accounts were named only by their bracketed id in 16 of 25.

The evaluation is report-only: it is part of no gate, since a model’s wording varies run to run. Each message is scored three ways — the expected tool call, an answer in words rather than a tool call written out as text, and every account and person named by reference — and the reply may quote only the reason codes and facts its tools returned.

The three numbers mean different things. The tools are what decide what the assistant knows, and the three misses were one question — “Does Acme have SSO?” — whose replies neither named both accounts by their bracketed ids nor checked both, either of which the scorer accepts. Words in every turn means no reply wrote a tool call as text. The bracketed ids are a habit: the model often writes an account’s name beside its id, against its instructions, but the screen writes the stored name, and the record says no wrong name was written.

The run found one cause of a miss and fixed it: the model wrote an id’s hyphen as a typographic one, so a reference did not resolve on screen; references now accept the dashes that look alike. The model itself was chosen by measurement. Llama 3.3, the first, wrote 7 of 20 replies as tool calls in an early run; gpt-oss-120b looked accounts up first in 20 of 20, and answered faster.

The runs before it show how the numbers were reached. Each is a row in the feature’s discoveries, with what it found and what changed after it.

The first run’s weakness was fixed in two places, and both are recorded. The instructions now require looking an account up by name before using it and forbid writing a tool call as text; and the screen, if a reply still writes one out, shows that no answer was written and points to the details, which are built from the tools’ results and were right in every run.

The comparison that chose the model ran the same messages five times each against both, with the same instructions. Llama 3.3 70B chose the right tool every time but wrote two of 20 replies as tool calls and looked an account up first in 13 of 20, at 6.2 seconds a turn; gpt-oss-120b looked up first in all 20, answered in words in all 20, at 4.5 seconds, and asked which Acme was meant, as instructed.

The other features were checked live as well. Grants and approvals’ first live check found a gap — a revocation by a grant id the model had read before an extension — which was fixed, and the second run passed all 13 checks. The case lifecycle passed its own, with one finding: a draft summary named the contact’s email, which led to the draft tool’s check. Workers AI’s free daily allocation ran out twice: once after about 140 turns of evaluations, every turn after it answering that the assistant could not answer — as designed, the cause in the logs alone — and once 19 turns into an evaluation of 25, the last 6 refused.

2. How it was built

The project was built by its owner with an AI coding assistant, under written working agreements: a constitution, a workflow, and rules for the architect, the developer, the tester and the writer. Each feature went through a proposal, a verification specification of scenarios, a plan, tests written first, the code, and adversarial reviews by feature, round after round, until a reviewer accepted.

Every feature followed one path. Its scenarios, from the proposal, became a verification specification with a complete given and an exact then for each, and the owner approved it before anything was built. An architect’s plan followed, reviewed; then the verification tests, which had to fail first, for the reason the specification predicted. The code came phase by phase, each phase reviewed, its documents updated with it; a clean tree and every check passing closed it; then the adversarial review.

The work was split into roles, each with its reviewer. The assistant played the architect, the tester, the coder and the technical writer, and the workflow gave each a reviewer: the plan, the tests and each phase reviewed, and the adversarial reviewer — the strongest model available — trying to refute the whole feature. A phase’s review could reject three times before the work stopped and went to the owner; the adversarial review had no cap, and each repair round had to leave fewer findings than the one before.

To finish the demo in time, the project ran in demo mode from its first commit: the architect’s, the tests’ and each phase’s reviews were the assistant’s own, and only the adversarial review of the whole feature ran as a separate review agent, both gates unchanged. The record says so beside the rule it qualifies.

The tests are fixed once they fail as predicted: changing one needs the owner’s agreement, because it changes what was approved. The final gate runs the linter with warnings as errors, the type checker, the dependency rules and every test, on a clean tree, and its result is quoted with its commit.

Five features were built: employee sign-in, explaining access, the case lifecycle, grants and approvals, and the usage guards, an operational feature. Ninety-eight decisions are recorded, 89 accepted and 9 superseded, each with its context, its alternatives and its reason. About 660 tests run: units, use cases against PostgreSQL, HTTP, and the Workers runtime with a scripted model and real Workflows.

The records are kept beside the code: the proposal and its glossary, a verification specification a feature with its plan, progress and discoveries, the decisions, the data model, the API and diagrams. The coding history is kept too; its export, redacted, is under way.

Each feature leaves its record in one place: the approved scenarios, the plan and its review, a progress table, every review’s findings and how each was settled, and its discoveries — every defect, gap, decision or deferral it surfaced. At its close the discoveries are reconciled, each item resolved, filed or put to the owner, and the decisions join the project’s session record.

The coding history is exported by a script of the project’s: each session’s transcript, with every value it is told to redact replaced, written twice — raw, and as a readable rendering of the prompts, the replies and the tool calls. The redacted history is published in the repository: PROMPTS.md, and tech-docs/prompt-history/.

A review is adversarial by design: a reviewer reads the feature’s whole change against its specification and tries to break it. Explaining access, for one, was rejected in its first round — a lookup had no bound, a stream skipped a check, refused arguments were logged with customers’ emails — and accepted in its second, with every finding fixed.

The second round of that review left three minor findings and weighed each: a search cut at 50 accounts could answer from the first 50 while a readable match lay beyond them, and the owner chose to keep the bound rather than give a signal that depended on accounts the representative may not read. The record keeps the choice and its reason beside the finding.

3. Not built

Changing a subscription is the feature the project planned and did not build: an upgrade, a downgrade or a cancellation, prepared and confirmed as a grant is. Its five policies are written and tested, so the decision point already knows how such a change would be decided.

The planned feature would also have let a representative review an account’s history — what changed, who asked, who did it, and why it was allowed — from the audit chain the built features already write. The records are there; the screen that reads them is what is not built.

Beyond it, the project lists what a production system would add, domain by domain. One limit is the platform’s own: passwords are hashed with PBKDF2-SHA256 at 100,000 iterations, the most Workers’ WebCrypto allows, below the 600,000 that current guidance recommends.

A few gaps stand out because the lab chose them knowingly. Customers’ personal data is stored as it is, where production would encrypt it and still need to look it up; the audit chain is immutable while a person may ask to be forgotten, which the chain’s lack of personal data makes possible but production would have to settle. Each inbound adapter — the browser, the chat, a future server of the Model Context Protocol — shares one principal in the demo, where production would give each its own kind of credential and its own namespace.

4. Other perspectives

Availability and resilience rest on the outbox and the sweep. A change and its message to the approval workflow commit together; the relay delivers the message after the commit, and the sweep delivers it again if the relay did not, so a workflow is never lost to a failure between the two. A workflow that misses its moment is woken by the sweep, and its deadline is its own timer.

Performance and scalability have one known ceiling: the audit chain’s single head. Every read writes a record under its lock, and through Hyperdrive each statement is a round trip, so under load the head limits throughput; the project names asynchronous chaining, or batching the records of reads, as the way past it. Case events at scale would move to Queues.

Evolution is planned for in the seams. The policy engine sits behind a port, with its 35 cases as the suite another engine must pass; the policy set is versioned, and every recorded decision names its version; the subscription changes have their policies already. A new tool or API endpoint must pass the architecture test the day it is added, or the build fails. Each seam is where a later change lands without touching the rest: another evaluator behind the port, a new policy beside its test cases, a new feature’s tools under the same test. None of them asks for any of the others to change at all.

The model’s capacity is its own concern: Workers AI’s daily allocation bounds a free account, and the evaluations ran into it. The demo runs on Workers Paid, and its allowance is a setting; a production system would size the model’s capacity to its representatives, and measure the cost of a case.

Two gaps sit beside the audit chain rather than in it. Operators, migrations and seeding reach the database without passing through a use case, so nothing they do is in the chain; and the security events the Worker logs — failed sign-ins, refused requests — are written but not yet collected, correlated or alerted on. Both are listed, with why they matter, among what production adds.

Location is the perspective the lab leaves furthest from production. It runs in one region of PostgreSQL, behind one chain head; serving several regions would need data residency decided and the chain split into partitions combined by a Merkle root, both listed among what production adds rather than built.

References

• The project’s production concerns: tech-docs/production-readiness.md.
• Its features’ discoveries: tech-docs/features/*/workflow/discoveries.md.
• OWASP’s password storage guidance.