CCAR-F / Free hands-on exercise / Version 1

Can you trust a well-formed extraction?

Normalize synthetic shipping notes, pin their source versions, then keep successful results while investigating failed siblings. A correct shape and a real excerpt can still accompany an unsupported claim.

Run locally with Python 3.10 or newer and its standard library. No package installation, sign-in, key or network service is needed after download. Basic dictionaries, exceptions and unit tests are helpful. Your code, prompt and written review stay on your computer; this page accepts no submissions or completion history.

Download the Python starter (ZIP)

Start with failing tests

  1. Check python3 --version (Windows: py -3 --version).
  2. Extract into a new empty folder. Read README.md, CONTRACT.md, the supplied fixtures and tests.
  3. Open a terminal in the extracted folder and run the command below. Four supplied checks pass; 24 intentionally fail because three functions are unfinished. These counts describe the starter, not your score. Keep the supplied tests unchanged.
python3 -m unittest -v

Windows: py -3 -m unittest -v. No setup script or package installation is needed. Use a disposable folder separate from real documents or customer work.

1. Normalize without guessing

Implement normalize_record in extraction.py. Raw amounts are explicit decimal major-unit strings. Use integer/string arithmetic, preserve the source identity and validate against the finite annotations.

" usd " → "USD"
"7" + "5.0" → [700, 500] cents
"0.29" → 29 cents, without float rounding
EUR discount "-2.00" → -200 cents
Explicit "2026-11-12" → same ISO date
Date absent in note → null

Change from note A's USD/no-date case to note B's EUR/discount/date case. Reject bool-as-integer, comma locale, ambiguous dates, invented values and mismatched source revisions. Do not infer a currency from $ or turn 11/12/2026 into a guessed date.

Note D explicitly states EUR 5.40; its line and surcharge calculate EUR 5.50. Inspect both values separately. Reject acceptance and keep the original discrepancy for review. Never repair the printed claim silently.

2. Write a prompt and pin a pilot

Refine PROMPT.md with an original USD/no-date example, the changed EUR example and explicit missing/ambiguous-value rules. Update its revision in fixtures.py. This is a handwritten prompt artifact; no model runs and the tests do not grade its quality.

Implement approve_pilot for the two designated samples before constructing local batch descriptors. Bind the exact prompt, normalization/schema revisions, source IDs/revisions/text hashes and sample IDs. Change the prompt after approval: the earlier pilot must fail.

The local pilot validates this finite application contract. It does not submit a synchronous API request, validate a provider schema or establish representative extraction performance.

3. Reconcile partial results by identity

Implement reconcile_batch. Shuffled results contain one success, one expiry and one toy transient error; a fourth result is missing. Retain the successful sibling and prepare only eligible retries with the same ID, source and plan at attempt 2.

  • Duplicate, unknown or unrequested IDs and malformed envelopes reject the entire file before accepting new records.
  • Only expiry and the toy transient_server_error are unchanged retry candidates, once. Permanent/unknown errors, canceled requests and missing results require review.
  • On retry, another ID succeeds while invalid_request_error stays unresolved. A prior success cannot complete another ID or be rebound to a newer attempt. No third attempt is allowed.

Run python3 demo.py (Windows: py -3 demo.py) after implementation to inspect actual local normalized records, distinct totals, partial state and retry descriptors. Nothing is sent. Add an adversarial case.

Explain what your checks establish

Write a local MY_REVIEW.md: prompt revision/rationale, exact source and plan pins, raw-to-canonical traces, the printed discrepancy, actual commands/results and each ID's final disposition. Explain why a literal excerpt is insufficient and what these fixtures cannot establish. A person must assess your reasoning.

A new source, an old approval

Add a written reasoning case after your Python attempt: note B changes revision, explicitly names an unsupported currency and contains an ambiguous date, even though its totals agree. Decide what the earlier result establishes before proposing a new prompt or plan. This is a separate written case, not another executable v1 fixture.

The companion gives six human-review criteria and examples of adequate and insufficient reasoning. Use your existing MY_REVIEW.md as a starting point. A person must inspect the evidence; there is no automatic score, submission or measured prompt improvement.

Download the changed case and human rubric (Markdown)
Reveal the worked solution download

Keep your attempt. Extract the solution into a different empty folder, run the same 28 tests and demo, then compare WALKTHROUGH.md with your explanation. Its passing checks are finite artifact diagnostics, not a learner score.

Download the separate worked solution (ZIP)

Compare your written decision with a separate worked rationale. It explains why correct arithmetic and a real excerpt still cannot reuse the old approval.

Download the worked reasoning example (Markdown)

Evaluate review routing on held-out examples

Try a separate Python companion after the extraction lab. Select a review threshold using development rows, freeze that policy, then evaluate different held-out rows without tuning on them. Compare reference and date fields, account for their supplied population proportions, and route an unseen template to review even when its score is high.

The data and selection targets are synthetic; scores are uncalibrated. A high score is not a probability of correctness. Weighting assumes representative rows within each field; it cannot repair biased examples, unreliable labels or a changed template.

  1. Extract the starter into a new empty folder and open a terminal there. Read README.md, CONTRACT.md and WORKSHEET.md. Use Python 3.10 or newer; no packages, account or network service are needed.
  2. Run python3 -m unittest -v. All 15 starter checks intentionally error because three functions are unfinished. Implement them in evaluation.py, keeping the supplied tests unchanged.
  3. Run python3 run_exercise.py to inspect the frozen policy and actual local development, held-out and changed-case tables. On Windows, use py -3 in place of python3.
  4. Explain the accepted subset's weighted error rate, why an empty accepted subset has no conditional error estimate, and what new evidence a future policy needs. Use the worksheet's human rubric to review your reasoning.
Download the evaluation starter (ZIP)
Reveal the evaluation worked solution

Keep your own attempt and extract this into another empty folder. Run the same tests and script, then compare WALKTHROUGH.md with your arithmetic and written decisions. Passing tests verify this finite contract; a person must assess the reasoning. No submissions or learner scores are collected.

Download the evaluation worked solution (ZIP)

Pinned evidence has limits

The source annotations are supplied adjudication for four synthetic notes. Literal excerpt containment and a source hash do not establish general entailment or external truth. The schema is a finite application descriptor with handcrafted Python checks, not a general JSON Schema decoder.

Dictionaries remain editable. SHA pins bytes; it does not authenticate their author or provide immutable storage. IDs correlate local results; they do not create provider idempotency, concurrency safety or durable receipts. No Claude SDK, model, Messages API, real batch or financial transaction is executed. Passing this offline exercise does not establish model accuracy, mastery, transfer effectiveness or exam readiness. These are original CertLabs materials, not official exam questions.

Check your downloaded version

SHA-256 identifies the versioned ZIP bytes for comparison. It is not a signature or proof of correctness.

Starter ZIP SHA-256
41a251bd7c4983fd942f9d0474ba346f52c86c3f12c022b520d907b6a684bba3
Solution ZIP SHA-256
9f6de0cf2c146e1af340955206201f8097c94196be3d65328d1afcfdb4679c11
Reasoning exercise SHA-256
3cf3a6f92e23c991e26fb30c91a0aad47085b825b1eb554927ff5ecf5d34b88f
Worked reasoning SHA-256
62b8d29974320424924086237004b626e4c84bb43212a11a1ad71c6960ed22e6
Evaluation starter ZIP SHA-256
5e4a2378566450e1ad38bb449d5c5ff47beee34967335a2156c0284ab58bb110
Evaluation solution ZIP SHA-256
2ebe0d21cafe6f78a9ff4fdbb6e5a48d00fab18ca0af61775f20ae95a0ed33b2

For actual provider behavior, consult Anthropic's batch documentation. Our outcome envelopes, error allowlist and pilot are explicitly local teaching contracts.