CCAR-F / Free hands-on exercise / Version 1
Can you trust a well-formed extraction?
Normalize synthetic shipping notes, pin their source versions, then keep successful results while investigating failed siblings. A correct shape and a real excerpt can still accompany an unsupported claim.
Run locally with Python 3.10 or newer and its standard library. No package installation, sign-in, key or network service is needed after download. Basic dictionaries, exceptions and unit tests are helpful. Your code, prompt and written review stay on your computer; this page accepts no submissions or completion history.
Download the Python starter (ZIP)Start with failing tests
- Check
python3 --version(Windows:py -3 --version). - Extract into a new empty folder. Read
README.md,CONTRACT.md, the supplied fixtures and tests. - Open a terminal in the extracted folder and run the command below. Four supplied checks pass; 24 intentionally fail because three functions are unfinished. These counts describe the starter, not your score. Keep the supplied tests unchanged.
python3 -m unittest -v
Windows: py -3 -m unittest -v. No setup script or package installation is needed. Use a disposable folder separate from real documents or customer work.
1. Normalize without guessing
Implement normalize_record in extraction.py. Raw amounts are explicit decimal major-unit strings. Use integer/string arithmetic, preserve the source identity and validate against the finite annotations.
" usd " → "USD" "7" + "5.0" → [700, 500] cents "0.29" → 29 cents, without float rounding EUR discount "-2.00" → -200 cents Explicit "2026-11-12" → same ISO date Date absent in note → null
Change from note A's USD/no-date case to note B's EUR/discount/date case. Reject bool-as-integer, comma locale, ambiguous dates, invented values and mismatched source revisions. Do not infer a currency from $ or turn 11/12/2026 into a guessed date.
Note D explicitly states EUR 5.40; its line and surcharge calculate EUR 5.50. Inspect both values separately. Reject acceptance and keep the original discrepancy for review. Never repair the printed claim silently.
2. Write a prompt and pin a pilot
Refine PROMPT.md with an original USD/no-date example, the changed EUR example and explicit missing/ambiguous-value rules. Update its revision in fixtures.py. This is a handwritten prompt artifact; no model runs and the tests do not grade its quality.
Implement approve_pilot for the two designated samples before constructing local batch descriptors. Bind the exact prompt, normalization/schema revisions, source IDs/revisions/text hashes and sample IDs. Change the prompt after approval: the earlier pilot must fail.
The local pilot validates this finite application contract. It does not submit a synchronous API request, validate a provider schema or establish representative extraction performance.
3. Reconcile partial results by identity
Implement reconcile_batch. Shuffled results contain one success, one expiry and one toy transient error; a fourth result is missing. Retain the successful sibling and prepare only eligible retries with the same ID, source and plan at attempt 2.
- Duplicate, unknown or unrequested IDs and malformed envelopes reject the entire file before accepting new records.
- Only expiry and the toy
transient_server_errorare unchanged retry candidates, once. Permanent/unknown errors, canceled requests and missing results require review. - On retry, another ID succeeds while
invalid_request_errorstays unresolved. A prior success cannot complete another ID or be rebound to a newer attempt. No third attempt is allowed.
Run python3 demo.py (Windows: py -3 demo.py) after implementation to inspect actual local normalized records, distinct totals, partial state and retry descriptors. Nothing is sent. Add an adversarial case.
Explain what your checks establish
Write a local MY_REVIEW.md: prompt revision/rationale, exact source and plan pins, raw-to-canonical traces, the printed discrepancy, actual commands/results and each ID's final disposition. Explain why a literal excerpt is insufficient and what these fixtures cannot establish. A person must assess your reasoning.
A new source, an old approval
Add a written reasoning case after your Python attempt: note B changes revision, explicitly names an unsupported currency and contains an ambiguous date, even though its totals agree. Decide what the earlier result establishes before proposing a new prompt or plan. This is a separate written case, not another executable v1 fixture.
The companion gives six human-review criteria and examples of adequate and insufficient reasoning. Use your existing MY_REVIEW.md as a starting point. A person must inspect the evidence; there is no automatic score, submission or measured prompt improvement.
Reveal the worked solution download
Keep your attempt. Extract the solution into a different empty folder, run the same 28 tests and demo, then compare WALKTHROUGH.md with your explanation. Its passing checks are finite artifact diagnostics, not a learner score.
Compare your written decision with a separate worked rationale. It explains why correct arithmetic and a real excerpt still cannot reuse the old approval.
Download the worked reasoning example (Markdown)Evaluate review routing on held-out examples
Try a separate Python companion after the extraction lab. Select a review threshold using development rows, freeze that policy, then evaluate different held-out rows without tuning on them. Compare reference and date fields, account for their supplied population proportions, and route an unseen template to review even when its score is high.
The data and selection targets are synthetic; scores are uncalibrated. A high score is not a probability of correctness. Weighting assumes representative rows within each field; it cannot repair biased examples, unreliable labels or a changed template.
- Extract the starter into a new empty folder and open a terminal there. Read
README.md,CONTRACT.mdandWORKSHEET.md. Use Python 3.10 or newer; no packages, account or network service are needed. - Run
python3 -m unittest -v. All 15 starter checks intentionally error because three functions are unfinished. Implement them inevaluation.py, keeping the supplied tests unchanged. - Run
python3 run_exercise.pyto inspect the frozen policy and actual local development, held-out and changed-case tables. On Windows, usepy -3in place ofpython3. - Explain the accepted subset's weighted error rate, why an empty accepted subset has no conditional error estimate, and what new evidence a future policy needs. Use the worksheet's human rubric to review your reasoning.
Reveal the evaluation worked solution
Keep your own attempt and extract this into another empty folder. Run the same tests and script, then compare WALKTHROUGH.md with your arithmetic and written decisions. Passing tests verify this finite contract; a person must assess the reasoning. No submissions or learner scores are collected.
Pinned evidence has limits
The source annotations are supplied adjudication for four synthetic notes. Literal excerpt containment and a source hash do not establish general entailment or external truth. The schema is a finite application descriptor with handcrafted Python checks, not a general JSON Schema decoder.
Dictionaries remain editable. SHA pins bytes; it does not authenticate their author or provide immutable storage. IDs correlate local results; they do not create provider idempotency, concurrency safety or durable receipts. No Claude SDK, model, Messages API, real batch or financial transaction is executed. Passing this offline exercise does not establish model accuracy, mastery, transfer effectiveness or exam readiness. These are original CertLabs materials, not official exam questions.
Check your downloaded version
SHA-256 identifies the versioned ZIP bytes for comparison. It is not a signature or proof of correctness.
- Starter ZIP SHA-256
- 41a251bd7c4983fd942f9d0474ba346f52c86c3f12c022b520d907b6a684bba3
- Solution ZIP SHA-256
- 9f6de0cf2c146e1af340955206201f8097c94196be3d65328d1afcfdb4679c11
- Reasoning exercise SHA-256
- 3cf3a6f92e23c991e26fb30c91a0aad47085b825b1eb554927ff5ecf5d34b88f
- Worked reasoning SHA-256
- 62b8d29974320424924086237004b626e4c84bb43212a11a1ad71c6960ed22e6
- Evaluation starter ZIP SHA-256
- 5e4a2378566450e1ad38bb449d5c5ff47beee34967335a2156c0284ab58bb110
- Evaluation solution ZIP SHA-256
- 2ebe0d21cafe6f78a9ff4fdbb6e5a48d00fab18ca0af61775f20ae95a0ed33b2
For actual provider behavior, consult Anthropic's batch documentation. Our outcome envelopes, error allowlist and pilot are explicitly local teaching contracts.