Pattern 19, Episode 2: the full roadmap

The PROOF Roadmap: 8 weeks, 8 data and risk proof projects

One real job post, read as a test list. One project per requirement line. Ninety minutes a day, five days a week, sixty hours total. Every project ships in public, and every project ends in evidence a risk team can read: a log, a repo, a report.

Target post: Senior AI Engineer, Tangerine, Toronto (requisition 268884), checked 2026-10-19. The official page prints no salary band, so this plan makes no pay claim.

The theme of this post is different from a demo post. The duties name governance, risk culture, and root cause analysis next to the model work. So the roadmap builds the evidence habit, not just the models: gates, logs, postmortems, and a risk pack.

What is inside: a starting-skill check, a free setup guide, the exact prompt, eight week-by-week builds with done tests, failure fixes and a backup source for every week, a worked Week 1 build you can clone, and slower and private paths for readers with less time or a privacy need.

How to run this plan

  1. Open the target post. Save it as a PDF. Posts close; your copy does not.
  2. Run the exact prompt below on the post. Check every project against the real post lines. Cut anything the post does not ask for.
  3. Build in order. Each week ends with a public repo, a short README, and one evidence file the week is named for (a gate log, a leakage test, a postmortem).
  4. Apply wide from week 6. Save the dream post for last, when the proof can survive questions.

Before Week 1: check your starting skill

The 8 builds assume six base skills. Test yourself in ten minutes. For each no, take the short path first. Pre-week work sits outside the 60 hours. Budget 1 to 12 hours, by how many gaps you have.

  1. Can you load a CSV with pandas, filter rows, and compute a group average? If not: pandas intro tutorials 1 to 3, about 2 hours.
  2. Can you write a SQL query with a join and a group by? If not: SQLBolt, lessons 1 to 6 and 8, about 2 hours. This post lives on data products; SQL is not optional here.
  3. Can you init a git repo, push to GitHub, and read a GitHub Actions run log? If not: GitHub get started docs, about 2 hours.
  4. Can you say what a train and test split is, and why testing on training data lies? If not: Kaggle Learn, Intro to Machine Learning, about 3 hours.
  5. Can you call an HTTP API from Python and read the JSON that comes back? If not: the MDN HTTP overview, about 1 hour, then one practice call.
  6. Can you define, in one sentence each: an eval, a model card, and data drift? If not: Hugging Face LLM course, chapter 1, about 2 hours, plus the Week 1 brief's glossary box.

Rule: do not start Week 1 with more than one open gap. The builds stack. A gap in Week 1 becomes a wall in Week 5.

Setup: one time, about 2 hours, all free

Do these steps in order. The plan needs no paid tier. Free tiers have rate limits; the builds are sized to fit inside them.

  1. Install Python 3.11 or newer (python.org). Check it: run python3 --version.
  2. Install VS Code (free) and its Python extension.
  3. Install git. Make a free GitHub account. You will make one repo per week. GitHub Actions free minutes cover every pipeline in this plan.
  4. Install Docker Desktop (free). You need it in Week 4 and Week 5, but install it now so a slow download never eats a build day.
  5. Get the public datasets: the UCI Default of Credit Card Clients dataset and the UCI German Credit dataset (archive.ics.uci.edu, free). Both are public, old, and safe. Never train these builds on real client data, yours or an employer's.
  6. Get one free model API key (Google AI Studio free tier, Groq free tier, or OpenRouter free models) for the weeks that call a model. Weeks 1 and 3 need no key at all.
  7. Make a free Kaggle account for dataset copies and notebooks if you want a second run of any week.
  8. Plan your observability: local MLflow (pip install, runs on your machine, free) for Weeks 1 and 6, or plain log files. Both paths are in the briefs.

Store every key in an environment variable or a secrets file that git ignores. Never commit a key.

If your time or privacy is limited

The 60 hours are 60 hours. The daily dose is your choice.

Less time

At 30 minutes a day, five days a week, each project takes about three weeks and the full plan takes 24 weeks. For a 16-week path, the dose is about 45 minutes a day, five days a week. Same projects, same done tests.

Build in private

Privacy is a hard rule in this plan, not a preference. Every dataset named here is public. If you adapt a week to your own domain, use synthetic data or public data only. Never use employer data, employer code, bank client data, or screens from work. Check your contract first; some contracts claim side work. Keep each repo private while you build and publish the write-up instead: the method, the failure log, and the results table.

The exact prompt

Paste a job post after this prompt:

Copy the prompt
Read this job post as a test list. For each requirement line that names a skill, a duty, or a tool, write one small project that proves it. Rules: one project per line, eight projects at most, each project must fit in one week at ninety minutes a day, each project must end in a public artifact (a repo, a live demo, or a written report), each project must produce at least one piece of evidence a risk reviewer can check (a log, a test, a report, or a dashboard), and each project brief must name the post line it answers. End with a calendar: week, project, post line, artifact, evidence. No project may repeat a skill another project already proves.

Week by week

W1A training pipeline with a gate that can block a bad model

Post line it answersEngineering delivery of the AI platform: MLOps/LLMOps pipelines, model training, deployment, monitoring, governance.
BuildOne small pipeline that trains a credit-risk scorer on the public UCI credit dataset, scores it on a frozen test split, writes a model card and an eval record, and then faces a promotion gate: a script that promotes the model only if the eval record exists, beats a floor score you set before training, a data card exists, and an approval line with your name and date sits in the approvals log.
Done whenYou run the demo twice. With the eval record removed, the gate prints BLOCKED with the reason. With the records complete, the gate prints PASS and the model version. Both runs are saved in the repo.
Time split (7.5 hours)2 hours course, 4 hours pipeline and gate, 1.5 hours demo runs and README.
Worked exampleA real build of this week, with both gate runs in the README: the first run BLOCKED on missing records, the second PASS (AUC 0.724 on 6,000 frozen rows, hash verified). Clone it, watch the gate block a model, then rebuild it your way.
Glossary boxEval: a scored run of a model on data it did not train on, saved as a record with the score, the data hash, and the date. Model card: a short file that states what a model is, what data trained it, its score, and its limits. Data drift: the slow change of real input data away from the data a model trained on, which makes old scores stop holding. Promotion gate: a script that blocks a model from shipping until its evidence records check out.
Free courseMLOps Zoomcamp, DataTalksClub (free, self-paced)
Backup sourceMade With ML, the open production ML course
Common failures and fixes
  • The gate passes everything: the check runs after the copy step, so promotion already happened. Fix: order the script read records, decide, then copy. A gate that cannot say no is a decoration.
  • The eval score looks great and means nothing: the model was scored on its training rows. Fix: split first, freeze the test file, hash it, and let the gate check the hash so a swapped test set shows.
  • A retrain silently overwrites last week's model: no version rule. Fix: name every artifact with the date and the data hash, and never write to an existing version folder.

W2Retrieval over the rulebooks, with a scrub for private data

Post line it answersInfrastructure and tooling for model training and inference, Retrieval-Augmented Generation (RAG), context engineering, and intelligent agents.
BuildA RAG system over public banking rulebook texts (public regulator guidance PDFs, for example published OSFI and Bank of Canada documents), chunked with page numbers kept, that answers with citations, scrubs personal data patterns out of every question before it is embedded, and refuses with "not in the documents" when the texts are silent.
Done when20 test questions return cited answers, a test set with planted fake personal data is scrubbed and logged on every row, and three no-answer questions get the refusal, not a guess.
Time split (7.5 hours)1.5 hours tutorial, 4.5 hours pipeline, scrub, and refusal rule, 1.5 hours test sets and README.
Free courseLangChain RAG tutorial
Backup sourceMicrosoft Presidio documentation (the open-source personal-data detection tool)
Common failures and fixes
  • Answers cite the wrong page or no page: chunks are too big and metadata is dropped. Fix: cut 300 to 600 token chunks, keep the source file and page on every chunk, and print the top chunks for five questions before you trust a score.
  • Planted personal data sails into the prompt: the scrub runs after embedding, or not at all. Fix: scrub the raw question first, log what was removed, and keep the planted test set in CI so a refactor cannot quietly drop the step.
  • The system answers questions the rulebook never covers: no abstain rule. Fix: set a similarity floor; under it, the only allowed answer is "not in the documents", and test that case by name.

W3A feature store that cannot leak the future

Post line it answersData products, feature stores, and other data sources optimized for ML/AI training, inference, and agents.
BuildA miniature feature store over a public transactions-style dataset: three features computed by one shared code path, a registry file that versions each feature, point-in-time joins for training rows, and a leakage test that plants a future feature and must fail.
Done whenThe leakage test catches the planted feature by name, the same feature code serves both the training table and a single-row lookup, and the registry shows a version history.
Time split (7.5 hours)1.5 hours guide, 4.5 hours features and point-in-time joins, 1.5 hours leakage test and README.
Free courseFeast quickstart and docs (the open-source feature store)
Backup sourceHopsworks feature store documentation
Common failures and fixes
  • The offline score is wonderful and the live score is not: a feature used data from after the outcome. Fix: join features as of the event time only, and keep the planted-leak test green forever.
  • Training and serving disagree: two copies of the feature logic drifted apart. Fix: one module computes the feature; both paths import it. No copy-paste.
  • Stale features get served in silence: no freshness record. Fix: store the compute time with every feature value and alert when the newest value is older than your limit.

W4One model, three roads

Post line it answersDesign and implementation of the standard AI pipelines: real-time, batch, and hybrid.
BuildTake the Week 1 scorer and serve it three ways: a real-time API, a nightly batch job over a file, and a hybrid path that replays the day's requests through the batch scorer and compares. One shared request and response schema, and one contract test suite that runs against all three.
Done whenThe contract suite passes on all three roads, a schema change fails all three in CI, and the replay comparison reports where live and batch disagreed.
Time split (7.5 hours)1.5 hours docs, 4.5 hours the three paths and the contract suite, 1.5 hours load check and README.
Free courseApache Airflow official tutorial (the batch road)
Backup sourceFastAPI tutorial (the real-time road)
Common failures and fixes
  • Three roads, three different answers: each path grew its own preprocessing. Fix: the schema and the preprocessing live in one package that all three import.
  • The batch job skips bad rows and says nothing: no dead-letter rule. Fix: count rejected rows, write them to a dead-letter file, and fail the run past a threshold you set.
  • The API dies under a small load test: no timeout or fallback. Fix: set a request timeout, return a cached safe answer past it, and keep the load test in the repo.

W5A delivery pipeline that fails the build when quality drops

Post line it answersEngineering excellence in coding, architecture, DevOps, CI/CD, observability, test automation, and incident response.
BuildA GitHub Actions pipeline for the Week 4 stack: unit tests, the Week 1 golden cases as an eval gate, a schema check, and an image build. Then plant a regression (a worse prompt or a shifted threshold) and watch the eval gate turn the build red.
Done whenThe planted regression fails the build with the eval report attached to the run, the fix turns it green, and both runs are linked in the README.
Time split (7.5 hours)1.5 hours Actions and evals docs, 4.5 hours pipeline and gates, 1.5 hours regression drill and README.
Free courseGitHub Actions documentation, including the MLOps walkthroughs
Backup sourceOpenAI evals guide
Common failures and fixes
  • The eval gate is so slow the team disables it: the full set runs on every push. Fix: a 10-case smoke set on push, the full set on merge and nightly.
  • Gate results flip between runs with no code change: the judge or the set is noisy. Fix: grade with plain rules where you can, pin the judge model version, and hand-check 10 cases to audit the judge.
  • The gate grades a stale artifact: the eval step reads last week's model. Fix: the gate takes the artifact ID from the same run that built it, and prints that ID in the report header.

W6A cockpit that watches cost, drift, and errors

Post line it answersPlatform resilience: extensive monitoring, proactive issue detection, detailed root cause analysis for incidents.
BuildTracing on the Week 4 services (MLflow or plain structured logs), one dashboard page with cost per run, latency, and accuracy per model version, two alerts (input drift, error budget burn), and a weekly risk report page generated from the logs.
Done whenAn injected drift spike fires the drift alert, the spike and the alert both appear in the generated risk report, and the dashboard shows real runs, not sample data.
Time split (7.5 hours)1.5 hours docs, 4.5 hours tracing, dashboard, and alerts, 1.5 hours drift drill and the report.
Free courseEvidently AI documentation (open-source data and model monitoring)
Backup sourceLangfuse docs and quickstart
Common failures and fixes
  • The dashboard shows sample data: traces were never wired to the real service. Fix: run one request end to end and follow its trace ID from the first log line to the last.
  • The drift alert fires every day: the baseline window is too short or the threshold is a guess. Fix: build the baseline from a full validation period and set the threshold from its spread.
  • Cost reads zero: token and usage counts are not captured. Fix: read usage from each model response and store it on the run record before anything else reads it.

W7Break it on purpose, publish the root cause

Post line it answersRisk culture and risk appetite in day-to-day work; secure, compliant operation of the platform.
BuildA game day against your own Week 6 stack, on synthetic data only: kill the model server mid-traffic, poison one feature, and triple the request rate, one fault at a time. Recover each against the clock. Then write the root cause analysis the way a bank postmortem reads: timeline from the logs, detection, root cause, fix, and the prevention you actually shipped. End with a one-page runbook.
Done whenAll three drills recover inside a time limit you set before you start, the postmortem is public, and the runbook's steps are the steps you really used.
Time split (7.5 hours)1 hour reading, 4 hours drills and recovery, 2.5 hours postmortem and runbook.
Free courseGoogle SRE Book, the chapter on postmortem culture (free online)
Backup sourcePagerDuty Incident Response Guide
Common failures and fixes
  • The drill breaks something real: the game day ran on the only copy of the stack. Fix: clone the stack for game day, synthetic data only, and write the blast radius down before the first fault.
  • The postmortem blames a person: it stops at "someone deployed". Fix: ask why five times, and end each chain at a system change you shipped, not a name.
  • Recovery works once, by luck: the steps live in one head. Fix: run the drill a second time from the runbook alone, and time it.

W8A demo wrapped in a risk pack a bank can sign

Post line it answersMentoring, technical leadership, and partnership with technology, risk, analytics, and business teams.
BuildA hosted demo of the full stack, and the model risk pack that goes with it: intended use, the data card, the limits list (what the model must never do), the monitoring plan from Week 6, the approval checklist from Week 1, and the Week 7 postmortem as the honesty exhibit. Write the pack so a risk partner who has never met you can review it.
Done whenSomeone who is not you runs the demo from the README alone, and reviews the pack against its own checklist without asking you a question.
Time split (7.5 hours)1.5 hours framework reading, 3.5 hours pack and demo polish, 2.5 hours second-person test and fixes.
Free courseNIST AI Risk Management Framework and Playbook (the public US framework banks reference)
Backup sourceGradio quickstart (the demo surface)
Common failures and fixes
  • The pack reads like marketing: it lists strengths and hides limits. Fix: write the limits page first. A reviewer trusts the pack that names its own walls.
  • The demo runs only on your machine: paths and keys are hard-coded. Fix: read keys from environment variables, use relative paths, and test a clean clone in a fresh folder.
  • The reviewer cannot find the evidence: pack claims point at nothing. Fix: every claim in the pack links to a repo, a log, or a report from Weeks 1 to 7. Proof, not promises.

After week 8

  1. Rewrite the application profile: one line per project, each line with its repo link and its evidence file.
  2. Lead with the risk pack in applications to regulated teams. It answers the post's own language: governance, monitoring, root cause.
  3. Keep the Week 1 gate. It is now the first step of every model you ship, and the story you tell in interviews.
Disclosure: A story, a real method. The Episode 2 story is fiction performed by characters; Northloop Financial, the company named in that story, is fictional. Tangerine and its job post are real, and this roadmap is real. This roadmap is free. Use it, ship the projects, and send us the repo links.