# Five AI Projects Worth Building Five original AI engineering projects designed to help you show how you think: the boundary you chose, the failure you handled, and the evidence you collected. This page is the quick-read version. The [seven-page PDF](https://www.agenticamit.com/resources/five-ai-projects-worth-building/five-ai-projects-worth-building.pdf) adds an architecture diagram, suggested stack, build sequence, proof criteria, and metrics for every project. ## Pick the skill you want to prove | Project | What it demonstrates | Level | | --- | --- | --- | | Output Contract Firewall | Reliable model outputs and backend judgment | Intermediate | | Context Budget Studio | Context engineering and controlled experimentation | Intermediate | | Grounding Ledger | Retrieval, provenance, and trustworthy answers | Intermediate | | Agent Replay Lab | Agent observability and systems design | Advanced | | AI Change Impact Mapper | Developer tooling, CI, and evaluation design | Advanced | --- ## 1. Output Contract Firewall ### What it is A validation gateway between an LLM and downstream software. Every candidate response is accepted, repaired once, or quarantined after schema, business-rule, and policy checks. ### Build it 1. Define one versioned JSON contract and a set of valid and invalid examples. 2. Implement separate schema, semantic, and policy gates. 3. Add one constrained repair attempt, followed by full revalidation. 4. Log every decision and run normal, edge, and adversarial fixtures. ### You are done when - No known high-severity fixture is accepted. - Every request creates an audit record. - Validator failure defaults to quarantine. ### What to show Demo one clean pass, one successful repair, and one quarantine. Include the schema, rules, fixtures, metric report, and a redacted audit record. ### References - [OpenAI Structured Outputs](https://developers.openai.com/api/docs/guides/structured-outputs) - [OpenAI Evaluation Best Practices](https://developers.openai.com/api/docs/guides/evaluation-best-practices) - [OWASP Prompt Injection](https://genai.owasp.org/llmrisk/llm01-prompt-injection/) --- ## 2. Context Budget Studio ### What it is A laboratory for comparing which context should reach a model under a hard token budget. It makes quality, cost, latency, redundancy, and access decisions visible. ### Build it 1. Create a small dataset of queries, candidate context items, and expected facts. 2. Apply access controls before ranking anything. 3. Compare recency, similarity, value-per-token, and diversity-aware selection. 4. Plot answer quality against tokens, cost, and latency. ### You are done when - Every plan stays inside its budget. - No out-of-scope context reaches the model. - You can show at least one case where more context performs worse. ### What to show Run one query through every strategy. Include the dataset, context manifests, grader notes, and a quality-cost-latency comparison. ### References - [OpenAI Evals](https://developers.openai.com/api/docs/guides/evals) - [OpenAI Evaluation Best Practices](https://developers.openai.com/api/docs/guides/evaluation-best-practices) - [Redis Semantic Cache](https://redis.io/docs/latest/develop/use-cases/semantic-cache/) --- ## 3. Grounding Ledger ### What it is A claim-level evidence system. It connects every important answer claim to exact, versioned source spans and withholds answers when critical support is missing. ### Build it 1. Ingest a small corpus with stable source IDs, versions, and hashes. 2. Generate answers that reference source IDs. 3. Split answers into claims and link each claim to exact evidence. 4. Classify support as supported, partial, unsupported, or conflict. ### You are done when - Every critical claim opens to an exact source excerpt. - A changed source triggers rechecking. - Unsupported critical claims cause abstention. ### What to show Demo one supported answer, one conflict, and one abstention. Include the corpus manifest, claim schema, labeled verification sample, and exported ledger. ### References - [OpenAI Evaluation Best Practices](https://developers.openai.com/api/docs/guides/evaluation-best-practices) - [OpenAI Evals](https://developers.openai.com/api/docs/guides/evals) - [OWASP Prompt Injection](https://genai.owasp.org/llmrisk/llm01-prompt-injection/) --- ## 4. Agent Replay Lab ### What it is An append-only event system for reconstructing, replaying, forking, and comparing agent runs. Recorded replay is safe by default and performs no external writes. ### Build it 1. Define versioned events for model calls, tools, state changes, approvals, and errors. 2. Instrument one agent and store referenced artifacts by hash. 3. Rebuild state using recorded model and tool outputs. 4. Fork before a failure and compare output, tools, state, tokens, latency, and cost. ### You are done when - The final state can be rebuilt from recorded events. - Default replay has no external side effects. - Secrets are redacted before persistence. ### What to show Demo a failed run, recorded replay, fork, and first point of divergence. Include the event schema, redacted trace, migration test, and replay threat model. ### References - [OpenTelemetry GenAI Semantic Conventions](https://github.com/open-telemetry/semantic-conventions-genai) - [OWASP Prompt Injection](https://genai.owasp.org/llmrisk/llm01-prompt-injection/) - [OpenAI Evaluation Best Practices](https://developers.openai.com/api/docs/guides/evaluation-best-practices) --- ## 5. AI Change Impact Mapper ### What it is A dependency graph that maps prompt, model, tool, retrieval, policy, and dataset changes to the smallest defensible set of evaluations required in CI. ### Build it 1. Inventory prompts, tools, models, datasets, evals, dashboards, and owners. 2. Discover dependencies from source code and configuration. 3. Classify a pull-request diff and traverse its impact graph. 4. Generate required checks with a visible rule ID and reason path. ### You are done when - Every required check has an inspectable reason path. - High-risk changes need evaluation evidence or an audited waiver. - Stale and low-confidence dependencies remain visible. ### What to show Demo one pull request from diff to impact graph to CI decision. Include scanner rules, the fixture repository, a precision-recall report, and one audited waiver. ### References - [GitHub Actions Workflows](https://docs.github.com/en/actions/concepts/workflows-and-actions/workflows) - [OpenAI Evals](https://developers.openai.com/api/docs/guides/evals) - [OpenAI Evaluation Best Practices](https://developers.openai.com/api/docs/guides/evaluation-best-practices) --- ## The portfolio standard Whichever project you choose, make four things easy to inspect: 1. The task is specific. 2. The mechanism is visible. 3. The failure path is handled. 4. The evidence is attached. Build the system. Keep the receipts. Explain the tradeoffs.