Production Monitoring & Career Portfolio · 3 hours · Python · pytest · Hypothesis
System Architecture
┌─────────────────────────────────────────────────────────────┐
│ END-TO-END TESTING SYSTEM │
├─────────────────────────────────────────────────────────────┤
│ │
│ 1. TDD Workflow (Phase 1) │
│ ├── Write failing test → Generate code → Verify pass │
│ └── Coverage > 90% │
│ │
│ 2. Static Analysis Gauntlet (Phase 2) │
│ ├── Ruff (lint) │
│ ├── Bandit (Python SAST) │
│ ├── Semgrep (multi-language SAST) │
│ └── mypy --strict (type checking) │
│ │
│ 3. PBT Suite (Phase 3) │
│ ├── Hypothesis strategies for all functions │
│ ├── Roundtrip, monotonicity, bounded output properties │
│ └── Edge case generation (negative, zero, None, max) │
│ │
│ 4. Mutation Testing Pipeline (Phase 4) │
│ ├── mutmut in nightly CI │
│ ├── Mutation score > 80% │
│ └── Surviving mutant analysis │
│ │
│ 5. Security Gates (Phase 5) │
│ ├── SAST (Bandit + Semgrep) │
│ ├── Z3 formal verification for critical functions │
│ ├── Security Review prompt │
│ └── 0 CRITICAL vulnerabilities │
│ │
│ 6. Supply Chain Security (Phase 6) │
│ ├── Hallucinated package detection │
│ ├── Dependency allowlist │
│ ├── SBOM generation │
│ └── Lockfile enforcement │
│ │
│ 7. CI/CD Quality Gates (Phase 7) │
│ ├── 13 quality gates in GitHub Actions │
│ ├── 3-layer parallel execution │
│ ├── Intent verification prompt │
│ └── Block merge on any gate failure │
│ │
│ 8. Verification Framework (Phase 8) │
│ ├── YAML configuration │
│ ├── verify command │
│ ├── 13 testing + review prompts │
│ └── Markdown dashboard │
│ │
│ 9. Production Monitoring (Phase 9) │
│ ├── 7 monitoring layers │
│ ├── Drift detection │
│ ├── Issue survival tracking │
│ └── Prometheus + Grafana dashboard │
│ │
└─────────────────────────────────────────────────────────────┘
The integration challenge
Each phase of the course teaches a single technique. The capstone project integrates all 9 techniques into a single system. The challenge is not learning each technique (you've already done that) -- it's making them work together. The system must: (1) Run all checks in a single pipeline. (2) Produce a unified report. (3) Block merges that don't meet quality thresholds. (4) Monitor production after deployment. (5) Be reusable across projects.
What are the 9 components of the end-to-end testing system, and how do they integrate?
The 9 components of the end-to-end testing system integrate all techniques from Phases 0-8 into a single, cohesive pipeline. Here's how they work and integrate: (1) TDD Workflow (Phase 1): What it does: Ensures all AI-generated code is developed test-first. The TDD workflow is: (a) Write a failing test for the desired behavior. (b) Ask AI to generate code that passes the test. (c) Run the test to verify it passes. (d) Refactor if needed. (e) Repeat for each feature. Quality threshold: Coverage > 90%. Integration: The TDD workflow is the entry point -- all AI-generated code enters the system through TDD. The tests written during TDD are the foundation for all subsequent checks (PBT, mutation, CI/CD). (2) Static Analysis Gauntlet (Phase 2): What it does: Runs 4 static analysis tools on all AI-generated code: (a) Ruff: Linting (PEP 8, import sorting, unused variables). (b) Bandit: Python-specific SAST (finds common security issues). (c) Semgrep: Multi-language SAST (custom rules for AI-specific patterns). (d) mypy --strict: Type checking (catches hallucinated APIs, type errors). Quality threshold: 0 errors from all 4 tools. Integration: The static analysis gauntlet runs first in the CI pipeline (before tests) because it's fast (< 30 seconds) and catches obvious issues early. If any tool finds errors, the pipeline stops and the PR is blocked. (3) PBT Suite (Phase 3): What it does: Runs property-based tests using Hypothesis. For each AI-generated function: (a) Define properties (roundtrip, monotonicity, bounded output). (b) Define Hypothesis strategies (custom value generators). (c) Run PBT to generate 100+ test cases per property. (d) Report any property violations. Quality threshold: All properties pass. Integration: PBT runs after static analysis (before mutation testing) because it's medium-speed (1-5 minutes) and catches behavioral bugs that static analysis misses. PBT complements unit tests by generating edge cases that the developer might not think of. (4) Mutation Testing Pipeline (Phase 4): What it does: Runs mutmut to generate mutants (small code changes) and checks if the test suite catches them. (a) Generate mutants (mutmut run). (b) Run test suite against each mutant. (c) Identify surviving mutants (tests don't catch them). (d) Calculate mutation score (killed mutants / total mutants). (e) Write tests to kill surviving mutants. Quality threshold: Mutation score > 80%. Integration: Mutation testing runs nightly (not on every PR) because it's slow (10-60 minutes). The nightly mutation report is posted to the dashboard. If mutation score < 80%, a warning is issued (not a merge block) because mutation testing is for continuous improvement, not gating. (5) Security Gates (Phase 5): What it does: Runs security checks on all AI-generated code: (a) SAST (Bandit + Semgrep): Catches known vulnerability patterns. (b) Z3 formal verification: Proves correctness of critical functions (e.g., payment processing, authentication). (c) Security Review prompt: AI reviews code for novel vulnerabilities SAST misses. (d) Vulnerability report: Severity, CWE ID, fix for each finding. Quality threshold: 0 CRITICAL vulnerabilities. Integration: Security gates run in the CI pipeline after static analysis and before tests. If any CRITICAL vulnerability is found, the PR is blocked. HIGH vulnerabilities are warnings (not blocks). MEDIUM and LOW are informational. (6) Supply Chain Security (Phase 6): What it does: Checks all dependencies for supply chain attacks: (a) Hallucinated package detection: Verify all imports against PyPI registry. (b) Dependency allowlist: Only allow pre-approved packages. (c) SBOM generation: Create Software Bill of Materials. (d) Lockfile enforcement: Require lockfile with hash verification. Quality threshold: 0 hallucinated packages, all dependencies on allowlist. Integration: Supply chain checks run in the CI pipeline before tests. If any hallucinated package is detected, the PR is blocked (prevents slopsquatting attacks). The SBOM is generated and stored as an artifact. (7) CI/CD Quality Gates (Phase 7): What it does: Runs 13 quality gates in a 3-layer parallel pipeline: Layer 1 (parallel, fast): Ruff, Bandit, Semgrep, mypy, supply chain check. Layer 2 (parallel, medium): pytest, Hypothesis PBT, security review prompt. Layer 3 (sequential, slow): mutation testing (nightly), intent verification, coverage check, quality scoring. Gate policy: Block merge if any Layer 1 or Layer 2 gate fails. Warn (don't block) if Layer 3 gates fail. Integration: The CI/CD pipeline orchestrates all previous components. It runs them in the correct order, with the correct parallelism, and enforces the gate policy. The pipeline is defined in GitHub Actions YAML and runs on every PR. (8) Verification Framework (Phase 8): What it does: Provides a reusable framework for running all checks: (a) YAML configuration: Define which checks to run, thresholds, and tools. (b) verify command: Run 'python verify.py' to execute all checks. (c) 13 prompts: 7 testing prompts + 6 review prompts, all reusable. (d) Markdown dashboard: Generate a unified report with all results. Integration: The verification framework wraps all previous components into a single, reusable system. Instead of running each tool separately, the developer runs 'python verify.py' and gets a unified report. The YAML configuration makes it easy to customize for different projects. (9) Production Monitoring (Phase 9): What it does: Monitors AI code health in production: (a) 7 monitoring layers: Error rate, latency, drift, regression, coverage, mutation, issue survival. (b) Drift detection: Input/output distribution shifts > 15%. (c) Issue survival tracking: 22.7% survival rate, age distribution. (d) Prometheus + Grafana: Metrics collection and dashboard. Integration: Production monitoring runs after deployment (not in CI). It monitors the AI-generated code that has been deployed to production. If any metric exceeds its threshold, an alert is triggered (Slack for warnings, PagerDuty for critical). How the 9 components integrate: (1) Entry point: AI-generated code enters through TDD (1). (2) Pre-merge checks: Static analysis (2), PBT (3), security gates (5), supply chain (6), and CI/CD quality gates (7) run on every PR. (3) Nightly checks: Mutation testing (4) runs nightly for continuous improvement. (4) Unified interface: The verification framework (8) wraps all checks into a single command. (5) Post-deployment: Production monitoring (9) runs after deployment. (6) Reporting: All components report to the unified markdown dashboard (8). (7) Prompt library: 13 prompts (8) are used by security gates (5) and CI/CD (7). The result is a system that: (a) Catches issues before merge (pre-merge checks). (b) Continuously improves test quality (nightly mutation). (c) Monitors production health (post-deployment). (d) Is reusable across projects (YAML config). (e) Produces a unified report (markdown dashboard). This is the end-to-end testing system that you'll build as the capstone project."