Production Monitoring & Career Portfolio · 50 min · Python · Prometheus · Grafana
The Problem
AI-generated code that passes all quality gates can still fail in production. Behavior drifts over time as data changes, edge cases emerge, and AI models update. Without monitoring, issues accumulate silently. The "Debt Behind the AI Boom" study found that 22.7% of AI-introduced issues (105,364 of 464,900) still survive at HEAD 9+ months later.
Monitoring Layers
| Layer | Metric | Tool | Alert Threshold | Trigger |
|---|
| Error Rate | 5xx errors per 1K requests | Prometheus | > 1% | Page |
| Latency | p95 response time | Prometheus | > 500ms | Warn |
| Drift | Input distribution change | Custom | > 15% shift | Warn |
| Regression | Test pass rate trend | CI metrics | < 95% | Warn |
| Coverage | Code coverage trend | pytest-cov | < 90% | Warn |
| Mutation | Mutation score trend | mutmut | < 80% | Warn |
| Issue Survival | Open issues from AI code | GitHub API | > 100 open | Warn |
The 22.7% issue survival rate
The "Debt Behind the AI Boom" study analyzed 464,900 AI-introduced issues across open-source repositories. They found that 105,364 (22.7%) of these issues still survive at HEAD 9+ months after introduction. This means nearly 1 in 4 AI-introduced issues are never fixed. Without monitoring, these issues accumulate as technical debt, degrading code quality over time. The monitoring dashboard tracks issue survival rate to ensure AI-introduced issues are identified and addressed promptly.
What is behavior drift in AI-generated code, and how do you detect it?
Behavior drift in AI-generated code is a subtle but critical production issue. It occurs when code that worked correctly at deployment time stops working correctly over time, even though the code itself hasn't changed. This happens because the environment around the code changes. Here are the 4 causes and detection methods for each: (1) Input data distribution shift: The AI code was tested with certain types of inputs, but production receives different inputs over time. Example: An AI-generated payment processing function was tested with: (a) US phone numbers (10 digits). (b) USD amounts (positive, < $10,000). (c) US zip codes (5 digits). But in production, after expanding to international markets: (a) International phone numbers (variable length, + prefix). (b) JPY amounts (larger numbers, no decimals). (c) International postal codes (alphanumeric). The AI code may not handle these new input types correctly -- it was never tested with them. Detection: (a) Track input distribution: Log the types of inputs received (phone number format, currency, postal code format). (b) Compare current distribution to baseline: If 95% of inputs were US format at deployment but now 30% are international, that's a 65% shift. (c) Alert if shift > 15%: 'Input distribution has shifted significantly. Re-test with current input types.' (d) Re-run PBT with current inputs: Use the actual production inputs as Hypothesis strategies to find edge cases. (2) Dependency updates: A library that the AI code depends on changes behavior in a new version. Example: The AI code uses json.loads() to parse JSON. In Python 3.12, json.loads() changed how it handles duplicate keys (keeps the last instead of the first). The AI code assumed the first value would be kept. After a Python upgrade, the behavior changes silently. Detection: (a) Track dependency versions: Log the versions of all dependencies in production. (b) Alert on version change: When a dependency version changes, re-run the full test suite. (c) Use lockfile with hashes: Ensure the exact same versions are used in production as in testing. (d) Integration tests with production dependencies: Run integration tests against the actual production dependency versions. (3) Environment changes: The production environment differs from the test environment. Example: (a) Different Python version: Test uses 3.11, production uses 3.12. (b) Different OS: Test uses macOS, production uses Linux. (c) Different timezone: Test uses UTC, production uses PST. (d) Different locale: Test uses en_US, production uses ja_JP. The AI code may behave differently in different environments. Detection: (a) Track environment metadata: Log Python version, OS, timezone, locale. (b) Alert on environment change: When environment changes, re-run tests. (c) Test in production-like environment: Use Docker to match production environment. (d) Feature flags: Deploy new code behind a feature flag and monitor error rates before full rollout. (4) Model updates: If the AI code calls an AI model (e.g., GPT-4 for text classification), model updates can change behavior. Example: The AI code uses GPT-4 to classify support tickets as 'urgent' or 'normal'. After a GPT-4 model update: (a) The classification threshold changes. (b) Some tickets that were 'urgent' are now 'normal'. (c) Urgent tickets are missed, causing customer dissatisfaction. Detection: (a) Track model version: Log the model version used for each call. (b) Alert on model update: When the model version changes, re-run evaluation suite. (c) Golden master testing: Compare current model output to historical output for the same inputs. If outputs change significantly, investigate. (d) A/B testing: Run old and new model versions in parallel and compare results. Detection methods summary: (a) Input distribution monitoring: Track the distribution of inputs over time. Alert if distribution shifts > 15%. Use Prometheus to track input types and Grafana to visualize trends. (b) Output distribution monitoring: Track the distribution of outputs. Alert if output patterns change. Example: If 95% of payments were 'success' but now 85% are 'success', alert on the 10% drop. (c) Golden master testing: Store historical outputs for a set of representative inputs. Periodically re-run the code with the same inputs and compare outputs. If outputs differ, investigate. (d) Canary deployment: Deploy new code to 5% of traffic. Monitor error rates for 1 hour. If error rate < 1%, deploy to 50%. Monitor for 1 hour. If error rate < 1%, deploy to 100%. This catches drift before it affects all users."