AI / LLM Testing & Automation Roadmap

Goal: Recognize which SDET skills transfer directly, so you can see exactly where the new skills in Stage 1 onward have to attach.

These skills transfer directly and are assumed throughout the rest of the roadmap:

• Test design fundamentals — equivalence classes, boundary values, positive/negative cases.
• Scripting and automation frameworks (Selenium, Playwright, REST-assured, Postman/Newman, etc.).
• CI/CD pipelines, version control, and treating test assets as code.
• API testing — most LLM interactions happen over REST/streaming APIs, so this is directly reusable.
• Reading logs, debugging failures, and writing clear defect reports.

What's new is not how to test but what you're testing: systems that are non-deterministic, graded on quality rather than pass/fail, and vulnerable to failure modes traditional software doesn't have (hallucination, prompt injection, bias).

After this stage, you can: name exactly which of your existing SDET skills carry over, and where AI-specific testing has to add something new.

Goal: Understand what a large language model actually is, how LLM-powered applications are built, and the vocabulary you'll need for every later stage.

1.1 What is an LLM, really?

• Tokens & context window — text is broken into tokens; the model can only "see" a limited number at once. Long inputs get truncated or pushed out of context.
• Prompt & completion — the prompt is your input (instructions + data); the completion is what the model generates.
• Temperature & top-p — settings controlling randomness. Higher temperature = more varied, less predictable output — directly relevant to why the same test can pass once and fail the next run.
• Inference vs. training — as a tester you work almost entirely at inference time (calling an already-trained model).
• Non-determinism — the same input can legitimately produce different, equally "correct" outputs. This single fact is why exact-match assertions stop working, and it's the thread running through this entire roadmap.

1.2 How LLM applications are actually built

• Plain prompting — instructions and examples crafted directly in the prompt; no extra data sources.
• Retrieval-Augmented Generation (RAG) — the app fetches relevant documents (via embeddings + a vector database) and feeds them as context, so answers are grounded in real data.
• Fine-tuning — further training the base model's weights on domain data. Rarer than prompting/RAG, but know when it's chosen: consistent style/format, domain jargon, cost reduction at scale.
• Agents — the model is given tools/functions it can call and reasons across multiple steps. Multi-agent systems chain several specialized agents together.

1.3 Talking to LLMs programmatically

• REST APIs from Anthropic, OpenAI, Google, plus locally-hosted open models (e.g. via Ollama) for cheaper, offline experimentation.
• The system/user/assistant message structure, and how system prompts set behavior boundaries.
• Function/tool calling — how a model requests that your code run a function and returns the result.
• Streaming responses — most production chat UIs stream tokens, which affects how you capture output for testing.
• Basic use of a provider SDK in Python or JS/TS — whichever matches your existing automation stack.

1.4 Why LLM testing breaks traditional assumptions

Keep this list handy for the rest of the roadmap — every later stage is really just building tooling to handle these points systematically.

• No single "correct" output — quality is graded on a spectrum, not asserted pass/fail.
• Non-determinism — identical inputs can produce different outputs, so a single run proves very little.
• New failure modes: hallucination, bias, prompt injection, inconsistency across runs, unpredictable cost/latency.
• The system under test can be updated silently by the vendor (a model version change), shifting behavior with no code change on your side.

After this stage, you can: explain, in plain terms, why the same prompt run three times can legitimately produce three different, equally correct answers — and why exact-match assertions break down as a result.

Goal: Learn the central skill of AI testing: designing a reliable oracle for a probabilistic, quality-graded system, then automating that judgment so it runs on every change. Every later stage is this same discipline applied to a new layer.

2.1 What exactly are we testing? The AI application stack

An AI application is not just "the model." A wrong answer can originate in any layer — a senior tester's first question is never "was this correct," it's "which layer caused this."

A single symptom — "the answer was wrong" — can trace back to any of these causes:

• Model failure — the model reasoned incorrectly even with the right information.
• Prompt failure — instructions were ambiguous, contradictory, or missing a needed constraint.
• Retrieval/chunking/embedding failure — the wrong (or no) supporting content was fetched.
• Context-selection failure — the right content was retrieved but didn't make it into the final prompt (truncation, ranking, context-window limits).
• Tool or orchestration failure — a function call used the wrong arguments, or steps ran in the wrong order.
• Safety-filter failure — a guardrail over- or under-blocked legitimate content.
• Stale data — the underlying knowledge source was out of date.
• Evaluation failure — the output was actually fine and your test oracle scored it wrong (see 2.5).

Stage 3 applies this to the retrieval layer, Stage 5 to guardrails/safety, Stage 6 to orchestration/tools.

2.2 The testing types you now need to add

• Functional/output-quality — does the response answer the question and follow instructions (format, tone, constraints)?
• Consistency & regression — does behavior stay acceptably stable across prompt edits and model/version upgrades?
• Safety & compliance — toxicity, bias, leakage of personal or sensitive information.
• Robustness — edge cases, malformed input, adversarial phrasing, out-of-scope requests.
• Performance & cost — latency and token cost per request, more like a variable resource budget than fixed compute cost.

2.3 Building test oracles: golden, regression, adversarial, boundary & synthetic datasets

A golden dataset is your evaluation suite's raw material — version-controlled inputs paired with expected behavior or scoring criteria, often a rubric rather than an exact string. Treat it as several purpose-built slices:

• Golden cases — known-good, carefully reviewed examples that define "correct" for the feature.
• Regression cases — real failures found in testing or production, added the moment you find them so they can never silently reappear.
• Adversarial cases — inputs deliberately designed to break the system: contradictory instructions, prompt-injection attempts, edge-of-policy requests.
• Boundary cases — inputs near a system limit: max context length, empty input, extreme values, unsupported languages.
• Synthetic/generated cases — LLM- or script-generated cases to expand coverage quickly; always have a human validate a sample before trusting them.

2.4 Evaluation methods & metrics — the testing pyramid

You'll typically combine several methods — no single one covers everything. The guiding rule: use the cheapest, most deterministic check that can actually answer the question, and reach for a model-based judge only when nothing below it in the pyramid can.

Narrower, more expensive, more subjective checks belong near the top — not as the default.

• Deterministic checks — exact match, regex, JSON-schema validation, contains/excludes a keyword. Fast, cheap, the right first line of defense wherever output has a predictable shape.
• Similarity-based scoring — embedding/cosine similarity; BLEU/ROUGE exist but are of limited use for open-ended text — know they exist, don't over-rely on them.
• LLM-as-judge — a second LLM call scores output against a written rubric (e.g. 1–5 on relevance, correctness, tone). Scales well, captures nuance, but costs money and can carry its own biases.
• Human evaluation — still the gold standard for subjective quality. Use structured rubrics and, where possible, more than one rater to check agreement.

Two examples of picking the right level: if output must be valid JSON, parse it — don't ask a second LLM "is this valid JSON?" If output must contain a specific ID, assert on it directly. Reserve LLM-as-judge and human review for genuinely subjective quality — tone, helpfulness, nuance — where no deterministic check is possible.

2.5 The evaluator is itself a system that needs testing

The moment you introduce LLM-as-judge, you've added a second model call between "the system did something" and "you know whether it was right" — and that judge is itself non-deterministic and can be wrong.

A judge score is an opinion from another model, not ground truth — test it like any other oracle:

• Judge consistency — same score on the same input across repeated runs?
• Judge bias — does it systematically favor longer answers, certain phrasing, or its own model family's style?
• Rubric quality — specific enough that two reviewers would apply it the same way?
• Calibration — do scores track human judgment across the whole range, not just the extremes?
• False positives/negatives — track cases the judge passed that a human would fail, and vice versa.
• Human agreement — periodically sample judge scores against human ratings; alert if agreement drops.
• Judge/model version drift — a judge model upgrade can silently shift your whole scoring scale.

2.6 Prompt regression & version control

Treat prompts as code: store them in version control, diff every change, and re-run your golden-dataset suite before any prompt or model change ships. A prompt regression suite turns "someone tweaked the prompt and something broke" from a production incident into a caught pull-request failure.

2.7 Tools to learn — one per category

After this stage, you can: build an automated evaluation suite that determines whether an LLM response meets defined quality criteria.

Goal: Test the most common production LLM pattern — retrieval-augmented generation — end to end, not just the final generated answer.

3.1 RAG architecture in plain terms

Documents are split into chunks, converted to embeddings, and stored in a vector database. At query time, the user's question is embedded and used to retrieve the most relevant chunks, which are inserted into the prompt as context so the LLM can generate a grounded answer. Bad chunking, poor retrieval ranking, or a context window too small to hold everything relevant can all produce a wrong answer even when the underlying model is fine — which is why RAG needs its own layer of testing.

3.2 RAG quality has three separate dimensions

Resist scoring a RAG system with one blended number. "Wrong answer" can mean retrieval fetched the wrong thing, the model didn't use what was retrieved, or the answer didn't address the question — three different bugs with three different owners.

Treat chunking strategy and retrieval tuning as a testing concern, not purely an engineering one — a failing faithfulness score is often a retrieval bug wearing a generation-bug costume.

3.3 Tools to learn

• Ragas — purpose-built RAG metrics (faithfulness, context precision/recall, answer relevancy) — the standard starting point.
• TruLens — tracing plus configurable "feedback functions" for scoring RAG and general LLM app behavior.
• Arize Phoenix — open-source observability and evaluation, with strong tracing visualizations.
• Langfuse — open-source tracing and analytics, useful for connecting production traces back into your eval datasets.

After this stage, you can: diagnose whether a RAG failure originated in retrieval, grounding, or answer generation.

Goal: Move LLM evaluation out of a one-off notebook and into a repeatable, automated, production-grade quality gate.

4.1 Wiring LLM evals into CI/CD

• Run your golden-dataset regression suite automatically on every prompt or model change, the same way you'd run unit tests.
• Set score thresholds as build gates — similar in spirit to a code-coverage gate, but for quality/safety scores.
• Use shadow or canary testing to run a new prompt or model version against a sample of real traffic before full rollout.

4.2 Observability & tracing

Multi-step LLM calls (prompt → retrieval → tool call → final answer) are hard to debug from logs alone — you need tracing that shows the full chain for each request.

• LangSmith — tracing, dataset management, and evaluation, tightly integrated with the LangChain ecosystem.
• Arize Phoenix / Langfuse — open-source alternatives for tracing and evaluation, useful when avoiding vendor lock-in.
• Helicone — lightweight logging/observability proxy for LLM API calls, useful for quick cost and latency visibility.

A strong habit: every production bug you trace should become a new regression case in your golden dataset — this closes the loop between what breaks in the wild and what your automated suite catches next time.

4.3 Performance, load & cost testing

• Track latency at P50/P95, not just averages — LLM latency is far less uniform than typical API latency.
• Load-test streaming endpoints with tools like k6 or Locust, adapted to handle streamed token responses rather than single JSON payloads.
• Track token usage and cost per request/session as a first-class metric — a prompt change that improves quality but doubles token cost is a real regression.

4.4 Production monitoring

• Watch for silent drift — a vendor's model update changing behavior with no code change on your side.
• Build feedback loops from real user signals (thumbs up/down, support escalations) back into your eval datasets.
• Alert on quality or safety score regressions the same way you'd alert on error-rate spikes.

After this stage, you can: put AI evaluations into CI/CD and monitor behavior in production.

Goal: Test for the failure modes that never show up in normal functional testing: safety, security, and deliberate misuse.

5.1 The OWASP LLM Top 10 as a living test checklist

Use the OWASP GenAI Security Project's current LLM Top 10 as a standing checklist rather than a one-time read — it's actively revised, so always work from whichever edition is current, not a dated PDF fixed in your notes. Core categories to map test cases against:

• Prompt injection (direct and indirect, via retrieved documents or tool output).
• Insecure output handling (unsafely trusting or executing model output).
• Training data poisoning and supply-chain risk for third-party models/plugins.
• Model denial-of-service and excessive resource consumption.
• Sensitive information disclosure (leaking PII, secrets, or system prompts).
• Insecure plugin/tool design and excessive agency (an agent doing more than it should).
• Overreliance — humans trusting AI output without adequate verification.

5.2 Prompt injection & jailbreak testing

• Direct injection — an attacker puts malicious instructions straight into their prompt to override system rules.
• Indirect injection — malicious instructions hidden inside a document, webpage, or tool result the model later reads — especially dangerous in RAG and agent systems.
• Build (or reuse) automated jailbreak test suites rather than relying on manually trying a handful of tricks.

5.3 Bias, fairness, toxicity & PII testing

• Maintain structured bias test sets that vary a sensitive attribute while holding everything else constant, and compare outputs.
• Use toxicity classifiers as an automated first pass, backed by human review for anything flagged.
• Explicitly test for PII leakage — both in normal use and under adversarial prompting designed to extract it.

5.4 Red-teaming tools

• Promptfoo (red-team module) — generates adversarial test cases automatically, maps results to known vulnerability categories.
• DeepTeam — an LLM red-teaming framework explicitly organized around the OWASP LLM Top 10.
• Garak — an automated LLM vulnerability scanner for probing a wide range of known attack types.

Manual, structured red-teaming exercises still matter — document methodology, findings, and severity the way a security team would.

5.5 Governance basics

• Keep audit trails for high-risk agent actions.
• Require human-in-the-loop approval before an agent can take irreversible or high-impact actions.
• Adopt a lightweight responsible-AI checklist for any new LLM feature before it ships.

The OWASP GenAI Security Project also maintains guidance specific to agentic AI, covering excessive agency, unsafe tool chaining, and identity/authorization across multi-agent systems. Once you reach Stage 6, layer that guidance on top of this checklist rather than treating agent security as a separate topic.

After this stage, you can: systematically test AI applications for security and safety failures.

Goal: Apply everything so far to the hardest LLM testing problem: autonomous agents that plan, call tools, and take multi-step actions.

6.1 What's different about agents

• Multi-step reasoning and planning, rather than a single prompt/response exchange.
• Tool/function calling — the agent decides which tool to use and with what arguments.
• Memory and state carried across turns or steps.
• Non-linear execution paths — the same task can legitimately be solved via different sequences of steps.

6.2 Agent evaluation dimensions

• Task completion rate — did the agent actually accomplish the goal, end to end?
• Tool-call accuracy — did it choose the right tool, and pass correct arguments?
• Trajectory/path evaluation — was the sequence of steps reasonable and efficient, not just the final answer?
• Efficiency — how many steps, tool calls, and tokens did it take?
• Action safety — did it avoid destructive or irreversible actions it wasn't authorized to take?

These dimensions score the path and the outcome — but an agent can produce a correct final answer while having done something you never wanted along the way. That's the subject of the next section.

6.3 State & side-effect validation

Testing only the final response isn't enough for an agent that can act on the world. A correct closing message can still sit on top of a real-world action that should never have happened — e.g. "cancel my subscription."

A correct-looking "Done" tells you nothing about whether every step above it was safe and authorized. This is where AI testing turns into systems testing rather than response grading. For any agent that can take real actions, test each link explicitly:

• Intent — was the user's request interpreted correctly, including what they did not ask for?
• Plan — is the sequence of steps reasonable, minimal, and reversible where possible?
• Tool selection & arguments — right tool, right parameters, no dangerous defaults.
• Authorization — did the agent check it was actually allowed to take this action, for this user, before acting?
• Side effect — did real-world state change exactly as intended, and only as intended (verify against the system of record, not the agent's own claim)?
• Final response — does what the agent reports match what it actually did?

6.4 Tools to learn

• LangSmith — tracing and evaluation for agent trajectories, tightly coupled with LangChain/LangGraph agents.
• MLflow (GenAI/agent evaluation) — experiment tracking extended to prompts, evals, and agent traces.
• Arize Phoenix — agent-aware tracing and evaluation, framework-agnostic.
• Dedicated agent-eval platforms (e.g. Maxim AI, Galileo) — a growing category worth knowing exists; evaluate against your team's specific needs.

6.5 MCP is another test boundary, not another framework to learn

The Model Context Protocol (MCP) is a common standard for connecting agents to external tools and data sources. Don't treat "learning MCP" as a separate skill — treat every MCP connection as a boundary your agent crosses, and test it like any integration boundary.

Each item below is a concrete thing to write a test case for — the same way you'd test any API boundary:

• Schema — does the agent correctly interpret a tool's schema and pass valid, correctly-typed arguments?
• Input/output validation — does it reject or safely handle malformed tool output instead of hallucinating a plausible-looking result?
• Failure handling — does it handle a tool timing out or erroring gracefully rather than guessing?
• Authentication & authorization — are tool permissions scoped correctly, so the agent can't reach data or actions outside its intended boundary?

This is exactly where the OWASP agentic AI guidance from Stage 5.5 applies most directly — excessive agency and unsafe tool chaining are boundary-crossing failures.

After this stage, you can: evaluate not only what an agent says, but what it does.

Goal: Flip the perspective — use AI and LLMs as part of your own automation toolkit to write tests faster, maintain them with less effort, and automate work beyond testing.

7.1 AI-assisted test authoring

• AI coding assistants (GitHub Copilot, Cursor, Claude Code, and similar) for writing and maintaining Selenium/Playwright/API test code faster.
• Natural-language-to-test-script generation — turning a manual test case or user story directly into a first draft of automated test code.

7.2 Self-healing & AI-powered execution

• Self-healing locators — tools that automatically adapt when a UI selector breaks, instead of failing the whole run.
• AI visual testing — perceptual, AI-driven visual regression (e.g. Applitools Eyes) that understands "meaningfully different" rather than doing brittle pixel-by-pixel diffing.
• Flaky-test triage — ML/LLM-assisted analysis of failure logs to auto-classify a failure as a real bug versus environmental flakiness.

7.3 AI-generated test data & exploratory testing

• Synthetic test-data generation with LLMs, constrained to your schema and business rules rather than free-form output.
• LLM-driven exploratory testing agents that navigate an application and report anomalies a scripted suite wouldn't think to check.

7.4 Building your own automation agents

This is where the two tracks in this roadmap meet: use agent frameworks — LangGraph, CrewAI, n8n with LLM nodes, or the Claude Agent SDK — to automate QA workflows end to end. A realistic example pipeline: read a bug ticket → attempt to reproduce it → draft a regression test → open a pull request for review. MCP servers can connect such an agent to your test framework, CI system, and ticketing tool in a standardized way.

7.5 Guardrails when AI writes or executes automation

• Always review AI-generated test code before merging — treat it exactly like a PR from a junior teammate.
• Sandbox any agent that can execute actions; never give it direct production access.
• Treat AI-generated tests as a first draft that needs validation, not a finished, trustworthy artifact.

After this stage, you can: use AI coding assistants and agent frameworks to draft, maintain, and extend your own test automation faster, with the review discipline to catch what they get wrong.

Goal: Consolidate everything into a portfolio-ready project, and set up a system for staying current in a field that changes fast.

8.1 Capstone project ideas — pick one

• An end-to-end eval pipeline for a RAG chatbot (your Stage 2 LibreChat project, or similar): golden dataset + Ragas metrics + a CI quality gate + a tracing dashboard.
• A red-team report for a public demo LLM app, structured around the OWASP LLM Top 10.
• An autonomous QA agent that turns bug tickets into draft regression tests via a pull request.
• A self-healing Playwright suite enhanced with an AI-driven visual-diff step.

8.2 Building your portfolio

• Publish on GitHub with a clear README, sample reports/dashboards, and a short write-up of what you measured and why.
• Write one short post per stage explaining what you learned and built — it reinforces the learning and doubles as visible proof of the work.

8.3 Staying current

• Follow the OWASP GenAI Security Project and the engineering blogs of major model providers and agent-framework maintainers.
• Join MLOps/LLMOps-focused communities and any local AI-testing meetups.
• Revisit this roadmap's tool list and the OWASP Top 10 every 6 months — this landscape moves quickly and tool names will change even where the underlying principles don't.

8.4 Optional courses & certifications

• Prompt-engineering courses (e.g. on Coursera or via DeepLearning.AI) to deepen Stage 1 foundations.
• Dedicated AI/LLM-testing bootcamps and courses are emerging — vet any course against the topics in this roadmap rather than assuming it's comprehensive.
• Treat certifications as supplementary. In a field moving this fast, a strong GitHub portfolio of real projects carries more weight than a badge.

After this stage, you have: a public, portfolio-ready project that demonstrates the full evaluation-engineering discipline end to end.