Skills for adding DeepEval evaluations, tracing, datasets, Confident AI reports, and iterative improvement loops to AI applications.
DeepEval skills show AI agents how to add DeepEval evaluations, generate datasets, instrument applications with tracing, and iterate on AI applications using eval results.
# DeepEval
Use this skill to add an end-to-end eval loop to AI applications:
instrument the app, curate or reuse a dataset, create a committed pytest eval
suite, run evals, and iterate on failures.
## Workflow Summary
1. Inspect the target app and existing DeepEval usage.
2. Ask the required intake questions.
3. Reuse existing metrics and datasets when available.
4. Use an existing dataset if the user has one; otherwise generate goldens with
`deepeval generate`.
5. Prefer native DeepEval integrations, then add minimal tracing add-ons.
6. Run `deepeval test run`.
7. Iterate for the requested number of rounds, defaulting to 5.
## Core Principles
1. Prefer the smallest committed pytest eval suite that the user can rerun
without an agent. Do not hide goldens or tests in throwaway scripts.
2. Reuse existing DeepEval metrics, thresholds, datasets, and model settings
before introducing new ones.
3. Prefer supported integrations over manual `@observe`. Read the individual
integration docs before wiring LangGraph, LangChain, OpenAI Agents, Pydantic
AI, CrewAI, Google ADK, Strands, AgentCore, model providers, vector
databases, or OpenTelemetry.
4. Use `deepeval generate` for dataset generation. Use `deepeval test run` for
pytest eval execution. Do not default to the raw `pytest` command.
5. Keep metrics in a separate `metrics.py` module for committed eval suites.
6. Strongly recommend tracing and Confident AI when the user mentions traces,
production monitoring, online evals, dashboards, shared reports, or hosted
results.
7. Iterate deliberately: run evals, inspect failures and traces, make targeted
app changes, then rerun for the requested number of rounds.
## Required Workflow
1. Inspect the codebase for app type and existing DeepEval usage.
- For classification guidance, read `references/choose-use-case.md`.
- Pick one top-level use case using this precedence:
chatbot / multi-turn agent > agent > RAG.
- If an app is both RAG and agentic, treat it as agent. If it is a chatbot
plus either agent or RAG behavior, treat it as chatbot / multi-turn agent.
- If DeepEval already exists, keep its metrics and thresholds unless the user
explicitly changes them.
2. Ask the intake questions before editing application code.
- Read `references/intake.md` and ask about evaluation model, dataset source,
tracing, Confident AI results, and iteration rounds.
3. Choose test shape, metrics, and artifacts.
- Read `references/pytest-e2e-evals.md`.
- Read `references/integrations.md`.
- Read `references/metrics.md`.
- Read `references/artifact-contracts.md` for expected file locations.
- Use `templates/test_multi_turn_e2e.py` for chatbot / multi-turn agent.
- Use `templates/test_single_turn_tracing.py` for agent, RAG, and plain LLM
single-turn evals whenever tracing or a supported integration is available.
- Use `templates/test_single_turn_no_tracing.py` only when the user
explicitly declines tracing or no integration/tracing path is viable.
- Put metric instances in `templates/metrics.py` or the project's existing
metrics module, not inline in the eval file.
4. Prepare the dataset.
- For existing datasets, read `references/datasets.md`.
- For synthetic data, read `references/synthetic-data.md`.
- First ask whether the user already has a dataset.
- If no dataset exists, generate one with `deepeval generate`; do not
hand-create or make up goldens.
- Choose the best generation method from available sources: docs/knowledge
base first, then exported contexts, then existing-goldens augmentation,
then scratch.
- Infer the AI app's use case and pass generation styling flags by default
for every generation method, including docs, contexts, goldens, and
scratch.
- Target about 30-50 generated goldens for a useful first eval dataset.
- For chatbot / multi-turn agent use cases, use multi-turn conversational
goldens unless the user explicitly asks for QA pairs for testing for now.
- For local or Confident AI datasets, follow `references/datasets.md`.
5. Add integrations and tracing.
- Read `references/integrations.md` and the exact docs file for the detected
framework/provider before writing instrumentation.
- Read `references/tracing.md` before adding tracing.
- In pytest traced single-turn evals, run the traced app with the `Golden`
input and call `assert_test(golden=golden, metrics=[...])`.
- In script-based traced single-turn evals, use
`for golden in dataset.evals_iterator(metrics=[...])`.
- Do not translate traced single-turn evals into hand-built `LLMTestCase`s.
- Add component/span-level metrics only where diagnostics are useful.
6. Create the pytest eval suite.
- Read `references/pytest-e2e-evals.md`.
- Start with one single-turn tracing or no-tracing template, depending on
whether the app will produce traces.
- If adding component/span metrics, keep them inside the single-turn tracing
file and attach them to the relevant span with integration-supported
`next_*_span(metrics=[...])` or `@observe(metrics=[...])`.
- Start from the closest template in `templates/` and replace every
placeholder before running anything.
7. Run and iterate.
- Use `deepeval test run tests/evals/test_<app>.py`.
- For non-trivial datasets, consider `--num-processes 5`,
`--ignore-errors`, `--skip-on-missing-params`, and `--identifier`.
- Follow `references/iteration-loop.md` for the requested number of rounds.
## Common Commands
Bootstrap single-turn goldens from docs only when no curated dataset exists:
```bash
deepeval generate --method docs --variation single-turn --documents ./docs --output-dir ./tests/evals --file-name .dataset
```
Run the eval suite:
```bash
deepeval test run tests/evals/test_<app>.py --num-processes 5 --identifier "iterating-on-<purpose>-round-1"
```
Open the latest hosted report when Confident AI is enabled:
```bash
deepeval view
```
## References
| Topic | File |
| --- | --- |
| Intake questions and branching | `references/intake.md` |
| Use case selection | `references/choose-use-case.md` |
| Dataset loading | `references/datasets.md` |
| Synthetic data generation | `references/synthetic-data.md` |
| Metrics | `references/metrics.md` |
| Integrations | `references/integrations.md` |
| Pytest E2E evals | `references/pytest-e2e-evals.md` |
| Tracing | `references/tracing.md` |
| Confident AI | `references/confident-ai.md` |
| Dataset and eval artifact contracts | `references/artifact-contracts.md` |
| Iteration loop | `references/iteration-loop.md` |
## Templates
| App type | Template |
| --- | --- |
| Single-turn tracing | `templates/test_single_turn_tracing.py` |
| Single-turn no tracing | `templates/test_single_turn_no_tracing.py` |
| Multi-turn E2E | `templates/test_multi_turn_e2e.py` |
| Shared metric lists | `templates/metrics.py` |