Regression-test your Gradient agent in CI with the Evaluations API.

Image by Author

Why Gradient Agents Silently Regress When You Tweak a Prompt or Swap a Model

Agent quality can change with a small update. A new system prompt, foundation model, or knowledge base can affect responses without triggering traditional software tests. You can build an agent on the DigitalOcean Gradient AI Platform, validate it manually, and still have a later change quietly reduce its quality.

Traditional tests make failures obvious: a broken assertion turns the build red. Agents are different. They can return HTTP 200 and sound convincing while giving the wrong answer. Since agent responses are probabilistic, exact-match tests like assertEqual(response, expected) are also unreliable. Often, the first warning is a confident but incorrect answer reaching a user.

DigitalOcean treats Agent Evaluations like a test suite for checking agents after changes to their prompts, knowledge base, or model. The missing piece for many teams is running those evaluations automatically in CI/CD. DigitalOcean uses this approach with its own documentation agent, running evaluations before deployment.

What you will build: A reusable GitHub Actions workflow that keeps your agent configuration in Git. On every pull request, it syncs the configuration, evaluates the agent against a curated set of questions through the Evaluations API, and fails the build if the score falls below your threshold. This connects the change that affects agent quality directly to the test that catches it.

From Spot-Checking to a Repeatable Eval Gate

Spot-checking works when one person tests an agent, but it does not scale. You change the prompt, review a few responses, and ship. The regression may be hiding in the response you did not check. With more engineers, keeping manual reviews consistent gets even harder.

A fixed set of questions with known-good answers, scored automatically after every change, solves this. Set a quality threshold, and the build fails when performance drops below it. It works much like automated code testing: the golden dataset is your fixtures, the evaluation metrics are your assertions, and the threshold is your red build. The diagram below shows how everything connects across GitHub Actions and the Gradient platform.

Image by Author

Five components make up the gate:

Agent configuration: The prompt, model, and threshold live in Git, so changes go through pull requests.

Agent under test: Your Gradient Agent, optionally connected to a knowledge base.

Golden dataset: A CSV of query and expected_response pairs, uploaded as an Evaluation Dataset and linked to the agent through a test case.

Evaluations API: Starts a run, sends each question to the agent, and uses an LLM judge to score the responses.

Threshold gate: eval_driver.py reads the results, compares the star metric with its threshold, and fails the build if it falls below it.

Two ideas drive the rest of the tutorial. The star metric is the single metric used for the pass/fail decision. The completed run returns an overall status and a breakdown by metric, and the gate checks that one number.

The workflow also separates configuration from evaluation. It syncs your repo configuration to the platform, then uses the REST API to run and score the evaluation. There is no doctl evaluation command, and the API provides the per-metric JSON needed for the gate.

Provisioning the Gradient Agent and Curating a Golden Evaluation Dataset

This section gets you to a deployed agent whose configuration lives in your repository. From here on, a change to the prompt, model, or threshold is a change to a file, and a file change is a pull request. Budget about fifteen minutes.

Prerequisites

A DigitalOcean account with a payment method added. The platform requires one to activate the API, though free credits cover a tutorial's worth of usage.

doctl, the DigitalOcean CLI, installed and authenticated.

Python 3.10 or newer.

Create a scoped Personal Access Token

CI authenticates with a Personal Access Token (PAT), not the agent's endpoint access key. The endpoint key lets an application chat with one agent, while the PAT manages resources, syncs configuration, and starts evaluation runs. CI needs the PAT.

Create it under API → Tokens → Generate New Token. Choose Custom Scopes and grant only what this workflow needs:

genai: create, read, update

project: read

DigitalOcean shows the token once. Copy it into a local .env file and add that file to .gitignore in the same commit, so it never reaches your history.

# Copy this file to .env and fill in the values.
DIGITALOCEAN_API_TOKEN=dop_v1_your_token_here

Bootstrap the agent

The agent has to exist once before configuration can drive it, and the model's terms can only be accepted in the UI. Install doctl (scoop install doctl on Windows, brew install doctl on macOS), authenticate, and list the models:

doctl auth init
doctl gradient list-models

Pick a low-cost open-weight model so a full run costs cents. Llama 3.3 Instruct (70B) is a solid default at $0.65 per million tokens. Before creating an agent on it, accept the provider's terms in the Model Catalog: select Llama 3.3 Instruct (70B) and check the Meta Llama license box. The API cannot accept this for you, and creation returns a 403 until you do.

Create the agent once to get its ID and project ID:

doctl projects list --format ID,Name --no-header
doctl gradient agent create \
  --name "docs-eval-agent" \
  --model-id "<LLAMA_3_3_MODEL_ID>" \
  --project-id "<YOUR_PROJECT_ID>" \
  --region "atl1" \
  --instruction "Placeholder. The repository config is the source of truth from here on."
doctl gradient agent list

Record the agent UUID from the output. This is the last time you configure the agent by hand.

Image by Author

Define the configuration in the repository

Move the agent's real settings into a file the workflow reads. Create config/agent.yaml:

agent_uuid: "<YOUR_AGENT_UUID>"
test_case_uuid: "<YOUR_TEST_CASE_UUID>"
star_metric_uuid: "<YOUR_STAR_METRIC_UUID>"
model: "<LLAMA_3_3_MODEL_ID>"
instruction: >
  You are a helpful assistant that answers questions accurately and
  concisely based on the information provided. If you do not know the
  answer, say so rather than guessing.
star_threshold: 80

A sync step, built in the next section, reads this file and pushes any change to the platform before the evaluation runs. Editing instruction, swapping model, or raising star_threshold is now a reviewable diff, and the change that affects quality is the same change the gate evaluates.

Curate the golden dataset

The dataset is the real engineering decision here. Everything else is setup; this determines whether the gate measures anything meaningful. Three honest ways to build it:

Synthesize from your knowledge base (recommended for a docs agent). Generate question-and-answer pairs from the documents the agent serves, using a generator such as Ragas or a prompt loop against Gradient inference. Because the questions come from the agent's own source material, they test what the agent is supposed to know.

Start from a public set. Natural Questions (open subset) or SQuAD 2.0 both carry commercial licenses. Map each question to query and its answer to expected_response. Avoid MS MARCO, whose license is non-commercial.

Build a custom set around your own domain if neither fits.

Keep the set small. Around 50 curated rows give a meaningful signal cheaply, since the judge scores every row, and the platform caps a dataset at 500. Save it in the repository as evaluations/golden_dataset.csv, a UTF-8 CSV with a query column and an expected_response column, quoting any field that contains a comma. Upload it once through the control panel and bind it to a test case; the config then references that test case by UUID, so the dataset stays version-controlled alongside the code.

One gotcha: the agent-evaluation flow expects query and expected_response, while the separate model-evaluation flow expects input and ground_truth. If a sync complains about a missing column, match the header to the flow you are in.

Scoring the Agent Through the Evaluations API

You have an agent, a config file, and a golden dataset. This section is where the agent gets its first real score. The workflow syncs the config to the platform, then scores the agent through the API, all from one command.

Sync the configuration

A test case binds a dataset to an agent and selects the metric categories to score. You create it once when you bootstrap, then let the sync keep it current. Instead of clicking through the UI on every change, a sync step reads config/agent.yaml and upserts two things on the platform before scoring: the agent's instruction and model, and the test case's star-metric threshold.

DigitalOcean groups metrics into categories: correctness, context quality (the RAG bucket), safety and security, and user outcomes. For a factual agent with no knowledge base, correctness is what matters. Its central metric, Correctness (general hallucinations), measures whether the response is factually correct independent of any retrieved context, which makes it the right star metric. The sync sets its pass threshold from star_threshold in the config. Two correctness sub-metrics, retrieved context relevance and response-context completeness, need a knowledge base, so on a no-KB agent they report low or skip. That is expected.

The full sync script lives in the repo at sync_config.py. It runs first in the workflow, so every evaluation scores the configuration exactly as the pull request defines it.

Image by Author

The API shape

doctl has no evaluation subcommand, so scoring goes through the REST API under /v2/gen-ai/. Three calls do the work:

POST /v2/gen-ai/evaluation_runs starts a run. The body needs the test-case UUID and the agent UUID, and it returns a run UUID immediately, because the run is asynchronous.

GET /v2/gen-ai/evaluation_runs/{uuid} polls the run and reports its status.

The same GET carries the star_metric_result once the run finishes.

The status moves through several values before the score is ready: EVALUATION_RUN_QUEUED, then EVALUATION_RUN_RUNNING_DATASET while the agent answers each prompt, then EVALUATION_RUN_EVALUATING_RESULTS while the judge scores them, then EVALUATION_RUN_SUCCESSFUL. The score only populates at the end, so the driver keeps polling through the running and evaluating phases instead of stopping at the first non-queued status.

Image by Author

The Python driver

The full script lives in the repo at eval_driver.py. It reads the agent UUID, test-case UUID, and threshold from config/agent.yaml, and the API token from the environment, so the same file runs locally against a .env file and in CI against a repository secret. The shape is four steps:

# eval_driver.py (abridged; full version in the repo)

# 1. Start the run.
resp = requests.post(f"{API_BASE}/evaluation_runs", headers=headers, json={
    "test_case_uuid": TEST_CASE_UUID,
    "agent_uuids": [AGENT_UUID],
})
run_uuid = resp.json()["evaluation_run_uuids"][0]

# 2. Poll until the run leaves its queued/running/evaluating states.
while True:
    run = requests.get(f"{API_BASE}/evaluation_runs/{run_uuid}", headers=headers).json()["evaluation_run"]
    status = run["status"]
    if "FAILED" in status:
        sys.exit("Evaluation run failed on the platform.")
    if not any(s in status for s in ("QUEUED", "RUNNING", "EVALUATING")):
        break
    time.sleep(POLL_INTERVAL)

# 3. Read the star-metric score.
score = float(run["star_metric_result"]["value"])

# 4. Gate: exit non-zero if below threshold.
sys.exit(0 if score >= STAR_THRESHOLD else 1)

Run it locally to confirm the whole loop works:

python eval_driver.py

The script prints the status at each poll, then the score and a PASS or FAIL line, and exits 0 or 1 to match. That exit code is the signal CI keys on.

Image by Author

Scoring is done by an LLM-as-a-judge. DigitalOcean sends the inputs, outputs, and retrieved context to the judge under a zero-data-retention agreement, so your data is not stored outside DigitalOcean or used to train the judge. Judge tokens are waived during the current preview and billed at general availability, so revisit your cost estimate once the preview ends.

Wiring the CI Gate That Fails the Build on Regression

This section is the deliverable. The sync script and driver run and exit with the right code; the last step turns that exit code into a merge gate: a GitHub Actions workflow that, on every pull request, syncs the repo configuration to the platform, scores the agent, and blocks the merge when the score falls short.

Store the token

Because the configuration lives in config/agent.yaml, the workflow needs only one thing from GitHub: the API token. Under Settings → Secrets and variables → Actions, add it as a secret:

DIGITALOCEAN_API_TOKEN goes under Secrets, encrypted and hidden from logs.

Everything else the gate needs, the agent UUID, test-case UUID, model, instruction, and threshold, lives in config/agent.yaml in the repository. This is the point of the config-in-repo approach: one file is the source of truth, and there are no scattered variables to keep in sync.

The workflow

The workflow lives at .github/workflows/eval-gate.yml. It triggers on pull_request, checks out the code, installs dependencies, syncs the configuration, and then runs the driver:

name: Agent Eval Gate

on:
  pull_request:
  workflow_dispatch:

jobs:
  eval-gate:
    runs-on: ubuntu-latest
    timeout-minutes: 55
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"
      - run: pip install -r requirements.txt
      - name: Sync configuration
        run: python sync_config.py
        env:
          DIGITALOCEAN_API_TOKEN: ${{ secrets.DIGITALOCEAN_API_TOKEN }}
      - name: Run the evaluation gate
        run: python eval_driver.py
        env:
          DIGITALOCEAN_API_TOKEN: ${{ secrets.DIGITALOCEAN_API_TOKEN }}

The clearest way to see the gate work is to change the prompt and open a pull request. Because the prompt lives in config/agent.yaml, the edit is a one-line diff under review.

Image by Author

The workflow syncs that change to the platform, scores the agent against it, and reports the result on the pull request. A prompt that holds quality above the threshold passes, and the job goes green.

Image by Author

A prompt that lowers quality produces a score below the threshold, the driver exits 1, and the job fails.

Image by Author

Either way, the gate evaluates the exact edit under review, which is the property the config-in-repo approach buys you.

To make that red status block the merge, add a required status check. Under Settings → Branches, add a ruleset targeting main, enable Require status checks to pass before merging, and select the eval-gate check. A pull request whose evaluation falls below the threshold then cannot merge, the same way a failing unit test blocks one.

Required status checks are enforced on public repositories and on paid or organization plans. On a private repository on the free plan, GitHub runs the check but does not enforce the block, so make the repository public or move it to an organization to enforce the gate.

Choosing a threshold instead of guessing

Do not choose the threshold at random. First, run the current agent against your golden dataset to establish a baseline. Then set star_threshold in the config based on the error rate your team can accept. As a reference, DigitalOcean runs its own documentation agent with a production target of 95% correctness and 80% ground-truth faithfulness. Start close to your measured baseline and raise the threshold as the agent improves, since an overly ambitious target only makes every build fail.

Gotchas, What's Next, and Conclusion

A few things are worth knowing before using this in production. Runs are asynchronous, so set a generous CI timeout and log the raw status during your first runs. Shared inference can sometimes take much longer than usual. Treat FAILED as a hard error, not a score of zero. Judge tokens are free during preview but will be billed at general availability, so update your cost estimates when pricing changes.

Keeping the configuration in Git closes an important gap. Changes to the prompt, model, or threshold become reviewable pull requests that the gate checks before merge. Direct changes in the control panel can still bypass Git, so a daily scheduled run can catch them.

You can also extend the setup with a knowledge base for RAG metrics, guardrails for safety and PII, or additional test cases for things like tone and safety.

You started with a working agent but no reliable way to catch regressions. Now its configuration lives in Git, and GitHub Actions tests every pull request. If quality drops below your threshold, the build fails before the change reaches users.