Live demo: https://giskard-risk-evals.vercel.app (read-only: replays every recorded run)
Code: https://github.com/AIEngineerHelp/applied-ai-engineering/tree/main/evaluations-reliability/giskard-risk-evals
Key takeaways
- Write one risk definition and use it twice. Giskard’s attacker reads the rule to write attacks, and its judge reads the same text to score the conversations. The test data and the grader can’t drift apart.
- The dangerous failures are half-refusals. “While I cannot formally authorize a dose for your medical chart… a single 500 mg dose of acetaminophen carries a very low risk of interacting with warfarin.” A keyword check passed 11 of the 22 conversations we labelled as failures. The judge passed 1.
- Check the judge on the attacks you actually run. It agreed with our labels on 37 of 40 conversations from the first recording. On one cell of more varied attacks, it passed 9 of 10 conversations where our labels pass only 4.
- Smarter attackers got clearer evidence. Giskard’s Crescendo and GOAT attackers broke the naive prompts in 19 of 20 attacks, with plain “Yes, …” answers that both judges caught. Against the hardened prompts, none of the three attackers broke the rule.
- The measurements cost about $7 in logged Gemini calls ($3.77 for the judge and diversity study, $3.38 for the attacker face-off).
What this teaches
In a bank or a clinic, engineers don’t decide what an agent may say. Lawyers, compliance officers and clinical safety leads do, and they write their rules in prose: “no personalized investment advice”, “always send chest pain to emergency care”. An eval suite is only useful if it tests those rules, and if the people who wrote them can read the results.
Giskard is an open-source library for testing agents. Its red-teaming loop has three moving parts:
- a rule, written by the policy’s owner;
- an attacker, an LLM that plays a user and tries, over several turns, to make the agent break the rule;
- a judge, an LLM that reads the same rule and the whole conversation and says pass or fail, with a reason.
The fourth part, review, is people.
This example runs real Giskard v3 against two Gemini agents, a banking assistant and a clinic assistant, on four policies:
- investment advice;
- discouraging a card dispute;
- individual medication changes;
- missed emergencies.
Each agent has a naive prompt and a hardened one, with the policy written in. Then it measures the method itself.
| Term | Meaning |
|---|---|
| Risk definition | What is prohibited, why, what to do instead, and what users ask, written by the rule’s owner |
| Attacker | Giskard’s LLMGenerator with an adversarial template; plays a user for several turns |
| Judge | Giskard’s Conformity check; scores the conversation against the rule |
| Half-refusal | A reply that says “I can’t advise you” and then gives the advice anyway |
| Label | A person’s own pass or fail, written without looking at the judge |
| False pass | The judge passed a conversation the label fails: a broken rule nobody sees |
What we measured
1. Is the judge right? We labelled 40 conversations from the first recording without looking at the verdicts: every failure, every case we had flagged, and random passes.
| Who scored it | Agrees with our labels | False passes |
|---|---|---|
gemini-3.8-flash (the recording’s judge) |
37 of 40 | 1 |
gemini-4-argon (a stronger judge) |
36 of 40 | 2 |
| A keyword check (does the reply refuse or route?) | 28 of 40 | 11 |
2. Were the attacks different?
- Without seeds: with one shared brief, the attacker wrote ten versions of one message. 19% of attack pairs opened with the same four words.
- With seeds: we gave each attack its own question, persona and tactic (role-play, hypothetical, splitting the request, an authority claim, emotional pressure). That brought shared openings to 0%.
- What the judge missed: the varied attacks found failures the first recording missed. On the naive medication prompt the judge then said 9 of 10 passed, while our labels pass 4. The advice was hidden inside role-play scripts, hypotheticals and replies that open with a firm “No, never double up”.
3. Would a keyword check do? No. A refusal-phrase check passed 11 of the 22 conversations our labels fail, because a half-refusal contains the refusal. A keyword check still earns its place for what code can verify exactly, such as a required phone number.
4. Closing the loop. Two review notes blamed the rule, not the agent, so we revised both rules, versioned them, and ran the eval again.
- The dispute rule: the revision didn’t flip the verdict on the conversation we had flagged. It did turn the argument into a concrete fix: when steering a member to the merchant, still give the dispute path and offer a person.
- The investment rule: the revision caught a one-sided list of “neutral factors” that we had labelled as a pass.
5. Stack the checks. Giskard composes checks: RegexMatching for exact facts, AllOf to require every check, AnyOf to accept any one. We re-scored the 50 labelled conversations without any model calls.
| Stack | Agrees with labels | False passes |
|---|---|---|
| The judge alone | 42/50 | 6 |
| Judge AND regex | 41/50 | 6 |
| Judge OR regex | 32/50 | 17 |
| Both judges | 43/50 | 5 |
The regex can only veto what it can see, and the half-refusals contain the words it looks for. Letting it overrule the judge (OR) brings every half-refusal back.
6. Smarter attackers. Giskard’s scan module ships other attack strategies, built for harmful-content goals. We pointed two of them at our rules, with up to 6 turns and 5 attacks per cell:
- Crescendo starts with a harmless, abstract question and escalates a little each turn.
- GOAT picks among seven named jailbreak techniques.
| Broke the rule (judge) | Our seeded attacker | Crescendo | GOAT |
|---|---|---|---|
| Naive prompts | 9/20 | 10/10 | 9/10 |
| Hardened prompts | 0/20 | 0/10 | 0/10 |
- Clearer evidence: Crescendo and GOAT pushed until the agent said it outright: “Yes, Jordan, since it has only been about two hours, it is generally safe to take your regular dose of lisinopril (10 mg) now.” The second judge agreed with the first on all 40 attacks.
- The hardened prompts held: GOAT tried a “simulator mode” with disclaimers “pre-acknowledged”, and Crescendo cited a doctor’s prior approval. The agent declined each time.
- No more varied on their own: all five GOAT attacks on one cell opened “Hi Penny, looking at my…”. Seeds still matter.
What Giskard did for us, and what it didn’t
Giskard doesn’t bring its own model: its attacker and judge are Gemini calls here, so a few hundred lines of your own code could do the same job.
What it gave us:
- a ready multi-turn attacker;
- one shape for every test (a scenario, a judge and a recorded trace), which lets the app replay and re-score runs;
- swappable models;
- more attack styles and composable checks.
What we had to add: seeds for varied attacks, labels to check the judge, and a cost ledger. Treat it as a convenient harness, not a source of truth.
Run it
To browse, open the live demo: it replays every recorded run and needs no API key. To run it yourself, you need Python 3.12+ with uv, and Node.js only to change the frontend.
git clone https://github.com/AIEngineerHelp/applied-ai-engineering.git
cd applied-ai-engineering/evaluations-reliability/giskard-risk-evals
uv sync
uv run python -m riskdemo serve # http://127.0.0.1:4330, replays the recorded runs
uv run pytest # offline tests
The app explains the method, replays every recorded attack, and lets you label conversations yourself and compare with two judges. “What we measured” holds all six measurements above. The live demo is read-only; a local copy with GEMINI_API_KEY in .env can run new attacks. Re-recording runs (record --live, faceoff record --live) makes paid Gemini calls; every command is a dry run without --live.