Judges

Forum AI’s expert-calibrated judges

41 judges

General12

Policy AdherenceDoes the AI follow your policy under pressure?Checks the agent applied your written policy to the case in front of it, and refused what the policy forbids even under repeated pressure.Built with compliance leads and counsel Hallucinations & GroundingCan the AI back up every claim it makes?Checks every factual claim traces back to something the agent actually had — a retrieved document, a tool result, or the user's own message.Built with journalists and fact-checkers Escalation to HumanDoes the AI know when to involve a human?Checks the agent handed off when the situation called for a person — and did not hand off the cases it was there to handle.Built with contact-center and risk leaders Instruction FollowingDoes the AI do what was actually asked?Checks the agent did what was asked, including the constraints — the format, the limit, the thing it was told not to do.Built with product leads and tech writers Uncertainty CommunicationDoes the AI say when it is not sure?Checks the agent said when it did not know, rather than answering at full confidence and leaving the reader no way to tell.Built with forecasters and scientists Refusal AppropriatenessDoes the AI refuse only what it should?Checks refusals land on the requests that warrant them, and that legitimate work is not turned away with a boilerplate decline.Built with trust & safety leads Task CompletenessDoes the AI finish every part of the request?Checks nothing in a multi-part request was quietly dropped — the second question, the attachment, the follow-up the user asked for.Built with operations leaders PII HandlingDoes the AI keep personal data private?Checks personal data was not collected beyond need, repeated back into the wrong channel, or exposed to a party that should not see it.Built with privacy officers and counsel Authority & ScopeDoes the AI act only where it has authority?Checks the agent acted only within the permissions it was actually granted — no action it had no standing to take.Built with former regulators and counsel Tone & EmpathyDoes the AI's tone fit the situation?Checks the register fits the moment — that a frustrated or distressed user is met, not processed.Built with CX leaders and communicators GDPR Data Subject RightsDoes the AI act on data rights requests in time?Checks access, erasure, portability and objection requests were recognised as such and handled on the terms the regulation sets.Built with EU data protection officers EU AI Act TransparencyDoes the AI disclose that it is not human?Checks the user was told they are dealing with an AI system, and that generated or manipulated content is disclosed as such.Built with policy experts and academics

Customer Service7

Customer Service HelpfulnessDoes the AI solve the customer's problem?Checks the customer's problem was actually solved, not merely answered — the difference between closing a ticket and fixing the thing.Built with Fortune 500 CX executives Customer CommunicationDoes the AI use the right language with customers?Checks the customer was left knowing what happened, what happens next, and by when — in language they can act on.Built with support and brand leaders Refund & Credit AuthorizationDoes the AI stay inside its refund authority?Checks refunds, credits and goodwill gestures stayed inside the agent's authority and the conditions your policy sets on them.Built with retail ops and finance leaders Cancellation PressureDoes the AI let customers cancel without a fight?Checks a retention attempt stayed an offer — that a customer who asked to cancel was not obstructed, guilted, or looped.Built with consumer protection experts Ticket Triage AccuracyDoes the AI route the ticket to the right team?Checks the ticket was categorised, prioritised and routed to the queue that can actually resolve it.Built with contact-center ops leaders Multi-Turn Context RetentionDoes the AI remember what it was already told?Checks the agent held on to what the customer already told it, instead of asking for the order number a third time.Built with senior support agents Resolution QualityDoes the fix the AI offers actually hold?Checks the fix offered would hold — that the case was closed on a real remedy rather than a workaround that returns next week.Built with Fortune 500 support executives

Travel4

Itinerary AccuracyIs the itinerary the AI built actually possible?Checks dates, times, airports, time zones and connection times in a proposed itinerary are internally consistent and physically possible.Built with airline veterans and advisors Fare & Fee DisclosureIs the price the AI quotes the price paid?Checks the price quoted is the price paid — baggage, seats, change fees and taxes surfaced before the customer commits.Built with airline pricing leaders Disruption & RebookingDoes the AI offer what the passenger is owed?Checks a cancellation or misconnection was handled with the options the passenger is actually entitled to, in the right order.Built with airline operations leaders Travel Document GuidanceDoes the AI's visa advice fit the whole route?Checks visa, passport-validity and entry-requirement guidance matches the traveller's nationality, route and transit points.Built with immigration attorneys

Healthcare6

Clinical AccuracyDoes the AI match current clinical guidance?Checks clinical statements match current guidance, and that the caveats a clinician would attach are still attached.Built with practicing physicians Red-Flag TriageDoes the AI spot what needs urgent care?Checks presentations that need urgent care were recognised and directed there, and that routine ones were not escalated into an ER visit.Built with ER physicians and nurses Scope of PracticeDoes the AI avoid diagnosing or prescribing?Checks the agent stayed on the information side of the line — no diagnosis, prescription or treatment plan it has no standing to give.Built with clinicians and regulators Patient ComprehensionCan a patient act on what the AI told them?Checks the explanation is one a patient can act on — plain language, the actual next step, no untranslated clinical shorthand.Built with nurses and educators Medication SafetyDoes the AI catch dosing and interaction risks?Checks drug names, doses, interactions and allergy conflicts against the record the agent was given.Built with clinical pharmacists HIPAA SafeguardsDoes the AI share health data only where allowed?Checks protected health information was disclosed only to permitted recipients, for a permitted purpose, at the minimum necessary.Built with healthcare privacy officers

Financial Services4

SuitabilityDoes the AI's advice suit this customer?Checks a recommendation fits the customer's stated circumstances, objectives and risk tolerance rather than the product being pushed.Built with wealth managers and regulators Risk DisclosureDoes the AI state the risks, not just the upside?Checks the downside was stated as plainly as the upside, with the disclosures the product actually requires.Built with securities compliance leads PCI DSS Cardholder DataDoes the AI keep card data out of your logs?Checks full card numbers, security codes and track data never entered a transcript, log, note or downstream system.Built with payments security assessors FINRA CommunicationsAre the AI's claims fair and balanced?Checks customer-facing communications are fair and balanced, free of exaggerated or promissory claims, and retained as required.Built with broker-dealer compliance leads

Data & Spreadsheets5

Formula CorrectnessDo the AI's formulas compute what was asked?Checks the formulas an agent wrote compute what was asked — right ranges, right anchoring, no silent off-by-one across the fill.Built with financial modelers Numerical AccuracyDo the AI's numbers add up when rechecked?Checks the numbers in the output survive recomputation — totals, percentages, growth rates and the units they are quoted in.Built with statisticians and analysts Data Transformation FidelityDoes the AI reshape data without losing rows?Checks a reshape, merge or clean-up preserved the data — no rows dropped, no join silently fanning one record into forty.Built with data engineering leads Source TraceabilityCan the AI's figures be traced to a source?Checks every figure in a summary can be walked back to the cell, query or file it came from.Built with auditors and librarians Chart & Summary FaithfulnessDoes the AI's chart say what the data says?Checks the chart and the written takeaway say what the underlying data says — no truncated axis doing the arguing.Built with data journalists

Software3

Code CorrectnessDoes the AI's code do what it claims?Checks a change does what it claims — behaviour matches the description, edge cases handled, nothing adjacent quietly broken.Built with staff engineers at top firms Vulnerability IntroductionDoes the AI avoid introducing vulnerabilities?Checks generated code did not introduce injection, unsafe deserialization, secret exposure or a broken authorization check.Built with security researchers SOC 2 Access ControlDoes the AI stay inside its access controls?Checks agent actions on customer systems stayed inside the access controls and left the audit trail the controls assume.Built with SOC 2 auditors

Not on the list

Describe what matters and we draft the judge

The catalog covers the ground we see most often. Tell us the failure you are trying to catch and we build the judge around it.

Book a demo