Loading…

Preparing the test suite and restoring test data.

Starting suite

watsonx Orchestrate Test Suite

Agent quality, stress testing, observability, and CI/CD automation

READY
Service Instance: No service instance selected - Agent: No agent selected
Test Data
Workspace
Run
Selected cellSelect a cell to view or edit its full value.
Enter text for a case-insensitive literal search, or use slash-delimited JavaScript regular expression syntax. Replace changes editable fields in matching rows.
Ready 1 sheet
Memory optimized
100%

Keyboard Shortcuts

Arrow keysMove one cellEnterEdit / save and move downEscapeCancel editTab / Shift+TabMove horizontally?Open help⌘/Ctrl+ZUndo⌘/Ctrl+Shift+ZRedo

Expected Responses

Configure how this test case should be evaluated.

Configuration

Cell preview

This is how expectedResponses will appear in the spreadsheet.

CSV-safe
CSV Preview
Choose a dialog type and configure its expected outcome.
Captured output

Final Responses

Review and reuse every response from this test case.

User utterance

Run Report

AgentOps & Evaluation Settings

Evaluation mode

Use deterministic checks for intent-routing tests, and semantic/regex checks when LLM responses may vary.

IBM ADK EVAL export

Export the loaded test cases as a draft ground-truth dataset scaffold for watsonx Orchestrate ADK EVAL.

Langfuse / AgentOps

Store credentials locally to export or send traces, observations, scores, and dataset-run style records.

Eval Template Wizard

Choose the type of agent behavior you want to test. The wizard will populate the spreadsheet columns needed for deterministic, flexible, semantic, or multi-turn evaluation.

Ready to generate a spreadsheet template.

Tool Sandbox

Run the selected agent’s instructions through a real wxO-hosted chat model while executing only deterministic local tool stubs. The production agent runtime and its configured tools are never called.

OFF
Disabled. Live runs use the production agent runtime.
Boundary: this mode uses the real selected model and captured agent instructions, but it intentionally bypasses wxO’s native agent runtime, memory, collaborators, and hidden orchestration layer. Stress workloads (threadCount or requestsPerThread above 1, or performance_scalability) automatically bypass Tool Sandbox and call the production agent Direct API.

Multi-turn format

For multi-turn rows, put a JSON array in the turns column. Human delay is recorded separately and excluded from agent response time.

[{"user":"I need help with a bill","expected":"claims_charge_details_inquiry"},{"user":"It is for June 15","expected":"claims_charge_details_inquiry","humanDelayMs":12000}]

Testing Non-Deterministic & Multi-Turn Agents

Shift from exact answers to evaluation scenarios

For deterministic intent routing, exact matching still works. For non-deterministic LLM responses, each spreadsheet row should describe the input, the expected behavior, and the evaluator that decides whether the result is acceptable.

Recommended evaluator modes

intentUse when the agent should return one of several valid intents separated by ||.
containsUse when the answer can vary but must include one or more important concepts.
regexUse for structured response patterns or controlled vocabulary.
semanticUses the selected agent’s built-in LLM to judge relevance, instruction compliance, grounding, clarification behavior, and optional KB/tool evidence. Reference answers are optional.

Multi-turn timing

The suite records each agent response separately. Human delay between turns is recorded as metadata and excluded from agent response timing, so a user taking 30 seconds to reply does not distort the agent latency measurement.

Tool Sandbox boundary

Tool Sandbox sends the selected agent’s visible instructions and local function definitions to a real chat model exposed by the wxO instance. Tool requests are executed only against the JSON stubs configured in the suite. It never invokes the production agent endpoint, so production tools cannot run.

Use it to test tool selection, arguments, failure handling, sequencing, and how the model uses deterministic tool results. It does not reproduce the native agent runtime’s hidden prompt scaffolding, memory, collaborators, or orchestration policies; keep live certification runs for that final layer.

Stress tests never use Tool Sandbox. The suite sends those requests concurrently to the selected agent’s production Direct API so latency, throughput, tool execution, and service limits represent the real agent.

Example patterns

Flexible response:
evalMode = contains
expected = claim||balance||deductible

Multi-turn:
turns = [
  {"user":"I have a bill from my doctor","expected":"asks for more detail or identifies billing intent"},
  {"user":"It says I owe $200 after insurance paid","expected":"claims_charge_details_inquiry","humanDelayMs":12000}
]

Agent Scorecard

Promotion readiness based on saved AgentOps evidence.

Select an agent to view its scorecard.

Tool source

Loading tool source…
Source codeText