Test with clarity.
Release with confidence.

This tutorial explains how to use the watsonx Orchestrate Test Suite in this workspace. It covers the browser application used to design, run, diagnose, record, and compare agent tests, and it also covers the repository's automated developer checks.

The instructions were verified against the workspace on August 19, 2026. At that time the root package was wxots-standalone 469.0.0, the UI identified itself as Suite Version 0.7 / Build 24, and the workspace required Node.js 22.20 or newer.

#1. What this workspace contains

The repository has two meanings of “test suite”:

1. The wxO Test Suite application is the primary product. It is a local Node/Express server plus a browser spreadsheet used to test watsonx Orchestrate agents.
2. The developer regression suite validates the application source itself. Its main command is npm run check.

The important directories are:

wxots-standalone is still the live package name, server identity, launcher/build identity, and published artifact prefix. Do not remove those references. What the current suite does not use are extracted historical source copies such as wxots-standalone-v*/ or BUILD-OLDER/; the application, developer checks, and release scripts all use the root source tree. BUILD/ is ignored, reproducible output from the current build script, not a second source tree.

The earlier independent wxots-landingpage/ application is also no longer part of this workspace. Current update metadata, release history, and downloadable standalone ZIPs are owned by wxots-version/. Those published ZIPs are live version-catalog data and must not be confused with the removed extracted source copies.

The typical execution flow is:

Profile + runtime
       ↓
Spreadsheet row + expected-response contract
       ↓
Agent invocation or captured-response evaluation
       ↓
Response evaluator + optional tool/trace checks
       ↓
Status, score, timing, debugger evidence, report, recording, exports

#2. The fastest successful workflow

Use this short path the first time. The rest of the tutorial explains every step in depth.

1. Open a terminal in the repository root.
2. Run npm install.
3. Run npm start.
4. Open the URL printed by the process, normally http://127.0.0.1:3038.
5. In Profiles, choose New.
6. Paste a watsonx Orchestrate service-instance URL and IBM Cloud API key.
7. Refresh the agents, select one, and choose Save Profile.
8. Under Test Data, choose Open and select ui/intents/sample_multiturn_clarification_tests.csv, or create one row manually.
9. Set Thread Count and Requests Per Thread to 1 while learning.
10. Double-click Expected Results and configure the response contract.
11. Under Run, leave Run Mode set to Run and evaluate immediately.
12. Select a row and choose Probe.
13. Review Status, Final Response(s), and the Agent Debugger.
14. Choose Report after a batch run, and save a .wxotsw workspace before making major changes.

#3. Install and start the suite

#3.1 Prerequisites

You need:

• Node.js 22.20 or newer.
• npm.
• A modern browser.
• For live watsonx Orchestrate testing: a service-instance URL, an IBM Cloud API key, and permission to invoke the selected agent.

Check the tools:

node --version
npm --version

#3.2 Install dependencies

From the repository root:

npm install

The application has no browser build step for ordinary development. The files in ui/ are served directly.

#3.3 Start on the default port

npm start

The terminal prints the actual address, local state directory, optional wxots-db connection, and a machine-readable WXOTS_BOUND_PORT line. The preferred port is 3038; if it is busy, the application searches later ports within its configured range.

Open the exact URL printed by the process. Do not assume 3038 if the server reports a different port.

#3.4 Start on a chosen port

WXOTS_PORT=3040 npm start

To bind to another interface, set WXOTS_HOST. Keep the default 127.0.0.1 unless remote access is deliberately required.

#3.5 macOS launcher

On macOS, Launch wxots.command starts the local harness, opens a browser, and leaves the server log in a Terminal window.

#3.6 Check server health

With the default port:

curl -fsS http://127.0.0.1:3038/api/health

For a compact view when jq is installed:

curl -fsS http://127.0.0.1:3038/api/health \
  | jq '{status, service, version, transport, trafficGovernor}'

The response should report status: "ok". It also exposes the active outbound traffic-governor limits.

#3.7 Stop the suite

Press Ctrl-C in the terminal that is running npm start. A normal stop preserves the disk-backed workspace state.

#3.8 Install an automatic update without losing local work

When the configured version service reports a newer release, the Test Suite displays an update prompt. Stop any active run and choose Update now.

Automatic replacement is available only from a packaged standalone BUILD started with its launcher or npm start. A Git/source checkout is protected: it may report that a newer version exists, but it offers manual download instead of overwriting tracked source files.

The standalone updater:

1. flushes the current disk-backed state;
2. saves a recovery copy of the live worksheet, results, debugger details, and profile configuration under the local state directory;
3. downloads and validates the stateless release in a temporary staging directory;
4. transactionally replaces the application files while preserving the existing state directory, legacy local-state locations, recordings, .env, and local GGUF model directory, rolling back the old application if replacement fails;
5. restarts the backend as a fresh supervised process on the same port and hard-reloads the browser; and
6. restores the saved live worksheet into the updated interface.

The default .wxots-harness-state/ directory is beside the installed BUILD/, so it is outside the application files being replaced. A custom WXO_STATE_DIR or WXOTS_NATIVE_MODEL_DIR is also left untouched when it is outside the installation and is explicitly preserved when it is located inside the installation tree. Downloaded releases do not contain another user's state or models and therefore cannot replace the state already on the local machine.

#4. Understand the interface

The main screen is a workbook-style test environment.

#4.1 Profiles panel

The left panel holds saved service-instance and agent bindings. Its actions are:

• New: create a profile.
• Save: persist profiles and, when a writable file handle exists, update that file.
• Save As: export profiles as JSON.
• Open: import a profile JSON file.
• Delete: remove the active profile.

Exported API keys are encrypted for re-import by the suite, but profile exports should still be handled as sensitive artifacts.

#4.2 Ribbon groups

The upper ribbon contains these major groups:

The ribbon groups are draggable and resizable. The entire ribbon area and its icon height are also resizable.

#4.3 Spreadsheet

The center grid is the executable test-case sheet. Workbook tabs appear below it. You can open several CSVs as separate tabs, reorder tabs, close tabs, and merge tabs into a new sheet.

The bottom bar provides:

• current workbook status;
• sheet count;
• direct row jump;
• memory-optimization state;
• spreadsheet zoom from 50% to 200%.

#4.4 Search, replace, and status filter

The search field accepts either case-insensitive text or a JavaScript-style regular expression such as /^claim-\d+$/i. The adjacent replacement field applies changes to editable fields in matching rows.

The status filter can show all rows or only Pending, Running, Captured, Passed, Failed, Error, Stopped, or Skipped rows. Snapshot exports honor the current filters.

#4.5 Agent Debugger

The right panel has three primary tabs:

• Run Review: row outcome, conversation, evaluator decision, execution path, warnings, and timing.
• Evaluation: evaluator verdicts, tool evidence, reasoning/trace evidence, and JSON or Markdown export.
• REST API: health, active profile, endpoints, and the latest manual request/response.

The debugger can be docked on the right or along the bottom and resized.

#4.6 Team Activity

When wxots-db is configured and reachable, the people icon opens:

• online activity;
• shared artifacts;
• team chat.

The test suite still works locally when wxots-db is absent.

#4.7 Keyboard shortcuts

#5. Connect a watsonx Orchestrate profile

Use a profile when running the production watsonx Orchestrate agent or when an auxiliary workflow needs that instance's model or metadata.

#5.1 Create the profile

1. In the left Profiles panel, choose New.
2. Optionally enter a profile name. If left blank, the suite derives one from the instance URL.
3. Paste the IBM Cloud API key into API Key.
4. Paste the full service-instance URL into watsonx Orchestrate Service Instance URL. A typical value looks like:

``text https://api.us-south.watson-orchestrate.cloud.ibm.com/instances/<instance-id> ``

1. Choose the refresh button beside the agent list.
2. Wait until loading finishes. The list is intentionally disabled while refresh is in progress.
3. Select the intended agent and review its preview.
4. Choose Save Profile.

#5.2 Verify the active context

The badge above the sheet should show both the service instance and selected agent. If you switch agents after running tests, the suite preserves result contexts per profile/agent instead of mixing results.

#5.3 Refresh or repair a profile

Reopen the profile if:

• agent discovery fails;
• the selected agent was deleted or replaced;
• the API key changed;
• a restored/shared workspace says credentials are required;
• the debugger reports a missing selected agent ID.

Discovery uses the direct REST API. Closing the profile dialog cancels an in-progress refresh, and starting a new refresh supersedes the previous one.

#5.4 Save and move profiles

Use Save As to export profiles. The suite removes plaintext apiKey values and stores encrypted equivalents in the JSON payload. Use Open to restore them. Do not hand-edit the encrypted fields.

Cloud/shared workspaces deliberately omit reusable API keys. Each recipient must enter their own credentials before executing tests.

#6. Choose a runtime

Configuration → Runtime -- Unavailable --

Runtime switching is currently unavailable. The suite always uses watsonx Orchestrate through the active profile and selected hosted agent. Standard builds do not download or package llama.cpp.

#6.1 watsonx Orchestrate

This is the default and the correct choice for production certification.

It uses:

• the active service-instance profile;
• the selected hosted agent;
• the hosted runtime;
• production tools, collaborators, and knowledge services.

Developers can explicitly package the dormant native engine for future-feature work with WXOTS_BUNDLE_NATIVE_RUNTIME=true npm run build. This opt-in is not required or used by the currently available product workflow.

#6.2 Stress tests

A row is a stress workload when threadCount > 1, requestsPerThread > 1, or testType is performance_scalability. Stress workloads bypass Tool Sandbox and use the production agent Direct API so latency, throughput, tool execution, and service limits are real. Confirm the active profile and traffic estimate before continuing.

#7. Create or import test data

There are four main ways to build a sheet.

#7.1 Start with one manual row

The default seed row is enabled and contains hello. Replace it with a meaningful request, then configure its expected result.

For a minimal functional row, fill:

• Execute?: checked;
• Thread Count: 1;
• Requests Per Thread: 1;
• User Utterance: the user message;
• Expected Results: a configured response contract;
• Max Turns: normally 1 for single-turn tests.

#7.2 Import a CSV

1. Choose Test Data → Open.
2. Select a .csv file.
3. The suite parses the rows and auto-maps recognized aliases.
4. Choose Mapper to review source-to-harness mappings.
5. Drag a source field onto a target or use the dropdowns.
6. Edit display names if needed.
7. Choose Apply mapping.

Recognized aliases include utterance, question, and userUtterance for input; expectedResponses, expectedChoices, and allowedIntents for expected results; and evaluationMode or evaluator for the evaluator.

Unmapped source columns are hidden rather than destroyed. The suite can preserve supplemental CSV fields for round-tripping.

Useful samples:

• ui/intents/sample_multiturn_clarification_tests.csv
• ui/intents/IntentAgent_InputTestData.csv
• ui/intents/IA_MasterRegressionSuite.csv

#7.3 Generate test data

1. Choose Test Data → Generate.
2. Describe the domain, such as “billing questions, claim status, password reset.”
3. Set the row count, from 1 to 500.
4. Keep thread/request counts at 1 unless load is intentional.
5. Optionally set subsegment and channel.
6. Choose expected-result style: intent labels, contains keywords, semantic notes, or blank.
7. Choose append, replace, or new worksheet.
8. Choose Generate.

Generated data is a deterministic local scaffold. Review and correct the utterances, expected contracts, and metadata before treating it as ground truth.

#7.4 Add a workbook tab

Choose the + beside the workbook tabs to open another CSV as a new sheet. Tabs can be dragged into a different order.

To merge sheets:

1. Open at least two tabs.
2. Choose the merge icon.
3. Select the source sheets.
4. Name the new sheet.
5. Select a duplicate policy, including deduplication by input with a priority sheet.
6. Optionally tag each row with its source sheet.
7. Create the merged sheet.

#7.5 Clean imported text

Choose Test Data → Clean to review unreadable symbols or invalid text markers. The suite presents proposed fixes and applies only those you select.

#8. Edit the spreadsheet and expected-response contract

#8.1 Main editable columns

Supplemental fields include testType, riskLevel, testObjective, expected tool sequences, trace assertions, RAG/transaction assertions, resilience/fault settings, safety settings, and regression triggers. Use Workspace → Show Columns to reveal or hide them.

#8.2 Read-only result columns

The suite fills:

• Final Response(s);
• Harness Replies;
• actual tool sequence and validation details;
• trace assertion status/details;
• Status;
• average turn latency;
• average request latency;
• maximum request latency;
• timestamp.

#8.3 Edit cells

• Double-click an editable cell, or select it and press Enter.
• Use the Cell Editor for long values.
• Press Escape to cancel.
• Use Undo/Redo after structural or cell changes.
• The Execute? header toggles all rows.
• Numeric inputs in the Thread Count and Requests Per Thread headers can apply a value to the entire column.

#8.4 Configure Expected Results

Double-click an Expected Results cell to open the three-panel editor.

1. Choose a dialog type.
2. Complete its configuration.
3. For intent-based types, load or use intents.txt, search, and select acceptable intents.
4. Review the cell preview and serialized expectedResponsesConfig CSV preview.
5. Optionally choose Apply this rule to all rows.
6. Choose Apply Expected Responses.

The serialized configuration is exported as one CSV field and restored on import. Use the dialog rather than hand-editing that JSON whenever possible.

#9. Choose the right evaluator

The expected-response dialog exposes six principal contract types. The lower-level evaluation modes also include contains and informational for imported or wizard-created rows.

#9.1 Exact and intent alternatives

Separate acceptable alternatives with || in CSV data:

claims_request_claim_status||claims_request_claim_details

#9.2 Contains evaluator

Use short, meaningful concepts rather than full prose:

claim||under review||next step

The row passes if the output contains an alternative. Do not use common words that could cause accidental passes.

#9.3 Regex evaluator

Use regex for controlled structures, for example:

/^CLAIM-\d{6}$/i

Test the expression independently. Invalid regex is reported instead of silently treated as a match.

#9.4 Semantic evaluator

Select at least one requirement:

• address the user's utterance;
• follow relevant agent instructions;
• avoid unsupported or fabricated claims;
• permit an appropriate clarification;
• require Knowledge Base/retrieval evidence;
• be voice-friendly, optionally with a word cap.

Reference answers are optional and can be separated by a line containing ---. Temperature 0 is recommended for repeatable pass/fail decisions.

Knowledge requirements are evidence-based: a claim in the final answer is not enough. The suite looks for captured retrieval, citation, or tool evidence, optionally for a named tool.

Semantic evaluation needs a directly callable evaluator model. Production-agent fallback for auxiliary evaluation is disabled by default for safety.

#9.5 Disambiguation

Select the allowed intent family and the minimum number of related choices. Chat allows up to five choices. Voice-friendly mode limits the clarification to at most three concise choices and can impose a word cap.

#9.6 Tool-call-only evaluation

Select one or more tools in the required order. All required tools must appear in order; unrelated calls may occur between them. Arguments, results, and response text do not affect this specific check.

For stricter end-to-end validation, reveal and populate the supplemental tool assertion columns and enable strict validation in Test Plan.

#10. Build single-turn, multi-turn, tool, and stress tests

#10.1 A single-turn intent test

1. Create a row with input What is the status of my claim?.
2. Set both counts to 1 and Max Turns to 1.
3. Open Expected Results.
4. Choose Intents.
5. Select claims_request_claim_status.
6. Apply the contract.
7. Use Probe.
8. Confirm Passed or inspect the evaluator explanation.

#10.2 A flexible answer test

1. Use an input whose wording can produce several correct answers.
2. Open Apps → AgentOps → Eval Template Wizard.
3. Choose Flexible non-deterministic response or Generative / semantic response.
4. Enter sample input, expected concepts or references, and measurable success criteria.
5. Apply the template to the active row or create a new row.
6. Review the generated fields before running.

#10.3 A multi-turn test

There are two supported styles.

Dialog contract through the UI:

1. Choose Multi-turn Dialog in Expected Results.
2. Select one or more allowed final intents.
3. Set Maximum Dialog Turns between 2 and 20.
4. Apply and run.

Scripted CSV/row turns: place a JSON array in the turns field:

[
  {
    "user": "I need help with a bill",
    "expected": "claims_charge_details_inquiry"
  },
  {
    "user": "It is for June 15",
    "expected": "claims_charge_details_inquiry",
    "humanDelayMs": 12000
  }
]

Human delay is recorded separately and excluded from agent response timing. If the agent asks for information and no scripted reply exists, model-generated user simulation requires a directly callable model; unsafe production-agent fallback is off by default.

#10.4 A tool-routing test

1. Select a profile whose agent metadata includes tools.
2. Open Expected Results and choose Verify Tool Calls Only.
3. Add the required tools in order.
4. Apply the contract.
5. Run once with Probe.
6. Inspect actual tool sequence and details in Run Review.
7. If tool mismatch must fail the whole row, enable Strict tool validation in Test Plan.

#10.5 A local Tool Sandbox test

Use this when production tools must not execute.

1. Open Apps → AgentOps.
2. Enable Tool Sandbox.
3. Load and select a chat model.
4. Set the maximum model/tool steps.
5. Choose Generate from Active Agent, or enter local stub JSON.
6. Add deterministic cases, outputs, and errors.
7. Save the sandbox.
8. Keep concurrency at 1 × 1 and run or chat.

Example stub:

[
  {
    "name": "lookup_claim",
    "parameters": {
      "type": "object",
      "properties": {
        "claimId": { "type": "string" }
      }
    },
    "cases": [
      {
        "when": { "claimId": "C-100" },
        "output": { "status": "pending" }
      },
      {
        "when": { "claimId": "/^ERR-/" },
        "error": "Claim service unavailable"
      }
    ]
  }
]

The sandbox uses the real selected model and visible instructions but bypasses the production agent endpoint, production tools, native memory, collaborators, and hidden orchestration.

#10.6 A stress test

Before increasing counts, compute:

iterations per row = threadCount × requestsPerThread
base suite iterations = sum(iterations for every enabled row)

This is not necessarily the total outbound-call count. Multi-turn conversations, semantic evaluation, model-generated user replies, endpoint-shape fallback, and the agent's own tool fan-out can add calls.

Start with a small canary:

1. One enabled row.
2. threadCount=1 and requestsPerThread=1.
3. Run and validate correctness.
4. Increase to a modest repeat count.
5. Review traffic estimates and the server's governor policy.
6. Obtain environment/SRE approval before a high-volume run.

The default process-wide governor allows at most 4 concurrent outbound requests and queues at most 250. It is process-local; multiple suite processes multiply aggregate traffic.

#11. Plan a complete test campaign

Open Apps → Test Plan.

#11.1 Select disciplines

The plan contains:

• Unit Testing;
• Functional Testing;
• Agent Quality Testing;
• Adversarial & Safety Testing;
• Performance & Scalability Testing;
• Regression Testing.

For each applicable discipline:

1. Set Applies? to Yes.
2. Choose Manual, Automated, or Hybrid.
3. Name an owner.
4. Choose Dev, QA/QC, UAT, or Prod-like.
5. Describe required data, traces, and tools.
6. Write measurable acceptance criteria.
7. Add dependencies or manual notes.

#11.2 Configure readiness gates

The plan includes:

• strict tool validation;
• strict trace validation;
• instruction warning/failure token estimates;
• tool-count warning;
• context token warning;
• P95 latency budget;
• retry cap.

Readiness findings are advisory and do not themselves block execution. Strict tool and trace switches can convert those specific validation failures into row failures.

#11.3 Generate a test matrix

Choose Generate Test Matrix. The suite creates or updates one draft row for each applicable discipline.

Generated rows:

• receive the matching testType;
• get initial risk metadata;
• copy acceptance criteria to testObjective, not Expected Results;
• receive trace/ground-truth declarations where appropriate;
• are marked with source=test_plan;
• remain disabled with blank executable inputs and expected values.

Complete and review every draft before enabling it.

#11.4 Generate a resilience suite

Choose a representative active row, then choose Generate Resilience Suite. The suite creates disabled drafts for ambiguity, conflicting requests, mid-turn correction, non-cooperation, prompt injection, unauthorized tool use, timeouts, errors, empty/malformed/partial tool results, and recovery behavior.

Fault profiles activate only in Tool Sandbox. Review the inherited expected contract before enabling any generated row.

#11.5 Import or export the plan

Use Import JSON and Export JSON in the Test Plan workspace to share planning metadata separately from executable sheets.

#12. Run tests safely

#12.1 Choose a run mode

Open Run → Run Mode.

Capture-first is useful when evaluator criteria are still being tuned or when you want to compare several contracts against the same model output without paying for another production invocation.

#12.2 Use Probe first

1. Select one row.
2. Choose Probe.
3. Confirm the response, status, timing, and tool/trace evidence.
4. Correct the row contract or profile before running the full sheet.

#12.3 Run all enabled rows

Choose All. Only checked Execute rows run.

#12.4 Run from the current row

Select a row and choose Rows N–end. Enabled rows before the selection are excluded from the scope and may show Skipped when they had no prior result.

#12.5 Apply a throttle

Choose Throttle and enter a delay in milliseconds. The pause occurs between test-case rows and is excluded from agent elapsed time. Valid values are 0 through 600,000.

Throttle is useful for demonstrations, conservative production traffic, or downstream systems with rate limits. It does not replace the server's global concurrency governor.

#12.6 Pause, continue, and stop

• Pause waits at a safe boundary; choose Continue to resume.
• Stop cancels queued work and propagates cancellation to active upstream HTTP requests where supported.
• Recording remains active through a pause and closes when the run completes or stops.

#12.7 Interpret statuses

For stress rows, one failed constituent request fails the aggregate row. The grid can show a compact final value while reports retain individual request results.

#13. Read results and debug failures

#13.1 Start with the spreadsheet

Review:

• status;
• score;
• final responses;
• average and maximum request latency;
• tool sequence status;
• trace assertion status.

Double-click Final Response(s) to open Captured Output. It lists individual thread/request responses, timings, status, evaluator explanations, and copy actions.

#13.2 Use Run Review

1. Open Run → Debug.
2. Select the failed row.
3. In Run Review, compare the expected contract and actual response.
4. Inspect each turn and request.
5. Review the execution path, warnings, harness-generated replies, and timing.
6. For stress tests, identify the first failed thread/request instead of relying only on the aggregate cell.

#13.3 Use Evaluation

The Evaluation tab can report on:

• selected row;
• all completed rows;
• failures and errors;
• the entire suite.

Choose Load trace when a trace ID exists. Export the report as JSON for tooling or Markdown for human review.

#13.4 Use the REST API debugger

The debugger's REST API tab shows local health, transport, the active profile, known endpoints, and the latest request/response for manual diagnostics. Credentials are redacted from diagnostics and recordings.

#13.5 Separate latency measurements

The timing contract is important:

agentResponseMs = Direct API HTTP round-trip minus governor queue time
harnessOverheadMs = client/server/evaluation/persistence work outside agent latency

Average Request Latency averages independent thread/request executions. Maximum Request Latency is the slowest one. Human delay in multi-turn scripts is excluded from agent timing.

#14. Reports and exports

#14.1 Run Report

After a run, choose Report. The report includes outcome counts, pass rate, latency/performance summaries, readiness dimensions, and run metadata. Choose Export PDF from the report dialog for a portable review artifact.

#14.2 Excel Snapshot

Exports:

• only currently filtered rows;
• only visible worksheet columns;
• columns in their current on-screen order;
• formatted .xlsx with filters;
• no internal debug payloads.

#14.3 CSV Snapshot

Uses the same visible/filter-aware view as Excel Snapshot but writes CSV.

#14.4 Response Excel and Response CSV

These tuning-oriented exports create one output row per captured thread/request response. Fields include prompt, actual response, status, thread, request, expected result, evaluator mode/reason, score, latency, agent, and timestamp.

Use a Response export for model tuning or error analysis. Use a Snapshot export to reproduce what an analyst currently sees in the sheet.

#14.5 Test-case CSV save

Test Data → Save/Save As writes the complete round-trip schema, including hidden configuration, results, execution context, and packed per-agent result context. It is the best format for continued spreadsheet editing, but not as easy to read as the filtered snapshots.

#15. Record, replay, analyze, and compare runs

#15.1 Record a run

1. Choose Record before starting the run.
2. Confirm the recording icon is armed.
3. Run Probe, All, or From Current.
4. The indicator changes while recording.
5. When the run completes or stops, the suite asynchronously saves a .wxorec file.

The recording includes rows, results, request/response diagnostics, evaluation evidence, traces, reasoning metadata, runtime events, benchmarks, and telemetry when available. Credential-shaped fields are redacted.

#15.2 Playback

1. Choose Playback.
2. Select a local or shared recording.
3. Choose Play Selected.
4. The suite restores recorded row results and debugger evidence without rerunning the production agent.

The dialog also offers Summary, Pattern Insights, Inspector, refresh, and sharing for local recordings.

#15.3 Compare recordings

1. Choose Compare.
2. Select between 2 and 12 recordings.
3. Review pass rate, failure/error rate, average latency, tool sequence validation, repeated failing inputs, intent-family failures, and stable wins.

Comparisons are most useful when the input sheet is held constant across agents, model versions, instruction changes, or environments.

#15.4 Recording format

New .wxorec files are lossless binary containers with:

• magic header WXOREC2\n;
• a JSON metadata header;
• a gzip level-1 payload containing the compact complete recording.

Legacy JSON recordings remain readable.

Convert or inspect from the repository root:

node app/scripts/wxorec-convert.js inspect path/to/run.wxorec
node app/scripts/wxorec-convert.js to-json path/to/run.wxorec /tmp/run.json
node app/scripts/wxorec-convert.js to-binary path/to/legacy.json /tmp/run.binary.wxorec

The utility refuses to overwrite an existing output file. The older documentation mentions an npm recordings:convert script, but the current root package.json does not define it; use the direct Node commands above.

#16. Use AgentOps, Tool Sandbox, and Inspector

#16.1 AgentOps defaults

Open Apps → AgentOps to set:

• default evaluator: intent, contains, regex, or semantic;
• case-sensitive matching;
• per-turn timing capture;
• Langfuse configuration;
• Eval Template Wizard;
• Tool Sandbox;
• multi-turn schema reference.

#16.2 IBM ADK EVAL export

Choose Export IBM EVAL Dataset to create a draft ground-truth scaffold from enabled rows. Enrich trajectories, dependency graphs, and reviewed ground truth before using it as a certification dataset.

#16.3 Langfuse

Enter the host, public key, and secret key, then optionally enable trace sending. You can also export Langfuse JSONL without live publication. Credentials are stored locally; protect the machine and workspace accordingly.

#16.4 Inspector

Open Apps → Inspector to analyze the selected agent using:

• visible instructions and agent metadata;
• connected tool metadata and available source;
• loaded test rows and expected contracts;
• selected prior recordings;
• optional uploaded instruction and tool/source files;
• optional live probes;
• optional grounded model-review preparation.

Recommended workflow:

1. Select the target profile/agent.
2. Load representative test cases.
3. Record at least one baseline run.
4. Open Inspector.
5. Keep loaded rows and selected recordings enabled as evidence.
6. Upload full instructions/tool source when hidden runtime metadata is incomplete.
7. Leave live probes off for a static review, or deliberately enable up to four governed probes after checking credentials and traffic.
8. Run the inspection.
9. Review contradictions, duplicate/circular rules, tool-contract gaps, deterministic JSON/decision contracts, readiness, and failure clusters.
10. Copy suggested wording or prepare a complete rewrite draft.
11. Review any draft manually. Inspector does not automatically change the live agent.

The clone workflow creates a draft clone and does not modify or deploy the original agent. It uses governed management calls and requires confirmation.

#17. Save, restore, and share work

#17.1 Know the artifact types

#17.2 Save a workspace

Choose Workspace → Save As. In browsers with File System Access support, later Save updates the same file. Otherwise, each save downloads a new .wxotsw bundle.

A workspace captures the current active sheet, not every open workbook tab. When opened, it replaces the current active worksheet; other tabs remain available.

#17.3 Open a workspace

1. Stop any active run.
2. Choose Workspace → Open.
3. Select .wxotsw or compatible JSON.
4. Confirm replacement of the active worksheet.
5. Reenter missing API keys for imported profiles.

#17.4 Automatic local persistence

By default, profiles, rows, workspaces, preferences, reports, and recordings use .wxots-harness-state/ or the configured state directory. Large state is compacted and stored through local server endpoints so browser quota is not the primary limit.

Do not delete the state directory when you intend to preserve local work. The build process keeps it outside BUILD/ so rebuilding does not remove it.

#17.5 Optional wxots-db sharing

To run the collaboration service locally, use two terminals.

Terminal 1:

npm run start:db

Terminal 2:

WXOTS_DB_URL=http://127.0.0.1:3041 npm start

Then use Share or Cloud Open in the Test Data/Workspace groups. Recordings can be shared from Playback or Compare. wxots-db stores presence, chat, and JSON artifacts; it is optional, not the primary test execution engine.

Shared profile data excludes reusable API keys. That is a permission boundary, not an inconvenience to bypass.

#18. Use the REST API workspace

Open Apps → REST API for endpoint discovery, generated examples, and exposure configuration.

#18.1 Local authenticated/internal routes

Important routes include:

GET  /api/health
GET  /api/harness/capabilities
GET  /api/wxo/debug-endpoints
POST /api/wxo/agents
POST /api/wxo/invoke
POST /api/wxo/semantic-evaluate
POST /api/wxo/conversation-reply
POST /api/wxo/tool-sandbox-invoke
GET  /api/recordings
POST /api/recordings
GET  /api/runtime/status
POST /api/runtime/models
POST /api/runtime/invoke

Inspect the live capability catalog:

curl -fsS http://127.0.0.1:3038/api/harness/capabilities | jq

#18.2 Invoke an agent

The core request shape is:

{
  "profile": {
    "baseUrl": "https://api.us-south.watson-orchestrate.cloud.ibm.com/instances/<id>",
    "apiKey": "<credential>",
    "selectedAgentId": "<agent-id>"
  },
  "input": "What is the status of my claim?",
  "timeoutMs": 90000
}

Send POST requests with Content-Type: application/json. Avoid placing credentials in shell history; use a protected request file, a secret manager, or an approved encrypted client workflow.

#18.3 Public API gateway

/api/public/* endpoints are deny-by-default. The REST API workspace can map approved capabilities to public paths and mark only deliberate operations as exposed. An unexposed path returns 404 with “This suite API is not exposed.”

Treat clone, delete, live invocation, telemetry, and alert routes as privileged operations. Do not expose them merely to simplify a script.

#18.4 Automation limitation

The current capability manifest lists full “run test suite,” upload, and report automation as future public APIs. Today, the UI is the primary batch-run orchestrator; the REST endpoints expose lower-level invocation, evaluation, recordings, diagnostics, and AgentOps building blocks.

#19. Run the developer regression suite

The root package does not define npm test. Use npm run check.

#19.1 Full root check

npm run check

This performs JavaScript syntax checks and runs the root regression scripts in sequence. It also runs npm --prefix wxots-version run check.

The suite is self-contained: runtime/network behavior is exercised through local fixture servers and temporary directories, not production watsonx Orchestrate traffic.

Expected successful output ends with messages such as:

Native runtime release selection, bundle layout, and private runner resolution checks passed.
Ollama and native runtime model discovery, agent loops, evaluation, and conversation checks passed.
Large-state persistence check passed: ...
Final response formatting, copy, and flat-export checks passed.
...
Version catalog ordering check passed.

The command exits 0 on success and stops at the first failure because the package script uses &&.

#19.2 What the root checks cover

Before functional scripts run, npm run check also runs node --check against the server, browser modules, native bundler, database service, and version service.

#19.3 Run one regression check

Use direct Node execution:

node scripts/test-search-filter.js
node scripts/test-xlsx-export.js
node scripts/test-built-in-runtime.js
node scripts/test-presence-expiration.js

This is useful during development, but always run the aggregate command before handing off a change.

#19.4 Version service only

npm --prefix wxots-version run check

#19.5 Verified result for this tutorial

On August 19, 2026, npm run check passed completely with the historical standalone copies and retired landing-page package absent.

The check includes the current wxots-version ordering test. There is no separate wxots-landingpage command in the current workspace. In a restricted sandbox, the root check can fail with listen EPERM because two tests bind ephemeral loopback servers; run it in a normal terminal or an environment that permits 127.0.0.1 binding.

#20. Add or troubleshoot developer tests

#20.1 Current test style

The checks use Node's built-in assert; there is no Jest/Mocha runner. Most scripts are ordinary CommonJS programs that exit nonzero on a thrown assertion or rejected promise.

#20.2 Add a focused test

1. Create scripts/test-<feature>.js.
2. Add 'use strict';.
3. Import assert and the smallest testable module.
4. Use temporary directories from os.tmpdir() for filesystem cases.
5. Bind local fixture servers to 127.0.0.1 on port 0 so the OS chooses a free port.
6. Use try/finally to close servers and remove temporary directories.
7. Print one clear success line.
8. Add node scripts/test-<feature>.js to the root check script in the right dependency order.
9. Run the new script directly.
10. Run npm run check.

Minimal example:

'use strict';

const assert = require('assert');
const { functionUnderTest } = require('../path/to/module');

assert.strictEqual(functionUnderTest('input'), 'expected');

console.log('Feature checks passed.');

#20.3 Understand failures

Common patterns:

• AssertionError: behavior or a source/UI contract changed. Read the assertion's message before updating the test.
• MODULE_NOT_FOUND: dependencies were not installed or the require path is wrong.
• EADDRINUSE: a test or app attempted a fixed occupied port. Tests should normally use port 0.
• EPERM ... listen 127.0.0.1: the execution environment forbids loopback servers; run in an allowed terminal/environment.
• Syntax failure before test output: the relevant node --check command failed.
• Root sequence stopped early: run the last printed node scripts/... command directly to isolate it.

Do not weaken a source-contract assertion merely because copy, markup, or a CSS selector changed. First decide whether the intended product behavior changed, then update implementation and test together.

#20.4 Run cleanly in a dirty worktree

The test scripts should use temporary locations and avoid rewriting tracked application data. Before and after a test run:

git status --short

Investigate unexpected file changes rather than discarding them automatically.

#21. Troubleshooting the application

#Server does not start

1. Check the Node version.
2. Run npm install.
3. Look for the first error in the terminal.
4. Try a different port:

``bash WXOTS_PORT=3040 npm start ``

1. Confirm the chosen port in WXOTS_BOUND_PORT.

#Browser shows an old build

Use the version badge/update flow or hard-reload the browser. The app uses versioned immutable caching for static JS/CSS.

#Agent list does not load

Check:

• full service-instance URL;
• valid IBM Cloud API key;
• network access to the instance;
• user permission to list agents;
• debugger/terminal HTTP status;
• whether a previous refresh was cancelled by closing the dialog.

#Row says Error immediately

Select it and inspect Debugger → REST API. Common causes are an incomplete profile, selected agent ID no longer present, expired credential, malformed custom context, missing local model, or a directly callable evaluator model not being available.

#Semantic row is Pending Evaluation or cannot evaluate a capture

Configure a directly callable evaluator model through the selected runtime/agent metadata. Capture mode itself can save responses without one; semantic testing later cannot.

#Native model is not detected

1. Confirm the filename ends in .gguf.
2. Put it under .wxots-harness-state/models or WXOTS_NATIVE_MODEL_DIR.
3. Return focus to the browser or reopen Runtime.
4. Check whether the source-tree/standalone copy contains an engine compatible with the host OS/architecture.
5. Check port 11435 or change WXOTS_NATIVE_MODEL_PORT.

#Tool Sandbox unexpectedly bypassed

Stress workloads always bypass it. Reset both counts to 1 and avoid testType=performance_scalability for a sandboxed functional test.

#Expected result looks correct but fails

Confirm the evaluator:

• intent and strict are exact contracts;
• contains accepts a substring alternative;
• regex syntax must be valid;
• semantic requirements may demand instruction, grounding, voice, or KB evidence beyond prose similarity;
• tool-call-only ignores response text and checks ordered calls.

#CSV columns are wrong

Open Mapper, reset to auto-map defaults, then explicitly map ambiguous headers. Save As after mapping to preserve the new canonical round-trip schema.

#Workspace import requires credentials

This is expected for shared/sanitized workspaces. Open each restored profile and enter your own API key.

#wxots-db controls are unavailable

Check WXOTS_DB_URL, start the service, and verify /health. Local execution, local workspace saving, and local recordings remain available.

#Large run feels slow before or after agent calls

Separate agent latency from harness overhead. Large CSV parsing, report aggregation, persistence, compression, and rendering can consume time without increasing agentResponseMs. Use pagination, hide unnecessary columns, filter rows, and save compact workspace/recording formats.

#22. Recommended operating practices

1. Use 1 × 1 canaries. Verify one row before running a suite or any stress load.
2. Make the contract explicit. Every automated row should have a deterministic expected value or a semantic policy with at least one enabled requirement.
3. Separate metadata from scoring. testObjective, risk, safety category, and readiness flags describe the test but do not automatically evaluate the answer.
4. Use capture-first for evaluator tuning. Re-test the same captured output instead of repeatedly invoking production.
5. Keep production certification. Local runtimes and Tool Sandbox are excellent for iteration but cannot reproduce hidden hosted orchestration by IDs alone.
6. Record baselines. Save .wxorec evidence before instruction, model, or tool changes.
7. Compare like with like. Hold the dataset, evaluator contract, runtime, and environment stable when comparing agents.
8. Review the first constituent failure. Aggregate stress failures need thread/request-level diagnosis.
9. Protect credentials and evidence. Profiles, workspaces, traces, and recordings can contain sensitive operational data even when credentials are encrypted/redacted.
10. Respect traffic amplification. Multiply rows, threads, requests, turns, auxiliary model calls, and internal agent tool fan-out.
11. Save both CSV and workspace. CSV is portable test data; .wxotsw preserves richer investigation context.
12. Run the aggregate developer check before release. npm run check is the authoritative regression command and includes the current version-service check.

#23. End-to-end certification checklist

Use this repeatable workflow for a release-quality agent test.

#Prepare

• Run npm run check.
• Start one known suite process and record its URL/version.
• Confirm /api/health and traffic-governor settings.
• Create/refresh the correct profile and agent.
• Choose the intended runtime.
• Import the reviewed test dataset.
• Save a pre-run .wxotsw workspace.

#Validate contracts

• Every enabled row has input.
• Every automated row has an expected value or semantic policy.
• Intent alternatives use ||.
• Multi-turn rows have allowed final intents and a turn cap.
• Tool-dependent rows define ordered tools where needed.
• Strict tool/trace settings match the certification objective.
• Readiness warnings are reviewed.
• Thread/request counts are intentional.

#Canary

• Probe one deterministic row.
• Probe one semantic or multi-turn row.
• Probe one tool row.
• Inspect request/response and evaluator evidence.
• Confirm timing semantics.

#Execute

• Arm recording.
• Set an approved throttle.
• Review the traffic estimate.
• Run enabled rows.
• Monitor progress and stop on systematic errors.

#Review

• Open the Run Report.
• Filter failures/errors.
• Review the first failing constituent request per aggregate failure.
• Load available traces.
• Export Evaluation Markdown or JSON.
• Export Response Excel/CSV for tuning.
• Export PDF and visible snapshot for stakeholders.

#Preserve and compare

• Confirm the .wxorec saved.
• Save the final .wxotsw workspace.
• Save the round-trip CSV.
• Compare against the prior baseline recording.
• Run Inspector with reviewed evidence.
• Document accepted risks and follow-up changes.

#24. Reference: environment variables and formats

#24.1 Core server

#24.2 Runtimes

#24.3 Traffic safety

Change traffic controls only with environment-owner/SRE approval.

#24.4 Collaboration and recordings

#24.5 Minimal canonical CSV

execute,threadCount,requestsPerThread,input,subSegment,channel,watsonIntent,expectedResponses,expectedResponseType,maxTurns,evaluationMode,status
YES,1,1,What is the status of my claim?,group,chat,claims_request_claim_status,claims_request_claim_status,intents,1,intent,Pending
YES,1,1,I have a bill from my doctor.,group,chat,,claims_charge_details_inquiry||claims_claim_bill_validation,disambiguation,3,disambiguation,Pending

For complete round-tripping, let the suite export the CSV after configuring rows. Its canonical export includes expectedResponsesConfig, semantic flags, tool/trace/safety fields, results, execution context, and packed per-agent result data.

#24.6 Source references

For implementation-level details, consult:

• README.md for startup, runtimes, database, build, and deployment;
• app/docs/REST_API.md for REST concepts and request examples;
• app/docs/RECORDING_FORMAT.md for the binary recording container;
• app/docs/WXO_TRAFFIC_SAFETY_AUDIT.md for call amplification and governor design;
• app/docs/PERFORMANCE_AUDIT.md for persistence, recording, and UI performance behavior;
• package.json for the authoritative root check sequence.