# Agent reliability

Updated 13 September 2026. Open **Agent reliability** from the administrator navigation. All endpoints are under `/api/v1/reliability`, require an active tenant administrator session, and use the existing CSRF protection for writes. External provider accounts must be configured before paid tests run.

## Runtime traces

Website and connected-channel chat entry points record observable provider and agent routing, model calls, published knowledge retrieval, sales-tool invocation, verified outbound API calls, and final responses. Workflow execution, mass simulation comparisons, batch investigations and voice tests have trace roots. Nested operations retain parent span identifiers, elapsed milliseconds, completion state and failure codes. Model spans record the usage information returned by the existing runtime; unavailable token prices remain unavailable. Hidden model reasoning is neither requested nor recorded.

The workspace retains the latest 100 completed trace records with at most 80 spans each. Each redacted detail is limited to 600 characters. These are bounded diagnostic records, not full archival transcripts; truncation and scope matter when investigating a problem. Redaction is best-effort and is not a guarantee that all personal information has been removed. Administrators should treat traces as customer data. Full messages remain in their source conversations. Deleting a conversation removes its associated traces and reports; bulk investigations associate each trace with its source batch. Provider-internal decisions, audio streaming events and code paths outside the instrumented entry points are not represented.

Trace persistence failure is logged without turning an already completed customer action into a retryable error. A runtime interruption marks unfinished spans as failed. Operations retain their existing no-replay behavior.

## Mass simulation and regression

1. Enroll an agent in Agent releases and create a candidate with business scenarios and required or forbidden phrases.
2. Select the candidate in Simulations. The supplied 50 customer contexts and 20 concerns produce 1,000 combinations per distinct seed. Review or replace these inputs for the business. Up to 5,000 scenarios are supported.
3. Preview the scenarios. Preview calls no model and performs no customer action. Changing the form invalidates the preview.
4. Authorize the displayed ceiling: one baseline and one candidate request per scenario, up to 10,000 model calls. Normal workspace AI usage rules continue to apply. There is no dollar-cost estimate where provider pricing is unavailable.
5. Queue the job and inspect counts, regressions and retained examples. Only compact seeds and generation settings are stored. Each comparison reserves its calls before execution and saves progress afterward.

The worker performs comparisons sequentially and can take hours for a large job. A workspace permits one active mass simulation and three new jobs per day. Ten recent jobs are retained, with up to 100 example results per job, prioritizing failures. Counts cover the complete processed set even when examples are omitted. A provider error or restart stops the job; there is no automatic retry. Cancellation stops after the current comparison. It cannot undo calls already sent to the provider.

Checks cover literal required/forbidden phrases, empty answers, unapproved product identifiers and recognizable credential patterns. These are not semantic correctness guarantees. The generated matrix combines administrator-reviewed cases with context variations; it does not claim model-generated diversity or full customer-journey simulation. No real carts, refunds, messages, bookings or custom tools execute during these response tests.

An administrator can require a minimum passing scenario count before publishing. The minimum can only increase. Publishing requires a passing complete job from the previous 24 hours against the current baseline, agent settings, provider, catalog, knowledge and runtime revision, in addition to existing release checks. Changed sources invalidate the result. Emergency rollback remains available.

## Voice lab

Upload consented test audio as a mono uncompressed 16-bit PCM WAV, 8–48 kHz, up to 40 seconds and 1.3 MB. Supply the expected transcript and authorize two speech-recognition requests. The lab compares the original with deterministic seeded white noise and a midpoint audio dropout. It reports transcripts, word-edit counts and word error rate. Word error rate can exceed 100% when insertions exceed reference length; segmentation uses Unicode words and is not a language-specific benchmark.

Accents, spoken interruptions, competing speakers and language switching must be present in the recording. The lab does not generate an accent or claim that text perturbation is an acoustic test. It does not test real-time barge-in, latency to stop speaking, live telephony transfer, TTS quality or an entire voice-agent conversation. Those require a separate real-time duplex harness.

The backend uses the configured `ELEVENLABS_API_KEY` with the fixed official speech-to-text endpoint and `scribe_v2`. See [ElevenLabs create transcript documentation](https://elevenlabs.io/docs/api-reference/speech-to-text/convert). The original and impaired audio are sent to that provider; ConveRact does not persist the uploaded audio. Provider retention and billing settings still apply. Calls have a 60-second timeout and tests have a one-minute per-workspace cooldown.

## Knowledge gap engine

Scan up to 90 days of customer messages. The engine groups normalized questions, ranks them by observed frequency and checks up to 500 distinct question groups against currently published knowledge retrieval. It records questions with no matching passage, source conversations and counts. The scan reports selected coverage and truncation.

This is retrieval-based triage. A missing passage can reflect wording, catalog-only answers, missing context or an actual knowledge gap. English question cues plus question marks determine candidates; broader multilingual and semantic coverage remain future work. The current scan does not prove that an answer was wrong or automatically infer a new business policy. Review the source, dismiss irrelevant gaps or enter a verified answer as an unpublished FAQ draft. Publish through the existing Knowledge review flow. Up to 500 potential gaps are retained.

## Explorer

The period comparison scans all matching recorded public messages and active customer conversations. It compares the chosen period with the immediately preceding period. A linked paid conversation requires an explicit customer link and a verified, non-test, currently-paid store order within that period. This is an observed linkage, not causal sales attribution. Missing provider imports or links undercount conversions; multiple conversations linked to a customer can each count. Historical order status and conversation-status snapshots are not reconstructed.

Quick analysis sends those aggregates and up to 48 relevant excerpts to the existing cited investigation endpoint. Batch investigation provides broader coverage: select up to 5,000 recent matching conversations and authorize one model call per 25 conversations, up to 200 calls. Every selected conversation contributes at most its last four public messages within the period, truncated to 400 characters each. Older and longer content is omitted explicitly. Batch findings cite supplied messages; invalid citations or changed evidence stop the job. The interface exposes every retained batch rather than presenting a small sample as complete analysis.

Jobs persist progress, support cancellation after the current batch, stop after provider failure/restart and retain partial results. A workspace supports one active batch investigation, two new jobs per day, and three retained jobs. Batch findings are hypotheses and review aids; they do not establish why a conversion change happened. No mass investigation is started by simply opening the page.

## Reusable tool and journey builder

Configure a custom public HTTPS API action in Service Operations, then add it as a visual building block. Compose it with existing workflow blocks, reorder blocks and set the supported conversation conditions for individual API blocks. Existing workflows are copied at the displayed version; source-version changes between loading and saving are rejected. The expanded result is a paused workflow with at most 12 steps. Review, enable and execute it using the existing human approval flow.

This extends reusable composition without requiring frontend development for each approved API action. It is not yet a general graph engine: arbitrary branching, loops, output-to-input data bindings, delayed timers, OAuth lifecycle and native business-result contracts remain separate engineering work. A generic successful HTTP response does not prove that a refund, payment or other business effect completed.

## Additional controls included

Tenant-admin access, CSRF checks, scoped reads, bounded records, call ceilings, deliberate paid-test acknowledgements, cancellation, citation checks, source deletion handling, stale-release rejection and publishing gates accompany the six capabilities. Test accounts and provider-backed acceptance are still required before relying on them for business-critical decisions.

The deployment continues using the existing hosted-demo configuration. Its `/ready` check reports that configuration until the account migration is completed. Self-service billing, SMTP, a verified public legal email and provider-specific credentials remain launch requirements. No release claim implies all 15 original product areas or every requested enterprise feature is complete.