Routing customer work across OpenAI and Gemini

Classify requests by capability: fast classification, grounded question answering, structured extraction, multimodal understanding, complex reasoning, or copy generation. Choose a tested route for each task instead of sending every message to one default.

Keep tenant policy above provider configuration so the same behavior, tool permissions, and audit fields apply whether OpenAI or Gemini handles the turn.

Create one internal request shape for instructions, messages, tools, attachments, safety context, and response constraints. Translate it inside provider adapters and normalize usage, finish reasons, errors, and structured outputs on return.

This boundary prevents UI and business services from depending on provider details and makes model changes testable rather than invasive.

Use evaluation scores, latency, context needs, modality, regional availability, and effective cost per successful outcome. Token price alone is misleading when a cheaper route causes retries, failed tool calls, or more human escalation.

Expose routing policy by complexity and allow tenant-specific approved providers. Track actual usage against the service and conversation that created it.

Validate tool arguments against strict schemas, enforce identity and tenant scope server-side, and return stable error structures. The model may request a tool, but the application decides whether it is allowed and whether approval is required.

Use idempotency keys for commercial actions. A provider retry or fallback must never create a second quote, payment link, booking, or refund.

Fallback should distinguish temporary provider failure from invalid input, policy refusal, context overflow, or a failing business dependency. Retry only errors that can improve, use bounded timeouts, and preserve the same idempotency key.

If quality would drop below the service threshold, explain the delay or escalate instead of silently switching to an unsuitable model.

Run the same representative conversation set across providers and versions. Compare factuality, instruction adherence, tool selection, structured validity, safety, language quality, latency, and cost.

Require a passing regression before changing a production route. Store model and adapter versions in the audit trail so later review can reproduce which path generated a response.