devLap

Your AI Gateway Needs a Conversion Graph, Not a Pile of If Statements

go dev.to

The innocent-looking requirement

"One endpoint for every AI model" sounds like a routing problem.

Receive JSON, change a few field names, forward the request, and convert the response back. That approach can work while every provider looks approximately like OpenAI Chat Completions.

It breaks once the gateway must support several real protocols:

  • OpenAI Chat Completions
  • OpenAI Responses
  • Anthropic Messages
  • Gemini generateContent

These are not four spellings of the same schema. They differ in message structure, tool calls, reasoning content, usage accounting, finish conditions, and streaming event lifecycles.

With four protocols, there are:

4 * (4 - 1) = 12
Enter fullscreen mode Exit fullscreen mode

possible directed conversions.

Direction matters. Converting Anthropic Messages to OpenAI Chat is not the same implementation as converting OpenAI Chat to Anthropic Messages.

A large switch provider may handle the first two integrations. At twelve routes, it becomes difficult to answer basic questions:

  • Which conversions actually exist?
  • Which routes are direct?
  • Which routes pass through an intermediate format?
  • How much information can each route preserve?
  • Does the request converter have a matching response converter?
  • Which streaming paths need persistent state?

The more useful abstraction is a directed conversion graph.

Model protocols as nodes and conversions as edges

In the Go gateway implementation behind Aivrae, each protocol is represented as a format, and each supported conversion is registered as a directed route.

The six core direct edges are:

Anthropic Messages  -> OpenAI Chat
OpenAI Chat         -> Anthropic Messages

Gemini Content      -> OpenAI Chat
OpenAI Chat         -> Gemini Content

OpenAI Chat         -> OpenAI Responses
OpenAI Responses    -> OpenAI Chat
Enter fullscreen mode Exit fullscreen mode

OpenAI Chat acts as a practical interchange format, but it is not treated as a perfect universal representation.

The remaining protocol pairs are declared explicitly as composed routes. For example:

Anthropic Messages
    -> OpenAI Chat
    -> Gemini Content
Enter fullscreen mode Exit fullscreen mode

and:

Gemini Content
    -> OpenAI Chat
    -> OpenAI Responses
Enter fullscreen mode Exit fullscreen mode

This gives the registry all twelve directed routes while limiting the number of core converters that need independent maintenance.

A simplified route definition looks like this:

type Route struct {
    ID      string
    From    Format
    To      Format
    Quality Quality

    Request  RequestPlan
    Response ResponsePlan
}

type RequestPlan struct {
    Convert DirectRequestConverter
    Steps   []RouteID
}

type ResponsePlan struct {
    Convert      DirectResponseConverter
    NewState     StreamStateFactory
    ConvertChunk StreamChunkConverter
    Finalize     StreamFinalizer
    Steps        []RouteID
}
Enter fullscreen mode Exit fullscreen mode

This is pseudocode, but the separation is important. A route is more than a request transformer. It describes request conversion, non-streaming response conversion, and the streaming lifecycle.

Do not let the graph choose paths silently

Once conversions are modeled as a graph, automatic pathfinding looks tempting.

If no direct edge exists, run breadth-first search and choose the shortest route.

That is dangerous for protocol conversion.

Two paths with the same number of edges may preserve different semantics. An intermediate representation may discard a field that another route could retain. A shorter path may also have less mature streaming support.

Instead, composed routes should list their steps explicitly:

Route{
    ID:   "anthropic_to_responses",
    From: AnthropicMessages,
    To:   OpenAIResponses,
    Request: RequestPlan{
        Steps: []RouteID{
            "anthropic_to_chat",
            "chat_to_responses",
        },
    },
}
Enter fullscreen mode Exit fullscreen mode

Registration should fail early if:

  • a route ID is empty or duplicated;
  • a source and target pair is registered twice;
  • a referenced step does not exist;
  • a composed route points to another composed route unexpectedly;
  • one step ends in a format that the next step does not accept;
  • the last step does not reach the declared target.

In the Go implementation, these checks happen while built-in routes are registered. A broken conversion chain becomes a startup failure instead of a malformed response discovered under traffic.

Request and response plans do not always need the same internal strategy. One route can use a specialized direct request translator while composing its response through OpenAI Chat. Keeping both plans under one route specification makes that asymmetry visible.

Compatibility needs a quality label

A boolean supported flag is not enough.

Some conversions preserve most of the useful semantics. Others require compromises. A gateway should not imply that both have equal fidelity.

The registry uses three quality levels:

const (
    QualityGood        Quality = "good"
    QualityFair        Quality = "fair"
    QualityDiscouraged Quality = "discouraged"
)
Enter fullscreen mode Exit fullscreen mode

For example, the direct conversions between OpenAI Chat and OpenAI Responses are marked good. Several provider-to-Chat routes are fair. Cross-provider routes such as Anthropic-to-Gemini can be marked discouraged.

The label does not magically enforce a routing policy. It gives callers, logs, tests, and future admin tools a shared fact to work with.

It also forces an honest distinction:

A route exists != every feature survives the route
Enter fullscreen mode Exit fullscreen mode

Tool schemas, reasoning blocks, cache-related usage fields, media parts, and provider-specific options do not always have exact equivalents. Quality should therefore describe semantic confidence, not whether the converter returned valid JSON.

Streaming is a state machine

Non-streaming conversion can often be represented as:

output, usage, err := convert(input)
Enter fullscreen mode Exit fullscreen mode

Streaming cannot.

An SSE chunk usually has meaning only in relation to previous chunks. Consider converting OpenAI Chat output into Anthropic Messages events.

Anthropic expects content blocks to follow a lifecycle:

content_block_start
content_block_delta*
content_block_stop
Enter fullscreen mode Exit fullscreen mode

OpenAI tool calls may arrive as indexed argument fragments across several chunks. Multiple tool calls may be streamed in parallel. A reasoning block can switch to text. A finish reason may appear before a final usage-only chunk.

A stateless function cannot reliably determine:

  • whether a content block is currently open;
  • what type that block is;
  • which output index belongs to a tool call;
  • whether a stop event has already been emitted;
  • whether usage is still expected;
  • which terminal events must be synthesized at EOF.

Each direct streaming edge therefore needs its own state.

For a composed route, the pipeline stores one state slot per conversion step:

type StreamPipeline struct {
    Steps  []StreamEdge
    States []any
    Usage  *Usage
}

func NewPipeline(steps []StreamEdge, opts Options) *StreamPipeline {
    states := make([]any, len(steps))

    for i, step := range steps {
        if step.NewState != nil {
            states[i] = step.NewState(opts)
        }
    }

    return &StreamPipeline{
        Steps:  steps,
        States: states,
    }
}
Enter fullscreen mode Exit fullscreen mode

A stateless edge can leave its slot empty. A stateful edge owns its slot and does not share mutable conversion state with another step.

That isolation matters when a route is composed. The first edge may be assembling tool-call fragments while the second edge is tracking a completely different output event lifecycle.

One input chunk may produce many output events

Another easy mistake is giving stream conversion the signature:

func ConvertChunk(input Event) (Event, error)
Enter fullscreen mode Exit fullscreen mode

That assumes a one-to-one relationship between protocols.

Real conversions can be zero-to-one, one-to-one, or one-to-many.

One upstream chunk may require the target protocol to emit a block stop, a new block start, and a delta. Another chunk may update internal state without producing any client-visible event.

The converter should therefore return a collection:

func ConvertChunk(
    input Event,
    state any,
) ([]Event, *Usage, error)
Enter fullscreen mode Exit fullscreen mode

The pipeline then fans every emitted event into the next edge:

func (p *StreamPipeline) Push(input Event) ([]Event, error) {
    current := []Event{input}

    for i, step := range p.Steps {
        next := make([]Event, 0)

        for _, event := range current {
            emitted, usage, err := step.ConvertChunk(event, p.States[i])
            if err != nil {
                return nil, err
            }

            if usage != nil {
                p.Usage = usage
            }

            next = append(next, emitted...)
        }

        current = next
        if len(current) == 0 {
            break
        }
    }

    return current, nil
}
Enter fullscreen mode Exit fullscreen mode

This fan-out behavior makes composed streaming routes possible without hiding protocol-specific behavior inside the orchestration layer.

EOF is an event too

Some output events cannot be emitted until the upstream stream ends.

A converter may need to close an open content block, emit a final response event, or attach usage that only became known at the end. Waiting for another data chunk will never work because no such chunk is coming.

Each streaming edge can therefore define a finalizer:

func Finalize(state any) ([]Event, *Usage, error)
Enter fullscreen mode Exit fullscreen mode

Finalization of a composed route requires one extra detail: events emitted by an earlier step must still pass through every later step.

Conceptually:

for i, step := range pipeline.Steps {
    tail := step.Finalize(pipeline.States[i])
    output = append(output, runRemainingSteps(tail, i+1)...)
}
Enter fullscreen mode Exit fullscreen mode

Calling finalizers is not the same as declaring every termination successful. EOF, an explicit provider completion, a timeout, a scanner failure, and a disconnected client are different outcomes. The stream lifecycle should record which one occurred.

Test protocol invariants, not only JSON shapes

Table tests for individual fields are useful, but streaming failures usually involve event order and history.

A practical conversion test matrix should include:

  • ordinary text from start through terminal event;
  • a switch between reasoning and visible text;
  • tool arguments split across multiple chunks;
  • multiple tool calls streamed in parallel;
  • a finish reason followed by a usage-only chunk;
  • one input chunk producing several output events;
  • a chunk that intentionally produces no output;
  • EOF while a target content block is still open;
  • finalizer output passing through the remaining composed steps;
  • upstream scanner errors and timeouts;
  • client disconnection during the stream;
  • usage preservation across each route.

Registration also deserves tests. Duplicate routes, missing step IDs, discontinuous chains, and incorrect final formats should be rejected deterministically.

The strongest assertions describe observable protocol behavior:

start(index=0)
delta(index=0)
stop(index=0)
Enter fullscreen mode Exit fullscreen mode

That is stronger than asserting that a private currentIndex field became 1.

Log the route that actually ran

When a converted request fails, the provider name and HTTP status are not enough.

The gateway should retain:

  • the original request format;
  • the conversion chain;
  • the final upstream request format;
  • the selected route and its quality;
  • the stream termination reason;
  • any conversion or scanner error.

The implementation records a human-readable request conversion chain and keeps the final request format in relay state for downstream behavior. Streaming logs distinguish normal completion and EOF from outcomes such as timeout, scanner error, panic, ping failure, or client disconnection.

That turns "the stream broke" into a narrower question:

Did Responses -> Chat fail,
did Chat -> Anthropic fail,
or did the client leave before finalization?
Enter fullscreen mode Exit fullscreen mode

Observability is part of the conversion architecture, not an optional dashboard feature.

What this design does not promise

A conversion graph does not make heterogeneous APIs lossless.

It does not mean every model supports every parameter. It does not guarantee that provider-specific reasoning, caching, media, or tool semantics have an equivalent in another protocol. Composed routes can accumulate information loss even when every individual step is valid.

The graph provides something more modest and more useful: an explicit map of supported routes, declared conversion quality, testable streaming lifecycles, and enough execution metadata to diagnose failures.

Once an AI gateway supports more than two protocols, that explicitness is worth far more than another branch in a provider switch.

What this architecture looks like for Aivrae users

The conversion graph is internal infrastructure. The user-facing goal is much simpler: reduce the work required to try and operate supported models.

Aivrae provides:

  • one OpenAI-compatible base URL for supported text models;
  • model switching that usually requires changing only the model value;
  • selected native Claude Messages and Gemini generateContent formats for workflows that need them;
  • one console for API keys, prepaid credits, request logs, and model usage;
  • compatibility with SDKs and developer tools that support a custom OpenAI base URL.

New users get $1 in trial credit to create a key and make their first API calls before deciding whether to top up.

Price is another practical reason to use the gateway. As checked on July 21, 2026, Aivrae lists GPT-5.5 at $0.71 per million input tokens and $4.24 per million output tokens, about 85% below OpenAI's published standard API prices of $5 and $30. Selected GPT models are available at savings of up to 85%; pricing varies by model, so check the live table before choosing one.

Start with $1 in trial credit and compare current model pricing on Aivrae.

What has been the hardest part of multi-provider compatibility in your own projects: tools, usage accounting, or the stream lifecycle?

Source: dev.to

arrow_back Back to Tutorials