Build a Provider Policy Linter for AI API Gateways

python dev.to

Multi-provider AI gateways fail in oddly quiet ways.

The SDK still compiles. The chat completion call still returns JSON. The dashboard still shows a model name that looks familiar. But a small routing assumption has moved underneath you: a model alias was renamed, a cache rule changed, a route only supports the OpenAI-compatible endpoint and not the Responses API, or the pricing file in your repo is older than the public rate card.

That is painful for teams in the US, UK, EU, Japan, Singapore, and other Tier 1/2 markets because production buyers usually do not ask, "Can we call this model?" They ask:

  • Which route did this SDK actually target?
  • Which dated pricing source did the estimate use?
  • Does this endpoint support the request shape we are about to ship?
  • Can finance reproduce the budget row after the run?
  • Can support explain a failed migration without reading the whole codebase?

A provider policy linter is a small CI gate that answers those questions before a release goes out.

It does not replace live monitoring. It does not promise that an upstream provider will never change behavior. It simply blocks SDK changes that depend on unstated model, endpoint, pricing, or billing-group assumptions.

For this article I rechecked AIWave's public pricing endpoints on September 14, 2026. The static pricing endpoint at https://aiwave.live/api/v1/pricing reported a source-dated 64-model snapshot with checked and updated_at set to September 10, 2026, plus a pricing version. The dynamic pricing endpoint at https://aiwave.live/api/pricing reported 64 rows and current group ratios of default: 1 and vip: 0.9. Treat those as source facts with dates, not evergreen copy.

What the linter should catch

Start with checks that are boring enough to run on every pull request.

The first check is source freshness. If an SDK example estimates cost, the estimate should include the pricing source URL, the source date, and the version or content fingerprint. A pull request that adds a price row without those fields should fail.

The second check is route existence. If the SDK config references deepseek-v4-pro, glm-5, kimi-k2.5, or any other model ID, the linter should verify that the model exists in the current route table used by the gateway. If your source has both public catalog rows and internal-only route rows, only lint against the public or approved deployment surface for that package.

The third check is endpoint compatibility. A model that works for /v1/chat/completions may not be verified for another API shape. Do not infer capability from a model name. Require an explicit supported_endpoint_types field, a local override, or a dated test receipt.

The fourth check is billing-group clarity. AIWave's current public base price is the default group. A VIP key can use a 0.9 multiplier where the application and account configuration allow it. Your docs and SDK examples should not silently mix those two views. The linter should force every example to declare whether it is showing a base rate, an effective account rate, or a runtime estimate.

The fifth check is fallback policy. If a route is removed, disabled, or missing a required endpoint type, the SDK should not silently pick a nearby model. A fallback may be acceptable, but only if the code names the fallback, records the reason, and preserves the original requested model in a receipt.

A compact policy file

Keep the file close to the SDK code. It should be reviewed like source, not edited by a hidden spreadsheet export.

{"policy_version":"2026-09-14","pricing_source":"https://aiwave.live/api/v1/pricing","pricing_checked":"2026-09-10","allowed_endpoint_types":["openai"],"billing_view":"base_default_rate","routes":[{"model":"deepseek-v4-pro","required_endpoint_type":"openai","allow_fallback":false},{"model":"glm-5","required_endpoint_type":"openai","allow_fallback":true,"fallback_model":"glm-4.7"}]}
Enter fullscreen mode Exit fullscreen mode

That policy is intentionally small. It avoids live secrets, request bodies, user IDs, prompts, and response content. It is safe to put in a public SDK repository because it records operational assumptions, not credentials or customer data.

A second file can hold the fetched route facts. Generate it during CI or a scheduled refresh job:

{"fetched_at":"2026-09-14T13:20:00Z","pricing_version":"8c7a0c0b30661ccbc13d142cb54d1e4ae445fe774b2c6fa501080db97c7a3e56","checked":"2026-09-10","models":{"deepseek-v4-pro":{"provider":"DeepSeek","unit":"per_1m_text_tokens","effective_date":"2026-08-27"},"glm-5":{"provider":"GLM","unit":"per_1m_text_tokens","effective_date":"2026-08-27"}}}
Enter fullscreen mode Exit fullscreen mode

Notice what is absent: no sample API key, no customer route, no request transcript, and no claim that the table proves runtime reliability.

Implement the check

Here is a minimal Python linter. It reads the policy, fetches the public pricing JSON, and emits actionable failures. In a real SDK, you would add retries, pin TLS behavior through your normal HTTP client, and write the result as a build artifact.

import json
import os
import sys
import urllib.request
from dataclasses import dataclass
from pathlib import Path


@dataclass
class Failure:
    code: str
    message: str


def load_json(path: str) -> dict:
    return json.loads(Path(path).read_text(encoding="utf-8"))


def fetch_json(url: str) -> dict:
    req = urllib.request.Request(
        url,
        headers={"User-Agent": "provider-policy-linter/1.0"},
    )
    with urllib.request.urlopen(req, timeout=20) as response:
        return json.loads(response.read().decode("utf-8"))


def model_index(pricing: dict) -> dict:
    rows = pricing.get("models") or pricing.get("data") or []
    index = {}
    for row in rows:
        model_id = row.get("id") or row.get("model_name")
        if model_id:
            index[model_id] = row
    return index


def check_policy(policy: dict, pricing: dict) -> list[Failure]:
    failures: list[Failure] = []
    models = model_index(pricing)

    checked = pricing.get("checked") or pricing.get("updated_at")
    expected_checked = policy.get("pricing_checked")
    if expected_checked and checked and expected_checked != checked:
        failures.append(
            Failure(
                "pricing_source_date_changed",
                f"policy expects pricing date {expected_checked}, live source reports {checked}",
            )
        )

    if policy.get("billing_view") not in {"base_default_rate", "effective_account_rate"}:
        failures.append(
            Failure(
                "billing_view_missing",
                "policy must declare whether examples use base or effective rates",
            )
        )

    for route in policy.get("routes", []):
        model = route.get("model")
        if not model:
            failures.append(Failure("route_model_missing", "route entry is missing model"))
            continue

        fact = models.get(model)
        if not fact:
            failures.append(Failure("route_not_found", f"{model} is not in the pricing source"))
            continue

        required_endpoint = route.get("required_endpoint_type")
        supported = set(fact.get("supported_endpoint_types") or policy.get("allowed_endpoint_types") or [])
        if required_endpoint and required_endpoint not in supported:
            failures.append(
                Failure(
                    "endpoint_not_verified",
                    f"{model} does not declare support for {required_endpoint}",
                )
            )

        if route.get("allow_fallback") and not route.get("fallback_model"):
            failures.append(
                Failure(
                    "fallback_model_missing",
                    f"{model} allows fallback but does not name a fallback model",
                )
            )

    return failures


def main() -> int:
    policy_path = os.environ.get("POLICY_FILE", "provider-policy.json")
    policy = load_json(policy_path)
    pricing = fetch_json(policy["pricing_source"])
    failures = check_policy(policy, pricing)

    for failure in failures:
        print(f"{failure.code}: {failure.message}", file=sys.stderr)

    return 1 if failures else 0


if __name__ == "__main__":
    raise SystemExit(main())
Enter fullscreen mode Exit fullscreen mode

This is deliberately stricter than a smoke test. A smoke test tells you that one request worked. A policy linter tells you that the repository's documented assumptions still match the source of record.

Add a receipt for every lint run

The useful artifact is not the console output. It is a receipt that support and finance can inspect later.

{"linted_at":"2026-09-14T13:30:00Z","policy_version":"2026-09-14","pricing_source":"https://aiwave.live/api/v1/pricing","pricing_checked":"2026-09-10","pricing_version":"8c7a0c0b30661ccbc13d142cb54d1e4ae445fe774b2c6fa501080db97c7a3e56","routes_checked":2,"failures":[]}
Enter fullscreen mode Exit fullscreen mode

Store that receipt with your CI artifacts. If the release later causes confusion, you can answer a precise question: "Which price source and route table did the SDK believe when this package shipped?"

The receipt also keeps your docs honest. If a markdown example claims a dated base rate, the receipt should point to the same date. If the live source changes, the next run fails and asks a human to decide whether to update docs, change examples, or pin the old behavior in a migration note.

Where to put the linter in your release path

Run the policy linter in three places.

First, run it on pull requests that change examples, model aliases, route configs, pricing tables, or SDK defaults. Make it fast and deterministic. The goal is to stop missing metadata, not to run a full model benchmark.

Second, run it in nightly CI against the current public pricing or route source. Nightly failures should open a review item, not auto-change code. Pricing and capability updates deserve a human review because a provider page update may be a layout change, a new optional mode, or a real contract change.

Third, run it before publishing external docs. Public articles, README snippets, and marketplace docs often outlive the SDK release that created them. A small linter can block stale rate-card dates, unsupported route names, and old billing-group language before those pages spread.

Here is the release rule I like:

No SDK release may publish a model example unless it has:
1. a route name that exists in the approved source,
2. a dated pricing source or an explicit "no price claim" marker,
3. an endpoint compatibility statement,
4. a declared billing view,
5. and a receipt from the latest policy-lint run.
Enter fullscreen mode Exit fullscreen mode

That rule is not glamorous. It is what prevents a demo from becoming an accidental support burden.

Keep the policy narrow

Do not turn the policy linter into a secret scanner, a benchmark suite, and a billing simulator all at once. Those are separate tools.

The linter should answer: "Are our declared assumptions still valid enough to publish this SDK change?"

It should not answer: "Which provider is globally best?" That question usually depends on workload shape, privacy posture, regional requirements, cache behavior, and budget controls.

It should not answer: "What will the exact invoice be?" Runtime token counts, cache hits, retries, selected account group, and provider-side behavior all matter. The linter can enforce dated sources and billing-view clarity, but the final bill still needs request-level receipts.

It should not infer private business metrics. Avoid customer counts, revenue, internal usage totals, or support details. The public artifact should be a technical contract, not a growth report.

A practical migration pattern

If you already have many examples, do not rewrite everything in one sprint.

Start by adding pricing_source, pricing_checked, and billing_view to the top ten SDK examples that get the most traffic. Then add required_endpoint_type to each model route used by those examples. Then create a CI job that warns, not fails.

After one week of warnings, flip the job to blocking for changed files only. That gives maintainers a way to improve the surface without breaking every historical sample in one day.

Finally, add the receipt artifact to releases. This is the part buyers appreciate during procurement and incident review. When someone asks how an SDK estimate was produced, you can point to a dated, reproducible policy run instead of a half-remembered spreadsheet.

For an AI API gateway, that is the real value of a linter. It turns model access into a reviewed contract: source-dated, endpoint-aware, billing-view explicit, and small enough that engineers will actually keep it alive.

Source: dev.to

arrow_back Back to Tutorials