Copy page

AI Gateway

Build a Complete AI Gateway, End to End

This tutorial builds a production-ready AI Gateway with the v4 API, starting from nothing: a new project, three provider assignments, a pool hierarchy with budgets and a policy template, people with roles, apps with model routing, fallbacks, guardrails, a semantic cache and tracing, app keys that rotate without downtime, and personal keys for coding agents.

The tutorial has three parts. Part 1 sets up the gateway, and both other parts need it. Then follow Part 2 for software that calls the gateway, such as a product feature, a service or an agent, Part 3 for people who use it with their own keys, or both. Parts 2 and 3 do not depend on each other.

Each step links to the operation to call in the API reference, which has the request and response of every operation, and says what to send and what to keep for the next step. The quickstart is the short version.

Before you start

Base URL and authentication. Every operation is on https://api.zuplo.com. Send Authorization: Bearer $TOKEN, where $TOKEN is the OAuth token of an account admin. Zuplo API keys (zpka_…) come later.

Conventions you will rely on.

• A create returns 201, a Location header and the whole resource. Keep its id.
• PATCH is JSON Merge Patch: omitted fields stay, null clears, and arrays are replaced as a whole. policies, policyTemplate and budgetRules are arrays, so always send the complete list.
• v4 has no If-Match, so the last write wins. Read a resource right before you change it, and send the change straight away.
• Every response property is present. A missing value is null.
• Errors are application/problem+json. A 400 lists every invalid field.
• People are identified by their Zuplo user ID (user_…), in fields and paths named userId. Someone new gets one the moment you add them to the account.

Retries. Reads, updates, deletes, adding a pool member and setting a budget (the gateway's or a person's) are safe to retry. Other creates are not, and v4 has no Idempotency-Key: a create whose response was lost may have happened, so list before you create again.

Concurrent edits. v4 has no ETag or If-Match: when two people (or agents) change the same resource, the last write wins. PATCH only the fields you change.

During the preview, api.zuplo.com validates every request against the contract, and an operation whose backend is not ready yet answers 501 Not Implemented with the change it waits for in requiredChanges.

What you will build

Code
acme-corp                                 account
└── acme-ai-gateway                       project, type ai-gateway
    └── production bucket                 everything below lives here
        ├── gateway                       warn at $800 a day, stop at $1,000
        ├── provider assignments          openai, anthropic, local-llama (self-hosted)
        ├── pool Engineering              $2,000 a month, policy template, Jane is admin
        │   └── pool Customer Support     $500 a month, Raj is a member
        │       └── app Support bot       routing, fallbacks, per-customer budgets,
        │                                 prompt-injection guardrail, DLP, cache, tracing
        ├── pool Platform
        │   └── app Guardrail classifier  internal only, runs the guardrail
        └── user app                      personal keys, $20 a day per person,
                                          Jane gets $50

Part 1 builds the project, the gateway's budget and the provider assignments. Part 2 builds the pools, their people and the apps. Part 3 sets up the user app.

Part 1: Set up the gateway

1. Create the AI Gateway project

Call Create a project in your account with a name (acme-ai-gateway) and type: ai-gateway.

This creates the project's production environment, that environment's bucket, and the gateway's user app, and starts deploying the gateway. No Git repository is needed. The production environment (List environments with type=production) has the gateway's url. Its status is null for an AI Gateway: to know when the gateway is deployed, read its newest deployment (List deployments with its environmentId) until it has succeeded. That does not guarantee every edge serves it yet; only a request to the gateway shows what it serves.

The bucket, the only one an AI Gateway project has, is created right after the response, so the response's buckets is usually still empty. List buckets in the account with the project's projectId until it is there. Keep the project id and the bucket's id. Everything the gateway holds lives in that bucket, so every operation from here on is under /v4/buckets/{bucketId}.

The name is part of the gateway's URL, so it is unique in the account and cannot change. Creating the same name again fails with 409, which also makes a retry after a lost response safe.

2. Define the custom provider

Zuplo's built-in providers (OpenAI, Anthropic, Amazon Bedrock, …) are in every bucket: List AI providers shows them. A self-hosted model is a custom provider, an OpenAI-compatible endpoint you define once with Create a custom AI provider: name: Llama on our cluster, the baseUrl of its API, and models: llama-3.1-70b (kind: completions) and bge-large-en (kind: embeddings), each with pricing per million input and output tokens. A provider holds no key. Keep its id (aiprov_…).

3. Assign the providers

The gateway routes to a provider once it is assigned. An assignment has a label, the prefix clients use to pick a model (openai/gpt-4o-mini), the provider's models it serves, and the environment variable that holds the provider's key. The key itself is never sent to the API: add OPENAI_API_KEY, ANTHROPIC_API_KEY and LLAMA_API_KEY as environment variables of the project's production environment. Then create each assignment with Create an AI provider assignment:

Name the models as GET …/ai/providers/{providerId}/models lists them, with their prices, which drive cost budgets. A built-in provider's model can also carry negotiatedPricing, your own rates, which replace the list prices. The variables are not checked when you write: while one is missing, requests to its models fail.

Some providers take one more field: azure-ai takes deploymentModels, which maps each Azure deployment to the catalog model it serves, vertex-ai takes the gcpProjectId to bill, and both, like bedrock-mantle, take the baseUrl of your endpoint.

Labels are unique in the bucket, so creating an assignment again after a lost response fails with 409 instead of adding a second one.

Call List AI provider assignments and collect the ref of every entry in models. A ref names the model by its assignment's label, <label>/<model>, as clients do in their requests (model) and as policies do: openai/gpt-4o-mini, anthropic/claude-haiku-4-5, local-llama/llama-3.1-70b and so on. Because requests and policies name models by it, a label cannot be changed yet.

4. Cap spend across the whole gateway

Call Set the gateway's budget rules with the list of rules as the body: one rule, budgetBy: app, whose limits warn at $800 and stop at $1,000 a day. The list replaces the rules the gateway had, and [] removes them.

A limit has a meter (cost in US dollars, requests or tokens), a period (hourly, daily, weekly or monthly, in UTC), an action and a value. warn records the breach and lets requests through. block rejects requests until the period resets. A warn value must be below the block value for the same meter and period.

The gateway's budget counts all of its traffic, every app and every personal key, on top of the budgets of its pools and apps. A request is blocked when any of them is exhausted.

Part 2: Pools and apps

For software that calls the gateway. Each client gets an app with its own keys and policies, in a pool that holds its budget, its policy template and the people who manage it. This part needs only Part 1. If you only need personal keys, skip to Part 3.

1. Build the pool hierarchy

A pool groups apps that share a budget, a policy template and people. Create each pool with Create an AI pool.

Engineering, a top-level pool. Send name, budgetRules that warn at $1,500 and stop at $2,000 a month, and a policyTemplate. The template is the list of policies every new app in the pool starts with, in order. Each entry says whether apps may change its options (canEdit) or remove it (canRemove):

The pool's own budgetRules count all of its apps and sub-pools together. The metering policy's budget counts each app on its own.

Customer Support, a sub-pool. Send name, parentPoolId (Engineering's id) and budgetRules that stop at $500 a month. Without a policyTemplate of its own, it uses Engineering's (inheritPolicyTemplate defaults to true). Its $500 applies on top of Engineering's $2,000. parentPoolId cannot change later.

Platform, a top-level pool for internal services. Send only its name. Without a template, its apps start with auth only.

To see the hierarchy, call List AI pools, with parentPoolId to list one pool's sub-pools.

2. Give people access

A pool member is an admin, who manages the pool, its apps and members, or a member, who sees the pool and uses its apps. Roles are inherited by sub-pools. Pool members are account members. Someone who is not in the account yet is added to it by email first, which gives you their userId right away. The member role needs role-based access control, an enterprise add-on. Without it every account member is an admin, so give Raj and Priya admin below.

1. Find Jane's userId with List account members and [email protected].
2. Make her an admin of Engineering with Add or update a pool member: her userId and role: admin.
3. Find Raj the same way, and make him a member of Customer Support.
4. Priya is not in the account yet, so the lookup returns no one. Add her with Add or update an account member and email: [email protected]. The response has her userId and status: invited, and Zuplo emails her a link that works for seven days.
5. Make her a member of Customer Support with her new userId, in the next request. She can use the pool once she accepts the email (or signs in through your SSO connection), and she gets access to the project with it.

List pool members of Customer Support now shows both: Raj with his own role, and Jane with the inheritedRole she gets from Engineering. To change a role, add the person again with the new role: the response is 200 instead of 201, and sending it twice changes nothing. To take a role away, call Remove a pool member.

Priya is a pool member already, but she cannot use the pool until she accepts. Until then List account members shows her with status: invited. To send her the email again, add her again with the same email: she gets a new link and seven new days.

3. Create the guardrail classifier app

An app is one client of the gateway. The support bot in the next step screens every prompt for prompt injection with a small model, and that classification runs on an app of its own, which keeps the classifier's routing and budget in one place.

Call Create an app with Platform's poolId, the name Guardrail classifier, and these policies:

Keep the app's id. When its budget runs out, classifications fail, and the guarded app's onError decides what happens to its requests.

4. Create the support bot

Call Create an app with Customer Support's poolId, the name Support bot, and a full list of policies. They run in the order you send them:

What each one does for a request to the support bot:

• auth checks the app's API key.
• prompt-injection classifies the newest message on the classifier app, inside the gateway, while the other policies run. Injection attempts are rejected before they reach the model. If the classifier fails, the request is rejected too.
• dlp masks personal data in what customers send, and stops a response that contains an AWS secret.
• model-filtering lets clients select only these four models. List the fallback models here too: a fallback never bypasses the filter.
• model-override uses Claude Haiku when a request names no model.
• fallback-model retries on openai/gpt-4o-mini after a retryable error, or when the model sends no response headers within 20 seconds, and switches to the self-hosted model once a block budget is exhausted.
• metering gives the whole app its budget, and each end user their own, by the x-end-user-id header the bot sends.
• semantic-cache answers a prompt close to one answered in the last day from the cache, without calling a provider. It runs after metering, so cache hits still count toward request limits.
• opik-tracing sends every request and response to Comet Opik. Its apiKey is write-only: it is never returned, and omitting it later keeps the stored one.

The app's budget applies together with Customer Support's, Engineering's and the gateway's. A policy with enabled: false keeps its place and options without running.

The response includes the app's baseUrl, where its clients send requests. Keep it with the id.

More policies

Use these where they fit. Each policy's options are in the API reference.

5. Issue app keys and call the gateway

Creating an app does not create a key. An app's keys belong to its consumers, one for each client or environment, which are consumers in the bucket linked to the app. Call Create a consumer for the support bot with a name, such as support-bot-production, a description, and include[]=apiKey in the query. For a key that should stop working on a date, set its expiresOn with Update an API key. The classifier gets no consumer: it runs internal-only.

The response has the consumer and its first key, with the key's secret. This is the only time you see it. Store it in your secret manager now. Every later read shows only maskedKey. A retry after a lost response fails with 409, because the name is taken: roll the consumer's key to get a new secret.

Then link the consumer to the app with a POST of its consumerId to …/ai/apps/{appId}/consumers. From then on the app lists it in its consumers when you ask for them with include[]=consumers.

The app's clients use their usual OpenAI or Anthropic SDK with the app's baseUrl as the base URL and the key as the API key, for example POST {baseUrl}/v1/chat/completions. The support bot also sends the end user's ID in x-end-user-id, for its per-customer budget. Every request draws from the budgets of the app, its pools and the gateway.

6. Rotate an app key without downtime

Rotating is creating the replacement. Call Roll a consumer's API key on the production consumer with previousKeyGracePeriodSeconds=86400 in the query, so the current key keeps working for a day while clients switch over. Without it, the current key stops working right away. Other consumers of the app, such as staging, are not affected.

List a consumer's API keys now shows both: the new key, and the old one with an expiresOn a day from now. Once every client uses the new key, end the grace period with Revoke an API key on the old one. A repeated revoke returns 404. Treat that as success.

To change a key's description or expiresOn, call Update an API key.

Part 3: Personal keys

For people. The user app gives everyone in the account their own API key and their own budget, for Claude Code, Cursor, scripts and notebooks. This part needs only Part 1.

1. Set everyone's models and budget

Every gateway has exactly one user app, created with the project. Find its id and baseUrl with List user apps.

Call Update a user app with its complete policies:

2. Create personal keys

Each person creates their own consumer and its first key with Create a personal consumer, with their own userId in the path and include[]=apiKey, signed in with their own OAuth token, for example from the portal or the Zuplo CLI. The consumer is theirs. Jane points Claude Code at the user app's baseUrl:

Code
export ANTHROPIC_BASE_URL="https://acme-ai-gateway-main-b8e2f1a.zuplo.app/u/UsTroIp2e0K8mWYGj1GT8U13"
export ANTHROPIC_AUTH_TOKEN="zpka_…"
claude

Personal keys work only on the user app's URL. A person can have several consumers, for example one per laptop, and rolls each one like an app's, with previousKeyGracePeriodSeconds. Rolling the laptop's key leaves the desktop's alone.

A user app admin, someone who can edit the user app, can issue a key for each person instead: one Create a personal consumer per person, with their userId in the path. Find everyone's userId with List account members, or one person's with their email. The consumer belongs to the person. The admin sees the key's secret only in the response, and the person reads it whenever they need it: they find the key with List a consumer's API keys and read it with Get an API key's secret.

Everyone lists their own consumers with List a person's consumers and their own userId, and can roll, update or revoke them. Admins can do the same for anyone.

3. Raise one person's budget, and watch usage

Jane leads a migration and needs $50 a day. Find her userId with List account members and [email protected], then call Set a person's budget with it in the path, and limits: one cost limit, period: daily, action: block, value: 50. Her budget replaces only cost limits with action block from the per-person rule. Her other limits keep the defaults, and sending the same budget again changes nothing.

List the people in a user app with hasBudgetOverride=true shows everyone with limits of their own, and each person's budget shows what those limits are. Setting someone's budget to limits: [] puts them back on the defaults.

Jane checks her own usage with Get a person's usage and her own userId. Each of her limits shows its limit, what she has used, what is remaining, when it resetsOn, and whether it comes from the default rule or her budget. User app admins pass anyone's userId. Usage is aggregated asynchronously, so measuredOn can lag by a few seconds.

Not in v4 yet

These have no v4 operation yet. Each is tracked in Decisions and Open Questions.

• Accepting an invitation. Accepting happens in the Zuplo portal (question 5).
• Usage of apps and pools, analytics and request logs. Only personal usage is in v4 so far (phase 2).
• Audit logs (phase 2).
• Waiting for configuration changes. A provider assignment change creates a deployment of the production environment: wait for the newest one. Changes that apply through the configuration cache (pools, apps, policies, budgets, keys) are not tracked, and apply within about a minute (question 7).
• Environments and source control: other environments than production, and connecting a Git repository to the gateway.