# Build a Complete AI Gateway, End to End

This tutorial builds a production-ready AI Gateway with the v4 API, starting from
nothing: a new project, three provider assignments, a pool hierarchy with budgets and a policy
template, people with roles, apps with model routing, fallbacks, guardrails, a
semantic cache and tracing, app keys that rotate without downtime, and personal keys
for coding agents.

The tutorial has three parts. [Part 1](#part-1-set-up-the-gateway) sets up the
gateway, and both other parts need it. Then follow
[Part 2](#part-2-pools-and-apps) for software that calls the gateway, such as a
product feature, a service or an agent, [Part 3](#part-3-personal-keys) for people
who use it with their own keys, or both. Parts 2 and 3 do not depend on each other.

Each step links to the operation to call in the [API reference](/api), which has the
request and response of every operation, and says what to send and what to keep for
the next step. The [quickstart](/ai-gateway/quickstart) is the short version.

## Before you start

**Base URL and authentication.** Every operation is on `https://api.zuplo.com`.
Send `Authorization: Bearer $TOKEN`, where `$TOKEN` is the OAuth token of an account
admin. Zuplo API keys (`zpka_…`) come later.

**Conventions you will rely on.**

- A create returns `201`, a `Location` header and the whole resource. Keep its `id`.
- `PATCH` is JSON Merge Patch: omitted fields stay, `null` clears, and arrays are
  replaced as a whole. `policies`, `policyTemplate` and `budgetRules` are arrays, so
  always send the complete list.
- v4 has no `If-Match`, so the last write wins. Read a resource right before you
  change it, and send the change straight away.
- Every response property is present. A missing value is `null`.
- Errors are `application/problem+json`. A `400` lists every invalid field.
- People are identified by their Zuplo user ID (`user_…`), in fields and paths
  named `userId`. Someone new gets one the moment you add them to the account.

**Retries.** Reads, updates, deletes, adding a pool member and setting a budget
(the gateway's or a person's) are safe to retry. Other creates are not, and v4 has no
`Idempotency-Key`: a create whose response was lost may have happened, so list
before you create again.

**Concurrent edits.** v4 has no `ETag` or `If-Match`: when two people (or agents)
change the same resource, the last write wins. `PATCH` only the fields you change.

During the preview, `api.zuplo.com` validates every request against the contract,
and an operation whose backend is not ready yet answers `501 Not Implemented` with
the change it waits for in `requiredChanges`.

## What you will build

```text
acme-corp                                 account
└── acme-ai-gateway                       project, type ai-gateway
    └── production bucket                 everything below lives here
        ├── gateway                       warn at $800 a day, stop at $1,000
        ├── provider assignments          openai, anthropic, local-llama (self-hosted)
        ├── pool Engineering              $2,000 a month, policy template, Jane is admin
        │   └── pool Customer Support     $500 a month, Raj is a member
        │       └── app Support bot       routing, fallbacks, per-customer budgets,
        │                                 prompt-injection guardrail, DLP, cache, tracing
        ├── pool Platform
        │   └── app Guardrail classifier  internal only, runs the guardrail
        └── user app                      personal keys, $20 a day per person,
                                          Jane gets $50
```

Part 1 builds the project, the gateway's budget and the provider assignments. Part 2 builds the
pools, their people and the apps. Part 3 sets up the user app.

## Part 1: Set up the gateway

### 1. Create the AI Gateway project

Call [Create a project](/api/projects#create-a-project) in your account with a `name`
(`acme-ai-gateway`) and `type: ai-gateway`.

This creates the project's production environment, that environment's bucket, and
the gateway's user app, and starts deploying the gateway. No Git repository is
needed. The production environment ([List environments](/api/environments#list-environments)
with `type=production`) has the gateway's `url`. Its `status` is null for an AI
Gateway: to know when the gateway is deployed, read its newest deployment
([List deployments](/api/deployments#list-deployments) with its `environmentId`)
until it has `succeeded`. That does not guarantee every edge serves it yet; only a
request to the gateway shows what it serves.

The bucket, the only one an AI Gateway project has, is created right after the
response, so the response's `buckets` is usually still empty.
[List buckets](/api/buckets#list-buckets) in the account with the project's
`projectId` until it is there. Keep the project `id` and the bucket's `id`.
Everything the gateway holds lives in that bucket, so every operation from here on
is under `/v4/buckets/{bucketId}`.

The name is part of the gateway's URL, so it is unique in the account and cannot
change. Creating the same name again fails with `409`, which also makes a retry
after a lost response safe.

### 2. Define the custom provider

Zuplo's built-in providers (OpenAI, Anthropic, Amazon Bedrock, …) are in every
bucket: [List AI providers](/api/providers#list-ai-providers) shows them. A
self-hosted model is a custom provider, an OpenAI-compatible endpoint you define
once with [Create a custom AI provider](/api/providers#create-a-custom-ai-provider):
`name: Llama on our cluster`, the `baseUrl` of its API, and `models`:
`llama-3.1-70b` (`kind: completions`) and `bge-large-en` (`kind: embeddings`),
each with `pricing` per million input and output tokens. A provider holds no key.
Keep its `id` (`aiprov_…`).

### 3. Assign the providers

The gateway routes to a provider once it is assigned. An assignment has a `label`,
the prefix clients use to pick a model (`openai/gpt-4o-mini`), the provider's
`models` it serves, and the environment variable that holds the provider's key.
The key itself is never sent to the API: add `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`
and `LLAMA_API_KEY` as environment variables of the project's production
environment. Then create each assignment with
[Create an AI provider assignment](/api/provider-assignments#create-an-ai-provider-assignment):

| Provider | Send |
| --- | --- |
| OpenAI | `providerId: OpenAI`, `label: openai`, `apiKeyEnvironmentVariable: OPENAI_API_KEY`, and `models`: `gpt-4o-mini` |
| Anthropic | `providerId: Anthropic`, `label: anthropic`, `apiKeyEnvironmentVariable: ANTHROPIC_API_KEY`, and `models`: `claude-opus-4-1`, `claude-sonnet-4-5` and `claude-haiku-4-5` |
| The self-hosted model | the custom provider's `id` as `providerId`, `label: local-llama`, `apiKeyEnvironmentVariable: LLAMA_API_KEY`, and `models`: `llama-3.1-70b` and `bge-large-en` |

Name the models as `GET …/ai/providers/{providerId}/models` lists them, with
their prices, which drive cost budgets. A built-in provider's model can also carry
`negotiatedPricing`, your own rates, which replace the list prices. The variables
are not checked when you write: while one is missing, requests to its models fail.

Some providers take one more field: `azure-ai` takes `deploymentModels`, which maps
each Azure deployment to the catalog model it serves, `vertex-ai` takes the
`gcpProjectId` to bill, and both, like `bedrock-mantle`, take the `baseUrl` of
your endpoint.

Labels are unique in the bucket, so creating an assignment again after a lost
response fails with `409` instead of adding a second one.

Call [List AI provider assignments](/api/provider-assignments#list-ai-provider-assignments)
and collect the `ref` of every entry in `models`. A `ref` names the model by its
assignment's label, `<label>/<model>`, as clients do in their requests (`model`)
and as policies do: `openai/gpt-4o-mini`, `anthropic/claude-haiku-4-5`,
`local-llama/llama-3.1-70b` and so on. Because requests and policies name models
by it, a label cannot be changed yet.

### 4. Cap spend across the whole gateway

Call [Set the gateway's budget rules](/api/budget-rules#set-the-gateways-budget-rules)
with the list of rules as the body: one rule, `budgetBy: app`, whose `limits` warn
at $800 and stop at $1,000 a day. The list replaces the rules the gateway had, and
`[]` removes them.

A limit has a `meter` (`cost` in US dollars, `requests` or `tokens`), a `period`
(`hourly`, `daily`, `weekly` or `monthly`, in UTC), an `action` and a `value`.
`warn` records the breach and lets requests through. `block` rejects requests until
the period resets. A `warn` value must be below the `block` value for the same meter
and period.

The gateway's budget counts all of its traffic, every app and every personal key,
on top of the budgets of its pools and apps. A request is blocked when any of them
is exhausted.

## Part 2: Pools and apps

For software that calls the gateway. Each client gets an app with its own keys and
policies, in a pool that holds its budget, its policy template and the people who
manage it. This part needs only [Part 1](#part-1-set-up-the-gateway). If you only
need personal keys, skip to [Part 3](#part-3-personal-keys).

### 1. Build the pool hierarchy

A pool groups apps that share a budget, a policy template and people. Create each
pool with [Create an AI pool](/api/pools#create-an-ai-pool).

**Engineering**, a top-level pool. Send `name`, `budgetRules` that warn at $1,500
and stop at $2,000 a month, and a `policyTemplate`. The template is the list of
policies every new app in the pool starts with, in order. Each entry says whether
apps may change its options (`canEdit`) or remove it (`canRemove`):

| Template entry | Options | Apps may |
| --- | --- | --- |
| `auth` | none: callers authenticate with the app's API keys | neither change nor remove it |
| `model-filtering` | `completions.allowList`: `anthropic/claude-haiku-4-5` and `openai/gpt-4o-mini` | change it |
| `fallback-model` | `completions.fallback`: `openai/gpt-4o-mini` | change or remove it |
| `metering` | a `budgetBy: app` rule that stops each app at $250 a month | change it |

The pool's own `budgetRules` count all of its apps and sub-pools together. The
`metering` policy's budget counts each app on its own.

**Customer Support**, a sub-pool. Send `name`, `parentPoolId` (Engineering's `id`)
and `budgetRules` that stop at $500 a month. Without a `policyTemplate` of its own,
it uses Engineering's (`inheritPolicyTemplate` defaults to `true`). Its $500 applies
on top of Engineering's $2,000. `parentPoolId` cannot change later.

**Platform**, a top-level pool for internal services. Send only its `name`. Without
a template, its apps start with `auth` only.

To see the hierarchy, call [List AI pools](/api/pools#list-ai-pools), with
`parentPoolId` to list one pool's sub-pools.

### 2. Give people access

A pool member is an `admin`, who manages the pool, its apps and members, or a
`member`, who sees the pool and uses its apps. Roles are inherited by sub-pools.
Pool members are account members. Someone who is not in the account yet is added
to it by email first, which gives you their `userId` right away. The `member` role
needs role-based access control, an enterprise add-on. Without it every account
member is an `admin`, so give Raj and Priya `admin` below.

1. Find Jane's `userId` with [List account members](/api/account-members#list-account-members)
   and `email=jane@acme.com`.
2. Make her an admin of Engineering with
   [Add or update a pool member](/api/pool-members#add-or-update-a-pool-member):
   her `userId` and `role: admin`.
3. Find Raj the same way, and make him a `member` of Customer Support.
4. Priya is not in the account yet, so the lookup returns no one. Add her with
   [Add or update an account member](/api/account-members#add-or-update-an-account-member)
   and `email: priya@acme.com`. The response has her `userId` and
   `status: invited`, and Zuplo emails her a link that works for seven days.
5. Make her a `member` of Customer Support with her new `userId`, in the next
   request. She can use the pool once she accepts the email (or signs in through
   your SSO connection), and she gets access to the project with it.

[List pool members](/api/pool-members#list-pool-members) of Customer Support now
shows both: Raj with his own `role`, and Jane with the `inheritedRole` she gets from
Engineering. To change a role, add the person again with the new `role`: the
response is `200` instead of `201`, and sending it twice changes nothing. To take a
role away, call [Remove a pool member](/api/pool-members#remove-a-pool-member).

Priya is a pool member already, but she cannot use the pool until she accepts.
Until then [List account members](/api/account-members#list-account-members)
shows her with `status: invited`. To send her the email again, add her again with
the same `email`: she gets a new link and seven new days.

### 3. Create the guardrail classifier app

An app is one client of the gateway. The support bot in the next step screens every
prompt for prompt injection with a small model, and that classification runs on an
app of its own, which keeps the classifier's routing and budget in one place.

Call [Create an app](/api/apps#create-an-app) with Platform's `poolId`, the `name`
`Guardrail classifier`, and these `policies`:

| Policy | Options | Why |
| --- | --- | --- |
| `internal-only` | none | Only calls from inside the gateway are accepted, so the app needs no API key and nobody outside can use it. It replaces `auth`. |
| `model-override` | `completions.force`: `openai/gpt-4o-mini` | Every classification uses the same small model, whatever it asks for. |
| `metering` | a `budgetBy: app` rule that stops at $25 a day | Classifications cost money too. |

Keep the app's `id`. When its budget runs out, classifications fail, and the guarded
app's `onError` decides what happens to its requests.

### 4. Create the support bot

Call [Create an app](/api/apps#create-an-app) with Customer Support's `poolId`, the
`name` `Support bot`, and a full list of `policies`. They run in the order you send
them:

| # | Policy | Options |
| --- | --- | --- |
| 1 | `auth` | none, as the template requires |
| 2 | `prompt-injection` | `appId`: the classifier's `id`, `model`: `openai/gpt-4o-mini`, `execution: parallel`, `onError: block` |
| 3 | `dlp` | `rules.inbound`: `pii` with `action: mask`, and `rules.outbound`: `secret-aws` with `action: block` |
| 4 | `model-filtering` | `completions.allowList`: `anthropic/claude-haiku-4-5`, `anthropic/claude-sonnet-4-5`, `openai/gpt-4o-mini` and `local-llama/llama-3.1-70b` |
| 5 | `model-override` | `completions.default`: `anthropic/claude-haiku-4-5` |
| 6 | `fallback-model` | `completions.fallback`: `openai/gpt-4o-mini`, `completions.quotaFallback`: `local-llama/llama-3.1-70b`, `fallbackTimeoutSeconds: 20` |
| 7 | `metering` | a `budgetBy: app` rule that warns at $300 and stops at $400 a month, with a cap of 100,000 requests a day, and a `budgetBy: expression` rule on `request.headers.get("x-end-user-id")` that allows 60 requests an hour and $2 a day |
| 8 | `semantic-cache` | `semanticTolerance: 0.1`, `ttlSeconds: 86400` |
| 9 | `opik-tracing` | your Opik `workspace`, `projectName` and `apiKey` |

What each one does for a request to the support bot:

- **`auth`** checks the app's API key.
- **`prompt-injection`** classifies the newest message on the classifier app, inside
  the gateway, while the other policies run. Injection attempts are rejected before
  they reach the model. If the classifier fails, the request is rejected too.
- **`dlp`** masks personal data in what customers send, and stops a response that
  contains an AWS secret.
- **`model-filtering`** lets clients select only these four models. List the
  fallback models here too: a fallback never bypasses the filter.
- **`model-override`** uses Claude Haiku when a request names no model.
- **`fallback-model`** retries on `openai/gpt-4o-mini` after a retryable error, or
  when the model sends no response headers within 20 seconds, and switches to the
  self-hosted model once a `block` budget is exhausted.
- **`metering`** gives the whole app its budget, and each end user their own, by
  the `x-end-user-id` header the bot sends.
- **`semantic-cache`** answers a prompt close to one answered in the last day from
  the cache, without calling a provider. It runs after `metering`, so cache hits
  still count toward request limits.
- **`opik-tracing`** sends every request and response to Comet Opik. Its `apiKey`
  is write-only: it is never returned, and omitting it later keeps the stored one.

The app's budget applies together with Customer Support's, Engineering's and the
gateway's. A policy with `enabled: false` keeps its place and options without
running.

The response includes the app's `baseUrl`, where its clients send requests. Keep it
with the `id`.

#### More policies

Use these where they fit. Each policy's options are in the API reference.

| Policy | What it does | Requires |
| --- | --- | --- |
| [`load-balancing`](/api/~schemas#ailoadbalancingpolicyinput) | Spreads requests across several models, `round-robin` or `sticky` per user, conversation or expression. | `models` |
| [`smart-router`](/api/~schemas#aismartrouterpolicyinput) | Classifies each prompt on another app and routes it to a model by its complexity. | `classifier` (an app and its model) and `modelsByComplexity` (`low`, `medium`, `high`) |
| [`akamai-firewall`](/api/~schemas#aiakamaifirewallpolicyinput) | Checks requests and responses with Akamai's AI firewall. | `configurationId` and `apiKey` |
| [`galileo-tracing`](/api/~schemas#aigalileotracingpolicyinput) | Sends requests and responses to Galileo. | `projectId`, `logStreamId` and `apiKey` |
| [`custom`](/api/~schemas#aicustompolicyinput) | Runs a policy declared in the project's `config/policies.json`, such as your own code. | its `name` |

### 5. Issue app keys and call the gateway

Creating an app does not create a key. An app's keys belong to its consumers, one
for each client or environment, which are consumers in the bucket linked to the
app. Call [Create a consumer](/api/consumers#create-a-consumer) for the support
bot with a `name`, such as `support-bot-production`, a `description`, and
`include[]=apiKey` in the query. For a key that should stop working on a date, set
its `expiresOn` with [Update an API key](/api/keys#update-an-api-key). The
classifier gets no consumer: it runs `internal-only`.

The response has the consumer and its first key, with the key's `secret`. This is
the only time you see it. Store it in your secret manager now. Every later read
shows only `maskedKey`. A retry after a lost response fails with `409`, because
the name is taken: roll the consumer's key to get a new secret.

Then [link the consumer to the app](/api/consumers#link-a-consumer-to-an-app)
with a `POST` of its `consumerId` to `…/ai/apps/{appId}/consumers`. From then on
the app lists it in its `consumers` when you ask for them with
`include[]=consumers`.

The app's clients use their usual OpenAI or Anthropic SDK with the app's `baseUrl`
as the base URL and the key as the API key, for example
`POST {baseUrl}/v1/chat/completions`. The support bot also sends the end user's ID in
`x-end-user-id`, for its per-customer budget. Every request draws from the budgets of
the app, its pools and the gateway.

### 6. Rotate an app key without downtime

Rotating is creating the replacement. Call
[Roll a consumer's API key](/api/keys#roll-a-consumers-api-key) on the
production consumer with `previousKeyGracePeriodSeconds=86400` in the query, so the
current key keeps working for a day while clients switch over. Without it, the current key
stops working right away. Other consumers of the app, such as staging, are not
affected.

[List a consumer's API keys](/api/keys#list-a-consumers-api-keys) now shows
both: the new key, and the old one with an `expiresOn` a day from now. Once every
client uses the new key, end the grace period with
[Revoke an API key](/api/keys#revoke-an-api-key) on the old one. A repeated
revoke returns `404`. Treat that as success.

To change a key's `description` or `expiresOn`, call
[Update an API key](/api/keys#update-an-api-key).

## Part 3: Personal keys

For people. The user app gives everyone in the account their own API key and their
own budget, for Claude Code, Cursor, scripts and notebooks. This part needs only
[Part 1](#part-1-set-up-the-gateway).

### 1. Set everyone's models and budget

Every gateway has exactly one user app, created with the project. Find its `id` and
`baseUrl` with [List user apps](/api/user-apps#list-user-apps).

Call [Update a user app](/api/user-apps#update-a-user-app) with its complete
`policies`:

| Policy | Options |
| --- | --- |
| `auth` | none |
| `model-filtering` | `completions.blockList`: `anthropic/claude-opus-4-1`, so everyone can use every other model on the gateway |
| `model-override` | `completions.default`: `anthropic/claude-sonnet-4-5` |
| `metering` | a `budgetBy: expression` rule on `request.user.sub`, so each person gets their own budget: warn at $15 and stop at $20 a day, and stop at $300 a month |

### 2. Create personal keys

Each person creates their own consumer and its first key with
[Create a personal consumer](/api/consumers#create-a-personal-consumer), with their
own `userId` in the path and `include[]=apiKey`, signed in with their own OAuth
token, for example from the portal or the Zuplo CLI. The consumer is theirs. Jane
points Claude Code at the user app's `baseUrl`:

```bash
export ANTHROPIC_BASE_URL="https://acme-ai-gateway-main-b8e2f1a.zuplo.app/u/UsTroIp2e0K8mWYGj1GT8U13"
export ANTHROPIC_AUTH_TOKEN="zpka_…"
claude
```

Personal keys work only on the user app's URL. A person can have several
consumers, for example one per laptop, and rolls each one like an app's, with
`previousKeyGracePeriodSeconds`. Rolling the laptop's key leaves the desktop's
alone.

A user app admin, someone who can edit the user app, can issue a key for each
person instead: one [Create a personal consumer](/api/consumers#create-a-personal-consumer)
per person, with their `userId` in the path. Find everyone's `userId` with
[List account members](/api/account-members#list-account-members), or one person's
with their `email`. The consumer belongs to the person. The admin sees the key's
secret only in the response, and the person reads it whenever they need it: they
find the key with [List a consumer's API keys](/api/keys#list-a-consumers-api-keys)
and read it with [Get an API key's secret](/api/keys#get-an-api-keys-secret).

Everyone lists their own consumers with
[List a person's consumers](/api/consumers#list-a-persons-consumers) and their own
`userId`, and can roll, update or revoke them. Admins can do the same for anyone.

### 3. Raise one person's budget, and watch usage

Jane leads a migration and needs $50 a day. Find her `userId` with
[List account members](/api/account-members#list-account-members) and
`email=jane@acme.com`, then call
[Set a person's budget](/api/user-apps#set-a-persons-budget) with it in the path,
and `limits`: one `cost` limit, `period: daily`, `action: block`, `value: 50`. Her
budget replaces only `cost` limits with action `block` from the per-person rule.
Her other limits keep the defaults, and sending the same budget again changes
nothing.

[List the people in a user app](/api/user-apps#list-the-people-in-a-user-app) with
`hasBudgetOverride=true` shows everyone with limits of their own, and each
person's `budget` shows what those limits are. Setting someone's budget to
`limits: []` puts them back on the defaults.

Jane checks her own usage with [Get a person's usage](/api/user-apps#get-a-persons-usage)
and her own `userId`. Each of her limits shows its `limit`, what she has `used`,
what is `remaining`, when it `resetsOn`, and whether it comes from the default rule
or her budget. User app admins pass anyone's `userId`. Usage is aggregated asynchronously, so
`measuredOn` can lag by a few seconds.

## Not in v4 yet

These have no v4 operation yet. Each is tracked in
[Decisions and Open Questions](/guidelines/decisions).

- **Accepting an invitation.** Accepting happens in the Zuplo portal
  (question 5).
- **Usage of apps and pools**, analytics and request logs. Only personal usage is in
  v4 so far (phase 2).
- **Audit logs** (phase 2).
- **Waiting for configuration changes.** A provider assignment change creates a deployment of
  the production environment: wait for the newest one. Changes that apply through
  the configuration cache (pools, apps, policies, budgets, keys) are not tracked,
  and apply within about a minute (question 7).
- **Environments and source control**: other environments than production, and
  connecting a Git repository to the gateway.