Reliability

More than one route for each model

Each model has a list of supply routes in a fixed order. We send a request to the first route. If that route fails before the answer starts, we send the same request to the next route. You do not have to change your code, and you get one answer.

We try the next route when the first route:

• does not answer, or closes the connection.

• answers with 401, 402, 403, 404, 408, 409, 425, 429 or a 5xx status.

• says that it cannot serve this model at this time.

• starts a stream, but the first event of the stream is an error.

• does not send the first part of the answer in the time limit. See the next section.

We do not try the next route when the model refuses your request for a different reason, for example a prompt that is too long or a tool schema that is not correct. The next route would refuse it too. You get a 400 error with the code upstream_invalid_request and the message from the model.

The time limit for the first byte

Some routes do not refuse a request. They keep the request and do not send an answer. Thus, each route has a time limit for the first byte of the answer:

Request	First byte	Time limit
A stream	The first event of the stream	15 seconds
A stream from a reasoning model on /v1/chat/completions	The first event of the stream	45 seconds
No stream	The response. This is the full answer.	15 seconds plus 1 second for each 20 tokens of max_tokens (maximum 300 seconds)

Some routes are slow but usually answer. These routes have a time limit of 30 seconds for a stream.

The last route in the order has a time limit of 60 seconds or more. If that route is slow, a slow answer is better than no answer. If the last route is at the end of the order because it failed (see the next sections), the route before it gets the 60 second limit. A reasoning model can think before it sends the first event of a /v1/chat/completions stream, thus it has a longer limit. The /v1/messages and /v1/responses streams start with an event at once, thus they keep the 15 second limit.

If a route does not send the first byte in the time limit, we stop the request to that route and send the request to the next route. We do not charge for the stopped request, because it did not give you a part of the answer. After the first byte gets to you, the time limit stops. We do not stop an answer after it starts.

One more try for a slow route

Sometimes all the routes fail, and one or more of them failed only because of the time limit. These routes did not refuse the request. They were slow, and their answer possibly was about to start. Thus, we send the request one more time to one of these routes, with the 60 second limit. A request without a stream also gets 1 second for each 20 tokens of max_tokens, to a maximum of 300 seconds.

We choose the route that failed the fewest times in sequence. If two routes have the same record, we choose the route that is first in the usual order. We make this try one time only for each request. If you close the connection, we do not make this try.

If the route answers, you pay for that answer one time, as usual. If it does not answer, you get the 502 error and we do not charge you.

The route order changes when a route fails

We keep a record of each route for each model. When a route fails three times in sequence, we move it to the end of the order for 5 minutes. A route that did not answer in the time limit has one failure. The one more try for a slow route does not add a second failure. If the route says that it has too much load (a 429 or 503 error with that message), we move it to the end after one failure.

After the 5 minutes, one request tries the route in its usual position:

• If the route answers, it stays in its usual position.
• If the route fails, it goes to the end of the order again. The time doubles each time, to a maximum of 30 minutes.

We do not remove a route from the order. If all the other routes fail, we try the routes at the end too. If all the routes are at the end, we try them all, and the route that failed first is first. Thus, the route order does not stop a request.

An error that comes from your request, for example a 400 for a prompt that is too long, does not change the route order.

A failed request costs no money

Before a request starts, we hold the maximum cost from your balance. If no route can answer, we give back all of that amount. The error is a 502 with this message:

{
  "error": {
    "message": "Model claude-sonnet-5-5 is not available right now. Nothing was charged. Retry in a moment.",
    "type": "api_error",
    "param": null,
    "code": "PROVIDER_ERROR"
  }
}

A 503 with the code SUPPLY_NOT_CONFIGURED means that the model has no route at this time. We do not charge for that request either. Send the request again later, or use a different model.

The OpenAI SDKs and the Anthropic SDKs retry 502 and 503 errors with backoff.

A stream that stops after it starts

After the first part of a stream gets to you, we cannot change the route. A second route would start a different answer. Thus, if the route fails after that point, the stream stops:

Endpoint	What you get
/v1/chat/completions	If the model sends an error event, the last frame has finish_reason: "error", and there is no usage frame. If the connection closes, the stream stops without data: [DONE]
/v1/messages, /v1/responses	The error event of the model, as the model sent it, or a closed connection

If the stream did not give a final token count, we count the output that we received. That count is never more than the max_tokens value of your request. We record that the count is an estimate.

If you close the connection

If you close the connection during a stream, the model can continue for a short time. We wait for 15 seconds:

• If the answer completes in that time, we charge for the token count of the model.
• If the answer does not complete, we stop it and charge for the output that the model wrote.

Thus, a closed connection does not make a request cost less.

Check the status

The status page shows if the API answers. For each model, it also shows the last hour of real requests:

Word	Meaning
Operational	95% or more of the requests got an answer, and the median time to the first byte is 10 seconds or less
Degraded	Less than 95% of the requests got an answer, or the median time to the first byte is more than 10 seconds
Down	Less than half of the requests got an answer, and three or more requests did not get an answer
No recent requests	No request used the model in the last hour

A request that a second route answered is a request that got an answer. The median time to the first byte uses streams only, because the first byte of a request without a stream is the full answer. Errors that come from your request, and 402 and 429 errors, are not in the count.

If you think that a model fails too frequently, send the x-request-id of a failed request to team@kontinentai.com.