Suppose you have two AI applications. One answers questions about published product documentation. The other searches internal documents containing customer information. You might allow the first to use a public model, while keeping the second on your own inference server.
That decision needs to survive a busy GPU, a model upgrade, and a developer changing a client setting. Putting the permitted routes in one gateway gives you a place to enforce it.
Where the request goes
With Anchras, applications connect to the AI Gateway. It routes requests to a private runtime, such as Ollama or vLLM, or to an approved public provider. Each application can use its own scoped key while the gateway holds the provider configuration.
The gateway and control plane are hosted by Anchras in the EU. A local model runs on infrastructure you control, but its requests still pass through that hosted gateway. If your requirement is that prompts never leave your premises, this request path does not meet it. Check gateway processing and logging alongside the model's location; our data-access guide covers the broader review.
Follow a request.
See how the permitted model route changes with the data.
AI GatewayHosted in the EU
This request stays on the approved local route. If that model is unavailable, the request is rejected.
Set permissions per application
Give the documentation assistant access to the specific public models you have approved. Give the internal search application access only to its local model route. Keep provider credentials out of the clients, and restrict direct access to the inference server so that clients cannot bypass the gateway.
The data classification should come from a setting you control, such as the application or scoped key. A model inspecting the prompt should not be able to lower a classification already assigned to it.
Here is an example policy to agree and configure for a deployment. These are proposed rules, not a statement of the platform's default settings.
| Data class | Route | If the model is unavailable |
|---|---|---|
| Public | An approved local or public model | Try another approved model |
| Internal | Local; external only for specifically approved uses | Use only an alternative approved for this use |
| Confidential | Local model | Reject the request |
| Restricted | Named local model with limited access | Reject the request |
For the internal search application in this example, a failed local request stays failed. Sending it to a public model would change who processes the customer information. An availability setting should not make that decision.
Check the runtime with your actual workload
Anchras offers Ollama and vLLM as catalog applications for deployment on your Kubernetes cluster. Choose a model that supports the task and hardware, then test the context lengths and number of simultaneous requests you expect. A model that fits in GPU memory for a short conversation may leave too little room for longer contexts and concurrent users.
Both runtimes offer OpenAI-compatible interfaces, but compatibility has limits. Check the Ollama API documentation or vLLM server documentation for the features your client uses. Test streaming, tool calls, and structured output where your application depends on them. Changing a base URL is only the connection step.
Test the refused requests too
Start with synthetic documents and two application keys. Confirm that the documentation assistant can reach its approved public model. Then try the same route with the internal search key and check that it is refused.
Stop the local runtime and repeat the internal request. Check the gateway record to verify that no external provider was selected. Also try reaching the runtime directly from a client that should have access only through the gateway.
Before adding real documents, decide what the application, gateway, and runtime may log, who can read those logs, and when they are deleted. Test latency and queueing under load, and record a rollback procedure for model updates. Those checks give you a useful first deployment to expand from; the hybrid AI blueprint shows the surrounding infrastructure.