A private, GDPR-compliant AI hub for a whole team.

The team wanted GPT-4. The legal team wanted GDPR. The CFO wanted it affordable. Open WebUI + LiteLLM + Azure OpenAI made all three happy.

Flügger Group A/S - internal 2023–2024

Brief

A way to use frontier AI models inside a corporate environment - without leaking data, and without the per-seat costs that make a roll-out economically painful. Built on Open WebUI, LiteLLM and Azure OpenAI. Significant cost savings, healthier compliance posture, and a happier team.

Overview

When ChatGPT launched, the headlines that followed weren't all about productivity. There was, famously, the Samsung incident: confidential code pasted into a public chatbot, fed into training data.

"Leak" is a generous word - nothing crossed into another company. But for corporate environments, the idea that your conversations could end up training someone else's model is often a deal-breaker.

My colleagues felt the same. Many had tried ChatGPT, agreed it had potential, but were hesitant to use it on anything serious. And the best value came from GPT-4 - $25/month per head. For a team of 100, that's $30k/year before anyone has touched the keyboard.

The problem

1. The hype/risk paradox - those who'd benefit most from frontier AI are also the most exposed if it leaks. GDPR and confidentiality both fail loudly here.
2. Cost - paying per seat for the best models is hard to justify against the value most users will draw out of them.

Solution

Honestly, I had the answer before the question was fully formed. As a developer, I wanted both the best models and the freedom to swap when a new state-of-the-art arrived. Without breaking my employer's bank.

The pieces fit together like Lego. After a weekend of hacking I had something that was secure, cost-effective, and easy to maintain. Three pieces:

Why these pieces

Open WebUI replaced Hugging Face's Chat UI, which I'd started with - felt more tuned to my use case. LiteLLM beat out Portkey Gateway for self-hosting: one config file holds all the API keys, so the request just needs a model name. Portkey is great too - better if you're hosting on the edge (Cloudflare Workers, say) - but requires the client to ship endpoint and key headers per request.

Azure OpenAI got us a custom data handling agreement easily, with Gemini on Vertex AI later joining the same model. Both crucial for a Danish company adhering to European standards.

How it was done

Once the pieces were connected, the chat interface sat on our own servers, behind an internal subdomain. Users could hot-swap between models per message - Claude for writing, Gemini for long context, GPT-4 for everything else. For anything touching confidential data, the Azure-backed models were the default.

Worth noting: in the very early days, GPT-4 access through Azure was more exclusive than direct OpenAI access. That gave us an operational edge - especially for the color pages project.

Implementation challenges

1. Integration complexity - wiring Open WebUI, LiteLLM and Azure OpenAI together cleanly took careful configuration and some troubleshooting.
2. Authentication - making sure only authorized employees could reach the chat interface took its own pass.
3. Model selection - choosing the right mix of models for performance vs cost required real testing, not just vibes.
4. Compliance review - each component's data handling needed legal sign-off.
5. Onboarding - even a beautiful tool fails if people don't know how to use it well, especially with sensitive data. Materials and sessions, not just a Slack post.

Results

As I wrote this in my last month at Flügger, I learned it was perhaps the most-loved little piece of software I set up. The brief from the team was clear: keep it running. It had run without a hitch for six months at that point.

The logs say the same thing. Roughly a thousand conversations over those six months, around 200 a month once it settled in, with activity on nine out of ten calendar days. Tens of millions of tokens went through the private endpoints, and a quarter of all traffic ran on the confidential-by-default models - the ones people reached for when the material was actually sensitive.

Fifteen models from five vendors sat behind the same interface, so anyone could switch to Gemini 1.5 Pro for long context or Claude 3 Opus for writing without leaving the tab. A handful of people built their own assistants on top of it, most of them wired to their own documents. Around $555/month saved versus everyone having a paid sub.

Adoption was not uniform, and it rarely is: a small power-user core drove most of the volume, with the busiest single person behind about a third of all conversations. That is the honest shape of internal tooling - a few people lean on it every day, and everyone else reaches for it when it matters.

DIY approach

If you want to replicate this:

1. Get Open WebUI - grab the latest from the GitHub repo.
2. Set up LiteLLM - follow the Azure provider docs.
3. Configure Azure OpenAI - reach out to Microsoft for a custom data-handling agreement. One-time, through the Azure Portal.
4. Connect the dots - drop your Azure API key and endpoint into LiteLLM. Point Open WebUI at LiteLLM. Done.

Links

Chat UIs:

• github.com/open-webui/open-webui
• github.com/huggingface/chat-ui

Proxies:

• github.com/BerriAI/litellm
• github.com/Portkey-AI/gateway

Azure opt-out from data collection:

• Azure OpenAI data privacy docs