AI glossary · Building and connecting

Local model

Also called: Local AI · Self-hosted model · Local LLM

What is a local AI model?

When you use ChatGPT, Claude or Gemini, the language model runs on the provider's computers. Your question travels there and back.

With a local model, the AI sits with you: on an office computer or on a server you control. Think of your own coffee machine instead of the café on the corner: you decide what goes in, but you also descale it.

How does it work?

This works with open models that their makers release for download, such as Llama, Mistral, Gemma, Qwen and DeepSeek. A tool such as Ollama downloads a model and starts it on your own computer. It then answers like a chat program, and your texts stay with you.

Model size depends on the hardware. Smaller models run on a good laptop; large ones need a server with plenty of graphics memory.

Why does it matter for your business?

The main advantage is data protection: customer data, contracts or health details stay with you. That matters for sensitive documents, say in a tax office, medical practice or property management.

The limits are just as real. Large cloud models are often stronger on difficult tasks than what runs on your own machine. Add purchase, power, updates and someone to look after it. For many tasks, a cloud service with EU hosting and a data processing agreement is more practical.

How does OsAI use it?

In client projects OsAI prefers EU hosting or locally run models so sensitive data stays in the EU legal area. Where US services are needed, EU standard contractual clauses and a data processing agreement (Art. 28 GDPR) are added.

Which route fits depends on the task and how sensitive the data is. OsAI works this out in AI consulting. The free AI data readiness check gives a first view.

Frequently asked questions

Is a local AI model GDPR compliant?

It helps, because your data does not go to an AI provider. The EU GDPR still applies: you need a purpose, a legal basis and protection against outside access. If a service provider runs the server, you need a data processing agreement with them too. This is an overview, not legal advice.

Are local models as good as ChatGPT or Claude?

On difficult tasks usually not; on clearly defined ones often good enough. Summarising texts, pulling details from emails or answering questions about your documents with RAG often works with smaller models too. Test with real tasks before buying hardware.

What do I need to run an AI model locally?

A powerful enough computer, a tool such as Ollama and an open model. A current laptop is often enough to try it out; several users in a business need a server and someone who handles updates and security.

Sources

Last updated: