Local and on-premise AI deployment
Aadhib Nasser · AI Engineer and AI Consultant
Who this is for
- 01Organisations whose contracts, HR records, client files or internal documents should not leave the building
- 02Teams deciding between local hardware, a hosted API or a split between the two
- 03Companies that want a model tested on their own documents before it replaces anything
What the work covers
- 01Deciding which workloads belong on local hardware and which are better hosted
- 02Model selection for local use: open-weight models, quantisation and the quality it costs
- 03Hardware sizing, from a single Mac mini or Mac Studio to on-premise GPU servers
- 04Hybrid designs where sensitive steps run locally and everything else uses a hosted model
- 05Deployment, monitoring and updates inside your own network
- 06Evaluating a local model on your own documents before it replaces anything
How I approach it
Local hosting is not compliance
Running a model locally does not by itself make a deployment compliant with PDPL or any other regulation. It makes certain architectures possible. Whether they are sufficient is a question for you and your counsel.
Responsibility against flexibility
The real comparison is not hardware price against instance price. It is total responsibility against total flexibility, and the answer changes per client.
Measure the model you will actually run
A compressed model's headline benchmark is an average of short tasks. Long, multi-step agent work is where the gap shows, so the evaluation runs on your tasks, at your context length, on your hardware.
Related work
Case studies and projects
Further reading
Notes on this subject
Questions
Asked before an engagement
Does local AI make us PDPL compliant?
Not by itself. Keeping data on hardware you control is one input into a compliance assessment, not the conclusion of one. The legal and security judgement belongs to you and your counsel.
Is a local model as good as ChatGPT or Claude?
Not across the board. Open-weight models are strong on bounded work such as drafting, classification and extraction, and still trail on long, multi-step agent tasks. The usual answer is a split: local for the sensitive steps, hosted for the rest.
What hardware do we need?
It depends on the model and on the context length you will actually use, not on the model's headline size. The weights are only part of the memory budget. The context cache and everything else resident on the machine are the rest.
Start a conversation
I usually reply within a day
Send a few sentences about the problem: what the process is today, where the data lives, and what would change if it worked.