Cloud-based AI services have become increasingly difficult to ignore. They produce impressive results, but they also come with serious trade-offs. Subscription costs keep rising, data is sent to remote servers, and the environmental cost of training and running enormous models is difficult to defend. For some users, the solution is to build a private AI system that runs entirely on hardware they control. That approach is still niche, but it is becoming far more practical than it used to be.

One technology editor recently decided to test this idea with a compact mini PC and a more powerful server. The original plan was simple: take a small desktop computer with a neural processing unit (NPU), install local AI software, and see if it could replace the cloud services that many people rely on. The plan quickly grew into a two-machine setup that includes an AI agent, a local model server, and a surprisingly easy way to control everything from a smartphone.

Key facts at a glance

• A mini PC with a Core 9 processor and an NPU acts as the host for a local AI agent called Hermes.
• Heavy AI workloads are handled by a separate server using a Framework Desktop with Ryzen AI Max hardware.
• The server runs Ubuntu and LM Studio, using the Quinn language model.
• Users interact with the system through the Telegram messaging app from a phone, which makes it accessible from anywhere.
• The setup is being used to gather and read news for technology podcast production.

The hardware setup

The project starts with a mini PC from MSI's Cubi NUC line. The Cubi NUC is a small desktop designed to save space, but this particular version includes a Core 9 processor and an NPU capable of running smaller localized AI models. Mini PCs like this are often used for basic productivity, media playback, or lightweight server tasks. The addition of an NPU makes them much more interesting for on-device artificial intelligence because NPUs are designed to accelerate machine-learning operations while consuming relatively little power.

As the project took shape, it became clear that the mini PC alone would not be enough for the kind of AI work the editor had in mind. A separate, far more powerful machine had to be brought into the picture. That server is based on the Framework Desktop, a modular desktop computer that can be configured with Ryzen AI Max hardware. This AMD platform is almost purpose-built for local AI workloads, with high-performance CPU cores, a strong integrated GPU, and a dedicated AI engine that can handle larger models.

The mini PC runs the Hermes agent, which is the control layer. Hermes communicates with the larger server, which runs Ubuntu and LM Studio. LM Studio is a popular application for downloading and running open-source language models locally. Inside that environment, the server uses the Quinn model, a relatively streamlined large language model that balances response quality with the hardware constraints of a desktop system. Together, the two machines create a distributed AI setup where the agent handles interaction and the server handles the heavy computational lifting.

Why local AI matters

There are plenty of reasons someone might want to avoid cloud AI services. Privacy is near the top of the list. Sending prompts to a remote server means sharing whatever is in those prompts, whether that is personal information, business documents, or draft writing. With a local model, everything stays on the user's own network. No data is uploaded to a third party, and no company can log the conversation for training or advertising purposes.

Cost is another factor. While some free AI services exist, the most useful models are often behind subscription paywalls. Token-based pricing can also get expensive quickly for heavy users. A local setup requires a hardware investment up front, but once the equipment is in place, there are no monthly fees or usage caps. The model runs as many times as the user wants without metering.

Control and autonomy are also important. Cloud providers can change their terms of service, deprecate features, or shut down entirely. Local AI models are not held hostage by company policy. Users can choose the exact model they want, update it when they choose, or replace it with another architecture. For journalists, researchers, and other people who handle sensitive information, this level of control can be essential.

The role of the AI agent

An AI agent is more than just a chatbot. It is a piece of software that can take instructions, coordinate tools, and perform multi-step tasks. In this setup, Hermes serves as the intermediary between the user and the language model. Instead of asking the user to open a terminal or interact with LM Studio directly, the agent interprets requests, sends them to the right model, and returns the results in a friendly format.

This kind of architecture makes the system easier to use and extends its capabilities. For example, the agent could be given a task that involves retrieving information from a file, summarizing it, and then reporting back. The agent can also be connected to external services, such as messaging platforms, without exposing the underlying model infrastructure.

A smartphone as the remote control

The final piece of the puzzle is the smartphone. The editor chose to control the local AI through Telegram, the popular messaging app. Telegram exposes a straightforward bot API, and the same API can be used for home automation or personal assistant projects. By creating a Telegram bot tied to the Hermes agent, the user can send a message from anywhere and receive an answer as if they were texting a friend.

This is a clever workaround because it avoids building a custom mobile app or setting up a complicated web dashboard. Telegram handles authentication, message delivery, and notifications. The user does not need to be on the same Wi-Fi network or connected through a VPN, although exposing a bot to the internet always requires proper security considerations. The bot front-end makes the local AI feel like a cloud service, when in fact it is running entirely on private hardware.

Practical uses and experiments

The editor built this local AI to help with news gathering and reading for two technology podcasts. Podcast production involves reading a large number of articles, monitoring announcements, and summarizing developments into show notes or segments. A local AI that can collect stories, summarize them, and highlight key points could save a significant amount of time.

However, making the workflow truly useful took a lot of effort. Setting up the server, configuring the agent, and connecting the messaging front-end was not a simple plug-and-play process. Even after all that work, the editor is still experimenting with ways to make the system reliably practical. Model responses are not always perfect, and prompts may need to be tuned to produce useful results. The project is a work in progress, but it already demonstrates how far local AI has come.

Challenges and trade-offs

Running a local AI is not without its drawbacks. The hardware required for good performance can be expensive, and the setup process demands a level of technical knowledge that many users do not have. Installing an operating system, configuring drivers, choosing a model, and connecting an agent are all tasks that can intimidate beginners.

Model quality is another issue. Small and medium-sized language models have improved dramatically, but they still may lag behind the largest cloud models in complex reasoning, creative writing, or up-to-date knowledge. Users who need state-of-the-art responses might find local alternatives lacking. The Quinn model used in this setup is a solid choice for efficient local inference, but it is not designed to compete with massive data-center-scale systems.

There are also maintenance burdens. Local models need to be updated or replaced as new versions are released. The operating system, AI tooling, and agent software all require periodic attention. If something breaks, the user is the only person responsible for fixing it. That is a trade-off many enthusiasts are willing to accept in exchange for privacy and independence.

The road ahead

What this experiment shows is that local AI is no longer a distant dream or an enterprise-only luxury. A small mini PC with an NPU, a powerful desktop server, and a messaging app can form the foundation of a private AI assistant that is available from a phone at any time. As neural processing units become standard in more computers, and as open-source models become even more efficient, this kind of setup will likely become accessible to a broader audience.

For now, the project remains an experiment. The editor who built it is still refining the workflow and looking for better ways to turn raw news feeds into podcast-ready material. That is exactly the right way to approach a new technology: not as a magic solution, but as a tool that can be tuned and improved over time. The ability to control a home-grown AI from a phone is a useful reminder that the most powerful technology is often the technology we can control.

Source: PCWorld News