Build a Compact LLM Workstation | SFF PC Guide for AI

8/12/2025

A Guide to Building a Compact LLM Inference Workstation for Small Spaces

Hey everyone, let's talk about something pretty cool: running your own powerful AI models, like the ones that power chatbots & other neat tools, right from your desk. Not in the cloud, not on some massive server farm, but on a small, quiet machine that fits neatly into your workspace. It sounds a bit sci-fi, but honestly, it's more achievable than ever.

I've been going down the rabbit hole of local Large Language Model (LLM) inference lately, & the idea of having a dedicated, compact workstation for it is SUPER appealing. We're talking about a small form factor (SFF) PC, a little powerhouse that you can build yourself. The main reasons for doing this? Privacy, for one. Your data stays with you. Cost is another big one; you buy the hardware once & avoid ongoing API fees. Plus, it's an incredible learning experience.

But here's the thing: building a PC is one thing, but building a small one meant for the specific demands of AI presents its own unique set of challenges. You're cramming a lot of power into a tiny box, & that means you have to be smart about every single component you choose.

So, I've put together this guide to walk you through it. We'll cover everything from picking the right parts to understanding the trade-offs. Think of it as a roadmap for building your own personal AI engine.

Why Go Local & Why Go Small?

First off, why even bother running LLMs locally? The big cloud platforms are powerful, no doubt. But running your own setup has some serious perks:

Total Privacy: When you run a model on your own machine, your data, your prompts, & your results never leave your hardware. This is HUGE if you're working with sensitive information or just value your privacy.
No More API Bills: Those cloud computing costs can add up, especially if you're experimenting a lot. A local setup is a one-time hardware investment.
Customization & Control: You can tinker to your heart's content. Fine-tune models for specific tasks, experiment with different software, & truly understand how things work under the hood.
Offline Access: Your AI assistant works even if your internet is down. Pretty neat, right?

And the "small" part? That's about practicality. Not everyone has space for a giant tower. A small form factor (SFF) build is ideal for home offices, dorm rooms, or just keeping your workspace minimalist. They're also more portable & often more energy-efficient.

The Heart of the Machine: CPU vs. GPU

Alright, let's get into the nitty-gritty. The first major decision you'll make is where the main processing power for your LLM will come from. It's a choice between the CPU (Central Processing Unit) & the GPU (Graphics Processing Unit).

The CPU-Only Approach: The Accessible Starter Kit

Turns out, you don't strictly NEED a monster graphics card to run LLMs. Thanks to amazing software like llama.cpp, you can get surprisingly good performance using just your computer's main processor. This is a fantastic entry point because it's cheaper & simpler.

The key here is RAM. Lots & lots of fast RAM. When you run an LLM on the CPU, the model's parameters get loaded into your system's memory. The more RAM you have, the larger & more complex the models you can run. For a decent experience, you should be aiming for a MINIMUM of 32GB, but honestly, 64GB or even 96GB is where you'll see the real magic happen, allowing you to run very capable models. The new Framework Desktop, for example, is built around this idea, offering up to 96GB of memory to run huge models like Llama 3.3 70B.

For a CPU-centric build, you'll want a modern processor with a good core count. Something from AMD's Ryzen 7 or 9 series or Intel's Core i7 or i9 families would be a great choice.

The GPU-Accelerated Approach: The High-Performance Path

If you want speed & the ability to run larger models with lower latency (quicker responses), a powerful GPU is the way to go. For LLMs, the most important spec on a graphics card isn't its clock speed; it's the VRAM (Video RAM).

Think of VRAM as the GPU's dedicated, ultra-fast memory. Just like with the CPU approach, the model has to fit into this memory. This is why you see so much buzz around cards like NVIDIA's RTX 3090 (24GB), RTX 4090 (24GB), or even professional cards like the A6000 Ada (48GB). More VRAM = more room for bigger, smarter models.

If you're serious about this route, 12GB of VRAM is the absolute minimum you should consider. But 24GB is the real sweet spot for running popular, high-performance open-source models.

One common misconception is that you need the most expensive GPU. Sometimes, it's actually more cost-effective to use multiple, cheaper GPUs instead of one high-end one. However, for a compact build, we're almost always limited to a single GPU slot, so you'll want to get the best single card you can afford with the most VRAM possible.

Let's Build It: A Component-by-Component Guide for SFF

Building a small form factor PC is like playing Tetris with high-tech components. Every millimeter counts. Here’s a breakdown of what you need to look for.

1. The Case: Your PC's Tiny Home

This is your starting point. The case determines the size of everything else. We're talking about cases under 20 liters in volume. Brands like Fractal Design, Lian Li, Cooler Master, & Sliger make some incredible SFF cases.

The Fractal Design Terra, for example, is a popular choice that looks great & is designed for powerful components. When choosing, pay CLOSE attention to the maximum supported GPU length & CPU cooler height. These are the two most critical clearance measurements in an SFF build.

2. The Motherboard: The Central Hub

For an SFF build, you're going to be using a Mini-ITX motherboard. These are the smallest standard-size motherboards, and they are marvels of engineering. Because of their size, they have some limitations you need to be aware of:

Only two RAM slots: This means you have to plan your total RAM capacity from the start. If you want 64GB, you'll need two 32GB sticks.
Usually only one or two M.2 slots: These are for your super-fast NVMe SSDs. I'd recommend getting at least a 2TB drive to start, so you have plenty of room for models, software, & your OS.
A single PCIe slot: This is for your graphics card. No room for expansion cards here!

Brands like ASUS, Gigabyte, & ASRock all make excellent Mini-ITX boards for both AMD & Intel platforms.

3. The CPU Cooler: Keeping Things from Melting

Heat is the enemy of performance, especially in a cramped SFF case. A big tower cooler is out of the question. You'll need a low-profile CPU cooler.

Companies like Noctua (their NH-L9 or NH-L12 series are legendary), Thermalright (the AXP90 is a great option), & Be Quiet! specialize in coolers designed for tight spaces. ALWAYS check the cooler's height against your case's maximum supported height. Sometimes, a liquid cooler (AIO - All-in-One) with a small 120mm radiator can be an option, but you have to make sure your case supports it.

4. RAM: Fuel for the AI

As we discussed, this is critical. For an LLM machine, don't skimp on RAM.

For a CPU-focused build: Go for 64GB if you can swing it. 32GB is the minimum.
For a GPU-focused build: 32GB of system RAM is usually plenty, as the heavy lifting happens in the GPU's VRAM.

Make sure to get a kit of two sticks (e.g., 2x16GB for 32GB total) to take advantage of dual-channel memory speeds, which helps CPU performance.

5. Storage (SSD): Where Your Models Live

You want a fast NVMe M.2 SSD. This is a small stick that plugs directly into the motherboard. It's where your operating system, programs, & the LLM files will be stored. Loading a multi-billion parameter model takes time, & a fast SSD makes the whole experience snappier. A 1TB drive is a good starting point, but 2TB is a safer bet for longevity.

6. The Power Supply (PSU): Clean Power in a Small Box

You can't use a standard ATX power supply. You'll need an SFX or SFX-L power supply. These are specifically designed for SFF cases.

Lian Li & Corsair make some of the best SFX power supplies on the market. Get a modular one if you can. This means you only have to plug in the cables you actually need, which is a LIFESAVER for cable management in a tiny case. For a build with a high-end GPU, aim for at least a 750W or 850W 80+ Gold rated unit to ensure stable power delivery.

The Software Side: Bringing Your Workstation to Life

Once your tiny beast is built, you need to install the software to actually run the LLMs.

Operating System: You can use Windows or Linux. Both work great. Linux is often preferred by developers for its control & lower overhead, but Windows is perfectly capable.
GPU Drivers: If you have an NVIDIA GPU, installing the latest CUDA drivers is essential for getting that sweet GPU acceleration.
LLM Runners: This is the software that loads & runs the models.
- Ollama: This is probably the easiest way to get started. It's a fantastic tool that manages downloading & running a huge library of popular open-source models. It's well-optimized for both CPU & GPU setups.
- llama.cpp: This is the project that really kicked off the CPU inference revolution. It's incredibly efficient & supports a wide range of model types. It's a bit more hands-on than Ollama but offers maximum performance & control.

A key concept you'll run into is quantization. This is a process that "compresses" the LLM files to make them smaller, so they require less RAM/VRAM. A 4-bit quantized model, for instance, will be much smaller than the original 16-bit version & will run much faster on CPU, often with only a minor drop in quality. It's a trade-off, but it's what makes running these powerful models on consumer hardware possible.

Engaging with Your Local AI

So you've built your workstation, installed the software, & downloaded a model. Now what? You can interact with it through a command-line interface, a local web UI, or even integrate it into your own applications via an API.

This is also where the worlds of local AI & business applications can intersect. Imagine you're a small business owner who's built this machine. You could use it to power an internal tool for your team, like helping with coding or drafting documents.

But what about engaging with customers? You could train a custom model on your company's data—your product specs, your FAQs, your knowledge base. Now, you could run this model locally, but to make it accessible to website visitors 24/7, you'd need a bridge. This is where a platform like Arsturn comes in. You could take the insights & even the model architecture you've developed on your local workstation & use it as a foundation. Arsturn helps businesses build no-code AI chatbots trained on their own data. You can essentially deploy your specialized knowledge into a customer-facing chatbot that provides instant support, answers questions, & engages visitors around the clock, boosting conversions & providing a personalized experience. Your little inference workstation becomes the perfect R&D lab for creating a powerful, public-facing AI assistant.

Putting It All Together: The Build Process

A word of warning: building in an SFF case is more challenging than a standard tower. It requires patience & planning.

Read the Manuals: Seriously. Especially the one for your case. Every SFF case has a specific build order you should follow.
Build Outside the Box First: It's a good idea to assemble your motherboard, CPU, cooler, & RAM on top of the motherboard box & test that it boots up before you put it in the case. This is called "breadboarding" & it can save you a massive headache later.
Cable Management is Key: In a small case, stray cables can block airflow & raise temperatures. Plan your cable routing carefully. Modular SFX power supplies are a huge help here.
Thermals, Thermals, Thermals: Once it's running, monitor your CPU & GPU temperatures, especially under load. If things are getting too hot, you may need to adjust fan curves in your BIOS or even consider re-orienting fans.

Building a compact LLM workstation is an incredibly rewarding project. It puts you on the cutting edge of AI & gives you a powerful tool that's all your own. It takes some research & careful planning, but the result is a small, quiet, & mighty machine that can unlock a world of possibilities.

Hope this was helpful & gives you the confidence to start planning your own build. Let me know what you think