How to Set Up a Private AI Server with Ollama on a Jetson Orin Nano
Z
Zack Saadioui
8/11/2025
Your Own Private AI: A Step-by-Step Guide to Setting Up a Self-Hosted AI Server with Ollama on a Jetson Orin Nano
Hey everyone! Ever thought about having your own, completely private ChatGPT-style AI? I'm not talking about relying on cloud services where your data gets sent off to who-knows-where. I mean a powerful AI that runs right in your own home or office, on a tiny, power-efficient computer. It's the kind of setup that gives you TOTAL control over your data & your AI models.
Well, it's not just possible, it's actually gotten a whole lot easier. Today, I'm going to walk you through, step-by-step, how to turn a little beast of a machine, the NVIDIA Jetson Orin Nano, into your personal, self-hosted AI server using a fantastic tool called Ollama.
Honestly, the fact that you can run this kind of tech on a device that fits in the palm of your hand is just mind-blowing. We're talking about creating a completely offline, customizable AI assistant that can do everything from answering your questions to helping you write code, all without an internet connection. Let's dive in.
Why Bother with a Self-Hosted AI Server?
Before we get our hands dirty, let's talk about the "why." Why not just use the big online services? Here's the thing, while cloud AI is great, a local AI server has some pretty compelling advantages, especially for developers, tinkerers, & small businesses.
Ultimate Data Privacy: This is the big one. When you self-host, your data never leaves your device. For businesses handling sensitive customer information or individuals who are just privacy-conscious, this is a game-changer. You don't have to worry about your conversations being used to train some massive corporate model.
No More API Fees & Unpredictable Costs: Those "pay-per-token" fees can add up, FAST. With a local server, the only cost is the initial hardware & a little bit of electricity. You can run it 24/7 without a single subscription fee.
Blazing Fast & Low Latency: Since the AI is running on your local network, the response times are incredibly quick. There's no round trip to a server halfway across the world. This is critical for real-time applications like robotics or interactive assistants.
Offline Capability: Your AI works even when your internet is down. This is HUGE for remote locations, mobile setups, or just ensuring your tools are always available.
Customization & Control: You get to choose which models you run. You can fine-tune them, experiment with different ones, & tailor the entire experience to your exact needs without being locked into a specific vendor's ecosystem.
For a business, this opens up some incredible possibilities. Imagine having a customer service bot that's not only instant but also runs on your own secure hardware. You could use a platform like Arsturn to build a custom AI chatbot for your website, but instead of connecting to a generic cloud API, you could point it to your own private Jetson server. This way, Arsturn would help create that polished, no-code chatbot interface, while your Jetson & Ollama setup would handle the "thinking," ensuring all customer interactions remain completely within your control. It's the perfect blend of a user-friendly front-end & a secure, private back-end.
What You'll Need: The Hardware & Software
Okay, let's get our shopping list together. The Jetson Orin Nano is the star of the show here, but you'll need a few other bits to get it up & running.
Hardware Checklist:
NVIDIA Jetson Orin Nano Developer Kit: I'm using the 8GB model, which is more than capable for running some impressive small language models (SLMs). The new "Super" developer kits are also a fantastic option.
A Good Power Supply: Don't skimp on this. The Jetson needs a stable 5V 4A power supply, usually with a barrel jack connector. A weak power supply is a common source of weird, hard-to-diagnose problems.
NVMe SSD: An absolute MUST. Running the operating system & storing your AI models on a fast NVMe SSD is crucial for performance. A 256GB or 512GB drive is a good starting point. You'll need space for the OS, the Ollama container, & of course, the models themselves, which can be several gigabytes each.
MicroSD Card (initially): You'll need a small (32GB is fine) microSD card to initially flash the Jetson's operating system before transferring it to the SSD.
Peripherals: A keyboard, mouse, & monitor for the initial setup. You'll also want an Ethernet cable for the most stable connection during downloads.
Optional but Recommended: A case with a fan to keep the Jetson cool, especially if you plan on running it hard.
Software Checklist:
NVIDIA JetPack: This is the software development kit for the Jetson, which includes the Ubuntu-based operating system. You'll want JetPack 5 (L4T r35.x) or JetPack 6 (L4T r36.x). As of late 2024/early 2025, JetPack 6 is the way to go.
Raspberry Pi Imager (or similar): To flash the JetPack image onto your SD card.
Step 1: Flashing & First Boot
This is the foundational step. We need to get the operating system onto the Jetson.
Download the Jetson Orin Nano Developer Kit SD Card Image: Head over to the NVIDIA developer website & grab the latest JetPack image for your device.
Flash the Image: Use the Raspberry Pi Imager or a tool like Balena Etcher to write the downloaded image to your microSD card.
First Boot: Insert the microSD card into the Jetson, plug in your keyboard, mouse, monitor, & Ethernet cable, & then connect the power supply.
Ubuntu Setup: The Jetson will boot up into a standard Ubuntu setup process. Go through the steps to create a user account, set a password, connect to your network, etc. It's all pretty straightforward.
Move to SSD: Once you're at the Ubuntu desktop, the VERY next thing you should do is move your root file system from the slow microSD card to your fast NVMe SSD. NVIDIA provides scripts & instructions for this, & it makes a world of difference for performance. There are plenty of great video guides on YouTube that walk you through this process.
Step 2: Installing Ollama - The Heart of Our AI Server
With our Jetson up & running on the SSD, it's time to install Ollama. Ollama is a fantastic piece of software that makes downloading, managing, & running large language models incredibly simple. It handles all the complex backend stuff so you can just focus on using the AI.
You have two main paths here: a native install or a Docker install. Honestly, the Docker method is my preferred way because it keeps everything neat & containerized, preventing any conflicts with your base system. It's also the method recommended by many in the Jetson community.
The Docker Method (Recommended)
The folks at NVIDIA, particularly Dustin Franklin, have created a project called