8/27/2024

Deploying Ollama Using Docker Compose

Are you ready to dive into the exciting world of AI and LLM (Large Language Models) by deploying Ollama? 🔥 With the help of Docker Compose, we can quickly and easily set up Ollama and its dependencies, creating a local environment that can handle powerful AI applications. In this blog post, we’ll walk through the entire process, making it straightforward & accessible for even the most novice tech enthusiasts. Let’s get started!

What is Ollama?

Ollama is a cutting-edge open-source platform designed for running large language models effortlessly. It leverages technology to allow users to create, customize, and manage models tailored for a plethora of applications, ranging from chatbots to more complex AI programs. If you haven’t checked it out yet, I highly recommend you visit the Ollama website for a deeper understanding of its capabilities.

Why Use Docker?

Docker provides a portable, efficient, & isolated environment where you can deploy applications. It takes away the headaches of dependency conflicts & setup issues, allowing us to focus on what truly matters—building amazing AI models!

Here’s why deploying Ollama using Docker Compose is a game-changer:

EASY SETUP: No more messing with tedious configurations. Docker makes everything super convenient.
ISOLATION: Each application runs in its isolated container. You can easily maintain different versions of Ollama or different configurations without any conflict.
SIMPLE SCALING: Want to run multiple instances of Ollama? Docker makes it easy with just a few CLI commands!

Prerequisites

Before we launch into the deployment process, let’s ensure you have everything in place:

Docker: Make sure Docker Desktop is installed. You can grab it from the official Docker website.
Docker Compose: This is typically included with Docker Desktop, but you can install it separately if needed. Check the Docker Docs for installation instructions.
Optional GPU Support: If you want to leverage GPU capabilities for faster inference, you’ll need to install the NVIDIA Container Toolkit. Here’s how to do it:
1 2 3 4 5 6 7 8 9 10bash curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \ && curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \ sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \ sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list sudo apt-get update sudo apt-get install -y nvidia-container-toolkit sudo nvidia-ctk runtime configure --runtime=docker sudo systemctl restart docker docker run --gpus nvidia/cuda:11.5.2-base-ubuntu20.04 nvidia-smi

Now your system is prepped & ready to roll! 🤓

Getting Started with Ollama

Step 1: Setting Up the Docker Compose Repository

Start by cloning the Ollama Docker repository to your local machine such as:

1
2
3

bash
git clone https://github.com/valiantlynx/ollama-docker.git
cd ollama-docker

Step 2: Understanding Docker Compose Configuration

In the cloned repository, you’ll find a file named

docker-compose.yml

. This file is vital as it defines all the services that are part of your application. Let’s quickly go through the main parts of it:

Ollama Server: This is where the primary application logic resides. It serves the language models.
Web UI: It allows you to interact with the AI models visually.
Network Configuration: It handles how these components communicate with each other.

Here’s a sample

docker-compose.yml

configuration: ```yml version: '3.8' services: ollama: image: ollama/ollama:latest ports:

1
2
3
- "11434:11434"
volumes:
  - ollama:/root/.ollama

webui: image: ghcr.io/open-webui/open-webui:main ports:

"8000:8000" environment:
OLLAMA_BASE_URL=http://host.docker.internal:11434 volumes: ollama: ```

ep 3: Deploying Ollama

Let’s bring up Ollama and all of its dependencies using Docker Compose. In your terminal, run:

1
2

bash
docker-compose up -d

This command will start the services defined in your

docker-compose.yml

file in detached mode (running in the background). You can view the status of your containers using:

1
2

bash
docker-compose ps

Step 4: Accessing the Ollama WebUI

Once everything is up & running, you should be able to access the Ollama WebUI by navigating to http://localhost:8000 in your web browser. Here, you can explore the capabilities of your language models & even set up new configurations.

Step 5: Interacting with Models

After accessing the WebUI, you can choose from various models available and interact with them. You can install new models directly through the UI interface. For example, navigate to Settings -> Model Installation to add models like

llava-phi3

, which provides an evident ChatGPT-like experience.

Step 6: Stop & Clean Up

Once you finish experimenting, you may want to stop the containers running. Simply execute the following command:

1
2

bash
docker-compose down

This will stop & remove the containers as well as the network created by Docker Compose. Cleanup is vital to keeping your environment orderly.

Troubleshooting Common Issues

As simple as this process is, issues may arise. Here are some common problems & solutions:

Container won't start: Check the logs using
1docker-compose logs
& look for error messages that indicate what went wrong, such as dependency conflicts.
Cannot access WebUI: Ensure that the Docker containers are running & that your firewall is not blocking the relevant ports.
Performance issues: If you’re facing slowness using the Ollama image without GPU, consider optimizing your Docker settings. Run Ollama in GPU mode to enhance output speed and efficiency.

Benefits of Using Ollama with Docker

Efficient Management: Using Docker, developers can easily manage containers or subsequent deployments, scaling them as per need. This flexibility proves extremely beneficial in dynamic development environments.
Cost-effective: You need not invest in expensive servers or infrastructure while using containers; you can run them on any local machine supporting Docker.
Robust Community Support: Both Docker, along with Ollama, enjoy active communities. This translates to forums, documentation, and guides like this one, making it a steady choice for developers.

Arsturn: Leveling Up Your AI Experience!

As you embark on your journey with Ollama, enhancing your AI efforts can truly skyrocket your project’s effectiveness. Have you considered integrating an AI chatbot for your business? With Arsturn, you can instantly create custom ChatGPT chatbots tailored to your specific needs and engage your audience effortlessly. Arsturn empowers brands to develop meaningful connections through engaging conversations, keeping your audience involved & converting leads into loyal customers.

Key Features of Arsturn:

Effortless AI Chatbot Creation
Instant Responses to customer inquiries
Full Customization to reflect your brand identity
Insightful Analytics to refine your strategies

Ready to experience the power of AI? Emoji-💡Start using Arsturn today! No credit card required—just check it out & elevate your brand’s engagement!

Conclusion

Deploying Ollama with Docker Compose is not only simple but it’s also an efficient way to harness the power of LLMs for various applications. By following the outlined steps, you can swiftly create a robust AI development environment. Combine this with Arsturn’s powerful chatbot creation tools, and you're well on your way to transforming your audience engagement. Happy coding! 🚀