Enabling GPU Support in Ollama Deployments

What is Ollama?

Understanding Your Hardware

1. Identify Your GPU: You need to ensure that your system is equipped with a compatible GPU. For Nvidia GPUs, Ollama supports compute capability versions of 5.0 and above. This means that older GPUs, like the NVIDIA GeForce GT710, may not be supported as they usually fall within the compute capabilities of 2.x or 3.x (as stated in this discussion on Reddit).
2. Check Your RAM: The requirements for running different models vary. For instance, 8GB of RAM is needed for 7B models, while 16GB is recommended for 13B models, and 32GB for 33B models. Always ensure you have enough memory available to avoid performance bottlenecks.

Setting up Your Environment

For Linux Users

1. Install NVIDIA Drivers: Ensure you have the appropriate NVIDIA drivers installed. The official NVIDIA Driver Downloads page can guide you through this process.
2. Install CUDA: You will need the CUDA toolkit. A minimum of CUDA 11.4 is typically required. You can download it from the CUDA Toolkit Archive.
3. Install Ollama: With drivers and CUDA installed, run the following command in your terminal:

   1 2 bash curl -fsSL https://ollama.com/install.sh | sh

   This will get Ollama up & running in no time.
4. Launch Ollama with GPU Support: After the installation, make sure to check that GPU support is enabled using:

   1 2 bash journalctl -u ollama -f --no-pager

   You should look for messages indicating that your GPU has been detected.

For Windows Users

1. Install NVIDIA Drivers: Similar to the Linux setup, the installation of NVIDIA drivers is crucial. Head over to the NVIDIA Driver Downloads for guidance.
2. Set Up WSL2: You might want to enable Windows Subsystem for Linux (WSL2) to access Ollama. Ensure your WSL2 is configured correctly.
   • Make sure you’re using WSL2 with GPU support. Check this guide to see how to enable GPU support for WSL2.
3. Install CUDA: You also will need CUDA installed to use the NVIDIA graphics card effectively. Download it here.
4. Install Ollama: Similar to Linux, open your command prompt or PowerShell and run:

   1 2 powershell curl -fsSL https://ollama.com/install.sh | sh
5. Check for GPU Availability: Again, check if Ollama is using the GPU by running:

   1 2 powershell nvidia-smi

   This command should display your GPU usage stats, indicating that Ollama is utilizing your GPU.

Configuring Ollama for GPU Usage

1. Using CUDA explicitly: Set the

   1 CUDA_VISIBLE_DEVICES

   environment variable if you want to specify which GPUs Ollama should use, especially in systems with multiple GPUs.

   1 2 bash export CUDA_VISIBLE_DEVICES=0,1

   This command tells Ollama to use the first two GPUs in your system.
2. Customizing Model Loading: You can adjust memory shopping settings and model specifics in the configuration files or during runtime.
   • Consider reducing the model size or using a more efficient architecture if memory is a limitation.
3. Monitoring Performance: Use tools like

   1 htop

   or

   1 nvidia-smi

   to monitor your resource usage, ensuring that Ollama is properly utilizing your GPU.

Troubleshooting Common Issues

1. No GPU Detected: If you see messages indicating that no GPU was detected, ensure that your drivers are installed correctly. In cases where Ollama still doesn’t find your GPU, try performing a system reboot.
2. Error Messages on Launch: If you encounter CUDA compatibility errors or messages like “no device,” it may signify that your driver versions are outdated or incompatible. Keeping drivers updated is usually the fix here.
3. Performance Is Still Slow: If you notice that the GPU is detected but performance is lacking, reconsider your model size or check your RAM settings. Ensure you meet the minimum specifications outlined above for different models.

Using Ollama with AMD GPUs

Leveraging Arsturn for Enhanced Efficiency

• Effortless No-Code AI Setup: You can set up an AI assistant in just a few clicks!
• Variety of Integrations: It supports multiple platforms helping you establish connections with your audience.
• Robust Analytics: Get insights into how users interact, enabling you to refine your approach continuously.

Conclusion