8/27/2024

Setting Up Ollama with MinIO for Object Storage

In this tech-savvy world, getting your hands dirty with cutting-edge tools can be both exciting & daunting, especially when it comes to setting up systems for object storage like MinIO & using models like Ollama. This guide will help you dive deep into the fascinating realm of object storage by showing you how to set up Ollama with MinIO so that you can create your very own Retrieval Augmented Generation (RAG) applications.

What is MinIO?

MinIO is an Open-source object storage solution that’s highly flexible & compatible with AWS S3. It allows you to store vast amounts of unstructured data, such as photos, videos, or any other files you can imagine. It’s like a magical vault for your data, making it easily accessible when you need it.

Here are some key features of MinIO:

Replication: Protects your data with multiple copies
Versioning: Keeps track of file versions over time
Compression: Reduces storage costs by compressing data
Observability: Insight into system performance with monitoring tools

What About Ollama?

Now, let's turn our gaze to Ollama. This nifty tool is designed to help you manage & run large language models (LLMs) locally, such as Phi-3 and others. If you’ve ever wanted to harness the power of LLMs without relying on cloud services, Ollama is your friend.

Benefits of using Ollama include:

Run models locally
Control over your data & implementations
Zero costs associated with hosted models

Pre-requisites: Tying the Knots

Before we gallivant on setting up the world’s best RAG application, there are a few prerequisites you need:

Download & Install MinIO: You can obtain the latest release here.
Set up Ollama: Make sure you grab the latest version from here.
Environment Setup: An environment where you can run Python, FastAPI, Gradio, etc. Python 3.8 or higher is recommended.

Step-by-Step Guide to Setting Up Ollama with MinIO

Step 1: Start the MinIO Server

To kick off, launch the MinIO server. Here’s the command:

1
2

bash
minio server ~/dev/data --console-address :9090 &

This will run the MinIO server in detached mode, allowing you to access it via the web UI at http://localhost:9090.

Step 2: Start Ollama Server + Download Necessary Models

Next up, fire up Ollama & pull the necessary models. Use these commands:

bash
ollama serve
ollama pull phi3:3.8b-mini-128k-instruct-q8_0
ollama pull nomic-embed-text:v1.5
ollama ls

The first command launches the Ollama server, while the latter two download the language models you’ll be using.

Step 3: Create a Basic Gradio App Using FastAPI

Here’s where the magic happens! You want to create a simple chat interface using Gradio. Here's a code snippet to kickstart the process: ```python from fastapi import FastAPI, Request, BackgroundTasks import gradio as gr import requests

app = FastAPI()

LLM_MODEL = "phi3:3.8b-mini-128k-instruct-q8_0" EMBEDDING_MODEL = "nomic-embed-text:v1.5" LLM_ENDPOINT = "http://localhost:11434/api/chat"

def llm_chat(user_question, history): history = history or [] user_message = f"You: {user_question}" llm_resp = requests.post(LLM_ENDPOINT, json={ "model": LLM_MODEL, "keep_alive": "48h", "messages": [{"role": "user", "content": user_question}] })

1
2
3
# Process the response
bot_response = "**AI:** " + llm_resp.json()['message']['content']
yield bot_response

app = gr.mount_gradio_app(app, demo, path="/chat") ```

Step 4: Create Menus for MinIO Buckets

Now that your chat interface is set up, let’s set up MinIO buckets to store our data. Use the following commands to create two buckets:

bash
!mc alias set 'myminio' 'http://localhost:9000' 'minioadmin' 'minioadmin'
!mc mb myminio/custom-corpus
!mc mb myminio/warehouse

Here’s the breakdown:

custom-corpus: For storing documents
warehouse: For saving metadata & chunk vector embeddings

Step 5: Set Up Webhook to Manage Notifications

It's crucial to automatically manage the bucket notifications, especially when new documents are added/removed from MinIO. Use the following commands to set up a webhook:

python
@app.post("/api/v1/document/notification")
async def receive_webhook(request: Request, background_tasks: BackgroundTasks):
    json_data = await request.json()
    print(json.dumps(json_data, indent=2))

This will enable you to receive notifications about the changes in your MinIO buckets.

Step 6: Monitor the Events

Once everything's set up, you’ll want to ensure you’re capturing the events correctly. Use the bucket's event notifications to process newly added documents using the webhook set up in the previous step.

For a sneak peek, go into the MinIO console > Buckets (Administrator) > custom-corpus > Events, and add event destinations for the type of notifications you want.

Step 7: Test the Configuration

It's time to flex those muscles! Add documents into the custom-corpus bucket and check whether the system handles it correctly. Verify if your webhook picks up the new documents and processes them accordingly!

Step 8: Generate embeddings & Save to LanceDB

You might want to store processed embeddings in a vector database like LanceDB to aid in efficient searching. This involves coding logic to handle the embeddings:

1
2
3
4
5
import lance

# Processing the embeddings
if new_documents_added:
    process_and_save_embeddings_to_lancedb(new_documents)

You’ll also need to integrate LanceDB for persisting embeddings from newly chunked documents.

Leveraging Arsturn’s Capabilities

After developing your custom application to serve constituents, consider extending its functionality with Arsturn. With Arsturn, you can instantly create custom ChatGPT chatbots that can significantly boost engagement & conversions. How?

No Code Required: Easily set up chatbots without worrying about a single line of code!
User Analytics: Gain insights into your audience’s needs, allowing you to tailor your content accordingly.
Instant Responses: Provide timely responses to user inquiries, improving engagement.
Customization Options: Fully customize your chatbot to reflect your brand identity and message.

Why not check out how easy it is to deploy your chatbot on Arsturn and enhance your audience's experience? Boost your digital presence today! 🎉

Securing Your Setup

Don’t forget about securing access to your MinIO servers with identity & access management tools. You can enable various features offered by MinIO to ensure your data safety & reliability. Explore options like encryption, versioning, and firewalls to enhance your data safety.

Conclusion

Setting up Ollama with MinIO isn’t just about creating a backend system; it’s about building RESILIENT applications that push boundaries & innovate. With the right tools like MinIO & Ollama, the possibilities are infinite! If you face any hiccups along the way, the community support from platforms like MinIO & Ollama can mean the difference. So dive in & make your project a success!

Feeling overwhelmed? Take a deep breath! You've got this, and there are resources available to help you tackle any obstacles.

Now, go ahead, unleash your creativity, & create something incredible. 🚀