8/27/2024

Setting Up Ollama with Azure Functions

Are you eager to dive into the world of Local Large Language Models (LLMs) while leveraging the prowess of Azure Functions? Look no further! In this blog post, we’ll take a detailed journey through the setup process for getting Ollama to run on Azure Functions, combining the power of both to create a seamless AI experience.

What is Ollama?

First off, let’s talk about Ollama. Ollama is a fantastic framework designed to simplify the deployment of LLMs without going through the complexities usually required in traditional setups. It supports multiple models like Llama (with various versions) and Mistral, based on transformer architectures. The best part? You can run your models without relying on third-party providers, thus keeping your information private while also making your costs predictable.

Why Choose Azure Functions?

Microsoft Azure's Functions allow developers to run event-driven serverless code without managing infrastructure. This makes it a GREAT fit for deploying Ollama as a backend to an interactive chatbot. Azure Functions allows for scalability & only charges for the amount of time your code is executed—perfect for handling various workloads, including LLMs. So, let’s roll up our sleeves & get to work!

Prerequisites

Before we get into the nitty-gritty of doing the setup, you’ll need a few things:

Azure account: If you don’t have one, you can easily create one for free.
Ollama installed: You need the Ollama CLI tool which you can download here.
Knowledge of basic command-line operations: This will be helpful throughout the setup process.
Azure CLI: Install Azure CLI to interact & manage resources— follow this guide.

Step 1: Set Up Azure Functions

To get started, we’ll create an Azure Function App. Here’s how:

Open the Azure portal & log in.
Click on Create a resource.
Select Compute > Function App.
Fill in the basic function details:
- Subscription: Choose your Azure subscription.
- Resource Group: Create a new one or use an existing one.
- Function App name: Create a unique name that’s easy to remember.
- Runtime Stack: Choose your preferred runtime (Node.js, Python, etc.).
- Region: Choose a region close to your users.
Click Review + Create, and once validated, select Create.

Once your Function App is created, you'll have a serverless environment set up to run your Ollama code.

Step 2: Deploying Ollama with Azure Functions

With the Azure Function App ready, let's deploy our Ollama instance.

Navigate to your newly created Function App.
Click on Functions in the left sidebar & then select Create.
Choose the HTTP Trigger template.
Name your function and set the Authorization Level to Anonymous if you want it to be publicly accessible.
Click Create.

Now, head to the Functions section to open the code editor for your newly created function. Here, you'll write your code to interact with Ollama using the HTTP endpoint.

Step 3: Writing Code to Connect Ollama

Your Azure Function needs to invoke Ollama’s API whenever it receives requests. Here’s a simplified version of how that code might look in a Node.js function:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
const axios = require('axios');
module.exports = async function (context, req) {
    const prompt = req.body.prompt;
    try {
        const response = await axios.post('http://localhost:11434/v1/chat/completions', {
            model: 'llama2',
            messages: [{ role: 'user', content: prompt }]
        }); 
        context.res = {
            status: 200,
            body: response.data,
        };
    } catch (error) {
        context.res = {
            status: 500,
            body: error.message,
        };
    }
};

In this example, the function takes a prompt from the request body, sends it to the Ollama endpoint, and returns the response. Make sure the Ollama server is running in the background as well. Run

ollama serve

to start Ollama locally.

Step 4: Testing Your Setup

To test your function, navigate back to the Azure Function's overview page & find the Function URL. You can use a tool like Postman or cURL to send POST requests to your Azure Function’s endpoint:

1
curl -X POST -H "Content-Type: application/json" -d '{"prompt": "Why's the sky blue?"}' https://<your-function-app-name>.azurewebsites.net/api/<your-function-name>

This should return the response generated by Ollama, effectively indicating everything is working! If you’re not quite getting the results you expect, double-check that both Ollama & Azure are set up correctly.

Step 5: Additional Configuration & Best Practices

Ensure that your Ollama & functions are secured right, especially if dealing with sensitive data. Use Azure's built-in authorization features to provide security where needed.
Monitor your function’s usage to ensure that you stay within the Azure Free Tier limits if you're using a trial.
Customize the HTTP trigger to handle different types of requests, depending on how you plan to use Ollama.
Use Azure Cosmos DB or Azure Storage for sustaining state or keeping track of user inputs and responses.

Conclusion: Join the AI Revolution with Arsturn

Setting up Ollama with Azure Functions can seem daunting at first, but it opens up a world of possibilities for interactive AI-driven applications. If you’re looking to supercharge engagement further, consider using Arsturn.

Arsturn enables you to instantly create custom ChatGPT chatbots that engage your audience before anyone else. With its user-friendly platform, you can build meaningful connections across your digital channels, enhancing brand image & customer experiences effortlessly. Join the growing community of organizations leveraging conversational AI to enhance engagement & streamline operations. Claim your chatbot now and see the magic for yourself!

FAQs

What if I want to connect other models or APIs?
You can easily modify the connection points in your Azure function code to use other AI services as needed.
Can Ollama work without an internet connection?
Yes! Ollama can run locally, and your Azure Function will still be able to interact with it through localhost. However, it's best suited for development & testing in such scenarios.
Will I incur costs while using Azure Functions?
As long as you stay within the Free Tier limits, you can experiment without worrying about costs. Always monitor your usage!

We hope you found this guide helpful! Let’s start leveraging Ollama’s strengths with Azure Functions today!