{/* */} {/* */} {/* */} {/* */} {/* */} {/* */} {/* */} {/* */} > Every company's business data is their gold mine > and there's every company sitting on these gold mines” > > — Jensen Huang, CEO & Founder @ NVIDIA Building an enterprise AI system means enhancing a Large Language Model (LLM) by augmenting it's knowledge through access to additional data or tools. LLMs are already trained on all the data publicly available on the internet, so in an enterprise AI context the most valuable thing you can do is connect an LLM to the huge amounts of unique business data you've accumulated over many years. It's also the biggest challenge in building enterprise AI systems as you have legitimate reasons for distributing all your data and tools across various locations and networks. It's unlikely that you have the required GPUs and compute to run the LLMs available in all these locations. Data is spread everywhere, LLMs and compute are somewhere else, nothing is where you need it to be! Ockam solves this problem through the use of "virtual adjacency": ![Ockam AI Virtual Adjacency for Gen AI apps](/blog/secure-llm-connections/ai-virtual-adjacency.png) Without the need to replatform, rearchitect your infrastructure, or even consolidate or migrate your data, Ockam makes all these dependent systems available to each other as though they were all virtually on the same machine. Your enterprise AI agents can establish secure and private point-to-point connections with any data or system they need to access, and you can rapidly build AI systems without the need to change your network or expose any of your data to the public internet. In this post I'm going to walk you through how to setup an Ockam Portal, and show how Ockam's approach to virtual adjacency can make your AI systems privately available wherever your data is. ## Data analytics & insight from LLMs Before we get into _how_ to set this up, I'm going to jump straight to the end and give you an example of what's possible, in less than 10 minutes. We're going to connect a private AI model running in the cloud with some private data in an "on-prem" (my laptop!) environment. ![Azure OpenAI + Ockam architecture](/blog/secure-llm-connections/final-architecture.png) **Remember that for all this I am _not_ using any public APIs or making any of my systems or data accessible from the public internet.** ### !!steps This demo is going to use a Large Language Model for data analysis tasks. I loaded a CSV file that contained a bunch of health information for 10,000 people into a data frame using Pythons. I then gave the LLM a prompt with some minimal detail about the data and then interpolated the data frame right into the prompt. ```markdown ! tour > The following is a dataset that represents health information > for a group of people, refer to it for future prompts: > {dataframe} ``` ### !!steps That's is _all_ the context I provided. Now it's time to watch the ✨magic✨ happen by asking the LLM to tell me what I just loaded. ```markdown ! tour > Explain this dataset ``` ### !!steps The results were impressive. Imagine all the data inside your organisation. All the times you've had to ask someone "what is this?" when you've been looking at a SQL table or an export. Now you can get a detailed answer within seconds. You can scroll the full output on this page to see the complete detail. It identified all the columns in the data and what they represented, summarised the number of rows and columns so I knew the size of the data, and offered some usage suggestions. ```markdown ! tour # Health Information Dataset Explanation This dataset contains comprehensive health information collected from a group of individuals. The focus is on various health metrics, lifestyle choices, and family medical history that can influence health outcomes, particularly concerning heart disease and other related conditions. ## Attributes Overview * **Age**: The age of the individual, which is a significant factor in determining health risks. * **Gender**: Specifies whether the individual is male or female, as health outcomes can differ by gender. * **Blood Pressure**: Recorded blood pressure value, a critical indicator of cardiovascular health. * **Cholesterol Level**: Measures the cholesterol level, which can indicate risks for heart disease. * **Exercise Habits**: Indicates the level of physical activity (High, Medium, Low), relating to cardiovascular and overall health. * **Smoking**: Indicates if the individual smokes, a significant risk factor for various health issues. **Family Heart Disease**: Indicates if there is a family history of heart disease, which can elevate individual risks. **Diabetes**: Diabetes status (Yes/No), an essential factor in cardiovascular health risk assessments. **BMI**: The Body Mass Index, which is a measure of body fat based on height and weight. **High Blood Pressure**: Indicates if the individual has been diagnosed with high blood pressure (Yes/No). **High LDL Cholesterol**: Indicates if the individual has high LDL cholesterol, which is linked to heart disease risk. **Alcohol Consumption**: The level of alcohol consumption (Low, Medium, High), which affects overall health and cardiovascular risk. **Stress Level**: Indicates the individual's stress level (Low, Medium, High), which can impact health significantly. **Sleep Hours**: Average hours of sleep per night, a crucial factor in health and recovery. **Sugar Consumption**: Indicates the level of sugar intake (Low, Medium, High), which can affect metabolic health. **Triglyceride Level**: Measures triglyceride levels, an important marker for heart health. **Fasting Blood Sugar**: Blood sugar level after fasting, which is essential for assessing diabetes risk. **CRP Level**: C-reactive protein level, a marker of inflammation in the body. **Homocysteine Level**: Homocysteine levels, which can indicate cardiovascular risk. **Heart Disease Status**: Indicates current heart disease status (Yes/No), providing a direct indication of health outcomes. ## Dataset Size The dataset contains a total of 10,000 entries (rows) and 21 features (columns), allowing for a diverse analysis of health-related attributes among the participants. ## Usage This dataset can be used for various analyses, including but not limited to, identifying risk factors for heart disease, studying lifestyle impacts on health, evaluating the effectiveness of health interventions, and understanding demographic differences in health outcomes. ``` ### !!steps Now that the LLM knew what my data was, I asked it the question that's the first that comes to mind whenever I'm looking at a dataset for the first time: > What trends, observations, or outliers exist in this data? It came back within seconds with the kind of insight it would have taken me hours to uncover. ```markdown ! tour # Trends, Observations, and Outliers in the Health Information Dataset ## Trends **Aging Population Health Risk**: As age increases, there appears to be a strong upward trend in the prevalence of high blood pressure and high cholesterol levels, indicating increasing cardiovascular risks with age. **Gender Differences in Health Outcomes**: Males may exhibit higher levels of smoking and alcohol consumption compared to females, leading to elevated cardiovascular risk factors such as higher cholesterol and blood pressure readings. **Exercise and Health Metrics**: Individuals engaging in regular exercise tend to report lower BMI, blood pressure, and cholesterol levels, demonstrating the protective effects of physical activity on cardiovascular health. **Impact of Diet on Health**: Higher levels of sugar and alcohol consumption are associated with poorer fasting blood sugar levels and elevated triglyceride levels, indicating a potential link between diet and metabolic health. ## Observations **Smoking Status Influence**: Smokers in the dataset show higher occurrences of heart disease and related metrics, confirming the detrimental effects of smoking on overall health. **Family History Impact**: A significant portion of individuals with a family history of heart disease also displays higher instances of high blood pressure and cholesterol, suggesting that genetics play a vital role in health risks. **Psychological Stress**: Participants reporting high stress levels also tend to have lower sleep hours, which correlates with poorer health outcomes including elevated blood pressure and BMI. ## Outliers **Extreme BMI Values**: Some individuals may display severely high or low BMI readings, particularly those classified as underweight or obese, indicating potential health issues related to nutrition and lifestyle. **Unusually High Cholesterol**: A few entries may have atypically high cholesterol levels despite normal exercise and diet habits, warranting further investigation into other underlying health issues. **Low Blood Pressure Cases**: Instances of low blood pressure coupled with high stress levels or high cholesterol might be rare and could suggest unique health conditions or medication effects rather than general trends. **Anomalous Family History Reports**: Some participants might report a history of heart disease despite having healthy lifestyle metrics, raising questions about genetic predispositions versus lifestyle factors. ``` Adding AI into the process has taken minutes to setup and it now focuses my attention on validating the alternatives that have the highest probability of being "interesting". It both reduces the time it takes for me to get started by orders of magnitude and directs my effort to the most valuable places to start. Just imagine the amazing things you can do once I show you how quick it is to connect these things together! ## Bringing LLMs, AI Agents, and data closer to each other I achieved all this in less than 10 minutes on my laptop connected to my private home network, communicating with a private OpenAI service in a private Azure network, over an Ockam Portal. An Ockam Portal does something that feels like magic; It virtually moves your remote application through the Portal so that it's available on `localhost`, something we refer to as "virtual adjacency". I didn't need to change my network layer configurations. I didn't need to move a database into the cloud. I didn't need to have a bunch of GPUs or spare compute available locally to run a model. It does this by having both ends of the portal establish a mutually authenticated point-to-point connection with each other. The Ockam Protocol takes care of automatically generating and exchanging keys between those nodes, and then automatically and regularly rotating those keys. The end result is a secure and private connection that is end-to-end encrypted with none of the nodes directly accessible from the public internet. Once the portal is setup my python code is able to talk to the Azure OpenAI service as though it's a service running directly alongside my code on `localhost`.

Ockam Portals make remote services appear virtually adjacent to each other

This isn't limited to Python scripts pushing CSV files to Azure OpenAI from a laptop. _Anything_ that needs to connect data to an AI service can bring those systems virtually together in a secure way. Any agentic system, workflow, batch or ETL process, RAG system, and even product integrations can connect with whatever data you need wherever it's located — from those on-prem systems with decades of data in them like Oracle and SAP, to your BigQuery tables that just happen to be in different cloud to where you need them. ## Interacting with LLMs locally I've packaged a demonstration of this into a couple of scripts and a Docker configuration so you can try it yourself with minimal effort. The full implementation is [available on GitHub](/tree/develop/examples/command/portals/ai/azure_openai_jupyter), though I'll walk through the relevant parts in detail to explain how it all works. Included in the repo is: * A set of scripts that will: * Install the latest version of Ockam Command on your workstation, and enroll your machine as an administrator in your Ockam Project. * Generate enrollment tickets for the Ockam Nodes that will create each end of our Ockam Portal. * Create a new isolated resource group, VNet, Azure OpenAI service and all other dependencies within Azure required to complete the demo. * Run an Ockam Node within Azure that mutually authenticated clients can later use to securely and privately access the Azure OpenAI service. * Start an interactive playground on `https://127.0.0.1:8888` * A Docker Compose configuration file that will start a container that: * Runs an instance of Jupyter Lab to serve as our interactive playground to load data and communicate with the Azure OpenAI service. * Includes a notebook with all the commands and prompts I used to interact with the LLM. * Automatically start an Ockam Node that's pre-configured to create an Ockam Portal to the Azure OpenAI service. ### Prerequisites * An [Ockam account](https://www.ockam.io/signup). * A [Microsoft Azure account](https://azure.microsoft.com/en-us/pricing/purchase-options/azure-account) * [Azure CLI](https://learn.microsoft.com/en-us/cli/azure/) * [jq](https://jqlang.github.io/jq/) * [Docker & Docker Compose](https://docs.docker.com/compose/install/) Before running the script below you ensure you have an active Ockam account. If you don't already have an account, you can [sign up now](https://www.ockam.io/signup) to start a 14-day trial. To automatically provision everything clone the [Ockam GitHub repository](https://github.com/build-trust/ockam) and within the `/examples/command/portals/ai/azure_openai_jupyter` directory run: ```bash ./run.sh ``` This will provision the required resources within Azure and then start the Docker container that is running Ockam + Jupyter Lab. The log output will include a signed URL that you when you visit in your browse will automatically log you in. The URL will look like the one in this screenshot: ![Jupyter login URL](/blog/secure-llm-connections/jupyter-login.png) Once logged in, in the file explorer on the left, click [`Notebooks > azure-openai.ipynb`](http://127.0.0.1:8888/lab/tree/notebooks/azure-openai.ipynb) to view the notebook with all of the commands required to communicate with the Azure OpenAI service. You can run all of the commands and see the output live by selecting `Run > Run All Cells`. You've just securely connected your private workstation to a private Azure OpenAI endpoint, loaded in some private data, and started the interaction loop between AI and your data. It took only a few minutes! Feel free to keep playing with what you've setup, load in other data sources, experiment with what is possible. Once you're finished with the demo you can remove all the created resources by running: ```bash ./run.sh cleanup ``` Let's dig into what this script has done so you can understand how to replicate something similar for your own needs. ## Setup a private Azure OpenAI GPT4 endpoint The `run.sh` script will provision a new Azure OpenAI service in your Azure account, and then isolate it from both the internet and any other services by having all network access disabled: ![Azure OpenAI with network disabled](/blog/secure-llm-connections/azure-disable-network.png) It then creates a [private endpoint](https://learn.microsoft.com/en-us/azure/ai-services/cognitive-services-virtual-networks?tabs=portal#use-private-endpoints) that we can use to securely access the OpenAI service over Azure Private Link. ![Azure private endpoint](/blog/secure-llm-connections/azure-ai-private-endpoint.png) The reason we're using the Azure OpenAI service here is [because of the level of isolation and privacy Azure OpenAI is able to offer](https://learn.microsoft.com/en-us/legal/cognitive-services/openai/data-privacy?tabs=azure-portal). Our data isn't available to other customers, isn't shared with OpenAI, and isn't used to improve models. ## Setup Ockam ### !!steps The first step to using Ockam is [signing up and creating a new account](https://www.ockam.io/signup ). Once you're signed up, you can install Ockam Command on your local workstation and enroll that workstation as an administrator of your project. ```bash ! tour curl --proto '=https' --tlsv1.2 -sSfL \ https://install.command.ockam.io \ | bash && source "$HOME/.ockam/env" ``` This will install the Ockam Command and source in the required environment settings. ### !!steps Once Ockam Command installation is complete the `run.sh` script runs `ockam enroll`: ```bash ! tour ockam enroll ``` Wrapped up in this single command are several steps that will bootstrap your first project and get you ready to go. It will: * Generate an Ockam Identity and store its secret keys in a file system based Ockam Vault. * Create an account with Ockam Orchestrator. * Provision a trial Space and Project in the Orchestrator. * Make your Ockam Identity the administrator of your new Project. ### !!steps Create an enrollment ticket for Azure ![Ockam node in Azure](/blog/secure-llm-connections/minimal-final-arch-azure.png) Each node that we want add needs an enrollment ticket to be admitted into our project. We're going to need two nodes, one that runs alongside the Azure OpenAI service and another at the end of our portal running next to Jupyter Lab on our workstation. That means we're going to create a total of two tickets. ```shell ! tour ockam project ticket \ --usage-count 1 --expires-in 60m \ --attribute azure-openai-outlet \ --relay openai-relay ``` ### !!steps In this command we've set the new ticket to expire in `6o minutes`, and a usage count of `1`. A single usage ticket means there is low risk associated with mishandling this ticket as once it's used the ticket is no longer valid and is worthless; there's no means for an attacker to re-use it like an API token to access any system. ```shell ! tour ockam project ticket \ # !focus(1:1) --usage-count 1 --expires-in 60m \ --attribute azure-openai-outlet \ --relay openai-relay ``` ### !!steps The Project Membership Credential that's issued will include attributes that will be cryptographically attested to by the Project's Membership Authority. You can use these attributes to apply policies with Attribute Based Access Controls (ABAC) to allow or restrict specific actions and communication paths between nodes. In this example we assign a single attribute of `azure-openai-outlet`. ```bash ! tour ockam project ticket \ --usage-count 1 --expires-in 60m \ # !focus(1:1) --attribute azure-openai-outlet \ --relay openai-relay ``` ### !!steps The `--relay openai-relay` flag is a shortcut for creating a policy that allows this node to create a relay at the address `openai-relay`. This relay will allow your clients to establish a secure end-to-end encrypted connection to the Azure OpenAI service, without requiring you to expose any endpoints to the public Internet. The output of this command is temporarily stored in a variable that we pass to the Azure setup scripts. ```bash ! tour ockam project ticket \ --usage-count 1 --expires-in 60m \ --attribute azure-openai-outlet \ # !focus(1:1) --relay openai-relay ``` ### !!steps Create an enrollment ticket for Jupyter Lab ![Ockam node in Jupyter](/blog/secure-llm-connections/minimal-final-arch-jupyter.png) The node that is connecting as a client to the Azure OpenAI service, in this case the node running in our Docker container with Jupyter Lab, also needs a ticket to enroll with our Ockam Project. ```bash ! tour ockam project ticket \ --usage-count 1 --expires-in 60m \ --attribute azure-openai-inlet \ --tls ``` ### !!steps We use the same usage count of `1` and set an expiry of `60 minutes` to ensure this ticket can't be misused. ```bash ! tour ockam project ticket \ # !focus(1:1) --usage-count 1 --expires-in 60m \ --attribute azure-openai-inlet \ --tls ``` ### !!steps We again apply an attribute that we will use to restrict access using Attribute Based Access Control (ABAC) policies later. Any node that enrolls with this ticket will have the attribute `azure-openai-inlet` as part of their cryptographically attested credentials. ```bash ! tour ockam project ticket \ --usage-count 1 --expires-in 60m \ # !focus(1:1) --attribute azure-openai-inlet \ --tls ``` ### !!steps The communication between Ockam Nodes happens over and end-to-end mutually encrypted secure channel. If you want to also encrypt the communication to the port your node makes available an Ockam Node can present a TLS certificate so the final hop of communication within your infrastructure will also happen over an encrypted connection. This may not entirely be required if all communication is happening within `localhost`, though some client SDKs may enforce communication over HTTPS so this is a way to meet those requirements. Passing `--tls` here grants this node permission to generate and serve a TLS certificate once it enrolls. ```bash ! tour ockam project ticket \ --usage-count 1 --expires-in 60m \ --attribute azure-openai-inlet \ # !focus(1:1) --tls ``` ## Start an Ockam node next to Azure OpenAI ### !!steps ![Ockam node in Azure](/blog/secure-llm-connections/minimal-final-arch-azure.png) The `run.sh` will start a new virtual machine within Azure that runs an Ockam Node. Ockam Command will process this configuration file after the node starts, with the first instruction being to create a relay with the name `openai-relay`. This relay will allow any client to establish a secure end-to-end encrypted connection to our node running in Azure. ```javascript ! tour { // !focus(1:1) "relay": "openai-relay", "tcp-outlet": { "to": "ockam-ex-az-openai-aicorp-7394b13aec7d40c9ee70-openai.openai.azure.com:443", "allow": "azure-openai-inlet", "tls": true } } ``` ### !!steps The configuration specifies that the node will create a TCP outlet. An outlet decrypts any messages received by the node and then forwards them to the specified `to` address. The address here is the fully qualified domain name to the private endpoint we created for the Azure OpenAI service. ```javascript ! tour { "relay": "openai-relay", // !focus(1:3) "tcp-outlet": { "to": "ockam-ex-az-openai-aicorp-7394b13aec7d40c9ee70-openai.openai.azure.com:443", "allow": "azure-openai-inlet", "tls": true } } ``` ### !!steps This policy applies Attribute Based Access Controls (ABAC) to only allow other nodes that have the attribute `azure-openai-inlet` as part of their attested credentials. The system denies access to nodes without the `azure-openai-inlet` attribute. Communication between nodes is not only mutually authenticated, but also mutually _authorized_. We'll make a similar policy on the other node that verifies the server-side has the attributes we expect. ```javascript ! tour { "relay": "openai-relay", "tcp-outlet": { "to": "ockam-ex-az-openai-aicorp-7394b13aec7d40c9ee70-openai.openai.azure.com:443", // !focus(1:1) "allow": "azure-openai-inlet", "tls": true } } ``` ## Start an Ockam node next to your data ### !!steps ![Ockam node in Docker](/blog/secure-llm-connections/minimal-final-arch-jupyter.png) This side of our Ockam Portal does not need to create a relay as it will use the one created by the TCP Outlet that is running within Azure. Instead we create a TCP _Inlet_, which will listen for inbound request on port `443`. ```javascript ! tour { // !focus(1:2) "tcp-inlet": { "from": "0.0.0.0:443", "via": "openai-relay", "allow": "azure-openai-outlet", "tls": true } } ``` ### !!steps Any packets received by the inlet will be forwarded _via_ the relay named `openai-relay`, and ultimately arrive at the TCP outlet that was created earlier. ```javascript ! tour { "tcp-inlet": { "from": "0.0.0.0:443", // !focus(1:1) "via": "openai-relay", "allow": "azure-openai-outlet", "tls": true } } ``` ### !!steps This portal will only be permitted to be established if the TCP outlet at the other end has the cryptographically attestable credential of `azure-openai-outlet`. ```javascript ! tour { "tcp-inlet": { "from": "0.0.0.0:443", "via": "openai-relay", // !focus(1:1) "allow": "azure-openai-outlet", "tls": true } } ``` ### !!steps A TLS certificate will be served by this inlet, allowing any clients that connect to communicate with the node using HTTPS. ```javascript ! tour { "tcp-inlet": { "from": "0.0.0.0:443", "via": "openai-relay", "allow": "azure-openai-outlet", // !focus(1:1) "tls": true } } ``` Now that the two nodes are running, the Azure OpenAI service is available on `localhost:443` within the Docker container running our local workstation. All the systems are still private, we get the benefit of Azure running the models on infrastructure that is best suited for running LLMs, but everything looks like it's running locally. Even though explaining how all the magic works takes a while it can take only a few milliseconds to establish an Ockam Portal. These connections are so quick and easy to setup! ## Next steps If you've not been following along by running the demo then this is a great time to [sign up for Ockam](/signup) and give it a try. If you'd like to explore some other capabilities of Ockam I'd recommend: * [Adding security as a feature in your SaaS product](/blog/building-secure-saas-platforms) * [Connecting Snowflake to Everything](/for/snowflake)

Quickstarts

Secure & private connections to enterprise LLMs

Glenn Gillen

VP of Product, GTM

Secure & private connections to enterprise LLMs

Glenn Gillen

VP of Product, GTM