{/* */} {/* */} {/* */} {/* */} If your business is using Kafka there is already _a lot_ of messages travelling through your brokers. If you're also a Snowflake customer it would be great if those messages could make their way into your Snowflake data cloud. Kafka is in your private network, Snowflake in their cloud, and getting data between them isn't entirely straight-foward. It'd be much easier if those two systems could look and feel like they were next to each other. Today, we're excited to introduce a solution that makes this vision a reality: the Pull from Kafka Connector! ## Quick links * [Pull from Kafka Connector in the Snowflake Marketplace](https://app.snowflake.com/marketplace/listing/GZTYZ723Y3S7D/ockam-pull-from-kafka-connector) * [Ockam Node for Amazon MSK in the AWS Marketplace](https://aws.amazon.com/marketplace/pp/prodview-iztqd27voc4xg) ## Snowflake 💙 Apache Kafka Apache Kafka (and Kafka-compatible alternatives) is the system of choice for building flexible, scalable, and reliable streaming platforms to connect data producers and consumers. It's often used as the central nervous system for real-time data pipelines and streaming applications. Snowflake is The Data Cloud and the place to support workloads such as data warehouses, data lakes, data science / ML / AI, and even cybersecurity. This centralization brings a huge amount of convenience through breaking down data silos and allowing teams to make smart data-informed decisions. Connecting your Kafka broker to Snowflake can be problematic depending on your network topology. It would be convenient to give the broker a public address, but that's a significant increase in risk for a system that handles a lot of important data. Managing IP allow lists and updating firewall ingress rules improves security but can be cumbersome to manage. Alternatives like PrivateLink are better, but they too can be cumbersome to setup and require your systems to be on the same public cloud and in the same region. In this post I'm going to show you how to securely connect Snowflake to your private Apache Kafka broker, in just a few minutes. We will: * Setup a managed Kafka cluster in AWS * Prepare Snowflake to receive data from Kafka * Connect Snowflake to Kafka with a private encrypted connection - without needing to expose either system to the public internet! The final architecture diagram will look like this: ![Snowflake push to Kafka](/blog/snowflake-push-kafka/snowflake-push-kafka.png) ## Amazon Managed Streaming for Kafka (MSK) We're going to provision an MSK cluster so we can see an end-to-end experience of data moving from Snowflake to Kafka. If you have an existing Kafka broker you're able to use you can skip this step. ### Create an MSK cluster ### !!steps Within your [AWS Console](https://console.aws.amazon.com/) search for `MSK` in the search field at the top and select the matching result. Visit the `Clusters` screen, and then click `Create Cluster`. The `Quick Create` option provides a good set of defaults for creating a Kafka cluster, so unless you've previous knowledge or experience to know you might want something different I'd suggest just confirming the details and then clicking `Create cluster` at the bottom of the screen. Once you've started the cluster creation it may take about 15 minutes for provisioning to complete and for your broker to be available. ![!tour Select a warehouse](/blog/snowflake-push-kafka/create-msk.png) ## Connect Snowflake to Kafka ### Create a table to capture data Run the code here in a Snowflake worksheet. It will create a new table. ```sql CREATE OR REPLACE TABLE CUSTOMERS ( key STRING, id INTEGER, name STRING, email STRING ); ``` We've created a table, Kafka is running. It's now time to connect the two! The next stage is going to complete the picture below, by creating a point-to-point connection between the two systems — without the need to expose any systems to the public internet! ![Snowflake push to Amazon AWS Kafka](/blog/snowflake-push-kafka/aws-complete.png) ### Launch the Snowflake Native App ### !!steps The Snowflake Pull from Kafka Connector by Ockam is [available in the Snowflake Marketplace](https://app.snowflake.com/marketplace/listing/GZTYZ723Y3S7D/ockam-pull-from-kafka-connector). ![!tour Get the app](/blog/snowflake-pull-kafka/app-store.png) ### !!steps Select a warehouse The first screen you're presented with will ask you to select the warehouse to utilize to activate the app. ![!tour Select a warehouse](/blog/snowflake-pull-kafka/select-warehouse.png) ### !!steps Grant account privileges Click the `Grant` button to the right of this screen. The app will then be automatically granted permissions to create a warehouse and create a compute pool. ![!tour Grant account privileges](/blog/snowflake-pull-kafka/grant-account.png) ### !!steps Activate app Once the permissions grants complete, an `Activate` button will appear. Click it and the activation process will begin. ![!tour Activate app](/blog/snowflake-pull-kafka/activate-app.png) ### !!steps Launch app After the app activates you'll see a page that summarizes the privileges that the application now has. There's nothing we need to review or update on these screens yet, so proceed by clicking the `Launch app` button. ![!tour Launch app](/blog/snowflake-pull-kafka/launch-app.png) ### Setup Ockam ### !!steps Download Ockam Command Run the following command on your local workstation: ```bash ! tour curl --proto '=https' --tlsv1.2 -sSfL \ https://install.command.ockam.io \ | bash && source "$HOME/.ockam/env" ``` This will install the Ockam Command and source in the required environment settings. If you've installed Ockam Command before you can skip this step. ### !!steps Setup admin account Once Ockam Command installation is complete you can run: ```bash ! tour ockam enroll ``` Wrapped up in this single `ockam enroll` command are several steps that will bootstrap your first project and get you ready to go. It will: * Generate an Ockam Identity and store its secret keys in a file system based Ockam Vault. * Create an account with Ockam Orchestrator. * Provision a trial Space and Project in the Orchestrator. * Make your Ockam Identity the administrator of your new Project. ### !!steps Generate enrollment ticket for Kafka In this section we're going to provision an Ockam node that will run alongside our Kafka broker, and provide one of the ends of our point-to-point connection. ![Amazon MSK setup screen](/blog/snowflake-push-kafka/aws-kafka-setup.png) We need to generate an enrollment ticket to allow a new Ockam Node to join the project that was just created. This node will run alongside the Kafka broker, in the network where the Kafka is running: ```bash ! tour ockam project ticket \ --usage-count 1 --expires-in 24h \ --attribute kafka \ --relay kafka > kafka.ticket ``` ### !!steps In this command we've set the new ticket to expire in `24 hours`, and a usage count of `1`. This means the generated ticket is valid for a single use, and that it's valid for 24 hours. A single usage ticket means there is low risk associated with mishandling this ticket after use; there's no means for an attacker to re-use it like an API token to access any system. ```shell ! tour ockam project ticket \ # !focus(1:1) --usage-count 1 --expires-in 24h \ --attribute kafka \ --relay kafka > kafka.ticket ``` ### !!steps The Project Membership Credential that's issued will include attributes that will be cryptographically attested to by the Project's Membership Authority. You can use these attributes to apply policies with Attribute Based Access Controls (ABAC) to allow or restrict specific actions and communication paths between nodes. In this example we assign a single attribute of `kafka`. ```bash ! tour ockam project ticket \ --usage-count 1 --expires-in 24h \ # !focus(1:1) --attribute kafka \ --relay kafka > kafka.ticket ``` ### !!steps The `--relay kafka` flag is a shortcut for creating a policy that allows this node to create a relay at the address `kafka`. This relay will allow your Snowflake warehouse to establish a secure end-to-end encrypted connection to your Kafka broker, without requiring you to expose any endpoints to the public Internet. As the final part of this command we pipe the generated ticket to a file named `kafka.ticket`. ```bash ! tour ockam project ticket \ --usage-count 1 --expires-in 24h \ --attribute kafka \ # !focus(1:1) --relay kafka > kafka.ticket ``` ### !!steps Create a configuration file We're going to pass the configuration for our Ockam Node as a JSON file, so save the configuration to the right into a file named `kafka.json`. ```javascript ! tour { "http-server-port": 23345, "relay": "kafka", "kafka-outlet": { "bootstrap-server": "$BOOTSTRAP_SERVER_WITH_PORT", "allow": "snowflake" } } ``` ### !!steps Ockam Command will process the configuration file after the node starts, with the first instruction being to create a relay with the name `kafka`. This relay will allow Snowflake to establish a secure end-to-end encrypted connection to your Kafka broker. It provides the ability to have entirely isolated infrastructure running within a private network, without requiring you to expose any endpoints to the public Internet and no configuring of allow lists to custom network routes. ```javascript ! tour { "http-server-port": 23345, // !focus(1:1) "relay": "kafka", "kafka-outlet": { "bootstrap-server": "$BOOTSTRAP_SERVER_WITH_PORT", "allow": "snowflake" } } ``` ### !!steps An outlet decrypts any messages received by the node and then forwards them to the specified broker address. ```javascript ! tour { "http-server-port": 23345, "relay": "kafka", // !focus(1:4) "kafka-outlet": { "bootstrap-server": "$BOOTSTRAP_SERVER_WITH_PORT", "allow": "snowflake" } } ``` ### !!steps In this example we're giving the interpolated value from the environment variable `BOOTSTRAP_SERVER_WITH_PORT`, which we'll set in a later step when we start the node. ```javascript ! tour { "http-server-port": 23345, "relay": "kafka", "kafka-outlet": { // !focus(1:1) "bootstrap-server": "$BOOTSTRAP_SERVER_WITH_PORT", "allow": "snowflake" } } ``` ### !!steps We will setup the Kafka Inlet that runs alongside our Snowflake warehouse in a moment. Before we do that though, we can define a policy to restrict what nodes can connect to our outlet. This policy applies Attribute Based Access Controls (ABAC) to allow other nodes that have the attribute `snowflake` as part of their attested credentials. The system denies access to nodes without the `snowflake` attribute. ```javascript ! tour { "http-server-port": 23345, "relay": "kafka", "kafka-outlet": { "bootstrap-server": "$BOOTSTRAP_SERVER_WITH_PORT", // !focus(1:1) "allow": "snowflake" } } ``` ### Option 1 - Run an Ockam node next to Kafka via the AWS Marketplace ### !!steps The Ockam Node for Amazon MSK is a streamlined way to provision a managed Ockam Node within your private AWS VPC. If you'd prefer the ability to deploy your Ockam Node yourself (for example on an existing machine or in a container) jump to our instructions on how to [run an Ockam Node manually](#option-2---run-an-ockam-node-next-to-kafka-manually). To deploy the node that will allow Snowflake to reach your Kafka broker visit the [Ockam Node for Amazon MSK listing in the AWS Marketplace](https://aws.amazon.com/marketplace/pp/prodview-iztqd27voc4xg), and click the `Continue to Subscribe` button, and `Continue to Configuration`. On the configuration page choose the region that your Amazon MSK cluster is running in, and then click `Continue to Launch` followed by `Launch`. ![!tour Ockam node for Amazon MSK](/blog/snowflake-push-kafka/aws-marketplace-msk.png) ### !!steps Enter stack details The initial Create Stack screen pre-fills the fields with the correct information for your node, so you can press `Next` to proceed. ![!tour Ockam node for Amazon MSK - create stack screen](/blog/snowflake-push-kafka/create-stack.png) ### !!steps Enter node configuration This screen has important details to you need to fill in: * **Stack name:** Give this stack a recognisable name, you'll see this in various locations in the AWS Console. It'll also make it easier to clean these resources up later if you wish to remove them. * **VPC ID:** The ID of the Virtual Private Cloud network to deploy the node in. Make sure it's the same VPC where you've deployed your Amazon MSK cluster. * **Subnet ID:** Choose one of the subnets within your VPC, ensure MSK has a broker address available in that subnet. * **Enrollment ticket:** Copy the contents of the `kafka.ticket` file we created earlier and paste it in here. * **Amazon MSK Bootstrap Server with Port:** In the client configuration details for your Amazon MSK cluster you will have a list of bootstrap servers. Copy the `hostname:port` value for the Private endpoint that's in the same subnet you chose above. * **JSON Node Configuration:** Copy the contents of the `kafka.json` file we created earlier and paste it in here. We've now completed the highlighted part of the diagram below, and our Kafka broker node is waiting for another node to connect. ![Amazon MSK setup](/blog/snowflake-push-kafka/aws-kafka-setup.png) ![!tour Ockam node for Amazon MSK - node configuration screen](/blog/snowflake-push-kafka/stack-details.png) ### Option 2 - Run an Ockam Node next to Kafka manually ### !!steps Install Ockam Command The system where you plan to run your Ockam Node will need to be able to reach your Kafka cluster (so ensure your network access rules allow it) and needs to have Ockam Command installed. Run the following command to install and setup the environment. ```bash ! tour curl --proto '=https' --tlsv1.2 -sSfL \ https://install.command.ockam.io \ | bash && source "$HOME/.ockam/env" ``` ### !!steps Start the Kafka node Now that Ockam Command is available, move the `kafka.json` and `kafka.ticket` files we created earlier onto this system. The `kafka.json` file references an environment variable named `BOOTSTRAP_SERVER_WITH_PORT`. Ensure this environment variable is set by either exporting it or defining it inline when starting `ockam`. Be sure to replace the `kafka-broker-address.here:port` value with the hostname and port of your actual Kafka broker. You can now start your Ockam Node with the following command: ```javascript ! tour BOOTSTRAP_SERVER_WITH_PORT="kafka-broker-address.here:port" \ ockam node create \ kafka.json \ --enrollment-ticket kafka.ticket ``` ### Connect the Ockam node in Snowflake ### !!steps ![Setting up Ockam Node as native Snowflake app](/blog/snowflake-push-kafka/aws-snowflake-setup.png) One end of our connection is now setup, it's time to connect the Snowflake end. We need to generate an enrollment ticket to allow another Ockam Node to join our project. This node will run in our Snowflake warehouse: ```bash ockam project ticket \ --usage-count 1 --expires-in 2h \ --attribute snowflake > snowflake.ticket ``` ![!tour Create Snowflake ticket](/blog/snowflake-pull-kafka/snowflake-ticket.png) ### !!steps As we did with the earlier ticket, we've reduced the risk associated with mishandling a ticket by restricting it's permitted usage window. We've again set a usage count of `1`, but with a tighter expiry time of just `2 hours`. ```bash ! tour ockam project ticket \ # !focus(1:1) --usage-count 1 --expires-in 2h \ --attribute snowflake > snowflake.ticket ``` ### !!steps This credential will include an attested attribute of `snowflake`. You'll recall in an earlier step we applied a policy to say that nodes in our project must have an attribute of `snowflake` to connect to the outlet. The node that enrolls with this ticket will also have the attribute `snowflake`, and so it will be able to communicate with our Kafka node. The ticket is then saved to a file called `snowflake.ticket`. ```bash ! tour ockam project ticket \ --usage-count 1 --expires-in 2h \ # !focus(1:1) --attribute snowflake > snowflake.ticket ``` ### !!steps Configure connection details Take the contents of the file `snowflake.ticket` that we just created and paste it into "Provide the above Enrollment Ticket" form field in the "Configure app" setup screen in Snowflake. ![!tour Create Snowflake ticket](/blog/snowflake-pull-kafka/snowflake-ticket.png) ### !!steps Grant privileges To be able to authenticate with Ockam Orchestrator and then discover the route to our outlet, the Snowflake app needs to allow outbound connections to your Ockam project. Toggle the `Grant access to egress and reach your Project` and approve the connection by pressing `Connect`. Toggle the `Grant access to the tables where Kafka messages must be inserted` and select the `CUSTOMERS` table. ![!tour Grant egress](/blog/snowflake-pull-kafka/grant-privileges.png) ### !!steps Map Kafka topics to Snowflake tables Enter the name of the Kafka topic you want to map to the Snowflake table. In this example, the topic is `customers`. Check "Messages are encoded with a schema" if you have a schema registry and the messages are encoded with a schema. The configuration to use to Launch Ockam node for Amazon MSK will need to be updated to include the schema registry details. Update `$SCHEMA_REGISTRY_ADDRESS` with the address of the schema registry. Make sure the ockam node has access to the schema registry. ```bash { "http-server-port": 23345, "relay": "kafka", "kafka-outlet": { "bootstrap-server": "$BOOTSTRAP_SERVER_WITH_PORT", "allow": "snowflake" }, "tcp-outlet": { "to": "$SCHEMA_REGISTRY_ADDRESS:9081", "from": "schema_registry", "allow": "snowflake" } } ``` Update other options from default values if needed. ![!tour Grant egress](/blog/snowflake-pull-kafka/map-topics.png) With that, we've completed the last step in the setup. We've now got a complete point-to-point connection that allows our Snowflake warehouse to securely pull data through to our private Kafka broker. ![Snowflake push to Kafka setup complete](/blog/snowflake-push-kafka/aws-complete.png) ## Seeing it in action Any updates to your data in your Kafka topic will now create a new row in your Snowflake table. Post the below message to the Kafka topic to verify the setup. Replace `$BROKER_ADDRESS` with your actual Kafka broker address, and ensure the topic name (`customers` in this example) matches the one you've configured in your Snowflake Pull from Kafka Connector setup ```bash echo '{"key": "customer123", "id": 1001, "name": "John Doe", "email": "john.doe@example.com"}' | \ kafka-console-producer --broker-list $BROKER_ADDRESS:9092 --topic customers ``` The Snowflake connector will then pull these messages from Kafka and insert them into your `CUSTOMERS` table, mapping the JSON fields to the corresponding columns. ## Wrap up It's all done! In the course of a few minutes we've been able to: * Setup an Ockam node next to our Kafka broker. * Start an Ockam node within Snowflake. * Establish an Ockam Portal between our nodes — a secure point-to-point connection that is mutually authenticated with end-to-end encryption. With regular and automatic rotation of the encryption keys. * Then use the secure portal to consume messages from our private Kafka broker and load them directly into a Snowflake table. We've been able to achieve all of this without the need to expose our Kafka broker to the public internet, update firewall ingress rules, setup a VPN, or manage IP allow lists. If you'd like to explore some other capabilities of Ockam I'd recommend: * [Encrypting data _through_ Kafka](https://docs.ockam.io/portals/kafka) * [Pushing data from Snowflake to Kafka](/blog/snowflake-push-kafka) * [Zero-trust data streaming with Redpanda Connect](/blog/redpanda_connect_with_ockam#connect-secure-and-streamall-in-one-simple-platform) * [Adding security as a feature in your SaaS product](/blog/building-secure-saas-platforms)

Quickstarts