Connect to Apache Kafka running in Docker – 连接到运行在Docker中的Apache Kafka

1. Overview

1.概述

Apache Kafka is a very popular event streaming platform that is used with Docker frequently. Often, people experience connection establishment problems with Kafka, especially when the client is not running on the same Docker network or the same host. This is primarily due to the misconfiguration of Kafka’s advertised listeners.

Apache Kafka是一个非常流行的事件流平台，经常与Docker一起使用。经常有人遇到Kafka的连接建立问题，特别是当客户端不在同一Docker网络或同一主机上运行时。这主要是由于Kafka的广告监听器的配置错误造成的。

In this tutorial, we will learn how to configure the listeners so that clients can connect to a Kafka broker running within Docker.

在本教程中，我们将学习如何配置监听器，以便客户端能够连接到在Docker中运行的Kafka代理。

2. Setup Kafka

2.Kafka设置

Before we try to establish the connection, we need to run a Kafka broker using Docker. Here’s a snippet of our docker-compose.yaml file:

在我们尝试建立连接之前，我们需要运行一个使用Docker的Kafka代理。下面是我们的docker-compose.yaml文件的一个片段。

version: '2'
services:
  zookeeper:
    container_name: zookeeper
    networks: 
      - kafka_network
    ...
  
  kafka:
    container_name: kafka
    networks: 
      - kafka_network
    ports:
      - 29092:29092
    environment:
      KAFKA_LISTENERS: EXTERNAL_SAME_HOST://:29092,INTERNAL://:9092
      KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka:9092,EXTERNAL_SAME_HOST://localhost:29092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,EXTERNAL_SAME_HOST:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
     ... 

networks:
  kafka_network:
    name: kafka_docker_example_net

Here, we defined two must-have services – Kafka and Zookeeper. We also defined a custom network – kafka_docker_example_net, which our services will use.

在这里，我们定义了两个必须的服务–Kafka和Zookeeper。我们还定义了一个自定义的网络–kafka_docker_example_net，我们的服务将使用该网络。

We will look at the KAFKA_LISTENERS, KAFKA_ADVERTISED_LISTENERS, and KAFKA_LISTENER_SECURITY_PROTOCOL_MAP properties in more detail later.

我们将在后面更详细地查看KAFKA_LISTENERS、KAFKA_ADVERTISED_LISTENERS,和KAFKA_LISTENER_SECURITY_PROTOCOL_MAP属性。

With the above docker-compose.yaml file, we start the services:

通过上述docker-compose.yaml文件，我们启动服务。

docker-compose up -d
Creating network "kafka_docker_example_net" with the default driver
Creating zookeeper ... done
Creating kafka ... done

Also, we will be using the Kafka console producer utility as a sample client to test the connection to the Kafka broker. To use the Kafka-console-producer script without Docker, we need to have Kafka downloaded.

另外，我们将使用Kafka 控制台生产者实用程序作为样本客户端来测试与Kafka代理的连接。要在没有Docker的情况下使用Kafka-console-producer脚本，我们需要下载Kafka。

3. Listeners

3.听众

Listeners, advertised listeners, and listener protocols play a considerable role when connecting with Kafka brokers.

在与Kafka经纪人连接时，监听器、广告监听器和监听器协议发挥了相当大的作用。

We manage listeners with the KAFKA_LISTENERS property, where we declare a comma-separated list of URIs, which specify the sockets that the broker should listen on for incoming TCP connections.

我们用KAFKA_LISTENERS属性来管理监听器，在这里我们声明一个用逗号分隔的URI列表，它指定了代理服务器应该监听的TCP连接的套接字。

Each URI comprises a protocol name, followed by an interface address and a port:

每个URI包括一个协议名称，后面是一个接口地址和一个端口。

EXTERNAL_SAME_HOST://0.0.0.0:29092,INTERNAL://0.0.0.0:9092

Here, we specified a 0.0.0.0 meta address to bind the socket to all interfaces. Further, EXTERNAL_SAME_HOST and INTERNAL are the custom listener names that we need to specify when defining listeners in the URI format.

在这里，我们指定了一个0.0.0.0元地址，将套接字与所有接口绑定。此外，EXTERNAL_SAME_HOST和INTERNAL是我们在以URI格式定义监听器时需要指定的自定义监听器名称。

3.2. Bootstrapping

3.2.引导

For initial connections, Kafka clients need a bootstrap server list where we specify the addresses of the brokers. The list should contain at least one valid address to a random broker in the cluster.

对于初始连接，Kafka客户端需要一个引导服务器列表，在这里我们指定经纪商的地址。该列表应该包含至少一个有效的地址给集群中的一个随机经纪人。

The client will use that address to connect to the broker. If the connection is successful, the broker will return the metadata about the cluster, including the advertised listener lists for all the brokers in the cluster. For subsequent connections, the clients will use that list to reach the brokers.

客户端将使用该地址来连接到经纪人。如果连接成功，经纪商将返回关于集群的元数据，包括集群中所有经纪商的广告听众列表。对于后续的连接，客户端将使用该列表来联系经纪商。

3.3. Advertised Listeners

3.3.宣传的听众

Just declaring listeners is not enough because it’s just a socket configuration for the broker. We need a way to tell the clients (consumers and producers) how to connect to Kafka.

仅仅声明监听器是不够的，因为这只是经纪人的一个套接字配置。我们需要一种方法来告诉客户（消费者和生产者）如何连接到Kafka。

This is where advertised listeners come into the picture with the help of the KAFKA_ADVERTISED_LISTENERS property. It has a similar format as the listener’s property:

这就是广告监听器在KAFKA_ADVERTISED_LISTENERS属性的帮助下进入画面的地方。它的格式与监听器的属性类似。

<listener protocol>://<advertised host name>:<advertised port>

<监听器协议>://<广告主机名>:<广告端口>

The clients use the addresses specified as advertised listeners after the initial bootstrapping process.

客户端在初始引导过程后使用指定的地址作为广告听众.。

3.4. Listener Security Protocol Map

3.4.听众安全协议图

Apart from listeners and advertised listeners, we need to tell the clients about the security protocols to use when connecting to Kafka. In the KAFKA_LISTENER_SECURITY_PROTOCOL_MAP, we map our custom protocol names to valid security protocols.

除了监听器和广告监听器之外，我们还需要告诉客户端在连接Kafka时要使用的安全协议。在KAFKA_LISTENER_SECURITY_PROTOCOL_MAP中，我们将我们的自定义协议名称映射到有效的安全协议。

In the configuration in the previous section, we declared two custom protocol names – INTERNAL and EXTERNAL_SAME_HOST. We can name them as we want, but we need to map them to valid security protocols.

在上一节的配置中，我们声明了两个自定义协议名称–INTERNAL和EXTERNAL_SAME_HOST。我们可以随心所欲地命名它们，但我们需要将它们映射到有效的安全协议。

One of the security protocols we specified is PLAINTEXT, which means that the clients don’t need to authenticate with the Kafka broker. Also, the data exchanged is not encrypted.

我们指定的安全协议之一是PLAINTEXT，这意味着客户端不需要与Kafka代理进行认证。另外，交换的数据也没有加密。

4. Client Connecting from the Same Docker Network

4.客户端从同一Docker网络连接

Let’s start the Kafka console producer from another container and try to produce messages to the broker:

让我们从另一个容器中启动Kafka控制台生产者，并尝试生产消息给代理。

docker run -it --rm --network kafka_docker_example_net confluentinc/cp-kafka /bin/kafka-console-producer --bootstrap-server kafka:9092 --topic test_topic
>hello
>world

Here, we are attaching this container to the existing kafka_docker_example_net network to communicate to our broker freely. We also specify the broker’s address –  kafka:9092 and the name of the topic, which will be created automatically.

在这里，我们将这个容器附加到现有的kafka_docker_example_net网络中，以便与我们的经纪人自由沟通。我们还指定了代理的地址–kafka:9092 和主题的名称，它将被自动创建。

We were able to produce the messages to the topic, which means that the connection to the broker was successful.

我们能够向主题产生消息，这意味着与经纪人的连接是成功的。

5. Client Connecting from the Same Host

5.客户端从同一主机连接

Let’s connect to the broker from the host machine when the client is not containerized. For external connection, we advertised EXTERNAL_SAME_HOST listener, which we can use to establish the connection from the host. From the advertised listener property, we know that we have to use the localhost:29092 address to reach Kafka broker.

当客户端没有被容器化时，让我们从主机连接到代理。对于外部连接，我们公布了EXTERNAL_SAME_HOST监听器，我们可以用它来从主机上建立连接。从广告的监听器属性中，我们知道我们必须使用localhost:29092地址来到达Kafka代理。

To test connectivity from the same host, we will use a non-Dockerized Kafka console producer:

为了测试来自同一主机的连接性，我们将使用一个非Dockerized Kafka控制台生产者。

kafka-console-producer --bootstrap-server localhost:29092 --topic test_topic_2
>hi
>there

Since we managed to produce the topic, it means that both the initial bootstrapping and the subsequent connection (where advertised listeners are used by the client) to the broker were successful.

由于我们成功地产生了主题，这意味着最初的引导和随后与经纪人的连接（客户使用广告监听器）都很成功。

The port number 29092 that we configured in docker-compose.yaml earlier made the Kafka broker reachable outside Docker.

我们之前在 docker-compose.yaml中配置的端口号29092使得Kafka代理可以在Docker之外到达。

6. Client Connecting from a Different Host

6.客户端从不同的主机连接

How would we connect to a Kafka broker if it’s running on a different host machine? Unfortunately, we can’t re-use existing listeners because they are only for the same Docker network or host connection. So instead, we need to define a new listener and advertise it:

如果一个Kafka代理运行在不同的主机上，我们要如何连接到它呢？不幸的是，我们不能重新使用现有的监听器，因为它们只针对同一个Docker网络或主机连接。因此，我们需要定义一个新的监听器，并将其公布。

KAFKA_LISTENERS: EXTERNAL_SAME_HOST://:29092,EXTERNAL_DIFFERENT_HOST://:29093,INTERNAL://:9092
KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka:9092,EXTERNAL_SAME_HOST://localhost:29092,EXTERNAL_DIFFERENT_HOST://157.245.80.232:29093
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,EXTERNAL_SAME_HOST:PLAINTEXT,EXTERNAL_DIFFERENT_HOST:PLAINTEXT

We created a new listener called EXTERNAL_DIFFERENT_HOST with security protocol PLAINTEXT and port 29093 associated. In KAFKA_ADVERTISED_LISTENERS, we also added the IP address of the cloud machine Kafka is running on.

我们创建了一个名为EXTERNAL_DIFFERENT_HOST的新监听器，安全协议PLAINTEXT和端口29093关联。在KAFKA_ADVERTISED_LISTENERS中，我们还添加了Kafka所运行的云端机器的IP地址。

We have to keep in mind that we can’t use localhost because we are connecting from a different machine (local workstation in this case). Also, port 29093 is published under the ports section so that it’s reachable outside Docker.

我们必须记住，我们不能使用localhost，因为我们是从不同的机器（本例中是本地工作站）连接的。另外，29093端口被发布在端口部分，这样它在Docker外部就可以到达。

Let’s try producing a few messages:

让我们试着制作一些信息。

kafka-console-producer --bootstrap-server 157.245.80.232:29093 --topic test_topic_3
>hello
>REMOTE SERVER

We can see that we were able to connect to the Kafka broker and produce messages successfully.

我们可以看到，我们能够连接到Kafka代理，并成功产生消息。

7. Conclusion

7.结语

In this article, we learned how to configure the listeners so that clients can connect to a Kafka broker running within Docker. We looked at different scenarios where the client was running on the same Docker network, same host, different host, etc. We saw that the configurations for listeners, advertised listeners, and security protocol maps determine the connectivity.

在这篇文章中，我们学习了如何配置监听器，以便客户端可以连接到运行在Docker中的Kafka代理。我们研究了不同的场景，其中客户端运行在同一Docker网络、同一主机、不同主机等。我们看到，监听器、广告监听器和安全协议图的配置决定了连接性。