Security¶

Authentication¶

Kerberos is the main authentication mechanism supported by both Kafka Brokers and Zookeeper [1]. Kafka Brokers also permit authentication via SSL client certificates.

Kafka REST Proxy and Schema Registry support authentication only via SSL client certificates. Kafka Connect does not support client authentication yet.

It is important to understand that as these components interact which other, they also are clients themselves. Whilst their server part may support authentication, their client part doesn’t always follow [2]. In later sections every component’s setup will be explained in detail.

Authorization¶

Authorization is handled by ACLs in ZooKeeper and Kafka Topics, enforced by the brokers. Although you will usually deal with ACLs in the brokers, Kafka topics’ settings are stored inside ZooKeeper, thus knowledge of and occasionally intervention in ZooKeeper ACLs is a requisite. Authorization requires authentication to the brokers and zookeeper.

The rest of the stack has limited support for ACLs as client to the ZooKeepers and the Brokers.

Although we include options for ACLs and perform some ACL management, we do not support ACLs at the moment due to their complex nature. Once we test them adequately we will provide a guide on how to safely use them and administer your services. Probably some manual intervention —documented here— will be needed to run an ACL setup.

Transport Security¶

Transport security is handled by TLS/SSL listeners on the Brokers, the Schema Registry and the Kafka REST Proxy. Kafka Connect does not offer SSL for its server part but can use SSL to connect to the brokers and the Schema Registry.

As of Confluent Platform 3.1.2 some workarounds and plaintext listeners are still needed, especially for the Schema Registry and Kafka Brokers (if Kafka REST is used). They are documented in the schema registry and broker sections below.

Authentication Support Matrix¶

These are the possible authentication scenarios Confluent Platform supports. All can be setup and managed within the Fast Data CSD. Please note that TLS provides always transport encryption and optionally authentication via client SSL certificates.

Client ▼ | Server ►	Zookeeper	Broker	Schema Registry	Kafka Connect	REST Proxy
Broker	NONE,SASL	NONE,SASL,TLS	not applicable	not applicable	not applicable
Schema Registry	NONE,SASL	NONE,SASL,TLS	NONE [3]	not applicable	not applicable
Connect	NONE,SASL	NONE,SASL,TLS	NONE,TLS	not applicable [4]	not applicable
REST Proxy	NONE,SASL	NONE [5]	NONE,TLS	not applicable	not applicable
Clients	NONE,SASL	NONE,SASL,TLS	NONE,TLS	NONE	NONE,TLS

Authentication and Transport Security¶

TLS/SSL for Transport Encryption¶

As TLS/SSL for transport encryption is a requisite for SSL client authentication, we will present it first. To enable TLS for the brokers, you should use the broker specific settings ssl_enabled, ssl.keystore.location, ssl.keystore.password and ssl.key.password. These settings are the same for each Cloudera Managed, JVM-based role that can act as a TLS server and probably you are already familiar with.

Enable and setup TLS/SSL for brokers.

Once TLS/SSL is setup, the brokers will use the TLS/SSL port instead of the default TCP Port and, depending on whether Kerberos is enabled, may require authentication via GSSAPI/Kerberos.

TLS/SSL Port setting for Brokers

As with Kerberos, you have to set your roles to use the appropriate protocol to connect to the brokers —SSL or SASL_SSL in this case. Also you have to set them with a truststore that contains your Certificate Authorities’ certificates in order to accept the certificate provided by the brokers. Please see Inter Broker Communication, Schema Registry Client, Kafka REST Proxy Client, and Kafka Connect Client.

Important

TLS/SSL can have a substantial impact in the performance of your Kafka cluster, maybe up to 30% or even more. You may want to see Force Plaintext Listener if there are roles or clients in your setup that can skip transport encryption.

SSL Client Authentication¶

Once you enabled TLS/SSL for transport encryption, you can enable client authentication as well. The necessary configuration options are ssl.client.auth which sets the brokers to require authentication via SSL and ssl.truststore.location, ssl.truststore.password which are used to provide to the brokers a list of Certificate Authorities that are trusted to sign client certificates.

Set Brokers to require authentication via client certificates.

Provide a list of CAs that are trusted to sign client certificates.

You have to set up the rest of the roles to use client certificates. Please refer to Schema Registry Client, Kafka REST Proxy Client, Kafka Connect Client subsections for more information. If you want to test your setup, see Verify Brokers SSL Authentication.

Force Plaintext Listener¶

The Fast Data CSD offers the option for an extra plaintext listener on the brokers; that is a listener with no security whatsoever. Such a setup may be justified when the unsecured listener is kept secured within your cluster via a firewall or other network configuration, so that only the other Fast Data roles or trusted clients have access to it.

Force an extra, plaintext, listener on the brokers.

Important

If you enable the force plaintext listener port without having set SASL or SSl for your brokers, your brokers will fail to start due to being set with two plaintext listeners.

It is mandatory if you want to use Kafka REST Proxy with secured brokers. It is especially useful when you enable transport encryption (TLS), which can have a significant impact on performance. The default listener still works on its default port and can be used by clients —or certain roles— to access the cluster in a secure way.

Other uses include when some trusted clients —i.e your applications— do not support the authentication mechanism you have enabled in your brokers or it is difficult to implement support for the authentication in your CI/CD system.

Inter Broker Communication¶

You can set the protocol the brokers will use to communicate with each other via security.inter.broker.protocol.

Set the inter broker security protocol.

The brokers communicate via the same listeners that are available to clients. Due to replication, they may exchange large quantities of data at high throughput. It is often advised to avoid TLS/SSL for inter-broker communication due to the performance impact. If you have followed the Force Plaintext Listener section, you can safely choose PLAINTEXT in the setting above and the brokers will use the correct port.

Should you choose to utilize SSL between the brokers, you should also set a SSL truststore file, so the brokers can accept each other’s certificate.

Provide a list of CAs that are trusted to sign client certificates.

If you want to test your setup, see Verify Kerberos or Verify Brokers SSL Authentication.

Schema Registry Client¶

For Schema Registry you should choose the appropriate protocol for connecting to the brokers via the kafkastore.security.protocol setting. GSSAPI/Kerberos authentication does not require any other setting than this.

Schema Registry security protocol for communication with the brokers.

In order to be able to authenticate via client certificates, you have to enable SSL for the Schema Registry and provide it with a keystore that contains the certificate to use for authentication. Also in any SSL scenario (SSL, SASL_SSL), you should set up a truststore, so the Schema Registry can verify the brokers’ certificates.

Schema Registry SSL settings.

Kafka REST Proxy Client¶

Kafka REST Proxy still uses the old consumer, so it doesn’t support neither authentication nor transport security to the brokers. Please see Force Plaintext Listener about enabling a plaintext listener to use with REST Proxy. We follow the development of Confluent Platform closely and release an update when this issue is addressed.

Kafka Connect Client¶

For Kafka Connect workers you should choose the appropriate protocol for connecting to the brokers via the security.protocol setting. GSSAPI/Kerberos authentication does not require any other setting than this.

Kafka Connect worker security protocol for communication with the brokers.

In order to be able to authenticate via client certificates, you have to enable SSL for Kafka Connect workers and provide them with a keystore that contains the certificate to use for authentication. Also in any SSL scenario (SSL, SASL_SSL), you should set up a truststore, so the workers can verify the brokers’ certificates.

Kafka Connect SSL settings.

TLS/SSL for Transport Encryption¶

To setup Schema Registry to serve its endpoint via TLS/SSL, enable SSL and provide a SSL keystore. Then set the ssl_listener option.

Enable SSL for Schema Registry and provide a SSL keystore

Enable TLS/SSL listener for Schema Registry

Secondary HTTP Listener¶

An important issue even with the latest iteration of the Schema Registry, is that it does not support https for communication between its instances [GHI_1]. To work around this shortcoming, by default the Fast Data CSD enables a second, plaintext listener at port 18081. You may change this behaviour using the ssl_add_http_listener and ssl_secondary_http_listener_port. As you probably will run multiple Schema Registry instances, you will need this option enabled, so please make sure the plain http port is secured via a firewall or other network configuration and accessible only by trusted servers, especially the ones that run Schema Registry instances.

Configuring a second http listener so Schema Registry instances can communicate.

[GHI_1]

Github Issue (confluentinc/schema-registry/issues/386)

Client Authentication¶

Once TLS is setup, client authentication only needs ssl.client.auth option and a SSL truststore in order to know which certificates the server can trust. Client Authentication is yet unsupported by Kafka REST Proxy, Kafka Connect and the Schema Registry itself (communication between Registry instances).

Enable client authentication to the Schema Registry via client certificates.

Setup a truststore for the Schema Registry, in order to know which certificates it can trust.

If you want to test your setup, see Verify HTTPS Endpoints Authentication.

Kafka REST Proxy Client¶

REST Proxy uses the Schema Registry. It is automatically setup with the prime listener of each Registry instance. If you enable SSL for the Schema Registry server, then REST Proxy will use the https port to communicate with it.

You need to set a SSL truststore for Kafka REST Proxy so that it can trust the certificate provided by the Registry.

Add a SSL truststore to Kafka REST Proxy so it can validate the certificate provided by Schema Registry

Kafka REST Proxy does not support authentication to the registry. If you enable the Schema Registry’s secondary http listener, it will autodetect it and use that. Please see Secondary HTTP Listener for more information.

Kafka Connect Client¶

Kafka Connect workers use the Schema Registry. They are automatically setup with the prime listener of each Registry instance. If you enable SSL for the Schema Registry server part, then Connect workers will use the https port to communicate with it —ignoring the possible http secondary listener.

You must set a SSL truststore for Connect workers to be able to verify the certificate provided by the Schema Registry.

Set a SSL truststore for Kafka Connect workers, in order to be able to trust the certificate provided by the Schema Registry.

Kafka Connect does not support authentication to the Registry. If you enable the Schema Registry’s secondary http listener, it will autodetect it and use that. Please see Secondary HTTP Listener for more information.

TLS/SSL Transport Encryption¶

To enable https for REST Proxy enable SSL and provide the role with a keystore that contains the key-certificate pair to use. Then enable the ssl_listener option.

Enable SSL for REST Proxy and provide a keystore.

Set the REST Proxy to serve its API under https.

Client Authentication¶

Once transport encryption is enabled, you can setup client authentication via the ssl.client.auth option and provide a SSL truststore with the certificates of the CAs you trust to sign client certificates.

Enable client authentication for REST Proxy.

Provide a truststore to REST Proxy to validate clients’ certificates against.

The Proxy isn’t used by other roles, so you don’t have to setup anything else within Fast Data. If you use the Fast Data Tools, please read the note below. If you want to test your setup, see Verify HTTPS Endpoints Authentication.

listeners=https://0.0.0.0:8082,http://0.0.0.0:31443

Kafka Listeners¶

Kafka Brokers use the listeners string to determine on which address and port the service should listen for client requests, as well as the protocol and possible authentication it will require. The protocol and the authentication are set by one string that may take one of four values:

PLAINTEXT

No transport encryption, nor authentication is required.

SASL_PLAINTEXT

No transport encryption. Authentication via SASL (only GSSAPI/Kerberos method supported) is required.

SASL_SSL

Transport encryption via TLS/SSL. Authentication via SASL (only GSSAPI/Kerberos method supported) is required.

SSL

Transport encryption via TLS/SSL. Authentication optionally set via client certificates.

A typical listener string to instruct the brokers to list to all addresses at port 9092 without auth or encryption, would look like:

listeners=PLAINTEXT://:9092

More than one listeners may be set, it the need arises:

listeners=SASL_PLAINTEXT://:9092,PLAINTEXT://10.10.0.1:19092,SSL://:9093

In the CSD we set the listeners automatically, given your choices for kerberos.auth.enable, ssl_enabled and force.plaintext.listener. Should you need a more advanced scenario, you may set the listeners’ string in the brokers’ safety valve, which overrides the automatic settings.

As of Confluent Platform 3.1.1 the listeners string is supported by all roles, albeit with different protocol support. Also the manual override via the safety valve is supported in all roles.

Verify Kerberos¶

To verify that Kerberos authorization works, from a node of your cluster you may obtain a ticket-granting ticket from Kerberos and run a short performance test.

First create a JAAS configuration file that contains your Kerberos principal:

KafkaClient {
     com.sun.security.auth.module.Krb5LoginModule required
     useTicketCache=true
     serviceName="kafka"
     principal="user@DOMAIN";
};

Client {
     com.sun.security.auth.module.Krb5LoginModule required
     useTicketCache=true
     principal="user@DOMAIN";
};

Then proceed to obtain your TGT via kinit [7], create a test topic [8] and run the performance test:

$ kinit user@DOMAIN
$ kafka-topics \
      --zookeeper zk:2181/fastdata \
      --create \
      --topic test-krb \
      --partitions 9 \
      --replication-factor 3
$ KAFKA_OPTS=-Djava.security.auth.login.config=/path/to/jaas.conf kafka-producer-perf-test \
      --topic test-krb \
      --throughput 1000 \
      --record-size 1000 \
      --num-records 5000 \
      --producer-props \
          bootstrap.servers=broker1:9092,broker2:9092,broker3:9092 \
          security.protocol=SASL_PLAINTEXT
$ kafka-topics --zookeeper zk:2181/fastdata --delete --topic test-krb

Verify Brokers SSL Authentication¶

As with Kerberos, you can use a SSL keystore, a SSL truststore and run a small performance test to verify your brokers work as expected.

Once you obtain your keystore and truststore, create a topic and the test:

$ kafka-topics \
    --zookeeper zk:2181/fastdata \
    --create \
    --topic test-tls \                                                                                                                                                                          --partitions 9 \
    --replication-factor 3
$ kafka-producer-perf-test \
    --topic test-tls \
    --throughput 1000 \
    --record-size 1000 \
    --num-records 5000 \
    --producer-props \
        bootstrap.servers=broker1:9093,broker2:9093,broker3:9093 \
        security.protocol=SSL \
        ssl.keystore.location=/path/to/keystore.jks \
        ssl.keystore.password=changeit \
        ssl.key.password=changeit \
        ssl.truststore.location=/path/to/truststore.jks \
        ssl.truststore.password=changeit
$ kafka-topics --zookeeper zk:2181/fastdata --delete --topic test-tls

Verify HTTPS Endpoints Authentication¶

To verify your https listeners from Schema Registry or Kafka REST Proxy you will need a key-certificate pair in PEM format [9] and curl.

To verify Schema Registry once you obtain your PEM files, first verify that it doesn’t permit connections without autentication:

$ curl https://schema.registry.url:8081

Then use your PEM keys to verify that authentication works:

$ curl --cert certificate.pem --key key.pem https://schema.registry.url:8081

The same procedure applies to Kafka REST Proxy:

$ curl https://kafka.rest.url:8082
$ curl --cert certificate.pem --key key.pem https://kafka.rest.url:8082