1.8. Changelog

1.8.1. CSD 4.0 release 0

This release brings support for Kafka 1.0 (CP 4.0). Notable changes are that Schema Registry now supports a setup without Zookeeper, which is the default behaviour for the CSD as well, unless you choose otherwise.

The prometheus metrics feature got fine grained setup, by requiring to explicitly enable the capability for the brokers. This is a small but important change, as in production environments it may be preferrable to not augment the brokers with a java agent, but instead use the classic standalone mode of jmx_exporter.

A big addition is the release of our own parcel for Kafka, FAST_DATA_KAFKA_LANDOOP. This parcel is interchangeable with the FAST_DATA_KAFKA_CONFLUENT parcel. We build all components from source, without any changes to the source code but a few changes to the startup scripts. We hope to be able to provide faster updates on this parcel. As an example, the initial release of our parcel comes with Kafka 1.0.1, whilst the Confluent parcel comes with 1.0.0.

A nice addition is our Kafka bash completion script, which provides bash completion for various Kafka command line apps, such as kafka-topics and kafka-console-consumer. Have a look on how to make use of it.

Notable changes in the CSD:

  • Support for Kafka 1.0
  • Explicitly augment brokers with java agents for prometheus monitoring
  • Schema Registry by default is setup without Zookeeper, adjustable
  • Endpoint Exporter now exports more details, in order to support future release of Lenses parcel
  • Fix: some plugin.path settings lead to failure to start connect workers
  • Fix: for Schema Registry and REST Proxy SSL listeners use the server FQDN instead of 0.0.0.0

Changes in the parcels:

  • New, optional Kafka parcel: FAST_DATA_KAFKA_LANDOOP
  • New Stream Reactor release 1.0.0
  • Kafka parcels now include bash completion for kafka commands

1.8.2. CSD 3.3 release 1

This is a bug-fix release. It solves an issue where the scripts that start each role, failed when the password for the SSL key and truststore included certain ASCII characters.

1.8.3. CSD 3.3 release 0

Kafka-wise this version is a straightforward upgrade from CP 3.2.x (Kafka 0.10.2) to CP 3.3.x (Kafka 0.11.0). The configuration options for each role were adjusted to this version of kafka; some new options were added and old options that got deprecated were removed. These changes mostly should be transparent to end users.

A important change came to Kafka Connect, which replaced the CLASSPATH variable with the plugin.path configuration. Please have a look at adding connectors to learn how these two differ.

For the CSD, the main addition came through a new role, Endpoint Exporter. This role exports information about the service’s setup, so other roles in the future, such as Fast Data Tools and Fast Data Monitoring can autoconfigure themselves. Also a new option enables exporting prometheus metrics directly from the roles instead of having to setup metric exporters in the Fast Data Monitoring service.

Last a new action was added for the broker roles: Trigger Controller Election. This is a quick way to fix clusterized setups, when all brokers have been stopped and upon restart they fail to do a proper controller election.

Notable CSD changes:

  • Support for Kafka 0.11.0
  • Switch to plugin.path support instead of CLASSPATH for Kafka Connect
  • New Endpoint Exporter role
  • New option fd.monitoring.prometheus to augment roles with java agent of jmx_exporter
  • New action for broker roles, Trigger Controller Election

Changes in supporting parcels and services:

  • Parcel builder now creates connector parcels that work with both plugin.path and CLASSPATH settings, so they are supported by older and newer CSDs.
  • Newer connector parcels (stream reactor) utilize the plugin.path capability, so some connectors that couldn’t work alongside each other are now working.

1.8.4. CSD 3.2 release 0

This version brought in some big changes. Up until now we imitated the way Cloudera setups Kafka. As we provide more roles (Schema Registry, Kafka Connect, Kafka REST) and support all authentication scenarios, this way of setting up Kafka was more complex and counterintuitive to people who already know Kafka.

For this version we aligned the setup layout with Kafka, letting the administrator to configure broker listeners. This permits more complex setups (i.e set up any combination of listeners) with more ease and better understanding, encouraging knowledge transfer from most Kafka resources. From our point, it makes the setup scripts more robust and will accelerate future development.

A small drawback is that a best practice is now almost enforced: all instances of a role should have the same settings for each listener (basically use the same port) unless the safety valve is used. The mirrormaker role was removed. The kafka-mirror-maker application is still available and may be used. We plan to re-introduce the role in a future release.

List of visible chances:

  • Support for Kafka 0.10.2.x, Confluent 3.2.x
  • Create listener configuration options for all types of listeners of all roles and make them available service wide
  • All options of Kafka are now availabe in Cloudera Manager
  • Support all authentication scenarios for all roles and permit overrides by instance
  • Separate Java opts from performance Java opts and align performance Java opts with the latest Kafka (use of G1 Garbage Collector, etc)
  • For Kafka brokers, equate Xmx with Xms, to avoid GC pauses and make performance more predictable. Align memory limits (min, max) with new Kafka suggested values.
  • Mirrormaker role removed (temporarily)

Changes in supporting parcel:

  • Add new Kafka command line application
  • We do not inject kafka-log4j into Hadoop classpath anymore
  • HTTP metrics library removed

1.8.5. CSD 3.1 release 0

  • Added a suffix to automatically add to connect’s topics and group.id in order to easily support multiple connect clusters.
  • Add option to add a http listener to schema registry when ssl is enabled to work around SR’s github issue #386.
  • Permit service instances without brokers to facilitate multiple connect clusters.
  • Re-enable schema.registry.url properties for connect to facilitate multiple connect clusters. (Note: we didn’t have them due to autoconfiguration but we detected their overrides in the safety valve.)
  • Remove our automatic load balancing to Schema Registry for Connect and REST and use instead the included one from Kafka libraries (we pass the list with the schema registry instances intact).
  • Fix: when enabling broker metrics for rebalancer, a regexp would match and prevent zookeeper.connect from being set.
  • Fix: metric.reporters had wrong configuration property name. Do not enable by default http metrics reporter.
  • Fix: Kafka REST and Connect Distributed do not support auth to Schema Registry via client certificates, so now we try to detect schema registries with http listeners or non-auth ssl listeners and use them.
  • Workaround: Set default receive buffer size to 64KB to mitigate bug https://issues.apache.org/jira/browse/KAFKA-3135

Notable changes in the supporting parcel:

  • Make kafka-mirror-maker available to the system path.
  • Upgrade to CP 3.1.2

1.8.6. CSD 3.1 beta 0

  • Added support for all authentication scenarios supported by Confluent for all roles.
  • Added missing broker properties and removed deprecated ones.
  • Tested with Confluent 3.1.1 release.

1.8.7. CSD 3.0 beta 0

  • Rename CSD to Fast Data.
  • Decouple minor version from parcel.
  • Support Kerberos for Connect - Brokers connections.
  • Support CM’s JVM monitoring for JVM based apps.
  • Create manually Connect’s System Topics with correct settings (replication, partitions, compact).
  • Add more kafka cli tools from Confluent’s 3.0.1 release
  • Stop appending broker’s ID to log.dirs. The cluster admin is expected to set unique log.dirs for multiple kafka clusters.
  • Many small fixes and improvements.

1.8.8. CSD 3.0.0 beta 3

  • Instead of appending a unique id to each log file, now we append a service-wide, user-defined id. This provides a common log path for all components in all nodes, while still permitting multiple Confluent service instances to work alongside in the cluster.

1.8.9. CSD 3.0.0 beta 2

  • Fix issue where Confluent Parcel Repository wasn’t added automatically to CM.
  • Stream Reactor Parcel Repository has to be added manually since CSD specification does not allow us to add more than one repositories automatically to CM.

1.8.10. CSD 3.0.0 beta 1

  • Upgrade Confluent Platform to 3.0.0
  • Simplify kerberized setups. If you force a plaintext port that is accessible from all hosts in you cluster, all the services will use it automatically without need to manually adjust bootstrap servers. Also now you can set the brokers to communicate through the plaintext port if you set a different security model for your clients.
  • Connect Classpath option lets you add custom connectors to Kafka Connect. You only have to add them to the same path for every host that runs Connect.
  • Datamountaineer’s Stream Reactor is available as a parcel. Should you choose to activate it, restart Connect Distributed and it will find automatically the new connectors.

1.8.11. CSD 2.0.1

Initial release.