1.8.1. CSD 3.2 release 0¶
This version brought in some big changes. Up until now we imitated the way Cloudera setups Kafka. As we provide more roles (Schema Registry, Kafka Connect, Kafka REST) and support all authentication scenarios, this way of setting up Kafka was more complex and counterintuitive to people who already know Kafka.
For this version we aligned the setup layout with Kafka, letting the administrator to configure broker listeners. This permits more complex setups (i.e set up any combination of listeners) with more ease and better understanding, encouraging knowledge transfer from most Kafka resources. From our point, it makes the setup scripts more robust and will accelerate future development.
A small drawback is that a best practice is now almost enforced: all instances
of a role should have the same settings for each listener (basically use the
same port) unless the safety valve is used. The mirrormaker role was
kafka-mirror-maker application is still available and may be used.
We plan to re-introduce the role in a future release.
List of visible chances:
- Support for Kafka 0.10.2.x, Confluent 3.2.x
- Create listener configuration options for all types of listeners of all roles and make them available service wide
- All options of Kafka are now availabe in Cloudera Manager
- Support all authentication scenarios for all roles and permit overrides by instance
- Separate Java opts from performance Java opts and align performance Java opts with the latest Kafka (use of G1 Garbage Collector, etc)
- For Kafka brokers, equate
Xms, to avoid GC pauses and make performance more predictable. Align memory limits (min, max) with new Kafka suggested values.
- Mirrormaker role removed (temporarily)
Changes in supporting parcel:
- Add new Kafka command line application
- We do not inject kafka-log4j into Hadoop classpath anymore
- HTTP metrics library removed
1.8.2. CSD 3.1 release 0¶
- Added a suffix to automatically add to connect’s topics and group.id in order to easily support multiple connect clusters.
- Add option to add a http listener to schema registry when ssl is enabled to work around SR’s github issue #386.
- Permit service instances without brokers to facilitate multiple connect clusters.
- Re-enable schema.registry.url properties for connect to facilitate multiple connect clusters. (Note: we didn’t have them due to autoconfiguration but we detected their overrides in the safety valve.)
- Remove our automatic load balancing to Schema Registry for Connect and REST and use instead the included one from Kafka libraries (we pass the list with the schema registry instances intact).
- Fix: when enabling broker metrics for rebalancer, a regexp would match and prevent zookeeper.connect from being set.
- Fix: metric.reporters had wrong configuration property name. Do not enable by default http metrics reporter.
- Fix: Kafka REST and Connect Distributed do not support auth to Schema Registry via client certificates, so now we try to detect schema registries with http listeners or non-auth ssl listeners and use them.
- Workaround: Set default receive buffer size to 64KB to mitigate bug https://issues.apache.org/jira/browse/KAFKA-3135
Notable changes in the supporting parcel:
- Make kafka-mirror-maker available to the system path.
- Upgrade to CP 3.1.2
1.8.3. CSD 3.1 beta 0¶
- Added support for all authentication scenarios supported by Confluent for all roles.
- Added missing broker properties and removed deprecated ones.
- Tested with Confluent 3.1.1 release.
1.8.4. CSD 3.0 beta 0¶
- Rename CSD to Fast Data.
- Decouple minor version from parcel.
- Support Kerberos for Connect - Brokers connections.
- Support CM’s JVM monitoring for JVM based apps.
- Create manually Connect’s System Topics with correct settings (replication, partitions, compact).
- Add more kafka cli tools from Confluent’s 3.0.1 release
- Stop appending broker’s ID to log.dirs. The cluster admin is expected to set unique log.dirs for multiple kafka clusters.
- Many small fixes and improvements.
1.8.5. CSD 3.0.0 beta 3¶
- Instead of appending a unique id to each log file, now we append a service-wide, user-defined id. This provides a common log path for all components in all nodes, while still permitting multiple Confluent service instances to work alongside in the cluster.
1.8.6. CSD 3.0.0 beta 2¶
- Fix issue where Confluent Parcel Repository wasn’t added automatically to CM.
- Stream Reactor Parcel Repository has to be added manually since CSD specification does not allow us to add more than one repositories automatically to CM.
1.8.7. CSD 3.0.0 beta 1¶
- Upgrade Confluent Platform to 3.0.0
- Simplify kerberized setups. If you force a plaintext port that is accessible from all hosts in you cluster, all the services will use it automatically without need to manually adjust bootstrap servers. Also now you can set the brokers to communicate through the plaintext port if you set a different security model for your clients.
- Connect Classpath option lets you add custom connectors to Kafka Connect. You only have to add them to the same path for every host that runs Connect.
- Datamountaineer’s Stream Reactor is available as a parcel. Should you choose to activate it, restart Connect Distributed and it will find automatically the new connectors.
1.8.8. CSD 2.0.1¶