2.1. Installation ¶

Contents

Installation

2.1.1. Add CSD to Cloudera Manager ¶

To install Landoop’s Fast Data Tools CSD all you have to do is copy a small jar file into your Cloudera CSD directory and restart Cloudera Manager. The CSD is available upon request.

Important

Fast Data Tools is currently available only for Linux x86_64 hosts. If you are on a different architecture or a 32bit OS and interested to try Fast Data Tools contact us.

It is up to the cluster administrator to copy the CSD to the server that hosts the Cloudera Manager. For reference, in a vanilla Cloudera installation, you would first ssh into the server that houses the Cloudera Manager and run:

sudo wget https://url/of/fast-data-tools.jar \
          -O /opt/cloudera/csd/FASTDATATOOLS-1.2-release-0.jar
sudo chown cloudera-scm:cloudera-scm /opt/cloudera/csd/FASTDATATOOLS-1.2-release-0.jar

If you installed cloudera to a different directory adjust accordingly.

Next you have to either restart CM (Cloudera Manager) or use CM’s CSD API to install without restart. The command line to restart CM depends on your Linux distribution of choice. For most modern distributions which rely on systemd, you would run something like:

sudo systemctl restart cloudera-scm-server

If you prefer the API, you should visit the following two locations in order. The first asks CM to reload the CSD list and the second to install Landoop’s Fast Data Tools CSD. Cloudera strongly recommends to avoid this method as the CSD may not work properly until CM restart:

http://cloudera.manager.server:7180/cmf/csd/refresh
http://cloudera.manager.server:7180/cmf/csd/install?csdName=FASTDATATOOLS-1.2-release-0.jar

Once you complete the above steps, two things will change in your CM; it now includes a Fast Data Tools service type and also has Landoop’s repository for the Fast Data Tools parcel, that installs the UI tools and support software.

Note

If you want Grafana to render properly graphs in alerts (these graphs will be embedded to email and/or slack notifications), you have to make sure freetype, fontconfig and urw-fonts are installed in your grafana node. In our experience usually just urw-fonts will be missing from a server install.

You are ready to proceed to Add a Fast Data Tools Service.

2.1.2. Add a Fast Data Tools Service ¶

The CSD installation made your CM aware of the Fast Data Tools programs and their management. You may use CM now to install the actual software to your cluster hosts and then setup the service.

2.1.2.1. Install the Parcel ¶

A parcel is a package format for Cloudera Manager. It contains both files and system setup instructions. We provide a range of Confluent parcels for various Linux x86_64 distros.

The Fast Data Tools CSD you installed earlier, added the parcel repository and now you can install our parcel directly from CM’s parcel management page.

At the parcel management page of your Cloudera Manager you proceed to download, distribute and activate the Fast Data Tools parcel.

../../../../_images/cm-fast-data-tools-parcel1.png

Fast Data Tools Parcel available in Cloudera Manager

Note

If your cluster does not have internet access, Cloudera provides alternative installation methods. Please consult Cloudera’s documentation

2.1.2.2. Add a Service Instance ¶

A service is a set of roles running on one or more cluster hosts, managed by CM. Our CSD provides the following roles:

Kafka Topics UI
Schema Registry UI
Kafka Connect UI

You may add as many of each to a service instance as you want. The applications are stateless and usually you don’t need more than one instance of each role.

To start with a Fast Data Tools service, inside CM, at your cluster’s menu, select add a service.

../../../../_images/cm-fast-data-tools-add-service1.png

Cloudera Manager, Add a Service

At the first step, choose Fast Data Tools.

../../../../_images/cm-fast-data-tools-service-select1.png

Fast Data Tools service installation option in Cloudera Manager

The second step asks you to assign roles. As mentioned before the applications are stateless and their target audience is human users —a limited audience that does not require significant resources. One instance of each should be enough. Should your host ever need to go offline, just add another instance to another host.

../../../../_images/cm-fast-data-tools-assign-roles1.png

Fast Data Tools role, instances assignment

The third step has the main settings for your Fast Data Tools service. The most important options for functionality are the addresses of your Schema Registry —important if you use the Schema Registry UI—, Kafka REST Proxy —important if you use the Kafka Topics UI— and Kafka Connect —important if you use the Kafka Connect UI.

Note

If you enabled authentication via client certificates for the Schema Registry and Kafka REST Proxy, please see Schema Registry’s Client Authentication and REST Proxy’s Client Authentication in order to use them with the UIs.

There are also some important security options that you must be aware off and can set at this step or later. You should read about them at Security.

../../../../_images/cm-fast-data-tools-wizard-settings1.png

Cloudera Manager Install Wizard settings for Fast Data Tools.

The last step will start the roles you assigned if possible. In case it doesn’t start them or there are issues, you can leave the wizard and head straight to the service settings to start it or fix the issues.

../../../../_images/cm-fast-data-tools-service-started1.png

Fast Data Tools Services Started Succesfully

Note

Our default ports are 13031 for Kafka Topics UI, 13032 for Schema Registry UI and 13033 for Kafka Connect UI.

2.1.3. Access the UIs ¶

Once your roles are started you can immediately begin using our software. Visit the Fast Data Tools service and you will find links for Kafka Topics UI and Schema Registry UI.

../../../../_images/cm-fast-data-tools-open-tool1.png

Quick Access to Fast Data Tools UIs

../../../../_images/cm-fast-data-tools-kt-ui1.png

Kafka Topics UI

../../../../_images/cm-fast-data-tools-sr-ui1.png

Schema Registry UI

../../../../_images/cm-fast-data-tools-cd-ui1.png

Kafka Connect UI

2.1.4. Cleanup, Reinstall, Remove ¶

Whether you want to remove completely Fast Data Tools or re-install the service, the process is simple.

2.1.4.1. Uninstall ¶

To uninstall the CSD, you should first stop any running Fast Data Tools services and then you delete them from your cluster.

Remove Service

Next visit the CM’s Parcels tab and deactivate, remove from hosts and uninstall our parcel.

Last stop the Cloudera Manager server and delete the CSD file from your Cloudera CSD directory (usually /opt/cloudera/csd/) and restart the Cloudera Manager for the change to take effect.

In our experiments, in one occassion Cloudera Manager wasn’t able to start again after the CSD removal due to missing metric sources. The solution was to copy the CSD back, start CM with it, stop it and remove the CSD again.

2.1.4.2. Clean Data Directories ¶

Fast Data Tools are stateless and do not store any data. They do store access logs though under /var/log/fastdatatools. You may freely remove this directory.