Apache Kafka

Installing Apache Kafka and Zookeeper from the RPM repository

Install Apache Kafka and Zookeeper with its default configuration using RPM.

Before you begin

• Apache Kafka and Zookeeper requires Java 8. You can use either Oracle Java or OpenJDK.
• Make sure that the JAVA_HOME variable is correctly set to the Java home directory. For example: /usr/java/jdk1.8.0_181-amd64.

  Tip: You can set it in the /root/.bashrc file by adding the following line to it: export JAVA_HOME=$(dirname $(dirname $(readlink -e /usr/bin/java))).

About this task

The default installation also installs the following dependencies:

• which
• sed
• gawk
• coreutils

In case of custom installation, these dependencies must be installed beforehand.

Procedure

1. Create a file called talend.repo in the /etc/yum.repos.d directory, containing the following configuration:

   [talend-7.1.1]
   name=Talend 7.1.1
   baseurl='https://<user>:<password>@www.opensourceetl.net/rpms/talend/7.1.1/base/x86_64/'
   enabled=1
   gpgcheck=1
   gpgkey=http://www.opensourceetl.net/rpms/GPG-KEY-talend

   Credentials (user and password) are provided in the license email sent by Talend.

   Your repository is now ready for use.

2. Install Apache Kafka and Zookeeper.
   • To install the package with its default configuration, use the following command:

     sudo yum install talend-kafka

     This command does not require any additional parameter. It installs the package and its dependencies with their default configuration in the default /opt/talend directory.

   • If the default parameters do not match your requirements, install the package with custom parameters using the RPM command.

     For example, the following command installs the module in a specific directory:

     rpm -i --prefix=/opt/folderpath talend-kafka-7.1.1-1-x86_64.rpm

     The list of configuration parameters is detailed in Apache Kafka and Zookeeper RPM configuration parameters.

     Note: When installing the package with custom parameters, the dependencies listed above are not installed. You need to install them beforehand.

   The package is now installed. You can start the service and use it.

Running Apache Kafka and Zookeeper with systemd

Start, stop and monitor the status of the Apache Kafka and Zookeeper services using systemd.

Procedure

• Start the services using the following commands:

  sudo systemctl start talend-zookeeper

  sudo systemctl start talend-kafka

  Note: Start the Zookeeper service before the Kafka service.

• Stop the services using the following commands:

  sudo systemctl stop talend-zookeeper

  sudo systemctl stop talend-kafka

• Check the status of the services using the following commands:

  sudo systemctl status talend-zookeeper

  sudo systemctl status talend-kafka

• Check logging information using the journalctl command.For example:
  • To list service journal entries:

    sudo journalctl --unit talend-zookeeper

    sudo journalctl --unit talend-kafka

  • To list service journal entries after a specific date:

    sudo journalctl --unit talend-zookeeper --since "2018-08-17 13:15:17"

    sudo journalctl --unit talend-kafka --since "2018-08-17 13:15:17"

Apache Kafka and Zookeeper RPM configuration parameters

The Apache Kafka and Zookeeper RPM uses a set of parameters to perform the installation.

To use custom values, set up these parameters in environment variables before performing the installation.

Variable	Default value	Description
TALEND_INSTALL_USER	talend	This user is set as the owner of base folder for the package. The user is created if missing.
TALEND_INSTALL_GROUP	talend	This group is set as the owner of base folder for the package. The group is created if missing.
TALEND_INSTALL_SYSTEMD	1	Whether to install SystemD services. Possible values are 0 (false) or 1 (true). Services are created and enabled, but not started.

Directory layout of the Apache Kafka and Zookeeper RPM

The RPM installs the module with the following directory layout:

Type	Description	Default location
Shell scripts	Several Shell scripts are available to start and stop the Apache Kafka and Zookeeper services: start_zookeeper.sh stop_zookeeper.sh start_kafka.sh stop_kafka.sh	/opt/talend/kafka
Configuration files	Apache Kafka and Zookeeper configuration files, including: server.properties zookeeper.properties	/opt/talend/kafka/config
Logs	Log file location for Apache Kafka and Zookeeper.