Running the adapter

Prerequisites

  • Java installation (min Java 8)

  • A running Kafka cluster

  • A running Diffusion server (or server cluster)

Download the adapter bundle

Download this zip file that contains the adapter jar and other resources.

Run script

The adapter can be started with any of the following:

  • just bootstrap configurations to connect to diffusion

  • a configuration file

  • with a combination of both

Provided scripts can be used to run the application which runs the executable jar of the Kafka adapter.

Starting with bootstrap configuration only

To start the adapter with the minimum required bootstrap configuration:

  • Run the kafka-adapter-bootstrap.sh script in the bin directory.

The Diffusion connection details can be updated as needed in the script. By default, it is set to use the connection details of a local Diffusion server.

./bin/kafka-adapter-bootstrap.sh
If the adapter is initiated with only the bootstrap configuration, it will establish a session with the Diffusion server. Nevertheless, to fully leverage its functionality, a service needs to be added for data consumption or publication. Therefore, after starting the adapter with the bootstrap configuration, use the Diffusion management console to add a service to commence data publication or consumption.

Starting with a configuration file

To start the adapter with a configuration file:

  • Run the kafka-adapter.sh script in the bin directory.

This script uses a sample configuration file, configuration.json, which is located in the example folder. The configuration file contains general Gateway Framework configuration along with configuration that is specific to the Kafka adapter - contained within the application sections. Modify this file as needed to configure services and sharedConfigs.

./bin/kafka-adapter.sh

The JSON schema files for service types and sharedConfig type in schema folder can be used to understand the configuration requirements in more detail.

For Windows, use the kafka-adapter.bat script instead.

.\bin\kafka-adapter.bat

Run the docker image

To run the docker image:

  1. Pull the docker image.

    docker pull pushtechnology/gateway-kafka-adapter:2.1.0

  2. Run the image as follows:

    1. With minimal bootstrap configuration:

      docker run \
 -e gateway.diffusion.url=ws://90.247.210.62:8080 \
 -e gateway.diffusion.principal=admin \
 -e gateway.diffusion.password=password \
 pushtechnology/gateway-kafka-adapter:2.1.0

    2. With a configuration file:

      docker run \
--mount type=bind,source=configuration.json,target=/opt/kafka-adapter/configuration.json \
-e gateway.config.file=/opt/kafka-adapter/configuration.json \
pushtechnology/gateway-kafka-adapter:2.1.0
Here an external config file is mounted in /opt/kafka-adapter/configuration.json path and is configured as a value for the gateway.config.file environment variable to set the configuration file for the java application.

The docker image allows setting the EXTRA_JAVA_OPTIONS environment variable, which can be set to specify any additional JVM option or override the pre-configured JVM options for the docker container.

External libraries

Any external JAR files containing payload converters or custom publishers that are not part of the Kafka adapter can be added to the 'ext' directory available in the Kafka adapter bundle. The JARs in the 'ext' directory are configured to be loaded during startup by the scripts in the 'bin/' directory.