Issued payload converters

The Gateway Framework provides a collection of payload converters intended for converting data within source, sink, or hybrid services into the necessary formats. These converters can be categorized into two groups: parameterized converters, which require additional parameters for construction, and Non-parameterized converters, which do not require any additional parameters. When defining the converters for a service, such configuration parameters can be passed if required. See configuring services for more details on configuring converters in service configuration.

All Framework-issued payload converter names start with "$".

Non-parameterized payload converters

Converters listed in the table below can be used with Diffusion topic type-specific data. These converters do not require any configuration parameters and are also implicitly used by the framework for the configured topic type of the service. For instance, if the topic type of a source service is configured as 'String', then the $Object_to_String converter will be employed internally. These converters can also be explicitly used in the service configuration.

Diffusion topic type specific, non-parameterized payload converters
Input type	Output type	Payload converter Name	Description
Object	String	$Object_to_String	Converts an Object value into a String. If the value is an instance of String, it is passed through as is. If the value is `null`, `null` will be returned. For any other type the value of its `toString` method is used. It can be used to produce a value for a Diffusion STRING topic type.
Object	Long	$Object_to_Long	Converts an Object value into a Long. If the value is an instance of Long then it is passed through as is. If the value is an instance of Number, the product of its `longValue` method is used. If the value is `null`, `null` will be returned. For any other type, the `Long.parseLong` method is used on the `toString` value of the supplied object. It can be used to produce a value for a Diffusion INT64 topic type.
Object	Double	$Object_to_Double	Converts an Object value into a Double. If the value is an instance of Double, it is passed through as is. If the value is an instance of Java Number, the product of its `doubleValue` method is used. If the value is `null`, `null` will be returned. For any other type, the `Double.parseDouble` method is used on the `toString` value of the supplied object. It can be used to produce a value for a Diffusion DOUBLE topic type.
Object	Diffusion BINARY topic type	$Object_to_BINARY	Converts an Object value into a Diffusion BINARY topic type. If the value is an instance of the Diffusion `Binary` type, the value is passed through as is. If the value is a `byte[]` it will be converted to a `Binary` object.
Object	JSON	$Object_to_JSON	Converts an Object value into an instance of the Diffusion JSON topic type. If the value is an instance of JSON, it is passed through as is. If the value is `null`, a JSON instance containing `null` will be returned. If the value is an instance of `CharSequence`, it’s `toString` method will be used to create a JSON instance to be returned. The supported types of input values for conversion include basic POJOs, primitives, and collections such as Map and List.
Diffusion JSON topic type	String	$JSON_to_String	Converts a value of the Diffusion JSON topic type into a JSON String. This converter is used by the framework for sink services when a payload converter is neither explicitly defined for the service in the configuration nor specified in `SinkServiceProperties`, and an update from a JSON Diffusion topic is received for the subscribed topic.
Diffusion BINARY topic type	byte[]	$BINARY_to_byte_array	Produces a `byte[]` representation of the topic’s `Binary` value. This is to avoid copying contents to a new array (which is inefficient with BINARY topics dealing with large value). Hence, the returned `byte[]` must not be modified. This converter is used by the framework for sink services when a payload converter is neither explicitly defined for the service in the configuration nor specified in `SinkServiceProperties`, and an update from a Binary Diffusion topic is received for the subscribed topic.

Other payload converters

CSV to JSON payload converter

A converter is available to convert simple CSV(Comma Separated Values) data to the Diffusion JSON topic type. The name of this converter is $CSV_to_JSON. It can be configured with or without configuration parameters.

By default, if no parameters are configured for this converter, it expects the CSV data it receives to contain a header in the first row and treats the first row as the header.

The supported parameter for the converter is defined below.

Supported configuration parameter for **$CSV_to_JSON** converter
Parameter Name	Type	Description	Mandatory	Default value
headers	List of String	List of headers to be used for CSV data. If headers are set, it is assumed that the CSV data will not contain a header row, and the provided headers will be used to create the JSON object. If the data contains a header row, that row will also be treated as a CSV data row. If no headers field is set in the parameters, the default behaviour is assumed, which is similar to setting the converter without any parameters. If an empty list is passed in headers, it is assumed that the CSV data will not contain any header row. Hence, all rows in the CSV data will be used to create the final JSON array.	No	N/A

The converter can take either a File or a String as input, containing CSV data, and convert it into a Diffusion JSON value. Depending on the configured headers parameter, the CSV data will be converted either into a JSON array of JSON objects or into a JSON array of row values.

JSON to CSV payload converter

Input data of the Diffusion JSON topic type can be converted into a CSV string using the $JSON_to_CSV_String payload converter. It can be configured with or without configuration parameters.

If the converter is not configured with any parameters, it only supports the conversion of JSON data containing arrays of arrays. This will create CSV data without any header fields. If a JSON array of objects is passed, a PayloadConverterException will be thrown.

The supported parameters are defined below.

Supported configuration parameter for **$JSON_to_CSV_String** converter
Parameter Name	Type	Description	Mandatory	Default value
headers	List of String	The headers are used to validate JSON input data, ensuring they contain the expected field names whose values will be used to create the CSV row data. Additionally, these headers are used to form the CSV header in the final CSV data.	No	N/A
ignoreUnknownJsonFields	Boolean	Flag to specify whether to ignore unknown fields in JSON data to be converted, which are not specified in the headers. Defaults to false. Hence, if any field in the JSON data is not present in the specified headers, the conversion will fail. If set to true, the additional fields in the JSON data will be ignored, and the CSV data will only contain data with the specified headers.	No	false

Avro data conversion

Avro to JSON converter

The framework provides a payload converter to transform Apache Avro data into Diffusion JSON topic format. The name of this converter is $Avro_to_JSON, and it requires no configuration.

This converter expects an input type of GenericContainer and generates data of the Diffusion JSON topic type.

Avro to Avro bytes converter

The framework provides a payload converter to transform Apache Avro data into an Avro byte array. The name of this converter is $Avro_to_Avro_bytes, and it requires no configuration.

This converter expects an input type of GenericContainer and converts it into an Avro byte array.

Avro to JSON String converter

The framework provides a payload converter to transform Apache Avro data into a JSON String. The name of this converter is $Avro_to_JSON_String, and requires no configuration.

This converter expects an input type of GenericContainer and converts it into Json byte array.

Avro bytes to Diffusion JSON topic format converter

The framework also provides a payload converter to transform Apache Avro byte array into Diffusion JSON topic format. The name of this converter is $Avro_bytes_to_JSON.

This converter expects an input type of Avro byte array and generates data of the Diffusion JSON topic type. It requires a configuration parameter to be used for a service, which is explained below.

Supported configuration parameter for **$Avro_bytes_to_JSON** converter
Parameter Name	Type	Description	Mandatory	Default value
schemaFilePath	String	Path of the schema file, which is required to convert Avro byte array into Diffusion JSON topic format.	yes	N/A

JSON to Avro converter

For the conversion of Diffusion JSON topic format to Avro, the $JSON_to_Avro converter is provided. It can take an object of the Diffusion JSON format and convert it into the org.apache.avro.generic.GenericContainer format.

It requires a configuration parameter to be used for a service, which is explained below.

Supported configuration parameter for **$JSON_to_Avro** converter
Parameter Name	Type	Description	Mandatory	Default value
schemaFilePath	String	Path of the schema file, which is required to create the Avro data from JSON.	yes	N/A

Custom Payload Converters

Users and developers can create custom Payload Converters for specific conversions or transformations as needed and use them in the application. For more details, refer to writing custom payload converter.