Payload converter

A Payload Converter is a component capable of transforming data from one format to another or converting it from one form to another. Typically, data from an external source is formatted differently from what the Diffusion topic requires, or vice versa. Payload Converters are essential components used within the Gateway Framework to handle data conversion and/or transformation for both source and sink services.

A Payload Converter can:

Convert back-end formatted data to a format supported by Diffusion, or vice versa.
Perform any necessary data reformatting or sanitization as required.

A Payload Converter can be configured programmatically by a developer for a particular service type within the application. Alternatively, a user of the application has the option to explicitly configure a Payload Converter for a specific service within the framework’s service configuration.

By default, the converter specified by the application will be applied to services of the designated type. However, user-specific configurations take precedence over the application’s settings.

If a Payload Converter is not explicitly designated for a service or service type, it will be inferred from the configured Diffusion topic type. By default, JSON topic type will be used for both source and sink services.

Each converter has a specific input value type and output value type. The input value type is the type of input value passed to the converter, and the output value type is the type of output value it creates. Converters can be identified by their names, which can be used in the configuration for a service.

As presented in the image above, the input value of type X from the external system is passed to the source service where a converter converts it into another type Y. This converted value is subsequently published to the Diffusion topic. Similarly, in the case of the sink service, Diffusion publishes a value of type A, which is then converted by the converter into type B and subsequently sent to the external system.

If a converter is unable to convert the value, a PayloadConversionException will be thrown.

Payload converter chaining

The Gateway Framework 2.0 offers support for chaining multiple payload converters for a service. This means that various converters can be employed to transform values into the desired final structure and format. Developers can define a sequence of converters to be applied to a specific service type through the framework API. Users, on the other hand, can achieve this by configuring an array of converter details within the service’s configuration.

The converters in a chain will be applied sequentially to convert the payload from one converter to another. Consequently, it’s important to ensure that the input type of the converter matches the output type of its predecessor and the input type of its successor in the chain.

In the image above, a series of converters are sequentially chained together to convert and/or transform data into the necessary format for the services. Initially, a value of type X from the external system is sent to the source service. Here, it is processed by PayloadConverter_1 to transform it into type Y. Subsequently, this transformed value is passed to PayloadConverter_2, which further converts it into type Z. This value is then forwarded to the final converter in the chain, PayloadConverter_3, where it is transformed into the required type, 0. Finally, this resulting value is published to the Diffusion topic.

Similarly, for the sink service, a value of type A is published from a Diffusion topic. This value is then routed to PayloadConverter_4 for conversion into type B. PayloadConverter_4 performs this initial conversion. Subsequently, the transformed value of type B is passed to PayloadConverter_5, which conducts the final conversion to the ultimate required type, C. This resulting value of type C is then published to the external system.

Defining valid payload converters

For a source service, the payload converter responsible for publishing data to the Diffusion topic, or the final payload converter in a chain, should generate data that aligns with Diffusion topic types. This generated data must also be compatible with the topicType configured for the service.

Similarly, for a sink service, the payload converter that receives data from the Diffusion topic, or the initial payload converter in a chain, should have an input type that matches Diffusion topic types. This input type should also be compatible with the configured topicType for the service.

Therefore, the choice of a payload converter is interdependent with the configured topic type, and vice versa.

Payload converters for a service can be explicitly defined in the service configuration as defined here. If not explicitly defined, they are inferred from the specified topic type in the configuration and used for the service. See issued payload converters in the framework for a list of converters that are available in the framework for specific Diffusion topic types.

Similarly, if a topic type is not defined for a service, it is inferred from the payload converter. If both topicType and payload converter are not defined, the default value is used, which is JSON topic type for both source and sink service.

If neither topicType nor payload converter is defined in the service configuration, these details defined for the service types will be used.

Restricting specific converters for use in the application

It is possible to restrict specific payload converters for use in an application when they do not make sense. For example, if a source service in the application does not publish CSV data, configuring the service to use the CSV to JSON converter will not work, as this converter will be expecting CSV string data.

To allow only a set of converters to be used in the application, a file containing a JSON array of converter names must be created, containing converter names, as follows.

[
  "$JSON_to_String",
  "$Object_to_String",
  "$Object_to_JSON",
  "$Object_to_Long",
  "$Object_to_BINARY",
  "$BINARY_to_byte_array",
  "$Object_to_Double"
]

The path of this file should then be set as the value of the system property: gateway.allowed.payload.converters.file.

If this system property is set, and a converter name that is not listed in the file is used in the configuration, an InvalidConfigurationException will be thrown.