Rest polling service type

The restPollingServiceType is a service type of Polling Source mode. It can be used to poll REST endpoints and publish the obtained data into configured Diffusion topics.

There can be multiple instances of this service type added to the adapter, to consume data from different REST endpoints or APIs and publish them to different Diffusion topics, using different configurations.

Complete configuration for a service consists of the framework required configuration for polling source service and the service type specific configuration.

Example 1. Sample configuration for a service of type restPollingServiceType

{
  "serviceName": "rates-of-exchange",
  "serviceType": "restPollingServiceType",
  "description": "",
  "config": {
    "framework": {
      "pollIntervalMs": 15000,
      "pollTimeoutMs": 10000
    },
    "application": {
      "restApi": {
        "baseUrl": "https://api.fiscaldata.treasury.gov/services/api/fiscal_service",
        "path": "/v1/accounting/od/rates_of_exchange"
      },
      "topic": {
        "path": "rates-of-exchange",
        "updateMode": "STREAMING"
      }
    }
  }
}

Polling service Framework configuration

Details about supported configuration parameters for framework configuration for the Polling service can be found here.

Application configuration

The supported application configuration parameters for this service are defined in the table below:

'restPollingServiceType' service application configuration
Property	Description	Default
`restApi`	REST API configuration	-
`topic`	Topic configuration	-
`retryIntervalInMinutes`	"An integer value indicating the minutes to wait before making another call to the REST API after receiving a `429` HTTP status, which indicates too many requests. Setting this configuration will resume the service after this interval, which would have been paused after receiving the `429` HTTP status code."	-

`restApi` JSON property

restApi JSON property
Property	Description	Default
`baseUrl`	The REST API base URL, for example: `api.fiscaldata.treasury.gov/services/api/fiscal_service`.	-
`path`	The REST API path, for example: `/v1/accounting/od/rates_of_exchange`. Note: Not all REST APIs expose a path	-
`method`	HTTP method. Options available, one of: `GET`, `POST`	`GET`
`basicAuth`	Username and password, if the REST API requires basic authentication.	-
`oauth`	OAuth details to fetch the access token if the REST API requires OAuth authorisation.	-
`headers`	HTTP headers as key/value pairs.	-
`queryParameters`	HTTP query parameters as key/value pairs.	-
`postContent`	A string value to be posted to a REST API.	-

`topic` JSON property

topic JSON property
Property	Description	Default
`path`	The Diffusion topic path.	-
`updateMode`	Update mode, can be one of: SIMPLE, STREAMING.	`SIMPLE`

Configuration of restSharedConfigType is the same as that of restPollingServiceType service type.

If a sharedConfig instance is defined in the configuration and is referred to in the service configuration, the overall configuration of the service is formed using a combination of configuration defined in the sharedConfig instance and the service instance. When a duplicate configuration is defined in the sharedConfig instance and in the service instance, the configuration in service configuration will take precedence.

In the overall configuration of the service, the mandatory configurations required are: restApi.baseUrl and topic.path.

Below is a sample configuration of a service for the application JSON property:

{
  "restApi": {
    "baseUrl": "service-base-url",
    "path": "service-path",
    "method": "post",
    "basicAuth": {
      "username": "username",
      "password": "password"
    },
    "headers": {
      "shk1": "shv1",
      "shk2": true
    },
    "queryParameters": {
      "sqk1": "sqv1",
      "sqk2": [
        1,
        2,
        3
      ]
    },
    "postContent": "post-content"
  },
  "topic": {
    "path": "topic-path",
    "updateMode": "STREAMING"
  }
}

A JSON schema is provided in the bundle to assist with validation of the restPollingServiceType service configuration.

Handling REST response

Different HTTP response status codes from the REST endpoint are handled accordingly.

200 OK

If the response from the REST endpoint contains a 200 HTTP status, the response body will be published to the configured Diffusion topic.

429 Too Many Requests

If the response consists of a 429 HTTP status code, which implies too many requests, the service making such a request will be paused. A status will be set for the service indicating the received HTTP status. If the retryIntervalInMinutes configuration parameter is set for the service, then after the specified time interval, the service will resume, and the REST calls will be restarted. If retryIntervalInMinutes is not configured, the service will not be automatically resumed and must be restarted via the Diffusion management console or by using the Gateway control REST API.

Other HTTP statuses

If any other HTTP status is received in the REST call, it will be logged, and the status of the service will be updated with the received HTTP code and response body if available.

Integration with Custom Publisher

The REST adapter supports using Custom Publishers with the RESTPolling services.

The custom publisher for the REST adapter should process update of type Object with UpdateContext of type RestUpdateContext. The RestUpdateContext contains the base URL and path of the REST service, in addition to the default Diffusion topic path and default topic properties. The update supplied to the process method of the custom publisher will be the update received after invoking the REST endpoint.

Below is a sample implementation of the custom publisher for REST adapter:

public final class RestCustomPublisher
    extends CustomPublisher<Object, RestUpdateContext> {

    private final Publisher publisher;
    private final PayloadConverter<Object, JSON> objectToJsonConverter;

    /**
     * Constructor.
     */
    public RestCustomPublisher(
        Publisher publisher,
        Map<String, Object> configContext,
        Map<String, ? extends PayloadConverter<?, ?>> payloadConverterMap) {

        super(publisher);
        this.publisher = publisher;

        this.objectToJsonConverter =
            (PayloadConverter<Object, JSON>) payloadConverterMap.get("$Object_to_JSON");

        if (objectToJsonConverter == null) {
            throw new IllegalArgumentException(
                "No object to JSON converter found");
        }
    }

    @Override
    public CompletableFuture<?> process(
        Object update,
        RestUpdateContext updateContext) throws PayloadConversionException {

        String baseUrl = updateContext.getBaseUrl();
        String path = updateContext.getPath();
        TopicProperties topicProperties = updateContext.getTopicProperties();
        String diffusionTopic = updateContext.getDiffusionTopic();

        // TODO process the update as required, convert with the payload
        //  converter and publish using the 'publisher'
        return CompletableFuture.completedFuture(null);
    }
}

In the example above, the custom publisher expects an instance of a payload converter that converts objects into JSON, in its constructor. An adapter user should configure the customPublisher for the service to include the required payload converter configuration. Expecting a payload converter in the payload converter map argument within the constructor is optional and can be expected and documented as required.

The RestUpdateContext is available in the REST adapter JAR, which should be supplied to the custom publisher as a dependency. For details on including the adapter JAR in the custom publisher, refer to the userguide for Custom Publishers.