Create components for REST API

In this tutorial we will show how to create components that consume a REST API.

As an example, we will develop an input component that will provide a search functionality for Zendesk using there Search API.

We use lambok. to get ride of getters, setters and constructors from our classes.
You can generate a project using the components kit starter as described in this tutorial.

Http client setup

As our input component will relay on Zendesk Search API. We will need an http client to consume it.

Zendesk Search API takes the following query parameters on this endpoint /api/v2/search.json.

  • query : The search query.

  • sort_by : One of updated_at, created_at, priority, status, or ticket_type. Defaults to sorting by relevance.

  • sort_order: One of asc or desc. Defaults to desc.

So let’s create our http client according to that.

Talend component kit provides a built-in service to create an easy to use http client in a declarative manner using java annotations.

public interface SearchClient extends HttpClient { (1)

    @Request(path = "api/v2/search.json", method = "GET") (2)
    Response<JsonObject> search(@Header("Authorization") String auth,(3) (4)
            @Header("Content-Type") String contentType, (5)
            @Query("query") String query, (6)
            @Query("sort_by") String sortBy,
            @Query("sort_order") String sortOrder,
            @Query("page") Integer page
    );
}
1 Our interface need to extend org.talend.sdk.component.api.service.http.HttpClient to be known as an http client by the component framework. This interface also provides void base(String base) method that will let us set the base uri for the http request. In our case, it will be the Zendesk instance url.
2 @Request annotation let us define two things. the http request path and method (GET, POST, PUT,…​).
3 At this line we have two important things. The method return type and a header param. At this point we will explain the method return that is of type Response<JsonObject>. The Response object let us access to the http response status code, headers, error payload and the response body that will be of type JsonObject here. The response body will be decoded according to the content type returned by the API. The component framework provides codec for json content. If you want to consume specific content type, you will need to provide your personalized codec using the @Codec annotation.
4 We define the Authorization http request header that will let us provide the authorization token.
5 We define another http request header to provide the content type.
6 We define the query parameters using the @Query annotation that will provide the parameter name.

And that all what we need to do to create our http client. No implementation is needed for the interface, as it will be provided by the component framework according to what we have defined.

This http client can be injected into a mapper or a processor to perform http requests.

Component Configuration

For the sake of simplicity, we will use the basic authentication supported by the API.

Let’s start setting up the configuration for the basic authentication. To be able to consume the Search API, we will need to provide the Zendesk instance URL, the username and the password.

@Data
@DataStore (1)
@GridLayout({ (2)
        @GridLayout.Row({ "url" }),
        @GridLayout.Row({ "username", "password" })
})
@Documentation("Basic authentication for Zendesk API")
public class BasicAuth {

    @Option
    @Documentation("Zendesk instance url")
    private final String url;

    @Option
    @Documentation("Zendesk account username (e-mail).")
    private final String username;

    @Option
    @Credential (3)
    @Documentation("Zendesk account password")
    private final String password;

    public String getAuthorizationHeader() { (4)
        try {
            return "Basic " + Base64.getEncoder()
                    .encodeToString((this.getUsername() + ":" + this.getPassword()).getBytes("UTF-8"));
        } catch (UnsupportedEncodingException e) {
            throw new RuntimeException(e);
        }
    }
}
1 As This configuration class provide the authentication information. We can type it as Datastore, so that it can be validated using services (a kind of test connection feature) or used by Talend studio or web application metadata.
2 This is the UI layout of this configuration.
3 We mark the password as Credential to that it can be handled as sensitive data in Talend Studio and web application. Read more about sensitive data handling.
4 This method generate a basic authentication token using the username and the password. This token will be used to authenticate our http call to the Search API.

Now that we have our data store configuration. that will provide us with the basic authentication token. We need to setup our data set configuration. i.e the search query that will define the records that our input component will provide.

@Data
@DataSet (1)
@GridLayout({ (2)
        @GridLayout.Row({ "dataStore" }),
        @GridLayout.Row({ "query" }),
        @GridLayout.Row({ "sortBy", "sortOrder" })
})
@Documentation("Data set that define a search query for Zendesk Search API. See api reference https://developer.zendesk.com/rest_api/docs/core/search")
public class SearchQuery {

    @Option
    @Documentation("Authentication information.")
    private final BasicAuth dataStore;

    @Option
    @TextArea (3)
    @Documentation("Search query.") (4)
    private final String query;

    @Option
    @DefaultValue("relevance") (5)
    @Documentation("One of updated_at, created_at, priority, status, or ticket_type. Defaults to sorting by relevance")
    private final String sortBy;

    @Option
    @DefaultValue("desc")
    @Documentation("One of asc or desc. Defaults to desc")
    private final String sortOrder;
}
1 This mark this configuration class as a DataSet type. Read more about configuration type.
2 The UI layout of this configuration.
3 We bind a text area widget to the search query field. See all the available widgets.
4 Note the usage of @Documentation annotation. this annotation let us document our component (configuration in this scope). There is a Talend component maven plugin that can be used to generate the component documentation with all the configuration description and the default values.
5 Here we give the field a default value.

That’s all for the configuration part. Let’s create the component logic.

The component mapper

We will not split the http calls on many workers. so our mappers will not implement the split part. 
@Version
@Icon(value = Icon.IconType.CUSTOM, custom = "zendesk")
@PartitionMapper(name = "search")
@Documentation("Search component for zendesk query")
public class SearchMapper implements Serializable {

    private final SearchQuery configuration; (1)
    private final SearchClient searchClient; (2)

    public SearchMapper(@Option("configuration") final SearchQuery configuration, final SearchClient searchClient) {
        this.configuration = configuration;
        this.searchClient = searchClient;
    }

    @PostConstruct
    public void init() {
        searchClient.base(configuration.getDataStore().getUrl()); (3)
    }

    @Assessor
    public long estimateSize() {
        return 1L;
    }

    @Split
    public List<SearchMapper> split(@PartitionSize final long bundles) {
        return Collections.singletonList(this); (4)
    }

    @Emitter
    public SearchSource createWorker() {
        return new SearchSource(configuration, searchClient); (5)
    }
}
1 The component configuration, that will be injected by the component framework
2 The http client that we have created above. it will also be injected by the framework via the mapper constructor.
3 We setup the base URL of our http client using the configuration url.
4 As we will not split the http requests we return this mapper in the split method.
5 We create a source that will perform the http request and return the search result.

The component source

Now we create the source that will perform the http request to the search api and convert the result to JsonObject records.

public class SearchSource implements Serializable {

    private final SearchQuery config; (1)
    private final SearchClient searchClient; (2)
    private BufferizedProducerSupport<JsonValue> bufferedReader; (3)

    private transient int page = 0;
    private transient int previousPage = -1;

    public SearchSource(final SearchQuery configuration, final SearchClient searchClient) {
        this.config = configuration;
        this.searchClient = searchClient;
    }

    @PostConstruct
    public void init() { (4)
        bufferedReader = new BufferizedProducerSupport<>(() -> {
            JsonObject result = null;
            if (previousPage == -1) {
                result = search(config.getDataStore().getAuthorizationHeader(),
                        config.getQuery(), config.getSortBy(),
                        config.getSortBy() == null ? null : config.getSortOrder(), null);
            } else if (previousPage != page) {
                result = search(config.getDataStore().getAuthorizationHeader(),
                        config.getQuery(), config.getSortBy(),
                        config.getSortBy() == null ? null : config.getSortOrder(), page);
            }
            if (result == null) {
                return null;
            }
            previousPage = page;
            String nextPage = result.getString("next_page", null);
            if (nextPage != null) {
                page++;
            }

            return result.getJsonArray("results").iterator();
        });
    }

    @Producer
    public JsonObject next() { (5)
        final JsonValue next = bufferedReader.next();
        return next == null ? null : next.asJsonObject();
    }

    (6)
    private JsonObject search(String auth, String query, String sortBy, String sortOrder, Integer page) {
        final Response<JsonObject> response = searchClient.search(auth, "application/json",
                query, sortBy, sortOrder, page);
        if (response.status() == 200 && response.body().getInt("count") != 0) {
            return response.body();
        }

        final String mediaType = extractMediaType(response.headers());
        if (mediaType != null && mediaType.contains("application/json")) {
            final JsonObject error = response.error(JsonObject.class);
            throw new RuntimeException(error.getString("error") + "\n" + error.getString("description"));
        }
        throw new RuntimeException(response.error(String.class));
    }

    (7)
    private String extractMediaType(final Map<String, List<String>> headers) {
        final String contentType = headers == null || headers.isEmpty()
                || !headers.containsKey(HEADER_Content_Type) ? null :
                headers.get(HEADER_Content_Type).iterator().next();

        if (contentType == null || contentType.isEmpty()) {
            return null;
        }
        // content-type contains charset and/or boundary
        return ((contentType.contains(";")) ? contentType.split(";")[0] : contentType).toLowerCase(ROOT);
    }
}
1 The component configuration injected from the component mapper.
2 The http client injected from the component mapper.
3 A buffer utility that we will use to buffer search result and iterate on theme one by one
4 In the init method we initialize our record buffer by providing the logic to iterate on the search result. we get the first result page and convert the results to json records. The buffer will retrieve the next result page if needed.
5 This method return the next record from the buffer. when no more record is present the buffer return null.
6 In this method we use the http client to perform the http request to the search api. According to the http response status code we get get the results or we throw an error if needed.
7 This method let us extract the media type returned by the API.

That all you will need to do to create a simple Talend component that consume a REST API.

In a next tutorial, we will show how to test this kind of component and use the component framework API simulation tools to create unit tests.

Scroll to top