Creating components for a REST API

This tutorial shows how to create components that consume a REST API.

The component developed as example in this tutorial is an input component that provides a search functionality for Zendesk using its Search API.
Lombok is used to avoid writing getter, setter and constructor methods.

You can generate a project using the Talend Components Kit starter, as described in this tutorial.

Setting up the HTTP client

The input component relies on Zendesk Search API and requires an HTTP client to consume it.

The Zendesk Search API takes the following parameters on the /api/v2/search.json endpoint.

  • query : The search query.

  • sort_by : The sorting type of the query result. Possible values are updated_at, created_at, priority, status, ticket_type, or relevance. It defaults to relevance.

  • sort_order: The sorting order of the query result. Possible values are asc (for ascending) or desc (for descending). It defaults to desc.

Talend Component Kit provides a built-in service to create an easy-to-use HTTP client in a declarative manner, using Java annotations.

public interface SearchClient extends HttpClient { (1)

    @Request(path = "api/v2/search.json", method = "GET") (2)
    Response<JsonObject> search(@Header("Authorization") String auth,(3) (4)
            @Header("Content-Type") String contentType, (5)
            @Query("query") String query, (6)
            @Query("sort_by") String sortBy,
            @Query("sort_order") String sortOrder,
            @Query("page") Integer page
    );
}
1 The interface needs to extend org.talend.sdk.component.api.service.http.HttpClient to be recognized as an HTTP client by the component framework. This interface also provides the void base(String base) method, that allows to set the base URI for the HTTP request. In this tutorial, it is the Zendesk instance URL.
2 The @Request annotation allows to define the HTTP request path and method (GET, POST, PUT, and so on).
3 The method return type and a header parameter are defined.
The method return type is a JSON object: Response<JsonObject>. The Response object allows to access the HTTP response status code, headers, error payload and the response body that are of the JsonObject type in this case.
The response body is decoded according to the content type returned by the API. The component framework provides the codec to decode JSON content.
If you want to consume specific content types, you need to specify your custom codec using the @Codec annotation.
4 The Authorization HTTP request header allows to provide the authorization token.
5 Another HTTP request header defined to provide the content type.
6 Query parameters are defined using the @Query annotation that provides the parameter name.

No additional implementation is needed for the interface, as it is provided by the component framework, according to what is defined above.

This HTTP client can be injected into a mapper or a processor to perform HTTP requests.

Configuring the component

This example uses the basic authentication that supported by the API.

Configuring basic authentication

The first step is to set up the configuration for the basic authentication. To be able to consume the Search API, the Zendesk instance URL, the username and the password are needed.

@Data
@DataStore (1)
@GridLayout({ (2)
        @GridLayout.Row({ "url" }),
        @GridLayout.Row({ "username", "password" })
})
@Documentation("Basic authentication for Zendesk API")
public class BasicAuth {

    @Option
    @Documentation("Zendesk instance url")
    private final String url;

    @Option
    @Documentation("Zendesk account username (e-mail).")
    private final String username;

    @Option
    @Credential (3)
    @Documentation("Zendesk account password")
    private final String password;

    public String getAuthorizationHeader() { (4)
        try {
            return "Basic " + Base64.getEncoder()
                    .encodeToString((this.getUsername() + ":" + this.getPassword()).getBytes("UTF-8"));
        } catch (UnsupportedEncodingException e) {
            throw new RuntimeException(e);
        }
    }
}
1 This configuration class provides the authentication information. Type it as Datastore so that it can be validated using services (similar to connection test) and used by Talend Studio or web application metadata.
2 @GridLayout defines the UI layout of this configuration.
3 The password is marked as Credential so that it is handled as sensitive data in Talend Studio and web applications. Read more about sensitive data handling.
4 This method generates a basic authentication token using the username and the password. This token is used to authenticate the HTTP call on the Search API.

The data store is now configured. It provides a basic authentication token.

Configuring the dataset

Once the data store is configured, you can define the dataset by configuring the search query. It is that query that defines the records processed by the input component.

@Data
@DataSet (1)
@GridLayout({ (2)
        @GridLayout.Row({ "dataStore" }),
        @GridLayout.Row({ "query" }),
        @GridLayout.Row({ "sortBy", "sortOrder" })
})
@Documentation("Data set that defines a search query for Zendesk Search API. See API reference https://developer.zendesk.com/rest_api/docs/core/search")
public class SearchQuery {

    @Option
    @Documentation("Authentication information.")
    private final BasicAuth dataStore;

    @Option
    @TextArea (3)
    @Documentation("Search query.") (4)
    private final String query;

    @Option
    @DefaultValue("relevance") (5)
    @Documentation("One of updated_at, created_at, priority, status, or ticket_type. Defaults to sorting by relevance")
    private final String sortBy;

    @Option
    @DefaultValue("desc")
    @Documentation("One of asc or desc. Defaults to desc")
    private final String sortOrder;
}
1 The configuration class is marked as a DataSet. Read more about configuration types.
2 @GridLayout defines the UI layout of this configuration.
3 A text area widget is bound to the Search query field. See all the available widgets.
4 The @Documentation annotation is used to document the component (configuration in this scope). A Talend Component Kit Maven plugin can be used to generate the component documentation with all the configuration description and the default values.
5 A default value is defined for sorting the query result.

Your component is configured. You can now create the component logic.

Defining the component mapper

Mappers defined with this tutorial don’t implement the split part because HTTP calls are not split on many workers in this case. 
@Version
@Icon(value = Icon.IconType.CUSTOM, custom = "zendesk")
@PartitionMapper(name = "search")
@Documentation("Search component for zendesk query")
public class SearchMapper implements Serializable {

    private final SearchQuery configuration; (1)
    private final SearchClient searchClient; (2)

    public SearchMapper(@Option("configuration") final SearchQuery configuration, final SearchClient searchClient) {
        this.configuration = configuration;
        this.searchClient = searchClient;
    }

    @PostConstruct
    public void init() {
        searchClient.base(configuration.getDataStore().getUrl()); (3)
    }

    @Assessor
    public long estimateSize() {
        return 1L;
    }

    @Split
    public List<SearchMapper> split(@PartitionSize final long bundles) {
        return Collections.singletonList(this); (4)
    }

    @Emitter
    public SearchSource createWorker() {
        return new SearchSource(configuration, searchClient); (5)
    }
}
1 The component configuration that is injected by the component framework
2 The HTTP client created earlier in this tutorial. It is also injected by the framework via the mapper constructor.
3 The base URL of the HTTP client is defined using the configuration URL.
4 The mapper is returned in the split method because HTTP requests are not split.
5 A source is created to perform the HTTP request and return the search result.

Defining the component source

Once the component logic implemented, you can create the source in charge of performing the HTTP request to the search API and converting the result to JsonObject records.

public class SearchSource implements Serializable {

    private final SearchQuery config; (1)
    private final SearchClient searchClient; (2)
    private BufferizedProducerSupport<JsonValue> bufferedReader; (3)

    private transient int page = 0;
    private transient int previousPage = -1;

    public SearchSource(final SearchQuery configuration, final SearchClient searchClient) {
        this.config = configuration;
        this.searchClient = searchClient;
    }

    @PostConstruct
    public void init() { (4)
        bufferedReader = new BufferizedProducerSupport<>(() -> {
            JsonObject result = null;
            if (previousPage == -1) {
                result = search(config.getDataStore().getAuthorizationHeader(),
                        config.getQuery(), config.getSortBy(),
                        config.getSortBy() == null ? null : config.getSortOrder(), null);
            } else if (previousPage != page) {
                result = search(config.getDataStore().getAuthorizationHeader(),
                        config.getQuery(), config.getSortBy(),
                        config.getSortBy() == null ? null : config.getSortOrder(), page);
            }
            if (result == null) {
                return null;
            }
            previousPage = page;
            String nextPage = result.getString("next_page", null);
            if (nextPage != null) {
                page++;
            }

            return result.getJsonArray("results").iterator();
        });
    }

    @Producer
    public JsonObject next() { (5)
        final JsonValue next = bufferedReader.next();
        return next == null ? null : next.asJsonObject();
    }

    (6)
    private JsonObject search(String auth, String query, String sortBy, String sortOrder, Integer page) {
        final Response<JsonObject> response = searchClient.search(auth, "application/json",
                query, sortBy, sortOrder, page);
        if (response.status() == 200 && response.body().getInt("count") != 0) {
            return response.body();
        }

        final String mediaType = extractMediaType(response.headers());
        if (mediaType != null && mediaType.contains("application/json")) {
            final JsonObject error = response.error(JsonObject.class);
            throw new RuntimeException(error.getString("error") + "\n" + error.getString("description"));
        }
        throw new RuntimeException(response.error(String.class));
    }

    (7)
    private String extractMediaType(final Map<String, List<String>> headers) {
        final String contentType = headers == null || headers.isEmpty()
                || !headers.containsKey(HEADER_Content_Type) ? null :
                headers.get(HEADER_Content_Type).iterator().next();

        if (contentType == null || contentType.isEmpty()) {
            return null;
        }
        // content-type contains charset and/or boundary
        return ((contentType.contains(";")) ? contentType.split(";")[0] : contentType).toLowerCase(ROOT);
    }
}
1 The component configuration injected from the component mapper.
2 The HTTP client injected from the component mapper.
3 A utility used to buffer search results and iterate on them one after another.
4 The record buffer is initialized with the init by providing the logic to iterate on the search result. The logic consists in getting the first result page and converting the result into JSON records. The buffer then retrieves the next result page, if needed, and so on.
5 The next method returns the next record from the buffer. When there is no record left, the buffer returns null.
6 In this method, the HTTP client is used to perform the HTTP request to the search API.
Depending on the HTTP response status code, the results are retrieved or an error is thrown.
7 The extractMediaType method allows to extract the media type returned by the API.

You now have created a simple Talend component that consumes a REST API.

To learn how to test this component, refer to this tutorial.

Scroll to top