Talend Component Kit Developer Reference Guide

The intent of the framework is to be able to fit in a Java UI as well as in a web UI.

It must be understood as colocalized and remote UI. The goal is to move as much as possible the logic to the UI side for UI-related actions. For example, validating a pattern, a size, and so on, should be done on the client side rather than on the server side. Being static encourages this practice.

The other goal of being static in the API definition is to ensure that the model will not be mutated at runtime and that all the auditing and modeling can be done before, at the design phase.

Being static also ensures that the development can be validated as much as possible through build tools.
This does not replace the requirement to test the components but helps developers to maintain components with automated tools.

By design, the framework must run in DI (plain standalone Java program) and in Beam pipelines.
It is out of scope of the framework to handle the way the runtime serializes - if needed - the data.

For that reason, it is critical not to import serialization constraints to the stack. As an example, this is the reason why JsonObject is not an IndexedRecord from Avro.

Any serialization concern should either be hidden in the framework runtime (outside of the component developer scope) or in the runtime integration with the framework (for example, Beam integration).

In this context, JSON-P can be good compromise because it brings a powerful API with very few constraints.

A PartitionMapper is a component able to split itself to make the execution more efficient.

This concept is borrowed from big data and useful in this context only (BEAM executions). The idea is to divide the work before executing it in order to reduce the overall execution time.

The process is the following:

A Producer is a component interacting with a physical source. It produces input data for the processing flow.

A producer is a simple component that must have a @Producer method without any parameter. It can return any data:

A Processor is a component that converts incoming data to a different model.

A processor must have a method decorated with @ElementListener taking an incoming data and returning the processed data:

Processors must be Serializable because they are distributed components.

If you just need to access data on a map-based ruleset, you can use JsonObject as parameter type.
From there, Talend Component Kit wraps the data to allow you to access it as a map. The parameter type is not enforced.
This means that if you know you will get a SuperCustomDto, then you can use it as parameter type. But for generic components that are reusable in any chain, it is highly encouraged to use JsonObject until you have an evaluation language-based processor that has its own way to access components.

For example:

A processor also supports @BeforeGroup and @AfterGroup methods, which must not have any parameter and return void values. Any other result would be ignored. These methods are used by the runtime to mark a chunk of the data in a way which is estimated good for the execution flow size.

It is recommended to batch records, for performance reasons:

In some cases, you may need to split the output of a processor in two. A common example is to have "main" and "reject" branches where part of the incoming data are passed to a specific bucket to be processed later.

To do that, you can use @Output as replacement of the returned value:

Alternatively, you can pass a string that represents the new branch:

Having multiple inputs is similar to having multiple outputs, except that an OutputEmitter wrapper is not needed:

@Input takes the input name as parameter. If no name is set, it defaults to the "main (default)" input branch. It is recommended to use the default branch when possible and to avoid naming branches according to the component semantic.

An Output is a Processor that does not return any data.

Conceptually, an output is a data listener. It matches the concept of processor. Being the last component of the execution chain or returning no data makes your processor an output component:

Currently, Talend Component Kit does not allow you to define a Combiner. A combiner is the symmetric part of a partition mapper and allows to aggregate results in a single partition.

Every component family and component needs to have a representative icon.
You can use one of the icons provided by the framework or you can use a custom icon.

For the component family the icon is defined in the package-info.java file. For the component itself, you need to declare it in the component class.

To use a custom icon, you need to have the icon file placed in the resources/icons folder of the project. The icon file needs to have a name following the convention IconName_icon32.png, where you can replace IconName by the name of your choice.

A primitive is a class which can be directly converted from a String to the expected type.

It includes all Java primitives, like the String type itself, but also all types with a org.apache.xbean.propertyeditor.Converter:

The conversion from property to object uses the Dot notation.

For example, assuming the method parameter was configured with @Option("file"):

matches

You can use metadata to specify that a field is required or has a minimum size, and so on. This is done using the validation metadata in the org.talend.sdk.component.api.configuration.constraint package:

Also note that these validations are executed before the runtime is started (when loading the component instance) and that the execution will fail if they don’t pass. If somehow it breaks your application you can disable that validation on the JVM by setting the system property talend.component.configuration.validation.skip to true.

It is common to classify the incoming data. It is similar to tagging data with several types. Data can commonly be categorized as follows:

Those configuration types can be composed to provide one configuration item. For example, a dataset type often needs a datastore type to be provided. A datastore type (that provides the connection information) is used to create a dataset type.

Those configuration types are also used at design time to create shared configurations that can be stored and used at runtime.

For example, in the case of a relational database that supports JDBC:

The component server scans all configuration types and returns a configuration type index. This index can be used for the integration into the targeted platforms (Studio, web applications, and so on).

The configuration type index is represented as a flat tree that contains all the configuration types, which themselves are represented as nodes and indexed by ID.

Every node can point to other nodes. This relation is represented as an array of edges that provides the child IDs.

As an illustration, a configuration type index for the example above can be defined as follows:

If you need to define a binding between properties, you can use a set of annotations:

The target element location is specified as a relative path to the current location, using Unix path characters. The configuration class delimiter is /.
The parent configuration class is specified by ...
Thus, ../targetProperty denotes a property, which is located in the parent configuration class and is named targetProperty.

In some cases, you may need to add metadata about the configuration to let the UI render that configuration properly.
For example, a password value that must be hidden and not a simple clear input box. For these cases - if you want to change the UI rendering - you can use a particular set of annotations:

Target support should cover org.talend.core.model.process.EParameterFieldType but you need to ensure that the web renderer is able to handle the same widgets.

You can also use other types of validation that are similar to @Pattern:

Components can require metadata to be integrated in Talend Studio or Cloud platforms. Metadata is set on the component class and belongs to the org.talend.sdk.component.api.component package.

Example:

Out of the box components are internationalized using the same location logic for the resource bundle. The supported keys are:

Example of configuration for a component named list and belonging to the memory family (@Emitter(family = "memory", name = "list")):

Configuration classes can be translated using the simple class name in the messages properties file. This is useful in case of common configurations shared by multiple components.

For example, if you have a configuration class as follows :

You can give it a translatable display name by adding ${simple_class_name}.${property_name}._displayName to Messages.properties under the same package as the configuration class.

A plugin is a JAR file that was enriched with the list of its dependencies. By default, Talend Component Kit runtime is able to read the output of maven-dependency-plugin in TALEND-INF/dependencies.txt. You just need to make sure that your component defines the following plugin:

Once build, check the JAR file and look for the following lines:

What is important to see is the scope related to the artifacts:

Even if a flat classpath deployment is possible, it is not recommended because it would then reduce the capabilities of the components.

The framework uses two kind of filterings when scanning your component. One based on the JAR name and one based on the package name. Make sure that your component definitions (including services) are in a scanned module if they are not registered manually using ComponentManager.instance().addPlugin(), and that the component package is not excluded.

The first goal is a shortcut for the maven-dependency-plugin. It creates the TALEND-INF/dependencies.txt file with the compile and runtime dependencies, allowing the component to use it at runtime:

This goal helps you validate the common programming model of the component. To activate it, you can use following execution definition:

It is bound to the process-classes phase by default. When executed, it performs several validations that can be disabled by setting the corresponding flags to false in the <configuration> block of the execution:

This goal generates an Asciidoc file documenting your component from the configuration model (@Option) and the @Documentation property that you can add to options and to the component itself.

Testing the rendering of your component configuration into the Studio requires deploying the component in Talend Studio (refer to Studio Documentation.

In the case where you need to deploy your component into a Cloud (web) environment, you can test its web rendering by using the web goal of the plugin:

Two parameters are available with the plugin:

The Mojo generate (Maven plugin goal) of the same plugin also embeds a generator that you can use to bootstrap any input or output component:

It is intended to be used from the command line (or IDE Maven integration) as follows:

For this command to work, you need to register the plugin as follows:

Component ARchive (.car) is the way to bundle a component to share it in the Talend ecosystem. It is a plain Java ARchive (.jar) containing a metadata file and a nested Maven repository containing the component and its depenencies.

This command creates a .car file in your build directory. This file can be shared on Talend platforms.

This CAR is executable and exposes the studio-deploy command which takes a Talend Studio home path as parameter. When executed, it installs the dependencies into the Studio and registers the component in your instance. For example:

You can also upload the dependencies to your Nexus server using the following command:

In this command, Nexus URL and repository name are mandatory arguments. All other arguments are optional. If arguments contain spaces or special symbols, you need to quote the whole value of the argument. For example:

The Internationalization API is the mechanism to use to internationalize your own messages in your own components.

The principle of the API is to design messages as methods returning String values and get back a template using a ResourceBundle named Messages and located in the same package than the interface that defines these methods.

To ensure your internationalization API is identified, you need to mark it with the @Internationalized annotation:

Some common actions need a clear contract so they are defined as API first-class citizen. For example, this is the case for wizards or health checks. Here is the list of the available actions:

Internationalization is supported through the injection of the $lang parameter, which allows you to get the correct locale to use with an @Internationalized service:

The HttpClient usage is described in this section by using the REST API example below. It is assume that it requires a basic authentication header.

To create an HTTP client that is able to consume the REST API above, you need to define an interface that extends HttpClient.

The HttpClient interface lets you set the base for the HTTP address that the client will hit.

The base is the part of the address that needs to be added to the request path to hit the API.

Every method annotated with @Request in the interface defines an HTTP request. Every request can have a @Codec parameter that allows to encode or decode the request/response payloads.

In the codec classes (that implement Encoder/Decoder), you can inject any of your service annotated with @Service or @Internationalized into the constructor. Internationalization services can be useful to have internationalized messages for errors handling.

The interface can be injected into component classes or services to consume the defined API.

The Job builder lets you create a job pipeline programmatically using Talend components (Producers and Processors). The job pipeline is an acyclic graph, allowing you to build complex pipelines.

Let’s take a simple use case where two data sources (employee and salary) are formatted to CSV and the result is written to a file.

A job is defined based on components (nodes) and links (edges) to connect their branches together.

Every component is defined by a unique id and an URI that identify the component.

The URI follows the form [family]://[component][?version][&configuration], where:

For Beam case, you need to rely on Beam pipeline definition and use the component-runtime-beam dependency, which provides Beam bridges.

To extend the UI, add an annotation that can be put on @Option fields, and that is decorated with @Ui. All its members are then put in the metadata of the parameter. For example:

Parameterized tests are a great solution to repeat the same test multiple times. This method of testing requires defining a test scenario (I test function F) and making the input/output data dynamic.

component-runtime-junit is a test library that allows you to validate simple logic based on the Talend Component Kit tooling.

To import it, add the following dependency to your project:

This dependency also provides mocked components that you can use with your own component to create tests.

The mocked components are provided under the test family:

The following artifact allows you to test against a Spark cluster:

The HTTP JUnit module allows you to mock REST API very simply. The module coordinates are:

It supports both JUnit 4 and JUnit 5. The concept is the exact same one: the extension/rule is able to serve precomputed responses saved in the classpath.

You can plug your own ResponseLocator to map a request to a response, but the default implementation - which should be sufficient in most cases - looks in talend/testing/http/<class name>_<method name>.json. Note that you can also put it in talend/testing/http/<request path>.json.

The MultiEnvironmentsRunner executes the tests for each defined environments. With the example above, it means that it runs test1 for Env1 and Env2.

By default, the JUnit4 runner is used to execute the tests in one environment, but you can use @DelegateRunWith to use another runner.

The multi-environment configuration with JUnit 5 is similar to JUnit 4:

The main differences are that no runner is used because they do not exist in JUnit 5, and that you need to replace @Test by @EnvironmentalTest.

The provided environment sets the contextual classloader in order to load the related runner of Apache Beam.

Package: org.talend.sdk.component.junit.environment.builtin.beam

If the environment extends BaseEnvironmentProvider and therefore defines an environment name - which is the case of the default ones - you can use EnvironmentConfiguration to customize the system properties used for that environment:

Here is a test example, which validates a connection URI using ConnectionService:

The testing method is always the same. Only values are changing. It can therefore be rewritten using JUnit Parameterized runner, as follows:

With JUnit 5, parameterized tests are easier to use. The full documentation is available at junit.org/junit5/docs/current/user-guide/#writing-tests-parameterized-tests.

The main difference with JUnit 4 is that you can also define inline that the test method is a parameterized test as well as the values to use:

However, you can still use the previous behavior with a method binding configuration:

This last option allows you to inject any type of value - not only primitives - which is common to define scenarios.

You can define a standard JUnit test and use the SimpleComponentRule rule:

To go further, you can add the ServiceInjectionRule rule, which allows to inject all the component family services into the test class by marking test class fields with @InjectService:

The JUnit 5 integration is very similar to JUnit 4, except that it uses the JUnit 5 extension mechanism.

The entry point is the @WithComponents annotation that you add to your test class, and which takes the component package you want to test. You can use @Injected to inject an instance of ComponentsHandler - which exposes the same utilities than the JUnit 4 rule - in a test class field :

As for JUnit 4, you can go further by injecting test class fields marked with @InjectService, but there is no additional extension to specify in this case:

Using the "test"/"collector" component as shown in the previous sample stores all records emitted by the chain (typically your source) in memory. You can then access them using theSimpleComponentRule.getCollectedData(type).

Note that this method filters by type. If you don’t need any specific type, you can use Object.class.

The input mocking is symmetric to the output. In this case, you provide the data you want to inject:

The component configuration is a POJO (using @Option on fields) and the runtime configuration (ExecutionChainBuilder) uses a Map<String, String>. To make the conversion easier, the JUnit integration provides a SimpleFactory.configurationByExample utility to get this map instance from a configuration instance.

The same factory provides a fluent DSL to create the configuration by calling configurationByExample without any parameter. The advantage is to be able to convert an object as a Map<String, String> or as a query string in order to use it with the Job DSL:

It handles the encoding of the URI to ensure it is correctly done.

The SimpleComponentRule also allows to test a mapper unitarily. You can get an instance from a configuration and execute this instance to collect the output.

As for a mapper, a processor is testable unitary. However, this case can be more complex in case of multiple inputs or outputs.

The rule allows you to instantiate a Processor from your code, and then to collect the output from the inputs you pass in. There are two convenient implementations of the input factory:

The testing relies on a JUnit TestRule. It is recommended to use it as a @ClassRule, to make sure that a single instance of a Spark cluster is built. You can also use it as a simple @Rule, to create the Spark cluster instances per method instead of per test class.

The @ClassRule takes the Spark and Scala versions to use as parameters. It then forks a master and N slaves. Finally, the submit* method allows you to send jobs either from the test classpath or from a shade if you run it as an integration test.

For example:

The integration of that Spark cluster logic with JUnit 5 is done using the @WithSpark marker for the extension. Optionally, it allows you to inject—through @SparkInject—the BaseSpark<?> handler to access the Spark cluster meta information. For example, its host/port.

Currently, SparkClusterRule does not allow to know when a job execution is done, even by exposing and polling the web UI URL to check. The best solution at the moment is to make sure that the output of your job exists and contains the right value.

awaitability or any equivalent library can help you to implement such logic:

To wait until a file exists and check that its content (for example) is the expected one, you can use the following logic:

JUnit 4 setup is done through two rules:

JUnit 5 uses a JUnit 5 extension based on the HttpApi annotation that you can add to your test class. You can inject the test handler - which has some utilities for advanced cases - through @HttpApiInject:

The strength of this implementation is to run a small proxy server and to auto-configure the JVM: http[s].proxyHost, http[s].proxyPort, HttpsURLConnection#defaultSSLSocketFactory and SSLContext#default are auto-configured to work out-of-the-box with the proxy.

It allows you to keep the native and real URLs in your tests. For example, the following test is valid:

If you execute this test, it fails with an HTTP 400 error because the proxy does not find the mocked response.
You can create it manually, as described in component-runtime-http-junit, but you can also set the talend.junit.http.capture property to the folder storing the captures. It must be the root folder and not the folder where the JSON files are located (not prefixed by talend/testing/http by default).

In most cases, use src/test/resources. If new File("src/test/resources") resolves the valid folder when executing your test (Maven default), then you can just set the system property to true. Otherwise, you need to adjust accordingly the system property value.

When the tests run with this system property, the testing framework creates the correct mock response files. After that, you can remove the system property. The tests will still pass, using google.com, even if you disconnect your machine from the Internet.

If you set the talend.junit.http.passthrough system property to true, the server acts as a proxy and executes each request to the actual server - similarly to the capturing mode.

This usage assumes that Beam 2.4.0 is used.

The following dependencies bring the JUnit testing toolkit, the Beam integration and the multi-environment testing toolkit for JUnit into the test scope.

Using the fluent DSL to define jobs, you can write a test as follows:

It executes the chain twice:

Input and output components are particular because they can be linked to a set of actions. It is recommended to wire all the actions you can apply to ensure the consumers of your component can provide a rich experience to their users.

The most common actions are the following ones:

Until the studio integration is complete, it is recommended to limit processors to one input.

Light validations are all the validations you can execute on the client side. They are listed in the UI hint section.

Use light validations first before going with custom validations because they are more efficient.

Custom validations enforce custom code to be executed, they are more heavy to process, so prefer using light validations when possible.

Define an action with the parameters needed for the validation and link the option you want to validate to this action. For example, to validate a dataset for a JDBC driver:

You can also define a Validable class and use it to validate a form by setting it on your whole configuration:

This endpoint will execute any UI action and serialize the response as a JSON (pojo model) It takes as input the family, type and name of the related action to identify it and its configuration as a flat key value set using the same kind of mapping than for components (option path as key).

This endpoint returns the list of available actions for a certain family and potentially filters the " output limiting it to some families and types of actions.

Returns a list of dependencies for the given components.

Then you can use /dependency/{id} to download the binary.

Return a binary of the dependency represented by id. It can be maven coordinates for dependencies or a component id.

Returns the set of metadata about a few components identified by their 'id'.

Returns a particular family icon in raw bytes.

Returns a particular component icon in raw bytes.

Returns the list of available components.

Allows to migrate a component configuration without calling any component execution.

Returns the set of metadata about a few configurations identified by their 'id'.

Returns all available configuration type - storable models. Note that the lightPayload flag allows to load all of them at once when you eagerly need to create a client model for all configurations.

Allows to migrate a configuration without calling any component execution.

Returns an asciidoctor version of the documentation for the component represented by its identifier id.

Format can be either asciidoc or html - if not it will fallback on asciidoc - and if html is selected you get a partial document.

The documentation will likely be the family documentation but you can use anchors to access a particular component (_componentname_inlowercase).

Returns the environment of this instance. Useful to check the version or configure a healthcheck for the server.

Read inputs from an instance of mapper. The number of returned records if enforced to be limited to 1000. The format is a JSON based format where each like is a json record.

Sends records using a processor instance. Note that the processor should have only an input. Behavior for other processors is undefined. The input format is a JSON based format where each like is a json record - same as for the symmetric endpoint.

As shown in the table above, you can customize the forms by type. The format reuses Talend Component Kit REST API (properties model) and defines two main types of extensions:

If you don’t specify a name, the path is used to deduce the name automatically.

When developing a org.talend.sdk.component.form.internal.converter.CustomPropertyConverter CDI, the proxy adds it to the UiSpecService service and uses it with a high priority to convert the server model to a UiSpec model.

This allows to use a custom @Ui API and advanced modeling when specific to applications.
Converters are sorted respecting to the @Priority value. If the annotation is missing, the priority defaults to 0.

The client to use to connect to the Talend Component Kit server is the CXF client, using HttpClient HC (NIO) transport. When you use the Play module, it can be configured with its standard properties prefixed by talend.component.proxy..

You can find more information on CXF website.

The special dynamic_values action builtin::roots can be used for a dropdown filled with all available root types.

Here is a sample patch file:

The builtin::root::reloadFromId action, with the jsonpatch type, allows to reload the whole form:

It is common to have a dropdown with the list of roots and to reload the form when one is selected.

For example, the UIForm part (JavaScript side) can be implemented as follows: