Talend Component Kit Developer Reference Guide

Intent of the framework is to be able to fit java UI as well as web UI. It must be understood as colocalized and remote UI. The direct impact of that choice is to try to move as much as possible the logic to the UI side for UI related actions. Typically we want to validate a pattern, a size, …​ on the client side and not on the server side. Being static encourages this practise.

The other goal to be really static in its definition is to ensure the model will not be mutated at runtime and all the auditing and modelling can be done before, in the design phase.

Being static also ensures the development can be validated as much as possible through build tools. This doesn’t replace the requirement to test the components but helps the developer to maintain its components with automated tools.

By design the framework must run in DI (plain standalone java program) but also in Beam pipelines. It is also out of scope of the framework to handle the way the runtime serializes - if needed - the data. For that reason it is primordial to not import serialization constraint in the stack. This is why JsonObject is not an IndexedRecord from avro for instance, to not impose any implementation. Any actual serialization concern - implementation - should either be hidden in the framework runtime (= outside component developer scope) or in the runtime integration with the framework (beam integration for instance). In this context, JSON-P is a good compromise because it brings a very powerful API with very few constraints.

This concept is borrowed to big data world and useful only in this context (BEAM executions). Overall idea is to divide the work before executing it to try to reduce the overall execution time.

The process is the following:

A partition mapper requires 3 methods marked with specific annotations:

A producer is a very simple component which MUST have a @Producer method without any parameter and returning any data:

A processor MUST have a method decorated with @ElementListener taking an incoming data and returning the processed data:

A processor also supports @BeforeGroup and @AfterGroup which MUST be methods without parameters and returning void (result would be ignored). This is used by the runtime to mark a chunk of the data in a way which is estimated good for the execution flow size.

The common usage is to batch records for performance reasons:

In some case you may want to split the output of a processor in two. A common example is "main" and "reject" branches where part of the incoming data are put in a specific bucket to be processed later.

This can be done using @Output. This can be used as a replacement of the returned value:

Or you can pass it a string which will represent the new branch:

Having multiple inputs is closeto the output case excep it doesn’t require a wrapper OutputEmitter:

@Input takes the input name as parameter, if not set it uses the main (default) input branch.

Conceptually an output is a listener of data. It perfectly matches the concept of processor. Being the last of the execution chain or returning no data will make your processor an output:

For now Talend Component doesn’t enable you to define a Combiner. It would be the symmetric part of the partition mapper and allow to aggregate results in a single one.

Every component family and component need to have a representative icon. You can use one of the icons provided by the component framework or you can use a custom icon. For the component family the icon is defined in package-info.java and for the component it need to be declared in the component class.

To use a custom icon, you need to have the icon file placed in resources/icons folder of the project. The icon file need to have a name following the convention IconName_icon32.png

What is considered as a primitive in this mecanism is a class which can be directly converted from a String to the expected type.

It obviously includes all java primitives, String type itself but also all the types with a org.apache.xbean.propertyeditor.Converter.

This includes out of the box:

The conversion from properties to object is using the dotted notation. For instance:

will match

assuming the method parameter was configured with @Option("file").

It is common to need to add as metadata a field is required, another has a minimum size etc. This is done with the validation in org.talend.sdk.component.api.configuration.constraint package:

It is common to classify the incoming data. You can see it as tagging them in several types. The most common ones are the:

Those configuration types can be composed to provide one configuration item. For example a dataset type will often need a datastore type to be provided. and a datastore type (that provides the connection information) will be used to create a dataset type.

Those configuration types will also be used at design time to create shared configuration that can be stored and used at runtime.

For example, we can think about a relational database that support JDBC:

The component server will scan all those configuration types and provide a configuration type index. This index can be used for the integration into the targeted platforms (studio, web applications…​)

The configuration type index is represented as a flat tree that contains all the configuration types represented as nodes and indexed by their ids.

Also, every node can point to other nodes. This relation is represented as an array of edges that provide the childes ids.

For example, a configuration type index for the above example will be:

It can be needed to define a binding between properties, a set of annotations allows to do it:

Target element location is specified as a relative path to current location using Unix path characters. Configuration class delimiter is /. Parent configuration class is specified by ... Thus ../targetProperty denotes a property, which is located in parent configuration class and has name targetProperty.

In some case it can be needed to add some metadata about the configuration to let the UI render properly the configuration. A simple example is a password value must be hidden and not a simple clear input box. For these cases - when the component developper wants to influence the UI rendering - you can use a particular set of annotations:

There are also other types of validation similar to @Pattern that you can use :

Components can require a few metadata to be integrated in Talend Studio or Cloud platform. Here is how to provide these information. These metadata are set on the component class and belongs to org.talend.sdk.component.api.component package.

Example:

Out of the box components are internationalized using the same location logic for the resource bundle and here is the list of supported keys:

Example of configuration for a component named list belonging to the family memory (@Emitter(family = "memory", name = "list")):

Configuration class are also translatable using the simple class name in the messages properties file. This useful when you have some common configuration shared within multiple components.

If you have a configuration class like :

You can give it a translatable display name by adding ${simple_class_name}.${property_name}._displayName to Messages.properties under the same package as the config class.

A plugin is just a jar which was enriched with the list of its dependencies. By default Talend Component runtime is able to read the output of maven-dependency-plugin in TALEND-INF/dependencies.txt location so you just need to ensure your component defines the following plugin:

If you check your jar once built you will see that the file contains something like:

What is important to see is the scope associated to the artifacts:

Even if a flat classpath deployment is possible, it is not recommended because it would then reduce the capabilities of the components.

The framework uses two kind of filtering when scanning your component. One based on the jar name and one based on the package name. Ensure your component definitions (including services) are in a scanned module if not registered manually using ComponentManager.instance().addPlugin() and that its package is not excluded.

The first goal is a shortcut for the maven-dependency-plugin, it will create the TALEND-INF/dependencies.txt file with the compile and runtime dependencies to let the component use it at runtime:

The most important goal is here to help you to validate the common programming model of the component. Here is the execution definition to activate it:

By default it will be bound to process-classes phase. When executing it will do several validations which can be switched off adding the corresponding flags to false in the <configuration> block of the execution:

This goal generates an Asciidoc file documenting your component from the configuration model (@Option) and @Documentation you can put on options and the component itself.

Testing the rendering of your component(s) configuration into the Studio is just a matter of deploying a component in Talend Studio (you can have a look to link::studio.html[Studio Documentation] page. But don’t forget the component can also be deployed into a Cloud (web) environment. To ease the testing of the related rendering, you can use the goal web of the plugin:

Then you can test your component going on localhost:8080. You need to select which component form you want to see using the treeview on the left, then on the right the form will be displayed.

The two available configurations of the plugin are serverPort which is a shortcut to change the default, 8080, port of the embedded server and serverArguments to pass Meecrowave options to the server. More on that configuration is available at openwebbeans.apache.org/meecrowave/meecrowave-core/cli.html.

The Mojo generate (maven plugin goal) of the same plugin also embeds a generator you can use to bootstrap any input or output component:

It is intended to be used from the command line (or IDE Maven integration):

For this command to work you will need to just register the plugin:

Component ARchive (.car) is the way to bundle a component to share it in Talend ecosystem. It is a plain Java ARchive (.jar) containing a metadata file and a nested maven repository containing the component and its depenencies.

It will create a .car in your build directory which is shareable on Talend platforms.

Note that this CAR is executable and exposes the command studio-deploy which takes as parameter a Talend Studio home location. Executed it will install the dependencies into the studio and register the component in your instance. Here is a sample launch command:

You can also upload the dependencies to your nexus server using the following command:

In this command Nexus Url and repository name are mandatory arguments. All other arguments are optional. If arguments contain spaces or special symbols, you need to quote the whole value of the argument, for example

Overal idea is to design its messages as methods returning String values and back the template by a ResourceBundle located in the same package than the interface defining these methods and named Messages.

To ensure you internationalization API is identified you need to mark it with @Internationalized:

Some actions are that common and need a clear contract so they are defined as API first citizen, this is the case for wizards or healthchecks for instance. Here is the list of all actions:

Let assume that we have a REST API defined like below, and that it requires a basic authentication header.

To create an http client able to consume this REST API, we will define an interface that extends HttpClient,

The HttpClient interface lets you set the base for the http address that our client will hit.

The base is the part of the address that we will need to add to the request path to hit the api.

Every method annotated with @Request of our interface will define an http request. Also every request can have @Codec that let us encode/decode the request/response playloads.

In the codec classes (class that implement Encoder/Decoder) you can inject any of your services annotated with @Service or @Internationalized into the constructor. The i18n services can be useful to have i18n messages for errors handling for example.

This interface can be injected into our Components classes or Services to consume the defined api.

Note: by default /+json are mapped to JSON-P and /+xml to JAX-B if the model has a @XmlRootElement annotation.

The Job builder let you create a job pipeline programmatically using Talend components (Producers and Processors). The job pipeline is an acyclic graph, so you can built complex pipelines.

Let’s take a simple use case where we will have 2 data source (employee and salary) that we will format to csv and write the result to a file.

A job is defined based on components (nodes) and links (edges) to connect their branches together.

Every component is defined by an unique id and an URI that identify the component.

The URI follow the form : [family]://[component][?version][&configuration]

Here is a more concrete job example:

For beam case, you need to rely on beam pipeline definition and use component-runtime-beam dependency which provides Beam bridges.

To extend the UI just add an annotation which can be put on @Option fields which is decorated with @Ui. All its members will be put in the metadata of the parameter. Example:

This is a great solution to repeat the same test multiple times. Overall idea is to define a test scenario (I test function F) and to make the input/output data dynamic.

component-runtime-junit is a small test library allowing you to validate simple logic based on Talend Component tooling.

To import it add to your project the following dependency:

This dependency also provide some mocked components that you can use with your own component to create tests.

The mocked components are provided under the family test :

The folowing artifact will allow you to test against a spark cluster:

The HTTP JUnit module allows you to mock REST API very easily. Here are its coordinates:

It supports JUnit 4 and JUnit 5 as well but the overall concept is the exact same one: the extension/rule is able to serve precomputed responses saved in the classpath.

You can plug your own ResponseLocator to map a request to a response but the default implementation - which should be sufficient in most cases - will look in talend/testing/http/<class name>_<method name>.json. Note that you can also put it in talend/testing/http/<request path>.json.

The MultiEnvironmentsRunner will execute the test(s) for each defined environments. It means it will run test1 for Env1 and Env2 in previous example.

By default JUnit4 runner will be used to execute the tests in one environment but you can use @DelegateRunWith to use another runner.

JUnit 5 configuration is close to JUnit 4 one:

The main difference is you don’t use a runner (it doesn’t exist in JUnit 5) and you replace @Test by @EnvironmentalTest.

The provided environment setup the contextual classloader to load the related runner of Apache Beam.

Package: org.talend.sdk.component.junit.environment.builtin.beam

If the environment extends BaseEnvironmentProvider and therefore defines an environment name - which is the case of the default ones, you can use EnvironmentConfiguration to customize the system properties used for that environment:

Here is an example. Let’s assume we have this test which validates the connection URI using ConnectionService:

We clearly identify the test method is always the same except the value. It can therefore be rewritter using JUnit Parameterized runner like that:

JUnit 5 reworked this feature to make it way easier to use. The full documentation is available at junit.org/junit5/docs/current/user-guide/#writing-tests-parameterized-tests.

The main difference is you can also define inline on the test method that it is a parameterized test and which are the values:

However you can still use the previous behavior using a method binding configuration:

This last option allows you to inject any type of value - not only primitives - which is very common to define scenarii.

Then you can define a standard JUnit test and use the SimpleComponentRule rule:

The JUnit 5 integration is mainly the same as for JUnit 4 except it uses the new JUnit 5 extension mecanism.

The entry point is the @WithComponents annotation you put on your test class which takes the component package you want to test and you can use @Injected to inject in a test class field an instance of ComponentsHandler which exposes the same utilities than the JUnit 4 rule:

Using the component "test"/"collector" as in previous sample stores all records emitted by the chain (typically your source) in memory, you can then access them using theSimpleComponentRule.getCollectedData(type). Note that this method filters by type, if you don’t care of the type just use Object.class.

The input mocking is symmetric to the output but here you provide the data you want to inject:

The component configuration is a POJO (using @Option on fields) and the runtime configuration (ExecutionChainBuilder) uses a Map<String, String>. To make the conversion easier, the JUnit integration provides a SimpleFactory.configurationByExample utility to get this map instance from a configuration instance.

Example:

The same factory provides a fluent DSL to create configuration calling configurationByExample without any parameter. The advantage is to be able to convert an object as a Map<String, String> as seen previously or as a query string to use it with the Job DSL:

It handles the encoding of the URI to ensure it is correctly done.

The SimpleComponentRule also allows to test a mapper unitarly, you can get an instance from a configuration and you can execute this instance to collect the output. Here is a snippet doing that:

As for the mapper a processor is testable unitary. The case is a bit more complex since you can have multiple inputs and outputs:

Here again the rule allows you to instantiate a Processor from your code and then to collect the output from the inputs you pass in. There are two convenient implementation of the input factory:

The usage relies on a JUnit TestRule. It is recommended to use it as a @ClassRule to ensure a single instance of a spark cluster is built but you can also use it as a simple @Rule which means it will be created per method instead of per test class.

It takes as parameter the spark and scala version to use. It will then fork a master and N slaves. Finally it will give you submit* method allowing you to send jobs either from the test classpath or from a shade if you run it as an integration test.

Here is a sample:

The integration with JUnit 5 of that spark cluster logic uses @WithSpark marker for the extension and let you, optionally, inject through @SparkInject, the BaseSpark<?> handler to access te spark cluster meta information - like its host/port.

Here is a basic test using it:

In current state, SparkClusterRule doesn’t allow to know a job execution is done - even if it exposes the webui url so you can poll it to check. The best at the moment is to ensure the output of your job exists and contains the right value.

awaitability or equivalent library can help you to write such logic.

Here are the coordinates of the artifact:

And here is how to wait a file exists and its content (for instance) is the expected one:

JUnit 4 setup is done through two rules: JUnit4HttpApi which is responsible to start the server and JUnit4HttpApiPerMethodConfigurator which is responsible to configure the server per test and also handle the capture mode (see later).

Most of the test will look like:

JUnit 5 uses a JUnit 5 extension based on the HttpApi annotation you can put on your test class. You can inject the test handler (which has some utilities for advanced cases) through @HttpApiInject:

The strength of this implementation is to run a small proxy server and auto configure the JVM: http[s].proxyHost, http[s].proxyPort, HttpsURLConnection#defaultSSLSocketFactory and SSLContext#default are auto configured to work out of the box with the proxy.

It allows you to keep in your tests the native and real URLs. For instance this test is perfectly valid:

If you execute this test, it will fail with a HTTP 400 because the proxy doesn’t find the mocked response. You can create it manually as seen in the introduction of the module but you can also set the property talend.junit.http.capture to the folder where to store the captures. It must be the root folder and not the folder where the json are (ie not prefixed by talend/testing/http by default).

Generally you will want to use src/test/resources. If new File("src/test/resources") resolves to the valid folder when executing your test (Maven default), then you can just set the system property to true, otherwise you need to adjust accordingly the system property value.

Once you ran the tests with this system property, the testing framework will have created the correct mock response files and you can remove the system property. The test will still pass, using google.com…​even if you disconnect your machine from the internet.

The rule (extension) is doing all the work for you :).

Setting talend.junit.http.passthrough system property to true, the server will just be a proxy and will execute each request to the actual server - like in capturing mode.

Dependencies:

These dependencies brings into the test scope the JUnit testing toolkit, the Beam integration and the multi-environment testing toolkit for JUnit.

Then using the fluent DSL to define jobs - which assumes your job is linear and each step sends a single value (no multi-input/multi-output), you can write this kind of test:

It will execute the chain twice:

The light validations are all the validations you can execute on the client side. They are listed in the UI hint part.

This is the ones to use first before going with custom validations since they will be more efficient.

These ones will enforce custom code to be executed, they are more heavy so try to avoid to use them for simple validations you can do with the previous part.

Here you define an action taking some parameters needed for the validation and you link the option you want to validate to this action. Here is an example to validate a dataset. For example for our JDBC driver we could have:

Note that you can also make a class validable and you can use it to validate a form if you put it on your whole configuration:

This endpoint will execute any UI action and serialize the response as a JSON (pojo model) It takes as input the family, type and name of the related action to identify it and its configuration as a flat key value set using the same kind of mapping than for components (option path as key).

This endpoint returns the list of available actions for a certain family and potentially filters the " output limiting it to some families and types of actions.

Returns a list of dependencies for the given components.

Then you can use /dependency/{id} to download the binary.

Return a binary of the dependency represented by id. It can be maven coordinates for dependencies or a component id.

Returns the set of metadata about a few components identified by their 'id'.

Returns a particular family icon in raw bytes.

Returns a particular component icon in raw bytes.

Returns the list of available components.

Allows to migrate a component configuration without calling any component execution.

Returns the set of metadata about a few configurations identified by their 'id'.

Returns all available configuration type - storable models. Note that the lightPayload flag allows to load all of them at once when you eagerly need to create a client model for all configurations.

Allows to migrate a configuration without calling any component execution.

Returns an asciidoctor version of the documentation for the component represented by its identifier id.

Format can be either asciidoc or html - if not it will fallback on asciidoc - and if html is selected you get a partial document.

The documentation will likely be the family documentation but you can use anchors to access a particular component (_componentname_inlowercase).

Returns the environment of this instance. Useful to check the version or configure a healthcheck for the server.

Read inputs from an instance of mapper. The number of returned records if enforced to be limited to 1000. The format is a JSON based format where each like is a json record.

Sends records using a processor instance. Note that the processor should have only an input. Behavior for other processors is undefined. The input format is a JSON based format where each like is a json record - same as for the symmetric endpoint.