| A test environment is available on Heroku and can be browsed using Talend Component Kit Server instance on Restlet Studio. |
HTTP API
The HTTP API intends to expose most Talend Component Kit features over HTTP. It is a standalone Java HTTP server.
The WebSocket protocol is activated for the endpoints. Endpoints then use /websocket/v1 as base instead of /api/v1. See WebSocket for more details.
|
Here is the API:
REST resources of Component Runtime :: Server Parent :: Server
1.0.3-SNAPSHOT
POST api/v1/action/execute
This endpoint will execute any UI action and serialize the response as a JSON (pojo model) It takes as input the family, type and name of the related action to identify it and its configuration as a flat key value set using the same kind of mapping than for components (option path as key).
Request
Content-Type: application/json
Request Body: (java.util.Map<java.lang.String, java.lang.String>)
Query Param: action, java.lang.String
Query Param: family, java.lang.String
Query Param: lang, java.lang.String
Query Param: type, java.lang.String
Response
Content-Type: application/json
400 Bad Request
Response Body: (org.talend.sdk.component.server.front.model.error.ErrorPayload)
{
"code": "ACTION_ERROR|ACTION_MISSING|BAD_FORMAT|COMPONENT_MISSING|CONFIGURATION_MISSING|DESIGN_MODEL_MISSING|FAMILY_MISSING|ICON_MISSING|PLUGIN_MISSING|UNAUTHORIZED|UNEXPECTED",
"description": "string"
}GET api/v1/action/index
This endpoint returns the list of available actions for a certain family and potentially filters the " output limiting it to some families and types of actions.
Request
No body
Query Param: family, java.lang.String
Query Param: language, java.lang.String
Query Param: type, java.lang.String
Response
Content-Type: application/json
200 OK
Response Body: (org.talend.sdk.component.server.front.model.ActionList)
{
"items": [
{
"component": "string",
"name": "string",
"properties": [
{
"defaultValue": "string",
"displayName": "string",
"metadata": {
},
"name": "string",
"path": "string",
"placeholder": "string",
"proposalDisplayNames": {
},
"type": "string",
"validation": {
"enumValues": [
"string"
],
"max": 0,
"maxItems": 0,
"maxLength": 0,
"min": 0,
"minItems": 0,
"minLength": 0,
"pattern": "string",
"required": false,
"uniqueItems": false
}
}
],
"type": "string"
}
]
}GET api/v1/component/dependencies
Returns a list of dependencies for the given components.
| don’t forget to add the component itself since it will not be part of the dependencies. |
Then you can use /dependency/{id} to download the binary.
GET api/v1/component/dependency/{id}
Return a binary of the dependency represented by id.
It can be maven coordinates for dependencies or a component id.
GET api/v1/component/details
Returns the set of metadata about a few components identified by their 'id'.
Response
Content-Type: application/json
200 OK
Response Body: (org.talend.sdk.component.server.front.model.ComponentDetailList)
{
"details": [
{
"actions": [
{
"family": "string",
"name": "string",
"properties": [
{
"defaultValue": "string",
"displayName": "string",
"metadata": {
},
"name": "string",
"path": "string",
"placeholder": "string",
"proposalDisplayNames": {
},
"type": "string",
"validation": {
"enumValues": [
"string"
],
"max": 0,
"maxItems": 0,
"maxLength": 0,
"min": 0,
"minItems": 0,
"minLength": 0,
"pattern": "string",
"required": false,
"uniqueItems": false
}
}
],
"type": "string"
}
],
"displayName": "string",
"icon": "string",
"id": {
"family": "string",
"familyId": "string",
"id": "string",
"name": "string",
"plugin": "string",
"pluginLocation": "string"
},
"inputFlows": [
"string"
],
"links": [
{
"contentType": "string",
"name": "string",
"path": "string"
}
],
"outputFlows": [
"string"
],
"properties": [
{
"defaultValue": "string",
"displayName": "string",
"metadata": {
},
"name": "string",
"path": "string",
"placeholder": "string",
"proposalDisplayNames": {
},
"type": "string",
"validation": {
"enumValues": [
"string"
],
"max": 0,
"maxItems": 0,
"maxLength": 0,
"min": 0,
"minItems": 0,
"minLength": 0,
"pattern": "string",
"required": false,
"uniqueItems": false
}
}
],
"type": "string",
"version": 0
}
]
}GET api/v1/component/icon/family/{id}
Returns a particular family icon in raw bytes.
GET api/v1/component/icon/{id}
Returns a particular component icon in raw bytes.
GET api/v1/component/index
Returns the list of available components.
Response
Content-Type: application/json
200 OK
Response Body: (org.talend.sdk.component.server.front.model.ComponentIndices)
{
"components": [
{
"categories": [
"string"
],
"displayName": "string",
"familyDisplayName": "string",
"icon": {
"customIcon": {
},
"customIconType": "string",
"icon": "string"
},
"iconFamily": {
"customIcon": {
},
"customIconType": "string",
"icon": "string"
},
"id": {
"family": "string",
"familyId": "string",
"id": "string",
"name": "string",
"plugin": "string",
"pluginLocation": "string"
},
"links": [
{
"contentType": "string",
"name": "string",
"path": "string"
}
],
"version": 0
}
]
}POST api/v1/component/migrate/{id}/{configurationVersion}
Allows to migrate a component configuration without calling any component execution.
GET api/v1/configurationtype/details
Returns the set of metadata about a few configurations identified by their 'id'.
GET api/v1/configurationtype/index
Returns all available configuration type - storable models. Note that the lightPayload flag allows to load all of them at once when you eagerly need to create a client model for all configurations.
POST api/v1/configurationtype/migrate/{id}/{configurationVersion}
Allows to migrate a configuration without calling any component execution.
GET api/v1/documentation/component/{id}
Returns an asciidoctor version of the documentation for the component represented by its identifier id.
Format can be either asciidoc or html - if not it will fallback on asciidoc - and if html is selected you get a partial document.
| it is recommended to use asciidoc format and handle the conversion on your side if you can, the html flavor handles a limited set of the asciidoc syntax only like plain arrays, paragraph and titles. |
The documentation will likely be the family documentation but you can use anchors to access a particular component (_componentname_inlowercase).
GET api/v1/environment
Returns the environment of this instance. Useful to check the version or configure a healthcheck for the server.
POST api/v1/execution/read/{family}/{component}
| deprecated |
Read inputs from an instance of mapper. The number of returned records if enforced to be limited to 1000. The format is a JSON based format where each like is a json record.
POST api/v1/execution/write/{family}/{component}
| deprecated |
Sends records using a processor instance. Note that the processor should have only an input. Behavior for other processors is undefined. The input format is a JSON based format where each like is a json record - same as for the symmetric endpoint.
Request
Content-Type: talend/stream
Request Body: (java.io.InputStream)
Path Param: component, java.lang.String
Path Param: family, java.lang.String
Query Param: group-size, long
Deprecated endpoints
Endpoints that are intended to disappear will be deprecated. A X-Talend-Warning header will be returned with a message as value.
WebSocket transport
You can connect yo any endpoint by:
-
Replacing
/apiwith/websocket -
Appending
/<http method>to the URL -
Formatting the request as:
SEND
destination: <endpoint after v1>
<headers>
<payload>^@For example:
SEND
destination: /component/index
Accept: application/json
^@The response is formatted as follows:
MESSAGE
status: <http status code>
<headers>
<payload>^@| All endpoints are logged at startup. You can then find them in the logs if you have a doubt about which one to use. |
If you don’t want to create a pool of connections per endpoint/verb, you can use the bus endpoint: /websocket/v1/bus.
This endpoint requires that you add the destinationMethod header to each request with the verb value (GET by default):
SEND
destination: /component/index
destinationMethod: GET
Accept: application/json
^@HTTPS activation
Using the server ZIP (or Docker image), you can configure HTTPS by adding properties to MEECROWAVE_OPTS. Assuming that you have a certificate in /opt/certificates/component.p12 (don’t forget to add/mount it in the Docker image if you use it), you can activate it as follows:
# use -e for Docker
#
# this skips the http port binding and only binds https on the port 8443, and setups the correct certificate
export MEECROWAVE_OPTS="-Dskip-http=true -Dssl=true -Dhttps=8443 -Dkeystore-type=PKCS12 -Dkeystore-alias=talend -Dkeystore-password=talend -Dkeystore-file=/opt/certificates/component.p12"Web forms and REST API
The component-form library provides a way to build a component REST API facade that is compatible with React form library.
for example:
@Path("tacokit-facade")
@ApplicationScoped
public class ComponentFacade {
private static final String[] EMPTY_ARRAY = new String[0];
@Inject
private Client client;
@Inject
private ActionService actionService;
@Inject
private UiSpecService uiSpecService;
@Inject // assuming it is available in your app, use any client you want
private WebTarget target;
@POST
@Path("action")
public void action(@Suspended final AsyncResponse response, @QueryParam("family") final String family,
@QueryParam("type") final String type, @QueryParam("action") final String action,
final Map<String, Object> params) {
client.action(family, type, action, params).handle((r, e) -> {
if (e != null) {
onException(response, e);
} else {
response.resume(actionService.map(type, r));
}
return null;
});
}
@GET
@Path("index")
public void getIndex(@Suspended final AsyncResponse response,
@QueryParam("language") @DefaultValue("en") final String language) {
target
.path("component/index")
.queryParam("language", language)
.request(APPLICATION_JSON_TYPE)
.rx()
.get(ComponentIndices.class)
.toCompletableFuture()
.handle((index, e) -> {
if (e != null) {
onException(response, e);
} else {
index.getComponents().stream().flatMap(c -> c.getLinks().stream()).forEach(
link -> link.setPath(link.getPath().replaceFirst("/component/", "/application/").replace(
"/details?identifiers=", "/detail/")));
response.resume(index);
}
return null;
});
}
@GET
@Path("detail/{id}")
public void getDetail(@Suspended final AsyncResponse response,
@QueryParam("language") @DefaultValue("en") final String language, @PathParam("id") final String id) {
target
.path("component/details")
.queryParam("language", language)
.queryParam("identifiers", id)
.request(APPLICATION_JSON_TYPE)
.rx()
.get(ComponentDetailList.class)
.toCompletableFuture()
.thenCompose(result -> uiSpecService.convert(result.getDetails().iterator().next()))
.handle((result, e) -> {
if (e != null) {
onException(response, e);
} else {
response.resume(result);
}
return null;
});
}
private void onException(final AsyncResponse response, final Throwable e) {
final UiActionResult payload;
final int status;
if (WebException.class.isInstance(e)) {
final WebException we = WebException.class.cast(e);
status = we.getStatus();
payload = actionService.map(we);
} else if (CompletionException.class.isInstance(e)) {
final CompletionException actualException = CompletionException.class.cast(e);
log.error(actualException.getMessage(), actualException);
status = Response.Status.BAD_GATEWAY.getStatusCode();
payload = actionService.map(new WebException(actualException, -1, emptyMap()));
} else {
log.error(e.getMessage(), e);
status = Response.Status.BAD_GATEWAY.getStatusCode();
payload = actionService.map(new WebException(e, -1, emptyMap()));
}
response.resume(new WebApplicationException(Response.status(status).entity(payload).build()));
}
}
the Client can be created using ClientFactory.createDefault(System.getProperty("app.components.base", "http://localhost:8080/api/v1")) and the service can be a simple new UiSpecService<>(). The factory uses JAX-RS if the API is available (assuming a JSON-B provider is registered). Otherwise, it tries to use Spring.
|
The conversion from the component model (REST API) to the uiSpec model is done through UiSpecService. It is based on the object model which is mapped to a UI model. Having a flat model in the component REST API allows to customize layers easily.
You can completely control the available components, tune the rendering by switching the uiSchema, and add or remove parts of the form.
You can also add custom actions and buttons for specific needs of the application.
The /migrate endpoint was not shown in the previous snippet but if you need it, add it as well.
|
Using the UiSpec model without the tooling
<dependency>
<groupId>org.talend.sdk.component</groupId>
<artifactId>component-form-model</artifactId>
<version>${talend-component-kit.version}</version>
</dependency>This Maven dependency provides the UISpec model classes. You can use the Ui API (with or without the builders) to create UiSpec representations.
For Example:
final Ui form1 = ui()
// (1)
.withJsonSchema(JsonSchema.jsonSchemaFrom(Form1.class).build())
// (2)
.withUiSchema(uiSchema()
.withKey("multiSelectTag")
.withRestricted(false)
.withTitle("Simple multiSelectTag")
.withDescription("This data list accepts values that are not in the list of suggestions")
.withWidget("multiSelectTag")
.build())
// (3)
.withProperties(myFormInstance)
.build();
// (4)
final String json = jsonb.toJson(form1);| 1 | The JsonSchema is extracted from reflection on the Form1 class. Note that @JsonSchemaIgnore allows to ignore a field and @JsonSchemaProperty allows to rename a property. |
| 2 | A UiSchema is programmatically built using the builder API. |
| 3 | An instance of the form is passed to let the serializer extract its JSON model. |
| 4 | The Ui model, which can be used by UiSpec compatible front widgets, is serialized. |
The model uses the JSON-B API to define the binding. Make sure to have an implementation in your classpath. To do that, add the following dependencies:
<dependency>
<groupId>org.apache.geronimo.specs</groupId>
<artifactId>geronimo-jsonb_1.0_spec</artifactId>
<version>1.0</version>
</dependency>
<dependency>
<groupId>org.apache.geronimo.specs</groupId>
<artifactId>geronimo-json_1.1_spec</artifactId>
<version>1.0</version>
</dependency>
<dependency>
<groupId>org.apache.johnzon</groupId>
<artifactId>johnzon-jsonb</artifactId>
<version>${johnzon.version}</version> <!-- 1.1.5 for instance -->
</dependency>JavaScript integration
Default JavaScript integration goes through the Talend UI Forms library.
It is bundled as a NPM module called component-kit.js. It provides a default trigger implementation for UIForm.
Here is how to use it:
import React from 'react';
import UIForm from '@talend/react-forms/lib/UIForm/UIForm.container';
import TalendComponentKitTrigger from 'component-kit.js';
export default class ComponentKitForm extends React.Component {
constructor(props) {
super(props);
this.trigger = new TalendComponentKitTrigger({ url: '/api/to/component/server/proxy' });
this.onTrigger = this.onTrigger.bind(this);
// ...
}
onTrigger(event, payload) {
return this.trigger.onDefaultTrigger(event, payload);
}
// ...
render() {
if(! this.state.uiSpec) {
return (<div>Loading ...</div>);
}
return (
<UIForm
data={this.state.uiSpec}
onTrigger={this.onTrigger}
onSubmit={this.onSubmit}
/>
);
}
}Logging
The logging uses Log4j2. You can specify a custom configuration by using the -Dlog4j.configurationFile system property or by adding a log4j2.xml file to the classpath.
Here are some common configurations:
-
Console logging:
<?xml version="1.0"?>
<Configuration status="INFO">
<Appenders>
<Console name="Console" target="SYSTEM_OUT">
<PatternLayout pattern="[%d{HH:mm:ss.SSS}][%highlight{%-5level}][%15.15t][%30.30logger] %msg%n"/>
</Console>
</Appenders>
<Loggers>
<Root level="INFO">
<AppenderRef ref="Console"/>
</Root>
</Loggers>
</Configuration>Output messages look like:
[16:59:58.198][INFO ][ main][oyote.http11.Http11NioProtocol] Initializing ProtocolHandler ["http-nio-34763"]-
JSON logging:
<?xml version="1.0"?>
<Configuration status="INFO">
<Properties>
<!-- DO NOT PUT logSource there, it is useless and slow -->
<Property name="jsonLayout">{"severity":"%level","logMessage":"%encode{%message}{JSON}","logTimestamp":"%d{ISO8601}{UTC}","eventUUID":"%uuid{RANDOM}","@version":"1","logger.name":"%encode{%logger}{JSON}","host.name":"${hostName}","threadName":"%encode{%thread}{JSON}","stackTrace":"%encode{%xThrowable{full}}{JSON}"}%n</Property>
</Properties>
<Appenders>
<Console name="Console" target="SYSTEM_OUT">
<PatternLayout pattern="${jsonLayout}"/>
</Console>
</Appenders>
<Loggers>
<Root level="INFO">
<AppenderRef ref="Console"/>
</Root>
</Loggers>
</Configuration>Output messages look like:
{"severity":"INFO","logMessage":"Initializing ProtocolHandler [\"http-nio-46421\"]","logTimestamp":"2017-11-20T16:04:01,763","eventUUID":"8b998e17-7045-461c-8acb-c43f21d995ff","@version":"1","logger.name":"org.apache.coyote.http11.Http11NioProtocol","host.name":"TLND-RMANNIBUCAU","threadName":"main","stackTrace":""}-
Rolling file appender:
<?xml version="1.0"?>
<Configuration status="INFO">
<Appenders>
<RollingRandomAccessFile name="File" fileName="${LOG_PATH}/application.log" filePattern="${LOG_PATH}/application-%d{yyyy-MM-dd}.log">
<PatternLayout pattern="[%d{HH:mm:ss.SSS}][%highlight{%-5level}][%15.15t][%30.30logger] %msg%n"/>
<Policies>
<SizeBasedTriggeringPolicy size="100 MB" />
<TimeBasedTriggeringPolicy interval="1" modulate="true"/>
</Policies>
</RollingRandomAccessFile>
</Appenders>
<Loggers>
<Root level="INFO">
<AppenderRef ref="File"/>
</Root>
</Loggers>
</Configuration>More details are available in the RollingFileAppender documentation.
| You can compose previous layout (message format) and appenders (where logs are written). |
UiSpec Server
The UiSpec server is a companion application for the Component Server.
It provides a client to the Component Server which serves UiSpec payload to integrate with the client JavaScript UiForm library.
Coordinates
<dependency>
<groupId>org.talend.sdk.component</groupId>
<artifactId>component-server-proxy</artifactId>
<version>${server-proxy.version}</version>
</dependency>Configuring the UiSpec server
the configuration is read from system properties, environment variables, ….
If you use playx-microprofile-config, you can also use typesafe configuration.
|
| Key | Description | Default |
|---|---|---|
talend.component.proxy.actions.proposable.cached |
If true the proposable (suggestion lists only depending on the server state) will be cached, otherwise they will be requested for each form rendering. |
true |
talend.component.proxy.application.home |
A home location for relative path resolution (optional). |
${playx.application.home} |
talend.component.proxy.client.providers |
List of JAX-RS providers to register on the client, at least a JSON-B one should be here. |
- |
talend.component.proxy.client.timeouts.connect |
The connect timeout for the communication with the server.base in ms. |
60000 |
talend.component.proxy.client.timeouts.read |
The read timeout for the communication with the server.base in ms. |
600000 |
talend.component.proxy.jcache.active |
Should the server use jcache to store catalog information and refresh it with some polling. If so the keys |
true |
talend.component.proxy.jcache.provider |
Caching provider implementation to use (only set it if ambiguous). |
- |
talend.component.proxy.jcache.refresh.period |
Number of seconds used to check if the server must be refreshed. |
60 |
talend.component.proxy.processing.headers |
The headers to append to the request when contacting the server. Format is a properties one. You can put a hardcoded value or a placeholder ( |
- |
talend.component.proxy.processing.uiSpec.patch |
An optional location (absolute or resolved from The values can be keys in the resource bundle |
component-uispec-metadata.%s.json?force=false |
talend.component.proxy.server.base |
The base to contact the remote server (NOTE: it is recommanded to put a load balancer if you have multiple instances.) |
- |
Adding custom entries to the forms
As shown in the table above, you can customize the forms by type. The format reuses Talend Component Kit REST API (properties model) and defines two main types of extensions:
-
prependProperties: Lists all the models of properties added to the form before the actual underlying form. -
appendProperties: Lists all the models of properties added to the form after the actual underlying form.
If you don’t specify a name, the path is used to deduce the name automatically.
Always make sure to define a root object for these properties. Do not use dots in the path value. It is recommended to prefix it with a $ character.
|
Adding custom converters (selecting the widget or rendering)
When developing a org.talend.sdk.component.form.internal.converter.CustomPropertyConverter CDI, the proxy adds it to the UiSpecService service and uses it with a high priority to convert the server model to a UiSpec model.
To make it a CDI bean, add @Dependent to the class and if you use the Play integration, customize the bean array: playx.cdi.beans.customs += {className: org.talend.myapp.MyConverter}.
|
This allows to use a custom @Ui API and advanced modeling when specific to applications.
Converters are sorted respecting to the @Priority value. If the annotation is missing, the priority defaults to 0.
Client in Play
The client to use to connect to the Talend Component Kit server is the CXF client, using HttpClient HC (NIO) transport. When you use the Play module, it can be configured with its standard properties prefixed by talend.component.proxy..
You can find more information on CXF website.
Defining a dropdown with all root configurations
The special dynamic_values action builtin::roots can be used for a dropdown filled with all available root types.
Here is a sample patch file:
{
"prependProperties": [
{
"path": "$datasetMetadata",
"type": "OBJECT"
},
{
"path": "$datasetMetadata.type",
"displayName": "Types",
"type": "ENUM",
"metadata": {
"action::dynamic_values": "builtin::roots"
}
}
]
}Reloading the form based on the selected root
The builtin::root::reloadFromId action, with the jsonpatch type,
allows to reload the whole form:
{
"path": "$datasetMetadata.type",
"displayName": "Types",
"type": "STRING",
"metadata": {
"action::dynamic_values": "builtin::roots", (1)
"action::reloadForm": "builtin::root::reloadFromId", (2)
"action::reloadForm::parameters": "."
}
}| 1 | Prepopulating the dropdown with the list of datastores. |
| 2 | On selection of a datastore, refreshing the form with the new parameters. |
It is common to have a dropdown with the list of roots and to reload the form when one is selected.
For example, the UIForm part (JavaScript side) can be implemented as follows:
import kit from 'component-kit.js';
// ...
constructor(props) {
super(props);
this.state = {};
this.trigger = kit.createTriggers({
url: '/componentproxy/api/v1/actions/execute',
customRegistry: {
reloadForm: ({ body }) => { (1)
const { _datasetMetadata } = this.state.uiSpec.properties;
return {
...body, (2)
properties: { _datasetMetadata }, (3)
};
}
}
});
// ...
}
// ...
onTrigger(event, payload) {
return this.trigger(event, payload)
.then(result => {
if (result.properties || result.errors || result.uiSchema || result.jsonSchema) { (4)
this.setState({
uiSpec: {
...this.state.uiSpec,
...result,
}
});
}
});
}| 1 | Adding a custom handler for the specific reloadForm action. |
| 2 | Passing the uiSchema and jsonSchema to the next step in the response processing chain. |
| 3 | Resetting the dynamic part of the form. Only the static part is kept. |
| 4 | Merging back the result of the handler into the current state. You can use redux or cmf. |
HTTP API
Component UiSpec Server
Overview
These endpoints allow to obtain UiSpec representations of the component/configuration types properties.
Contact information
Contact : Talend
Contact Email : contact@talend.com
License information
License : Apache 2.0
License URL : www.apache.org/licenses/LICENSE-2.0.html
Terms of service : null
Paths
This endpoint execute an action required by a form.
POST /actions/execute
| Type | Name | Schema |
|---|---|---|
Query |
action |
string |
Query |
family |
string |
Query |
language |
string |
Query |
type |
string |
| HTTP Code | Description | Schema |
|---|---|---|
200 |
successful operation |
< string, CompletionStageMapStringObject > map |
400 |
This response is returned when the action is null |
|
404 |
This response is returned when no action is found |
|
520 |
This response is returned when the action raise an unhandled error |
Return all the available root configuration (Data store like) from the component server
GET /configurations
Every configuration has an icon. In the response an icon key is returned. this icon key can be one of the bundled icons or a custom one. The consumer of this endpoint will need to check if the icon key is in the icons bundle otherwise the icon need to be gathered using the familyId from this endpoint configurations/{id}/icon
Return a form description ( Ui Spec ) without a specific configuration
GET /configurations/form/initial/{type}
Return a form description ( Ui Spec ) of a specific configuration
GET /configurations/form/{id}
Return the configuration icon file in png format
GET /configurations/icon/{id}
Update a configuration.
POST /configurations/persistence/edit/{id}
Saves a configuration based on a type. Concretely it is the same as /persistence/save/{formId} but the formId is contained into the payload itself and marked in the metadata as such.
POST /configurations/persistence/save-from-type/{type}
Saves a configuration based on a form identifier.
POST /configurations/persistence/save/{formId}
Definitions
EntityRef
| Name | Description | Schema |
|---|---|---|
id |
The identifier of the entity related to current request. It is generally thecreated entity of updated one. |
string |
JsonSchema
| Name | Schema |
|---|---|
defaultValue |
object |
description |
string |
enumValues |
< string > array |
id |
string |
items |
|
maxItems |
integer (int32) |
maxLength |
integer (int32) |
maximum |
number (double) |
minItems |
integer (int32) |
minLength |
integer (int32) |
minimum |
number (double) |
pattern |
string |
properties |
< string, JsonSchema > map |
ref |
string |
required |
< string > array |
schema |
string |
title |
string |
type |
string |
uniqueItems |
boolean |
Node
| Name | Description | Schema |
|---|---|---|
children |
The list of configuration reusing this one as a reference (can be created "next"). |
< string > array |
familyId |
The identifier of the family of this configuration. |
string |
familyLabel |
The display name of the family of this configuration. |
string |
icon |
The icon of this configuration. If you use an existing bundle (@talend/ui/icon), ensure it is present by default and if not do a request using the family on the related endpoint. |
string |
id |
The identifier of this configuration/node. |
string |
label |
The display name of this configuration. |
string |
name |
The technical name of this node (it is human readable but not i18n friendly), useful for debug purposes. |
string |
version |
The version of this configuration for the migration management. |
integer (int32) |
Nodes
| Name | Description | Schema |
|---|---|---|
nodes |
The list of nodes matching the request. The key is the node identifier. |
< string, Node > map |
ProxyErrorPayload
| Name | Description | Schema |
|---|---|---|
code |
The error code independently of the locale and not as precise as a message (not context aware). |
string |
message |
A human readable message to help understanding the error |
string |
Trigger
| Name | Schema |
|---|---|
action |
string |
family |
string |
onEvent |
string |
options |
< Option > array |
parameters |
< Parameter > array |
type |
string |
Ui
| Name | Schema |
|---|---|
jsonSchema |
|
properties |
object |
uiSchema |
< UiSchema > array |
UiNode
| Name | Description | Schema |
|---|---|---|
metadata |
The metadata associated to the node if needed by the UI. |
|
ui |
The ui specification corresponding to the requested node. It is literally the form representing this configuration. |
UiSchema
| Name | Schema |
|---|---|
autoFocus |
boolean |
conditions |
< Condition > array |
description |
string |
disabled |
boolean |
itemWidget |
string |
items |
< UiSchema > array |
key |
string |
options |
< string, string > map |
placeholder |
string |
readOnly |
boolean |
required |
boolean |
restricted |
boolean |
title |
string |
titleMap |
< NameValue > array |
triggers |
< Trigger > array |
type |
string |
widget |
string |
There are two ways to call the save endpoint. If you don’t want to pass the form identifier and prefer to use a generic endpoint that simply passes the type of configuration you are configuring, then you need to modify your enrichment configuration to ensure that the form identifier is present and to specify which form field it is.
To do that, add the proxyserver::formId Boolean to the metadata:
{
"path": "$datasetMetadata.type",
"displayName": "Types",
"type": "STRING",
"metadata": {
// other metadata as seen previously
"proxyserver::formId": "true"
}
}
Only the first property with proxyserver::formId set to "true" is used. The path cannot contain any array.
|
Integrating with Play
Thanks to Playx, you can deploy this server in a Play! application.
-
Import the pre-configuration of the Play integration.
<dependency>
<groupId>org.talend.sdk.component</groupId>
<artifactId>component-server-proxy-play</artifactId>
<version>${server-proxy.version}</version>
</dependency>-
Configure the integration in your
application.conffile:
include "conf/component-proxy.play.conf" (1)
talend.component.proxy { (2)
server {
base = "http://localhost:8080/api/v1"
}
processing {
headers = """
X-ServiceName = proxy
X-ID = ${someRequestAttributeNameForTheId}
"""
// ...
}
}| 1 | Importing the global configuration that maps the endpoints on /componentproxy/*. |
| 2 | Customizing the configuration to specialize it for the current deployment. |
If you are using Guice
The org.talend.sdk.component.guice.ComponentUiSpecServerModule Guice module allows to import Services from the server and to use some of them locally for the parts that are out of the scope of the endpoint delivery. For example:
-
Converting
JsonObjecttoMap<String, String>and the other way around. -
Getting the properties model for a configuration or component.
Use a Provider<T> injection to keep it lazy.
|
The server does not handle two layers. It is possible that you need to add:
-
Security.
-
Persistence.
For the security layer, it is recommended to create a Play filter that you configure before the proxy (ServletFilter).
Make sure that you pass all contextual data in the request attributes to be able to get them back after.
For the persistence layer, the framework fires particular events (OnPersist, OnEdit and OnFindById) that you should observe in order to implement the persistence as you wish.
Reusing the CDI context of the server just a plain CDI observer works, but if you used the Guice/CDI setup (using the Play module of the integration) you need to write Guice observers.
OnPersist needs to become identifier-aware of the created entity. Make sure to call composeId. In a similar way, OnFindById requires you to pass the found instance with composeProperties() and composeFormId().
|
Optionally, you can implement OnFindByFormId, which allows to have completion for references. If your application does not allow to edit references, you can bypass this event. If you implement it, make sure to call composeResult no more than once.
@Singleton
public class PersistenceHandler {
@Subscribe
public void onPersist(final OnPersist event) {
// do persist, ensure to call event.composeId(...)
}
@Subscribe
public void onEdit(final OnEdit event) {
// do update
}
@Subscribe
public void onFind(final OnFindById event) {
// do find
}
@Subscribe
public void onFindByFormId(final OnFindByFormId event) {
// do find
}
}| Deletion is not handled by the server because there is no encapsulation of any format at that moment. |
Configuring the server module
The server module contains several configurations that you can set in:
-
Environment variables.
-
System properties.
-
A file located based on the
--component-configurationCLI option.
| the configuration is read from system properties, environment variables, …. |
| Key | Description | Default |
|---|---|---|
talend.component.server.component.coordinates |
A comma separated list of gav to locate the components |
- |
talend.component.server.component.extend.dependencies |
Should the component extensions add required dependencies. |
true |
talend.component.server.component.registry |
A property file where the value is a gav of a component to register (complementary with |
- |
talend.component.server.documentation.active |
Should the /documentation endpoint be activated. |
true |
talend.component.server.execution.dataset.retriever.timeout |
How long the read execution endpoint can last (max) |
180 |
talend.component.server.execution.pool.size |
The size of the execution pool for runtime endpoints. |
64 |
talend.component.server.execution.pool.wait |
How long the application waits during shutdown for the execution tasks to complete |
PT10S |
talend.component.server.jaxrs.exceptionhandler.defaultMessage |
If set it will replace any message for exceptions. Set to |
false |
talend.component.server.maven.repository |
The local maven repository used to locate components and their dependencies |
- |
talend.component.server.monitoring.brave.reporter.async |
When using url or kafka reporter, you can configure the async reporter with properties passed to this configuration entry.Ex: |
console |
talend.component.server.monitoring.brave.reporter.type |
The brave reporter to use to send the spans. Supported values are [log, noop]. When configuration is needed,you can use this syntax to configure the repoter if needed: In |
noop |
talend.component.server.monitoring.brave.sampling.action.rate |
The accuracy rate of the sampling for action endpoints. |
-1 |
talend.component.server.monitoring.brave.sampling.component.rate |
The accuracy rate of the sampling for component endpoints. |
-1 |
talend.component.server.monitoring.brave.sampling.configurationtype.rate |
The accuracy rate of the sampling for environment endpoints. |
-1 |
talend.component.server.monitoring.brave.sampling.documentation.rate |
The accuracy rate of the sampling for documentation endpoint. |
-1 |
talend.component.server.monitoring.brave.sampling.environment.rate |
The accuracy rate of the sampling for environment endpoints. |
-1 |
talend.component.server.monitoring.brave.sampling.execution.rate |
The accuracy rate of the sampling for execution endpoints. |
1 |
talend.component.server.monitoring.brave.sampling.rate |
The accuracy rate of the sampling. |
-1. |
talend.component.server.monitoring.brave.service.name |
The name used by the brave integration (zipkin) |
component-server |
talend.component.server.security.command.handler |
How to validate a command/request. Accepted values: securityNoopHandler. |
securityNoopHandler |
talend.component.server.security.connection.handler |
How to validate a connection. Accepted values: securityNoopHandler. |
securityNoopHandler |
Docker
The server image is deployed on Docker. Its version is suffixed with a timestamp to ensure images are not overriden and can break your usage. You can check the available version on Docker hub.
Run
You can run the docker image by executing this command :
$ sudo docker run -p 8080:8080 tacokit/component-starterConfigure
You can set the env variable MEECROWAVE_OPTS to customize the server, by default it is installed in /opt/talend/component-kit.
Maven repository
The maven repository is the default one of the machine, you can change it setting the system property
talend_component_server_maven_repository=/path/to/your/m2.
Deploy components to the server
If you want to deploy some components you can configure which ones in MEECROWAVE_OPTS (see server doc online) and redirect your local m2:
$ docker run \
-p 8080:8080 \
-v ~/.m2:/root/.m2 \
-e MEECROWAVE_OPTS="-Dtalend.component.server.component.coordinates=g:a:v,g2:a2:v2,..." \
component-serverLogging
The component server docker image comes with two log4j2 profile default and kafka.
The logging profile can be changed by setting the environment variable TALEND_COMPONENT_LOG4J2_PROFILE to kafka
the default profile is active by default.
default profile
The default profile has file and console logging capabilities.
The console logging is off by default and you can activate it by setting CONSOLE_LOG_LEVEL environment variable
to DEBUG, INFO, WARN or any other log level supported by log4j2. In practise and during development you will want
to see the logs without connecting to the server by activating console logging.
Run docker image with console logging
sudo docker run -p 8080:8080 \
-e CONSOLE_LOG_LEVEL=INFO \
component-serverKafka profile
Kafka profile let you send log to Kafka servers. The log are formatted in json and follow the layout defined by Talend and described here github.com/Talend/daikon/tree/master/daikon-logging/logging-event-layout
This profile require two environment variables
-
LOG_KAFKA_TOPIC: Kafka topic. -
LOG_KAFKA_URL: A list of host/port pairs to use for establishing the initial connection to the Kafka cluster. This list should be in the formurl:portseparated by,
Run docker image with kafka profile
sudo docker run -p 8080:8080 \
-e TALEND_COMPONENT_LOG4J2_PROFILE=kafka \
-e LOG_KAFKA_URL=`log kafka url:port` \
-e LOG_KAFKA_TOPIC=`log kafka topic` \
-e TRACING_KAFKA_URL=`tracing kafka url:port` \
-e TRACING_KAFKA_TOPIC=`tracing kafka topic` \
tacokit/component-serverNote : LOG_KAFKA_TOPIC will receive the application and the access logs
and TRACING_KAFKA_TOPIC will receive brave tracing logs.
Tracing (Brave Monitoring)
The component server use github.com/openzipkin/brave to monitor request.
The tracing can be activated by setting environment variable TRACING_ON to true.
You can choose the reporter type by setting talend_component_server_monitoring_brave_reporter_type environment variable
to log (this is the default value in this docker image) or to noop
which will deactivate the tracing. Other type of reporter may be added in the future.
The tracing rate is configurable by setting environment variable TRACING_SAMPLING_RATE.
This is the default sample rate for all the endpoints and has a default value of 0.1
You can define more accurate rate for every component server endpoint using those environment variables :
| Environment variable | Endpoint |
|---|---|
|
|
|
|
|
|
|
|
|
|
|
|
Run docker image with tracing on
sudo docker run -p 8080:8080 \
-e TRACING_ON=true \
-e TRACING_SAMPLING_RATE = 0.1 \
tacokit/component-serverBuild the image yourself
You can build component starter server in docker following those instructions :
docker build --build-arg ARTIFACT_ID=component-starter-server \
--build-arg SERVER_VERSION=`component starter server version` \
--tag tacokit/component-server .| this assumes the project is built before you run that command. |