| a test environment is available on Heroku and browable using Talend Component Kit Server Restlet Studio instance. |
HTTP API
The HTTP API intends to expose over HTTP most of Talend Component features, it is a standalone Java HTTP server.
WebSocket protocol is activated for the endpoints as well, instead of /api/v1 they uses the base /websocket/v1, see WebSocket part for more details.
|
Here is the API:
REST resources of Component Runtime :: Server
0.0.6-SNAPSHOT
POST api/v1/action/execute
This endpoint will execute any UI action and serialize the response as a JSON (pojo model) It takes as input the family, type and name of the related action to identify it and its configuration as a flat key value set using the same kind of mapping than for components (option path as key).
Request
Content-Type: application/json
Request Body: (java.util.Map<java.lang.String, java.lang.String>)
Query Param: action, java.lang.String
Query Param: family, java.lang.String
Query Param: type, java.lang.String
Response
Content-Type: application/json
GET api/v1/action/index
This endpoint returns the list of available actions for a certain family and potentially filters the " output limiting it to some families and types of actions.
Request
No body
Query Param: family, java.lang.String
Query Param: language, java.lang.String
Query Param: type, java.lang.String
Response
Content-Type: application/json
200 OK
Response Body: (org.talend.sdk.component.server.front.model.ActionList)
{
"items": [
{
"component": "string",
"name": "string",
"properties": [
{
"defaultValue": "string",
"displayName": "string",
"metadata": {
},
"name": "string",
"path": "string",
"placeholder": "string",
"proposalDisplayNames": {
},
"type": "string",
"validation": {
"enumValues": [
"string"
],
"max": 0,
"maxItems": 0,
"maxLength": 0,
"min": 0,
"minItems": 0,
"minLength": 0,
"pattern": "string",
"required": false,
"uniqueItems": false
}
}
],
"type": "string"
}
]
}GET api/v1/component/dependencies
Returns a list of dependencies for the given components.
| don’t forget to add the component itself since it will not be part of the dependencies. |
Then you can use /dependency/{id} to download the binary.
GET api/v1/component/dependency/{id}
Return a binary of the dependency represented by id.
It can be maven coordinates for dependencies or a component id.
GET api/v1/component/details
Returns the set of metadata about a few components identified by their 'id'.
Response
Content-Type: application/json
200 OK
Response Body: (org.talend.sdk.component.server.front.model.ComponentDetailList)
{
"details": [
{
"actions": [
{
"family": "string",
"name": "string",
"properties": [
{
"defaultValue": "string",
"displayName": "string",
"metadata": {
},
"name": "string",
"path": "string",
"placeholder": "string",
"proposalDisplayNames": {
},
"type": "string",
"validation": {
"enumValues": [
"string"
],
"max": 0,
"maxItems": 0,
"maxLength": 0,
"min": 0,
"minItems": 0,
"minLength": 0,
"pattern": "string",
"required": false,
"uniqueItems": false
}
}
],
"type": "string"
}
],
"displayName": "string",
"icon": "string",
"id": {
"family": "string",
"familyId": "string",
"id": "string",
"name": "string",
"plugin": "string",
"pluginLocation": "string"
},
"inputFlows": [
"string"
],
"links": [
{
"contentType": "string",
"name": "string",
"path": "string"
}
],
"outputFlows": [
"string"
],
"properties": [
{
"defaultValue": "string",
"displayName": "string",
"metadata": {
},
"name": "string",
"path": "string",
"placeholder": "string",
"proposalDisplayNames": {
},
"type": "string",
"validation": {
"enumValues": [
"string"
],
"max": 0,
"maxItems": 0,
"maxLength": 0,
"min": 0,
"minItems": 0,
"minLength": 0,
"pattern": "string",
"required": false,
"uniqueItems": false
}
}
],
"type": "string",
"version": 0
}
]
}GET api/v1/component/icon/family/{id}
Returns a particular family icon in raw bytes.
GET api/v1/component/icon/{id}
Returns a particular component icon in raw bytes.
GET api/v1/component/index
Returns the list of available components.
Response
Content-Type: application/json
200 OK
Response Body: (org.talend.sdk.component.server.front.model.ComponentIndices)
{
"components": [
{
"categories": [
"string"
],
"displayName": "string",
"familyDisplayName": "string",
"icon": {
"customIcon": {
},
"customIconType": "string",
"icon": "string"
},
"iconFamily": {
"customIcon": {
},
"customIconType": "string",
"icon": "string"
},
"id": {
"family": "string",
"familyId": "string",
"id": "string",
"name": "string",
"plugin": "string",
"pluginLocation": "string"
},
"links": [
{
"contentType": "string",
"name": "string",
"path": "string"
}
],
"version": 0
}
]
}POST api/v1/component/migrate/{id}/{configurationVersion}
Allows to migrate a component configuration without calling any component execution.
GET api/v1/configurationtype/details
Returns the set of metadata about a few configurations identified by their 'id'.
GET api/v1/configurationtype/index
Returns all available configuration type - storable models. Note that the lightPayload flag allows to load all of them at once when you eagerly need to create a client model for all configurations.
POST api/v1/configurationtype/migrate/{id}/{configurationVersion}
Allows to migrate a configuration without calling any component execution.
GET api/v1/documentation/component/{id}
Returns an asciidoctor version of the documentation for the component represented by its identifier id.
Format can be either asciidoc or html - if not it will fallback on asciidoc - and if html is selected you get a partial document.
| it is recommended to use asciidoc format and handle the conversion on your side if you can, the html flavor handles a limited set of the asciidoc syntax only like plain arrays, paragraph and titles. |
The documentation will likely be the family documentation but you can use anchors to access a particular component (_componentname_inlowercase).
GET api/v1/environment
Returns the environment of this instance. Useful to check the version or configure a healthcheck for the server.
POST api/v1/execution/read/{family}/{component}
| deprecated |
Read inputs from an instance of mapper. The number of returned records if enforced to be limited to 1000. The format is a JSON based format where each like is a json record.
POST api/v1/execution/write/{family}/{component}
| deprecated |
Sends records using a processor instance. Note that the processor should have only an input. Behavior for other processors is undefined. The input format is a JSON based format where each like is a json record - same as for the symmetric endpoint.
Deprecated endpoints
If some endpoints are intended to disappear they will be deprecated. In practise it means a header X-Talend-Warning
will be returned with some message as value.
WebSocket transport
You can connect on any endpoint replacing /api by /websocket and appending /<http method> for the URL and formatting the request as:
SEND
destination: <endpoint after v1>
<headers>
<payload>^@For instance:
SEND
destination: /component/index
Accept: application/json
^@The response is formatted as follow:
MESSAGE
status: <http status code>
<headers>
<payload>^@| if you have a doubt about the endpoint, they are all logged during startup and you can find them in the logs. |
If you don’t want to create a pool of connection per endpoint/verb you can use the bus endpoint: /websocket/v1/bus.
This endpoint requires that you add the header destinationMethod to each request with the verb value - default would be GET:
SEND
destination: /component/index
destinationMethod: GET
Accept: application/json
^@HTTPS activation
Using the server zip (or Docker image)
you can configure HTTPS adding properties to MEECROWAVE_OPTS. Assuming you have a certificate in /opt/certificates/component.p12
(don’t forget to add/mount it in the Docker image if you use it) you can activate it through:
# use -e for Docker
#
# this will skip the http port binding to only bind https on the port 8443 and setup the right certificate
export MEECROWAVE_OPTS="-Dskip-http=true -Dssl=true -Dhttps=8443 -Dkeystore-type=PKCS12 -Dkeystore-alias=talend -Dkeystore-password=talend -Dkeystore-file=/opt/certificates/component.p12"Web forms and REST API
component-form library provides a way to build a component REST API facade compatible with react form library.
A trivial facade can be:
@Path("tacokit-facade")
@ApplicationScoped
public class ComponentFacade {
private static final String[] EMPTY_ARRAY = new String[0];
@Inject
private Client client;
@Inject
private ActionService actionService;
@Inject
private UiSpecService uiSpecService;
@Inject // assuming it is available in your app, use any client you want
private WebTarget target;
@POST
@Path("action")
public void action(@Suspended final AsyncResponse response, @QueryParam("family") final String family,
@QueryParam("type") final String type, @QueryParam("action") final String action,
final Map<String, Object> params) {
client.action(family, type, action, params).handle((r, e) -> {
if (e != null) {
onException(response, e);
} else {
response.resume(actionService.map(type, r));
}
return null;
});
}
@GET
@Path("index")
public void getIndex(@Suspended final AsyncResponse response,
@QueryParam("language") @DefaultValue("en") final String language) {
target
.path("component/index")
.queryParam("language", language)
.request(APPLICATION_JSON_TYPE)
.rx()
.get(ComponentIndices.class)
.toCompletableFuture()
.handle((index, e) -> {
if (e != null) {
onException(response, e);
} else {
index.getComponents().stream().flatMap(c -> c.getLinks().stream()).forEach(
link -> link.setPath(link.getPath().replaceFirst("/component/", "/application/").replace(
"/details?identifiers=", "/detail/")));
response.resume(index);
}
return null;
});
}
@GET
@Path("detail/{id}")
public void getDetail(@Suspended final AsyncResponse response,
@QueryParam("language") @DefaultValue("en") final String language, @PathParam("id") final String id) {
target
.path("component/details")
.queryParam("language", language)
.queryParam("identifiers", id)
.request(APPLICATION_JSON_TYPE)
.rx()
.get(ComponentDetailList.class)
.toCompletableFuture()
.thenCompose(result -> uiSpecService.convert(result.getDetails().iterator().next()))
.handle((result, e) -> {
if (e != null) {
onException(response, e);
} else {
response.resume(result);
}
return null;
});
}
private void onException(final AsyncResponse response, final Throwable e) {
final UiActionResult payload;
final int status;
if (WebException.class.isInstance(e)) {
final WebException we = WebException.class.cast(e);
status = we.getStatus();
payload = actionService.map(we);
} else if (CompletionException.class.isInstance(e)) {
final CompletionException actualException = CompletionException.class.cast(e);
log.error(actualException.getMessage(), actualException);
status = Response.Status.BAD_GATEWAY.getStatusCode();
payload = actionService.map(new WebException(actualException, -1, emptyMap()));
} else {
log.error(e.getMessage(), e);
status = Response.Status.BAD_GATEWAY.getStatusCode();
payload = actionService.map(new WebException(e, -1, emptyMap()));
}
response.resume(new WebApplicationException(Response.status(status).entity(payload).build()));
}
}
the Client can be created using ClientFactory.createDefault(System.getProperty("app.components.base", "http://localhost:8080/api/v1"))
and the service can be a simple new UiSpecService(). The factory uses JAX-RS if the API is available (assuming a JSON-B provider is registered) otherwise it tries to use Spring.
|
All the conversion between component model (REST API) and uiSpec model is done through the UiSpecService. It is based on the object model
which will be mapped to a ui model. The advantage to have a flat model in the component REST API is to make these layers easy to customize.
You can completely control the available components, tune the rendering switching the uiSchema if desired or add/remove part of the form.
You can also add custom actions/buttons for specific needs of the application.
the /migrate endpoint has nothing special so was not shown in previous snippet but if you need it you must add it as well.
|
Use UiSpec model without all the tooling
<dependency>
<groupId>org.talend.sdk.component</groupId>
<artifactId>component-form-model</artifactId>
<version>${talend-component-kit.version}</version>
</dependency>This maven dependency provides the UISpec model classes. You can use the Ui API (with or without the builders)
to create UiSpec representations.
Example:
final Ui form1 = ui()
// (1)
.withJsonSchema(JsonSchema.jsonSchemaFrom(Form1.class).build())
// (2)
.withUiSchema(uiSchema()
.withKey("multiSelectTag")
.withRestricted(false)
.withTitle("Simple multiSelectTag")
.withDescription("This datalist accepts values that are not in the list of suggestions")
.withWidget("multiSelectTag")
.build())
// (3)
.withProperties(myFormInstance)
.build();
// (4)
final String json = jsonb.toJson(form1);-
We extract the
JsonSchemafrom reflection on the classForm1. Note that@JsonSchemaIgnoreallows to ignore a field and@JsonSchemaPropertyallows to rename a property, -
We build programmatically using the builder API a
UiSchema, -
We pass an instance of the form to let the serializer extracts it JSON model,
-
We serialize the
Uimodel which can be used by UiSpec compatible front widgets.
| the model uses JSON-B API to define the binding, ensure to have an implementation in your classpath. This can be done adding these dependencies: |
<dependency>
<groupId>org.apache.geronimo.specs</groupId>
<artifactId>geronimo-jsonb_1.0_spec</artifactId>
<version>1.0</version>
</dependency>
<dependency>
<groupId>org.apache.geronimo.specs</groupId>
<artifactId>geronimo-json_1.1_spec</artifactId>
<version>1.0</version>
</dependency>
<dependency>
<groupId>org.apache.johnzon</groupId>
<artifactId>johnzon-jsonb</artifactId>
<version>${johnzon.version}</version> <!-- 1.1.5 for instance -->
</dependency>Javascript integration
Default javascript integration goes through Talend UI Forms library.
It is bundled as a NPM module called component-kit.js. It provides a default trigger implementation for the UIForm.
Here is how to use it:
import React from 'react';
import UIForm from '@talend/react-forms/lib/UIForm/UIForm.container';
import TalendComponentKitTrigger from 'component-kit.js';
export default class ComponentKitForm extends React.Component {
constructor(props) {
super(props);
this.trigger = new TalendComponentKitTrigger({ url: '/api/to/component/server/proxy' });
this.onTrigger = this.onTrigger.bind(this);
// ...
}
onTrigger(event, payload) {
return this.trigger.onDefaultTrigger(event, payload);
}
// ...
render() {
if(! this.state.uiSpec) {
return (<div>Loading ...</div>);
}
return (
<UIForm
data={this.state.uiSpec}
onTrigger={this.onTrigger}
onSubmit={this.onSubmit}
/>
);
}
}Logging
The logging uses Log4j2, you can specify a custom configuration using the system property -Dlog4j.configurationFile
or adding a log4j2.xml file into the classpath.
Here are some common configurations:
-
Console logging:
<?xml version="1.0"?>
<Configuration status="INFO">
<Appenders>
<Console name="Console" target="SYSTEM_OUT">
<PatternLayout pattern="[%d{HH:mm:ss.SSS}][%highlight{%-5level}][%15.15t][%30.30logger] %msg%n"/>
</Console>
</Appenders>
<Loggers>
<Root level="INFO">
<AppenderRef ref="Console"/>
</Root>
</Loggers>
</Configuration>This outputs messages looking like:
[16:59:58.198][INFO ][ main][oyote.http11.Http11NioProtocol] Initializing ProtocolHandler ["http-nio-34763"]-
JSON logging:
<?xml version="1.0"?>
<Configuration status="INFO">
<Properties>
<!-- DO NOT PUT logSource there, it is useless and slow -->
<Property name="jsonLayout">{"severity":"%level","logMessage":"%encode{%message}{JSON}","logTimestamp":"%d{ISO8601}{UTC}","eventUUID":"%uuid{RANDOM}","@version":"1","logger.name":"%encode{%logger}{JSON}","host.name":"${hostName}","threadName":"%encode{%thread}{JSON}","stackTrace":"%encode{%xThrowable{full}}{JSON}"}%n</Property>
</Properties>
<Appenders>
<Console name="Console" target="SYSTEM_OUT">
<PatternLayout pattern="${jsonLayout}"/>
</Console>
</Appenders>
<Loggers>
<Root level="INFO">
<AppenderRef ref="Console"/>
</Root>
</Loggers>
</Configuration>Output messages look like:
{"severity":"INFO","logMessage":"Initializing ProtocolHandler [\"http-nio-46421\"]","logTimestamp":"2017-11-20T16:04:01,763","eventUUID":"8b998e17-7045-461c-8acb-c43f21d995ff","@version":"1","logger.name":"org.apache.coyote.http11.Http11NioProtocol","host.name":"TLND-RMANNIBUCAU","threadName":"main","stackTrace":""}-
Rolling file appender
<?xml version="1.0"?>
<Configuration status="INFO">
<Appenders>
<RollingRandomAccessFile name="File" fileName="${LOG_PATH}/application.log" filePattern="${LOG_PATH}/application-%d{yyyy-MM-dd}.log">
<PatternLayout pattern="[%d{HH:mm:ss.SSS}][%highlight{%-5level}][%15.15t][%30.30logger] %msg%n"/>
<Policies>
<SizeBasedTriggeringPolicy size="100 MB" />
<TimeBasedTriggeringPolicy interval="1" modulate="true"/>
</Policies>
</RollingRandomAccessFile>
</Appenders>
<Loggers>
<Root level="INFO">
<AppenderRef ref="File"/>
</Root>
</Loggers>
</Configuration>More details are available on RollingFileAppender documentation.
| of course you can compose previous layout (message format) and appenders (where logs are written). |
Server Configuration
The server module contains several configuration you can set in:
-
Environment variables
-
System properties
-
A file located based on the
--component-configurationCLI option
| the configuration is read from system properties, environment variables, …. |
| Key | Description | Default |
|---|---|---|
talend.component.server.component.coordinates |
A comma separated list of gav to locate the components |
- |
talend.component.server.component.registry |
A property file where the value is a gav of a component to register (complementary with |
- |
talend.component.server.documentation.active |
Should the /documentation endpoint be activated. |
true |
talend.component.server.execution.dataset.retriever.timeout |
How long the read execution endpoint can last (max) |
180 |
talend.component.server.execution.pool.size |
The size of the execution pool for runtime endpoints. |
64 |
talend.component.server.execution.pool.wait |
How long the application waits during shutdown for the execution tasks to complete |
PT10S |
talend.component.server.jaxrs.exceptionhandler.defaultMessage |
If set it will replace any message for exceptions. Set to |
false |
talend.component.server.maven.repository |
The local maven repository used to locate components and their dependencies |
- |
talend.component.server.monitoring.brave.reporter.async |
When using url or kafka reporter, you can configure the async reporter with properties passed to this configuration entry.Ex: |
console |
talend.component.server.monitoring.brave.reporter.type |
The brave reporter to use to send the spans. Supported values are [auto, console, noop, url]. When configuration is needed,you can use this syntax to configure the repoter if needed: In |
auto |
talend.component.server.monitoring.brave.sampling.action.rate |
The accuracy rate of the sampling for action endpoints. |
-1 |
talend.component.server.monitoring.brave.sampling.component.rate |
The accuracy rate of the sampling for component endpoints. |
-1 |
talend.component.server.monitoring.brave.sampling.configurationtype.rate |
The accuracy rate of the sampling for environment endpoints. |
-1 |
talend.component.server.monitoring.brave.sampling.documentation.rate |
The accuracy rate of the sampling for documentation endpoint. |
-1 |
talend.component.server.monitoring.brave.sampling.environment.rate |
The accuracy rate of the sampling for environment endpoints. |
-1 |
talend.component.server.monitoring.brave.sampling.execution.rate |
The accuracy rate of the sampling for execution endpoints. |
1 |
talend.component.server.monitoring.brave.sampling.rate |
The accuracy rate of the sampling. |
-1. |
talend.component.server.monitoring.brave.service.name |
The name used by the brave integration (zipkin) |
component-server |
talend.component.server.security.command.handler |
How to validate a command/request. Accepted values: securityNoopHandler. |
securityNoopHandler |
talend.component.server.security.connection.handler |
How to validate a connection. Accepted values: securityNoopHandler. |
securityNoopHandler |