Components are designed to manipulate data (access, read, create). Talend Component Kit can handle several types of data, described in this document. By design, the framework must run in DI (plain standalone Java program) and in Beam pipelines. It is out of scope of the framework to handle the way the runtime serializes - if needed - the data. For that reason, it is critical not to import serialization constraints to the stack. As an example, this is one of the reasons why Record or jsonObject were preferred to Avro IndexedRecord. Any serialization concern should either be hidden in the framework runtime (outside of the component developer scope) or in the runtime integration with the framework (for example, Beam integration). Record is the default format. It offers many possibilities and can evolve depending on the Talend platform needs. Its structure is data-driven and exposes a schema that allows to browse it. Projects generated from the Talend Component Kit Starter are by default designed to handle this format of data. Record is a Java interface but never implement it yourself to ensure compatibility with the different Talend products. Follow the guidelines below. You can build records using the newRecordBuilder method of the RecordBuilderFactory (see here). For example: In the example above, the schema is dynamically computed from the data. You can also do it using a pre-built schema, as follows: The example above uses a schema that was pre-built using factory.newSchemaBuilder(Schema.Type.RECORD). When using a pre-built schema, the entries passed to the record builder are validated. It means that if you pass a null value null or an entry type that does not match the provided schema, the record creation fails. It also fails if you try to add an entry which does not exist or if you did not set a not nullable entry. Using a dynamic schema can be useful on the backend but can lead users to more issues when creating a pipeline to process the data. Using a pre-built schema is more reliable for end-users. You can access and read data by relying on the getSchema method, which provides you with the available entries (columns) of a record. The Entry exposes the type of its value, which lets you access the value through the corresponding method. For example, the Schema.Type.STRING type implies using the getString method of the record. For example: The Record format supports the following data types: String Boolean Int Long Float Double DateTime Array Bytes Record A map can always be modelized as a list (array of records with key and value entries). For example: For example, you can use the API to provide the schema. The following method needs to be implemented in a service. Manually constructing the schema without any data: Returning the schema from an already built record: MyDataset is the class that defines the dataset. Learn more about datasets and datastores in this document. Entry names for Record and jsonObject types must comply with the following rules: The name must start with a letter or with _. If not, the invalid characters are ignored until the first valid character. Following characters of the name must be a letter, a number, or . If not, the invalid character is replaced with . For example: 1foo becomes foo. f@o becomes f_o. 1234f5@o becomes ___f5_o. foo123 stays foo123. Each array uses only one schema for all of its elements. If an array contains several elements, they must be of the same data type. For example, the following array is not correct as it contains a string and an object: The runtime also supports jsonObject as input and output component type. You can rely on the JSON services (Jsonb, JsonBuilderFactory) to create new instances. This format is close to the Record format, except that it does not natively support the Datetime type and has a unique Number type to represent Int, Long, Float and Double types. It also does not provide entry metadata like nullable or comment, for example. It also inherits the Record format limitations. The runtime also accepts any POJO as input and output component type. In this case, it uses JSON-B to treat it as a jsonObject.

This part is limited to specific kinds of Beam PTransform: PTransform> for inputs. PTransform, PDone> for outputs. Outputs must use a single (composite or not) DoFn in their apply method. To illustrate the input wrapping, this procedure uses the following input as a starting point (based on existing Beam inputs): To wrap the Read in a framework component, create a transform delegating to that Read with at least a @PartitionMapper annotation and using @Option constructor injections to configure the component. Also make sure to follow the best practices and to specify @Icon and @Version. To illustrate the output wrapping, this procedure uses the following output as a starting point (based on existing Beam outputs): You can wrap this output exactly the same way you wrap an input, but using @Processor instead of: Note that the org.talend.sdk.component.runtime.beam.transform.DelegatingTransform class fully delegates the "expansion" to another transform. Therefore, you can extend it and implement the configuration mapping: In terms of classloading, when you write an I/O, the Beam SDK Java core stack is assumed as provided in Talend Component Kit runtime. This way, you don’t need to include it in the compile scope, it would be ignored anyway. If you need a JSonCoder, you can use the org.talend.sdk.component.runtime.beam.factory.service.PluginCoderFactory service, which gives you access to the JSON-P and JSON-B coders. There is also an Avro coder, which uses the FileContainer. It ensures it is self-contained for IndexedRecord and it does not require—as the default Apache Beam AvroCoder—to set the schema when creating a pipeline. It consumes more space and therefore is slightly slower, but it is fine for DoFn, since it does not rely on serialization in most cases. See org.talend.sdk.component.runtime.beam.transform.avro.IndexedRecordCoder. If your PCollection is made of jsonObject records, and you want to convert them to IndexedRecord, you can use the following PTransforms: converts an IndexedRecord to a jsonObject. converts a jsonObject to an IndexedRecord. converts a jsonObject to an IndexedRecord with AVRO schema inference. There are two main provided coder for Record: it will unwrap the record as an Avro IndexedRecord and serialize it with its schema. This can indeed have a performance impact but, due to the structure of component, it will not impact the runtime performance in general - except with direct runner - because the runners will optimize the pipeline accurately. it will serialize the Avro IndexedRecord as well but it will ensure the schema is in the SchemaRegistry to be able to deserialize it when needed. This implementation is faster but the default implementation of the registry is "in memory" so will only work with a single worker node. You can extend it using Java SPI mecanism to use a custom distributed implementation. Sample input based on Beam Kafka: Because the Beam wrapper does not respect the standard Talend Component Kit programming model ( for example, there is no @Emitter), you need to set the false property in your pom.xml file (or equivalent for Gradle) to skip the component programming model validations of the framework.

TCOMP-2289: Payload validator fails on regexp matching component-manager TCOMP-2186: Guess schema service for processors api schema-record TCOMP-2276: Make Streaming stop strategy configurable component-manager studio TCOMP-1998: Remove component-server-vault-proxy from cloud environments component-server component-server-vault-proxy helm-charts vault-client TCOMP-2256: Replace nashorn javascript engine by rhino component-manager studio TCOMP-2259: Support db column name/length/precision with guess schema for all types studio TCOMP-2268: Upgrade jib to 0.22.0 remote-engine-customizer starter tsbi TCOMP-2273: Upgrade TSBI to 3.0.8-20220928070500 component-server starter tsbi TCOMP-2274: Move component-starter-server to TSBI starter TCOMP-2275: Use java 17 TSBI image for CI build build tsbi TCOMP-2286: Upgrade jackson to 2.13.4 TCOMP-2287: Upgrade commons-text to 1.10.0 documentation TCOMP-2276: Make Streaming stop strategy configurable component-manager studio TCOMP-2239: Fix Record.Builder interface to avoid API break schema-record TCOMP-2226: Implement a default UI for streaming sources for user configuration of a StopStrategy component-form component-manager TCOMP-2234: Override blocking read process in streaming connectors component-manager studio TCOMP-2258: @Documentation to tooltips in uiSchema component-form component-server TCOMP-2147: Decrease log level for blacklisted dependencies component-manager TCOMP-2228: Upgrade git-commit-id-plugin to 4.9.10 TCOMP-2232: Upgrade slf4j to 1.7.34 TCOMP-2238: Upgrade jib-core to 0.16.0 TCOMP-2249: Upgrade johnzon to 1.2.19 TCOMP-2251: Upgrade jackson to 2.13.3 TCOMP-2252: MavenRepositoryResolver call fallback only it’s needed component-manager TCOMP-2257: Upgrade meecrowave to 1.2.14 component-manager TCOMP-2263: Upgrade openwebbeans-se to 2.0.27 component-manager TCOMP-2264: Upgrade TSBI to 3.0.5-20220907120958 tsbi TCOMP-2239: Fix Record.Builder interface to avoid API break schema-record TCOMP-2182: Guess Schema in Studio always uses version of component 1 studio studio-integration TCOMP-2190: Handle partial messages for large payloads in websocket communications component-server studio TCOMP-2107: Implement a stop strategy for streaming input connectors component-manager studio TCOMP-2177: Suppress illegal reflective access operation has occurred warnings component-manager TCOMP-2163: [QA] Component Runtime API test Framework testing TCOMP-2187: Introduce IntegerConstraintEnricher component-form TCOMP-2204: Upgrade netty to 4.1.79.Final TCOMP-2205: Upgrade crawler-commons to 1.3 TCOMP-2206: Upgrade guava to 31.1-jre TCOMP-2207: Upgrade maven to 3.8.6 TCOMP-2208: Upgrade maven-shade-plugin to 3.3.0 build TCOMP-2209: Upgrade junit5 to 5.9.0 TCOMP-2210: Upgrade tomcat to 9.0.63 TCOMP-2211: Upgrade cxf to 3.5.2 TCOMP-2212: Upgrade bndlib to 5.2.0 TCOMP-2217: Update rat-plugin to 0.14 build TCOMP-2219: Add API to convert data in Record schema-record TCOMP-2223: Upgrade log4j to 2.18.0 TCOMP-2227: Upgrade commons-io to 2.9.0 TCOMP-2229: Upgrade jcommander to 1.81 TCOMP-2230: Allow specific context UI TCOMP-2233: support decimal type TCOMP-2190: Handle partial messages for large payloads in websocket communications component-server studio TCOMP-2177: Suppress illegal reflective access operation has occurred warnings component-manager TCOMP-2177: Suppress illegal reflective access operation has occurred warnings component-manager TCOMP-2176: Record : Infinite loop schema-record TCOMP-2146: Car bundler improvements car-bundler maven-plugin TCOMP-2151: Add documentation translation to metadata component-server TCOMP-2132: Optimisation for preparation schema-record TCOMP-2143: [JDBC TCK]: Support MODULE_LIST field for studio in tck connector ui for driver jars choose studio TCOMP-2152: Upgrade jackson to 2.13.2 beam bom maven-plugin TCOMP-2153: Bump netty to 4.1.77.Final due to CVE CVE-2022-24823 testing TCOMP-2154: Upgrade maven-settings to 3.8.5 due to CVE-2021-26291 build TCOMP-2155: Upgrade jdom2 to 2.0.6.1 due to CVE-2021-33813 beam TCOMP-2164: Ensure that decryption is done only on credential fields component-server vault-client TCOMP-2171: Add component type to ComponentIndex component-server TCOMP-2152: Upgrade jackson to 2.13.2 beam bom maven-plugin TCOMP-2146: Car bundler improvements car-bundler maven-plugin TCOMP-2111: [Runtime convergence] : Join connector fails in cloud environment with hybrid tck/beam connectors api beam TCOMP-2123: Bug on order columns for Avro Impl beam schema-record TCOMP-2127: Fix avro records where array contains nullable array beam schema-record TCOMP-2131: starter-toolkit fails when generating a connector from openapi description starter TCOMP-2133: component-registry uses detailed version not baseVersion in snapshot case build maven-plugin TCOMP-2134: Activate intellij plugin by default intellij starter TCOMP-2138: starter-toolkit github repository creation process fails starter TCOMP-2135: Component web tester in non interactive mode component-server maven-plugin testing TCOMP-2126: give default implementation to Record.Builder to not break api api TCOMP-2130: Add git informations in starter-toolkit’s environment starter TCOMP-2127: Fix avro records where array contains nullable array beam schema-record TCOMP-2130: Add git informations in starter-toolkit’s environment starter TCOMP-2126: give default implementation to Record.Builder to not break api api TCOMP-2085: Add extras manipulations on Record BuilderImpl beam schema-record TCOMP-2102: Wrong maven resolution with car when using snapshot in prepare-repository goal build maven-plugin TCOMP-2119: Avro Record : array containing Null. beam schema-record TCOMP-2112: [JDBC] discover schema API is failing on production. build maven-plugin TCOMP-2103: Link affected jira components to issue in changelog as keywords for search documentation TCOMP-2098: Improve m2 discovery process documentation TCOMP-2104: Header link should be linked to latest path documentation TCOMP-2105: Upgrade Tomcat to 9.0.60 component-server maven-plugin starter TCOMP-2108: Upgrade maven plugins TCOMP-2109: Upgrade git-commit-id-plugin to 4.0.5 TCOMP-2110: Replace log4j by reload4j stitch TCOMP-2114: Upgrade TSBI to 2.9.27-20220331162145 component-server component-server-vault-proxy starter tsbi TCOMP-2115: Upgrade jackson to 2.12.6 due to CVE-2020-36518 bom TCOMP-2116: Upgrade log4j2 to 2.17.2 TCOMP-2117: Upgrade slf4j to 1.7.33 TCOMP-2118: Upgrade tomcat to 9.0.62 (mitigation for CVE-2022-22965) component-server component-server-vault-proxy starter TDI-47693 : fix misaligned openwebbeans-spi dependency studio TCOMP-2003: Maven dependency classifier considered as version in dependencies.txt by Studio TCOMP-2096: Support BigDecimal type in DI integration TCOMP-2087: Upgrade Tomcat to 9.0.59 due to CVE-2022-23181 TCOMP-2088: Upgrade OpenWebBeans to 2.0.26 TCOMP-2089: Upgrade meecrowave to 1.2.13 TCOMP-2090: Upgrade johnzon to 1.2.16 TCOMP-2091: Upgrade Beam to 2.36.0 TCOMP-2092: MvnCoordinateToFileConverter fakes classifiers' support TCOMP-2093: Improve component-runtime documentation site TCOMP-2097: Upgrade cxf to 3.5.1 TCOMP-1803: RecordBuilder.withRecord(final String name, final Record value) doesn’t accept null value TCOMP-2079: Intellij plugin fails on plugin startup TCOMP-2080: AvroRecord refuses Union[null, RecordSchema] TCOMP-2082: ComponentManager’s findDefaultM2 method takes comment as granted TCOMP-2058: Add dependencies on config TCOMP-2074: Change JSON log format to conform to ECS TCOMP-2083: Give component-runtime version on ComponentManager startup TCOMP-2084: Allow use of i18n in connectors' metadata for custom labels TCOMP-2079: Intellij plugin fails on plugin startup TCOMP-2080: AvroRecord refuses Union[null, RecordSchema] TCOMP-2082: ComponentManager’s findDefaultM2 method takes comment as granted TCOMP-2063: Avro Record Constructor TCOMP-2064: NPE with lookup missconfiguration in Join processor TCOMP-2067: Bug on order columns TCOMP-2071: Define default methods on Schema / Entry / Record interfaces TCOMP-2045: Pass and read meta information about columns. TCOMP-2072: Ligthen parameters for component-server docker image TCOMP-2057: AvroSchema : optimize getType by using type fields TCOMP-2060: Upgrade log4j2 to 2.17.0 due to CVE-2021-45105 TCOMP-2061: Upgrade netty to 4.1.72.Final due to CVE-2021-43797 TCOMP-2065: Internationalized Services as Serializable TCOMP-2068: Upgrade log4j2 to 2.17.1 due to CVE-2021-44832 TCOMP-2069: Create a latest tag for component-runtime images TCOMP-2070: Upgrade TSBI to 2.9.18-20220104141654 TCOMP-2073: Upgrade maven-core to 3.8.4 due to CVE TCOMP-2047: RecordBuilder in RowstructVisitor keeps values TCOMP-2048: RowstructVisitor should respect case in member not java convention TCOMP-2049: Incompatible class change on Entry TCOMP-2053: Migration failing when using custom java code in configuration TCOMP-2018: Optimize Avro Record TCOMP-2054: Upgrade log4j2 to 2.16.0 due to CVE-2021-44228 TCOMP-2053: Migration failing when using custom java code in configuration TCOMP-2054: Upgrade log4j2 to 2.16.0 due to CVE-2021-44228 TCOMP-2049: Incompatible class change on Entry TCOMP-2047: RecordBuilder in RowstructVisitor keeps values TCOMP-2048: RowstructVisitor should respect case in member not java convention TCOMP-2019: Sanitized columns name collision support TCOMP-2021: Missing logic when handling null date values in Record TCOMP-2046: Rowstruct visitor recreates schema at each incoming row TCOMP-2004: [Runtime convergence] New tck/API to retrieve dataset full content TCOMP-2008: Add ability to insert a schema entry on Record BuilderImpl TCOMP-1924: Support Java 17 runtime TCOMP-2023: Upgrade gradle to 6.9.1 TCOMP-2024: Upgrade maven-bundle-plugin to 4.2.1 TCOMP-2025: Upgrade documentation to latest TCOMP-2027: Upgrage junit to 5.8.1 TCOMP-2028: Provide nashorn scripting engine when using java15+ TCOMP-2029: Upgrade jaxb to 2.3.5 TCOMP-2030: Upgrade Tomcat to 9.0.54 due to CVE-2021-42340 TCOMP-2031: Upgrade Beam to 2.33.0 TCOMP-2032: Upgrade Spark to 3.2.0 TCOMP-2035: Check build w/ Java 17 on CI TCOMP-2036: Upgrade cxf to 3.4.5 TCOMP-2037: Upgrade johnzon to 1.2.15 TCOMP-2038: Upgrade bouncycastle to 1.69 TCOMP-2042: Return a key related to version of connector services and its content TCOMP-2043: Upgrade spotless to 2.17.3 and talend-java-formatter to 0.2.2 TCOMP-2044: Upgrade TSBI to 2.9.2-20211106085418 TCOMP-2045: Pass and read meta information about columns. studio-integration TCOMP-2096: Support BigDecimal type in DI integration schema-record studio studio-integration TCOMP-2070: Upgrade TSBI to 2.9.18-20220104141654 build component-server component-server-vault-proxy tsbi TCOMP-2105: Upgrade Tomcat to 9.0.60 component-server maven-plugin starter TCOMP-2030: Upgrade Tomcat to 9.0.54 due to CVE-2021-42340 TCOMP-2053: Migration failing when using custom java code in configuration TCOMP-2054: Upgrade log4j2 to 2.16.0 due to CVE-2021-44228 TCOMP-2048: RowstructVisitor should respect case in member not java convention TCOMP-2047: RecordBuilder in RowstructVisitor keeps values TCOMP-2046: Rowstruct visitor recreates schema at each incoming row TCOMP-1963: Missing IMetaDataColumn fields in guess schema TCOMP-1987: Avro record : Array of Array of records issue TCOMP-1988: Unable to run component-runtime connectors in Studio with JDK 17 TCOMP-2005: Non defined columns appear in schema TCOMP-2006: Support empty values for Numbers case TCOMP-2010: Error on Documentation build on "less" usage TCOMP-2020: talend-component-kit-intellij-plugin module build fails using Bintray (decomissioned) TCOMP-1900: Create jenkins release process for component-runtime TCOMP-1997: Enable plugins reloading according criteria TCOMP-2000: Upgrade netty to 4.1.68.Final TCOMP-2001: Upgrade Beam to 2.32.0 TCOMP-2007: connectors as a json object in Environment TCOMP-2009: Upgrade dockerfile-maven-plugin to 1.4.13 TCOMP-2016: UiSchema can’t hold advanced titleMap for more advanded datalist widgets TCOMP-2007: connectors as a json object in Environment TCOMP-1957: Avro schema builder issue TCOMP-1994: WebSocketClient$ClientException when executing action in Studio TCOMP-1923: Record : add metadata TCOMP-1990: Update jsoup to 1.14.2 due to CVE-2021-37714 TCOMP-1991: Update groovy to 3.0.9 due to CVE-2021-36373 / CVE-2021-36374 TCOMP-1992: Update lombok to 1.18.20 TCOMP-1993: Update TSBI to 2.9.0-20210907155713 TCOMP-1995: Expose the connectors (global) version in the "Environment" response TCOMP-1996: BaseService must not define equals & hashcode TCOMP-1994: WebSocketClient$ClientException when executing action in Studio TCOMP-1904: Delegate Avro record in AvroRecord seems to be invalid TCOMP-1967: goal uispec generation failure TCOMP-1983: fix module inclusion in dependencies.txt when build is java9+ TCOMP-1981: Allow to filter artifacts in car file generation TCOMP-1982: Allow to include extra artifacts in car file generation TCOMP-1876: Make schemaImpl immutable TCOMP-1885: Service Serializable TCOMP-1906: Redefine equals on RecordImpl TCOMP-1955: Upgrade cxf to 3.4.4 due to CVE-2021-30468 TCOMP-1966: Upgrade Tomcat to 9.0.50 due to CVE-2021-33037 TCOMP-1968: Upgrade maven to 3.8.1 TCOMP-1969: Upgrade Beam to 2.31.0 TCOMP-1970: Upgrade jackson to 2.12.1 TCOMP-1971: Upgrade Junit to 5.8.0-M1 TCOMP-1972: Upgrade slf4j to 1.7.32 TCOMP-1973: Upgrade log4j to 2.14.1 TCOMP-1974: Upgrade commons-compress to 1.21 due to CVE-2021-36090 TCOMP-1975: Upgrade TSBI to 2.8.2-20210722144648 TCOMP-1976: Upgrade meecrowave to 1.2.11 TCOMP-1977: Upgrade OpenWebBeans to 2.0.23 TCOMP-1978: Upgrade tomcat to 9.0.44 TCOMP-1979: Upgrade xbean to 4.20 TCOMP-1980: Upgrade meecrowave to 1.2.12 TCOMP-1967: goal uispec generation failure TCOMP-1935: After Variables doesn’t support custom object types TCOMP-1941: Maven goal talend-component:web fails on startup TCOMP-1947: Implement a retry strategy on failure in vault-client TCOMP-1948: Raised exception in component-server(s) should be serialized in json TCOMP-1952: IllegalArgumentException when the http response return duplicated header. TCOMP-1939: Upgrade TSBI to Talend 2.7.2-20210616074048 TCOMP-1940: Upgrade Beam to 2.30.0 TCOMP-1941: Maven goal talend-component:web fails on startup TCOMP-1939: Upgrade TSBI to Talend 2.7.2-20210616074048 TCOMP-1919: Sanitize must force encoding file TCOMP-1925: Incorrect mapping of the parameters after arrays TCOMP-1937: Classpath not fully parsed in TSBI images TCOMP-1917: Add DatasetDiscovery annotation TCOMP-1707: Upgrade Geronimo :: Simple JCache to 1.0.5 TCOMP-1850: component-server with vault feature TCOMP-1907: Service monitor implementation & cleaning of grafana dashboard TCOMP-1921: Upgrade TSBI to 2.7.0-20210527090437 TCOMP-1930: Remove jsoup 1.7.x transitive dependency due to CVE-2015-6748 TCOMP-1936: Extend properties in Schema to use JsonValue TCOMP-1938: Add the german locale in the locale mapping TCOMP-1938: Add the german locale in the locale mapping TCOMP-1937: Classpath not fully parsed in TSBI images TCOMP-1919: Sanitize must force encoding file TCOMP-1886: Errors on Schema.sanitizeConnectionName TCOMP-1905: component-runtime fails to build with Java 11 TCOMP-1893: Upgrade to Beam 2.29.0 and use Beam’s Spark 3 specific module TCOMP-705: Support After variables TCOMP-1898: Add method to Record.Builder TCOMP-1910: Upgrade commons-io to 2.8.0 due to CVE-2021-29425 TCOMP-1911: Upgrade cxf to 3.4.3 due to CVE-2021-22696 TCOMP-1912: Upgrade TSBI to 2.6.7-20210503202416 TCOMP-1938: Add the german locale in the locale mapping TCOMP-1937: Classpath not fully parsed in TSBI images TCOMP-1880: Engine Server returns binary data instead of json (aka does not respect the compressed header) TCOMP-1886: Errors on Schema.sanitizeConnectionName TCOMP-1815: Support of ComponentException in migration TCOMP-1873: Add method getEntry on TCK Record Schema class TCOMP-1892: Upgrade Spark to 3.0.1 TCOMP-1888: Remove/change validation of ComponentException TCOMP-1894: Uniformize docker images entrypoints TCOMP-1895: Enhance coercion in RecordConverters TCOMP-1896: Upgrade TSBI to 2.6.4-20210331133410 TCOMP-1806: Double values are rounded to 5 decimal places in studio TCOMP-1851: HttpClient implementation class is a Service with State TCOMP-1864: JsonSchemaConverter and johnzon-jsonschema 1.2.9+ look incompatible TCOMP-1866: Invalid number coercion on primitive type TCOMP-1869: byte[] handling is incorrect in dynamic column TCOMP-1871: Dynamic metadata name is not sanitized TCOMP-1861: Add a 'props' property in the Schema TCOMP-1863: Upgrade batik-codec to 1.14 due to CVE-2020-11988 TCOMP-1865: Upgrade cxf to 3.4.2 TCOMP-1867: Upgrade Apache Beam to 2.28.0 TCOMP-1878: Upgrade TSBI to 2.6.3-20210304090015 TCOMP-1688: Rewrite JsonSchema required rules to reflect component’s validation rules TCOMP-1857: Pojo conversion don’t support nested Objects TCOMP-1841: Add a SPI that would allow to add metadata to components TCOMP-1847: Upgrade Apache Beam to 2.27.0 TCOMP-1848: Upgrade bouncycastle to 1.68 due to CVE 2020-28052 TCOMP-1849: Proxify metrics component-server’s endpoint TCOMP-1852: Upgrade netty to v4.1.58.Final and ensure default http testing module is java 11 friendly over ssl TCOMP-1854: Upgrade netty to 4.1.59.Final due to CVE-2021-21290 TCOMP-1855: Upgrade johnzon to 1.2.10 TCOMP-1856: Upgrade tomcat to 9.0.43 TCOMP-1841: Add a SPI that would allow to add metadata to components TCOMP-1852: Upgrade netty to v4.1.58.Final and ensure default http testing module is java 11 friendly over ssl TCOMP-1854: Upgrade netty to 4.1.59.Final due to CVE-2021-21290 TCOMP-1848: Upgrade bouncycastle to 1.68 due to CVE 2020-28052 TCOMP-1839: Tomcat websocket server fails to start after tomcat 9.0.40 and meecrowave 1.2.10 TCOMP-1836: Upgrade OpenWebBeans to 2.0.20 TCOMP-1837: Upgrade xbean to 4.18 TCOMP-1838: Upgrade cxf to 3.4.1 TCOMP-1840: Upgrade tomcat to 9.0.41 TCOMP-1842: Upgrade jgit to 5.10.0.202012080955-r TCOMP-1844: Upgrade johnzon to 1.2.9 TCOMP-1845: Upgrade guava to 30.1-jre due to CVE-2020-8908 TCOMP-1848: Upgrade bouncycastle to 1.68 due to CVE 2020-28052 TCOMP-1839: Tomcat websocket server fails to start after tomcat 9.0.40 and meecrowave 1.2.10 TCOMP-1836: Upgrade OpenWebBeans to 2.0.20 TCOMP-1837: Upgrade xbean to 4.18 TCOMP-1827: Upgrade lombok to 1.18.16 TCOMP-1828: Change project’s versioning scheme TCOMP-1829: Upgrade TSBI to 2.5.3-20201201131449 TCOMP-1830: Upgrade Apache Beam to 2.26.0 TCOMP-1832: Upgrade httpclient to 4.5.13 due to CVE-2020-13956 TCOMP-1833: Upgrade spark to 2.4.7 TCOMP-1834: Upgrade groovy to 3.0.7 due to CVE-2020-17521 TCOMP-1787: ComponentManager can’t be re-created after it’s been closed TCOMP-1788: Invalid properties validation TCOMP-1801: Can’t look for resources in the classpath on Windows TCOMP-1761: Support of complete schema definition TCOMP-1725: Upgrade Tomcat to 9.0.40 TCOMP-1792: Uniform error message on component validation TCOMP-1808: Upgrade log4j2 to 2.14.0 TCOMP-1809: Update CXF to 3.3.8 due to CVE-2020-13954 TCOMP-1812: Upgrade junit to 4.13.1 due to CVE-2020-15250 TCOMP-1813: Upgrade jupiter to 5.7.0 TCOMP-1816: Apache Maven Shared Utils: OS Command Injection in Talend/component-runtime (master) and Talend/cloud-components TCOMP-1817: Upgrade gmavenplus-plugin to 1.11.0 TCOMP-1722: REST - Last / in endpoint is removed TCOMP-1757: Studio - context not set when call a @suggestable service TCOMP-1772: Code widget doesn’t allow multiline text TCOMP-1726: Update logos and colors TCOMP-1771: Record builder optimization (with static schema) TCOMP-1773: Upgrade log4j2 to 2.13.3 TCOMP-1774: Upgrade johnzon to 1.2.8 TCOMP-1775: Upgrade commons-lang3 to 3.11 TCOMP-1776: Upgrade commons-codec to 1.15 TCOMP-1777: Upgrade jgit to 5.9.0.202009080501-r TCOMP-1778: Upgrade jib-core to 0.15.0 TCOMP-1779: Upgrade batik to 1.13 TCOMP-1780: Upgrade TSBI to 2.4.0-20200925092052 TCOMP-1781: Upgrade asciidoctorj to 2.4.1 TCOMP-1782: Upgrade rrd4j to 3.7 TCOMP-1783: Upgrade netty to 5.0.0.Alpha2 TCOMP-1784: Upgrade ziplock to 8.0.4 TCOMP-1785: Upgrade JRuby to 9.2.13.0 TCOMP-1786: Upgrade to Apache Beam 2.24.0 TCOMP-1804: Upgrade to Apache Beam 2.25.0 TCOMP-1805: Upgrade TSBI to 2.5.0-20201030171201 TCOMP-1770: Performance loss on Ouput components in Studio TCOMP-1750: Deadlock at TPD job startup using the Component SDK and using the Workday component TCOMP-1759: Guess schema mixes columns returned by tck service TCOMP-1752: Make component-runtime class loader find classes in RemoteEngine JobServer TCOMP-1764: Upgrade to Apache Beam 2.23.0 TCOMP-1719: Header responses for icon not propagated correctly from Component-server-vault-proxy TCOMP-1733: NPE in Studio metadata connection with activeif on different layouts TCOMP-1734: Studio froze when installing a patch with azure-dls-gen2-1.10.0-component.car TCOMP-1736: JobImpl retrieves more than streaming.maxRecords parameter TCOMP-1739: Use scala version defined on parent for Spark related components TCOMP-1695: Support List type in Studio TCOMP-1737: Allow to force installation of an already existing component with the car bundle TCOMP-1728: Enforce use of the defined error contract in connectors TCOMP-1731: Make connectors docker image TSBI compliant TCOMP-1738: Upgrade to Apache Beam 2.22.0 TCOMP-1742: Upgrade johnzon to 1.2.7 TCOMP-1727: WebSocketContainer not present in ServletContext TCOMP-1696: Definition of an error contract to handle expected errors TCOMP-1729: Upgrade to Apache Beam 2.21.0 TCOMP-1730: Upgrade johnzon to 1.2.6 TCOMP-1719: Header responses for icon not propagated correctly from Component-server-vault-proxy TCOMP-1649: Tomcat bump to 9.0.31 broke talend-component:web goal TCOMP-1676: Starter-toolkit mvn package throws error when running for the first time TCOMP-1677: Using other types than String in Studio’s context values causes compilation error TCOMP-1679: Combination of @Required and @Suggestable on a field creates strange behaviour TCOMP-1682: Remove key attribute in UISchema for containers TCOMP-1686: antora helper function relativize corrupts documentation TCOMP-1694: [MAVEN PLUGIN] validateSvg argument is ineffective TCOMP-1698: UiSpecService injects a wrong property for suggestions and dynamic_values TCOMP-1718: Duplicated code in RecordConverters TCOMP-1702: Improve columns name TCOMP-1655: Upgrade jib-core to 0.13.1 TCOMP-1656: Upgrade log4j2 to 2.13.1 TCOMP-1657: Upgrade maven to 3.6.3 TCOMP-1658: Upgrade groovy to 3.0.2 TCOMP-1659: Upgrade lombok to 1.18.12 TCOMP-1660: Upgrade commons-compress to 1.20 TCOMP-1661: Upgrade commons-codec to 1.14 TCOMP-1662: Upgrade guava to 28.2-jre TCOMP-1663: Upgrade ziplock to 8.0.1 TCOMP-1664: Upgrade asciidoctorj to 2.2.0 and its dependencies TCOMP-1665: Upgrade jackson to 2.10.3 TCOMP-1666: Upgrade batik-codec to 1.12 TCOMP-1667: Upgrade jgit to 5.6.1.202002131546-r TCOMP-1668: Upgrade junit to 4.13 TCOMP-1669: Upgrade bouncycastle to 1.64 TCOMP-1670: Upgrade spark-core_2.11 to 2.4.5 TCOMP-1671: Upgrade maven-shade-plugin to 3.2.2 TCOMP-1672: Upgrade httpclient to 4.5.12 TCOMP-1673: Upgrade component-runtime-testing dependencies TCOMP-1674: Upgrade tomitribe-crest to 0.14 TCOMP-1678: Upgrade jgit to 5.7.0.202003090808-r TCOMP-1685: Provide docker images based on TSBI TCOMP-1687: More explicit exception messsage on reflection for findField TCOMP-1690: Upgrade netty to 4.1.48.Final TCOMP-1692: Update CXF to 3.3.6 due to CVE-2020-1954 TCOMP-1697: Update BouncyCastle to 1.65 TCOMP-1703: Upgrade log4j-2 to 2.13.2 TCOMP-1705: Upgrade to Apache Beam 2.20.0 TCOMP-1706: Upgrade OpenWebBeans to 2.0.16 TCOMP-1708: Upgrade groovy to 3.0.3 TCOMP-1710: Upgrade johnzon to 1.2.5 TCOMP-1711: Upgrade guava to 29.0-jre TCOMP-1712: Upgrade commons-lang3 to 3.10 TCOMP-1713: Upgrade jackson to 2.11.0 TCOMP-1714: Upgrade junit to 5.7.0-M1 TCOMP-1716: Upgrade maven shade plugin to 3.2.3 and misc libs TCOMP-1639: component-server incorrect response set in request TCOMP-1640: Ensure Intellij plugin works with Intellij Idea IU-201 TCOMP-1641: Upgrade OpenWebBeans to 2.0.15 TCOMP-1642: Upgrade Groovy to 3.0.1 TCOMP-1643: Add automatic scheduling eviction system on LocalCache TCOMP-1644: Upgrade log4j to 2.13.0 TCOMP-1645: Ensure correct wording is used in @Documentation TCOMP-1647: Upgrade netty to 4.1.45.Final TCOMP-1648: Unsafe Dependancy Resolution on jcommander TCOMP-1638: Inject services to delegate in proxy TCOMP-1619: Handle correctly DATETIME field type on AvroRecord TCOMP-1622: [DOC] @Icon is not supported on datastore/dataset TCOMP-1623: Change scheme for maven repos TCOMP-1628: Manage BigDecimal in RecordConverter TCOMP-1629: Ensure LocalConfiguration environment source replace dot with _ TCOMP-1630: Avoid NPE when configurationByExample() is called in a list of primitive without values TCOMP-1631: int attribute in pojo is transformed to double in a Record TCOMP-1632: Add a way to evict cached data from LocalCache TCOMP-1616: Upgrade OpenWebBeans to 2.0.14 in component-server and component-server-vault-proxy TCOMP-1617: Move mocked api results to github pages TCOMP-1618: Upgrade Junit to 5.6.0 TCOMP-1620: Upgrade to Apache Beam 2.18.0 TCOMP-1621: Upgrade to Johnzon 1.2.3 TCOMP-1624: @Service does not support list injections TCOMP-1625: Upgrade to xbean 4.16 TCOMP-1626: Ensure ContainerListenerExtensions can be sorted TCOMP-1627: Upgrade to Apache Beam 2.19.0 TCOMP-1633: Upgrade Groovy to 3.0.0 TCOMP-1634: Upgrade tomcat to 9.0.31 TCOMP-1596: Windows URI are broken TCOMP-1597: Httpclient does not support multi query parameters TCOMP-1598: validator task uses ENGLISH locale to validate instead of root one TCOMP-1612: Starter toolkit shouldn’t use the default 'STAR' icon in demo component TCOMP-1585: Upgrade netty to 4.1.43.Final TCOMP-1586: Upgrade ziplock to v8.0.0 TCOMP-1587: Upgrade jib to v0.12.0 TCOMP-1588: Upgrade JRuby to v9.2.9.0 TCOMP-1589: Upgrade crest to v0.11.0 TCOMP-1591: Update to Tomcat 9.0.29 TCOMP-1592: Update to Johnzon 1.2.2 TCOMP-1593: Update to OpenWebBeans 2.0.13 TCOMP-1595: Infinite partitionmapper shouldn’t require assesor TCOMP-1599: More unsafe usage tolerance on JVM versions TCOMP-1600: Upgrade to Tomcat 9.0.30 TCOMP-1606: Ensure job dsl can stop infinite inputs TCOMP-1608: Upgrade geronimo openapi to 1.0.12 TCOMP-1609: Ensure Intellij plugin works with Intellij Idea 2019 TCOMP-1611: Upgrade to Apache Beam 2.17.0 TCOMP-1613: Upgrade cxf to 3.3.5 TCOMP-1614: Upgrade groovy to 3.0.0-rc3 TCOMP-1615: Upgrade OpenWebBeans to 2.0.14 TCOMP-1560: Min and Max error message during configuration validation are reversed TCOMP-1563: Web Tester does not work anymore (maven/gradle goal/task) TCOMP-1573: Body encoder is called twice for each query TCOMP-1582: Deploy to Nexus 3.15 caused "Provided url doesn’t respond neither to Nexus 2 nor to Nexus 3 endpoints" TCOMP-1576: Add the possibility to desactivate http client redirection in HTTP Configurer TCOMP-1559: Support configuration of the maxBatchSize enablement TCOMP-1561: Custom action type shouldn’t need to be enforced to define a family method TCOMP-1562: Support jsonObject type in actions TCOMP-1564: Move to java.nio.Path instead of java.io.File in component-runtime-manager stack where possible TCOMP-1565: Upgade to Junit Jupiter 5.6.0-M1 TCOMP-1566: Don’t compute jvmMarkers per component module but once for all TCOMP-1567: Cache Artifact path in case of reuse TCOMP-1568: Lazily create the container services TCOMP-1569: Upgrade starter to gradle 6.0-rc1 TCOMP-1570: Ensure starter adds _placeholder entries in Messages.properties TCOMP-1571: Support [length] syntax to change array configuration TCOMP-1572: Validate that @Option is not used on final fields TCOMP-1574: Upgrade to CXF 3.3.4 TCOMP-1575: Upgrade to Spark 2.4.4 TCOMP-1577: Upgrade to xbean 4.15 TCOMP-1578: Upgrade asciidoctor-pdf to v1.5.0-beta.7 TCOMP-1581: Support JUnit5 meta annotations for our extensions TCOMP-1752: Make component-runtime class loader find classes in RemoteEngine JobServer TCOMP-1702: Improve columns name TCOMP-1685: Provide docker images based on TSBI TCOMP-1558: org.talend.sdk.component.api.service.record.RecordService must be serializable TCOMP-1548: Basic Remote Engine Customizer TCOMP-1550: Component configuration instantiation can be slow for complex configurations TCOMP-1551: ObjectFactory should default to fieldproperties when field injection is activated TCOMP-1553: Simplify and widden excluded classes for with transformer support TCOMP-1555: Upgrade to Tomcat 9.0.27 TCOMP-1556: Studio short, byte, BigDecimal and char types are wrong handled TCOMP-1557: Upgrade to Beam 2.16.0 TCOMP-1509: Intellij plugin does not declare java module preventing the plugin to run under last versions TCOMP-1526: Upgrade talend UI bundle (js) to 4.6.0 TCOMP-1533: JSON-B API does not enable to combine multiple adapters or (de)serializers in JsonbConfig TCOMP-1536: @DefaultValue ignored in documentation generation TCOMP-1541: Studio integration enforces JSON<→Record conversion instead of relying on rowStruct making number precision lost TCOMP-1542: Validator plugin uses family instead of pluginId (artifactId) to validate local-configuration TCOMP-1508: Don’t let Talend Starter Toolkit loose state on Enter in intellij TCOMP-1543: Add a uispec mapper TCOMP-1544: Update Geronimo JSON-P spec bundle to v1.3 TCOMP-1545: Update OpenWebBeans to version 2.0.12 TCOMP-1546: Update Meecrowave to 1.2.9 TCOMP-1547: Update Johnzon to 1.2.1 TCOMP-1279: Rewrite the pojo <→ record mapping to keep number types TCOMP-1504: Apache Beam 2.14.0 upgrade TCOMP-1505: Upgrade jackson-databind to 2.9.9.3 TCOMP-1506: Enable actions in bulk endpoint TCOMP-1507: Upgrade to johnzon 1.1.13 TCOMP-1511: Upgrade cxf to v3.3.3 TCOMP-1513: Upgrade to Tomcat 9.0.24 TCOMP-1514: Provide a RecordService to simplify record enrichment coding in processors TCOMP-1515: Record visitor API TCOMP-1517: Use netty 4.1.39.Final in junit http tools TCOMP-1518: Upgrade to slf4j 1.7.28 TCOMP-1519: Upgrade to jib-core 0.10.1 TCOMP-1520: Don’t use JsonNode with Avro Fields anymore TCOMP-1521: Upgrade to Beam 2.15.0 TCOMP-1522: Basic singer/tap/stitch integration with kit components TCOMP-1523: Upgrade Apache Geronimo OpenAPI to v1.0.11 TCOMP-1524: Upgrade starter to gradle 5.6 TCOMP-1525: Upgrade commons-compress to v1.19 TCOMP-1527: Remove beam Mapper/Processor wrapping support TCOMP-1528: Upgrade to maven 3.6.2 TCOMP-1529: Asciidoctor 2.1.0 upgrade TCOMP-1530: geronimo-annotation 1.2 upgrade TCOMP-1532: Upgrade to Junit 5.5.2 TCOMP-1535: Upgrade to johnzon 1.2.0 TCOMP-1537: Upgrade to Tomcat 9.0.26 TCOMP-1538: Upgrade to jackson 2.9.10 TCOMP-1539: Rework default direct runner/spark classloader rules TCOMP-1540: Ensure Asciidoctor documentation rendering releases properly JRuby threads (main usage only) TCOMP-1478: /documentation/component/{id} internationalization does not work when embedded TCOMP-1479: When generating the documentation, it can happen the lang is wrong due to ResourceBundle usage TCOMP-1480: Servers docker images don’t have curl or wget available TCOMP-1497: POJO to Record mapping is not supported in processors TCOMP-1498: SVG2Mojo wrongly log the source file as being created TCOMP-1499: component-form does not support array of object of object if 2 levels use the same field name TCOMP-1500: Ensure component-form button have a key to have an id and propagate errors in the front TCOMP-1503: EnvironmentSecuredFilter not working on /environment/ TCOMP-1482: Enable web tester to switch the language TCOMP-1483: Enable to expose the documentation through the web tester TCOMP-1485: Asciidoctor documentation does not enable titles (component name and configuration ones) to be translated TCOMP-1486: Ensure locale mapping is configurable in component-server TCOMP-1484: Junit 5.5.0 upgrade TCOMP-1487: AsciidocMojo should only use ROOT locale by default TCOMP-1488: Enable to translate gridlayout names TCOMP-1489: Upgrade Tomcat to v9.0.22 TCOMP-1491: Upgrade JIB to v1.4.0 TCOMP-1492: Upgrade jackson-databind to 2.9.9.1 TCOMP-1493: Rewrite component exception to ensure they can be loaded after a serialization TCOMP-1494: Upgrade to junit jupiter 5.5.1 TCOMP-1495: Upgrade to Geronimo OpenAPI 1.0.10 TCOMP-1496: [testing tool] MainInputFactory does not support Record TCOMP-1501: Remove generate mojo TCOMP-1502: [maven plugin] upgrade jib-core to 0.10.0 TCOMP-1469: Studio maven repository not found OOTB TCOMP-1472: Connectors maven goal does not work in 1.1.10 TCOMP-1473: Docker image text log setup should use ISO8601 and not HH:mm:ss.SSS TCOMP-1470: Upgrade Tomcat to v9.0.21 TCOMP-1471: Upgrade Geronimo OpenAPI to v1.0.9 TCOMP-1474: Ensure proxies definition are java >=11 friendly TCOMP-1425: Spark classes not excluded anymore in component-runtime-beam leading to classloading issues TCOMP-1427: dependencies.txt mojo uses timestamped versions for snapshots instead of just -SNAPSHOT TCOMP-1431: [maven] Asciidoctor files should be attached with adoc extension and not jar one TCOMP-1433: [form-model] itemwidget ignored from uischema builder TCOMP-1438: Index cache can lead to invalid index list of component TCOMP-1440: Bulk components without @ElementListener when used with component-extension (default in the server) TCOMP-1441: Missing parameter init in the UiSchema Trigger builder TCOMP-1446: Rework gradle lifecycle TCOMP-1419: Upgrade build to groovy 2.5.7 TCOMP-1420: Upgrade maven compiler to 3.1.2 TCOMP-1422: Filter allowed beam classes in component-server image TCOMP-1423: Enable to customize studio maven repository for deploy-studio maven and gradle goal/task TCOMP-1426: Ensure Spark rule and @WithSpark uses a default version consistent with the runtime TCOMP-1430: Deprecate built-in icons in favor of vendor specific icons TCOMP-1432: basic dita generation for the component documentation TCOMP-1434: [form-model] Add withCondition to UISchema builder TCOMP-1435: Dont use beam_sdks_java_core shaded libraries TCOMP-1437: Add infinite metadata to ComponentDetail TCOMP-1444: Remove KnownJarsFilter since it is no more used to discover components TCOMP-1445: Icon must support SVG TCOMP-1448: [starter] provide a basic OpenAPI integration TCOMP-1449: Upgrade XBean to v4.14 TCOMP-1450: Add a read-only bulk endpoint in component-server TCOMP-1451: [upgrade] Johnzon 1.1.12 TCOMP-1452: [upgrade] Meecrowave 1.2.8 TCOMP-1453: Upgrade to CXF 3.3.2 TCOMP-1455: Prepare DateTime support in configurations TCOMP-1457: Upgrade to Apache Beam 2.13.0 TCOMP-1458: Ensure _placeholder presence is encouraged and validated TCOMP-1459: Experimental way to patch a component dependency TCOMP-1461: Extension API for the validator plugin TCOMP-1462: Validate through the corresponding build task provided SVG TCOMP-1464: Upgrade to OpenWebBeans 2.0.11 TCOMP-1465: Upgrade to JUnit 5.5.0-RC1 TCOMP-1466: Upgrade to ziplock 8.0.0-M2 TCOMP-1467: Upgrade mock server (testing tool) to netty 5.0.0.Alpha2 TCOMP-1468: Support docker-compose >= 1.23 in vault-proxy TCOMP-1374: ensure Utf8 avro strings don’t leak in AvroRecord API, even using get(Object.class, …) TCOMP-1375: When two sources use the same dataset and one source has additional required parameter the validation fails TCOMP-1384: Enhance studio guess schema algorithm to find implicitly the action to call if needed TCOMP-1388: Can’t change the dataset name in starter TCOMP-1389: Intellij starter fails to generate a project TCOMP-1398: Using after option of @updateable can lead to a null pointer exception in component-form TCOMP-1401: Documentation table is broken TCOMP-1407: Databricks: interface javax.json.stream.JsonGeneratorFactory is not visible from class loader TCOMP-1386: Add withRecord(String,Record) in Record.Builder TCOMP-1387: Use icon bundle version 3.1.0 TCOMP-1412: Add rest and couchbase icon to component api TCOMP-1376: Upgrade jupiter to 5.4.2 TCOMP-1385: talend.component.server.component.registry must be a list TCOMP-1390: Move component-api to component-runtime repository TCOMP-1392: Tomcat 9.0.19 upgrade TCOMP-1402: Provide a placeholder for classpath extensions in docker images TCOMP-1403: Upgrade asciidoctor to 2.0.0 and asciidoctor-pdf to alpha17 TCOMP-1404: Upgrade to Apache Beam 2.12.0 TCOMP-1408: Starter does not support types starting with a lowercase TCOMP-1411: ComponentManager relies on beam jar name. This is unlikely and should move to beam integration module. TCOMP-1417: Upgrade to Geronimo OpenAPI 1.0.8 TCOMP-1326: Avro Schema is not serializable as JSON so guess schema action does not work when compoennt-runtime-beam is present TCOMP-1330: Shade extensions don’t inherit from pluginrepositories TCOMP-1340: Tools webapp (talend-component:web) does not support changing the locale anymore TCOMP-1343: Use LogicalTypes.timestampMillis() on DATETIME for avro record builder TCOMP-1360: Renaming an option (@Option("custom")) does not work on fields of type object TCOMP-1370: ImageM2Mojo does not set timestamp in the docker image leading to component-server having a wrong lastUpdated value TCOMP-1372: Nested components don’t expose their doc deterministicly until it is overriden TCOMP-1341: Register deploy in studio task OOTB in gradle extension TCOMP-1325: Upgrade CXF to 3.3.1 TCOMP-1327: /environment iterates over deployed plugin for each call, this is not needed TCOMP-1328: Upgrade to Beam 2.11.0 TCOMP-1329: Lazy initialize parameter model to have a quicker cold start in plain main(String[]) TCOMP-1331: Use java 8u191 as base docker image TCOMP-1332: Provide a simple way to filter configurations and component on /index endpoints TCOMP-1334: Add a mojo to generate the list of components/services classes TCOMP-1335: Add in doc mojo table the type of configuration the parameter belongs to TCOMP-1336: Allow output processors to only have an @AfterGroup taking the list of record of the group in parameter TCOMP-1346: Upgrade to Tomcat 9.0.17 TCOMP-1347: Upgrade to Slf4j 1.7.26 TCOMP-1348: [form-core] Ensure suggestions trigger is bound to "change" event too TCOMP-1349: [form-core] When a tab is empty, don’t show it TCOMP-1350: talend.component.server.component.registry should support glob pattern TCOMP-1351: Upgrade jsoup for Spark Cluster Testing module TCOMP-1353: component-server must not use TALEND-INF/dependencies.txt but another path TCOMP-1354: Enforce services to belong to the delcaring service class TCOMP-1361: Upgrade to asciidoctorj 2.0.0-RC.1 TCOMP-1362: Beam Wrapped Components should throw shared exception types TCOMP-1366: Upgrade to XBean 4.13 to not track all classes scanned TCOMP-1371: Upgrade to Apache Geronimo OpenAPI 1.0.7 TCOMP-1307: support char and character types in configuration. TCOMP-1312: Component-form-core shouldn’t trigger validation of object due to conditional visibility (only individual fields are validable) TCOMP-1314: category field of the starter is broken TCOMP-1316: [build] Ensure snapshot use timestamped versions in dependencies.txt TCOMP-1306: Add RecordPointerFactory to enable to extract data from Record using json pointer spec TCOMP-1315: Ensure @Internationalized can use shortnames too in Messages.properties TCOMP-1303: Support docker configs/secrets in docker images TCOMP-1304: Vault proxy should support token configuration TCOMP-1305: Upgrade to beam 2.10.0 TCOMP-1308: Upgrade to Talend UI 2.6.0 TCOMP-1309: Upgrade to Component API 1.1.5 TCOMP-1310: Ensure there is a basic secured mecanism to store configuration data TCOMP-1317: Use Apache Geronimo Microprofile Config extensions (docker and secured string) TCOMP-1318: Upgrade to Apache Meecrowave 1.2.7 TCOMP-1319: Upgrade Apache Geronimo Metrics to 1.0.3 TCOMP-1320: Upgrade to Apache Geronimo OpenAPI 1.0.6 TCOMP-1321: Upgrade to Apache Geronimo OpenTracing 1.0.2 TCOMP-1322: Upgrade to Apache Geronimo Config 1.2.2 TCOMP-1263: When using @Updateable(after=xxx) the visibility condition (@ActiveIf) of the after field shouldn’t be inherited TCOMP-1264: AvroSchema does not unwrap null(able types) to map to Schema model TCOMP-1265: dataset / datastore cloud validation : allow nested configuration types TCOMP-1267: /documentation does not filter properly component TCOMP-1281: Add jackson-mapper-asl in docker image of the server TCOMP-1298: Support restricted lists for @Proposable TCOMP-1297: make max batch size property configurable for family and components through LocalConfiguration TCOMP-1266: Enhance starter to support dataset and datastore TCOMP-1268: Ensure /environment is not callable if not local or secured TCOMP-1269: Ensure ErrorReportValve does not leak Tomcat version OOTB TCOMP-1271: Upgrade to talend UI 2.3.0 TCOMP-1272: Move multiSelectTag to multiSelect for web environment TCOMP-1273: [build/dev plugin] Automatically open the browser for talend-component:web task/goal TCOMP-1276: Exclude xerces from component loadable resources for XMLReaderFactory TCOMP-1282: Upgrade meecrowave to 1.2.6 TCOMP-1283: Upgrade cxf to 3.3.0 TCOMP-1284: Upgrade to johnzon 1.1.11 TCOMP-1292: Provide a vault friendly integration for the server TCOMP-1293: Upgrade to Tomcat 9.0.16 TCOMP-1295: Ensure local-configuration.properties of a container are merged TCOMP-1296: Ensure user can enrich families with custom jar+configuration TCOMP-1245: Provided services (SPI) by tacokit not available TCOMP-1246: Rework docker image setup to use jib TCOMP-1247: Upgrade geronimo metrics to 1.0.2 TCOMP-1248: Upgrade to geronimo opentracing 1.0.3 TCOMP-1249: Provide segment extractor for doc endpoint TCOMP-1250: Make component documentation (@Documentation on component) i18n friendly TCOMP-1251: cache avrocoders used in SchemaRegistryCoder TCOMP-1252: Remove html support in documentation endpoint TCOMP-1253: Refine OpenAPI documentation TCOMP-1256: Add mapDescriptorToClassLoader to create a classloader from a list of gav TCOMP-1258: Support to build a Record from a provided Schema TCOMP-1259: Add getOptional to Record TCOMP-1223: byte[] not supported in AvroRecord (beam) TCOMP-1222: Ensure @WithComponents and @Environment are compatible TCOMP-1234: Upgrade to beam 2.9.0 TCOMP-1235: Upgrade to antora 2 TCOMP-1237: Upgrade component-api to 1.1.2 TCOMP-1238: Upgrade metrics and opentracing microprofile libraries in docker image to use Geronimo extensions TCOMP-1239: OpenWebBeans 2.0.9 upgrade TCOMP-1240: Johnzon 1.1.11 upgrade TCOMP-1242: Runtime validation error message wrongly interpolated TCOMP-1243: Ensure component classloader isolates the system classloader resources except for the JVM ones TCOMP-1170: [regression] http testing module pom imports netty and jsonb stack TCOMP-1181: tacokit can’t pass the long type field from ui rightly TCOMP-1187: Job DSL does not support correctly parameters when they are URI/URL TCOMP-1189: Ensure primitive are not nullable in Record model (builder) TCOMP-1191: [beam] BeamIOTransformer does not support serialization of complex objects correctly TCOMP-1192: Ensure Avro schema union is interpreted as nullable in Record Schema model TCOMP-1194: [testing] Ensure BeamEnvironment adds component-runtime-beam TCOMP-1196: Nested maven repository not used for component module TCOMP-1197: Tacokit beam tests. NPE when creating the schema with RECORD type. TCOMP-1198: Tacokit beam tests. SchemaParseException ⇒ drop unsupported characters TCOMP-1200: Packages not defined from nested repository classes TCOMP-1201: includeTransitiveDependencies option of nested-maven-repository does not work TCOMP-1202: Refine avro classloading exclusion to accept hadoop and mapred packages TCOMP-1205: Empty JSon object lead to NPE TCOMP-1209: Ensure SerializableCoder is replaced with a contextual version to support Talend Component Kit classloading model TCOMP-1210: BeamComponentExtension should let the exception go back to the caller when the transform fails TCOMP-1215: Nested maven repository in jars don’t go through transformers TCOMP-1218: Record entries order shouldn’t be sorted by the runtime TCOMP-1185: Support maxBatchSize in Job test runner for standalone mode TCOMP-1171: Remove component proxy server from the project TCOMP-1182: Ensure the property editor for the configuration registers the default converters TCOMP-1183: Upgrade JRuby to 9.2.4.0 TCOMP-1184: Avoid to do a group by key in BeamExecutor (job DSL) when not needed TCOMP-1188: Tolerate null for dates in Records TCOMP-1190: Enable secure processing for DocumentBuilderFactory instances TCOMP-1193: Add injectable ContainerInfo with the containerId (plugin) in services TCOMP-1195: Enable user to extend BeamEnvironment test tempalte more easily TCOMP-1199: Nested repository not used when the classpath is not composed of a single jar TCOMP-1204: [dependency upgrade] XBean 4.12 TCOMP-1207: [beam] add ContextualSerializableCoder TCOMP-1213: Upgrade guava to v27 for testing modules TCOMP-1216: Take into account the visibility for the parameter validation TCOMP-1217: Add JVM system property talend.component.runtime.serialization.java.inputstream.whitelist for our custom object input stream TCOMP-1219: Upgrade starter to gradle 5 TCOMP-1220: Upgrade Maven to 3.6.0 in starter TCOMP-1121: [tacokit proxy] suggestion trigger creation issue TCOMP-1122: [tacokit proxy] slefRefrence filter configuration type by name, type and family TCOMP-1123: Processor component onNext duplicate columns in record for rowStructs TCOMP-1126: UiSpecService shouldn’t show the documentation by default TCOMP-1129: form core - $selfReference breaks triggers TCOMP-1130: component form - default value of maxBatchSize prop loose it type. TCOMP-1131: [beam integration] Ensure Coder is contextual (classloader) TCOMP-1132: Ensure beam custom Coders implement equals.hashCode for beam contract TCOMP-1148: Asciidoctor documentation fails for collection of objects TCOMP-1149: [testing] BeamEnvironment does not reset PipelineOptionsFactory properly for beam > 2.4 TCOMP-1155: [proxy server] arrays not supporting null values in ConfigurationFormatter TCOMP-1159: AvroSchema does not support DATETTIME type (beam module) TCOMP-1168: Avro record implementation ignores nullable/union TCOMP-1143: Ensure icons are validated and fail the build if a custom one is missing (validate mojo) TCOMP-1112: Let beam PTransform define an @ElementListener method to set the component design (inputs/outputs) TCOMP-1113: Simplify the scanning by assuming there is a TALEND-INF/dependencies.txt in components TCOMP-1120: BeamMapperImpl.isStream not accurate for UnboundedSource TCOMP-1124: Add /metrics endpoint TCOMP-1125: Extend CustomPropertyConverter to pass the convertion context TCOMP-1127: Record doesn’t support null values TCOMP-1133: CXF 3.2.7 upgrade TCOMP-1134: Ensure any input/output have a dataset TCOMP-1135: Ensure any dataset has a datastore TCOMP-1136: deprecate "generate" mojo TCOMP-1145: [dependency upgrade] Beam 2.8.0 TCOMP-1146: implement infinite=true in PartitionMapper/Input TCOMP-1150: Upgrade rat plugin to 0.13 TCOMP-1154: Required validation at runtime ignores lists and nested objects TCOMP-1157: [dependency upgrade] Tomcat 9.0.13 TCOMP-1158: Enable JUnit test collector to use a static storage instead of thread related one TCOMP-1160: Upgrade spark to 2.4.0 TCOMP-1161: Upgrade shade plugin to 3.2.1 TCOMP-1162: Upgrade nested-maven-repository shade transformers to support last maven versions TCOMP-1163: Upgrade openwebbeans to 2.0.8 TCOMP-1164: Validate mojo does not log any success information TCOMP-1165: Dependency mojo does not log any success information TCOMP-1166: Documentation mojo does not log generated files properly TCOMP-1167: Beam-Avro record name generation should use avro fingerprint to be more unique than current logic TCOMP-1086: Fix documentation about DiscoverSchema TCOMP-1064: Update action can’t receive List parameter TCOMP-1110: When a configuration has no layout and uses @AfterGroup the configuration is lost TCOMP-1111: Move to PropertyEditorRegistry from xbean instead of using the deprecated static class TCOMP-1000: @Option name value is not respected on fields TCOMP-1008: Enum order is lost TCOMP-1009: (web) OptionsOrder ignored for tables (List), fields located in random order TCOMP-1028: [tools-webapp] submit button no more functional TCOMP-1031: DiscoverSchema parameters are not correctly mapped in Studio GuessSchema runtime TCOMP-1044: Fix java.lang.ClassCastException in TableActionParameter TCOMP-1046: String option can’t set default value from a file TCOMP-1056: ActiveIf doesn’t work in advanced settings TCOMP-1072: Metadata migration issues TCOMP-1074: talend-component mvn plugin : deploy-in-studio need to rise an error when component is already installed TCOMP-1075: component reload file on windows after deploying a modified jar TCOMP-1076: component starter - fix mapper generation (Record integration) TCOMP-1077: component starter - ensure kit version are updated atomically. TCOMP-1078: Guess Schema button is not shown on Basic Settings view TCOMP-1082: Fix Exception during HealthCheck parameter deserialization TCOMP-1085: [classloader] com.sun is too wide as exclusion TCOMP-1104: Fix drag and drop issue for dataset/datastore metadata TCOMP-779: Drop down list Java type in configuration class TCOMP-819: Processor doesn’t produce more than 1 row on each iteration TCOMP-917: Migration handler need only to receive component configuration TCOMP-941: Default and init values are ignored in connection wizzard (datastore/dataset) TCOMP-968: Trigger AsyncValidation call only when option annotated with Validable is changed TCOMP-970: Add support for complex parameter types for AsyncValidation methods TCOMP-973: component migration - the configuration version need to be serialized in addition to the version of the component TCOMP-984: Integrate ParameterizedTest with component-runtime-http-junit capture mode TCOMP-988: component migration - fix nested configuration migration TCOMP-989: .car studio install command breaks config.ini of the studio TCOMP-991: metadat : ignore activations from config not being part of the form while creating metadata TCOMP-996: metadata : migration issues TCOMP-1001: [proxy] ConfigurationClient should expose a migrate method TCOMP-1011: Ensure datastore/dataset i18n names are validated by the maven/gradle plugins TCOMP-1013: Add an operator support in @ActiveIfs (OR/AND switch) TCOMP-1014: Ensure a dataset has a source which has no other required parameters in the validator TCOMP-1029: Extend ActiveIf EvaluationStrategy with CONTAINS strategy TCOMP-1063: Integrate Record API to the studio TCOMP-1069: restrict input branches for output components to only one. TCOMP-1071: support actions i18n display name TCOMP-1092: Ensure @Configuration POJO are injectable as Supplier in services TCOMP-1094: Add FullSerializationRecordCoder coder for Record in beam module TCOMP-1095: Ensure all configuration type models root entries are named "configuration" TCOMP-993: [proxy] Propagate UiSpecContext in referenceservice#findByTypeAndName TCOMP-994: [dependency upgrade] CXF 3.2.6 TCOMP-1003: [dependency upgrade] Tomcat 9.0.12 TCOMP-1004: [dependency upgrade] Log4j2 2.11.1 TCOMP-1015: Upgrade icons to 1.0.0 TCOMP-1019: (form) enum should lead to restricted datalist TCOMP-1037: [dependency upgrade] Johnzon 1.1.9 TCOMP-1038: Drop spring client from component-form-core TCOMP-1041: HttpClient should enable to process InputStream directly TCOMP-1042: Upgrade to JUnit 5.3.1 TCOMP-1045: Add documentation in metadata and enable to use it in the UI on configuration TCOMP-1047: Make Suggestable text field editable (align with web) TCOMP-1048: Add update API for configuration TCOMP-1049: Add completion support for actions displayname in intellij plugin TCOMP-1050: Provide simple OAuth1 integration TCOMP-1051: Remove brave and move to geronimo-opentracing TCOMP-1054: Introduce @Configuration API TCOMP-1055: remove the ExecutionResource TCOMP-1057: Add ActiveIf on @Proposable test-case TCOMP-1058: Add DefaultValue on proposable/dynamicValue testcase TCOMP-1059: Rework generic record format TCOMP-1073: [maven/gradle plugin] Add configuration support in web goal TCOMP-1079: Document new Record structure TCOMP-1080: [dependency upgrade] Meecrowave 1.2.4 TCOMP-1081: ComponentManager should ignore engine classes in its filtering TCOMP-1087: Jsonb service should serialize byte[] as BASE64 TCOMP-1089: [starter] Upgrade gradle to 4.10.2 TCOMP-1090: [form] Main/Advanced order not respected when some remote action are involved TCOMP-1091: Ensure main component is preferred over test ones in a maven project TCOMP-1093: [dependency upgrade] netty 4.1.30.Final for junit http testing module TCOMP-1096: [dependency upgrade] xbean 4.10 TCOMP-1097: [dependency upgrade] Beam 2.7.0 TCOMP-1099: Upgrade web ui bundle to 1.0.2 TCOMP-1101: Add conditional rendering in the generated documentation TCOMP-1102: Reflect in documentation that Validable/AsyncValidation doesn’t support Object types TCOMP-1106: Enable to generate the component documentation in multiple languages TCOMP-1107: ConfigurableClassLoader does not priviledges container classloader for getResourceAsStream TCOMP-877: [documentation] Sample implementation of bulk/batch/commit-interval using groups TCOMP-980: Provide a ValidationService in server-proxy TCOMP-985: Align docker git metada on out Standard TCOMP-998: [dependency upgrade] Apache Commons Compress 1.18 TCOMP-911: Suggestions callback doesn’t support Configuration parameters TCOMP-921: String cannot be cast to Boolean when adding table with checkboxes TCOMP-922: component manager : support loading dependencies from job lib folder. TCOMP-924: component-kit.js errors are not sent to the error handler TCOMP-927: talend-component:web errors are not always unwrapped and understandable TCOMP-934: Ensure Studio rely on category and doesn’t append family name TCOMP-960: Suggestions parameters are not correctly resolved in Studio TCOMP-961: Default value of Suggestions method parameter is ignored TCOMP-964: ClassCastException is thrown when non-string values are used as Suggestions method parameter TCOMP-825: Provide component server proxy TCOMP-928: Add negate and evaluation strategy to @ActiveIf TCOMP-929: Ensure category contains the family TCOMP-816: Check migration feature and implement missing use-cases TCOMP-918: create a mvn bom with tacokit stack to keep some dependencies aligned between component-runtime and it’s studio integration TCOMP-932: Avoid Kafka recursive logging for component server TCOMP-933: Drop component-kit.js module TCOMP-935: Component server should log application and service in kafka mode TCOMP-938: Add a builtin::http trigger in the server proxy TCOMP-939: Ensure the proxy server can lookup references with a SPI TCOMP-943: (web) Grand parent references for triggers not well resolved TCOMP-944: (proxy server) Ensure the trigger are well resolved for references TCOMP-947: (maven/gradle) ensure web task logs there is a UI TCOMP-953: Upgrade to ziplock 7.0.5 TCOMP-954: Upgrade netty to 4.1.28.Final for the test stack TCOMP-958: Componentvalidator error message in case of an unsupported type is misleading TCOMP-959: [dependency upgrade] Upgrade to icon bundle 0.202.0 TCOMP-962: .car deploy-in-studio command (CarMain) should support to override an existing version TCOMP-965: [dependency upgrade] Apache Beam 2.6.0 TCOMP-966: Ensure Studio integration renames HTTP threads to identify them more explicitly TCOMP-967: Ensure parameter index is in metadata for services and constructors TCOMP-919: Starter doesn’t synchronize correctly with central versions TCOMP-920: Use Meecrowave 1.2.3 TCOMP-888: Designer pipeline records counter are wrong for tacokit components with multiples outputs TCOMP-899: Update Beam 2.5.0 compatibility TCOMP-903: [tacokit studio integration] - Guess schema - better handling of number types recognition TCOMP-904: [tacokit studio integration] - fix job classpath generation TCOMP-913: Fix absolute path resolution for child of child use-case TCOMP-900: [tacokit studio integration] - Handle conditional outputs TCOMP-898: Ensure starter will be able to auto update its versions to avoid redeployments TCOMP-905: Enrich scanning exclusion set TCOMP-906: Minimalist jsonObject to IndexeredRecord utilities for beam TCOMP-907: Support maxBatchSize as in the studio in Beam TCOMP-910: Add maxbatchsize as built in parameter to Processor meta model TCOMP-915: Upgrade Apache Meecrowave to 1.2.2 TCOMP-822: [Windows] deploy-in-studio & car copy jar command in mvn plugin - don’t work if the studio is running TCOMP-844: Service default method forwarded to interface method instead of implementation one if exists TCOMP-848: [junit5] implicit mock collector and emitter are not resetted per method TCOMP-851: [form] UiSchema shouldn’t have a JsonSchema TCOMP-858: @OptionsOrder not respected by form-core TCOMP-862: [form-core] ".." path is not correctly resolved TCOMP-863: Job DSL doesn’t support multiple outputs TCOMP-873: Fix shade junit-http module : remove shaded dependencies from generated artifact TCOMP-889: [form] arrays are lost in trigger paths TCOMP-890: Merge the component outputs (by name) from @AfterGroup and @ElementListener TCOMP-893: Don’t log a warning for services when parameters don’t have i18n support TCOMP-834: Ensure that component has only one configuration argument. TCOMP-845: [junit] ComponentsHandler misses findService TCOMP-846: [junit] allow to inject current plugin services in test class TCOMP-847: Support gzip in JUnit HTTP tooling TCOMP-849: [junit http] support to match the request payload TCOMP-850: MavenDecrypter should tolerate ${env.xxx} syntax TCOMP-861: Ensure Car Mojo can be skipped TCOMP-887: [studio] add chunk size advanced common param for processors & output TCOMP-892: Validate runtime configuration before executing the runtime TCOMP-829: Configuration Type tree is not correctly computed TCOMP-830: Move all configuration to Microprofile Config instead of DeltaSpike TCOMP-832: Provide a way to access lastUpdatedTimestamp in rest api TCOMP-833: Upgrade gradle+maven for the starter TCOMP-839: Add an API to load lazily the potential values of a list TCOMP-840: Upgrade icon bundle to 0.190.2 TCOMP-841: Add validation of option names in the validator TCOMP-852: [dependency upgrade] Upgrade shrinkwrap-resolver-impl-maven to 3.1.3 TCOMP-855: Support service injections in services TCOMP-856: [dependency upgrade] OpenWebBeans 2.0.6 TCOMP-857: SimpleCollector must not depend on junit 4 TCOMP-864: Mojo should be thread safe for car/dependencies.txt generation TCOMP-867: Expose Injector service TCOMP-868: Create an ObjectFactory service TCOMP-869: Ensure actions can get injected the requested lang TCOMP-870: Provide Beam DoFn to simplify the migration from IndexedRecord to jsonObject TCOMP-876: Allow custom converters in form-core TCOMP-878: Add beam in the docker image OOTB TCOMP-879: CarMojo doesn’t use car extension to attach the artifact TCOMP-880: [dependency upgrade] Maven 3.5.4 TCOMP-881: [dependency upgrade] CXF 3.2.5 TCOMP-882: [dependency upgrade] Tomcat 9.0.10 TCOMP-883: [dependency upgrade] Beam 2.5.0 TCOMP-884: [dependency upgrade] Upgrade to icon bundle 0.197.0 TCOMP-894: [dependency upgrade] Johnzon 1.1.8 TCOMP-895: [dependency upgrade] xbean 4.9 TCOMP-827: Fix Automatic-Module-Name TCOMP-811: Upgrade to tomcat 9.0.8 TCOMP-826: Extract component model from component server to a new artifact TCOMP-763: Add a dev mode in the studio for tacokit TCOMP-802: Add method to upload dependencies from .car to nexus TCOMP-808: Upgrade to JUnit 5.2.0 TCOMP-809: compress js and css for the starter TCOMP-810: ui spec service uses a multiselecttag for a proposable on a string field TCOMP-804: Idea plugin doesn’t render properly configuration inputs TCOMP-798: intellij plugin - add official starter url TCOMP-799: @Checkable expects the datastore name to match the validation name TCOMP-806: Ensure server and starter support gzip TCOMP-643: Deployment TCOMP-770: Removing component from web UI causes wrong number of components in summary TCOMP-775: Starter - Fix properties keys generation TCOMP-776: component-kit.js ignore credentials TCOMP-783: ActiveIfs doesn’t make option visible TCOMP-796: Datastore check (@Checkable) should default meta parameters to "datastore" if none is found TCOMP-773: Extend the http client api to handle more generic use cases TCOMP-771: ConfigurableClassLoader should skip scala.* classes TCOMP-772: Upgrade icon set to ui/icons 0.179.0 TCOMP-774: Upgrade xbean to 4.8 TCOMP-768: More tolerance of configuration prefix for implicit migration of configuration node in form core library TCOMP-756: Setup maven clirr plugin for component-api +testing TCOMP-762: Starter should only propose a single category level in the ui TCOMP-767: Ensure the configurationtype endpoints have matching name/path values TCOMP-761: Merge component-runtime-manager and component-runtime-standalone TCOMP-764: Clean up component-form-core dependencies TCOMP-765: Upgrade to batik 1.9.1 TCOMP-752: Fix Advanced settings and Test connection button appearance in repository wizard TCOMP-757: Duplicate method name "writeReplace" with signature "()Ljava.lang.Object;" in class file TCOMP-751: Support gzip compression on component-server TCOMP-753: Make classpath scanning to find component configurable TCOMP-758: Support component-server server configuration from system properties TCOMP-759: Enum must be i18n TCOMP-738: Component Server should respect ~/.m2/settings.xml local repository if it exists TCOMP-739: SerializationTransformer shouldn’t use ComponentManager to avoid ClassNotFoundException TCOMP-740: UISpecService should be reactive and use a CompletionStage based API TCOMP-741: UISpecService configuration support TCOMP-742: Configuration Type properties should be rooted TCOMP-744: Ensure wrapped BeamIO uses the right TCCL TCOMP-745: [Dependency Upgrade] CXF 3.2.4 TCOMP-746: [Dependency Upgrade] Tomcat 9.0.6 TCOMP-747: [Dependency Upgrade] Log4j2 2.11.0 TCOMP-748: Make configurationtype index endpoint lighter OOTB TCOMP-749: Intellij Idea plugin TCOMP-750: Unify @Pattern using javascript regex instead of a mixed mode TCOMP-734: Add support for context and globalMap values in Tacokit component settings TCOMP-733: support to use a beam pipeline under the hood for beam components in di TCOMP-693: Integrate Migration API TCOMP-737: upgrade to beam 2.4.0 TCOMP-731: Configuration Type migration handler skipped TCOMP-725: MavenDecrypter doesn’t support comments in settings.xml TCOMP-726: When a component is not found the error message can be misleading TCOMP-728: Http client doesn’t ignore empty query parameters TCOMP-722: WebSocket connection fails with a NPE when the endpoint doesn’t exists TCOMP-723: Adding configurationByExample utility to create query string for Job DSL TCOMP-724: Documentation endpoint doesn’t support HTML TCOMP-446: Support Embedded Documentation TCOMP-650: Ensure component can be executed in beam pipelines TCOMP-651: Ensure beam components can be wrapped and used through the Talend Component Kit Framework TCOMP-653: Web Form metamodel service TCOMP-655: Catalog service TCOMP-656: UISpec compatibility TCOMP-658: Add test Source/Sink collectors in JUnit integration TCOMP-659: Basic job builder API to simplify JUnit tests TCOMP-662: Validation Mojo TCOMP-664: Local testing server for dev TCOMP-675: Select a communication solution for Talend Component Kit server TCOMP-680: Register components into the Studio Palette TCOMP-681: Studio parameters form integration TCOMP-682: Studio Metadata integration TCOMP-683: Studio Runtime integration TCOMP-691: Create context menu for Tacokit node in repository panel TCOMP-719: Support Input Definition TCOMP-720: Support Output Definition TCOMP-721: Initial Widget Definitions

In some cases you can need to add some actions that are not related to the runtime. For example, enabling users of the plugin/library to test if a connection works properly. To do so, you need to define an @Action, which is a method with a name (representing the event name), in a class decorated with @Service: Services are singleton. If you need some thread safety, make sure that they match that requirement. Services should not store any status either because they can be serialized at any time. Status are held by the component. Services can be used in components as well (matched by type). They allow to reuse some shared logic, like a client. Here is a sample with a service used to access files: The service is automatically passed to the constructor. It can be used as a bean. In that case, it is only necessary to call the service method. Some common actions need a clear contract so they are defined as API first-class citizen. For example, this is the case for wizards or health checks. Here is the list of the available actions: Mark an action works for closing runtime connection, returning a close helper object which do real close action. The functionality is for the Studio only, studio will use the close object to close connection for existed connection, and no effect for cloud platform. Type: close_connection API: @org.talend.sdk.component.api.service.connection.CloseConnection Returned type: org.talend.sdk.component.api.service.connection.CloseConnectionObject Sample: Mark an action works for creating runtime connection, returning a runtime connection object like jdbc connection if database family. Its parameter MUST be a datastore. Datastore is configuration type annotated with @DataStore. The functionality is for the Studio only, studio will use the runtime connection object when use existed connection, and no effect for cloud platform. Type: create_connection API: @org.talend.sdk.component.api.service.connection.CreateConnection This class marks an action that explore a connection to retrieve potential datasets. Type: discoverdataset API: @org.talend.sdk.component.api.service.discovery.DiscoverDataset Returned type: org.talend.sdk.component.api.service.discovery.DiscoverDatasetResult Sample: Mark a method as being useful to fill potential values of a string option for a property denoted by its value. You can link a field as being completable using @Proposable(value). The resolution of the completion action is then done through the component family and value of the action. The callback doesn’t take any parameter. Type: dynamic_values API: @org.talend.sdk.component.api.service.completion.DynamicValues Returned type: org.talend.sdk.component.api.service.completion.Values Sample: This class marks an action doing a connection test Type: healthcheck API: @org.talend.sdk.component.api.service.healthcheck.HealthCheck Returned type: org.talend.sdk.component.api.service.healthcheck.HealthCheckStatus Sample: Mark an action as returning a discovered schema. Its parameter MUST be a dataset. Dataset is configuration type annotated with @DataSet. If component has multiple datasets, then dataset used as action parameter should have the same identifier as this @DiscoverSchema. Type: schema API: @org.talend.sdk.component.api.service.schema.DiscoverSchema Returned type: org.talend.sdk.component.api.record.Schema Sample: Mark a method as returning a Schema resulting from a connector configuration and some other parameters.Parameters can be an incoming schema and/or an outgoing branch.`value' name should match the connector’s name. Type: schema_extended API: @org.talend.sdk.component.api.service.schema.DiscoverSchemaExtended Returned type: org.talend.sdk.component.api.record.Schema Sample: Mark a method as being useful to fill potential values of a string option. You can link a field as being completable using @Suggestable(value). The resolution of the completion action is then done when the user requests it (generally by clicking on a button or entering the field depending the environment). Type: suggestions API: @org.talend.sdk.component.api.service.completion.Suggestions Returned type: org.talend.sdk.component.api.service.completion.SuggestionValues Sample: This class marks an action returning a new instance replacing part of a form/configuration. Type: update API: @org.talend.sdk.component.api.service.update.Update Extension point for custom UI integrations and custom actions. Type: user API: @org.talend.sdk.component.api.service.Action Mark a method as being used to validate a configuration. this is a server validation so only use it if you can’t use other client side validation to implement it. Type: validation API: @org.talend.sdk.component.api.service.asyncvalidation.AsyncValidation Returned type: org.talend.sdk.component.api.service.asyncvalidation.ValidationResult Sample: These actions are provided - or not - by the application the UI runs within. always ensure you don’t require this action in your component. Mark the decorated field as supporting suggestions, i.e. dynamically get a list of valid values the user can use. It is however different from @Suggestable by looking up the implementation in the current application and not the services. Finally, it is important to note that it can do nothing in some environments too and that there is no guarantee the specified action is supported. API: @org.talend.sdk.component.api.configuration.action.BuiltInSuggestable Internationalization is supported through the injection of the $lang parameter, which allows you to get the correct locale to use with an @Internationalized service: You can combine the $lang option with the @Internationalized and @Language parameters.

The component API is declarative (through annotations) to ensure it is: Evolutive. It can get new features without breaking old code. As static as possible. Because it is fully declarative, any new API can be added iteratively without requiring any change to existing components. For example, in the case of Beam potential evolution: would not be affected by the addition of the new Timer API, which can be used as follows: The intent of the framework is to be able to fit in a Java UI as well as in a web UI. It must be understood as colocalized and remote UI. The goal is to move as much as possible the logic to the UI side for UI-related actions. For example, validating a pattern, a size, and so on, should be done on the client side rather than on the server side. Being static encourages this practice. The other goal of being static in the API definition is to ensure that the model will not be mutated at runtime and that all the auditing and modeling can be done before, at the design phase. Being static also ensures that the development can be validated as much as possible through build tools. This does not replace the requirement to test the components but helps developers to maintain components with automated tools. Refer to this document. The components must be able to execute even if they have conflicting libraries. For that purpose, classloaders must be isolated. A component defines its dependencies based on a Maven format and is always bound to its own classloader. The definition payload is as flat as possible and strongly typed to ensure it can be manipulated by consumers. This way, consumers can add or remove fields with simple mapping rules, without any abstract tree handling. The execution (runtime) configuration is the concatenation of framework metadata (only the version) and a key/value model of the instance of the configuration based on the definition properties paths for the keys. It enables consumers to maintain and work with the keys/values according to their need. The framework not being responsible for any persistence, it is very important to make sure that consumers can handle it from end to end, with the ability to search for values (update a machine, update a port and so on) and keys (for example, a new encryption rule on key certificate). Talend Component Kit is a metamodel provider (to build forms) and a runtime execution platform. It takes a configuration instance and uses it volatilely to execute a component logic. This implies it cannot own the data nor define the contract it has for these two endpoints and must let the consumers handle the data lifecycle (creation, encryption, deletion, and so on). A new mime type called talend/stream is introduced to define a streaming format. It matches a JSON object per line: Icons (@Icon) are based on a fixed set. Custom icons can be used but their display cannot be guaranteed. Components can be used in any environment and require a consistent look that cannot be guaranteed outside of the UI itself. Defining keys only is the best way to communicate this information. Once you know exactly how you will deploy your component in the Studio, then you can use `@Icon(value = CUSTOM, custom = "…") to use a custom icon file.

singer-java module provide a Java API to write a custom singer. To import it, add the following dependency: Then you have access to the Singer class and its companion which provides the primitives to output properly the data: To build schema, keys, bookmarks, json and state which are all either jsonObject or JsonArray you can rely on JsonBuilderFactory which can be instantiated with this snippet: component-kitap is the name of the integration between singer-java and Talend Component Kit. It enables to run native Talend Component Kit components through a tap. The module relies on a proper setup of the component and classpath: Classpath is well setup - composed of component-kitap with a default SLF4J binding configured to log only errors on stderr. For convenience, you can use the all in one bundle provided by the module: org.talend.sdk.component:component-kitapp:${kit.version}:fatjar. Component is "deployed" - i.e. its maven repository is set provisionned with its dependencies. If you downloaded a component as a .car then you can run the car to do it. You can enforce the maven location through the system property talend.component.manager.m2.repository. Regarding SLF4J, the fatjar uses slf4j-standard of the framework which enables to set the logged level to error through a system property: -Dorg.talend.sdk.component.slf4j.StdLogger.level=err. To automatically register a component plugin/family you must add to the classpath a TALEND-INF/plugins.properties: The file only need to contain the registration of the plugin jar: Therefore the launch command can look like: Alternatively you can use org.talend.sdk.component.singer.kitap.Carpates main to launch the application, it differs from Kitap in the sense it takes a .car as option and avoids to pre-build the m2 repository: Here is an example with a real component: The config.json must respect this format: in some environment, such a JSON is not desirable, it is possible to put component_config attribute as a string containing the full json (escaped indeed) too.

This tutorial shows how to create components that consume a REST API. The component developed as example in this tutorial is an input component that provides a search functionality for Zendesk using its Search API. Lombok is used to avoid writing getter, setter and constructor methods. You can generate a project using the Talend Components Kit starter, as described in this tutorial. The input component relies on Zendesk Search API and requires an HTTP client to consume it. The Zendesk Search API takes the following parameters on the /api/v2/search.json endpoint. query : The search query. sort_by : The sorting type of the query result. Possible values are updated_at, created_at, priority, status, ticket_type, or relevance. It defaults to relevance. sort_order: The sorting order of the query result. Possible values are asc (for ascending) or desc (for descending). It defaults to desc. Talend Component Kit provides a built-in service to create an easy-to-use HTTP client in a declarative manner, using Java annotations. No additional implementation is needed for the interface, as it is provided by the component framework, according to what is defined above. This HTTP client can be injected into a mapper or a processor to perform HTTP requests. This example uses the basic authentication that supported by the API. The first step is to set up the configuration for the basic authentication. To be able to consume the Search API, the Zendesk instance URL, the username and the password are needed. The data store is now configured. It provides a basic authentication token. Once the data store is configured, you can define the dataset by configuring the search query. It is that query that defines the records processed by the input component. Your component is configured. You can now create the component logic. Mappers defined with this tutorial don’t implement the split part because HTTP calls are not split on many workers in this case. Once the component logic implemented, you can create the source in charge of performing the HTTP request to the search API and converting the result to jsonObject records. You now have created a simple Talend component that consumes a REST API. To learn how to test this component, refer to this tutorial.

This tutorial is the continuation of Talend Input component for Hazelcast tutorial. We will not walk through the project creation again, So please start from there before taking this one. This tutorial shows how to create a complete working output component for Hazelcast As seen before, in Hazelcast there is multiple data source type. You can find queues, topics, cache, maps… In this tutorials we will stick with the Map dataset and all what we will see here is applicable to the other types. Let’s assume that our Hazelcast output component will be responsible of inserting data into a distributed Map. For that, we will need to know which attribute from the incoming data is to be used as a key in the map. The value will be the hole record encoded into a json format. Bu that in mind, we can design our output configuration as: the same Datastore and Dataset from the input component and an additional configuration that will define the key attribute. Let’s create our Output configuration class. Let’s add the i18n properties of our configuration into the Messages.properties file The skeleton of the output component looks as follows: @Version annotation indicates the version of the component. It is used to migrate the component configuration if needed. @Icon annotation indicates the icon of the component. Here, the icon is a custom icon that needs to be bundled in the component JAR under resources/icons. @Processor annotation indicates that this class is the processor (output) and defines the name of the component. constructor of the processor is responsible for injecting the component configuration and services. Configuration parameters are annotated with @Option. The other parameters are considered as services and are injected by the component framework. Services can be local (class annotated with @Service) or provided by the component framework. The method annotated with @PostConstruct is executed once by instance and can be used for initialization. The method annotated with @PreDestroy is used to clean resources at the end of the execution of the output. Data is passed to the method annotated with @ElementListener. That method is responsible for handling the data output. You can define all the related logic in this method. If you need to bulk write the updates accordingly to groups, see Processors and batch processing. Now, we will need to add the display name of the Output to the i18n resources file Messages.properties Let’s implement all of those methods We will create the outpu contructor to inject the component configuration and some additional local and built in services. Built in services are services provided by TCK. Here we find: configuration is the component configuration class hazelcastService is the service that we have implemented in the input component tutorial. it will be responsible of creating a hazelcast client instance. jsonb is a built in service provided by tck to handle json object serialization and deserialization. We will use it to convert the incoming record to json format before inseting them into the map. Nothing to do in the post construct method. but we could for example initialize a hazle cast instance there. but we will do it in a lazy way on the first call in the @ElementListener method Shut down the Hazelcast client instance and thus free the Hazelcast map reference. We get the key attribute from the incoming record and then convert the hole record to a json string. Then we insert the key/value into the hazelcast map. Let’s create a unit test for our output component. The idea will be to create a job that will insert the data using this output implementation. So, let’s create out test class. Here we start by creating a hazelcast test instance, and we initialize the map. we also shutdown the instance after all the test are executed. Now let’s create our output test. Here we start preparing the emitter test component provided bt TCK that we use in our test job to generate random data for our output. Then, we use the output component to fill the hazelcast map. By the end we test that the map contains the exact amount of data inserted by the job. Run the test and check that it’s working. Congratulation you just finished your output component.

A Processor is a component that converts incoming data to a different model. A processor must have a method decorated with @ElementListener taking an incoming data and returning the processed data: Processors must be Serializable because they are distributed components. If you just need to access data on a map-based ruleset, you can use Record or jsonObject as parameter type. From there, Talend Component Kit wraps the data to allow you to access it as a map. The parameter type is not enforced. This means that if you know you will get a SuperCustomDto, then you can use it as parameter type. But for generic components that are reusable in any chain, it is highly encouraged to use Record until you have an evaluation language-based processor that has its own way to access components. For example: A processor also supports @BeforeGroup and @AfterGroup methods, which must not have any parameter and return void values. Any other result would be ignored. These methods are used by the runtime to mark a chunk of the data in a way which is estimated good for the execution flow size. Because the size is estimated, the size of a group can vary. It is even possible to have groups of size 1. It is recommended to batch records, for performance reasons: You can optimize the data batch processing by using the maxBatchSize parameter. This parameter is automatically implemented on the component when it is deployed to a Talend application. Only the logic needs to be implemented. You can however customize its value setting in your LocalConfiguration the property _maxBatchSize.value - for the family - or ${component simple class name}._maxBatchSize.value - for a particular component, otherwise its default will be 1000. If you replace value by active, you can also configure if this feature is enabled or not. This is useful when you don’t want to use it at all. Learn how to implement chunking/bulking in this document. In some cases, you may need to split the output of a processor in two or more connections. A common example is to have "main" and "reject" output connections where part of the incoming data are passed to a specific bucket and processed later. Talend Component Kit supports two types of output connections: Flow and Reject. Flow is the main and standard output connection. The Reject connection handles records rejected during the processing. A component can only have one reject connection, if any. Its name must be REJECT to be processed correctly in Talend applications. You can also define the different output connections of your component in the Starter. To define an output connection, you can use @Output as replacement of the returned value in the @ElementListener: Alternatively, you can pass a string that represents the new branch: Having multiple inputs is similar to having multiple outputs, except that an OutputEmitter wrapper is not needed: @Input takes the input name as parameter. If no name is set, it defaults to the "main (default)" input branch. It is recommended to use the default branch when possible and to avoid naming branches according to the component semantic. Batch processing refers to the way execution environments process batches of data handled by a component using a grouping mechanism. By default, the execution environment of a component automatically decides how to process groups of records and estimates an optimal group size depending on the system capacity. With this default behavior, the size of each group could sometimes be optimized for the system to handle the load more effectively or to match business requirements. For example, real-time or near real-time processing needs often imply processing smaller batches of data, but more often. On the other hand, a one-time processing without business contraints is more effectively handled with a batch size based on the system capacity. Final users of a component developed with the Talend Component Kit that integrates the batch processing logic described in this document can override this automatic size. To do that, a maxBatchSize option is available in the component settings and allows to set the maximum size of each group of data to process. A component processes batch data as follows: Case 1 - No maxBatchSize is specified in the component configuration. The execution environment estimates a group size of 4. Records are processed by groups of 4. Case 2 - The runtime estimates a group size of 4 but a maxBatchSize of 3 is specified in the component configuration. The system adapts the group size to 3. Records are processed by groups of 3. Batch processing relies on the sequence of three methods: @BeforeGroup, @ElementListener, @AfterGroup, that you can customize to your needs as a component Developer. The group size automatic estimation logic is automatically implemented when a component is deployed to a Talend application. Each group is processed as follows until there is no record left: The @BeforeGroup method resets a record buffer at the beginning of each group. The records of the group are assessed one by one and placed in the buffer as follows: The @ElementListener method tests if the buffer size is greater or equal to the defined maxBatchSize. If it is, the records are processed. If not, then the current record is buffered. The previous step happens for all records of the group. Then the @AfterGroup method tests if the buffer is empty. You can define the following logic in the processor configuration: You can also use the condensed syntax for this kind of processor: When writing tests for components, you can force the maxBatchSize parameter value by setting it with the following syntax: .$maxBatchSize=10. You can learn more about processors in this document. Defining a processor/output logic General component execution logic Implementing bulk processing Best practices For the case of output components (not emitting any data) using bulking you can pass the list of records to the after group method:

Processors and output components are the components in charge of reading, processing and transforming data in a Talend job, as well as passing it to its required destination. Before implementing the component logic and defining its layout and configurable fields, make sure you have specified its basic metadata, as detailed in this document. A Processor is a component that converts incoming data to a different model. A processor must have a method decorated with @ElementListener taking an incoming data and returning the processed data: Processors must be Serializable because they are distributed components. If you just need to access data on a map-based ruleset, you can use Record or jsonObject as parameter type. From there, Talend Component Kit wraps the data to allow you to access it as a map. The parameter type is not enforced. This means that if you know you will get a SuperCustomDto, then you can use it as parameter type. But for generic components that are reusable in any chain, it is highly encouraged to use Record until you have an evaluation language-based processor that has its own way to access components. For example: A processor also supports @BeforeGroup and @AfterGroup methods, which must not have any parameter and return void values. Any other result would be ignored. These methods are used by the runtime to mark a chunk of the data in a way which is estimated good for the execution flow size. Because the size is estimated, the size of a group can vary. It is even possible to have groups of size 1. It is recommended to batch records, for performance reasons: You can optimize the data batch processing by using the maxBatchSize parameter. This parameter is automatically implemented on the component when it is deployed to a Talend application. Only the logic needs to be implemented. You can however customize its value setting in your LocalConfiguration the property _maxBatchSize.value - for the family - or ${component simple class name}._maxBatchSize.value - for a particular component, otherwise its default will be 1000. If you replace value by active, you can also configure if this feature is enabled or not. This is useful when you don’t want to use it at all. Learn how to implement chunking/bulking in this document. In some cases, you may need to split the output of a processor in two or more connections. A common example is to have "main" and "reject" output connections where part of the incoming data are passed to a specific bucket and processed later. Talend Component Kit supports two types of output connections: Flow and Reject. Flow is the main and standard output connection. The Reject connection handles records rejected during the processing. A component can only have one reject connection, if any. Its name must be REJECT to be processed correctly in Talend applications. You can also define the different output connections of your component in the Starter. To define an output connection, you can use @Output as replacement of the returned value in the @ElementListener: Alternatively, you can pass a string that represents the new branch: Having multiple inputs is similar to having multiple outputs, except that an OutputEmitter wrapper is not needed: @Input takes the input name as parameter. If no name is set, it defaults to the "main (default)" input branch. It is recommended to use the default branch when possible and to avoid naming branches according to the component semantic. Batch processing refers to the way execution environments process batches of data handled by a component using a grouping mechanism. By default, the execution environment of a component automatically decides how to process groups of records and estimates an optimal group size depending on the system capacity. With this default behavior, the size of each group could sometimes be optimized for the system to handle the load more effectively or to match business requirements. For example, real-time or near real-time processing needs often imply processing smaller batches of data, but more often. On the other hand, a one-time processing without business contraints is more effectively handled with a batch size based on the system capacity. Final users of a component developed with the Talend Component Kit that integrates the batch processing logic described in this document can override this automatic size. To do that, a maxBatchSize option is available in the component settings and allows to set the maximum size of each group of data to process. A component processes batch data as follows: Case 1 - No maxBatchSize is specified in the component configuration. The execution environment estimates a group size of 4. Records are processed by groups of 4. Case 2 - The runtime estimates a group size of 4 but a maxBatchSize of 3 is specified in the component configuration. The system adapts the group size to 3. Records are processed by groups of 3. Batch processing relies on the sequence of three methods: @BeforeGroup, @ElementListener, @AfterGroup, that you can customize to your needs as a component Developer. The group size automatic estimation logic is automatically implemented when a component is deployed to a Talend application. Each group is processed as follows until there is no record left: The @BeforeGroup method resets a record buffer at the beginning of each group. The records of the group are assessed one by one and placed in the buffer as follows: The @ElementListener method tests if the buffer size is greater or equal to the defined maxBatchSize. If it is, the records are processed. If not, then the current record is buffered. The previous step happens for all records of the group. Then the @AfterGroup method tests if the buffer is empty. You can define the following logic in the processor configuration: You can also use the condensed syntax for this kind of processor: When writing tests for components, you can force the maxBatchSize parameter value by setting it with the following syntax: .$maxBatchSize=10. You can learn more about processors in this document. Defining a processor/output logic General component execution logic Implementing bulk processing Best practices For the case of output components (not emitting any data) using bulking you can pass the list of records to the after group method: An Output is a Processor that does not return any data. Conceptually, an output is a data listener. It matches the concept of processor. Being the last component of the execution chain or returning no data makes your processor an output component: Currently, Talend Component Kit does not allow you to define a Combiner. A combiner is the symmetric part of a partition mapper. It allows to aggregate results in a single partition.