Talend Component Kit Developer Reference Guide :: Component Server Vault Proxy

Browse the API description using OpenAPI.

A Vault proxy dedicated to the component server allows to safely manage credential encryption when operating in a Cloud environment.

The Vault Proxy only concerns data marked as @Credential in the component configuration.

This Vault proxy:

receives an encrypted payload containing sensitive data from the remote engine, via HTTP.
decrypts the data
caches in memory the decrypted value of the data for performance reasons.
sends unencrypted data to the component server using HTTPS. An SSL certificate can be automatically generated and secures the data.

Vault proxy configuration

the configuration is read from system properties, environment variables, ….

talend.vault.cache.client.certificate.acceptAny: Default value: false. Should any certificate be accepted - only for dev purposes.
talend.vault.cache.client.executor.server.core: Default value: 64. Thread pool core size for Component Server client.
talend.vault.cache.client.executor.server.keepAlive: Default value: 60000. Thread keep alive (in ms) for Component Server client thread pool.
talend.vault.cache.client.executor.server.max: Default value: 256. Thread pool max size for Component Server client.
talend.vault.cache.client.executor.vault.core: Default value: 64. Thread pool core size for Vault client.
talend.vault.cache.client.executor.vault.keepAlive: Default value: 60000. Thread keep alive (in ms) for Vault client thread pool.
talend.vault.cache.client.executor.vault.max: Default value: 256. Thread pool max size for Vault client.
talend.vault.cache.client.providers: JAX-RS fully qualified name of the provides (message body readers/writers) for vault and component-server clients.
talend.vault.cache.client.server.authorization: The token to use to call component-server if any.
talend.vault.cache.client.server.certificate.keystore.location: Where the keystore to use to connect to Component Server is located.
talend.vault.cache.client.server.certificate.keystore.password: Default value: changeit. The keystore password for talend.vault.cache.client.server.certificate.keystore.location.
talend.vault.cache.client.server.certificate.keystore.type: The keystore type for talend.vault.cache.client.server.certificate.keystore.location.
talend.vault.cache.client.server.certificate.truststore.type: The truststore type for talend.vault.cache.client.server.certificate.keystore.location.
talend.vault.cache.client.server.hostname.accepted: Default value: localhost,127.0.0.1,0:0:0:0:0:0:0:1. Valid hostnames for the Component Server certificates (see java.net.ssl.HostnameVerifier).
talend.vault.cache.client.timeout.connect: Default value: 30000. HTTP connection timeout to vault server.
talend.vault.cache.client.timeout.read: Default value: 30000. HTTP read timeout to vault server.
talend.vault.cache.client.vault.certificate.keystore.location: Where the keystore to use to connect to vault is located.
talend.vault.cache.client.vault.certificate.keystore.password: Default value: changeit. The keystore password for talend.vault.cache.client.vault.certificate.keystore.location.
talend.vault.cache.client.vault.certificate.keystore.type: The keystore type for talend.vault.cache.client.vault.certificate.keystore.location.
talend.vault.cache.client.vault.certificate.truststore.type: The truststore type for talend.vault.cache.client.vault.certificate.keystore.location.
talend.vault.cache.client.vault.hostname.accepted: Default value: localhost,127.0.0.1,0:0:0:0:0:0:0:1. Valid hostnames for the Vault certificates (see java.net.ssl.HostnameVerifier).
talend.vault.cache.jcache.cache.expiry: Default value: 3600. JCache expiry for decrypted values (ms).
talend.vault.cache.jcache.cache.management: Default value: false. Should JCache MBeans be registered.
talend.vault.cache.jcache.cache.statistics: Default value: false. Should JCache statistics be enabled.
talend.vault.cache.jcache.manager.properties: Default value: `. JCache `CacheManager properties used to initialized the instance.
talend.vault.cache.jcache.manager.uri: Default value: geronimo://simple-jcache.properties. Configuration for JCache setup, default implementation is Geronimo Simple Cache.
talend.vault.cache.jcache.maxCacheSize: Default value: 100000. JCache max size per cache.
talend.vault.cache.jcache.refresh.period: Default value: 30000. How often (in ms) the Component Server should be checked to invalidate the caches on the component parameters (to identify credentials).
talend.vault.cache.security.allowedIps: Default value: localhost,127.0.0.1,0:0:0:0:0:0:0:1. The IP or hosts allowed to call that server on /api/* if no token is passed.
talend.vault.cache.security.hostname.sanitizer: Default value: none. Enable to sanitize the hostname before testing them. Default to none which is a noop. Supported values are docker (for <folder>_<service>_<number>.<folder>_<network> pattern) and weave (for <prefix>_dataset_<number>.<suffix> pattern).
talend.vault.cache.security.tokens: Default value: -. The tokens enabling a client to call this server without being in allowedIp whitelist.
talend.vault.cache.service.auth.cantDecipherStatusCode: Default value: 422. Status code sent when vault can’t decipher some values.
talend.vault.cache.service.auth.refreshDelayMargin: Default value: 600000. How often (in ms) to refresh the vault token.
talend.vault.cache.service.auth.refreshDelayOnFailure: Default value: 10000. How often (in ms) to refresh the vault token in case of an authentication failure.
talend.vault.cache.service.decipher.skip.regex: Default value: vault\:v[0-9]+\:.*. The regex to whitelist ciphered keys, others will be passthrough in the output without going to vault.
talend.vault.cache.talendComponentKit.url: Base URL to connect to Component Server
talend.vault.cache.vault.auth.endpoint: Default value: v1/auth/engines/login. The vault path to retrieve a token.
talend.vault.cache.vault.auth.roleId: Default value: -. The vault role identifier to use to log in (if token is not set). - means it is ignored.
talend.vault.cache.vault.auth.secretId: Default value: -. The vault secret identifier to use to log in (if token is not set). - means it is ignored.
talend.vault.cache.vault.auth.token: Default value: -. The vault token to use to log in (will make roleId and secretId ignored). - means it is ignored.
talend.vault.cache.vault.decrypt.endpoint: Default value: v1/tenants-keyrings/decrypt/{x-talend-tenant-id}. The vault path to decrypt values. You can use the variable {x-talend-tenant-id} to replace by x-talend-tenant-id header value.
talend.vault.cache.vault.url: Base URL to connect to Vault.

Configuration mechanism

The configuration uses Microprofile Config for most entries. It means it can be passed through system properties and environment variables (by replacing dots with underscores and making the keys uppercase).

To configure a Docker image rather than a standalone instance, Docker Config and secrets integration allow you to read the configuration from files. You can customize the configuration of these integrations through system properties, as described below in this document.

Also note that Docker integration provides a secure: support to encrypt values when required. it also works for system properties.

Docker Config

org.talend.sdk.component.docker.secret.config.DockerConfigConfigSource.base: configuration location. Defaults to /.
org.talend.sdk.component.docker.secret.config.DockerConfigConfigSource.prefixes: file name prefixes to take into account. Defaults to nothing, which means that all prefixes but the common UNIx exclusions are taken into account.

For example, creating a /talend.foo.bar file makes its content available under the talend.foo.bar key.

Docker Secrets

org.talend.sdk.component.docker.secret.config.DockerSecretConfigSource: secrets location. Defaults to /run/secrets.

For example, creating a /run/secrets/talend.foo.bar file makes its content available under the talend.foo.bar key.

Secure values support

A security mechanism allows to use a ciphered value instead of a clear value for passing sensitive data (password, secret, token, etc.).

It relies on org.talend.sdk.component.configuration.converter.secured.ConfigurationMain, which allows you to:

create a master_key file
encrypt a value
decrypt a value (for testing purposes)

A master password is stored and obfuscated in a master_key file. The algorithm AES/CBC/PKCS5Padding ciphers the value and encodes the result in base64. This mechanism allows to manage the value as plain text and easily pass it through all potential ways you can set the configuration.

The secure values support is managed through the org.talend.sdk.component:secured-string-converter artifact. You can find it for released versions on Central.

The command examples in the sections below assume you have set the alias in your profile:

alias tssc="java -cp secured-string-converter.jar org.talend.sdk.component.configuration.converter.secured.ConfigurationMain"

If it is not the case, replace tssc by the full command.

Generating the master key

To generate the master key, use the provided configuration in the org.talend.sdk.component:secured-string-converter artifact:

tssc --master-key /path/to/my_master_key [secret-value]

Set the talend.component.server.configuration.master_key.location system property on the server to ensure it uses this master key. Make sure that only the applications needing this key can read it.

Encrypting a value

To encrypt a value, use the provided configuration in the org.talend.sdk.component:secured-string-converter artifact:

tssc --encrypt /path/to/my_master_key my_credential_to_encrypt

Decrypting a value

To decrypt a value, use the provided configuration in the org.talend.sdk.component:secured-string-converter artifact:

tssc --decrypt /path/to/my_master_key secure:my_credential_to_encrypt

Adding the Vault Proxy to your Docker Compose

The YAML below is the recommended configuration to enable the Vault Proxy and Component Server to communicate over HTTPS.

Docker Compose

version: '3.2'

services:
  component-server: (1)
    healthcheck:
      timeout: 3s
      interval: 3s
      retries: 3
      test: curl --fail http://localhost:8080/api/v1/environment
    image: tacokit/component-server:${COMPONENT_SERVER_IMAGE_VERSION:-1.1.6_20190208104207}
    environment:
    - _JAVA_OPTIONS=
        -Dtalend.component.server.filter.secured.tokens=vault-proxy (2)
        -Dtalend.component.server.ssl.active=true (3)
        -Dtalend.component.server.ssl.keystore.location=/opt/talend/configuration/https.p12 (3)
        -Dtalend.component.server.ssl.keystore.type=PKCS12 (3)
        -Dtalend.component.server.component.registry=/opt/talend/connectors/component-registry.properties (4)
        -Dtalend.component.server.maven.repository=/opt/talend/connectors (4)
    volumes:
    - connectors:/opt/talend/connectors:ro  (4)
    - vault-proxy-configuration:/opt/talend/configuration (3)


  component-server-vault-proxy: (5)
    healthcheck:
      timeout: 3s
      interval: 3s
      retries: 3
      test: curl --fail http://localhost:8080/api/v1/proxy/environment
    image: tacokit/component-server-vault-proxy:${COMPONENT_SERVER_VAULT_PROXY_IMAGE_VERSION:-1.1.6_20190208104221}
    environment:
    - _JAVA_OPTIONS=
        -Dtalend.vault.cache.client.server.certificate.keystore.location=/opt/talend/configuration/https.p12 (6)
        -Dtalend.vault.cache.client.server.certificate.keystore.type=PKCS12 (6)
        -Dtalend.vault.cache.client.server.hostname.accepted=component-server (6)
        -Dtalend.vault.cache.client.server.authorization=vault-proxy (7)
        -Dtalend.vault.cache.talendComponentKit.url=https://component-server:8080/api/v1 (7)
        -Dtalend.vault.cache.vault.url=http://vault:8200 (8)
        -Dtalend.vault.cache.vault.auth.roleId=myrole (8)
        -Dtalend.vault.cache.vault.decrypt.endpoint=v1/something/decrypt/00000001 (8)
        -Dtalend.vault.cache.security.allowedIps=${COMPONENT_SERVER_VAULT_PROXY_CLIENT_IP:-127.0.0.1} (9)
    ports:
    - 9090:8080/tcp
    links: (10)
    - "component-server:component-server"
    # - "vault:vault"
    volumes:
    - vault-proxy-configuration:/opt/talend/configuration:ro

  connectors: (11)
    image: registry.datapwn.com/talend/connectors:${CONNECTORS_IMAGE_VERSION:-1.0.0_master_20190208091312}
    environment:
    - CONNECTORS_SETUP_OPTS=setup --wait-for-end
    volumes:
    - connectors:/opt/talend/connectors:ro

volumes: (12)
  connectors:
  vault-proxy-configuration:

1	The standard Component Server entry.
2	Ensures only a client with a particular token can call the server. It is similar to a shared secret and only allows to call the server in "remote" mode, since only the local mode is enabled by default.
3	Activates and configures the auto generation of a X.509 certificate which is used for the HTTPS connector on the server.
4	Binds the components to deploy into the server.
5	Definition of the Vault proxy service which handles the mediation between Vault and the Component Server.
6	Since both servers are colocalized, the generated certificate is inherited from the Component Server, which allows to create the client that connects to it.
7	Configuration of the base URL to connect to the server - see <10>.
8	Configuration of the vault connection and security information.
9	Ensuring that connecting from `$COMPONENT_SERVER_VAULT_PROXY_CLIENT_IP` to vault proxy is possible. Any other IP will be rejected.
10	Linking both services so they can communicate. It allows not to expose the Component Server port outside of its own container (no `ports` mapping in the Component Server service definition). Note that if your vault is a service, you can link it here as well.
11	Common component image service definition.
12	Volumes used by the services. The `connectors` has not changed but `vault-proxy-configuration` was added for the automatic HTTPS configuration.

This setup enables the Vault Proxy and Component Server to communicate. You can now use the Vault Proxy as if it was the Component Server, by using localhost:9090 (or any other host matching your deployment) instead of the Component Server directly.

Linking the Vault Proxy to the Component Server through HTTPS

When the Vault Proxy is enabled, ensure you configure HTTPS on the Component Server using the following parameters:

talend.component.server.ssl.active: true or false. Indicates if the SSL protocol is enabled.
talend.component.server.ssl.password: Keystore password.
talend.component.server.ssl.keystore.location: Path to Keystore.
talend.component.server.ssl.keystore.alias: Private key/certificate alias.
talend.component.server.ssl.keystore.type: Keystore type.
talend.component.server.ssl.keystore.generation.force: true or false.
talend.component.server.ssl.keystore.generation.command: Specifies if a custom command is to be used to generate the certificate.
talend.component.server.ssl.keypair.algorithm: Encryption algorithm. RSA by default.
talend.component.server.ssl.certificate.dname: Distinguished name.
talend.component.server.ssl.keypair.size: Size of the key. 2048 by default.
talend.component.server.ssl.port: SSL port to use.