Talend Component Kit Developer Reference Guide :: Component Server Vault Proxy

A Vault proxy dedicated to the component server allows to safely manage credential encryption when operating in a Cloud environment.

The Vault Proxy only concerns data marked as @Credential in the component configuration.

This Vault proxy:

receives an encrypted payload containing sensitive data from the remote engine, via HTTP.
decrypts the data
caches in memory the decrypted value of the data for performance reasons.
sends unencrypted data to the component server using HTTPS. An SSL certificate can be automatically generated and secures the data.

Vault proxy configuration

The configuration mechanism is the same as for the component server (microprofile config).

the configuration is read from system properties, environment variables, ….

talend.vault.cache.client.certificate.acceptAny: Default value: false. Should any certificate be accepted - only for dev purposes.
talend.vault.cache.client.executor.server.core: Default value: 64. Thread pool core size for Component Server client.
talend.vault.cache.client.executor.server.keepAlive: Default value: 60000. Thread keep alive (in ms) for Component Server client thread pool.
talend.vault.cache.client.executor.server.max: Default value: 256. Thread pool max size for Component Server client.
talend.vault.cache.client.executor.vault.core: Default value: 64. Thread pool core size for Vault client.
talend.vault.cache.client.executor.vault.keepAlive: Default value: 60000. Thread keep alive (in ms) for Vault client thread pool.
talend.vault.cache.client.executor.vault.max: Default value: 256. Thread pool max size for Vault client.
talend.vault.cache.client.providers: JAX-RS fully qualified name of the provides (message body readers/writers) for vault and component-server clients.
talend.vault.cache.client.server.authorization: The token to use to call component-server if any.
talend.vault.cache.client.server.certificate.keystore.location: Where the keystore to use to connect to Component Server is located.
talend.vault.cache.client.server.certificate.keystore.password: Default value: changeit. The keystore password for talend.vault.cache.client.server.certificate.keystore.location.
talend.vault.cache.client.server.certificate.keystore.type: The keystore type for talend.vault.cache.client.server.certificate.keystore.location.
talend.vault.cache.client.server.certificate.truststore.type: The truststore type for talend.vault.cache.client.server.certificate.keystore.location.
talend.vault.cache.client.server.hostname.accepted: Default value: localhost,127.0.0.1,0:0:0:0:0:0:0:1. Valid hostnames for the Component Server certificates (see java.net.ssl.HostnameVerifier).
talend.vault.cache.client.timeout.connect: Default value: 30000. HTTP connection timeout to vault server.
talend.vault.cache.client.timeout.read: Default value: 30000. HTTP read timeout to vault server.
talend.vault.cache.client.vault.certificate.keystore.location: Where the keystore to use to connect to vault is located.
talend.vault.cache.client.vault.certificate.keystore.password: Default value: changeit. The keystore password for talend.vault.cache.client.vault.certificate.keystore.location.
talend.vault.cache.client.vault.certificate.keystore.type: The keystore type for talend.vault.cache.client.vault.certificate.keystore.location.
talend.vault.cache.client.vault.certificate.truststore.type: The truststore type for talend.vault.cache.client.vault.certificate.keystore.location.
talend.vault.cache.client.vault.hostname.accepted: Default value: localhost,127.0.0.1,0:0:0:0:0:0:0:1. Valid hostnames for the Vault certificates (see java.net.ssl.HostnameVerifier).
talend.vault.cache.jcache.cache.expiry: Default value: 3600. JCache expiry for decrypted values (ms).
talend.vault.cache.jcache.cache.management: Default value: false. Should JCache MBeans be registered.
talend.vault.cache.jcache.cache.statistics: Default value: false. Should JCache statistics be enabled.
talend.vault.cache.jcache.manager.properties: Default value: `. JCache `CacheManager properties used to initialized the instance.
talend.vault.cache.jcache.manager.uri: Default value: geronimo://simple-jcache.properties. Configuration for JCache setup, default implementation is Geronimo Simple Cache.
talend.vault.cache.jcache.maxCacheSize: Default value: 100000. JCache max size per cache.
talend.vault.cache.jcache.refresh.period: Default value: 30000. How often (in ms) the Component Server should be checked to invalidate the caches on the component parameters (to identify credentials).
talend.vault.cache.security.allowedIps: Default value: localhost,127.0.0.1,0:0:0:0:0:0:0:1. The IP or hosts allowed to call that server on /api/* if no token is passed.
talend.vault.cache.security.tokens: Default value: -. The tokens enabling a client to call this server without being in allowedIp whitelist.
talend.vault.cache.service.auth.refreshDelayMargin: Default value: 600000. How often (in ms) to refresh the vault token.
talend.vault.cache.service.auth.refreshDelayOnFailure: Default value: 10000. How often (in ms) to refresh the vault token in case of an authentication failure.
talend.vault.cache.talendComponentKit.url: Base URL to connect to Component Server
talend.vault.cache.vault.auth.endpoint: Default value: v1/auth/approle/login. The vault path to retrieve a token.
talend.vault.cache.vault.auth.roleId: The vault role identifier to use to log in.
talend.vault.cache.vault.auth.secretId: The vault secret identifier to use to log in.
talend.vault.cache.vault.decrypt.endpoint: The vault path to decrypt values.
talend.vault.cache.vault.url: Base URL to connect to Vault.

Adding the Vault Proxy to your Docker Compose

The YAML below is the recommended configuration to enable the Vault Proxy and Component Server to communicate over HTTPS.

Docker Compose

version: '3.2'

services:
  component-server: (1)
    healthcheck:
      timeout: 3s
      interval: 3s
      retries: 3
      test: curl --fail http://localhost:8080/api/v1/environment
    image: tacokit/component-server:${COMPONENT_SERVER_IMAGE_VERSION:-1.1.6_20190208104207}
    environment:
    - _JAVA_OPTIONS=
        -Djava.library.path=/opt/talend/component-kit/work/sigar/sigar:/usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64/server:/usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64:/usr/lib/jvm/java-1.8-openjdk/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
        -Dtalend.component.server.filter.secured.tokens=vault-proxy (2)
        -Dtalend.component.server.ssl.active=true (3)
        -Dtalend.component.server.ssl.keystore.location=/opt/talend/configuration/https.p12 (3)
        -Dtalend.component.server.ssl.keystore.type=PKCS12 (3)
        -Dtalend.component.server.component.registry=/opt/talend/connectors/component-registry.properties (4)
        -Dtalend.component.server.maven.repository=/opt/talend/connectors (4)
    volumes:
    - connectors:/opt/talend/connectors:ro  (4)
    - vault-proxy-configuration:/opt/talend/configuration (3)


  component-server-vault-proxy: (5)
    healthcheck:
      timeout: 3s
      interval: 3s
      retries: 3
      test: curl --fail http://localhost:8080/api/v1/proxy/environment
    image: tacokit/component-server-vault-proxy:${COMPONENT_SERVER_VAULT_PROXY_IMAGE_VERSION:-1.1.6_20190208104221}
    environment:
    - _JAVA_OPTIONS=
        -Djava.library.path=/opt/talend/component-kit/work/sigar/sigar:/usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64/server:/usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64:/usr/lib/jvm/java-1.8-openjdk/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
        -Dtalend.vault.cache.client.server.certificate.keystore.location=/opt/talend/configuration/https.p12 (6)
        -Dtalend.vault.cache.client.server.certificate.keystore.type=PKCS12 (6)
        -Dtalend.vault.cache.client.server.hostname.accepted=component-server (6)
        -Dtalend.vault.cache.client.server.authorization=vault-proxy (7)
        -Dtalend.vault.cache.talendComponentKit.url=https://component-server:8080/api/v1 (7)
        -Dtalend.vault.cache.vault.url=http://vault:8200 (8)
        -Dtalend.vault.cache.vault.auth.roleId=myrole (8)
        -Dtalend.vault.cache.vault.decrypt.endpoint=v1/something/decrypt/00000001 (8)
        -Dtalend.vault.cache.security.allowedIps=${COMPONENT_SERVER_VAULT_PROXY_CLIENT_IP:-127.0.0.1} (9)
    ports:
    - 9090:8080/tcp
    links: (10)
    - "component-server:component-server"
    # - "vault:vault"
    volumes:
    - vault-proxy-configuration:/opt/talend/configuration:ro

  connectors: (11)
    image: registry.datapwn.com/talend/connectors:${CONNECTORS_IMAGE_VERSION:-1.0.0_master_20190208091312}
    environment:
    - CONNECTORS_SETUP_OPTS=setup --wait-for-end
    volumes:
    - connectors:/opt/talend/connectors:ro

volumes: (12)
  connectors:
  vault-proxy-configuration:

1	The standard Component Server entry
2	Ensure only a client with a particular token can call the server (it is a kind of shared secret and only enables to call the server in "remote" mode since only local mode is enabled by default)
3	Activates and configure the auto generation of a X.509 certificate which will be used for HTTPS connector on the server
4	Bind as usual the components to deploy into the server
5	This is where we will define the vault proxy service which will do the mediation between Vault and Component Server.
6	Since both server are colocalized we inherit from the Component Server certificate to create the client which will connect on it
7	We configure the base url to connect to the server - see <10>
8	We configure vault connection and security informations
9	We ensure we can connect from `$COMPONENT_SERVER_VAULT_PROXY_CLIENT_IP` to vault proxy (other IP will be rejected)
10	We link both services to let them communicate between them, it allows to not expose the Component Server port outside its own container (no `ports` mapping in Component Server service definition). Note that if your vault is a service as well you can link it here as well.
11	The common component image service definition (as previously)
12	The volumes used by the services, the `connectors` one is the same as before but we added `vault-proxy-configuration` for the automatic HTTPS configuration.

This setup enables the Vault Proxy and Component Server to communicate, now you can use the Vault Proxy as if it was the Component Server, just use localhost:9090 - or any other host matching your deployment - instead of the Component Server directly.

Linking the Vault Proxy to the Component Server through HTTPS

When the Vault Proxy is enabled, ensure you configure HTTPS on the Component Server using the following parameters:

talend.component.server.ssl.active: true or false. Indicates if the SSL protocol is enabled.
talend.component.server.ssl.password: Keystore password.
talend.component.server.ssl.keystore.location: Path to Keystore.
talend.component.server.ssl.keystore.alias: Private key/certificate alias.
talend.component.server.ssl.keystore.type: Keystore type.
talend.component.server.ssl.keystore.generation.force: true or false.
talend.component.server.ssl.keystore.generation.command: Specifies if a custom command is to be used to generate the certificate.
talend.component.server.ssl.keypair.algorithm: Encryption algorithm. RSA by default.
talend.component.server.ssl.certificate.dname: Distinguished name.
talend.component.server.ssl.keypair.size: Size of the key. 2048 by default.
talend.component.server.ssl.port: SSL port to use.