Running Component Kit components on a Remote Engine

The Remote Engine is a key piece for executing components with Talend Pipeline Designer. To customize it for testing purposes, a helper tool is available.

The following content and the tool it describes are not officially supported as a Talend Cloud feature and are provided "as it" without any guarantee.

Remote Engine Customizer

The remote-engine-customizer is a simple tool that reads the Remote Engine docker-compose.yml file, extracts the connectors image and rebuilds it adding configured connectors to the image. By default, it replaces the original image with the newly built one and saves the old file in the .remote_engine_customizer/backup/ directory of the Remote Engine.

Using the tool

The customizer is provided as a docker container.

You can run the tool using the register-component-archive command:

docker run tacokit/remote-engine-customizer:${version} \
    register-component-archive [options]

You can get more details about the tool with the following command:

docker run tacokit/remote-engine-customizer:${version} help
You can find out all the available versions at hub.docker.com/r/tacokit/remote-engine-customizer/tags.

The command accepts the following parameters:

Usage: register-component-archive [options]

Options:
  --base-image=<String>                           Base connector image to enrich with components.
                                                  (default: auto)
  --cache-dir=<String>                            Where to cache the image layers - useful when the command is launched multiple times.
                                                  (default: ${remote.engine.dir}/.remote_engine_customizer/cache)
  --component-archive=<String[]>                  Component Archive (.car) to register/add in the Remote Engine.
  --docker-configuration-environment=<String>     Docker daemon environment variables (as properties) if custom.
                                                  (default: )
  --docker-configuration-path=<String>            Docker daemon executable path if custom.
  --from-image-type=<ImageType>                   Type of source image, can be useful if you don't use the default version generation.
                                                  (default: AUTO)
                                                  (enum: AUTO, DOCKER, REGISTRY)
  --registry-configuration-baseUrl=<String>       Registry base URL when image type is REGISTRY.
  --registry-configuration-password=<String>      Registry password when image type is REGISTRY.
  --registry-configuration-username=<String>      Registry username when image type is REGISTRY.
  --remote-engine-dir=<String>                    Where the remote engine folder is, it should host a docker-compose.yml file.
  --target-image=<String>                         Name of the target image, if not set it will be generated.
                                                  (default: auto)
  --to-image-type=<ImageType>                     Type of target image, it is generally just a local DOCKER image.
                                                  (default: DOCKER)
                                                  (enum: AUTO, DOCKER, REGISTRY)
  --no-update-original-docker-compose             Should the Remote Engine docker-compose.yml be updated once the new image is built.
  --work-dir=<String>                             Where to create temporary files needed for the build of the new image.
                                                  (default: ${java.io.tmpdir}/remote-engine-customizer)

When running using the docker container, you must mount several files/folders. For example:

docker run \
    -v /opt/talend/pipeline-remote-engine/:/opt/talend/pipeline-remote-engine/ \ (1)
    -v /opt/talend/downloads_exchange/:/opt/talend/downloads_exchange/ \ (2)
    -v /var/run/docker.sock:/var/run/docker.sock \ (3)
    tacokit/remote-engine-customizer:${version} \
    register-component-archive \
    --remote-engine-dir=/opt/talend/pipeline-remote-engine/
    --component-archive=/opt/talend/downloads_exchange/mailio-1.0.0.car
1 Mount the Remote Engine folder to let the docker image read and modify it.
2 Mount the folder hosting your local component if you want the Docker image to be able to use it.
3 Mount the docker daemon to let the program create a new docker image on the host.
Once you have built the new image, you must restart the Remote Engine. You can verify that the new image is available by running docker run --entrypoint sh -it [newimage] before.
Scroll to top