Setup

In this chapter the setup of a new VS stack is detailed. Before this step can be done, the configuration and environment files need to be present. These files can be added manually or be created as described in the Initialization step.

Docker

In order to deploy the Docker Swarm stack to the target machine, Docker and its facilities need to be installed. This step depends on the systems architecture. On a Debian based system this may look like this:

sudo apt-get install \
    apt-transport-https \
    ca-certificates \
    curl \
    gnupg-agent \
    software-properties-common

curl -fsSL https://download.docker.com/linux/debian/gpg | sudo apt-key add -

# add the apt repository
sudo add-apt-repository \
    "deb [arch=amd64] https://download.docker.com/linux/debian \
    $(lsb_release -cs) \
    stable"

# fetch the package index and install Docker
sudo apt-get update
sudo apt-get install docker-ce docker-ce-cli containerd.io

Docker Swarm

Now that Docker is installed, the machine can either create a new swarm or join an existing one.

To create a new Swarm, the following command is used:

docker swarm init

Alternatively, an existing Swarm can be joined. The easiest way to do this, is to obtain a join-token. On an existing Swarm manager (where a Swarm was initialized or already joined as manager) run this command:

docker swarm join-token worker

This prints out a command that can be run on a machine to join the swarm:

docker swarm join --token <obtained token>

It is possible to dedicate certain workers for example to contribute to ingestion exclusively, while others can take care only for rendering. This setup has benefits, when a mixed setup of nodes with different parameters is available.

In order to set a node for example as external, to contribute in rendering only, one can simply run:

docker node update --label-add type=external <node-id>

Additionally, it is necessary to modify placement parameter in the docker compose file.

renderer:
  deploy:
    placement:
      constraints:
        - node.labels.type == external

Additional information for swarm management can be obtained in the official documentation of the project.

Docker Image Retrieval

Before the Docker images can be used, they have to be retrieved first. Currently, all images used in VS that are not off-the-shelf are hosted on the public registry.gitlab.eox.at registry. For production deployment, tagged releases in format release-<major.minor.patch> should be used.

To pull the relevant images:

docker pull registry.gitlab.eox.at/esa/prism/vs/pvs_core:release-x.x.x
docker pull registry.gitlab.eox.at/esa/prism/vs/pvs_cache:release-x.x.x
docker pull registry.gitlab.eox.at/esa/prism/vs/pvs_preprocessor:release-x.x.x
docker pull registry.gitlab.eox.at/esa/prism/vs/pvs_client:release-x.x.x
docker pull registry.gitlab.eox.at/esa/prism/vs/fluentd:release-x.x.x
docker pull registry.gitlab.eox.at/esa/prism/vs/pvs_ingestor:release-x.x.x
docker pull registry.gitlab.eox.at/esa/prism/vs/pvs_sftp:release-x.x.x
docker pull registry.gitlab.eox.at/esa/prism/vs/pvs_shibauth:release-x.x.x

Logging

For production, the docker images in the compose files use the default logging driver. Therefore we configure the default logging driver for the docker daemon to be fluent by createing the file /etc/docker/daemon.json with the following content:

{
    "log-driver": "fluentd"
}

For development, we don’t want to redirect all of the docker logging output, so the respective compose files for dev configure the logging driver for each container.

Stack Deployment

Before the stack deployment step, some environment variables and configurations -which are considered sensitive- should be created beforehand, this can done following the Sensitive variables steps that are included in the next Configuration section.

Now that a Docker Swarm is established and docker secrets and configs are created, it is time to deploy the VS as a stack. This is done using the created Docker Compose configuration files. In order to enhance the re-usability, these files are split into multiple parts to be used for both development and final service deployment.

For a development deployment one would do (replace name with the actual service identifier:

docker stack deploy -c docker-compose.<name>.yml -c docker-compose.<name>.dev.yml <stack-name>

This command actually performs a variety of tasks. First off, it obtains any missing images, such as the image for the reverse proxy, the database, or the redis key-value store.

When all relevant images are pulled from their respective repository the services of the stack are initialized. In the default setting, each service is represented by a single container of its respective service type. When starting for the first time, the startup procedure takes some time, as everything needs to be initialized. This includes the creation of the database, user, required tables, and the Django instance.

That process can be supervised using the docker service ls command, which lists all available services and their respective status.

Continue to the next section Configuration to read how a running VS stack can be configured.