.. _setup:

Setup
=====

In this chapter the setup of a new VS stack is detailed. Before this step can
be done, the configuration and environment files need to be present. These
files can be added manually or be created as described in the
:ref:`initialization` step.


Docker
------

In order to deploy the Docker Swarm stack to the target machine, Docker and
its facilities need to be installed. This step depends on the systems
architecture. On a Debian based system this may look like this:

.. code-block:: bash

    sudo apt-get install \
        apt-transport-https \
        ca-certificates \
        curl \
        gnupg-agent \
        software-properties-common

    curl -fsSL https://download.docker.com/linux/debian/gpg | sudo apt-key add -

    # add the apt repository
    sudo add-apt-repository \
        "deb [arch=amd64] https://download.docker.com/linux/debian \
        $(lsb_release -cs) \
        stable"

    # fetch the package index and install Docker
    sudo apt-get update
    sudo apt-get install docker-ce docker-ce-cli containerd.io

.. ## TODO: RHEL?

Docker Swarm
------------

Now that Docker is installed, the machine can either create a new swarm or join
an existing one.

To create a new Swarm, the following command is used:

.. code-block:: bash

    docker swarm init


Alternatively, an existing Swarm can be joined. The easiest way to do this, is
to obtain a ``join-token``. On an existing Swarm manager (where a Swarm was
initialized or already joined as manager) run this command:


.. code-block:: bash

    docker swarm join-token worker


This prints out a command that can be run on a machine to join the swarm:

.. code-block:: bash

    docker swarm join --token <obtained token>

It is possible to dedicate certain workers for example to contribute to ingestion exclusively, while others can take care only for rendering. This setup has benefits, when a mixed setup of nodes with different parameters is available.

In order to set a node for example as `external`, to contribute in rendering only, one can simply run: 

.. code-block:: bash

    docker node update --label-add type=external <node-id>
    
Additionally, it is necessary to modify `placement` parameter in the docker compose file.

.. code-block:: yaml

  renderer:
    deploy:
      placement:
        constraints:
          - node.labels.type == external

Additional information for swarm management can be obtained in the official
`documentation of the project
<https://docs.docker.com/engine/reference/commandline/swarm/>`_.


Docker Image Retrieval
----------------------

Before the Docker images can be used, they have to be retrieved first.
Currently, all images used in VS that are not off-the-shelf are hosted on the public ``registry.gitlab.eox.at`` registry. For production deployment, tagged releases in format ``release-<major.minor.patch>`` should be used.  

To pull the relevant images:

.. code-block:: bash

    docker pull registry.gitlab.eox.at/esa/prism/vs/pvs_core:release-x.x.x
    docker pull registry.gitlab.eox.at/esa/prism/vs/pvs_cache:release-x.x.x
    docker pull registry.gitlab.eox.at/esa/prism/vs/pvs_preprocessor:release-x.x.x
    docker pull registry.gitlab.eox.at/esa/prism/vs/pvs_client:release-x.x.x
    docker pull registry.gitlab.eox.at/esa/prism/vs/fluentd:release-x.x.x
    docker pull registry.gitlab.eox.at/esa/prism/vs/pvs_ingestor:release-x.x.x
    docker pull registry.gitlab.eox.at/esa/prism/vs/pvs_sftp:release-x.x.x
    docker pull registry.gitlab.eox.at/esa/prism/vs/pvs_shibauth:release-x.x.x




Logging
-------

For production, the docker images in the compose files use the default logging
driver. Therefore we configure the default logging driver for the docker daemon to
be fluent by createing the file ``/etc/docker/daemon.json`` with the following content:

.. code-block:: json

    {
        "log-driver": "fluentd"
    }

For development, we don't want to redirect all of the docker logging output,
so the respective compose files for dev configure the logging driver for each
container.


Stack Deployment
----------------

Before the stack deployment step, some environment variables and configurations -which
are considered sensitive- should be created beforehand, this can done following 
the ``Sensitive variables`` steps that are included in the next :ref:`configuration` section.

Now that a Docker Swarm is established and docker secrets and configs are created,
it is time to deploy the VS as a stack. This is done using the created Docker Compose
configuration files. In order to enhance the re-usability, these files are split into multiple parts to be used
for both development and final service deployment.

For a development deployment one would do (replace ``name`` with the actual
service identifier:

.. code-block:: bash

    docker stack deploy -c docker-compose.<name>.yml -c docker-compose.<name>.dev.yml <stack-name>


This command actually performs a variety of tasks. First off, it obtains any
missing images, such as the image for the reverse proxy, the database, or the
redis key-value store.

When all relevant images are pulled from their respective repository the
services of the stack are initialized. In the default setting, each service is
represented by a single container of its respective service type. When starting
for the first time, the startup procedure takes some time, as everything needs
to be initialized. This includes the creation of the database, user,
required tables, and the Django instance.

That process can be supervised using the ``docker service ls`` command, which
lists all available services and their respective status.

Continue to the next section :ref:`configuration` to read how a running VS
stack can be configured.