Deploy Arkindex with docker-compose

This documentation is written for system administrators.

We'll use different terms for the components of our product:

• Platform server is the server that will run the Backend code responsible for the Rest API,
• Arkindex needs to run some specific asynchronous tasks that require direct access to the database: the local worker will execute these tasks,
• Some intensive Machine Learning tasks will be executed by Remote workers, using a proprietary software called Ponos. One instance of Ponos is called an Agent.

Requirements

• A bare metal server running Linux Ubuntu LTS (20.04 or 22.04) for the platform
  • If you plan to run Machine Learning processes, you'll need another server
• Docker installed on that server
• docker-compose
• A domain name for the platform server:
  • ideally this is a public domain name if your server is reachable on Internet (like arkindex.company.com),
    • or an internal domain name, provided by your company's system administrator.
• An SSL certificate for that domain name:
  • it can be provided by Let's Encrypt freely and automatically if your server is reachable on Internet
    • otherwise an internal certificate , provided by your company's system administrator.

Third-party services

You'll need to setup multiple companion services that support Arkindex. All these services are open source and freely available.

Required services:

• a load balancer, Traefik that will control traffic from your users towards the different services.
• a message broker for asynchronous service: redis
• a relational database for all data stored in Arkindex: postgres
  • the postgis extension is also required

Optional services:

• a remote storage server S3-compatible, MinIO
  • You can use AWS S3 or any other API-compatible provider instead
• a IIIF server for your images, cantaloupe
• a search engine to lookup your transcriptions: Apache Solr

Arkindex software

Teklia will provide you with several docker images (to load using docker load):

• the backend image, tagged registry.gitlab.teklia.com/arkindex/backend:X.Y.Z, must be present on your application server,
• the tasks image, registry.gitlab.teklia.com/arkindex/tasks:X.Y.Z, will be used to by the remote workers (file imports, thumbnails generation, ...).
• the ponos image, registry.gitlab.teklia.com/arkindex/ponos-agent:X.Y.Z will be used to actually run the asynchronous tasks across all your remote workers.

The backend image mentioned above will run in two containers on your application server:

1. for the API, this is really the heart of Arkindex,
2. for the local asynchronous tasks that can directly reach the database (sqlite export, element deletion, ...)

We recommend you to use our own CDN for the frontend files. Simply use the assets.teklia.com as source for static files in the backend configuration (you can look into our own example).

Docker-Compose

We documented a working example of docker-compose setup in a dedicated public repository. You can clone this repository and use it as a starting point for your own deployment.

Of course your setup may differ, you could use external services (databases, search engine, file storage, ...). You just need to run our own software through Docker, the other parts can be externalized.

Configuration

All the configuration options for the backend are detailed on this page.

A minimal configuration file is also available in the public repository.

Ponos

If your setup requires Machine Learning process, you'll need at least one Ponos Agent on a dedicated server.

The setup of this kind of server is easier, as it only requires to run the agent (from Docker image registry.gitlab.teklia.com/arkindex/ponos-agent) and configure it. The tasks will then be triggered by the agent automatically.

To begin the setup, you'll need 2 private keys: one for the backend, another for the agent. Each agent needs a dedicated key to authenticate itself.

To generate a valid private key:

openssl ecparam -name secp384r1 -genkey -noout > agent.key

A YAML configuration file is also required:

---
# Save as agent.yml
url: https://ark.localhost/
farm_id: XXXXX
seed: YYYYY
data_dir: /data
private_key: /etc/ponos/agent.key

The farm_id and seed information can be found in the Arkindex administration interface under the section Ponos > Farms.

You can then run the agent as:

docker run \
  --name=ponos \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v ./agent.yml:/etc/ponos/agent.yml:ro \
  -v ./agent.key:/etc/ponos/agent.key:ro \
  -v ponos_data:/data
  registry.gitlab.teklia.com/arkindex/ponos-agent:X.Y.Z

Please note that the agent requires a write access on the local Docker socket in order to create new containers that will run the tasks.

The ponos_data docker volume is not required, but will allow to retrieve debug logs outside of the agent container.