Skip to content

Advanced setup and CI

Arkindex backend

Dockerized stack

It is possible to run the whole Arkindex stack through Docker containers. This is useful to quickly test the platform.

The following command builds all the required Docker images (backend & frontend) and runs them as Docker containers:

make stack

You can access the platform at the URL https://ark.localhost.

Local image server

Arkindex splits up image URLs into image servers and image paths. For example, a IIIF server at http://iiif.irht.cnrs.fr/iiif/ and an image at /Paris/JJ042/1.jpg would be represented as an ImageServer instance holding one Image. Since Arkindex has a local IIIF server for image uploads and thumbnails, a special instance of ImageServer is required to point to this local server. In local development, this server should be available at https://ark.localhost/iiif. You will therefore need to create an ImageServer via the Django admin or the Django shell with this URL. To set the local server ID, you can add a custom setting in arkindex/config.yml:

local_imageserver_id: 42

Here is how to quickly create the ImageServer using the shell:

$ arkindex shell
>>> from arkindex.images.models import ImageServer
>>> ImageServer.objects.create(id=42, display_name='local', url='https://ark.localhost/iiif')

Note that this local server will only work inside Docker.

Common operations

Makefile

At the root of the repository is a Makefile that provides commands for common operations:

  • make or make all: Clean and build;
  • make base: Create and push the arkindex-base Docker image that is used to build the arkindex-app image;
  • make clean: Cleanup the Python package build and cache files;
  • make clean-docker: Deletes all running containers to avoid naming and network ports conflicts;
  • make build: Build the arkindex Python package and recreate the arkindex-app:latest without pushing to the GitLab container registry;
  • make test-fixtures: Create the unit tests fixtures on a temporary PostgreSQL database and save them to the data.json file used by most Django unit tests.

Django commands

Aside from the usual Django commands, some custom commands are available via arkindex:

  • build_fixtures: Create a set of database elements designed for use by unit tests in a fixture (see make test-fixtures).
  • delete_corpus: Delete a big corpus using an RQ task.
  • reindex: Reindex elements into Solr.
  • move_lines_to_parents: Moves element children to their geographical parents.

See arkindex <command> --help to view more details about a specific command.

Code validation

Once your code appears to be working on a local server, a few checks have to be performed:

  • Migrations: Ensure that all migrations have been created by running the arkindex makemigrations command.
  • Unit tests: Run arkindex test to perform unit tests.
  • Use arkindex test module_name to perform tests on a single module, if you wish to spend less time waiting for all tests to complete.

Known issues

Worker run parents field issue in unit tests

After re-generating the unit test fixtures with make test-fixtures, the unit tests may fail with the following error:

Can't add or update WorkerRun [worker run UUID] because parents field isn't properly defined. It can be either because one or several UUIDs don't refer to existing WorkerRuns or either because listed WorkerRuns doesn't belong to the same Process than this WorkerRun.

When the fixtures get written to the data.json file, we have no way of specifying the order in which they get written. Some worker runs in the test fixtures have other worker runs as parents: if these parent worker runs are written not before, but after their child run, then you get the error above.

To fix it, you need to manually edit the data.json file in backend/arkindex/documents/fixtures:

  1. Find all the worker runs (look for process.workerrun) that have a non-empty "parents" property.
  2. If the worker run that is set as a parent for another worker run is written after it in the JSON file, move it so that it comes before.

Once this is done, the code creating the fixtures data from the JSON file will no longer run into an error when creating worker runs, because it will have created the parent worker runs before their children.

Linting

We use pre-commit to check the Python source code syntax of this project.

You need to install pre-commit, and then initialize it for the backend repository by running the following commands:

pip install pre-commit
pre-commit install

The linting workflow will now run on modified files before committing, and may fix issues for you.

To run the full workflow on all the files, use this command: pre-commit run -a. You should always run pre-commit before committing, as you cannot commit unless your code passes the formatting checks.

Debugging tools

Run pip install ipython django-debug-toolbar django_extensions to install all the available optional dev tools for the backend.

IPython will give you a nicer shell with syntax highlighting, auto reloading and much more via arkindex shell.

Django Debug Toolbar provides you with a neat debug sidebar that will help diagnosing slow API endpoints or weird template bugs. Since the Arkindex frontend is completely decoupled from the backend, you will need to browse to an API endpoint to see the debug toolbar.

Django Extensions adds a lot of arkindex commands ; the most important one is arkindex shell_plus which runs the usual shell but with all the available models pre-imported. You can add your own imports with the local_settings.py file. Here is an example that imports some of the backend’s enums and some special QuerySet features:

SHELL_PLUS_POST_IMPORTS = [
    ('django.db.models', ('Value', )),
    ('django.db.models.functions', '*'),
    ('arkindex.documents.models', (
        'ElementType',
        'Right',
    )),
    ('arkindex.process.models', (
        'ProcessMode',
    )),
    ('arkindex.project.aws', (
        'S3FileStatus',
    ))
]

Arkindex frontend

Common operations

  • npm start: Start a local development server on localhost:8080
  • npm run test or npm t: Run unit tests
  • npm run build: Build for development, no Docker image
  • npm run production: Build for production, no Docker image
  • make build: Build to a local Docker image using latest GitLab CI artifact
  • make clean: Clean up files from previous builds
  • make all: Clean up then build

Pre-commit and linting

Pre-commit

If you want to contribute a patch to Arkindex’s frontend, in order for your contribution to be accepted your code must pass a CI pipeline including not only unit tests but linting rules as well.

These formatting checks are run using pre-commit. You need to install pre-commit by running the following command:

pip install pre-commit

You then need to initialize pre-commit by running the following command, in the frontend repository:

pre-commit install

Once this is done, all the pre-commit checks will run whenever you commit some code. To verify that your code passes the pre-commit checks, you can and should run them using the following command:

pre-commit run -a

You cannot commit code if the pre-commit checks fail.

Linting rules

See the .eslintrc.yml file for the exact ESLint settings. The linting rules are based on the following sets:

Some rules have been customized: