🔎

    Advanced setup and CI

    Arkindex backend🔗

    Dockerized stack🔗

    It is possible to run the whole Arkindex stack through Docker containers. This is useful to quickly test the platform.

    The following command builds all the required Docker images (backend & frontend) and runs them as Docker containers:

    make stack

    You can access the platform at the URL https://ark.localhost.

    Local image server🔗

    Arkindex splits up image URLs into image servers and image paths. For example, a IIIF server at http://iiif.irht.cnrs.fr/iiif/ and an image at /Paris/JJ042/1.jpg would be represented as an ImageServer instance holding one Image. Since Arkindex has a local IIIF server for image uploads and thumbnails, a special instance of ImageServer is required to point to this local server. In local development, this server should be available at https://ark.localhost/iiif. You will therefore need to create an ImageServer via the Django admin or the Django shell with this URL. To set the local server ID, you can add a custom setting in arkindex/config.yml:

    local_imageserver_id: 42

    Here is how to quickly create the ImageServer using the shell:

    $ arkindex shell
>>> from arkindex.images.models import ImageServer
>>> ImageServer.objects.create(id=42, display_name='local', url='https://ark.localhost/iiif')

    Note that this local server will only work inside Docker.

    Common operations🔗

    Makefile🔗

    At the root of the repository is a Makefile that provides commands for common operations:

    • make or make all: Clean and build;
    • make base: Create and push the arkindex-base Docker image that is used to build the arkindex-app image;
    • make clean: Cleanup the Python package build and cache files;
    • make clean-docker: Deletes all running containers to avoid naming and network ports conflicts;
    • make build: Build the arkindex Python package and recreate the arkindex-app:latest without pushing to the GitLab container registry;
    • make test-fixtures: Create the unit tests fixtures on a temporary PostgreSQL database and save them to the data.json file used by most Django unit tests.

    Django commands🔗

    Aside from the usual Django commands, some custom commands are available via arkindex:

    • build_fixtures: Create a set of database elements designed for use by unit tests in a fixture (see make test-fixtures).
    • delete_corpus: Delete a big corpus using an RQ task.
    • reindex: Reindex elements into Solr.
    • move_lines_to_parents: Moves element children to their geographical parents.

    See arkindex <command> --help to view more details about a specific command.

    Code validation🔗

    Once your code appears to be working on a local server, a few checks have to be performed:

    • Migrations: Ensure that all migrations have been created by running the arkindex makemigrations command.
    • Unit tests: Run arkindex test to perform unit tests.
      • Use arkindex test module_name to perform tests on a single module, if you wish to spend less time waiting for all tests to complete.

    Linting🔗

    We use pre-commit to check the Python source code syntax of this project.

    You need to install pre-commit, and then initialize it for the backend repository by running the following commands:

    pip install pre-commit
pre-commit install

    The linting workflow will now run on modified files before committing, and may fix issues for you.

    To run the full workflow on all the files, use this command: pre-commit run -a. You should always run pre-commit before committing, as you cannot commit unless your code passes the formatting checks.

    Debugging tools🔗

    Run pip install ipython django-debug-toolbar django_extensions to install all the available optional dev tools for the backend.

    IPython will give you a nicer shell with syntax highlighting, auto reloading and much more via arkindex shell.

    Django Debug Toolbar provides you with a neat debug sidebar that will help diagnosing slow API endpoints or weird template bugs. Since the Arkindex frontend is completely decoupled from the backend, you will need to browse to an API endpoint to see the debug toolbar.

    Django Extensions adds a lot of arkindex commands ; the most important one is arkindex shell_plus which runs the usual shell but with all the available models pre-imported. You can add your own imports with the local_settings.py file. Here is an example that imports some of the backend's enums and some special QuerySet features:

    SHELL_PLUS_POST_IMPORTS = [
    ('django.db.models', ('Value', )),
    ('django.db.models.functions', '*'),
    ('arkindex.documents.models', (
        'ElementType',
        'Right',
    )),
    ('arkindex.process.models', (
        'ProcessMode',
    )),
    ('arkindex.project.aws', (
        'S3FileStatus',
    ))
]

    Arkindex frontend🔗

    Common operations🔗

    • npm start: Start a local development server on localhost:8080
    • npm run test or npm t: Run unit tests
    • npm run build: Build for development, no Docker image
    • npm run production: Build for production, no Docker image
    • make build: Build to a local Docker image using latest GitLab CI artifact
    • make clean: Clean up files from previous builds
    • make all: Clean up then build

    Pre-commit and linting🔗

    Pre-commit🔗

    If you want to contribute a patch to Arkindex's frontend, in order for your contribution to be accepted your code must pass a CI pipeline including not only unit tests but linting rules as well.

    These formatting checks are run using pre-commit. You need to install pre-commit by running the following command:

    pip install pre-commit

    You then need to initialize pre-commit by running the following command, in the frontend repository:

    pre-commit install

    Once this is done, all the pre-commit checks will run whenever you commit some code. To verify that your code passes the pre-commit checks, you can and should run them using the following command:

    pre-commit run -a

    You cannot commit code if the pre-commit checks fail.

    Linting rules🔗

    See the .eslintrc.yml file for the exact ESLint settings. The linting rules are based on the following sets:

    Some rules have been customized: