Creating ground truth for segmentation

In this tutorial, you will learn how to create ground truth data to train an image segmentation model, using the Callico collaborative annotation platform.

This section should be followed and performed after you have completed the first steps described on this page.

Preliminary requirement¶

You are about to learn how to manually segment text lines and illustrations on pages from the Pellet dataset using Callico.

Prior to working with Callico, you have to create a new element type in Arkindex. To do so, navigate to your Arkindex project details page that you have created in the previous steps.

Once on your project details page, open the Types tab and add a new type as presented below:

Display name - The name of your element type, you can use Illustration.
Slug - The slug of your element type, you can use illustration.
Folder - Whether your type is a folder or not, you have to leave it unchecked.
Color (optional) - The color to display elements holding this type in Arkindex.
Actions - Click on the + button to validate your type creation.

Create the illustration type in Arkindex

As you can see, the Text line element type already exists by default, so you don’t need to create it.

Import data to segment in Callico¶

Info

This section expects you to have a Callico account. You can follow this link to register on Callico’s demonstration instance.

Create a Callico project¶

Once logged in to Callico, you have to create a new annotation project by clicking the Create a project green button at the top-right of the homepage.

Then, fill in the creation form as presented below:

Callico's pre-filled project creation form

Name - Name of your project, should be unique, do not copy word by word the name from the screenshot above.
Description (optional) - Description of your project, it supports Markdown.
Illustration (optional) - Illustration of your project to display on various pages across Callico, you can use this image for example.
Provider - Provider from which to import and export your project data, in this tutorial we will pick Arkindex demo.
Object identifier in provider - The identifier of the object, in the provider, your Callico project should be linked to. As shown in the screenshot above, we are talking about an Arkindex project UUID, you have to replace the aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee value by yours which can be copied from your Arkindex project details page, just below its name.

Import an Arkindex dataset¶

Once your project is created, you are ready to import an Arkindex dataset and its elements to Callico.

To do so, you can click on the Import from Arkindex action in the Elements section of the menu on the left side of the project details page:

Then, fill in the import form as presented below, to import all of the Page elements from your dataset containing data from Europeana:

Callico's pre-filled Arkindex import form

Process name - Name of your process to import elements from Arkindex.
~~Element~~ - Not relevant when importing an Arkindex dataset.
Dataset - UUID of your Arkindex dataset, you have to replace the aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee value by yours which can be copied from your Arkindex dataset details page, just below its name.

~~Filter sets to import~~ - To filter the sets from your Arkindex dataset that should be imported to Callico, we do not use it in this tutorial.
Filter types to import - To filter the Arkindex elements to import by their type, here we only want to import Page elements from your dataset.
All subsequent fields from this form are to be ignored in this tutorial, you can learn more about them in a dedicated page from Callico’s documentation if you wish to.

Track your import’s progress¶

Once you have started the data import, you will be redirected to a new page where you can track its progress. Note that this page is not dynamically refreshed. You will need to reload it manually to see updated status and logs. When the import is complete, its status will be updated to Completed.

Track the Arkindex import progress in Callico

Create segmentation tasks to annotate¶

While Arkindex elements are being imported into your Callico project, you can start setting up your annotation campaign.

Create a segmentation campaign¶

First, navigate back to your Callico’s project details page by using the navbar at the top of the page and clicking on your project name.

Navigate back to your project from the process page

From there, you can click on the Create action in the Campaigns section of the menu on the left side of the project details page:

Then, fill in the creation form as presented below:

Callico's pre-filled campaign creation form

Name - Name of your campaign, you can copy the name from the screenshot above.
Mode - Mode of your campaign, you have to pick the Elements one to follow this tutorial.
Description (optional) - Description of your campaign, it supports Markdown.

Configure the newly created campaign¶

Once your segmentation campaign is created, you will be redirected to its configuration page. Fill in the configuration form as presented below:

Callico's pre-filled campaign configuration form

Name - Keep it unchanged.
Description (optional) - Keep it unchanged.
Number of tasks to assign per volunteer - Set this field value to 10, this will allow annotators to request tasks by batch containing 10 pages.
~~Number of allowed assignments for available tasks~~ - Ignore this field.
Element types to use to annotate - The element types that will be used to draw elements on your pages. Keep Illustration and Text line types and uncheck all others.

Create annotation tasks¶

After configuring your campaign, you will be redirected to its details page. From there, you can access the form to create annotation tasks by clicking the Create action in the Tasks section of the menu on the left:

Warning

Please make sure your import process is complete before creating your annotation tasks, otherwise you may miss pages while annotating.

Then, fill in the creation form as presented below:

Element type - The element type to create your tasks on, here we are annotating Pages.
~~Users~~ - Ignore this field (do not worry, we will add users to your project in the next step).
Sequential - Keep it unchanged, pages will be annotated following their import order in Callico.
Elements to use - Keep it unchanged, we want to annotate all the imported pages.
~~Maximum number of tasks per user~~ - Ignore this field.
Create unassigned tasks - This option allows the creation of annotation tasks which will be requested by the annotators as they go, in this tutorial, you must check it.

Once the tasks are created, you will be redirected to the task list which should contain many items, one for each page to be annotated from your dataset.

You can navigate back to your Callico’s project details page by using the navbar at the top of the page and clicking on your project name.

Navigate back to your project from the task list

Invite collaborators¶

To invite users to join your project, you can click on the Invite users action in the Project section of the menu on the left side of the project details page:

From there, you can copy the invite link of your project to your clipboard by clicking on the Copy button on the right:

Once the link is copied, you can share it with users to invite them to collaborate on your project.

By opening the link, they will be asked to login into Callico (or register if they do not already have an account) before being added as Contributors to your project, meaning they will have the rights to annotate tasks from the campaign you just created.

If you wish to annotate the tasks by yourself, you can:

Logout from Callico
Open the invite link that was just copied
Register using a different email address
Accept the invitation to join the project as a contributor

Annotate the segmentation tasks¶

In this section, we will put ourselves in the shoes of a Contributor user whose role is to annotate tasks from one or more campaigns.

Request annotation tasks¶

Once you have joined the project as a contributor, you can request tasks.

Callico's project details page for contributors

To request a task on your campaign, you can click on the My tasks blue button displayed next to the Segment text lines and illustrations from pages campaign name on the current page:

You will be redirected to the task list for this campaign, showing available tasks, i.e. the ones that can be requested for annotation. From there, you can click on the Annotate green button on any task to request it.

You will be redirected to the annotation page of this specific task.

Redirection after requesting a single task

You can annotate this task, before being redirected to your task list for this campaign.

Redirection to the task list after a single annotation

Info

You can also requests 10 tasks simultaneously by clicking on the Requests tasks button instead of My tasks described above. You will be redirected towards the annotation page of your first task, then to the next one in the stack, and so on.

From there, you can request one or more tasks to continue annotating.

Annotate your tasks¶

Now that you know how to request tasks, you will learn how to annotate segmentation tasks. Here is an annotation page:

Callico's annotation page for a segmentation campaign

Select a type¶

At any time during the annotation, you can pick the element type that will be assigned to the next elements that you will draw. In this tutorial, we want to segment:

illustrations and
text lines.

Pick an element type before drawing elements

Draw a rectangle¶

Once you have selected a type, you can start drawing rectangles and/or polygons.

To draw a rectangle, you need to select the Rectangle tool (highlighted in blue just below) and start dragging your mouse on the image:

Then, simply release your mouse to validate your drawing:

Draw a polygon¶

Once you have selected a type, you can start drawing polygons and/or rectangles.

To draw a polygon, you need to select the Polygon tool (highlighted in blue just below) and start adding points on the image by clicking your mouse:

Then, finish your polygon by clicking on the very first point you added:

Click on the first added point to validate your drawing

Edit drawn elements¶

If one of your rectangles or polygons is poorly drawn:

You can edit its placement or its points using the Mouse tool (highlighted in blue just below):

Delete wrongfully drawn elements¶

Danger

Deleting a drawn element is irreversible, be careful when using this feature, you can always edit an element if it is poorly placed.

When adding elements, you may have forgotten to select the right element type. Such elements can be deleted using the Trash red icon displayed in the drawn elements table on the right side of the annotation page:

Delete the drawn illustration that should be a text line

Other tools on the image component¶

A few other tools are available to ease the annotation process:

A slider to Zoom in or Zoom out the image being worked on,
An Open in a new tab tool to better visualize large images,
Two Rotate left and Rotate right tools to pivot your image.

Warning

Do not forget to validate your task by clicking the Submit green button when you are done annotating.

Correct an annotated task¶

If you have submitted a task without finishing your annotation or want to correct drawn elements, you can edit it by going to the Annotated tab in your task list and clicking the Change annotation green button:

You will be redirected to the task annotation page, pre-filled with the last annotations you made:

Annotation page pre-filled with a previous version

In this case, we may have forgotten to segment the stamps as illustrations, we can add them on the image and submit a new version for our task:

Add illustrations and submit a new version

The last version of an annotation task is the one that is exported to the provider, the one published back to Arkindex in our tutorial.

Track and export annotations back to Arkindex¶

Info

If necessary, logout from your Contributor account and login with your first email address.

Back to your Manager account, you can track the progress of your segmentation campaign from its details page:

Callico's campaign details page showing the ongoing progress

Once it is completed, i.e. when all tasks from this tutorial are annotated, you can proceed with the export to Arkindex.

Export results to Arkindex¶

To export your results back to Arkindex, you will need to click on the To Arkindex action in the Export results section of the menu on the left of the campaign details page.

Then, fill in the export form as presented below:

Callico's pre-filled Arkindex export form

Process name - Name of your process to export annotations to Arkindex.
Status of tasks to be exported - Pick the Annotated value to export your tasks.
~~Force the republication of annotations~~ - Ignore this field.
~~Publish each annotation separately~~ - Ignore this field.

Track your export’s progress¶

Once you have started the results export, you will be redirected to a new page where you can track its progress. Note that this page is not dynamically refreshed. You will need to reload it manually to see updated status and logs. When the export is complete, its status will be updated to Completed.

Track the progress of the export to Arkindex in Callico

Check that your Arkindex export went smoothly¶

Once the export process is complete, you should check that the annotations for your segmentation tasks have been properly published to Arkindex by browsing your dataset elements:

Arkindex dataset details page showing its elements

Congratulations, you have successfully segmented pages in Callico and exported the annotations back to Arkindex!

Annotated text lines and illustrations are available in Arkindex

Next step¶

Now that the ground truth has been annotated on Callico and collected in Arkindex, you are ready to train a Machine Learning segmentation model.