idmatcher.ai#

idmatcher.ai, published in [1], matches identities across separate tracking sessions of the same group of animals. Track each video independently with idtracker.ai, then run idmatcher.ai to find which identity in one session corresponds to which identity in another.

Before matching#

For reliable matches, all sessions should have:

  • The same identification image size. An identification network cannot process images of a different size from the ones it was trained on. To keep the size equal across sessions, set id_image_size or use knowledge_transfer_folder (see Knowledge transfer).

  • The same, or very similar, segmentation parameters (intensity thresholds, background subtraction…). Small changes in how the animals look in the identification images make matching less reliable.

  • The same idtracker.ai version. Different versions can produce slightly different identification images.

  • The identification model and images, which are kept with the default data_policy (see Data policies).

Running idmatcher.ai#

Pass the folders of two or more tracked sessions:

idmatcherai path/to/session_MASTER path/to/session_A path/to/session_B ...

The first session is the master. Every other session is matched against it. In the example above, idmatcher.ai matches A with MASTER and B with MASTER.

The results of each match are saved inside the matched session, in session_A/matching_results/session_MASTER/:

  • assignments.csv: the result. Each row pairs an identity of A (first column) with its identity in MASTER (second column), with the direct and indirect scores of the match.

  • csv/ and png/: the direct, indirect and joined matching matrices (see below), as CSV tables and as plots.

How it works#

To match session A with session MASTER:

  1. All the images of A are identified with the identification network of MASTER. This gives the direct matching matrix: row i counts how many images of identity i in A were identified as each identity of MASTER.

  2. In reverse, all the images of MASTER are identified with the network of A. This gives the indirect matching matrix.

  3. The two matrices are added into a joined matching matrix, like the one below.

  4. The Hungarian algorithm assigns each identity of A to one identity of MASTER, maximizing the total number of matches. The assignment is drawn as red crosses in the plots.

  5. For each assigned pair, a direct and an indirect score measure how clear the match is (see the dropdown below). An overall agreement is also computed: the fraction of matches that agree with the assignment.

../_images/idmatcherai_example.png

Example of a joined matching matrix from idmatcher.ai#

Caution

An identification network always gives a wrong, unpredictable answer for an animal it has never seen.

For this reason, when the two sessions have different numbers of animals, only the images from the session with fewer animals are used for the assignment. In this case, check both the direct and the indirect scores to confirm the assignment.

How are the scores computed?

Suppose we match MATCHING with MASTER.

\(D_{n,m}\) is the direct matching matrix: the number of images of identity \(n\) in MATCHING identified as identity \(m\) by MASTER. \(I_{n,m}\) is the indirect matching matrix: the number of images of identity \(m\) in MASTER identified as identity \(n\) by MATCHING.

Every assignment \(i \rightarrow j\) has two associated vectors:

  • the direct vector \(D_{n=i,m}\), the \(i_{\text{th}}\) row of the direct matching matrix,

  • the indirect vector \(I_{n,m=j}\), the \(j_{\text{th}}\) column of the indirect matching matrix,

which contain the assigned values \(D_{n=i,m=j}\) and \(I_{n=i,m=j}\).

The direct and indirect scores \(S\) of the assignment \(i\rightarrow j\) are the difference between the assigned value and its strongest competitor, normalized by the largest value:

\[S_{i\rightarrow j} = \left[ \frac{D_{n=i,m=j}-\max{\left( D_{n=i,m\ne j}\right) }}{\max{\left( D_{n=i,m}\right) }}, \frac{I_{n=i,m=j}-\max{\left( I_{n\ne i,m=j}\right) }}{\max{\left( I_{n,m=j}\right) }}\right]\]

A score of 1 means a clean match with no competitor, a score near 0 means the match is ambiguous, and a negative score means that another identity received more matches than the assigned one.

Funded by FCT under project PTDC/BIA-COM/5770/2020

References