Usage#

Basic Usage#

Activate idtracker.ai’s Conda environment using conda activate idtrackerai, then run the command:

idtrackerai

to launch the Segmentation app, a graphical application designed to help you define the correct input parameters for your videos. There you can select the desired video(s) to track, set the basic parameters and start the tracking process.

Terminal usage#

From the Segmentation app, you can start tracking directly or you can save the specified parameters in a .toml file like this one, enabling their reuse or automation in future tracking sessions:

example.toml#

name = 'example'
video_paths = ['/home/user/idtrackerai/video_A.avi']
intensity_ths = [0, 155]
area_ths = [100.0, inf]
tracking_intervals = ""
number_of_animals = 8
use_bkg = false
check_segmentation = false
track_wo_identities = false
roi_list = ['+ Polygon [[138.0, 50.1], [992.9, 62.1], [996.9, 878.9]]']
exclusive_roi = false

This file contains the full configuration defined in the Segmentation app and it can be loaded anytime with

idtrackerai --load example.toml

to recover the app as you left it, or with

idtrackerai --load example.toml --track

to load the parameters from example.toml and start the tracking process without any prior graphical interface. This feature allows the control of idtracker.ai in remote via ssh and the capability to write custom scripts to run sequences of tracking sessions.

Parameter log

Every loaded parameter will be notified in the Tracking log, always read it while checking your parameters have been properly read.

More advanced parameters can be used to extend idtracker.ai’s capabilities. These can be loaded from a .toml file by using the same --load argument (see the details of these Advanced parameters below in this page).

Finally, any additional parameter can be passed in the command line as --PARAMETER VALUE.

An example of an advanced idtracker.ai command could be:

idtrackerai --load my_basic_settings.toml example.toml --track_wo_identities true --number_of_animals 15 --track

Note

Parameter files specified with --load are processed in the order they are listed, with later files overriding earlier ones for the same parameter. For example, in the command above, settings in example.toml will take precedence over those in my_basic_settings.toml. Additionally, any parameter specified directly on the command line will override all settings from the loaded files.

Tip

In the case of running idtracker.ai in remote (where the session parameters may have been created in another computer), it could be helpful to override, for example, the video paths from example.toml:

idtrackerai --load example.toml --video_paths path/in/remote/computer.avi --track

Tracking log#

Take care of your machine

Pay attention to your computer status during tracking (CPU, RAM and GPU usage). Idtracker.ai can be very memory-intensive in some parts (see Parallel processing) and your computer can struggle on very long high resolution videos.

During tracking, idtracker.ai will communicate with the user through the log. This log will be displayed live in the terminal (Anaconda prompt on Windows) and written in the idtrackerai.log file in the current working directory. Users should keep an eye on the log, checking its status and warnings.

When a critical error occurs, the log contains all the information needed to solve it. Read the last lines of it to know more about what went wrong or send it to us at info@idtracker.ai so that we can help you. For more information on common errors after the installation, please refer to Installation Troubleshooting.

Advanced parameters#

Besides the basic parameters from the segmentation app (the ones in example.toml), more advanced parameters can be used.

Important

All parameter names are case-insensitive.
Define path variables using 'single quotes' instead of "double ones" in the toml files to avoid backslashes (\) to trigger special characters (see TOML documentation to know more)
The value '' in a toml file is loaded as a Python’s None in idtrackerai.

Output#

OUTPUT_DIR. Sets the directory path where the output session folder will be saved, by default it is the input video directory.
TRAJECTORIES_FORMATS. The output trajectory files can be saved in four different formats: H5DF ("h5"), Numpy ("npy"), Python’s pickle ("pickle") and CSV ("csv"). Use this parameter to indicate the desired format(s) as a list of strings. Know more about these formats in Trajectory files.
BOUNDING_BOX_IMAGES_IN_RAM If true, bounding box images, a middle step to generate the final identification images, will be kept in RAM until no longer needed. Else, they are saved in disk and loaded when needed. We recommend setting this to true only when working with very slow disks (HDD) to speed up segmentation.
DATA_POLICY. The tracking algorithm generates a significant amount of data, which is stored in the session folder along with the trajectory files. While some of this data can be large, it is also essential for running various additional tools included with the software.

The available data policy options are: "all", "idmatcher.ai", "knowledge_transfer", "validation" and "trajectories".

This setting determines which data is retained and, thus, which tools remain accessible. The default option, "idmatcher.ai", provides a balanced approach between storage efficiency and tool availability.

The table below details which data is preserved under each policy and which tools remain functional.

Data available with different data policies#
	`all`	`idmatcher.ai`	`knowledge_transfer`	`validation`	`trajectories`
Trajectories	✅	✅	✅	✅	✅
Pre-processing folder	✅	✅	✅	✅	❌
Identification model folder	✅	✅	✅	❌	❌
Identification images	✅	✅	❌	❌	❌
Crossing detector folder	✅	❌	❌	❌	❌
Bounding box images	✅	❌	❌	❌	❌

Functional tools with different data policies#
	`all`	`idmatcher.ai`	`knowledge_transfer`	`validation`	`trajectories`
Validator	✅	✅	✅	✅	✅
Video Generators	✅	✅	✅	✅	✅
Knowledge transfer	✅	✅	✅	❌	❌
Idmatcher.ai	✅	✅	❌	❌	❌
Cluster inspection	✅	✅	❌	❌	❌

Output defaults#

output_dir = ''
trajectories_formats = ["h5", "npy", "csv"]
bounding_box_images_in_ram = false
data_policy = "idmatcher.ai"

Background subtraction#

The animal segmentation can be done by subtracting the background to each frame and thresholding this difference. To do this, a stack of sample frames is generated to later compute the background estimation using some statistical method.

BACKGROUND_SUBTRACTION_STAT. Sets the statistic method to compute the background from the stack of sample frames, choices are "median" (default), "mean", "max" (for dark animals on bright backgrounds) and "min" (for bright animals on dark backgrounds).
NUMBER_OF_FRAMES_FOR_BACKGROUND. Sets the number of frames used to generate the stack of sample frames. These are equally spaced along the tracking intervals. More frames means more accuracy but also more computing time and RAM usage.

Background subtraction defaults#

background_subtraction_stat = "median"
number_of_frames_for_background = 50

Tracking checks#

CHECK_SEGMENTATION. The presence of frames with more blobs than animals means a bad segmentation with non-animal blobs detected. Idtracker.ai is not built to deal with non-animal blobs (shadows, reflections, dust…), these can contaminate the algorithms harming the identification. To ensure a proper segmentation, set this to true and idtracker.ai will abort the tracking session if a bad segmentation is detected.
```
check_segmentation = false
```
Note

This parameter appears on the segmentation app as Stop tracking if #blobs > #animals.

Parallel processing#

Some parts of idtracker.ai are parallelized (segmentation and identification images creation). This is done by slicing the video into different chunks and sending them to a group of independent workers to process.

NUMBER_OF_PARALLEL_WORKERS. Sets the number of workers used in the parallel parts. A negative value means using as many workers as the total number of CPUs minus the specified value. Zero value means running half of the total number of CPUs in the system or 8 if the system has more than 16 cores (using more than 8 cores doesn’t provide any significant speed up). One means not using multiprocessing at all. The default value is 0.

Warning

During segmentation, every worker can use up to 4GB of memory, using too many workers might fill your RAM memory very fast. Computers with a large number of CPU cores (>10) should be monitored and the number of parallel workers should be adjusted accordingly. For users with limited RAM, consider reducing the number of parallel workers by setting the number_of_parallel_workers parameter to a lower value. Additionally, use monitoring tools like htop, top, or free on Linux, Task Manager or Resource Monitor on Windows, and Activity Monitor on macOS to keep an eye on your system’s resource usage.
FRAMES_PER_EPISODE. Sets the size of the video chunks (episodes). Less frames per episode means more parallel chunks.

Parallel processing defaults#

number_of_parallel_workers = 0
frames_per_episode = 500

Knowledge transfer#

You can use the knowledge acquired by the identification model of a previous video as a starting point for the training of the current one. This speeds up the identification training when the videos are very similar (same light conditions, distance from camera to arena, type and size of animals).

KNOWLEDGE_TRANSFER_FOLDER.: Sets the path to a session or accumulation folder from a previous tracked video. For example "/home/username/session_test" or "/home/username/session_test/accumulation". This will load the weights of the models trained in the previous video as a starting point for the current session. It will also adopt the same ID_IMAGE_SIZE and RESOLUTION_REDUCTION as the previous video. By default, no knowledge is transferred and every identification model starts from scratch.
IDENTITY_TRANSFER.: If the animals in your video are the same as the ones from the knowledge_transfer session, set this parameter to true to perform identity transfer. If so, idtracker.ai will use the network from the knowledge_transfer session to assign identities in the current session. In our experience, for this to work the video conditions need to be almost identical to the previous video.
ID_IMAGE_SIZE. Identification images are squares, the size of which is, by default, optimized to match the size of the animals in each video. You can override this optimization by defining this parameter to an integer (the size in pixels of the side of the square images). Check the note below for more information about the behavior of this parameter.
RESOLUTION_REDUCTION. Very big identification images (> 80 pixels per side) are usually unnecessarily heavy to work with. In this case, this parameter can scale down the images of the animals to fit them into smaller identification images, speeding up the tracking. It can go from 0 (limit to infinite reduction) to 1 (no reduction at all). Check the note below for more information about the behavior of this parameter.

Note

The automatic values of ID_IMAGE_SIZE and RESOLUTION_REDUCTION are codependent in the following way:

If None of them are defined (default): the ID_IMAGE_SIZE is set based on the average size of the animals and the RESOLUTION_REDUCTION is used to limit this size to 80 pixels only if necessary.
Only ID_IMAGE_SIZE is defined by the user: only in case the animals average size is bigger than the stated image size, the resolution reduction is used to fit those animals in the images.
Only RESOLUTION_REDUCTION is defined by the user: the ID_IMAGE_SIZE is set based on the rescaled average size of the animals.

We recommend to let idtrackerai define both parameters automatically, or to use the KNOWLEDGE_TRANSFER_FOLDER to inherit the parameters from a previously tracked video.

Knowledge transfer defaults#

knowledge_transfer_folder = ''
identity_transfer = false
id_image_size = ''
resolution_reduction = ''

Tip

There are alternative ways of transferring identities between tracking sessions. Check our tool Idmatcher.ai, it requires the identification image size and the resolution reduction factor to be equal for all the sessions.

Contrastive#

Contrastive learning has been introduced in version 6.0.0 as the new identification algorithm (publication in progress). In it, all individual blobs are used to train ResNet to embed images in an embedded space by using positive and negative pairs of images (this is why it’s called contrastive learning). Positive pairs of images come from the same fragment and negative pairs come from different but coexisting fragments. With training, images from the same animal start clustering in the embedded space and their silhouette score increases reaching the target score. After contrastive training, images are embedded, clustered, identified and accumulated if possible. If enough images have been accumulated, the identification is completed, else the accumulation protocol starts by training the small idtrackerai’s idCNN with the accumulated images from contrastive as a first synthetic global fragment.

DISABLE_CONTRASTIVE. Skips the contrastive step to go directly to accumulation protocol.
CONTRASTIVE_MAX_MBYTES. Maximum number of megabytes the identification images can weight to be preloaded in RAM during contrastive training. The default is half of the available memory in the system when contrastive is initialized.
CONTRASTIVE_BATCHSIZE. Number of pairs of images contained in a contrastive training batch. The more pairs of images, the more GPU memory will be needed. A batch of size \(N\) will contain \(N\) positive and \(N\) negative pairs of images, so \(4N\) images in total.
CONTRASTIVE_SILHOUETTE_TARGET. Minimum silhouette score required for contrastive to finish training. This score, since coming from a K-Means clustering, is ranged from zero to one. Set it to one (unachievable value) to maximize the quality of the contrastive model and stopping the training only because of the triggering of the patience.
CONTRASTIVE_PATIENCE. Number of steps without an improvement on the silhouette score to trigger the patience and early stopping the training during contrastive learning.

Contrastive defaults#

disable_contrastive = false
contrastive_max_mbytes = ''
contrastive_batchsize = 400
contrastive_silhouette_target = 0.91
contrastive_patience = 30

Basic parameters#

The assignment of any basic parameter (like the ones in example.toml) in the settings file acts as a default. For example, if you always track videos with 8 animals, you can set number_of_animals = 8 in you settings file and, when running idtrackerai --load settings.toml, the segmentation app will run with 8 animals as default.

Advanced hyper-parameters#

Warning

These parameters change the way the CNN is trained, use with care.

THRESHOLD_EARLY_STOP_ACCUMULATION.: Fraction of accumulated images needed to early stop the accumulation process.
MAXIMAL_IMAGES_PER_ANIMAL.: Maximum number of images per animal that will be used to train the CNN in each accumulation step.
DEVICE.: Device name passed to torch.device() to indicate where to perform machine learning operations, typically "cpu", "cuda", "cuda:0"… See Torch documentation. (default: empty string, automatic device selection).

threshold_early_stop_accumulation = 0.999
maximal_images_per_animal = 3000
device = ""

File example#

An example settings file with all parameters as default (no effect) is

example settings.toml#

# Segmentation app defaults
name = ''
video_paths = ''
intensity_ths = [0, 130]
area_ths = [50.0, inf]
tracking_intervals = ""
number_of_animals = 0
use_bkg = false
check_segmentation = false
track_wo_identities = false
roi_list = []

# Output
output_dir = ''
trajectories_formats = ["h5", "npy", "csv"]
bounding_box_images_in_ram = false
data_policy = 'idmatcher.ai'

# Background subtraction
background_subtraction_stat = 'median'
number_of_frames_for_background = 50

# Parallel processing
number_of_parallel_workers = 0
frames_per_episode = 500

# Knowledge and identity transfer
knowledge_transfer_folder = ''
identity_transfer = false
id_image_size = ''
resolution_reduction = ''

# Contrastive
disable_contrastive = false
contrastive_max_mbytes = ''
contrastive_batchsize = 400
contrastive_silhouette_target = 0.91
contrastive_patience = 30

# Advanced hyper-parameters
threshold_early_stop_accumulation = 0.999
maximal_images_per_animal = 3000
device= ""

`idtrackerai -h`#

Use the command idtrackerai -h to print the list of all possible command line arguments in your terminal:

Output of idtrackerai -h

options:

-h, --help: show this help message and exit

General:

--load <path …>: A list of .toml files to load session parameters in increasing priority order.
--track: Track the video without launching the GUI
--name <str>: Name of the session (default: name of the video files).
--video_paths <path …>: List of paths to the video files to track (default: []).
--intensity_ths <float float>: Blob’s intensity thresholds. When using background subtraction, the background difference threshold is the second value of these intensity thresholds (default: None).
--area_ths <float float>: Blob’s areas thresholds (default: None).
--tracking_intervals <pair_of_ints …>: Tracking intervals in frames. Examples: “0,100”, “[0,100]”, “[0,100] [150,200] …”. If none, the whole video is tracked (default: None).
--number_of_animals <int>: Number of different animals that appear in the video (default: 0).
--use_bkg <bool>: Compute and extract background to improve blob identification (default: False).
--resolution_reduction <float>: Video resolution reduction factor used in the creation of the identification images from 0 (limit of infinite reduction) to 1 (no reduction) (default: None).
--exclusive_rois <bool>: Treat each separate ROI as closed identities groups (default: False).
--track_wo_identities <bool>: Track the video ignoring identities (without AI) (default: False).
--roi_list <str …>: List of polygons defining the Region Of Interest (default: None).

Output:

--output_dir <path>: Output directory where session folder will be saved to, default is video paths parent directory (default: None).
--trajectories_formats <str …>: A sequence of strings defining in which formats the trajectories should be saved (choices: h5, npy, csv, pickle) (default: [‘h5’, ‘npy’, ‘csv’]).
--bounding_box_images_in_ram <bool>: If true, bounding box images, a middle step to generate the identification images, will be kept in RAM until no longer needed. Else, they are saved in disk and loaded when needed (default: False).
--data_policy <str>: Type of data policy indicating the data in the session folder not to beerased when successfully finished a tracking (choices: trajectories, validation, knowledge_transfer, idmatcher.ai, all) (default: idmatcher.ai).

Background Subtraction:

--background_subtraction_stat <str>: Statistical method to compute the background (choices: median, mean, max, min) (default: median).
--number_of_frames_for_background <int>: Number of frames used to compute the background (default: 50).

Parallel processing:

--number_of_parallel_workers <int>: Maximum number of jobs to parallelize segmentation and identification image creation. A negative value means using the number of CPUs in the system minus the specified value. Zero means using half of the number of CPUs in the system (limited to 8). One means no multiprocessing at all (default: 0).
--frames_per_episode <int>: Maximum number of frames for each video episode (used to parallelize some processes) (default: 500).

Knowledge and identity transfer:

--knowledge_transfer_folder <path>: Path to the session to transfer knowledge from (default: None).
--identity_transfer <bool>: If true, identities from knowledge transfer folder are transferred (default: False).
--id_image_size <int>: The size of the identification images used in the tracking (default: []).

Checks:

--check_segmentation <bool>: Check all frames have less or equal number of blobs than animals (default: False).

Contrastive:

--disable_contrastive <bool>: Disable the contrastive first step to go directly to accumulation protocol (default: False).
--contrastive_max_mbytes <float>: Maximum number of megabytes the identification images can weight to be preloaded in RAM during contrastive training (default: None).
--contrastive_batchsize <int>: Number of pairs of images a training batch contains in contrastive training. The more pairs of images, the more GPU memory will be needed (default: 400).
--contrastive_silhouette_target <float>: Minimum silhouette score required for contrastive to finish. From zero to one. (default: 0.91).
--contrastive_patience <int>: The maximum number of training steps without an improvement on the silhouette score to trigger the patience and early stopping the contrastive training (default: 30).

Advanced hyperparameter:

--threshold_early_stop_accumulation <float>: Ratio of accumulated images needed to early stopping the accumulation process (default: 0.999).
--maximal_images_per_animal <int>: Maximum number of images per animal that will be used to train the IdCNN in each accumulation step (default: 3000).
--device <str>: Device name passed to torch.device() to indicate where machine learning computations will be performed, typically “cpu”, “cuda”, “cuda:0”… See https://pytor ch.org/docs/stable/tensor_attributes.html#torch- device. (default: empty string, automatic device selection)..

Deprecated:

--add_time_column_to_csv <str>: The parameter ‘add_time_column_to_csv’ has been removed.
--convert_trajectories_to_csv_and_json <str>: The parameter ‘convert_trajectories_to_csv_and_json’ has been removed.
--protocol3_action <str>: The parameter ‘protocol3_action’ has been removed.
--threshold_acceptable_accumulation <str>: The parameter ‘threshold_acceptable_accumulation’ has been removed.
--maximum_number_of_parachute_accumulations <str>: The parameter ‘maximum_number_of_parachute_accumulations’ has been removed.
--max_ratio_of_pretrained_images <str>: The parameter ‘max_ratio_of_pretrained_images’ has been removed.

Usage Analytics#

idtracker.ai collects usage analytics to improve the software. This data is anonymized, does not include any sensitive or personally identifiable information, and is collected in every command line call to an idtracker.ai module (not collected for API calls). It is used solely for research purposes and contains general information such as the operating system, the version of idtracker.ai being used, and the command used to run it.

You may check out the source code to see exactly what data is collected.

If you want to opt-out of this data collection, go to the “About” menu located in the top-left corner of any idtracker.ai graphical application and uncheck the box for analytics. Alternatively, set the environment variable IDTRACKERAI_DISABLE_ANALYTICS to true or 1. For example:

On Linux/macOS: export IDTRACKERAI_DISABLE_ANALYTICS=true
On Windows: set IDTRACKERAI_DISABLE_ANALYTICS=true

Any of these actions will keep analytics disabled until you enable them again.