Command Line Interfaces

sl-configure

This Command-Line Interface allows configuring major components of the Sun lab data workflow.

Usage

sl-configure [OPTIONS] COMMAND [ARGS]...

directory

Sets the input directory as the local Sun lab’s working directory.

Usage

sl-configure directory [OPTIONS]

Options

-d, --directory <directory>: Required The absolute path to the directory where to cache Sun lab configuration and local runtime data.

experiment

Configures the local data acquisition system to execute the specified project’s experiment.

Usage

sl-configure experiment [OPTIONS]

Options

-p, --project <project>: Required The name of the project for which to generate the new experiment configuration file.

-e, --experiment <experiment>: Required The name of the experiment for which to create the configuration file.

-sc, --state_count <state_count>: Required The number of runtime states supported by the experiment.

-wc, --water_reward_count <water_reward_count>: Required The number of water reward (reinforcing) trial types supported by the experiment.

-gc, --gas_puff_count <gas_puff_count>: Required The number of gas puff (aversive) trial types supported by the experiment.

google

Sets the path to the Google service account credentials file.

Usage

sl-configure google [OPTIONS]

Options

-c, --credentials <credentials>: Required The absolute path to the Google service account credentials .JSON file.

project

Configures the local data acquisition system to acquire data for the specified project.

Usage

sl-configure project [OPTIONS]

Options

-p, --project <project>: Required The name of the project to be created.

server

Creates the remote compute server configuration file.

Usage

sl-configure server [OPTIONS]

Options

-u, --username <username>: Required The username to use for server authentication.

-p, --password <password>: Required The password to use for server authentication.

-h, --host <host>

Required The host name or IP address of the server.

Default:: 'cbsuwsun.biohpc.cornell.edu'

-sr, --storage-root <storage_root>

Required The absolute path to to the server’s slow HDD RAID volume.

Default:: '/local/storage'

-wr, --working-root <working_root>

Required The absolute path to to the server’s fast NVME RAID volume.

Default:: '/local/workdir'

-sd, --shared-directory <shared_directory>

Required The name of the shared directory used to store all Sun lab’s projects on both server’s volumes.

Default:: 'sun_data'

system

Creates the specified data acquisition system’s configuration file.

Usage

sl-configure system [OPTIONS]

Options

-s, --system <system>

Required The type (name) of the data acquisition system for which to create the configuration file.

Default:: <AcquisitionSystems.MESOSCOPE_VR: 'mesoscope'>
Options:: mesoscope

Data Transfer Assets

This package provides assets for safely transferring data between destinations available on the local filesystem and efficiently removing it from the local filesystem.

sl_shared_assets.data_transfer.calculate_directory_checksum(directory, num_processes=None, *, progress=False, save_checksum=True)

Calculates the xxHash3-128 checksum for the input directory.

Note

The function can be configured to write the generated checksum as a hexadecimal string to the ax_checksum.txt file stored at the highest level of the input directory.

The xxHash3 checksum is not suitable for security purposes and is only used to ensure data integrity.

The returned checksum accounts for both the contents of each file and the layout of the input directory structure.

Parameters:

directory (Path) – The path to the directory for which to generate the checksum.
num_processes (int | None, default: None) – The number of processes to use for parallelizing checksum calculation. If set to None, the function uses all available CPU cores.
progress (bool, default: False) – Determines whether to track the checksum calculation progress using a progress bar.
save_checksum (bool, default: True) – Determines whether the checksum should be saved (written to) a .txt file.

Return type:

str

Returns:

The xxHash3-128 checksum for the input directory as a hexadecimal string.

sl_shared_assets.data_transfer.delete_directory(directory_path)

Deletes the target directory and all its subdirectories using parallel processing.

Parameters:: directory_path (Path) – The path to the directory to delete.
Return type:: None

sl_shared_assets.data_transfer.transfer_directory(source, destination, num_threads=1, *, verify_integrity=False, remove_source=False, progress=False)

Copies the contents of the input source directory to the destination directory while preserving the underlying directory hierarchy.

Notes

This function recreates the moved directory hierarchy on the destination if the hierarchy does not exist. This is done before copying the files.

The function executes a multithreaded copy operation and does not by default remove the source data after the copy is complete.

If the function is configured to verify the transferred data’s integrity, it generates an xxHash-128 checksum of the data before and after the transfer and compares the two checksums to detect data corruption.

Parameters:

source (Path) – The path to the directory to be transferred.
destination (Path) – The path to the destination directory where to move the contents of the source directory.
num_threads (int, default: 1) – The number of threads to use for the parallel file transfer. Setting this value to a number below 1 instructs the function to use all available CPU threads.
verify_integrity (bool, default: False) – Determines whether to perform integrity verification for the transferred files.
remove_source (bool, default: False) – Determines whether to remove the source directory after the transfer is complete and (optionally) verified.
progress (bool, default: False) – Determines whether to track the transfer progress using a progress bar.

Raises:

RuntimeError – If the transferred files do not pass the xxHas3-128 checksum integrity verification.

Return type:

None

Data and Configuration Assets

This package provides the assets for storing the data acquired in the Sun lab and configuring all lab’s data workflow components.

class sl_shared_assets.data_classes.AcquisitionSystems(*values)

Bases: StrEnum

Defines the data acquisition systems currently used in the Sun lab.

MESOSCOPE_VR = 'mesoscope': This system is built around the 2-Photon Random Access Mesoscope (2P-RAM) and relies on Virtual Reality (VR) environments running in Unity game engine to conduct experiments.

class sl_shared_assets.data_classes.Cue(name, code, length_cm)

Bases: object

Defines a single visual cue used in the experiment task’s Virtual Reality (VR) environment.

Notes

Each cue has a unique name (used in the Unity segment (prefab) definitions) and a unique uint8 code (used during MQTT communication and analysis). Cues are not loaded as individual prefabs - they are baked into segment prefabs.

code: int: The unique uint8 code (0-255) that identifies the cue during MQTT communication and data analysis.

length_cm: float: The length of the cue in centimeters.

name: str: The visual identifier for the cue (e.g., ‘A’, ‘B’, ‘Gray’). Used to reference the cue in segment definitions.

class sl_shared_assets.data_classes.DatasetData(name, project, session_type, acquisition_system, sessions=<factory>, tracking_data=<factory>, dataset_data_path=<factory>, _session_data_cache=<factory>)

Bases: YamlConfig

Defines the structure and the metadata of an analysis dataset.

An analysis dataset aggregates multiple data acquisition sessions of the same type, recorded across different animals by the same acquisition system. This class encapsulates the information necessary to access the dataset’s assembled (forged) data stored on disk and functions as the entry point for all interactions with the dataset.

Notes

Do not initialize this class directly. Instead, use the create() method when creating new datasets or the load() method when accessing data for an existing dataset.

Datasets are created using a pre-filtered set of session + animal pairs, typically obtained through the session filtering functionality in sl-forgery. The dataset stores only the assembled data, not raw or processed data.

acquisition_system: str | AcquisitionSystems: The name of the data acquisition system used to acquire all sessions in the dataset.

property animals: tuple[str, ...]: Returns a tuple of unique animal identifiers included in the dataset.

classmethod create(name, project, session_type, acquisition_system, sessions, datasets_root)

Creates a new analysis dataset and initializes its data structure on disk.

Notes

To access the data of an already existing dataset, use the load() method.

Parameters:

name (str) – The unique name for the dataset.
project (str) – The name of the project from which the dataset’s sessions originate.
session_type (SessionTypes | str) – The type of data acquisition sessions included in the dataset.
acquisition_system (AcquisitionSystems | str) – The name of the data acquisition system used to acquire all sessions included in the dataset.
sessions (tuple[SessionMetadata, ...] | set[SessionMetadata]) – The set of SessionMetadata instances that define the sessions whose data should be included in the dataset.
datasets_root (Path) – The path to the root directory where to create the dataset’s hierarchy.

Return type:

DatasetData

Returns:

An initialized DatasetData instance that stores the structure and the metadata of the created dataset.

Raises:

ValueError – If the session_type or acquisition_system is invalid, or if no sessions are provided.
FileExistsError – If a dataset with the same name already exists.

dataset_data_path: Path: The path to the dataset.yaml file cached to disk.

get_session_data(animal, session)

Returns the DatasetSessionData instance for the specified session.

Parameters:

animal (str) – The unique identifier of the animal that participated in the session.
session (str) – The unique identifier of the session for which to return the data.

Return type:

DatasetSessionData

Returns:

The DatasetSessionData instance containing the paths to the session’s data folder and memory-mapped data files.

Raises:

ValueError – If the specified session is not found in the dataset.

get_sessions_for_animal(animal)

Returns the identification data for all sessions for the specified animal packaged into SessionMetadata instances.

Parameters:: animal (str) – The unique identifier of the animal for which to retrieve the session identification data.
Return type:: tuple[SessionMetadata, ...]
Returns:: A tuple of SessionMetadata instances for the specified animal.

classmethod load(dataset_path)

Loads the target dataset’s data from the specified dataset.yaml file.

Notes

To create a new dataset, use the create() method.

This method memory-maps the data.feather and metadata.feather files for each loaded session as Polars dataframes.

Parameters:: dataset_path (Path) – The path to the directory where to search for the dataset.yaml file. Typically, this is the path to the root dataset directory.
Return type:: DatasetData
Returns:: An initialized DatasetData instance that stores the loaded dataset’s data.
Raises:: FileNotFoundError – If multiple or no ‘dataset.yaml’ file instances are found under the input directory.

name: str: The unique name of the dataset.

project: str: The name of the project from which the dataset’s sessions originate.

save()

Caches the instance’s data to the dataset’s root directory as a ‘dataset.yaml’ file.

Return type:: None

Notes

This method releases all memory-mapped dataframes before saving and resets them to None.

session_type: str | SessionTypes: The type of data acquisition sessions included in the dataset. All sessions in a dataset must be of the same type.

sessions: tuple[SessionMetadata, ...]: Stores the SessionMetadata instances that define the metadata used to identify each session included in the dataset.

tracking_data: DatasetTrackingData: Defines the dataset’s tracking data hierarchy for forging and multi-day processing pipelines.

class sl_shared_assets.data_classes.DatasetSessionData(session_path=PosixPath('.'), data_path=PosixPath('.'), metadata_path=PosixPath('.'), data=None, metadata=None)

Bases: object

Provides the paths and access to the assembled data files for a single session within the dataset.

Notes

Each session in the dataset has its own directory containing the forged data and metadata feather files, in addition to other supplementary data files.

As part of loading the data, this class memory-maps the .feather files to Polars dataframes for efficient access.

data: DataFrame | None = None: The Polars dataframe that stores the session’s data memory-mapped from the dataset .feather file.

data_path: Path = PosixPath('.'): The path to the data.feather file containing the assembled and time-aligned session-specific dataset.

load_data()

Loads the session’s data by memory-mapping its feather files as Polars dataframes.

Return type:: None

make_directories()

Ensures that the session directory exists, creating it if missing.

Return type:: None

metadata: DataFrame | None = None: The Polars dataframe that stores the session’s metadata memory-mapped from the metadata .feather file.

metadata_path: Path = PosixPath('.'): The path to the metadata.feather file containing session’s metadata.

release_data()

Releases the memory-mapped dataframes by setting them to None.

Return type:: None

resolve_paths(session_directory)

Resolves all paths managed by the class instance.

Parameters:: session_directory (Path) – The path to the session’s directory within the dataset hierarchy.
Return type:: None

session_path: Path = PosixPath('.'): The path to the session’s directory within the dataset hierarchy (dataset/animal/session).

class sl_shared_assets.data_classes.DatasetTrackers(*values)

Bases: StrEnum

Defines the tracker files used by dataset forging pipelines currently supported by the Sun lab data workflow.

FORGING = 'forging.yaml': The tracker file used by the dataset forging pipeline.

MULTIDAY = 'multiday.yaml': The tracker file used by the multi-day suite2p registration pipeline.

class sl_shared_assets.data_classes.DatasetTrackingData(tracking_data_path=PosixPath('.'))

Bases: object

Provides the path to the directory that stores the .yaml and .lock files used by ProcessingTracker instances to track the runtime status of the dataset forging and multi-day processing pipelines.

make_directories()

Ensures that the root directory exists, creating it if missing.

Return type:: None

resolve_paths(root_directory_path)

Resolves all paths managed by the class instance based on the input root directory path.

Parameters:: root_directory_path (Path) – The path to the top-level tracking data directory of the dataset’s data hierarchy.
Return type:: None

tracking_data_path: Path = PosixPath('.'): The path to the root directory that stores the dataset’s tracking data.

class sl_shared_assets.data_classes.DrugData(lactated_ringers_solution_volume_ml, lactated_ringers_solution_code, ketoprofen_volume_ml, ketoprofen_code, buprenorphine_volume_ml, buprenorphine_code, dexamethasone_volume_ml, dexamethasone_code)

Bases: object

Stores information about all medical substances (drugs) administered to the subject before, during, and immediately after the surgical intervention.

buprenorphine_code: str: The manufacturer code or internal reference code for the administered buprenorphine.

buprenorphine_volume_ml: float: The volume of diluted buprenorphine administered during the surgery, in milliliters.

dexamethasone_code: str: The manufacturer code or internal reference code for the administered dexamethasone.

dexamethasone_volume_ml: float: The volume of diluted dexamethasone administered during the surgery, in milliliters.

ketoprofen_code: str: The manufacturer code or internal reference code for the administered ketoprofen.

ketoprofen_volume_ml: float: The volume of diluted ketoprofen administered during the surgery, in milliliters.

lactated_ringers_solution_code: str: The manufacturer code or internal reference code for the administered Lactated Ringer’s Solution (LRS).

lactated_ringers_solution_volume_ml: float: The volume of Lactated Ringer’s Solution (LRS) administered during the surgery, in milliliters.

class sl_shared_assets.data_classes.GasPuffTrial(segment_name, stimulus_trigger_zone_start_cm, stimulus_trigger_zone_end_cm, stimulus_location_cm, show_stimulus_collision_boundary=False, cue_sequence=<factory>, trial_length_cm=0.0, puff_duration_ms=100, occupancy_duration_ms=1000)

Bases: _MesoscopeBaseTrial

Defines a trial that delivers N2 gas puffs (aversive stimuli) when the animal fails to meet occupancy duration.

Notes

Trigger mode: The animal must occupy the stimulus trigger zone for the specified duration to disarm the stimulus boundary and avoid the gas puff. If the animal exits the zone early or collides with the boundary before meeting the occupancy threshold, the gas puff is delivered. Guidance mode: When the animal exits the zone early, an OccupancyFailed message is emitted, allowing sl-experiment to block movement and prevent the animal from reaching the armed boundary.

occupancy_duration_ms: int = 1000: The time, in milliseconds, the animal must occupy the trigger zone to disarm the stimulus boundary and avoid the gas puff.

puff_duration_ms: int = 100: The duration, in milliseconds, for which to deliver the N2 gas puff when the animal fails the trial.

class sl_shared_assets.data_classes.ImplantData(implant, implant_target, implant_code, implant_ap_coordinate_mm, implant_ml_coordinate_mm, implant_dv_coordinate_mm)

Bases: object

Stores information about a single implantation procedure performed during the surgical intervention.

Multiple ImplantData instances can be used at the same time if the surgery involved multiple implantation procedures.

implant: str: The descriptive name of the implant.

implant_ap_coordinate_mm: float: The implant’s antero-posterior stereotactic coordinate, in millimeters, relative to bregma.

implant_code: str: The manufacturer code or internal reference code for the implant.

implant_dv_coordinate_mm: float: The implant’s dorsal-ventral stereotactic coordinate, in millimeters, relative to bregma.

implant_ml_coordinate_mm: float: The implant’s medial-lateral stereotactic coordinate, in millimeters, relative to bregma.

implant_target: str: The name of the brain region or cranium section targeted by the implant.

class sl_shared_assets.data_classes.InjectionData(injection, injection_target, injection_volume_nl, injection_code, injection_ap_coordinate_mm, injection_ml_coordinate_mm, injection_dv_coordinate_mm)

Bases: object

Stores information about a single injection performed during the surgical intervention.

Multiple InjectionData instances can be used at the same time if the surgery involved multiple injections.

injection: str: The descriptive name of the injection.

injection_ap_coordinate_mm: float: The injection’s antero-posterior stereotactic coordinate, in millimeters, relative to bregma.

injection_code: str: The manufacturer code or internal reference code for the injected substance.

injection_dv_coordinate_mm: float: The injection’s dorsal-ventral stereotactic coordinate, in millimeters, relative to bregma.

injection_ml_coordinate_mm: float: The injection’s medial-lateral stereotactic coordinate, in millimeters, relative to bregma.

injection_target: str: The name of the brain region targeted by the injection.

injection_volume_nl: float: The volume of substance, in nanoliters, delivered during the injection.

class sl_shared_assets.data_classes.LickTrainingDescriptor(experimenter, mouse_weight_g, minimum_reward_delay_s=6, maximum_reward_delay_s=18, maximum_water_volume_ml=1.0, maximum_training_time_min=20, maximum_unconsumed_rewards=1, water_reward_size_ul=5.0, reward_tone_duration_ms=300, dispensed_water_volume_ml=0.0, pause_dispensed_water_volume_ml=0.0, experimenter_given_water_volume_ml=0.0, preferred_session_water_volume_ml=0.0, incomplete=True, experimenter_notes='Replace this with your notes.')

Bases: YamlConfig

Stores the task and outcome information specific to lick training sessions that use the Mesoscope-VR system.

dispensed_water_volume_ml: float = 0.0: The total water volume, in milliliters, dispensed during runtime. This excludes the water volume dispensed during the paused (idle) state.

experimenter: str: The ID of the experimenter running the session.

experimenter_given_water_volume_ml: float = 0.0: The additional volume of water, in milliliters, administered by the experimenter to the animal after the session.

experimenter_notes: str = 'Replace this with your notes.': Stores the experimenter’s notes made during runtime.

incomplete: bool = True: Tracks whether the session’s data is complete and eligible for unsupervised data processing.

maximum_reward_delay_s: int = 18: The maximum delay, in seconds, that can separate the delivery of two consecutive water rewards.

maximum_training_time_min: int = 20: The maximum time, in minutes, the system is allowed to run the training.

maximum_unconsumed_rewards: int = 1: The maximum number of consecutive rewards that can be delivered without the animal consuming them. If the animal receives this many rewards without licking (consuming) them, reward delivery is paused until the animal consumes the delivered rewards.

maximum_water_volume_ml: float = 1.0: The maximum volume of water the system is allowed to dispense during training.

minimum_reward_delay_s: int = 6: The minimum delay, in seconds, that can separate the delivery of two consecutive water rewards.

mouse_weight_g: float: The weight of the animal, in grams, at the beginning of the session.

pause_dispensed_water_volume_ml: float = 0.0: The total water volume, in milliliters, dispensed during the paused (idle) state.

preferred_session_water_volume_ml: float = 0.0: The volume of water, in milliliters, the animal should be receiving during the session runtime if its performance matches experimenter-specified threshold.

reward_tone_duration_ms: int = 300: The duration, in milliseconds, of the auditory tone played to the animal when it receives water rewards.

water_reward_size_ul: float = 5.0: The volume of water, in microliters, dispensed to the animal when it achieves the required running speed and duration thresholds.

class sl_shared_assets.data_classes.ManagingTrackers(*values)

Bases: StrEnum

Defines the tracker files used by the data managing pipelines currently supported by the Sun lab data workflow.

CHECKSUM = 'checksum.yaml': The tracker file used by the checksum resolution pipeline.

MANIFEST = 'manifest.yaml': The tracker file used by the project manifest generation pipeline.

class sl_shared_assets.data_classes.MesoscopeCameras(face_camera_index=0, body_camera_index=1, face_camera_quantization=20, face_camera_preset=7, body_camera_quantization=20, body_camera_preset=7)

Bases: object

Stores the video camera configuration of the Mesoscope-VR data acquisition system.

body_camera_index: int = 1: The index of the body camera in the list of all available Harvester-managed cameras.

body_camera_preset: int = 7: The encoding speed preset used by the body camera to encode acquired frames as video files. Must be one of the valid members of the EncoderSpeedPresets enumeration from the ataraxis-video-system library.

body_camera_quantization: int = 20: The quantization parameter used by the body camera to encode acquired frames as video files.

face_camera_index: int = 0: The index of the face camera in the list of all available Harvester-managed cameras.

face_camera_preset: int = 7: The encoding speed preset used by the face camera to encode acquired frames as video files. Must be one of the valid members of the EncoderSpeedPresets enumeration from the ataraxis-video-system library.

face_camera_quantization: int = 20: The quantization parameter used by the face camera to encode acquired frames as video files.

class sl_shared_assets.data_classes.MesoscopeExperimentConfiguration(cues, segments, trial_structures, experiment_states, vr_environment, unity_scene_name, cue_offset_cm=0.0)

Bases: YamlConfig

Defines an experiment session that uses the Mesoscope_VR data acquisition system.

This is the unified configuration that serves both the data acquisition system (sl-experiment), the analysis pipeline (sl-forgery), and the Unity VR environment (sl-unity-tasks).

cue_offset_cm: float = 0.0: Specifies the offset of the animal’s starting position relative to the Virtual Reality (VR) environment’s cue sequence origin, in centimeters.

cues: list[Cue]: Defines the Virtual Reality environment wall cues used in the experiment.

experiment_states: dict[str, MesoscopeExperimentState]: Defines the experiment’s flow by specifying the sequence of experiment and data acquisition system states executed during runtime.

segments: list[Segment]: Defines the Virtual Reality environment segments (sequences of wall cues) for the Unity corridor system.

trial_structures: dict[str, WaterRewardTrial | GasPuffTrial]: Defines experiment’s structure by specifying the types of trials used by the phases (states) of the experiment.

unity_scene_name: str: The name of the Virtual Reality task (Unity Scene) used during the experiment.

vr_environment: VREnvironment: Defines the Virtual Reality corridor used during the experiment.

class sl_shared_assets.data_classes.MesoscopeExperimentDescriptor(experimenter, mouse_weight_g, maximum_unconsumed_rewards=1, dispensed_water_volume_ml=0.0, pause_dispensed_water_volume_ml=0.0, experimenter_given_water_volume_ml=0.0, preferred_session_water_volume_ml=0.0, incomplete=True, experimenter_notes='Replace this with your notes.')

Bases: YamlConfig

Stores the task and outcome information specific to experiment sessions that use the Mesoscope-VR system.

dispensed_water_volume_ml: float = 0.0: The total water volume, in milliliters, dispensed during runtime. This excludes the water volume dispensed during the paused (idle) state.

experimenter: str: The ID of the experimenter running the session.

experimenter_given_water_volume_ml: float = 0.0: The additional volume of water, in milliliters, administered by the experimenter to the animal after the session.

experimenter_notes: str = 'Replace this with your notes.': Stores the experimenter’s notes made during runtime.

incomplete: bool = True: Tracks whether the session’s data is complete and eligible for unsupervised data processing.

maximum_unconsumed_rewards: int = 1: The maximum number of consecutive rewards that can be delivered without the animal consuming them. If the animal receives this many rewards without licking (consuming) them, reward delivery is paused until the animal consumes the delivered rewards.

mouse_weight_g: float: The weight of the animal, in grams, at the beginning of the session.

pause_dispensed_water_volume_ml: float = 0.0: The total water volume, in milliliters, dispensed during the paused (idle) state.

preferred_session_water_volume_ml: float = 0.0: The volume of water, in milliliters, the animal should be receiving during the session runtime if its performance matches experimenter-specified threshold.

class sl_shared_assets.data_classes.MesoscopeExperimentState(experiment_state_code, system_state_code, state_duration_s, supports_trials=True, reinforcing_initial_guided_trials=0, reinforcing_recovery_failed_threshold=0, reinforcing_recovery_guided_trials=0, aversive_initial_guided_trials=0, aversive_recovery_failed_threshold=0, aversive_recovery_guided_trials=0)

Bases: object

Defines the structure and runtime parameters of an experiment state (phase).

aversive_initial_guided_trials: int = 0: The number of aversive trials after the onset of the experiment state that use the guidance mode.

aversive_recovery_failed_threshold: int = 0: The number of sequentially failed aversive trials after which to enable the recovery guidance mode.

aversive_recovery_guided_trials: int = 0: The number of guided aversive trials to use in the recovery guidance mode.

experiment_state_code: int: The unique identifier code of the experiment state.

reinforcing_initial_guided_trials: int = 0: The number of reinforcing trials after the onset of the experiment state that use the guidance mode.

reinforcing_recovery_failed_threshold: int = 0: The number of sequentially failed reinforcing trials after which to enable the recovery guidance mode.

reinforcing_recovery_guided_trials: int = 0: The number of guided reinforcing trials to use in the recovery guidance mode.

state_duration_s: float: The time, in seconds, to maintain the experiment state while executing the experiment.

supports_trials: bool = True: Determines whether trials are executed during this experiment state. When False, no trial-related processing occurs during this phase.

system_state_code: int: The data acquisition system’s state (configuration snapshot) code associated with the experiment state.

class sl_shared_assets.data_classes.MesoscopeExternalAssets(headbar_port='/dev/ttyUSB0', lickport_port='/dev/ttyUSB1', wheel_port='/dev/ttyUSB2', unity_ip='127.0.0.1', unity_port=1883)

Bases: object

Stores the third-party asset configuration of the Mesoscope-VR data acquisition system.

headbar_port: str = '/dev/ttyUSB0': The USB port used by the HeadBar Zaber motor controllers.

lickport_port: str = '/dev/ttyUSB1': The USB port used by the LickPort Zaber motor controllers.

unity_ip: str = '127.0.0.1': The IP address of the MQTT broker used to communicate with the Unity game engine.

unity_port: int = 1883: The port number of the MQTT broker used to communicate with the Unity game engine.

wheel_port: str = '/dev/ttyUSB2': The USB port used by the Wheel Zaber motor controllers.

class sl_shared_assets.data_classes.MesoscopeFileSystem(root_directory=PosixPath('.'), server_directory=PosixPath('.'), nas_directory=PosixPath('.'), mesoscope_directory=PosixPath('.'))

Bases: object

Stores the filesystem configuration of the Mesoscope-VR data acquisition system.

mesoscope_directory: Path = PosixPath('.'): The absolute path to the local-filesystem-mounted directory where all Mesoscope-acquired data is aggregated during acquisition by the PC that manages the Mesoscope during runtime.

nas_directory: Path = PosixPath('.'): The absolute path to the local-filesystem-mounted directory where all projects are stored on the NAS backup storage volume.

root_directory: Path = PosixPath('.'): The absolute path to the directory where all projects are stored on the main data acquisition system PC.

server_directory: Path = PosixPath('.'): The absolute path to the local-filesystem-mounted directory where all projects are stored on the remote compute server.

class sl_shared_assets.data_classes.MesoscopeGoogleSheets(surgery_sheet_id='', water_log_sheet_id='')

Bases: object

Stores the identifiers for the Google Sheets used by the Mesoscope-VR data acquisition system.

surgery_sheet_id: str = '': The identifier of the Google Sheet that stores information about surgical interventions performed on the animals that participate in data acquisition sessions.

water_log_sheet_id: str = '': The identifier of the Google Sheet that stores information about water restriction and handling for all animals that participate in data acquisition sessions.

class sl_shared_assets.data_classes.MesoscopeHardwareState(cm_per_pulse=None, maximum_brake_strength=None, minimum_brake_strength=None, lick_threshold=None, valve_scale_coefficient=None, valve_nonlinearity_exponent=None, torque_per_adc_unit=None, screens_initially_on=None, recorded_mesoscope_ttl=None, delivered_gas_puffs=None, system_state_codes=None)

Bases: YamlConfig

Stores configuration parameters (states) of the Mesoscope-VR system hardware modules used during training or experiment runtimes.

Notes

All hardware parameters are stored using the appropriate datatypes and rounding methods that ensure their complete equivalence to the values used by the data acquisition system during runtime.

cm_per_pulse: float | None = None: The conversion factor used to translate encoder pulses into centimeters.

delivered_gas_puffs: bool | None = None: Tracks whether the session delivered any gas puffs to the animal.

lick_threshold: int | None = None: Determines the threshold, in 12-bit Analog to Digital Converter (ADC) units reported by the lick sensor, for considering the reported signal a lick.

maximum_brake_strength: float | None = None: The braking torque, in Newton centimeters, applied by the brake to the edge of the running wheel when it is maximally engaged.

minimum_brake_strength: float | None = None: The braking torque, in Newton centimeters, applied by the brake to the edge of the running wheel when it is completely disengaged.

recorded_mesoscope_ttl: bool | None = None: Tracks whether the session recorded brain activity data with the mesoscope.

screens_initially_on: bool | None = None: Stores the initial state of the Virtual Reality screens at the beginning of the session’s runtime.

system_state_codes: dict[str, int] | None = None: Maps integer state-codes used by the Mesoscope-VR system to communicate its states (system states) to human-readable state names.

torque_per_adc_unit: float | None = None: The conversion factor used to translate torque values reported by the sensor as 12-bit Analog to Digital Converter (ADC) units into Newton centimeters (N·cm).

valve_nonlinearity_exponent: float | None = None: The nonlinearity exponent of the power law equation that describes the relationship between the time the valve is kept open and the dispensed water volume.

valve_scale_coefficient: float | None = None: The scale coefficient of the power law equation that describes the relationship between the time the valve is kept open and the dispensed water volume.

class sl_shared_assets.data_classes.MesoscopeMicroControllers(actor_port='/dev/ttyACM0', sensor_port='/dev/ttyACM1', encoder_port='/dev/ttyACM2', keepalive_interval_ms=500, minimum_brake_strength_g_cm=43.2047, maximum_brake_strength_g_cm=1152.1246, wheel_diameter_cm=15.0333, lick_threshold_adc=600, lick_signal_threshold_adc=300, lick_delta_threshold_adc=300, lick_averaging_pool_size=2, torque_baseline_voltage_adc=2048, torque_maximum_voltage_adc=3443, torque_sensor_capacity_g_cm=720.0779, torque_report_cw=True, torque_report_ccw=True, torque_signal_threshold_adc=150, torque_delta_threshold_adc=100, torque_averaging_pool_size=4, wheel_encoder_ppr=8192, wheel_encoder_report_cw=False, wheel_encoder_report_ccw=True, wheel_encoder_delta_threshold_pulse=15, wheel_encoder_polling_delay_us=500, cm_per_unity_unit=10.0, screen_trigger_pulse_duration_ms=500, sensor_polling_delay_ms=1, valve_calibration_data=((15000, 1.1), (30000, 3.0), (45000, 6.25), (60000, 10.9)))

Bases: object

Stores the microcontroller configuration of the Mesoscope-VR data acquisition system.

actor_port: str = '/dev/ttyACM0': The USB port used by the Actor Microcontroller.

cm_per_unity_unit: float = 10.0: The length of each Virtual Reality (VR) environment’s distance ‘unit’ (Unity unit) in real-world centimeters.

encoder_port: str = '/dev/ttyACM2': The USB port used by the Encoder Microcontroller.

keepalive_interval_ms: int = 500: The interval, in milliseconds, at which the microcontrollers are expected to receive and send the keepalive messages used to ensure that all controllers function as expected during runtime.

lick_averaging_pool_size: int = 2: The number of lick sensor readouts to average together to produce the final lick sensor readout value.

lick_delta_threshold_adc: int = 300: The minimum absolute difference between two consecutive lick sensor readouts, in raw analog units recorded by a 3.3 Volt 12-bit Analog-to-Digital-Converter (ADC), for the change to be reported to the PC.

lick_signal_threshold_adc: int = 300: The minimum voltage, in raw analog units recorded by a 3.3 Volt 12-bit Analog-to-Digital-Converter (ADC), reported to the PC as a non-zero value. Voltages below this level are interpreted as ‘no-lick’ noise and are pulled to 0.

lick_threshold_adc: int = 600: The threshold voltage, in raw analog units recorded by a 3.3 Volt 12-bit Analog-to-Digital-Converter (ADC), interpreted as the animal’s tongue contacting the lick sensor.

maximum_brake_strength_g_cm: float = 1152.1246: The torque applied by the running wheel brake at the maximum operational voltage, in gram centimeter.

mesoscope_frame_averaging_pool_size = 0: The number of digital pin readouts to average together when determining the current logic level of the incoming TTL signal sent by the mesoscope at the onset of each frame’s acquisition.

minimum_brake_strength_g_cm: float = 43.2047: The torque applied by the running wheel brake at the minimum operational voltage, in gram centimeter.

screen_trigger_pulse_duration_ms: int = 500: The duration, in milliseconds, of the TTL pulse used to toggle the VR screen power state.

sensor_polling_delay_ms: int = 1: The delay, in milliseconds, between any two successive readouts of any sensor other than the encoder.

sensor_port: str = '/dev/ttyACM1': The USB port used by the Sensor Microcontroller.

torque_averaging_pool_size: int = 4: The number of torque sensor readouts to average together to produce the final torque sensor readout value.

torque_baseline_voltage_adc: int = 2048: The voltage level, in raw analog units measured by a 3.3 Volt 12-bit Analog-to-Digital-Converter (ADC) after the AD620 amplifier, that corresponds to no torque (0) readout.

torque_delta_threshold_adc: int = 100: The minimum absolute difference between two consecutive torque sensor readouts, in raw analog units recorded by a 3.3 Volt 12-bit Analog-to-Digital-Converter (ADC), for the change to be reported to the PC.

torque_maximum_voltage_adc: int = 3443: The voltage level, in raw analog units measured by a 3.3 Volt 12-bit Analog-to-Digital-Converter (ADC) after the AD620 amplifier, that corresponds to the absolute maximum torque detectable by the sensor.

torque_report_ccw: bool = True: Determines whether the sensor should report torque in the Counter-Clockwise (CCW) direction.

torque_report_cw: bool = True: Determines whether the torque sensor should report torques in the Clockwise (CW) direction.

torque_sensor_capacity_g_cm: float = 720.0779: The maximum torque detectable by the sensor, in grams centimeter (g cm).

torque_signal_threshold_adc: int = 150: The minimum voltage, in raw analog units recorded by a 3.3 Volt 12-bit Analog-to-Digital-Converter (ADC), reported to the PC as a non-zero value. Voltages below this level are interpreted as noise and are pulled to 0.

valve_calibration_data: dict[int | float, int | float] | tuple[tuple[int | float, int | float], ...] = ((15000, 1.1), (30000, 3.0), (45000, 6.25), (60000, 10.9)): Maps water delivery solenoid valve open times, in microseconds, to the dispensed volumes of water, in microliters.

wheel_diameter_cm: float = 15.0333: The diameter of the running wheel, in centimeters.

wheel_encoder_delta_threshold_pulse: int = 15: The minimum absolute difference between two consecutive encoder readouts, in encoder pulse counts, for the change to be reported to the PC.

wheel_encoder_polling_delay_us: int = 500: The delay, in microseconds, between consecutive encoder state readouts.

wheel_encoder_ppr: int = 8192: The resolution of the wheel’s quadrature encoder, in Pulses Per Revolution (PPR).

wheel_encoder_report_ccw: bool = True: Determines whether the encoder should report rotation in the CounterClockwise (CCW) direction.

wheel_encoder_report_cw: bool = False: Determines whether the encoder should report rotation in the Clockwise (CW) direction.

class sl_shared_assets.data_classes.MesoscopePositions(mesoscope_x=0.0, mesoscope_y=0.0, mesoscope_roll=0.0, mesoscope_z=0.0, mesoscope_fast_z=0.0, mesoscope_tip=0.0, mesoscope_tilt=0.0, laser_power_mw=0.0, red_dot_alignment_z=0.0)

Bases: YamlConfig

Stores the positions of real and virtual Mesoscope imaging axes reused between experiment sessions that use the Mesoscope-VR system.

laser_power_mw: float = 0.0: The laser excitation power at the sample, in milliwatts.

mesoscope_fast_z: float = 0.0: The ScanImage’s FastZ (virtual Z-axis) position, in micrometers.

mesoscope_roll: float = 0.0: The Mesoscope objective’s Roll-axis position, in degrees.

mesoscope_tilt: float = 0.0: The ScanImage’s Tip position, in degrees.

mesoscope_tip: float = 0.0: The ScanImage’s Tilt position, in degrees..

mesoscope_x: float = 0.0: The Mesoscope objective’s X-axis position, in micrometers.

mesoscope_y: float = 0.0: The Mesoscope objective’s Y-axis position, in micrometers.

mesoscope_z: float = 0.0: The Mesoscope objective’s Z-axis position, in micrometers.

red_dot_alignment_z: float = 0.0: The Mesoscope objective’s Z-axis position, in micrometers, used for the red-dot alignment procedure.

class sl_shared_assets.data_classes.MesoscopeSystemConfiguration(name='mesoscope', filesystem=<factory>, sheets=<factory>, cameras=<factory>, microcontrollers=<factory>, assets=<factory>)

Bases: YamlConfig

Defines the hardware and software asset configuration for the Mesoscope-VR data acquisition system.

assets: MesoscopeExternalAssets: Stores the third-party hardware and firmware assets configuration.

cameras: MesoscopeCameras: Stores the video cameras configuration.

filesystem: MesoscopeFileSystem: Stores the filesystem configuration.

microcontrollers: MesoscopeMicroControllers: Stores the microcontrollers configuration.

name: str = 'mesoscope': The descriptive name of the data acquisition system.

save(path)

Saves the instance’s data to disk as a .YAML file.

Parameters:: path (Path) – The path to the .YAML file to save the data to.
Return type:: None

sheets: MesoscopeGoogleSheets: Stores the identifiers and access credentials for the Google Sheets.

class sl_shared_assets.data_classes.ProcedureData(surgery_start_us, surgery_end_us, surgeon, protocol, surgery_notes, post_op_notes, surgery_quality=0)

Bases: object

Stores general information about the surgical intervention.

post_op_notes: str: The surgeon’s notes taken during the post-surgery recovery period.

protocol: str: The number (ID) of the experiment protocol used during the surgery.

surgeon: str: The surgeon’s name or ID. If the intervention was carried out by multiple surgeons, the data for all surgeons is stored as part of the same string.

surgery_end_us: int: The surgery’s stop date and time as microseconds elapsed since UTC epoch onset.

surgery_notes: str: The surgeon’s notes taken during the surgery.

surgery_quality: int = 0: The quality of the surgical intervention ob a scake from 0 to 3 inclusive. 0 indicates unusable (bad) result, 1 indicates usable result that does not meet the publication threshold, 2 indicates publication-grade result, 3 indicates high-tier publication grade result.

surgery_start_us: int: The surgery’s start date and time as microseconds elapsed since UTC epoch onset.

class sl_shared_assets.data_classes.ProcessedData(processed_data_path=PosixPath('.'), camera_data_path=PosixPath('.'), mesoscope_data_path=PosixPath('.'), behavior_data_path=PosixPath('.'))

Bases: object

Provides the paths to the directories and files that store the data generated by the processing pipelines from the raw data.

behavior_data_path: Path = PosixPath('.'): The path to the directory that contains the non-video behavior data extracted from the .npz log archives by the sl-behavior log processing pipeline.

camera_data_path: Path = PosixPath('.'): The path to the directory that contains video tracking data generated by the Sun lab DeepLabCut-based video processing pipeline(s).

make_directories()

Ensures that all major subdirectories and the root directory exist, creating any missing directories.

Return type:: None

mesoscope_data_path: Path = PosixPath('.'): The path to the directory that contains processed brain activity (cell) data generated by sl-suite2p processing pipelines (single-day and multi-day).

processed_data_path: Path = PosixPath('.'): The path to the root directory that stores the session’s processed data.

resolve_paths(root_directory_path)

Resolves all paths managed by the class instance based on the input root directory path.

Parameters:: root_directory_path (Path) – The path to the top-level processed data directory of the session’s data hierarchy.
Return type:: None

class sl_shared_assets.data_classes.ProcessingPipelines(*values)

Bases: StrEnum

Defines the data processing pipelines currently supported by the Sun lab data workflow.

ADOPTION = 'adoption': The session data adoption pipeline. This pipeline copies session’s data stored in the shared Sun lab data directory on the remote compute server to the user’s working directory. This is the entry-point for all further interactions with the data, as it gives the calling user the permission to modify the copied data.

BEHAVIOR = 'behavior': The behavior data processing pipeline. This pipeline parses the non-video behavior data from the .npz log archives generated by the sl-experiment library during the session’s data acquisition.

CHECKSUM = 'checksum': The session’s raw data integrity checksum verification pipeline. This pipeline verifies or regenerates the session’s data integrity checksum.

FORGING = 'forging': The dataset assembly (forging) pipeline. This pipeline integrates the single-day and multi-day data from all available sources for each processed session into a unified analysis Dataset structure. The assembled dataset then serves as an entry-point for all further data analysis tasks.

MANIFEST = 'manifest': The project manifest generation pipeline. This pipeline generates a .feather file that stores the snapshot of the target project’s data processing state. The created manifest file is then used as the entry-point for all other data processing pipelines other than the ADOPTION pipeline.

MULTIDAY = 'multiday': The suite2p multi-day cell tracking pipeline. This pipeline uses the output of the ‘suite2p’ pipeline for each of the processed sessions to identify and track the activity of stably present cells across days.

SUITE2P = 'suite2p': The suite2p calcium imaging data processing pipeline. This pipeline motion-corrects, identifies, and extracts the calcium fluorescence traces from cells recorded with 2-Photon Random Access Mesoscope (2P-RAM) during the session’s data acquisition.

VIDEO = 'video': The video data processing pipeline. This pipeline identifies and extracts the animal’s pose and movement data from the frames recorded by behavior video cameras during the session’s data acquisition.

class sl_shared_assets.data_classes.ProcessingStatus(*values)

Bases: IntEnum

Defines the status codes used by the ProcessingTracker instances to communicate the runtime state of each job making up the managed data processing pipeline.

FAILED = 3: The job encountered a runtime error and was not completed.

RUNNING = 1: The job is currently being executed.

SCHEDULED = 0: The job is scheduled for execution.

SUCCEEDED = 2: The job has been completed.

class sl_shared_assets.data_classes.ProcessingTracker(file_path, jobs=<factory>)

Bases: YamlConfig

Tracks the state of a data processing pipeline and provides tools for communicating this state between multiple processes and host-machines.

Note

All modifications to the tracker file require the acquisition of the .lock file, which ensures exclusive access to the tracker’s data, allowing multiple independent processes (jobs) to safely work with the same tracker file.

property complete: bool

Returns True if the tracked processing pipeline has been completed successfully.

Notes

The pipeline is considered complete if all jobs have been marked as succeeded.

complete_job(job_id)

Marks a target job as successfully completed.

Parameters:

job_id (str) – The unique identifier of the job to mark as complete.

Raises:

TimeoutError – If the .LOCK file for the tracker .YAML file cannot be acquired within the timeout period.
ValueError – If the specified job ID is not found in the managed tracker file.

Return type:

None

property encountered_error: bool: Returns True if the tracked processing pipeline has been terminated due to a runtime error.

Note

The pipeline is considered to have encountered an error if any job has been marked as failed.

fail_job(job_id)

Marks the target job as failed.

Parameters:

job_id (str) – The unique identifier of the job to mark as failed.

Raises:

TimeoutError – If the .LOCK file for the tracker .YAML file cannot be acquired within the timeout period.
ValueError – If the specified job ID is not found in the managed tracker file.

Return type:

None

file_path: Path: The path to the .YAML file used to cache the tracker’s data on disk.

static generate_job_id(session_path, job_name)

Generates a unique hexadecimal job identifier based on the session’s data path and the job’s name using the xxHash64 checksum generator.

Parameters:

session_path (Path) – The path to the processed session’s data directory.
job_name (str) – The unique name for the processing job.

Return type:

str

Returns:

The unique hexadecimal identifier for the target job.

get_job_status(job_id)

Queries the current runtime status of the target job.

Parameters:

job_id (str) – The unique identifier of the job for which to query the runtime status.

Return type:

ProcessingStatus

Returns:

The current runtime status of the job.

Raises:

TimeoutError – If the .LOCK file for the tracker .YAML file cannot be acquired within the timeout period.
ValueError – If the specified job ID is not found in the managed tracker file.

initialize_jobs(job_ids)

Configures the tracker with the list of jobs to be executed during the pipeline’s runtime.

Parameters:: job_ids (list[str]) – The list of unique identifiers for all jobs that make up the tracked pipeline.
Raises:: TimeoutError – If the .LOCK file for the tracker .YAML file cannot be acquired within the timeout period.
Return type:: None

jobs: dict[str, JobState]: Maps the unique identifiers of the jobs that make up the processing pipeline to their current state and metadata.

lock_path: str: The path to the .LOCK file used to ensure thread-safe access to the tracker’s data.

reset()

Resets the tracker file to the default state.

Return type:: None

start_job(job_id)

Marks the target job as running and captures the SLURM-assigned job’s ID from the environment, if called under the SLURM job manager.

Parameters:

job_id (str) – The unique identifier of the job mark as started.

Raises:

TimeoutError – If the .LOCK file for the tracker .YAML file cannot be acquired within the timeout period.
ValueError – If the specified job ID is not found in the managed tracker file.

Return type:

None

class sl_shared_assets.data_classes.ProcessingTrackers(*values)

Bases: StrEnum

Defines the tracker files used by data processing pipelines currently supported by the Sun lab data workflow.

BEHAVIOR = 'behavior.yaml': The tracker file used by the behavior extraction pipeline.

SUITE2P = 'suite2p.yaml': The tracker file used by the suite2p processing pipeline.

VIDEO = 'video.yaml': The tracker file used by the video (DeepLabCut) processing pipeline.

class sl_shared_assets.data_classes.RawData(raw_data_path=PosixPath('.'), camera_data_path=PosixPath('.'), mesoscope_data_path=PosixPath('.'), behavior_data_path=PosixPath('.'), zaber_positions_path=PosixPath('.'), session_descriptor_path=PosixPath('.'), hardware_state_path=PosixPath('.'), surgery_metadata_path=PosixPath('.'), session_data_path=PosixPath('.'), experiment_configuration_path=PosixPath('.'), mesoscope_positions_path=PosixPath('.'), window_screenshot_path=PosixPath('.'), system_configuration_path=PosixPath('.'), checksum_path=PosixPath('.'), nk_path=PosixPath('.'))

Bases: object

Provides the paths to the directories and files that store the data acquired and losslessly preprocessed during the session’s data acquisition runtime.

behavior_data_path: Path = PosixPath('.'): The path to the directory that contains the non-video behavior data acquired during the session’s runtime.

camera_data_path: Path = PosixPath('.'): The path to the directory that contains the video camera data acquired during the session’s runtime.

checksum_path: Path = PosixPath('.'): The path to the ax_checksum.txt file that stores the xxHash-128 checksum of the data used to verify its integrity during transmission.

experiment_configuration_path: Path = PosixPath('.'): The path to the experiment_configuration.yaml file that contains the snapshot of the experiment’s configuration used during the session’s runtime. This file is only created for experiment sessions.

hardware_state_path: Path = PosixPath('.'): The path to the hardware_state.yaml file that contains the partial snapshot of the configuration parameters used by the data acquisition system’s hardware modules during the session’s runtime.

make_directories()

Ensures that all major subdirectories and the root directory exist, creating any missing directories.

Return type:: None

mesoscope_data_path: Path = PosixPath('.'): The path to the directory that contains the Mesoscope data acquired during the session’s runtime.

mesoscope_positions_path: Path = PosixPath('.'): The path to the mesoscope_positions.yaml file that contains the snapshot of the imaging axes positions used by the Mesoscope at the end of the session’s runtime.

nk_path: Path = PosixPath('.'): The path to the nk.bin file used by the sl-experiment library to mark sessions undergoing runtime initialization.

raw_data_path: Path = PosixPath('.'): The path to the root directory that stores the session’s raw data.

resolve_paths(root_directory_path)

Resolves all paths managed by the class instance based on the input root directory path.

Parameters:: root_directory_path (Path) – The path to the top-level raw data directory of the session’s data hierarchy.
Return type:: None

session_data_path: Path = PosixPath('.'): The path to the session_data.yaml file. This path is used by the SessionData instance to save itself to disk as a .yaml file.

session_descriptor_path: Path = PosixPath('.'): The path to the session_descriptor.yaml file that contains session-specific information, such as the specific task parameters and the notes made by the experimenter during the session’s runtime.

surgery_metadata_path: Path = PosixPath('.'): The path to the surgery_metadata.yaml file that contains the information about the surgical intervention(s) performed on the animal prior to the session’s runtime.

system_configuration_path: Path = PosixPath('.'): The path to the system_configuration.yaml file that contains the exact snapshot of the data acquisition system configuration parameters used to acquire the session’s data.

window_screenshot_path: Path = PosixPath('.'): The path to the .png screenshot of the ScanImagePC screen that communicates the visual snapshot of the cranial window alignment and cell appearance at the beginning of the session’s runtime.

zaber_positions_path: Path = PosixPath('.'): The path to the zaber_positions.yaml file that contains the snapshot of all Zaber motor positions at the end of the session’s runtime.

class sl_shared_assets.data_classes.RunTrainingDescriptor(experimenter, mouse_weight_g, final_run_speed_threshold_cm_s=1.5, final_run_duration_threshold_s=1.5, initial_run_speed_threshold_cm_s=0.8, initial_run_duration_threshold_s=1.5, increase_threshold_ml=0.1, run_speed_increase_step_cm_s=0.05, run_duration_increase_step_s=0.1, maximum_water_volume_ml=1.0, maximum_training_time_min=40, maximum_unconsumed_rewards=1, maximum_idle_time_s=0.3, water_reward_size_ul=5.0, reward_tone_duration_ms=300, dispensed_water_volume_ml=0.0, pause_dispensed_water_volume_ml=0.0, experimenter_given_water_volume_ml=0.0, preferred_session_water_volume_ml=0.0, incomplete=True, experimenter_notes='Replace this with your notes.')

Bases: YamlConfig

Stores the task and outcome information specific to run training sessions that use the Mesoscope-VR system.

dispensed_water_volume_ml: float = 0.0: The total water volume, in milliliters, dispensed during runtime. This excludes the water volume dispensed during the paused (idle) state.

experimenter: str: The ID of the experimenter running the session.

experimenter_given_water_volume_ml: float = 0.0: The additional volume of water, in milliliters, administered by the experimenter to the animal after the session.

experimenter_notes: str = 'Replace this with your notes.': Stores the experimenter’s notes made during runtime.

final_run_duration_threshold_s: float = 1.5: The running duration threshold, in seconds, at the end of training.

final_run_speed_threshold_cm_s: float = 1.5: The running speed threshold, in centimeters per second, at the end of training.

incomplete: bool = True: Tracks whether the session’s data is complete and eligible for unsupervised data processing.

increase_threshold_ml: float = 0.1: The threshold volume of water delivered to the animal, in milliliters, that triggers the increase in the running speed and duration thresholds.

initial_run_duration_threshold_s: float = 1.5: The initial running duration threshold, in seconds.

initial_run_speed_threshold_cm_s: float = 0.8: The initial running speed threshold, in centimeters per second.

maximum_idle_time_s: float = 0.3: The maximum time, in seconds, the animal can dip below the running speed threshold to still receive the reward. This allows animals that ‘run’ by taking a series of large steps, briefly dipping below speed threshold at the end of each step, to still get water rewards.

maximum_training_time_min: int = 40: The maximum time, in minutes, the system is allowed to run the training.

maximum_unconsumed_rewards: int = 1: The maximum number of consecutive rewards that can be delivered without the animal consuming them. If the animal receives this many rewards without licking (consuming) them, reward delivery is paused until the animal consumes the delivered rewards.

maximum_water_volume_ml: float = 1.0: The maximum volume of water the system is allowed to dispense during training.

mouse_weight_g: float: The weight of the animal, in grams, at the beginning of the session.

pause_dispensed_water_volume_ml: float = 0.0: The total water volume, in milliliters, dispensed during the paused (idle) state.

preferred_session_water_volume_ml: float = 0.0: The volume of water, in milliliters, the animal should be receiving during the session runtime if its performance matches experimenter-specified threshold.

reward_tone_duration_ms: int = 300: The duration, in milliseconds, of the auditory tone played to the animal when it receives water rewards.

run_duration_increase_step_s: float = 0.1: The value, in seconds, used by the system to increment the duration threshold each time the animal receives ‘increase_threshold’ volume of water.

run_speed_increase_step_cm_s: float = 0.05: The value, in centimeters per second, used by the system to increment the running speed threshold each time the animal receives ‘increase_threshold’ volume of water.

water_reward_size_ul: float = 5.0: The volume of water, in microliters, dispensed to the animal when it achieves the required running speed and duration thresholds.

class sl_shared_assets.data_classes.Segment(name, cue_sequence, transition_probabilities=None)

Bases: object

Defines a visual segment (sequence of cues) used in the experiment task’s Virtual Reality (VR) environment.

Notes

Segments are the building blocks of the infinite corridor, each containing a sequence of visual cues and optional transition probabilities for segment-to-segment transitions.

cue_sequence: list[str]: The ordered sequence of cue names that comprise this segment.

name: str: The unique identifier of the segment’s Unity prefab file.

transition_probabilities: list[float] | None = None: Optional transition probabilities to other segments that make up the task’s corridor environment. If provided, must sum to 1.0.

class sl_shared_assets.data_classes.ServerConfiguration(username='', password='', host='cbsuwsun.biohpc.cornell.edu', storage_root='/local/storage', working_root='/local/workdir', shared_directory_name='sun_data')

Bases: YamlConfig

Defines the access credentials and the filesystem layout of the Sun lab’s remote compute server.

host: str = 'cbsuwsun.biohpc.cornell.edu': The hostname or IP address of the server to connect to.

password: str = '': The password to use for server authentication.

shared_directory_name: str = 'sun_data': The name of the shared directory that stores Sun lab’s project data on both server volumes.

shared_storage_root: str: The path to the root Sun lab’s shared directory on the storage server’s volume.

shared_working_root: str: The path to the root Sun lab’s shared directory on the working server’s volume.

storage_root: str = '/local/storage': The path to the server’s storage (slow) HDD RAID volume.

user_data_root: str: The path to the root user’s directory on the storage server’s volume.

user_working_root: str: The path to the root user’s directory on the working server’s volume.

username: str = '': The username to use for server authentication.

working_root: str = '/local/workdir': The path to the server’s working (fast) NVME RAID volume.

class sl_shared_assets.data_classes.SessionData(project_name, animal_id, session_name, session_type, acquisition_system=AcquisitionSystems.MESOSCOPE_VR, experiment_name=None, python_version='3.11.13', sl_experiment_version='3.0.0', raw_data=<factory>, processed_data=<factory>, tracking_data=<factory>)

Bases: YamlConfig

Defines the structure and the metadata of a data acquisition session.

This class encapsulates the information necessary to access the session’s data stored on disk and functions as the entry point for all interactions with the session’s data.

Notes

Do not initialize this class directly. Instead, use the create() method when starting new data acquisition sessions or the load() method when accessing data for an existing session.

When this class is used to create a new session, it generates the new session’s name using the current UTC timestamp, accurate to microseconds. This ensures that each session ‘name’ is unique and preserves the overall session order.

acquisition_system: str | AcquisitionSystems = 'mesoscope': The name of the data acquisition system used to acquire the session’s data

animal_id: str: The unique identifier of the animal that participates in the session.

classmethod create(project_name, animal_id, session_type, python_version, sl_experiment_version, experiment_name=None)

Initializes a new data acquisition session and creates its data structure on the host-machine’s filesystem.

Notes

To access the data of an already existing session, use the load() method.

Parameters:

project_name (str) – The name of the project for which the session is acquired.
animal_id (str) – The unique identifier of the animal participating in the session.
session_type (SessionTypes | str) – The type of the session.
python_version (str) – The Python version used to acquire the session’s data.
sl_experiment_version (str) – The sl-experiment library version used to acquire the session’s data.
experiment_name (str | None, default: None) – The name of the experiment performed during the session or None, if the session is not an experiment session.

Return type:

SessionData

Returns:

An initialized SessionData instance that stores the structure and the metadata of the created session.

experiment_name: str | None = None: The name of the experiment performed during the session or Null (None), if the session is not an experiment session.

classmethod load(session_path)

Loads the target session’s data from the specified session_data.yaml file.

Notes

To create a new session, use the create() method.

Parameters:: session_path (Path) – The path to the directory where to search for the session_data.yaml file. Typically, this is the path to the root session’s directory, e.g.: root/project/animal/session.
Return type:: SessionData
Returns:: An initialized SessionData instance that stores the loaded session’s data.
Raises:: FileNotFoundError – If multiple or no ‘session_data.yaml’ file instances are found under the input directory.

processed_data: ProcessedData: Defines the session’s processed data hierarchy.

project_name: str: The name of the project for which the session was acquired.

python_version: str = '3.11.13': The Python version used to acquire session’s data.

raw_data: RawData: Defines the session’s raw data hierarchy.

runtime_initialized()

Ensures that the ‘nk.bin’ marker file is removed from the session’s raw_data directory.

Return type:: None

Notes

This service method is used by the sl-experiment library to acquire the session’s data. Do not call this method manually.

save()

Caches the instance’s data to the session’s ‘raw_data’ directory as a ‘session_data.yaml’ file.

Return type:: None

session_name: str: The unique identifier (name) of the session.

session_type: str | SessionTypes: The type of the session.

sl_experiment_version: str = '3.0.0': The sl-experiment library version used to acquire the session’s data.

tracking_data: TrackingData: Defines the session’s tracking data hierarchy.

class sl_shared_assets.data_classes.SessionMetadata(session, animal)

Bases: object

Encapsulates the identity metadata for a single data acquisition session.

This class is used to identify sessions included in an analysis dataset. It provides the minimum information necessary to locate and access the session’s data within a project’s data hierarchy.

animal: str: The unique identifier of the animal that participated in the session.

session: str: The unique identifier of the session. Session names follow the format ‘YYYY-MM-DD-HH-MM-SS-microseconds’ and encode the session’s acquisition timestamp.

class sl_shared_assets.data_classes.SessionTypes(*values)

Bases: StrEnum

Defines the data acquisition session types supported by all data acquisition systems used in the Sun lab.

LICK_TRAINING = 'lick training': A Mesoscope-VR session designed to teach animals to use the water delivery port while being head-fixed.

MESOSCOPE_EXPERIMENT = 'mesoscope experiment': A Mesoscope-VR experiment session. The session uses the Unity game engine to run virtual reality tasks and collects brain activity data using 2-Photon Random Access Mesoscope (2P-RAM).

RUN_TRAINING = 'run training': A Mesoscope-VR session designed to teach animals to run on the treadmill while being head-fixed.

WINDOW_CHECKING = 'window checking': A Mesoscope-VR session designed to evaluate the quality of the cranial window implantation procedure and the suitability of the animal for experiment sessions. The session uses the Mesoscope to assess the quality of the cell activity data.

class sl_shared_assets.data_classes.SubjectData(id, ear_punch, sex, genotype, date_of_birth_us, weight_g, cage, location_housed, status)

Bases: object

Stores information about the subject of the surgical intervention.

cage: int: The unique identifier (number) of the cage used to house the subject after the surgery.

date_of_birth_us: int: The subject’s date of birth, stored as the number of microseconds elapsed since the UTC epoch onset.

ear_punch: str: The number and the locations of ear-tags used to distinguish the subject from its cage-mates.

genotype: str: The subject’s genotype.

id: int: The subject’s unique identifier.

location_housed: str: The location (room) used to house the subject after the surgery.

sex: str: The subject’s gender.

status: str: The current subject’s status (alive / deceased).

weight_g: float: The subject’s pre-surgery weight, in grams.

class sl_shared_assets.data_classes.SurgeryData(subject, procedure, drugs, implants, injections)

Bases: YamlConfig

Stores information about a surgical intervention performed on an animal before data acquisition session(s).

drugs: DrugData: Stores information about all medical substances (drugs) administered to the subject before, during, and immediately after the surgical intervention.

implants: list[ImplantData]: Stores information about all implantation procedures performed during the surgical intervention.

injections: list[InjectionData]: Stores information about all injections (brain infusions) performed during the surgical intervention.

procedure: ProcedureData: Stores general information about the surgical intervention.

subject: SubjectData: Stores information about the subject of the surgical intervention.

class sl_shared_assets.data_classes.TrackingData(tracking_data_path=PosixPath('.'))

Bases: object

Provides the path to the directory that stores the .yaml and .lock files used by ProcessingTracker instances to track the runtime status of the data processing pipelines working with the session’s data.

make_directories()

Ensures that all major subdirectories and the root directory exist, creating any missing directories.

Return type:: None

resolve_paths(root_directory_path)

Resolves all paths managed by the class instance based on the input root directory path.

Parameters:: root_directory_path (Path) – The path to the top-level tracking data directory of the session’s data hierarchy.
Return type:: None

tracking_data_path: Path = PosixPath('.'): The path to the root directory that stores the session’s tracking data.

class sl_shared_assets.data_classes.VREnvironment(corridor_spacing_cm=20.0, segments_per_corridor=3, padding_prefab_name='Padding', cm_per_unity_unit=10.0)

Bases: object

Defines the Unity VR corridor system configuration.

Notes

This class is primarily used by Unity to configure the task environment. Python parses these values from the YAML configuration file but does not use them at runtime.

cm_per_unity_unit: float = 10.0: The conversion factor from centimeters to Unity units.

corridor_spacing_cm: float = 20.0: The horizontal spacing between corridor instances in centimeters.

padding_prefab_name: str = 'Padding': The name of the Unity prefab used for corridor padding.

segments_per_corridor: int = 3: The number of segments visible in each corridor instance (corridor depth).

class sl_shared_assets.data_classes.WaterRewardTrial(segment_name, stimulus_trigger_zone_start_cm, stimulus_trigger_zone_end_cm, stimulus_location_cm, show_stimulus_collision_boundary=False, cue_sequence=<factory>, trial_length_cm=0.0, reward_size_ul=5.0, reward_tone_duration_ms=300)

Bases: _MesoscopeBaseTrial

Defines a trial that delivers water rewards (reinforcing stimuli) when the animal licks in the trigger zone.

Notes

Trigger mode: The animal must lick while inside the stimulus trigger zone to receive the water reward. Guidance mode: The animal receives the reward upon colliding with the stimulus boundary (no lick required).

reward_size_ul: float = 5.0: The volume of water, in microliters, to deliver when the animal successfully completes the trial.

reward_tone_duration_ms: int = 300: The duration, in milliseconds, to sound the auditory tone when delivering the water reward.

class sl_shared_assets.data_classes.WindowCheckingDescriptor(experimenter, surgery_quality=0, incomplete=True, experimenter_notes='Replace this with your notes.')

Bases: YamlConfig

Stores the outcome information specific to window checking sessions that use the Mesoscope-VR system.

experimenter: str: The ID of the experimenter running the session.

experimenter_notes: str = 'Replace this with your notes.': Stores the experimenter’s notes made during runtime.

incomplete: bool = True: Tracks whether the session’s data is complete and eligible for unsupervised data processing.

surgery_quality: int = 0: The quality of the cranial window and surgical intervention on a scale from 0 (non-usable) to 3 (high-tier publication grade) inclusive.

class sl_shared_assets.data_classes.ZaberPositions(headbar_z=0, headbar_pitch=0, headbar_roll=0, lickport_z=0, lickport_y=0, lickport_x=0, wheel_x=0)

Bases: YamlConfig

Stores Zaber motor positions reused between data acquisition sessions that use the Mesoscope-VR system.

headbar_pitch: int = 0: The absolute position, in native motor units, of the HeadBar pitch-axis motor.

headbar_roll: int = 0: The absolute position, in native motor units, of the HeadBar roll-axis motor.

headbar_z: int = 0: The absolute position, in native motor units, of the HeadBar z-axis motor.

lickport_x: int = 0: The absolute position, in native motor units, of the LickPort x-axis motor.

lickport_y: int = 0: The absolute position, in native motor units, of the LickPort y-axis motor.

lickport_z: int = 0: The absolute position, in native motor units, of the LickPort z-axis motor.

wheel_x: int = 0: The absolute position, in native motor units, of the running wheel platform x-axis motor.

sl_shared_assets.data_classes.create_server_configuration_file(username, password, host='cbsuwsun.biopic.cornell.edu', storage_root='/local/workdir', working_root='/local/storage', shared_directory_name='sun_data')

Creates the .YAML configuration file for the Sun lab compute server and configures the local machine (PC) to use this file for all future server-related calls.

Notes

This function creates the configuration file inside the shared Sun lab’s working directory on the local machine.

Parameters:

username (str) – The username to use for server authentication.
password (str) – The password to use for server authentication.
host (str, default: 'cbsuwsun.biopic.cornell.edu') – The hostname or IP address of the server to connect to.
storage_root (str, default: '/local/workdir') – The path to the server’s storage (slow) HDD RAID volume.
working_root (str, default: '/local/storage') – The path to the server’s working (fast) NVME RAID volume.
shared_directory_name (str, default: 'sun_data') – The name of the shared directory that stores Sun lab’s project data on both server volumes.

Return type:

None

sl_shared_assets.data_classes.create_system_configuration_file(system)

Creates the .YAML configuration file for the requested Sun lab’s data acquisition system and configures the local machine (PC) to use this file for all future acquisition-system-related calls.

Notes

This function creates the configuration file inside the local Sun lab’s working directory.

Parameters:: system (AcquisitionSystems | str) – The name (type) of the data acquisition system for which to create the configuration file.
Raises:: ValueError – If the input acquisition system name (type) is not recognized.
Return type:: None

sl_shared_assets.data_classes.get_google_credentials_path()

Resolves and returns the path to the Google service account credentials .JSON file.

Return type:: Path
Returns:: The path to the Google service account credentials .JSON file.
Raises:: FileNotFoundError – If the Google service account credentials path has not been configured for the host-machine, or if the previously configured credentials file no longer exists at the expected path.

sl_shared_assets.data_classes.get_server_configuration()

Resolves and returns the Sun lab compute server’s configuration data as a ServerConfiguration instance.

Return type:

ServerConfiguration

Returns:

The loaded and validated server configuration data, stored in a ServerConfiguration instance.

Raises:

FileNotFoundError – If the configuration file does not exist in the local Sun lab’s working directory.
ValueError – If the configuration file exists, but is not properly configured.

sl_shared_assets.data_classes.get_system_configuration_data()

Resolves the path to the local data acquisition system configuration file and loads the configuration data as a SystemConfiguration instance.

Return type:: MesoscopeSystemConfiguration
Returns:: The initialized SystemConfiguration class instance that stores the loaded configuration parameters.
Raises:: FileNotFoundError – If the local machine does not have a valid data acquisition system configuration file.

sl_shared_assets.data_classes.get_working_directory()

Resolves and returns the path to the local Sun lab’s working directory.

Return type:: Path
Returns:: The path to the local working directory.
Raises:: FileNotFoundError – If the local working directory has not been configured for the host-machine.

sl_shared_assets.data_classes.set_google_credentials_path(path)

Configures the local machine (PC) to use the provided Google Sheets service account credentials .JSON file for all future interactions with the Google’s API.

Notes

This function caches the path to the Google Sheets credentials file in the user’s data directory.

Parameters:: path (Path) – The path to the .JSON file containing the Google Sheets service account credentials.
Raises:: FileNotFoundError – If the specified .JSON file does not exist at the provided path.
Return type:: None

sl_shared_assets.data_classes.set_working_directory(path)

Sets the specified directory as the Sun lab’s working directory for the local machine (PC).

Notes

This function caches the path to the working directory in the user’s data directory.

If the input path does not point to an existing directory, the function creates the requested directory.

Parameters:: path (Path) – The path to the directory to set as the local Sun lab’s working directory.
Return type:: None