data

Autopilot’s data handling system was revamped as of v0.5.0, and now is based on pydantic models and a series of interfaces that allow us to write data from the same abstract structures to several formats, initially pytables and hdf5, but we have laid the groundwork for exporting to nwb and datajoint natively.

A brief narrative overview here, and more detailed documentations within the relevant module documentation.

modeling - Basic Data Types

digraph inheritancee2533af439 { bgcolor=transparent; rankdir=LR; size="8.0, 12.0"; "abc.ABC" [fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",tooltip="Helper class that provides a standard way to create an ABC using"]; "autopilot.data.modeling.base.Attributes" [URL="../modeling/index.html#autopilot.data.modeling.base.Attributes",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="A set of attributes that is intended to have a single representation per usage:"]; "autopilot.data.modeling.base.Data" -> "autopilot.data.modeling.base.Attributes" [arrowsize=0.5,style="setlinewidth(0.5)"]; "autopilot.data.modeling.base.Data" [URL="../modeling/index.html#autopilot.data.modeling.base.Data",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="The top-level container for Data."]; "autopilot.root.Autopilot_Type" -> "autopilot.data.modeling.base.Data" [arrowsize=0.5,style="setlinewidth(0.5)"]; "autopilot.data.modeling.base.Group" [URL="../modeling/index.html#autopilot.data.modeling.base.Group",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="A generic representation of a \"Group\" if present in a given interface."]; "autopilot.root.Autopilot_Type" -> "autopilot.data.modeling.base.Group" [arrowsize=0.5,style="setlinewidth(0.5)"]; "autopilot.data.modeling.base.Node" [URL="../modeling/index.html#autopilot.data.modeling.base.Node",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Abstract representation of a Node in a treelike or linked data structure."]; "autopilot.root.Autopilot_Type" -> "autopilot.data.modeling.base.Node" [arrowsize=0.5,style="setlinewidth(0.5)"]; "autopilot.data.modeling.base.Schema" [URL="../modeling/index.html#autopilot.data.modeling.base.Schema",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="A special type of type intended to be a representation of an"]; "autopilot.root.Autopilot_Type" -> "autopilot.data.modeling.base.Schema" [arrowsize=0.5,style="setlinewidth(0.5)"]; "autopilot.data.modeling.base.Table" [URL="../modeling/index.html#autopilot.data.modeling.base.Table",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Tabular data: each field will have multiple values -- in particular an equal number across fields."]; "autopilot.data.modeling.base.Data" -> "autopilot.data.modeling.base.Table" [arrowsize=0.5,style="setlinewidth(0.5)"]; "autopilot.root.Autopilot_Type" [URL="../../root/index.html#autopilot.root.Autopilot_Type",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Root autopilot model for types"]; "pydantic.main.BaseModel" -> "autopilot.root.Autopilot_Type" [arrowsize=0.5,style="setlinewidth(0.5)"]; "abc.ABC" -> "autopilot.root.Autopilot_Type" [arrowsize=0.5,style="setlinewidth(0.5)"]; "pydantic.main.BaseModel" [fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled"]; "pydantic.utils.Representation" -> "pydantic.main.BaseModel" [arrowsize=0.5,style="setlinewidth(0.5)"]; "pydantic.utils.Representation" [fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",tooltip="Mixin to provide __str__, __repr__, and __pretty__ methods. See #884 for more details."]; }

Autopilot’s models are built from pydantic models.

The autopilot.root module defines some of Autopilot’s basic metaclasses, one of which is Autopilot_Type. The data.modeling module extends Autopilot_Type into several abstract modeling classes used for different types of data:

  • modeling.base.Data - Containers for data, generally these are used as containers for data, or else used to specify how data should be handled and typed. Its subtypes indicate different classes of data that have different means of storage and representation depending on the interface.

    • modeling.base.Attributes - Static (usually metadata) attributes that are intended to be specified once per instance they are used (eg. the Biography class is used once per Subject)

    • modeling.base.Table - Tabular data specifies that there should be multiple values for each of the fields defined: in particular equal numbers of each of them. This is used for most data collected, as most data can be framed in a tabular format.

  • modeling.base.Group and modeling.base.Node - Abstract specifications for hierarchical data interfaces - a Node is a particular element in a tree/network-like system, and a Group is a collection of Nodes. Some transitional work is still being done to generalize Autopilot’s former data structures from H5F-specific groups and nodes, so for the moment there is some parallel functionality in the H5F_Node and H5F_Group classes

  • modeling.base.Schema - Specifications for organization of other data structures, for data that isn’t expected to ever be instantiated in its described form, but for scaffolding building other data structures together. Some transitional work is also being done here, eventually moving the Subject Schema to an abstract form (Subject_Schema) vs one tied to HDF5 (Subject_Structure)

models - The Models Themselves

digraph inheritanceacc1831564 { bgcolor=transparent; rankdir=LR; size="8.0, 12.0"; "Autopilot_Type" [URL="../../root/index.html#autopilot.root.Autopilot_Type",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Root autopilot model for types"]; "interfaces.tables.H5F_Group" [URL="../interfaces/tables.html#autopilot.data.interfaces.tables.H5F_Group",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Description of a pytables group and its location"]; "interfaces.tables.H5F_Node" -> "interfaces.tables.H5F_Group" [arrowsize=0.5,style="setlinewidth(0.5)"]; "interfaces.tables.H5F_Node" [URL="../interfaces/tables.html#autopilot.data.interfaces.tables.H5F_Node",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Base class for H5F Nodes"]; "modeling.base.Node" -> "interfaces.tables.H5F_Node" [arrowsize=0.5,style="setlinewidth(0.5)"]; "modeling.base.Attributes" [URL="../modeling/index.html#autopilot.data.modeling.base.Attributes",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="A set of attributes that is intended to have a single representation per usage:"]; "modeling.base.Data" -> "modeling.base.Attributes" [arrowsize=0.5,style="setlinewidth(0.5)"]; "modeling.base.Data" [URL="../modeling/index.html#autopilot.data.modeling.base.Data",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="The top-level container for Data."]; "Autopilot_Type" -> "modeling.base.Data" [arrowsize=0.5,style="setlinewidth(0.5)"]; "modeling.base.Group" [URL="../modeling/index.html#autopilot.data.modeling.base.Group",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="A generic representation of a \"Group\" if present in a given interface."]; "Autopilot_Type" -> "modeling.base.Group" [arrowsize=0.5,style="setlinewidth(0.5)"]; "modeling.base.Node" [URL="../modeling/index.html#autopilot.data.modeling.base.Node",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Abstract representation of a Node in a treelike or linked data structure."]; "Autopilot_Type" -> "modeling.base.Node" [arrowsize=0.5,style="setlinewidth(0.5)"]; "modeling.base.Schema" [URL="../modeling/index.html#autopilot.data.modeling.base.Schema",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="A special type of type intended to be a representation of an"]; "Autopilot_Type" -> "modeling.base.Schema" [arrowsize=0.5,style="setlinewidth(0.5)"]; "modeling.base.Table" [URL="../modeling/index.html#autopilot.data.modeling.base.Table",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Tabular data: each field will have multiple values -- in particular an equal number across fields."]; "modeling.base.Data" -> "modeling.base.Table" [arrowsize=0.5,style="setlinewidth(0.5)"]; "models.biography.Baselines" [URL="../models/biography.html#autopilot.data.models.biography.Baselines",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Baseline health measurements for animal care regulation. In the future this"]; "modeling.base.Data" -> "models.biography.Baselines" [arrowsize=0.5,style="setlinewidth(0.5)"]; "models.biography.Biography" [URL="../models/biography.html#autopilot.data.models.biography.Biography",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="The combined biographical, health, genetic, and other details that define an experimental subject."]; "modeling.base.Attributes" -> "models.biography.Biography" [arrowsize=0.5,style="setlinewidth(0.5)"]; "models.biography.Breeding" [URL="../models/biography.html#autopilot.data.models.biography.Breeding",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Information about the breeding conditions of the subject"]; "modeling.base.Data" -> "models.biography.Breeding" [arrowsize=0.5,style="setlinewidth(0.5)"]; "models.biography.Enclosure" [URL="../models/biography.html#autopilot.data.models.biography.Enclosure",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Where does the subject live?"]; "modeling.base.Data" -> "models.biography.Enclosure" [arrowsize=0.5,style="setlinewidth(0.5)"]; "models.biography.Gene" [URL="../models/biography.html#autopilot.data.models.biography.Gene",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="An individual (trans)gene that an animal may have."]; "modeling.base.Data" -> "models.biography.Gene" [arrowsize=0.5,style="setlinewidth(0.5)"]; "models.biography.Genotype" [URL="../models/biography.html#autopilot.data.models.biography.Genotype",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Genotyping information, information about a subject's background and (potentially multiple) :class:`.Gene` s of interest"]; "modeling.base.Data" -> "models.biography.Genotype" [arrowsize=0.5,style="setlinewidth(0.5)"]; "models.protocol.Protocol_Data" [URL="../models/protocol.html#autopilot.data.models.protocol.Protocol_Data",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top"]; "modeling.base.Schema" -> "models.protocol.Protocol_Data" [arrowsize=0.5,style="setlinewidth(0.5)"]; "models.protocol.Protocol_Group" [URL="../models/protocol.html#autopilot.data.models.protocol.Protocol_Group",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="The group and subgroups for a given protocol."]; "interfaces.tables.H5F_Group" -> "models.protocol.Protocol_Group" [arrowsize=0.5,style="setlinewidth(0.5)"]; "models.protocol.Step_Data" [URL="../models/protocol.html#autopilot.data.models.protocol.Step_Data",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Schema for storing data for a single step of a protocol"]; "modeling.base.Schema" -> "models.protocol.Step_Data" [arrowsize=0.5,style="setlinewidth(0.5)"]; "models.protocol.Step_Group" [URL="../models/protocol.html#autopilot.data.models.protocol.Step_Group",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="An hdf5 group for an individual step within a protocol."]; "interfaces.tables.H5F_Group" -> "models.protocol.Step_Group" [arrowsize=0.5,style="setlinewidth(0.5)"]; "models.protocol.Task_Params" [URL="../models/protocol.html#autopilot.data.models.protocol.Task_Params",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Metaclass for storing task parameters"]; "Autopilot_Type" -> "models.protocol.Task_Params" [arrowsize=0.5,style="setlinewidth(0.5)"]; "models.protocol.Trial_Data" [URL="../models/protocol.html#autopilot.data.models.protocol.Trial_Data",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Base class for declaring trial data."]; "modeling.base.Table" -> "models.protocol.Trial_Data" [arrowsize=0.5,style="setlinewidth(0.5)"]; "models.researcher.Researcher" [URL="../models/researcher.html#autopilot.data.models.researcher.Researcher",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top"]; "modeling.base.Data" -> "models.researcher.Researcher" [arrowsize=0.5,style="setlinewidth(0.5)"]; "models.subject.Hashes" [URL="../models/subject.html#autopilot.data.models.subject.Hashes",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Table to track changes in version over time"]; "modeling.base.Table" -> "models.subject.Hashes" [arrowsize=0.5,style="setlinewidth(0.5)"]; "models.subject.History" [URL="../models/subject.html#autopilot.data.models.subject.History",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Table to describe parameter and protocol change history"]; "modeling.base.Table" -> "models.subject.History" [arrowsize=0.5,style="setlinewidth(0.5)"]; "models.subject.History_Group" [URL="../models/subject.html#autopilot.data.models.subject.History_Group",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Group for collecting subject history tables."]; "modeling.base.Group" -> "models.subject.History_Group" [arrowsize=0.5,style="setlinewidth(0.5)"]; "models.subject.Protocol_Status" [URL="../models/subject.html#autopilot.data.models.subject.Protocol_Status",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Status of assigned protocol. Accessible from the :attr:`.Subject.protocol` getter/setter"]; "modeling.base.Attributes" -> "models.subject.Protocol_Status" [arrowsize=0.5,style="setlinewidth(0.5)"]; "models.subject.Subject_Schema" [URL="../models/subject.html#autopilot.data.models.subject.Subject_Schema",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Structure of the :class:`.Subject` class's hdf5 file"]; "modeling.base.Schema" -> "models.subject.Subject_Schema" [arrowsize=0.5,style="setlinewidth(0.5)"]; "models.subject.Subject_Structure" [URL="../models/subject.html#autopilot.data.models.subject.Subject_Structure",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Structure of the :class:`.Subject` class's hdf5 file"]; "modeling.base.Schema" -> "models.subject.Subject_Structure" [arrowsize=0.5,style="setlinewidth(0.5)"]; "models.subject.Weights" [URL="../models/subject.html#autopilot.data.models.subject.Weights",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Class to describe table for weight history"]; "modeling.base.Table" -> "models.subject.Weights" [arrowsize=0.5,style="setlinewidth(0.5)"]; }

Specific models are then built out of the basic modeling components! This will serve as the point where data models can be added or modified by plugins (stay tuned).

Each of the modules contains several classes that are used together in some particular context:

  • models.biography - Defines biographical information for an individual Subject

  • models.protocol - Defines the data structure of how multiple Tasks are stacked together into a training protocol, as well as how they are represented in the Subject’s h5f file.

  • models.subject - Schemas that define how the multiple models that go into a subject are combined and structured on disk

  • models.researcher - Stubs for researcher information that will be used in future versions for giving explicit credit for data gathered by a particular researcher or research group…

interfaces - Bridging to Multiple Representations

digraph inheritance96efe7ce9e { bgcolor=transparent; rankdir=LR; size="8.0, 12.0"; "abc.ABC" [fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",tooltip="Helper class that provides a standard way to create an ABC using"]; "autopilot.data.interfaces.base.Interface" [URL="../interfaces/base.html#autopilot.data.interfaces.base.Interface",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Create a representation of a given Schema"]; "autopilot.root.Autopilot_Type" -> "autopilot.data.interfaces.base.Interface" [arrowsize=0.5,style="setlinewidth(0.5)"]; "autopilot.data.interfaces.base.Interface_Map" [URL="../interfaces/base.html#autopilot.data.interfaces.base.Interface_Map",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Statement of equivalence between two things, potentially with some"]; "autopilot.root.Autopilot_Type" -> "autopilot.data.interfaces.base.Interface_Map" [arrowsize=0.5,style="setlinewidth(0.5)"]; "autopilot.data.interfaces.base.Interface_Mapset" [URL="../interfaces/base.html#autopilot.data.interfaces.base.Interface_Mapset",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Metaclass for mapping base types to another format."]; "autopilot.root.Autopilot_Type" -> "autopilot.data.interfaces.base.Interface_Mapset" [arrowsize=0.5,style="setlinewidth(0.5)"]; "autopilot.data.interfaces.nwb.NWB_Interface" [URL="../interfaces/nwb.html#autopilot.data.interfaces.nwb.NWB_Interface",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top"]; "autopilot.root.Autopilot_Type" -> "autopilot.data.interfaces.nwb.NWB_Interface" [arrowsize=0.5,style="setlinewidth(0.5)"]; "autopilot.data.interfaces.tables.H5F_Group" [URL="../interfaces/tables.html#autopilot.data.interfaces.tables.H5F_Group",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Description of a pytables group and its location"]; "autopilot.data.interfaces.tables.H5F_Node" -> "autopilot.data.interfaces.tables.H5F_Group" [arrowsize=0.5,style="setlinewidth(0.5)"]; "autopilot.data.interfaces.tables.H5F_Node" [URL="../interfaces/tables.html#autopilot.data.interfaces.tables.H5F_Node",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Base class for H5F Nodes"]; "autopilot.data.modeling.base.Node" -> "autopilot.data.interfaces.tables.H5F_Node" [arrowsize=0.5,style="setlinewidth(0.5)"]; "autopilot.data.interfaces.tables.H5F_Table" [URL="../interfaces/tables.html#autopilot.data.interfaces.tables.H5F_Table",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top"]; "autopilot.data.interfaces.tables.H5F_Node" -> "autopilot.data.interfaces.tables.H5F_Table" [arrowsize=0.5,style="setlinewidth(0.5)"]; "autopilot.data.interfaces.tables.Tables_Interface" [URL="../interfaces/tables.html#autopilot.data.interfaces.tables.Tables_Interface",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top"]; "autopilot.data.interfaces.base.Interface" -> "autopilot.data.interfaces.tables.Tables_Interface" [arrowsize=0.5,style="setlinewidth(0.5)"]; "autopilot.data.modeling.base.Node" [URL="../modeling/index.html#autopilot.data.modeling.base.Node",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Abstract representation of a Node in a treelike or linked data structure."]; "autopilot.root.Autopilot_Type" -> "autopilot.data.modeling.base.Node" [arrowsize=0.5,style="setlinewidth(0.5)"]; "autopilot.root.Autopilot_Type" [URL="../../root/index.html#autopilot.root.Autopilot_Type",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Root autopilot model for types"]; "pydantic.main.BaseModel" -> "autopilot.root.Autopilot_Type" [arrowsize=0.5,style="setlinewidth(0.5)"]; "abc.ABC" -> "autopilot.root.Autopilot_Type" [arrowsize=0.5,style="setlinewidth(0.5)"]; "pydantic.main.BaseModel" [fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled"]; "pydantic.utils.Representation" -> "pydantic.main.BaseModel" [arrowsize=0.5,style="setlinewidth(0.5)"]; "pydantic.utils.Representation" [fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",tooltip="Mixin to provide __str__, __repr__, and __pretty__ methods. See #884 for more details."]; }

Interfaces define mappings between basic python types and the classes in modeling.

This set of classes is still growing, and we’re still exploring the best strategy to make generalizable interfaces between very different formats, but in general, each interface consists of mappings between types and some means of converting the particular data structures of one format and another.

The only interface that is actively used within Autopilot is that for tables, but we have started interfaces for nwb and datajoint (using a parallel project datajoint-babel). Both of these are provisional and very incomplete, but it is possible to generate a datajoint schema from any table, and there are mappings and conversions for their different representations of types.

Our goal for future versions is to generalize data interfaces to the point where a similar API can be shared across them, so a subject’s data can be stored in HDF5 or in a datajoint database equivalently.

Subject - The Main Interface to Data Collection

Subject is the main object that most people will use to interact with their data, and it is used throughout Autopilot to keep track of individual subjects data, the protocols they are run on, changes in code version over time, etc.

See the main data.subject module page for further information.

units - Explicit SI Unit representation

This too is just a stub, but we will be moving more of our models to using specific SI units when appropriate rather than using generic floats and ints with human-readable descriptions of when they are a mL or a ms vs. second or Liter, etc.

Transition Status

Transitioning to a uniform data modeling system is in progress! The following need to still be transitioned to formal models.

  • Task.PARAMS and Task.HARDWARE

  • Task.PLOT which should be merged into the TrialData field descriptions

  • autopilot.prefs - which currently has a large dictionary of default prefs

  • Hardware parameter descriptions - Need to find better way of having models that represent class arguments.

  • graduation objects.

  • Verious GUI widgets need to use models rather than the zillions of ad-hoc representations:

  • utils.plugins needs its own model to handle dependencies, etc.

  • agents needs models for defining basic agent attributes.