modeling

Inheritance diagram of autopilot.data.modeling.base

Autopilot’s models are built from pydantic models.

The autopilot.root module defines some of Autopilot’s basic metaclasses, one of which is Autopilot_Type. The data.modeling module extends Autopilot_Type into several abstract modeling classes used for different types of data:

  • modeling.base.Data - Containers for data, generally these are used as containers for data, or else used to specify how data should be handled and typed. Its subtypes indicate different classes of data that have different means of storage and representation depending on the interface.

    • modeling.base.Attributes - Static (usually metadata) attributes that are intended to be specified once per instance they are used (eg. the Biography class is used once per Subject)

    • modeling.base.Table - Tabular data specifies that there should be multiple values for each of the fields defined: in particular equal numbers of each of them. This is used for most data collected, as most data can be framed in a tabular format.

  • modeling.base.Group and modeling.base.Node - Abstract specifications for hierarchical data interfaces - a Node is a particular element in a tree/network-like system, and a Group is a collection of Nodes. Some transitional work is still being done to generalize Autopilot’s former data structures from H5F-specific groups and nodes, so for the moment there is some parallel functionality in the H5F_Node and H5F_Group classes

  • modeling.base.Schema - Specifications for organization of other data structures, for data that isn’t expected to ever be instantiated in its described form, but for scaffolding building other data structures together. Some transitional work is also being done here, eventually moving the Subject Schema to an abstract form (Subject_Schema) vs one tied to HDF5 (Subject_Structure)

basic classes

Base classes for data models – the Data class itself.

Classes:

Data

The top-level container for Data.

Table

Tabular data: each field will have multiple values -- in particular an equal number across fields.

Attributes

A set of attributes that is intended to have a single representation per usage: eg.

Schema

A special type of type intended to be a representation of an abstract structure/schema of data, rather than a live container of data objects themselves.

Node

Abstract representation of a Node in a treelike or linked data structure.

Group

A generic representation of a "Group" if present in a given interface.

Data:

BASE_TYPES

Base Python types that should be suppported by every interface

pydantic model Data[source]

Bases: autopilot.root.Autopilot_Type

The top-level container for Data.

Subtypes will define more specific formats and uses of data, but this is the most general form used to represent the type and meaning of data.

The Data class is not intended to contain individual fields, but collections of data that are collected as a unit, whether that be a video frame along with its timestamp and encoding, or a single trial of behavioral data.

This class is also generally not intended to be used for the literal transport of data when performance is necessary: this class by default does type validation on instantiation that takes time (see the construct method for validation-less creation). It is usually more to specify the type, grouping, and annotation for a given unit of data – though users should feel free to dump their data in a Data object if it is not particularly performance sensitive.

Show JSON schema
{
   "title": "Data",
   "description": "The top-level container for Data.\n\nSubtypes will define more specific formats and uses of data, but this is the most general\nform used to represent the type and meaning of data.\n\nThe Data class is not intended to contain individual fields, but collections of data that are collected\nas a unit, whether that be a video frame along with its timestamp and encoding, or a single trial of behavioral data.\n\nThis class is also generally not intended to be used for the literal transport of data when performance is\nnecessary: this class by default does type validation on instantiation that takes time (see the `construct <https://pydantic-docs.helpmanual.io/usage/models/#creating-models-without-validation>`_\nmethod for validation-less creation). It is usually more to specify the type, grouping, and annotation for\na given unit of data -- though users should feel free to dump their data in a :class:`.Data` object if\nit is not particularly performance sensitive.",
   "type": "object",
   "properties": {}
}

pydantic model Table[source]

Bases: autopilot.data.modeling.base.Data

Tabular data: each field will have multiple values – in particular an equal number across fields.

Used for trialwise data, and can be used to create pytables descriptions.

Todo

To make this usable as a live container of data, the fields need to be declared as Lists (eg. instead of just declaring something an int, it must be specified as a List[int] to pass validation. We should expand this model to relax that constraint and effectively treat every field as containing a list of values.

Show JSON schema
{
   "title": "Table",
   "description": "Tabular data: each field will have multiple values -- in particular an equal number across fields.\n\nUsed for trialwise data, and can be used to create pytables descriptions.\n\n.. todo::\n\n    To make this usable as a live container of data, the fields need to be declared as Lists (eg. instead of just\n    declaring something an ``int``, it must be specified as a ``List[int]`` to pass validation. We should expand this\n    model to relax that constraint and effectively treat every field as containing a list of values.",
   "type": "object",
   "properties": {}
}

classmethod to_pytables_description() Type[tables.description.IsDescription][source]

Convert the fields of this table to a pytables description.

See model_to_description()

classmethod from_pytables_description(description: Type[tables.description.IsDescription]) autopilot.data.modeling.base.Table[source]

Create an instance of a table from a pytables description

See description_to_model()

Parameters

description (tables.IsDescription) – A Pytables description

to_df() pandas.core.frame.DataFrame[source]

Create a dataframe from the lists of fields

Returns

pandas.DataFrame

pydantic model Attributes[source]

Bases: autopilot.data.modeling.base.Data

A set of attributes that is intended to have a single representation per usage: eg. a subject has a single set of biographical information.

Useful to specify a particular type of storage that doesn’t need to include variable numbers of each field (eg. the tables interface stores attribute objects as metadata on a node, rather than as a table).

Show JSON schema
{
   "title": "Attributes",
   "description": "A set of attributes that is intended to have a single representation per usage:\neg. a subject has a single set of biographical information.\n\nUseful to specify a particular type of storage that doesn't need to include variable\nnumbers of each field (eg. the tables interface stores attribute objects as metadata on a node, rather than as a table).",
   "type": "object",
   "properties": {}
}

pydantic model Schema[source]

Bases: autopilot.root.Autopilot_Type

A special type of type intended to be a representation of an abstract structure/schema of data, rather than a live container of data objects themselves. This class is used for constructing data containers, translating between formats, etc. rather than momentary data handling

Show JSON schema
{
   "title": "Schema",
   "description": "A special type of type intended to be a representation of an\nabstract structure/schema of data, rather than a live container of\ndata objects themselves. This class is used for constructing data containers,\ntranslating between formats, etc. rather than momentary data handling",
   "type": "object",
   "properties": {}
}

pydantic model Node[source]

Bases: autopilot.root.Autopilot_Type

Abstract representation of a Node in a treelike or linked data structure. This should be extended by interfaces when relevant and needed to implement an abstract representation of their structure.

This class purposely lacks structure like a path or parents pending further usage in interfaces to see what would be the best means of implementing them.

Show JSON schema
{
   "title": "Node",
   "description": "Abstract representation of a Node in a treelike or linked data structure.\nThis should be extended by interfaces when relevant and needed to implement\nan abstract representation of their structure.\n\nThis class purposely lacks structure like a path or parents pending further\nusage in interfaces to see what would be the best means of implementing them.",
   "type": "object",
   "properties": {}
}

pydantic model Group[source]

Bases: autopilot.root.Autopilot_Type

A generic representation of a “Group” if present in a given interface. Useful for when, for example in a given container format you want to make an empty group that will be filled later, or one that has to be present for syntactic correctness.

A children attribute is present because it is definitive of groups, but should be overridden by interfaces that use it.

Show JSON schema
{
   "title": "Group",
   "description": "A generic representation of a \"Group\" if present in a given interface.\nUseful for when, for example in a given container format you want to\nmake an empty group that will be filled later, or one that has to be\npresent for syntactic correctness.\n\nA children attribute is present because it is definitive of groups, but\nshould be overridden by interfaces that use it.",
   "type": "object",
   "properties": {
      "children": {
         "title": "Children",
         "type": "array",
         "items": {
            "$ref": "#/definitions/Node"
         }
      }
   },
   "definitions": {
      "Node": {
         "title": "Node",
         "description": "Abstract representation of a Node in a treelike or linked data structure.\nThis should be extended by interfaces when relevant and needed to implement\nan abstract representation of their structure.\n\nThis class purposely lacks structure like a path or parents pending further\nusage in interfaces to see what would be the best means of implementing them.",
         "type": "object",
         "properties": {}
      }
   }
}

Fields
field children: Optional[List[autopilot.data.modeling.base.Node]] = None
BASE_TYPES = (   <class 'bool'>,     <class 'int'>,     <class 'float'>,     <class 'str'>,     <class 'bytes'>,     <class 'datetime.datetime'>)

Base Python types that should be suppported by every interface