v0.5.0a0 - ~The Data Modeling Edition~
A lot. Namely a whole rewriting of the autopilot.data
module.
This is being released as an alpha version while we finish working out a few of the kinks in the data modeling system because Jonny is finishing their dissertation and has a deadline determined by the viscera of institutional science rather than software development!
Upgrading From <v0.5.0
The subject class will attempt to update any of your existing .h5 files to the new format – it should be nondestructive (making a backup of the whole file first and then making backups up any tables that it can’t create a new version of that fully preserves the data from the previous version), but don’t be alarmed when your subject files start updating! We advise, as we always do, making a full backup of your data directory before making this update.
Change references in
launch_autopilot.sh
scripts fromautopilot.core.terminal
orautopilot.core.pilot
toautopilot.agents.terminal
orautopilot.agents.pilot
, respectivelyUpdate any
TrialData
declarations in plugin tasks to use the newTrial_Data
model. SeeNafc.TrialData
for an example. The old pytables IsDescriptor style descriptions will still work, but you will be given a warning every time.We advise updating pigpio to at least
3c23715
to avoid warnings and make use of some new functionality.
Package Structure
The
autopilot.core
module was removed entirely, and was split intoautopilot.agent
: which contains theTerminal
andPilot
classes and a stub metaclassautopilot.utils.loggers
: formerlyautopilot.core.loggers
autopilot.gui
(described in module documentation) now has all the GUI modules, split out by type rather than all in one file.autopilot.data
was created fromautopilot.core.subject
, see below.
autopilot.root
contains root classes:Autopilot_Type
- types and data models for handling and saving live dataAutopilot_Object
- General root metaclass, beneath which will be top-level metaclasses for each of the major class hierarchies that are currently separate.Autopilot_Pref
- Stub for later converting prefs and environmental configuration parameters from their scattered state.
autopilot.exceptions
will be used for more explicit exception handling.Code files were given
755
permissions to allow execution (as opposed to644
)The unused git submodules for pigpio and an old temperature sensor were removed.
Major updates
autopilot.data
is the major change in this release! See the module-level doc page for more detailsautopilot.core.subject
was moved toautopilot.data.subject
and remains the main interface to autopilot data. It was effectively rewritten to accomodate the use of data models, and many of its public methods were made private to make its use more obvious.The subject structure that determines which things are located where has been made into its own class
Subject_Structure
Making new subjects is now done with the
Subject.new()
class method rather than from the__init__
methodThe
open_hdf
method has been replaced with the_h5f
private method that is a more explicit context manager allowing locking read/write access and nonlocking read accessThe subject file now explicitly handles updates between different versions of the Subject class.
Subject will check if the protocol
.json
file has been updated from when it was assigned and automatically update to the new version, logging the change.All trial data is now saved with a
session_uuid
unique identifier column that is an always-unique key in case of any overlapping session IDs from reassignments, etc.Many attributes were replaced by properties that return data models:
Subject.bio
andSubject.info
->Biography
Subject.protocol
->Protocol_Status
, which manages the current trial, session, step, etc. rather than being treated separately
autopilot.data.interfaces
contains classes to relate abstract data models to different representations and formatsinterfaces.datajoint
allows creating datajoint schema definitions from autopilot models using the companion datajoint-babel projectinterfaces.tables
translates pydantic models to HDF5 files
autopilot.data.modeling
has basic types for use in data modelsautopilot.data.models
has the models themselvesautopilot.data.units
stub module for using explicit units instead of anonymous floats
A new set of general model filling widgets -
widgets.input
andwidgets.model
- were created that will eventually replace much of the existing GUI which suffers from code duplication problems from minor variations between representations of parameters, etc.Log
,Log_Format
,LogEntry
,ParseError
,MESSAGE_FORMATS
,LOG_FORMATS
were added to allow programmatic loading of logfiles, andutils.log_parsers
was added to allow recovery of structured data (eg. from logged trial data). Loggers now use the rich logging handler for much more readable logs in stderr.Jack_Sound
classes now put all frames at once into the processing queue, rather than one at a time for much less variability in sound onset jitter. TheJackClient
class was updated to reflect that as well by adding aplay_q_size
argument that controls the size of the deque used to store frames to play (rather than pulling each from a Queue as was done before).JackClient
also has an optionaldisable_gc
(False by default) parameter that allows the garbage collector to be disabled in the process for further (unquantified) jitter reduction.
Minor updates
The bandwidth test (
Bandwidth_Test
widget andPilot.l_bandwidth()
listen) was updated to allow selecting compression, use of a random array, or preserialized messagesThe
Terminal
now closes cleanly (still with a warning of leaked semaphors) without hanging.External processes
external.start_pigpiod()
andexternal.start_jackd()
first check if the process is already startedDigital_Out.set()
now has aresult
argument for faster setting that doesn’t confirm the result whenFalse
Message
s now provide clearer support for compression in serialization, and automatically detect and handle a compressed message when deserializing. This is also carried through to theNet_Node.send()
method.Station.l_kill()
was moved up to the top levelStation
class, rather than being something the Terminal station owns.The
run_script
functions no longer uses the totally unreadable white on green ANSI code combo that I have no idea why i ever used in the first place.A
picamera_legacy
script was added for enabling the picamera on bullseye.The
sounds.Gammatone
sound now accepts afilter_kwargs
dictionary that is passed on to thetimeseries.Gammatone
filter.The
Task
andNafc
classes use the newTrial_Data
style data declarations.Two utility functions
common.walk_dicts()
andcommon.flatten_dicts()
were added to make handling nested dictionaries a bit easier.
Bugfixes
The Subject class would incorrectly overwrite data given a mismatch in system times between the Terminal and Pilot. The subject class should now ideally no longer overwrite anything ever.
The subject class would drop trial data silently if it was not in the TrialData description. En route to making the table automatically expand to accomodate unexpected data, dropped data is now logged as an exception.
The prefs manager handles being launched from within ipython and other processes better, but is still a bit buggy. Now it uses the check used internally in the multiprocessing module to see if a manager can be launched, and falls back to using a standard dictionary if not.
jackd_source
script uses correcthttps://
rather thangit://
protocol.A
_TASK_LIST
was added toutils.registry
to support deprecated task specifications.
Regressions
With additional checking for monotonic increases in
trial_num
and checks that prevent data overwriting, individual writes of trial data are now a bit slower, which should be optimized for when we complete the transition to uniform data models throughout the library.
Prefs
PIGPIOD
-bool
- ifTrue
, start pigpiod on pilot startStubs were created for converting the prefs to using data models, but they have not been filled yet.
A
AUTOPILOT_NO_PREFS_MANAGER
environment variable now controls the use of a multiprocessing manager explicit. Documentation for environmental variables is forthcoming.A
AUTOPILOT_WARN_DEFAULTS
environment variable controls whether warnings should be printed for when a default pref value is retrieved, because that warning is a good one but can be really annoying.
Packaging & Dependencies
Autopilot is now packaged with Poetry! This allows for fully deterministic installation with the poetry.lock file and updates from the old setuptools style of dependency specification.
The source repository has moved from https://github.com/wehr-lab/autopilot to https://github.com/auto-pi-lot/autopilot
MANIFEST.in
has been replaced by theinclude
field inpyproject.toml
autopilot.__version__
is now determined byimportlib.metadata
and specified in thepyproject.toml
file rather than in the__init__.py
fileblosc
was replaced withblosc2
New dependencies
global
pydantic (^1.9.0)
parse (^1.19.0)
rich (^11.2.0)
validators (^0.18.2)
docs
autodoc_pydantic (^1.7.0)
myst_parser (^0.17.2)
Version Bumps
pyzmq 18.1.* -> ^22.3.0
tornado >=5.0.0 -> ^6.1.0
numpy >=1.16.5 -> ^1.20.0
scipy >=1.6.0 -> ^1.7.0
pandas >=0.19.2 -> ^1.3.0 on python 3.7 and ^1.4.0 on >=3.8
tables >=3.4.2 -> ^3.7.0
Sphinx >=3.1.2 -> ^4.3.1
A lot more dependencies were taken from being implicit versions to explicit by the conversion to using Poetry…
Docs
Configuration was moved to its own page, documenting setting up the system as well as the contents of the user directory.
A faq page was stubbed out (but is still pretty skeletal)
The overview was updated with some more information in the module tour
Virtual environment usage was moved from the setup page to its own subpage linked from the FAQ.
A
make serve
option was added to the docs makefile that makes use of sphinx-autobuild to livereload docs while editing them.autopilot_theme.css
was updated to be compatible with the new version of sphinx-rtd-theme that apparently changed the way that TOC buttons were made, as well as remove incorrect references to fonts no longer packaged.The
autodoc_pydantic
andmyst_parser
extensions were added – and we will be moving towards using MyST rather than hellish ReST for future narrative docs!Private methods and functions are now no longer rendered in the main documentation, and the library will over time use the public/private distinction more systematically to make it more understandable.
Examples was split off into its own folder and links to wiki plugins. Blink was moved with it
Tests
We have started importing some of the pigpio mocking tools from the People’s Ventilator Project to start writing GPIO tests!