v0.5.0

v0.5.1 - Maintenance patch

Fixes to keep it sort of running and catch up on dependency drift….

Still haven’t gotten pyside6 to work yet though which is a PROBLEM

  • #211

    • Fix: #209 - update location of dtoverlays

    • Convert travis-based tests to github actions so they actually run

    • Don’t raise NotImplementedError in datajoint module on import - what the hell was i thinking lmao

    • don’t access deprecated alias np.float, just use float

    • fix a bunch of docs stuff to make it build

    • update dep versions and remove some upper bounds

v0.5.0a0 - ~The Data Modeling Edition~

A lot. Namely a whole rewriting of the autopilot.data module.

This is being released as an alpha version while we finish working out a few of the kinks in the data modeling system because Jonny is finishing their dissertation and has a deadline determined by the viscera of institutional science rather than software development!

Upgrading From <v0.5.0

  • The subject class will attempt to update any of your existing .h5 files to the new format – it should be nondestructive (making a backup of the whole file first and then making backups up any tables that it can’t create a new version of that fully preserves the data from the previous version), but don’t be alarmed when your subject files start updating! We advise, as we always do, making a full backup of your data directory before making this update.

  • Change references in launch_autopilot.sh scripts from autopilot.core.terminal or autopilot.core.pilot to autopilot.agents.terminal or autopilot.agents.pilot, respectively

  • Update any TrialData declarations in plugin tasks to use the new Trial_Data model. See Nafc.TrialData for an example. The old pytables IsDescriptor style descriptions will still work, but you will be given a warning every time.

  • We advise updating pigpio to at least 3c23715 to avoid warnings and make use of some new functionality.

Package Structure

  • The autopilot.core module was removed entirely, and was split into

    • autopilot.agent: which contains the Terminal and Pilot classes and a stub metaclass

    • autopilot.utils.loggers: formerly autopilot.core.loggers

    • autopilot.gui (described in module documentation) now has all the GUI modules, split out by type rather than all in one file.

    • autopilot.data was created from autopilot.core.subject, see below.

  • autopilot.root contains root classes:

    • Autopilot_Type - types and data models for handling and saving live data

    • Autopilot_Object - General root metaclass, beneath which will be top-level metaclasses for each of the major class hierarchies that are currently separate.

    • Autopilot_Pref - Stub for later converting prefs and environmental configuration parameters from their scattered state.

  • autopilot.exceptions will be used for more explicit exception handling.

  • Code files were given 755 permissions to allow execution (as opposed to 644)

  • The unused git submodules for pigpio and an old temperature sensor were removed.

Major updates

  • autopilot.data is the major change in this release! See the module-level doc page for more details

    • autopilot.core.subject was moved to autopilot.data.subject and remains the main interface to autopilot data. It was effectively rewritten to accomodate the use of data models, and many of its public methods were made private to make its use more obvious.

      • The subject structure that determines which things are located where has been made into its own class Subject_Structure

      • Making new subjects is now done with the Subject.new() class method rather than from the __init__ method

      • The open_hdf method has been replaced with the _h5f private method that is a more explicit context manager allowing locking read/write access and nonlocking read access

      • The subject file now explicitly handles updates between different versions of the Subject class.

      • Subject will check if the protocol .json file has been updated from when it was assigned and automatically update to the new version, logging the change.

      • All trial data is now saved with a session_uuid unique identifier column that is an always-unique key in case of any overlapping session IDs from reassignments, etc.

      • Many attributes were replaced by properties that return data models:

    • autopilot.data.interfaces contains classes to relate abstract data models to different representations and formats

    • autopilot.data.modeling has basic types for use in data models

    • autopilot.data.models has the models themselves

    • autopilot.data.units stub module for using explicit units instead of anonymous floats

  • A new set of general model filling widgets - widgets.input and widgets.model - were created that will eventually replace much of the existing GUI which suffers from code duplication problems from minor variations between representations of parameters, etc.

  • Log, Log_Format, LogEntry, ParseError, MESSAGE_FORMATS, LOG_FORMATS were added to allow programmatic loading of logfiles, and utils.log_parsers was added to allow recovery of structured data (eg. from logged trial data). Loggers now use the rich logging handler for much more readable logs in stderr.

  • Jack_Sound classes now put all frames at once into the processing queue, rather than one at a time for much less variability in sound onset jitter. The JackClient class was updated to reflect that as well by adding a play_q_size argument that controls the size of the deque used to store frames to play (rather than pulling each from a Queue as was done before). JackClient also has an optional disable_gc (False by default) parameter that allows the garbage collector to be disabled in the process for further (unquantified) jitter reduction.

Minor updates

  • The bandwidth test (Bandwidth_Test widget and Pilot.l_bandwidth() listen) was updated to allow selecting compression, use of a random array, or preserialized messages

  • The Terminal now closes cleanly (still with a warning of leaked semaphors) without hanging.

  • External processes external.start_pigpiod() and external.start_jackd() first check if the process is already started

  • Digital_Out.set() now has a result argument for faster setting that doesn’t confirm the result when False

  • Message s now provide clearer support for compression in serialization, and automatically detect and handle a compressed message when deserializing. This is also carried through to the Net_Node.send() method.

  • Station.l_kill() was moved up to the top level Station class, rather than being something the Terminal station owns.

  • The run_script functions no longer uses the totally unreadable white on green ANSI code combo that I have no idea why i ever used in the first place.

  • A picamera_legacy script was added for enabling the picamera on bullseye.

  • The sounds.Gammatone sound now accepts a filter_kwargs dictionary that is passed on to the timeseries.Gammatone filter.

  • The Task and Nafc classes use the new Trial_Data style data declarations.

  • Two utility functions common.walk_dicts() and common.flatten_dicts() were added to make handling nested dictionaries a bit easier.

Bugfixes

  • The Subject class would incorrectly overwrite data given a mismatch in system times between the Terminal and Pilot. The subject class should now ideally no longer overwrite anything ever.

  • The subject class would drop trial data silently if it was not in the TrialData description. En route to making the table automatically expand to accomodate unexpected data, dropped data is now logged as an exception.

  • The prefs manager handles being launched from within ipython and other processes better, but is still a bit buggy. Now it uses the check used internally in the multiprocessing module to see if a manager can be launched, and falls back to using a standard dictionary if not.

  • jackd_source script uses correct https:// rather than git:// protocol.

  • A _TASK_LIST was added to utils.registry to support deprecated task specifications.

Regressions

  • With additional checking for monotonic increases in trial_num and checks that prevent data overwriting, individual writes of trial data are now a bit slower, which should be optimized for when we complete the transition to uniform data models throughout the library.

Prefs

  • PIGPIOD - bool - if True, start pigpiod on pilot start

  • Stubs were created for converting the prefs to using data models, but they have not been filled yet.

  • A AUTOPILOT_NO_PREFS_MANAGER environment variable now controls the use of a multiprocessing manager explicit. Documentation for environmental variables is forthcoming.

  • A AUTOPILOT_WARN_DEFAULTS environment variable controls whether warnings should be printed for when a default pref value is retrieved, because that warning is a good one but can be really annoying.

Packaging & Dependencies

  • Autopilot is now packaged with Poetry! This allows for fully deterministic installation with the poetry.lock file and updates from the old setuptools style of dependency specification.

  • The source repository has moved from https://github.com/wehr-lab/autopilot to https://github.com/auto-pi-lot/autopilot

  • MANIFEST.in has been replaced by the include field in pyproject.toml

  • autopilot.__version__ is now determined by importlib.metadata and specified in the pyproject.toml file rather than in the __init__.py file

  • blosc was replaced with blosc2

  • New dependencies

    • global

      • pydantic (^1.9.0)

      • parse (^1.19.0)

      • rich (^11.2.0)

      • validators (^0.18.2)

    • docs

      • autodoc_pydantic (^1.7.0)

      • myst_parser (^0.17.2)

  • Version Bumps

    • pyzmq 18.1.* -> ^22.3.0

    • tornado >=5.0.0 -> ^6.1.0

    • numpy >=1.16.5 -> ^1.20.0

    • scipy >=1.6.0 -> ^1.7.0

    • pandas >=0.19.2 -> ^1.3.0 on python 3.7 and ^1.4.0 on >=3.8

    • tables >=3.4.2 -> ^3.7.0

    • Sphinx >=3.1.2 -> ^4.3.1

    • A lot more dependencies were taken from being implicit versions to explicit by the conversion to using Poetry…

Docs

  • Configuration was moved to its own page, documenting setting up the system as well as the contents of the user directory.

  • A faq page was stubbed out (but is still pretty skeletal)

  • The overview was updated with some more information in the module tour

  • Virtual environment usage was moved from the setup page to its own subpage linked from the FAQ.

  • A make serve option was added to the docs makefile that makes use of sphinx-autobuild to livereload docs while editing them.

  • autopilot_theme.css was updated to be compatible with the new version of sphinx-rtd-theme that apparently changed the way that TOC buttons were made, as well as remove incorrect references to fonts no longer packaged.

  • The autodoc_pydantic and myst_parser extensions were added – and we will be moving towards using MyST rather than hellish ReST for future narrative docs!

  • Private methods and functions are now no longer rendered in the main documentation, and the library will over time use the public/private distinction more systematically to make it more understandable.

  • Examples was split off into its own folder and links to wiki plugins. Blink was moved with it

Tests