v0.3.0 (August 4, 2020)¶
Python 3 - We’ve finally made it to Python 3! Specifically we have brought Autopilot up to compatibility with Python 3.8. The Spinnaker SDK is currently only available through Python 3.7, so we have tested Autopilot on 3.6, 3.7, and 3.8. I will not attempt to keep Autopilot compatible with Python 2, but no decision has been made about compatibility with other versions of Python 3. Until then, expect that Autopilot will attempt to keep up with major version changes. The switch also let up update PySide (Qt library used for the GUI) to PySide2, which uses Qt5 and has a whole raft of other improvements.
Continuous Data Handling - The
networkingmodules have been improved to handle continuous data (eg. streaming data, generally non-trialwise or non-event-sampled data). Continuous data can be set in a Task description either with a
tablescolumn descriptor as trial data is, but also can be set as
'infer', for which the
Subjectclass will wait until it receives the first data and automatically create a
tablescolumn depending on its type and shape. While previously we intended to nudge users to be explicit about declaring their data, this was necessary to allow for data that might be variable in type and shape to be included in a Task – eg. it should be possible to record video data without needing to specify the resolution or bit depth as a hardcoded parameter in a task class. I have come to like type inference, and may make it a general practice for all types of data. That would potentially allow tasks to be written without explicitly declaring the data that they produce at all, but I haven’t decided if that’s a good thing or not yet.
The GPIO engine has been rebuilt, relying more on
pigpio’s function interface. This means that GPIO timing is now ~microsecond precise, important for reward delivery, LED flashing, and a number of other basic infrastructural needs. The reorganization of hardware modules resulted in general
Digital_Outmetaclasses, making common operations like setting polarity, triggers, and pullup/down resistors much easier.
Setup has been greatly improved. This includes proper packaging and installation with setuptools & sk-build, allowing us to finally join PyPI :) https://pypi.org/project/auto-pi-lot/ . Setup has been unified into a single npyscreen-based set of prompts that allow the user to run scripts to install libraries or configure their environment (also see
prefs, configure hardware objects (based on some very fun signature introspection), setup autopilot as a systemd service, etc. Getting started with Autopilot is now three commands!:
pip install auto-pi-lot autopilot.setup.setup_autopilot ~/autopilot/launch_autopilot.sh
Logging level is now set from
prefs, so where before, eg. every message through the networking modules would be logged to stdout, now only warnings and exceptions are. This gives a surprisingly large performance boost.
Logging has also been much improved in
networkingmodules, where rather than an awkward
do_loggingflag that was used to avoid logging performance-critical events like streaming data, logging is controlled by log level throughout the system. By default, logging of most messages is set at
debuglevel so they don’t drown out important messages in the logs as they used to.
Networking modules now only deserialize messages if they are the final recipient, saving lots of processing time – particularly with streamed arrays.
Messageobjects also only re-serialize messages if they have been changed. Message structure has been changed such that serialized messages are now of the general format:
[sender, (optional) intermediate_node_1, intermediate_node_2, ... final_recipient, message_contents]
Configuration will continue to be a point of improvement, but a few minor updates were made:
prefs.CONFIGwill be used to signal multiple, potentially overlapping agent configurations, each of which may have their own system dependencies, external daemons, etc. Eg. a Pilot could be configured to play audio (which requires a jackd daemon to be started before Autopilot) and video (which requires Autopilot to be started in a X session). Checks of
==to reflect that.
prefs.HARDWARE, and now allows hardware to be configured with dictionaries rather than integers only. Initially
PINSwas meant to just contain pin numbering for GPIO objects, but having a single point of hardware configuration is preferable.
Task.init_hardware()now respects all parameters set in
Throughout the code, minimal
get_thistype methods have begun to be replaced with
@propertyattributes. This is because a) I love them and think they are magical, but b) will also be building Autopilot’s closed-loop infrastructure around a Qt-style signal/slot architecture that wraps
@propertyattributes so they can be
.connectedto one another easily.
Previously it was possible to control presentation by groups of stimuli, but now it is possible to control the presentation frequency of individual stimuli.
PySide2has proper support for CSS Stylesheets, so the design of Autopilot’s GUI has been marginally improved, a process that will continue in the ceaseless quest for aesthetic perfection.
Several setup routines have been added to make installation of opencv, pyspin, etc. easier. I also wrote a routine to
download_box()files from a URL, which is mysteriously hard to do.
I have also opened up a message board in google groups to make feature requests and discuss use and development, hope to see you there :)
TRANSFORMS have been introduced!!!
Transformobjects have a
process()method that, well, transforms data in some way. Multiple transforms can be added together to make a transformation chain. This module is still very young and doesn’t have a developed API, but will be built to to automatic type compatibility checking, coersion, parallelization, and rhythm (FIFO/FILO) control. Transforms are implemented with different modalities (image, selection, logical) that imply different types of input and output data structures, but the hierarchical structure of the modules is still quite flat.
HARDWARE has been substantially refactored to give objects an appropriate inheritance structure. This substantially reduces effort duplication across hardware objects and makes a bunch of obvious capabilities available to all of them, for example all hardware objects are now network (
init_networking()) and logging (
cameras.Camera_CVclass allows webcams/other simple cameras to be accessed through OpenCV, and the
cameras.Camera_Spinnakerclass allows FLIR and other cameras to be accessed through the Spinnaker SDK. Cameras are capable of encoding videos locally (with x264), streaming frames over the network, and making acquired frames available to other objects on the same computer. The
Camera_Spinnakerclass provides simple
@propertysetter/getter methods for common parameters, but also makes all
PySpinattributes available to the user with its
cameras.Camerametaclass is written so that new camera types can be added by overriding a few methods. A new
Video_Childcan be used to run a camera on a Child agent.
9DOF Motion Sensor: The
i2c.I2C_9DOFclass can use the LSM9DS1 sensor to collect accelerometer, magnetometer, and gyroscopic data to compute unambiguous position and orientation information. We will be including calibration and computation routines that make it easier to extract properties of interest – eg. computing vertical motion by combining readings from the three sensors.
Temperature Sensor: The
i2c.MLX90640class can use the MLX90640 sensor to measure temperature. The sensor is 32x24px, which the class can
interpolate(). The class also allows frames to be integrated and averaged over time, substantially reducing noise. I modified the driver library to enable capture at the full 64fps on the Raspberry Pi.
NETWORKING modules can stream continuous data better in a few ways:
Net_Nodemodules were given a
get_stream()method that lets objects, well, stream data. Specifically, they are given a
queue.Queueto shovel data into, which is then picked up by a dedicated
zmq.Socketin its own thread, which handles batching, serialization, and load balancing. Streamed messages are batched (ie. contain multiple messages), but behave like normal message when received – they are split and contain an
inner_keythat is used to call the
listenwith each message (see
networkingobjects also now compress arrays-in-transit with the superfast blosc compression library. This increases their throughput dramatically, as many data streams in neuroscience are relatively low-entropy (eg. the pixels in a video of a mostly-white arena are mostly unchanged frame-to-frame and are thus highly compressible). See the
STIMULI - The
JackClientcan now play continuous sounds rather than discrete sounds. An example can be found in the
Nafc_Gaptask, which plays continuous white noise. All sounds now have a
play_continuous()method, which continually dumps samples in a cycle into a queue for the
JackClient. The continuous sound will be interrupted if another sound has its
Jack_Sound.play()method called, but the continuous sound will resume seamlessly even if number of samples in the played sound aren’t a multiple of the jack buffer size. We use this for gaps in noise (using the new
Gapclass), which we have confirmed are sample-accurate.
UI & VIZ
Videowindow has been created to display streaming video. The
Terminal_Networking.l_continuous()method meters frames such that even if high-speed video is being acquired, frames are only sent at a rate of
Videoclass uses the
ImageItem_TimedUpdateobject, a slight modification of
pyqtgraph.ImageItem, that calls its
updatemethod according to a
plots_menumenu has been added to the Terminal, and a GUI dialog (
gui.Psychometric) has been added to create simple psychometric curves with the
viz.psychometricmodule, which uses altair. Plans for developing visualization are described in To-Do.
gui.pop_dialog()function simplifies displaying messages to the user using the Terminal UI. This was an initial step towards improving status/error reporting from other agents, further detailed in To-Do.
Some objects, particularly several
guiobjects, had the old mouse/mice terminology updated to subject/subjects.
Net_Nodeobjects were only implicitly destroyed by their
releasemethod which ends the threaded loop by setting the
Pilotobjects were not prevented from running multiple tasks at a time. This led to some very confusing and hard-to-debug problems, as well as frequent conflicts over hardware access and resources. Typically what would happen is the Terminal would send a
STARTmessage to begin a task, and if it wouldn’t received a message receipt quickly enough would resend it, resulting in two tasks being started – but this would happen whenever two
STARTmessages were sent to a pilot. This was fixed with a simple check of
Pilot.statebefore a task is initialized. Similar bugs were fixed in
Subjectclass would sometimes fail to get and increment the trial session. This has been fixed by saving the session number as an attribute in the
Subjectclass would reset the session counter even when the same task was being reassigned (eg. if updated), now it preserves session number if the protocol name is unchanged.
update_protocols()method didn’t report which subjects had their protocols updated, and so if there was some exception when setting new protocols it happened silently, making it so a user would never know their task was never updated. This was fixed with a noisier protocol update method for the Subject class and by displaying a list of subjects that were updated after the method is called.
Correction trials were being calculated incorrectly by the
Stim_Manager, such that rather than only repeating a stimulus if the subject got the previous trial incorrect, the stimulus was always repeated at least once.
Modified versions of external libraries have been added as git submodules in autopilot/external.
Requirements files have been split out to better differentiate between different agents and use-cases. eg. requirements for Terminal agents are in
requirements/requirements_terminal.txt, requirements for build the docs are in
requirements/requirements_docs.txt, etc. This is a temporary arrangement, as a future design goal is restructuring setup routines so that they can flexibly install components as-needed (see To-Do)
autopilot.core.hardwarehas been refactored into its own module,
autopilot.hardware, and split by device type, currently…
autopilot.gpio- devices that use the GPIO pins for standard digital I/O logic
autopilot.i2c- devices that use the GPIO pins for I2C
The docs are hosted on readthedocs again, so the docs structure has been collapsed to a single folder without built documentation
The autopilot user directory is now
/usr/autopilot, which was always a mistake anyway. Autopilot creates a wayfinder
~/.autopilotfile that is used to find the user directory if it’s set elsewhere
External libraries can now be built and packaged along with autopilot using cmake, see CMakeLists.txt. Still uh having a little bit of trouble getting this to work, so code is in place to build and package the custom pigpio repo and jack audio but this will likely need some more work.
Added the ability to return absolute timestamps rather than system ticks. pigpio typically returns 1 32-bit integer of ticks since the daemon started, absolute timestamps are 64-bit, so the pigpio daemon and python interface (pi) were given two new methods:
synchronize gets several (default 5) sets of paired timestamps and ticks using get_sync_time. It then computes an offset for translating ticks to timestamps
ticks_to_timestamp converts ticks to timestamps based on the offset found with synchronize
get_current_time sends two requests to the daemon to get the seconds and microseconds of the complete timestamp and returns an isoformatted string
Removed building examples by default which require additional dependencies
When using the raspi I2C driver, the baudrate would never be set to 1MHz, which is necessary to achieve full 64fps. This was fixed to use 1MHz by default.
Message confirmation (holding a message to resend if confirmation isn’t received) was causing a huge amount of problems and needed to be rethought. There are in general very low rates (near-zero) of messages being dropped without some larger bug causing them, so confirmation has been disabled for now.
The same is true of
heartbeat()- which polled for status of connected pilots. this will be repaired and restored, as the terminal currently has a pretty bad idea of the status of what’s connected to it. this will be part of a broader networking overhaul