Welcome!¶
Welcome to Samlab: the Sandia Machine Learning Laboratory. Samlab provides a set of tools for creating, monitoring, and analyzing machine learning experiments.
Documentation¶
Installation¶
Using Pip / Easy Install¶
If your package manager doesn’t support Samlab, or doesn’t have the latest version, your next option should be Python setup tools like pip. You can always install the latest stable version of Samlab and its required dependencies using:
$ pip install samlab
… following that, you’ll be able to use all of Samlab’s features.
From Source¶
Finally, if you want to work with the latest, bleeding-edge Samlab goodness, you can install it using the source code:
$ git clone https://github.com/sandialabs/samlab
$ cd samlab
$ sudo python setup.py install
The setup script installs Samlab’s required dependencies and copies Samlab into your Python site-packages directory, ready to go.
Overview¶
Samlab provides a set of tools that you can use to create, monitor, and analyze your machine learning experiments:
Command Line Tools¶
samlab-gputop provides a colorful, updating display of how hard your GPUs are working.
Dashboard¶
The Samlab dashboard is a web server providing a graphical user interface for annotating images, monitoring experiments, and analyzing results. Unlike comparable tools (Tensorboard, Visdom, Vott, …), Samlab adapts to your data instead of forcing you to store it in databases, binary records, or other organizations. In most cases, a Samlab data adapter can be written in fewer lines of code than you’d need to convert your data to a tool-specific format, and avoids the pitfalls of data duplication.
Interaction¶
Samlab functions make it easy to see your experiments’ progress, and interrupt an experiment gracefully, without data loss.
Notebooks¶
Samlab provides convenience functions to simplify displaying data in Jupyter notebooks.
Torch¶
Samlab provides convenience functions for working with PyTorch, such as a function to automatically choose a Torch device based on load, making it easy to run multiple experiments simultaneously without accidentally selecting the same device.
API Reference¶
samlab module¶
Tools for managing machine learning experiments.
samlab.dashboard module¶
Warning
module ‘samlab.dashboard’ undocumented
- class samlab.dashboard.AllowConverter(map, *items)[source]¶
Bases:
werkzeug.routing.BaseConverterWarning
class ‘samlab.dashboard.AllowConverter’ undocumented
- class samlab.dashboard.ExcludeConverter(map, *items)[source]¶
Bases:
werkzeug.routing.BaseConverterWarning
class ‘samlab.dashboard.ExcludeConverter’ undocumented
samlab.dashboard.acl module¶
Warning
module ‘samlab.dashboard.acl’ undocumented
- class samlab.dashboard.acl.ForbidAll[source]¶
Bases:
objectAccess control list strategy that prevents anyone from doing anything.
samlab.dashboard.authentication module¶
Warning
module ‘samlab.dashboard.authentication’ undocumented
samlab.dashboard.backend module¶
Warning
module ‘samlab.dashboard.backend’ undocumented
samlab.dashboard.backend.documents module¶
Warning
module ‘samlab.dashboard.backend.documents’ undocumented
- class samlab.dashboard.backend.documents.Directory(*, name, root, pattern='.*\\.(html|txt)')[source]¶
Bases:
samlab.dashboard.backend.documents.DocumentCollection,watchdog.events.FileSystemEventHandlerWarning
class ‘samlab.dashboard.backend.documents.Directory’ undocumented
- get(index)[source]¶
Return a document by index.
- Parameters
index (int, required) – The index of the image to return.
- Returns
image – If
str, the filesystem path of the image.- Return type
strornumpy.ndarray
- property name¶
Warning
property ‘samlab.dashboard.backend.documents.Directory.name’ undocumented
- class samlab.dashboard.backend.documents.DocumentCollection[source]¶
Bases:
abc.ABCWarning
class ‘samlab.dashboard.backend.documents.DocumentCollection’ undocumented
- abstract get(index)[source]¶
Return a document by index.
- Parameters
index (int, required) – The index of the image to return.
- Returns
image – If
str, the filesystem path of the image.- Return type
strornumpy.ndarray
- abstract property name¶
Warning
property ‘samlab.dashboard.backend.documents.DocumentCollection.name’ undocumented
- property service¶
Warning
property ‘samlab.dashboard.backend.documents.DocumentCollection.service’ undocumented
samlab.dashboard.backend.favorites module¶
Warning
module ‘samlab.dashboard.backend.favorites’ undocumented
- class samlab.dashboard.backend.favorites.Favorites[source]¶
Bases:
abc.ABCWarning
class ‘samlab.dashboard.backend.favorites.Favorites’ undocumented
- property name¶
Return the name of this backend.
- property service¶
Return the service type implemented by this backend.
- class samlab.dashboard.backend.favorites.JSONFile(storage)[source]¶
Bases:
samlab.dashboard.backend.favorites.FavoritesWarning
class ‘samlab.dashboard.backend.favorites.JSONFile’ undocumented
samlab.dashboard.backend.images module¶
Warning
module ‘samlab.dashboard.backend.images’ undocumented
- class samlab.dashboard.backend.images.COCO(*, name, annotations, images)[source]¶
Bases:
samlab.dashboard.backend.images.ImageCollectionWarning
class ‘samlab.dashboard.backend.images.COCO’ undocumented
- property categories¶
Return a list of existing categories for the collection.
- metadata(index)[source]¶
Warning
method ‘samlab.dashboard.backend.images.COCO.metadata’ undocumented
- property name¶
Warning
property ‘samlab.dashboard.backend.images.COCO.name’ undocumented
- class samlab.dashboard.backend.images.Directory(*, name, root, pattern='.*\\.(png|jpg|jpeg|PNG|JPG|JPEG)')[source]¶
Bases:
samlab.dashboard.backend.images.ImageCollectionWarning
class ‘samlab.dashboard.backend.images.Directory’ undocumented
- property name¶
Warning
property ‘samlab.dashboard.backend.images.Directory.name’ undocumented
- class samlab.dashboard.backend.images.ImageCollection[source]¶
Bases:
abc.ABCWarning
class ‘samlab.dashboard.backend.images.ImageCollection’ undocumented
- property categories¶
Return a list of existing categories for the collection.
- metadata(index)[source]¶
Warning
method ‘samlab.dashboard.backend.images.ImageCollection.metadata’ undocumented
- abstract property name¶
Warning
property ‘samlab.dashboard.backend.images.ImageCollection.name’ undocumented
- put_bboxes(index, bboxes)[source]¶
Warning
method ‘samlab.dashboard.backend.images.ImageCollection.put_bboxes’ undocumented
- put_tags(index, tags)[source]¶
Warning
method ‘samlab.dashboard.backend.images.ImageCollection.put_tags’ undocumented
- property service¶
Warning
property ‘samlab.dashboard.backend.images.ImageCollection.service’ undocumented
- class samlab.dashboard.backend.images.ImageNet(*, name, root, split)[source]¶
Bases:
samlab.dashboard.backend.images.ImageCollectionWarning
class ‘samlab.dashboard.backend.images.ImageNet’ undocumented
- property name¶
Warning
property ‘samlab.dashboard.backend.images.ImageNet.name’ undocumented
samlab.dashboard.backend.layouts module¶
Warning
module ‘samlab.dashboard.backend.layouts’ undocumented
- class samlab.dashboard.backend.layouts.JSONFile(storage)[source]¶
Bases:
samlab.dashboard.backend.layouts.LayoutsWarning
class ‘samlab.dashboard.backend.layouts.JSONFile’ undocumented
- class samlab.dashboard.backend.layouts.Layouts[source]¶
Bases:
abc.ABCWarning
class ‘samlab.dashboard.backend.layouts.Layouts’ undocumented
- abstract get(*, lid)[source]¶
Warning
method ‘samlab.dashboard.backend.layouts.Layouts.get’ undocumented
- property name¶
Warning
property ‘samlab.dashboard.backend.layouts.Layouts.name’ undocumented
- abstract put(*, content)[source]¶
Warning
method ‘samlab.dashboard.backend.layouts.Layouts.put’ undocumented
- property service¶
Warning
property ‘samlab.dashboard.backend.layouts.Layouts.service’ undocumented
samlab.dashboard.backend.timeseries module¶
Warning
module ‘samlab.dashboard.backend.timeseries’ undocumented
- class samlab.dashboard.backend.timeseries.Directory(*, name, root, pattern='.*\\.(csv|CSV)')[source]¶
Bases:
samlab.dashboard.backend.timeseries.TimeseriesCollection,watchdog.events.FileSystemEventHandlerWarning
class ‘samlab.dashboard.backend.timeseries.Directory’ undocumented
- get(index)[source]¶
Return a timeseries by index.
- Parameters
index (int, required) – The index of the timeseries to return.
- Returns
timeseries – Numpy array containing timeseries fields.
- Return type
- property name¶
Warning
property ‘samlab.dashboard.backend.timeseries.Directory.name’ undocumented
- class samlab.dashboard.backend.timeseries.TimeseriesCollection[source]¶
Bases:
abc.ABCWarning
class ‘samlab.dashboard.backend.timeseries.TimeseriesCollection’ undocumented
- abstract get(index)[source]¶
Return a timeseries by index.
- Parameters
index (int, required) – The index of the timeseries to return.
- Returns
timeseries – Numpy array containing timeseries fields.
- Return type
- abstract property name¶
Warning
property ‘samlab.dashboard.backend.timeseries.TimeseriesCollection.name’ undocumented
- property service¶
Warning
property ‘samlab.dashboard.backend.timeseries.TimeseriesCollection.service’ undocumented
samlab.dashboard.credentials module¶
Warning
module ‘samlab.dashboard.credentials’ undocumented
- class samlab.dashboard.credentials.ExactMatch(username='test', password='test')[source]¶
Bases:
objectAllow credentials that match the given username and password.
- class samlab.dashboard.credentials.LDAP(server, user_dn, timeout=5)[source]¶
Bases:
objectCheck credentials against an LDAP server.
samlab.dashboard.service module¶
Warning
module ‘samlab.dashboard.service’ undocumented
samlab.dashboard.service.asynchronous module¶
Warning
module ‘samlab.dashboard.service.asynchronous’ undocumented
- samlab.dashboard.service.asynchronous.connect()[source]¶
Warning
function ‘samlab.dashboard.service.asynchronous.connect’ undocumented
samlab.dashboard.service.backends module¶
Warning
module ‘samlab.dashboard.service.backends’ undocumented
samlab.dashboard.service.basic module¶
Warning
module ‘samlab.dashboard.service.basic’ undocumented
- samlab.dashboard.service.basic.get_identity()[source]¶
Warning
function ‘samlab.dashboard.service.basic.get_identity’ undocumented
- samlab.dashboard.service.basic.get_index()[source]¶
Warning
function ‘samlab.dashboard.service.basic.get_index’ undocumented
- samlab.dashboard.service.basic.get_permissions()[source]¶
Warning
function ‘samlab.dashboard.service.basic.get_permissions’ undocumented
samlab.dashboard.service.documents module¶
Warning
module ‘samlab.dashboard.service.documents’ undocumented
samlab.dashboard.service.favorites module¶
Warning
module ‘samlab.dashboard.service.favorites’ undocumented
samlab.dashboard.service.images module¶
Warning
module ‘samlab.dashboard.service.images’ undocumented
- samlab.dashboard.service.images.get_image(name, index)[source]¶
Warning
function ‘samlab.dashboard.service.images.get_image’ undocumented
- samlab.dashboard.service.images.get_image_collection(name)[source]¶
Warning
function ‘samlab.dashboard.service.images.get_image_collection’ undocumented
- samlab.dashboard.service.images.get_image_collection_categories(name)[source]¶
Warning
function ‘samlab.dashboard.service.images.get_image_collection_categories’ undocumented
- samlab.dashboard.service.images.get_image_metadata(name, index)[source]¶
Warning
function ‘samlab.dashboard.service.images.get_image_metadata’ undocumented
samlab.dashboard.service.layouts module¶
Warning
module ‘samlab.dashboard.service.layouts’ undocumented
samlab.dashboard.service.notify module¶
Warning
module ‘samlab.dashboard.service.notify’ undocumented
samlab.dashboard.service.timeseries module¶
samlab.debounce module¶
Adapted from https://github.com/salesforce/decorator-operations.
samlab.interactive module¶
Utilities to simplify user interaction during experiments.
- class samlab.interactive.Progress(desc=None, unit=None)[source]¶
Bases:
objectDisplay a graphical progress bar while iterating over a sequence.
- class samlab.interactive.Stop(timeout=5.0)[source]¶
Bases:
objectHandle interrupts so training can be interrupted gracefully.
Create an instance of
samlab.interactive.Stopand check its triggered property periodically during training. If triggered is True then the user has interrupted the process, either with CTRL-C or the Jupyter Interrupt Kernel button.- property triggered¶
True if the user has interrupted the process, False otherwise.
samlab.notebook module¶
Integration with Jupyter notebooks.
samlab.torch module¶
Functionality for simplifying work with Pytorch.
- samlab.torch.select_device()[source]¶
Automatically choose a Torch device to be used for subsequent computation.
Automatically chooses GPUs over CPUs, prioritizing GPUs that are lightly loaded.
Note
Always returns a valid device, but there is no guarantee that a given device will have sufficient resources for a given computation.
samlab.train module¶
Utilities to simplify training artifacts.
- class samlab.train.EarlyStop(patience=10, delta=0)[source]¶
Bases:
objectStop training if a loss doesn’t improve within N iterations.
- Parameters
patience (integer, optional) –
- property triggered¶
Warning
property ‘samlab.train.EarlyStop.triggered’ undocumented
- class samlab.train.Loss(delta=0)[source]¶
Bases:
objectKeeps track of whether a loss value has improved.
- property value¶
Warning
property ‘samlab.train.Loss.value’ undocumented
- samlab.train.k_fold(dataset, n=5, k=2, validation=0.2, count=None)[source]¶
Return sets of indices partitioning a dataset for K-fold cross validation.
Compatibility¶
A quick disclaimer on backwards-compatibility for Samlab users:
Samlab follows the Semantic Versioning standard for assigning version numbers in a way that has specific meaning. As of this writing Samlab releases are still in the 0.y.z development phase, which means (among other things) that the API may change at any time. We try not to be abusive about it, but you should be prepared for the occasional bump on the road to the 1.0 release.
Developers¶
Even if you’re not in a position to contribute code to Samlab, there are many ways you can help the project out:
If you use Samlab let us know about it!
Even better, cite Samlab when you publish your work.
Let us know if there are any problems.
Help us document Samlab.
Spread the word!
Getting Started¶
If you haven’t already, you’ll want to get familiar with the Samlab repository at http://github.com/sandialabs/samlab … there, you’ll find the Samlab sources, issue tracker, and wiki.
Next, you’ll need to install Samlab’s dependencies, which you can find in setup.py. Then, you’ll be ready to get Samlab’s source code and use setuptools to install it. To do this, you’ll almost certainly want to use “develop mode”. Develop mode is a a feature provided by setuptools that links the Samlab source code into the install directory instead of copying it … that way you can edit the source code in your git sandbox, and you don’t have to re-install it to test your changes:
$ git clone https://github.com/sandialabs/samlab.git
$ cd samlab
$ python setup.py develop
Versioning¶
Samlab version numbers follow the Semantic Versioning standard.
Coding Style¶
The Samlab source code follows the PEP-8 Style Guide for Python Code.
Running Regression Tests¶
To run the Samlab test suite, simply run regression.py from the top-level source directory:
$ cd samlab
$ python regression.py
The tests will run, providing feedback on successes / failures.
Test Coverage¶
When you run the test suite with regression.py, it also automatically generates code coverage statistics. To see the coverage results, open .cover/index.html in a web browser.
Building the Documentation¶
To build the documentation, run:
$ cd samlab/docs
$ make html
Note that significant subsets of the documentation are written using Jupyter notebooks, so the docs/setup.py script requires Jupyter to convert the notebooks into restructured text files for inclusion with the rest of the documentation.
Once the documentation is built, you can view it by opening docs/_build/html/index.html in a web browser.
Support¶
The Samlab documentation:
Visit our GitHub repository for access to source code, issue tracker, regression tests, and the wiki:
And here are our test coverage stats, updated automatically when modifications are committed:
For Samlab questions, comments, or suggestions, get in touch with the team at:
Otherwise, you can contact Tim directly:
Timothy M. Shead - tshead@sandia.gov
Release Notes¶
Samlab 0.2.0 - October 13th, 2021¶
Complete rewrite of the dashboard server, replacing the MongoDB storage with a more flexible system of backend data adapters - now, Samlab adjusts to work with your data, instead of the other way around.
Samlab 0.1.0 - May 24th, 2018¶
Initial Release