_images/samlab.png

Welcome!

Welcome to Samlab: the Sandia Machine Learning Laboratory. Samlab provides a set of tools for creating, monitoring, and analyzing machine learning experiments.

Documentation

_images/samlab.png

Installation

Using Pip / Easy Install

If your package manager doesn’t support Samlab, or doesn’t have the latest version, your next option should be Python setup tools like pip. You can always install the latest stable version of Samlab and its required dependencies using:

$ pip install samlab

… following that, you’ll be able to use all of Samlab’s features.

From Source

Finally, if you want to work with the latest, bleeding-edge Samlab goodness, you can install it using the source code:

$ git clone https://github.com/sandialabs/samlab
$ cd samlab
$ sudo python setup.py install

The setup script installs Samlab’s required dependencies and copies Samlab into your Python site-packages directory, ready to go.

_images/samlab.png

Overview

Samlab provides a set of tools that you can use to create, monitor, and analyze your machine learning experiments:

Command Line Tools

samlab-gputop provides a colorful, updating display of how hard your GPUs are working.

Dashboard

The Samlab dashboard is a web server providing a graphical user interface for annotating images, monitoring experiments, and analyzing results. Unlike comparable tools (Tensorboard, Visdom, Vott, …), Samlab adapts to your data instead of forcing you to store it in databases, binary records, or other organizations. In most cases, a Samlab data adapter can be written in fewer lines of code than you’d need to convert your data to a tool-specific format, and avoids the pitfalls of data duplication.

Interaction

Samlab functions make it easy to see your experiments’ progress, and interrupt an experiment gracefully, without data loss.

Notebooks

Samlab provides convenience functions to simplify displaying data in Jupyter notebooks.

Torch

Samlab provides convenience functions for working with PyTorch, such as a function to automatically choose a Torch device based on load, making it easy to run multiple experiments simultaneously without accidentally selecting the same device.

_images/samlab.png

API Reference

samlab module

Tools for managing machine learning experiments.

exception samlab.DeprecationWarning[source]

Bases: Warning

Warning

exception ‘samlab.DeprecationWarning’ undocumented

samlab.deprecated(message)[source]

Warning

function ‘samlab.deprecated’ undocumented

samlab.dashboard module

Warning

module ‘samlab.dashboard’ undocumented

class samlab.dashboard.AllowConverter(map, *items)[source]

Bases: werkzeug.routing.BaseConverter

Warning

class ‘samlab.dashboard.AllowConverter’ undocumented

to_python(value)[source]

Warning

method ‘samlab.dashboard.AllowConverter.to_python’ undocumented

class samlab.dashboard.ExcludeConverter(map, *items)[source]

Bases: werkzeug.routing.BaseConverter

Warning

class ‘samlab.dashboard.ExcludeConverter’ undocumented

to_python(value)[source]

Warning

method ‘samlab.dashboard.ExcludeConverter.to_python’ undocumented

samlab.dashboard.require_auth(f)[source]

Warning

function ‘samlab.dashboard.require_auth’ undocumented

samlab.dashboard.require_permissions(permissions)[source]

Warning

function ‘samlab.dashboard.require_permissions’ undocumented

samlab.dashboard.acl module

Warning

module ‘samlab.dashboard.acl’ undocumented

class samlab.dashboard.acl.ForbidAll[source]

Bases: object

Access control list strategy that prevents anyone from doing anything.

class samlab.dashboard.acl.List(**permissions)[source]

Bases: object

Access control list strategy based on lists of usernames.

class samlab.dashboard.acl.PermitAll[source]

Bases: object

Access control list strategy that allows anyone to do anything.

samlab.dashboard.authentication module

Warning

module ‘samlab.dashboard.authentication’ undocumented

class samlab.dashboard.authentication.Basic(realm)[source]

Bases: object

Instruct clients to provide basic authentication.

class samlab.dashboard.authentication.Null[source]

Bases: object

Warning

class ‘samlab.dashboard.authentication.Null’ undocumented

samlab.dashboard.backend module

Warning

module ‘samlab.dashboard.backend’ undocumented

samlab.dashboard.backend.documents module

Warning

module ‘samlab.dashboard.backend.documents’ undocumented

class samlab.dashboard.backend.documents.Directory(*, name, root, pattern='.*\\.(html|txt)')[source]

Bases: samlab.dashboard.backend.documents.DocumentCollection, watchdog.events.FileSystemEventHandler

Warning

class ‘samlab.dashboard.backend.documents.Directory’ undocumented

get(index)[source]

Return a document by index.

Parameters

index (int, required) – The index of the image to return.

Returns

image – If str, the filesystem path of the image.

Return type

str or numpy.ndarray

property name

Warning

property ‘samlab.dashboard.backend.documents.Directory.name’ undocumented

on_any_event(event)[source]

Catch-all event handler.

Parameters

event (FileSystemEvent) – The event object representing the file system event.

reload()[source]

Warning

method ‘samlab.dashboard.backend.documents.Directory.reload’ undocumented

class samlab.dashboard.backend.documents.DocumentCollection[source]

Bases: abc.ABC

Warning

class ‘samlab.dashboard.backend.documents.DocumentCollection’ undocumented

abstract get(index)[source]

Return a document by index.

Parameters

index (int, required) – The index of the image to return.

Returns

image – If str, the filesystem path of the image.

Return type

str or numpy.ndarray

abstract property name

Warning

property ‘samlab.dashboard.backend.documents.DocumentCollection.name’ undocumented

property service

Warning

property ‘samlab.dashboard.backend.documents.DocumentCollection.service’ undocumented

samlab.dashboard.backend.favorites module

Warning

module ‘samlab.dashboard.backend.favorites’ undocumented

class samlab.dashboard.backend.favorites.Favorites[source]

Bases: abc.ABC

Warning

class ‘samlab.dashboard.backend.favorites.Favorites’ undocumented

abstract create(service, name, label)[source]

Mark an item as a favorite.

Parameters
  • service (str, required) – Service type.

  • name (str, required.) – Service name of the item to be favorited.

  • label (str, required.) – Human-readable label for the favorite.

abstract delete(service, name)[source]

Un-favorite an item.

Parameters
  • service (str, required) – Service type.

  • name (str, required.) – Service name of the item to be un-favorited.

abstract get()[source]

Return a sequence of favorites.

property name

Return the name of this backend.

property service

Return the service type implemented by this backend.

class samlab.dashboard.backend.favorites.JSONFile(storage)[source]

Bases: samlab.dashboard.backend.favorites.Favorites

Warning

class ‘samlab.dashboard.backend.favorites.JSONFile’ undocumented

create(service, name, label)[source]

Mark an item as a favorite.

Parameters
  • service (str, required) – Service type.

  • name (str, required.) – Service name of the item to be favorited.

  • label (str, required.) – Human-readable label for the favorite.

delete(service, name)[source]

Un-favorite an item.

Parameters
  • service (str, required) – Service type.

  • name (str, required.) – Service name of the item to be un-favorited.

get()[source]

Return a sequence of favorites.

samlab.dashboard.backend.images module

Warning

module ‘samlab.dashboard.backend.images’ undocumented

class samlab.dashboard.backend.images.COCO(*, name, annotations, images)[source]

Bases: samlab.dashboard.backend.images.ImageCollection

Warning

class ‘samlab.dashboard.backend.images.COCO’ undocumented

bboxes(index)[source]

Return a list of bounding box annotations for an image.

Parameters

index (int, required) – Index of the image.

Returns

bboxes – Sequence of dict, one per bounding box. Each dict must have “left”, “top”, “width”, “height”, “category”, and “color” keys.

Return type

list

property categories

Return a list of existing categories for the collection.

Returns

categories – Sequence of str, one per category, with a unique category name.

Return type

list

get(index)[source]

Return an image.

Parameters

index (int, required) – The index of the image to return.

Returns

image – If str, the filesystem path of the image.

Return type

str

metadata(index)[source]

Warning

method ‘samlab.dashboard.backend.images.COCO.metadata’ undocumented

property name

Warning

property ‘samlab.dashboard.backend.images.COCO.name’ undocumented

tags(index)[source]

Warning

method ‘samlab.dashboard.backend.images.COCO.tags’ undocumented

class samlab.dashboard.backend.images.Directory(*, name, root, pattern='.*\\.(png|jpg|jpeg|PNG|JPG|JPEG)')[source]

Bases: samlab.dashboard.backend.images.ImageCollection

Warning

class ‘samlab.dashboard.backend.images.Directory’ undocumented

get(index)[source]

Return an image.

Parameters

index (int, required) – The index of the image to return.

Returns

image – If str, the filesystem path of the image.

Return type

str

property name

Warning

property ‘samlab.dashboard.backend.images.Directory.name’ undocumented

tags(index)[source]

Warning

method ‘samlab.dashboard.backend.images.Directory.tags’ undocumented

class samlab.dashboard.backend.images.ImageCollection[source]

Bases: abc.ABC

Warning

class ‘samlab.dashboard.backend.images.ImageCollection’ undocumented

bboxes(index)[source]

Return a list of bounding box annotations for an image.

Parameters

index (int, required) – Index of the image.

Returns

bboxes – Sequence of dict, one per bounding box. Each dict must have “left”, “top”, “width”, “height”, “category”, and “color” keys.

Return type

list

property categories

Return a list of existing categories for the collection.

Returns

categories – Sequence of str, one per category, with a unique category name.

Return type

list

abstract get(index)[source]

Return an image.

Parameters

index (int, required) – The index of the image to return.

Returns

image – If str, the filesystem path of the image.

Return type

str

metadata(index)[source]

Warning

method ‘samlab.dashboard.backend.images.ImageCollection.metadata’ undocumented

abstract property name

Warning

property ‘samlab.dashboard.backend.images.ImageCollection.name’ undocumented

put_bboxes(index, bboxes)[source]

Warning

method ‘samlab.dashboard.backend.images.ImageCollection.put_bboxes’ undocumented

put_tags(index, tags)[source]

Warning

method ‘samlab.dashboard.backend.images.ImageCollection.put_tags’ undocumented

property service

Warning

property ‘samlab.dashboard.backend.images.ImageCollection.service’ undocumented

tags(index)[source]

Warning

method ‘samlab.dashboard.backend.images.ImageCollection.tags’ undocumented

class samlab.dashboard.backend.images.ImageNet(*, name, root, split)[source]

Bases: samlab.dashboard.backend.images.ImageCollection

Warning

class ‘samlab.dashboard.backend.images.ImageNet’ undocumented

get(index)[source]

Return an image.

Parameters

index (int, required) – The index of the image to return.

Returns

image – If str, the filesystem path of the image.

Return type

str

property name

Warning

property ‘samlab.dashboard.backend.images.ImageNet.name’ undocumented

tags(index)[source]

Warning

method ‘samlab.dashboard.backend.images.ImageNet.tags’ undocumented

samlab.dashboard.backend.layouts module

Warning

module ‘samlab.dashboard.backend.layouts’ undocumented

class samlab.dashboard.backend.layouts.JSONFile(storage)[source]

Bases: samlab.dashboard.backend.layouts.Layouts

Warning

class ‘samlab.dashboard.backend.layouts.JSONFile’ undocumented

get(*, lid)[source]

Warning

method ‘samlab.dashboard.backend.layouts.JSONFile.get’ undocumented

put(*, content)[source]

Warning

method ‘samlab.dashboard.backend.layouts.JSONFile.put’ undocumented

class samlab.dashboard.backend.layouts.Layouts[source]

Bases: abc.ABC

Warning

class ‘samlab.dashboard.backend.layouts.Layouts’ undocumented

abstract get(*, lid)[source]

Warning

method ‘samlab.dashboard.backend.layouts.Layouts.get’ undocumented

property name

Warning

property ‘samlab.dashboard.backend.layouts.Layouts.name’ undocumented

abstract put(*, content)[source]

Warning

method ‘samlab.dashboard.backend.layouts.Layouts.put’ undocumented

property service

Warning

property ‘samlab.dashboard.backend.layouts.Layouts.service’ undocumented

samlab.dashboard.backend.timeseries module

Warning

module ‘samlab.dashboard.backend.timeseries’ undocumented

class samlab.dashboard.backend.timeseries.Directory(*, name, root, pattern='.*\\.(csv|CSV)')[source]

Bases: samlab.dashboard.backend.timeseries.TimeseriesCollection, watchdog.events.FileSystemEventHandler

Warning

class ‘samlab.dashboard.backend.timeseries.Directory’ undocumented

get(index)[source]

Return a timeseries by index.

Parameters

index (int, required) – The index of the timeseries to return.

Returns

timeseries – Numpy array containing timeseries fields.

Return type

numpy.ndarray

property name

Warning

property ‘samlab.dashboard.backend.timeseries.Directory.name’ undocumented

on_any_event(event)[source]

Catch-all event handler.

Parameters

event (FileSystemEvent) – The event object representing the file system event.

reload()[source]

Warning

method ‘samlab.dashboard.backend.timeseries.Directory.reload’ undocumented

class samlab.dashboard.backend.timeseries.TimeseriesCollection[source]

Bases: abc.ABC

Warning

class ‘samlab.dashboard.backend.timeseries.TimeseriesCollection’ undocumented

abstract get(index)[source]

Return a timeseries by index.

Parameters

index (int, required) – The index of the timeseries to return.

Returns

timeseries – Numpy array containing timeseries fields.

Return type

numpy.ndarray

abstract property name

Warning

property ‘samlab.dashboard.backend.timeseries.TimeseriesCollection.name’ undocumented

property service

Warning

property ‘samlab.dashboard.backend.timeseries.TimeseriesCollection.service’ undocumented

samlab.dashboard.credentials module

Warning

module ‘samlab.dashboard.credentials’ undocumented

class samlab.dashboard.credentials.ExactMatch(username='test', password='test')[source]

Bases: object

Allow credentials that match the given username and password.

class samlab.dashboard.credentials.ForbidAll[source]

Bases: object

Forbid all credentials.

class samlab.dashboard.credentials.LDAP(server, user_dn, timeout=5)[source]

Bases: object

Check credentials against an LDAP server.

class samlab.dashboard.credentials.PermitAll[source]

Bases: object

Permit all non-empty credentials.

class samlab.dashboard.credentials.PermitAny[source]

Bases: object

Permit any credentials, including empty.

samlab.dashboard.service module

Warning

module ‘samlab.dashboard.service’ undocumented

samlab.dashboard.service.register_backend(backend)[source]

Warning

function ‘samlab.dashboard.service.register_backend’ undocumented

samlab.dashboard.service.require_backend(service, name=None)[source]

Warning

function ‘samlab.dashboard.service.require_backend’ undocumented

samlab.dashboard.service.asynchronous module

Warning

module ‘samlab.dashboard.service.asynchronous’ undocumented

samlab.dashboard.service.asynchronous.connect()[source]

Warning

function ‘samlab.dashboard.service.asynchronous.connect’ undocumented

samlab.dashboard.service.asynchronous.disconnect()[source]

Warning

function ‘samlab.dashboard.service.asynchronous.disconnect’ undocumented

samlab.dashboard.service.asynchronous.test()[source]

Warning

function ‘samlab.dashboard.service.asynchronous.test’ undocumented

samlab.dashboard.service.backends module

Warning

module ‘samlab.dashboard.service.backends’ undocumented

samlab.dashboard.service.backends.get_backends()[source]

Warning

function ‘samlab.dashboard.service.backends.get_backends’ undocumented

samlab.dashboard.service.basic module

Warning

module ‘samlab.dashboard.service.basic’ undocumented

samlab.dashboard.service.basic.get_identity()[source]

Warning

function ‘samlab.dashboard.service.basic.get_identity’ undocumented

samlab.dashboard.service.basic.get_index()[source]

Warning

function ‘samlab.dashboard.service.basic.get_index’ undocumented

samlab.dashboard.service.basic.get_permissions()[source]

Warning

function ‘samlab.dashboard.service.basic.get_permissions’ undocumented

samlab.dashboard.service.basic.get_ready()[source]

Warning

function ‘samlab.dashboard.service.basic.get_ready’ undocumented

samlab.dashboard.service.basic.get_server()[source]

Warning

function ‘samlab.dashboard.service.basic.get_server’ undocumented

samlab.dashboard.service.documents module

Warning

module ‘samlab.dashboard.service.documents’ undocumented

samlab.dashboard.service.documents.get_document(name, index)[source]

Warning

function ‘samlab.dashboard.service.documents.get_document’ undocumented

samlab.dashboard.service.documents.get_document_count(name)[source]

Warning

function ‘samlab.dashboard.service.documents.get_document_count’ undocumented

samlab.dashboard.service.favorites module

Warning

module ‘samlab.dashboard.service.favorites’ undocumented

samlab.dashboard.service.favorites.get_favorites()[source]

Warning

function ‘samlab.dashboard.service.favorites.get_favorites’ undocumented

samlab.dashboard.service.favorites.put_delete_favorites_service_name(service, name)[source]

Warning

function ‘samlab.dashboard.service.favorites.put_delete_favorites_service_name’ undocumented

samlab.dashboard.service.images module

Warning

module ‘samlab.dashboard.service.images’ undocumented

samlab.dashboard.service.images.get_image(name, index)[source]

Warning

function ‘samlab.dashboard.service.images.get_image’ undocumented

samlab.dashboard.service.images.get_image_collection(name)[source]

Warning

function ‘samlab.dashboard.service.images.get_image_collection’ undocumented

samlab.dashboard.service.images.get_image_collection_categories(name)[source]

Warning

function ‘samlab.dashboard.service.images.get_image_collection_categories’ undocumented

samlab.dashboard.service.images.get_image_metadata(name, index)[source]

Warning

function ‘samlab.dashboard.service.images.get_image_metadata’ undocumented

samlab.dashboard.service.images.get_put_image_bboxes(name, index)[source]

Warning

function ‘samlab.dashboard.service.images.get_put_image_bboxes’ undocumented

samlab.dashboard.service.images.get_put_image_tags(name, index)[source]

Warning

function ‘samlab.dashboard.service.images.get_put_image_tags’ undocumented

samlab.dashboard.service.layouts module

Warning

module ‘samlab.dashboard.service.layouts’ undocumented

samlab.dashboard.service.layouts.get_layouts_lid(lid)[source]

Warning

function ‘samlab.dashboard.service.layouts.get_layouts_lid’ undocumented

samlab.dashboard.service.layouts.get_post_layouts()[source]

Warning

function ‘samlab.dashboard.service.layouts.get_post_layouts’ undocumented

samlab.dashboard.service.notify module

Warning

module ‘samlab.dashboard.service.notify’ undocumented

samlab.dashboard.service.notify.post_notify()[source]

Warning

function ‘samlab.dashboard.service.notify.post_notify’ undocumented

samlab.dashboard.service.timeseries module

samlab.debounce module

Adapted from https://github.com/salesforce/decorator-operations.

samlab.debounce.debounce(wait)[source]

Warning

function ‘samlab.debounce.debounce’ undocumented

samlab.interactive module

Utilities to simplify user interaction during experiments.

class samlab.interactive.Progress(desc=None, unit=None)[source]

Bases: object

Display a graphical progress bar while iterating over a sequence.

class samlab.interactive.Stop(timeout=5.0)[source]

Bases: object

Handle interrupts so training can be interrupted gracefully.

Create an instance of samlab.interactive.Stop and check its triggered property periodically during training. If triggered is True then the user has interrupted the process, either with CTRL-C or the Jupyter Interrupt Kernel button.

trigger()[source]

Programmatically trigger an interruption.

property triggered

True if the user has interrupted the process, False otherwise.

samlab.notebook module

Integration with Jupyter notebooks.

class samlab.notebook.Progress(desc=None, unit=None)[source]

Bases: object

Display a graphical progress bar while iterating over a sequence.

samlab.notebook.gallery(paths, captions=None, row_height='auto')[source]

Warning

function ‘samlab.notebook.gallery’ undocumented

samlab.torch module

Functionality for simplifying work with Pytorch.

samlab.torch.select_device()[source]

Automatically choose a Torch device to be used for subsequent computation.

Automatically chooses GPUs over CPUs, prioritizing GPUs that are lightly loaded.

Note

Always returns a valid device, but there is no guarantee that a given device will have sufficient resources for a given computation.

samlab.train module

Utilities to simplify training artifacts.

class samlab.train.EarlyStop(patience=10, delta=0)[source]

Bases: object

Stop training if a loss doesn’t improve within N iterations.

Parameters

patience (integer, optional) –

property triggered

Warning

property ‘samlab.train.EarlyStop.triggered’ undocumented

class samlab.train.Loss(delta=0)[source]

Bases: object

Keeps track of whether a loss value has improved.

improved(loss)[source]

Warning

method ‘samlab.train.Loss.improved’ undocumented

property value

Warning

property ‘samlab.train.Loss.value’ undocumented

samlab.train.k_fold(dataset, n=5, k=2, validation=0.2, count=None)[source]

Return sets of indices partitioning a dataset for K-fold cross validation.

samlab.train.random_split(dataset, split=0.1)[source]

Return indices that randomly partition a dataset into two sets.

samlab.train.train_validate_test_split(dataset, test=0.2, validation=0.2)[source]

Return indices that randomly partition a dataset into training, validation, and test sets.

_images/samlab.png

Compatibility

A quick disclaimer on backwards-compatibility for Samlab users:

Samlab follows the Semantic Versioning standard for assigning version numbers in a way that has specific meaning. As of this writing Samlab releases are still in the 0.y.z development phase, which means (among other things) that the API may change at any time. We try not to be abusive about it, but you should be prepared for the occasional bump on the road to the 1.0 release.

_images/samlab.png

Developers

Even if you’re not in a position to contribute code to Samlab, there are many ways you can help the project out:

  • If you use Samlab let us know about it!

  • Even better, cite Samlab when you publish your work.

  • Let us know if there are any problems.

  • Help us document Samlab.

  • Spread the word!

Getting Started

If you haven’t already, you’ll want to get familiar with the Samlab repository at http://github.com/sandialabs/samlab … there, you’ll find the Samlab sources, issue tracker, and wiki.

Next, you’ll need to install Samlab’s dependencies, which you can find in setup.py. Then, you’ll be ready to get Samlab’s source code and use setuptools to install it. To do this, you’ll almost certainly want to use “develop mode”. Develop mode is a a feature provided by setuptools that links the Samlab source code into the install directory instead of copying it … that way you can edit the source code in your git sandbox, and you don’t have to re-install it to test your changes:

$ git clone https://github.com/sandialabs/samlab.git
$ cd samlab
$ python setup.py develop

Versioning

Samlab version numbers follow the Semantic Versioning standard.

Coding Style

The Samlab source code follows the PEP-8 Style Guide for Python Code.

Running Regression Tests

To run the Samlab test suite, simply run regression.py from the top-level source directory:

$ cd samlab
$ python regression.py

The tests will run, providing feedback on successes / failures.

Test Coverage

When you run the test suite with regression.py, it also automatically generates code coverage statistics. To see the coverage results, open .cover/index.html in a web browser.

Building the Documentation

To build the documentation, run:

$ cd samlab/docs
$ make html

Note that significant subsets of the documentation are written using Jupyter notebooks, so the docs/setup.py script requires Jupyter to convert the notebooks into restructured text files for inclusion with the rest of the documentation.

Once the documentation is built, you can view it by opening docs/_build/html/index.html in a web browser.

_images/samlab.png

Support

The Samlab documentation:

Visit our GitHub repository for access to source code, issue tracker, regression tests, and the wiki:

And here are our test coverage stats, updated automatically when modifications are committed:

For Samlab questions, comments, or suggestions, get in touch with the team at:

Otherwise, you can contact Tim directly:

_images/samlab.png

Release Notes

Samlab 0.2.0 - October 13th, 2021

Complete rewrite of the dashboard server, replacing the MongoDB storage with a more flexible system of backend data adapters - now, Samlab adjusts to work with your data, instead of the other way around.

Samlab 0.1.0 - May 24th, 2018

  • Initial Release

Indices and tables