Skip to content
This repository has been archived by the owner on Sep 16, 2024. It is now read-only.

A ContentsManager wrapper for using multiple ContentsManager in Jupyter

License

Notifications You must be signed in to change notification settings

viaduct-ai/hybridcontents

 
 

Repository files navigation

HybridContents

The HybridContentManager was originally created by Quantopian as part of pgcontents; however, the usage of HybridContentsManager was restricted to the compatibility requirements of pgcontents. These restrictions included postgres dependencies and no support for the latest notebook version (>6).

At Viaduct we used pgcontents exclusively for the HybridContentsManager and wanted to extend its functionality, so we created this fork hybridcontents.

See related pgcontents issues:

Getting Started

Prerequisites:

Installation:

pip install hybridcontents
conda install -c viaduct hybridcontents

See instructions here

Features

  • Mix and match different content managers for different directories
  • Easily move files between different content managers (i.e local files to s3 backed manager)
  • Path validation to keep consistent naming scheme and/or prevent illegal characters

Usage

For a detailed example see, hybrid_manager_example.py

The following code snippet creates a HybridContentsManager with two directories with different content managers.

c = get_config()

c.NotebookApp.contents_manager_class = HybridContentsManager

c.HybridContentsManager.manager_classes = {
    # NOTE: LargFileManager only exists in notebook > 5
    # If using notebook < 5, use FileContentManager instead
    "": LargeFileManager,
    "shared": S3ContentsManager
}

# Each item will be passed to the constructor of the appropriate content manager.
c.HybridContentsManager.manager_kwargs = {
    # Args for root LargeFileManager
    "": {
        "root_dir": read_only_dir
    },
    # Args for the shared S3ContentsManager directory
    "shared": {
        "access_key_id": ...,
        "secret_access_key": ...,
        "endpoint_url":  ...,
        "bucket": ...,
        "prefix": ...
    },
}

def only_allow_notebooks(path):
  return path.endswith('.ipynb')

# Only allow notebook files to be stored in S3
c.HybridContentsManager.path_validators = {
    "shared": only_allow_notebooks
}

Testing

To run unit tests,

tox

This will run all unit tests for python versions 2.7, 3.6, 3.7 and jupyter notebook versions 4, 5, and 6.

Publishing a Release

  1. Create a new release on Github
  2. Update the version in setup.py
  3. Run ./scripts/pip_publish.sh
  4. Update the version meta.yaml
  5. Update the sha256 in meta.yaml
  6. Run ./scripts/anaconda_publish.sh
  7. Update on Conda Forge

About

A ContentsManager wrapper for using multiple ContentsManager in Jupyter

Resources

License

Stars

Watchers

Forks

Packages

No packages published

Languages

  • Python 99.6%
  • Shell 0.4%