Skip to content

Instantly share code, notes, and snippets.

@blink1073
Last active October 4, 2019 21:05
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save blink1073/59967d616332a78afe9ddc217165991a to your computer and use it in GitHub Desktop.
Save blink1073/59967d616332a78afe9ddc217165991a to your computer and use it in GitHub Desktop.
Jupyter Enterprise JEP (Draft)

When submitting an enhancement proposal, individuals will include the following information in their submission.

The problem that this enhancement addresses. If possible include code or anecdotes to describe this problem to readers.

A brief (1-2 sentences) overview of the enhancement you are proposing. If possible include hypothetical code sample to describe how the solution would work to readers.

A detailed explanation covering relevant algorithms, data structures, an API spec, and any other relevant technical information
A list of pros that this implementation has over other potential implementations.
A list of cons that this implementation has.
A list of individuals who would be interested in contributing to this enhancement should it be accepted.

https://github.com/jupyterlab/jupyterlab/blob/master/CORPORATE.md

Enterprise Jupyter will be a separate Jupyter Org to handle the unquie demands of groups of users using Jupyter for Enterprise purposes. Enterprise includes groups that have a large number of users with multiple user classifications.

Three pieces will be added to Enterprise Jupyter. Two additions are code packages. One addition is a solution to connect the Enterprise Jupyter community.

  1. paperboy - Scheduling and parameterizing notebooks in production to produce reports
  2. nbgallery - A community created GitHub repo that creates user-friendly Jupyter notebook sharing and collaboration platform
  3. community for Enterprise Jupyter

Communication

Challenge

Existing platforms to connect the Enterprise Jupyter community and their challenges

  • Discourse -- Blocked by firewalls -- Not real time -- No security model
  • Gitter -- Blocked by firewalls -- Not sufficient for complex issues -- No threading -- Too real time -- Not searchable

Goals

  • Maximum participation
  • Threaded conversation
  • Searchable and archivable
  • Organized by thread type (question vs discussion/ideas)
  • Rating system for threads and responses (likes)
  • Accessible from Enterprises
  • Notification Settings (daily, per message, etc.)
  • Secure channels (by invitation)

As a User answering questions, I want an incentive for questions with supporting information and categorization. As a Maintainer, I want the ability to assign secure access and appropriately categorize questions, and to provide incentives for participation of questions and answers. As a User with questions, I want a searchable, easily navigable interface and a strong possibility of getting a prompt, helpful answer.
As a Person with an idea, I want a system that encourages dialog while amplifying the most relevant or helpful topics/discussion. As an Enterprise System Administrator, I want insights into what my users are asking, access to secure channels for communication, and perhaps access to paid support.

Goals and Tenets

The ideal community for Enterprise Jupyter would have:

  • A strong set of communication channels
  • A reasonable cadence of in-person meeting opportunities for collaboration around ideas and best practices
  • Established expectations of support levels and opportunities
  • A secure communication channel for security discussion

Support Levels

Support of Jupyter has historically from from three sources: community, sponsors, and institional partners. Enterprise Jupyter should have a clear concept of support

  • Community - incentivized by visible status in the community (badges, points, etc).
  • Paid - Job boards and advertisement of paid support methods for enterprises (like Open Collective and OpenTeams)

In Person Meetings

There are several avenues currently for in-person meetings, but none of them is ideal for Enterprise collaboration and engagement. Future JupyterCONs are intended for all user types of Juptyerc. Jupyter Community Workshops are intended to support under-represented groups or specific topics. Jupyter Days are meant to be more of a mini-conference with presentations from local practioners.

WE might want a recurring Jupyter Enterprise CON that has sponsorship.....

Today I met with Tim Paine and Vidar Fauske and their boss from JP Morgan, as well as Dave Stuart and a few others from the intelligence community.

Dave's group made nbgallery [1][2], and had been exploring some of the same space as Tim Paine in making a production workflow for notebooks.

Tim has been working on paperboy, a web UI and configurable back-end for generating parameterized scheduled reports using papermill [3].

The thrust of the call was to get on the same page and collaborate moving forward.

I had suggested that we make a JEP to set up a jupyter_enterprise org and move the Enterprise Kernel Gateway there, along with nbgallery and paperboy.

We would also use a team_compass repo there to organize thoughts and discussion around enterprise use of Jupyter.

We also talked about a more enterprise-friendly communication channel (discourse is blocked both at JPM and in the DoD).

Darian later suggested that we set up a weekly video chat time.

Finally, we talked about setting up more regular in-person meetings, that are not ad-hoc like the event that the Argonne folks are planning [4].

Any thoughts or objections to me starting the JEP?

Regards,

Steve

[1] https://github.com/nbgallery/nbgallery [2] https://conferences.oreilly.com/jupyter/jup-ny/public/schedule/detail/68450 [3] https://github.com/timkpaine/paperboy [4] https://discourse.jupyter.org/t/jupyter-security-best-practices-workshop/1496

Item Value
JEP number 30
Title Enterprise Jupyter
Authors Steven Silvester (steven.silvester@ieee.org), Gabriel Robinson (robinson.gabe@gmail.com), Justin Champeau (justinchampeau@gmail.com), Timothy Paine (t.paine154@gmail.com)
Status Draft
Type P - Process
Created 6-Sep-2019
History 06-Sep-2019, 06-Sep-2019

Summary

Enterprise users encounter unique issues that are not currently adequately addressed by the community. These include barriers to participation and lack of clear information and guidance. They also lack a clear set of applications that can be used in an enterprise surrounding Jupyter notebooks, outside of JupyterHub.

We would like to consolidate and provide a focal point for the relationship between Jupyter and Enterprises. Enterprise includes groups that have a large number of users with multiple user classifications.

The start of this focal point is a new GitHub organization, jupyter-enterprise. This org will have a team-compass repository where the roadmap and discussion around this topic can generally take place.

The new org will be seeded with two projects produced by enterprises that we feel have broad applicability. These are paperboy, a Web-UI and configurable backend for scheduling production reports using parameterized notebooks, and nbgallery, a Web-UI for production sharing of notebooks.

The new org will be a springboard to address such issues as optimal communication channels, paid support, in-person meetings, and other enterprise-specific topics.

Background

Below we describe the key features of the proposal, broken into the intitial set of GitHub repositories as well as the initial set of focal points for the organization.

GitHub Repositories

The jupyter-enterprise org will initiallly consist of three repositories: team-compass, paperboy, and nbgallery. Together these provide an initial set of capabilities as well as a central channel communication.

Team Compass

The team-compass repositority provides a central channel communication about roadmaps, discussions, governance, and org-wide issues that pertain to Jupyter and the enterprise.

paperboy

paperboy provides a web frontend for scheduling, parameterizing, and running Jupyter notebooks to produce custom reports.
It is designed to be used in constrained production enviroments with appropriate controls.

nbgallery

nbgallery provides a web frontend for Jupyter notebook sharing and collaboration. It is also designed to be used in constrained production enviroments with appropriate controls.

Focal Points

Here we describe some suggested initial focal points for the new organization to discuss.

Communication Channels

The organization should establish the right mixture of communication channels that satisfy the needs of enterprise users and administrators. For example, discourse is not available behind some corporate firewalls.

Support

The organization should establish norms around the expectation and advertisement of support and enterprise engagement.
This includes job listings, partnerships, as well as offers for contract work to implement features or support deployments.

In-Person Meetings

The organization should establish an appropriate cadence and scope for regular meetings to share best practices and ideas between enterprise users and administrators. The exisiting set of Jupyter meeting avenues does not explicitly address this need.

Identifying and On-boarding Other Enterprise Libraries

The organization should have a clearly established guideline for the adoption of third party libraries that provide value to enterprise users of Jupyter to be brought into the organization.

Pros

  • Consolidates and focuses existing communication and coordination channels for enterprise users and administrators
  • Clear set of supported applications that can be used within an enterprise
  • Encourages more collaboration of ideas and code for enterprises

Cons

  • Adds an additional burden of communication and coordination
  • Maintenance burden of additional enterprise applications

Support

The following individuals have commited to support this new organization

  • Steven Silvester
  • Tim Paine
@tgeorgeux
Copy link

I don't know a ton about JEPS in general, but does it make sense to outline a little more in the area of 'pain points'. I think it would be interesting to know the problems being addressed by paperboy and nbgallery, instead of just the solution offered.

I think it also makes sense to give a section on what challenges do enterprise end users have that others don't face. Here's a rough example:

User Needs Analysis

Enterprise users of Jupyter often view reproducible computing differently from scientists. Analysts take advantage of Jupyter to find quick insights from rapidly evolving datasets. You're not always looking for long-term study results and as such it's useful to create shorter, punctuated results more often, then catalog them. It's often necessary when looking at data over the long term to extract usable insights in throughout the process. Other challenges include transparency (not highly valued between companies), trade secrets, sensitive information stewardship (GDPR/HIPAA compliance), and working behind a firewall.

This is rough and it's probably not accurate but I think it adds some context to the 'why' behind this proposal.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment