Jupyter Resource Usage Roadmap

[jtpio]

Let's keep discussions on what to do next with nbresuse in this meta issue.

---

• Restore the /metrics endpoint to keep backward compatibility for the 0.3.x releases: Adding support for jupyterlab statusbar-extension #36 #45
• Decide where the repo should live: keep under yuvipanda or more to another GitHub organization (or move to Jupyter Server)? Seeking new maintainers #24 -> moved to https://github.com/jupyter-server/jupyter-resource-usage
• Keep the /metrics endpoint for the 0.3.x releases to not break backward compatibility with the lab status bar and other extensions that rely on nbresuse.
• Drop Deprecate the /metrics endpoint in 0.4.x so it doesn't shadow the Prometheus endpoint anymore: Soft-deprecate /metrics endpoint #68
• Either in a 0.3.x release or in 0.4.x: add a new /api/metrics/v1 endpoint to retrieve the metrics as JSON (just like the current /metrics endpoint). However the format of the response is still to be defined - Add a new /api/nbresuse/v1 endpoint #52 - new endpoint added in Soft-deprecate /metrics endpoint #68
• Change the endpoint used in the JupyterLab status bar: https://github.com/jupyterlab/jupyterlab/blob/4fe4dcfe5c9dc329bed2dcf2602f569ddef8a8a0/packages/statusbar/src/defaults/memoryUsage.tsx#L291 -> remove from core lab and create a federated lab extension for nbresuse: Add a JupyterLab extension for the memory usage status bar item #69
• Drop the deprecated /metrics endpoint: Drop the deprecated /metrics endpoint #75
• Add kernel metrics: Report kernel metrics #31
• Add more metrics (Network I/O or other metrics supported by psutil).
• Switch to event stream? Switch to using eventstreams #7. This will change the way frontends consume the API and will be a breaking change
• Add tests: Add tests for the nbresuse endpoints #49
• Document the nbresuse API: Document the nbresuse API #48
• Document the Prometheus metrics: Document the Prometheus metrics #51

[shreddd]

May also need to reflect official clients in the roadmap (i.e. statusbar)
that need to consume these metrics

Specifically - this endpoint will need to change:
https://github.com/jupyterlab/jupyterlab/blob/4fe4dcfe5c9dc329bed2dcf2602f569ddef8a8a0/packages/statusbar/src/defaults/memoryUsage.tsx#L291

Noting it here since it is a cross repo change.

[jtpio]

Thanks @shreddd, added to the list.

[kevin-bates]

Given that per kernel metrics are in high demand, could we consider the possibility that this be baked directly into jupyter_server? Moreover, I'd love to see the kernel metrics just extend the existing kernel activity API, with either extending/api/kernels/{kernel-id} or via a filter parameter (e.g., /api/kernels/{kernel-id}?metrics) or a slightly new endpoint (/api/kernels/metrics/{kernel-id}). Of course, the advantage of the former is that no applications would need to change (thinking Jupyter Kernel/Enterprise Gateway) and/or add support for server extensions.

The server metrics could remain at /metrics or become /api/server/metrics (for possible other endpoints off /api/server) but would be separate from the kernels if that's possible.

We'll also want to plumb the gathering of metrics into MappingKernelManager - which has access to the kernel process (for local kernels) via the KernelManager instance. Eventually, either via a process abstraction layer or Kernel Providers,
we'll have a specific method to call to gather metrics and let the abstraction layer/provider to the right thing, including metrics for remote kernels, containerized kernels, Spark driver kernels (w/ executors), etc.

It feels like multiple metrics implementations are popping up and it would be good to have a single destination/goal of where this should land.

cc: @Zsailer @lresende

[jtpio]

Given that per kernel metrics are in high demand, could we consider the possibility that this be baked directly into jupyter_server?

Yes absolutely 👍

For now it was still unclear whether this should stay in its own project until it matures, be moved to the jupyterhub organization, or something else. But I agree that if the demand is increasing it would be better to have a more streamlined support, and having it backed into jupyter_server should help.

Also repo2docker recently added nbresuse by default so the memory usage can be displayed automatically in the classic notebook interface and JupyterLab status bar: jupyterhub/repo2docker#904

Moreover, I'd love to see the kernel metrics just extend the existing kernel activity API, with either extending/api/kernels/{kernel-id} or via a filter parameter (e.g., /api/kernels/{kernel-id}?metrics) or a slightly new endpoint (/api/kernels/metrics/{kernel-id}). Of course, the advantage of the former is that no applications would need to change (thinking Jupyter Kernel/Enterprise Gateway) and/or add support for server extensions

It sounds like extending the existing kernel endpoints would be convenient.

/metrics will have to be removed at some point to not shadow the Prometheus endpoint.

It feels like multiple metrics implementations are popping up and it would be good to have a single destination/goal of where this should land.

Should we list these other metrics implementations (for example in this issue) so we can keep an eye of them?

+1 for having a single source of truth for gathering kernel and server metrics.

[kevin-bates]

Awesome, it sounds like we're on the same page here!

Should we list these other metrics implementations (for example in this issue) so we can keep an eye of them?

Actually, I may be conflating the discussion from the lab issue and your work in jupyterlab-system-monitor with nbresuse a bit. But there was this thread of discussion in jupyter_client - which derived from another in notebook. Not to mention this Kernel Gateway question - but that's nbreuse-based as well.

So, although there may not be lots of implementations, there is definitely lots of interest and this should find its way into the server at some point.

[mlucool]

+1 for baking this into jupyter_server. Feels like a great feature to have out of the box, especially if it includes per kernel!

[jtpio]

Decide where the repo should live: keep under yuvipanda or more to another GitHub organization (or move to Jupyter Server)? #24

A candidate for now could be the recent jupyter-server organization, as a new home for this repo?