Skip to content

Commit c15af2a

Browse files
OriolAbrilOriolAbril
authored
Update jupyter style guide (#5267)
* add extra dependencies guidance to jupyter style doc * add variable name suggestions * fix typo * extra tweaks * add note on myst * add note on extra dependencies
1 parent 3c6e7f3 commit c15af2a

File tree

2 files changed

+271
-9
lines changed

2 files changed

+271
-9
lines changed

‎docs/source/conf.py

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -158,6 +158,8 @@
158158
"aeppl": ("https://aesara-devs.github.io/aeppl/", None),
159159
"numpy": ("https://numpy.org/doc/stable/", None),
160160
"nb": ("https://pymc-examples.readthedocs.io/en/latest/", None),
161+
"myst": ("https://myst-parser.readthedocs.io/en/latest", None),
162+
"myst-nb": ("https://myst-nb.readthedocs.io/en/latest/", None),
161163
}
162164

163165

‎docs/source/contributing/jupyter_style.md

Lines changed: 269 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -1,21 +1,49 @@
11
(jupyter_style)=
22
# Jupyter Style Guide
33

4-
These guidelines should be followed by all notebooks in the documentation, independently of
5-
the repository where the notebook is in (pymc or pymc-examples).
4+
These guidelines should be followed by notebooks in the documentation.
5+
All notebooks in pymc-examples must follow this to the letter, the style
6+
is more permissive for the ones on pymc where not everything is available.
7+
8+
The documentation websites are generated by Sphinx, which uses
9+
{doc}`myst:index` and {doc}`myst-nb:index`
10+
to parse the notebooks.
611

712
## General guidelines
813

914
* Don't use abbreviations or acronyms whenever you can use complete words. For example, write "random variables" instead of "RVs".
1015

1116
* Explain the reasoning behind each step.
1217

13-
* Use the glossary whenever possible. If you use a term that is defined in the Glossary, link to it the first time that term appears in a significant manner. Use [this syntax](https://jupyterbook.org/content/content-blocks.html?highlight=glossary#glossaries) to add a term reference. [Link to glossary source](https://github.com/pymc-devs/pymc/blob/main/docs/source/glossary.md) where new terms should be added.
14-
1518
* Attribute quoted text or code, and link to relevant references.
1619

1720
* Keep notebooks short: 20/30 cells for content aimed at beginners or intermediate users, longer notebooks are fine at the advanced level.
1821

22+
### MyST guidelines
23+
Using MyST allows taking advantage of all sphinx features from markdown cells in the notebooks.
24+
All markdown should be valid MyST (note that MyST is a superset of recommonmark).
25+
This guide does not teach nor cover MyST extensively, only gives some opinionated guidelines.
26+
27+
* **Never** use url links to refer to other notebooks, PyMC documentation or other python
28+
libraries documentations. Use [sphinx cross-references](https://docs.readthedocs.io/en/stable/guides/cross-referencing-with-sphinx.html)
29+
instead.
30+
31+
:::{caution}
32+
Using urls links breaks self referencing in versioned docs! And at the same time they are
33+
less robust than sphinx cross-references.
34+
:::
35+
36+
* When linking to other notebooks, always use a `ref` type cross-reference pointing
37+
to the target in the {ref}`jupyter_style_first_cell`.
38+
39+
* If the output (or even code and output) of a cell is not necessary to follow the
40+
notebook or it is very long and can break the flow of reading, consider hiding
41+
it with a {doc}`toggle button <myst-nb:use/hiding>`
42+
43+
* Consider using {ref}`myst:syntax/figures` to add captions to images used in the notebook.
44+
45+
* Use the glossary whenever possible. If you use a term that is defined in the Glossary, link to it the first time that term appears in a significant manner. Use [this syntax](https://jupyterbook.org/content/content-blocks.html?highlight=glossary#glossaries) to add a term reference. [Link to glossary source](https://github.com/pymc-devs/pymc/blob/main/docs/source/glossary.md) where new terms should be added.
46+
1947
### Variable names
2048

2149
* Above all, stay consistent with variable names within the notebook. Notebooks using multiple names for the same variable will not be merged.
@@ -28,7 +56,119 @@ the repository where the notebook is in (pymc or pymc-examples).
2856

2957
* When using non meaningful names such as single letters, add bullet points with a 1-2 sentence description of each variable below the equation where they are first introduced.
3058

59+
Choosing variable names can sometimes be difficult, tedious or annoying.
60+
In case it helps, the dropdown below has some suggestions so you can focus on writing the actual content
61+
62+
:::::::{dropdown} Variable name suggestions
63+
:icon: light-bulb
64+
65+
**Models and sampling results**
66+
* Use `idata` for sampling results, always containing a variable of type InferenceData.
67+
* Store inferecedata groups as variables to ease writing and reading of code operating on sampling results.
68+
Use underscore separated 3-5 word abbreviations or the group name. Some examples of `abbrebiation`/`group_name`:
69+
`post`/`posterior`, `const`/`constant_data`, `post_pred`/`posterior_predictive` or `obs_data`/`observed_data`
70+
* For stats and diagnostics, use the ArviZ function name as variable name: `ess = az.ess(...)`, `loo = az.loo(...)`
71+
* If there are multiple models in a notebook, assign a prefix to each model,
72+
and use it throughout to identify which variables map to each model.
73+
Taking the famous eight school as example, with a `centered` and `non_centered` model
74+
to compare parametrizations, use `centered_model` (pm.Model object), `centered_idata`, `centered_post`, `centered_ess`... and `non_centered_model`, `non_centered_idata`...
75+
76+
**Dimension and random variable names**
77+
* Use singular dimension names, following ArviZ `chain` and `draw`.
78+
For example `cluster`, `axis`, `component`, `forest`, `time`...
79+
* If you can't think of a meaningful name for the dimension representing the number of observations such as time, fall back to `obs_id`.
80+
* For matrix dimensions, as xarray doesn't allow repeated dimension names, add a `_bis` suffix. i.e. `param, param_bis`.
81+
* For the dimension resulting from stacking `chain` and `draw` use `sample`, that is `.stack(sample=("chain", "draw"))`.
82+
* We often need to encode a categorical variable as integers. add `_idx` to the name of the variable it's encoding.
83+
i.e. from `floor` and `county` to `floor_idx` and `county_idx`.
84+
* To avoid clashes and overwriting variables when using `pm.Data`, use the following pattern:
85+
86+
```
87+
x = np.array(...)
88+
with pm.Model():
89+
x_ = pm.Data("x", x)
90+
...
91+
```
92+
93+
This avoids overwriting the original `x` while having `idata.constant_data["x"]`,
94+
and within the model `x_` is still available to play the role of `x`.
95+
Otherwise, always try to use the same variable name as the string name given to the PyMC random variable.
96+
97+
**Plotting**
98+
* Matplotlib figures and axes. Use:
99+
* `fig` for matplotlib figures
100+
* `ax` for a single matplotib axes object
101+
* `axs` for arrays of matplotlib axes objects
102+
103+
When manually working with multiple matplotlib axes, use local `ax` variables:
104+
105+
::::{tab-set}
106+
:::{tab-item} Local `ax` variables
107+
```{code-block} python
108+
:emphasize-lines: 3, 7
109+
fig, axs = pyplot.subplots()
110+
111+
ax = axs[0, 1]
112+
ax.plot(...)
113+
ax.set(...)
114+
115+
ax = axs[1, 2]
116+
ax.scatter(...)
117+
```
118+
:::
119+
:::{tab-item} Instead of subsetting every time
120+
```
121+
fig, axs = pyplot.subplots()
31122
123+
axs[0, 1].plot(...)
124+
axs[0, 1].set(...)
125+
126+
axs[1. 2].scatter(...)
127+
```
128+
:::
129+
::::
130+
131+
This makes editing the code if restructuring the subplots easier, only one change per subplot
132+
is needed instead of one change per matplotlib function call.
133+
134+
* It is often useful to make a numpy linspace into an {class}`~xarray.DataArray`
135+
for xarray to handle aligning and broadcasing automatically and ease computation.
136+
* If a dimension name is needed, use `x_plot`
137+
* If a variable name is needed for the original array and DataArray to coexist, add `_da` suffix
138+
139+
Thus, ending up with code like:
140+
141+
```
142+
x = xr.DataArray(np.linspace(0, 10, 100), dims=["x_plot"])
143+
# or
144+
x = np.linspace(0, 10, 100)
145+
x_da = xr.DataArray(x)
146+
```
147+
148+
**Looping**
149+
* When using enumerate, take the first letter of the variable as the count:
150+
151+
```
152+
for p, person in enumerate(persons)
153+
```
154+
155+
* When looping, if you need to store a variable after subsetting with the loop index,
156+
append the index variable used for looping to the original variable name:
157+
158+
```{code-block} python
159+
:emphasize-lines: 4, 6
160+
variable = np.array(...)
161+
x = np.array(...)
162+
for i in range(N):
163+
variable_i = variable[i]
164+
for j in range(K):
165+
x_j = x[j]
166+
...
167+
```
168+
169+
:::::::
170+
171+
(jupyter_style_first_cell)=
32172
## First cell
33173
The first cell of all example notebooks should have a MyST target, a level 1 markdown title (that is a title with a single `#`) followed by the post directive.
34174
The syntax is as follows:
@@ -46,7 +186,7 @@ The syntax is as follows:
46186

47187
The date should correspond to the latest update/execution date, at least roughly (it's not a problem if the date is a few days off due to the review process before merging the PR). This will allow users to see which notebooks have been updated lately and will help the PyMC team make sure no notebook is left outdated for too long.
48188

49-
The [MyST target](https://myst-parser.readthedocs.io/en/latest/syntax/syntax.html#targets-and-cross-referencing)
189+
The {ref}`MyST target <myst:syntax/targets>`
50190
is important to ease referencing and linking notebooks between each other.
51191

52192
Tags can be anything, but we ask you to try to use [existing tags](https://github.com/pymc-devs/pymc/wiki/Categories-and-Tags-for-PyMC-Examples)
@@ -58,6 +198,124 @@ Choose a category from [existing categories](https://github.com/pymc-devs/pymc/w
58198
Authors should list people who authored, adapted or updated the notebook. See {ref}`jupyter_authors`
59199
for more details.
60200

201+
## Extra dependencies
202+
If the notebook uses libraries that are not PyMC dependencies, these extra dependencies should
203+
be indicated together with some advise on how to install them.
204+
This ensures readers know what they'll need to install beforehand and can for example
205+
decide between running it locally or on binder.
206+
207+
To make things easier for notebook writers and maintainers, pymc-examples contains
208+
a template for this that warns about the extra dependencies and provides specific
209+
installation instructions inside a dropdown.
210+
211+
Thus, notebooks with extra dependencies should:
212+
213+
1. list the extra dependencies as notebook metadata using the `myst_substitutions` category
214+
and then either the `extra_dependencies` or the `pip_dependencies` and `conda_dependencies`.
215+
In addition, there is also an `extra_install_notes` to include custom text inside the dropdown.
216+
217+
* notebook metadata can be edited from the menubar `Edit` -> `Edit notebook metadata`
218+
in the dropdown
219+
220+
This will open a window with json formatted text that might look a bit like:
221+
222+
::::{tab-set}
223+
:::{tab-item} No myst_substitutions
224+
225+
```json
226+
{
227+
"kernelspec": {
228+
"name": "python3",
229+
"display_name": "Python 3 (ipykernel)",
230+
"language": "python"
231+
},
232+
"language_info": {
233+
"name": "python",
234+
"version": "3.9.7",
235+
"mimetype": "text/x-python",
236+
"codemirror_mode": {
237+
"name": "ipython",
238+
"version": 3
239+
},
240+
"pygments_lexer": "ipython3",
241+
"nbconvert_exporter": "python",
242+
"file_extension": ".py"
243+
}
244+
}
245+
```
246+
:::
247+
248+
:::{tab-item} extra_dependencies key
249+
250+
```{code-block} json
251+
:emphasize-lines: 19-21
252+
{
253+
"kernelspec": {
254+
"name": "python3",
255+
"display_name": "Python 3 (ipykernel)",
256+
"language": "python"
257+
},
258+
"language_info": {
259+
"name": "python",
260+
"version": "3.9.7",
261+
"mimetype": "text/x-python",
262+
"codemirror_mode": {
263+
"name": "ipython",
264+
"version": 3
265+
},
266+
"pygments_lexer": "ipython3",
267+
"nbconvert_exporter": "python",
268+
"file_extension": ".py"
269+
},
270+
"myst_substitutions": {
271+
"extra_dependencies": "bambi seaborn"
272+
}
273+
}
274+
```
275+
:::
276+
277+
:::{tab-item} pip and conda specific keys
278+
```{code-block} json
279+
:emphasize-lines: 19-22
280+
{
281+
"kernelspec": {
282+
"name": "python3",
283+
"display_name": "Python 3 (ipykernel)",
284+
"language": "python"
285+
},
286+
"language_info": {
287+
"name": "python",
288+
"version": "3.9.7",
289+
"mimetype": "text/x-python",
290+
"codemirror_mode": {
291+
"name": "ipython",
292+
"version": 3
293+
},
294+
"pygments_lexer": "ipython3",
295+
"nbconvert_exporter": "python",
296+
"file_extension": ".py"
297+
},
298+
"myst_substitutions": {
299+
"pip_dependencies": "graphviz",
300+
"conda_dependencies": "python-graphviz",
301+
}
302+
}
303+
```
304+
305+
The pip and conda spcific keys overwrite the `extra_installs` one, so it doesn't make
306+
sense to use `extra_installs` if using them. Either both pip and conda substitutions
307+
are defined or none of them is.
308+
:::
309+
::::
310+
311+
1. include the warning and installation advise template with the following markdown right before
312+
the extra dependencies are imported:
313+
314+
```markdown
315+
:::{include} ../extra_installs.md
316+
:::
317+
```
318+
61319
## Code preamble
62320

63321
In a cell just below the cell where you imported matplotlib and/or ArviZ (usually the first one),
@@ -185,7 +443,7 @@ References can be cited twice within a single notebook. Two common reference for
185443

186444
which can be added inline, within the text itself. At the end of the notebook, add the bibliography with the following markdown
187445

188-
```
446+
```markdown
189447
## References
190448

191449
:::{bibliography}
@@ -195,7 +453,7 @@ which can be added inline, within the text itself. At the end of the notebook, a
195453

196454
or alternatively, if you wanted to add extra references that have not been cited within the text, use:
197455

198-
```
456+
```markdown
199457
## References
200458

201459
:::{bibliography}
@@ -211,12 +469,14 @@ Once you're finished with your NB, add a very last cell with [the watermark pack
211469

212470
```python
213471
%load_ext watermark
214-
%watermark -n -u -v -iv -w -p theano,xarray
472+
%watermark -n -u -v -iv -w -p aesara,aeppl,xarray
215473
```
216474

217475
This second to last code cell should be preceded by a markdown cell with the `## Watermark` title only so it appears in the table of contents.
218476

219-
`watermark` should be in your virtual environment if you installed our `requirements-dev.txt`. Otherwise, just run `pip install watermark`. The `p` flag is optional but should be added if Theano (or Aesara if in `v4`) or xarray are not imported explicitly.
477+
`watermark` should be in your virtual environment if you installed our `requirements-dev.txt`.
478+
Otherwise, just run `pip install watermark`.
479+
The `p` flag is optional but should be added if Aesara or xarray are not imported explicitly.
220480
This will also be checked by `pre-commit` (because we all forget to do things sometimes 😳).
221481

222482
## Epilogue

0 commit comments

Comments
 (0)