Skip to content

Commit 2ff7d88

Browse files
d4l3kfacebook-github-bot
d4l3k
authored and
facebook-github-bot
committed
docs: add documentation for workspaces + general cleanup (#402)
Summary: This makes some of the changes listed in https://docs.google.com/document/d/14icG0GJG1nPHF3DW7yMreZpe3S8FwYdrI_XYIzDTSbI/edit#heading=h.6ixj5co4aocb * Adds workspaces documention * Adds coverage test to docs build to ensure all exported methods/classes are documented in the docs. * Adds previously undocumented methods/classes to docs * Moves torchconfig from experimental to beta * Splits docker/local scheduler docs * Reshuffles the sidebar to prioritize the schedulers over some of the other content Next steps: * update readme/index/quickstart guides to highlight workspaces + patching w/ ddp builtin * deprioritize/remove some of the more complex custom component examples Pull Request resolved: #402 Test Plan: ``` $ make -C docs coverage; and cat docs/build/0.1.2dev0/coverage/python.txt $ pyre $ pytest ``` CI ![Screenshot 2022-02-28 at 17-21-17 torchx workspace — PyTorch TorchX main documentation](https://user-images.githubusercontent.com/909104/156086755-8330fece-dc86-442d-a656-aaef775bd3e5.png) Reviewed By: kiukchung Differential Revision: D34540003 Pulled By: d4l3k fbshipit-source-id: 585d2ad5f35d87c0451ba6fb1e3fd68b5e0e6e0e
1 parent ca86f97 commit 2ff7d88

28 files changed

+304
-88
lines changed

‎.github/workflows/doc-build.yaml

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -42,6 +42,17 @@ jobs:
4242
run: |
4343
cd docs
4444
make papermill
45+
- name: Coverage
46+
run: |
47+
set -ex
48+
cd docs
49+
make coverage
50+
if [ "$(wc -l build/*/coverage/python.txt)" -ne 2 ]
51+
then
52+
cat build/*/coverage/python.txt
53+
echo "missing documentation coverage!"
54+
exit 1
55+
fi
4556
4657
docpush:
4758
runs-on: ubuntu-18.04

‎docs/source/basics.rst

Lines changed: 6 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -10,11 +10,12 @@ The top level modules in TorchX are:
1010

1111
1. :mod:`torchx.specs`: application spec (job definition) APIs
1212
2. :mod:`torchx.components`: predefined (builtin) app specs
13-
3. :mod:`torchx.runner`: given an app spec, submits the app as a job on a scheduler
14-
4. :mod:`torchx.schedulers`: backend job schedulers that the runner supports
15-
5. :mod:`torchx.pipelines`: adapters that convert the given app spec to a "stage" in an ML pipeline platform
16-
6. :mod:`torchx.runtime`: util and abstraction libraries you can use in authoring apps (not app spec)
17-
7. :mod:`torchx.cli`: CLI tool
13+
3. :mod:`torchx.workspace`: handles patching images for remote execution
14+
4. :mod:`torchx.cli`: CLI tool
15+
5. :mod:`torchx.runner`: given an app spec, submits the app as a job on a scheduler
16+
6. :mod:`torchx.schedulers`: backend job schedulers that the runner supports
17+
7. :mod:`torchx.pipelines`: adapters that convert the given app spec to a "stage" in an ML pipeline platform
18+
8. :mod:`torchx.runtime`: util and abstraction libraries you can use in authoring apps (not app spec)
1819

1920
Below is a UML diagram
2021

‎docs/source/components/utils.rst

Lines changed: 7 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -4,9 +4,10 @@ Utils
44
.. automodule:: torchx.components.utils
55
.. currentmodule:: torchx.components.utils
66

7-
.. autofunction:: torchx.components.utils.echo
8-
.. autofunction:: torchx.components.utils.touch
9-
.. autofunction:: torchx.components.utils.sh
10-
.. autofunction:: torchx.components.utils.copy
11-
.. autofunction:: torchx.components.utils.python
12-
.. autofunction:: torchx.components.utils.booth
7+
.. autofunction:: echo
8+
.. autofunction:: touch
9+
.. autofunction:: sh
10+
.. autofunction:: copy
11+
.. autofunction:: python
12+
.. autofunction:: booth
13+
.. autofunction:: binary

‎docs/source/conf.py

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -66,6 +66,12 @@
6666
"IPython.sphinxext.ipython_console_highlighting",
6767
]
6868

69+
# coverage options
70+
71+
coverage_ignore_modules = [
72+
"torchx.components.component_test_base",
73+
]
74+
6975
# katex options
7076
#
7177
#

‎docs/source/index.rst

Lines changed: 32 additions & 35 deletions
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,7 @@ most unique applications can be serviced without customizing the whole vertical
1616

1717

1818
**GETTING STARTED?** First learn the :ref:`basic concepts<basics:Basic Concepts>` and
19-
follow the :ref:`quickstart guide<quickstart:Quickstart>`.
19+
follow the :ref:`quickstart guide<quickstart:Quickstart - Custom Components>`.
2020

2121
.. image:: torchx_index_diag.png
2222

@@ -47,8 +47,37 @@ Documentation
4747
quickstart.md
4848
cli
4949

50+
runner.config
51+
5052
advanced
5153

54+
55+
Works With
56+
---------------
57+
58+
.. _Schedulers:
59+
.. toctree::
60+
:maxdepth: 1
61+
:caption: Schedulers
62+
63+
schedulers/local
64+
schedulers/docker
65+
schedulers/kubernetes
66+
schedulers/slurm
67+
schedulers/ray
68+
schedulers/aws_batch
69+
70+
.. _Pipelines:
71+
.. toctree::
72+
:maxdepth: 1
73+
:caption: Pipelines
74+
75+
pipelines/kfp
76+
77+
78+
Examples
79+
------------
80+
5281
.. toctree::
5382
:maxdepth: 1
5483
:caption: Examples
@@ -58,6 +87,7 @@ Documentation
5887
examples_pipelines/index
5988

6089

90+
6191
Components Library
6292
---------------------
6393
.. _Components:
@@ -85,28 +115,6 @@ Runtime Library
85115
runtime/tracking
86116

87117

88-
Works With
89-
---------------
90-
91-
.. _Schedulers:
92-
.. toctree::
93-
:maxdepth: 1
94-
:caption: Schedulers
95-
96-
schedulers/local
97-
schedulers/kubernetes
98-
schedulers/slurm
99-
schedulers/ray
100-
schedulers/aws_batch
101-
102-
.. _Pipelines:
103-
.. toctree::
104-
:maxdepth: 1
105-
:caption: Pipelines
106-
107-
pipelines/kfp
108-
109-
110118
Reference
111119
-----------
112120

@@ -118,6 +126,7 @@ Reference
118126
specs
119127
runner
120128
schedulers
129+
workspace
121130
pipelines
122131

123132
.. toctree::
@@ -126,15 +135,3 @@ Reference
126135

127136
app_best_practices
128137
component_best_practices
129-
130-
131-
Experimental
132-
---------------
133-
.. toctree::
134-
:maxdepth: 1
135-
:caption: Experimental Features
136-
137-
experimental/runner.config
138-
139-
140-

‎docs/source/quickstart.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,7 @@ jupyter:
1212
name: python3
1313
---
1414

15-
# Quickstart
15+
# Quickstart - Custom Components
1616

1717
This is a self contained guide on how to build a simple app and component spec
1818
and launch it via two different schedulers.

‎docs/source/experimental/runner.config.rst renamed to ‎docs/source/runner.config.rst

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
(beta) .torchxconfig file
1+
.torchxconfig
22
-----------------------------
33

44
.. automodule:: torchx.runner.config
@@ -10,3 +10,7 @@ Config API Functions
1010
.. autofunction:: apply
1111
.. autofunction:: load
1212
.. autofunction:: dump
13+
.. autofunction:: find_configs
14+
.. autofunction:: get_configs
15+
.. autofunction:: get_config
16+
.. autofunction:: load_sections

‎docs/source/runtime/hpo.rst

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,4 +14,7 @@ Ax (Adaptive Experimentation)
1414
.. currentmodule:: torchx.runtime.hpo.ax
1515

1616
.. autoclass:: TorchXRunner
17+
:members:
18+
1719
.. autoclass:: AppMetric
20+
:members:

‎docs/source/schedulers/aws_batch.rst

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,17 @@ AWS Batch
22
=================
33

44
.. automodule:: torchx.schedulers.aws_batch_scheduler
5+
56
.. currentmodule:: torchx.schedulers.aws_batch_scheduler
67

78
.. autoclass:: AWSBatchScheduler
89
:members:
10+
:show-inheritance:
11+
12+
.. autoclass:: BatchJob
13+
:members:
14+
15+
Reference
16+
~~~~~~~~~~~~
17+
18+
.. autofunction:: create_scheduler

‎docs/source/schedulers/docker.rst

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
Docker
2+
=================
3+
4+
.. automodule:: torchx.schedulers.docker_scheduler
5+
6+
.. currentmodule:: torchx.schedulers.docker_scheduler
7+
8+
.. autoclass:: DockerScheduler
9+
:members:
10+
:show-inheritance:
11+
12+
.. autoclass:: DockerJob
13+
:members:
14+
15+
Reference
16+
~~~~~~~~~~~~
17+
18+
.. autofunction:: create_scheduler
19+
20+
.. autoclass:: DockerContainer
21+
:members:
22+
23+
.. autofunction:: has_docker

‎docs/source/schedulers/kubernetes.rst

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,8 +2,22 @@ Kubernetes
22
=================
33

44
.. automodule:: torchx.schedulers.kubernetes_scheduler
5+
56
.. currentmodule:: torchx.schedulers.kubernetes_scheduler
67

78
.. autoclass:: KubernetesScheduler
89
:members:
10+
:show-inheritance:
11+
12+
.. autoclass:: KubernetesJob
13+
:members:
14+
15+
Reference
16+
~~~~~~~~~~~~
917

18+
.. autofunction:: create_scheduler
19+
.. autofunction:: app_to_resource
20+
.. autofunction:: cleanup_str
21+
.. autofunction:: pod_labels
22+
.. autofunction:: role_to_pod
23+
.. autofunction:: sanitize_for_serialization

‎docs/source/schedulers/local.rst

Lines changed: 22 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -2,24 +2,38 @@ Local
22
=================
33

44
.. automodule:: torchx.schedulers.local_scheduler
5+
56
.. currentmodule:: torchx.schedulers.local_scheduler
67

78
.. autoclass:: LocalScheduler
89
:members:
9-
10-
.. automodule:: torchx.schedulers.docker_scheduler
11-
.. currentmodule:: torchx.schedulers.docker_scheduler
12-
13-
.. autoclass:: DockerScheduler
14-
:members:
10+
:show-inheritance:
1511

1612
Image Providers
1713
~~~~~~~~~~~~~~~~~
1814

19-
.. currentmodule:: torchx.schedulers.local_scheduler
20-
2115
.. autoclass:: ImageProvider
2216
:members:
2317

2418
.. autoclass:: CWDImageProvider
2519
:members:
20+
21+
.. autoclass:: LocalDirectoryImageProvider
22+
:members:
23+
24+
Reference
25+
~~~~~~~~~~~~
26+
27+
.. autofunction:: create_cwd_scheduler
28+
29+
.. autoclass:: LogIterator
30+
:members:
31+
32+
.. autoclass:: PopenRequest
33+
:members:
34+
35+
.. autoclass:: ReplicaParam
36+
:members:
37+
38+
.. autoclass:: SignalException
39+
:members:

‎docs/source/schedulers/ray.rst

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,16 @@ Ray
22
=================
33

44
.. automodule:: torchx.schedulers.ray_scheduler
5+
56
.. currentmodule:: torchx.schedulers.ray_scheduler
67

78
.. autoclass:: RayScheduler
89
:members:
10+
:show-inheritance:
11+
12+
.. autofunction:: create_scheduler
13+
.. autofunction:: has_ray
14+
.. autofunction:: serialize
15+
16+
.. autoclass:: RayJob
17+
:members:

‎docs/source/schedulers/slurm.rst

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,17 @@ Slurm
22
=================
33

44
.. automodule:: torchx.schedulers.slurm_scheduler
5+
56
.. currentmodule:: torchx.schedulers.slurm_scheduler
67

78
.. autoclass:: SlurmScheduler
89
:members:
10+
:show-inheritance:
11+
12+
.. autofunction:: create_scheduler
13+
14+
.. autoclass:: SlurmBatchRequest
15+
:members:
16+
17+
.. autoclass:: SlurmReplicaRequest
18+
:members:

0 commit comments

Comments
 (0)