Skip to content

Commit 0e3af4c

Browse files
d4l3kd4l3k
committed
docs: add documentation for workspaces + general cleanup
1 parent ca86f97 commit 0e3af4c

28 files changed

+291
-77
lines changed

‎.github/workflows/doc-build.yaml

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -42,6 +42,17 @@ jobs:
4242
run: |
4343
cd docs
4444
make papermill
45+
- name: Coverage
46+
run: |
47+
set -ex
48+
cd docs
49+
make coverage
50+
if [ "$(wc -l build/*/coverage/python.txt)" -ne 2 ]
51+
then
52+
cat build/*/coverage/python.txt
53+
echo "missing documentation coverage!"
54+
exit 1
55+
fi
4556
4657
docpush:
4758
runs-on: ubuntu-18.04

‎docs/source/basics.rst

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,7 @@ The top level modules in TorchX are:
1515
5. :mod:`torchx.pipelines`: adapters that convert the given app spec to a "stage" in an ML pipeline platform
1616
6. :mod:`torchx.runtime`: util and abstraction libraries you can use in authoring apps (not app spec)
1717
7. :mod:`torchx.cli`: CLI tool
18+
8. :mod:`torchx.workspace`: handles patching images for remote execution
1819

1920
Below is a UML diagram
2021

‎docs/source/components/utils.rst

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,3 +10,4 @@ Utils
1010
.. autofunction:: torchx.components.utils.copy
1111
.. autofunction:: torchx.components.utils.python
1212
.. autofunction:: torchx.components.utils.booth
13+
.. autofunction:: binary

‎docs/source/conf.py

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -66,6 +66,12 @@
6666
"IPython.sphinxext.ipython_console_highlighting",
6767
]
6868

69+
# coverage options
70+
71+
coverage_ignore_modules = [
72+
"torchx.components.component_test_base",
73+
]
74+
6975
# katex options
7076
#
7177
#

‎docs/source/index.rst

Lines changed: 32 additions & 35 deletions
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,7 @@ most unique applications can be serviced without customizing the whole vertical
1616

1717

1818
**GETTING STARTED?** First learn the :ref:`basic concepts<basics:Basic Concepts>` and
19-
follow the :ref:`quickstart guide<quickstart:Quickstart>`.
19+
follow the :ref:`quickstart guide<quickstart:Quickstart - Custom Components>`.
2020

2121
.. image:: torchx_index_diag.png
2222

@@ -47,8 +47,37 @@ Documentation
4747
quickstart.md
4848
cli
4949

50+
runner.config
51+
5052
advanced
5153

54+
55+
Works With
56+
---------------
57+
58+
.. _Schedulers:
59+
.. toctree::
60+
:maxdepth: 1
61+
:caption: Schedulers
62+
63+
schedulers/local
64+
schedulers/docker
65+
schedulers/kubernetes
66+
schedulers/slurm
67+
schedulers/ray
68+
schedulers/aws_batch
69+
70+
.. _Pipelines:
71+
.. toctree::
72+
:maxdepth: 1
73+
:caption: Pipelines
74+
75+
pipelines/kfp
76+
77+
78+
Examples
79+
------------
80+
5281
.. toctree::
5382
:maxdepth: 1
5483
:caption: Examples
@@ -58,6 +87,7 @@ Documentation
5887
examples_pipelines/index
5988

6089

90+
6191
Components Library
6292
---------------------
6393
.. _Components:
@@ -85,28 +115,6 @@ Runtime Library
85115
runtime/tracking
86116

87117

88-
Works With
89-
---------------
90-
91-
.. _Schedulers:
92-
.. toctree::
93-
:maxdepth: 1
94-
:caption: Schedulers
95-
96-
schedulers/local
97-
schedulers/kubernetes
98-
schedulers/slurm
99-
schedulers/ray
100-
schedulers/aws_batch
101-
102-
.. _Pipelines:
103-
.. toctree::
104-
:maxdepth: 1
105-
:caption: Pipelines
106-
107-
pipelines/kfp
108-
109-
110118
Reference
111119
-----------
112120

@@ -118,6 +126,7 @@ Reference
118126
specs
119127
runner
120128
schedulers
129+
workspace
121130
pipelines
122131

123132
.. toctree::
@@ -126,15 +135,3 @@ Reference
126135

127136
app_best_practices
128137
component_best_practices
129-
130-
131-
Experimental
132-
---------------
133-
.. toctree::
134-
:maxdepth: 1
135-
:caption: Experimental Features
136-
137-
experimental/runner.config
138-
139-
140-

‎docs/source/quickstart.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,7 @@ jupyter:
1212
name: python3
1313
---
1414

15-
# Quickstart
15+
# Quickstart - Custom Components
1616

1717
This is a self contained guide on how to build a simple app and component spec
1818
and launch it via two different schedulers.

‎docs/source/experimental/runner.config.rst renamed to ‎docs/source/runner.config.rst

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
(beta) .torchxconfig file
1+
.torchxconfig
22
-----------------------------
33

44
.. automodule:: torchx.runner.config
@@ -10,3 +10,7 @@ Config API Functions
1010
.. autofunction:: apply
1111
.. autofunction:: load
1212
.. autofunction:: dump
13+
.. autofunction:: find_configs
14+
.. autofunction:: get_configs
15+
.. autofunction:: get_config
16+
.. autofunction:: load_sections

‎docs/source/runtime/hpo.rst

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,4 +14,7 @@ Ax (Adaptive Experimentation)
1414
.. currentmodule:: torchx.runtime.hpo.ax
1515

1616
.. autoclass:: TorchXRunner
17+
:members:
18+
1719
.. autoclass:: AppMetric
20+
:members:

‎docs/source/schedulers/aws_batch.rst

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,17 @@ AWS Batch
22
=================
33

44
.. automodule:: torchx.schedulers.aws_batch_scheduler
5+
56
.. currentmodule:: torchx.schedulers.aws_batch_scheduler
67

78
.. autoclass:: AWSBatchScheduler
89
:members:
10+
:show-inheritance:
11+
12+
.. autoclass:: BatchJob
13+
:members:
14+
15+
Reference
16+
~~~~~~~~~~~~
17+
18+
.. autofunction:: create_scheduler

‎docs/source/schedulers/docker.rst

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
Docker
2+
=================
3+
4+
.. automodule:: torchx.schedulers.docker_scheduler
5+
6+
.. currentmodule:: torchx.schedulers.docker_scheduler
7+
8+
.. autoclass:: DockerScheduler
9+
:members:
10+
:show-inheritance:
11+
12+
.. autoclass:: DockerJob
13+
:members:
14+
15+
Reference
16+
~~~~~~~~~~~~
17+
18+
.. autofunction:: create_scheduler
19+
20+
.. autoclass:: DockerContainer
21+
:members:
22+
23+
.. autofunction:: has_docker

‎docs/source/schedulers/kubernetes.rst

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,8 +2,22 @@ Kubernetes
22
=================
33

44
.. automodule:: torchx.schedulers.kubernetes_scheduler
5+
56
.. currentmodule:: torchx.schedulers.kubernetes_scheduler
67

78
.. autoclass:: KubernetesScheduler
89
:members:
10+
:show-inheritance:
11+
12+
.. autoclass:: KubernetesJob
13+
:members:
14+
15+
Reference
16+
~~~~~~~~~~~~
917

18+
.. autofunction:: create_scheduler
19+
.. autofunction:: app_to_resource
20+
.. autofunction:: cleanup_str
21+
.. autofunction:: pod_labels
22+
.. autofunction:: role_to_pod
23+
.. autofunction:: sanitize_for_serialization

‎docs/source/schedulers/local.rst

Lines changed: 22 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -2,24 +2,38 @@ Local
22
=================
33

44
.. automodule:: torchx.schedulers.local_scheduler
5+
56
.. currentmodule:: torchx.schedulers.local_scheduler
67

78
.. autoclass:: LocalScheduler
89
:members:
9-
10-
.. automodule:: torchx.schedulers.docker_scheduler
11-
.. currentmodule:: torchx.schedulers.docker_scheduler
12-
13-
.. autoclass:: DockerScheduler
14-
:members:
10+
:show-inheritance:
1511

1612
Image Providers
1713
~~~~~~~~~~~~~~~~~
1814

19-
.. currentmodule:: torchx.schedulers.local_scheduler
20-
2115
.. autoclass:: ImageProvider
2216
:members:
2317

2418
.. autoclass:: CWDImageProvider
2519
:members:
20+
21+
.. autoclass:: LocalDirectoryImageProvider
22+
:members:
23+
24+
Reference
25+
~~~~~~~~~~~~
26+
27+
.. autofunction:: create_cwd_scheduler
28+
29+
.. autoclass:: LogIterator
30+
:members:
31+
32+
.. autoclass:: PopenRequest
33+
:members:
34+
35+
.. autoclass:: ReplicaParam
36+
:members:
37+
38+
.. autoclass:: SignalException
39+
:members:

‎docs/source/schedulers/ray.rst

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,16 @@ Ray
22
=================
33

44
.. automodule:: torchx.schedulers.ray_scheduler
5+
56
.. currentmodule:: torchx.schedulers.ray_scheduler
67

78
.. autoclass:: RayScheduler
89
:members:
10+
:show-inheritance:
11+
12+
.. autofunction:: create_scheduler
13+
.. autofunction:: has_ray
14+
.. autofunction:: serialize
15+
16+
.. autoclass:: RayJob
17+
:members:

‎docs/source/schedulers/slurm.rst

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,17 @@ Slurm
22
=================
33

44
.. automodule:: torchx.schedulers.slurm_scheduler
5+
56
.. currentmodule:: torchx.schedulers.slurm_scheduler
67

78
.. autoclass:: SlurmScheduler
89
:members:
10+
:show-inheritance:
11+
12+
.. autofunction:: create_scheduler
13+
14+
.. autoclass:: SlurmBatchRequest
15+
:members:
16+
17+
.. autoclass:: SlurmReplicaRequest
18+
:members:

‎docs/source/specs.rst

Lines changed: 28 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -24,14 +24,22 @@ Resource
2424
.. autoclass:: Resource
2525
:members:
2626

27+
.. autofunction:: resource
28+
2729
.. autofunction:: get_named_resources
2830

31+
2932
AWS Named Resources
3033
^^^^^^^^^^^^^^^^^^^^^
3134
.. automodule:: torchx.specs.named_resources_aws
3235

3336
.. currentmodule:: torchx.specs.named_resources_aws
3437

38+
.. autofunction:: aws_m5_2xlarge
39+
.. autofunction:: aws_p3_2xlarge
40+
.. autofunction:: aws_p3_8xlarge
41+
.. autofunction:: aws_t3_medium
42+
3543
Macros
3644
------------
3745
.. currentmodule:: torchx.specs
@@ -60,4 +68,23 @@ Component Linter
6068
.. automodule:: torchx.specs.file_linter
6169
.. currentmodule:: torchx.specs.file_linter
6270

63-
.. autofunction:: validate
71+
.. autofunction:: validate
72+
.. autofunction:: get_fn_docstring
73+
74+
.. autoclass:: LinterMessage
75+
:members:
76+
77+
.. autoclass:: TorchFunctionVisitor
78+
:members:
79+
80+
.. autoclass:: TorchXArgumentHelpFormatter
81+
:members:
82+
83+
.. autoclass:: TorchxFunctionArgsValidator
84+
:members:
85+
86+
.. autoclass:: TorchxFunctionValidator
87+
:members:
88+
89+
.. autoclass:: TorchxReturnValidator
90+
:members:

0 commit comments

Comments
 (0)