docs: add airflow example #519
Conversation
facebook-github-bot
added
the
CLA Signed
Codecov Report
@@ Coverage Diff @@
## main #519 +/- ##
=======================================
Coverage 94.78% 94.79%
=======================================
Files 65 65
Lines 3932 3939 +7
=======================================
+ Hits 3727 3734 +7
Misses 205 205
Continue to review full report at Codecov.
|
|
@d4l3k has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
|
@d4l3k has updated the pull request. You must reimport the pull request before landing. |
|
@d4l3k has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
| assert ti.state == TaskInstanceState.SUCCESS | ||
| ``` | ||
|
|
||
| If all goes well you should see `Hello, TorchX!` printed above. |
There was a problem hiding this comment.
What do you think on recommending links on things to do next after this example? For more complex apps that can be used in the pipeline?
docs/source/pipelines/airflow.md
Outdated
|
|
||
| ```python | ||
| @task(task_id=f'hello_torchx') | ||
| def run_torchx(echo): |
There was a problem hiding this comment.
Is the arg 'echo' used in the function? I see "foo" passed in from line 81 but the echo command used in the body at line 52 looks like a string literal
There was a problem hiding this comment.
It is -- utils.sh runs a shell command which in this case is "echo"
There was a problem hiding this comment.
sorry, you're right! I was thinking about the lower echo! my bad :)
|
@d4l3k has updated the pull request. You must reimport the pull request before landing. |
|
@d4l3k has updated the pull request. You must reimport the pull request before landing. |
|
@d4l3k has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
|
@d4l3k has updated the pull request. You must reimport the pull request before landing. |
|
@d4l3k has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
|
@d4l3k has updated the pull request. You must reimport the pull request before landing. |
|
@d4l3k has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
This adds an example for how to use Airflow with TorchX. This example requires airflow server to be running on the local machine during the docs build process.
This also adds in a
AppStatus.raise_for_status()method to match the requests/urllib3 behavior which just makes checking for whether or not a job failed easier.Build is working locally -- might need to fiddle with CI to get airflow running there correctly still
Test plan: