1
1
# How dstack works
2
2
3
- ` dstack ` enables you to define ML pipelines in YAML and run them through CLI.
4
- You can run them either locally or remotely, for instance, in any configured cloud account, without the need for
5
- Kubernetes or custom Docker images.
3
+ ` dstack ` allows YAML-defined ML pipelines to be run locally or remotely in any configured cloud accounts without
4
+ Kubernetes or custom Docker images via CLI.
6
5
7
- When you run a workflow remotely in a configured cloud account, ` dstack ` creates and destroys instances automatically,
8
- based on the resource requirements. The workflows run in containers that have pre-configured Conda environments,
9
- CUDA drivers, and other necessary components. To optimize costs, ` dstack ` offers the option of using spot instances.
6
+ Workflows can be scripts for data preparation or model training, web apps like Streamlit or Gradio, or development
7
+ environments like JupyterLab or VS Code.
8
+
9
+ !!! info "NOTE:"
10
+ When running a workflow remotely (e.g. in a configured cloud account), ` dstack ` automatically creates and
11
+ destroys instances based on resource requirements and cost strategy, such as using spot instances.
12
+
13
+ ## Remotes
14
+
15
+ By default, workflows run locally. To run workflows remotely, you need to first configure a remote using the `dstack
16
+ config` command. Once a remote is configured, use the ` --remote` flag with the ` dstack run` command to run a workflow in
17
+ the remote.
18
+
19
+ !!! info "NOTE:"
20
+ Currently, a remote can be an AWS or GCP account only. Support for Azure, and Hub[ ^ 1 ] are coming soon.
21
+
22
+ Remotes facilitate collaboration by allowing multiple team members to access the same remote, view each other's runs,
23
+ and reuse each other's artifacts.
10
24
11
25
## Workflows
12
26
@@ -33,33 +47,21 @@ workflows:
33
47
- path : ./lightning_logs
34
48
` ` `
35
49
36
- As workflows are defined using YAML, there's no need for modifying the code in your scripts. You have the freedom to
37
- choose any frameworks, experiment trackers, or cloud providers.
38
-
39
- Workflows can take the form of regular scripts, which may involve tasks such as data preparation or model training, as
40
- well as web applications such as Streamlit or Gradle. Additionally, they can also be development environments such as
41
- JupyterLab or VS Code.
42
-
43
- ## Remotes
44
-
45
- By default, workflows run locally. However, to run workflows remotely (such as in a cloud), you need to
46
- configure a remote by using the ` dstack config` command and then use the `--remote` flag with the `dstack run` command.
50
+ YAML-defined workflows eliminate the need to modify code in your scripts, giving you the freedom to choose frameworks,
51
+ experiment trackers, and cloud providers.
47
52
48
53
!!! info "NOTE:"
49
- Currently, `dstack` supports AWS and GCP as remotes. Support for Azure and Hub are coming soon.
50
-
51
- If multiple members on your team have the same remote configured, they can see each other's runs and reuse
52
- each other's artifacts.
54
+ Workflows run in containers with pre-configured Conda environments, and CUDA drivers.
53
55
54
56
## Artifacts
55
57
56
- Artifacts can be utilized to preserve the output of a workflow for later use in other workflows. Artifacts may comprise
57
- data, model checkpoints, or even a pre-configured Conda environment.
58
+ Artifacts enable you to save any files produced by a workflow for later reuse in other workflows. They may include data,
59
+ model checkpoints, or even a pre-configured Conda environment.
58
60
59
- When executing a workflow locally, the artifacts are saved locally. If you want to use the artifacts of a local run
60
- outside your machine, you can push them to a configured remote using the `dstack push` command.
61
+ When running a workflow locally, the artifacts are saved locally. To push the artifacts of a local to a configured remote,
62
+ use the ` dstack push` command.
61
63
62
- When running a workflow remotely, the artifacts are automatically pushed to the remote.
64
+ When running a workflow remotely, the artifacts are pushed to the remote automatically .
63
65
64
66
# # CLI
65
67
@@ -74,4 +76,9 @@ runs, among others.
74
76
from your preferred IDE.
75
77
76
78
Unlike end-to-end MLOps platforms, `dstack` is lightweight, developer-friendly, and designed to facilitate collaboration
77
- without imposing any particular approach.
79
+ without imposing any particular approach.
80
+
81
+ [^1] :
82
+ Use the `dstack hub start --port PORT` command (coming soon) to host a web application that provides a UI for configuring cloud
83
+ accounts and managing user tokens. Configure this hub as a remote for the CLI to enable the hub to act as a proxy
84
+ between the CLI and the configured account. This setup offers improved security and collaboration.
0 commit comments