Skip to content

gracefully handle \r carriage returns to support progress bars #424

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
2 of 10 tasks
d4l3k opened this issue Mar 15, 2022 · 0 comments
Closed
2 of 10 tasks

gracefully handle \r carriage returns to support progress bars #424

d4l3k opened this issue Mar 15, 2022 · 0 comments
Milestone
0.1.2 release

Comments

@d4l3k
Copy link
Member

d4l3k commented Mar 15, 2022

🐛 Bug

We currently strip new lines from each line and always append a new line to each write. This isn't strictly correct

Module (check all that applies):

  • torchx.spec
  • torchx.component
  • torchx.apps
  • torchx.runtime
  • torchx.cli
  • torchx.schedulers
  • torchx.pipelines
  • torchx.aws
  • torchx.examples
  • other

To Reproduce

from tqdm import tqdm
import time

for i in tqdm(range(100)):
    time.sleep(0.03)
@d4l3k d4l3k mentioned this issue Mar 15, 2022
Closed
d4l3k added a commit that referenced this issue Mar 22, 2022
@d4l3k @facebook-github-bot
schedulers,cli: persist newline breaks in log_iter (#425)
9f81b85 
Summary:
This resolves #424

This makes it so the torchx scheduler `log_iter` method keeps the line breaks so downstream log streams can handle them gracefully. The current solution strips all `\n` characters and always adds them so it makes it impossible to do streaming visualizations of progress bars which use `\r` without a new line break.

WARNING: This is a change in the log_iter interface and all schedulers/downstream consumers will need to be updated.

If someone is logging from multiple workers this gets dangerous since the progress bar `\r` lines can clobber each other.

Pull Request resolved: #425

Test Plan:
(torchx-3.10.2) tristanr@tristanr-arch2 ~/D/torchx-proj> torchx run --scheduler local_docker --wait --log utils.python --script test_tqdm.py
torchx 2022-03-15 14:26:42 INFO     loaded configs from /home/tristanr/Developer/torchx-proj/.torchxconfig
torchx 2022-03-15 14:26:42 INFO     Building workspace: file:///home/tristanr/Developer/torchx-proj for role[0]: python, image: ghcr.io/pytorch/torchx:0.1.2
dev0
torchx 2022-03-15 14:26:43 INFO     Done building workspace
torchx 2022-03-15 14:26:43 INFO     New image: sha256:9cfaf70f7143b4caef383b46c23635eaf001cbd3d9ff55335aa1ff8c5e236388 built from workspace
local_docker://torchx/torchx_utils_python-bprr9rb4k764nd
torchx 2022-03-15 14:26:44 INFO     Waiting for the app to finish...
python/0 100%|██████████| 100/100 [00:03<00:00, 32.95it/s]
torchx 2022-03-15 14:26:48 INFO     Job finished: SUCCEEDED
(torchx-3.10.2) tristanr@tristanr-arch2 ~/D/torchx-proj> torchx run --scheduler local_cwd --wait --log utils.python --script test_tqdm.py
torchx 2022-03-15 14:26:52 INFO     loaded configs from /home/tristanr/Developer/torchx-proj/.torchxconfig
torchx 2022-03-15 14:26:52 INFO     Log files located in: /tmp/torchx_0nqvqm1d/torchx/torchx_utils_python-x217jjqhbkkrgd/python/0
local_cwd://torchx/torchx_utils_python-x217jjqhbkkrgd
torchx 2022-03-15 14:26:52 INFO     Waiting for the app to finish...
python/0 100%|██████████| 100/100 [00:03<00:00, 32.95it/s]
torchx 2022-03-15 14:26:56 INFO     Job finished: SUCCEEDED

Differential Revision: D34907682

Pulled By: d4l3k

fbshipit-source-id: 2c2619b05366074870434444acf1e0b02787fb77
d4l3k added a commit that referenced this issue Mar 23, 2022
@d4l3k @facebook-github-bot
schedulers,cli: persist newline breaks in log_iter (#425)
fff5382 
Summary:
This resolves #424

This makes it so the torchx scheduler `log_iter` method keeps the line breaks so downstream log streams can handle them gracefully. The current solution strips all `\n` characters and always adds them so it makes it impossible to do streaming visualizations of progress bars which use `\r` without a new line break.

WARNING: This is a change in the log_iter interface and all schedulers/downstream consumers will need to be updated.

If someone is logging from multiple workers this gets dangerous since the progress bar `\r` lines can clobber each other.

Pull Request resolved: #425

Test Plan:
(torchx-3.10.2) tristanr@tristanr-arch2 ~/D/torchx-proj> torchx run --scheduler local_docker --wait --log utils.python --script test_tqdm.py
torchx 2022-03-15 14:26:42 INFO     loaded configs from /home/tristanr/Developer/torchx-proj/.torchxconfig
torchx 2022-03-15 14:26:42 INFO     Building workspace: file:///home/tristanr/Developer/torchx-proj for role[0]: python, image: ghcr.io/pytorch/torchx:0.1.2
dev0
torchx 2022-03-15 14:26:43 INFO     Done building workspace
torchx 2022-03-15 14:26:43 INFO     New image: sha256:9cfaf70f7143b4caef383b46c23635eaf001cbd3d9ff55335aa1ff8c5e236388 built from workspace
local_docker://torchx/torchx_utils_python-bprr9rb4k764nd
torchx 2022-03-15 14:26:44 INFO     Waiting for the app to finish...
python/0 100%|██████████| 100/100 [00:03<00:00, 32.95it/s]
torchx 2022-03-15 14:26:48 INFO     Job finished: SUCCEEDED
(torchx-3.10.2) tristanr@tristanr-arch2 ~/D/torchx-proj> torchx run --scheduler local_cwd --wait --log utils.python --script test_tqdm.py
torchx 2022-03-15 14:26:52 INFO     loaded configs from /home/tristanr/Developer/torchx-proj/.torchxconfig
torchx 2022-03-15 14:26:52 INFO     Log files located in: /tmp/torchx_0nqvqm1d/torchx/torchx_utils_python-x217jjqhbkkrgd/python/0
local_cwd://torchx/torchx_utils_python-x217jjqhbkkrgd
torchx 2022-03-15 14:26:52 INFO     Waiting for the app to finish...
python/0 100%|██████████| 100/100 [00:03<00:00, 32.95it/s]
torchx 2022-03-15 14:26:56 INFO     Job finished: SUCCEEDED

Reviewed By: kiukchung

Differential Revision: D34907682

Pulled By: d4l3k

fbshipit-source-id: 1750a301ca7c8319df4fbeca35fa8eb29b8ecaae
d4l3k added a commit that referenced this issue Mar 23, 2022
@d4l3k @facebook-github-bot
schedulers,cli: persist newline breaks in log_iter (#425)
0ab00b2 
Summary:
This resolves #424

This makes it so the torchx scheduler `log_iter` method keeps the line breaks so downstream log streams can handle them gracefully. The current solution strips all `\n` characters and always adds them so it makes it impossible to do streaming visualizations of progress bars which use `\r` without a new line break.

WARNING: This is a change in the log_iter interface and all schedulers/downstream consumers will need to be updated.

If someone is logging from multiple workers this gets dangerous since the progress bar `\r` lines can clobber each other.

Pull Request resolved: #425

Test Plan:
(torchx-3.10.2) tristanr@tristanr-arch2 ~/D/torchx-proj> torchx run --scheduler local_docker --wait --log utils.python --script test_tqdm.py
torchx 2022-03-15 14:26:42 INFO     loaded configs from /home/tristanr/Developer/torchx-proj/.torchxconfig
torchx 2022-03-15 14:26:42 INFO     Building workspace: file:///home/tristanr/Developer/torchx-proj for role[0]: python, image: ghcr.io/pytorch/torchx:0.1.2
dev0
torchx 2022-03-15 14:26:43 INFO     Done building workspace
torchx 2022-03-15 14:26:43 INFO     New image: sha256:9cfaf70f7143b4caef383b46c23635eaf001cbd3d9ff55335aa1ff8c5e236388 built from workspace
local_docker://torchx/torchx_utils_python-bprr9rb4k764nd
torchx 2022-03-15 14:26:44 INFO     Waiting for the app to finish...
python/0 100%|██████████| 100/100 [00:03<00:00, 32.95it/s]
torchx 2022-03-15 14:26:48 INFO     Job finished: SUCCEEDED
(torchx-3.10.2) tristanr@tristanr-arch2 ~/D/torchx-proj> torchx run --scheduler local_cwd --wait --log utils.python --script test_tqdm.py
torchx 2022-03-15 14:26:52 INFO     loaded configs from /home/tristanr/Developer/torchx-proj/.torchxconfig
torchx 2022-03-15 14:26:52 INFO     Log files located in: /tmp/torchx_0nqvqm1d/torchx/torchx_utils_python-x217jjqhbkkrgd/python/0
local_cwd://torchx/torchx_utils_python-x217jjqhbkkrgd
torchx 2022-03-15 14:26:52 INFO     Waiting for the app to finish...
python/0 100%|██████████| 100/100 [00:03<00:00, 32.95it/s]
torchx 2022-03-15 14:26:56 INFO     Job finished: SUCCEEDED

Reviewed By: kiukchung

Differential Revision: D34907682

Pulled By: d4l3k

fbshipit-source-id: c87f7931be3d90cf006080d3cb1e54b17016e930
@d4l3k d4l3k added this to the 0.1.2 release milestone Mar 23, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
0.1.2 release
Development

Successfully merging a pull request may close this issue.

schedulers,cli: persist newline breaks in log_iter pytorch/torchx
1 participant
@d4l3k