schedulers/aws_batch: fix thread local sessions + raise error on missing memory resource

[d4l3k]

This behavior was noticed in #429 and this is intended to clean it up.

Previously the thread local logic was incorrect so we would create a new session for every request which caused a lot of spam:

torchx 2022-03-21 14:19:45 INFO     Found credentials in environment variables.
torchx 2022-03-21 14:19:45 INFO     Found credentials in environment variables.
torchx 2022-03-21 14:19:45 INFO     Found credentials in environment variables.

Test plan:

Updated unit tests

torchx run --scheduler aws_batch --wait --log dist.ddp --memMB 2000 -j 1x1 --script large-shm.py

[codecov]

Codecov Report

Merging #430 (29ebdb3) into main (0a66255) will increase coverage by 0.16%.
The diff coverage is 85.00%.

@@            Coverage Diff             @@
##             main     #430      +/-   ##
==========================================
+ Coverage   94.41%   94.57%   +0.16%     
==========================================
  Files          67       67              
  Lines        3830     3836       +6     
==========================================
+ Hits         3616     3628      +12     
+ Misses        214      208       -6     

Impacted Files	Coverage Δ
torchx/components/serve.py	91.66% <ø> (ø)
torchx/schedulers/aws_batch_scheduler.py	87.62% <84.21%> (+3.44%)	⬆️
torchx/components/utils.py	51.72% <100.00%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0a66255...29ebdb3. Read the comment docs.

[facebook-github-bot]

@d4l3k has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[facebook-github-bot]

@d4l3k has updated the pull request. You must reimport the pull request before landing.

[facebook-github-bot]

@d4l3k has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[facebook-github-bot]

@d4l3k has updated the pull request. You must reimport the pull request before landing.

[facebook-github-bot]

@d4l3k has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[facebook-github-bot]

@d4l3k has updated the pull request. You must reimport the pull request before landing.

[facebook-github-bot]

@d4l3k has updated the pull request. You must reimport the pull request before landing.

[facebook-github-bot]

@d4l3k has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.