[RL] [Ray] Update #568 errors

[Guilherme-B]

First of all, thank you for updating the Ray container to add checkpoint support, it was definitely missing and rather important.

Today I proceeded to test said modifications #568 , and a few things seem to be out of order:
Both checkpoint and outputs seem to be saved properly. However, the overall process fails, throwing the following:

TERMINATED trials:
- PPO_RoboschoolReacher-v1_0:#011TERMINATED [pid=92], 1774 s, 65 iter, 1638000 ts, 18.4 rew
Saved model configuration.
Traceback (most recent call last):
File "train-reacher.py", line 51, in <module>
MyLauncher().train_main()
File "/opt/ml/code/sagemaker_rl/ray_launcher.py", line 250, in train_main
launcher.launch()
File "/opt/ml/code/sagemaker_rl/ray_launcher.py", line 242, in launch
config=config)
File "/opt/ml/code/sagemaker_rl/ray_launcher.py", line 208, in save_checkpoint_and_serving_model
self.copy_checkpoints_to_model_output()
File "/opt/ml/code/sagemaker_rl/ray_launcher.py", line 183, in copy_checkpoints_to_model_output
raise RuntimeError("Failed to save checkpoint files - .tune_metadata or .extra_data")

As for the evaluation process, it too seems to be malfunctioning, not being able to retrieve checkpoints:

LocalMultiGPUOptimizer devices ['/cpu:0']
Traceback (most recent call last):
File "evaluate-ray.py", line 110, in <module>
run(args, parser)
File "evaluate-ray.py", line 79, in run
agent.restore(args.checkpoint)
File "/usr/local/lib/python3.6/dist-packages/ray/tune/trainable.py", line 253, in restore
self._restore(checkpoint_path)
File "/usr/local/lib/python3.6/dist-packages/ray/rllib/agents/ppo/ppo.py", line 153, in _restore
extra_data = pickle.load(open(checkpoint_path + ".extra_data", "rb"))
FileNotFoundError: [Errno 2] No such file or directory: '/opt/ml/input/data/model/checkpoint.extra_data'

All of the previous happened whilst running an exact copy of the repository, no changes have been included.

Side note 1 - Recall to modify the examples - available through SageMaker's notebook instances - to mirror the updates, otherwise some people might mistakenly bind their code to the old sources.

Side note 2 - Callbacks can be quite a useful tool as well, not sure if their support is planned.

[saurabh3949]

@GBanlik : We are working on a fix.

[w601sxs]

Running a ray example for lunar lander (box2d) environment .. started with an earlier version of the SM RL container for ray-tesorflow-gpu and had this checkpoint error. I've attached the ray launcher code.

thought updating container to 0.6 or 0.65 will fix that.

With version 0.65 I get

Traceback (most recent call last):
File "train-lunarlander-PPO.py", line 10, in
from sagemaker_rl.ray_launcher import SageMakerRayLauncher
File "/opt/ml/code/sagemaker_rl/ray_launcher.py", line 13, in
from ray.rllib.agents.agent import get_agent_class

19:37:10
ImportError: cannot import name 'get_agent_class'

And with 0.6 I get

ImportError: cannot import name 'get_agent_class' (SAME ERROR)

train-lunarlander-PPO.py.zip

[saurabh3949]

Hi,
It looks like you are using the old ray_launcher.py file. Could you please check the file ../common/sagemaker_rl/ray_launcher.py and confirm if it's the same as https://github.com/awslabs/amazon-sagemaker-examples/blob/master/reinforcement_learning/common/sagemaker_rl/ray_launcher.py#L216 . If not, please try using the latest ray_launcher.py from the above link.

[w601sxs]

Lines 42 - 62 of the example given in the link pasted above looks the same to me as our launched code below:

Again, we have the same issue as the original post here, and we are trying to get around it by setting checkpoint frequency. However, the way the launcher code is structured, it doesn't seem like it will take user parameters for certain fields. This is confirmed in the TODO

    # TODO: move this to before customer-specified so they can override
    hyperparams_dict["rl.training.local_dir"] = INTERMEDIATE_DIR
    hyperparams_dict["rl.training.checkpoint_at_end"] = True
    hyperparams_dict["rl.training.checkpoint_freq"] = config['training'].get('checkpoint_freq', 10)

---

import json
import os

import gym
import ray
from ray.tune import run_experiments
from ray.tune.registry import register_env
import roboschool

from sagemaker_rl.ray_launcher import SageMakerRayLauncher

def create_environment(env_config):
# This import must happen inside the method so that worker processes import this code
import roboschool
return gym.make('LunarLander-v2')

class MyLauncher(SageMakerRayLauncher):

def register_env_creator(self):
    register_env("LunarLander-v2", create_environment)

def get_experiment_config(self):
    return {
      "training": { 
        "env": "LunarLander-v2",
        "run": "PPO",
        "checkpoint_freq":100000, # Tried 1, 0 and 10000 doesn't work
        "stop": {
            "episode_reward_mean": 0,
            "training_iteration": 50,
        },
        "config": {
          "num_sgd_iter": 50,
          "lr": 5e-5,
          "lambda":0.896973,
          "vf_loss_coeff":0.786165,
          "kl_target":0.01,
          "kl_coeff":0.62396,
          "entropy_coeff":0.001268,
          "clip_param":0.384411,
          "sgd_minibatch_size": 5000,
          "train_batch_size": 25000,
          "monitor": True,  # Record videos.
          "model": {
            "free_log_std": True
          },
          "num_workers": (self.num_cpus-1),
          "num_gpus": self.num_gpus,
          "batch_mode": "complete_episodes",
        }
      }
    }

if name == "main":
MyLauncher().train_main()

[annaluo676]

Hi,
What Ray version did you use? Can you please try both including the checkpoint_freq option and using latest ray_luancher on the robomaker-ray example and paste the error?

[w601sxs]

• Changed container to 0.6.5
• Replaced entire ray launcher file in common subdirectory with https://github.com/awslabs/amazon-sagemaker-examples/blob/master/reinforcement_learning/common/sagemaker_rl/ray_launcher.py
• commented out checkpoint save line Fix typos in linear learner multiclass notebook #322

now works!

[annaluo676]

Cool. If you do want to save checkpoint the set up hosting, please make sure the stop criteria is set up appropriately, so that at least one checkpoint gets saved before the job finishes / gets stopped.