questions about evaluation of training model by using built-in algorithm

[giggle0312]

After I execute the command "linear.fit({'train': s3_train_data})", so many logs were printed on the screen. However, I was confused to analysis the content of log. The logs are like followings:
1、I want to know the exploitation of The metrics information meaning?

#metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 92, "sum": 92.0, "min": 92}, "Number of Batches Since Last Reset": {"count": 1, "max": 0, "sum": 0.0, "min": 0}, "Number of Records Since Last Reset": {"count": 1, "max": 0, "sum": 0.0, "min": 0}, "Total Batches Seen": {"count": 1, "max": 93, "sum": 93.0, "min": 93}, "Total Records Seen": {"count": 1, "max": 922, "sum": 922.0, "min": 922}, "Max Records Seen Between Resets": {"count": 1, "max": 912, "sum": 912.0, "min": 912}, "Reset Count": {"count": 1, "max": 2, "sum": 2.0, "min": 2}}, "EndTime": 1528936458.033175, "Dimensions": {"Host": "algo-1", "Meta": "init_train_data_iter", "Operation": "training", "Algorithm": "Linear Learner"}, "StartTime": 1528936458.033125}

2、I also want to know how to call the result of training result ,like precision or recall, I want to figure the change of different epoch and hyperparameters

3、The built-in method realize the cross-validation by self? I don't know how to pass the test data to the training model?

[superprachi]

I second this issue. Is it possible to see validation accuracy during training (i.e. the fit method)? I'm using SageMaker's tensorflow estimator but this seems like a general issue. I even considered writing my own docker image to implement this basic functionality... it seems very odd that this is missing? Why does the model_fn need an EVAL mode if there's no way to run it??

I realize it would be possible to read the output checkpoint files in from another notebook or machine and manually calculate the test accuracy in real time, but is there an existing function to do this? Although it probably isn't that difficult, it seems like a very basic feature.

Even after training completes, how do we get the test accuracy? Do we have to deploy and then use the predict command to manually compute it. This seems like a strange setup, and I hope I'm just missing something here!

[djarpin]

Thanks, @giggle0312 .

1. In general, our logs to tend to be a bit verbose, and some of the information printed are less about model's performance and more about the processing status of the algorithm.

2. Each SageMaker algorithm has a defined list of evaluation metrics that are reported to CloudWatch. The documentation on hyperparameters and hyperparameter tuning are good sources to see which metrics are provided. Unfortunately, any metric not provided in CloudWatch would need to be calculated manually.

3. Currently, none of the built-in SageMaker algorithms support cross-validation. Typically a separate validation or test dataset can be passed in as a separate channel. Example notebooks like this one show how to specify separate channels for a training job. There's more detail on which algorithms support which channels here.

[djarpin]

Hi @superprachi ,
If you're using the SageMaker Python SDK and the pre-built TensorFlow container, then the CloudWatch logs should be printed to your notebook output for the .fit() cell in close to real-time. In most of our example TensorFlow notebooks after your model is trained you should see evaluation metrics printed there (which is the purpose of EVAL mode). If you're interested in seeing evaluations on a validation dataset between epochs, maybe this TensorFlow issue could point you in the right direction?

If you'd like to test your model on a third, separate dataset (that you didn't supply during training), then yes, deploying to an endpoint and generating predictions is currently the best way to get the test accuracy.

[superprachi]

Hi @djarpin,
Yes, I'm particularly interested in seeing evaluations on a validation dataset between epochs. The issue that you referenced unfortunately doesn't help because it's about running evaluation during a tensorflow session, which I know how to do. My issue is whether a TensorFlow estimator in SageMaker can do this during training, since I do not explicitly start a tf session. (That must happen in the train docker image for a sagemaker tensorflow estimator).

As far as I can tell, I have two options:

1. Rewrite the train docker image to do this during training directly (seems a bit excessive)
2. Reload the graph from a different notebook/machine using the .ckpt files that are periodically output and compute the accuracy by providing the validation data there.

Option 2 seems preferable since it allows training to continue uninterrupted without wasting time on computing the validation accuracy, and also seems easier to implement. Given that this is a simple feature, I was wondering if there was built in support for this. Basically, is there something like a model.evaluate function (similar to normal tensorflow estimators) that could easily read in the latest model ckpt (using the format output by SageMaker), use the same "entry point" python file which indicates the validation set, and spit out the accuracy? It should be possible to import manually, but seems like there should be a better way, reusing the entrypoint code directly, so I was wondering if one was out there yet.

[andremoeller]

@superprachi

Currently, there's no way to do this with the SageMaker TensorFlow container, and I don't think we can commit to delivering this feature, but we've incorporated this feature request into our plans for SageMaker TensorFlow. Thanks!

[hongshanli23]

Dear SageMaker Community,
In our attempts to provide this repository with a better level of support going forward, we’re closing issues that were opened prior to the v2 release of the SDK. This is because we believe that over time many of the issues posted were solved with the latest release or other recent changes to the repo. This will help us reallocate resources towards issues that are more likely to still be relevant today. Some of the issues experienced now can be resolved by referencing the v2 guide: https://sagemaker.readthedocs.io/en/stable/v2.html In this guide, you can find simple solutions to common notebook errors, like the renaming of parameters and classes. If you believe your issue is still ongoing and you have updated error messaging or other info, please re-open it, and we investigate the issue.
Best Regards,
AWS SageMaker Team