image-classification-fulltraining.ipynb - Image Classification with images saved in S3

[lauracozma]

Hi,

I am trying to use your mxnet provided example to train an image classifier using images already saved into S3. My images come as .png so I have decided to create the .lst files.
The steps I´m doing:

1. Import S3 images into Sagemaker Notebook Instance (you only provide an example where data is downloaded from the internet)
2. Create the .lst files
3. Upload train, test, my_data_val.lst and my_data_train.lst files to S3 channels as in your examples (this seems redundant in my case since I already do have the training and testing images in S3)
4. Run the algorithm

Now, I did this once and then decided to drop the lst folders from S3 and to re-run the script. Then step 1 starts failing. An error message is thrown: cannot find the file my_data_train.lst. (which is not phisically there anymore)

Thanks,
Laura

[djarpin]

Hi @lauracozma
In general, if your images are already in S3, you shouldn't have to re-upload them to S3 to run the algorithm. Instead, you can just point your "train" and "validation" channels at their original location in S3. You may still have to download them into the notebook instance to create the .lst file if you don't already have it though.

When you say "re-run the script", do you mean im2rec.py? Or the notebook itself? Also, exactly which step is the "cannot find the file my_data_train.lst" error thrown? You mention step 1, but your step 1 of importing S3 images doesn't appear to involve the .lst file in any way. I was able to run the im2rec.py file multiple times without seeing an error, so it seems like this shouldn't be an issue. If the .lst file isn't in S3, the training job will fail. But, it sounds like your error may be happening earlier than that.

Thanks.

[lauracozma]

Hi @djarpin ,

1. I agree. :) This is the way I run the analysis as well.

2. By re-running the script I mean the notebook itself. I get the error thrown when trying to re-download the bucket from S3. Imagine I have a folder like this:
   data
   |_ train
   |_ validation
   |_ train_lst
   |_ validation_lst

If I empty the above lst folders, which contain the train and test lst files, somehow there is some metadata that is still kept. Hence there will be an object called "my_data_lst" but this object wont have any correspondent on the S3 disk. That is why when I try to re-download, it points to these files that are not there anymore. Indeed, the error is happening when I try to re-download the S3 bucket to the sagemaker instance.

Thanks,
Laura

[djarpin]

Can you provide the code you're using to download the data from S3? If you've deleted the lst files then I think it makes sense to get an error if you're trying to download them into your notebook instance. Let me know if I'm misunderstanding the problem. Thanks.

[hongshanli23]

Dear SageMaker Community,
In our attempts to provide this repository with a better level of support going forward, we’re closing issues that were opened prior to the v2 release of the SDK. This is because we believe that over time many of the issues posted were solved with the latest release or other recent changes to the repo. This will help us reallocate resources towards issues that are more likely to still be relevant today. Some of the issues experienced now can be resolved by referencing the v2 guide: https://sagemaker.readthedocs.io/en/stable/v2.html In this guide, you can find simple solutions to common notebook errors, like the renaming of parameters and classes. If you believe your issue is still ongoing and you have updated error messaging or other info, please re-open it, and we investigate the issue.
Best Regards,
AWS SageMaker Team