Seq2Seq Word Embeddings (Not an Issue)

[duasahil8]

Hi
This is for information purposes only and not an issue. I was trying to train the seq2seq for a summarization task. How do we introduce word embedding-s using glove / word2vec before submitting the training job - the example notebook only maps words to integers - is it possible to also provide vector embeddings corresponding to those indices. There are hyperparameters to specify the embedding size but could not figure out a way to provide the actual embeddings.
Thanks in advance.

[orchidmajumder]

Hi, you are correct that currently there is no provision for SageMaker seq2seq to accept pre-trained word embeddings as part of input. For tasks like Machine Translation where there is already enough labeled data for certain language pairs, the model is actually capable of learning the representations when initialized from scratch. However for other tasks where the amount of dataset available is limited, using pre-trained embeddings can indeed help in performance. We'll consider this as a feature request and try to prioritize in our roadmap. Let me know if you face any other issue in terms of training or hosting. Thanks for using Amazon SageMaker.

[duasahil8]

Thanks for a prompt response. I also had a quick question about padding for the string to integer arrays - do we have to explicitly pad it to the max length of source and target or the hyper parameters related to those take care of that.

One issue I have been facing is that if I limit the training to a few epochs / batches - the logs show that training stopped and waiting for decoder to finish - but there is no update after that for hours - and I have to manually terminate the training job. It does not give access to any checkpoint till there and those 4-5 hours of GPU training goes wasted.

Apologies for a late reply, you may close the thread after your response.

[orchidmajumder]

Thanks for your reply. About your first question, you don't need to pad sequences. The algorithm internally pads the source and target sentences to maximum length.
About the second issue, thanks for bringing this to our notice. I have also heard about the same issue from another user recently and trying to reproduce the issue on our side. It'll be helpful if you can share the hyper-parameters of your job for which you have faced the issue. I am not closing the issue as I'm guessing we might still have some communication going on about the training getting stuck issue.

[duasahil8]

Thanks Orchid. I tried a conversational model again on a very small data-set - even if the training is not stuck in this case, there is a long time (~ 1 hour) between two decoder processes. Any thoughts ?

[orchidmajumder]

Hi Sahil, could you please tell me in your previous case where the training got stuck, did it also happen in the part of when the decoder was computing the BLEU score? As an intermediate solution, there is a parameter called bleu_sample_size which controls the portion of the validation dataset on which BLEU score is being computed, you can try to choose a smaller value for the number and it should reduce the decoding time.

[duasahil8]

Hey Orchid
Thanks a lot for keeping this open. Yes, you are correct - it got stuck while calculating the bleu score. The optimized metric being bleu and the sample size not being set - it must have been taking the entire validation set. Since we are only trying to finalize an approach and targeting under-trained models, I tried keeping perplexity as the metric. I have come across another time consuming step - in this snapshot after the log shows bucketing there is a gap of 2.5 hours when the training gets started. I have tried reducing the number of layers, taking a smaller vocab size and number of documents and using a 8x gpu but still there is a long gap after the bucket logs. Is there something trivial I am missing.

We need to finalize an approach for summarization and a Q/A model soon, so have to conclude these two experiments. I am happy to share both the script and the public data-set being used. Also do you know any example - blog etc which explains deploying tensorflow seq2seq model using Sagemaker - I was able to deploy feed forward examples using the example notebooks but could not figure out a way for the seq2seq one.

Thanks again for your help.

[orchidmajumder]

Hi Sahil, it'll be helpful for me to debug if you can share the parameters and the public dataset that you are trying to leverage to finalize your model.
About deploying TensorFlow seq2seq model, I agree with you that it's not straightforward like a feed forward network to deploy for inference. We don't have an example notebook and I believe @ragavvenkatesan / @mvsusp / @djarpin are the best persons to tell whether it's possible or not to deploy a generative model for inference (feed forward ones are discriminative) using current SageMaker Python SDK for TensorFlow.

[duasahil8]

Hi Orchid
This is the link to the dataset.
I have uploaded the notebook as well. This last training job failed after 9 hours.
summarization.ipynb.gz

[orchidmajumder]

Hi Sahil, from the error it looks like the training failed when trying to save the model parameters to disk as there was not sufficient space. Please try after increasing your volumeSizeInGB parameter to a higher value.

[orchidmajumder]

About your parameters, I'd recommend a few things which might help to improve your training:

• Limit the source side to a length around 300-500. Generally, for summarization, the useful information contain within first few lines of the document. My hunch is that this will drastically improve your training time with a minimal/no drop in accuracy (you may even gain some accuracy).
• You are using a P2.8x large machine which has 8 GPUs. You can use a bigger batch size (e.g. 512) to improve your training speed. This will not go out of memory if you restrict the sequence length as I suggested above.
• Learning rate of 0.001 for adam is a bit high. Try with something around 0.0003-0.0005.

[duasahil8]

Thanks Orchid. I will try limiting the source length.
Yeah the batch size carried over from the p2.xlarge machine parameters - which I ll be using most of the times. I will reduce the learning rate as well - was just trying to reduce the training time. I will let you know how the next iteration goes. Really appreciate your help.

[duasahil8]

Thanks Orchid, the model is turning out better. You may close this.