Batch size

[royak]

I’m trying to run Monailabel’s training pipeline on CT lymph node data. I started using spleen segmentation as the trainer. The training works fine using the spleen data, but when I’m trying to run it on the CT lymph node data I’m getting the following error:
RuntimeError: Unable to find a valid cuDNN algorithm to run convolution
I did some research and there is a high chance that the error is due to the fact that the data does not fit into the GPU memory (I’m using a single Tesla K80 GPU with 11.4GB of memory).
I tried to find the setting for batch size in Train Task but could not find it. Please let me know where I can set/change the batch size for training. And if the batch size is already 1, what do you suggest to change in the trainer to make the training working? The lymph node CT scan images are huge and currently, I’m using the default parameters for spleen segmentation.

[diazandr3s]

Hi @royak,

By default, the batch size is set to one. You can change this value by redefining the self._config in the training class (https://github.com/Project-MONAI/MONAILabel/blob/main/monailabel/tasks/train/basic_train.py#L139-L152) or by setting it in the Options tab (https://youtu.be/KjwuFx0pTXU?t=235).

My suggestion is to use the segmentation model, change the ROI size to (96, 96, 96) https://github.com/Project-MONAI/MONAILabel/blob/main/sample-apps/radiology/lib/configs/segmentation.py#L65 and reduce the network size https://github.com/Project-MONAI/MONAILabel/blob/main/sample-apps/radiology/lib/configs/segmentation.py#L72

Hope this helps,