Mismatch in torch.Size --> how to fix it?

[MrMarkusJ]

Hi,

I am trying to work on µCT data and make use of the already manually annotated data in order to train a model in MonAiLabel.

The server is started with the following command:

• monailabel start_server --app apps --studies datasets/skull_images/imagesTr --conf models segmentation --conf use_pretrained_model false

The server starts and it also finds the images & manually labelled data.

But once I try to start the training by clicking on "TRAIN" the server will stop by stating the issue:

• size mismatch for model.2.0.conv.weight: copying a param with shape torch.Size([32, 3, 3, 3, 3]) from checkpoint, the shape in current model is torch.Size([32, 4, 3, 3, 3]).

How to solve this issue?

• does the CT data require an appropriate dimension?
• how to adapt / find out the right torch.Size?

AND.
I also tried to run the heuristac planner (--conf heuristic_planner true) but then the server was not even able to start.

Any ideas?

THANKS!

Best,
Markus

[diazandr3s]

Hi @MrMarkusJ,

Thanks for opening this question.
Usually, this happens when the ROI size (https://github.com/Project-MONAI/MONAILabel/blob/main/sample-apps/radiology/lib/configs/segmentation.py#L66) is bigger than the smallest axis dimension of the dataset. By default we used an ROI size of 128x128x128 - This might be too big for the size of the images you're working with.

My suggestion is to reduce the ROI size to 64x64x64 and see how that goes. You could also reduce the target spacing to 0.8x0.8x0.8 as that will increase the image size.

Hope this helps, please let us know.

[MrMarkusJ]

Hi @diazandr3s,

thanks for you suggestions!

The data/ label format was *.nrrd; the heuristic planner was not able to read the meta data and hence was not running.

• The format is now changed to Nifti format. Now the heuristic planner runs.

[2022-08-28 10:22:05,314] [10212] [MainThread] [INFO] (monailabel.utils.others.planner:35) - Reading datastore metadata for heuristic planner...100%|
[2022-08-28 10:22:17,460] [10212] [MainThread] [INFO] (monailabel.utils.others.generic:147) - Using nvidia-smi command
[2022-08-28 10:22:17,541] [10212] [MainThread] [INFO] (monailabel.utils.others.planner:71) - Available GPU memory: {0: 7136} in MB
[2022-08-28 10:22:17,542] [10212] [MainThread] [INFO] (monailabel.utils.others.generic:147) - Using nvidia-smi command
[2022-08-28 10:22:17,621] [10212] [MainThread] [INFO] (monailabel.utils.others.planner:75) - Spacing: [0. 0. 0.]; Spatial Size: [8, 8, 1024]_

Unfortunately the training is not working. Still same issue:

size mismatch for model.2.0.conv.weight: copying a param with shape torch.Size([32, 3, 3, 3, 3]) from checkpoint, the shape in current model is torch.Size([32, 4, 3, 3, 3]).

used this setting:

• self.target_spacing = (0.6, 0.6, 0.6) # target space for image
• self.roi_size = (32, 32, 32) # sliding window size for infer

Could it be that the image data is simply too large for the available GPU memory of 8 GB?
The images are ~ 550 x 500 x 750.

• Should the images be reduced in size?
• why does the INFO tells "Spacing: [0. 0. 0.] ? ([2022-08-28 10:22:17,621] [10212] [MainThread] [INFO] (monailabel.utils.others.planner:75) - Spacing: [0. 0. 0.]; Spatial Size: [8, 8, 1024]_)

Best,
Markus

[diazandr3s]

Hi @MrMarkusJ,

These images are quite big. I was misled by the name microCT :D
The heuristic planner suggested:

Spacing: [0. 0. 0.]; Spatial Size: [8, 8, 1024]

which can be interpreted as it is a small GPU to run the segmentation model on these images :/

But first, let's make sure we don't have other things creating issues:

• Are you using a pretrained model? If yes, do they have the same number of labels? If not, please remove any files located in the model folder

• Are these images multi-modality? Is it only CT or CT + PET?

• Please try

  self.target_spacing = (4, 4, 4) # target space for image
  self.roi_size = (32, 32, 32) # sliding window size for infer

You won't get the best results, but at least we can see if this works.

[MrMarkusJ]

Hi @diazandr3s ,

thanks for your advice!
Indeed, microCT images can be quiet big ;-)

The trainig is now RUNNING !

What was done to achieve this:

• I am using already labelled images --> these data had to be stored as NIFTI files (used ITK-Snap)
• the heuristic planner can read the metadata of NIFTI files, but has issues with the NRRD files
• as suggested by @diazandr3s: empty the folder "model" in case one works without a pretrained model (the model folder
  was not empty in my case since I tried different options/parameters to get the training started; during this trails a model
  was created, which was "incompatible" (probably, there was some info/data from the NRRD label data included..))
• images size 550 x 500 x 750 / GPU RAM 8 GB & heuristic planner true -->
  AND target_spacing = (1.0, 1.0, 1.0) & self.roi_size = (128, 128, 128) was accepted and ~ 2.3 RAM GB of the GPU were used
  NO need to change (maybe a change would improve the training.... not sure about this)

Now I am gonna supply more labelled data to the model and train....
Currently the training needs to be improved (nursery stage) :

[2022-08-29 11:12:51,926] [17860] [MainThread] [INFO] (monailabel.tasks.train.basic_train:478) - Running cleanup...
[2022-08-29 11:12:51,927] [17860] [MainThread] [INFO] (main:62) - Result: {"total_epochs": 50, "total_iterations": 4, "epoch": 50, "start_ts": 1661764181, "total_time": "0:03:10", "best_metric": 0.07049167156219482, "train": {"metrics": {"train_dice": 0.08518235385417938, "train_skull_dice": 0.1791563630104065, "train_jaw_l_dice": 0.055941998958587646, "train_jaw_r_dice": 0.020448699593544006}, "key_metric_name": "train_dice", "best_metric": 0.09252568334341049, "best_metric_epoch": 49}, "eval": {"metrics": {"val_mean_dice": 0.06291116774082184, "val_skull_dice": 0.14640432596206665, "val_jaw_l_dice": 0.02804570645093918, "val_jaw_r_dice": 0.014283479191362858}, "key_metric_name": "val_mean_dice", "best_metric": 0.07049167156219482, "best_metric_epoch": 44}}
[2022-08-29 11:12:52,462] [14760] [ThreadPoolExecutor-0_0] [INFO] (monailabel.utils.async_tasks.utils:76) - Return code: 0

Thanks for your advices!

Best regards,
Markus

[diazandr3s]

Many thanks for the update!