Distributed model initialisation - CIFAR 10 example

[ryanwongsa]

❓ Questions/Help/Support

In the cifar10 example when a model is defined inside the process, is the model updated and shared between the nproc_per_node?

In most of the TPU examples I have seen they define the model before the Parallel process.

• tpu-training-super-fast-xlmroberta
• i-like-clean-tpu-training-kernels-i-can-not-lie

Does the way ignite handles the model ensure the same model is used throughout the training?

[vfdev-5]

@ryanwongsa in the examples you cite seems like they define a model on CPU once and on each proc they move it to the corresponding device. If you take a look at one of "official" pytorch/xla examples, you can see that the model is created inside spawned function: https://github.com/pytorch/xla/blob/master/test/test_train_mp_imagenet.py#L179
I think this should be equivalent.

• Grads synchronization happens inside xm.optimizer_step(optimizer).
• There is also MpModelWrapper provided by pytorch/xla which helps to reduce memory usage (see class docstring for details). In this case, we can indeed define a single model instance and then move it to corresponding device inside spawned function.

HTH

EDIT: maybe we need to make it also clear by setting manual_seed just before creation of the model.

[ryanwongsa]

Ah, I see thanks for the links.

I have a second question relating to the CIFAR example. What is the best way to update it support mixed-precision with the ignite auto distribution methods.
Should something like:

model, optimizer = amp.initialize(model, optimizer, opt_level="O1")

be done before or after the below code

optimizer = idist.auto_optim(optimizer)
model = idist.auto_optim(model)

[vfdev-5]

Good question! It should be like here:
https://github.com/pytorch/ignite/blob/master/examples/references/segmentation/pascal_voc2012/code/scripts/training.py#L30

model, optimizer = amp.initialize(model, optimizer, opt_level="O1")
model = idist.auto_model(model)  # This performs wrapping by DDP if `torch.distributed` settings is used. 
optimizer = idist.auto_optim(optimizer)  # For torch.distributed this does nothing. This is only for XLA to replace `xm.optimizer_step(optimizer)` by `optimizer.step()`. 

We need to update the docs, probably, if these functions are not clear.

[ryanwongsa]

Nice. Thanks for the help.