mmdetection implementation

[dqawami]

Hi,

I'm currently working on an EfficientDet implementation for mmdetection that uses this repo as a reference. While training efficientdet-d0, I noticed that after 5-10 epochs, the AP values drop drastically and stay that way for about 20 epochs. Afterwards the AP results start increasing again. After testing different configurations, I believe it's an issue with my HeadNet implementation. I haven't replicated the issue when training using this repo, so I was curious if this trend is what is expected, or if this issue has a known solution. Any and all feedback is appreciated (whether it be for this issue or my implementation in general). Thanks in advance!

Link to the repo: https://github.com/dqawami/mmdetection/tree/feature/ik/efficientdet
Link to the HeadNet file: https://github.com/dqawami/mmdetection/blob/feature/ik/efficientdet/mmdet/models/dense_heads/headnet.py

[rwightman]

@dqawami haven't observed that exact behaviour here, sometimes in early stages of training are a bit unstable, especially with small batch sizes... ultimately it's the end result that matters though.

[dqawami]

Thanks for the response! Typically what would you consider the "early stages" of training? (50 epochs? 100 epochs? etc.)

[rwightman]

First 5-10 epochs coming out of warmup is most likely time for a blow up, experimenting with loss fn, which opt, lr is significant there. Have seen some random later blows > 100 epochs once or twice. Sometimes it seems the BN stats are slow to settle in first 50 or so epochs of a 300-400 epoch schedule.