Model predictions incorrect -> possible dataloader issue?

[rtaori]

Hi,

I ran code/predict.py with the PGD_1step/eps_512/noise_0.25 noise model and the predictions seem to be always wrong (the "correct" column in the output is always 0). Upon further inspection, it seems that the predictions are agreeing, just that the label index is wrong (for example instead of prediction index 0, it predicts 828).
To confirm this, I ran the baseline noise_0.25 model from https://github.com/locuslab/smoothing, but with the code in this repo. The predictions are correct, ie the "correct" column is almost always 1.

I think probably the way your models were trained did not use the standard imagenet directories, and so the sort order was different, causing the labels to be different as well.
If possible, could you investigate this and let me know which standard imagenet indices correspond to the indices which the model outputs?

Thanks,
Rohan

[Hadisalman]

@rtaori I think I know the problem.

Can you please go to https://github.com/Hadisalman/smoothing-adversarial/blob/master/code/architectures.py
and use (uncomment)

smoothing-adversarial/code/architectures.py

Line 37 in d2a71bf

# normalize_layer = get_input_center_layer(dataset)

instead of using (comment)

smoothing-adversarial/code/architectures.py

Line 36 in d2a71bf

normalize_layer = get_normalize_layer(dataset)

For some of the early models in the repo, I was using get_input_center_layer instead of get_normalize_layer as the first layer of the NN. Later on I, switched to get_normalize_layer as I found that it really doesn't matter which one to use.

In short, during certificaiton/predicition, make sure that you use the same normalization layer as the one used during training.

Please let me know if this solves your issue.

[rtaori]

I see. Could you let me know which models you used which setting for?
Also, for InputCenterLayer, why isn't there normalization using stddev?

smoothing-adversarial/code/datasets.py

Line 190 in d2a71bf

return input - means

Thanks

[rtaori]

Specifically, I am looking at these imagenet models:
resnet50-smoothing_adversarial_DNN_2steps_eps_512_noise_0.25
resnet50-smoothing_adversarial_DNN_2steps_eps_512_noise_0.50
resnet50-smoothing_adversarial_DNN_2steps_eps_512_noise_1.00
resnet50-smoothing_adversarial_PGD_1step_eps_512_noise_0.25
resnet50-smoothing_adversarial_PGD_1step_eps_512_noise_0.50
resnet50-smoothing_adversarial_PGD_1step_eps_512_noise_1.00

If you could tell me which ones need this change and which ones don't that would be very helpful.

Thanks

[Hadisalman]

The InputCenterLayer doesn't actually normalize the images, it just does mean subtraction. The only difference between the two layers is that one divides by the stddev, and the other doesn't. In practice both give similar results.

All the models you mentioned use InputCenterLayer. I am checking the rest of the models too, and I will include a detailed list soon here and update the README accordingly.

[rtaori]

I see, great thanks

[Hadisalman]

So when you download our trained models, you will find these folders

imagenet32/ --> NormalizeLayer
cifar10/finetune_cifar_from_imagenetPGD2steps/ --> NormalizeLayer
cifar10/self_training/ --> NormalizeLayer

imagenet/--> InputCenterLayer
cifar10/"everythingelse"/ --> InputCenterLayer

Hope this helps. I will include these details in the README as well. Thanks for catching this!

[rtaori]

Thanks Hadi!