Supported PyTorch Optimizers

Supported PyTorch OptimizersΒΆ

These are drop-in replacements from modelzoo.common.optim.* and can replace torch.nn.optim.* without any modifications.

The following optimizers work only in weight streaming mode:

modelzoo.common.optim.Adadelta

Implements Adadelta algorithm.

modelzoo.common.optim.Adagrad

Implements Adagrad algorithm.

modelzoo.common.optim.Adam

Implements Adam algorithm.

modelzoo.common.optim.AdamW

Implements AdamW algorithm.

modelzoo.common.optim.Adamax

Implements Adamax algorithm (a variant of Adam based on infinity norm).

modelzoo.common.optim.ASGD

Implements Averaged Stochastic Gradient Descent.

modelzoo.common.optim.NAdam

Implements NAdam algorithm.

modelzoo.common.optim.RAdam

Implements RAdam algorithm.

modelzoo.common.optim.RMSProp

Implements RMSprop algorithm.

modelzoo.common.optim.Rprop

Implements the resilient backpropagation algorithm.

modelzoo.common.optim.SGD

Implements stochastic gradient descent (optionally with momentum).