InverseSquareRootDecayLR

InverseSquareRootDecayLR

class modelzoo.common.pytorch.optim.lr_scheduler.InverseSquareRootDecayLR (optimizer: torch.optim.optimizer.Optimizer, initial_learning_rate: float, scale: float, warmup_steps: int, disable_lr_steps_reset: bool = False)

Decays the learning rate inverse-squareroot over time, as described in the following equation:

../../../_images/inverse-square-root-decay-LR.png
Parameters:

optimizer – The optimizer to schedule

initial_learning_rate – The initial learning rate

scale – Multiplicative factor to scale the result

warmup_steps – use initial_learning_rate for the first warmup_steps