modelzoo.common.pytorch.model_utils.activations#

Functions

geglu

gelu_fast

gelu_new

Implementation of the GELU activation function currently in Google BERT repo (identical to OpenAI GPT).

get_activation

glu_bivariate_base_fn

is_glu_activation

liglu

linear_act

quick_gelu

reglu

squared_gelu

swiglu