cerebras.modelzoo.common.utils.model.lora.LoraConfig

class cerebras.modelzoo.common.utils.model.lora.LoraConfig(r=0, alpha=1, dropout=0.0, fan_in_fan_out=False, merge_weights=False, target_modules=None)

Bases: object

r: Rank of LoRA matrix projections alpha: Scaling factor (see paper for additional details) dropout: Dropout to apply to LoRA updates fan_in_fan_out: merge_weights: Determines whether lora weights should be merged/folded

into underlying layers

target_modules: A list of module names that must all exist in layers

that will be converted to LoRA. For example, setting target_modules to [“TransformerDecoderLayer”, “Linear”] would mean that all linear layers that were children of a TransformerDecoderLayer would be converted to LoRA.

Methods

Attributes

alpha
dropout
fan_in_fan_out
merge_weights
r
target_modules

previous

cerebras.modelzoo.common.utils.model.lora.LoRA_Linear

next

cerebras.modelzoo.common.utils.model.mup_utils