pytabkit.models.optim package

Submodules

pytabkit.models.optim.adopt module

class pytabkit.models.optim.adopt.ADOPT

Bases: Optimizer

__init__(params, lr=0.001, betas=(0.9, 0.9999), eps=1e-06, weight_decay=0.0, decoupled=False, *, foreach=None, maximize=False, capturable=False, differentiable=False, fused=None)
Parameters:

  • params (Iterable[Tensor] | Iterable[dict[str, Any]] | Iterable[tuple[str, Tensor]])

  • lr (float | Tensor)

  • betas (Tuple[float, float])

  • eps (float)

  • weight_decay (float)

  • decoupled (bool)

  • foreach (bool | None)

  • maximize (bool)

  • capturable (bool)

  • differentiable (bool)

  • fused (bool | None)

step(closure=None)

Perform a single optimization step.

Args:
closure (Callable, optional): A closure that reevaluates the model

and returns the loss.

pytabkit.models.optim.adopt.adopt(params, grads, exp_avgs, exp_avg_sqs, state_steps, foreach=None, capturable=False, differentiable=False, fused=None, grad_scale=None, found_inf=None, has_complex=False, *, beta1, beta2, lr, weight_decay, decoupled, eps, maximize)

Functional API that performs ADOPT algorithm computation.

Parameters:

  • params (List[Tensor])

  • grads (List[Tensor])

  • exp_avgs (List[Tensor])

  • exp_avg_sqs (List[Tensor])

  • state_steps (List[Tensor])

  • foreach (bool | None)

  • capturable (bool)

  • differentiable (bool)

  • fused (bool | None)

  • grad_scale (Tensor | None)

  • found_inf (Tensor | None)

  • has_complex (bool)

  • beta1 (float)

  • beta2 (float)

  • lr (float | Tensor)

  • weight_decay (float)

  • decoupled (bool)

  • eps (float)

  • maximize (bool)

pytabkit.models.optim.optimizers module

class pytabkit.models.optim.optimizers.AMSGradOptimizer

Bases: OptimizerBase

__init__(param_groups, hp_manager)
class pytabkit.models.optim.optimizers.AdamOptimizer

Bases: OptimizerBase

__init__(param_groups, hp_manager)
class pytabkit.models.optim.optimizers.AdamaxOptimizer

Bases: OptimizerBase

__init__(param_groups, hp_manager)
class pytabkit.models.optim.optimizers.AdoptOptimizer

Bases: OptimizerBase

__init__(param_groups, hp_manager)
Parameters:

hp_manager (HyperparamManager)

class pytabkit.models.optim.optimizers.MoMoAdamOptimizer

Bases: OptimizerBase

__init__(param_groups, hp_manager)
Parameters:

hp_manager (HyperparamManager)

class pytabkit.models.optim.optimizers.OptimizerBase

Bases: Optimizer

__init__(opt, hyper_mappings, hp_manager)
Parameters:

hp_manager (HyperparamManager)

eval()
get_hyper_values(name, i, use_hyper_factor=True)
step(closure=None, loss=None)

Perform a single optimization step to update parameter.

Args:
closure (Callable): A closure that reevaluates the model and

returns the loss. Optional for most optimizers.

Parameters:

loss (Tensor | None)

train()
class pytabkit.models.optim.optimizers.SFAdamOptimizer

Bases: OptimizerBase

__init__(param_groups, hp_manager)
Parameters:

hp_manager (HyperparamManager)

class pytabkit.models.optim.optimizers.SGDOptimizer

Bases: OptimizerBase

__init__(param_groups, hp_manager)
class pytabkit.models.optim.optimizers.SchedulingAdamOptimizer

Bases: OptimizerBase

__init__(param_groups, hp_manager)
pytabkit.models.optim.optimizers.get_opt_class(opt_name)

pytabkit.models.optim.scheduling_adam module

class pytabkit.models.optim.scheduling_adam.SchedulingAdam

Bases: Optimizer

Implements Adam algorithm.

It has been proposed in Adam: A Method for Stochastic Optimization. The implementation of the L2 penalty follows changes proposed in Decoupled Weight Decay Regularization.

Args:
params (iterable): iterable of parameters to optimize or dicts defining

parameter groups

lr (float, optional): learning rate (default: 1e-3) betas (Tuple[float, float], optional): coefficients used for computing

running averages of gradient and its square (default: (0.9, 0.999))

eps (float, optional): term added to the denominator to improve

numerical stability (default: 1e-8)

weight_decay (float, optional): weight decay (L2 penalty) (default: 0) amsgrad (boolean, optional): whether to use the AMSGrad variant of this

algorithm from the paper On the Convergence of Adam and Beyond (default: False)

__init__(params, lr=0.001, betas=(0.9, 0.999), eps=1e-08, weight_decay=0, amsgrad=False)
step(closure=None)

Performs a single optimization step.

Args:
closure (callable, optional): A closure that reevaluates the model

and returns the loss.

Module contents