pytabkit.models.nn_models package

Submodules

pytabkit.models.nn_models.activations module

class pytabkit.models.nn_models.activations.ActivationFactory

Bases: FitterFactory

__init__(**config)
class pytabkit.models.nn_models.activations.MishJitAutoFn

Bases: Function

static backward(ctx, grad_output)

Define a formula for differentiating the operation with backward mode automatic differentiation.

This function is to be overridden by all subclasses. (Defining this function is equivalent to defining the vjp function.)

It must accept a context ctx as the first argument, followed by as many outputs as the forward() returned (None will be passed in for non tensor outputs of the forward function), and it should return as many tensors, as there were inputs to forward(). Each argument is the gradient w.r.t the given output, and each returned value should be the gradient w.r.t. the corresponding input. If an input is not a Tensor or is a Tensor not requiring grads, you can just pass None as a gradient for that input.

The context can be used to retrieve tensors saved during the forward pass. It also has an attribute ctx.needs_input_grad as a tuple of booleans representing whether each input needs gradient. E.g., backward() will have ctx.needs_input_grad[0] = True if the first input to forward() needs gradient computed w.r.t. the output.

static forward(ctx, x)

Define the forward of the custom autograd Function.

This function is to be overridden by all subclasses. There are two ways to define forward:

Usage 1 (Combined forward and ctx):

@staticmethod
def forward(ctx: Any, *args: Any, **kwargs: Any) -> Any:
    pass

  • It must accept a context ctx as the first argument, followed by any number of arguments (tensors or other types).

  • See combining-forward-context for more details

Usage 2 (Separate forward and ctx):

@staticmethod
def forward(*args: Any, **kwargs: Any) -> Any:
    pass


@staticmethod
def setup_context(ctx: Any, inputs: Tuple[Any, ...], output: Any) -> None:
    pass

  • The forward no longer accepts a ctx argument.

  • Instead, you must also override the torch.autograd.Function.setup_context() staticmethod to handle setting up the ctx object. output is the output of the forward, inputs are a Tuple of inputs to the forward.

  • See extending-autograd for more details

The context can be used to store arbitrary data that can be then retrieved during the backward pass. Tensors should not be stored directly on ctx (though this is not currently enforced for backward compatibility). Instead, tensors should be saved either with ctx.save_for_backward() if they are intended to be used in backward (equivalently, vjp) or ctx.save_for_forward() if they are intended to be used for in jvp.

class pytabkit.models.nn_models.activations.ParametricActivationFitter

Bases: Fitter

__init__(f, **config)
Parameters:

needs_tensors – Set to true if the fitter needs to have the tensors in fit() or fit_transform().

If false, then in fit(ds) or fit_transform(ds), ds.tensors is allowed to be None. :param is_individual: Set to false if fit(ds) deterministically produces a non-trainable layer. (In this case, this Fitter only needs to be called once in k-fold CV on the train+val set.) :param scope_names: List of names to add to the scope (will be present in the names of Variables constructed in this Fitter) :param modified_tensors: List of names of tensors that are modified by this Fitter, e.g., [‘x_cont’]. This is used for the default implementation of get_n_forward(), which is used to get a RAM estimate for the forward pass. The default RAM estimate is simply the size of all modified tensors.

get_n_params(tensor_infos)

Should be overridden if the fitter produces layers with trainable parameters. :param tensor_infos: Tensor infos. :return: Returns the number of parameters of the fitted layer for the given tensor_infos.

Parameters:

tensor_infos (Dict[str, TensorInfo])

Return type:

int

class pytabkit.models.nn_models.activations.ParametricActivationLayer

Bases: Layer

__init__(f, weight)

Constructor. Puts the layer in eval mode, since it might be used inside the fit_transform() of the Fitter. The parameters provide different opportunities to specify a default implementation for forward_tensor_infos(). The default implementation is: ```

if self.fitter is not None:

return self.fitter.forward_tensor_infos(tensor_infos)

return utils.update_dict(tensor_infos, self.new_tensor_infos, remove_keys=self.remove_keys)

```

forward_cont(x)
pytabkit.models.nn_models.activations.golu(x)
pytabkit.models.nn_models.activations.mish(x)
pytabkit.models.nn_models.activations.swish(x)

pytabkit.models.nn_models.base module

class pytabkit.models.nn_models.base.BiasLayer

Bases: Layer

__init__(bias, factor=1.0)

Constructor. Puts the layer in eval mode, since it might be used inside the fit_transform() of the Fitter. The parameters provide different opportunities to specify a default implementation for forward_tensor_infos(). The default implementation is: ```

if self.fitter is not None:

return self.fitter.forward_tensor_infos(tensor_infos)

return utils.update_dict(tensor_infos, self.new_tensor_infos, remove_keys=self.remove_keys)

```

Parameters:
forward_cont(x)
class pytabkit.models.nn_models.base.ConcatParallelFactory

Bases: FitterFactory

__init__(factories)
Parameters:

factories (List[FitterFactory])

class pytabkit.models.nn_models.base.ConcatParallelFitter

Bases: Fitter

__init__(fitters)
Parameters:

  • needs_tensors – Set to true if the fitter needs to have the tensors in fit() or fit_transform().

  • fitters (List[Fitter])

If false, then in fit(ds) or fit_transform(ds), ds.tensors is allowed to be None. :param is_individual: Set to false if fit(ds) deterministically produces a non-trainable layer. (In this case, this Fitter only needs to be called once in k-fold CV on the train+val set.) :param scope_names: List of names to add to the scope (will be present in the names of Variables constructed in this Fitter) :param modified_tensors: List of names of tensors that are modified by this Fitter, e.g., [‘x_cont’]. This is used for the default implementation of get_n_forward(), which is used to get a RAM estimate for the forward pass. The default RAM estimate is simply the size of all modified tensors.

forward_tensor_infos(tensor_infos)

Should be overridden if the fitter changes the tensor shapes. :param tensor_infos: Tensor infos (for shapes and category sizes). :return: Transformed tensor infos.

Parameters:

tensor_infos (Dict[str, TensorInfo])

Return type:

Dict[str, TensorInfo]

get_n_forward(tensor_infos)

Should be overridden if the fitter does more than just one operation. :param tensor_infos: Ingoing tensor infos. :return: Should return the number of bytes used in the forward pass per batch element

Parameters:

tensor_infos (Dict[str, TensorInfo])

Return type:

int

get_n_params(tensor_infos)

Should be overridden if the fitter produces layers with trainable parameters. :param tensor_infos: Tensor infos. :return: Returns the number of parameters of the fitted layer for the given tensor_infos.

Parameters:

tensor_infos (Dict[str, TensorInfo])

Return type:

int

class pytabkit.models.nn_models.base.ConcatParallelLayer

Bases: Layer

Executes all layers on the given input and combines the resulting output tensors by concatenating along the last dimension (as in DenseNet, for example). Not all layers need to output the same tensors, e.g., one can output only ‘x_cont’ and the other can output ‘x_cont’ and ‘y’, in which case ‘y’ will not be concatenated with another tensor.

__init__(layers, fitter)

Constructor. Puts the layer in eval mode, since it might be used inside the fit_transform() of the Fitter. The parameters provide different opportunities to specify a default implementation for forward_tensor_infos(). The default implementation is: ```

if self.fitter is not None:

return self.fitter.forward_tensor_infos(tensor_infos)

return utils.update_dict(tensor_infos, self.new_tensor_infos, remove_keys=self.remove_keys)

```

Parameters:
forward_tensors(tensors)

Transforms the given tensors. :param tensors: :return:

class pytabkit.models.nn_models.base.ContextAware

Bases: object

__init__(scope_names=None)
Parameters:

scope_names (List[str] | None)

add_others_scope(other)
Parameters:

other (ContextAware)

add_scope(name)
Parameters:

name (str)

set_context()
class pytabkit.models.nn_models.base.ContextRecorder

Bases: object

__init__()
set_context()
class pytabkit.models.nn_models.base.FilterTensorsFactory

Bases: Fitter, FitterFactory

__init__(include_keys=None, exclude_keys=None)
Parameters:

  • needs_tensors – Set to true if the fitter needs to have the tensors in fit() or fit_transform().

  • include_keys (List[str] | None)

  • exclude_keys (List[str] | None)

If false, then in fit(ds) or fit_transform(ds), ds.tensors is allowed to be None. :param is_individual: Set to false if fit(ds) deterministically produces a non-trainable layer. (In this case, this Fitter only needs to be called once in k-fold CV on the train+val set.) :param scope_names: List of names to add to the scope (will be present in the names of Variables constructed in this Fitter) :param modified_tensors: List of names of tensors that are modified by this Fitter, e.g., [‘x_cont’]. This is used for the default implementation of get_n_forward(), which is used to get a RAM estimate for the forward pass. The default RAM estimate is simply the size of all modified tensors.

forward_tensor_infos(tensor_infos)

Should be overridden if the fitter changes the tensor shapes. :param tensor_infos: Tensor infos (for shapes and category sizes). :return: Transformed tensor infos.

Parameters:

tensor_infos (Dict[str, TensorInfo])

Return type:

Dict[str, TensorInfo]

class pytabkit.models.nn_models.base.FilterTensorsLayer

Bases: Layer

Only returns those tensors whose name is in a list of names

__init__(include_keys, exclude_keys, fitter)
Parameters:

  • keys – List of tensor names that is allowed to pass through

  • include_keys (List[str] | None)

  • exclude_keys (List[str] | None)

  • fitter (Fitter)

forward_tensors(tensors)

Transforms the given tensors. :param tensors: :return:

Parameters:

tensors (Dict[str, Tensor])

Return type:

Dict[str, Tensor]

class pytabkit.models.nn_models.base.Fitter

Bases: ContextAware, StringConvertible

Fitters produce Layer objects given a data set (of inputs to the fitter at initialization)

__init__(needs_tensors=True, is_individual=True, scope_names=None, modified_tensors=None)
Parameters:

  • needs_tensors (bool) – Set to true if the fitter needs to have the tensors in fit() or fit_transform().

  • is_individual (bool)

  • scope_names (List[str] | None)

  • modified_tensors (List[str] | None)

If false, then in fit(ds) or fit_transform(ds), ds.tensors is allowed to be None. :param is_individual: Set to false if fit(ds) deterministically produces a non-trainable layer. (In this case, this Fitter only needs to be called once in k-fold CV on the train+val set.) :param scope_names: List of names to add to the scope (will be present in the names of Variables constructed in this Fitter) :param modified_tensors: List of names of tensors that are modified by this Fitter, e.g., [‘x_cont’]. This is used for the default implementation of get_n_forward(), which is used to get a RAM estimate for the forward pass. The default RAM estimate is simply the size of all modified tensors.

fit(ds)

Produces a layer initialized based on a given data set. This method should not be overridden, override _fit() instead. :param ds: Data set. :return: Layer object.

Parameters:

ds (DictDataset)

Return type:

Layer

fit_transform(ds, needs_tensors=True)

Produces a layer initialized based on a given data set. This method should not be overridden, override _fit_transform() instead. :param ds: Data set. :param needs_tensors: Whether the transformed data set should also contain transformed tensors

(compared to only transformed tensor_infos).

Returns:

Layer object and the data set transformed by the Layer.

Parameters:
Return type:

Tuple[Layer, DictDataset]

fit_transform_subsample(ds, ram_limit_gb, needs_tensors=True)

Similar to fit_transform(), but may subsample the data set in order to stay within a given RAM limit. This method should not be overridden, override _fit_transform_subsample() instead. :param ds: Data set. :param ram_limit_gb: RAM limit in GB. :param needs_tensors: Whether the transformed tensors should be output. :return: Tuple of the resulting Layer and the transformed DictDataset.

Parameters:

  • ds (DictDataset)

  • ram_limit_gb (float)

  • needs_tensors (bool)

Return type:

Tuple[Layer, DictDataset]

forward_tensor_infos(tensor_infos)

Should be overridden if the fitter changes the tensor shapes. :param tensor_infos: Tensor infos (for shapes and category sizes). :return: Transformed tensor infos.

Parameters:

tensor_infos (Dict[str, TensorInfo])

Return type:

Dict[str, TensorInfo]

get_n_forward(tensor_infos)

Should be overridden if the fitter does more than just one operation. :param tensor_infos: Ingoing tensor infos. :return: Should return the number of bytes used in the forward pass per batch element

Parameters:

tensor_infos (Dict[str, TensorInfo])

Return type:

int

get_n_params(tensor_infos)

Should be overridden if the fitter produces layers with trainable parameters. :param tensor_infos: Tensor infos. :return: Returns the number of parameters of the fitted layer for the given tensor_infos.

Parameters:

tensor_infos (Dict[str, TensorInfo])

Return type:

int

split_off_dynamic()

Can be overridden by subclasses if a trivial split based on self.needs_tensors and self.is_individual is not desired. :return: Returns a tuple of a static and a dynamic transform such that self is equivalent to SequentialFitter([static, dynamic]) and such that the static transform does not need data and is not trainable. The idea is that in the vectorized setting, the static transform only needs to be applied once to the data set, while the dynamic transform needs to be applied separately for each of the vectorized models.

Return type:

Tuple[Fitter, Fitter]

split_off_individual()

Can be overridden by subclasses if a trivial split based on self.is_individual is not desired. :return: Returns a tuple of a non-individual and an individual transform such that self is equivalent to SequentialFitter([non_individual, individual]) and such that the non_individual transform deterministically produces a non-trainable layer. The idea is that the non-individual transform only needs to be applied once in k-fold cross-validation.

class pytabkit.models.nn_models.base.FitterFactory

Bases: ContextAware, StringConvertible

Class that allows to create Fitter objects depending on tensor_infos (the shape and category sizes of the tensors).

__init__(scope_names=None)
Parameters:

scope_names (List[str] | None)

create(tensor_infos)

Creates a Fitter object with the scope given in the constructor. Do not override this method, override _create() or _create_transform() instead. :param tensor_infos: Tensor infos (shapes etc.) :return: Fitter object.

Parameters:

tensor_infos (Dict[str, TensorInfo])

Return type:

Fitter

create_transform(tensor_infos)

Creates a Fitter object with the scope given in the constructor. Do not override this method, override _create() or _create_transform() instead. :param tensor_infos: Tensor infos (shapes etc.) :return: Fitter object and the transformed tensor infos.

Parameters:

tensor_infos (Dict[str, TensorInfo])

Return type:

Tuple[Fitter, Dict[str, TensorInfo]]

class pytabkit.models.nn_models.base.FunctionFactory

Bases: FitterFactory

__init__(f)
class pytabkit.models.nn_models.base.FunctionFitter

Bases: Fitter

__init__(f, **config)
Parameters:

needs_tensors – Set to true if the fitter needs to have the tensors in fit() or fit_transform().

If false, then in fit(ds) or fit_transform(ds), ds.tensors is allowed to be None. :param is_individual: Set to false if fit(ds) deterministically produces a non-trainable layer. (In this case, this Fitter only needs to be called once in k-fold CV on the train+val set.) :param scope_names: List of names to add to the scope (will be present in the names of Variables constructed in this Fitter) :param modified_tensors: List of names of tensors that are modified by this Fitter, e.g., [‘x_cont’]. This is used for the default implementation of get_n_forward(), which is used to get a RAM estimate for the forward pass. The default RAM estimate is simply the size of all modified tensors.

class pytabkit.models.nn_models.base.FunctionLayer

Bases: Layer

__init__(f)

Constructor. Puts the layer in eval mode, since it might be used inside the fit_transform() of the Fitter. The parameters provide different opportunities to specify a default implementation for forward_tensor_infos(). The default implementation is: ```

if self.fitter is not None:

return self.fitter.forward_tensor_infos(tensor_infos)

return utils.update_dict(tensor_infos, self.new_tensor_infos, remove_keys=self.remove_keys)

```

forward_cont(x)
Parameters:

x (Tensor)

Return type:

Tensor

class pytabkit.models.nn_models.base.IdentityFactory

Bases: FitterFactory

class pytabkit.models.nn_models.base.IdentityFitter

Bases: Fitter

__init__(**config)
Parameters:

needs_tensors – Set to true if the fitter needs to have the tensors in fit() or fit_transform().

If false, then in fit(ds) or fit_transform(ds), ds.tensors is allowed to be None. :param is_individual: Set to false if fit(ds) deterministically produces a non-trainable layer. (In this case, this Fitter only needs to be called once in k-fold CV on the train+val set.) :param scope_names: List of names to add to the scope (will be present in the names of Variables constructed in this Fitter) :param modified_tensors: List of names of tensors that are modified by this Fitter, e.g., [‘x_cont’]. This is used for the default implementation of get_n_forward(), which is used to get a RAM estimate for the forward pass. The default RAM estimate is simply the size of all modified tensors.

class pytabkit.models.nn_models.base.IdentityLayer

Bases: Layer

forward_tensors(x)

Transforms the given tensors. :param tensors: :return:

class pytabkit.models.nn_models.base.Layer

Bases: ContextRecorder, StringConvertible, Module

Extended version of nn.Module, allowing vectorization, processing data sets with multiple tensors, using Variable instead of Parameter, …

The following methods need to be overridden: - forward_tensor_infos (but if the output is constant, we can just set new_tensor_infos in the constructor) - forward_tensor or forward_cont (the latter if only x_cont is changed) - _stack() - optionally __repr__() and __str__()

__init__(new_tensor_infos=None, fitter=None, remove_keys=None)

Constructor. Puts the layer in eval mode, since it might be used inside the fit_transform() of the Fitter. The parameters provide different opportunities to specify a default implementation for forward_tensor_infos(). The default implementation is: ```

if self.fitter is not None:

return self.fitter.forward_tensor_infos(tensor_infos)

return utils.update_dict(tensor_infos, self.new_tensor_infos, remove_keys=self.remove_keys)

```

Parameters:

  • new_tensor_infos (Dict[str, TensorInfo] | None)

  • fitter (Fitter | None)

  • remove_keys (str | List[str] | None)

forward(data)

This is an implementation of the nn.Module forward() function, which is called by __call__(). Don’t override this method. :param data: data set or dict of tensors. :return: Transformed version of the data set or dict of tensors.

Parameters:

data (DictDataset | Dict[str, Tensor])

Return type:

DictDataset | Dict[str, Tensor]

forward_cont(x)
Parameters:

x (Tensor)

Return type:

Tensor

forward_ds(ds)
Parameters:

ds (DictDataset)

Return type:

DictDataset

forward_tensor_infos(tensor_infos)

Override this method if the information from constructor is not sufficient. :param tensor_infos: Tensor infos (shapes etc.) :return: Transformed tensor infos.

Parameters:

tensor_infos (Dict[str, TensorInfo])

Return type:

Dict[str, TensorInfo]

forward_tensors(tensors)

Transforms the given tensors. :param tensors: :return:

Parameters:

tensors (Dict[str, Tensor])

Return type:

Dict[str, Tensor]

stack(layers)

Vectorizes the given layers. The given layers should all have the same structure. Do not override this method, override _stack() instead. :param layers: Layers that should be stacked for vectorization. :return: Returns the stacked Layer object

Parameters:

layers (List[Layer])

Return type:

Layer

class pytabkit.models.nn_models.base.RenameTensorFactory

Bases: Fitter, FitterFactory

__init__(old_name, new_name, **config)
Parameters:

  • needs_tensors – Set to true if the fitter needs to have the tensors in fit() or fit_transform().

  • old_name (str)

  • new_name (str)

If false, then in fit(ds) or fit_transform(ds), ds.tensors is allowed to be None. :param is_individual: Set to false if fit(ds) deterministically produces a non-trainable layer. (In this case, this Fitter only needs to be called once in k-fold CV on the train+val set.) :param scope_names: List of names to add to the scope (will be present in the names of Variables constructed in this Fitter) :param modified_tensors: List of names of tensors that are modified by this Fitter, e.g., [‘x_cont’]. This is used for the default implementation of get_n_forward(), which is used to get a RAM estimate for the forward pass. The default RAM estimate is simply the size of all modified tensors.

forward_tensor_infos(tensor_infos)

Should be overridden if the fitter changes the tensor shapes. :param tensor_infos: Tensor infos (for shapes and category sizes). :return: Transformed tensor infos.

Parameters:

tensor_infos (Dict[str, TensorInfo])

Return type:

Dict[str, TensorInfo]

get_n_forward(tensor_infos)

Should be overridden if the fitter does more than just one operation. :param tensor_infos: Ingoing tensor infos. :return: Should return the number of bytes used in the forward pass per batch element

Parameters:

tensor_infos (Dict[str, TensorInfo])

Return type:

int

class pytabkit.models.nn_models.base.RenameTensorLayer

Bases: Layer

__init__(old_name, new_name, fitter)

Constructor. Puts the layer in eval mode, since it might be used inside the fit_transform() of the Fitter. The parameters provide different opportunities to specify a default implementation for forward_tensor_infos(). The default implementation is: ```

if self.fitter is not None:

return self.fitter.forward_tensor_infos(tensor_infos)

return utils.update_dict(tensor_infos, self.new_tensor_infos, remove_keys=self.remove_keys)

```

Parameters:

  • old_name (str)

  • new_name (str)

  • fitter (Fitter)

forward_tensors(tensors)

Transforms the given tensors. :param tensors: :return:

Parameters:

tensors (Dict[str, Tensor])

Return type:

Dict[str, Tensor]

class pytabkit.models.nn_models.base.ResidualFitter

Bases: Fitter

__init__(inner_fitter, **config)
Parameters:

  • needs_tensors – Set to true if the fitter needs to have the tensors in fit() or fit_transform().

  • inner_fitter (Fitter)

If false, then in fit(ds) or fit_transform(ds), ds.tensors is allowed to be None. :param is_individual: Set to false if fit(ds) deterministically produces a non-trainable layer. (In this case, this Fitter only needs to be called once in k-fold CV on the train+val set.) :param scope_names: List of names to add to the scope (will be present in the names of Variables constructed in this Fitter) :param modified_tensors: List of names of tensors that are modified by this Fitter, e.g., [‘x_cont’]. This is used for the default implementation of get_n_forward(), which is used to get a RAM estimate for the forward pass. The default RAM estimate is simply the size of all modified tensors.

forward_tensor_infos(tensor_infos)

Should be overridden if the fitter changes the tensor shapes. :param tensor_infos: Tensor infos (for shapes and category sizes). :return: Transformed tensor infos.

Parameters:

tensor_infos (Dict[str, TensorInfo])

get_n_forward(tensor_infos)

Should be overridden if the fitter does more than just one operation. :param tensor_infos: Ingoing tensor infos. :return: Should return the number of bytes used in the forward pass per batch element

Parameters:

tensor_infos (Dict[str, TensorInfo])

get_n_params(tensor_infos)

Should be overridden if the fitter produces layers with trainable parameters. :param tensor_infos: Tensor infos. :return: Returns the number of parameters of the fitted layer for the given tensor_infos.

Parameters:

tensor_infos (Dict[str, TensorInfo])

split_off_dynamic()

Can be overridden by subclasses if a trivial split based on self.needs_tensors and self.is_individual is not desired. :return: Returns a tuple of a static and a dynamic transform such that self is equivalent to SequentialFitter([static, dynamic]) and such that the static transform does not need data and is not trainable. The idea is that in the vectorized setting, the static transform only needs to be applied once to the data set, while the dynamic transform needs to be applied separately for each of the vectorized models.

split_off_individual()

Can be overridden by subclasses if a trivial split based on self.is_individual is not desired. :return: Returns a tuple of a non-individual and an individual transform such that self is equivalent to SequentialFitter([non_individual, individual]) and such that the non_individual transform deterministically produces a non-trainable layer. The idea is that the non-individual transform only needs to be applied once in k-fold cross-validation.

class pytabkit.models.nn_models.base.ResidualLayer

Bases: Layer

__init__(inner_layer)

Constructor. Puts the layer in eval mode, since it might be used inside the fit_transform() of the Fitter. The parameters provide different opportunities to specify a default implementation for forward_tensor_infos(). The default implementation is: ```

if self.fitter is not None:

return self.fitter.forward_tensor_infos(tensor_infos)

return utils.update_dict(tensor_infos, self.new_tensor_infos, remove_keys=self.remove_keys)

```

Parameters:

inner_layer (Layer)

forward_tensor_infos(tensor_infos)

Override this method if the information from constructor is not sufficient. :param tensor_infos: Tensor infos (shapes etc.) :return: Transformed tensor infos.

forward_tensors(tensors)

Transforms the given tensors. :param tensors: :return:

Parameters:

tensors (Dict[str, Tensor])

class pytabkit.models.nn_models.base.ScaleLayer

Bases: Layer

__init__(scale)

Constructor. Puts the layer in eval mode, since it might be used inside the fit_transform() of the Fitter. The parameters provide different opportunities to specify a default implementation for forward_tensor_infos(). The default implementation is: ```

if self.fitter is not None:

return self.fitter.forward_tensor_infos(tensor_infos)

return utils.update_dict(tensor_infos, self.new_tensor_infos, remove_keys=self.remove_keys)

```

Parameters:

scale (Variable)

forward_cont(x)
class pytabkit.models.nn_models.base.Scope

Bases: object

__init__(names=None)
Parameters:

names (List[str] | None)

get_sub_scope(name)
Parameters:

name (str)

Return type:

Scope

matches(regex)
Parameters:

regex (str | Pattern)

Return type:

bool

class pytabkit.models.nn_models.base.SequentialFactory

Bases: FitterFactory

__init__(factories)
Parameters:

factories (List[FitterFactory])

class pytabkit.models.nn_models.base.SequentialFitter

Bases: Fitter

__init__(fitters, **config)
Parameters:

  • needs_tensors – Set to true if the fitter needs to have the tensors in fit() or fit_transform().

  • fitters (List[Fitter])

If false, then in fit(ds) or fit_transform(ds), ds.tensors is allowed to be None. :param is_individual: Set to false if fit(ds) deterministically produces a non-trainable layer. (In this case, this Fitter only needs to be called once in k-fold CV on the train+val set.) :param scope_names: List of names to add to the scope (will be present in the names of Variables constructed in this Fitter) :param modified_tensors: List of names of tensors that are modified by this Fitter, e.g., [‘x_cont’]. This is used for the default implementation of get_n_forward(), which is used to get a RAM estimate for the forward pass. The default RAM estimate is simply the size of all modified tensors.

forward_tensor_infos(tensor_infos)

Should be overridden if the fitter changes the tensor shapes. :param tensor_infos: Tensor infos (for shapes and category sizes). :return: Transformed tensor infos.

Parameters:

tensor_infos (Dict[str, TensorInfo])

get_n_forward(tensor_infos)

Should be overridden if the fitter does more than just one operation. :param tensor_infos: Ingoing tensor infos. :return: Should return the number of bytes used in the forward pass per batch element

Parameters:

tensor_infos (Dict[str, TensorInfo])

get_n_params(tensor_infos)

Should be overridden if the fitter produces layers with trainable parameters. :param tensor_infos: Tensor infos. :return: Returns the number of parameters of the fitted layer for the given tensor_infos.

Parameters:

tensor_infos (Dict[str, TensorInfo])

split_off_dynamic()

Can be overridden by subclasses if a trivial split based on self.needs_tensors and self.is_individual is not desired. :return: Returns a tuple of a static and a dynamic transform such that self is equivalent to SequentialFitter([static, dynamic]) and such that the static transform does not need data and is not trainable. The idea is that in the vectorized setting, the static transform only needs to be applied once to the data set, while the dynamic transform needs to be applied separately for each of the vectorized models.

split_off_individual()

Can be overridden by subclasses if a trivial split based on self.is_individual is not desired. :return: Returns a tuple of a non-individual and an individual transform such that self is equivalent to SequentialFitter([non_individual, individual]) and such that the non_individual transform deterministically produces a non-trainable layer. The idea is that the non-individual transform only needs to be applied once in k-fold cross-validation.

class pytabkit.models.nn_models.base.SequentialLayer

Bases: Layer

__init__(tfms)

Constructor. Puts the layer in eval mode, since it might be used inside the fit_transform() of the Fitter. The parameters provide different opportunities to specify a default implementation for forward_tensor_infos(). The default implementation is: ```

if self.fitter is not None:

return self.fitter.forward_tensor_infos(tensor_infos)

return utils.update_dict(tensor_infos, self.new_tensor_infos, remove_keys=self.remove_keys)

```

Parameters:

tfms (List[Layer])

forward_ds(ds)
Parameters:

ds (DictDataset)

forward_tensor_infos(tensor_infos)

Override this method if the information from constructor is not sufficient. :param tensor_infos: Tensor infos (shapes etc.) :return: Transformed tensor infos.

forward_tensors(tensors)

Transforms the given tensors. :param tensors: :return:

class pytabkit.models.nn_models.base.StringConvertible

Bases: object

__init__()
class pytabkit.models.nn_models.base.TrainContext

Bases: object

__init__(scope=None, hp_manager=None)
Parameters:
clone()
static get_global_context()
Return type:

TrainContext

class pytabkit.models.nn_models.base.Variable

Bases: ContextRecorder, Parameter

__init__(data=None, trainable=True, requires_grad=None, hyper_factors=None)
static __new__(cls, data=None, trainable=True, requires_grad=None, hyper_factors=None)
static stack(vars, dim=0)
Parameters:

vars (List[Variable])

class pytabkit.models.nn_models.base.WeightLayer

Bases: Layer

__init__(weight, factor=1.0)

Constructor. Puts the layer in eval mode, since it might be used inside the fit_transform() of the Fitter. The parameters provide different opportunities to specify a default implementation for forward_tensor_infos(). The default implementation is: ```

if self.fitter is not None:

return self.fitter.forward_tensor_infos(tensor_infos)

return utils.update_dict(tensor_infos, self.new_tensor_infos, remove_keys=self.remove_keys)

```

Parameters:
forward_cont(x)
pytabkit.models.nn_models.base.set_hp_context(hp_manager)
Parameters:

hp_manager (HyperparamManager | None)

pytabkit.models.nn_models.base.set_scope_context(scope)
Parameters:

scope (Scope)

pytabkit.models.nn_models.base.sub_scope_context(name)
Parameters:

name (str)

pytabkit.models.nn_models.base.sub_scopes_context(names)
Parameters:

names (List[str])

pytabkit.models.nn_models.categorical module

class pytabkit.models.nn_models.categorical.ConstantFunction

Bases: object

__init__(value)
Parameters:

value (Any)

class pytabkit.models.nn_models.categorical.EncodingFactory

Bases: FitterFactory

__init__(single_encoder_factory, enc_output_name='x_cont')
Parameters:

enc_output_name (str)

class pytabkit.models.nn_models.categorical.EncodingFitter

Bases: Fitter

__init__(single_encoder_fitters, enc_output_name='x_cont', **config)
Parameters:

  • needs_tensors – Set to true if the fitter needs to have the tensors in fit() or fit_transform().

  • single_encoder_fitters (List[Fitter])

  • enc_output_name (str)

If false, then in fit(ds) or fit_transform(ds), ds.tensors is allowed to be None. :param is_individual: Set to false if fit(ds) deterministically produces a non-trainable layer. (In this case, this Fitter only needs to be called once in k-fold CV on the train+val set.) :param scope_names: List of names to add to the scope (will be present in the names of Variables constructed in this Fitter) :param modified_tensors: List of names of tensors that are modified by this Fitter, e.g., [‘x_cont’]. This is used for the default implementation of get_n_forward(), which is used to get a RAM estimate for the forward pass. The default RAM estimate is simply the size of all modified tensors.

forward_tensor_infos(tensor_infos)

Should be overridden if the fitter changes the tensor shapes. :param tensor_infos: Tensor infos (for shapes and category sizes). :return: Transformed tensor infos.

get_n_forward(tensor_infos)

Should be overridden if the fitter does more than just one operation. :param tensor_infos: Ingoing tensor infos. :return: Should return the number of bytes used in the forward pass per batch element

Parameters:

tensor_infos (Dict[str, TensorInfo])

Return type:

int

get_n_params(tensor_infos)

Should be overridden if the fitter produces layers with trainable parameters. :param tensor_infos: Tensor infos. :return: Returns the number of parameters of the fitted layer for the given tensor_infos.

Parameters:

tensor_infos (Dict[str, TensorInfo])

Return type:

int

class pytabkit.models.nn_models.categorical.EncodingLayer

Bases: Layer

__init__(single_enc_layers, enc_output_name, fitter)

Constructor. Puts the layer in eval mode, since it might be used inside the fit_transform() of the Fitter. The parameters provide different opportunities to specify a default implementation for forward_tensor_infos(). The default implementation is: ```

if self.fitter is not None:

return self.fitter.forward_tensor_infos(tensor_infos)

return utils.update_dict(tensor_infos, self.new_tensor_infos, remove_keys=self.remove_keys)

```

Parameters:

  • single_enc_layers (Iterable[Layer])

  • enc_output_name (str)

forward_tensors(tensors)

Transforms the given tensors. :param tensors: :return:

class pytabkit.models.nn_models.categorical.SingleEmbeddingFactory

Bases: SingleEncodingFactory

__init__(embedding_size=None, min_embedding_cat_size=0, max_embedding_cat_size=-1, **config)
class pytabkit.models.nn_models.categorical.SingleEmbeddingFitter

Bases: Fitter

__init__(embedding_size=None, **config)
Parameters:

needs_tensors – Set to true if the fitter needs to have the tensors in fit() or fit_transform().

If false, then in fit(ds) or fit_transform(ds), ds.tensors is allowed to be None. :param is_individual: Set to false if fit(ds) deterministically produces a non-trainable layer. (In this case, this Fitter only needs to be called once in k-fold CV on the train+val set.) :param scope_names: List of names to add to the scope (will be present in the names of Variables constructed in this Fitter) :param modified_tensors: List of names of tensors that are modified by this Fitter, e.g., [‘x_cont’]. This is used for the default implementation of get_n_forward(), which is used to get a RAM estimate for the forward pass. The default RAM estimate is simply the size of all modified tensors.

forward_tensor_infos(tensor_infos)

Should be overridden if the fitter changes the tensor shapes. :param tensor_infos: Tensor infos (for shapes and category sizes). :return: Transformed tensor infos.

get_n_params(tensor_infos)

Should be overridden if the fitter produces layers with trainable parameters. :param tensor_infos: Tensor infos. :return: Returns the number of parameters of the fitted layer for the given tensor_infos.

Parameters:

tensor_infos (Dict[str, TensorInfo])

Return type:

int

class pytabkit.models.nn_models.categorical.SingleEmbeddingLayer

Bases: Layer

__init__(emb)

Constructor. Puts the layer in eval mode, since it might be used inside the fit_transform() of the Fitter. The parameters provide different opportunities to specify a default implementation for forward_tensor_infos(). The default implementation is: ```

if self.fitter is not None:

return self.fitter.forward_tensor_infos(tensor_infos)

return utils.update_dict(tensor_infos, self.new_tensor_infos, remove_keys=self.remove_keys)

```

Parameters:

emb (Variable)

forward_tensors(tensors)

Transforms the given tensors. :param tensors: :return:

class pytabkit.models.nn_models.categorical.SingleEncodingFactory

Bases: FitterFactory

__init__(create_fitter, min_cat_size=0, max_cat_size=-1)
apply_on(cat_size, n_classes)
Parameters:

  • cat_size (int)

  • n_classes (int)

class pytabkit.models.nn_models.categorical.SingleOneHotFactory

Bases: SingleEncodingFactory

__init__(use_missing_zero=True, bin_onoff=(1.0, 0.0), multi_onoff=(1.0, 0.0), min_one_hot_cat_size=0, max_one_hot_cat_size=-1, max_one_hot_size_by_n_classes=False, use_1d_binary_onehot=True, **config)
Parameters:

use_1d_binary_onehot (bool)

apply_on(cat_size, n_classes)
Parameters:

  • cat_size (int)

  • n_classes (int)

class pytabkit.models.nn_models.categorical.SingleOneHotFitter

Bases: Fitter

__init__(use_missing_zero, bin_onoff, multi_onoff, use_1d_binary_onehot)
Parameters:

  • needs_tensors – Set to true if the fitter needs to have the tensors in fit() or fit_transform().

  • use_missing_zero (bool)

  • bin_onoff (Tuple[float, float])

  • multi_onoff (Tuple[float, float])

  • use_1d_binary_onehot (bool)

If false, then in fit(ds) or fit_transform(ds), ds.tensors is allowed to be None. :param is_individual: Set to false if fit(ds) deterministically produces a non-trainable layer. (In this case, this Fitter only needs to be called once in k-fold CV on the train+val set.) :param scope_names: List of names to add to the scope (will be present in the names of Variables constructed in this Fitter) :param modified_tensors: List of names of tensors that are modified by this Fitter, e.g., [‘x_cont’]. This is used for the default implementation of get_n_forward(), which is used to get a RAM estimate for the forward pass. The default RAM estimate is simply the size of all modified tensors.

forward_tensor_infos(tensor_infos)

Should be overridden if the fitter changes the tensor shapes. :param tensor_infos: Tensor infos (for shapes and category sizes). :return: Transformed tensor infos.

class pytabkit.models.nn_models.categorical.SingleOneHotLayer

Bases: Layer

__init__(fitter, onoff, cat_size, use_missing_zero, use_1d_binary_onehot)

Constructor. Puts the layer in eval mode, since it might be used inside the fit_transform() of the Fitter. The parameters provide different opportunities to specify a default implementation for forward_tensor_infos(). The default implementation is: ```

if self.fitter is not None:

return self.fitter.forward_tensor_infos(tensor_infos)

return utils.update_dict(tensor_infos, self.new_tensor_infos, remove_keys=self.remove_keys)

```

Parameters:

  • fitter (Fitter)

  • use_missing_zero (bool)

  • use_1d_binary_onehot (bool)

forward_tensors(tensors)

Transforms the given tensors. :param tensors: :return:

class pytabkit.models.nn_models.categorical.SingleOrdinalEncodingFactory

Bases: SingleEncodingFactory

__init__(min_labelenc_cat_size=0, max_labelenc_cat_size=-1, **config)
class pytabkit.models.nn_models.categorical.SingleOrdinalEncodingFitter

Bases: Fitter

__init__(permute_ordinal_encoding=False, **config)
Parameters:

  • needs_tensors – Set to true if the fitter needs to have the tensors in fit() or fit_transform().

  • permute_ordinal_encoding (bool)

If false, then in fit(ds) or fit_transform(ds), ds.tensors is allowed to be None. :param is_individual: Set to false if fit(ds) deterministically produces a non-trainable layer. (In this case, this Fitter only needs to be called once in k-fold CV on the train+val set.) :param scope_names: List of names to add to the scope (will be present in the names of Variables constructed in this Fitter) :param modified_tensors: List of names of tensors that are modified by this Fitter, e.g., [‘x_cont’]. This is used for the default implementation of get_n_forward(), which is used to get a RAM estimate for the forward pass. The default RAM estimate is simply the size of all modified tensors.

forward_tensor_infos(tensor_infos)

Should be overridden if the fitter changes the tensor shapes. :param tensor_infos: Tensor infos (for shapes and category sizes). :return: Transformed tensor infos.

class pytabkit.models.nn_models.categorical.SingleOrdinalEncodingLayer

Bases: Layer

__init__(fitter, cat_size, permute_ordinal_encoding=False)

Constructor. Puts the layer in eval mode, since it might be used inside the fit_transform() of the Fitter. The parameters provide different opportunities to specify a default implementation for forward_tensor_infos(). The default implementation is: ```

if self.fitter is not None:

return self.fitter.forward_tensor_infos(tensor_infos)

return utils.update_dict(tensor_infos, self.new_tensor_infos, remove_keys=self.remove_keys)

```

Parameters:

  • cat_size (int)

  • permute_ordinal_encoding (bool)

forward_tensors(tensors)

Transforms the given tensors. :param tensors: :return:

class pytabkit.models.nn_models.categorical.SingleTargetEncodingFactory

Bases: SingleEncodingFactory

__init__(min_targetenc_cat_size=0, max_targetenc_cat_size=-1, **config)
class pytabkit.models.nn_models.categorical.SingleTargetEncodingFitter

Bases: Fitter

__init__(n_classes, **config)
Parameters:

needs_tensors – Set to true if the fitter needs to have the tensors in fit() or fit_transform().

If false, then in fit(ds) or fit_transform(ds), ds.tensors is allowed to be None. :param is_individual: Set to false if fit(ds) deterministically produces a non-trainable layer. (In this case, this Fitter only needs to be called once in k-fold CV on the train+val set.) :param scope_names: List of names to add to the scope (will be present in the names of Variables constructed in this Fitter) :param modified_tensors: List of names of tensors that are modified by this Fitter, e.g., [‘x_cont’]. This is used for the default implementation of get_n_forward(), which is used to get a RAM estimate for the forward pass. The default RAM estimate is simply the size of all modified tensors.

forward_tensor_infos(tensor_infos)

Should be overridden if the fitter changes the tensor shapes. :param tensor_infos: Tensor infos (for shapes and category sizes). :return: Transformed tensor infos.

get_n_params(tensor_infos)

Should be overridden if the fitter produces layers with trainable parameters. :param tensor_infos: Tensor infos. :return: Returns the number of parameters of the fitted layer for the given tensor_infos.

Parameters:

tensor_infos (Dict[str, TensorInfo])

Return type:

int

pytabkit.models.nn_models.categorical.fastai_emb_size_fn(n_cat)
Parameters:

n_cat (int)

pytabkit.models.nn_models.categorical.get_embedding_size(fn)
Parameters:

fn (int | str | Callable[[int], int] | None)

Return type:

Callable[[int], int]

pytabkit.models.nn_models.models module

class pytabkit.models.nn_models.models.BlockFactory

Bases: FitterFactory

__init__(out_features, block_str='w-b-a', **config)
Parameters:

  • out_features (int)

  • block_str (str)

class pytabkit.models.nn_models.models.NNFactory

Bases: FitterFactory

__init__(**config)
class pytabkit.models.nn_models.models.PreprocessingFactory

Bases: FitterFactory

__init__(**config)
pytabkit.models.nn_models.models.smooth_clip_func(x, max_abs_value=3.0)
Parameters:

max_abs_value (float)

pytabkit.models.nn_models.models.tanh_clip_func(x)

pytabkit.models.nn_models.nn module

class pytabkit.models.nn_models.nn.AntisymmetricInitializationFactory

Bases: FitterFactory

__init__(factory, **config)
class pytabkit.models.nn_models.nn.AntisymmetricInitializationFitter

Bases: Fitter

Implements the antisymmetric initialization trick from http://proceedings.mlr.press/v107/zhang20a/zhang20a.pdf

__init__(fitter, **config)
Parameters:

  • needs_tensors – Set to true if the fitter needs to have the tensors in fit() or fit_transform().

  • fitter (Fitter)

If false, then in fit(ds) or fit_transform(ds), ds.tensors is allowed to be None. :param is_individual: Set to false if fit(ds) deterministically produces a non-trainable layer. (In this case, this Fitter only needs to be called once in k-fold CV on the train+val set.) :param scope_names: List of names to add to the scope (will be present in the names of Variables constructed in this Fitter) :param modified_tensors: List of names of tensors that are modified by this Fitter, e.g., [‘x_cont’]. This is used for the default implementation of get_n_forward(), which is used to get a RAM estimate for the forward pass. The default RAM estimate is simply the size of all modified tensors.

forward_tensor_infos(tensor_infos)

Should be overridden if the fitter changes the tensor shapes. :param tensor_infos: Tensor infos (for shapes and category sizes). :return: Transformed tensor infos.

Parameters:

tensor_infos (Dict[str, TensorInfo])

get_n_forward(tensor_infos)

Should be overridden if the fitter does more than just one operation. :param tensor_infos: Ingoing tensor infos. :return: Should return the number of bytes used in the forward pass per batch element

Parameters:

tensor_infos (Dict[str, TensorInfo])

get_n_params(tensor_infos)

Should be overridden if the fitter produces layers with trainable parameters. :param tensor_infos: Tensor infos. :return: Returns the number of parameters of the fitted layer for the given tensor_infos.

Parameters:

tensor_infos (Dict[str, TensorInfo])

class pytabkit.models.nn_models.nn.BiasFitter

Bases: Fitter

__init__(**config)
Parameters:

needs_tensors – Set to true if the fitter needs to have the tensors in fit() or fit_transform().

If false, then in fit(ds) or fit_transform(ds), ds.tensors is allowed to be None. :param is_individual: Set to false if fit(ds) deterministically produces a non-trainable layer. (In this case, this Fitter only needs to be called once in k-fold CV on the train+val set.) :param scope_names: List of names to add to the scope (will be present in the names of Variables constructed in this Fitter) :param modified_tensors: List of names of tensors that are modified by this Fitter, e.g., [‘x_cont’]. This is used for the default implementation of get_n_forward(), which is used to get a RAM estimate for the forward pass. The default RAM estimate is simply the size of all modified tensors.

get_n_params(tensor_infos)

Should be overridden if the fitter produces layers with trainable parameters. :param tensor_infos: Tensor infos. :return: Returns the number of parameters of the fitted layer for the given tensor_infos.

Parameters:

tensor_infos (Dict[str, TensorInfo])

Return type:

int

heplus_bias(x, n_simplex)
class pytabkit.models.nn_models.nn.ClampLayer

Bases: Layer

__init__(low, high)

Constructor. Puts the layer in eval mode, since it might be used inside the fit_transform() of the Fitter. The parameters provide different opportunities to specify a default implementation for forward_tensor_infos(). The default implementation is: ```

if self.fitter is not None:

return self.fitter.forward_tensor_infos(tensor_infos)

return utils.update_dict(tensor_infos, self.new_tensor_infos, remove_keys=self.remove_keys)

```

Parameters:
forward_cont(x)
class pytabkit.models.nn_models.nn.ClampOutputFactory

Bases: Fitter, FitterFactory

__init__(**config)
Parameters:

needs_tensors – Set to true if the fitter needs to have the tensors in fit() or fit_transform().

If false, then in fit(ds) or fit_transform(ds), ds.tensors is allowed to be None. :param is_individual: Set to false if fit(ds) deterministically produces a non-trainable layer. (In this case, this Fitter only needs to be called once in k-fold CV on the train+val set.) :param scope_names: List of names to add to the scope (will be present in the names of Variables constructed in this Fitter) :param modified_tensors: List of names of tensors that are modified by this Fitter, e.g., [‘x_cont’]. This is used for the default implementation of get_n_forward(), which is used to get a RAM estimate for the forward pass. The default RAM estimate is simply the size of all modified tensors.

get_n_params(tensor_infos)

Should be overridden if the fitter produces layers with trainable parameters. :param tensor_infos: Tensor infos. :return: Returns the number of parameters of the fitted layer for the given tensor_infos.

Parameters:

tensor_infos (Dict[str, TensorInfo])

Return type:

int

class pytabkit.models.nn_models.nn.DropoutFitter

Bases: Fitter

__init__()
Parameters:

needs_tensors – Set to true if the fitter needs to have the tensors in fit() or fit_transform().

If false, then in fit(ds) or fit_transform(ds), ds.tensors is allowed to be None. :param is_individual: Set to false if fit(ds) deterministically produces a non-trainable layer. (In this case, this Fitter only needs to be called once in k-fold CV on the train+val set.) :param scope_names: List of names to add to the scope (will be present in the names of Variables constructed in this Fitter) :param modified_tensors: List of names of tensors that are modified by this Fitter, e.g., [‘x_cont’]. This is used for the default implementation of get_n_forward(), which is used to get a RAM estimate for the forward pass. The default RAM estimate is simply the size of all modified tensors.

class pytabkit.models.nn_models.nn.DropoutLayer

Bases: Layer

__init__()

Constructor. Puts the layer in eval mode, since it might be used inside the fit_transform() of the Fitter. The parameters provide different opportunities to specify a default implementation for forward_tensor_infos(). The default implementation is: ```

if self.fitter is not None:

return self.fitter.forward_tensor_infos(tensor_infos)

return utils.update_dict(tensor_infos, self.new_tensor_infos, remove_keys=self.remove_keys)

```

forward_cont(x)
class pytabkit.models.nn_models.nn.FeatureImportanceFactory

Bases: Fitter, FitterFactory

__init__()
Parameters:

needs_tensors – Set to true if the fitter needs to have the tensors in fit() or fit_transform().

If false, then in fit(ds) or fit_transform(ds), ds.tensors is allowed to be None. :param is_individual: Set to false if fit(ds) deterministically produces a non-trainable layer. (In this case, this Fitter only needs to be called once in k-fold CV on the train+val set.) :param scope_names: List of names to add to the scope (will be present in the names of Variables constructed in this Fitter) :param modified_tensors: List of names of tensors that are modified by this Fitter, e.g., [‘x_cont’]. This is used for the default implementation of get_n_forward(), which is used to get a RAM estimate for the forward pass. The default RAM estimate is simply the size of all modified tensors.

class pytabkit.models.nn_models.nn.FixedScaleFactory

Bases: Fitter, FitterFactory

__init__(scale)
Parameters:

  • needs_tensors – Set to true if the fitter needs to have the tensors in fit() or fit_transform().

  • scale (Tensor)

If false, then in fit(ds) or fit_transform(ds), ds.tensors is allowed to be None. :param is_individual: Set to false if fit(ds) deterministically produces a non-trainable layer. (In this case, this Fitter only needs to be called once in k-fold CV on the train+val set.) :param scope_names: List of names to add to the scope (will be present in the names of Variables constructed in this Fitter) :param modified_tensors: List of names of tensors that are modified by this Fitter, e.g., [‘x_cont’]. This is used for the default implementation of get_n_forward(), which is used to get a RAM estimate for the forward pass. The default RAM estimate is simply the size of all modified tensors.

class pytabkit.models.nn_models.nn.FixedWeightFactory

Bases: Fitter, FitterFactory

__init__()
Parameters:

needs_tensors – Set to true if the fitter needs to have the tensors in fit() or fit_transform().

If false, then in fit(ds) or fit_transform(ds), ds.tensors is allowed to be None. :param is_individual: Set to false if fit(ds) deterministically produces a non-trainable layer. (In this case, this Fitter only needs to be called once in k-fold CV on the train+val set.) :param scope_names: List of names to add to the scope (will be present in the names of Variables constructed in this Fitter) :param modified_tensors: List of names of tensors that are modified by this Fitter, e.g., [‘x_cont’]. This is used for the default implementation of get_n_forward(), which is used to get a RAM estimate for the forward pass. The default RAM estimate is simply the size of all modified tensors.

class pytabkit.models.nn_models.nn.LabelSmoothingFactory

Bases: FitterFactory

__init__(**config)
class pytabkit.models.nn_models.nn.LabelSmoothingFitter

Bases: Fitter

__init__(use_ls_prior=False, **config)
Parameters:

needs_tensors – Set to true if the fitter needs to have the tensors in fit() or fit_transform().

If false, then in fit(ds) or fit_transform(ds), ds.tensors is allowed to be None. :param is_individual: Set to false if fit(ds) deterministically produces a non-trainable layer. (In this case, this Fitter only needs to be called once in k-fold CV on the train+val set.) :param scope_names: List of names to add to the scope (will be present in the names of Variables constructed in this Fitter) :param modified_tensors: List of names of tensors that are modified by this Fitter, e.g., [‘x_cont’]. This is used for the default implementation of get_n_forward(), which is used to get a RAM estimate for the forward pass. The default RAM estimate is simply the size of all modified tensors.

class pytabkit.models.nn_models.nn.LabelSmoothingLayer

Bases: Layer

__init__(ls_dist)

Constructor. Puts the layer in eval mode, since it might be used inside the fit_transform() of the Fitter. The parameters provide different opportunities to specify a default implementation for forward_tensor_infos(). The default implementation is: ```

if self.fitter is not None:

return self.fitter.forward_tensor_infos(tensor_infos)

return utils.update_dict(tensor_infos, self.new_tensor_infos, remove_keys=self.remove_keys)

```

Parameters:

ls_dist (Variable)

forward_tensors(tensors)

Transforms the given tensors. :param tensors: :return:

class pytabkit.models.nn_models.nn.NoiseFitter

Bases: Fitter

__init__(**config)
Parameters:

needs_tensors – Set to true if the fitter needs to have the tensors in fit() or fit_transform().

If false, then in fit(ds) or fit_transform(ds), ds.tensors is allowed to be None. :param is_individual: Set to false if fit(ds) deterministically produces a non-trainable layer. (In this case, this Fitter only needs to be called once in k-fold CV on the train+val set.) :param scope_names: List of names to add to the scope (will be present in the names of Variables constructed in this Fitter) :param modified_tensors: List of names of tensors that are modified by this Fitter, e.g., [‘x_cont’]. This is used for the default implementation of get_n_forward(), which is used to get a RAM estimate for the forward pass. The default RAM estimate is simply the size of all modified tensors.

class pytabkit.models.nn_models.nn.NoiseLayer

Bases: Layer

__init__()

Constructor. Puts the layer in eval mode, since it might be used inside the fit_transform() of the Fitter. The parameters provide different opportunities to specify a default implementation for forward_tensor_infos(). The default implementation is: ```

if self.fitter is not None:

return self.fitter.forward_tensor_infos(tensor_infos)

return utils.update_dict(tensor_infos, self.new_tensor_infos, remove_keys=self.remove_keys)

```

forward_cont(x)
class pytabkit.models.nn_models.nn.NormWeightLayer

Bases: Layer

__init__(weight, factor, fitter, transpose=False)

Constructor. Puts the layer in eval mode, since it might be used inside the fit_transform() of the Fitter. The parameters provide different opportunities to specify a default implementation for forward_tensor_infos(). The default implementation is: ```

if self.fitter is not None:

return self.fitter.forward_tensor_infos(tensor_infos)

return utils.update_dict(tensor_infos, self.new_tensor_infos, remove_keys=self.remove_keys)

```

Parameters:
forward_cont(x)
class pytabkit.models.nn_models.nn.NormalizeOutputFactory

Bases: Fitter, FitterFactory

__init__(**config)
Parameters:

needs_tensors – Set to true if the fitter needs to have the tensors in fit() or fit_transform().

If false, then in fit(ds) or fit_transform(ds), ds.tensors is allowed to be None. :param is_individual: Set to false if fit(ds) deterministically produces a non-trainable layer. (In this case, this Fitter only needs to be called once in k-fold CV on the train+val set.) :param scope_names: List of names to add to the scope (will be present in the names of Variables constructed in this Fitter) :param modified_tensors: List of names of tensors that are modified by this Fitter, e.g., [‘x_cont’]. This is used for the default implementation of get_n_forward(), which is used to get a RAM estimate for the forward pass. The default RAM estimate is simply the size of all modified tensors.

get_n_params(tensor_infos)

Should be overridden if the fitter produces layers with trainable parameters. :param tensor_infos: Tensor infos. :return: Returns the number of parameters of the fitted layer for the given tensor_infos.

Parameters:

tensor_infos (Dict[str, TensorInfo])

Return type:

int

class pytabkit.models.nn_models.nn.NormalizeOutputLayer

Bases: Layer

__init__(mean, std)

Constructor. Puts the layer in eval mode, since it might be used inside the fit_transform() of the Fitter. The parameters provide different opportunities to specify a default implementation for forward_tensor_infos(). The default implementation is: ```

if self.fitter is not None:

return self.fitter.forward_tensor_infos(tensor_infos)

return utils.update_dict(tensor_infos, self.new_tensor_infos, remove_keys=self.remove_keys)

```

Parameters:
forward_tensors(tensors)

Transforms the given tensors. :param tensors: :return:

class pytabkit.models.nn_models.nn.PLREmbeddingsFactory

Bases: Fitter, FitterFactory

__init__(plr_sigma=1.0, plr_hidden_1=8, plr_hidden_2=8, plr_lr_factor=1.0, plr_lr_factor_1=1.0, plr_lr_factor_2=1.0, plr_wd_factor=1.0, plr_act_name='relu', plr_use_densenet=False, plr_use_cos_bias=False, **config)
Parameters:

  • needs_tensors – Set to true if the fitter needs to have the tensors in fit() or fit_transform().

  • plr_sigma (float)

  • plr_hidden_1 (int)

  • plr_hidden_2 (int)

  • plr_lr_factor (float)

  • plr_lr_factor_1 (float)

  • plr_lr_factor_2 (float)

  • plr_wd_factor (float)

  • plr_act_name (str)

  • plr_use_densenet (bool)

  • plr_use_cos_bias (bool)

If false, then in fit(ds) or fit_transform(ds), ds.tensors is allowed to be None. :param is_individual: Set to false if fit(ds) deterministically produces a non-trainable layer. (In this case, this Fitter only needs to be called once in k-fold CV on the train+val set.) :param scope_names: List of names to add to the scope (will be present in the names of Variables constructed in this Fitter) :param modified_tensors: List of names of tensors that are modified by this Fitter, e.g., [‘x_cont’]. This is used for the default implementation of get_n_forward(), which is used to get a RAM estimate for the forward pass. The default RAM estimate is simply the size of all modified tensors.

forward_tensor_infos(tensor_infos)

Should be overridden if the fitter changes the tensor shapes. :param tensor_infos: Tensor infos (for shapes and category sizes). :return: Transformed tensor infos.

Parameters:

tensor_infos (Dict[str, TensorInfo])

Return type:

Dict[str, TensorInfo]

get_n_forward(tensor_infos)

Should be overridden if the fitter does more than just one operation. :param tensor_infos: Ingoing tensor infos. :return: Should return the number of bytes used in the forward pass per batch element

Parameters:

tensor_infos (Dict[str, TensorInfo])

Return type:

int

get_n_params(tensor_infos)

Should be overridden if the fitter produces layers with trainable parameters. :param tensor_infos: Tensor infos. :return: Returns the number of parameters of the fitted layer for the given tensor_infos.

Parameters:

tensor_infos (Dict[str, TensorInfo])

Return type:

int

class pytabkit.models.nn_models.nn.PLREmbeddingsLayer

Bases: Layer

__init__(fitter, weight_1, weight_2, bias_2, plr_act_name, plr_use_densenet=False)

Constructor. Puts the layer in eval mode, since it might be used inside the fit_transform() of the Fitter. The parameters provide different opportunities to specify a default implementation for forward_tensor_infos(). The default implementation is: ```

if self.fitter is not None:

return self.fitter.forward_tensor_infos(tensor_infos)

return utils.update_dict(tensor_infos, self.new_tensor_infos, remove_keys=self.remove_keys)

```

Parameters:
forward_cont(x)
class pytabkit.models.nn_models.nn.PLREmbeddingsLayerCosBias

Bases: Layer

__init__(fitter, weight_1, bias_1, weight_2, bias_2, plr_act_name, plr_use_densenet=False)

Constructor. Puts the layer in eval mode, since it might be used inside the fit_transform() of the Fitter. The parameters provide different opportunities to specify a default implementation for forward_tensor_infos(). The default implementation is: ```

if self.fitter is not None:

return self.fitter.forward_tensor_infos(tensor_infos)

return utils.update_dict(tensor_infos, self.new_tensor_infos, remove_keys=self.remove_keys)

```

Parameters:
forward_cont(x)
class pytabkit.models.nn_models.nn.PeriodicEmbeddingsFactory

Bases: Fitter, FitterFactory

__init__(periodic_emb_sigma=1.0, periodic_emb_dim=8, periodic_emb_lr_factor=1.0, periodic_emb_wd_factor=1.0, periodic_emb_only_cos=False, periodic_emb_densenet=False, **config)
Parameters:

  • needs_tensors – Set to true if the fitter needs to have the tensors in fit() or fit_transform().

  • periodic_emb_sigma (float)

  • periodic_emb_dim (int)

  • periodic_emb_lr_factor (float)

  • periodic_emb_wd_factor (float)

  • periodic_emb_only_cos (bool)

  • periodic_emb_densenet (bool)

If false, then in fit(ds) or fit_transform(ds), ds.tensors is allowed to be None. :param is_individual: Set to false if fit(ds) deterministically produces a non-trainable layer. (In this case, this Fitter only needs to be called once in k-fold CV on the train+val set.) :param scope_names: List of names to add to the scope (will be present in the names of Variables constructed in this Fitter) :param modified_tensors: List of names of tensors that are modified by this Fitter, e.g., [‘x_cont’]. This is used for the default implementation of get_n_forward(), which is used to get a RAM estimate for the forward pass. The default RAM estimate is simply the size of all modified tensors.

forward_tensor_infos(tensor_infos)

Should be overridden if the fitter changes the tensor shapes. :param tensor_infos: Tensor infos (for shapes and category sizes). :return: Transformed tensor infos.

Parameters:

tensor_infos (Dict[str, TensorInfo])

Return type:

Dict[str, TensorInfo]

get_n_forward(tensor_infos)

Should be overridden if the fitter does more than just one operation. :param tensor_infos: Ingoing tensor infos. :return: Should return the number of bytes used in the forward pass per batch element

Parameters:

tensor_infos (Dict[str, TensorInfo])

Return type:

int

get_n_params(tensor_infos)

Should be overridden if the fitter produces layers with trainable parameters. :param tensor_infos: Tensor infos. :return: Returns the number of parameters of the fitted layer for the given tensor_infos.

Parameters:

tensor_infos (Dict[str, TensorInfo])

Return type:

int

class pytabkit.models.nn_models.nn.PeriodicEmbeddingsLayerSinCos

Bases: Layer

__init__(fitter, weight, periodic_emb_densenet)

Constructor. Puts the layer in eval mode, since it might be used inside the fit_transform() of the Fitter. The parameters provide different opportunities to specify a default implementation for forward_tensor_infos(). The default implementation is: ```

if self.fitter is not None:

return self.fitter.forward_tensor_infos(tensor_infos)

return utils.update_dict(tensor_infos, self.new_tensor_infos, remove_keys=self.remove_keys)

```

Parameters:
forward_cont(x)
class pytabkit.models.nn_models.nn.RFFeatureImportanceFactory

Bases: Fitter, FitterFactory

__init__()
Parameters:

needs_tensors – Set to true if the fitter needs to have the tensors in fit() or fit_transform().

If false, then in fit(ds) or fit_transform(ds), ds.tensors is allowed to be None. :param is_individual: Set to false if fit(ds) deterministically produces a non-trainable layer. (In this case, this Fitter only needs to be called once in k-fold CV on the train+val set.) :param scope_names: List of names to add to the scope (will be present in the names of Variables constructed in this Fitter) :param modified_tensors: List of names of tensors that are modified by this Fitter, e.g., [‘x_cont’]. This is used for the default implementation of get_n_forward(), which is used to get a RAM estimate for the forward pass. The default RAM estimate is simply the size of all modified tensors.

class pytabkit.models.nn_models.nn.ScaleFactory

Bases: FitterFactory

__init__(**config)
class pytabkit.models.nn_models.nn.ScaleFitter

Bases: Fitter

__init__(**config)
Parameters:

needs_tensors – Set to true if the fitter needs to have the tensors in fit() or fit_transform().

If false, then in fit(ds) or fit_transform(ds), ds.tensors is allowed to be None. :param is_individual: Set to false if fit(ds) deterministically produces a non-trainable layer. (In this case, this Fitter only needs to be called once in k-fold CV on the train+val set.) :param scope_names: List of names to add to the scope (will be present in the names of Variables constructed in this Fitter) :param modified_tensors: List of names of tensors that are modified by this Fitter, e.g., [‘x_cont’]. This is used for the default implementation of get_n_forward(), which is used to get a RAM estimate for the forward pass. The default RAM estimate is simply the size of all modified tensors.

get_n_params(tensor_infos)

Should be overridden if the fitter produces layers with trainable parameters. :param tensor_infos: Tensor infos. :return: Returns the number of parameters of the fitted layer for the given tensor_infos.

Parameters:

tensor_infos (Dict[str, TensorInfo])

Return type:

int

class pytabkit.models.nn_models.nn.StochasticGateFactory

Bases: Fitter, FitterFactory

__init__()
Parameters:

needs_tensors – Set to true if the fitter needs to have the tensors in fit() or fit_transform().

If false, then in fit(ds) or fit_transform(ds), ds.tensors is allowed to be None. :param is_individual: Set to false if fit(ds) deterministically produces a non-trainable layer. (In this case, this Fitter only needs to be called once in k-fold CV on the train+val set.) :param scope_names: List of names to add to the scope (will be present in the names of Variables constructed in this Fitter) :param modified_tensors: List of names of tensors that are modified by this Fitter, e.g., [‘x_cont’]. This is used for the default implementation of get_n_forward(), which is used to get a RAM estimate for the forward pass. The default RAM estimate is simply the size of all modified tensors.

get_n_forward(tensor_infos)

Should be overridden if the fitter does more than just one operation. :param tensor_infos: Ingoing tensor infos. :return: Should return the number of bytes used in the forward pass per batch element

Parameters:

tensor_infos (Dict[str, TensorInfo])

Return type:

int

get_n_params(tensor_infos)

Should be overridden if the fitter produces layers with trainable parameters. :param tensor_infos: Tensor infos. :return: Returns the number of parameters of the fitted layer for the given tensor_infos.

Parameters:

tensor_infos (Dict[str, TensorInfo])

Return type:

int

class pytabkit.models.nn_models.nn.StochasticGateLayer

Bases: Layer

__init__(mu)

Constructor. Puts the layer in eval mode, since it might be used inside the fit_transform() of the Fitter. The parameters provide different opportunities to specify a default implementation for forward_tensor_infos(). The default implementation is: ```

if self.fitter is not None:

return self.fitter.forward_tensor_infos(tensor_infos)

return utils.update_dict(tensor_infos, self.new_tensor_infos, remove_keys=self.remove_keys)

```

Parameters:

mu (Variable)

forward_cont(x)
class pytabkit.models.nn_models.nn.StochasticLabelNoiseFactory

Bases: FitterFactory

class pytabkit.models.nn_models.nn.StochasticLabelNoiseFitter

Bases: Fitter

__init__()
Parameters:

needs_tensors – Set to true if the fitter needs to have the tensors in fit() or fit_transform().

If false, then in fit(ds) or fit_transform(ds), ds.tensors is allowed to be None. :param is_individual: Set to false if fit(ds) deterministically produces a non-trainable layer. (In this case, this Fitter only needs to be called once in k-fold CV on the train+val set.) :param scope_names: List of names to add to the scope (will be present in the names of Variables constructed in this Fitter) :param modified_tensors: List of names of tensors that are modified by this Fitter, e.g., [‘x_cont’]. This is used for the default implementation of get_n_forward(), which is used to get a RAM estimate for the forward pass. The default RAM estimate is simply the size of all modified tensors.

class pytabkit.models.nn_models.nn.StochasticLabelNoiseLayer

Bases: Layer

__init__()

Constructor. Puts the layer in eval mode, since it might be used inside the fit_transform() of the Fitter. The parameters provide different opportunities to specify a default implementation for forward_tensor_infos(). The default implementation is: ```

if self.fitter is not None:

return self.fitter.forward_tensor_infos(tensor_infos)

return utils.update_dict(tensor_infos, self.new_tensor_infos, remove_keys=self.remove_keys)

```

forward_tensors(tensors)

Transforms the given tensors. :param tensors: :return:

class pytabkit.models.nn_models.nn.SubtractionLayer

Bases: Layer

__init__(layer1, layer2)

Constructor. Puts the layer in eval mode, since it might be used inside the fit_transform() of the Fitter. The parameters provide different opportunities to specify a default implementation for forward_tensor_infos(). The default implementation is: ```

if self.fitter is not None:

return self.fitter.forward_tensor_infos(tensor_infos)

return utils.update_dict(tensor_infos, self.new_tensor_infos, remove_keys=self.remove_keys)

```

Parameters:
forward_tensor_infos(tensor_infos)

Override this method if the information from constructor is not sufficient. :param tensor_infos: Tensor infos (shapes etc.) :return: Transformed tensor infos.

forward_tensors(tensors)

Transforms the given tensors. :param tensors: :return:

class pytabkit.models.nn_models.nn.ToSoftLabelFitter

Bases: Fitter

__init__()
Parameters:

needs_tensors – Set to true if the fitter needs to have the tensors in fit() or fit_transform().

If false, then in fit(ds) or fit_transform(ds), ds.tensors is allowed to be None. :param is_individual: Set to false if fit(ds) deterministically produces a non-trainable layer. (In this case, this Fitter only needs to be called once in k-fold CV on the train+val set.) :param scope_names: List of names to add to the scope (will be present in the names of Variables constructed in this Fitter) :param modified_tensors: List of names of tensors that are modified by this Fitter, e.g., [‘x_cont’]. This is used for the default implementation of get_n_forward(), which is used to get a RAM estimate for the forward pass. The default RAM estimate is simply the size of all modified tensors.

forward_tensor_infos(tensor_infos)

Should be overridden if the fitter changes the tensor shapes. :param tensor_infos: Tensor infos (for shapes and category sizes). :return: Transformed tensor infos.

class pytabkit.models.nn_models.nn.ToSoftLabelLayer

Bases: Layer

__init__(y_tensor_info, fitter)

Constructor. Puts the layer in eval mode, since it might be used inside the fit_transform() of the Fitter. The parameters provide different opportunities to specify a default implementation for forward_tensor_infos(). The default implementation is: ```

if self.fitter is not None:

return self.fitter.forward_tensor_infos(tensor_infos)

return utils.update_dict(tensor_infos, self.new_tensor_infos, remove_keys=self.remove_keys)

```

Parameters:

fitter (Fitter)

forward_tensors(tensors)

Transforms the given tensors. :param tensors: :return:

class pytabkit.models.nn_models.nn.WeightFitter

Bases: Fitter

__init__(out_features, **config)
Parameters:

needs_tensors – Set to true if the fitter needs to have the tensors in fit() or fit_transform().

If false, then in fit(ds) or fit_transform(ds), ds.tensors is allowed to be None. :param is_individual: Set to false if fit(ds) deterministically produces a non-trainable layer. (In this case, this Fitter only needs to be called once in k-fold CV on the train+val set.) :param scope_names: List of names to add to the scope (will be present in the names of Variables constructed in this Fitter) :param modified_tensors: List of names of tensors that are modified by this Fitter, e.g., [‘x_cont’]. This is used for the default implementation of get_n_forward(), which is used to get a RAM estimate for the forward pass. The default RAM estimate is simply the size of all modified tensors.

forward_tensor_infos(tensor_infos)

Should be overridden if the fitter changes the tensor shapes. :param tensor_infos: Tensor infos (for shapes and category sizes). :return: Transformed tensor infos.

get_n_params(tensor_infos)

Should be overridden if the fitter produces layers with trainable parameters. :param tensor_infos: Tensor infos. :return: Returns the number of parameters of the fitted layer for the given tensor_infos.

Parameters:

tensor_infos (Dict[str, TensorInfo])

Return type:

int

pytabkit.models.nn_models.pipeline module

class pytabkit.models.nn_models.pipeline.CircleCodingFactory

Bases: Fitter, FitterFactory

__init__(circle_coding_scale=1.0, **config)
Parameters:

needs_tensors – Set to true if the fitter needs to have the tensors in fit() or fit_transform().

If false, then in fit(ds) or fit_transform(ds), ds.tensors is allowed to be None. :param is_individual: Set to false if fit(ds) deterministically produces a non-trainable layer. (In this case, this Fitter only needs to be called once in k-fold CV on the train+val set.) :param scope_names: List of names to add to the scope (will be present in the names of Variables constructed in this Fitter) :param modified_tensors: List of names of tensors that are modified by this Fitter, e.g., [‘x_cont’]. This is used for the default implementation of get_n_forward(), which is used to get a RAM estimate for the forward pass. The default RAM estimate is simply the size of all modified tensors.

forward_tensor_infos(tensor_infos)

Should be overridden if the fitter changes the tensor shapes. :param tensor_infos: Tensor infos (for shapes and category sizes). :return: Transformed tensor infos.

class pytabkit.models.nn_models.pipeline.CircleCodingLayer

Bases: Layer

__init__(scale, fitter)

Constructor. Puts the layer in eval mode, since it might be used inside the fit_transform() of the Fitter. The parameters provide different opportunities to specify a default implementation for forward_tensor_infos(). The default implementation is: ```

if self.fitter is not None:

return self.fitter.forward_tensor_infos(tensor_infos)

return utils.update_dict(tensor_infos, self.new_tensor_infos, remove_keys=self.remove_keys)

```

Parameters:

  • scale (float)

  • fitter (Fitter)

forward_cont(x)
class pytabkit.models.nn_models.pipeline.GlobalScaleNormalizeFactory

Bases: Fitter, FitterFactory

__init__(global_scale_factor=1.0, **config)
Parameters:

needs_tensors – Set to true if the fitter needs to have the tensors in fit() or fit_transform().

If false, then in fit(ds) or fit_transform(ds), ds.tensors is allowed to be None. :param is_individual: Set to false if fit(ds) deterministically produces a non-trainable layer. (In this case, this Fitter only needs to be called once in k-fold CV on the train+val set.) :param scope_names: List of names to add to the scope (will be present in the names of Variables constructed in this Fitter) :param modified_tensors: List of names of tensors that are modified by this Fitter, e.g., [‘x_cont’]. This is used for the default implementation of get_n_forward(), which is used to get a RAM estimate for the forward pass. The default RAM estimate is simply the size of all modified tensors.

get_n_params(tensor_infos)

Should be overridden if the fitter produces layers with trainable parameters. :param tensor_infos: Tensor infos. :return: Returns the number of parameters of the fitted layer for the given tensor_infos.

Parameters:

tensor_infos (Dict[str, TensorInfo])

Return type:

int

class pytabkit.models.nn_models.pipeline.L1NormalizeFactory

Bases: Fitter, FitterFactory

__init__(trainable=False, eps=1e-08, **config)
Parameters:

needs_tensors – Set to true if the fitter needs to have the tensors in fit() or fit_transform().

If false, then in fit(ds) or fit_transform(ds), ds.tensors is allowed to be None. :param is_individual: Set to false if fit(ds) deterministically produces a non-trainable layer. (In this case, this Fitter only needs to be called once in k-fold CV on the train+val set.) :param scope_names: List of names to add to the scope (will be present in the names of Variables constructed in this Fitter) :param modified_tensors: List of names of tensors that are modified by this Fitter, e.g., [‘x_cont’]. This is used for the default implementation of get_n_forward(), which is used to get a RAM estimate for the forward pass. The default RAM estimate is simply the size of all modified tensors.

get_n_params(tensor_infos)

Should be overridden if the fitter produces layers with trainable parameters. :param tensor_infos: Tensor infos. :return: Returns the number of parameters of the fitted layer for the given tensor_infos.

Parameters:

tensor_infos (Dict[str, TensorInfo])

Return type:

int

class pytabkit.models.nn_models.pipeline.L2NormalizeFactory

Bases: Fitter, FitterFactory

__init__(trainable=False, l2_normalize_eps=1e-08, **config)
Parameters:

needs_tensors – Set to true if the fitter needs to have the tensors in fit() or fit_transform().

If false, then in fit(ds) or fit_transform(ds), ds.tensors is allowed to be None. :param is_individual: Set to false if fit(ds) deterministically produces a non-trainable layer. (In this case, this Fitter only needs to be called once in k-fold CV on the train+val set.) :param scope_names: List of names to add to the scope (will be present in the names of Variables constructed in this Fitter) :param modified_tensors: List of names of tensors that are modified by this Fitter, e.g., [‘x_cont’]. This is used for the default implementation of get_n_forward(), which is used to get a RAM estimate for the forward pass. The default RAM estimate is simply the size of all modified tensors.

get_n_params(tensor_infos)

Should be overridden if the fitter produces layers with trainable parameters. :param tensor_infos: Tensor infos. :return: Returns the number of parameters of the fitted layer for the given tensor_infos.

Parameters:

tensor_infos (Dict[str, TensorInfo])

Return type:

int

class pytabkit.models.nn_models.pipeline.MeanCenterFactory

Bases: Fitter, FitterFactory

__init__(trainable=False, **config)
Parameters:

needs_tensors – Set to true if the fitter needs to have the tensors in fit() or fit_transform().

If false, then in fit(ds) or fit_transform(ds), ds.tensors is allowed to be None. :param is_individual: Set to false if fit(ds) deterministically produces a non-trainable layer. (In this case, this Fitter only needs to be called once in k-fold CV on the train+val set.) :param scope_names: List of names to add to the scope (will be present in the names of Variables constructed in this Fitter) :param modified_tensors: List of names of tensors that are modified by this Fitter, e.g., [‘x_cont’]. This is used for the default implementation of get_n_forward(), which is used to get a RAM estimate for the forward pass. The default RAM estimate is simply the size of all modified tensors.

get_n_params(tensor_infos)

Should be overridden if the fitter produces layers with trainable parameters. :param tensor_infos: Tensor infos. :return: Returns the number of parameters of the fitted layer for the given tensor_infos.

Parameters:

tensor_infos (Dict[str, TensorInfo])

Return type:

int

class pytabkit.models.nn_models.pipeline.MeanReplaceMissingContFactory

Bases: Fitter, FitterFactory

__init__(trainable=False, **config)
Parameters:

needs_tensors – Set to true if the fitter needs to have the tensors in fit() or fit_transform().

If false, then in fit(ds) or fit_transform(ds), ds.tensors is allowed to be None. :param is_individual: Set to false if fit(ds) deterministically produces a non-trainable layer. (In this case, this Fitter only needs to be called once in k-fold CV on the train+val set.) :param scope_names: List of names to add to the scope (will be present in the names of Variables constructed in this Fitter) :param modified_tensors: List of names of tensors that are modified by this Fitter, e.g., [‘x_cont’]. This is used for the default implementation of get_n_forward(), which is used to get a RAM estimate for the forward pass. The default RAM estimate is simply the size of all modified tensors.

get_n_params(tensor_infos)

Should be overridden if the fitter produces layers with trainable parameters. :param tensor_infos: Tensor infos. :return: Returns the number of parameters of the fitted layer for the given tensor_infos.

Parameters:

tensor_infos (Dict[str, TensorInfo])

Return type:

int

class pytabkit.models.nn_models.pipeline.MedianCenterFactory

Bases: Fitter, FitterFactory

__init__(median_center_trainable=False, **config)
Parameters:

needs_tensors – Set to true if the fitter needs to have the tensors in fit() or fit_transform().

If false, then in fit(ds) or fit_transform(ds), ds.tensors is allowed to be None. :param is_individual: Set to false if fit(ds) deterministically produces a non-trainable layer. (In this case, this Fitter only needs to be called once in k-fold CV on the train+val set.) :param scope_names: List of names to add to the scope (will be present in the names of Variables constructed in this Fitter) :param modified_tensors: List of names of tensors that are modified by this Fitter, e.g., [‘x_cont’]. This is used for the default implementation of get_n_forward(), which is used to get a RAM estimate for the forward pass. The default RAM estimate is simply the size of all modified tensors.

get_n_params(tensor_infos)

Should be overridden if the fitter produces layers with trainable parameters. :param tensor_infos: Tensor infos. :return: Returns the number of parameters of the fitted layer for the given tensor_infos.

Parameters:

tensor_infos (Dict[str, TensorInfo])

Return type:

int

class pytabkit.models.nn_models.pipeline.MinMaxScaleFactory

Bases: Fitter, FitterFactory

__init__(trainable=False, eps=1e-08, **config)
Parameters:

needs_tensors – Set to true if the fitter needs to have the tensors in fit() or fit_transform().

If false, then in fit(ds) or fit_transform(ds), ds.tensors is allowed to be None. :param is_individual: Set to false if fit(ds) deterministically produces a non-trainable layer. (In this case, this Fitter only needs to be called once in k-fold CV on the train+val set.) :param scope_names: List of names to add to the scope (will be present in the names of Variables constructed in this Fitter) :param modified_tensors: List of names of tensors that are modified by this Fitter, e.g., [‘x_cont’]. This is used for the default implementation of get_n_forward(), which is used to get a RAM estimate for the forward pass. The default RAM estimate is simply the size of all modified tensors.

get_n_params(tensor_infos)

Should be overridden if the fitter produces layers with trainable parameters. :param tensor_infos: Tensor infos. :return: Returns the number of parameters of the fitted layer for the given tensor_infos.

Parameters:

tensor_infos (Dict[str, TensorInfo])

Return type:

int

class pytabkit.models.nn_models.pipeline.ReplaceMissingContLayer

Bases: Layer

__init__(means)

Constructor. Puts the layer in eval mode, since it might be used inside the fit_transform() of the Fitter. The parameters provide different opportunities to specify a default implementation for forward_tensor_infos(). The default implementation is: ```

if self.fitter is not None:

return self.fitter.forward_tensor_infos(tensor_infos)

return utils.update_dict(tensor_infos, self.new_tensor_infos, remove_keys=self.remove_keys)

```

Parameters:

means (Variable)

forward_cont(x)
class pytabkit.models.nn_models.pipeline.RobustScaleFactory

Bases: Fitter, FitterFactory

__init__(robust_scale_trainable=False, robust_scale_eps=1e-30, **config)
Parameters:

needs_tensors – Set to true if the fitter needs to have the tensors in fit() or fit_transform().

If false, then in fit(ds) or fit_transform(ds), ds.tensors is allowed to be None. :param is_individual: Set to false if fit(ds) deterministically produces a non-trainable layer. (In this case, this Fitter only needs to be called once in k-fold CV on the train+val set.) :param scope_names: List of names to add to the scope (will be present in the names of Variables constructed in this Fitter) :param modified_tensors: List of names of tensors that are modified by this Fitter, e.g., [‘x_cont’]. This is used for the default implementation of get_n_forward(), which is used to get a RAM estimate for the forward pass. The default RAM estimate is simply the size of all modified tensors.

get_n_params(tensor_infos)

Should be overridden if the fitter produces layers with trainable parameters. :param tensor_infos: Tensor infos. :return: Returns the number of parameters of the fitted layer for the given tensor_infos.

Parameters:

tensor_infos (Dict[str, TensorInfo])

Return type:

int

class pytabkit.models.nn_models.pipeline.RobustScaleV2Factory

Bases: Fitter, FitterFactory

__init__(robust_scale_trainable=False, **config)
Parameters:

needs_tensors – Set to true if the fitter needs to have the tensors in fit() or fit_transform().

If false, then in fit(ds) or fit_transform(ds), ds.tensors is allowed to be None. :param is_individual: Set to false if fit(ds) deterministically produces a non-trainable layer. (In this case, this Fitter only needs to be called once in k-fold CV on the train+val set.) :param scope_names: List of names to add to the scope (will be present in the names of Variables constructed in this Fitter) :param modified_tensors: List of names of tensors that are modified by this Fitter, e.g., [‘x_cont’]. This is used for the default implementation of get_n_forward(), which is used to get a RAM estimate for the forward pass. The default RAM estimate is simply the size of all modified tensors.

get_n_params(tensor_infos)

Should be overridden if the fitter produces layers with trainable parameters. :param tensor_infos: Tensor infos. :return: Returns the number of parameters of the fitted layer for the given tensor_infos.

Parameters:

tensor_infos (Dict[str, TensorInfo])

Return type:

int

class pytabkit.models.nn_models.pipeline.SklearnTransformFactory

Bases: Fitter, FitterFactory

__init__(tfm, **config)
Parameters:

  • needs_tensors – Set to true if the fitter needs to have the tensors in fit() or fit_transform().

  • tfm (BaseEstimator)

If false, then in fit(ds) or fit_transform(ds), ds.tensors is allowed to be None. :param is_individual: Set to false if fit(ds) deterministically produces a non-trainable layer. (In this case, this Fitter only needs to be called once in k-fold CV on the train+val set.) :param scope_names: List of names to add to the scope (will be present in the names of Variables constructed in this Fitter) :param modified_tensors: List of names of tensors that are modified by this Fitter, e.g., [‘x_cont’]. This is used for the default implementation of get_n_forward(), which is used to get a RAM estimate for the forward pass. The default RAM estimate is simply the size of all modified tensors.

class pytabkit.models.nn_models.pipeline.SklearnTransformLayer

Bases: Layer

__init__(tfms, fitter)

Constructor. Puts the layer in eval mode, since it might be used inside the fit_transform() of the Fitter. The parameters provide different opportunities to specify a default implementation for forward_tensor_infos(). The default implementation is: ```

if self.fitter is not None:

return self.fitter.forward_tensor_infos(tensor_infos)

return utils.update_dict(tensor_infos, self.new_tensor_infos, remove_keys=self.remove_keys)

```

Parameters:

  • tfms (BaseEstimator | List)

  • fitter (Fitter)

forward_cont(x)
class pytabkit.models.nn_models.pipeline.ThermometerCodingFactory

Bases: Fitter, FitterFactory

__init__(tc_low=-1.0, tc_high=1.0, tc_num=3, tc_scale=1.0, **config)
Parameters:

needs_tensors – Set to true if the fitter needs to have the tensors in fit() or fit_transform().

If false, then in fit(ds) or fit_transform(ds), ds.tensors is allowed to be None. :param is_individual: Set to false if fit(ds) deterministically produces a non-trainable layer. (In this case, this Fitter only needs to be called once in k-fold CV on the train+val set.) :param scope_names: List of names to add to the scope (will be present in the names of Variables constructed in this Fitter) :param modified_tensors: List of names of tensors that are modified by this Fitter, e.g., [‘x_cont’]. This is used for the default implementation of get_n_forward(), which is used to get a RAM estimate for the forward pass. The default RAM estimate is simply the size of all modified tensors.

forward_tensor_infos(tensor_infos)

Should be overridden if the fitter changes the tensor shapes. :param tensor_infos: Tensor infos (for shapes and category sizes). :return: Transformed tensor infos.

get_n_params(tensor_infos)

Should be overridden if the fitter produces layers with trainable parameters. :param tensor_infos: Tensor infos. :return: Returns the number of parameters of the fitted layer for the given tensor_infos.

Parameters:

tensor_infos (Dict[str, TensorInfo])

Return type:

int

class pytabkit.models.nn_models.pipeline.ThermometerCodingLayer

Bases: Layer

__init__(centers, scale, fitter)

Constructor. Puts the layer in eval mode, since it might be used inside the fit_transform() of the Fitter. The parameters provide different opportunities to specify a default implementation for forward_tensor_infos(). The default implementation is: ```

if self.fitter is not None:

return self.fitter.forward_tensor_infos(tensor_infos)

return utils.update_dict(tensor_infos, self.new_tensor_infos, remove_keys=self.remove_keys)

```

Parameters:
forward_cont(x)
pytabkit.models.nn_models.pipeline.apply_tfms_rec(tfms, x)
Parameters:

  • tfms (BaseEstimator | List)

  • x (Tensor)

pytabkit.models.nn_models.rtdl_num_embeddings module

On Embeddings for Numerical Features in Tabular Deep Learning.

class pytabkit.models.nn_models.rtdl_num_embeddings.LinearEmbeddings

Bases: Module

Linear embeddings for continuous features.

Shape

  • Input: (*, n_features)

  • Output: (*, n_features, d_embedding)

Examples

>>> batch_size = 2
>>> n_cont_features = 3
>>> x = torch.randn(batch_size, n_cont_features)
>>> d_embedding = 4
>>> m = LinearEmbeddings(n_cont_features, d_embedding)
>>> m(x).shape
torch.Size([2, 3, 4])
__init__(n_features, d_embedding)
Args:

n_features: the number of continuous features. d_embedding: the embedding size.

Parameters:

  • n_features (int)

  • d_embedding (int)

Return type:

None

forward(x)

Do the forward pass.

Parameters:

x (Tensor)

Return type:

Tensor

reset_parameters()
Return type:

None

class pytabkit.models.nn_models.rtdl_num_embeddings.LinearReLUEmbeddings

Bases: Sequential

Simple non-linear embeddings for continuous features.

Shape

  • Input: (*, n_features)

  • Output: (*, n_features, d_embedding)

Examples

>>> batch_size = 2
>>> n_cont_features = 3
>>> x = torch.randn(batch_size, n_cont_features)
>>>
>>> d_embedding = 32
>>> m = LinearReLUEmbeddings(n_cont_features, d_embedding)
>>> m(x).shape
torch.Size([2, 3, 32])
__init__(n_features, d_embedding=32)

Initialize internal Module state, shared by both nn.Module and ScriptModule.

Parameters:

  • n_features (int)

  • d_embedding (int)

Return type:

None

class pytabkit.models.nn_models.rtdl_num_embeddings.PeriodicEmbeddings

Bases: Module

Embeddings for continuous features based on periodic activations.

See README for details.

Shape

  • Input: (*, n_features)

  • Output: (*, n_features, d_embedding)

Examples

>>> batch_size = 2
>>> n_cont_features = 3
>>> x = torch.randn(batch_size, n_cont_features)
>>>
>>> d_embedding = 24
>>> m = PeriodicEmbeddings(n_cont_features, d_embedding, lite=False)
>>> m(x).shape
torch.Size([2, 3, 24])
>>>
>>> m = PeriodicEmbeddings(n_cont_features, d_embedding, lite=True)
>>> m(x).shape
torch.Size([2, 3, 24])
>>>
>>> # PL embeddings.
>>> m = PeriodicEmbeddings(n_cont_features, d_embedding=8, activation=False, lite=False)
>>> m(x).shape
torch.Size([2, 3, 8])
__init__(n_features, d_embedding=24, *, n_frequencies=48, frequency_init_scale=0.01, activation=True, lite)
Args:

n_features: the number of features. d_embedding: the embedding size. n_frequencies: the number of frequencies for each feature.

(denoted as “k” in Section 3.3 in the paper).

frequency_init_scale: the initialization scale for the first linear layer

(denoted as “sigma” in Section 3.3 in the paper). This is an important hyperparameter, see README for details.

activation: if False, the ReLU activation is not applied.

Must be True if lite=True.

lite: if True, the outer linear layer is shared between all features.

See README for details.

Parameters:

  • n_features (int)

  • d_embedding (int)

  • n_frequencies (int)

  • frequency_init_scale (float)

  • activation (bool)

  • lite (bool)

Return type:

None

forward(x)

Do the forward pass.

Parameters:

x (Tensor)

Return type:

Tensor

class pytabkit.models.nn_models.rtdl_num_embeddings.PiecewiseLinearEmbeddings

Bases: Module

Piecewise-linear embeddings.

Shape

  • Input: (batch_size, n_features)

  • Output: (batch_size, n_features, d_embedding)

__init__(bins, d_embedding, *, activation, version=None)
Args:

bins: the bins computed by compute_bins. d_embedding: the embedding size. activation: if True, the ReLU activation is additionally applied in the end. version: the preset for various implementation details, such as

parametrization and initialization. See README for details.

Parameters:

  • bins (list[Tensor])

  • d_embedding (int)

  • activation (bool)

  • version (Literal[None, 'A', 'B'] | None)

Return type:

None

forward(x)

Do the forward pass.

Parameters:

x (Tensor)

Return type:

Tensor

class pytabkit.models.nn_models.rtdl_num_embeddings.PiecewiseLinearEncoding

Bases: Module

Piecewise-linear encoding.

See README for detailed explanation.

Shape

  • Input: (*, n_features)

  • Output: (*, total_n_bins), where total_n_bins is the total number of bins for all features: total_n_bins = sum(len(b) - 1 for b in bins).

Technically, the output of this module is the flattened output of _PiecewiseLinearEncoding with all “padding” values removed.

__init__(bins)
Args:

bins: the bins computed by compute_bins.

Parameters:

bins (list[Tensor])

Return type:

None

forward(x)

Do the forward pass.

Parameters:

x (Tensor)

Return type:

Tensor

pytabkit.models.nn_models.rtdl_num_embeddings.compute_bins(X, n_bins=48, *, tree_kwargs=None, y=None, regression=None, verbose=False)

Compute the bin boundaries for PiecewiseLinearEncoding and PiecewiseLinearEmbeddings.

Usage

Compute bins using quantiles (Section 3.2.1 in the paper):

>>> X_train = torch.randn(10000, 2)
>>> bins = compute_bins(X_train)

Compute bins using decision trees (Section 3.2.2 in the paper):

>>> X_train = torch.randn(10000, 2)
>>> y_train = torch.randn(len(X_train))
>>> bins = compute_bins(
...     X_train,
...     y=y_train,
...     regression=True,
...     tree_kwargs={'min_samples_leaf': 64, 'min_impurity_decrease': 1e-4},
... )
Args:

X: the training features. n_bins: the number of bins. tree_kwargs: keyword arguments for sklearn.tree.DecisionTreeRegressor

(if regression=True) or sklearn.tree.DecisionTreeClassifier (if regression=False). NOTE: requires scikit-learn>=1.0,>2 to be installed.

y: the training labels (must be provided if tree is not None). regression: whether the labels are regression labels

(must be provided if tree is not None).

verbose: if True and tree_kwargs is not None, than tqdm

(must be installed) will report the progress while fitting trees.

Returns:

A list of bin edges for all features. For one feature:

  • the maximum possible number of bin edges is n_bins + 1.

  • the minimum possible number of bin edges is 1.

Parameters:

  • X (Tensor)

  • n_bins (int)

  • tree_kwargs (dict[str, Any] | None)

  • y (Tensor | None)

  • regression (bool | None)

  • verbose (bool)

Return type:

list[Tensor]

pytabkit.models.nn_models.rtdl_resnet module

class pytabkit.models.nn_models.rtdl_resnet.EarlyStoppingCustomError

Bases: EarlyStopping

on_epoch_end(net, **kwargs)

Called at the end of each epoch.

class pytabkit.models.nn_models.rtdl_resnet.FT_Transformer

Bases: Module

Transformer.

References: - https://pytorch.org/docs/stable/generated/torch.nn.Transformer.html - https://github.com/facebookresearch/pytext/tree/master/pytext/models/representations/transformer - https://github.com/pytorch/fairseq/blob/1bba712622b8ae4efb3eb793a8a40da386fe11d0/examples/linformer/linformer_src/modules/multihead_linear_attention.py#L19

__init__(*, d_in, categories, token_bias, n_layers, d_token, n_heads, d_ffn_factor, attention_dropout, ffn_dropout, residual_dropout, activation, prenormalization, initialization, kv_compression, kv_compression_sharing, d_out, regression, categorical_indicator)

Initialize internal Module state, shared by both nn.Module and ScriptModule.

Parameters:

  • d_in (int)

  • categories (List[int] | None)

  • token_bias (bool)

  • n_layers (int)

  • d_token (int)

  • n_heads (int)

  • d_ffn_factor (float)

  • attention_dropout (float)

  • ffn_dropout (float)

  • residual_dropout (float)

  • activation (str)

  • prenormalization (bool)

  • initialization (str)

  • kv_compression (float | None)

  • kv_compression_sharing (str | None)

  • d_out (int)

  • regression (bool)

Return type:

None

forward(x)

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

Return type:

Tensor

class pytabkit.models.nn_models.rtdl_resnet.InputShapeSetterResnet

Bases: Callback

__init__(regression=False, batch_size=None, cat_features=None, categories=None)
on_train_begin(net, X, y)

Called at the beginning of training.

class pytabkit.models.nn_models.rtdl_resnet.LearningRateLogger

Bases: Callback

on_epoch_begin(net, dataset_train=None, dataset_valid=None, **kwargs)

Called at the beginning of each epoch.

class pytabkit.models.nn_models.rtdl_resnet.MultiheadAttention

Bases: Module

__init__(d, n_heads, dropout, initialization)

Initialize internal Module state, shared by both nn.Module and ScriptModule.

Parameters:

  • d (int)

  • n_heads (int)

  • dropout (float)

  • initialization (str)

Return type:

None

forward(x_q, x_kv, key_compression, value_compression)

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

Parameters:

  • x_q (Tensor)

  • x_kv (Tensor)

  • key_compression (Linear | None)

  • value_compression (Linear | None)

Return type:

Tensor

exception pytabkit.models.nn_models.rtdl_resnet.MyCustomError

Bases: Exception

class pytabkit.models.nn_models.rtdl_resnet.NeuralNetClassifierCustomOptim

Bases: NeuralNetClassifierWrapped

initialize_optimizer(triggered_directly=None)

Initialize the model optimizer. If self.optimizer__lr is not set, use self.lr instead.

Parameters

triggered_directly

Deprecated, don’t use it anymore.

set_partial_fit_request(*, classes='$UNCHANGED$')

Configure whether metadata should be requested to be passed to the partial_fit method.

Note that this method is only relevant when this estimator is used as a sub-estimator within a meta-estimator and metadata routing is enabled with enable_metadata_routing=True (see sklearn.set_config()). Please check the User Guide on how the routing mechanism works.

The options for each parameter are:

  • True: metadata is requested, and passed to partial_fit if provided. The request is ignored if metadata is not provided.

  • False: metadata is not requested and the meta-estimator will not pass it to partial_fit.

  • None: metadata is not requested, and the meta-estimator will raise an error if the user provides it.

  • str: metadata should be passed to the meta-estimator with this given alias instead of the original name.

The default (sklearn.utils.metadata_routing.UNCHANGED) retains the existing request. This allows you to change the request for some parameters and not others.

Added in version 1.3.

Parameters

classesstr, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED

Metadata routing for classes parameter in partial_fit.

Returns

selfobject

The updated object.

Parameters:
Return type:

NeuralNetClassifierCustomOptim

set_score_request(*, sample_weight='$UNCHANGED$')

Configure whether metadata should be requested to be passed to the score method.

Note that this method is only relevant when this estimator is used as a sub-estimator within a meta-estimator and metadata routing is enabled with enable_metadata_routing=True (see sklearn.set_config()). Please check the User Guide on how the routing mechanism works.

The options for each parameter are:

  • True: metadata is requested, and passed to score if provided. The request is ignored if metadata is not provided.

  • False: metadata is not requested and the meta-estimator will not pass it to score.

  • None: metadata is not requested, and the meta-estimator will raise an error if the user provides it.

  • str: metadata should be passed to the meta-estimator with this given alias instead of the original name.

The default (sklearn.utils.metadata_routing.UNCHANGED) retains the existing request. This allows you to change the request for some parameters and not others.

Added in version 1.3.

Parameters

sample_weightstr, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED

Metadata routing for sample_weight parameter in score.

Returns

selfobject

The updated object.

Parameters:
Return type:

NeuralNetClassifierCustomOptim

class pytabkit.models.nn_models.rtdl_resnet.NeuralNetClassifierWrapped

Bases: NeuralNetClassifier

__init__(*args, **kwargs)
fit(X, y)

See NeuralNet.fit.

In contrast to NeuralNet.fit, y is non-optional to avoid mistakenly forgetting about y. However, y can be set to None in case it is derived dynamically from X.

get_default_callbacks()
partial_fit(X, y=None, classes=None, **fit_params)

Fit the module.

If the module is initialized, it is not re-initialized, which means that this method should be used if you want to continue training a model (warm start).

Parameters

Xinput data, compatible with skorch.dataset.Dataset

By default, you should be able to pass:

  • numpy arrays

  • torch tensors

  • pandas DataFrame or Series

  • scipy sparse CSR matrices

  • a dictionary of the former three

  • a list/tuple of the former three

  • a Dataset

If this doesn’t work with your data, you have to pass a Dataset that can deal with the data.

ytarget data, compatible with skorch.dataset.Dataset

The same data types as for X are supported. If your X is a Dataset that contains the target, y may be set to None.

classesarray, sahpe (n_classes,)

Solely for sklearn compatibility, currently unused.

**fit_paramsdict

Additional parameters passed to the forward method of the module and to the self.train_split call.

set_categorical_indicator(categorical_indicator)
set_n_classes(n_classes)
set_partial_fit_request(*, classes='$UNCHANGED$')

Configure whether metadata should be requested to be passed to the partial_fit method.

Note that this method is only relevant when this estimator is used as a sub-estimator within a meta-estimator and metadata routing is enabled with enable_metadata_routing=True (see sklearn.set_config()). Please check the User Guide on how the routing mechanism works.

The options for each parameter are:

  • True: metadata is requested, and passed to partial_fit if provided. The request is ignored if metadata is not provided.

  • False: metadata is not requested and the meta-estimator will not pass it to partial_fit.

  • None: metadata is not requested, and the meta-estimator will raise an error if the user provides it.

  • str: metadata should be passed to the meta-estimator with this given alias instead of the original name.

The default (sklearn.utils.metadata_routing.UNCHANGED) retains the existing request. This allows you to change the request for some parameters and not others.

Added in version 1.3.

Parameters

classesstr, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED

Metadata routing for classes parameter in partial_fit.

Returns

selfobject

The updated object.

Parameters:
Return type:

NeuralNetClassifierWrapped

set_score_request(*, sample_weight='$UNCHANGED$')

Configure whether metadata should be requested to be passed to the score method.

Note that this method is only relevant when this estimator is used as a sub-estimator within a meta-estimator and metadata routing is enabled with enable_metadata_routing=True (see sklearn.set_config()). Please check the User Guide on how the routing mechanism works.

The options for each parameter are:

  • True: metadata is requested, and passed to score if provided. The request is ignored if metadata is not provided.

  • False: metadata is not requested and the meta-estimator will not pass it to score.

  • None: metadata is not requested, and the meta-estimator will raise an error if the user provides it.

  • str: metadata should be passed to the meta-estimator with this given alias instead of the original name.

The default (sklearn.utils.metadata_routing.UNCHANGED) retains the existing request. This allows you to change the request for some parameters and not others.

Added in version 1.3.

Parameters

sample_weightstr, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED

Metadata routing for sample_weight parameter in score.

Returns

selfobject

The updated object.

Parameters:
Return type:

NeuralNetClassifierWrapped

class pytabkit.models.nn_models.rtdl_resnet.NeuralNetRegressorCustomOptim

Bases: NeuralNetRegressorWrapped

initialize_optimizer(triggered_directly=None)

Initialize the model optimizer. If self.optimizer__lr is not set, use self.lr instead.

Parameters

triggered_directly

Deprecated, don’t use it anymore.

set_partial_fit_request(*, classes='$UNCHANGED$')

Configure whether metadata should be requested to be passed to the partial_fit method.

Note that this method is only relevant when this estimator is used as a sub-estimator within a meta-estimator and metadata routing is enabled with enable_metadata_routing=True (see sklearn.set_config()). Please check the User Guide on how the routing mechanism works.

The options for each parameter are:

  • True: metadata is requested, and passed to partial_fit if provided. The request is ignored if metadata is not provided.

  • False: metadata is not requested and the meta-estimator will not pass it to partial_fit.

  • None: metadata is not requested, and the meta-estimator will raise an error if the user provides it.

  • str: metadata should be passed to the meta-estimator with this given alias instead of the original name.

The default (sklearn.utils.metadata_routing.UNCHANGED) retains the existing request. This allows you to change the request for some parameters and not others.

Added in version 1.3.

Parameters

classesstr, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED

Metadata routing for classes parameter in partial_fit.

Returns

selfobject

The updated object.

Parameters:
Return type:

NeuralNetRegressorCustomOptim

set_score_request(*, sample_weight='$UNCHANGED$')

Configure whether metadata should be requested to be passed to the score method.

Note that this method is only relevant when this estimator is used as a sub-estimator within a meta-estimator and metadata routing is enabled with enable_metadata_routing=True (see sklearn.set_config()). Please check the User Guide on how the routing mechanism works.

The options for each parameter are:

  • True: metadata is requested, and passed to score if provided. The request is ignored if metadata is not provided.

  • False: metadata is not requested and the meta-estimator will not pass it to score.

  • None: metadata is not requested, and the meta-estimator will raise an error if the user provides it.

  • str: metadata should be passed to the meta-estimator with this given alias instead of the original name.

The default (sklearn.utils.metadata_routing.UNCHANGED) retains the existing request. This allows you to change the request for some parameters and not others.

Added in version 1.3.

Parameters

sample_weightstr, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED

Metadata routing for sample_weight parameter in score.

Returns

selfobject

The updated object.

Parameters:
Return type:

NeuralNetRegressorCustomOptim

class pytabkit.models.nn_models.rtdl_resnet.NeuralNetRegressorWrapped

Bases: NeuralNetRegressor

__init__(*args, **kwargs)
fit(X, y)

See NeuralNet.fit.

In contrast to NeuralNet.fit, y is non-optional to avoid mistakenly forgetting about y. However, y can be set to None in case it is derived dynamically from X.

get_default_callbacks()
partial_fit(X, y=None, classes=None, **fit_params)

Fit the module.

If the module is initialized, it is not re-initialized, which means that this method should be used if you want to continue training a model (warm start).

Parameters

Xinput data, compatible with skorch.dataset.Dataset

By default, you should be able to pass:

  • numpy arrays

  • torch tensors

  • pandas DataFrame or Series

  • scipy sparse CSR matrices

  • a dictionary of the former three

  • a list/tuple of the former three

  • a Dataset

If this doesn’t work with your data, you have to pass a Dataset that can deal with the data.

ytarget data, compatible with skorch.dataset.Dataset

The same data types as for X are supported. If your X is a Dataset that contains the target, y may be set to None.

classesarray, sahpe (n_classes,)

Solely for sklearn compatibility, currently unused.

**fit_paramsdict

Additional parameters passed to the forward method of the module and to the self.train_split call.

predict(X)

Where applicable, return class labels for samples in X.

If the module’s forward method returns multiple outputs as a tuple, it is assumed that the first output contains the relevant information and the other values are ignored. If all values are relevant, consider using forward() instead.

Parameters

Xinput data, compatible with skorch.dataset.Dataset

By default, you should be able to pass:

  • numpy arrays

  • torch tensors

  • pandas DataFrame or Series

  • scipy sparse CSR matrices

  • a dictionary of the former three

  • a list/tuple of the former three

  • a Dataset

If this doesn’t work with your data, you have to pass a Dataset that can deal with the data.

Returns

y_pred : numpy ndarray

set_categorical_indicator(categorical_indicator)
set_partial_fit_request(*, classes='$UNCHANGED$')

Configure whether metadata should be requested to be passed to the partial_fit method.

Note that this method is only relevant when this estimator is used as a sub-estimator within a meta-estimator and metadata routing is enabled with enable_metadata_routing=True (see sklearn.set_config()). Please check the User Guide on how the routing mechanism works.

The options for each parameter are:

  • True: metadata is requested, and passed to partial_fit if provided. The request is ignored if metadata is not provided.

  • False: metadata is not requested and the meta-estimator will not pass it to partial_fit.

  • None: metadata is not requested, and the meta-estimator will raise an error if the user provides it.

  • str: metadata should be passed to the meta-estimator with this given alias instead of the original name.

The default (sklearn.utils.metadata_routing.UNCHANGED) retains the existing request. This allows you to change the request for some parameters and not others.

Added in version 1.3.

Parameters

classesstr, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED

Metadata routing for classes parameter in partial_fit.

Returns

selfobject

The updated object.

Parameters:
Return type:

NeuralNetRegressorWrapped

set_predict_mean(predict_mean)
set_score_request(*, sample_weight='$UNCHANGED$')

Configure whether metadata should be requested to be passed to the score method.

Note that this method is only relevant when this estimator is used as a sub-estimator within a meta-estimator and metadata routing is enabled with enable_metadata_routing=True (see sklearn.set_config()). Please check the User Guide on how the routing mechanism works.

The options for each parameter are:

  • True: metadata is requested, and passed to score if provided. The request is ignored if metadata is not provided.

  • False: metadata is not requested and the meta-estimator will not pass it to score.

  • None: metadata is not requested, and the meta-estimator will raise an error if the user provides it.

  • str: metadata should be passed to the meta-estimator with this given alias instead of the original name.

The default (sklearn.utils.metadata_routing.UNCHANGED) retains the existing request. This allows you to change the request for some parameters and not others.

Added in version 1.3.

Parameters

sample_weightstr, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED

Metadata routing for sample_weight parameter in score.

Returns

selfobject

The updated object.

Parameters:
Return type:

NeuralNetRegressorWrapped

set_y_train_mean(y_train_mean)
class pytabkit.models.nn_models.rtdl_resnet.RTDL_MLP

Bases: Module

__init__(*, d_in, n_layers, d_layers, d_first_layer, d_last_layer, dropout, d_out, categories, d_embedding, regression, categorical_indicator, num_emb_type='none', num_emb_dim=24, num_emb_hidden_dim=48, num_emb_sigma=0.01, num_emb_lite=False)

Initialize internal Module state, shared by both nn.Module and ScriptModule.

Parameters:

  • d_in (int)

  • n_layers (int)

  • d_layers (int | List[int])

  • d_first_layer (int)

  • d_last_layer (int)

  • dropout (float)

  • d_out (int)

  • categories (List[int] | None)

  • d_embedding (int)

  • regression (bool)

  • num_emb_type (str)

  • num_emb_dim (int)

  • num_emb_hidden_dim (int)

  • num_emb_sigma (float)

  • num_emb_lite (bool)

Return type:

None

forward(x)

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class pytabkit.models.nn_models.rtdl_resnet.ResNet

Bases: Module

__init__(*, d_in, categories, d_embedding, d, d_hidden_factor, n_layers, activation, normalization, hidden_dropout, residual_dropout, d_out, regression, categorical_indicator)

Initialize internal Module state, shared by both nn.Module and ScriptModule.

Parameters:

  • d_in (int)

  • categories (List[int] | None)

  • d_embedding (int)

  • d (int)

  • d_hidden_factor (float)

  • n_layers (int)

  • activation (str)

  • normalization (str)

  • hidden_dropout (float)

  • residual_dropout (float)

  • d_out (int)

  • regression (bool)

Return type:

None

forward(x)

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

Return type:

Tensor

class pytabkit.models.nn_models.rtdl_resnet.Tokenizer

Bases: Module

__init__(d_numerical, categories, d_token, bias)

Initialize internal Module state, shared by both nn.Module and ScriptModule.

Parameters:

  • d_numerical (int)

  • categories (List[int] | None)

  • d_token (int)

  • bias (bool)

Return type:

None

category_offsets: Tensor | None
forward(x_num, x_cat)

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

Parameters:

  • x_num (Tensor)

  • x_cat (Tensor | None)

Return type:

Tensor

property n_tokens: int
class pytabkit.models.nn_models.rtdl_resnet.UniquePrefixCheckpoint

Bases: Checkpoint

This class has two purposes: - add a unique prefix to the checkpoint file to avoid conflicts between different runs in parallel - remove the checkpoint file when training is finished to avoid having too many files

initialize()

(Re-)Set the initial state of the callback. Use this e.g. if the callback tracks some state that should be reset when the model is re-initialized.

This method should return self.

on_train_end(net, **kwargs)

Called at the end of training.

pytabkit.models.nn_models.rtdl_resnet.create_classifier_skorch(id=None, wandb_run=None, use_checkpoints=True, cat_features=None, model_name='resnet', checkpoint_dir='skorch_cp', val_metric_name='class_error', **kwargs)
Parameters:

val_metric_name (str)

pytabkit.models.nn_models.rtdl_resnet.create_ft_transformer_classifier_skorch(id=None, wandb_run=None, *, use_checkpoints=True, cat_features=None, model_name='ft_transformer', checkpoint_dir='skorch_cp', val_metric_name='class_error', **kwargs)
Parameters:

val_metric_name (str)

pytabkit.models.nn_models.rtdl_resnet.create_ft_transformer_regressor_skorch(id=None, wandb_run=None, *, use_checkpoints=True, cat_features=None, model_name='ft_transformer', checkpoint_dir='skorch_cp', **kwargs)
pytabkit.models.nn_models.rtdl_resnet.create_mlp_classifier_skorch(id=None, wandb_run=None, *, use_checkpoints=True, cat_features=None, model_name='mlp', checkpoint_dir='skorch_cp', val_metric_name='class_error', **kwargs)
Parameters:

val_metric_name (str)

pytabkit.models.nn_models.rtdl_resnet.create_mlp_regressor_skorch(id=None, wandb_run=None, *, use_checkpoints=True, cat_features=None, model_name='mlp', checkpoint_dir='skorch_cp', **kwargs)
pytabkit.models.nn_models.rtdl_resnet.create_regressor_skorch(id=None, wandb_run=None, use_checkpoints=True, cat_features=None, model_name='resnet', checkpoint_dir='skorch_cp', **kwargs)
pytabkit.models.nn_models.rtdl_resnet.create_resnet_classifier_skorch(id=None, wandb_run=None, *, use_checkpoints=True, cat_features=None, model_name='resnet', checkpoint_dir='skorch_cp', val_metric_name='class_error', **kwargs)
Parameters:

val_metric_name (str)

pytabkit.models.nn_models.rtdl_resnet.create_resnet_regressor_skorch(id=None, wandb_run=None, *, use_checkpoints=True, cat_features=None, model_name='resnet', checkpoint_dir='skorch_cp', **kwargs)
pytabkit.models.nn_models.rtdl_resnet.geglu(x)
Parameters:

x (Tensor)

Return type:

Tensor

pytabkit.models.nn_models.rtdl_resnet.get_activation_fn(name)
Parameters:

name (str)

Return type:

Callable[[Tensor], Tensor]

pytabkit.models.nn_models.rtdl_resnet.get_nonglu_activation_fn(name)
Parameters:

name (str)

Return type:

Callable[[Tensor], Tensor]

pytabkit.models.nn_models.rtdl_resnet.initialize_optimizer_ft_transformer(self, triggered_directly=None)

Initialize the model optimizer. If self.optimizer__lr is not set, use self.lr instead.

Parameters

triggered_directly

Deprecated, don’t use it anymore.

pytabkit.models.nn_models.rtdl_resnet.mse_constant_predictor(model, X, y)
pytabkit.models.nn_models.rtdl_resnet.print_but_serializable(*args, **kwargs)
pytabkit.models.nn_models.rtdl_resnet.reglu(x)
Parameters:

x (Tensor)

Return type:

Tensor

pytabkit.models.nn_models.tabm module

class pytabkit.models.nn_models.tabm.LinearEfficientEnsemble

Bases: Module

This layer is a more configurable version of the “BatchEnsemble” layer from the paper “BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learning” (link: https://arxiv.org/abs/2002.06715).

First, this layer allows to select only some of the “ensembled” parts: - the input scaling (r_i in the BatchEnsemble paper) - the output scaling (s_i in the BatchEnsemble paper) - the output bias (not mentioned in the BatchEnsemble paper,

but is presented in public implementations)

Second, the initialization of the scaling weights is configurable through the scaling_init argument.

NOTE The term “adapter” is used in the TabM paper only to tell the story. The original BatchEnsemble paper does NOT use this term. So this class also avoids the term “adapter”.

__init__(in_features, out_features, bias=True, *, k, ensemble_scaling_in, ensemble_scaling_out, ensemble_bias, scaling_init)

Initialize internal Module state, shared by both nn.Module and ScriptModule.

Parameters:

  • in_features (int)

  • out_features (int)

  • bias (bool)

  • k (int)

  • ensemble_scaling_in (bool)

  • ensemble_scaling_out (bool)

  • ensemble_bias (bool)

  • scaling_init (Literal['ones', 'random-signs'])

bias: Tensor | None
forward(x)

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

Parameters:

x (Tensor)

Return type:

Tensor

r: Tensor | None
reset_parameters()
s: Tensor | None
class pytabkit.models.nn_models.tabm.MLP

Bases: Module

__init__(*, d_in=None, d_out=None, n_blocks, d_block, dropout, activation='ReLU')

Initialize internal Module state, shared by both nn.Module and ScriptModule.

Parameters:

  • d_in (int | None)

  • d_out (int | None)

  • n_blocks (int)

  • d_block (int)

  • dropout (float)

  • activation (str)

Return type:

None

forward(x)

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

Parameters:

x (Tensor)

Return type:

Tensor

class pytabkit.models.nn_models.tabm.Model

Bases: Module

MLP & TabM.

__init__(*, n_num_features, cat_cardinalities, n_classes, backbone, bins, num_embeddings=None, arch_type, k=None, share_training_batches=True)

Initialize internal Module state, shared by both nn.Module and ScriptModule.

Parameters:

  • n_num_features (int)

  • cat_cardinalities (List[int])

  • n_classes (int | None)

  • backbone (dict)

  • bins (List[Tensor] | None)

  • num_embeddings (Dict | None)

  • arch_type (Literal['plain', 'tabm', 'tabm-mini', 'tabm-packed', 'tabm-normal', 'tabm-mini-normal'])

  • k (int | None)

  • share_training_batches (bool)

Return type:

None

forward(x_num=None, x_cat=None)

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

Parameters:

  • x_num (Tensor | None)

  • x_cat (Tensor | None)

Return type:

Tensor

class pytabkit.models.nn_models.tabm.NLinear

Bases: Module

N linear layers applied in parallel to N disjoint parts of the input.

Shape

  • Input: (B, N, in_features)

  • Output: (B, N, out_features)

The i-th linear layer is applied to the i-th matrix of the shape (B, in_features).

Technically, this is a simplified version of delu.nn.NLinear: https://yura52.github.io/delu/stable/api/generated/delu.nn.NLinear.html. The difference is that this layer supports only 3D inputs with exactly one batch dimension. By contrast, delu.nn.NLinear supports any number of batch dimensions.

__init__(n, in_features, out_features, bias=True)

Initialize internal Module state, shared by both nn.Module and ScriptModule.

Parameters:

  • n (int)

  • in_features (int)

  • out_features (int)

  • bias (bool)

Return type:

None

forward(x)

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

Parameters:

x (Tensor)

Return type:

Tensor

reset_parameters()
class pytabkit.models.nn_models.tabm.OneHotEncoding0d

Bases: Module

__init__(cardinalities)

Initialize internal Module state, shared by both nn.Module and ScriptModule.

Parameters:

cardinalities (List[int])

Return type:

None

forward(x)

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

Parameters:

x (Tensor)

Return type:

Tensor

class pytabkit.models.nn_models.tabm.ScaleEnsemble

Bases: Module

__init__(k, d, *, init)

Initialize internal Module state, shared by both nn.Module and ScriptModule.

Parameters:

  • k (int)

  • d (int)

  • init (Literal['ones', 'normal', 'random-signs'])

Return type:

None

forward(x)

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

Parameters:

x (Tensor)

Return type:

Tensor

reset_parameters()
Return type:

None

pytabkit.models.nn_models.tabm.default_zero_weight_decay_condition(module_name, module, parameter_name, parameter)
Parameters:

  • module_name (str)

  • module (Module)

  • parameter_name (str)

  • parameter (Parameter)

pytabkit.models.nn_models.tabm.init_random_signs_(x)
Parameters:

x (Tensor)

Return type:

Tensor

pytabkit.models.nn_models.tabm.init_rsqrt_uniform_(x, d)
Parameters:

  • x (Tensor)

  • d (int)

Return type:

Tensor

pytabkit.models.nn_models.tabm.make_efficient_ensemble(module, EnsembleLayer, **kwargs)

Replace linear layers with efficient ensembles of linear layers.

NOTE In the paper, there are no experiments with networks with normalization layers. Perhaps, their trainable weights (the affine transformations) also need “ensemblification” as in the paper about “FiLM-Ensemble”. Additional experiments are required to make conclusions.

Parameters:

module (Module)

Return type:

None

pytabkit.models.nn_models.tabm.make_module(type, *args, **kwargs)
Parameters:

type (str)

Return type:

Module

pytabkit.models.nn_models.tabm.make_parameter_groups(module, zero_weight_decay_condition=<function default_zero_weight_decay_condition>, custom_groups=None)
Parameters:

  • module (Module)

  • custom_groups (List[Dict[str, Any]] | None)

Return type:

List[Dict[str, Any]]

pytabkit.models.nn_models.tabr module

class pytabkit.models.nn_models.tabr.NTPLinearLayer

Bases: Module

__init__(in_features, out_features, bias=True, bias_factor=0.1, linear_init_type='default')

Initialize internal Module state, shared by both nn.Module and ScriptModule.

Parameters:

  • in_features (int)

  • out_features (int)

  • bias (bool)

  • bias_factor (float)

  • linear_init_type (str)

forward(x)

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class pytabkit.models.nn_models.tabr.ParametricMishActivationLayer

Bases: Module

__init__(n_features, lr_factor=1.0)

Initialize internal Module state, shared by both nn.Module and ScriptModule.

Parameters:

  • n_features (int)

  • lr_factor (float)

f(x)
forward(x)

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class pytabkit.models.nn_models.tabr.ParametricReluActivationLayer

Bases: Module

__init__(n_features, lr_factor=1.0)

Initialize internal Module state, shared by both nn.Module and ScriptModule.

Parameters:

  • n_features (int)

  • lr_factor (float)

f(x)
forward(x)

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class pytabkit.models.nn_models.tabr.ScalingLayer

Bases: Module

__init__(n_features, lr_factor=6.0)

Initialize internal Module state, shared by both nn.Module and ScriptModule.

Parameters:

  • n_features (int)

  • lr_factor (float)

forward(x)

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class pytabkit.models.nn_models.tabr.TabrLightning

Bases: LightningModule

__init__(model, train_dataset, val_dataset, C, n_classes)
configure_optimizers()

Choose what optimizers and learning-rate schedulers to use in your optimization. Normally you’d need one. But in the case of GANs or similar you might have multiple. Optimization with multiple optimizers only works in the manual optimization mode.

Return:

Any of these 6 options.

  • Single optimizer.

  • List or Tuple of optimizers.

  • Two lists - The first list has multiple optimizers, and the second has multiple LR schedulers (or multiple lr_scheduler_config).

  • Dictionary, with an "optimizer" key, and (optionally) a "lr_scheduler" key whose value is a single LR scheduler or lr_scheduler_config.

  • None - Fit will run without any optimizer.

The lr_scheduler_config is a dictionary which contains the scheduler and its associated configuration. The default configuration is shown below.

lr_scheduler_config = {
    # REQUIRED: The scheduler instance
    "scheduler": lr_scheduler,
    # The unit of the scheduler's step size, could also be 'step'.
    # 'epoch' updates the scheduler on epoch end whereas 'step'
    # updates it after a optimizer update.
    "interval": "epoch",
    # How many epochs/steps should pass between calls to
    # `scheduler.step()`. 1 corresponds to updating the learning
    # rate after every epoch/step.
    "frequency": 1,
    # Metric to monitor for schedulers like `ReduceLROnPlateau`
    "monitor": "val_loss",
    # If set to `True`, will enforce that the value specified 'monitor'
    # is available when the scheduler is updated, thus stopping
    # training if not found. If set to `False`, it will only produce a warning
    "strict": True,
    # If using the `LearningRateMonitor` callback to monitor the
    # learning rate progress, this keyword can be used to specify
    # a custom logged name
    "name": None,
}

When there are schedulers in which the .step() method is conditioned on a value, such as the torch.optim.lr_scheduler.ReduceLROnPlateau scheduler, Lightning requires that the lr_scheduler_config contains the keyword "monitor" set to the metric name that the scheduler should be conditioned on.

Metrics can be made available to monitor by simply logging it using self.log('metric_to_track', metric_val) in your LightningModule.

Note:

Some things to know:

  • Lightning calls .backward() and .step() automatically in case of automatic optimization.

  • If a learning rate scheduler is specified in configure_optimizers() with key "interval" (default “epoch”) in the scheduler configuration, Lightning will call the scheduler’s .step() method automatically in case of automatic optimization.

  • If you use 16-bit precision (precision=16), Lightning will automatically handle the optimizer.

  • If you use torch.optim.LBFGS, Lightning handles the closure function automatically for you.

  • If you use multiple optimizers, you will have to switch to ‘manual optimization’ mode and step them yourself.

  • If you need to control how often the optimizer steps, override the optimizer_step() hook.

get_Xy(part, idx)
Parameters:

part (str)

Return type:

tuple[dict[str, Tensor], Tensor]

predict_step(batch, batch_idx, dataloader_idx=None)

Step function called during predict(). By default, it calls forward(). Override to add any processing logic.

The predict_step() is used to scale inference on multi-devices.

To prevent an OOM error, it is possible to use BasePredictionWriter callback to write the predictions to disk or database after each batch or on epoch end.

The BasePredictionWriter should be used while using a spawn based accelerator. This happens for Trainer(strategy="ddp_spawn") or training on 8 TPU cores with Trainer(accelerator="tpu", devices=8) as predictions won’t be returned.

Args:

batch: The output of your data iterable, normally a DataLoader. batch_idx: The index of this batch. dataloader_idx: The index of the dataloader that produced this batch.

(only if multiple dataloaders used)

Return:

Predicted output (optional).

Example

class MyModel(LightningModule):

    def predict_step(self, batch, batch_idx, dataloader_idx=0):
        return self(batch)

dm = ...
model = MyModel()
trainer = Trainer(accelerator="gpu", devices=2)
predictions = trainer.predict(model, dm)
setup(stage=None)

Called at the beginning of fit (train + validate), validate, test, or predict. This is a good hook when you need to build models dynamically or adjust something about them. This hook is called on every process when using DDP.

Args:

stage: either 'fit', 'validate', 'test', or 'predict'

Example:

class LitModel(...):
    def __init__(self):
        self.l1 = None

    def prepare_data(self):
        download_data()
        tokenize()

        # don't do this
        self.something = else

    def setup(self, stage):
        data = load_data(...)
        self.l1 = nn.Linear(28, data.num_classes)
train_dataloader()

An iterable or collection of iterables specifying training samples.

For more information about multiple dataloaders, see this section.

The dataloader you return will not be reloaded unless you set :paramref:`~pytorch_lightning.trainer.trainer.Trainer.reload_dataloaders_every_n_epochs` to a positive integer.

For data processing use the following pattern:

  • download in prepare_data()

  • process and split in setup()

However, the above are only necessary for distributed processing.

Warning

do not assign state in prepare_data

Note:

Lightning tries to add the correct sampler for distributed and arbitrary hardware. There is no need to set it yourself.

training_step(batch, batch_idx)

Here you compute and return the training loss and some additional metrics for e.g. the progress bar or logger.

Args:

batch: The output of your data iterable, normally a DataLoader. batch_idx: The index of this batch. dataloader_idx: The index of the dataloader that produced this batch.

(only if multiple dataloaders used)

Return:

  • Tensor - The loss tensor

  • dict - A dictionary which can include any keys, but must include the key 'loss' in the case of automatic optimization.

  • None - In automatic optimization, this will skip to the next batch (but is not supported for multi-GPU, TPU, or DeepSpeed). For manual optimization, this has no special meaning, as returning the loss is not required.

In this step you’d normally do the forward pass and calculate the loss for a batch. You can also do fancier things like multiple forward passes or something model specific.

Example:

def training_step(self, batch, batch_idx):
    x, y, z = batch
    out = self.encoder(x)
    loss = self.loss(out, x)
    return loss

To use multiple optimizers, you can switch to ‘manual optimization’ and control their stepping:

def __init__(self):
    super().__init__()
    self.automatic_optimization = False


# Multiple optimizers (e.g.: GANs)
def training_step(self, batch, batch_idx):
    opt1, opt2 = self.optimizers()

    # do training_step with encoder
    ...
    opt1.step()
    # do training_step with decoder
    ...
    opt2.step()
Note:

When accumulate_grad_batches > 1, the loss returned here will be automatically normalized by accumulate_grad_batches internally.

val_dataloader()

An iterable or collection of iterables specifying validation samples.

For more information about multiple dataloaders, see this section.

The dataloader you return will not be reloaded unless you set :paramref:`~pytorch_lightning.trainer.trainer.Trainer.reload_dataloaders_every_n_epochs` to a positive integer.

It’s recommended that all data downloads and preparation happen in prepare_data().

  • fit()

  • validate()

  • prepare_data()

  • setup()

Note:

Lightning tries to add the correct sampler for distributed and arbitrary hardware There is no need to set it yourself.

Note:

If you don’t need a validation dataset and a validation_step(), you don’t need to implement this method.

validation_step(batch, batch_idx)

Operates on a single batch of data from the validation set. In this step you’d might generate examples or calculate anything of interest like accuracy.

Args:

batch: The output of your data iterable, normally a DataLoader. batch_idx: The index of this batch. dataloader_idx: The index of the dataloader that produced this batch.

(only if multiple dataloaders used)

Return:

  • Tensor - The loss tensor

  • dict - A dictionary. Can include any keys, but must include the key 'loss'.

  • None - Skip to the next batch.

# if you have one val dataloader:
def validation_step(self, batch, batch_idx): ...


# if you have multiple val dataloaders:
def validation_step(self, batch, batch_idx, dataloader_idx=0): ...

Examples:

# CASE 1: A single validation dataset
def validation_step(self, batch, batch_idx):
    x, y = batch

    # implement your own
    out = self(x)
    loss = self.loss(out, y)

    # log 6 example images
    # or generated text... or whatever
    sample_imgs = x[:6]
    grid = torchvision.utils.make_grid(sample_imgs)
    self.logger.experiment.add_image('example_images', grid, 0)

    # calculate acc
    labels_hat = torch.argmax(out, dim=1)
    val_acc = torch.sum(y == labels_hat).item() / (len(y) * 1.0)

    # log the outputs!
    self.log_dict({'val_loss': loss, 'val_acc': val_acc})

If you pass in multiple val dataloaders, validation_step() will have an additional argument. We recommend setting the default value of 0 so that you can quickly switch between single and multiple dataloaders.

# CASE 2: multiple validation dataloaders
def validation_step(self, batch, batch_idx, dataloader_idx=0):
    # dataloader_idx tells you which dataset this is.
    x, y = batch

    # implement your own
    out = self(x)

    if dataloader_idx == 0:
        loss = self.loss0(out, y)
    else:
        loss = self.loss1(out, y)

    # calculate acc
    labels_hat = torch.argmax(out, dim=1)
    acc = torch.sum(y == labels_hat).item() / (len(y) * 1.0)

    # log the outputs separately for each dataloader
    self.log_dict({f"val_loss_{dataloader_idx}": loss, f"val_acc_{dataloader_idx}": acc})
Note:

If you don’t need to validate you don’t need to implement this method.

Note:

When the validation_step() is called, the model has been put in eval mode and PyTorch gradients have been disabled. At the end of validation, the model goes back to training mode and gradients are enabled.

class pytabkit.models.nn_models.tabr.TabrModel

Bases: Module

__init__(*, n_num_features, n_bin_features, cat_cardinalities, n_classes, num_embeddings, d_main, d_multiplier, encoder_n_blocks, predictor_n_blocks, mixer_normalization, context_dropout, dropout0, dropout1, normalization, activation, memory_efficient=False, candidate_encoding_batch_size=None, add_scaling_layer=False, scale_lr_factor=6.0, use_ntp_linear=False, linear_init_type='default', use_ntp_encoder=False)

Initialize internal Module state, shared by both nn.Module and ScriptModule.

Parameters:

  • n_num_features (int)

  • n_bin_features (int)

  • cat_cardinalities (list[int])

  • n_classes (int | None)

  • num_embeddings (dict | None)

  • d_main (int)

  • d_multiplier (float)

  • encoder_n_blocks (int)

  • predictor_n_blocks (int)

  • mixer_normalization (bool | Literal['auto'])

  • context_dropout (float)

  • dropout0 (float)

  • dropout1 (float | Literal['dropout0'])

  • normalization (str)

  • activation (str)

  • memory_efficient (bool)

  • candidate_encoding_batch_size (int | None)

  • add_scaling_layer (bool)

  • scale_lr_factor (float)

  • use_ntp_linear (bool)

  • linear_init_type (str)

  • use_ntp_encoder (bool)

Return type:

None

forward(*, x_, y, candidate_x_, candidate_y, context_size, is_train)

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

Parameters:

  • x_ (dict[str, Tensor])

  • y (Tensor | None)

  • candidate_x_ (dict[str, Tensor])

  • candidate_y (Tensor)

  • context_size (int)

  • is_train (bool)

Return type:

Tensor

reset_parameters()
pytabkit.models.nn_models.tabr.bce_with_logits_and_label_smoothing(inputs, *args, ls_eps, **kwargs)
Parameters:

ls_eps (float)

pytabkit.models.nn_models.tabr.zero_wd_condition(module_name, module, parameter_name, parameter)
Parameters:

  • module_name (str)

  • module (Module)

  • parameter_name (str)

  • parameter (Parameter)

pytabkit.models.nn_models.tabr_context_freeze module

class pytabkit.models.nn_models.tabr_context_freeze.TabrLightningContextFreeze

Bases: LightningModule

__init__(model, train_dataset, val_dataset, C, n_classes)
apply_model(part, batch, batch_idx, training)
configure_optimizers()

Choose what optimizers and learning-rate schedulers to use in your optimization. Normally you’d need one. But in the case of GANs or similar you might have multiple. Optimization with multiple optimizers only works in the manual optimization mode.

Return:

Any of these 6 options.

  • Single optimizer.

  • List or Tuple of optimizers.

  • Two lists - The first list has multiple optimizers, and the second has multiple LR schedulers (or multiple lr_scheduler_config).

  • Dictionary, with an "optimizer" key, and (optionally) a "lr_scheduler" key whose value is a single LR scheduler or lr_scheduler_config.

  • None - Fit will run without any optimizer.

The lr_scheduler_config is a dictionary which contains the scheduler and its associated configuration. The default configuration is shown below.

lr_scheduler_config = {
    # REQUIRED: The scheduler instance
    "scheduler": lr_scheduler,
    # The unit of the scheduler's step size, could also be 'step'.
    # 'epoch' updates the scheduler on epoch end whereas 'step'
    # updates it after a optimizer update.
    "interval": "epoch",
    # How many epochs/steps should pass between calls to
    # `scheduler.step()`. 1 corresponds to updating the learning
    # rate after every epoch/step.
    "frequency": 1,
    # Metric to monitor for schedulers like `ReduceLROnPlateau`
    "monitor": "val_loss",
    # If set to `True`, will enforce that the value specified 'monitor'
    # is available when the scheduler is updated, thus stopping
    # training if not found. If set to `False`, it will only produce a warning
    "strict": True,
    # If using the `LearningRateMonitor` callback to monitor the
    # learning rate progress, this keyword can be used to specify
    # a custom logged name
    "name": None,
}

When there are schedulers in which the .step() method is conditioned on a value, such as the torch.optim.lr_scheduler.ReduceLROnPlateau scheduler, Lightning requires that the lr_scheduler_config contains the keyword "monitor" set to the metric name that the scheduler should be conditioned on.

Metrics can be made available to monitor by simply logging it using self.log('metric_to_track', metric_val) in your LightningModule.

Note:

Some things to know:

  • Lightning calls .backward() and .step() automatically in case of automatic optimization.

  • If a learning rate scheduler is specified in configure_optimizers() with key "interval" (default “epoch”) in the scheduler configuration, Lightning will call the scheduler’s .step() method automatically in case of automatic optimization.

  • If you use 16-bit precision (precision=16), Lightning will automatically handle the optimizer.

  • If you use torch.optim.LBFGS, Lightning handles the closure function automatically for you.

  • If you use multiple optimizers, you will have to switch to ‘manual optimization’ mode and step them yourself.

  • If you need to control how often the optimizer steps, override the optimizer_step() hook.

evaluate(eval_batch_size, *, progress_bar=False)
Parameters:

  • eval_batch_size (int)

  • progress_bar (bool)

get_Xy(part, idx)
Parameters:

part (str)

Return type:

tuple[dict[str, Tensor], Tensor]

predict_step(batch, batch_idx, dataloader_idx=None)

Step function called during predict(). By default, it calls forward(). Override to add any processing logic.

The predict_step() is used to scale inference on multi-devices.

To prevent an OOM error, it is possible to use BasePredictionWriter callback to write the predictions to disk or database after each batch or on epoch end.

The BasePredictionWriter should be used while using a spawn based accelerator. This happens for Trainer(strategy="ddp_spawn") or training on 8 TPU cores with Trainer(accelerator="tpu", devices=8) as predictions won’t be returned.

Args:

batch: The output of your data iterable, normally a DataLoader. batch_idx: The index of this batch. dataloader_idx: The index of the dataloader that produced this batch.

(only if multiple dataloaders used)

Return:

Predicted output (optional).

Example

class MyModel(LightningModule):

    def predict_step(self, batch, batch_idx, dataloader_idx=0):
        return self(batch)

dm = ...
model = MyModel()
trainer = Trainer(accelerator="gpu", devices=2)
predictions = trainer.predict(model, dm)
setup(stage=None)

Called at the beginning of fit (train + validate), validate, test, or predict. This is a good hook when you need to build models dynamically or adjust something about them. This hook is called on every process when using DDP.

Args:

stage: either 'fit', 'validate', 'test', or 'predict'

Example:

class LitModel(...):
    def __init__(self):
        self.l1 = None

    def prepare_data(self):
        download_data()
        tokenize()

        # don't do this
        self.something = else

    def setup(self, stage):
        data = load_data(...)
        self.l1 = nn.Linear(28, data.num_classes)
train_dataloader()

An iterable or collection of iterables specifying training samples.

For more information about multiple dataloaders, see this section.

The dataloader you return will not be reloaded unless you set :paramref:`~pytorch_lightning.trainer.trainer.Trainer.reload_dataloaders_every_n_epochs` to a positive integer.

For data processing use the following pattern:

  • download in prepare_data()

  • process and split in setup()

However, the above are only necessary for distributed processing.

Warning

do not assign state in prepare_data

Note:

Lightning tries to add the correct sampler for distributed and arbitrary hardware. There is no need to set it yourself.

training_step(batch, batch_idx)

Here you compute and return the training loss and some additional metrics for e.g. the progress bar or logger.

Args:

batch: The output of your data iterable, normally a DataLoader. batch_idx: The index of this batch. dataloader_idx: The index of the dataloader that produced this batch.

(only if multiple dataloaders used)

Return:

  • Tensor - The loss tensor

  • dict - A dictionary which can include any keys, but must include the key 'loss' in the case of automatic optimization.

  • None - In automatic optimization, this will skip to the next batch (but is not supported for multi-GPU, TPU, or DeepSpeed). For manual optimization, this has no special meaning, as returning the loss is not required.

In this step you’d normally do the forward pass and calculate the loss for a batch. You can also do fancier things like multiple forward passes or something model specific.

Example:

def training_step(self, batch, batch_idx):
    x, y, z = batch
    out = self.encoder(x)
    loss = self.loss(out, x)
    return loss

To use multiple optimizers, you can switch to ‘manual optimization’ and control their stepping:

def __init__(self):
    super().__init__()
    self.automatic_optimization = False


# Multiple optimizers (e.g.: GANs)
def training_step(self, batch, batch_idx):
    opt1, opt2 = self.optimizers()

    # do training_step with encoder
    ...
    opt1.step()
    # do training_step with decoder
    ...
    opt2.step()
Note:

When accumulate_grad_batches > 1, the loss returned here will be automatically normalized by accumulate_grad_batches internally.

val_dataloader()

An iterable or collection of iterables specifying validation samples.

For more information about multiple dataloaders, see this section.

The dataloader you return will not be reloaded unless you set :paramref:`~pytorch_lightning.trainer.trainer.Trainer.reload_dataloaders_every_n_epochs` to a positive integer.

It’s recommended that all data downloads and preparation happen in prepare_data().

  • fit()

  • validate()

  • prepare_data()

  • setup()

Note:

Lightning tries to add the correct sampler for distributed and arbitrary hardware There is no need to set it yourself.

Note:

If you don’t need a validation dataset and a validation_step(), you don’t need to implement this method.

validation_step(batch, batch_idx)

Operates on a single batch of data from the validation set. In this step you’d might generate examples or calculate anything of interest like accuracy.

Args:

batch: The output of your data iterable, normally a DataLoader. batch_idx: The index of this batch. dataloader_idx: The index of the dataloader that produced this batch.

(only if multiple dataloaders used)

Return:

  • Tensor - The loss tensor

  • dict - A dictionary. Can include any keys, but must include the key 'loss'.

  • None - Skip to the next batch.

# if you have one val dataloader:
def validation_step(self, batch, batch_idx): ...


# if you have multiple val dataloaders:
def validation_step(self, batch, batch_idx, dataloader_idx=0): ...

Examples:

# CASE 1: A single validation dataset
def validation_step(self, batch, batch_idx):
    x, y = batch

    # implement your own
    out = self(x)
    loss = self.loss(out, y)

    # log 6 example images
    # or generated text... or whatever
    sample_imgs = x[:6]
    grid = torchvision.utils.make_grid(sample_imgs)
    self.logger.experiment.add_image('example_images', grid, 0)

    # calculate acc
    labels_hat = torch.argmax(out, dim=1)
    val_acc = torch.sum(y == labels_hat).item() / (len(y) * 1.0)

    # log the outputs!
    self.log_dict({'val_loss': loss, 'val_acc': val_acc})

If you pass in multiple val dataloaders, validation_step() will have an additional argument. We recommend setting the default value of 0 so that you can quickly switch between single and multiple dataloaders.

# CASE 2: multiple validation dataloaders
def validation_step(self, batch, batch_idx, dataloader_idx=0):
    # dataloader_idx tells you which dataset this is.
    x, y = batch

    # implement your own
    out = self(x)

    if dataloader_idx == 0:
        loss = self.loss0(out, y)
    else:
        loss = self.loss1(out, y)

    # calculate acc
    labels_hat = torch.argmax(out, dim=1)
    acc = torch.sum(y == labels_hat).item() / (len(y) * 1.0)

    # log the outputs separately for each dataloader
    self.log_dict({f"val_loss_{dataloader_idx}": loss, f"val_acc_{dataloader_idx}": acc})
Note:

If you don’t need to validate you don’t need to implement this method.

Note:

When the validation_step() is called, the model has been put in eval mode and PyTorch gradients have been disabled. At the end of validation, the model goes back to training mode and gradients are enabled.

class pytabkit.models.nn_models.tabr_context_freeze.TabrModelContextFreeze

Bases: Module

class ForwardOutput

Bases: NamedTuple

ForwardOutput(y_pred, context_idx, context_probs)

static __new__(_cls, y_pred, context_idx, context_probs)

Create new instance of ForwardOutput(y_pred, context_idx, context_probs)

Parameters:

  • y_pred (Tensor)

  • context_idx (Tensor)

  • context_probs (Tensor)

context_idx: Tensor

Alias for field number 1

context_probs: Tensor

Alias for field number 2

y_pred: Tensor

Alias for field number 0

__init__(*, n_num_features, n_bin_features, cat_cardinalities, n_classes, num_embeddings, d_main, d_multiplier, encoder_n_blocks, predictor_n_blocks, mixer_normalization, context_dropout, dropout0, dropout1, normalization, activation, memory_efficient=False, candidate_encoding_batch_size=None, add_scaling_layer=False, scale_lr_factor=6.0)

Initialize internal Module state, shared by both nn.Module and ScriptModule.

Parameters:

  • n_num_features (int)

  • n_bin_features (int)

  • cat_cardinalities (list[int])

  • n_classes (int | None)

  • num_embeddings (dict | None)

  • d_main (int)

  • d_multiplier (float)

  • encoder_n_blocks (int)

  • predictor_n_blocks (int)

  • mixer_normalization (bool | Literal['auto'])

  • context_dropout (float)

  • dropout0 (float)

  • dropout1 (float | Literal['dropout0'])

  • normalization (str)

  • activation (str)

  • memory_efficient (bool)

  • candidate_encoding_batch_size (int | None)

  • add_scaling_layer (bool)

  • scale_lr_factor (float)

Return type:

None

forward(*, x_, y, idx, candidate_x_, candidate_y, candidate_idx, context_size, context_idx, is_train)

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

Parameters:

  • x_ (dict[str, Tensor])

  • y (Tensor | None)

  • idx (Tensor | None)

  • candidate_x_ (dict[str, Tensor])

  • candidate_y (Tensor)

  • candidate_idx (Tensor)

  • context_size (int)

  • context_idx (Tensor | None)

  • is_train (bool)

reset_parameters()
pytabkit.models.nn_models.tabr_context_freeze.zero_wd_condition(module_name, module, parameter_name, parameter)
Parameters:

  • module_name (str)

  • module (Module)

  • parameter_name (str)

  • parameter (Parameter)

pytabkit.models.nn_models.tabr_lib module

class pytabkit.models.nn_models.tabr_lib.CLSEmbedding

Bases: Module

__init__(d_embedding)

Initialize internal Module state, shared by both nn.Module and ScriptModule.

Parameters:

d_embedding (int)

Return type:

None

forward(x)

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

Parameters:

x (Tensor)

Return type:

Tensor

class pytabkit.models.nn_models.tabr_lib.CatEmbeddings

Bases: Module

__init__(_cardinalities_and_maybe_dimensions, d_embedding=None, *, stack=False)

Initialize internal Module state, shared by both nn.Module and ScriptModule.

Parameters:

  • _cardinalities_and_maybe_dimensions (list[int] | list[tuple[int, int]])

  • d_embedding (int | None)

  • stack (bool)

Return type:

None

forward(x)

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

Parameters:

x (Tensor)

Return type:

Tensor

reset_parameters()
Return type:

None

class pytabkit.models.nn_models.tabr_lib.LREmbeddings

Bases: Sequential

The LR embeddings from the paper ‘On Embeddings for Numerical Features in Tabular Deep Learning’.

__init__(n_features, d_embedding)

Initialize internal Module state, shared by both nn.Module and ScriptModule.

Parameters:

  • n_features (int)

  • d_embedding (int)

Return type:

None

class pytabkit.models.nn_models.tabr_lib.Lambda

Bases: Module

A wrapper for functions from torch and methods of torch.Tensor.

An important “feature” of this module is that it is intentionally limited:

  • Only the functions from the torch module and the methods of torch.Tensor are allowed.

  • The passed callable must accept a single torch.Tensor and return a single torch.Tensor.

  • The allowed keyword arguments must be of simple types (see the docstring).

Usage

>>> m = delu.nn.Lambda(torch.squeeze)
>>> m(torch.randn(2, 1, 3, 1)).shape
torch.Size([2, 3])
>>> m = delu.nn.Lambda(torch.squeeze, dim=1)
>>> m(torch.randn(2, 1, 3, 1)).shape
torch.Size([2, 3, 1])
>>> m = delu.nn.Lambda(torch.Tensor.abs_)
>>> m(torch.tensor(-1.0))
tensor(1.)

Custom functions are not allowed (technically, they are temporarily allowed, but this functionality is deprecated and will be removed in future releases):

>>> # xdoctest: +SKIP
>>> m = delu.nn.Lambda(lambda x: torch.abs(x))
Traceback (most recent call last):
    ...
ValueError: fn must be a function from `torch` or a method of `torch.Tensor`, but ...

Non-trivial keyword arguments are not allowed:

>>> m = delu.nn.Lambda(torch.mul, other=torch.tensor(2.0))
Traceback (most recent call last):
    ...
ValueError: For kwargs, the allowed value types include: ...
__init__(fn, /, **kwargs)
Args:

fn: the callable. kwargs: the keyword arguments for fn. The allowed values types include:

None, bool, int, float, bytes, str and (nested) tuples of these simple types.

Parameters:

fn (Callable[[...], Tensor])

Return type:

None

forward(x)

Do the forward pass.

Parameters:

x (Tensor)

Return type:

Tensor

class pytabkit.models.nn_models.tabr_lib.LinearEmbeddings

Bases: Module

__init__(n_features, d_embedding, bias=True)

Initialize internal Module state, shared by both nn.Module and ScriptModule.

Parameters:

  • n_features (int)

  • d_embedding (int)

  • bias (bool)

forward(x)

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

Parameters:

x (Tensor)

Return type:

Tensor

reset_parameters()
Return type:

None

class pytabkit.models.nn_models.tabr_lib.MLP

Bases: Module

class Block

Bases: Module

__init__(*, d_in, d_out, bias, activation, dropout)

Initialize internal Module state, shared by both nn.Module and ScriptModule.

Parameters:

  • d_in (int)

  • d_out (int)

  • bias (bool)

  • activation (str)

  • dropout (float)

Return type:

None

forward(x)

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

Parameters:

x (Tensor)

Return type:

Tensor

Head

alias of Linear

__init__(*, d_in, d_out, n_blocks, d_layer, activation, dropout)

Initialize internal Module state, shared by both nn.Module and ScriptModule.

Parameters:

  • d_in (int)

  • d_out (int | None)

  • n_blocks (int)

  • d_layer (int)

  • activation (str)

  • dropout (float)

Return type:

None

property d_out: int
forward(x)

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

Parameters:

x (Tensor)

Return type:

Tensor

class pytabkit.models.nn_models.tabr_lib.NLinear

Bases: Module

__init__(n_features, d_in, d_out, bias=True)

Initialize internal Module state, shared by both nn.Module and ScriptModule.

Parameters:

  • n_features (int)

  • d_in (int)

  • d_out (int)

  • bias (bool)

Return type:

None

forward(x)

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class pytabkit.models.nn_models.tabr_lib.OneHotEncoder

Bases: Module

__init__(cardinalities)

Initialize internal Module state, shared by both nn.Module and ScriptModule.

Parameters:

cardinalities (list[int])

Return type:

None

cardinalities: Tensor
forward(x)

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

Parameters:

x (Tensor)

Return type:

Tensor

class pytabkit.models.nn_models.tabr_lib.PBLDEmbeddings

Bases: Module

__init__(n_features, n_frequencies, frequency_scale, d_embedding, plr_act_name='linear', plr_use_densenet=True)

Initialize internal Module state, shared by both nn.Module and ScriptModule.

Parameters:

  • n_features (int)

  • n_frequencies (int)

  • frequency_scale (float)

  • d_embedding (int)

  • plr_act_name (str)

  • plr_use_densenet (bool)

forward(x)

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class pytabkit.models.nn_models.tabr_lib.PLREmbeddings

Bases: Sequential

The PLR embeddings from the paper ‘On Embeddings for Numerical Features in Tabular Deep Learning’.

Additionally, the ‘lite’ option is added. Setting it to False gives you the original PLR embedding from the above paper. We noticed that lite=True makes the embeddings noticeably more lightweight without critical performance loss, and we used that for our model.

__init__(n_features, n_frequencies, frequency_scale, d_embedding, lite)

Initialize internal Module state, shared by both nn.Module and ScriptModule.

Parameters:

  • n_features (int)

  • n_frequencies (int)

  • frequency_scale (float)

  • d_embedding (int)

  • lite (bool)

Return type:

None

class pytabkit.models.nn_models.tabr_lib.PeriodicEmbeddings

Bases: Module

__init__(n_features, n_frequencies, frequency_scale)

Initialize internal Module state, shared by both nn.Module and ScriptModule.

Parameters:

  • n_features (int)

  • n_frequencies (int)

  • frequency_scale (float)

Return type:

None

forward(x)

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

Parameters:

x (Tensor)

Return type:

Tensor

pytabkit.models.nn_models.tabr_lib.cat(data, /, dim=0)

Concatenate a sequence of collections of tensors.

delu.cat is a generalized version of torch.cat for concatenating not only tensors, but also (nested) collections of tensors.

Usage

Let’s see how a sequence of model outputs for batches can be concatenated into a output tuple for the whole dataset:

>>> from torch.utils.data import DataLoader, TensorDataset
>>> dataset = TensorDataset(torch.randn(320, 24))
>>> batch_size = 32
>>>
>>> # The model returns not only predictions, but also embeddings.
>>> def model(x_batch):
...     # A dummy forward pass.
...     embeddings_batch = torch.randn(batch_size, 16)
...     y_pred_batch = torch.randn(batch_size)
...     return (y_pred_batch, embeddings_batch)
...
>>> y_pred, embeddings = delu.cat(
...     [model(batch) for batch in DataLoader(dataset, batch_size, shuffle=True)]
... )
>>> len(y_pred) == len(dataset)
True
>>> len(embeddings) == len(dataset)
True

The same works for dictionaries:

>>> def model(x_batch):
...     return {
...         'y_pred': torch.randn(batch_size),
...         'embeddings': torch.randn(batch_size, 16)
...     }
...
>>> outputs = delu.cat(
...     [model(batch) for batch in DataLoader(dataset, batch_size, shuffle=True)]
... )
>>> len(outputs['y_pred']) == len(dataset)
True
>>> len(outputs['embeddings']) == len(dataset)
True

The same works for sequences of named tuples, dataclasses, tensors and nested combinations of all mentioned collection types.

Below, additional technical examples are provided.

The common setup:

>>> # First batch.
>>> x1 = torch.randn(64, 10)
>>> y1 = torch.randn(64)
>>> # Second batch.
>>> x2 = torch.randn(64, 10)
>>> y2 = torch.randn(64)
>>> # The last (incomplete) batch.
>>> x3 = torch.randn(7, 10)
>>> y3 = torch.randn(7)
>>> total_size = len(x1) + len(x2) + len(x3)

delu.cat can be applied to tuples:

>>> batches = [(x1, y1), (x2, y2), (x3, y3)]
>>> X, Y = delu.cat(batches)
>>> len(X) == total_size and len(Y) == total_size
True

delu.cat can be applied to dictionaries:

>>> batches = [
...     {'x': x1, 'y': y1},
...     {'x': x2, 'y': y2},
...     {'x': x3, 'y': y3},
... ]
>>> result = delu.cat(batches)
>>> isinstance(result, dict)
True
>>> len(result['x']) == total_size and len(result['y']) == total_size
True

delu.cat can be applied to named tuples:

>>> from typing import NamedTuple
>>> class Data(NamedTuple):
...     x: torch.Tensor
...     y: torch.Tensor
...
>>> batches = [Data(x1, y1), Data(x2, y2), Data(x3, y3)]
>>> result = delu.cat(batches)
>>> isinstance(result, Data)
True
>>> len(result.x) == total_size and len(result.y) == total_size
True

delu.cat can be applied to dataclasses:

>>> from dataclasses import dataclass
>>> @dataclass
... class Data:
...     x: torch.Tensor
...     y: torch.Tensor
...
>>> batches = [Data(x1, y1), Data(x2, y2), Data(x3, y3)]
>>> result = delu.cat(batches)
>>> isinstance(result, Data)
True
>>> len(result.x) == total_size and len(result.y) == total_size
True

delu.cat can be applied to nested collections:

>>> batches = [
...     (x1, {'a': {'b': y1}}),
...     (x2, {'a': {'b': y2}}),
...     (x3, {'a': {'b': y3}}),
... ]
>>> X, Y_nested = delu.cat(batches)
>>> len(X) == total_size and len(Y_nested['a']['b']) == total_size
True

Lists are not supported:

>>> # This does not work. Instead, use tuples.
>>> # batches = [[x1, y1], [x2, y2], [x3, y3]]
>>> # delu.cat(batches)  # Error
Args:
data: the list of collections of tensors.

All items of the list must be of the same type, structure and layout, only the dim dimension can vary (same as for torch.cat). All the “leaf” values must be of the type torch.Tensor.

dim: the dimension along which the tensors are concatenated.

Returns:

The concatenated items of the list.

Parameters:

  • data (List[T])

  • dim (int)

Return type:

T

pytabkit.models.nn_models.tabr_lib.default_zero_weight_decay_condition(module_name, module, parameter_name, parameter)
Parameters:

  • module_name (str)

  • module (Module)

  • parameter_name (str)

  • parameter (Parameter)

pytabkit.models.nn_models.tabr_lib.get_d_out(n_classes)
Parameters:

n_classes (int | None)

Return type:

int

pytabkit.models.nn_models.tabr_lib.get_lr(optimizer)
Parameters:

optimizer (Optimizer)

Return type:

float

pytabkit.models.nn_models.tabr_lib.get_n_parameters(m)
Parameters:

m (Module)

pytabkit.models.nn_models.tabr_lib.is_oom_exception(err)
Parameters:

err (RuntimeError)

Return type:

bool

pytabkit.models.nn_models.tabr_lib.iter_batches(data, /, batch_size, *, shuffle=False, generator=None, drop_last=False)

Iterate over a tensor or a collection of tensors by (random) batches.

The function makes batches along the first dimension of the tensors in data.

TL;DR (assuming that X and Y denote full tensors and xi and yi denote batches):

  • delu.iter_batches: X -> [x1, x2, ..., xN]

  • delu.iter_batches: (X, Y) -> [(x1, y1), (x2, y2), ..., (xN, yN)]

  • delu.iter_batches: {'x': X, 'y': Y} -> [{'x': x1, 'y': y1}, ...]

  • Same for named tuples.

  • Same for dataclasses.

Note

delu.iter_batches is significantly faster for in-memory tensors than torch.utils.data.DataLoader, because, when building batches, it uses batched indexing instead of one-by-one indexing.

Usage

>>> X = torch.randn(12, 32)
>>> Y = torch.randn(12)

delu.iter_batches can be applied to tensors:

>>> for x in delu.iter_batches(X, batch_size=5):
...     print(len(x))
5
5
2

delu.iter_batches can be applied to tuples:

>>> # shuffle=True can be useful for training.
>>> dataset = (X, Y)
>>> for x, y in delu.iter_batches(dataset, batch_size=5, shuffle=True):
...     print(len(x), len(y))
5 5
5 5
2 2
>>> # Drop the last incomplete batch.
>>> for x, y in delu.iter_batches(
...     dataset, batch_size=5, shuffle=True, drop_last=True
... ):
...     print(len(x), len(y))
5 5
5 5
>>> # The last batch is complete, so drop_last=True does not have any effect.
>>> batches = []
>>> for x, y in delu.iter_batches(dataset, batch_size=6, drop_last=True):
...     print(len(x), len(y))
...     batches.append((x, y))
6 6
6 6

By default, shuffle is set to False, i.e. the order of items is preserved:

>>> X2, Y2 = delu.cat(list(delu.iter_batches((X, Y), batch_size=5)))
>>> print((X == X2).all().item(), (Y == Y2).all().item())
True True

delu.iter_batches can be applied to dictionaries:

>>> dataset = {'x': X, 'y': Y}
>>> for batch in delu.iter_batches(dataset, batch_size=5, shuffle=True):
...     print(isinstance(batch, dict), len(batch['x']), len(batch['y']))
True 5 5
True 5 5
True 2 2

delu.iter_batches can be applied to named tuples:

>>> from typing import NamedTuple
>>> class Data(NamedTuple):
...     x: torch.Tensor
...     y: torch.Tensor
>>> dataset = Data(X, Y)
>>> for batch in delu.iter_batches(dataset, batch_size=5, shuffle=True):
...     print(isinstance(batch, Data), len(batch.x), len(batch.y))
True 5 5
True 5 5
True 2 2

delu.iter_batches can be applied to dataclasses:

>>> from dataclasses import dataclass
>>> @dataclass
... class Data:
...     x: torch.Tensor
...     y: torch.Tensor
>>> dataset = Data(X, Y)
>>> for batch in delu.iter_batches(dataset, batch_size=5, shuffle=True):
...     print(isinstance(batch, Data), len(batch.x), len(batch.y))
True 5 5
True 5 5
True 2 2
Args:
data: the tensor or the non-empty collection of tensors.

If data is a collection, then the tensors must be of the same size along the first dimension.

batch_size: the batch size. If drop_last is False,

then the last batch can be smaller than batch_size.

shuffle: if True, iterate over random batches (without replacement),

not sequentially.

generator: when shuffle is True, passing generator makes the function

reproducible.

drop_last: when True and the last batch is smaller then batch_size,

then this last batch is not returned (in other words, same as the drop_last argument for torch.utils.data.DataLoader).

Returns:

the iterator over batches.

Parameters:

  • data (T)

  • batch_size (int)

  • shuffle (bool)

  • generator (Generator | None)

  • drop_last (bool)

Return type:

Iterator[T]

pytabkit.models.nn_models.tabr_lib.make_module(spec, *args, **kwargs)
>>> make_module('ReLU')
>>> make_module(nn.ReLU)
>>> make_module('Linear', 1, out_features=2)
>>> make_module((lambda *args: nn.Linear(*args)), 1, out_features=2)
>>> make_module({'type': 'Linear', 'in_features' 1}, out_features=2)
Parameters:

spec (str | dict[str, Any] | Callable[[...], Module])

Return type:

Module

pytabkit.models.nn_models.tabr_lib.make_optimizer(module, type, *, zero_weight_decay_condition=<function default_zero_weight_decay_condition>, custom_parameter_groups=None, **optimizer_kwargs)
Parameters:

  • module (Module)

  • type (str)

  • custom_parameter_groups (dict[tuple[str], dict] | None)

Return type:

Optimizer

pytabkit.models.nn_models.tabr_lib.make_parameter_groups(model, zero_weight_decay_condition, custom_groups)
Parameters:

  • model (Module)

  • custom_groups (dict[tuple[str], dict])

Return type:

list[dict[str, Any]]

pytabkit.models.nn_models.tabr_lib.make_trainable_vector(d)
Parameters:

d (int)

Return type:

Parameter

pytabkit.models.nn_models.tabr_lib.register_module(key, f)
Parameters:

  • key (str)

  • f (Callable[[...], Module])

Return type:

None

pytabkit.models.nn_models.tabr_lib.set_lr(optimizer, lr)
Parameters:

  • optimizer (Optimizer)

  • lr (float)

Return type:

None

Module contents