pytabkit.bench.run package

Submodules

pytabkit.bench.run.results module

class pytabkit.bench.run.results.ResultManager

Bases: object

Stores experimental results and can save and load them.

__init__()
add_results(is_cv, results_dict)

Add a dictionary of results. :param is_cv: Whether these results are from cross-validation (True) or refitting (False). :param results_dict: Dictionary of results

Parameters:

  • is_cv (bool)

  • results_dict (Dict)

Return type:

None

static load(path, load_other=True, load_preds=True)

Load results. :param path: Data path. :param load_other: If True, load other_dict. :param load_preds: If True, load the model predictions. :return:

Parameters:

  • path (Path)

  • load_other (bool)

  • load_preds (bool)

save(path)
Parameters:

path (Path)

Return type:

None

pytabkit.bench.run.results.save_summaries(paths, task_infos, alg_name, n_cv, rerun=False)

Compress the results into result_summaries that can be loaded faster for evaluation. :param paths: Path configuration. :param task_infos: Task infos of tasks that should be summarized. :param alg_name: Name of the method whose results should be summarized. :param n_cv: Number of cross-validation splits for which the results should be summarized. :param rerun: Whether to re-compute the summaries even if summaries are already present.

Parameters:

  • paths (Paths)

  • task_infos (List[TaskInfo])

  • alg_name (str)

  • n_cv (int)

Return type:

None

pytabkit.bench.run.task_execution module

class pytabkit.bench.run.task_execution.RunConfig

Bases: object

This class stores some benchmark settings that a method can be run with.

__init__(n_tt_splits, n_cv=1, n_refit=0, use_default_split=False, trainval_fraction=0.8, train_fraction=0.75, save_y_pred=False, min_split_idx=0, metrics=None)
Parameters:

  • n_tt_splits (int) – Number of trainval-test-splits to evaluate the method with.

  • n_cv (int) – Number of cross-validation folds. If n_cv=1, use a single random split.

  • n_refit (int) – Number of models that should be refitted (and ensembled) on the training and validation set.

  • use_default_split (bool) – Whether the default split of the datasets should be used.

  • trainval_fraction (float) – Fraction in (0, 1) of the data that should be used for training and validation set.

  • train_fraction (float)

  • save_y_pred (bool)

  • min_split_idx (int)

  • metrics (Metrics | None)

The rest will be used for the test set. :param train_fraction: Only used if n_cv=1. In this case, out of the training+validation data, the given fraction of the data is used for training. :param save_y_pred: Whether the predictions on the whole dataset should be saved (can use a considerable amount of disk storage, e.g. 3 GB for running a single method on meta-train and meta-test benchmarks). :param min_split_idx: Minimum index of the split that should be used. Can be set larger than zero if only a sub-range of the splits should be run. :param metrics: Metrics object that specifies which metrics should be evaluated.

class pytabkit.bench.run.task_execution.TabBenchJob

Bases: AbstractJob

Internal helper class implementing AbstractJob for running tabular benchmarking jobs with our scheduling code.

__init__(alg_name, alg_wrapper, task_package, paths, metrics=None)
Parameters:

  • alg_name (str) – Unique name of the method (for saving results).

  • alg_wrapper (AlgWrapper) – Wrapper implementing the ML method.

  • task_package (TaskPackage) – Task package containing information on dataset and splits.

  • paths (Paths) – Data path configuration.

  • metrics (Metrics | None)

get_desc()
Returns:

Return a description that can be logged, e.g., when the job is started and when it finishes.

Return type:

str

get_group()
Returns:

Group name, in this case just the name of the AlgWrapper class.

Return type:

str

get_required_resources()
Returns:

Return the resources requested by this job.

Return type:

RequiredResources

class pytabkit.bench.run.task_execution.TabBenchJobManager

Bases: object

This class can be used to add and run jobs for tabular benchmarks.

__init__(paths)
Parameters:

paths (Paths) – Data path configuration.

add_jobs(task_infos, run_config, alg_name, alg_wrapper, tags=None, rerun=False)

Add jobs for the given method with the given run configuration on all task infos where results are not already available (except if rerun=True). Will also store the algorithm configuration and copy the current source files to the corresponding algorithm folder. :param task_infos: List of TaskInfo objects representing the datasets on which the method should be run. :param run_config: Run configuration. :param alg_name: Name of the method, should be unique (is used for storing and printing the results) :param alg_wrapper: Wrapper implementing the ML method. :param tags: List of tags associated to the method (can be used for selecting a subset of methods later). :param rerun: If True, run all combinations even if there are already computed results stored for it. (For large reruns, we rather recommend renaming the old method with rename_alg.py and then running the jobs again with the new name and rerun=False. This avoids problems if the rerun crashes and preserves the old results for comparison.)

Parameters:
Return type:

None

run_jobs(scheduler)

Runs the added jobs with the given scheduler. After all jobs are done, creates the result summaries for faster loading of results. :param scheduler: Scheduler for running the jobs.

Parameters:

scheduler (BaseJobScheduler)

Return type:

None

pytabkit.bench.run.task_execution.run_alg_selection(paths, config, task_infos, target_alg_name, alg_names, val_metric_name, tags=['paper'], rerun=False)
Parameters:

  • paths (Paths)

  • config (RunConfig)

  • task_infos (List[TaskInfo])

  • target_alg_name (str)

  • alg_names (List[str])

  • val_metric_name (str)

  • tags (List[str])

  • rerun (bool)

Module contents