synthcity.metrics.eval module

class Metrics

Bases: object

static evaluate(X_gt: Union[synthcity.plugins.core.dataloader.DataLoader, pandas.core.frame.DataFrame], X_syn: Union[synthcity.plugins.core.dataloader.DataLoader, pandas.core.frame.DataFrame], X_train: Optional[Union[synthcity.plugins.core.dataloader.DataLoader, pandas.core.frame.DataFrame]] = None, X_ref_syn: Optional[Union[synthcity.plugins.core.dataloader.DataLoader, pandas.core.frame.DataFrame]] = None, X_augmented: Optional[Union[synthcity.plugins.core.dataloader.DataLoader, pandas.core.frame.DataFrame]] = None, reduction: str = 'mean', n_histogram_bins: int = 10, metrics: Optional[Dict] = None, task_type: str = 'classification', random_state: int = 0, workspace: pathlib.Path = PosixPath('workspace'), use_cache: bool = True) → pandas.core.frame.DataFrame

Core evaluation logic for the metrics

X_gt: Dataloader or DataFrame

Reference real data

X_syn: Dataloader or DataFrame

Synthetic data

X_train: Dataloader or DataFrame

The data used to train the synthetic model (used for domias metrics only).

X_ref_syn: Dataloader or DataFrame

Reference synthetic data (used for domias metrics only).

X_augmented: Dataloader or DataFrame

Augmented data

metrics: dict

the dictionary of metrics to evaluate Full dictionary of metrics is: {

‘sanity’: [‘data_mismatch’, ‘common_rows_proportion’, ‘nearest_syn_neighbor_distance’, ‘close_values_probability’, ‘distant_values_probability’], ‘stats’: [‘jensenshannon_dist’, ‘chi_squared_test’, ‘feature_corr’, ‘inv_kl_divergence’, ‘ks_test’, ‘max_mean_discrepancy’, ‘wasserstein_dist’, ‘prdc’, ‘alpha_precision’, ‘survival_km_distance’], ‘performance’: [‘linear_model’, ‘mlp’, ‘xgb’, ‘feat_rank_distance’], ‘detection’: [‘detection_xgb’, ‘detection_mlp’, ‘detection_gmm’, ‘detection_linear’], ‘privacy’: [‘delta-presence’, ‘k-anonymization’, ‘k-map’, ‘distinct l-diversity’, ‘identifiability_score’]

}

reduction: str

The way to aggregate metrics across folds. Can be: ‘mean’, “min”, or “max”.

n_histogram_bins: int

The number of bins used in histogram calculation of a given metric. Defaults to 10.

task_type: str

The type of problem. Relevant for evaluating the downstream models with the correct metrics. Valid tasks are: “classification”, “regression”, “survival_analysis”, “time_series”, “time_series_survival”.

random_state: int

random seed

workspace: Path

The folder for caching intermediary results.

use_cache: bool

If the a metric has been previously run and is cached, it will be reused for the experiments. Defaults to True.

static list() → dict