synthcity.metrics.eval_performance module

class AugmentationPerformanceEvaluatorLinear(**kwargs: Any)

Bases: synthcity.metrics.eval_performance.PerformanceEvaluatorLinear

static direction() → str

evaluate(X_gt: synthcity.plugins.core.dataloader.DataLoader, X_syn: synthcity.plugins.core.dataloader.DataLoader) → Dict

evaluate_default(X_gt: synthcity.plugins.core.dataloader.DataLoader, X_syn: synthcity.plugins.core.dataloader.DataLoader) → float

classmethod fqdn() → str

static name() → str

reduction() → Callable

static standard_performance_output_keys() → List

static type() → str

use_cache(path: pathlib.Path) → bool

class AugmentationPerformanceEvaluatorMLP(**kwargs: Any)

Bases: synthcity.metrics.eval_performance.PerformanceEvaluatorMLP

static direction() → str

evaluate(X_gt: synthcity.plugins.core.dataloader.DataLoader, X_syn: synthcity.plugins.core.dataloader.DataLoader) → Dict

evaluate_default(X_gt: synthcity.plugins.core.dataloader.DataLoader, X_syn: synthcity.plugins.core.dataloader.DataLoader) → float

classmethod fqdn() → str

static name() → str

reduction() → Callable

static standard_performance_output_keys() → List

static type() → str

use_cache(path: pathlib.Path) → bool

class AugmentationPerformanceEvaluatorXGB(**kwargs: Any)

Bases: synthcity.metrics.eval_performance.PerformanceEvaluatorXGB

static direction() → str

evaluate(X_gt: synthcity.plugins.core.dataloader.DataLoader, X_syn: synthcity.plugins.core.dataloader.DataLoader) → Dict

evaluate_default(X_gt: synthcity.plugins.core.dataloader.DataLoader, X_syn: synthcity.plugins.core.dataloader.DataLoader) → float

classmethod fqdn() → str

static name() → str

reduction() → Callable

static standard_performance_output_keys() → List

static type() → str

use_cache(path: pathlib.Path) → bool

class FeatureImportanceRankDistance(distance: str = 'kendall', **kwargs: Any)

Bases: synthcity.metrics.core.metric.MetricEvaluator

Train an XGBoost classifier or regressor on the synthetic data and evaluate the feature importance. Train an XGBoost model on the real data and evaluate the feature importance.

Returns the rank distance between the feature importance Returns the average performance discrepancy between training on real data vs on synthetic data.

Score:

close to 1: similar performance close to 0: unrelated close to -1: the ranks have different monotony.

static direction() → str

distance(lhs: numpy.ndarray, rhs: numpy.ndarray) → Tuple[float, float]

evaluate(X_gt: synthcity.plugins.core.dataloader.DataLoader, X_syn: synthcity.plugins.core.dataloader.DataLoader) → Dict

evaluate_default(X_gt: synthcity.plugins.core.dataloader.DataLoader, X_syn: synthcity.plugins.core.dataloader.DataLoader) → float

classmethod fqdn() → str

static name() → str

reduction() → Callable

static type() → str

use_cache(path: pathlib.Path) → bool

class PerformanceEvaluator(**kwargs: Any)

Bases: synthcity.metrics.core.metric.MetricEvaluator

Evaluating synthetic data based on downstream performance.

This implements the train-on-synthetic test-on-real methodology for evaluation.

static direction() → str

abstract evaluate(X_gt: synthcity.plugins.core.dataloader.DataLoader, X_syn: synthcity.plugins.core.dataloader.DataLoader) → Dict

evaluate_default(X_gt: synthcity.plugins.core.dataloader.DataLoader, X_syn: synthcity.plugins.core.dataloader.DataLoader) → float

classmethod fqdn() → str

abstract static name() → str

reduction() → Callable

static standard_performance_output_keys() → List

static type() → str

use_cache(path: pathlib.Path) → bool

class PerformanceEvaluatorLinear(**kwargs: Any)

Bases: synthcity.metrics.eval_performance.PerformanceEvaluator

Train a Linear classifier or regressor on the synthetic data and evaluate the performance on real test data.

Returns the average performance discrepancy between training on real data vs on synthetic data.

Score:

close to 1: similar performance close to 0: massive performance degradation

static direction() → str

evaluate(X_gt: synthcity.plugins.core.dataloader.DataLoader, X_syn: synthcity.plugins.core.dataloader.DataLoader) → Dict

evaluate_default(X_gt: synthcity.plugins.core.dataloader.DataLoader, X_syn: synthcity.plugins.core.dataloader.DataLoader) → float

classmethod fqdn() → str

static name() → str

reduction() → Callable

static standard_performance_output_keys() → List

static type() → str

use_cache(path: pathlib.Path) → bool

class PerformanceEvaluatorMLP(**kwargs: Any)

Bases: synthcity.metrics.eval_performance.PerformanceEvaluator

Train a Neural Net classifier or regressor on the synthetic data and evaluate the performance on real test data.

Returns the average performance discrepancy between training on real data vs on synthetic data.

Score:

close to 1: similar performance close to 1: massive performance degradation

static direction() → str

evaluate(X_gt: synthcity.plugins.core.dataloader.DataLoader, X_syn: synthcity.plugins.core.dataloader.DataLoader) → Dict

evaluate_default(X_gt: synthcity.plugins.core.dataloader.DataLoader, X_syn: synthcity.plugins.core.dataloader.DataLoader) → float

classmethod fqdn() → str

static name() → str

reduction() → Callable

static standard_performance_output_keys() → List

static type() → str

use_cache(path: pathlib.Path) → bool

class PerformanceEvaluatorXGB(**kwargs: Any)

Bases: synthcity.metrics.eval_performance.PerformanceEvaluator

Train an XGBoost classifier or regressor on the synthetic data and evaluate the performance on real test data.

Returns the average performance discrepancy between training on real data vs on synthetic data.

Score:

close to 1: similar performance close to 0: massive performance degradation

static direction() → str

evaluate(X_gt: synthcity.plugins.core.dataloader.DataLoader, X_syn: synthcity.plugins.core.dataloader.DataLoader) → Dict

evaluate_default(X_gt: synthcity.plugins.core.dataloader.DataLoader, X_syn: synthcity.plugins.core.dataloader.DataLoader) → float

classmethod fqdn() → str

static name() → str

reduction() → Callable

static standard_performance_output_keys() → List

static type() → str

use_cache(path: pathlib.Path) → bool