synthcity.metrics.eval_performance module
- class AugmentationPerformanceEvaluatorLinear(**kwargs: Any)
Bases:
synthcity.metrics.eval_performance.PerformanceEvaluatorLinear
- static direction() str
- evaluate(X_gt: synthcity.plugins.core.dataloader.DataLoader, X_syn: synthcity.plugins.core.dataloader.DataLoader) Dict
- evaluate_default(X_gt: synthcity.plugins.core.dataloader.DataLoader, X_syn: synthcity.plugins.core.dataloader.DataLoader) float
- classmethod fqdn() str
- static name() str
- reduction() Callable
- static standard_performance_output_keys() List
- static type() str
- use_cache(path: pathlib.Path) bool
- class AugmentationPerformanceEvaluatorMLP(**kwargs: Any)
Bases:
synthcity.metrics.eval_performance.PerformanceEvaluatorMLP
- static direction() str
- evaluate(X_gt: synthcity.plugins.core.dataloader.DataLoader, X_syn: synthcity.plugins.core.dataloader.DataLoader) Dict
- evaluate_default(X_gt: synthcity.plugins.core.dataloader.DataLoader, X_syn: synthcity.plugins.core.dataloader.DataLoader) float
- classmethod fqdn() str
- static name() str
- reduction() Callable
- static standard_performance_output_keys() List
- static type() str
- use_cache(path: pathlib.Path) bool
- class AugmentationPerformanceEvaluatorXGB(**kwargs: Any)
Bases:
synthcity.metrics.eval_performance.PerformanceEvaluatorXGB
- static direction() str
- evaluate(X_gt: synthcity.plugins.core.dataloader.DataLoader, X_syn: synthcity.plugins.core.dataloader.DataLoader) Dict
- evaluate_default(X_gt: synthcity.plugins.core.dataloader.DataLoader, X_syn: synthcity.plugins.core.dataloader.DataLoader) float
- classmethod fqdn() str
- static name() str
- reduction() Callable
- static standard_performance_output_keys() List
- static type() str
- use_cache(path: pathlib.Path) bool
- class FeatureImportanceRankDistance(distance: str = 'kendall', **kwargs: Any)
Bases:
synthcity.metrics.core.metric.MetricEvaluator
Train an XGBoost classifier or regressor on the synthetic data and evaluate the feature importance. Train an XGBoost model on the real data and evaluate the feature importance.
Returns the rank distance between the feature importance Returns the average performance discrepancy between training on real data vs on synthetic data.
- Score:
close to 1: similar performance close to 0: unrelated close to -1: the ranks have different monotony.
- static direction() str
- distance(lhs: numpy.ndarray, rhs: numpy.ndarray) Tuple[float, float]
- evaluate(X_gt: synthcity.plugins.core.dataloader.DataLoader, X_syn: synthcity.plugins.core.dataloader.DataLoader) Dict
- evaluate_default(X_gt: synthcity.plugins.core.dataloader.DataLoader, X_syn: synthcity.plugins.core.dataloader.DataLoader) float
- classmethod fqdn() str
- static name() str
- reduction() Callable
- static type() str
- use_cache(path: pathlib.Path) bool
- class PerformanceEvaluator(**kwargs: Any)
Bases:
synthcity.metrics.core.metric.MetricEvaluator
Evaluating synthetic data based on downstream performance.
This implements the train-on-synthetic test-on-real methodology for evaluation.
- static direction() str
- abstract evaluate(X_gt: synthcity.plugins.core.dataloader.DataLoader, X_syn: synthcity.plugins.core.dataloader.DataLoader) Dict
- evaluate_default(X_gt: synthcity.plugins.core.dataloader.DataLoader, X_syn: synthcity.plugins.core.dataloader.DataLoader) float
- classmethod fqdn() str
- abstract static name() str
- reduction() Callable
- static standard_performance_output_keys() List
- static type() str
- use_cache(path: pathlib.Path) bool
- class PerformanceEvaluatorLinear(**kwargs: Any)
Bases:
synthcity.metrics.eval_performance.PerformanceEvaluator
Train a Linear classifier or regressor on the synthetic data and evaluate the performance on real test data.
Returns the average performance discrepancy between training on real data vs on synthetic data.
- Score:
close to 1: similar performance close to 0: massive performance degradation
- static direction() str
- evaluate(X_gt: synthcity.plugins.core.dataloader.DataLoader, X_syn: synthcity.plugins.core.dataloader.DataLoader) Dict
- evaluate_default(X_gt: synthcity.plugins.core.dataloader.DataLoader, X_syn: synthcity.plugins.core.dataloader.DataLoader) float
- classmethod fqdn() str
- static name() str
- reduction() Callable
- static standard_performance_output_keys() List
- static type() str
- use_cache(path: pathlib.Path) bool
- class PerformanceEvaluatorMLP(**kwargs: Any)
Bases:
synthcity.metrics.eval_performance.PerformanceEvaluator
Train a Neural Net classifier or regressor on the synthetic data and evaluate the performance on real test data.
Returns the average performance discrepancy between training on real data vs on synthetic data.
- Score:
close to 1: similar performance close to 1: massive performance degradation
- static direction() str
- evaluate(X_gt: synthcity.plugins.core.dataloader.DataLoader, X_syn: synthcity.plugins.core.dataloader.DataLoader) Dict
- evaluate_default(X_gt: synthcity.plugins.core.dataloader.DataLoader, X_syn: synthcity.plugins.core.dataloader.DataLoader) float
- classmethod fqdn() str
- static name() str
- reduction() Callable
- static standard_performance_output_keys() List
- static type() str
- use_cache(path: pathlib.Path) bool
- class PerformanceEvaluatorXGB(**kwargs: Any)
Bases:
synthcity.metrics.eval_performance.PerformanceEvaluator
Train an XGBoost classifier or regressor on the synthetic data and evaluate the performance on real test data.
Returns the average performance discrepancy between training on real data vs on synthetic data.
- Score:
close to 1: similar performance close to 0: massive performance degradation
- static direction() str
- evaluate(X_gt: synthcity.plugins.core.dataloader.DataLoader, X_syn: synthcity.plugins.core.dataloader.DataLoader) Dict
- evaluate_default(X_gt: synthcity.plugins.core.dataloader.DataLoader, X_syn: synthcity.plugins.core.dataloader.DataLoader) float
- classmethod fqdn() str
- static name() str
- reduction() Callable
- static standard_performance_output_keys() List
- static type() str
- use_cache(path: pathlib.Path) bool