synthcity.metrics.eval_privacy module

class DeltaPresence(**kwargs: Any)

Bases: synthcity.metrics.eval_privacy.PrivacyEvaluator

Inheritance diagram of synthcity.metrics.eval_privacy.DeltaPresence

Returns the maximum re-identification probability on the real dataset from the synthetic dataset.

For each dataset partition, we report the maximum ratio of unique sensitive information between the real dataset and in the synthetic dataset.

static direction() → str
evaluate(X_gt: synthcity.plugins.core.dataloader.DataLoader, X_syn: synthcity.plugins.core.dataloader.DataLoader, *args: Any, **kwargs: Any) → Dict
evaluate_default(X_gt: synthcity.plugins.core.dataloader.DataLoader, X_syn: synthcity.plugins.core.dataloader.DataLoader) → float
classmethod fqdn() → str
static name() → str
reduction() → Callable
static type() → str
use_cache(path: pathlib.Path) → bool
class DomiasMIA(**kwargs: Any)

Bases: synthcity.metrics.eval_privacy.PrivacyEvaluator

DOMIAS is a membership inference attacker model against synthetic data, that incorporates density estimation to detect generative model overfitting. That is it uses local overfitting to detect whether a data point was used to train the generative model or not.

Returns: A dictionary with a key for each of the synthetic_sizes values. For each synthetic_sizes value, the dictionary contains the keys:

  • MIA_performance : accuracy and AUCROC for each attack

  • MIA_scores: output scores for each attack

Reference: Boris van Breugel, Hao Sun, Zhaozhi Qian, Mihaela van der Schaar, AISTATS 2023. DOMIAS: Membership Inference Attacks against Synthetic Data through Overfitting Detection.

static direction() → str
evaluate(X_gt: synthcity.plugins.core.dataloader.DataLoader, X_syn: synthcity.plugins.core.dataloader.DataLoader, *args: Any, **kwargs: Any) → Dict
evaluate_default(X_gt: synthcity.plugins.core.dataloader.DataLoader, X_syn: synthcity.plugins.core.dataloader.DataLoader, X_train: synthcity.plugins.core.dataloader.DataLoader, X_ref_syn: synthcity.plugins.core.dataloader.DataLoader, reference_size: int) → float
abstract evaluate_p_R(synth_set: Union[synthcity.plugins.core.dataloader.DataLoader, Any], synth_val_set: Union[synthcity.plugins.core.dataloader.DataLoader, Any], reference_set: numpy.ndarray, X_test: numpy.ndarray, device: Any) → Any
classmethod fqdn() → str
static name() → str
reduction() → Callable
static type() → str
use_cache(path: pathlib.Path) → bool
class DomiasMIABNAF(**kwargs: Any)

Bases: synthcity.metrics.eval_privacy.DomiasMIA

static direction() → str
evaluate(X_gt: synthcity.plugins.core.dataloader.DataLoader, X_syn: synthcity.plugins.core.dataloader.DataLoader, *args: Any, **kwargs: Any) → Dict
evaluate_default(X_gt: synthcity.plugins.core.dataloader.DataLoader, X_syn: synthcity.plugins.core.dataloader.DataLoader, X_train: synthcity.plugins.core.dataloader.DataLoader, X_ref_syn: synthcity.plugins.core.dataloader.DataLoader, reference_size: int) → float
evaluate_p_R(synth_set: Union[synthcity.plugins.core.dataloader.DataLoader, Any], synth_val_set: Union[synthcity.plugins.core.dataloader.DataLoader, Any], reference_set: numpy.ndarray, X_test: numpy.ndarray, device: Any) → Tuple[numpy.ndarray, numpy.ndarray]
classmethod fqdn() → str
static name() → str
reduction() → Callable
static type() → str
use_cache(path: pathlib.Path) → bool
class DomiasMIAKDE(**kwargs: Any)

Bases: synthcity.metrics.eval_privacy.DomiasMIA

static direction() → str
evaluate(X_gt: synthcity.plugins.core.dataloader.DataLoader, X_syn: synthcity.plugins.core.dataloader.DataLoader, *args: Any, **kwargs: Any) → Dict
evaluate_default(X_gt: synthcity.plugins.core.dataloader.DataLoader, X_syn: synthcity.plugins.core.dataloader.DataLoader, X_train: synthcity.plugins.core.dataloader.DataLoader, X_ref_syn: synthcity.plugins.core.dataloader.DataLoader, reference_size: int) → float
evaluate_p_R(synth_set: Union[synthcity.plugins.core.dataloader.DataLoader, Any], synth_val_set: Union[synthcity.plugins.core.dataloader.DataLoader, Any], reference_set: numpy.ndarray, X_test: numpy.ndarray, device: Any) → Tuple[numpy.ndarray, numpy.ndarray]
classmethod fqdn() → str
static name() → str
reduction() → Callable
static type() → str
use_cache(path: pathlib.Path) → bool
class DomiasMIAPrior(**kwargs: Any)

Bases: synthcity.metrics.eval_privacy.DomiasMIA

static direction() → str
evaluate(X_gt: synthcity.plugins.core.dataloader.DataLoader, X_syn: synthcity.plugins.core.dataloader.DataLoader, *args: Any, **kwargs: Any) → Dict
evaluate_default(X_gt: synthcity.plugins.core.dataloader.DataLoader, X_syn: synthcity.plugins.core.dataloader.DataLoader, X_train: synthcity.plugins.core.dataloader.DataLoader, X_ref_syn: synthcity.plugins.core.dataloader.DataLoader, reference_size: int) → float
evaluate_p_R(synth_set: Union[synthcity.plugins.core.dataloader.DataLoader, Any], synth_val_set: Union[synthcity.plugins.core.dataloader.DataLoader, Any], reference_set: numpy.ndarray, X_test: numpy.ndarray, device: Any) → Tuple[numpy.ndarray, numpy.ndarray]
classmethod fqdn() → str
static name() → str
reduction() → Callable
static type() → str
use_cache(path: pathlib.Path) → bool
class IdentifiabilityScore(**kwargs: Any)

Bases: synthcity.metrics.eval_privacy.PrivacyEvaluator

Inheritance diagram of synthcity.metrics.eval_privacy.IdentifiabilityScore

Returns the re-identification score on the real dataset from the synthetic dataset.

We estimate the risk of re-identifying any real data point using synthetic data. Intuitively, if the synthetic data are very close to the real data, the re-identification risk would be high. The precise formulation of the re-identification score is given in the reference below.

Reference: Jinsung Yoon, Lydia N. Drumright, Mihaela van der Schaar, “Anonymization through Data Synthesis using Generative Adversarial Networks (ADS-GAN): A harmonizing advancement for AI in medicine,” IEEE Journal of Biomedical and Health Informatics (JBHI), 2019. Paper link: https://ieeexplore.ieee.org/document/9034117

static direction() → str
evaluate(X_gt: synthcity.plugins.core.dataloader.DataLoader, X_syn: synthcity.plugins.core.dataloader.DataLoader, *args: Any, **kwargs: Any) → Dict
evaluate_default(X_gt: synthcity.plugins.core.dataloader.DataLoader, X_syn: synthcity.plugins.core.dataloader.DataLoader) → float
classmethod fqdn() → str
static name() → str
reduction() → Callable
static type() → str
use_cache(path: pathlib.Path) → bool
class PrivacyEvaluator(**kwargs: Any)

Bases: synthcity.metrics.core.metric.MetricEvaluator

Inheritance diagram of synthcity.metrics.eval_privacy.PrivacyEvaluator
abstract static direction() → str
evaluate(X_gt: synthcity.plugins.core.dataloader.DataLoader, X_syn: synthcity.plugins.core.dataloader.DataLoader, *args: Any, **kwargs: Any) → Dict
evaluate_default(X_gt: synthcity.plugins.core.dataloader.DataLoader, X_syn: synthcity.plugins.core.dataloader.DataLoader) → float
classmethod fqdn() → str
abstract static name() → str
reduction() → Callable
static type() → str
use_cache(path: pathlib.Path) → bool
class kAnonymization(**kwargs: Any)

Bases: synthcity.metrics.eval_privacy.PrivacyEvaluator

Inheritance diagram of synthcity.metrics.eval_privacy.kAnonymization

Returns the k-anon ratio between the real data and the synthetic data. For each dataset, it is computed the value k which satisfies the k-anonymity rule: each record is similar to at least another k-1 other records on the potentially identifying variables.

static direction() → str
evaluate(X_gt: synthcity.plugins.core.dataloader.DataLoader, X_syn: synthcity.plugins.core.dataloader.DataLoader, *args: Any, **kwargs: Any) → Dict
evaluate_data(X: synthcity.plugins.core.dataloader.DataLoader) → int
evaluate_default(X_gt: synthcity.plugins.core.dataloader.DataLoader, X_syn: synthcity.plugins.core.dataloader.DataLoader) → float
classmethod fqdn() → str
static name() → str
reduction() → Callable
static type() → str
use_cache(path: pathlib.Path) → bool
class kMap(**kwargs: Any)

Bases: synthcity.metrics.eval_privacy.PrivacyEvaluator

Inheritance diagram of synthcity.metrics.eval_privacy.kMap

Returns the minimum value k that satisfies the k-map rule.

The data satisfies k-map if every combination of values for the quasi-identifiers appears at least k times in the reidentification(synthetic) dataset.

static direction() → str
evaluate(X_gt: synthcity.plugins.core.dataloader.DataLoader, X_syn: synthcity.plugins.core.dataloader.DataLoader, *args: Any, **kwargs: Any) → Dict
evaluate_default(X_gt: synthcity.plugins.core.dataloader.DataLoader, X_syn: synthcity.plugins.core.dataloader.DataLoader) → float
classmethod fqdn() → str
static name() → str
reduction() → Callable
static type() → str
use_cache(path: pathlib.Path) → bool
class lDiversityDistinct(**kwargs: Any)

Bases: synthcity.metrics.eval_privacy.PrivacyEvaluator

Inheritance diagram of synthcity.metrics.eval_privacy.lDiversityDistinct

Returns the distinct l-diversity ratio between the real data and the synthetic data.

For each dataset, it computes the minimum value l which satisfies the distinct l-diversity rule: every generalized block has to contain at least l different sensitive values.

We simulate a set of the cluster over the dataset, and we return the minimum length of unique sensitive values for any cluster.

static direction() → str
evaluate(X_gt: synthcity.plugins.core.dataloader.DataLoader, X_syn: synthcity.plugins.core.dataloader.DataLoader, *args: Any, **kwargs: Any) → Dict
evaluate_data(X: synthcity.plugins.core.dataloader.DataLoader) → int
evaluate_default(X_gt: synthcity.plugins.core.dataloader.DataLoader, X_syn: synthcity.plugins.core.dataloader.DataLoader) → float
classmethod fqdn() → str
static name() → str
reduction() → Callable
static type() → str
use_cache(path: pathlib.Path) → bool