Synthetic Data Evaluation Vocabulary Exercises

1 / 5

A team evaluates a synthetic dataset on two axes: fidelity and utility.

Which definitions are correct?

Fidelity vs. utility — two distinct evaluation axes:

Dimension	What it measures	How evaluated
Fidelity	Statistical similarity to real data	KS test, JSD, correlation matrix comparison
Utility	Downstream task performance	TSTR, TRTS evaluation frameworks
Privacy	Protection against re-identification	Membership inference attack, distance to nearest neighbour

Key vocabulary: statistical fidelity, downstream utility, privacy-utility-fidelity triangle.

2 / 5

A team runs a TSTR (Train on Synthetic, Test on Real) evaluation.

What does TSTR measure and why is it used?

TSTR evaluation framework:

Acronym	Train set	Test set	Measures
TSTR	Synthetic	Real	Utility of synthetic data for ML
TRTS	Real	Synthetic	Fidelity of synthetic distribution
TSTR ≈ TRTR	Both similar	—	High utility: synthetic ≈ real as training data

Key vocabulary: utility gap, held-out evaluation, TSTR score, downstream ML task.

3 / 5

A report uses the Kolmogorov-Smirnov (KS) test to compare real and synthetic feature distributions.

What does the KS test measure in this context?

Kolmogorov-Smirnov (KS) test for distribution comparison:

Metric	What it compares	Interpretation
KS statistic	Max difference between CDFs	0 = identical; 1 = completely different
Jensen-Shannon Divergence	Symmetric divergence between distributions	0 = same; 1 = maximally different (log-2 base)
Wasserstein distance	Earth mover's distance	Used in GAN training loss

Key vocabulary: marginal distribution, joint distribution fidelity, correlation preservation, per-column fidelity report.

4 / 5

A privacy audit runs a membership inference attack (MIA) against a synthetic dataset.

What does a membership inference attack test for synthetic data?

Membership inference attack (MIA) as a privacy metric:

Attack outcome	Privacy implication
Adversary accuracy ≈ 50%	Good — no better than guessing; strong privacy
Adversary accuracy > 50%	Concerning — model has memorised some training data

Other privacy metrics: nearest neighbour distance ratio (NNDR), attribute inference attack, singling-out risk, disclosure risk. Key vocabulary: privacy audit, adversarial evaluation, re-identification risk score.

5 / 5

A synthetic data report shows high fidelity but low utility.

What does this combination reveal about the synthetic dataset?

Fidelity ≠ utility — they measure different things:

Scenario	What it means
High fidelity, high utility	Ideal — data looks real and trains good models
High fidelity, low utility	Marginal distributions match but decision-relevant structure (e.g. class boundaries) is lost
Low fidelity, high utility	Rare — model still learns useful patterns despite imperfect statistics

Key vocabulary: class boundary preservation, rare event representation, conditional distribution fidelity, task-specific utility evaluation.