Synthetic Data Evaluation Vocabulary Exercises

1 / 5

A ML researcher presents synthetic data evaluation results using the TSTR (Train on Synthetic, Test on Real) framework.

What does the TSTR framework evaluate?

TSTR evaluation framework:

Framework	Protocol	What it measures
TSTR	Train on synthetic → Test on real	Downstream utility (does the synthetic data teach useful patterns?)
TRTS	Train on real → Test on synthetic	Fidelity (does synthetic data resemble real data?)

Key vocabulary: downstream task performance, utility score, TSTR gap, classification accuracy, generalisation, benchmark evaluation.

2 / 5

A data scientist explains: "Our synthetic data has high fidelity but low utility for the downstream classification task."

What does this statement mean?

Fidelity vs. Utility — they can diverge:

Metric type	What it measures	How measured
Fidelity	Statistical similarity to real data	JSD, KS test, correlation matrix comparison
Utility	Performance of models trained on synthetic data	TSTR accuracy, F1 score, AUC

Why the divergence? High-fidelity distributions may still lack the decision-relevant signal needed for the task. Key vocabulary: TSTR gap, downstream utility, evaluation benchmark, task-specific utility.

3 / 5

A privacy auditor asks: "Can a classifier distinguish synthetic from real data?" and sets up a binary classification experiment.

What does this experiment measure, and what result indicates high-quality synthetic data?

Classifier-based fidelity evaluation:

Classifier accuracy	Interpretation
≈ 50%	Cannot distinguish → very high fidelity synthetic data
60–70%	Some detectable differences; moderate fidelity
> 80%	Clear distributional gap; low fidelity

This is also called a discriminative score or detection test. Key vocabulary: discriminative score, detection test, synthetic vs. real classifier, fidelity metric, distributional gap.

4 / 5

A team evaluates synthetic data quality using a utility score computed from multiple downstream tasks.

Which of the following best describes what a "utility score" represents in synthetic data evaluation?

Utility score in synthetic data evaluation:

Component	How it contributes to the utility score
TSTR accuracy	Classification performance on real test data
Regression MSE	Error of models trained on synthetic vs. real data
Query accuracy	Accuracy of aggregate queries on synthetic vs. real data

Key vocabulary: utility score, downstream task performance, machine learning efficacy (MLE), TSTR, synthetic data benchmark, evaluation framework.

5 / 5

A privacy researcher evaluates a synthetic dataset for privacy leakage using a membership inference attack.

What does a membership inference attack test?

Membership inference attack (MIA):

Concept	Explanation
MIA	Attacker queries: "Was Alice's record in the training set?"
Success rate ≈ 50%	Attacker is guessing randomly → good privacy
Success rate > 60%	Attacker can distinguish members → privacy leakage

Related attacks: attribute inference, singling out, linkability attack. Key vocabulary: membership inference, privacy leakage, re-identification risk, singling out, privacy evaluation.