Learn the vocabulary of isolating an intermittently failing test without losing its coverage entirely.
0 / 5 completed
1 / 5
At standup, a dev mentions a test that intermittently passes or fails with no change to the underlying code, undermining trust in what a red CI run actually means. What is this kind of test called?
A flaky test intermittently passes or fails with no change to the underlying code, undermining the team's trust in what a red CI run actually signals. A regression test, by contrast, is specifically meant to fail only after a real behavioral change has been introduced, which is exactly the reliable signal a flaky test fails to provide. This intermittent, code-independent failure is what distinguishes a flaky test from a genuinely broken one.
2 / 5
During a design review, the team wants a known flaky test moved into its own tracked suite that doesn't block a pull request's main CI signal, rather than either deleting it outright or leaving it in the main suite. Which capability supports this?
Quarantining the flaky test moves it into its own separately tracked suite that doesn't block a pull request's main CI signal, while still keeping it visible and scheduled for investigation rather than simply deleted. Leaving the flaky test in the main suite keeps blocking every pull request's signal indefinitely, which is exactly the trust problem quarantining is meant to solve. This separate tracking is what lets a team keep CI trustworthy without losing the flaky test's coverage outright.
3 / 5
In a code review, a dev notices a quarantine dashboard listing each flaky test alongside an assigned owner and a deadline to either fix or remove it, rather than a list a team simply forgets about. What does this represent?
Owned, time-bound tracking of a quarantined flaky test assigns a specific owner and a deadline to either fix or remove it, ensuring the quarantine suite doesn't quietly become a permanent graveyard of ignored tests. Quarantining with no owner or deadline assigned leaves that test unaddressed indefinitely, which defeats the purpose of quarantining it in the first place. This accountability is what keeps quarantine a temporary holding area rather than a place tests go to be forgotten.
4 / 5
An incident report shows a genuine regression shipped to production because the team had grown accustomed to ignoring CI failures and simply rerunning until green, since a known flaky test had never been separated from the main suite. What practice would prevent this?
Quarantining a known flaky test into its own tracked suite restores the main CI run's reliability, so a red result there once again means a genuine failure worth investigating. Continuing to leave a known flaky test in the main suite is exactly what trained the team to ignore CI failures in this incident, letting a real regression slip through unnoticed. This separation is essential for keeping a team's trust in its CI signal intact over time.
5 / 5
During a PR review, a teammate asks why the team quarantines a flaky test into a separately tracked suite instead of just deleting it outright once it starts causing trouble. What is the reasoning?
Deleting the test outright loses whatever real coverage it did provide, even if its flakiness currently makes that coverage unreliable to depend on for CI gating. Quarantining preserves that coverage in a non-blocking suite while assigning an owner and a deadline to actually fix the underlying flakiness, rather than losing the test's intent entirely. The tradeoff is the ongoing discipline required to keep the quarantine suite from silently accumulating tests nobody ever circles back to fix.