Automated tests can sometimes fail due to transient issues like network instability or timing problems. Retrying failed tests can help mitigate false negatives and provide more stable builds. Let us delve into understanding how JUnit implements retry logic to handle flaky or intermittent test failures effectively.
1. What is JUnit?
JUnit is a widely-used testing framework for Java applications. It allows developers to write repeatable tests and supports features like test annotations, assertions, and test runners.
- JUnit 4: Uses annotations like
@Testand custom runners. - JUnit 5: Introduces a modular architecture with better extension support and new annotations.
2. How to Add Retry Logic in JUnit 5 Tests
JUnit 5 doesn’t provide built-in retry support, but you can create a custom extension by implementing the TestExecutionExceptionHandler and TestWatcher interfaces.
2.1 RetryExtension.java
This class defines a custom JUnit 5 extension that adds retry logic to test methods. It implements two interfaces:
TestExecutionExceptionHandler and BeforeTestExecutionCallback. The extension intercepts test failures and re-executes the test method up to a maximum number of retries if it fails.
import org.junit.jupiter.api.extension.*;
public class RetryExtension implements TestExecutionExceptionHandler, BeforeTestExecutionCallback {
private static final int MAX_RETRIES = 3;
private int retryCount = 0;
@Override
public void handleTestExecutionException(ExtensionContext context, Throwable throwable) throws Throwable {
if (retryCount < MAX_RETRIES) {
retryCount++;
System.out.println("Retrying test " + context.getDisplayName() + ", attempt " + retryCount);
context.getRequiredTestMethod().invoke(context.getRequiredTestInstance());
} else {
throw throwable;
}
}
@Override
public void beforeTestExecution(ExtensionContext context) throws Exception {
retryCount = 0;
}
}
2.2 Sample Test Using Retry
This sample test class demonstrates how to use the RetryExtension with JUnit 5. The test method is designed to fail initially and only pass after a few retries. This simulates flaky test scenarios and verifies that the retry logic works as expected.
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.ExtendWith;
@ExtendWith(RetryExtension.class)
public class JUnit5RetryTest {
private static int counter = 0;
@Test
void testWithRetry() {
counter++;
System.out.println("Test run #" + counter);
if (counter < 3) {
throw new RuntimeException("Failing test to trigger retry");
}
}
}
2.3 Output
When the above test is executed, you will see the following output. It shows that the test failed on the first two attempts and finally passed on the third attempt, demonstrating the retry mechanism in action.
Test run #1 Retrying test testWithRetry(), attempt 1 Test run #2 Retrying test testWithRetry(), attempt 2 Test run #3
3. How to Add Retry Logic in JUnit 4 Tests
JUnit 4 does not offer built-in support for retrying failed tests, but it provides the flexibility to implement custom retry mechanisms using rules. You can define a custom TestRule (or the older MethodRule) that controls test execution and retries upon failure.
3.1 RetryRule.java
This class implements the TestRule interface to provide retry capabilities for test methods in JUnit 4. It wraps the test execution logic in a retry loop and re-runs the test up to the specified number of times until it succeeds or exhausts all attempts.
import org.junit.rules.TestRule;
import org.junit.runner.Description;
import org.junit.runners.model.Statement;
public class RetryRule implements TestRule {
private int retryCount;
public RetryRule(int retryCount) {
this.retryCount = retryCount;
}
public Statement apply(Statement base, Description description) {
return new Statement() {
public void evaluate() throws Throwable {
Throwable caughtThrowable = null;
for (int i = 0; i <= retryCount; i++) {
try {
base.evaluate();
return;
} catch (Throwable t) {
caughtThrowable = t;
System.out.println(description.getDisplayName() + ": run " + (i + 1) + " failed");
}
}
System.out.println(description.getDisplayName() + ": giving up after " + (retryCount + 1) + " failures");
throw caughtThrowable;
}
};
}
}
3.2 Sample Test Using Retry
This test class uses the custom RetryRule to retry a failing test up to two times. The counter variable simulates a flaky test that only passes after multiple attempts. By using the @Rule annotation, the retry logic is automatically applied to the test method.
import org.junit.Rule;
import org.junit.Test;
public class JUnit4RetryTest {
private static int counter = 0;
@Rule
public RetryRule retryRule = new RetryRule(2);
@Test
public void testWithRetry() {
counter++;
System.out.println("Test run #" + counter);
if (counter < 3) {
throw new RuntimeException("Failing test to trigger retry");
}
}
}
3.3 Output
The following output shows how the test fails on the first two runs and succeeds on the third attempt. This demonstrates that the custom retry rule works as intended.
testWithRetry: run 1 failed testWithRetry: run 2 failed Test run #3
4. Guidelines for Reliable Test Retries
- Use retries sparingly – only for flaky or external-dependent tests. Retries are most appropriate when dealing with tests that rely on unstable resources like third-party APIs, UI tests, or asynchronous systems.
- Log retry attempts for better debugging. Always output the retry count, test name, and error message so developers can track which tests are unreliable and how often they fail.
- Avoid retrying tests with side effects unless they are idempotent. If a test modifies shared state (e.g., a database or file), it could lead to inconsistent results across retries unless it is safely repeatable.
- Prefer fixing unstable tests over relying on retries. Retry logic should be a temporary measure. Investigate root causes of test flakiness and resolve them to maintain test reliability and accuracy.
- Use retry settings from configuration if possible. Instead of hardcoding retry limits in code, make them configurable via environment variables or test framework settings for flexibility and environment-specific tuning.
- Combine retries with test reports and dashboards. Visibility is key—track flaky test metrics, visualize retry frequency, and prioritize which tests need attention based on failure patterns.
- Limit total retries to avoid masking frequent failures. Excessive retries can hide real issues. Keep retry attempts low (1–3), and fail fast if a test repeatedly fails.
- Use exponential backoff for service-dependent tests. When retrying tests that hit external systems or APIs, introduce a small delay between attempts to give the service time to stabilize or recover.
5. Conclusion
Adding retry logic to JUnit tests can be helpful in reducing false test failures, especially in CI environments. JUnit 5 uses custom extensions, while JUnit 4 uses rules to implement retries. Always aim to fix flaky tests rather than masking issues through retries.
