New way of handling “failing” tests for Jackson

Introduction

In this post, I’ll discuss recent improvements in managing “failing” tests in the Jackson project.

• Discussion : Changing way “failing” tests are implemented, run: test for working condition, except non-pass #243
• Commit : New failing test management using Jupiter extension #4696

Problem : How failing test fail

The Jackson team recently identified a small but annoying issue: failing tests can silently pass without notice. With the widespread use of Jackson projects, there are always so many bugs to fix and features to implement.

For about anything we rely heavily on tests, from bug reproduction to pinpointing specific behaviors . We write tests on the spot — often before knowing the solution — based on how we expect our libraries to behave.

Those tests might look like this, if written in Kotlin and using JUnit and AssertJ.

@Test
fun testThis() {
  // Given
  val input = generateInput()

  // When
  val actual = SUT.run()

  // Then : ***test will fail with AssertionError or sort
  assertThat(actual).isEqaulTo(expected)
}

The problem is, this test will fail, so we can’t include it in the build artifacts. To handle this, we use the maven-sure-fire plugin🔗 and <excludes> feature to exclude failing tests from builds, like so:

 <plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <version>${version.plugin.surefire}</version>
    <artifactId>maven-surefire-plugin</artifactId>
    <configuration>
      <classpathDependencyExcludes>
        <exclude>javax.measure:jsr-275</exclude>
      </classpathDependencyExcludes>
      <excludes>
        <exclude>com.fasterxml.jackson.databind.MapperFootprintTest</exclude>
        <exclude>**/failing/**/*.java</exclude>
      </excludes>
      // ....
      // rest of the config...
      // ...
    </configuration>
  </plugin>

With this setup, failing tests no longer block the build, allowing us to work on bug fixes asynchronously. However, there are downsides to this approach.

The main issue is the <exclude> section of the Surefire plugin. Even though we have failing tests, they aren't fully under our control because they aren't part of the build. If there's an unexpected (but important) change in behavior, the test results could change without us noticing.

This becomes even more problematic during refactors, like the recent property introspection rewrite. We need to know right away when behavior changes, and excluding tests doesn’t provide that feedback.

Additionally, “failing” tests can pile up. It’s impossible to manually track all behavior changes — currently, there are 82 failing tests in the jackson-databind project alone.

Requirements for a New System

So there were these requirements

• Run failing tests expecting them to fail
• Fail the build if a failing test passes
• Execution of failing tests be part of builds
• Keep test modifications minimal (e.g., using annotations)

Solution: New Failing Test Management

The Jackson team introduced a new solution to handle failing tests using JUnit 5 extensions. Here’s how the new system works:

@JacksonTestFailureExpected // <---- NEW!
@Test
fun testThis() {
  // Given
  val input = generateInput()

  // When
  val actual = SUT.run()

  // Then : ***test will fail with AssertionError or sort
  assertThat(actual).isEqaulTo(expected)
}

Almost same as before, but with the new @JacksonTestFailureExpected annotation, failing tests are now executed during project builds, and we are immediately notified if a previously failing test starts passing.

How It Works

We utilized JUnit 5 extension features🔗, specifically the InvocationInterceptor API🔗, to implement this system. InvocationInterceptor defines the API for Extensions that wish to intercept calls to test code.

First we created the annotation to mark the test should fail with an exception like following.

// Annotation to mark on a failing test
@Target({ElementType.METHOD})
@Retention(RetentionPolicy.RUNTIME)
@ExtendWith(JacksonTestFailureExpectedInterceptor.class)
public @interface JacksonTestFailureExpected {
    
}

// Exception to throw when target test suddenly passes
public class JacksonTestShouldFailException
    extends RuntimeException
{
    public JacksonTestShouldFailException(String msg) {
        super(msg);
    }
}

Then we implemented the interceptor like below. The interceptor ensures that tests marked with the @JacksonTestFailureExpected annotation fail if they unexpectedly pass:

public class JacksonTestFailureExpectedInterceptor
    implements InvocationInterceptor
{

    @Override
    public void interceptTestMethod(Invocation<Void> invocation,
             ReflectiveInvocationContext<Method> invocationContext, ExtensionContext extensionContext)
            throws Throwable
    {
        try {
            invocation.proceed(); // run the test
        } catch (Throwable t) {
            // do-nothing, as we expected an exception
            return;
        }
        // WARNING! test invocation did not throw an exception, so
        //          now we need to fail this test.
        handleUnexpectePassingTest(invocationContext);
    }

    private void handleUnexpectePassingTest(ReflectiveInvocationContext<Method> invocationContext) {
        // Collect information we need
        Object targetClass = invocationContext.getTargetClass();
        Object testMethod = invocationContext.getExecutable().getName();
        List<Object> arguments = invocationContext.getArguments();

        // Create message
        String message = String.format("Test method %s.%s() passed, but should have failed", targetClass, testMethod);

        // throw exception
        throw new JacksonTestShouldFailException(message);
    }

}

Super simple, but gets the job done, no need to complicate things!

Results

• The solution ensures that failing tests are always run and alerts us if they unexpectedly pass.
• It uses JUnit 5’s extension API, keeping the implementation simple but effective.

Retrospective

Initially, I considered complex ways to handle failing tests but realized that simplicity works best. The current solution is straightforward and does the job well. JUnit 5’s extension features have proven to be both powerful and easy to use, and I recommend trying them.

Feel free to reach out with questions via my LinkedIn🔗.

Happy Hacking!