Unit_Test_Concerns.md

Unit Test Concerns

Overview

• The following document poses a unit testing concern observed in a Java 17 codebase leveraging Spring 5.3, JUnit 5.9 and Mockito 5.1.

Details

Scenario

• Suppose you have the following class with a method consisting of a try block who's catch contains a single logging event:

   import org.slf4j.* // Underlying logging framework is logback.classic.Logger
  
   public class MyClass1 {
   	private Logger LOGGER = (Logger) LoggerFactory.getLogger(Logger.ROOT_LOGGER_NAME);
  
   	private void parseDocument(File theFile) {
   		try{
   		   // Some code
   		   var parsedDocument = documentBuilder.parse(theFile); // Can throw IOException
   		} catch(IOException e){
   		   LOGGER.error(LoggingFacade.formatError"ErrorCode", null, null, null, "IOException " + e);
   		}
   	}
   }

• Additionally, you have it's testing file that contains test cases which make assertions using the state of the Logger field:

   class MyClass1Test {
       final Logger logger = (Logger) LoggerFactory = getLogger(Logger.ROOT_LOGGER_NAME);
       @Mock
       private Appender<ILoggingEvents> mockAppender;
       @Captor
       private ArgumentCptor<LoggingEvents> captorLoggingEvents;
       @BeforeEach
       public void setUp { logger.addAppender(mockAppender); }
       @AfterEach
       public void tearDown { logger.detachAppender(mockAppender); }
  
       @Test
       void parseDocumentTest_givenBadDocument_expectIOException() {
   	// Mock decleration
  
   	// class.method under test instantiation
  
   	verify(mockAppender, atLeast(1)).doAppend(captorLoggingEvents.capture());
   	var allValues = captorLoggingEvent.getAllValues();
   	var isIOException = allValues.stream().anyMatch(event -> event.getFormattedMessage().contains("IOException"));
   	assertTrue(isIOException);
       }
   }

• Currently, this implementation works as antcipated and no issues are observed.

Concerns

• The primary concern is that this implementation may introduce non-consistent / indeterministic behavior.
• Suppose a nearly identical class, MyClass2, employs the same logger/assertion approach.
• Consider the possibility that, during build, MyClass1 and MyClass2 test cases execute in parellel. I believe the following pitfalls exist (but unsure if well founded):
  • MyClass1 and MyClass2 are now sharing the state of the Logger field.
  • MyClass1 could interfere with a write/read event for MyClass2 or vice versa.
  • Suppose MyClass1 doesn't execute the catch block for unbeknownst reasons; if MyClass2 logs an IOException, MyClass1 can still successfully assert isIOException.