How to write legible QA tests

Legible and understandable tests are a large result of how your production code is written.

Badly designed code can only get you so far in terms of test legibility and may be what’s preventing you from closing the gap between a regression suite that works just fine and a regression suite that not only works, but is the single source of truth of your application’s functionality — and serves as a source of confidence for subsequent changes.

Unfortunately, even when you master all of the best practices around code design, test design is still relegated to the background and not treated with same care as production code.

I’ve been a quality-minded software engineer for a very long time, and I even worked as a QA engineer for a couple of years. I’ve seen many different approaches to writing tests (be they unit tests or functional tests), but it’s very hard to find tests written with consistency, so I’m going to share with you my best practices for writing legible QA tests.

Conway’s Law

I believe that test writing is the ultimate example of Conway’s Law.

Conway’s Law was coined by Melvin Conway in 1967 and states:

Organizations which design systems … are constrained to produce designs which are copies of the communication structures of these organizations.

Much as how code design reflects the internal communication of an organization, test design mirrors behaviors and intentions.

About this article

In the end, this is a very opinionated article about how to write tests. It is not, by any means, the sole right way of doing things, but I feel that I was able to compile what I consider to be the best tips on how to write tests after many years of writing, observing, and reviewing them.

When I started writing this, my intention was to only write about unit tests, but in the end, most of the advice here can be applied to every type of testing.

If you are having trouble writing tests or if you think you can do a better job, ask yourself the following questions:

Are my test case’s components clear?

A test case is usually composed of these parts:

• Unit/Function under test
• Unit/Function under test dependencies
• Requirements
• Expectations
• Shared states

I’d say that having all those parts distinguishable and distinct from each other is crucial and fundamental for writing better tests. Let’s go through each one with examples:

Unit/Function under test

The unit/function under test is the piece of code that the test is expected to stress and explore. They usually have an input and output, and the interaction between those three components should be very clear. Try to avoid the following patterns:

Calling the unit/function and asserting on the same line

Look at this example:

 Assert.assertEquals("Mr. ABC", stringTitleGiver.apply("ABC"))

The function being called is on the same line as it asserts. This pattern is not only confusing from the standpoint of the assertion, which requires more interpretation than if the assertion was against a variable, but it also obfuscates the most important part of a test case: the unit or function under test. Fixing it is simple:

final String value = stringTitleGiver.apply("ABC");
Assert.assertEquals("Mr. ABC", value);

Creating complex inputs in the function parameters

The idea is always to declutter and streamline information so your brain can understand better the relationship between all the different parts in the test case. This is another example of a common anti-pattern:

final String title = titleGiver.apply(person().name("ABC").address("R. ABC").age(123).build());

You don’t even need to have very complex or elaborate constructs to make the unit/function under test less legible. Once again just assign the input for the function in a separate variable.

final Person person = ImmutablePerson.builder().name("ABC").address("R. ABC").age(123).build();
final String title = titleGiver.apply(person);

More than one when clause in a test case

In the example below two very different test cases are mingled together and may cause the reader to misunderstand the behavior of the production code and what the test case should describe.

final Person personA = person().name("ABC").address("R. ABC").age(123).build();
final Person personB = person().name("Mr. ABC").address("R. ABC").age(123).build();

final String titlePersonA = titleGiver.apply(personA);
final String titlePersonB = titleGiver.apply(personB);

Assert.assertEquals("Mr. ABC", titlePersonA);
Assert.assertEquals("Mr. ABC", titlePersonB);

For this, just break it down to how many test cases are necessary:

final Person person = person().name("ABC").address("R. ABC").age(123).build();

final String title = titleGiver.apply(person);

Assert.assertEquals("Mr. ABC", title);
final Person person = person().name("Mr. ABC").address("R. ABC").age(123).build();

final String title = titleGiver.apply(person);

Assert.assertEquals("Mr. ABC", title);

Unit/Function under test dependencies

Take a look at this excerpt:

private Function<String, Boolean> magicSizeIndicator;

private Function<String, MagicNumberResult> magicOrSad;

@Before
public void setup() {
  this.magicSizeIndicator = word -> (word.trim().length() - 3) % 2 == 0;
  this.magicOrSad = word -> this.magicSizeIndicator.apply(word) ? MAGIC : SAD;
}

@Test
public void isMagic() {
  final MagicNumberResult happy = this.magicOrSad.apply("HAPPY");
  Assert.assertEquals(MAGIC, happy);
}

@Test
public void isSad() {
  final MagicNumberResult happy = this.magicOrSad.apply("DISASTER");
  Assert.assertEquals(SAD, happy);
}

enum MagicNumberResult {
  MAGIC,
  SAD
}

Complexity increases

Complexity increases considerably every time that your tests touches different components. For example, the code above is testing the function that determines if a word is Magic or Sad. It does that by invoking a dependency which indicates if the word has a magic size (whatever that means). The problem with this approach is that you are basically indirectly testing the behavior of the dependency. The number of test cases necessary to cover all possibilities most of the times will be the combination of all the dependencies outcomes. Now imagine there is another dependency which causes side effects as well: the complexity will increase even more and by then your test is easily getting out of hand. More often than not, when it gets to this point, people will just hope for the best instead of adding more test cases.

Let’s rewrite:

private Function<String, Boolean> magicNumberIndicator;

private Function<String, MagicNumberResult> magicOrSad;

@Before
public void setup() {
  this.magicNumberIndicator = Mockito.mock(Function.class);
  this.magicOrSad = word -> this.magicNumberIndicator.apply(word) ? MAGIC : SAD;
}

@Test
public void isMagic() {
  given(this.magicNumberIndicator.apply("HAPPY")).willReturn(true);

  final MagicNumberResult happy = this.magicOrSad.apply("HAPPY");

  Assert.assertEquals(MAGIC, happy);
}

@Test
public void isSad() {
  given(this.magicNumberIndicator.apply("HAPPY")).willReturn(false);

  final MagicNumberResult happy = this.magicOrSad.apply("HAPPY");

  Assert.assertEquals(SAD, happy);
}

enum MagicNumberResult {
  MAGIC,
  SAD
}

By using a mocking framework (in this case mockito), the relationships between dependencies are much better described. The word that is being tested can even be the same, because what is important is that the dependency is indicating if the word has a magic size or not. By describing the relationship between the dependencies like this, you’re able to make its impact on the output of the unit under test much clearer.

Every main programming language has a few options for mocking frameworks to pick from that were designed to solve this particular problem.

Requirements & Expectations

The same logic from the Function/Unit Under Test applies here.

It is very important that the relation between your requirements and expectations are clear.

When writing requirements and expectations try to:

Only write requirements that are really important to the test case

This is probably the tip that most requires writing more than test code. If you are in a situation where you have lots of variables which are not directly linked to the test case you are writing, you may have to look into your production code.

It is also very common to have a function that receives an object which have restrictions for instantiating it, like:

• Mandatory constructor parameters
• Data objects which requires more steps to instantiate

You have to strive to be able to communicate a scenario with as much unrelated code as possible. This is one of those times that you might have to ask if your unit under test is doing too many things at the same time.

Isolate common values in a variable

In general, if it is a String, create a constant for it, but other requirements and expectations should be case by case. My first approach is always to put them in separate variables but sometimes leaving them inlined is much clearer. The advantage of putting common values into variables is that you can give them names, which will reinforce the purpose of that value from a functional point of view.

Describe assertions from the point of view of functionality

Try to add messages to your assertions to help in debugging but also to serve as documentation of the functionality.

Strive to write assertion messages that explains why that assertion in that test case is unique.

For Java, it is possible to use the default Junit API, but I’m a big fan of how complete and expressive AssertJ is.

Shared State

Sometimes it is necessary to write certain behavior that is shared between different test cases. This behavior can occur before and/or after each test case. Consider the rule of thumb to be:

Shared states should be true for EVERY test case.

It is very common to find information relevant to only one test case in a setup method. When this starts to accumulate in a class, understanding the relationship between requirements and expectations becomes near impossible. It is important to mention that test cases should strive to be independent from each other. Even if the same piece of code or information is similar, if not the same, those requirements mean different things for each test case.

Putting it all together

In the end you have to make your test case explain itself as a whole. Expectations must match requirements and dependencies must be clear in how they contribute to the unit under test. How you glue this all together is the final step in writing legible tests.

I would advise to separate all these different parts and be very clear what they are. The naming is important but being consistent with it is much more important. I usually use the following, in order:

• Given – for the requirements
• Orchestration – for the dependencies’s behavior and mocking setup
• When – for the unit/function under test
• Then – for the expectations
• Verification – for the dependencies’s checks

I like having separate sections for mocking because there can be some clutter depending on your production code and the mocking framework of your choice.

In the following example all sections are separated by line comments:

// GIVEN
final Word word = ImmutableWord.word().word(HAPPY).build();

// ORCHESTRATION
given(this.magicNumberIndicator.apply(HAPPY)).willReturn(true);

// WHEN
final MagicNumberResult happy = this.magicOrSad.apply(word);

// THEN
Assert.assertEquals("A magic number not happy should be sad", MAGIC, happy);

// VERIFICATION
verify(this.magicNumberIndicator, only()).apply(HAPPY);
verifyNoMoreInteractions(this.magicNumberIndicator);

I, personally, like using those useless labels in Java for this:

final Word word;
final MagicNumberResult happy;

given:
word = word().word(HAPPY).build();

orchestration:
given(this.magicNumberIndicator.apply(HAPPY)).willReturn(true);

when:
happy = this.magicOrSad.apply(word);

then:
Assert.assertEquals("A magic number not happy should be sad", MAGIC, happy);

verification:
verify(this.magicNumberIndicator, only()).apply(HAPPY);
verifyNoMoreInteractions(this.magicNumberIndicator);

I believe that by using labels, the separation becomes much more notable and hopefully won’t be ignored as much by future developers. Somehow they seem to be more ingrained into the code.

Is my test code obfuscated?

Another common bad practice is to obfuscate the behavior of your test, with:

Test class abstractions and excessive helper methods

There are valid reasons to isolate code that serves your test cases. But as you’d do with your production code, think twice before choosing inheritance over composition.

In Junit for example you can cleanup your test code by creating Rules.

If you believe your test code is too obfuscated, check if there is too much relevant information for the test case outside of the test case definition. It is tempting and very common to see test logic outside of the test case because they are seemingly similar in a lot of test cases, but this can be bad for legibility. This is one of those moments that legibility may trump the usual coding best practices.

As with all other advice in this article, you need to take this on a case-by-case basis, but a good way of doing this is by starting with all logic inside your test case and then moving outside only when you are 100% convinced that it will benefit not only the test case you are writing but also all the other test cases in the given scope.

Are my test cases becoming too repetitive?

It is very common to have test cases that have the same setup and the same assertions but with different requirements and expectations, for example, when testing boundaries and partitions.

In these cases applying data-driven testing is the way to go. Simply put, this pattern allows you to write the test logic separately from its requirements and expectations.

If you are writing higher level tests, then you can go to the omnipresent Cucumber framework, although there are other options out there.

Junit, for example, has this implemented in a feature called parameterized tests.

@RunWith(Parameterized.class)
public class TestCaseParts_ParameterizedTests_ExampleTest {

  private static final String HAPPY = "HAPPY";

  private Function<String, Boolean> magicNumberIndicator;

  private Function<String, MagicNumberResult> magicOrSad;

  private boolean inputWordMagicNumber;

  private MagicNumberResult expectedResult;

  public TestCaseParts_ParameterizedTests_ExampleTest(
      final boolean inputWordMagicNumber, final MagicNumberResult expectedResult) {
    this.inputWordMagicNumber = inputWordMagicNumber;
    this.expectedResult = expectedResult;
  }

  @Before
  public void setup() {
    this.magicNumberIndicator = Mockito.mock(Function.class);
    this.magicOrSad = word -> this.magicNumberIndicator.apply(word) ? MAGIC : SAD;
  }

  @Parameterized.Parameters(name = "{index}: Should be {1} when word is magic ({0})")
  public static Collection<Object[]> data() {
    return Arrays.asList(new Object[][] );
  }

  @Test
  public void isMagicOrSad() {
    orchestration:
    given(this.magicNumberIndicator.apply(HAPPY)).willReturn(inputWordMagicNumber);

    when:
    final MagicNumberResult happy = this.magicOrSad.apply(HAPPY);

    then:
    Assert.assertEquals(expectedResult, happy);
  }

  enum MagicNumberResult {
    MAGIC,
    SAD
  }
}

There are alternatives to Junit parameterized tests, including a refined version in Junit 5. I think this topic is very interesting, and I even developed an alternative myself, called Cheesecakes. I will explore this topic later in another article.

The most important thing with this approach is, again, to make the relation between input and output very clear, but also to make the differences between each test case standout.

Are my test cases too hard to write?

This usually means you need to go back to the drawing board. Your problem may be something related to the test apparatus, but more often than not it is related to how your production code was designed.

Bad production code is harder to test

Look into the SOLID principles for refactoring and starti to break your code into smaller parts.

Are my mocks too complex?

First of all…

Don’t be afraid of using mocking frameworks on your tests

Mocking frameworks help some of the principles I’ve talked about really shine. They help:

• Establish the relationship between input and outputs
• Establish the relationship between the Unit/Function under test and its dependencies

But mocks can get out of hand very quickly, and it is not uncommon to see mocks used for everything in a test case. The rule of thumb here is:

You only mock dependencies

It is very common to see people mocking data objects and that’s usually a code smell. Complex data objects may make your tests harder to write, understand, and maintain, which will usually be amplified when it comes to the production code.

Try to define nice and clean objects to be passed around in your API.

Conclusion

Making legibility the most important factor when writing tests will:

• Help you make better decisions on code design
• Improve maintainability
• Improve communication
• Create confidence over subsequent changes
• Serve as a single source of truth for functionalities
• Allow tests to scale for all layers

I hope this was helpful and I would love to hear any opinions and suggestions.

Fernando Chovich Correa is a Software Engineer at Codacy. Over his 13 year career, he has developed a taste for quality-driven development and Kanban management. In his personal time he likes to record and play music.