Flaky test tracking in Kotlin with JUnit

Example setup with Kotlin, JUnit, Github Actions, and Gradle

We will be setting up a Kotlin project with Gradle, Github Actions for CI, and JUnit for our tests. A demo repo is here if you want to see the full code.

kotlin_example

Overview of the steps:

  1. New Gradle project
  2. Add a test
  3. Run gradlew test
  4. Verify report xml has been generated in your build directory
  5. Setup our test script
  6. Setup Github Actions
  7. Sample action
  8. Update your TR_UPLOAD_TOKEN
  9. Push your changes

1) New Gradle project

Either create a new project or use an existing one, our sample app used Intellij for creating a new one. As long as you are able to create and run tests with gradlew test then you should be good to go.

2) Add a test

Create a simple test if you do not already have one, this is our hello world test

// src/test/kotlin/HelloTest.kt
import org.junit.jupiter.api.Assertions.assertEquals
import org.junit.jupiter.api.Test

class HelloTest {

  @Test
  fun simplest() {
    assertEquals(2, 1 + 1, "1 + 1 should equal 2")
  }
}

3) Run gradlew test

Run gradlew test hopefully your test suite passes, but either a pass or fail will work. As long as it runs we can go on to the next step.

4) Verify report xml has been generated in your build directory

The default location for test results is in the build directory of your app. There should be an xml file build/test-results/test/TEST-HelloTest.xml if you created the hello world test, as well as other tests in the build/test-results/test/ directory if you have other tests.

5) Setup our test script

The test results need to be uploaded after the test suite has finished running, regardless of pass or fail. There are many ways to achieve this, a simple one is to use a test script like this using trap:

#!/bin/bash

set -e
export TR_UPLOAD_TOKEN="<add_your_token_here>"
trap 'testrecall-reporter' EXIT

./gradlew test

6) Setup Github Actions

Here is an example github action:


	name: Java CI

	on: [push]

	jobs:
	 build:
	 runs-on: ubuntu-latest

	 steps:
	 - uses: actions/checkout@v2

	 - name: Set up java
	 uses: actions/setup-java@v1
	 with:
	 java-version: 11

	 - name: Setup test dependencies
	 run: |
	 curl -sL https://get.testrecall.com/reporter | bash

	 - name: Build with Gradle
	 run: ./test.sh

To start we check out our code and set up Java. Afterward, we install the TestRecall reporter, this makes the testrecall-reporter command available. Lastly, we call invoke our test.sh script from earlier. This runs our tests and uses trap to invoke testrecall-reporter.

7) Update your TR_UPLOAD_TOKEN

Our example used <add_your_token_here>, replace this with the value from your TestRecall project.

8) Push your changes

With everything set up, push your changes and you should see your test results in TestRecall.

kotlin_example