Unit Tests, Android Tests, and Code Coverage

Effective testing is one of the biggest challenges in the process of software engineering. Especially testing Android applications can be tricky because the code often calls Android specific methods which a normal unit-test cannot handle.
In this article I want to describe the different types of Android tests and how to measure the combined test coverage.

Android Counter

Android Counter is a very simple Android application that I developed to demonstrate different testing strategies.
It allows to increment and reset a counter displayed on the app's screen. The counter's value is persistent across multiple sessions.

The project can be found here: https://github.com/matthinc/AndroidTestDemo

Android counter consists of only three classes:

Counter.kt contains the counter logic itself:

class Counter(private var value: Int) {
    fun increment() {
        ++value
    }

    fun reset() {
        value = 0
    }

    val counterValue
        get() = value
}

The CounterPersistenceManager.kt stores the value of the counter using SharedPreferences:

class CounterPersistenceManager(private val context: Context) {
    private val preferences: SharedPreferences by lazy {
        context.getSharedPreferences("counter", Context.MODE_PRIVATE)
    }

    fun persistContent(counter: Counter) {
        preferences.edit().putInt(COUNTER_KEY, counter.counterValue).apply()
    }

    fun loadCounter(): Counter {
        return Counter(preferences.getInt(COUNTER_KEY, 0))
    }

    companion object {
        private const val COUNTER_KEY = "counter_value"
    }
}

MainActivity.kt connects the the counter logic to the app's user interface:

class MainActivity : AppCompatActivity() {
    private val counterPersistenceManager by lazy { CounterPersistenceManager(this) }
    private val counter by lazy { counterPersistenceManager.loadCounter() }

    override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)
        setContentView(R.layout.activity_main)

        findViewById<Button>(R.id.increment_button).setOnClickListener {
            counter.increment()
            updateCounter()
        }

        findViewById<Button>(R.id.reset_button).setOnClickListener {
            counter.reset()
            updateCounter()
        }

        updateCounter()
    }

    private fun updateCounter() {
        findViewById<TextView>(R.id.counter_value).text = "${counter.counterValue}"
        counterPersistenceManager.persistContent(counter)
    }
}

Testing the Counter

The class Counter does not contain any Android specific code. That means that it can be tested using a "traditional" unit-test.
Unit-Tests are much faster than Android tests so they should be used when possible.

class CounterTest {

    @Test
    fun testNewCounter() {
        val counter = Counter(0)
        Assert.assertEquals(0, counter.counterValue)
    }

    @Test
    fun testIncrement() {
        val counter = Counter(0)
        counter.increment()
        Assert.assertEquals(1, counter.counterValue)
    }

    @Test
    fun testIncrementInitialValue() {
        val counter = Counter(5)
        counter.increment()
        Assert.assertEquals(6, counter.counterValue)
    }

    @Test
    fun testReset() {
        val counter = Counter(5)
        counter.reset()
        Assert.assertEquals(0, counter.counterValue)
    }
    
}

The coverage of this test can simply be measured by using Android Studio's "Run Test with Coverage" feature. The generated coverage report shows that the class Counter now has 100% test coverage. Great!

Testing CounterPersistenceManager

The CounterPersistenceManager uses Android's SharedPreferences class which requires a Context. That means that a normal unit-test can't be used to test this class without mocking a lot of stuff.
That means that we have to use Android tests. Android tests work like normal unit-tests except that they have to be executed on a real device or an Android emulator. A valid context object can be optained by using

val appContext = InstrumentationRegistry.getInstrumentation().targetContext

The final test looks like that:

@RunWith(AndroidJUnit4::class)
class PersistenceTest {

    @Before
    fun resetCounter() {
        val appContext = InstrumentationRegistry.getInstrumentation().targetContext
        CounterPersistenceManager(appContext).persistContent(Counter(0))
    }

    @Test
    fun testLoadCounter() {
        val appContext = InstrumentationRegistry.getInstrumentation().targetContext

        val manager = CounterPersistenceManager(appContext)
        val counter = manager.loadCounter()

        assertEquals(0, counter.counterValue)
    }

    @Test
    fun testStoreCounter() {
        val appContext = InstrumentationRegistry.getInstrumentation().targetContext

        val counter = Counter(5)

        val manager = CounterPersistenceManager(appContext)
        manager.persistContent(counter)

        val counter2 = manager.loadCounter()
        assertEquals(5, counter2.counterValue)
    }

}

Testing the User Interface

Now, MainActivity remains the last untested class. The best way of testing MainActivity is an UI-test. UI-tests are a special kind of Android tests. Instead of testing the code directly, UI-tests simulate user interaction to the app's UI and check, whether the results match the app's expected behavior.

The most popular library for writing UI-tests is espresso.
It should be integrated into every Android application by default.

Android Counter could be UI-tested like that:

@RunWith(AndroidJUnit4::class)
class UITest {
    @get:Rule
    val activityRule = ActivityScenarioRule(MainActivity::class.java)

    @Before
    fun resetCounter() {
        onView(withId(R.id.reset_button)).perform(click())
    }

    @Test
    fun testIncrement() {
        onView(withId(R.id.counter_value)).check(matches(withText("0")))
        onView(withId(R.id.increment_button)).perform(click())
        onView(withId(R.id.counter_value)).check(matches(withText("1")))
    }

    @Test
    fun testReset() {
        onView(withId(R.id.increment_button)).perform(click())
        onView(withId(R.id.counter_value)).check(matches(withText("1")))
        onView(withId(R.id.reset_button)).perform(click())
        onView(withId(R.id.counter_value)).check(matches(withText("0")))
    }
}

Measuring the Coverage of Android Tests

You might have noticed that the "Run with Coverage" option is not available for Android tests.
The coverage of those tests can be measured by the library jacoco.
Jacoco can be installed by adding the following lines to the project's build.gradle file:

The coverage also has to be enabled in the module's build variants:

buildTypes {
    ...
    debug {
        testCoverageEnabled true
    }
}

This adds a new gradle task called createDebugCoverageReport.
Executing this task creates the new file:

build/reports/coverage/debug/index.html

This html file contains the coverage report for the Android tests. It looks like that:

Combining Unit-Test and Android-Test Coverage

Now we have the coverage of our unit-tests and Android-tests in two different places. It would be much more useful to have one unified coverage report. Luckily, this is also possible using jacoco.

It's archieved using a new Gradle task that runs all unit- and Android-tests and collects the coverage information from both test results.

The task has to be defined in the module's build.gradle file:

plugins {
    ...
    id 'jacoco'
}

task jacocoTestReport(type: JacocoReport, dependsOn: ['testDebugUnitTest', 'createDebugCoverageReport']) {
    reports {
        xml.enabled = true
        html.enabled = true
    }

    def fileFilter = [ '**/R.class', '**/R$*.class', '**/BuildConfig.*', '**/Manifest*.*', '**/*Test*.*', 'android/**/*.*' ]
    def debugTree = fileTree(dir: "$project.buildDir/tmp/kotlin-classes/debug", excludes: fileFilter)
    def mainSrc = "$project.projectDir/src/main/kotlin"

    sourceDirectories.from files([mainSrc])
    classDirectories.from files([debugTree])
    executionData.from fileTree(dir: project.buildDir, includes: [
            'jacoco/testDebugUnitTest.exec',
            'outputs/code_coverage/debugAndroidTest/connected/*coverage.ec'
    ])
}

Running this task creates another test report in

build/reports/jacoco/jacocoTestReport/html

which contains the combined coverage of all tests:

Tests and Coverage using GitHub Actions

For automated testing and coverage calculation we can use GitHub's integrated CI/CD service.
To active GitHub actions for the project we add the following file .github/workflows/android.yml to our project:

name: Android CI

on:
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]

jobs:
  build:

    runs-on: ubuntu-latest

    steps:
    - uses: actions/checkout@v2
    - name: set up JDK 11
      uses: actions/setup-java@v2
      with:
        java-version: '11'
        distribution: 'adopt'

    - name: Grant execute permission for gradlew
      run: chmod +x gradlew
    - name: Build with Gradle
      run: ./gradlew build

This triggers the Android build on every push to the project's main branch.
GitHub itself can't measure test coverage.To do so we need an external coverage service. There are various options available but for this demo, I decided to use Sonarcloud. In addition to test coverage, Sonarcloud can also measure different code-quality metrics which is quite handy.

I created the project android_test in sonarcloud for this demo.

I also created an access token so that our GitHub Action can access this project later.

Now, I added the following build job to android.yml:

  android-test:
  
      runs-on: ubuntu-latest
      
      steps:
        - uses: actions/checkout@v2
        - uses: reactivecircus/android-emulator-runner@v2
          name: Run tests with coverage
          with:
            api-level: 23
            target: default
            arch: x86
            profile: Nexus 6
            script: ./gradlew jacocoTestReport
        - name: Remove build.gradle
          run: rm build.gradle
        - name: SonarCloud Scan
          uses: sonarsource/sonarcloud-github-action@master
          env:
            SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }}
            GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          with:
            args: >
              -Dsonar.coverage.jacoco.xmlReportPaths=app/build/reports/jacoco/jacocoTestReport/jacocoTestReport.xml
              -Dsonar.organization=matthinc
              -Dsonar.projectKey=android_test
              -Dsonar.sources=app/src/main/java

You have to change the parameters sonar.organization and sonar.projectKey to match your Sonarcloud project.

After adding the SONAR_TOKEN as a secret in GitHub's repository settings, the build of the project succeeded.

The coverage for this buid can be checked in Sonarcloud:

Show Comments