STM32F4 – Unit Testing with CppUTest or GoogleTest [Part I.]

Embedded software testing can be complicated, because (unit) test harness is often unavailable on a native system. Consequently software is tested on the x86 system due to convenience. However, the difference between the test- and the target system opens a can of worms. To avoid potential problems I provide:

  • A project template with native unit testing support;
  • It includes:
    • CppUTest harness with memory leak detection;
    • GoogleTest harness;
    • Code coverage, execution time- and code size profilers.

AustinPowers-Testing1. Introduction

We catch software bugs using testing. The basic testing principle is to apply an input to software under test, to observe the software response and to check for the response acceptability. The hardest part is to select a good set of test inputs and acceptability checks (see CS258 Software Testing Course). We increase confidence in the software, if a test succeeds. Otherwise, we just caught a bug in one of the following.

  • Software under test;
  • Software specification;
  • Acceptability check (indeed, we caught a bug in the test);
  • Compiler, Libraries, OS, Hardware (can be buggy, too!).

In embedded software development things are even more complicated, because test harness is often unavailable on native ARM system. Consequently software is tested on the x86 system due to convenience. However, the difference between the test- and the target system opens a can of worms [1, 2, 3], because source code is complied with different compiler, linked against different libraries as well as underlying hardware and CPU is different. To avoid potential problems I provide:

2. Prerequisites

  • Ubuntu 14.04 LTS (x86 architecture).
  • STM32F4 Discovery Board (ARM architecture, costs less than 20 EUR).
  • Complete the mbed Your Code: Zero Switching Costs When Changing ARM Platforms tutorial.
    • Or at least go through chapter 2. Below is a quick-and-dirty instruction summary if you decide to stay on this page.
      cd ~
      # Remove the official package
      sudo apt-get purge binutils-arm-none-eabi \
                         gcc-arm-none-eabi \
                         gdb-arm-none-eabi \
                         libnewlib-arm-none-eabi
      
      # Add 3rd party repository
      sudo add-apt-repository ppa:terry.guo/gcc-arm-embedded
      sudo apt-get update
      # Check the GCC package version in the PPA repository
      sudo apt-cache policy gcc-arm-none-eabi
      
      # Install software requirements
      sudo apt-get install build-essential git openocd \
       gcc-arm-none-eabi qemu-system-arm \
       symlinks expect
      
      # Clone my git repository and init submodules
      git clone https://github.com/istarc/stm32.git
      cd ~/stm32
      git submodule update --init
      

3. CppUTest Project Template

3.1 Basic Usage

The CppUTest project template is easy to use. Just follow these steps to create and manage the template.

  1. Create new (and empty) project folder.
    mkdir -p ~/stm32/examples/cpput
    cd ~/stm32/examples/cpput
    
  2. Run the wizard to initialize a new CppUTest project. Pitfall: Run the script from the new project directory, otherwise it will not work as intended.
    export PATH=$PATH:~/stm32/mbed-project-wizard
    
    gen-stm32f407-GCC-project.sh mbed-none-shcpput
     
    # The project file organization
    #.
    #│   # Application File Organization
    #├── bin # Binary files
    #├── inc # Add your header files here
    #├── src # Add your source files here
    #├── lib # 3rd party source code
    #├── Makefile   # Application Makefile (DON'T TOUCH)
    #├── deploy.cfg # OpenOCD deploy script (DON'T TOUCH)
    #├── gprof.cfg  # Exec. time prof. script (DON'T TOUCH)
    #│
    #│   # Test Harness File Organization
    #├── test-bin       # Binary files
    #├── test-src       # Add your test suite here
    #├── test-cpputest  # 3rd party source code (CppUTest)
    #├── Makefile-test  # Unit test Makefile (DON'T TOUCH)
    #├── test-gprof.cfg # Exec. time prof. script (DON'T TOUCH)
    #├── check.cfg # OpenOCD Deploy Script (DON'T TOUCH)
    #└── check.exp # Expect Test Auto. Script (DON't TOUCH)
     
  3. It should generate the above project directory structure, which shows the relevant application and test harness file organization.
    1. The application file organization is in-depth described here. The project template is extensible. You may add or remove source file in the appropriate folders and even create a custom sub-directory hierarchy. The Makefile script finds source file automagically and builds them. Don’t touch the Makefile, unless really necessary.
    2. The test harness file organization includes CppUTest source code. Use test-src folder to create the application’s test suite. The Makefile-test script automatically finds the application’s source files (main.c or main.cpp are excluded). Don’t touch the Makefile-test script, unless really necessary.
  4. Build and deploy the application (see src/main.cpp) as follows.
    make clean
    make -j4 # Size optimized build
    # make -j4 debug # Debug build
    sudo make deploy
    # Press Ctrl+C to interrupt OpenOCD connection.
    
  5. The application prints “Hello World!” messages via ARM semihosting and toggles LED. The toggling/printing time-delay is increased using the add function (see src/add.cpp or the below source). The function takes two integers as an input, calculates absolute values and adds them.
    // Usage:
    // int i = 0;
    // i = add(i, 1);
    
    #include "add.h"
    
    int add(int x, int y) {
        if(x<0)
            x=-x;
        if(y<0)
            y=-y;
        return x+y;
    }
    
  6. The corresponding unit test is located in test-src/test-add.cpp file. It should verify the addition and absolute value operations (actually it doesn’t ;-), see the Section 3.2).
    #include "CppUTest/TestHarness.h"
    #include "add.h"
    
    TEST_GROUP(SecondTestGroup) {
    };
    
    TEST(SecondTestGroup, TestAdd) {
     LONGS_EQUAL(add(0,0), 0);
     LONGS_EQUAL(add(0,1), 1);
     LONGS_EQUAL(add(1,0), 1);
     LONGS_EQUAL(add(1,1), 2);
    }
    
  7. Build the test cases, deploy them on the native hardware and check the application. Caveat: Disable the code coverage profiler to reduce the binary size and to speed up the testing considerably (see test-src/test-main.cpp).
    make test-clean
    make test-deps  # Build CppUTest Library
    make -j4 test   # Build Unit Tests
    sudo make check # Check the Application
    
    # --- Test Start ---
    # TEST(FirstTestGroup, FirstTest) - 0 ms
    # TEST(SecondTestGroup, TestAdd) - 0 ms
    # TEST(ThirdTestGroup, TestDAdd) - 0 ms
    #
    # OK (3 tests, 3 ran, 8 checks, 0 ignored,
    #                 0 filtered out, 2000 ms)
    # --- Test End ---
    # Test Results: Success.
    
  8. The sudo make check exit code is 0 if successful, 1 if test execution fails (e.g. due to timeout in 240 seconds; see check.exp to adjust this value), 2 if one or more unit test failed. These exit codes can be used directly with Buildbot or Gitlab-CI test automation.

3.2 Code Coverage Profiling

HolesCode coverage profiler can be used to identify untested application code as follows.

make test-clean
make test-deps  # Build CppUTest Library
make -j4 test   # Build Unit Tests
sudo make check # Wait 2-3 minutes
make check-exec-coverage # Show the results

# File 'src/add.cpp'
# Lines executed:66.67% of 6
# Creating 'add.cpp.gcov'
# ---
#    Count  Line
#        -:    1:#include "add.h"
#        -:    2:
#        4:    3:int add(int x, int y)
#        -:    4:{
#        4:    5:    if(x<0)
#    #####:    6:        x=-x;
#        4:    7:    if(y<0)
#    #####:    8:        y=-y;
#        4:    9:    return x+y;
#        -:   10:}

The results show that conditional statements are not tested (see lines 6 and 8). The corresponding test fixture (see test-src/test-add.cpp) should be improved by adding two test inputs.

LONGS_EQUAL(add(0,-1), 1);
LONGS_EQUAL(add(-1,0), 1);

Caveat: The profiler is enabled by default (see test-src/test-main.cpp), but you may disable it to decrease the binary size and to speed up testing considerably.

3.3 Execution Time Profiling

LegDayExecution time profiler is used to identify where CPU spends most of its time to optimize bottlenecks. The profiler samples CPU’s program counter address to calculate execution time statistic and to determine corresponding function names.

The profiler usage is demonstrated below. The results show that the application spends most of the time in the wait function (see src/main.cpp), which is correct behaviour.

make clean
make -j4
sudo make profile-exec-time

# Flat profile:
#
# Each sample counts as 0.01 seconds.
# %     cum. self
# time  sec. sec. name
# 58.30 1.44 1.44 us_ticker_read
# 33.20 2.26 0.82 wait_us
# 6.48  2.42 0.16 us_ticker_init
# 0.81  2.44 0.02 initialise_monitor_handles
# 0.40  2.45 0.01 Reset_Handler
# 0.40  2.46 0.01 _swistat
# 0.40  2.47 0.01 _swiwrite

If necessary, the profiler can be used with unit testing as follows.

make test-clean
make test-deps
make -j4 test
sudo make check-exec-time

3.4 Code Size Profiling

TooBigCode size profiler is used to identify memory consuming symbols. Suppose the application (see src/main.cpp) records the history of LED events as indicated below.

#include "mbed.h"
#include "add.h"

int history[10000]; // Wasted space
DigitalOut myled(LED1);

int main() {
    int i = 0;
    while(1) {
        i = add(i,1);
        myled = i%3;
        if(i < 10000)
            history[i] = myled;
        wait(i%10);
    }
}

Then the profiler correctly shows the most memory consuming symbol, which is the prime candidate for optimization, and its location in the source code.

make clean
make -j4
make profile-code-size

# Top 10 space consuming symbols from the object code ...
#  1 bin/outp.elf:00040000 B history
#  2 bin/outp.elf:00007396 t d_print_comp
#  3 bin/outp.elf:00006356 t d_array_type
#  ...
#
# And corresponging source files to blame.
# 1 main.cpp:?
# 2 cp-demangle.c:?
# 3 cp-demangle.c:?

If necessary, the profiler can be used with unit testing as follows.

make test-clean
make test-deps
make -j4 test
sudo make check-code-size

4. Stay Tuned …

In the following posts I’ll describe:

  • Memory leak detection using CppUTest;
  • GoogleTest project template;
  • Unit testing on QEMU emulator instead of native system;
  • How to use GoogleMocks with GoogleTest or CppUTest.
Advertisements

About istarc

Embedded Systems Developer.
This entry was posted in Embedded Systems, STM32F4 and tagged , , , , , , , . Bookmark the permalink.

7 Responses to STM32F4 – Unit Testing with CppUTest or GoogleTest [Part I.]

  1. clarkli86 says:

    Hi istarc,

    Fantastic article!

    Have you successfully got GoogleTest to work on bare-metal targets? It is the test framework I prefer over CPPUnit.

    Cheers,

  2. Alex says:

    Hi istarc, really interesting article! Any plans on when you will post about Google Test and Google Mock?

    • istarc says:

      Thx, Alex! 🙂 Currently, I am really busy and don’t have much time to write the part II., but I’ll try to complete it in a few months. Best regards, Iztok.

  3. Pingback: Build googletest for Baremetal targets (stm32) | clarkli86

  4. Hannes says:

    Hi Iztok! Very good work!! I am looking forward to read your future posts.
    Regards
    Hannes

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s