STM32F4 – On Hunting Bugs with a Profiler

Villains sometimes misbehave and cause troubles. Luckily, Batman and Robin are constantly on the move. They monitor villains’ behaviour and act, if necessary.

Batman-Villains Batman-Surveillance Batman-Pow

Our code for embedded systems can misbehave, too. Even though it builds without errors and passes testing. Do the following run-time problems in a multitasking environment sound familiar to you?

  • An erratic error occurs that leads to a system failure. It is hard to reproduce the error.
  • A program behaves inappropriately. It is hogging CPU or memory resources. It drains more energy than anticipated. It decreases the autonomy of the system.

Can we use the same principle as Batman and Robin to hunt bugs? What tools do we have at our disposal?

  • In-circuit debugger provides insight into a program’s execution at the instruction-level. Unfortunately, this scope is too narrow to easily spot the problems that typically become apparent over time (like the above).
  • This is where the Tracealyzer for FreeRTOS comes in. It provides run-time analytic to combat the problems above …
Batman-OnlyFoolsAndHorses

… so you can party later in the evening. πŸ˜‰

In this tutorial, you will learn how to use the Tracealyzer to identify a “run-time bug” that hogs CPU on a simple FreeRTOS application example.

  1. I explain the big picture.
    1. I describe the example FreeRTOS application.
    2. I show how to identify the bug using Tracealyzer.
    3. I show how to fix the bug.
  2. I provide instructions to complete the tutorial on your home computer.
  3. I provide instructions to setup the environment from scratch in less than 10 minutes. 😎

1. Prerequisites

  • Ubuntu 14.04 LTS (x86_64 architecture).
  • Windows XP/Vista/7/8/8.1 (to run the Tracealyzer).
    • Update: In a few weeks, the upcoming Tracealyzer version will support Linux.
  • STM32F4 Discovery Board (ARM architecture, costs less than 20 EUR).
  • Complete the Behold the Project Wizard! and In-circuit Debugging tutorials.
    • Or at least go through this quick-and-dirty instruction summary if you decide to stay on the page.
      cd ~
      # Remove the official package
      sudo apt-get purge binutils-arm-none-eabi \
                         gcc-arm-none-eabi \
                         gdb-arm-none-eabi \
                         libnewlib-arm-none-eabi
      
      # Add 3rd party repository
      sudo add-apt-repository ppa:terry.guo/gcc-arm-embedded
      sudo apt-get update
      # Check the GCC package version in the PPA repository
      sudo apt-cache policy gcc-arm-none-eabi
      
      # Install software requirements
      sudo apt-get install build-essential git openocd \
      Β Β Β Β Β Β Β Β Β     gcc-arm-none-eabi qemu-system-arm \
                   symlinks expect
      
      # Clone my git repository and init submodules
      git clone https://github.com/istarc/stm32.git
      cd ~/stm32
      git submodule update --init
      
      # Sorry, you'll have to build OpenOCD from scratch.
      # Install OpenOCD build requirements
      sudo apt-get install libtool libftdi-dev \
                           libusb-1.0-0-dev automake \
                           pkg-config texinfo
      
      # Clone OpenOCD repository
      git clone --depth 1 \
       git://openocd.git.sourceforge.net/gitroot/openocd/openocd
      
      # Configure OpenOCD
      cd ~/openocd
      ./bootstrap
      ./configure --enable-maintainer-mode \
                  --disable-option-checking \
                  --disable-werror \
                  --prefix=/opt/openocd \
                  --enable-dummy \
                  --enable-usb_blaster_libftdi \
                  --enable-ep93xx \
                  --enable-at91rm9200 \
                  --enable-presto_libftdi \
                  --enable-usbprog \
                  --enable-jlink \
                  --enable-vsllink \
                  --enable-rlink \
                  --enable-stlink \
                  --enable-arm-jtag-ew
      
      # Build OpenOCD
      make
      sudo make install
      

2. Install Software Dependencies

Install the latest Tracealyzer for FreeRTOS (FreeRTOSplusTrace.exe from percepio.com) on your Windows machine.

3. The Big Picture

In this section, I first describe the example (buggy) FreeRTOS application. Next, I show how to identify the bug using the Tracealyzer tool. Finally, I show how to fix the bug and the corresponding performance improvement.

3.1 The Buggy FreeRTOS Application

The code snippet below is a part of a FreeRTOS task. It detects button press event and notifies other process via interprocess communication (IPC). You may find the complete source code here (see Task 2).

while (1) {
   /* Detect button press */
   if(pb == 1) {

      /* Debounce delay 10 ms */
      vTaskDelay(10 / portTICK_RATE_MS);

      /* Wait button release, probe every 10 ms */
      while(pb == 1)
         vTaskDelay(10 / portTICK_RATE_MS);

      /* Debounce Delay 10 ms */
      vTaskDelay(10 / portTICK_RATE_MS);

      /* Notify Task 3 */
      xQueueSendToBack(pbq, &sig, 0);
   }
}

A short explanation what it does. The button state change is captured by reading the pb variable (line 3). When the mechanical button is pressed or released, its mechanical contacts bounce many times before the contact is steady. This could be misinterpreted as repeated button press (see the figure below), so our code includes debounce delays to counteract this transient response (lines 5, 13). It waits for the button to be released (line 9, 10) and after the 2nd debounce (line 13) it notifies Task 3 about the button press event via IPC, which in turn toggles the LED diode on STM32F4-Discovery board.

Bouncy_Switch

Snapshot of mechanical switch contact bounce. Without a debounce delay, program could misinterpret this as a repeated button press.

I hope it is simple to understand as the pasulj recipe. And, it actually works as expected. Try it out! Build and deploy the code onto the STM32F4-Discovery board and press the blue button to toggle the LED.

The green LED toggles every 1.5 s. You can toggle the red LED by pressing the blue button.

cd ~/stm32/examples/FreeRTOS.mbed-trace
make clean
make -j4
sudo make deploy

3.2 Bug Identification using the Tracealyzer

The code seems to work just fine, but after close examination using the Tracealyzer tool we discover a bug. Just look at the figure below. Ideally, Task 2 (button press detection task) should not be CPU intensive, but in reality it dominates the execution trace and completely hogs the CPU. We created an energy draining monster! 😯

Tracealyzer-BasicNow, we could use in-circuit debugger to pinpoint the problem. The root-cause is, however, apparent. The program busy waits in the while loop (line 1).

3.3 The Bugfix

The bugfix is very simple. Instead of busy waiting in a while loop, we add a delay to poll the status of the button every 10 ms (line 3). You may find the complete patched source code here.

while (1) {
   /* Wait press, probe every 10 ms */
   vTaskDelay(10 / portTICK_RATE_MS);

   /* Detect button press */
   if(pb == 1) {

      /* Debounce delay 10 ms */
      vTaskDelay(10 / portTICK_RATE_MS);

      /* Wait button release, probe every 10 ms */
      while(pb == 1)
         vTaskDelay(10 / portTICK_RATE_MS);

      /* Debounce Delay 10 ms */
      vTaskDelay(10 / portTICK_RATE_MS);

      /* Notify Task 3 */
      xQueueSendToBack(pbq, &sig, 0);
   }
}

The improvement is remarkable (see the figure below).

Tracealyzer-ImprovedPreviously, the OS idled at less than 0.1%. Now, it idles at more than 99.9% and saves energy, not to mention, it’s environmental friendly.

Batman-Congrats

Congrats!

You may try it out as well! πŸ™‚

cd ~/stm32/examples/FreeRTOS.mbed-trace-v2
make clean
make -j4
sudo make deploy

4. Instructions to Repeat the Exercise

To repeat the exercise on your home computer you have to build the FreeRTOS application (which includes a special trace unit).

cd ~/stm32/examples/FreeRTOS.mbed-trace-v2
make clean
make -j4

Next, deploy the binary onto the STM32F4-Discovery board via OpenOCD. It is important you use the latest OpenOCD version, which is not available in the official Ubuntu 14.04 LTS repository, so you have to build it from the source (described in the 1st section).

cd ~/stm32/examples/FreeRTOS.mbed-trace-v2
sudo /opt/openocd/bin/openocd -f \
 /opt/openocd/share/openocd/scripts/board/stm32f4discovery.cfg

In a separate terminal window connect to the OpenOCD instance via GDB. Then (i) deploy the binary, (ii) run the program, (iii) wait a few seconds, then halt the execution, and (iv) dump the memory.

cd ~/stm32/examples/FreeRTOS.mbed-trace-v2
arm-none-eabi-gdb
# 1. Deploy the binary
(gdb) target remote localhost:3333
(gdb) monitor reset halt
(gdb) monitor flash protect 0 0 11 off
(gdb) monitor flash write_image erase /abs/path/to/outp.elf
# In my case the /abs/path/to/outp.elf is
# /home/iztok/stm32/examples/FreeRTOS.mbed-trace-v2/bin/outp.elf
# 2. Run the program
(gdb) monitor reset run
# 3. Wait a few second then halt the execution
(gdb) monitor reset halt
# 4. Dump the memory
(gdb) monitor dump_image /path/to/dump.bin 0x20000000 0x1FFFF
# In my case the /path/to/dump.bin is
# /home/iztok/dump.bin
(gdb) quit

The memory dump includes the execution trace information. Open the dump file in the Tracealyzer (File β†’ Open…).

Tracealyzer-FromScratch5. Setup the FreeRTOS+Trace project in Less than 10 Minutes

5.1 Setup the Basic Project Environment

A new FreeRTOS+Trace project is easy to setup if you use my project wizard.

export PATH=$PATH:~/stm32/mbed-project-wizard
cd ~/stm32/examples

# Create a new project folder
mkdir trace
cd trace

# Generate a new FreeRTOS project template
# with mbed SDK
gen-stm32f407-GCC-project.sh mbed-freertos

Add the trace unit to the project’s lib folder.

cp -r \
    ~/stm32/freertos/FreeRTOS-Plus/Source/FreeRTOS-Plus-Trace \
    lib/

ls lib/
# FreeRTOS FreeRTOS-Plus-Trace mbed README

5.2 Hook the Trace Unit onto the FreeRTOS

Follow the original instructions to hook the Trace unit onto the FreeRTOS.

  1. Hook the trace unit at the very end of the FreeRTOSConfig.h file (a complete example).
    gedit lib/FreeRTOS/config/FreeRTOSConfig.h
    
    ... line 163 ...
    #define configUSE_TRACE_FACILITY 1
    #include "trcKernelPort.h"
    
    #endif /* FREERTOS_CONFIG_H */
    ... EOF ...
    
  2. Start the trace unit at the very beginning of the main.cpp file (a complete example).
    gedit src/main.cpp
    
    ... line 20 ...
    int main(void)
    {
       vTraceInitTraceData ();
       if (!uiTraceStart())
          vTraceConsoleMessage("Could not start recorder!");
    ... line 26 ...
    

5.3 Apply Two Hotfixes

Apply the hotfixes below to the trace unit and the Makefile. These hotfixes are STM32F4 specific and extend the original instructions.

  1. Hotfix 1: safely call the recorder from any interrupt service routine (a complete example).
    cd ~/stm32/examples/trace/lib/FreeRTOS-Plus-Trace
    gedit ConfigurationTemplate/trcConfig.h
    
    ... line 516 ...
    #define USE_PRIMASK_CS 1
    ... line 518 ...
    
  2. Hotfix 2: use Gnu99 dialect instead of C99 to compile C code (a complete example).
    cd ~/stm32/examples/trace
    gedit Makefile
    
    ... line 52 ...
    CFLAGS=-c $(MCFLAGS) $(DEFS) $(INCLUDES) -std=gnu99
    ... line 54 ...
    

5.4 Build and Deploy

See section 4.

6. Acknowledgements

I would like to thank Percepio for providing the Tracealyzer tool for FreeRTOS! πŸ™‚

Advertisements

About istarc

Embedded Systems Developer.
This entry was posted in Embedded Systems, STM32F4 and tagged , , , . Bookmark the permalink.

2 Responses to STM32F4 – On Hunting Bugs with a Profiler

  1. percepiodotcom says:

    Hi
    Percepio are releasing Tracealyzer v2.7 in a few weeks at most. This version runs on Linux through Mono.
    /Johan

    • istarc says:

      Hi Johan! Even though the current version only runs on Windows, it is an invaluable tool for any embedded systems developer. I warmly recommend it and am looking forward to testing the upcoming version that runs on Linux. πŸ™‚ Keep up with the good work and best regards, Iztok.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s