What I learned about Zephyr threads

I'm very new to Zephyr, literally, like..., 4 weeks new. Wasn't sure about this new RTOS kid on the block at first, but now it's really starting to grow on me!

When you develop apps running on RTOS (such as FreeRTOS, Zephyr, ThreadX, VxWorks, etc.), most likely you will need to deal with threads of different priorities and determine what tasks should be performed inside the interrupt service routine (ISR) when the current thread is preempted. What's more challenging is when you need to synchronize those threads with Kernel IPC services such as mutexes and semaphores in order to protect critical sections.

To walk through those scenarios, we will use the nRF7002-DK dev board from Nordic Semiconductor for demonstration. I will also use the sample apps (philosophers & button) from Zephyr RTOS repository plus a professional tracing tool: Percepio Tracealyzer? with J-Link RTT streaming to walk through Zephyr's preemptive scheduling of threads in a GUI like this:

The scenario

With my nRF7002DK board, I merged the source code of 2 apps (Dining Philosophers & GPIO button) to demonstrate the Dining Philosophers problem (a classic multi-thread synchronization problem) with multiple preemptible and cooperative threads of differing priorities, as well as dynamic mutexes and thread sleeping. Here are the 4 transitional states of 6 philosophers with 6 forks: THINKING, HOLDING ONE FORK, STARVING, EATING:

Then I added a "twist" to the original dining philosopher application: manually preempt the current thread with random interrupts by pressing the GPIO Button #1 on the board:

When the current thread (could be any one of those six philosophers) is preempted by pressing Button 1, my ISR will execute k_wakeup() to wake up Philosopher #4 from thinking. We will then analyze what happens when ISR returns and how the kernel schedules the next current thread based on priorities.

You probably noticed in the short video above: philosophers 4 & 5 have negative priority values which made them cooperative threads. According to Zephyr's documentation:

A cooperative thread has a negative priority value. Once it becomes the current thread, a cooperative thread remains the current thread until it performs an action that makes it unready.

A preemptible thread has a non-negative priority value. Once it becomes the current thread, a preemptible thread may be supplanted at any time if a cooperative thread, or a preemptible thread of higher or equal priority, becomes ready.

The thread priority level in Zephyr looks like this:

Ok, let's come back to our story here. When Button #1 is pressed, whether the current thread is cooperative or preemptive, the execution of current thread will be replaced by ISR unless interrupts have been masked (well, that is an advanced topic, beyond a rookie like me can explain...)

I would recommend you pause and resume the video above at your own pace to closely observe the behavior of those threads before moving on.

Prerequisites

If you would like to follow along, here are the steps:

1. Prepare your own hardware, QEMU, or simply use Zephyr's native_sim with GPIO simulator. Merging the code from these two sample apps is actually very easy and straightforward to build.
2. Download a copy of Percepio? Tracealyzer and follow through the guide: Getting Started with Tracealyzer for Zephyr RTOS - Percepio.
3. Since my nRF7002DK already has an on-board J-Link interface, I am not using the Segger J-Link debug probes. Because the on-board J-Link is a lot slower than the external debug probe and there are a lot of missing events from the philosopher sample app, we will need to increase the CONFIG_PERCEPIO_TRC_CFG_STREAM_PORT_RTT_UP_BUFFER_SIZE in your .config. See my previous LinkedIn post.

Let's do it!

Adding a GPIO button

It's super easy to add a GPIO button. Carefully review the code in Button sample and then add the relevant lines into the Philosophers sample. The procedure to enable interrupt on a GPIO pin is done in a particular order:

• Retrieve the GPIO device pointer (button) and if gpio_is_ready_dt(&button) indicates the device pointer is ready, then configure the pin to be either an input or an output pin (in this case, GPIO_INPUT).

• Set up an interrupt on a GPIO pin by calling the function gpio_pin_interrupt_configure_dt().

• Next is a key step of this demo. Initialize and add the callback handler function button_pressed(), which will be called when an interrupt is triggered:

• The lines below define my ISR button_pressed(). At line 124, I purposely let the ISR wake up Philosopher #4, which has priority -1. While we are here, you will see I added 2 more lines (121 & 125) to enable tracing inside an ISR at the beginning and at the end. We will come back to this in details in the next section when we configure Tracealyzer for tracing.

Add support for Percepio? Tracealyzer

• Follow the instruction to enable tracing in our Zephyr app and select tracing format for Tracealyzer: Zephyr — Trace Recorder 4.8.0 documentation.
• We are leveraging J-Link RTT streaming mode for tracing, follow the instructions: Getting Started with Tracealyzer for Zephyr RTOS - Percepio
• This troubleshooting guide is super helpful (especially helped me addressed the missing events issue): Using J-Link RTT Streaming - Percepio

To enable tracing ISR with Tracealyzer, from the doc:

To trace interrupt handlers, first call xTraceSetISRProperties() to specify the interrupt name and priority. This is typically done in your main function, not in the interrupt handler. This returns a handle, that you need to store in a global variable. Then call vTraceStoreISRBegin() in the beginning of your handler, and vTraceStoreISREnd() in the end.

Here's the code in philosopher app's main():

Analyze the threads in Tracealyzer

Once you have built and flashed the enhanced philosopher app with GPIO button 1 enabled, start the Tracealzyer session to record the traces via J-Link RTT. Now it's time to press Button #1 a few times (I pressed the button as fast as I could in order to get meaningful trace records). Stop the session and let's look into the Trace View together.

Scroll the trace view back and forth in time so you can observe how the philosophers behave nicely when they are eating or thinking. By default, the philosopher sample app uses mutexes for thread synchronization. Also, you will see the ISRs are short and not introducing much latency.

We can also see the ISR trace property has "RICK's button ISR" in color red and Philosopher 4 in orange:

From the trace view screenshot above, at the end of button_pressed() ISR (RICK's button ISR in red), it executed k_wakeup(Philosopher 4) which is circled in purple. Looking good so far! The ISR is doing what was expected.

Next, let's look for the instance when Philosopher 5 (which has priority -2, the highest cooperative thread in this app) was preempted by GPIO button interrupt:

Now this (Test 1) is VERY interesting and looks like I may have found the answer to the questions I had for the past few weeks:

1. When a cooperative thread is preempted by an interrupt, where does the ISR return to? Will the kernel wake up Philosopher 4 as I specified in the button_pressed() ISR?
2. Or it will return to the cooperative thread that was preempted?

Let's look at the trace together:

• The green arrow shows the cooperative thread "Philosopher 5" with priority -2 was preempted by GPIO interrupt.
• The blue arrow shows the end of ISR, the execution returns back to "Philosopher 5". Not like the previous traces showing "Philosopher 4" was scheduled to run at the end of ISR.
• The purple arrow shows when Philosopher 5 with priority -2 is done eating (put down the forks and released the mutexes by k_mutex_unlock()), Philosopher 4 with priority -1, which is the next highest in the app, picked up the forks (by calling k_mutex_lock()) and started eating.

Next, let's do one more interesting test (Test 2). This time we will change the ISR to wake up Philosopher 5 (priority -2, the highest one) instead. We will look into the trace records and find the moment when Philosopher 4 was interrupted, will kernel wake up Philosopher 5 or schedule Philosopher 4 to run? The trace below revealed something that confirmed a very promising proof of the cooperative priority behavior:

• The green arrow shows the cooperative thread "Philosopher 4" with priority -1 was preempted by GPIO interrupt.
• The blue arrow shows the end of ISR, the execution returns back to "Philosopher 4", EVEN WHEN my ISR is programmed to wake up another cooperative thread (Philosopher 5, priority -2) with higher priority.
• The purple arrow shows when Philosopher 4 with priority -1 is done eating (put down the forks and released the mutexes by k_mutex_unlock()), Philosopher 5 with priority -2, which is the highest in the app, picked up the forks (by calling k_mutex_lock()) and started eating.

Conclusion

Through the tracing exercise with Percepio's Tracealyzer, within a few hours, I was able to find the answer I had for weeks! So, here's what I learned about the cooperative vs preemptible threads in Zephyr:

When a cooperative thread is preempted by an interrupt, the kernel will schedule the cooperative thread which was preempted as the current thread (EVEN WHEN there's another higher priority cooperative thread ready). This provides a guarantee of the cooperative priority promise: it will remain the current thread until it performed an action to make itself unready or it's preempted by an interrupt.

References

I really enjoyed working through the exercise above to learn something new! During the exercise, I found several great blog posts by a few folks I know. Kudos to you all who contributed to the success of Zephyr!

• Common multithreading problems and their fixes (Part 3) – Zephyr Project by Roy Jamil
• Zephyr Threads, Work Queues, Message Queues and how we use them - The Golioth Developer Blog by Mike Szczys
• Taking the next step: Debugging with SEGGER Ozone and SystemView on Zephyr - The Golioth Developer Blog by Chris Gammell
• Unlocking Zephyr Debugging: Tracealyzer Now Validated on 600+ Development Boards – Zephyr Project by Percepio? (This blog post explained how to diagnose the priority inversion problem using the Tracealyer tool.)