Multi-core: A new challenge for debugging

Jens Braunes, pls Development Tools
(10/14/2008 9:59 AM EDT)

A customer story
In a huge software project for an embedded application, a function behaved in a strange fashion. A variable, which must not be changed while the function is executed, was changed. The function itself did not write the variable, but rather an illegal write was caused from an-other process, an interrupt service routine or the DMA controller. To identify the culprit the source code of the function was instrumented by a costly monitoring approach. Monitor code was inserted every few lines and near branches to find the point of time the variable was overwritten. In addition, the actual challenge was to check all system activities at or right be-fore the illegal write access to find the writing process. Not a trivial task in a complex system of several cores, busses and bus masters etc. It took one week to identify the problem -- an incorrectly configured DMA controller. Other company projects based on a different hard-ware platform make use of an on-chip multi-core debug system to provide visibility to cores and bus transactions. "If we had a hardware debug system available for our project the prob-lem would have isolated within minutes" the customer told us.

What have we learned from this story? Development of complex systems with powerful hard-ware on one side and ambitious applications on the other side, benefits from on system-spanning on-chip support for debugging. In complex SoCs, just to observe and control a sin-gle core is insufficient. Rather the interactions of multiple cores, busses, and peripherals com-prising the SoC are needed when you want to detect, trace, and eliminate software problems or to profile system behavior for performance optimization.
Before determining an on-chip debug system suitable for multi-core systems , it is important to understand the user requirements.

Must each core and bus must be observable? Being able to see or reconstruct the program flow of each single core independently as well as the data flow on the system busses is critical to drawing conclusions about optimizations, interactions, bus accessing, and performance in other modes of operation.

Is it crucial for system analysis to recognize events that arise from interactions between the cores and busses? Single core operation is often insufficient, with events coming from several cores having to be considered. To address this challenge, cross triggers must be used, which combine events from different sources and make them available system-wide.

Click here to read more ...