Embedded Virtualization Changes Comm Software Development Landscape

Embedded Virtualization Changes Comm Software Development Landscape

Ross Wheeler, DoubleWide Software, Inc.
Mar 25, 2004 (6:00 AM)
URL: http://www.commsdesign.com/showArticle.jhtml?articleID=18401106

There's a big secret in the embedded communication space that needs to be let out. For the most part, embedded comm designers haven't changed the way they've been doing things since the late 1980's.

To illustrate this point, let's take look at the EDA space. If you asked a chip designer to use the same tools and process that he or she was using in the 1980s to design chips, the designer would throw you out of his/her cube.

Now ask the embedded communication designer the same question and the response will be "No problem — as long as it's a C compiler, I'm cool." Can you, or do you get your job done with these tools from the 1980s? For many, the answer is yes. But, is this the most effective approach? Is your current development environment scalable for the large complex embedded communication systems being developed today? That answer is probably no. In this article we'll show why traditional design approaches aren't the most effective and why embedded supersystem virtualization (ESV) may be a better option for development moving forward..

Lacking Innovation
For almost 20 years, enterprise application developers have reaped the productivity and code quality benefits of the dramatic advances in development tools and environments. But what new language has come out of the embedded system space first? C#? Java? C++? Most embedded communication software today is still written in C. Meanwhile, embedded software developers have been left behind the normal productivity curves in a kind of "Developers' Hell."

Do the following stories—all of them true—make you think that we've really been innovating for the embedded software space over the last 20 years?

Story 1:

Embedded Developer One: "The (only) prototype just came in and I'm going to need it until the end of the week."
Embedded Developer Two: "That's fine; I'll take it next Monday."
Embedded Developer Three: "Super. I wasn't doing anything this weekend so I guess I'll take it Saturday and Sunday. Grrrrrrrr."

Story 2: You're remote debugging some equipment in a lab away from your office/cube and your software isn't running correctly. You make changes over the next several days and decide that you're going to take a walk and check out the lab hardware. Once in the lab you notice a red RMA tag on the chassis board that you are using. The lab tech replies, "One of the other groups took your original board out and put this one in there. Is there a problem?"

Story 3: (A personal favorite) Spending two weeks getting your test rig setup properly only to have it torn apart by your co-workers while you are taking a week's vacation.

Story 4: A developer asked me to take a look at her program under the debugger because it was acting funny. I watched her trace her source code until she made an RTOS API function. Then she tried to keep stepping through her source and nothing happened. She asked what should she do and the obvious answer was to try to debug it. There was only one problem: The call crashed the debugger so she couldn't use it to debug the crash. How do you debug the debugger? The next time you speak with your enterprise developer friends, ask them how often they crash the debugger.

While many of these stories might appear humorous, the reality is that it's not funny when you're the developer and under pressure to deliver. These scenarios often create a blame-game situation in which hardware/software/QA developers are pointing fingers at each other.

Defensive Programming
Let's look at the embedded software landscape in a typical project. You'll have the truly "embedded" developers who write the device drivers, work out the board support packages (BSPs) and work with the hardware. Sometimes you're lucky and you have developers that are good at both "hardware stuff" and the application software, but most often they are only good in one area. Then you have the application developers who write code that interfaces to the device drivers and RTOS. But why are the embedded application developers worrying about what hardware they are running on? Shouldn't they be focused on building software that talks to the system APIs?

In this environment, the development processes take on an interesting cultural change. Development engineers become "defensive" programmers because any change they make can take minutes, hours, or even days to test. In a sense, they get stuck in the real-time "penalty box." Because of this time penalty, developers become tentative and overly conservative when it comes to making fixes, adding features and experimenting. Fixes are often a short term (safe at the time) patch as opposed to a more aggressive but better long-term solution. Later in the development cycle, adding features becomes very difficult because of the time pressure.

Speaking of being conservative, most embedded developers are still writing in C. However, the only developers that are talking to the hardware are the device driver writers and the BSP folks. Do you ever wonder why the enterprise folks aren't writing in C anymore? There is almost no other language we write in because of the defensive posture we're forced into. So next time you're asked, just tell your enterprise software friends that your programming environment is "hell" and that's why you write code in C.

The Hardware Problem
Currently, embedded comm developers have a unique dependence on access to the actual target hardware which at times during a project can be quite fluid ("fluid" equals "buggy"— I'm trying to be politically correct!). Imagine the enterprise developer trying to debug an application program and having to reboot the PC after every run. Wouldn't that be inefficient?

The same applies to enterprise developers who use multi-user systems. If another enterprise developer needs share the hardware, no problem—just log in. For the typical embedded project with a hardware prototype — forget it! Many embedded communications systems don't support multiple users simultaneously testing, debugging etc.

Companies have been using virtualization and simulation techniques for many years to get around the hardware problems. Typically an embedded project starts with development of a patchwork form of simulation to help jumpstart the development effort. Because simulation development isn't core to the embedded systems provider's business, the hardware simulation tool gets neglected as soon as the physical hardware is available.

Most embedded developers realize that there are benefits to simulation, but the ability of the developer's company to achieve those benefits is limited by time and manpower constraints. For some companies, these in-house tools are purposely under staffed or abandoned because they are not considered a profit center for the company.

To effectively test and debug their software, embedded developers require access to actual target hardware in various configurations. Typical real-time development tasks include setting up the hardware, connecting interfaces and rebooting/reinitializing the target.

Many of these tasks, especially configuration, are tedious and tend to be mistake-prone. Forcing software developers to waste valuable project time physically configuring prototype hardware and waiting for code to download to the target device is costly. Customers have said that their developers typically waste 10 to 30% of their time dealing with hardware limitations and complications. This time loss can have a huge impact on overall project costs.

Consider this a real-time development tax. If we assume that an engineer costs $150,000 per year with overhead, the chart below illustrates the impact this lost time can have on an annualized basis.

ESV: A Different Kind of Simulation
One of the reasons real-time Linux and embedded Windows have become popular as embedded operating systems is the simple fact that the embedded developers can leverage the enterprise tools and operating systems that are being used in the enterprise space as well. For the embedded communication developers, the simulation environment is, "Let's just run it on the desktop." Heck, at least you don't have to mess with the hardware. But as we'll find out, there are some shortcomings to doing this as well when it comes to embedded communication systems.

What if I said you can virtualize your hardware and run it on your desktop? Would you believe me? Well this is not only possible but in the coming years this could be the only way to do embedded development.

By applying system virtualization techniques, embedded communication software developers for complex systems can be extremely productive. Fast, accurate, virtualization of target environments is possible, and in fact now available. Embedded developers are now free from depending on serialized access to target hardware prototypes.

Many devices today don't function as standalone systems, but instead as a connected part of a larger whole. Supersystem virtualization techniques must go beyond just simulating a single hardware component or even one complete system. The virtualized system must be able to communicate with other systems and other devices. Examples of this can be a communication medium, such as a network connection, backplane or any other data interface.

Simulation of complex hardware and software systems is referred to as embedded supersystems virtualization (ESV), but you might prefer to think of it as finally being able to utilize innovative software tools in the embedded software environment, and as your "escape" from developer's hell. This new approach offers compelling advantages because developers can now develop and test software against multiple virtualized systems from a single desktop running on popular operating systems such as Windows, Linux or Solaris.

ESV is characterized by the ability to:

Run unmodified target code in the virtualized environment
Simulate complete single or multiple interconnected systems
Execute code at the speed of the developer's desktop CPU (3000+ MIPS)
Provide access to a wide array of mainstream development and testing tools not previously available for embedded software development

It isn't just the developers who benefit from ESV. Complex virtualized configurations can be saved and supported with scripts that move develop, debug, and test cycles from the lab to the test engineer's desktop. This scripting capability allows test engineers to quickly and reliably perform unit, stress and data loss tests without spending hours setting up and changing test system configurations. Complex test topologies can be set up in seconds simply by choosing the applicable virtual test configuration file.

How ESV Works
To illustrate the impact of ESV on a development process, let's take a telco equipment manufacturer. In this example, let's assume the development project involves a chassis-based switch with several types of line cards, each with its own processors.

In the past, the ability to perform software development was limited in advance of the availability of a given prototype card. Full chassis prototypes were a shared resource in limited supply, which made integration testing difficult. And the developers wasted precious time in the develop/build/target download/test/debug cycle. At this stage, many in-house tools are created and later discarded as inadequate (as described earlier).

The ESV approach changes the development process. Hardware components are virtualized well in advance of prototype availability. Consequently, the telco line card software developers can start development and testing well ahead of their tradition-bound counterparts.

Developers then develop their application using their embedded or enterprise tools and link in the libraries provided by the virtualization environment for the target operating system. A test configuration is constructed using the virtualized device and is executed enabling the software developer to:

Attach a debugger to the virtualized device
Check for memory leaks
Detect and debug errant/null pointers
Do post-mortems on crashed devices
Halt threads/processes on faults
Control timers within the virtualization
Profile the software performance

And this is all done on the developer's desktop, reducing the need to come into the office on the weekends to access prototypes.

ESV Benefits
For the equipment vendor, ESV means bringing products to market both faster and with higher quality. To the software developer, it means developing and testing any part of the software without connecting or configuring the corresponding physical hardware. To the QA team, it means more time for creative software testing and the ability to easily share complex test topologies.

Equipment vendors will achieve significant R&D cost savings using ESV. Integration testing and debugging is easily accomplished without conflict over shared hardware resources. For our telco equipment vendor, multiple full-chassis prototypes are available in virtual form to every developer, which eliminates scheduling and resource conflicts. Manufacturing fewer hardware prototypes provides significant hard ROI savings potentially in the millions of dollars.

ESV holds many other advantages for software projects. Fast, iterative testing for developers on the desktop dramatically improves software error detection and product quality for embedded products. Removing prototype availability as a testing and verification bottleneck speeds up project completion.

ESV lets developers spend most of their project time testing the target image on their desktop. Because the developer uses a virtual device instead of actual hardware, the develop/build/target download/test/debug cycle is compressed to develop/build/debug cycle helping avoid the real-time development tax. Target images run at the speed of the desktop CPU that is hosting it (3000+ MIPS), which can be ten or more times faster than actual target hardware, resulting in faster debugging and the ability to run multiple virtualized devices on a single desktop.

About the Author
Ross Wheeler is the founder and CEO of DoubleWide Software, Inc. Ross also founded and led network protocol supplier RouterWare. Hr Wheeler holds a BSCS and BS Computer Engineering from Trinity University and can be reached at ross@doublewidesoft.com.