Networking driver strategies for using embedded Linux

Whether that hardware is a programmable network processor or a configurable networking ASIC or ASSP, complicated software is required to bond the hardware and the system's CPU into a cohesive networking system. Such software typically consists of hardware access routines, application support libraries and services tasks that perform maintenance services related to supported applications of the hardware.

In many cases software written for these systems has been done with only certain embedded operating systems or environments in mind. While many of the current embedded operating systems may support one or more of the standard Posix applications programming interfaces and appea r to be similar in other areas, there are distinct differences in such areas as the driver model for hardware access and how the software components are separated across memory spaces. One must understand those differences when using embedded Linux and leveraging software written for other environments.

Linux is a protected system. The Linux kernel uses the memory management hardware available on modern CPUs to protect tasks from each other and to protect the kernel from errant userland tasks.

Further, to ensure system integrity the kernel allows only "root"-owned processes to access physical address ranges. These processes require specific programming interfaces to map physical address ranges into a process' virtual address space. Other types of hardware accesses, such as interrupt handlers, DMA transfers and other non-memory mapped hardware accesses, are restricted entirely to code running in the kernel's context. While protections have great value, they can serve to make it difficult to c ontrol hardware of any complexity entirely from userland.

Since modern networking hardware is highly functional and complex, simple memory-mapped register access is generally insufficient for control of these devices. Other forms of hardware access require execution in kernel context, and vendor-provided drivers tend to be poorly segregated between hardware access and higher-level functionality. The solution is to package the conglomerate drivers in a way that allows execution within the context of the kernel.

A vendor-provided driver tends to have a top-level API that is intended for use by developers in creating a working system. But in this model the actual conglomerate driver resides in a separate address space from the rest of the application, rendering traditional linking impossible.

One solution is to provide a library for the application to link against that mimics the published API of the conglomerate driver. Underneath, this doppelgänger library will communicate to the kern el-resident code by established means, such as an ioctl() call to an actual Linux device driver.

In practice, the approach works reasonably well to access a conglomerate driver's functionality from userland applications. But there are some drawbacks. The costs of switching between user and kernel context can accumulate rapidly. Also, there is no simple way to call userland functions from within the kernel's context, which precludes the use of simple callbacks. Perhaps the biggest problem is that standard Linux kernels are not preemptible. Existing code that was not written with preemptibility in mind generally will not make any effort to bound its own execution time and thus can easily starve all of the other processes in a system.

It is possible to run most or all of a conglomerate driver in a user's context. The strategy can produce a workable system without major changes to an existing conglomerate of software.

Hardware access

As discussed, the Lin ux kernel restricts many kinds of hardware accesses exclusively to code running in the kernel's context. Complex networking hardware will generally require more than simple memory-mapped register accesses to function properly. If the body of an existing conglomerate driver is to reside in a user's context, some means must be devised to account for the non-memory-mapped hardware accesses.

The general approach to providing this type of access is to implement a small formal device driver that executes in the kernel's context. The small driver provides the required functionality — for example, DMA transfer setup through a small suite of custom ioctl() calls. In general, only a few minor changes to the conglomerate driver are required to make the strategy work.

This strategy avoids many of the problems found with the previous approach. No special "shim" or doppelgänger library is required to allow the application to access the conglomerate driver. Further, callbacks, two-way sharing of pointe rs and other common software techniques are easily available to the body of the conglomerate driver's code. Thread preemption is not a problem, since code running in a user's context is always preemptible. Debugging is possible using standard means, and crashes do not automatically crash the whole system.

While the strategy resolves many of the problems of the earlier approach, it faces its own set of problems. It is likely that some hardware accesses will occur more frequently than required or be performed as a small part of a larger atomic operation that has no other need to run in the kernel's context. Hence, the strategy is still prone to excessive amounts of expensive context switching or nonpreemptible execution in the kernel's context. While this strategy is better than the first approach, it seems clear that another solution must be found.

The discussion so far has enumerated and demonstrated the problems with maintaining a conglomerate driver as a single entity in a Linux environmen t. The cost of excessive context switching becomes significant. So, what is the solution?

All of these problems stem from an architectural conflict: The Linux kernel restricts most hardware accesses to code running in the kernel's context, and conglomerate drivers do not adequately separate hardware access code from other conglomerate components. The simple answer is to remove this conflict between the Linux architecture and the architecture of the conglomerate driver. That means dismantling the conglomerate driver and reconstructing it in a form compatible with the Linux driver model. The task at hand is to separate the two application-oriented components from the system-oriented hardware access component.

By careful code and system analysis, the conglomerate driver can be separated into a hardware access group and an "other" group, which will contain application support routines and other service tasks. Reorganization of existing functions will be required to modify or remove unnecessary hardware accesses or to formalize those accesses in a way that is supportable in the Linux environment.

Library independent

The end result will produce two distinct object libraries. The hardware access library should have no dependencies on the "other" library, while the other library should at most require access to the functions defined in the hardware access library. In particular, there should be absolutely no use of shared global variables between the two libraries and no calling of functions defined in the other library from functions defined in the hardware access library.

With the former conglomerate driver broken into two distinct libraries, one has the basic ingredients necessary for implementing this solution. The hardware access library can be packaged to run in the Linux kernel's context using a wrapper that implements one or more of the standard Linux driver interfaces. The other library is used for linking against applications requiring access to the hardware controlled by the former conglomerate driver.

The interface between the hardware access library and the other library is implemented either by adding Linux-specific code to the other library or by creating a doppelgänger library that provides the same symbol definitions as the hardware access library but that uses Linux-specific code to drive the hardware access code running in the kernel's context. The end result is a functional solution in nearly every case.

This discussion has been about how to make use of an existing conglomerate driver in a Linux environment. But it is possible to architect a driver library so that it works equally well with both Linux and other embedded operating systems.

A proper driver architecture should provide for a hardware access layer that encapsulates all of the atomic operations one would want to perform on the hardware. Once defined, that layer should not be violated. Further, any dependencies on the hardware access must be "one-way" ; calls from the hardware access layer to higher layers are specifically not allowed.

In Linux, the hardware access layer will be run within the kernel's context. In an unprotected operating system, this layer will be just another part of the driver library.

A driver library architected in this fashion from the beginning should introduce neither extra run-time overhead nor extra complexity in either a protected or unprotected environment. Furthermore, this basic design framework is more in line with modern software design practices and should lead to fundamentally better software, with fewer bugs and greater stability.

There are a number of ways to adapt an existing conglomerate driver to a Linux environment. Following an approach in which the application-oriented components are separated from the system-oriented hardware access components will result in a driver design that works well in either a protected environment, such as Linux, or an unprotected environment.