Most Common Embedded Linux System Project Estimation Issues

Estimating your Linux embedded project is like starting a full renovation of your house: there are just too many things you need to take into account in order to plan everything right. What makes the process even more challenging is that you need to explain and rationalize your estimates to the customer.

At Apriorit, we often work with device vendors who need custom-made software for their specific devices or test stands. Due to the great number of factors to be taken into account, estimating such projects is a challenge for both developers and project leaders. We decided to share our experience planning and estimating such projects and cooperating with customer teams.

In this article, we talk about some of the most critical embedded Linux estimation issues and what you can do to avoid them. We tell you what problems might be waiting for you at the stage of estimating, preparing, and investigating a new project. This information will help you understand better what to focus on when planning a project and discussing project requirements with stakeholders.

Expectations vs reality

Product leaders and developers see things differently, so finding a compromise when estimating Linux embedded development projects is always a challenge. Usually, a business visionary has an idea in mind and can explain what result they want. But even if they have some vision of how to bring this idea to life, the implementation is left to developers.

For instance, an initial request for development may look like this:

We need someone who can develop an application for real-time IP camera video streaming. We want to broadcast the video stream over multiple Ethernet links, considering that each link will have a variable bandwidth and some instances may drop out completely.

Our main goal is to broadcast an RTSP stream and then recombine the stream from multiple Ethernet interfaces with low latency and a minimum number of dropped frames.

In this case, we have a task to be solved: to stream video at high speed. There may already be a specific motherboard and understanding that certain software should be installed in order to use it. But what kind of software will be used and whether it should be installed in the kernel or in the user space are questions that must be answered by the developer.

Here’s another example of a customer request that a development team may receive:

We’re looking for someone who can analyze and reverse engineer a Linux device driver. We use a WLAN processor from MediaTek (MT7612U), which is operated via USB bus from an FPGA. We need to add logging to it so that we can log the USB traffic of a WLAN USB dongle connected to a Linux PC with the driver operation. We also need to extract USB data packets that we can then use for configuring the WLAN processor and sending/receiving data packets on the MAC layer.

In this case, the final goal of the project is to configure a network adapter. And even though they have some considerations about the way this task should be accomplished, the final decision is still left up to developers. The main thing you should keep in mind here is that you’ll have to work with a specific device and a specific driver, which will affect the duration of the development stage.

The main problem is that project leaders are often too optimistic regarding embedded Linux development estimations.

So what do you need to pay special attention to when estimating a new embedded Linux project? We answer this question in the next section.

Looking for the right solution

When you start a new project, the first question you should ask is What is the final goal of the project? You need to mark the borders of your project.

Then, you can move to more specific details, such as whether to solve the task on the user space level or in the kernel. While it’s always preferable to solve embedded Linux development tasks in the user space, there are several cases when doing it in the kernel would be the better (if not the only possible) choice:

• When you don’t have a driver for a specific device you’re working with
• When the speed of network packet processing is crucial
• When it’s required by the task (for instance, when building a file system filter driver)

 

Figure 1. Three cases when we solve tasks in the kernel

Let’s look closer at some of the issues of Linux embedded estimation that you may face when working in the user space and in the kernel.

Solving tasks in the user space

When working on the user space level, there are many open-source frameworks that you can use, such as Buildroot and Yocto. By reusing even part of the code, you can significantly reduce time spent on the project.

Plus, when you develop a program in the user space, the system usually remains stable. The most basic algorithm for solving a task in the user space consists of two steps:

1. Writing a piece of code
2. Compiling and launching the code

Even in the worst case, the code will crash and the system won’t even give you a readable error message, but the system itself will stay functional. However, things work differently in the kernel.

Solving tasks in the kernel

When solving tasks in the kernel, borrowing a piece of code isn’t an option, especially if you need to work with a specific motherboard. Even if there are suitable drivers, they may not expect or even be able to process the behavior you want to program for a particular board model.

In this case, you’ll first need to examine the driver’s source code and then configure the driver for the specific task at hand. But debugging and realizing the improved driver will surely take additional time.

Plus, a driver failure can crash the entire system. In you’re lucky, after the reboot, the system will be recovered to the point before the problem driver was loaded. Otherwise, you’ll need to reconfigure the whole system from scratch.

This is why it’s better to check and debug the driver code using a virtual machine instead of a real-life system, so you can roll to a snapshot if something goes wrong. However, since your kernel runs on and is configured for a specific motherboard, finding a fitting emulator can be a challenge.

You also need to consider whether there are available interfaces for communicating with the board (UART, Ethernet, USB), as well as the availability of the graphical user interface.

The next thing to decide is whether you need to write a prototype before starting to work on the project itself.

Writing your own prototype

It’s noteworthy that in most cases, a Linux embedded development project is meant for a test stand built at the beginning of a bigger project. Therefore, creating a proof of concept (PoC) that will successfully perform on a motherboard used in the final project is very unlikely.

You need to find out if there’s a kernel you can use for this board or if you need to build it from scratch. You can also try to create sample PoCs in the current kernel for your device or take sample code from the kernel documentation and use it to process an event.

If your project can allocate user space code, then you can write a sample that you’ll later transfer to the kernel. For instance, in the case of the video streaming project, you’ll need a GStreamer pipeline to create a real-time streaming protocol (RTSP) media data flow and a receiver on the server side.

When your task requires focusing on a specific device, it’s best not to waste time on prototyping because writing a prototype may take just as much time as the project itself. Instead, turn your attention to similar realizations for other device models and research their source codes.

Also, if there’s a common driver that can be used instead of a chain of drivers responsible for the system interaction with the device, you should work with this driver. If there isn’t any, go to the specifications provided by the board manufacturer and consider its signal processing as the first iteration, then move iteratively.

When working on a graphics adapter device driver, you should remember about possible loading and debugging issues. You’ll need to debug the BIOS without being able to see what’s going on on the screen.

For instance, a graphics adapter driver doesn’t load and you can’t log the event because the file system isn’t installed yet. In this case, you can debug the driver remotely with the help of a QEMU emulator:

• Turn off the load of the graphics adapter driver on the target machine and add the text parameter to the kernel boot line.
• Start a Virtual Network Computing (VNC) server:

#!/bin/sh
DISPLAY=:1 & /usr/bin/Xvfb :1 -screen 0 1920x1080x24 & fluxbox -display :1 & x11vnc -display :1 -repeat

• Connect to the server using any VNC client and try to launch the QEMU emulator with a passthrough to your graphics card. You can get detailed information about different models and parameters here.
• It would also be helpful if you could unload the content of the quest console from the virtual machine. To do this, you need to add the -serial stdio command to the console parameters, and on the virtual machine, add console=ttyS0 to the kernel boot line and the config file: /etc/init/ttyS0.conf

start on stopped rc RUNLEVEL=[12345]
stop on runlevel [!12345]
respawn
  
exec /sbin/getty -L 115200 ttyS0 vt102

• As a result, we’ll see the whole log and even be able to debug the load of the graphics adapter driver.

Of course, this is only a small portion of the methods and ideas that we’ve found useful when working on our embedded Linux projects. We also have three small tips that will help you to estimate the time needed for realizing your project with greater accuracy.

Three more tips

Note that we aren’t talking about all the possible development risks; we’re only focusing on specific problems related to estimating embedded projects. Below, we list three things that you should take into account when estimating the technical part of the project.

 

Figure 2. Three things to take into account when estimating a Linux embedded project

Let’s look closer at each of these three factors.

Supported kernels and devices. The first thing you should clarify is the list of devices and kernels that your embedded solution should support. After all, even switching from one minor kernel version to another may lead to a fatal time overage.

The form of the final solution. The form you’re planning to deliver the final solution in is also important. Your prototype may be quite performant and the final product may launch successfully on your motherboard, but you’ll still need to make a board support package (BSP). The content of such a BSP should be specified at the start of the project. To make this task a bit easier, you can use tools and data provided by such open-source projects as Yocto.

Little-known aspects of development. Finally, remember to clarify the aspects of development that project leaders might be unaware of. For instance, it’s important to explain that in the kernel, there aren’t any high-level frameworks and tools available in the user space. So when you, for instance, load some files (as well as the file system) in the kernel, you can’t log information anywhere. And the time needed to find an alternative solution, such as the use of a QEMU emulator for debugging the graphics adapter driver in our example above, should be taken into account at the project estimation stage.

Conclusion

Linux embedded system project estimation is a challenging task for both project leaders and developers. There are lots of factors that should be taken into account in order to avoid unnecessary time overages and use your resources wisely.

Ideally, if you can solve the task in the user space, then you should. However, if you need to solve a task in the kernel, you need to clarify a few things:

• Does the project require a specific kernel?
• What problems may you face related to developing this particular device driver?
• What interfaces and what debugging options are there for this particular device?

In some cases, you might want to first write a prototype in the user space or find a similar driver for a different device that might work on the target model.

It’s also important to discuss the form of product delivery, including the versions of supported kernels and the set of drivers that might be needed for a specific motherboard. And finally, make sure to explain to your business partners what specific issues you might face when working on this project and how much time it may take to deal with them. Even if some of these issues seem obvious to you, stakeholders might have no idea about them.

At Apriorit, we have a team of dedicated Linux kernel and driver developers who’ll gladly assist you in accomplishing even the most complex and challenging tasks.