white paper

The Shortest Path to Embedded Linux Deployment:

But the distribution model is only one Introduction approaches for specifying and building
mode for propagating open source Starting from its inception in 1993, the Linux-based platforms to meet specific
platform software. The Linux kernel, GNU/Linux operating system has found developer requirements, with source
along with myriad libraries, tools, utilities, its way into the hands of developers code as starting point instead of fixed
middleware and applications, evolves at and end-users via “distributions” – collections of binary packages.
a rapid pace, offering ongoing improve­ aggre­gations of the Linux kernel, GNU
ments in code quality, performance libraries, utilities and tools, and selected Linux Distributions
and capabilities. To accompany this other software. History and Rationale
evolution configuration-based approaches Starting with Slackware in 1993, open Distributions originated in the early
have evolved for specifying and building source distribution projects, commercial 1990s, a time when Internet access
Linux-based platforms to meet specific vendors and other organizations made was slow and not widely available. The
developer requirements, with source a wide range of distributions available to source code to the Linux kernel and the
code as a starting point instead of fixed professional developers, IT departments rest of the OS, and also just the binaries
collections of binary packages. and hobbyists in the form of CD-ROMs, built from it, consisted of several million
This white paper explores the downloadable disk images and other lines of C and assembly language.
merits of distribution and configuration- repositories. Originally, these distribu- Source and/or binaries could days to
based approaches to embedded Linux tions targeted workstations and servers; download, fill up a stack of floppy disks
development. It compares and contrasts circa 1999 the definition of a distribution and require many hours to configure
the two paradigms, with particular expanded to include embedded and and build and install a working system.
attention to how each supports the mobile development toolkits. As a practical matter, the Linux de-
needs of today’s embedded systems veloper community sought to make it
developers with functionality and ”The motto of Open Source Software is: easier to install and run the open source
access to the most up-to-date open Release Early, Release Often” OS, when possible emulating positive
source code. To illustrate the compa- user experience from commercial OS
rison, it examines the popular Open The distribution model is only one mode offerings (Windows, MacOS, Solaris, etc.).
Embedded project as well as commercial for propagating open source platform The most straightforward approach
offerings derived from it. software, and may not be the most proved to be bundling a set of known-
appropriate for targeting embedded compatible kernel(s), tools, libraries and
Who Should Read development. The motto of open source other packages together, along with an
This White Paper? is “Release Early, Release Often”, and the installer to create a working file system
While this document should interest a Linux kernel, along with hundreds (or on a desktop PC or server. Distribution
broad audience of intelligent device thousands) of libraries, development disks were easily copied and handed
developers, the primary audience includes tools, utilities, middleware and ready- out at conferences, user groups and
n Embedded systems architects to-use applications, evolves at a rapid through other channels. Today the
n Device development team managers pace. Monthly, weekly or even daily, tradition continues. Dozens of distri-
n Systems integrators and in-house new software versions become availa-
integration teams ble, offering incremental improvements
n Quality assurance engineers in code quality, performance and capa-
bilities. To accompany this progression,
there have evolved configuration-based

Enea is a global software and services company focused on solutions for communication-driven products. With 40 years of experience Enea is a world leader in the development
of software platforms with extreme demands on high-availability and performance. Enea’s expertise in real-time operating systems and high availability middleware shortens
development cycles, brings down product costs and increases system reliability. Enea’s vertical solutions cover telecom handsets and infrastructure, medtech, automotive and
mil/aero. Enea is listed on Nasdaq OMX Nordic Exchange Stockholm AB. For more information please visit [Link] or contact us at info@[Link]. [Link]
 white paper

butions target mainstream and niche embedded distributions and tool kits Embedded Distribution Value-added
markets, including Debian, Ubuntu, duplicate the appearance of Linux host Embedded systems tools vendors (OSVs
Red Hat, Novell/SuSE and lesser-known file systems and contents, in microcosm. – Operating Systems Vendors) set out
ones like Linpus, MEPIS and Yellow Dog. to add value to community embedded
Cross-Embedded Functionality Linux in a variety of ways:
Device-specific and To support cross development, key n Validation of interoperability among
Embedded Distributions commercial and community developers kernel, tools and user packages
Years later, developers and embedded working on the Linux kernel, GNU tools n Commercial quality board support
systems commercial tools companies and other projects strove to support n Vendor and distribution-specific
sought to apply the same model to functionality, some taken for granted open-source and proprietary deve-
distributing, installing and developing by legacy embedded developers, other lopment tools
Linux-based software for intelligent relatively new to embedded: n Vendor-specific patches, patch sets
devices. The earliest efforts targeted n Cross compile from C, C++ and and system functionality
pre-built boards and handheld devices other languages to target binaries n Dedicated support and custom
– VME systems, PC/104, HP iPaqs, etc. n Create deployment images for use development services
using the desktop distribution model with JTAG emulators, device pro-
with slight modification to allow for grammers, and architecture specific Embedded Linux
Power Architecture, 68000, ARM and boot-ROMs Platform Builders
MIPS CPUs and non-standard booting n Build and populate file system Embedded Linux OSVs seek to supply
sequences. images on the host for target developers with an entire workshop,
deployment from tools to nails and screws to
Cross Development n Different tools for system and user lumber and sheet metal. Platform
Starting in the early 1970s, semiconductor space development (e.g., KGDB, builders, by contract, focus on tools and
suppliers and third-party software vendors GDB and gdbserver) configurations to work on Linux soft-
began supporting cross dev­elopment ware available directly from community
of embedded software, offering code Embedded Distribution Contents repositories. But platform builders are
creation and maintenance on minicom- An embedded Linux distribution can more than “lite” distributions.
puters, mainframes and eventually PCs resemble desktop or server distributions
and workstations. In the days of 8-bit with a few key differences: Configuration vs. Contents
CPUs, development hosts outstripped n One or more non-desktop architec- When commercial embedded Linux
embedded targets in available memory, ture support directories and utilities distributions first emerged (c. 2000),
clock speed, storage and other attributes n Cross development tools for Power the developer community was excited
– embedded systems were bare-bones Architecture, ARM, MIPS and other to see Linux designed into ubiquitous
affairs. Later, as 16 and 32-bit systems embedded CPU families consumer, infrastructure, industrial and
emerged, target systems achieved rough n Big-endian and Bi-endian architec- mobile applications. As with commer-
parity with development hosts, but still ture support, as appropriate cial desktop and server distribution,
departed from them in terms of CPU n Embedded versions of run-time however, community interests wanted
architecture and use cases. libraries and utilities, e.g., software- to preserve non-commercial open
By the time that embedded Linux only floating point, smaller footprint source options for embedded develop-
made its debut in 1999-2000, cross (uClibC instead of glibc, ash or ment. Initially, this desire was realized
development for bare-metal and RTOS busybox vs. bash) in creation of device-specific distribu-
based target systems was mature with n Hundreds rather than thousands of tion for iPAQ and similar hardware
de facto standards for building and cross user-space packages (RPMs, DEBs, etc.) ([Link]), and with low-level
loading. Most development occurred horizontal distributions like Embedded
on Windows and Solaris hosts with For hosting on non-Linux development Debian. These projects quickly faced
download mediated by in-circuit and systems (primarily Windows), add the same challenges as commercial
ROM emulators and boot ROMs. cross file system creation, mounting distribution suppliers – endless mul-
Embedded Linux perturbed this and population tools (Linux hosts can tiplication of supported hardware and
state of affairs: developers began swit- mount target file systems as “loopback” configurations and special-purpose
ching to Linux desktop host systems devices) and usually versions of NFS. patches and modules needing to keep
(initially Red Hat) and target systems But similar to the desktop, embedded up with rapid [Link] evolution.
took on the attributes of development distributions usually focus on binary/ In response to this difficult dynamic,
hosts – file systems, networking, etc. object code; source is available on a
However, host and target systems still separate disk and vendor SLAs often do
needed to support different architectures not support developer-derived binaries
and CPUs, have unique board support from source.
and other key attributes, requiring that
 white paper

a number of projects began to focus on (board support packages (BSPs), media projects”, with or without commercial
configuration instead of contents. The CODECs, drivers, etc.), platform builder organizations standing behind them.
most visible and successful of these suppliers’ hosted repositories (e.g., Today, the platform builder paradigm
projects has been [Link], Enea), other third-party system code and supporting technologies have
from which the [Link] project grew (ISVs, integrators et al.) and system code greatly matured, as has embedded
and upon which many commercial plat- built and maintained by device manu- industry commercial support for them.
­form builder implementations are derived. facturers themselves. If you haven’t already done so, it is a
A platform builder is used to narrow worthwhile exercise to re-evaluate the
Architecture this broad universe of source code into platform builder concept, especially in
Traditional embedded development takes a manageable subset and integrate it contrast to embedded Linux distribu-
place on a workstation and occasionally into a custom (or customer-specific) tions, which by comparison reflect a
on dedicat-ed/embedded hardware distribution for day-to-day development decades old technical approach.
itself. With embedded Linux, develop­ use. That customized platform can of
ment first required the installation of an course be further optimized and ultimately Comparing Distribution and
SDK and accompanying Linux distribution. integrated with value-added application Platform Builder Paradigms
A platform builder begins with a code to power a finished product. A viable comparison between distribu-
more minimalist installation – just the tion and platform builder paradigms
tools needed to create configurations Paths to Deployment covers a range of activites and capabili-
and build code, starting with cross Distributions carry a lot of baggage. ties, from building from source code to
compilers. It is a bootstrap approach, Their content aims to please a wide customization and beyond:
proceeding from meta-tools to cross range of developers and industries,
development tools and images to with highly targeted distributions Build from Source
integration and deployment. evolving to engage specific application Most embedded Linux distributions
The Enea Linux PlatformBuilder is types: Carrier Grade Linux and telecom provide kernels, packages, tools, etc. by
a desktop environment that can also infrastructure, Mobilinux™ and cell default in binary object and executable
be hosted on shared servers, in a data phones, etc. This omnibus philosophy form, with source in a separate repository
center or even in the Cloud. The main causes Linux distributions to grow or at least on a separated CD/DVD.
configuration interface is Eclipse-based over time. More content means more Starting from binaries instead of source
– when developers have established the development and more testing and stems from (legacy) needs to speed
particulars of a trial or final configuration, QA impose higher costs and longer builds and installation on desktop
build proceed with a click of button. product cycle on embedded toolkit distributions, and carries over into
vendors, which they pass along to embedded – a full embedded Linux
Work Flow device developers as higher “per seat” distribution can encompass 30-40
A platform builder, rather than force-fitting prices and fewer releases per year. million lines of code and were (and are
a set of “pre-configured bits” on deve- Platform builder technology has still) non-trivial to build from source on
loper projects, can draw from multiple long been available from open source older workstations.
source bases, including [Link] and projects like [Link] and In compliance with the GNU General
myriad FOSS project repositories, system more recently from [Link]. Develop- Public License (GPL) and other license
software supplied by semiconductor ment managers historically regarded regimes, embedded distributions also
vendors and board manufacturers these and derived products as “science include source for all (or most) binaries
shipped in them. In practice, however,
supplied sources often do not repro-
Other Community Semiconductor
[Link] ENEA duce actual distribution binaries, and in
Projects Suppliers
Linux Kernel, etc. Value-added SW
Tools, Utilities, etc. Drivers, BSPs, etc some cases are not buildable at all.
Best practices for platform builders
(e.g., Enea Linux PlatformBuilder) start
each system build fresh from source
Platform Builder
Internal code. Sources can be local/custom
Configure Check Build Development
or hosted on shared repositories from
suppliers and project hosting sites. To
accelerate builds, binary objects can be
Custom
cached for linking as needed.
Platform Develop Integrate Build End Product
Distribution

Device Manufacturer

Figure 1. Platform Build Workflow – from Source Repositories to End Product.

Platform Customization groupings of key software components, software components to match his
Distributions can take on lives of their but do not create the ossified core that project schedule and his company’s
own. To constrain them, suppliers do defines the distribution model. style. During R&D, a platform builder
not outright prohibit customization, can be used to integrate “cutting edge”
but usually greatly narrow its scope. Release Cycle and experimental versions of kernels,
When developers stray too far from Even extremely aggressive and agile drivers, middleware, utilities, etc. Dev­
default configurations (add/remove distribution suppliers succeed in provi- elopers can choose just the “right set”
packages, rebuild kernels with custom ding 2-3 updates to their product per of software components and versions
configurations, remove modules, etc.), year. By contrast, [Link] typically to meet their project needs. When an
they can descend into “dependency proceeds at a much faster rate, with OEM product ships, the configurations
hell”, spending hours or days resolving quarterly major updates and more and bills-of-material produced by a
versioning and interoperability issues frequent minor ones. Moreover, the platform build can help support that
among software components. And release cycles of the hundreds of product moving forward, throughout its
most or all such customization falls out- other projects of interest to embedded own life-cycle, integrating and docu-
side distribution supplier Service Level developers also advance at their own, ment the original contents and any
Agreements (SLAs). unsynchronized paces. changes to it.
Building from source with both pre- The result is that when a distribu-
determined configurations (templates) tion supplier actually ships a new Board Support
and fully custom options lets developers distribution version, it is by definition Industry analysts have termed em-
create and optimize a Linux-based plat- out of date: bedded board support “table stakes”
form to meet real embedded applica- n Included/supported kernel will – if you don’t support the CPU family
tion needs. Developers can update or have been chosen early on, months and/or a physical board that at least
remove default components and add before code freeze loosely corresponds to OEM customer
new ones, with automatic dependency n Ditto for the package sets included, hardware, then you are not invited
checks. Platform builders need to especially key elements like GNU tools to play. The result is that OSVs invest
resolve dependencies anyway, making n OEMs receiving a “new” distribution copious resources in integrating drivers
it easier for OEMs and distribution will test and integrate it over the and configurations needed to support
suppliers to support customization. course of weeks (or not at all, reference hardware “out of the box” as
depending on their own release part of standard embedded Linux dist-
Kernel and Package Versions cycles), arriving on developers’ desks ributions. Embedded Linux distribu-
Standard practice among distribution several months after the OSV ships it tion suppliers even compete with one
suppliers is to choose a stable, mostly another on the basis of “my board farm
up-to-date version of Linux from the The result is that the contents of a is bigger than yours”.
[Link] tree(s), apply a vendor- typical embedded Linux distribution BSP farming is actually a dange-
specific patch set, and aggregate and represents a snapshot of technology rous trap. The more boards a vendor
test libraries, utilities, and tools against 6-12 months behind current kernel supports, the more time it takes to pro-
it. This sensible procedure delivers a and project releases. While “stable”, ductize and ship a new release – every
stable, testable and easily supportable it may not be supportable – many board, every configuration and device
platform that meets vendor needs for smaller projects simply do not have driver must be rebuilt and validated
merchantability and support. However, the resources to fix bugs found in older against new kernels and other system
it does little to address real-world OEM releases and to back-port them; OSVs software. New kernels dictate new
customer requirements. do, but then end up supporting a set device drivers and/or porting old ones
Embedded developers want to of per-project forks, in some cases forward; needed modules that worked
be able to track, experiment with and different for each customer. The sheer with a legacy release may not work with
integrate new versions of the Linux kernel number of available project/packages a new one. Over time, vendors suffer
and other key projects (Webkit, file means that OSVs either integrate/sup- from “BSP fatigue” and typically scale
systems, gstreamer, etc.). Up until their port a small subset of them (~150) or back to fully supporting a small core of
own code freeze and delivery dates, provide minimal, best-effort support for hardware, leaving a majority of boards
OEMs need to be “looking forward” – to a greater number. and partners and customers in limbo.
be able to access and receive support To add insult to injury, distribution The BSP approach also does not
for building on the “freshest” open source suppliers typically support a given service actual customer requirements.
technology, not peering backward in product release for 12-14 months. If BSPs address the particulars of semi-
time at deprecated software versions. an OEM needs to maintain kernels and conductor vendor reference boards and
A platform builder methodology packages of an older vintage, they must
by definition lets developers choose pay a premium or proceed on their own.
among kernel and package versions. By contrast, platform builder users
Platform builder templates suggest valid can tailor the versioning of Linux and
 white paper

other commercial off-the-shelf (COTS) supports customer-specific CPU, perip­ port for them. OEMs could themselves
hardware. An additional step remains herals and software configuration. aggregate, integrate and support a
to support actual OEM product sys- Customizing and rebuilding a fixed Linux-based platform; OSVs are ef-
tems, which variously clone reference distribution is a thankless, laborious and fectively outsourcing the process and
platforms or substantially depart in exacting task. By contrast, a platform responsibility for productizing commu-
provisioning and configuration. This builder can automatically create a nity-based software development and
divergence requires additional engine- customized distribution – replete with distribution.
ering and integration, either on the part kernel, select packages, utilities, tools It is not, however, in the interest
of the OEM or as a service from the OSV. and board support – ready for use by of OSVs to facilitate changing vendors
BSPs, then, are really all-or-nothing an OEM, its channel and partners. OEMs or using broadly available FOSS in lieu
affairs: they support your particular can re-distribute this derived platform of or together with their commercial
hardware or they don’t. Platform and tools internally to support systems wares. Distributions by their very nature
builders, by contrast, take a more in- developers, application programmers, exert a “gravitational field” that keeps
cremental approach to board support. release engineering, and test and QA customers in orbit:
Once developers select a kernel version teams. They can also make it available n Selection of Linux kernel and
for a given CPU, they can choose sup- externally to resellers, support channel application of patch sets,and cross
ported devices one at a time and/or engineers and other partners. validation with other software
use vendor-supplied templates that n Establishment of vendor-specific
cover standard peripherals and known Support and SLAs build regimes that are hard to
boards (e.g., those present on SoCs Distribution vendors invest a lot in reproduce and that obfuscate inte-
and reference boards). They can also aggregating, integrating and produc- raction of open source components
integrate source code for community- tizing FOSS components to create a n Inclusion of proprietary compo-
supported drivers and for interfaces branded distribution. The return on nents and use of closed-source
sourced from third parties, especially that investment comes in the form of installation tools
semiconductor vendors. With the product subscriptions and support n Outsourcing “too much” and dis-
addition of memory maps and other contracts. To scale this model, OSVs couraging independence, leaving
configuration information, a platform must be very prudent with the scope of OEM customers overly dependent
builder then creates a custom distri- provided support, defined via Service upon OSVs
bution for real development and/or Level Agreements (SLAs). The result is The platform builder paradigm does
production hardware, speeding time an ever-narrowing band of functionality not guarantee that such lock-in won’t
from prototype to development. and configurations actually support by occur. However, by simplifying integra-
an OSV. Even with an impressive “board tion of more types of FOSS and com-
Custom Distribution Creation farm” – a stable of reference hardware mercial software from a greater variety
Distribution suppliers’ business models and BSPs to accompany it – OSVs of sources, and by producing clear
entail creating a set of static platforms over time try to turn everything and configuration files and software bills-
designed to cover as many types of anything that diverges from reference of-material, a platform builder eases
embedded designs as is practical. They hardware into an “extra”: an unsupported transitioning among FOSS components,
build and market distributions targeted feature that requires incremental support platform builder tools, and when neces-
at vertical applications (consumer contracts and fees. sary, among suppliers.
electronics, telecommunications, etc.), Platform builders start in a different
at specific CPU architecture (ARM, place. They enable OEMs to create their Summarizing the Two Approaches
Power, etc) and in some cases at very own distributions – tools, BSPs, kernels The following table summarizes the
specific hardware (e.g., ATCA chassis and configurations. Vendors offering plat­-­ difference between distribution-based
and evaluation boards). This hierarchy – form builders (like Enea) not only support and platform builder approaches:
application type ¢ CPU architecture ¢ the platform builder itself, but (within
reference board type – can come close reason) the platforms derived from them. Conclusion
to actual OEM system support, but
always leaves a gap in actual hardware Vendor Lock-In ”Platform builders are simply the logical
and software configurations. Freedom is the main tenet of Free and conclusion of distribution evolution”
In today’s world of highly optimized Open Source Software (FOSS), and the
hardware combined with standard and majority of the software supplied by The purpose of this white paper has
shrink-wrapped software, OEMs need OSVs in a distribution is “free” (in both been to examine the de facto practice
embedded Linux suppliers to come clo- senses of the word). Choice of OSV,
ser to supporting their end products. A then, is based on a number of factors,
straightforward mechanism to achieve including knowledge of and participa-
this end is through creation of a “custom tion in FOSS development, quality of
distribution”, that is, one that precisely derived commercial products, and sup-
 white paper

of developing intelligent devices and These critiques are not mean to damn core, modern multi CPU and multicore
applications based on embedded Linux distributions and banish them from devices based on the Linux operating
distributions. In particular, in compa- embedded development. Distributions system. A joint collaboration between
ring distribution-based development serve a valuable role in the ecosystem, Enea and Timesys Corp., Enea Linux
approaches with developing Linux sys- as reference aggregations and to sup- PlatformBuilder includes the award-
tem software using platform builders, port COTS hardware (e.g., Carrier Grade winning Timesys LinuxLink Factory
several important differences come to Linux on standard ATCA systems). But custom distribution build system, the
light: standard, fixed commercial distribu- Eclipse-based Enea Optima for Linux
n Distributions strive to be compre- tions actually impede development IDE, expert support for select embed-
hensive (and usually fail); platform and integration of value-added custom ded platforms and global profes-
builders are minimalist but cater to embedded applications by forcing sional services team for value-added
divergent developer needs OEMs to follow OSVs practices, by consulting and development. Enea
n Economics of distributions creation limiting technology options, and by Linux PlatformBuilder reduces the time
narrow the universe of available requiring incremental support and/or your embedded engineers spend on
open source components; platform customization expenditures. undifferentiated development work
builders can accommodate a wider Platform builders are simply the and allows them to focus on building
range of kernels, drivers, middle- logical conclusion of distribution evolu- and bringing competitively superior
ware, tools and utilities with lower tion. They are meta-distributions – sys- products to market faster.
overhead tems that create custom distributions n Start benchmarking and proto-
n Distributions approach OEM hard- to support the production hardware in typing applications on multiple
ware via broad-based reference actual OEM products. reference platforms in minutes/
platform support; platform builders hours without having to set up the
incremental target actual OEM Enea Linux PlatformBuilder hardware environment
customer hardware ”Easy-to-use, Elegantly Simple, n Perform advanced customizations
n Distribution-based software is out- and Cost Effective Deployment and easily integrate 3rd party soft-
of-date from before OSVs release it; of Embedded Linux Designs” ware (even binaries) in your existing
platform builders let OEMs choose build system
and integrate cutting-edge, stable Enea® Linux PlatformBuilder is a n Quickly create multiple, repeatable
and/or legacy versions of the Linux compelling, easy-to-use, cost-effective platform configurations
kernel, packages and tools offering that lets you quickly configure, n Delivers a wide array of online ex-
build, test, debug and maintain applica- pert development advice, proactive
tion and platform software for single updates, training, a vast repository
of Linux packages, ready for cross-
compile, and much more
Summarizing the Two Approches n Complete C/C++ IDE and source
Distribution Platform Builder code debug support based on
Build from Source included but often not Always from source
open source standards, such as
Source buildable Eclipse and GDB.
Platform Limited by SLAs and default Fully customizable n Value added professional services at
Customization configuration both the Linux OS and application
Kernel and Package Current at distribution release time – Cutting edge or conservative, as level are supported by Enea’s global
Versions 6-12 months old needed by OEMs software consulting organization.
Release Cycle Follows OSV release schedule, months Lets OEMs integrate latest FOSS
behind [Link] and other projects source bits and/or continue working
with older versions
Board Support Target reference hardware Configure to actual OEM systems
Custom Distro Only standard distributions Supports custom distribution creation
Creation supported – hard to customize by default
Support Terms Narrowly support distribution content Support creation of custom
distribution and its use
Lock-in Limit visibility, use modes to vendor- Impose minimal strictures and admit
specific code and build engines commercial, community and internal
code

Figure 2. Comparing Distribution and Platform Builder Paradigms.

Enea®, Enea OSE®, Netbricks®, Polyhedra® and Zealcore® are registered trademarks of Enea AB and its subsidiaries. Enea OSE®ck, Enea OSE® Epsilon, Enea® Element, Enea® Optima,
Enea® Optima Log Analyzer, Enea® Black Box Recorder, Enea® LINX, Enea® Accelerator, Polyhedra® Flashlite, Enea® dSPEED Platform, Enea® System Manager, Accelerating Network
Convergence(TM), Device Software Optimized(TM) and Embedded for Leaders(TM) are unregistered trademarks of Enea AB or its subsidiaries. Any other company, product or
service names mentioned above are the registered or unregistered trademarks of their respective owner. WP56 042011. © Enea AB 2011.