Емеленко А.Н., Маллачиев К.А., Пакулин Н.В. Разработка отладчика для операционной системы реального Emelenko A.N.

N., Mallachiev K.A., Pakulin N.V. Developing a Debugger for Real-Time Operating System. Trudy ISP RAN
времени. Труды ИСП РАН, 2016, том 28, выпуск 2, с. 193-204. /Proc. ISP RAS, 2016, vol. 28, no 2, pp. 193-204.

1. Introduction
Application debugger is an indispensable tool in developer’s hands. But debugger in
a real-time operating system is more than just plain debugger. In this paper we present
an on-going project on debugger development for JetOS, a real-time operating system
Developing a Debugger for Real-Time that is being developed in the Institute for System Programming of the Russian
Academy of Sciences.
Operating System JetOS is a prototype operating system for civil airborne avionics. It is designed to
work within Integrated Modular Avionics (IMA) architecture and implements
ARINC 653 API specification, the de-facto architecture for applied (functional)
1, 2
A.N. Emelenko <[email protected]> software.
1, 3
K.A. Mallachiev <[email protected]> The primary objectives of ARINC 653 are deterministic behavior and reliable
1, 3
N.V. Pakulin <[email protected]> execution of the functional software. To achieve this ARINC 653 imposes strict
1
Institute for System Programming of the Russian Academy of Sciences, requirements on time and space partitioning. For instance, all memory allocations and
25, Alexander Solzhenitsyn st., Moscow, 109004, Russia. execution schedules are pre-defined statically.
2
Moscow Institute of Physics and Technology (State University), The unit of partitioning in ARINC 653 is called partition. Every partition has its own
9 Institutskiy per., Dolgoprudny, Moscow Region, 141700, Russia memory space and is executed in user mode. Partitions consist of one or more
3
Lomonosov Moscow State University, processes, operating concurrently, that share the same address space. Processes have
GSP-1, Leninskie Gory, Moscow, 119991, Russia. data and stack areas and they resemble well-known concept of threads.
Embedded applications might be run in two different environments: in an emulator
Abstract. In this paper, we report on the work in progress on the debugger project for real-time
operating system JetOS for civil airborne systems. It is designed to work within Integrated
and on the target hardware. In our project we use QEMU system emulator. Although
Modular Avionics (IMA) architecture and implements ARINC 653 API specification. This QEMU has its own debugger support, its functionality proved to be insufficient for
operating system is being developed in the Institute for System Programming of the Russian debugging embedded applications. Therefore, we implemented a debugger not only
Academy of Sciences and next step in developing this system is to create a tool to debug user- for the target hardware, but for the emulator as well.
space applications on it. We also discuss the major requirements to such a debugger and show
the difference between it and typical debugger, used by desktop developers. Moreover, we 2. Main Targets for Debugger
review a number of debuggers for various embedded systems and study their functionality.
Finally, we present our solution that works both in emulator QEMU, which we use to emulate Debugger for an embedded operating system has a number of specific features
environment for our system, and on the target hardware. The presented debugger is based on compared to typical debugger used by desktop developers.
GDB debugging framework but contains a number of extensions specific for debugging Firstly, an embedded application runs under constrained conditions, such as limited
embedded applications. However, the implementation of the debugger is not complete yet and on-board resources and lack of interactive facilities – no keyboard and screen. This
there is a number of features that can improve debugger usability, but it is already more makes it impossible to do debugging on the same device where application runs.
functional than common GDB debugger for QEMU and, in contrast to other systems and their Therefore, the debugger for embedded applications has to be remote: the developer
debuggers, where developers can use some functions to debug applications, but not all we need,
interacts with workstation while the application runs on a target hardware.
our debugger meets the majority of our requirements and restrictions.
Secondly, an embedded application typically consists of a number of interacting
Keywords: debugger; GDB; real-time OS; remote debugger processes that needs to be debugged simultaneously. This means that the debugger
DOI: 10.15514/ISPRAS-2016-28(2)-13 must support dynamic and transparent switching between execution contexts during
debugging session.
For citation: A.N. Emelenko, K.A. Mallachiev, N.V. Pakulin. Developing a Debugger for
Real-Time Operating System. Trudy ISP RAN/Proc. ISP RAS, vol. 28, issue 2, 2016, pp. 193-
Thirdly, the debugged should support developers of system software, mostly device
204. DOI: 10.15514/ISPRAS-2016-28(2)-13 drivers and network stack. This requires switching between low-privilege code and
highly privileged kernel code in the same debugging.
It is also important to mention that embedded developers widely use emulators in
their work process. Typically most of development runs on top of emulators, therefore
the debugger must support corresponding emulators as well.
193 194
Емеленко А.Н., Маллачиев К.А., Пакулин Н.В. Разработка отладчика для операционной системы реального Emelenko A.N., Mallachiev K.A., Pakulin N.V. Developing a Debugger for Real-Time Operating System. Trudy ISP RAN
времени. Труды ИСП РАН, 2016, том 28, выпуск 2, с. 193-204. /Proc. ISP RAS, 2016, vol. 28, no 2, pp. 193-204.

The above mentioned features impose a number of restrictions on the design of the  sets breakpoints;
debugger that we considered:  does single step;
 There are many different applications compiled in OS, which can have  provides reading/writing in memory;
overlapping virtual address spaces.
 provides reading hardware registers;
 Typically target hardware board for embedded OS has only one port to
 support interprocess communication (IPC) monitoring.
communicate with the external world – a single serial port. Since it is used
to stream console output of the running applications we need to share it The Fiasco Kernel Debugger (JDB) is a debugger for Fiasco. It has the following
between debugger traffic and applications' output. special functionality:
 Multifunctional debugger is a complex program. It is very complicated to  It always freezes the system when it is working. It means that JDB disables
develop it from scratch, so we decided to base our debugger on an existing all interrupts and halts clock. All processes and kernel don't work when JDB
one. is invoked.
 Support debugging both on hardware and with emulator because this support  JDB doesn't use any part of Fiasco kernel, because it is a stand-alone
can expand developers' capabilities and improve their efficiency. debugger with drivers for keyboard, display, etc.
In general, JDB is not a part of Fiasco µ-kernel, and Fiasco µ-kernel can run without
 Support capabilities of debugging for kernel and for user mode code, as well
connection with JDB or another debugger.
as capabilities of multiprocess mode.
The debugger operates remotely over the serial line.
 It must excel QEMU debugger, which we use to emulate environment for
our system.
3.2 VxWorks
 Since the OS in question is real-time, it is important to minimize debugger's
impact on system during debugging. VxWorks [5] is a real-time operating system (RTOS) developed as proprietary
software by Wind River of Alameda, California, US. It supports Intel (x86, including
In order to meet these restrictions we selected the architecture of remote debugger
the new Intel Quark SoC and x86-64), MIPS, PowerPC, SH-4, and ARM
with server and client parts, that communicate over a serial port using multiplexer.
architectures.
We have chosen GDB (GNU debugger) for the client part of our debugger.
RTOS debugger for VxWorks implements the following set of features:
3. Related Works  Task Stack Coverage
 Task Related Breakpoints
We are not the first to consider the problem of remote debugging. For example,
Pistachio microkernel uses kdebug for debugging [4]; besides, there is Fiasco  Task Context Display
debugger [1] and many different debuggers for VxWorks, for example, RTOS  Debugging Modules (for example, Kernel module)
debugger [2].  Debugging Real-Time Processes
Here we briefly consider some debuggers for embedded OSes and their primary  Debugging Protection Domains
features.  Collecting statistics for function and tasks
RTOS debugger displays all system states, tasks, message queues, memory
3.1 Fiasco OS partitioning, modules and etc.
Fiasco OS is a 3rd-generation microkernel, based on L4 microkernel [1]. The kernel The key feature of the RTOS debugger is that is based on Lauterbach's TRACE32
is simplistic, it misses most of the features available in “big” operating systems like debugger [3] that utilizes hardware interfaces like JTAG. It does not use serial port
Linux or Windows: program loading, device drivers and file system. All these for communication with the target hardware but rather requires specific debug
features must be implemented in user-level programs on top of it (L4 Runtime module.
Environment provides a basic set such functions).
Fiasco OS has built-in support for debugger that: 3.3 L4Ka::Pistachio
 supports threads; L4Ka::Pistachio [4] is the latest L4 microkernel developed by the System
 provides stack backtrace Architecture Group at the University of Karlsruhe. It is the first available kernel

195 196
Емеленко А.Н., Маллачиев К.А., Пакулин Н.В. Разработка отладчика для операционной системы реального Emelenko A.N., Mallachiev K.A., Pakulin N.V. Developing a Debugger for Real-Time Operating System. Trudy ISP RAN
времени. Труды ИСП РАН, 2016, том 28, выпуск 2, с. 193-204. /Proc. ISP RAS, 2016, vol. 28, no 2, pp. 193-204.

implementation of the L4 Version 4 kernel API, which provides support of both 32- for processing. Then the server performs the action described in the packet and sends
bit and 64-bit architectures, multiprocessor and super-fast local IPC. The current its own packet to the client.
release supports x86-x64 (AMD64/ EM64T, K9 / P4 and higher), x86-x32 (IA32,
Pentium and higher), PowerPC 32bit (IBM 440, AMCC Ebony / Blue Gene P).
The debugger for Pistachio kernel can direct its I/O via the serial line or the
keyboard/screen. It is a local debugger and does not support remote debugging mode.
This debugger is also a low-level device with very limited amount of functions.
Debugger for Pistachio can:
 Set breakpoints
 Single step
 Dump memory
 Read registers
When the processor meets special instruction (for example, int3 instruction), it passes
control to interrupt handler, which is the part of Pistachio kernel. In turn, interrupt
handler checks instructions, which come next, and if they correspond to the special
layout, it prints special message before passing control to interrupt handler. This
feature is a simplistic implementation of a facility to trace execution.

4. Technical Description:
The primary goal of the debugger is Power PC platform, based on e500mc CPU core.
The debugger is based on GDB, it uses the GDB architecture to establish link to the
remote target.
The architecture includes three major components: front end, local client and remote
server. The front end provides user interface, it runs on the same workstation as the
client part. The latter translates the commands from the front end into GDB protocol
and communicates with the remote server. The server implements the actual Fig. 1. GDB messaging mechanism
command embedded into protocol messages such as reading memory regions, setting
breakpoints, processing debug interrupts, etc. Remote server is sometimes called Let us consider an example on Fig. 1. Here client sends to server packet
“stub”. “$m8000acac,4#1d”. This means that client wants to read 4 bytes of memory from
Gdb-stub for i386 was taken as a basis for our debugger. This stub was totally virtual address 0x8000acac. In this packet “1d” is the control sum, that is, the sum of
redesigned for e500mc processor, which belongs to PowerPC architecture family. We all bytes in message modulo 256. If the server fully receives this message, it sends
left only the packet exchange and some of the packet processing mechanisms. “+”, and the client knows that the message was accepted. After that, server sends 4
We use common gdb client, which was built for PowerPC with somewhat extended bytes of memory from that address to the client in the same way, and message
functional, to connect to our stub. This functional was developed using special user exchange continues. All these types of packets are described in GDB manual.
defines commands, so developers don't need to use special version of GDB. Instead,
they can use any version, but it needs to use gdb commands file by utilizing special 4.1 Implementation of the server side
“source” command in GDB. In general, debugger's work consists of packet exchange between client and server.
Accordingly, messaging mechanism between client and server doesn't change – the Client sends certain types of packets to perform the action, which the user needs. Our
client sends a special-type packet to the server and waits for the server's answer. The goal is to develop server part because we use client part from common GDB.
server receives this message, checks control sum, which was sent in this packet, and
if it matches the message contents, informs the client that the message was accepted
197 198
Емеленко А.Н., Маллачиев К.А., Пакулин Н.В. Разработка отладчика для операционной системы реального Emelenko A.N., Mallachiev K.A., Pakulin N.V. Developing a Debugger for Real-Time Operating System. Trudy ISP RAN
времени. Труды ИСП РАН, 2016, том 28, выпуск 2, с. 193-204. /Proc. ISP RAS, 2016, vol. 28, no 2, pp. 193-204.

During the connection between server part of the debugger with client part our system
stands in frozen state where no interrupts are available and the clock is halted. This
opportunity allows us to work with partitions and debugger as if there is no debugger
in the system.
We implemented functions in our debugger in the following way:
Breakpoints setting was implemented using special PowerPC instruction 'trap'. When
the trap instruction occurs, server code in interrupt handler is called.
For Single step operation, we can use two different methods. The first one is when
the system stops on the next instruction of the current partition. The second one is to
stop the system stops on the next instruction wherever it is. The difference is how
system calls are handled; the first method skis all kernel code and traverses
application only. The second method allows entering kernel and stepping through
system call implementation. Furthermore, it is sensitive to interrupts: if an interrupt
occurs during the step, the debugger switched to the interrupt handler.
However, GDB structure requires interrupts to be disabled during single step. This
requirement imposes restrictions on partition's work, so we gave up the second
method. Because of the lack of debug registers in QEMU we need to disable interrupts
and set trap instruction on the next instruction. Fig. 2. Multiplexer work
Watchpoints were implemented using special capabilities of hardware, such as Debug
registers. Unfortunately, QEMU doesn't have such registers, so we need to use 5. Debugger's Capabilities
another way to set watchpoints in emulator. This method isn't implemented yet, but
Our debugger supports all standard debugging features. Among them are:
we are working in this direction.We also developed multiplexer to use one serial port
for both GDB and another application. Multiplexer allows message exchange for
debugger and for internal system service. The transformation of one serial port into 5.1 Setting Breakpoints on Kernel and Partitions.
two serial ports with the help of our multiplexer is not so difficult. Setting breakpoints is the key feature of any debugger. Considering that client knows
There are two parts of multiplexer, local and remote. Local part is a superstructure only virtual addresses, the server part of the debugger must correctly translate this
responsible for information input/output in the system. During the output it puts a address into physical address. Our debugger can do this, that's why users can debug
special symbol before every printable symbol, determining to which of the two virtual partitions with overlapping virtual address spaces and debugger stops only on the
serial ports the next symbol should be sent. Working with input symbols is very partition that the user wants.
similar: two symbols are read, with the first of them specifying the application to
which we want to send the second symbol. Remote part of multiplexer looks the same. 5.2 Single Step.
This solution is not the fastest, but it provides smooth debugger's work via one serial Stepping through code step by step is a convenient way of finding bugs. However,
port together with other applications. This connection between remote and local parts there can be a situation in real-time OS, when the next instruction in code is not the
of multiplexer is shown on Fig. 2. next executable instruction, for example, because of timer interrupt. That's why we
disable interrupts during the single step.

5.3 Showing Information about Processes and Threads,

Inspecting Memory, Instructions and Registers. Memory Reading
and Writing.
Memory view must correctly translate virtual addresses into physical as with
breakpoints. The capability to find out all information about threads in OS, their
states, registers and memory is very important too.
199 200
Емеленко А.Н., Маллачиев К.А., Пакулин Н.В. Разработка отладчика для операционной системы реального Emelenko A.N., Mallachiev K.A., Pakulin N.V. Developing a Debugger for Real-Time Operating System. Trudy ISP RAN
времени. Труды ИСП РАН, 2016, том 28, выпуск 2, с. 193-204. /Proc. ISP RAS, 2016, vol. 28, no 2, pp. 193-204.

Support of memory writes allows changing process state as user's discretion. (http://www2.lauterbach.com/pdf/rtos_vxworks.pdf)
[3]. Lauterbach GmbH, “RTOS-VxWorks”, 18 August 2014
(http://www2.lauterbach.com/doc/rtosvxworks.pdf)
5.4 Setting Watchpoints. [4]. System Architecture Group University of Karlsruhe. “The L4Ka:: Pistachio Microkernel”.
Watchpoints are one of the most comfortable ways to control user's partition. They May 1, 2003
give the opportunity to follow changes in memory sectors and stop\pause while trying (http://www.l4ka.org/l4ka/pistachio-whitepaper.pdf)
to read or record memory. This opportunity increases the number of ways to control [5]. Wind River Systems, Inc “VxWorks Product Overview”, March 2016
partitions' states. (http://windriver.com/products/product-overviews/2691-VxWorks-Product-
Overview.pdf)
[6]. Free Software Foundation, Inc. “Debugging with gdb: the GNU Source-Level Debugger”,
5.5 Stack Inspection. The Tenth Edition
Stack inspection makes tracing possible: for example, tracing the queue of called (https://software.intel.com/sites/default/files/article/365160/gdb.pdf)
functions, which can help user to understand exactly what has happened in the system.

6. Future Work Разработка отладчика для операционной системы

Implementation of the debugger is not complete yet. There is a number of features реального времени
that can improve debugger usability:
1, 2
 Enhance debugging capabilities to the level of standard GDB functionality. А.Н. Емеленко <[email protected]>
1, 3
 Accelerate debugger interaction time with the system through multiplexer. К.А. Маллачиев <[email protected]>
1, 3
Н.В. Пакулин <[email protected]>
 Improve hardware support on bare metal. 1
Институт системного программирования РАН,
 Increase user convenience in multiplexer. Enhance its functionality for 109004, Россия, г. Москва, ул. А. Солженицына, д. 25.
working with more devices (now multiplexer supports only two devices). 2
Московский физико-технический институт (государственный
This solution allows us to work on bare metal with as many ports as we need, университет),
regardless of the actual amount of ports. 141701, Московская область, г. Долгопрудный, Институтский переулок, д.9.
 Add watchpoints implementation to QEMU, which doesn't support debug 3
Московский государственный университет имени М.В. Ломоносова,
registers. This is the reason why we can't use debug registers for setting 119991, Россия, Москва, Ленинские горы, д. 1.
watchpoints like we do on bare metal. In that case, we need to change code
handling in QEMU to develop instruction for watchpoints creation. Аннотация. В этой статье мы расскажем о проекте по разработке отладчика для
операционной системы реального времени JetOS, созданной для гражданских
7. Conclusion авиационных систем. Она предназначена для работы в рамках архитектуры
Интегрированной Модульной Авионики (ИМА) и реализует ARINC 653 спецификацию
In this paper, we have presented our project on implementation of the debugger for API. Эта операционная система разрабатывается в институте системного
real-time operating system JetOS. In contrast to other systems and their debuggers, программирования РАН, и следующим шагом в ее разработке стало создание
where developers can use some functions to debug applications, but not all we need, инструмента для отладки пользовательских приложений. Также в этой статье будут
our debugger meets the majority of our requirements and restrictions. However, we рассмотрены основные требования к такому отладчику и показана разница между ним и
will able to update our debugger in near future and increase its functionality, but it is обычным отладчиком, используемым разработчиками настольных приложений. Более
already more functional than common GDB debugger for QEMU. того, были рассмотрены другие встраиваемые операционные системы, такие как
WxWorks, Fiasco OS, L4Ka::Pistachio и отладчики для них, а также был изучен их
функционал. В заключение, мы представим наш отладчик, который может работать как
References в эмуляторе QEMU, используемом для эмуляции окружения для JetOS, так и на целевой
машине. Представленный отладчик является удаленным и построен с использованием
[1]. F. Mehnert, J. Glauber and J. Liedtke, “Fiasco Kernel Debugger Manual” Dresden структуры GDB, но содержит ряд расширений, специфичных для отладки встроенных
University of Technology, Department of Computer Science, November 2008 приложений. Однако реализация отладчика пока не завершена и существует целый ряд
(https://os.inf.tu-dresden.de/fiasco/doc/jdb.pdf) задач по улучшению удобства и возможностей отладчика, но на текущий момент он
[2]. Lauterbach GmbH, “RTOS debugger for VxWorks”, November 2015 является уже более функциональным, чем обычный отладчик GDB для QEMU и, в

201 202
Емеленко А.Н., Маллачиев К.А., Пакулин Н.В. Разработка отладчика для операционной системы реального
времени. Труды ИСП РАН, 2016, том 28, выпуск 2, с. 193-204.

отличие от других рассмотренных систем и их отладчиков, где разработчики могут

использовать некоторые функции для отладки приложений, но не все, что нам нужны,
наш отладчик удовлетворяет большинству поставленных требований и ограничений, а
также уже используется разработчиками приложений для JetOS.
Ключевые слова: отладчик; GDB; ОСРВ; удаленный отладчик; операционная система
реального времени
DOI: 10.15514/ISPRAS-2016-28(2)-13
Для цитирования: А.Н. Емеленко, К.А. Маллачиев, Н.В. Пакулин. Разработка
отладчика для операционной системы реального времени. Труды ИСП РАН, том 28, вып.
2, 2016 г., стр. 193-204 (на английском). DOI: 10.15514/ISPRAS-2016-28(2)-13

Список литературы
[7]. F. Mehnert, J. Glauber and J. Liedtke, “Fiasco Kernel Debugger Manual” Dresden
University of Technology, Department of Computer Science, November 2008
(https://os.inf.tu-dresden.de/fiasco/doc/jdb.pdf)
[8]. Lauterbach GmbH, “RTOS debugger for VxWorks”, November 2015
(http://www2.lauterbach.com/pdf/rtos_vxworks.pdf)
[9]. Lauterbach GmbH, “RTOS-VxWorks”, 18 August 2014
(http://www2.lauterbach.com/doc/rtosvxworks.pdf)
[10]. System Architecture Group University of Karlsruhe. “The L4Ka:: Pistachio Microkernel”.
May 1, 2003
(http://www.l4ka.org/l4ka/pistachio-whitepaper.pdf)
[11]. Wind River Systems, Inc “VxWorks Product Overview”, March 2016
(http://windriver.com/products/product-overviews/2691-VxWorks-Product-
Overview.pdf)
[12]. Free Software Foundation, Inc. “Debugging with gdb: the GNU Source-Level Debugger”,
The Tenth Edition
(https://software.intel.com/sites/default/files/article/365160/gdb.pdf)

203