L1/L2 SOFTWARE TRAINING

R-CAR LINUX BSP

MAY 28, 2021

HIEN DANG
ABU/RVC/RVSW/RVSSW
RENESAS ELECTRONICS CORPORATION

© 2021 Renesas Electronics Corporation. All rights reserved. RENESAS CONFIDENTIAL

AGENDA
▪ Introduction Page 03

▪ Yocto Package and BSP Composition Page 05

▪ Issue Analysis - Basic Page 16

▪ Debugging Page 23

▪ Performance Tuning and Analysis Page 28

© 2021 Renesas Electronics Corporation. All rights reserved. RENESAS CONFIDENTIAL Page 2
 INTRODUCTION
TRAINING TARGET AND SCOPE

© 2021 Renesas Electronics Corporation. All rights reserved. RENESAS CONFIDENTIAL Page 3
INTRODUCTION

❑ L1/L2 SW Training is necessary to reduce Q&A to L3

Engineer. The training will equip L1/L2 with R-Car
BSP knowledge so that they can solve some Q&A
without the help of L3.
❑ Training scope:
➢ Yocto package and BSP composition
➢ Debugging tools
➢ Performance analysis

© 2021 Renesas Electronics Corporation. All rights reserved. RENESAS CONFIDENTIAL Page 4
PREFACE
❑ This material is intended for L1/L2 Global FAE (Field Application Engineer) and customer support engineers.
❑ It is assumed that Reader/Trainee has some fundamental knowledge in below topics in advance.
o Embedded development environment e.g., cross compiler/toolchain, git, vim, Linux kernel source tree etc.
o Basic concept about kernel configuration, driver module and sysfs (entry to retrieve system information or
control system/module behavior)
o Be able to deploy system images (kernel image, DTB, IPL, rootfs etc.) on board.
o Linux command line interaction.
o Experience in IT/ST will be helpful to understand use case mentioned in this material.

© 2021 Renesas Electronics Corporation. All rights reserved. RENESAS CONFIDENTIAL Page 5
 YOCTO PACKAGE AND BSP COMPOSITION
SOFTWARE COMPONENTS AND VERSION CONTROL

© 2021 Renesas Electronics Corporation. All rights reserved. RENESAS CONFIDENTIAL Page 6
YOCTO PACKAGE

Renesas standard Yocto package

contains but not limited to the following
components:
❑ Userland
❑ Middleware and Libraries
❑ Secure package
❑ Multimedia package (MMP)
❑ Board support package (BSP)

© 2021 Renesas Electronics Corporation. All rights reserved. RENESAS CONFIDENTIAL Page 7
YOCTO PACKAGE VERSION CONTROL
Each of the components in Yocto package has a version control itself or defined/produced by a tool that generates/builds it.
Please refer to Yocto Release Note (e.g. RENESAS_RCH3M3M3NE3D3_YoctoReleaseNote_E_v5.[Link]) for details.

Yocto package v5.1.0

❑ “Yocto package” can
be considered as Userland
Renesas distribution
for Linux.
Wayland v1.18.0 GStreamer v1.16.3
❑ “yocto project” is a
build-tool used for
packaging all S/W yocto project v3.1.4 OpenGLES v3.2
components together. gcc v9.3

Linux BSP v5.0.0

U-Boot 2020.10

© 2021 Renesas Electronics Corporation. All rights reserved. RENESAS CONFIDENTIAL Page 8
LINUX BSP

❑ Linux BSP is usually comprised of Linux

kernel and some local patches or in-house
drivers on top of it.
❑ In a more general concept, BSP can also
include U-boot, IPL or even simple
userland that helps to boot up one platform
completely.
Local patches
❑ BSP version and change points are clearly
In-house drivers
described in BSP release note (e.g.
RENESAS_RCH3M3M3NE3D3V3UV3H_B + Linux BSP v5.0.0
SP_ReleaseNote_E_v5.[Link])
Linux kernel v5.10

U-Boot 2020.10

© 2021 Renesas Electronics Corporation. All rights reserved. RENESAS CONFIDENTIAL Page 9
 ❑ Boot ROM: Low level initialization code.
SYSTEM BOOT FLOW ❑ IPL (Initial Program Loader): Early system initialization and secure
boot.
❑ U-Boot: loads default kernel image, DTB, and invokes kernel image.
❑ Linux startup: establishes memory management, detect CPU type,
execute initialization functions, load built-in device drivers.
❑ INIT process: ‘Init’ is the first process responsible for starting
necessary system processes to configure the environment for
Boot ROM users and load necessary drivers/modules/user services.

IPL

U-boot
Linux BSP

Kernel

INIT process

© 2021 Renesas Electronics Corporation. All rights reserved. RENESAS CONFIDENTIAL Page 10
SYSTEM BOOT FLOW – BOOT LOG (1/2)

(1) BL2

(3) Kernel

(2) U-boot

© 2021 Renesas Electronics Corporation. All rights reserved. RENESAS CONFIDENTIAL Page 11
SYSTEM BOOT FLOW – BOOT LOG (2/2)

(4) Mount NFS and

start INIT process

(6)
Finish
(5) Load user
module

© 2021 Renesas Electronics Corporation. All rights reserved. RENESAS CONFIDENTIAL Page 12
DRIVER ARCHITECTURE
❑ In Linux BSP, there are four concepts that have close relationship with one driver.
▪ Kernel configuration: enables driver sub-system and features.
▪ Driver Sub-system or Framework: supports specific types of drivers e.g. serial,
network, USB …
▪ Device Tree Source (DTS): defines hardware module structure and properties.
▪ Device driver source: controls driver operation.

Serial driver sub-system

drivers/tty/serial/
DTS Device driver source
arch/arm64/boot/dts/renesas drivers/tty/serial/sh-sci.c

Entry
Exit

Power control

© 2021 Renesas Electronics Corporation. All rights reserved. RENESAS CONFIDENTIAL Page 13
DRIVER USER MANUAL (UM)
❑ Driver UM provides basic information about each driver including:
▪ Supported feature/function: in correspondence to hardware
UM. If one feature is not mentioned, it might not be supported.
▪ Module configuration: Driver block diagram. It usually shows
relation between driver and related S/W, H/W components.
▪ Integration: shows driver source file name and location.
▪ Others: maybe external connection, enable/disable channel or
changing driver setting etc.

© 2021 Renesas Electronics Corporation. All rights reserved. RENESAS CONFIDENTIAL Page 14
BSP RELEASE NOTE (RN)
❑ BSP RN contains essential information about the BSP (ONLY) including
▪ Change History: This is good reference to compare difference between
versions.
▪ Restriction: Detected issues of current release which are not fixed. All
the tickets here cannot be accessed by L1/L2.
▪ Notice and Hardware restriction: usage note and restriction related to
hardware.

© 2021 Renesas Electronics Corporation. All rights reserved. RENESAS CONFIDENTIAL Page 15
APPLICATION NOTE (AN)

❑ AN usually describes overall function of a sub-system

from user viewpoint. Not all driver has AN available.
Content includes the following:
▪ Overview, specification of supported features.
▪ Operation and use case description.

© 2021 Renesas Electronics Corporation. All rights reserved. RENESAS CONFIDENTIAL Page 16
 ISSUE ANALYSIS
BASIC APPROACH

© 2021 Renesas Electronics Corporation. All rights reserved. RENESAS CONFIDENTIAL Page 17
ISSUE ANALYSIS

❑ There is no standard workflow for issue analysis.

❑ Practically, steps can be mixed or adjusted depending on situation.

Compare with Debug

Reproduce Analyze error Support from
Identify error stable
L3
issue log environment (Best effort)

© 2021 Renesas Electronics Corporation. All rights reserved. RENESAS CONFIDENTIAL Page 18
 Normal Yocto build
(Please refer to Yocto Startup Guide)
REPRODUCE ISSUE
❑ Please try to make clear and replicate exactly Build only Linux BSP by Yocto
the condition/environment in which the error
occurs. $ bitbake –c cleansstate linux-renesas
$ bitbake linux-renesas
❑ Steps to reproduce Linux BSP issue:
▪ Identify BSP version or Yocto package
version that the issue is reported.
▪ Build kernel image and DTB either by
Yocto or standalone build.
▪ Boot up and check.

Wait the build to complete and get

output (kenel image, DTB, IPL, U-boot,
rootfs)

© 2021 Renesas Electronics Corporation. All rights reserved. RENESAS CONFIDENTIAL Page 19
BSP STANDALONE BUILD Easier and faster to build, deploy and customize kernel
image for debugging. This is specifically useful when
It is assumed that working directory is ‘/home/bsp_lab’ issue comes from LinuxBSP.

❑ Install toolchains to ‘/home/bsp_lab/poky/3.1.4’

$ sh poky-glibc-x86_64-[Link] /home/bsp_lab/poky/3.1.4

❑ Use git to get entire kernel source tree repository

$ git clone [Link]
❑ Change working directory to ‘[Link]’ and checkout branch
$ git checkout –b work rcar-5.0.0

❑ Setup build environment

$ export ARCH=arm64
$ export CROSS_COMPILE= /home/bsp_lab/poky/3.1.4/sysroots/x86_64-pokysdk-linux/usr/bin/aarch64-
poky-linux/aarch64-poky-linux-

❑ Configure Linux kernel

$ make defconfig

❑ Compile the kernel image and device tree blob

$ make –j8

© 2021 Renesas Electronics Corporation. All rights reserved. RENESAS CONFIDENTIAL Page 20
 [ 239.725204] bt_rfkill_set: blocked 1, Turning BT OFF
[ 239.728285] binder: undelivered TRANSACTION_ERROR: 29189
[ 239.739646] sh-sci [Link]: Failed preparing Tx DMA descriptor
[ 239.746365] Unable to handle kernel read from unreadable memory at virtual address 00000000

ANALYZE ERROR LOG [ 239.754767] Mem abort info:

[ 239.757578] Exception class = DABT (current EL), IL = 32 bits
[ 239.763637] SET = 0, FnV = 0
[ 239.766771] EA = 0, S1PTW = 0
[ 239.769963] Data abort info:
[ 239.772855] ISV = 0, ISS = 0x00000004
Driver error message. Usually in a form of [ 239.776943] CM = 0, WnR = 0
[ 239.779986] user pgtable: 4k pages, 48-bit VAs, pgd = 000000000cb58d73
<driver_name><device_address>.<device_type>: <message> [ 239.786537] [0000000000000000] *pgd=0000000000000000
[ 239.791547] Internal error: Oops: 96000004 [#1] PREEMPT SMP
[ 239.797119] Modules linked in: synaptics_tcm_device synaptics_tcm_core_module cdc_ncm r8152 rtl8150
usbnet snd_usb_audio snd_hwdep snd_usbmidi_lib crc_ccitt btusb btrtl btintel btbcm usb_storage usbhid
System error message. It tells what the error is. xhci_plat_d
[ 239.831379] Process kworker/6:0 (pid: 41, stack limit = 0x00000000b622275a)
[ 239.838336] CPU: 6 PID: 41 Comm: kworker/6:0 Tainted: G W O 4.14.133-gdf61d91d4cd0 #1
[ 239.847096] Hardware name: Lucid Motors CCC (DT)
[ 239.851724] Workqueue: events work_fn_tx
Last location/function executed before the error occurs [ 239.855651] task: 0000000069abb350 [Link]: 00000000b622275a
[ 239.861567] PC is at sci_tx_dma_release+0x44/0xe0
[ 239.866264] LR is at sci_tx_dma_release+0x44/0xe0
[ 239.870954] pc : [<ffff000008566994>] lr : [<ffff000008566994>] pstate: 40400145

<snipped>

Call trace: This will give more information about calling flow of the [ 240.173595] Call trace:
error function. …
[ 240.260660] [<000000008c7361e7>] sci_tx_dma_release+0x44/0xe0
[ 240.266415] [<000000006f95e9d6>] work_fn_tx+0x18c/0x1e8
[ 240.271642] [<00000000acac628b>] process_one_work+0x1d8/0x498 Should be read in
[ 240.277410] [<0000000053efdff2>] worker_thread+0x4c/0x478 reversed order
Error log contains useful info for analyzing issue. [ 240.282808] [<00000000193664b7>] kthread+0x138/0x140
[ 240.287769] [<000000000fc7dde0>] ret_from_fork+0x10/0x1c
Please try to: [ 240.293078] Code: b9023261 aa0003e1 aa1303e0 9416b1ea (f9400280)
[ 240.299165] ---[ end trace 45bc99e0c413589e ]---
[ 240.320506] Kernel panic - not syncing: Fatal exception
❑ Locate error function of driver. [ 240.325735] SMP: stopping secondary CPUs
[ 240.329660] Kernel Offset: disabled
[ 240.333143] CPU features: 0x1802004
❑ Identify error type. Memory/Interrupt/IO…? [ 240.336617] Memory Limit: none
[ 240.356055] Rebooting in 5 seconds..

© 2021 Renesas Electronics Corporation. All rights reserved. RENESAS CONFIDENTIAL Page 21
 ERROR AND DRIVER MAPPING
❑ There are several ways to map the error to driver source code

Find the error function or text pattern using egrep or grep Refer to driver UM to locate source file of the driver and necessary
egrep –r –e <searching_pattern> <folder_to_be_searched> kernel configurations for it.

Find error function or any other C language definition using cscope utility
▪ Installation and usage is here Cscope Home Page ([Link])
▪ When combining with Vim editor, tracing/searching/browsing C source code is
very easy and straight forward.
▪ Example: vim –t <func_name/C definition>
This command will open source file which contains the function and jump right there.
▪ Searching can be done in text interface also.

© 2021 Renesas Electronics Corporation. All rights reserved. RENESAS CONFIDENTIAL Page 22
COMPARE WITH STABLE ENVIRONMENT
❑ After successfully reproduce issue, try to find an environment (Yocto package version, BSP version) that the issue
does not occur.
❑ Use git to compare commit history between two BSP versions and locate the bad commit that causes the issue or
the good one that can fix. It is usually related to the function or driver source file.
Example: git log --oneline <branch_A>..<branch_B> <source file>
This will list all commit of a specific source file that is contained in branch B but not in branch A.

Find fix patch

BSP ver.(N-1) BSP ver.(N) BSP ver.(N+1)

Good Bad Good
Locate bad commit Upstream

Any solution?
BSP ver.(N-1) BSP ver.(N) BSP ver.(N+1)
Good Good Bad Latest

Locate bad commit

© 2021 Renesas Electronics Corporation. All rights reserved. RENESAS CONFIDENTIAL Page 23
 DEBUGGING
INTRODUCE SOME USEFUL TIPS AND TOOLS

© 2021 Renesas Electronics Corporation. All rights reserved. RENESAS CONFIDENTIAL Page 24
KERNEL BUILT-IN DEBUGGING OPTIONS
❑ Kernel parameters
▪ Utilize debugging log as much as possible by telling the kernel to print more log. This can be done by adding
‘ignore-loglevel’ to U-boot bootargs.
▪ Another useful parameter is ‘initcall_debug’. Trace initcalls as they are executed, very useful for working out
where the kernel is dying during startup.
▪ There are also many other interesting kernel parameters listed in this text file
<kernel_path>/Documentation/admin-guide/[Link]. The file comes with every kernel source tree.
(location may differ)
❑ Kernel configurations
▪ Beside kernel level debugging configs (spinlock/mutex, debug sysfs, memory …), each sub-system will have
their own configs which will provide even more log of a specific driver. For example,
✓ CONFIG_I2C_DEBUG_CORE
✓ CONFIG_I2C_DEBUG_BUS
✓ CONFIG_MMC_DEBUG
▪ Kernel OOPS, kernel to panic when it oopses. This feature is useful to ensure that the kernel does not do
anything erroneous after an oops which could result in data corruption or other issues.
▪ CONFIG_PANIC_ON_OOPS & CONFIG_PANIC_TIMEOUT

© 2021 Renesas Electronics Corporation. All rights reserved. RENESAS CONFIDENTIAL Page 25
KERNEL FUNCTION TRACER (FTRACE)
❑ Ftrace is an internal tracer designed to help ❑ Basic usage of Ftrace:
out developers and designers of systems to ▪ Default sysfs of Ftrace:
find what is going on inside the kernel. It can
cd /sys/kernel/debug/tracing/
be used for debugging or analyzing
latencies and performance issues that take ▪ Disable Ftrace: Output
place inside the kernel. echo 0 > tracing_on
❑ For more details, please visit ftrace - Function ▪ Clear previous trace history:
Tracer — The Linux Kernel documentation. echo 0 > trace
❑ To enable Ftrace, please enable below kernel
▪ Configure Ftrace:
configs and rebuild kernel image.
echo function > current_tracer
▪ CONFIG_FTRACE
echo <func_name> > set_ftrace_filter
▪ CONFIG_FUNCTION_TRACER
▪ Enable Ftrace:
▪ CONFIG_FUNCTION_GRAPH_TRACER echo 1 > tracing_on
▪ CONFIG_DYNAMIC_FTRACE ▪ Run process that needs to be traced.
▪ CONFIG_STACK_TRACER ▪ Read trace history
cat trace

© 2021 Renesas Electronics Corporation. All rights reserved. RENESAS CONFIDENTIAL Page 26
FTRACE – TRACING FUNCTION CALL
❑ What calls a specific function? ❑ What a function calls?
▪ Default sysfs of Ftrace: ▪ Default sysfs of Ftrace:
cd /sys/kernel/debug/tracing/ cd /sys/kernel/debug/tracing/
▪ Configure Ftrace: ▪ Configure Ftrace:
echo function > current_tracer echo function_graph > current_tracer
echo <func_name> > set_ftrace_filter echo <func_name> > set_graph_function
Echo 1 > options/func_stack_trace ▪ Enable Ftrace:
▪ Enable Ftrace: echo 1 > tracing_on
echo 1 > tracing_on ▪ Run process that needs to be traced.
▪ Run process that needs to be traced. ▪ Read trace history
▪ Read trace history cat trace
cat trace

© 2021 Renesas Electronics Corporation. All rights reserved. RENESAS CONFIDENTIAL Page 27
 FTRACE – TRACING FROM USER SPACE
Kernel-shark GUI

❑ Ftrace can be used with trace-cmd command

from Linux user space. The command will
interact with the Ftrace tracer in a more intuitive
way and provide similar output which is then can
be visualized by kernel-shark utility.

❑ Example: How long interrupts take in the kernel

$ trace-cmd record -p
function_graph -l do_IRQ -e
irq_handler_entry sleep 10
$ trace-cmd report

© 2021 Renesas Electronics Corporation. All rights reserved. RENESAS CONFIDENTIAL Page 28
 PERFORMANCE TUNING & ANALYSIS
OPTIMIZE BOOT TIME AND RESOLVE BOTTLENECK

© 2021 Renesas Electronics Corporation. All rights reserved. RENESAS CONFIDENTIAL Page 29
OPTIMIZE BOOT TIME
❑ Boot time is one the most concerned question from customer in demonstration or promotion activities.
❑ BSP is usually not optimized for this purpose because it is reference platform and requirement cannot be
satisfied all at once.
❑ Measuring boot time can be done in many ways e.g. stopwatch, GPIO toggle and oscilloscope, timestamp or
systemd-analyze utility … depending on use case and application.
❑ There are several ways to optimize boot time from BSP perspective.
▪ Limit system log output by adding “quiet” to U-boot bootargs. (ignore_loglevel should not be used.)
▪ Reduce number of loaded modules/services during kernel bootup.
▪ Optimize kernel image size to be loaded faster.

© 2021 Renesas Electronics Corporation. All rights reserved. RENESAS CONFIDENTIAL Page 30
 OPTIMIZE BOOT TIME – LOADABLE MODULES
❑ One driver can be built-in or
loadable mode.
▪ Built-in module (y): is a part of
kernel and loaded during kernel
bootup. It is usually essential
drivers for early system
initialization and function which is
deemed necessary. Example:
CPG, PFC, INTC, SCIF, DU
drivers.
▪ Loadable module (m): is placed
on user space (rootfs) and
loaded after kernel bootup
automatically or manually.
Example: GFX, ADSP, MMNGR,
UVCS drivers.

arch/arm64/configs/defconfig make menuconfig

© 2021 Renesas Electronics Corporation. All rights reserved. RENESAS CONFIDENTIAL Page 31
OPTIMIZE BOOT TIME – IMAGE SIZE
❑ For v5.10 kernel, image size is about 34 MB. The heavier the image,
the longer the boot time.
❑ Some ways to optimize kernel image size for faster loading.
▪ Reducing number of loaded modules also decreases kernel image
size.
▪ Disable redundant modules. For example: Intel network driver.
▪ Disable unnecessary kernel features. For example: debugging
features such as debugging file system, ftrace …

© 2021 Renesas Electronics Corporation. All rights reserved. RENESAS CONFIDENTIAL Page 32
DRIVER PERFORMANCE ISSUE

❑ Driver performance is measured individually.

❑ One of the most common way to measure performance of driver is using ‘time’ command.
time dd iflag=direct if=/dev/sda of=/dev/null bs=1M count=1024
❑ Usually, performance issue of driver comes from driver design and implementation.
❑ Ftrace can be used to analyze bottle neck.

© 2021 Renesas Electronics Corporation. All rights reserved. RENESAS CONFIDENTIAL Page 33
SUSPEND TO RAM

❑ “System Suspend to RAM” is a function in

which system operation is suspended while
its state at that time is retained to DRAM. On
release from suspension, the system is
restored to its prior state.
❑ This is one of the solution to reduce boot up
time because system does not need to
restart everything all over again.
❑ Improving resuming time of system is
important for demonstration purpose.
❑ How to measure S2RAM performance and
analyze it is described in next slide.

© 2021 Renesas Electronics Corporation. All rights reserved. RENESAS CONFIDENTIAL Page 34
SUSPEND TO RAM PERFORMANCE ANALYSIS
❑ Enable below configs and rebuild kernel image.
▪ CONFIG_PM_SLEEP_DEBUG
▪ CONFIG_FTRACE
▪ CONFIG_FUNCTION_TRACER
▪ CONFIG_FUNCTION_GRAPH_TRACER
❑ Add below param to bootargs ‘log_buf_len=16M’
❑ Download suspend/resume timing analysis tool from
[Link]
❑ Copy "analyze_suspend.py" to rootfs.
❑ Steps to run and collect S2RAM analysis on Salvator board.
▪ i2cset -f -y 7 0x30 0x20 0x0F
▪ Turn off SW23
▪ ./analyze_suspend.py -f -m mem
▪ Turn on SW23.
▪ The script will take several minutes to collect result. The result is
placed under folder named 'suspend-xxx-xxx’ in HTML format.

© 2021 Renesas Electronics Corporation. All rights reserved. RENESAS CONFIDENTIAL Page 35
REVISION HISTORY
Rev. Page Remark Date Author
0.1 - First issued version May 24, 2021 Hien Dang
0.2 5 Added required knowledge before reading this material Sep. 17, 2021 Hien Dang
36 Added revision history

© 2021 Renesas Electronics Corporation. All rights reserved. RENESAS CONFIDENTIAL Page 36
 [Link]

© 2021 Renesas Electronics Corporation. All rights reserved. RENESAS CONFIDENTIAL