Practical Firmware Reversing

and Exploit Development for

AVR-based Embedded Devices
Alexander @dark_k3y Bolshev
Boris @dukeBarman Ryutin

; cat /dev/user(s)
Alexander Bolshev (@dark_key), Security
Researcher, Ph.D., Assistant Professor @ SPbETU

Boris Ryutin (@dukeBarman), radare2 evangelist,

Security Engineer @ ZORSecurity

Agenda
Hour 1

Part 1: Quick RJMP to AVR + Introduction example

Hours 2-3:

Part 2: Pre-exploitation
Part 3: Exploitation and ROP-chains building
Part 4: Post-exploitation and tricks

Hour 4:

Mitigations
CFP! (Powered by Roman Bazhin)

If you have a
question, please
interrupt and ask
immediately

Disclaimer:
1) Workshop is VERY fast-paced.
2) Workshop is highly-practical
3) You may encounter information
overflow
Image Credit: Marac Kolodzinski

Part 1: What is AVR?

AVR
Alf (Egil Bogen) and Vegard (Wollan)s RISC processor
Modified Harvard architecture 8-bit RISC single-chip microcontroller
Developed by Atmel in 1996 (now Dialog/Atmel)

Image: https://de.wikipedia.org/wiki/Atmel AVR

AVR is almost everywhere

Industrial PLCs and gateways
Home electronics: kettles, irons, weather stations, etc
IoT
HID devices (ex.: Xbox hand controllers)
Automotive applications: security, safety, powertrain and
entertainment systems.
Radio applications (and also Xbee and Zwave)
Arduino platform
Your new shiny IoE fridge ;)

AVR inside industrial gateway

Synapse IoT module with Atmega128RFA1 inside

Philips Hue Bulb

http://www.eetimes.com/document.asp?doc_id=1323739&image_number=1

AVR inside home automation dimmer

Harvard Architecture

Harvard Architecture
Physically separated storage and signal pathways for instructions and
data
Originated from the Harvard Mark I relay-based computer

Image: https://en.wikipedia.org/wiki/Harvard architecture

Modified Harvard architecture

allows the contents of the instruction memory to be accessed as if it were data1

1but

not the data as code!

Introduction example:
Were still able to exploit!

AVR features

AVR-8
MCU (MicroController Unit) -- single computer chip designed for
embedded applications
Low-power
Integrated RAM and ROM (SRAM + EEPROM + Flash)
Some models could work with external SRAM
8-bit, word size is 16 bit (2 bytes)
Higher integration
Single core/Interrupts
Low-freq (<20MHz in most cases)

Higher Integration

Built-in SRAM, EEPROM an Flash

GPIO (discrete I/O pins)
UART(s)
I2C, SPI, CAN,
ADC
PWM or DAC
Timers
Watchdog
Clock generator and divider(s)
Comparator(s)
In-circuit programming and debugging support

AVRs are very different

AtTiny13
Up to 20 MIPS Througput at 20 MHz
64 SRAM/64 EEPROM/1k Flash
Timer, ADC, 2 PWMs, Comparator,
internal oscillator
0.24mA in active mode, 0.0001mA in
sleep mode

AVRs are very different

Atmega32U4
2.5k SRAM/1k EEPROM/32k Flash
JTAG
USB
PLL, Timers, PWMs, Comparators,
ADCs, UARTs, Temperatures sensors,
SPI, I2C, => tons of stuff

AVRs are very different

Atmega128
4k SRAM/4k EEPROM/128k Flash
JTAG
Tons of stuff:

In the rest of the workshop we will focus on this chip

Why Atmega128?
Old, but very widespread chip.
At90can128 popular analogue for CAN buses in automotive
application
Cheap JTAG programmer
Much SRAM == ideal for ROP-chain construction training

Lets look to the architecture of Atmega128

Ok, ok, lets simplify a bit

Image: http://www.cs.jhu.edu/~jorgev/cs333/usbkey/uC_3.JPG

Note: code is separated from data

Memory map

Memory: registers
R1-R25 GPR
X,Y,Z pair working
registers, e.g. for memory
addressing operations
I/O registers for accessing
different hardware

Memory: special registers

PC program counter, 16-bit register
SP stack pointer, 16-bit register (SPH:SPL)
SREG status register (8-bit)

Memory addressing
SRAM/EEPROM 16-bit addressing, 8-bit element
Flash 16(8)-bit addressing, 16-bit element

LPM
command!

Memory addressing directions

Direct to register
Direct to I/O
SRAM direct
SRAM indirect (pre- and post- increment)
Flash direct

Datasheets are your best friends!

Interrupts
Interrupts normal process of code
execution for handling something
or reacting to some event
Interrupt handler procedure to
be executed after interrupt;
address stored in the interrupt
vector
Examples of interrupts:
Timers
Hardware events
Reset

AVR assembly
In a very quick
manner

Instruction types
Arithmetic and logic
Bit manipulation/test
Memory manipulation
Unconditional jump/call
Branch commands
SREG manipulation
Special (watchdog, etc)

Instruction mnemonics
mov
out

r16,r0
PORTA, r16

; Copy r0 to r16
; Write r16 to PORTA

16-bit long
Intel syntax (destination before source)

A bit more about architecture

Fuses and Lock Bits

Several bytes of permanent storage
Set internal hardware and features
configuration, including oscillator
(int or ext), bootloader, pin, ability to
debug/programm, etc.
2 lock bits controls programming
protection.

AVR bootloader what is it?

Part of code that starts BEFORE RESET interrupt.
Could be used for self-programmable (i.e. without external device)
systems, in case you need to supply firmware update for your IoT
device.
Bootloader address and behavior configured via FUSEs.
BLB lock bits controls bootloader ability to update application and/or
bootloader parts of flash.

AVR bootloaders
Arduino bootloader
USB bootloaders (AVRUSBBoot)
Serial programmer bootloaders (STK500-compatible)
Cryptobootloaders

Tons of them!

Watchdog
Timer that could be used for interrupt or reset device.
Cleared with WDR instruction.

http://ardiri.com/blog/entries/20141028/watchdog.jpg

Development for AVR

Atmel studio

AVR-GCC
Main compiler/debugger kit for the platform
Used by Atmel studio
Use AVR libc -- http://www.nongnu.org/avr-libc/
Several optimization options, several memory models

Other tools
Arduino
CodeVision AVR
IAR Embedded workbench

Debugging AVR

JTAG
Joint Test Action Group (JTAG)
Special debugging interface added to a chip
Allows testing, debugging, firmware manipulation and boundary
scanning.
Requires external hardware

AVR JTAGIce3

JTAG for AVRs

AVR JTAG mkII

AVR JTAG mkI

Atmel ICE3
AVR Dragon

Avarice
Open-source interface between AVR JTAG and GDB
Also allow to flash/write eeprom, manipulate fuse and lock bits.
Could capture the exeuction flow to restore the firmware
Example usage:
avarice --program --file test.elf --part atmega128 --jtag /dev/ttyUSB0 :4444

AVR-GDB
Part of nongnu AVR gcc kit.
Roughly ported standard gdb to AVR platform
Doesnt understand Harvard architecture, i.e. to read flash you will
need to resolve it by reference of $pc:
(gdb) x/10b $pc + 100

Simulators
Atmel Studio simulator
Proteus simulator
Simavr
Simulavr

VM access:
Login: radare
Password: radare

Ex 1.1: Hello world!

Real hardware
cd /home/radare/workshop/ex1.1
avarice --mkI --jtag /dev/ttyUSB0 -p -e --file build-crumbuino128/ex1.1.hex -g :4242
avr-gdb

Communication: cutecom or screen /dev/ttyUSB1 9600

Simulator
cd /home/radare/workshop/ex1.1_simulator
simulavr -P atmega128 -F 16000000 f build-crumbuino128/ex1.1.elf
avr-gdb

Ex 1.2: Blink!
Real hardware
cd /home/radare/workshop/ex1.2
avarice --mkI --jtag /dev/ttyUSB0 -p -e --file build-crumbuino128/ex1.2.hex -g :4242
avr-gdb

AVR RE

Reverse engineering AVR binaries

Pure disassemblers:
avr-objdump gcc kit standard tool
Vavrdisasm -- https://github.com/vsergeev/vavrdisasm
ODAweb -- https://www.onlinedisassembler.com/odaweb/
Normal disassemblers:
IDA Pro
Radare

IDA PRO: AVR specifics

Incorrect AVR elf-handling
Incorrect LPM command behavior
Addressing issues
Sometimes strange output
...
However, usable, but with care

Radare2
Opensource reverse engineering framework (RE, debugger, forensics)
Crossplatform (Linux,Mac,Windows,QNX,Android,iOS, )
Scripting
A lot of Architectures / file-formats

Without habitual GUI

Radare2. Tools
radare2
rabin2
radiff2
rafind2
rasm2
r2pm

rarun2
rax2
r2agent
ragg2
rahash2
rasign2

Radare2. Using
Install from git
# git clone https://github.com/radare/radare2
# cd radare2
# sys/install.sh

Packages (yara, retdec / radeco decompilers, ):

# r2pm -i radare2

Console commands
# r2 -d /bin/ls debugging
# r2 a avr sample.bin architecture
# r2 b 16 sample.bin specify register size in bits
# r2 sample.bin i script include script

Radare2. Basic commands

aaa analyze
axt xrefs
s seek
p disassemble
~ - grep
! run shell commands
/ search
/R search ROP
/c search instruction
? help

Radare2. Disassembling
p?
pd/pD - dissamble
pi/pI print instructions
Examples:
> pd 35 @ function

Radare2. Options
~/.radarerc
e asm.describe=true
e scr.utf8=true
e asm.midflags=true
e asm.emu=true
eco solarized

Radare2. Interfaces
ASCII VV
Visual panels V! (vim like controls)
Web-server r2 -c=H file
Bokken

Training kit content

AVR JTAG mkI

Arduino (not included)

ESP8266 WiFi to serial

Atmega128 custom
devboard

Part 2: Pre-exploitation

You have a device. First steps?

Decide
what you
want

Determine
target
platform

Search for
I/O
point(s)

Search for
debug
point(s)

Acquire
the
firmware

Fuzz
and/or
static
analysis

Lets start with a REAL example

Lets use training kit board as an example.
Imagine that you know nothing about it
We will go through all steps, one by one

What we want?
Complexity

At first, decide what you want:

Abuse functionality
Read something from EEPROM/Flash/SRAM
Stay persistant

Determine target platform

Look at the board and search for all ICs
CP2102

ESP8266EX

Atmega128 16AU

Digikey/Octopart/Google

External connectors

Search for I/O(s)

USB

UART

External connectors

Antenna

Search for I/O(s): tools

Jtagulator

Bus pirate

Saleae logic analyzer

Arduino

Search for debug interface(s)

ISP

JTAG

Search for debug interface(s): tools

Or cheaper

Jtagulator

Arduino + JTAGEnum

JTAGEnum against
Atmega128 demoboard

Search for debug & I/O: real device

ICS bus

Connector

Button

2 JTAGs
ISPs

Ethernet
LEDs

Acquire the firmware

From vendor web-site
Sniffing the update process
From device

Acquire the firmware: sniff it!

Acquire the firmware: JTAG or ISP

Use JTAG or ISP programmer to connect to the board debug ports
Use:
Atmel Studio
AVRDude
Programmer-specific software to read flash
$ avrdude -p m128 -c jtagmkI P /dev/ttyUSB0 \
-U flash:r:/home/avr/flash.bin":r

Acquire the firmware: lock bits

AVR has lock bits that protects device from extracting flash

Removing this lockbits will erase entire device

If you have them set, youre not lucky, try to get firmware from other
sources
However, if you have lock bits set, but JTAG is enabled you could try partial
restoration of firmware with avarice capture (rare case)

Exercise 2.0: Acquire!

Real hardware
Read fuses and lock bits using avarice r
Acquire firmware using avrdude

Firmware reversing: formats

Raw binary format
ELF format for AVRs
Intel HEX format (often used by programmers)
Could be easily converted between with avr-objcopy, e.g.:
avr-objcopy -R .eeprom -O ihex test.elf test.hex"

Ex 2.1: Hello! RE

Real hardware & Simulator

cd /home/radare/workshop/ex2.1
avr-objcopy I ihex O binary ex2.1.hex ex2.1.bin
r2 a avr ex2.1.bin

Arithmetic instructions
add
add
and
clr
inc
neg

r1,r2
r28,r28
r2,r3
r18
r0
r0

;
;
;
;
;
;

r1
r28
r2
r18
r0
r0

=
=
=
=
=
=

r1 + r2
r28 + r28
r2 & r3
0
r0 + 1
-r0

Bit manipulation instructions

lsl
lsr
rol

r0
r1
r15

ror

r16

cbr
sbr
cbi

r18,1
r16, 3
$16, 1

; r0 << 2
; r1 >> 2
; cyclic shift r16 bits to the
left
; cyclic shift r16 bits to the
right
; clear bit 1 in r18
; set bits 0 and 1 in r16
; PORTB[1] = 0

Memory manipulation
mov
ldi
lds
sts
st
st
std
in
out

r1, r2
r0, 10
r2,$FA00
$FA00,r0
Z, r0
Z, r1
Z+5, r2
r15, $16
$16, r0

;
;
;
;
;
;
;
;
;

r1 = r2
r0 = 10
r2 = *0xFA00
*0xFA00 = r0
*Z(r31:r30) = r0
*Z-- = r0
*(Z+5) = r2
r15 = PORTB
PORTB = r0

Same
for LD*

Memory manipulation: stack

push

r14

; save r14 on the Stack

SP = SP - 1

pop

r15

; pop top of Stack to r15

SP = SP + 1

Memory manipulation: flash

lpm r16, Z

; r16 = *(r31:r30), but from flash

Note: code is separated from data

Unconditional jump/call
jmp
rjmp

$ABC1
5

; PC = 0xABC1
; PC = PC + 5 + 1

call

$ABC1

; push PC+2
; jmp $ABC

ret

; pop PC

Harvard architecture? But PC goes to DATA

memory

SREG 8-bit status register

C
Z
N
V
S
H
T
I

Carry flag
Zero flag
Negative flag
twos complement oVerflow indicator
N V, for Signed tests
Half carry flag
Transfer bit (BLD/BST)
global Interrupt enable/disable flag

Conditional jump
cpse

r1, r0

; r1 == r2 ?
PC PC + 2 : PC PC + 3

breq
brne

10
11

; Z ? PC PC + 1 + 10
; !Z ? PC PC + 1 + 10

SREG manipulations
sec/clc set/clear carry
sei/cli set/clear global interruption flag
se*/cl* set/clear * flag in SRGE

Special
break debugger break
nop no operation
sleep enter sleep mode
wdr watchdog reset

Ex 2.2: Blink! RE
Real hardware & Simulator
cd /home/radare/workshop/ex2.1

avr-objcopy I ihex O binary blink.hex blink.bin

r2 a avr ex2.1.bin

Questions:
1. Identify main() function and describe it using af
2. Find the LED switching command
3. What type of delay is used and why accurate frequency is required?
4. Locate interrupt vector and init code, explain what happens inside init code.

Reversing: function szignatures

Most of firmwares contains zero or little strings.
How to start?
Use function signatures.
However, in AVR world signatures may be to vary.
Be prepared to predict target compiler/library/RTOS and options or
bruteforce it.
In R2, signatures are called zignatures.

Embedded code priorities

Size
Speed
Hardware limits
Redundancy

Security

Fuzzing specifics
Fuzzing is Fuzzing. Everywhere.
But were in embedded world.
Sometimes you could detect crash through test/debug UART or pins
In most cases, you could detect crash only by noticing, that device is
no longer response
Moreover, watchdog timer will could limit your detection capabilities,
because it will reset device.
So how to detect crash?

Fuzzing: ways to detect crash

JTAG debugger break on RESET
External analysis of functionality detect execution pauses
Detect bootloader/initialization code (e.g. for SRAM) behavior with
logic analyzer and/or FPGA
Detect power consumption change with oscilloscope/DAQ

Sometimes Arduino is enough to detect

I2C and SPI init sequencies could be captured by Arduino GPIOs
If bootloader is slow and waits ~1 second, this power consumption
reduction could be reliably detected with cheap current sensor, e.g.:

+
SparkFun Low Current Sensor Breakout - ACS712
https://www.sparkfun.com/products/8883

Lets proof it.

Part 3: Exploitation

Quick intro to ROP-chains

Return Oriented Programming
Series of function returns
Were searching for primitives (gadgets) ending with ret that could
be transformed into useful chain
SP is our new PC

Notice: Arduino
The next examples/exercises will be based upon Arduio libc (in fact,
Non-GNU AVR libc + Arduino wiring libs)
Were using Arduino because its complex, full of gadgets but free
(against IAR or CV which are also complex and full of gadgets)
Also, Arduino is fairly popular today, due to enormous number of
libraries and quick start (e.g. quick bugs)

Ex 3.1 3.3
Real hardware
cd /home/radare/workshop/ex3.1
avarice --mkI --jtag /dev/ttyUSB0 -p -e --file build-crumbuino128/ex3.1.hex -g :4242
avr-gdb

Simulator
cd /home/radare/workshop/ex3.1_simulator
simulavr -P atmega128 -F 16000000 f build-crumbuino128/ex3.1.elf
avr-gdb
Or: node exploit.js

Example 3.1: Abusing

functionality: ret to function

Internal-SRAM only memory map

Overflowing the heap => Rewriting the stack!

http://www.atmel.com/webdoc/AVRLibcReferenceManual/malloc_1malloc_intro.html

How to connect data(string/binary) to code?

Standard model: with .data
variables
Determine data offset in flash
Find init code/firmware prologue where
.data is copied to SRAM
Using debugging or brain calculate offset of
data in SRAM
Search code for this address

Economy model: direct read with

lpm/elpm
Determine data offset in flash
Search code with *lpm addressing to this offset

ABI, Types and frame layouts (GCC)

Types: standard (short == int == 2, long == 4, except for double (4))
Int could be 8bit if -mint8 option is enforced.
Call-used: R18R27, R30, R31
Call-saved: R2R17, R28, R29
R29:R28 used as frame pointer
Frame layout after function prologue:
incoming arguments
return address
saved registers
stack slots, Y+1 points at the bottom

Calling convention: arguments

An argument is passed either completely in registers or completely in memory.
To find the register where a function argument is passed, initialize the register
number Rn with R26 and follow this procedure:
1. If the argument size is an odd number of bytes, round up the size to the next even number.
2. Subtract the rounded size from the register number Rn.
3. If the new Rn is at least R18 and the size of the object is non-zero, then the low-byte of the argument is
passed in Rn. Other bytes will be passed in Rn+1, Rn+2, etc.
4. If the new register number Rn is smaller than R18 or the size of the argument is zero, the argument will
be passed in memory.
5. If the current argument is passed in memory, stop the procedure: All subsequent arguments will also
be passed in memory.
6. If there are arguments left, goto 1. and proceed with the next argument.

Varagrs are passed on the stack.

Calling conventions: returns

Return values with a size of 1 byte up to and including a size of 8
bytes will be returned in registers.
For example, an 8-bit value is returned in R24 and an 32-bit value is
returned R22...R25.
Return values whose size is outside that range will be returned in
memory.

Example
For
int func (char a, long b);
a will be passed in R24.
b will be passed in R20, R21, R22 and R23 with the LSB in R20 and the
MSB in R23.
the result is returned in R24 (LSB) and R25 (MSB).

Example 3.2: Abusing

functionality: simple ROP

ROP gadget sources

User functions
Standard or RTOS functions
Data segment
Bootloader section

More code => more gadgets

ROP chain size

Its MCU
SRAM is small
SRAM is divided between register file, heap and stack
Stack size is small
Were low on chain size
Obviously, you will be limited with 20-40 bytes (~15-30 gadgets)
However it all depends on compiler and memory model

http://www.atmel.com/webdoc/AVRLibcReferenceManual/malloc_1malloc_tunables.html

Memory maps external SRAM/separated stack

Memory maps external SRAM/mixed stack

Detecting standard functions

In AVR we have bunch of compilers, libraries and even RToSes
So, standard function could be vary.
More bad news: memory model and optimization options could
change function.
The best approach is try to detect functions like malloc/str(n)cpy and
then find the exact compiler/options that generates such code
After it, use function signatures to restore the rest of the code
In Radare2, you could use zignatures or Yara.

Example 3.3: more complex

ROP

Exercise 3.1: ret 2 function

build exploit that starts with ABC but calls switchgreen() function

Exercise 3.3: print something

else
3.3.1) build exploit that prints a few seconds
3.3.2 (homework) build exploit that prints blink a few seconds

Ex 3.4
Real hardware
cd /home/radare/workshop/ex3.1
in Blink.ino change APNAME constant from esp_123 to esp_your3digitnumber
make
avr-objdump I ihex O binary build-crumbuino128/ex3.4.hex ex3.4.bin
avarice --mkI --jtag /dev/ttyUSB0 -p -e --file build-crumbuino128/ex3.4.hex -g :4242
avr-gdb
Connect to esp_your3digitnumber and type http://192.168.4.1 in your browser

Simulator
cd /home/radare/workshop/ex3.4_simulator
On 1st terminal: node exploit.js
On 2nd terminal: tail f serial1.txt

In your browser: http://127.0.0.1:5000

Example 3.4: Blinking

through HTTP GET

Exercise 3.4: UARTing

through HTTP query

Exercise 3.5: Blinking

through HTTP Post

Its possible to construct ROP with debugger

But if I dont have some, how I could
determine the overflow point?

Reverse and use external analysis to find function that

overflows
Bruteforce it!

Arduino blink (ROP without

debugger)

Part 4: Post-exploitation
&& Tricks

What do we want? (again)

Evade watchdog
Work with persistent memory (EEPROM and Flash)
Stay persistent in device
Control device for a long time

Evade the watchdog

In most cases, there three ways:
1. Find a ROP with WDR and periodically jump to it.
2. Find watchdog disable code and try to jump to it.
3. Construct watchdog disable code over watchdog enable code.
Set r18 to 0 and JMP here

Fun and scary things to do with memory

Read/write EEPROM (and extract cryptography keys)
Read parts of flash (e.g., reading locked bootloader section) could
be more useful than it seems
Staying persistent (writing flash)

Reading EEPROM/Flash
Ok, in most cases its almost easy to find gadget(s) that reads byte
from EEPROM or flash and stores it somewhere.
We could send it back over UART or any external channel gadgets
Not always possible, but there are good chances

Writing flash
Writing flash is locked during normal program execution
However, if you use jump-to-bootloader trick, you could write flash
from bootloader sections.
To do this, you need bootloader of that has enough gadgets.
However, modern bootloaders are big and sometimes you could be
lucky (e.g. Arduino bootloader)
Remember to disable interrupts before jumping to bootloader.

Infinite-ROP trick*
1. Set array to some upper stack address (A1) and N to some value
(128/256/etc) and JMP to read(..)
2. Output ROP-chain from UART to A1.
3. Set SPH/SPL to A1 (gadgets could be got from init code)
4. JMP to RET.
5. ???
6. Profit!
Dont forget to include 1 and 3-4 gadgets in the ROP-chain that you are
sending by UART.
*Possible on firmwares with read(array, N) from UART functions and complex init code

Mitigations

Mitigations (software)
Safe code/Dont trust external data (read 24 deadly sins of computer
security)
Reduce code size (less code -> less ROP gadgets)
Use rjmp/jmp instead of call/ret (ofc, it wont save you from ret2
function)
Use inconvenient memory models with small stack
Use stack canaries in your RTOS
Limit external libraries
Use watchdogs
Periodically check stack limits (to avoid stack expansion tricks)

Mitigations (hardware)
Disable JTAG/debuggers/etc, remove pins/wires of JTAG/ISP/UART
Write lock bits to 0/0
Use multilayered PCBs
Use external/hardware watchdogs
Use new ICs (more secure against various hardware attacks)
Use external safety controls/processors
And last, but not least:
Beware of Dmitry Nedospasov ;)

Part 4: Post-exploitation
&& Tricks

Conclusions
RCE on embedded systems isnt so hard as it seems.
Abusing of functionality is the main consequence of such attacks
However, more scary things like extracting cipherkeys or rewriting the
flash is possible
When developing embedded system remember that security also
should be part of the Software DLC process.

Books/links
.. AVR
Atmega128 disasm thread: http://www.avrfreaks.net/forum/disassembly-atmega128-bin-file
Exploiting buffer overflows on arduino: http://electronics.stackexchange.com/questions/78880/exploitingstack-buffer-overflows-on-an-arduino
Code Injection Attacks on Harvard-Architecture Devices: http://arxiv.org/pdf/0901.3482.pdf

Buffer overflow attack on an Atmega2560: http://www.avrfreaks.net/forum/buffer-overflow-attackatmega2560?page=all

Jump to bootloader: http://www.avrfreaks.net/forum/jump-bootloader-app-help-needed
AVR Libc reference manual:
http://www.atmel.com/webdoc/AVRLibcReferenceManual/overview_1overview_avr-libc.html
AVR GCC calling conventions: https://gcc.gnu.org/wiki/avr-gcc
Travis Goodspeed, Nifty Tricks and Sage Advice for Shellcode on Embedded Systems:
https://conference.hitb.org/hitbsecconf2013ams/materials/D1T1%20-%20Travis%20Goodspeed%20%20Nifty%20Tricks%20and%20Sage%20Advice%20for%20Shellcode%20on%20Embedded%20Systems.pdf
Pandoras Cash Box: The Ghost Under Your POS: https://recon.cx/2015/slides/recon2015-17-nitayartenstein-shift-reduce-Pandora-s-Cash-Box-The-Ghost-Under-Your-POS.pdf

Radare2. Links
http://radare.org
https://github.com/pwntester/cheatsheets/blob/master/radare2.
md
https://www.gitbook.com/book/radare/radare2book/details
https://github.com/radare/radare2ida

Any Q?
@dark_k3y

@dukeBarman
http://radare.org/r/

http://dsec.ru

http://eltech.ru

http://zorsecurity.ru

Now its CTF time!