pynvme: an open, fast and
extensible NVMe SSD test tool
Crane Chu, Engineer, Founder
GENG YUN Technology Pte Ltd
2020 Storage Developer Conference. © GENG YUN Technology Pte Ltd. All Rights Reserved. 1
Requirement
2020 Storage Developer Conference. © GENG YUN Technology Pte Ltd. All Rights Reserved. 2
Changes in SSD
• SSD has been changing for the decade:
• media:
• SLC, MLC, TLC, QLC
• 2D => 3D
• PCM, 3D-Xpoint...
• host:
• PATA, SATA, PCIe/NVMe
• open-channel
• up coming: ZNS, KV, ... ?
• DRAM
• form factor
• Agile: good for the constant change and uncertainty
2020 Storage Developer Conference. © GENG YUN Technology Pte Ltd. All Rights Reserved. 3
Agile Testing
• Developing and Testing are done interactively and iteratively.
• QA verifies the product of Dev/Test for customers.
• Testing and QA are different. Testing tools and QA tools are also different.
• Most available tools in the market are QA tools.
test QA
Developing
for developer for customer
before checkin before release
automatic manual Testing
white-box black-box
short long
changing stable QA Release
2020 Storage Developer Conference. © GENG YUN Technology Pte Ltd. All Rights Reserved. 4
Experience with dnvme
• extended dnvme with a python wrapper in user space, thus
developers can write test scripts in Python.
• Some essential problems:
• performance:
• IOPS, latency, consistency
• test efficiency
• stress tests
• maintainness: kernel module
• memory: user can only allocate virtual memory
• Then, SPDK comes ...
• high performance
• user space driver
• DMA memory
• python + nvme: pynvme https://github.com/pynvme/pynvme
2020 Storage Developer Conference. © GENG YUN Technology Pte Ltd. All Rights Reserved. 5
Design
2020 Storage Developer Conference. © GENG YUN Technology Pte Ltd. All Rights Reserved. 6
SPDK
2020 Storage Developer Conference. © GENG YUN Technology Pte Ltd. All Rights Reserved. 7
Architecture
library (ongoing): ZNS TCG psd
API: controller namespace qpair pcie buffer
SPDK: nvme driver cmdlog checksum ioworker
2020 Storage Developer Conference. © GENG YUN Technology Pte Ltd. All Rights Reserved. 8
Open Ecosystem
2020 Storage Developer Conference. © GENG YUN Technology Pte Ltd. All Rights Reserved. 9
Hardware
2020 Storage Developer Conference. © GENG YUN Technology Pte Ltd. All Rights Reserved. 10
High Performance
4K read IOPS
400
350
300
250
200
150
100
50
0
1 2 4
fio test 1 fio test 2 fio test 3
pynvme test 1 pynvme test 2 pynvme test 3
2020 Storage Developer Conference. © GENG YUN Technology Pte Ltd. All Rights Reserved. 11
Low Latency
latency
1600
1400
1200
1000
800
600
400
200
0
1 10 50 90 99 99.9 99.99
fio test 1 fio test 2 fio test 3 pynvme test 1 pynvme test 2 pynvme test 3
2020 Storage Developer Conference. © GENG YUN Technology Pte Ltd. All Rights Reserved. 12
Design
FIO pynvme
NVMe driver
in userspace
NVMe SSD
2020 Storage Developer Conference. © GENG YUN Technology Pte Ltd. All Rights Reserved. 13
psd: Python Space Driver
psd: IOSQ IOCQ SQE CQE PRPList
API: controller namespace qpair pcie buffer
SPDK: nvme driver cmdlog checksum ioworker
NVMe SSD
2020 Storage Developer Conference. © GENG YUN Technology Pte Ltd. All Rights Reserved. 14
Features
• access PCI configuration space
• access NVMe registers in BAR space
• send any NVMe admin/IO commands
• support callback functions for NVMe commands
• support MSI/MSIx interrupts
• transparent checksum verification on every LBA
• generates IO workload of high performance and low latency
• support multiple namespaces
• support multiple tests on different controllers
• integrate with the test framework pytest
• integrate with VSCode to display cmdlog in GUI
• support NVMe over TCP targets
• doc: https://pynvme.readthedocs.io/
2020 Storage Developer Conference. © GENG YUN Technology Pte Ltd. All Rights Reserved. 15
Examples
2020 Storage Developer Conference. © GENG YUN Technology Pte Ltd. All Rights Reserved. 16
Example: hello world
• pytest fixtures
• callback functions for commands
• sync point: waitdone()
• run test:
> make setup
> make test TESTS=scripts/test_examples.py::test_hello_world
> make test TESTS=scripts/test_examples.py::test_hello_world
pciaddr=0000:3d:00.0
> make test TESTS=scripts/test_examples.py::test_hello_world
pciaddr=172.168.5.44
2020 Storage Developer Conference. © GENG YUN Technology Pte Ltd. All Rights Reserved. 17
Example: psd
• create IOCQ/IOSQ respectively with
specified sqid and cqid
• fill commands in SQ entries
• manually trigger doorbell
2020 Storage Developer Conference. © GENG YUN Technology Pte Ltd. All Rights Reserved. 18
Example: ioworker
• fio-like IO generator
• Python API
• better performance
• sending IO in separated processes
• define IO pattern in parameters
• support multiple ioworkers
• support workload from simple to
complex
2020 Storage Developer Conference. © GENG YUN Technology Pte Ltd. All Rights Reserved. 19
Example: dirty power cycle
• ioworker sending IO in a separated
process
• cut power in main process when
ioworker is working
2020 Storage Developer Conference. © GENG YUN Technology Pte Ltd. All Rights Reserved. 20
Example: customized nvme init
• pynvme has a default nvme
initializaiton process
• users can define different nvme
initialization process in test scripts
2020 Storage Developer Conference. © GENG YUN Technology Pte Ltd. All Rights Reserved. 21
Example: latency of JEDEC workload
• ioworker generates IO according to
JEDEC enterprise workload
• scripts get latency at different
percentile (99%, 99.99%, 99.9999%)
2020 Storage Developer Conference. © GENG YUN Technology Pte Ltd. All Rights Reserved. 22
Example: check on error code
• pynvme through a warning when
command completes with an error
• scripts can use pytest to capture and
check the warning
2020 Storage Developer Conference. © GENG YUN Technology Pte Ltd. All Rights Reserved. 23
Example: sanitize an aer
• check if sanitize is
supported
• start sanitize operation
• send a write command,
and check if it is
aborted due to the
sanitize operation in
progress
• monitor the sanitize
progress till it is
completed
• check the final sanitize
status in logpage
2020 Storage Developer Conference. © GENG YUN Technology Pte Ltd. All Rights Reserved. 24
Example: multiple processes
• process 1: ioworker with flush, read,
trim commands
• process 2: ioworker with flush, write,
trim commands
• main process: print temperature
value in SMART data for every
second
2020 Storage Developer Conference. © GENG YUN Technology Pte Ltd. All Rights Reserved. 25
Live Demo in VSCode
2020 Storage Developer Conference. © GENG YUN Technology Pte Ltd. All Rights Reserved. 26
pynvme builds your own tests.
Ecosystem Scripts Extendability
Hardware Performance Service
2020 Storage Developer Conference. © GENG YUN Technology Pte Ltd. All Rights Reserved. 27
Please take a moment
to rate this session.
Your feedback matters to us.
Thank you!!!
2020 Storage Developer Conference. © GENG YUN Technology Pte Ltd. All Rights Reserved. 28
