Linux on z Systems and LinuxONE IBM

Device Drivers, Features, and Commands

on SUSE Linux Enterprise Server 12 SP2

SC34-2745-02
Linux on z Systems and LinuxONE IBM

Device Drivers, Features, and Commands

on SUSE Linux Enterprise Server 12 SP2

SC34-2745-02
 Note
Before using this document, be sure to read the information in Notices on page 697.

This edition applies to SUSE Linux Enterprise Server 12 SP2 and to all subsequent releases and modifications until
otherwise indicated in new editions.
Copyright IBM Corporation 2000, 2016.
US Government Users Restricted Rights Use, duplication or disclosure restricted by GSA ADP Schedule Contract
with IBM Corp.
Contents
Summary of changes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vii

About this publication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi

Part 1. General concepts . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

Chapter 1. How devices are accessed by Linux . . . . . . . . . . . . . . . . . . . 3

Chapter 2. Devices in sysfs . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

Chapter 3. Kernel and module parameters . . . . . . . . . . . . . . . . . . . . 23

Part 2. Booting and shutdown . . . . . . . . . . . . . . . . . . . . . . . . 29

Chapter 4. Console device drivers . . . . . . . . . . . . . . . . . . . . . . . . 31

Chapter 5. Booting Linux . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53

Chapter 6. Suspending and resuming Linux. . . . . . . . . . . . . . . . . . . . 75

Chapter 7. Shutdown actions . . . . . . . . . . . . . . . . . . . . . . . . . . 81

Chapter 8. Remotely controlling virtual hardware - snipl . . . . . . . . . . . . . . 85

Part 3. Storage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107

Chapter 9. DASD device driver . . . . . . . . . . . . . . . . . . . . . . . . . 109

Chapter 10. SCSI-over-Fibre Channel device driver . . . . . . . . . . . . . . . . 145

Chapter 11. Storage-class memory device driver supporting Flash Express . . . . . 187

Chapter 12. Channel-attached tape device driver . . . . . . . . . . . . . . . . . 191

Chapter 13. XPRAM device driver . . . . . . . . . . . . . . . . . . . . . . . . 201

Part 4. Networking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 205

Chapter 14. qeth device driver for OSA-Express (QDIO) and HiperSockets . . . . . . 209

Chapter 15. OSA-Express SNMP subagent support . . . . . . . . . . . . . . . . 277

Chapter 16. LAN channel station device driver . . . . . . . . . . . . . . . . . . 287

Chapter 17. CTCM device driver . . . . . . . . . . . . . . . . . . . . . . . . 293

Chapter 18. NETIUCV device driver . . . . . . . . . . . . . . . . . . . . . . . 305

Copyright IBM Corp. 2000, 2016 iii

 Chapter 19. AF_IUCV address family support. . . . . . . . . . . . . . . . . . . 315

Chapter 20. RDMA over Converged Ethernet . . . . . . . . . . . . . . . . . . . 319

Part 5. System resources . . . . . . . . . . . . . . . . . . . . . . . . . . 321

Chapter 21. Managing CPUs . . . . . . . . . . . . . . . . . . . . . . . . . . 323

| Chapter 22. NUMA emulation. . . . . . . . . . . . . . . . . . . . . . . . . . 329

Chapter 23. Managing hotplug memory . . . . . . . . . . . . . . . . . . . . . 333

Chapter 24. Large page support . . . . . . . . . . . . . . . . . . . . . . . . 339

Chapter 25. S/390 hypervisor file system . . . . . . . . . . . . . . . . . . . . 343

Chapter 26. ETR- and STP-based clock synchronization . . . . . . . . . . . . . . 349

Chapter 27. Identifying the z Systems hardware . . . . . . . . . . . . . . . . . 353

Chapter 28. The diag288 watchdog device driver . . . . . . . . . . . . . . . . . 355

Chapter 29. HMC media device driver . . . . . . . . . . . . . . . . . . . . . . 359

| Chapter 30. Data compression with GenWQE and zEDC Express . . . . . . . . . . 363

Part 6. z/VM virtual server integration . . . . . . . . . . . . . . . . . . . . 373

Chapter 31. z/VM concepts . . . . . . . . . . . . . . . . . . . . . . . . . . 375

Chapter 32. Writing kernel APPLDATA records . . . . . . . . . . . . . . . . . . 379

Chapter 33. Writing z/VM monitor records . . . . . . . . . . . . . . . . . . . . 385

Chapter 34. Reading z/VM monitor records. . . . . . . . . . . . . . . . . . . . 389

Chapter 35. z/VM recording device driver . . . . . . . . . . . . . . . . . . . . 395

Chapter 36. z/VM unit record device driver . . . . . . . . . . . . . . . . . . . . 403

Chapter 37. z/VM DCSS device driver . . . . . . . . . . . . . . . . . . . . . . 405

Chapter 38. z/VM CP interface device driver . . . . . . . . . . . . . . . . . . . 417

Chapter 39. z/VM special messages uevent support . . . . . . . . . . . . . . . . 419

Chapter 40. Cooperative memory management . . . . . . . . . . . . . . . . . . 425

Part 7. Security . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 427

Chapter 41. Generic cryptographic device driver . . . . . . . . . . . . . . . . . 429

Chapter 42. Pseudo-random number device driver . . . . . . . . . . . . . . . . 445

iv Device Drivers, Features, and Commands on SLES 12 SP2

| Chapter 43. Hardware-accelerated in-kernel cryptography . . . . . . . . . . . . . 449

Part 8. Performance measurement using hardware facilities. . . . . . . . . . 453

Chapter 44. Channel measurement facility . . . . . . . . . . . . . . . . . . . . 455

Chapter 45. OProfile hardware sampling support . . . . . . . . . . . . . . . . . 459

Chapter 46. Using the CPU-measurement counter facility . . . . . . . . . . . . . 463

Part 9. Diagnostics and troubleshooting . . . . . . . . . . . . . . . . . . . 469

Chapter 47. Logging I/O subchannel status information . . . . . . . . . . . . . . 471

Chapter 48. Control program identification . . . . . . . . . . . . . . . . . . . . 473

Chapter 49. Activating automatic problem reporting . . . . . . . . . . . . . . . . 477

Chapter 50. Avoiding common pitfalls . . . . . . . . . . . . . . . . . . . . . . 479

| Chapter 51. Displaying system information . . . . . . . . . . . . . . . . . . . 483

Chapter 52. Kernel messages . . . . . . . . . . . . . . . . . . . . . . . . . 485

Part 10. Reference . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 489

Chapter 53. Commands for Linux on z Systems . . . . . . . . . . . . . . . . . 491

Chapter 54. Selected kernel parameters . . . . . . . . . . . . . . . . . . . . . 669

Chapter 55. Linux diagnose code use . . . . . . . . . . . . . . . . . . . . . . 689

Part 11. Appendixes . . . . . . . . . . . . . . . . . . . . . . . . . . . . 691

Appendix A. Accessibility . . . . . . . . . . . . . . . . . . . . . . . . . . . 693

Appendix B. Understanding syntax diagrams. . . . . . . . . . . . . . . . . . . 695

Notices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 697

Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 699

Glossary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 703

Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 709

Contents v
vi Device Drivers, Features, and Commands on SLES 12 SP2
Summary of changes
This revision reflects changes for SUSE Linux Enterprise Server 12 Service Pack 2.

SUSE Linux Enterprise Server 12 SP2 changes

This editions contains changes related to SUSE Linux Enterprise Server 12 SP2.

New information
v You can now read measurement data for PCIe devices, see Reading statistics for
a PCIe device on page 22.
v The snipl tool now supports IPv6 connections between the Linux instance
where snipl runs and the SE, HMC, or z/VM CP instance that controls the
LPARs or guest virtual machines you want to work with, see Chapter 8,
Remotely controlling virtual hardware - snipl, on page 85.
v A HiperSockets port can be configured as a member of a Linux software
bridge, see Layer 2 bridge port function on page 221.
v Priority queueing for QDIO devices now supports IPv6. There are also two new
values that you can set, prio_queueing_vlan for VLANs and prio_queueing_skb
for other cases. See Using priority queueing on page 230.
v NUMA emulation is now available for Linux on z Systems. See Chapter 22,
NUMA emulation, on page 329.
v A new device driver facilitates hardware-accelerated data compression and
decompression through a PCIe-attached Field Programmable Gate Array (FPGA)
acceleration adapter, see Chapter 30, Data compression with GenWQE and
zEDC Express, on page 363.
v Information has been added about using hardware-acceleration for in-kernel
cryptographic operations, see Chapter 43, Hardware-accelerated in-kernel
cryptography, on page 449.
v There is a new section about obtaining information about your system and its
capabilities, Chapter 51, Displaying system information, on page 483.
v You can now view z Systems specific kernel messages through an app for
mobile devices. See Viewing messages with the IBM Doc Buddy app on page
486.
v A new command, cpacfstats, lets you monitor CPACF activity, see cpacfstats -
Monitor CPACF cryptographic activity on page 522.
v With new parameters for the vmur command, you can control the CLASS, DEST,
FORM, and DIST spooling options for virtual unit record devices. See vmur -
Work with z/VM spool file queues on page 652.
v A new section describes the fips kernel parameter, which enables the FIPS mode
of operation, fips - Run Linux in FIPS mode on page 675.

Changed Information
v You can now IPL from subchannel sets other than 0, see Booting from DASD
on page 61 and Attributes for ccw on page 70.
v The format of SCSI device nodes that are based on bus IDs has changed, see
SCSI device nodes on page 147.

Copyright IBM Corp. 2000, 2016 vii

 v The storage-class memory device driver now supports submitting more
concurrent I/O requests than the current limit, see Setting up the storage-class
memory device driver on page 188.
v The qeth device driver now supports offloading checksum operations in layer 2
as well as layer 3, see Configuring offload operations on page 238.

This revision also includes maintenance and editorial changes. Technical changes
or additions to the text and illustrations are indicated by a vertical line to the left
of the change.

Deleted Information
v CLAW devices are no longer supported and the description of the CLAW device
driver has been removed.

SUSE Linux Enterprise Server 12 SP1 changes

This editions contains changes related to SUSE Linux Enterprise Server 12 SP1.

New information
v Linux on z Systems now includes a HMC media device driver to access files on
removable media at systems that run the HMC. Installers on suitably prepared
installation DVDs can use this device driver to install Linux in an LPAR. See the
following sections:
Loading Linux from removable media or from an FTP server on page 65
Chapter 29, HMC media device driver, on page 359
hmcdrvfs - Mount a FUSE file system for remote access to media in the
HMC media drive on page 560
lshmc - List media contents in the HMC media drive on page 586
v You can now, if needed, tune the behavior of the automatic port scan, see
Controlling automatic port scanning on page 161.
v New attributes for FCP-attached SCSI devices let you check whether a device is
trying to recover, recovery failed, or access is denied. See Table 29 on page 172,
and Recovering failed SCSI devices on page 175.
v Linux in LPAR mode now supports simultaneous multithreading, see
Simultaneous multithreading on page 323.
v The cryptographic device driver now exploits the Crypto Express5S (CEX5S)
feature, see Hardware and software prerequisites on page 430.
v The pseudorandom number generator device driver now supports version 5 of
the Message Security Assist (MSA), available as of the EC12 with the latest
firmware level. See Chapter 42, Pseudo-random number device driver, on
page 445.
v The support for the z/Architecture CPU-measurement facilities now includes
the CPU-measurement sampling facility, see Chapter 46, Using the
CPU-measurement counter facility, on page 463. A new command helps you to
display details about supported and authorized counters and sampling modes,
see lscpumf - Display information about the CPU-measurement facilities on
page 577.
v The hyptop command can now display additional data, see hyptop - Display
hypervisor performance data on page 564.
Time data by thread for LPARs with multithreading.
Management time for z/VM mode.

viii Device Drivers, Features, and Commands on SLES 12 SP2

Changed Information
v You can now display the supported forwarding modes of a switch, see Isolating
data connections on page 240.
v The z/VM watchdog device driver has been replaced by the diag288 watchdog
device driver. The new device driver can be used for both Linux on z/VM and
Linux in LPAR mode. See Chapter 28, The diag288 watchdog device driver, on
page 355.

This revision also includes maintenance and editorial changes. Technical changes
or additions to the text and illustrations are indicated by a vertical line to the left
of the change.

Deleted Information
v The mem= kernel parameter has become obsolete and is no longer described.

Summary of changes ix
x Device Drivers, Features, and Commands on SLES 12 SP2
About this publication
This publication describes the device drivers, features, and commands available to
SUSE Linux Enterprise Server 12 SP2 for the control of IBM z Systems devices
and attachments. Unless stated otherwise, in this publication the terms device
drivers and features are understood to refer to device drivers and features for SUSE
Linux Enterprise Server 12 SP2 for z Systems.

Unless stated otherwise, all z/VM related information in this document assumes a
current z/VM version, see www.vm.ibm.com/techinfo/lpmigr/vmleos.html.

In this publication, z Systems is taken to include all IBM mainframe systems

supported by SUSE Linux Enterprise Server 12 SP2 for z Systems. In particular,
this includes IBM z13 (z13), IBM zEnterprise BC12 (zBC12), IBM zEnterprise
EC12 (zEC12), IBM zEnterprise 196 (z196), and IBM zEnterprise 114 (z114)
mainframes.

For more specific information about the device driver structure, see the documents
in the kernel source tree at /usr/src/linux-<version>/Documentation/s390

For what is new, known issues, prerequisites, restrictions, and frequently asked
questions, see the SUSE Linux Enterprise Server 12 SP2 release notes at
www.suse.com/releasenotes

You can find the latest version of this publication on the developerWorks website
at www.ibm.com/developerworks/linux/linux390/documentation_suse.html

How this document is organized

The first part of this document contains general and overview information for the
z Systems device drivers for SUSE Linux Enterprise Server 12 SP2 for z Systems.

Part two contains chapters about device drivers and features that are used in the
context of booting and shutting down Linux.

Part three contains chapters specific to individual storage device drivers.

Part four contains chapters specific to individual network device drivers.

Part five contains chapters about device drivers and features that help to manage
the resources of the real or virtual hardware.

Part six contains chapters that describe device drivers and features in support of
z/VM virtual server integration.

Part seven contains chapters about device drivers and features that support
security aspects of SUSE Linux Enterprise Server 12 SP2 for z Systems.

Part eight contains chapters about assessing the performance of Linux on z

Systems.

Part nine contains chapters about device drivers and features that are used in the
context of diagnostics and problem solving.

Copyright IBM Corp. 2000, 2016 xi

 Part ten contains chapters with reference information about commands, kernel
parameters, and Linux use of z/VM DIAG calls.

Who should read this document

Most of the information in this document is intended for system administrators
who want to configure SUSE Linux Enterprise Server 12 SP2 for z Systems.

The following general assumptions are made about your background knowledge:
v You have an understanding of basic computer architecture, operating systems,
and programs.
v You have an understanding of Linux and z Systems terminology.
v You are familiar with Linux device driver software.
v You are familiar with the z Systems devices attached to your system.

Programmers: Some sections are of interest primarily to specialists who want to

program extensions to the Linux on z Systems device drivers and features.

Conventions and assumptions used in this publication

This section summarizes the styles, highlighting, and assumptions used throughout
this publication.

Authority
Most of the tasks described in this document require a user with root authority. In
particular, writing to procfs, and writing to most of the described sysfs attributes
requires root authority.

Throughout this document, it is assumed that you have root authority.

Using sysfs and YaST

This document describes how to change settings and options in sysfs. In most
cases, changes in sysfs are not persistent. To make your changes persistent, use
YaST. If you use a tool other than YaST, ensure that the tool makes persistent
changes. See SUSE Linux Enterprise Server 12 SP2 Deployment Guide and SUSE Linux
Enterprise Server 12 SP2 Administration Guide for details.

Terminology
In this publication, the term booting is used for running boot loader code that loads
the Linux operating system. IPL is used for issuing an IPL command to load boot
loader code or a stand-alone dump utility. See also IPL and booting on page 53.

sysfs and procfs

In this publication, the mount point for the virtual Linux file system sysfs is
assumed to be /sys. Correspondingly, the mount point for procfs is assumed to be
/proc.

debugfs
This document assumes that debugfs has been mounted at /sys/kernel/debug.

To mount debugfs, you can use this command:

xii Device Drivers, Features, and Commands on SLES 12 SP2

 # mount none -t debugfs /sys/kernel/debug

Number prefixes
In this publication, KB means 1024 bytes, MB means 1,048,576 bytes, and GB
means 1,073,741,824 bytes.

Hexadecimal numbers
Mainframe publications and Linux publications tend to use different styles for
writing hexadecimal numbers. Thirty-one, for example, would typically read X'1F'
in a mainframe publication and 0x1f in a Linux publication.

Because the Linux style is required in many commands and is also used in some
code samples, the Linux style is used throughout this publication.

Highlighting
This publication uses the following highlighting styles:
v Paths and URLs are highlighted in monospace.
v Variables are highlighted in <italics within angled brackets>.
v Commands in text are highlighted in monospace bold.
v Input and output as normally seen on a computer screen is shown
within a screen frame.
Prompts are shown as hash signs:
#

About this publication xiii

xiv Device Drivers, Features, and Commands on SLES 12 SP2
Part 1. General concepts
Chapter 1. How devices are accessed by Linux . . 3 Channel path measurement . . . . . . . . . 14
Device names, device nodes, and major/minor Channel path ID information . . . . . . . . 15
numbers . . . . . . . . . . . . . . . . 3 CCW hotplug events . . . . . . . . . . . 19
Network interfaces . . . . . . . . . . . . 4 PCI Express support . . . . . . . . . . . 19

Chapter 2. Devices in sysfs . . . . . . . . . 7 Chapter 3. Kernel and module parameters . . . 23

Device categories . . . . . . . . . . . . . 7 Kernel parameters . . . . . . . . . . . . 23
Device directories. . . . . . . . . . . . . 9 Module parameters . . . . . . . . . . . . 26
Device views in sysfs . . . . . . . . . . . 11

This information at an overview level describes concepts that apply across different
device drivers and kernel features.

Newest version

You can find the newest version of this publication on IBM Knowledge Center at
www.ibm.com/support/knowledgecenter/linuxonibm/liaaf/lnz_r_suse.html

Copyright IBM Corp. 2000, 2016 1

2 Device Drivers, Features, and Commands on SLES 12 SP2
Chapter 1. How devices are accessed by Linux
Applications on Linux access character and block devices through device nodes,
and network devices through network interfaces.

Device names, device nodes, and major/minor numbers

The Linux kernel represents character and block devices as pairs of numbers
<major>:<minor>.

Some major numbers are reserved for particular device drivers. Other device nodes
are dynamically assigned to a device driver when Linux boots. For example, major
number 94 is always the major number for DASD devices while the device driver
for channel-attached tape devices has no fixed major number. A major number can
also be shared by multiple device drivers. See /proc/devices to find out how
major numbers are assigned on a running Linux instance.

The device driver uses the minor number <minor> to distinguish individual
physical or logical devices. For example, the DASD device driver assigns four
minor numbers to each DASD: one to the DASD as a whole and the other three for
up to three partitions.

Device drivers assign device names to their devices, according to a device

driver-specific naming scheme (see, for example, DASD naming scheme on page
115). Each device name is associated with a minor number (see Figure 1).

Figure 1. Minor numbers and device names

User space programs access character and block devices through device nodes also
referred to as device special files. When a device node is created, it is associated with
a major and minor number (see Figure 2).

Figure 2. Device nodes

SUSE Linux Enterprise Server 12 SP2 uses udev to create device nodes for you.
Standard device nodes match the device name that is used by the kernel, but

Copyright IBM Corp. 2000, 2016 3

 different or additional nodes might be created by special udev rules. See SUSE
Linux Enterprise Server 12 SP2 Administration Guide and the udev man page for
more details.

Network interfaces
The Linux kernel representation of a network device is an interface.

Figure 3. Interfaces

When a network device is defined, it is associated with a real or virtual network

adapter (see Figure 3). You can configure the adapter properties for a particular
network device through the device representation in sysfs (see Device directories
on page 9).

You activate or deactivate a connection by addressing the interface with ifconfig

or an equivalent command. All interfaces that are provided by the z Systems
specific network device drivers are interfaces for the Internet Protocol (IP).

Interface names
The interface names are assigned by the Linux network stack.

Interface names are of the form <base_name><n> where <base_name> is a base name
that is used for a particular interface type. <n> is an index number that identifies
an individual interface of a particular type.

Table 1 summarizes the base names that are used for the network device drivers
for interfaces that are associated with real hardware.
Table 1. Interface base names for real devices.

This table lists interface type and applicable device driver for the available base names.
Base name Interface type Device driver Hardware
module
eth Ethernet qeth, lcs OSA-Express features
eth Ethernet mlx4_en RoCE Express feature

This table is intended as an overview only. For details about which version of a
particular hardware is supported by a device driver, see the applicable section
about the device driver.

Table 2 on page 5 summarizes the base names that are used for the network device
drivers for interfaces that are associated with virtual hardware:

4 Device Drivers, Features, and Commands on SLES 12 SP2

 Table 2. Interface base names for virtual devices.

This table lists interface type and applicable device driver for the available base names.
Base name Interface type Device driver Comment
module
hsi HiperSockets , virtual qeth Real HiperSockets or
NIC virtual NIC type
HiperSockets coupled
to a guest LAN
eth virtual NIC qeth QDIO virtual NIC
coupled to a guest
LAN or virtual
switch

When the first device for a particular interface name is set online, it is assigned the
index number 0, the second is assigned 1, the third 2, and so on. For example, the
first HiperSockets interface is named hsi0, the second hsi1, the third hsi2, and so
on.

When a network device is set offline, it retains its interface name. When a device is
removed, it surrenders its interface name and the name can be reassigned as
network devices are defined in the future. When an interface is defined, the Linux
kernel always assigns the interface name with the lowest free index number for the
particular type. For example, if the network device with an associated interface
name hsi1 is removed while the devices for hsi0 and hsi2 are retained, the next
HiperSockets interface to be defined becomes hsi1.

Matching devices with the corresponding interfaces

If you define multiple interfaces on a Linux instance, you must keep track of the
interface names assigned to your network devices.

SUSE Linux Enterprise Server 12 SP2 uses udev to track the network interface
name and preserves the mapping of interface names to network devices across
IPLs.

How you can keep track of the mapping yourself differs depending on the
network device driver. For qeth, you can use the lsqeth command (see lsqeth -
List qeth-based network devices on page 591) to obtain a mapping.

After setting a device online, read /var/log/messages or issue dmesg to find the
associated interface name in the messages that are issued in response to the device
being set online.

For each network device that is online, there is a symbolic link of the form
/sys/class/net/<interface>/device where <interface> is the interface name. This
link points to a sysfs directory that represents the corresponding network device.
You can read this symbolic link with readlink to confirm that an interface name
corresponds to a particular network device.

| Main steps for setting up a network interface

| The main steps apply to all Linux on z Systems network devices drivers that are
| based on ccwgroup devices (for example, qeth and lcs devices). How to perform a
| particular step can be different for the different device drivers.

Chapter 1. How devices are accessed 5

| The main steps are:
| 1. Create a network device by combining suitable subchannels into a group
| device. The device driver then creates directories that represent the device in
| sysfs.
| 2. Configure the device through its attributes in sysfs. See Device views in sysfs
| on page 11. Some devices have attributes that can or must be set later when the
| device is online or when the connection is active.
| 3. Set the device online. This step associates the device with an interface name
| and thus makes the device known to the Linux network stack. For devices that
| are associated with a physical network adapter it also initializes the adapter for
| the network interface.
| 4. Configure and activate the interface. This step adds interface properties like IP
| addresses, netmasks, and MTU to the network interface and moves the network
| interface into state "up". The interface is then ready for user space (socket)
| programs to run connections and transfer data across it.

| To configure a network device, use tools provided with SUSE Linux Enterprise
| Server 12 SP2. See SUSE Linux Enterprise Server 12 SP2 Administration Guide.

6 Device Drivers, Features, and Commands on SLES 12 SP2

Chapter 2. Devices in sysfs
Most of the device drivers create structures in sysfs. These structures hold
information about individual devices and are also used to configure and control
the devices.

Device categories
There are several Linux on z Systems specific device categories in the /sys/devices
directory.

Figure 4 illustrates a part of sysfs.

Figure 4. sysfs

/sys/bus and /sys/devices are common Linux directories. The directories

following /sys/bus sort the device drivers according to the categories of devices
they control. There are several categories of devices. The sysfs branch for a
particular category might be missing if there is no device for that category.
AP devices
are adjunct processors used for cryptographic operations.
CCW devices
are devices that can be addressed with channel-command words (CCWs).
These devices use a single subchannel on the mainframe's channel
subsystem.

Copyright IBM Corp. 2000, 2016 7

 CCW group devices
are devices that use multiple subchannels on the mainframe's channel
subsystem.
IUCV devices
are devices for virtual connections between z/VM guest virtual machines
within an IBM mainframe. IUCV devices do not use the channel
subsystem.
PCI devices
represent PCIe devices, for example, a 10GbE RoCE Express device. In
sysfs, PCIe devices are listed in the /pci directory rather than the /pcie
directory.

Table 3 lists the device drivers that have representation in sysfs:

Table 3. Device drivers with representation in sysfs
Device driver Category sysfs directories
3215 console CCW /sys/bus/ccw/drivers/3215
3270 console CCW /sys/bus/ccw/drivers/3270
DASD CCW /sys/bus/ccw/drivers/dasd-eckd
/sys/bus/ccw/drivers/dasd-fba
SCSI-over-Fibre Channel CCW /sys/bus/ccw/drivers/zfcp
Storage class memory SCM /sys/bus/scm/
supporting Flash Express
Channel-attached tape CCW /sys/bus/ccw/drivers/tape_34xx
/sys/bus/ccw/drivers/tape_3590
Cryptographic AP
/sys/bus/ap/drivers/cex5a
/sys/bus/ap/drivers/cex5c
/sys/bus/ap/drivers/cex5p
/sys/bus/ap/drivers/cex4a
/sys/bus/ap/drivers/cex4c
/sys/bus/ap/drivers/cex4p
/sys/bus/ap/drivers/cex3a
/sys/bus/ap/drivers/cex3c
/sys/bus/ap/drivers/pcixcc
DCSS n/a /sys/devices/dcssblk
XPRAM n/a /sys/devices/system/xpram
z/VM recording IUCV /sys/bus/iucv/drivers/vmlogrdr
qeth (OSA-Express features CCW group /sys/bus/ccwgroup/drivers/qeth
and HiperSockets )
LCS CCW group /sys/bus/ccwgroup/drivers/lcs
CTCM CCW group /sys/bus/ccwgroup/drivers/ctcm
NETIUCV IUCV /sys/bus/iucv/drivers/netiucv
CLAW CCW group /sys/bus/ccwgroup/drivers/claw
10GbE RoCE Express devices PCI sys/bus/pci/drivers/mlx4_core
(mlx4_en)

8 Device Drivers, Features, and Commands on SLES 12 SP2

 Some device drivers do not relate to physical devices that are connected through
the channel subsystem. Their representation in sysfs differs from the CCW and
CCW group devices, for example, the Cryptographic device drivers have their own
category, AP.

The following sections provide more details about devices and their representation
in sysfs.

Device directories
Each device that is known to Linux is represented by a directory in sysfs.

For CCW and CCW group devices the name of the directory is a bus ID that
identifies the device within the scope of a Linux instance. For a CCW device, the
bus ID is the device's device number with a leading 0.<n>., where <n> is the
subchannel set ID. For example, 0.1.0ab1.

CCW group devices are associated with multiple device numbers. For CCW group
devices, the bus ID is the primary device number with a leading 0.<n>., where
<n> is the subchannel set ID.

Device views in sysfs on page 11 tells you where you can find the device
directories with their attributes in sysfs.

Device attributes
The device directories contain attributes. You control a device by writing values to
its attributes.

Some attributes are common to all devices in a device category, other attributes are
specific to a particular device driver. The following attributes are common to all
CCW devices:
online
You use this attribute to set the device online or offline. To set a device online,
write the value 1 to its online attribute. To set a device offline, write the value
0 to its online attribute.
cutype
specifies the control unit type and model, if applicable. This attribute is
read-only.
cmb_enable
enables I/O data collection for the device. See Enabling, resetting, and
switching off data collection on page 456 for details.
devtype
specifies the device type and model, if applicable. This attribute is read-only.
availability
indicates whether the device can be used. The following values are possible:
good
This is the normal state. The device can be used.
boxed
The device is locked by another operating system instance and cannot be
used until the lock is surrendered or the DASD is accessed by force (see
Accessing DASD by force on page 125).

Chapter 2. Devices in sysfs 9

 no device
Applies to disconnected devices only. The device disappears after a
machine check and the device driver requests to keep the device online
anyway. Changes back to good when the device returns after another
machine check and the device driver accepts the device back.
no path
Applies to disconnected devices only. After a machine check or a logical
vary off, no path remains to the device. However, the device driver keeps
the device online. Changes back to good when the path returns after
another machine check or logical vary on and the device driver accepts the
device back.
modalias
contains the module alias for the device. It is of the format:
ccw:t<cu_type>m<cu_model>

or
ccw:t<cu_type>m<cu_model>dt<dev_type>dm<dev_model>

Setting attributes
Directly write to attributes or, for CCW devices, use the chccwdev command to set
attribute values.

Procedure
v You can set a writable attribute by writing the designated value to the
corresponding attribute file.
v For CCW devices, you can also use the chccwdev command (see chccwdev - Set
CCW device attributes on page 492) to set attributes.
With a single chccwdev command you can:
Set an attribute for multiple devices
Set multiple attributes for a device, including setting the device online
Set multiple attributes for multiple devices

Working with newly available devices

Errors can occur if you try to work with a device before its sysfs representation is
completely initialized.

About this task

When new devices become available to a running Linux instance, some time
elapses until the corresponding device directories and their attributes are created in
sysfs. Errors can occur if you attempt to work with a device for which the sysfs
structures are not present or are not complete. These errors are most likely to occur
and most difficult to handle when you are configuring devices with scripts.

Procedure

Use the following steps before you work with a newly available device to avoid
such errors:
1. Attach the device, for example, with a z/VM CP ATTACH command.
2. Assure that the sysfs structures for the new device are complete.

10 Device Drivers, Features, and Commands on SLES 12 SP2

 # echo 1 > /proc/cio_settle

This command returns control after all pending updates to sysfs are complete.

Tip: For CCW devices you can omit this step if you then use chccwdev (see
chccwdev - Set CCW device attributes on page 492) to work with the devices.
chccwdev triggers cio_settle for you and waits for cio_settle to complete.

Results

You can now work with the new device, For example, you can set the device
online or set attributes for the device.

Device views in sysfs

sysfs provides multiple views of device specific data.

The most important views are:

v Device driver view
v Device category view on page 12
v Device view on page 12
v Channel subsystem view on page 12

Many paths in sysfs contain device bus-IDs to identify devices. Device bus-IDs of
subchannel-attached devices are of the form:
0.<n>.<devno>

where <n> is the subchannel set-ID and <devno> is the device number.

Device driver view

This view groups devices by the device drivers that control them.

The device driver view is of the form:

/sys/bus/<bus>/drivers/<driver>/<device_bus_id>

where:
<bus> is the device category, for example, ccw or ccwgroup.
<driver>
is a name that specifies an individual device driver or the device driver
component that controls the device (see Table 3 on page 8).
<device_bus_id>
identifies an individual device (see Device directories on page 9).

Note: DCSSs and XPRAM are not represented in this view.

Examples
v This example shows the path for an ECKD type DASD device:
/sys/bus/ccw/drivers/dasd-eckd/0.0.b100
v This example shows the path for a qeth device:
/sys/bus/ccwgroup/drivers/qeth/0.0.a100

Chapter 2. Devices in sysfs 11

 v This example shows the path for a cryptographic device (a CEX4A card):
/sys/bus/ap/drivers/cex4a/card3b

Device category view

This view groups devices by major categories that can span multiple device
drivers.

The device category view does not sort the devices according to their device
drivers. All devices of the same category are contained in a single directory. The
device category view is of the form:
/sys/bus/<bus>/devices/<device_bus_id>

where:
<bus> is the device category, for example, ccw or ccwgroup.
<device_bus_id>
identifies an individual device (see Device directories on page 9).

Note: DCSSs and XPRAM are not represented in this view.

Examples
v This example shows the path for a CCW device.
/sys/bus/ccw/devices/0.0.b100
v This example shows the path for a CCW group device.
/sys/bus/ccwgroup/devices/0.0.a100
v This example shows the path for a cryptographic device:
/sys/bus/ap/devices/card3b

Device view
This view sorts devices according to their device drivers, but independent from the
device category. It also includes logical devices that are not categorized.

The device view is of the form:

/sys/devices/<driver>/<device>

where:
<driver>
is a name that specifies an individual device driver or the device driver
component that controls the device.
<device>
identifies an individual device. The name of this directory can be a device
bus-ID or the name of a DCSS or IUCV device.

Examples
v This example shows the path for a qeth device.
/sys/devices/qeth/0.0.a100
v This example shows the path for a DCSS block device.
/sys/devices/dcssblk/mydcss

Channel subsystem view

The channel subsystem view shows the relationship between subchannels and
devices.

12 Device Drivers, Features, and Commands on SLES 12 SP2

 The channel subsystem view is of the form:
/sys/devices/css0/<subchannel>

where:
<subchannel>
is a subchannel number with a leading 0.<n>., where <n> is the
subchannel set ID.

I/O subchannels show the devices in relation to their respective subchannel sets
and subchannels. An I/O subchannel is of the form:
/sys/devices/css0/<subchannel>/<device_bus_id>

where:
<subchannel>
is a subchannel number with a leading 0.<n>., where <n> is the
subchannel set ID.
<device_bus_id>
is a device number with a leading 0.<n>., where <n> is the subchannel
set ID (see Device directories on page 9).

Examples
v This example shows a CCW device with device number 0xb100 that is associated
with a subchannel 0x0001.
/sys/devices/css0/0.0.0001/0.0.b100
v This example shows a CCW device with device number 0xb200 that is associated
with a subchannel 0x0001 in subchannel set 1.
/sys/devices/css0/0.1.0001/0.1.b200
v The entries for a group device show as separate subchannels. If a CCW group
device uses three subchannels 0x0002, 0x0003, and 0x0004 the subchannel
information could be:
/sys/devices/css0/0.0.0002/0.0.a100
/sys/devices/css0/0.0.0003/0.0.a101
/sys/devices/css0/0.0.0004/0.0.a102

Each subchannel is associated with a device number. Only the primary device
number is used for the bus ID of the device in the device driver view and the
device view.
v This example lists the information available for a non-I/O subchannel with
which no device is associated:
ls /sys/devices/css0/0.0.ff00/
bus driver modalias subsystem type uevent

Subchannel attributes
There are sysfs attributes that represent subchannel properties, including common
attributes and information specific to the subchannel type.

Subchannels have two common attributes:

type
The subchannel type, which is a numerical value, for example:
v 0 for an I/O subchannel
v 1 for a CHSC subchannel
v 3 for an EADM subchannel
Chapter 2. Devices in sysfs 13
 modalias
The module alias for the device of the form css:t<n>, where <n> is the
subchannel type (for example, 0 or 1).

These two attributes are the only ones that are always present. Some subchannels,
like I/O subchannels, might contain devices and further attributes.

Apart from the bus ID of the attached device, I/O subchannel directories typically
contain these attributes:
chpids
is a list of the channel-path identifiers (CHPIDs) through with the device is
connected. See also Channel path ID information on page 15.
pimpampom
provides the path installed, path available, and path operational masks. See
z/Architecture Principles of Operation, SA22-7832 for details about the masks.

Channel path measurement

A sysfs attribute controls the channel path measurement facility of the channel
subsystem.

/sys/devices/css0/cm_enable

With the cm_enable attribute you can enable and disable the extended channel-path
measurement facility. It can take the following values:
0 Deactivates the measurement facility and remove the measurement-related
attributes for the channel paths. No action if measurements are not active.
1 Attempts to activate the measurement facility and create the
measurement-related attributes for the channel paths. No action if
measurements are already active.

If a machine does not support extended channel-path measurements the cm_enable

attribute is not created.

Two sysfs attributes are added for each channel path object:
cmg Specifies the channel measurement group or unknown if no characteristics
are available.
shared
Specifies whether the channel path is shared between LPARs or unknown
if no characteristics are available.

If measurements are active, two more sysfs attributes are created for each channel
path object:
measurement
A binary sysfs attribute that contains the extended channel-path
measurement data for the channel path. It consists of eight 32-bit values
and must always be read in its entirety, or 0 will be returned.
measurement_chars
A binary sysfs attribute that is either empty, or contains the channel
measurement group dependent characteristics for the channel path, if the
channel measurement group is 2 or 3. If not empty, it consists of five 32-bit
values.

14 Device Drivers, Features, and Commands on SLES 12 SP2

 Examples
v To turn measurements on issue:
# echo 1 > /sys/devices/css0/cm_enable

v To turn measurements off issue:

# echo 0 > /sys/devices/css0/cm_enable

Channel path ID information

All CHPIDs that are known to Linux are shown alongside the subchannels in the
/sys/devices/css0 directory.

The directories that represent the CHPIDs have the form:

/sys/devices/css0/chp0.<chpid>

where <chpid> is a two digit hexadecimal CHPID.

Example: /sys/devices/css0/chp0.4a

Setting a CHPID logically online or offline

Directories that represent CHPIDs contain a status attribute that you can use to
set the CHPID logically online or offline.

About this task

When a CHPID has been set logically offline from a particular Linux instance, the
CHPID is, in effect, offline for this Linux instance. A CHPID that is shared by
multiple operating system instances can be logically online to some instances and
offline to others. A CHPID can also be logically online to Linux while it has been
varied off at the SE.

Procedure

To set a CHPID logically online, set its status attribute to online by writing the
value on to it. To set a CHPID logically offline, set its status attribute to offline
by writing off to it.

Issue a command of this form:

# echo <value> > /sys/devices/css0/chp0.<CHPID>/status

where:
<CHPID>
is a two digit hexadecimal CHPID.
<value>
is either on or off.

Chapter 2. Devices in sysfs 15

 Examples
v To set a CHPID 0x4a logically offline issue:
# echo off > /sys/devices/css0/chp0.4a/status

v To read the status attribute to confirm that the CHPID is logically offline issue:
# cat /sys/devices/css0/chp0.4a/status
offline

v To set the same CHPID logically online issue:

# echo on > /sys/devices/css0/chp0.4a/status

v To read the status attribute to confirm that the CHPID is logically online issue:
# cat /sys/devices/css0/chp0.4a/status
online

Configuring a CHPID on LPAR

For Linux in LPAR mode, directories that represent CHPIDs contain a configure
attribute that you can use to query and change the configuration state of I/O
channel-paths.

About this task

The following configuration changes are supported:

v From standby to configured (configure)
v From configured to standby (deconfigure)

Procedure

To configure a CHPID, set its configure attribute by writing the value 1 to it. To
deconfigure a CHPID, set its configure attribute by writing 0 to it.

Issue a command of this form:

# echo <value> > /sys/devices/css0/chp0.<CHPID>/configure

where:
<CHPID>
is a two digit hexadecimal CHPID.
<value>
is either 1 or 0.

To query and set the configure value using commands, see chchp - Change
channel path status on page 494 and lschp - List channel paths on page 575.

Examples
v To set a channel path with the ID 0x40 to standby issue:
# echo 0 > /sys/devices/css0/chp0.40/configure

16 Device Drivers, Features, and Commands on SLES 12 SP2

 This operation is equivalent to performing a Configure Channel Path Off
operation on the hardware management console.
v To read the configure attribute to confirm that the channel path has been set to
standby issue:
# cat /sys/devices/css0/chp0.40/configure
0

v To set the same CHPID to configured issue:

# echo 1 > /sys/devices/css0/chp0.40/configure

This operation is equivalent to performing a Configure Channel Path On

operation on the hardware management console.
v To read the status attribute to confirm that the CHPID has been set to
configured issue:
# cat /sys/devices/css0/chp0.40/configure
1

Finding the physical channel associated with a CHPID

Use the mapping of physical channel IDs (PCHID) to CHPIDs to find the hardware
from the CHPID number or the CHPID numbers from the PCHID.

About this task

A CHPID is associated with either a physical port or with an internal connection

defined inside the mainframe, such as HiperSockets. See Figure 5. You can
determine the PCHID or internal channel ID number that is associated with a
CHPID number.

Figure 5. Relationships between CHPIDs, PCHIDs, and internal channel ID numbers.

Knowing the PCHID number can be useful in the following situations:

v When Linux indicates that a CHPID is in an error state, you can use the PCHID
number to identify the associated hardware.

Chapter 2. Devices in sysfs 17

 v When a hardware interface requires service action, the PCHID mapping can be
used to determine which CHPIDs and I/O devices will be affected.
The internal channel ID number can be useful to determine which CHPIDs are
connected to the same communication path, such as a HiperSockets link.

Procedure

To find the physical channel ID corresponding to a CHPID, either:

v Display the mapping of all CHPIDs to PCHIDs. Issue the lschp command:
# lschp

v Find the channel-ID related files for the CHPID. These sysfs files are located
under /sys/devices/css0/chp0.<num>, where <num> is the two-digit, lowercase,
hexadecimal CHPID number. There are two attribute files:
chid The channel ID number.
chid_external
A flag that indicates whether this CHPID is associated with an internal
channel ID (value 0) or a physical channel ID (value 1).
The sysfs attribute files are created only when channel ID information is
available to Linux. For Linux on z/VM, the availability of this information
depends on the z/VM version and configuration. For Linux in LPAR mode, this
information is always available.

Example

The lschp command shows channel ID information in a column labeled PCHID.

Internal channel IDs are enclosed in brackets. If no channel ID information is
available, the column shows "-".
# lschp
CHPID Vary Cfg. Type Cmg Shared PCHID
============================================
0.30 1 1 1b 2 1 0390
0.31 1 1 1b 2 1 0392
0.32 1 1 1b 2 1 0510
0.33 1 1 1b 2 1 0512
0.34 1 0 1b - - 0580
0.fc 1 1 24 3 1 (0702)
0.fd 1 1 24 3 1 (0703)
0.fe 1 1 24 3 1 (0704)

In this example, CHPID 30 is associated with PCHID 0390, and CHPID fe is

associated with internal channel ID 0704.

Alternatively, read the chid and chid_external sysfs attributes, for example for
CHPID 30:
# cat /sys/devices/css0/chp0.30/chid
0390
# cat /sys/devices/css0/chp0.30/chid_external
1

18 Device Drivers, Features, and Commands on SLES 12 SP2

CCW hotplug events
A hotplug event is generated when a CCW device appears or disappears with a
machine check.

The hotplug events provide the following variables:

CU_TYPE
for the control unit type of the device that appeared or disappeared.
CU_MODEL
for the control unit model of the device that appeared or disappeared.
DEV_TYPE
for the type of the device that appeared or disappeared.
DEV_MODEL
for the model of the device that appeared or disappeared.
MODALIAS
for the module alias of the device that appeared or disappeared. The module
alias is the same value that is contained in /sys/devices/css0/
<subchannel_id>/<device_bus_id>/modalias and is of the
formatccw:t<cu_type>m<cu_model> or
ccw:t<cu_type>m<cu_model>dt<dev_type>dm<dev_model>

Hotplug events can be used, for example, for:

v Automatically setting devices online as they appear
v Automatically loading driver modules for which devices have appeared

PCI Express support

The Peripheral Component Interconnect Express (PCIe) device driver provides
support of RDMA over Converged Ethernet (RoCE).

PCIe functions are seen by Linux as devices, hence devices is used here
synonymously. You can assign PCIe devices to LPARs in the IOCDS.

Setting up the PCIe support

Configure the PCIe support through the pci= kernel parameter.

PCIe devices are automatically configured during the system boot process. In
contrast to most z Systems devices, all PCIe devices that are in a configured state
are automatically set online. PCIe devices that are in stand-by state are not
automatically enabled.

Scanning of PCIe devices is enabled by default. To disable use of PCI devices, set
the kernel command line parameter pci=off.

Chapter 2. Devices in sysfs 19

 PCI kernel parameter syntax

pci=on

pci=off

where:
off
disables automatic scanning of PCIe devices.
on enables automatic scanning of PCIe devices (default).

Example

The following kernel parameter enables automatic scanning of PCIe devices.

pci=on

Using PCIe hotplug

Use PCIe hotplug to change the availability of a shared PCIe device.

About this task

Only one LPAR can access a PCIe device. Other LPARs can be candidates for
access. Use the HMC or SE to define which LPAR is connected and which LPARs
are on the candidate list. A PCIe device that is defined, but not yet used, is shown
as a PCIe slot in Linux.

On Linux, you use the power sysfs attribute of a PCIe slot to connect the device to
the LPAR where Linux runs. While a PCIe device is connected to one LPAR, it is in
the reserved state for all other LPARs that are in the candidates list. A reserved
PCIe device is invisible to the operating system. The slot is removed from sysfs.

Procedure

The power attribute of a slot contains 0 if a PCIe device is in stand-by state, or 1 if

the device is configured and usable.
1. Locate the slot for the card you want to work with. To locate the slot, read the
function_id attribute of the PCIe device from sysfs. For example, to read the
/sys/bus/pci/devices/0000:00:00.0/function_id issue:
# cat /sys/bus/pci/devices/0000:00:00.0/function_id
0x00000011

where 00000011 is the slot. Alternatively, you can use the lspci -v command to
find the slot.
2. Write the value that you want to the power attribute:
v Write 1 to power to connect the PCIe device to the LPAR in which your Linux
instance is running. Linux automatically scans the device, registers it, and
brings it online. For example:
echo 1 > /sys/bus/pci/slots/00000011/power

20 Device Drivers, Features, and Commands on SLES 12 SP2

 v Write 0 to power to stop using the PCIe device. The device state changes to
stand-by. The PCIe device is set offline automatically. For example:
echo 0 > /sys/bus/pci/slots/00000011/power
A PCIe device in standby is also in the standby state to all other LPARs in
the candidates list. A standby PCIe device appears as a slot, but without a
PCIe device.

Recovering a PCIe device

Use the recover sysfs attribute to recover a PCIe device.

About this task

A message is displayed when a PCIe device enters the error state. It is not possible
to automatically relieve the PCIe device from this state.

Procedure
1. Find the PCIe device directory in sysfs. PCIe device directories are of the form
/sys/devices/pci<dev> where <dev> is the device ID. For example:
/sys/devices/pci0000:00/0000:00:00.0/.
2. Write 1 to the recover attribute of the PCIe device. For example:
# echo 1 > /sys/devices/pci0000:00/0000:00:00.0/recover

After a successful recovery, the PCI device is de-registered and reprobed.

Displaying PCIe information

To display information about PCIe devices, read the attributes of the devices in
sysfs.

About this task

The sysfs representation of a PCIe device or slot is a directory of the form
/sys/devices/pci<device_bus_id>/<device_bus_id>, where <device_bus_id> is the
bus ID of the PCIe device. This sysfs directory contains a number of attributes with
information about the PCIe device.
Table 4. Attributes with PCIe device information
Attribute Explanation
function_handle Eight-character, hexadecimal PCI-function (device) handle.

This attribute is read-only.

function_id Eight-character, hexadecimal PCI-function (device) ID. The ID
identifies the PCIe device within the processor configuration.

This attribute is read-only.

pchid Four-character, hexadecimal, physical channel ID. Specifies the
slot of the PCIe adapter in the I/O drawer. Thus identifies the
adapter that provides the device.

This attribute is read-only.

pfgid Two-character, hexadecimal, physical function group ID.

This attribute is read-only.

Chapter 2. Devices in sysfs 21

| Reading statistics for a PCIe device
| Use the statistics attribute file to see measurement data for a PCIe device.

| About this task

| All PCIe devices collect measurement data by default. You can read the data in a
| sysfs attribute file in the debug file system, by default mounted at
| /sys/kernel/debug.

| You can turn data collection on and off. To switch off measurement data collecting
| for the current session, write "0" to the statistics attribute. To enable data
| collection again, write "1" to the statistics attribute.

| Example

| To read measurement data for a (RoCE) function named 0000:00:00.0 use:

| # cat /sys/kernel/debug/pci/0000:00:00.0/statistics

| The statistics attribute file might look similar to this example:

| FMB @ 0000000078cd8000
| Update interval: 4000 ms
| Samples: 14373
| Last update TOD: cefa44fa50006378
| Load operations: 1002780
| Store operations: 1950622
| Store block operations: 0
| Refresh operations: 0
| Received bytes: 0
| Received packets: 0
| Transmitted bytes: 0
| Transmitted packets: 0
| Allocated pages: 9104
| Mapped pages: 16633
| Unmapped pages: 2337

22 Device Drivers, Features, and Commands on SLES 12 SP2

Chapter 3. Kernel and module parameters
Kernel and module parameters are used to configure the kernel and kernel
modules.

Individual kernel parameters or module parameters are single keywords, or

keyword-value pairs of the form keyword=<value> with no blank. Blanks separate
consecutive parameters.

Kernel parameters and module parameters are encoded as strings of ASCII

characters.

Use kernel parameters to configure the base kernel and any optional kernel parts
that have been compiled into the kernel image. Use module parameters to configure
separate kernel modules. Do not confuse kernel and module parameters. Although
a module parameter can have the same syntax as a related kernel parameter,
kernel and module parameters are specified and processed differently.

Kernel parameters
Use kernel parameters to configure the base kernel and all modules that have been
compiled into the kernel.

Where possible, this document describes kernel parameters with the device driver
or feature to which they apply. Kernel parameters that apply to the base kernel or
cannot be attributed to a particular device driver or feature are described in
Chapter 54, Selected kernel parameters, on page 669. You can also find
descriptions for most of the kernel parameters in Documentation/kernel-
parameters.txt in the Linux source tree.

Specifying kernel parameters

You can use several interfaces to specify kernel parameters.
v Including kernel parameters in a boot configuration
v Adding kernel parameters when booting Linux
v z/VM reader only: Using a kernel parameter file

Avoid parameters that break GRUB 2

This section applies to all interfaces for specifying kernel parameters, except the
kernel parameter file that you can use when booting from the z/VM reader.

During the boot process, first the auxiliary kernel and GRUB 2 are started.
GRUB 2 then proceeds to start the target SUSE Linux Enterprise Server 12 SP2
kernel (see Figure 13 on page 53).

The auxiliary kernel and the target SUSE Linux Enterprise Server 12 SP2 kernel use
the same set of kernel parameters. Be cautious when making changes to the
parameters in the boot configuration.
v New or changed parameters might adversely affect the auxiliary kernel.
v Replacing the entire kernel parameter line eliminates parameters that are
required by the auxiliary kernel.

Copyright IBM Corp. 2000, 2016 23

 Including kernel parameters in a boot configuration
Use GRUB 2 to create or modify boot configurations for SUSE Linux Enterprise
Server 12 SP2 for z Systems.

See SUSE Linux Enterprise Server 12 SP2 Administration Guide about how to specify
kernel parameters with GRUB 2.

Adding kernel parameters when booting Linux

Depending on your platform, boot medium, and boot configuration, you can
provide kernel parameters when you start the boot process.

Note:
v Kernel parameters that you add when booting Linux are not persistent. Such
parameters enter the default reboot configuration, but are omitted after a regular
shutdown. To define a permanent set of kernel parameters for a Linux instance,
include these parameters in the boot configuration.
v Kernel parameters that you add when booting might interfere with parameters
that SUSE Linux Enterprise Server 12 SP2 sets for you. Read /proc/cmdline to
find out which parameters were used to start a running Linux instance.

If it is displayed, you can specify kernel parameters on the interactive GRUB 2

menu. See SUSE Linux Enterprise Server 12 SP2 Administration Guide for more
information.

Specifying kernel parameters before GRUB 2 takes control

Important: The preferred method for specifying kernel parameters when booting is
through the GRUB 2 interactive boot menu.

You might be able to use one or more of these interfaces for specifying kernel
parameters:
z/VM guest virtual machine with a CCW boot device
When booting Linux in a z/VM guest virtual machine from a CCW boot
device, you can use the PARM parameter of the IPL command to specify
kernel parameters. CCW boot devices include DASD and the z/VM reader.
For details, see the subsection of Booting Linux in a z/VM guest virtual
machine on page 56 that applies to your boot device.
z/VM guest virtual machine with a SCSI boot device
When booting Linux in a z/VM guest virtual machine from a SCSI boot
device, you can use the SET LOADDEV command with the SCPDATA
option to specify kernel parameters. See Booting from a SCSI device on
page 57 for details.
LPAR mode with a SCSI boot device
When booting Linux in LPAR mode from a SCSI boot device, you can
specify kernel parameters in the Operating system specific load
parameters field on the HMC Load panel. See Figure 17 on page 64.

Kernel parameters as entered from a CMS or CP session are interpreted as

lowercase on Linux.

How kernel parameters from different sources are combined

If kernel parameters are specified in a combination of methods, they are
concatenated in a specific order.

24 Device Drivers, Features, and Commands on SLES 12 SP2

 1. Kernel parameters that have been included in the boot configuration with
GRUB 2.
2. Kernel parameters that are specified with the GRUB 2 interactive boot menu.
The combined parameters that are specified in the boot configuration and
through the GRUB 2 interactive boot menu must not exceed 895 characters.
3. Kernel parameters that you specify through the HMC or through z/VM
interfaces (see Adding kernel parameters when booting Linux on page 24).
For DASD boot devices you can specify up to 64 characters (z/VM only); for
SCSI boot devices you can specify up to 3452 characters.

In total, the combined kernel parameter string that is passed to the Linux kernel
for booting can be up to 4096 characters.

Multiple specifications for the same parameter

For some kernel parameters, multiple instances in the kernel parameter string are
treated cumulatively. For example, multiple specifications for cio_ignore= are all
processed and combined.

Conflicting kernel parameters

If the kernel parameter string contains kernel parameters with mutually exclusive
settings, the last specification in the string overrides preceding ones. Thus, you can
specify a kernel parameter when booting to override an unwanted setting in the
boot configuration.

Examples:
v If the kernel parameters in your boot configuration include possible_cpus=8 but
you specify possible_cpus=2 when booting, Linux uses possible_cpus=2.
v If the kernel parameters in your boot configuration include resume=/dev/dasda2
to specify a disk from which to resume the Linux instance when it has been
suspended, you can circumvent the resume process by specifying noresume when
booting.

Parameters other than kernel parameters

Parameters on the kernel parameter string that the kernel does not recognize as
kernel parameters are ignored by the kernel and made available to user space
programs. How multiple specifications and conflicts are resolved for such
parameters depends on the program that evaluates them.

Using a kernel parameter file with the z/VM reader.

You can use a kernel parameter file for booting Linux from the z/VM reader.

See Booting from the z/VM reader on page 59 about using a kernel parameter
file in the z/VM reader.

Examples for kernel parameters

Typical parameters that are used for booting SUSE Linux Enterprise Server 12 SP2
configure the console, kdump, and the suspend and resume function.
conmode=<mode>, condev=<cuu>, console=<name>
to set up the Linux console. See Console kernel parameter syntax on page 38
for details.

Chapter 3. Kernel and module parameters 25

 crashkernel=<area>
reserves a memory area for a kdump kernel and its initial RAM disk (initrd).
resume=<partition>, noresume, no_console_suspend
to configure suspend-and-resume support (see Chapter 6, Suspending and
resuming Linux, on page 75).

See Chapter 54, Selected kernel parameters, on page 669 for more examples of
kernel parameters.

Displaying the current kernel parameter line

Read /proc/cmdline to find out with which kernel parameters a running Linux
instance was booted.

About this task

Apart from kernel parameters, which are evaluated by the Linux kernel, the kernel
parameter line can contain parameters that are evaluated by user space programs,
for example, modprobe.

See also Displaying current IPL parameters on page 68 about displaying the
parameters that were used to IPL and boot the running Linux instance.

Example:
# cat /proc/cmdline
root=UUID=93722c3c-85ed-4537-ac68-8528a5bdef0c hvc_iucv=8 TERM=dumb OsaMedium=eth crashkernel=204M-:102M

Kernel parameters for rebooting

When rebooting, you can use the current kernel parameters or an alternative set of
kernel parameters.

By default, Linux uses the current kernel parameters for rebooting. See Rebooting
from an alternative source on page 70 about setting up Linux to use different
kernel parameters for re-IPL and the associated reboot.

Module parameters
Use module parameters to configure kernel modules that are compiled as separate
modules that can be loaded by the kernel.

Separate kernel modules must be loaded before they can be used. Many modules
are loaded automatically by SUSE Linux Enterprise Server 12 SP2 when they are
needed and you use YaST to specify the module parameters.

To keep the module parameters in the context of the device driver or feature
module to which they apply, this information describes module parameters as part
of the syntax you would use to load the module with modprobe.

To find the separate kernel modules for SUSE Linux Enterprise Server 12 SP2, list
the contents of the subdirectories of /lib/modules/<kernel-release> in the Linux
file system. In the path, <kernel-release> denotes the kernel level. You can query
the value for <kernel-release> with uname -r.

26 Device Drivers, Features, and Commands on SLES 12 SP2

Specifying module parameters
How to specify module parameters depends on how the module is loaded, for
example, with YaST or from the command line.

YaST is the preferred tool for specifying module parameters for SUSE Linux
Enterprise Server 12 SP2. You can use alternative means to specify module
parameters, for example, if a particular setting is not supported by YaST. Avoid
specifying the same parameter through multiple means.

Specifying module parameters with modprobe

If you load a module explicitly with a modprobe command, you can specify the
module parameters as command arguments.

Module parameters that are specified as arguments to modprobe are effective only
until the module is unloaded.

Note: Parameters that you specify as command arguments might interfere with
parameters that SUSE Linux Enterprise Server 12 SP2 sets for you.

Module parameters on the kernel parameter line

Parameters that the kernel does not recognize as kernel parameters are ignored by
the kernel and made available to user space programs.

One of these programs is modprobe, which SUSE Linux Enterprise Server 12 SP2
uses to load modules for you. modprobe interprets module parameters that are
specified on the kernel parameter line if they are qualified with a leading module
prefix and a dot.

For example, you can include a specification with cmm.sender=TESTID on the kernel
parameter line. modprobe evaluates this specification as the sender= module
parameter when it loads the cmm module.

Including module parameters in a boot configuration

Module parameters for modules that are required early during the boot process
must be included in the boot configuration.

About this task

SUSE Linux Enterprise Server 12 SP2 uses an initial RAM disk when booting.

Procedure

Perform these steps to provide module parameters for modules that are included
in the initial RAM disk:
1. Make your configuration changes with YaST or an alternative method.
2. If YaST does not perform this task for you, run dracut -f to create an initial
RAM disk that includes the module parameters.

Displaying information about module parameters

Loaded modules can export module parameter settings to sysfs.

The parameters for modules are available as sysfs attributes of the form:
/sys/module/<module_name>/parameters/<parameter_name>

Chapter 3. Kernel and module parameters 27

 Before you begin

You can display information about modules that fulfill these prerequisites:
v The module must be loaded.
v The module must export the parameters to sysfs.

Procedure
To find and display the parameters for a module, follow these steps:
1. Optional: Confirm that the module of interest is loaded by issuing a command
of this form:
# lsmod | grep <module_name>

where <module_name> is the name of the module.

2. Optional: Get an overview of the parameters for the module by issuing a
command of this form:
# modinfo <module_name>

3. To check if a module exports settings to sysfs, try listing the module

parameters. Issue a command of the form:
# ls /sys/module/<module_name>/parameters

4. If the previous command listed parameters, you can display the value for the
parameter you are interested in. Issue a command of the form:
# cat /sys/module/<module_name>/parameters/<parameter_name>

Example
v To list the module parameters for the ap module, issue:
# ls /sys/module/ap/parameters
domain
...

v To display the value of the domain parameter, issue:

# cat /sys/module/ap/parameters/domain
1

28 Device Drivers, Features, and Commands on SLES 12 SP2

Part 2. Booting and shutdown
Chapter 4. Console device drivers . . . . . . 31 What you should know about suspend and resume 75
Console features . . . . . . . . . . . . . 32 Setting up Linux for suspend and resume . . . . 77
What you should know about the console device Suspending a Linux instance . . . . . . . . 79
drivers . . . . . . . . . . . . . . . . 33 Resuming a suspended Linux instance . . . . . 79
Setting up the console device drivers . . . . . . 37
Working with Linux terminals . . . . . . . . 43 Chapter 7. Shutdown actions . . . . . . . . 81
The shutdown configuration in sysfs . . . . . . 82
Chapter 5. Booting Linux . . . . . . . . . 53 | Configuring z/VM CP commands as a shutdown
IPL and booting . . . . . . . . . . . . . 53 | action . . . . . . . . . . . . . . . . 83
Control point and boot medium . . . . . . . 54
Boot data . . . . . . . . . . . . . . . 55 Chapter 8. Remotely controlling virtual hardware
Booting Linux in a z/VM guest virtual machine . . 56 - snipl . . . . . . . . . . . . . . . . 85
Booting Linux in LPAR mode . . . . . . . . 61 LPAR mode . . . . . . . . . . . . . . 85
Specifying GRUB 2 parameters . . . . . . . . 68 z/VM mode . . . . . . . . . . . . . . 95
Displaying current IPL parameters. . . . . . . 68 The snipl configuration file . . . . . . . . . 99
Rebooting from an alternative source . . . . . . 70 Connection errors and return codes . . . . . . 103
STONITH support (snipl for STONITH) . . . . 104
Chapter 6. Suspending and resuming Linux . . 75

These device drivers and features are useful for booting and shutting down SUSE
Linux Enterprise Server 12 SP2.

Newest version

You can find the newest version of this publication on IBM Knowledge Center at
www.ibm.com/support/knowledgecenter/linuxonibm/liaaf/lnz_r_suse.html

Restrictions

For prerequisites and restrictions see the z Systems architecture specific

information in the SUSE Linux Enterprise Server 12 SP2 release notes at
www.suse.com/releasenotes

Copyright IBM Corp. 2000, 2016 29

30 Device Drivers, Features, and Commands on SLES 12 SP2
Chapter 4. Console device drivers
The Linux on z Systems console device drivers support terminal devices for basic
Linux control, for example, for booting Linux, for troubleshooting, and for
displaying Linux kernel messages.

The only interface to a Linux instance in an LPAR before the boot process is
completed is the Hardware Management Console (HMC), see Figure 6. After the
boot process has completed, you typically use a network connection to access
Linux through a user login, for example, in an ssh session. The possible
connections depend on the configuration of your particular Linux instance.

Selected
mainframe system

Selected LPAR
Operating System Messages

Integrated ASCII Console

Figure 6. Hardware Management Console

With Linux on z/VM, you typically use a 3270 terminal or terminal emulator to
log in to z/VM first. From the 3270 terminal, you IPL the Linux boot device.
Again, after boot you typically use a network connection to access Linux through a
user login rather than a 3270 terminal.

Copyright IBM Corp. 2000, 2016 31

Console features
The console device drivers support several types of terminal devices.
HMC applets
You can use two applets.
Operating System Messages
This applet provides a line-mode terminal. See Figure 7 for an
example.
Integrated ASCII Console
This applet provides a full-screen mode terminal.

These HMC applets are accessed through the service-call logical processor
(SCLP) console interface.
3270 terminal
This terminal can be based on physical 3270 terminal hardware or a 3270
terminal emulation.
z/VM can use the 3270 terminal as a 3270 device or perform a protocol
translation and use it as a 3215 device. As a 3215 device it is a line-mode
terminal for the United States code page (037).
The iucvconn program
You can use the iucvconn program from Linux on z/VM to access terminal
devices on other Linux instances that run as guests of the same z/VM
system.
See How to Set up a Terminal Server Environment on z/VM, SC34-2596 for
information about the iucvconn program.

The console device drivers support these terminals as output devices for Linux
kernel messages.

Figure 7. Linux kernel messages on the HMC Operating System Messages applet

32 Device Drivers, Features, and Commands on SLES 12 SP2

What you should know about the console device drivers
The console concepts, naming conventions, and terminology overview help you to
understand the tasks you might have to perform with console and terminal
devices.

Console terminology
Terminal and console have special meanings in Linux.
Linux terminal
An input/output device through which users interact with Linux and
Linux applications. Login programs and shells typically run on Linux
terminals and provide access to the Linux system.
Linux console
An output-only device to which the Linux kernel can write kernel
messages. Linux console devices can be associated with Linux terminal
devices. Thus, console output can be displayed on a Linux terminal.
Mainframe terminal
Any device that gives a user access to operating systems and applications
that run on a mainframe. A mainframe terminal can be a physical device
such as a 3270 terminal hardware that is linked to the mainframe through
a controller. It can also be a terminal emulator on a workstation that is
connected through a network. For example, you access z/OS through a
mainframe terminal.
Hardware Management Console (HMC)
A device that gives a system programmer control over z Systems hardware
resources, for example, LPARs. The HMC is a web application on a web
server that is connected to the support element (SE). The HMC can be
accessed from the SE but more commonly is accessed from a workstation
within a secure network.

On the mainframe, the Linux console and Linux terminals can both be connected
to a mainframe terminal.

Before you have a Linux terminal - boot menus

Do not confuse boot menus with a Linux terminal.

Depending on your setup, a zipl boot menu, a GRUB 2 boot menu, or both might
be displayed when you perform an IPL.
zipl boot menu
The zipl boot menu is part of the boot loader for the auxiliary kernel that
provides GRUB 2 and is displayed before a Linux terminal is set up.
GRUB 2 boot menu
GRUB 2 might display a menu for selecting the target kernel to be booted.
For more information about GRUB 2, see SUSE Linux Enterprise Server 12
SP2 Administration Guide.

Device and console names

Each terminal device driver can provide a single console device.

Table 5 on page 34 lists the terminal device drivers with the corresponding device
names and console names.

Chapter 4. Console device drivers 33

 Table 5. Device and console names
Device driver Device name Console name
SCLP line-mode terminal device driver sclp_line0 ttyS0
SCLP VT220 terminal device driver ttysclp0 ttyS1
3215 line-mode terminal device driver ttyS0 ttyS0
3270 terminal device driver 3270/tty1 to tty3270
3270/tty<N>
z/VM IUCV HVC device driver hvc0 to hvc7 hvc0

As shown in Table 5, the console with name ttyS0 can be provided either by the
SCLP console device driver or by the 3215 line-mode terminal device driver. The
system environment and settings determine which device driver provides ttyS0.
For details, see the information about the conmode kernel parameter in Console
kernel parameter syntax on page 38.

Of the terminal devices that are provided by the z/VM IUCV HVC device driver
only hvc0 is associated with a console.

Of the 3270/tty<N> terminal devices only 3270/tty1 is associated with a console.

Device nodes
Applications, for example, login programs, access terminal devices by device
nodes.

For example, with the default conmode settings, udev creates the following device
nodes:
Table 6. Device nodes created by udev
Device driver On LPAR On z/VM
SCLP line-mode terminal device driver /dev/sclp_line0 n/a
SCLP VT220 terminal device driver /dev/ttysclp0 /dev/ttysclp0
3215 line-mode terminal device driver n/a /dev/ttyS0
3270 terminal device driver /dev/3270/tty1 to /dev/3270/tty1 to
/dev/3270/tty<N> /dev/3270/tty<N>
z/VM IUCV HVC device driver n/a /dev/hvc0 to /dev/hvc7

Terminal modes
The Linux terminals that are provided by the console device drivers include
line-mode terminals, block-mode terminals, and full-screen mode terminals.

On a full-screen mode terminal, pressing any key immediately results in data being
sent to the terminal. Also, terminal output can be positioned anywhere on the
screen. This feature facilitates advanced interactive capability for terminal-based
applications like the vi editor.

On a line-mode terminal, the user first types a full line, and then presses Enter to
indicate that the line is complete. The device driver then issues a read to get the
completed line, adds a new line, and hands over the input to the generic TTY
routines.

34 Device Drivers, Features, and Commands on SLES 12 SP2

 The terminal that is provided by the 3270 terminal device driver is a traditional
IBM mainframe block-mode terminal. Block-mode terminals provide full-screen
output support and users can type input in predefined fields on the screen. Other
than on typical full-screen mode terminals, no input is passed on until the user
presses Enter. The terminal that is provided by the 3270 terminal device driver
provides limited support for full-screen applications. For example, the ned editor is
supported, but not vi.

Table 7 summarizes when to expect which terminal mode.

Table 7. Terminal modes
Accessed through Environment Device driver Mode
Operating System Messages LPAR SCLP line-mode terminal Line mode
applet on the HMC device driver
z/VM emulation of the HMC z/VM SCLP line-mode terminal Line mode
Operating System Messages device driver
applet
Integrated ASCII Console z/VM or LPAR SCLP VT220 terminal device Full-screen mode
applet on the HMC driver
3270 terminal hardware or z/VM with 3215 line-mode terminal Line mode
emulation CONMODE=3215 device driver
3270 terminal hardware or z/VM with 3270 terminal device driver Block mode
emulation CONMODE=3270
iucvconn program z/VM z/VM IUCV HVC device Full-screen mode
driver

The 3270 terminal device driver provides three different views. See Switching the
views of the 3270 terminal device driver on page 45 for details.

How console devices are accessed

How you can access console devices depends on your environment.

The diagrams in the following sections omit device drivers that are not relevant for
the particular access scenario.

Using the HMC for Linux in an LPAR

You can use two applets on the HMC to access terminal devices on Linux instances
that run directly in an LPAR.

Network
HMC Linux
Operating System SCLP line-mode
ttyS0 terminal
Workstation Messages device driver

Browser Integrated ttyS1 SCLP VT220

ASCII Console terminal device driver

Figure 8. Accessing terminal devices on Linux in an LPAR from the HMC

Chapter 4. Console device drivers 35

 The Operating System Messages applet accesses the device that is provided by the
SCLP line-mode terminal device driver. The Integrated ASCII console applet
accesses the device that is provided by the SCLP VT220 terminal device driver.

Using the HMC for Linux on z/VM

You can use the HMC Integrated ASCII Console applet to access terminal devices
on Linux instances that run as z/VM guests.

While the ASCII system console is attached to the z/VM guest virtual machine
where the Linux instance runs, you can access the ttyS1 terminal device from the
HMC Integrated ASCII Console applet.

Workstation

Browser
Network

HMC z/VM
Operating System Linux
Messages
ATTACH SYSASCII
Integrated ttyS1 SCLP VT220
ASCII Console terminal device driver

Figure 9. Accessing terminal devices from the HMC for Linux on z/VM

Use the CP ATTACH SYSASCII command to attach the ASCII system console to
your z/VM guest virtual machine.

Using a 3270 terminal emulation

For Linux on z/VM, you can use 3270 terminal emulation to access a console
device.

Figure 10 illustrates how z/VM can handle the 3270 communication.

z/VM
Network Linux
protocol

Workstation
3215

CONMODE=3215 ttyS0 3215 line-mode

3270 terminal device driver
terminal
emulation
protocol

3270 terminal
3270

CONMODE=3270 tty3270 device driver

VINPUT ttyS0 SCLP line-mode

terminal device driver

Figure 10. Accessing terminal devices from a 3270 device

Note: Figure 10 shows two console devices with the name ttyS0. Only one of these
devices can be present at any one time.

36 Device Drivers, Features, and Commands on SLES 12 SP2

 CONMODE=3215
translates between the 3270 protocol and the 3215 protocol and connects the
3270 terminal emulation to the 3215 line-mode terminal device driver in the
Linux kernel.
CONMODE=3270
connects the 3270 terminal emulation to the 3270 terminal device driver in the
Linux kernel.
VINPUT
is a z/VM CP command that directs input to the ttyS0 device provided by the
SCLP line-mode terminal device driver. In a default z/VM environment, ttyS0
is provided by the 3215 line-mode terminal device driver. You can use the
conmode kernel parameter to make the SCLP line-mode terminal device driver
provide ttyS0 (see Console kernel parameter syntax on page 38).

The terminal device drivers continue to support 3270 terminal hardware, which, if
available at your installation, can be used instead of a 3270 terminal emulation.

Using iucvconn on Linux on z/VM

On Linux on z/VM, you can access the terminal devices that are provided by the
z/VM IUCV Hypervisor Console (HVC) device driver.

z/VM
Network
Linux Linux

shell
z/VM IUCV HVC device driver
Workstation hvc7
Terminal iucvconn hvc1
session hvc0

IUCV

Figure 11. Accessing terminal devices from a peer Linux instance

As illustrated in Figure 11, you access the devices with the iucvconn program from
another Linux instance. Both Linux instances are guests of the same z/VM system.
IUCV provides the communication between the two Linux instances. With this
setup, you can access terminal devices on Linux instances with no external
network connection.

Note: Of the terminal devices that are provided by the z/VM IUCV HVC device
driver only hvc0 can be activated to receive Linux kernel messages.

Setting up the console device drivers

You configure the console device drivers through kernel parameters. You also
might have to enable user logins on terminals and ensure that the TERM
environment variable has a suitable value.

Chapter 4. Console device drivers 37

 Console kernel parameter syntax
Use the console kernel parameters to configure the console device drivers,
line-mode terminals, and HVC terminal devices.

The sclp_con_pages= and sclp_con_drop= parameters apply only to the SCLP

line-mode terminal device driver and to the SCLP VT220 terminal device driver.

The hvc_iucv= and hvc_iucv_allow= kernel parameters apply only to terminal

devices that are provided by the z/VM IUCV HVC device driver.

Console kernel parameter syntax

console=<console_name>
conmode= hwc
sclp
3215
3270

sclp_con_drop=0 sclp_con_pages=6

sclp_con_drop=1 sclp_con_pages=<n>

hvc_iucv=1

hvc_iucv=<number_of_devices> ,

hvc_iucv_allow= <z/VM user ID>

Note: If you specify both the conmode= and the console= parameter, specify them
in the sequence that is shown, conmode= first.

where:
conmode
specifies which one of the line-mode or block-mode terminal devices is present
and provided by which device driver.
A Linux kernel might include multiple console device drivers that can provide
a line-mode terminal:
v SCLP line-mode terminal device driver
v 3215 line-mode terminal device driver
v 3270 terminal device driver
On a running Linux instance, only one of these device drivers can provide a
device. Table 8 shows how the device driver that is used by default depends
on the environment.
Table 8. Default device driver for the line-mode terminal device
Mode Default
LPAR SCLP line-mode terminal device driver

38 Device Drivers, Features, and Commands on SLES 12 SP2

Table 8. Default device driver for the line-mode terminal device (continued)
Mode Default
z/VM 3215 line-mode terminal device driver or 3270 terminal device
driver, depending on the z/VM guest's console settings (the
CONMODE field in the output of #CP QUERY TERMINAL).

If the device driver you specify with the conmode= kernel

parameter contradicts the CONMODE z/VM setting, z/VM is
reconfigured to match the specification for the kernel parameter.

You can use the conmode parameter to override the default.

sclp or hwc
specifies the SCLP line-mode terminal device driver.
You need this specification if you want to use the z/VM CP VINPUT
command (Using a z/VM emulation of the HMC Operating System
Messages applet on page 49).
3270
specifies the 3270 device driver.
3215
specifies the 3215 device driver.
console=<console_name>
specifies the console devices to be activated to receive Linux kernel messages.
If present, ttyS0 is always activated to receive Linux kernel messages and, by
default, it is also the preferred console.
The preferred console is used as an initial terminal device, beginning at the
stage of the boot process when the initialization procedures run. Messages that
are issued by programs that are run at this stage are therefore only displayed
on the preferred console. Multiple terminal devices can be activated to receive
Linux kernel messages but only one of the activated terminal devices can be
the preferred console.
If you specify conmode=3270, there is no console with name ttyS0.
If you want console devices other than ttyS0 to be activated to receive Linux
kernel messages, specify a console statement for each of these other devices.
The last console statement designates the preferred console.
If you specify one or more console parameters and you want to keep ttyS0 as
the preferred console, add a console parameter for ttyS0 as the last console
parameter. Otherwise, you do not need a console parameter for ttyS0.
<console_name> is the console name that is associated with the terminal device
to be activated to receive Linux kernel messages. Of the terminal devices that
are provided by the z/VM IUCV HVC device driver only hvc0 can be
activated. Specify the console names as shown in Table 5 on page 34.
sclp_con_drop
governs the behavior of the SCLP line-mode and VT220 terminal device driver
if either of them runs out of output buffer pages. The trade-off is between
slowing down Linux and losing console output. Possible values are 0 (default)
and 1.
0 assures complete console output by pausing until used output buffer pages
are written to an output device and can be reused without loss.

Chapter 4. Console device drivers 39

 1 avoids system pauses by overwriting used output buffer pages, even if the
content was never written to an output device.

You can use the sclp_con_pages= parameter to set the number of output
buffers.
sclp_con_pages=<n>
specifies the number of 4-KB memory pages to be used as the output buffer for
the SCLP line-mode and VT220 terminals. Depending on the line length, each
output buffer can hold multiple lines. Use many buffer pages for a kernel with
frequent phases of producing console output faster than it can be written to the
output device.
Depending on the setting for the sclp_con_drop=, running out of pages can
slow down Linux or cause it to lose console output.
The value is a positive integer. The default is 6.
hvc_iucv=<number_of_devices>
specifies the number of terminal devices that are provided by the z/VM IUCV
HVC device driver. <number_of_devices> is an integer in the range 0 - 8. Specify
0 to switch off the z/VM IUCV HVC device driver.
hvc_iucv_allow=<z/VM user ID>,<z/VM user ID>, ...
specifies an initial list of z/VM guest virtual machines that are allowed to
connect to HVC terminal devices. If this parameter is omitted, any z/VM guest
virtual machine that is authorized to establish the required IUCV connection is
also allowed to connect. On the running system, you can change this list with
the chiucvallow command. See How to Set up a Terminal Server Environment on
z/VM, SC34-2596 for more information.

Examples
v To activate ttyS1 in addition to ttyS0, and to use ttyS1 as the preferred console,
add the following specification to the kernel command line:
console=ttyS1
v To activate ttyS1 in addition to ttyS0, and to keep ttyS0 as the preferred
console, add the following specification to the kernel command line:
console=ttyS1 console=ttyS0
v To use an emulated HMC Operating System Messages applet in a z/VM
environment specify:
conmode=sclp
v To activate hvc0 in addition to ttyS0, use hvc0 as the preferred console,
configure the z/VM IUCV HVC device driver to provide four devices, and limit
the z/VM guest virtual machines that can connect to HVC terminal devices to
lxtserv1 and lxtserv2, add the following specification to the kernel command
line:
console=hvc0 hvc_iucv=4 hvc_iucv_allow=lxtserv1,lxtserv2
v The following specification selects the SCLP line-mode terminal and configures
32 4-KB pages (128 KB) for the output buffer. If buffer pages run out, the SCLP
line-mode terminal device driver does not wait for pages to be written to an
output device. Instead of pausing, it reuses output buffer pages at the expense of
losing content.
console=sclp sclp_con_pages=32 sclp_con_drop=1

40 Device Drivers, Features, and Commands on SLES 12 SP2

Setting up a z/VM guest virtual machine for iucvconn
Because the iucvconn program uses z/VM IUCV to access Linux, you must set up
your z/VM guest virtual machine for IUCV.

See Setting up your z/VM guest virtual machine for IUCV on page 316 for
details about setting up the z/VM guest virtual machine.

For information about accessing Linux through the iucvtty program rather than
through the z/VM IUCV HVC device driver, see How to Set up a Terminal Server
Environment on z/VM, SC34-2596 or the man pages for the iucvtty and iucvconn
commands.

Setting up a line-mode terminal

The line-mode terminals are primarily intended for booting Linux.

The preferred user access to a running SUSE Linux Enterprise Server 12 SP2
instance is through a user login that runs, for example, in an ssh session. See
Terminal modes on page 34 for information about the available line-mode
terminals.

Tip: If the terminal does not provide the expected output, ensure that dumb is
assigned to the TERM environment variable. For example, enter the following
command on the bash shell:
# export TERM=dumb

Setting up a full-screen mode terminal

The full-screen terminal can be used for full-screen text editors, such as vi, and
terminal-based full-screen system administration tools.

See Terminal modes on page 34 for information about the available full-screen
mode terminals.

Tip: If the terminal does not provide the expected output, ensure that linux is
assigned to the TERM environment variable. For example, enter the following
command on the bash shell:
# export TERM=linux

Setting up a terminal provided by the 3270 terminal device

driver
The terminal that is provided by the 3270 terminal device driver is not a line-mode
terminal, but it is also not a typical full-screen mode terminal.

The terminal provides limited support for full-screen applications. For example, the
ned editor is supported, but not vi.

Tip: If the terminal does not provide the expected output, ensure that linux is
assigned to the TERM environment variable. For example, enter the following
command on the bash shell:
# export TERM=linux

Chapter 4. Console device drivers 41

 Enabling user logins
Use systemd service units to enable terminals for user access.

About this task

You must explicitly enable user logins for the HVC terminals hvc1 to hvc7 and for
any dynamically attached virtual or real 3270 terminals. On other terminals that
are, typically, available in your environment, including hvc0 and 3270/tty1,
systemd automatically enables user logins for you.

Enabling user logins for 3270 terminals

Instantiate getty services for terminals to enable users access.

Procedure

Perform these steps to use a getty service for enabling user logins on any
dynamically added real or virtual 3270 terminals.
1. Enable the new getty service by issuing a command of this form:
# systemctl enable serial-getty@<terminal>.service

where <terminal> specifies one of the terminals 3270-tty<N> and <N> is an

integer greater than 1.

Note: You specify terminal 3270/tty<N> as 3270-tty<N>.

2. Optional: Start the new getty service by issuing a command of this form:
# systemctl start serial-getty@<terminal>.service

Results

At the next system start, systemd automatically starts the getty service for you.

Example

For 3270/tty2, issue:

# systemctl enable [email protected]
# systemctl start [email protected]

Preventing respawns for non-operational HVC terminals

If you enable user logins on a HVC terminal that is not available or not
operational, systemd keeps respawning the getty program.

About this task

If user logins are enabled on unavailable HVC terminals hvc1 to hvc7, systemd
might keep respawning the getty program. To be free to change the conditions that
affect the availability of these terminals, use the ttyrun service to enable user logins
for them. HVC terminals are operational only in a z/VM environment, and they
depend on the hvc_iucv= kernel parameter (see Console kernel parameter syntax
on page 38).

42 Device Drivers, Features, and Commands on SLES 12 SP2

 Any other unavailable terminals with enabled user login, including hvc0, do not
cause problems with systemd.

Procedure

Perform these steps to use a ttyrun service for enabling user logins on a terminal:
1. Enable the ttyrun service by issuing a command of this form:
# systemctl enable ttyrun-getty@hvc<n>.service

where hvc<n> specifies one of the terminals hvc1 to hvc7.

2. Optional: Start the new service by issuing a command of this form:
# systemctl start ttyrun-getty@hvc<n>.service

Results

At the next system start, systemd starts the ttyrun service for hvc<n>. The ttyrun
service starts a getty only if this terminal is available.

Example

For hvc1, issue:

# systemctl enable [email protected]
# systemctl start [email protected]

Setting up the code page for an x3270 emulation on Linux

For accessing z/VM from Linux through the x3270 terminal emulation, you must
add a number of settings to the .Xdefaults file to get the correct code translation.

Add these settings:

! X3270 keymap and charset settings for Linux
x3270.charset: us-intl
x3270.keymap: circumfix
x3270.keymap.circumfix: :<key>asciicircum: Key("^")\n

Working with Linux terminals

You might have to work with different types of Linux terminals, and use special
functions on these terminals.
v Using the terminal applets on the HMC on page 44
v Accessing terminal devices over z/VM IUCV on page 44
v Switching the views of the 3270 terminal device driver on page 45
v Setting a CCW terminal device online or offline on page 46
v Entering control and special characters on line-mode terminals on page 47
v Using the magic sysrequest feature on page 47
v Using a z/VM emulation of the HMC Operating System Messages applet on
page 49
v Using a 3270 terminal in 3215 mode on page 51

Chapter 4. Console device drivers 43

 Using the terminal applets on the HMC
You should be aware of some aspects of the line-mode and the full-screen mode
terminal when working with the corresponding applets on the HMC.

The following statements apply to both the line-mode terminal and the full-screen
mode terminal on the HMC:
v On an HMC, you can open each applet only once.
v Within an LPAR, there can be only one active terminal session for each applet,
even if multiple HMCs are used.
v A particular Linux instance supports only one active terminal session for each
applet.
v Security hint: Always end a terminal session by explicitly logging off (for
example, type exit and press Enter). Simply closing the applet leaves the
session active and the next user to open the applet resumes the existing session
without a logon.
v Slow performance of the HMC is often due to a busy console or increased
network traffic.

The following statements apply to the full-screen mode terminal only:

v Output that is written by Linux while the terminal window is closed is not
displayed. Therefore, a newly opened terminal window is always blank. For
most applications, like login or shell prompts, it is sufficient to press Enter to
obtain a new prompt.
v The terminal window shows only 24 lines and does not provide a scroll bar. To
scroll up, press Shift+PgUp; to scroll down, press Shift+PgDn.

Accessing terminal devices over z/VM IUCV

Use z/VM IUCV to access hypervisor console (HVC) terminal devices, which are
provided by the z/VM IUCV HVC device driver.

About this task

For information about accessing terminal devices that are provided by the iucvtty
program see How to Set up a Terminal Server Environment on z/VM, SC34-2596.

You access HVC terminal devices from a Linux instance where the iucvconn
program is installed. The Linux instance with the terminal device to be accessed
and the Linux instance with the iucvconn program must both run as guests of the
same z/VM system. The two guest virtual machines must be configured such that
IUCV communication is permitted between them.

Procedure

Perform these steps to access an HVC terminal device over z/VM IUCV:
1. Open a terminal session on the Linux instance where the iucvconn program is
installed.
2. Enter a command of this form:
# iucvconn <guest_ID> <terminal_ID>

where:

44 Device Drivers, Features, and Commands on SLES 12 SP2

 <guest_ID>
specifies the z/VM guest virtual machine on which the Linux instance with
the HVC terminal device to be accessed runs.
<terminal_ID>
specifies an identifier for the terminal device to be accessed. HVC terminal
device names are of the form hvcn where n is an integer in the range 0-7.
The corresponding terminal IDs are lnxhvcn.

Example: To access HVC terminal device hvc0 on a Linux instance that runs on
a z/VM guest virtual machine LXGUEST1, enter:
# iucvconn LXGUEST1 lnxhvc0

For more details and further parameters of the iucvconn command, see the
iucvconn man page or How to Set up a Terminal Server Environment on z/VM,
SC34-2596.
3. Press Enter to obtain a prompt.
Output that is written by Linux while the terminal window is closed, is not
displayed. Therefore, a newly opened terminal window is always blank. For
most applications, like login or shell prompts, it is sufficient to press Enter to
obtain a new prompt.

Security hint

Always end terminal sessions by explicitly logging off (for example, type exit and
press Enter). If logging off results in a new login prompt, press Control and
Underscore (Ctrl+_), then press D to close the login window. Simply closing the
terminal window for a hvc0 terminal device that was activated for Linux kernel
messages leaves the device active. The terminal session can then be reopened
without a login.

Switching the views of the 3270 terminal device driver

The 3270 terminal device driver provides three different views.

Use function key 3 (PF3) to switch between the views (see Figure 12).

Linux kernel
messages
view

PF3
Full-screen Terminal I/O
application view
view

Figure 12. Switching views of the 3270 terminal device driver

The Linux kernel messages view is available only if the terminal device is activated
for Linux kernel messages. The full-screen application view is available only if
there is an application that uses this view, for example, the ned editor.

Chapter 4. Console device drivers 45

 Be aware that the 3270 terminal provides only limited full-screen support. The
full-screen application view of the 3270 terminal is not intended for applications
that require vt220 capabilities. The application itself must create the 3270 data
stream.

For the Linux kernel messages view and the terminal I/O view, you can use the
PF7 key to scroll backward and the PF8 key to scroll forward. The scroll buffers are
fixed at four pages (16 KB) for the Linux kernel messages view and five pages
(20 KB) for the terminal I/O view. When the buffer is full and more terminal data
needs to be printed, the oldest lines are removed until there is enough room. The
number of lines in the history, therefore, vary. Scrolling in the full-screen
application view depends on the application.

You cannot issue z/VM CP commands from any of the three views that are
provided by the 3270 terminal device driver. If you want to issue CP commands,
use the PA1 key to switch to the CP READ mode.

Setting a CCW terminal device online or offline

The 3270 terminal device driver uses CCW devices and provides them as CCW
terminal devices.

About this task

This section applies to Linux on z/VM. A CCW terminal device can be:
v The tty3270 terminal device that can be activated for receiving Linux kernel
messages.
If this device exists, it comes online early during the Linux boot process. In a
default z/VM environment, the device number for this device is 0009. In sysfs, it
is represented as /sys/bus/ccw/drivers/3270/0.0.0009. You need not set this
device online and you must not set it offline.
v CCW terminal devices through which users can log in to Linux with the CP
DIAL command.
These devices are defined with the CP DEF GRAF command. They are
represented in sysfs as /sys/bus/ccw/drivers/3270/0.<n>.<devno> where <n> is
the subchannel set ID and <devno> is the virtual device number. By setting these
devices online, you enable them for user logins. If you set a device offline, it can
no longer be used for user login.

See z/VM CP Commands and Utilities Reference, SC24-6175 for more information
about the DEF GRAF and DIAL commands.

Procedure
You can use the chccwdev command (see chccwdev - Set CCW device attributes
on page 492) to set a CCW terminal device online or offline. Alternatively, you can
write 1 to the device's online attribute to set it online, or 0 to set it offline.

Examples
v To set a CCW terminal device 0.0.7b01 online, issue:
# chccwdev -e 0.0.7b01

Alternatively issue:

46 Device Drivers, Features, and Commands on SLES 12 SP2

 # echo 1 > /sys/bus/ccw/drivers/3270/0.0.7b01/online

v To set a CCW terminal device 0.0.7b01 offline, issue:

# chccwdev -d 0.0.7b01

Alternatively issue:
# echo 0 > /sys/bus/ccw/drivers/3270/0.0.7b01/online

Entering control and special characters on line-mode

terminals
Line-mode terminals do not have a control (Ctrl) key. Without a control key, you
cannot enter control characters directly.

Also, pressing the Enter key adds a newline character to your input string. Some
applications do not tolerate such trailing newline characters.

Table 9 summarizes how to use the caret character (^) to enter some control
characters and to enter strings without appended newline characters.
Table 9. Control and special characters on line-mode terminals
For the key Enter Usage
combination
Ctrl+C ^c Cancel the process that is running in the foreground of the
terminal.
Ctrl+D ^d Generate an end of file (EOF) indication.
Ctrl+Z ^z Stop a process.
n/a ^n Suppresses the automatic generation of a new line. Thus,
you can enter single characters; for example, the characters
that are needed for yes/no answers in some utilities.

Note: For a 3215 line-mode terminal in 3215 mode, you must use United States
code page (037).

Using the magic sysrequest feature

You can call the magic sysrequest functions from a line-mode terminal and,
depending on your setup, from the hvc0 terminal device.

To call the magic sysrequest functions on a line-mode terminal, enter the 2

characters ^- (caret and hyphen) followed by a third character that specifies the
particular function.

You can also call the magic sysrequest functions from the hvc0 terminal device if it
is present and is activated to receive Linux kernel messages. To call the magic
sysrequest functions from hvc0, enter the single character Ctrl+o followed by the
character for the particular function.

Table 10 on page 48 provides an overview of the commands for the magic

sysrequest functions:

Chapter 4. Console device drivers 47

 Table 10. Magic sysrequest functions
On line-mode On hvc0, enter To
terminals, enter
^-b Ctrl+o b Re-IPL immediately (see lsreipl - List IPL
and re-IPL settings on page 593).
^-s Ctrl+o s Emergency sync all file systems.
^-u Ctrl+o u Emergency remount all mounted file
systems read-only.
^-t Ctrl+o t Show task info.
^-m Ctrl+o m Show memory.
Ctrl+o Set the console log level.
^-
followed by a digit followed by a digit
(0 - 9) (0 - 9)
^-e Ctrl+o e Send the TERM signal to end all tasks
except init.
^-i Ctrl+o i Send the KILL signal to end all tasks except
init.
^-p Ctrl+o p See Obtaining debug information on page
467.

Note: In Table 10 Ctrl+o means pressing while holding down the control key.

Table 10 lists the main magic sysrequest functions that are known to work on
Linux on z Systems. For a more comprehensive list of functions, see
Documentation/sysrq.txt in the Linux source tree. Some of the listed functions
might not work on your system.

Activating and deactivating the magic sysrequest feature

Use the sysrq procfs attribute to activate or deactivate the magic sysrequest
feature.

Procedure

Issue the following command to activate the magic sysrequest feature:

echo 1 > /proc/sys/kernel/sysrq

Enter the following command to deactivate the magic sysrequest feature:

echo 0 > /proc/sys/kernel/sysrq

Tip: You can use YaST to activate and deactivate the magic sysrequest function.
Go to yast -> system -> Kernel Settings, select or clear the enable SYSRQ option
and leave YaST with OK.

Triggering magic sysrequest functions from procfs

If you are working from a terminal that does not support a key sequence or
combination to call magic sysrequest functions, you can trigger the functions
through procfs.

48 Device Drivers, Features, and Commands on SLES 12 SP2

 Procedure

Write the character for the particular function to /proc/sysrq-trigger.

You can use this interface even if the magic sysrequest feature is not activated as
described in Activating and deactivating the magic sysrequest feature on page
48.

Example

To set the console log level to 9, enter:

# echo 9 > /proc/sysrq-trigger

Using a z/VM emulation of the HMC Operating System

Messages applet
You can use the Operating System Messages applet emulation; for example, if the
3215 terminal is not operational.

About this task

The preferred terminal devices for Linux instances that run as z/VM guests are
provided by the 3215 or 3270 terminal device drivers.

The emulation requires a terminal device that is provided by the SCLP line-mode
terminal device driver. To use the emulation, you must override the default device
driver for z/VM environments (see Console kernel parameter syntax on page
38).

For the emulation, you use the z/VM CP VINPUT command instead of the
graphical user interface at the service element or HMC. Type any input to the
operating system with a leading CP VINPUT.

The examples in the sections that follow show the input line of a 3270 terminal or
terminal emulator (for example, x3270). Omit the leading #CP if you are in CP read
mode. For more information about VINPUT, see z/VM CP Commands and Utilities
Reference, SC24-6175.

Priority and non-priority commands

VINPUT commands require a VMSG (non-priority) or PVMSG (priority)
specification.

Operating systems that accept this specification, process priority commands with a
higher priority than non-priority commands.

The hardware console driver can accept both if supported by the hardware console
within the specific machine or virtual machine.

Linux does not distinguish priority and non-priority commands.

Example

The specifications:
#CP VINPUT VMSG LS -L

Chapter 4. Console device drivers 49

 and
#CP VINPUT PVMSG LS -L

are equivalent.

Case conversion
All lowercase characters are converted by z/VM to uppercase. To compensate for
this effect, the console device driver converts all input to lowercase.

For example, if you type VInput VMSG echo $PATH, the device driver gets ECHO
$PATH and converts it into echo $path.

Linux and bash are case-sensitive and require some specifications with uppercase
characters. To include uppercase characters in a command, use the percent sign (%)
as a delimiter. The console device driver interprets characters that are enclosed by
percent signs as uppercase.

Examples

In the following examples, the first line shows the user input. The second line
shows what the device driver receives after the case conversion by CP. The third
line shows the command that is processed by bash:
v
#cp vinput vmsg ls -l
CP VINPUT VMSG LS -L
ls -l
...

v The following input would result in a bash command that contains a variable
$path, which is not defined in lowercase:
#cp vinput vmsg echo $PATH
CP VINPUT VMSG ECHO $PATH
echo $path
...

To obtain the correct bash command enclose the uppercase string with the
conversion escape character:
#cp vinput vmsg echo $%PATH%
CP VINPUT VMSG ECHO $%PATH%
echo $PATH
...

Using the escape character

The quotation mark (") is the standard CP escape character. To include the escape
character in a command that is passed to Linux, you must type it twice.

For example, the following command passes a string in double quotation marks to
be echoed.
#cp vinput pvmsg echo ""here is ""$0
CP VINPUT PVMSG ECHO "HERE IS "$0
echo "here is "$0
here is -bash

50 Device Drivers, Features, and Commands on SLES 12 SP2

 In the example, $0 resolves to the name of the current process.

Using the end-of-line character

To include the end-of-line character in the command that is passed to Linux, you
must specify it with a leading escape character.

If you are using the standard settings according to Using a 3270 terminal in 3215
mode, you must specify "# to pass # to Linux.

If you specify the end-of-line character without a leading escape character, z/VM
CP interprets it as an end-of-line character that ends the VINPUT command.

Example

In this example, a number sign is intended to mark the begin of a comment in the
bash command. This character is misinterpreted as the beginning of a second
command.
#cp vinput pvmsg echo ""%N%umber signs start bash comments"" #like this one
CP VINPUT PVMSG ECHO "%N%UMBER SIGNS START BASH COMMENTS"
LIKE THIS ONE
HCPCMD001E Unknown CP command: LIKE
...

The escape character prevents the number sign from being interpreted as an
end-of-line character.
#cp vinput pvmsg echo ""%N%umber signs start bash comments"" "#like this one
VINPUT PVMSG ECHO "%N%UMBER SIGNS START BASH COMMENTS" #LIKE THIS ONE
echo "Number signs start bash comments" #like this one
Number signs start bash comments

Simulating the Enter and Spacebar keys

You can use the CP VINPUT command to simulate the Enter and Spacebar keys.

Simulate the Enter key by entering a blank followed by \n:

#CP VINPUT VMSG \n

Simulate the Spacebar key by entering two blanks followed by \n:

#CP VINPUT VMSG \n

Using a 3270 terminal in 3215 mode

The z/VM control program (CP) defines five characters as line-editing symbols.
Use the CP QUERY TERMINAL command to see the current settings.

The default line-editing symbols depend on your terminal emulator. You can
reassign the symbols by changing the settings of LINEND, TABCHAR, CHARDEL, LINEDEL,
or ESCAPE with the CP TERMINAL command. Table 11 on page 52 shows the most
commonly used settings:

Chapter 4. Console device drivers 51

 Table 11. Line edit characters
Character Symbol Usage
# LINEND The end of line character. With this character, you can enter several
logical lines at once.
| TABCHAR The logical tab character.
@ CHARDEL The character delete symbol deletes the preceding character.
[ or LINEDEL The line delete symbol deletes everything back to and including the
previous LINEND symbol or the start of the input. [ is common
for ASCII terminals and for EBCDIC terminals.
" ESCAPE The escape character. With this character, you can enter a line-edit
symbol as a normal character.

To enter a line-edit symbol, you must precede it with the escape character. In
particular, to enter the escape character, you must type it twice.

Examples

The following examples assume the settings of Table 11 with the opening bracket
character ([) as the delete line character.
v To specify a tab character, specify:
"|
v To specify a double quotation mark character, specify:
""
v If you type the character string:
#CP HALT#CP ZIPL 190[#CP IPL 1@290 PARM vmpoff=""MSG OP REBOOT"#IPL 290""

the actual commands that are received by CP are:

CP HALT
CP IPL 290 PARM vmpoff="MSG OP REBOOT#IPL 290"

52 Device Drivers, Features, and Commands on SLES 12 SP2

Chapter 5. Booting Linux
The options and requirements you have for booting Linux depend on your
platform, LPAR or z/VM, and on your boot medium.

The boot loader for SUSE Linux Enterprise Server 12 SP2 is GRUB 2. Use GRUB 2
to prepare DASD and SCSI devices as IPL devices for booting Linux. For details
about GRUB 2, see SUSE Linux Enterprise Server 12 SP2 Administration Guide.

IPL and booting

On z Systems, you usually start booting Linux by performing an Initial Program
Load (IPL).

Figure 13 illustrates the main steps of booting SUSE Linux Enterprise Server 12
SP2.

memory memory memory memory

Auxiliary Auxiliary
kernel image kernel image
GRUB 2 GRUB 2

zipl zipl Linux Linux

boot loader boot loader target kernel target kernel
code code image image

IPL device IPL device IPL device

zipl zipl zipl
boot loader boot loader boot loader
code code code
Auxiliary Auxiliary Auxiliary
kernel image kernel image kernel image
GRUB 2 GRUB 2 GRUB 2

Linux Linux Linux

target kernel target kernel target kernel
image image image

(1) IPL: (2) Boot process 1: (3) Boot process 2: (4) Result:
firmware loads zipl boot loader GRUB 2 loads target kernel
zipl boot loads auxiliary target kernel gets control
loader code kernel
Figure 13. IPL and boot process

First step: IPL

The IPL process is controlled by the z Systems firmware. In this step, zipl
boot loader code is loaded into memory.

Copyright IBM Corp. 2000, 2016 53

 Second step: boot process for the auxiliary kernel
In this step, the zipl boot loader gets control. It loads a Linux auxiliary
kernel into memory. This auxiliary kernel includes GRUB 2. Depending on
your configuration and boot device, a zipl boot menu might be displayed
during this step.
Third step: boot process for the target kernel
In this step, GRUB 2 gets control. It loads the target Linux kernel into
memory.
Fourth step: the target kernel takes over
When the boot process for the target Linux kernel has completed, the
target Linux kernel gets control.

If your Linux instance is to run in LPAR mode, you can also use the HMC or the
service element (SE) to copy the Linux kernel to the mainframe memory (see
Loading Linux from removable media or from an FTP server on page 65).
Typically, this method applies to an initial installation of a Linux instance.

Apart from starting a boot process, an IPL can also start a dump process. See Using
the Dump Tools on SUSE Linux Enterprise Server 12 SP1, SC34-2746 for more
information about dumps.

You can find the newest version of this publication on IBM Knowledge Center at
www.ibm.com/support/knowledgecenter/linuxonibm/liaaf/lnz_r_suse.html

Control point and boot medium

The control point from where you can start the boot process depends on the
environment where Linux is to run.

If your Linux instance is to run in LPAR mode, the control point is the mainframe's
Support Element (SE) or an attached Hardware Management Console (HMC). For
Linux on z/VM, the control point is the control program (CP) of the hosting
z/VM.

The media that can be used as boot devices also depend on where Linux is to run.
Table 12 provides an overview of the possibilities:
Table 12. Boot media
DASD SCSI z/VM reader CD-ROM/FTP
z/VM guest U U U
LPAR U U U

DASDs and SCSI devices that are attached through an FCP channel can be used for
both LPAR and z/VM guest virtual machines. A SCSI device can be a disk or an
FC-attached CD-ROM or DVD drive. The z/VM reader is available only in a
z/VM environment.

For Linux in LPAR mode, you can also boot from a CD-ROM drive on the SE or
HMC, or you can obtain the boot data from a remote FTP server.

Typically, booting from removable media applies to initial installation of Linux.

Booting from DASD or SCSI disk devices usually applies to previously installed
Linux instances.

54 Device Drivers, Features, and Commands on SLES 12 SP2

Boot data
To boot Linux, you generally need a kernel image, boot loader code, kernel
parameters, and an initial RAM disk image.

For the z/VM reader, as a sequential I/O boot device, the order in which this data
is provided is significant. For random access devices there is no required order.

On SUSE Linux Enterprise Server 12 SP2, kernel images are installed into the /boot
directory and are named image-<version>. For information about where to find the
images and how to start an installation, see SUSE Linux Enterprise Server 12 SP2
Deployment Guide.

Boot loader code

SUSE Linux Enterprise Server 12 SP2 kernel images are compiled to contain boot
loader code for IPL from z/VM reader devices.

If you want to boot a kernel image from a device that does not correspond to the
included boot loader code, you can provide alternate boot loader code separate
from the kernel image.

Use GRUB 2 to prepare boot devices with separate DASD or SCSI boot loader
code. You can then boot from these devices, regardless of the boot loader code in
the kernel image.

Kernel parameters
The kernel parameters are in form of an ASCII text string of up to 895 characters.
If the boot device is the z/VM reader, the string can also be encoded in EBCDIC.

Individual kernel parameters are single keywords or keyword/value pairs of the

form keyword=<value> with no blank. Blanks are used to separate consecutive
parameters.

You specify kernel parameters when you create your boot configuration with
GRUB 2. Depending on your boot method, you can add kernel parameters when
starting the boot process.

Important: Do not specify parameters that prevent SUSE Linux Enterprise Server
12 SP2 from booting. See Avoid parameters that break GRUB 2 on page 23.

Initial RAM disk image

An initial RAM disk holds files, programs, or modules that are not included in the
kernel image but are required for booting.

For example, booting from DASD requires the DASD device driver. If you want to
boot from DASD but the DASD device driver has not been compiled into your
kernel, you need to provide the DASD device driver module on an initial RAM
disk.

SUSE Linux Enterprise Server 12 SP2 provides a ramdisk in /boot and named
initrd-<kernel version>.

Chapter 5. Booting Linux 55

 Rebuilding the initial RAM disk image
Configuration changes might apply to components that are required in the boot
process before the root file system is mounted.

For SUSE Linux Enterprise Server 12, such components and their configuration are
provided through an initial RAM disk.

Procedure

Perform these steps to make configuration changes for components in the initrd
take effect:
1. Issue dracut -f to update the initial RAM disk of your target kernel.
2. Issue grub2-install to update the initial RAM disk of the auxiliary kernel and
to rewrite the zipl boot record.

Booting Linux in a z/VM guest virtual machine

You boot Linux in a z/VM guest virtual machine by issuing CP commands from a
CMS or CP session.

For more general information about z/VM guest environments for Linux, see z/VM
Getting Started with Linux on System z, SC24-6194.

Booting from a DASD

Boot Linux by issuing the IPL command with a DASD boot device.

Before you begin

You need a DASD boot device that is prepared with GRUB 2.

Procedure

Perform these steps to start the boot process:

1. Establish a CMS or CP session with the z/VM guest virtual machine where you
want to boot Linux.
2. Ensure that the boot device is accessible to your z/VM guest virtual machine.
3. Issue a command of this form:
#cp i <devno> clear loadparm <n>g<grub_parameters> parm <kernel_parameters>

where:
<devno>
specifies the device number of the boot device as seen by the guest.
<n>
selects the kernel to be booted.
0 or 1
immediately starts GRUB 2 for booting the target SUSE Linux
Enterprise Server 12 SP2 kernel.
2 boots a rescue kernel.

56 Device Drivers, Features, and Commands on SLES 12 SP2

 If you omit this specification, GRUB 2 is started after a timeout period has
expired. Depending on your configuration, a zipl boot menu might be
displayed during the timeout period. From this menu, you can choose
between starting GRUB 2 or booting a rescue kernel.
<grub_parameters>
specifies parameters for GRUB 2. Typically, this specification selects a boot
option from a GRUB 2 boot menu. For details, see Specifying GRUB 2
parameters on page 68.
<kernel_parameters>
is an optional 64-byte string of kernel parameters to be concatenated to the
end of the existing kernel parameters that are used by your boot
configuration.

Important: Do not specify parameters that prevent SUSE Linux Enterprise

Server 12 SP2 from booting. See Avoid parameters that break GRUB 2 on
page 23.

Example for the zipl menu

This example illustrates how a zipl menu is displayed on the z/VM guest virtual
machine console.
00: zIPL interactive boot menu
00:
00: 0. default (grub2)
00:
00: 1. grub2
00: 2. skip-grub
00:
00: Note: VM users please use #cp vi vmsg <number> <kernel-parameters>
00:
00: Please choose (default will boot in 30 seconds): #cp vi vmsg 1

Specify 0 or 1 to immediately start GRUB 2 to proceed with booting the target

kernel. Specify 2 to start a rescue kernel. If you do not select a menu item until the
timeout expires, GRUB 2 is started.

Example: To start GRUB 2 specify:

#cp vi vmsg 1

Booting from a SCSI device

Boot Linux by issuing the IPL command with an FCP channel as the IPL device.
You must specify the target port and LUN for the boot device in advance by
setting the z/VM CP LOADDEV parameter.

Before you begin

You need a SCSI boot device that is prepared with GRUB 2.

Procedure

Perform these steps to start the boot process:

1. Establish a CMS or CP session with the z/VM guest virtual machine where you
want to boot Linux.

Chapter 5. Booting Linux 57

 2. Ensure that the FCP channel that provides access to the SCSI boot disk is
accessible to your z/VM guest virtual machine.
3. Specify the target port and LUN of the SCSI boot disk. Enter a command of
this form:
#cp set loaddev portname <wwpn> lun <lun>

where:
<wwpn>
specifies the world wide port name (WWPN) of the target port in
hexadecimal format. A blank separates the first eight digits from the final
eight digits.
<lun>
specifies the LUN of the SCSI boot disk in hexadecimal format. A blank
separating the first eight digits from the final eight digits.

Example: To specify a WWPN 0x5005076300c20b8e and a LUN

0x5241000000000000:
#cp set loaddev portname 50050763 00c20b8e lun 52410000 00000000

4. Optional for menu configurations: Specify the boot configuration (boot program
in z/VM terminology) to be used. Enter a command of this form:
#cp set loaddev bootprog <n>

where <n> selects the kernel to be booted.

0 or 1
immediately starts GRUB 2 for booting the target SUSE Linux Enterprise
Server 12 SP2 kernel.
2 boots a rescue kernel.

If you omit this specification, GRUB 2 is started after a timeout period has
expired.

Example: To start GRUB 2 and proceed with booting the target kernel, issue
this command:
#cp set loaddev bootprog 0

5. Optional: Add kernel parameters.

Issue a command of this form:
#cp set loaddev scpdata <APPEND|NEW> <kernel_parameters>

where:
<kernel_parameters>
specifies a set of kernel parameters to be stored as system control program
data (SCPDATA). When booting Linux, these kernel parameters are
concatenated to the end of the existing kernel parameters that are used by
your boot configuration.

58 Device Drivers, Features, and Commands on SLES 12 SP2

 <kernel_parameters> must contain ASCII characters only. If characters other
than ASCII characters are present, the boot process ignores the SCPDATA.
<kernel_parameters> as entered from a CMS or CP session is interpreted as
lowercase on Linux. If you require uppercase characters in the kernel
parameters, run the SET LOADDEV command from a REXX script instead.
In the REXX script, use the address command statement. See REXX/VM
Reference, SC24-6221 and REXX/VM User's Guide, SC24-6222 for details.
Optional: APPEND
appends kernel parameters to existing SCPDATA. This is the default.
Optional: NEW
replaces existing SCPDATA.

Important: Do not specify parameters that prevent SUSE Linux Enterprise

Server 12 SP2 from booting. See Avoid parameters that break GRUB 2 on
page 23.
6. Start the IPL and boot process by entering a command of this form:
#cp i <devno> loadparm g<grub_parameters>

where
<devno>
is the device number of the FCP channel that provides access to the SCSI
boot disk.
loadparm g<grub_parameters>
optionally specifies parameters for GRUB 2. Typically, this specification
selects a boot option from a GRUB 2 boot menu. For details, see
Specifying GRUB 2 parameters on page 68.

Tip

You can specify the target port and LUN of the SCSI boot disk, a boot
configuration, and SCPDATA all with a single SET LOADDEV command. See
z/VM CP Commands and Utilities Reference, SC24-6175 for more information about
the SET LOADDEV command.

Booting from the z/VM reader

Boot Linux by issuing the IPL command with the z/VM reader as the IPL device.
You first must transfer the boot data to the reader.

Before you begin

You need the following files, all in record format fixed 80:
v Linux kernel image with built-in z/VM reader boot loader code. This is the case
for the default SUSE Linux Enterprise Server 12 SP2 kernel.
v Kernel parameters (optional)
v Initial RAM disk image (optional)

About this task

This information is a summary of how to boot Linux from a z/VM reader. For
more details, see Redpaper Building Linux Systems under IBM VM, REDP-0120.

Chapter 5. Booting Linux 59

 Tip: On the SUSE Linux Enterprise Server 12 SP2 DVD under /boot/s390x there is
a sample script (SLES12 EXEC) for booting from the z/VM reader.

Procedure

Proceed like this to boot Linux from a z/VM reader:

1. Establish a CMS session with the guest where you want to boot Linux.
2. Transfer the kernel image, kernel parameters, and the initial RAM disk image
to your guest. You can obtain the files from a shared minidisk or use:
v The z/VM sendfile facility.
v An FTP file transfer in binary mode.
Files that are sent to your reader contain a file header that you must remove
before you can use them for booting. Receive files that you obtain through your
z/VM reader to a minidisk.
3. Set up the reader as a boot device.
a. Ensure that your reader is empty.
b. Direct the output of the punch device to the reader. Issue:
spool pun * rdr

c. Use the CMS PUNCH command to transfer each of the required files to the
reader. Be sure to use the no header option to omit the file headers.
First transfer the kernel image.
Second transfer the kernel parameters.
Third transfer the initial RAM disk image, if present.
For each file, issue a command of this form:
pun <file_name> <file_type> <file_mode> (noh

d. Optional: Ensure that the contents of the reader remain fixed.

change rdr all keep nohold

If you omit this step, all files are deleted from the reader during the IPL
that follows.
4. Issue the IPL command:
ipl 000c clear parm <kernel_parameters>

where:
0x000c
is the device number of the reader.
parm <kernel_parameters>
is an optional 64-byte string of kernel parameters to be concatenated to the
end of the existing kernel parameters that are used by your boot
configuration.
See also Adding kernel parameters when booting Linux on page 24.

60 Device Drivers, Features, and Commands on SLES 12 SP2

Booting Linux in LPAR mode
You can boot Linux in LPAR mode from a Hardware Management Console (HMC)
or Support Element (SE).

About this task

The following description refers to an HMC, but the same steps also apply to an
SE.

Booting from DASD

Use the SE or HMC to boot Linux in LPAR from a DASD boot device.

Before you begin

You need a boot device that is prepared with GRUB 2.

Procedure

Perform these steps to boot from a DASD:

1. In the navigation pane of the HMC, expand Systems Management and Servers
and select the mainframe system that you want to work with. A table of LPARs
is displayed on the Images tab in the content area.
2. Select the LPAR where you want to boot Linux.
3. In the Tasks area, expand Recovery and click Load (see Figure 14).

1) Select mainframe system

2) Select
LPAR 3) Click Load

Figure 14. Load task on the HMC

4. Select load type Normal (see Figure 15 on page 62).

Chapter 5. Booting Linux 61

 0g2

Figure 15. Load panel for booting from DASD

5. Enter the device number of the DASD boot device in the Load address field.
6. Enter a specification of the form <n>g<grub_parameters> in the Load parameter
filed.
<n>
selects the kernel to be booted.
0 or 1
immediately starts GRUB 2 for booting the target SUSE Linux
Enterprise Server 12 SP2 kernel.
2 boots a rescue kernel.

If you omit this specification, GRUB 2 is started after a timeout period has
expired. Depending on your configuration, a zipl boot menu might be
displayed during the timeout period. From this menu, you can choose
between starting GRUB 2 or booting a rescue kernel.
<grub_parameters>
specifies parameters for GRUB 2. Typically, this specification selects a boot
option from a GRUB 2 boot menu. For details, see Specifying GRUB 2
parameters on page 68.
7. Click OK to start the boot process.

62 Device Drivers, Features, and Commands on SLES 12 SP2

 Example for the zipl menu

This example illustrates how a zipl menu is displayed on the HMC or SE.
zIPL interactive boot menu

0. default (grub2)

1. grub2
2. skip-grub

Note: VM users please use #cp vi vmsg <number> <kernel-parameters>

Please choose (default will boot in 30 seconds): 1

Specify 0 or 1 to immediately start GRUB 2 and proceed with booting the target
kernel. Specify 2 to start a rescue kernel. If you do not select a menu item before
the timeout expires, GRUB 2 is started.

What to do next
Check the output on the preferred console (see Console kernel parameter syntax
on page 38) to monitor the boot progress.

Booting from SCSI

Use the SE or HMC to boot Linux in LPAR from a SCSI boot device.

Before you begin

You need a boot device that is prepared with GRUB 2.

Procedure

Perform these steps to boot from a SCSI boot device:

1. In the navigation pane of the HMC, expand Systems Management and
Servers and select the mainframe system that you want to work with. A table
of LPARs is displayed on the Images tab in the content area.
2. Select the LPAR where you want to boot Linux.
3. In the Tasks area, expand Recovery and click Load (see Figure 16 on page 64).

Chapter 5. Booting Linux 63

 1) Select mainframe system

2) Select
LPAR 3) Click Load

Figure 16. Load task on the HMC

4. Select load type SCSI (see Figure 17).

g2

Figure 17. Load panel with SCSI feature enabled - for booting from a SCSI device

5. Enter the device number of the FCP channel through which the SCSI device is
accessed in the Load address field.
6. Enter the WWPN of the SCSI device in the World wide port name field.
7. Enter the LUN of the SCSI device in the Logical unit number field.

64 Device Drivers, Features, and Commands on SLES 12 SP2

 8. Optional: In the Boot program selector field, enter 0, 1, or 2.
0 or 1
immediately starts GRUB 2 for booting the target SUSE Linux Enterprise
Server 12 SP2 kernel.
2 boots a rescue kernel.
If you omit this specification, the target kernel is booted after a timeout period
has expired.
9. Optional: In the Load parameter field specify g<grub_parameters> where
<grub_parameters> are parameters to be evaluated by GRUB 2.
Typically, this specification selects a boot option from a GRUB 2 boot menu.
For details, see Specifying GRUB 2 parameters on page 68.
10. Optional: Type kernel parameters in the Operating system specific load
parameters field. These parameters are concatenated to the end of the existing
kernel parameters used by your boot configuration when booting Linux.
Use ASCII characters only. If you enter characters other than ASCII characters,
the boot process ignores the data in the Operating system specific load
parameters field.

Important: Do not specify parameters that prevent SUSE Linux Enterprise

Server 12 SP2 from booting. See Avoid parameters that break GRUB 2 on
page 23.
11. Accept the defaults for the remaining fields.
12. Click OK to start the boot process.

What to do next
Check the output on the preferred console (see Console kernel parameter syntax
on page 38) to monitor the boot progress.

Loading Linux from removable media or from an FTP server

Instead of a boot loader, you can use SE functions to copy the Linux kernel image
to your LPAR memory.

After the Linux kernel is loaded, Linux is started using restart PSW.

Before you begin

You need installation data that includes a special file with installation information
(with extension ins). This file can be in different locations:
v On a disk that is inserted in the CD-ROM or DVD drive of the system where the
HMC runs
v In the file system of an FTP server that you can access through FTP from your
HMC system

The .ins file contains a mapping of the location of installation data on the disk or
FTP server and the memory locations where the data is to be copied.

For SUSE Linux Enterprise Server 12 SP2 this file is called suse.ins and located in
the root directory of the file system on the DVD 1.

Chapter 5. Booting Linux 65

 Procedure

Perform these steps:

1. In the navigation pane of the HMC, expand Systems Management and Servers
and select the mainframe system you want to work with. A table of LPARs is
displayed on the Images tab in the content area.
2. Select the LPAR where you want to boot Linux.
3. In the Tasks area, expand Recovery and click Load from Removable Media or
Server (see Figure 18).

1) Select mainframe system

2) Select
3) Click
LPAR
Load from Removable Media or Server

Figure 18. Load from Removable Media or Server on the HMC

4. Specify the source of the code to be loaded.

v For loading from a CD-ROM drive:
a. Select Hardware Management Console CD-ROM/DVD (see Figure 19 on
page 67).

66 Device Drivers, Features, and Commands on SLES 12 SP2

Figure 19. Load from Removable Media or Server panel

b. Leave the File location field blank.

v For an initial installation from removable media at the HMC
a. Select Hardware Management Console CD / DVD and assign for
operating system use (see Figure 19).
b. Enter the path for the directory where the ins-file is in the File location
field. You can leave this field blank if the ins-file is in the root directory
of the file system on the removable media.
The installation CD or DVD must hold a distribution that supports an
installation from the HMC.
v For loading from an FTP server
a. Select the FTP Source radio button.
b. Enter the IP address or host name of the FTP server with the installation
code resides in the Host computer entry field.
c. Enter your user ID for the FTP server in the User ID entry field.
d. Enter your password for the FTP server in the Password entry field.
e. If required by your FTP server, enter your account information in the
Account entry field.
f. Enter the path for the directory where the suse.ins resides in the file
location entry field. You can leave this field blank if the file is in the FTP
server's root directory.
5. Click Continue to display the Select Software to Install panel (Figure 20 on
page 68).

Chapter 5. Booting Linux 67

 Load from Removable Media or Server - Select Software
to Install
Select the software to install.
Select Name Description
SLES-12/DVD/suse.ins SUSE Linux Enterp
OK Cancel Help

Figure 20. Select Software to Install panel

6. Select suse.ins.
7. Click OK to start loading Linux.

Results

The kernel has started and the SUSE Linux Enterprise Server 12 SP2 boot process
continues.

Specifying GRUB 2 parameters

When you IPL from SCSI or DASD, you can use the loadparm parameter to, for
example, select a boot option from a GRUB 2 boot menu.

About this task

For DASD the syntax is <0|1|2>g<grub_parameters>, for SCSI it is

g<grub_parameters>, where <grub_parameters> specifies a boot configuration from
a GRUB 2 boot menu.

Procedure
1. Optional: To select a GRUB 2 boot option, first use grub2-once --enum to list
the GRUB 2 boot entries, for example:
# grub2-once --enum
0 SLES12
1>0 Advanced options for SLES12>SLES12, with Linux 3.12.49-3-default
1>1 Advanced options for SLES12>SLES12, with Linux 3.12.49-3-default (recovery mode)

2. To specify a GRUB 2 boot entry, replace the greater than (>) character with the
full stop (.) character. For example, specify loadparm g1.1 for the 1>1 boot
entry.

Displaying current IPL parameters

To display the IPL parameters, use the lsreipl command with the -i option.
Alternatively, a sysfs interface is available.

For more information about the lsreipl command, see lsreipl - List IPL and
re-IPL settings on page 593. In sysfs, information about IPL parameters is
available in subdirectories of /sys/firmware/ipl.

/sys/firmware/ipl/ipl_type

68 Device Drivers, Features, and Commands on SLES 12 SP2

The /sys/firmware/ipl/ipl_type attribute contains the device type from which the
kernel was booted. The following values are possible:
ccw The IPL device is a CCW device, for example, a DASD or the z/VM
reader.
fcp The IPL device is an FCP device.
unknown
The IPL device is not known.

Depending on the IPL type, there might be more attributes in /sys/firmware/ipl/.

If the device is a CCW device, the additional attributes device and loadparm are
present.
device Contains the bus ID of the CCW device that is used for IPL, for example:
# cat /sys/firmware/ipl/device
0.0.1234

loadparm
Contains up to 8 characters for the loadparm that is used for IPL, for
example:
# cat /sys/firmware/ipl/loadparm
0g2

parm
Contains additional kernel parameters that are specified with the PARM
parameter when booting with the z/VM CP IPL command. See also
Adding kernel parameters when booting Linux on page 24.

If the device is FCP, a number of additional attributes are present (also see
Chapter 10, SCSI-over-Fibre Channel device driver, on page 145 for details):
device Contains the bus ID of the FCP device that is used for IPL, for example:
# cat /sys/firmware/ipl/device
0.0.50dc

wwpn Contains the WWPN used for IPL, for example:

# cat /sys/firmware/ipl/wwpn
0x5005076300c20b8e

lun Contains the LUN used for IPL, for example:

# cat /sys/firmware/ipl/lun
0x5010000000000000

br_lba Contains the logical block address of the boot record on the boot device
(usually 0).
bootprog
Contains the boot program number. For example:
# cat /sys/firmware/ipl/bootprog
0

Chapter 5. Booting Linux 69

 loadparm
Contains up to 8 characters as parameters for GRUB 2. Typically, this
specification selects a boot option from a GRUB 2 boot menu. For example:
# cat /sys/firmware/ipl/loadparm
g2

scp_data
Contains additional kernel parameters, if any, that are used when booting
from a SCSI device. For information about how SCPDATA can be set see
the following sections:
v Booting from a SCSI device on page 57 for z/VM
v Booting from SCSI on page 63 for LPAR
v chreipl - Modify the re-IPL configuration on page 499
binary_parameter
Contains the information of the preceding attributes in binary format.

Rebooting from an alternative source

When you reboot Linux, the system conventionally boots from the last used
location. However, you can configure an alternative device to be used for re-IPL
instead of the last used IPL device.

When the system is re-IPLed, the alternative device is used to boot the kernel.

To configure the re-IPL device, use the chreipl tool (see chreipl - Modify the
re-IPL configuration on page 499).

Alternatively, you can use the sysfs attributes in /sys/firmware/reipl. To

configure, write strings into the attributes. The following re-IPL types can be set
with the /sys/firmware/reipl/reipl_type attribute:
ccw For ccw devices such as DASDs that are attached through ESCON or
FICON.
fcp For FCP SCSI devices, including SCSI disks and CD or DVD drives
(Hardware support is required.)
nss For Named Saved Systems (z/VM only)

For each supported re-IPL type a sysfs directory is created under

/sys/firmware/reipl that contains the configuration attributes for the device. The
directory name is the same as the name of the re-IPL type.

When Linux is booted, the re-IPL attributes are set by default to the values of the
boot device, which can be found under /sys/firmware/ipl.

Attributes for ccw

You can find the attributes for re-IPL type ccw in the /sys/firmware/reipl/ccw
sysfs directory.
| device Device number of the re-IPL device. For example 0.0.7412 or 0.1.5119.
loadparm
Up to eight characters for the loadparm used to select the boot
configuration in the zipl menu (if available).

70 Device Drivers, Features, and Commands on SLES 12 SP2

 If the re-IPL target kernel is SUSE Linux Enterprise Server 12 or later, the
specification must be of the form <n>g<grub_parameters>, where
<n>
selects the kernel to be booted.
0 or 1
immediately starts GRUB 2 for booting the target kernel.
2 boots a rescue kernel.

If you omit this specification, GRUB 2 is started after a timeout period

has expired.
<grub_parameters>
specifies parameters for GRUB 2. Typically, this specification selects a
boot option from a GRUB 2 boot menu. For details, see Specifying
GRUB 2 parameters on page 68.
parm A 64-byte string of kernel parameters that is concatenated to the boot
command line. The PARM parameter can be set only for Linux on z/VM.
See also Adding kernel parameters when booting Linux on page 24.
A leading equal sign (=) means that the existing kernel parameter line in
the boot configuration is ignored and the boot process uses the kernel
parameters in the scp_data attribute only.

Important:
v If the re-IPL kernel is SUSE Linux Enterprise Server 12 or later, be sure
not to specify kernel parameters that prevent the target kernel from
booting.
v In particular, do not use a leading equal sign if the re-IPL kernel is SUSE
Linux Enterprise Server or later.
See Avoid parameters that break GRUB 2 on page 23.

Attributes for fcp

You can find the attributes for re-IPL type fcp in the /sys/firmware/reipl/fcp
sysfs directory.
device Device number of the FCP device that is used for re-IPL. For example,
0.0.7412.

Note: IPL is possible only from subchannel set 0.

wwpn World wide port number of the FCP re-IPL device.
lun Logical unit number of the FCP re-IPL device.
loadparm
If the re-IPL target is a SUSE Linux Enterprise Server 12 SP2 kernel, up to
eight characters to specify parameters for GRUB 2. The specification must
be of the form g<grub_parameters>. Typically, <grub_parameters> is a
specification that selects an entry in the GRUB 2 menu. For details, see
Specifying GRUB 2 parameters on page 68.
bootprog
Boot program selector to select the kernel to be booted.
0 or 1
immediately starts GRUB 2 for booting the target kernel.

Chapter 5. Booting Linux 71

 2 boots a rescue kernel.

If you omit this specification, GRUB 2 is started after a timeout period has
expired.
br_lba Boot record logical block address. Master boot record. Is always 0 for
Linux.
scp_data
Kernel parameters to be used for the next FCP re-IPL.
A leading equal sign (=) means that the existing kernel parameter line in
the boot configuration is ignored and the boot process uses the kernel
parameters in the scp_data attribute only.

Important:
v If the re-IPL kernel is SUSE Linux Enterprise Server 12 or later, be sure
not to specify kernel parameters that prevent the target kernel from
booting.
v In particular, do not use a leading equal sign if the re-IPL kernel is SUSE
Linux Enterprise Server or later.
See Avoid parameters that break GRUB 2 on page 23.

Attributes for nss

You can find the attributes for re-IPL type nss in the /sys/firmware/reipl/nss
sysfs directory.
name Name of the NSS. The NSS name can be 1-8 characters long and must
consist of alphabetic or numeric characters. The following examples are all
valid NSS names: 73248734, NSSCSITE, or NSS1234.
parm A 56-byte string of parameters for the operating system in the NSS.

You cannot load SUSE Linux Enterprise Server 12 or later from an NSS. If the NSS
contains a Linux distribution that supports NSS, the value could be a string of
kernel parameters.

Kernel panic settings

Set the attribute /sys/firmware/shutdown_actions/on_panic to reipl to make the
system re-IPL with the current re-IPL settings if a kernel panic occurs.

See also the description of the dumpconf tool in Using the Dump Tools on SUSE Linux
Enterprise Server 12 SP1, SC34-2746 on the IBM Knowledge Center website at
www.ibm.com/support/knowledgecenter/linuxonibm/com.ibm.trouble.doc/
serviceandsupport.html

Examples for configuring re-IPL

Typical examples include configuring re-IPL from an FCP device and specifying
parameters for re-IPL.
v To configure an FCP re-IPL device 0.0.5711 with a LUN 0x1711000000000000 and
a WWPN 0x5005076303004715 with an additional kernel parameter noresume:

72 Device Drivers, Features, and Commands on SLES 12 SP2

 # echo 0.0.5711 > /sys/firmware/reipl/fcp/device
# echo 0x5005076303004715 > /sys/firmware/reipl/fcp/wwpn
# echo 0x1711000000000000 > /sys/firmware/reipl/fcp/lun
# echo 0 > /sys/firmware/reipl/fcp/bootprog
# echo 0 > /sys/firmware/reipl/fcp/br_lba
# echo fcp > /sys/firmware/reipl/reipl_type

v To set up re-IPL from a CMS NSS:

1. Set the reipl_type to nss:
# echo nss > /sys/firmware/reipl/reipl_type

2. Set up the attributes in the nss directory:

# echo CMSNSS > /sys/firmware/reipl/reipl_type/nss/name
# echo "AUTOCR" > /sys/firmware/reipl/reipl_type/nss/parm

Chapter 5. Booting Linux 73

74 Device Drivers, Features, and Commands on SLES 12 SP2
Chapter 6. Suspending and resuming Linux
With suspend and resume support, you can stop a running Linux on z Systems
instance and later continue operations.

When Linux is suspended, data is written to a swap partition. The resume process
uses this data to make Linux continue from where it left off when it was
suspended. A suspended Linux instance does not require memory or processor
cycles.

Linux on z Systems suspend and resume support applies to both Linux on z/VM
and Linux instances that run directly in an LPAR.

While a Linux instance is suspended, you can run another Linux instance in the
z/VM guest virtual machine or in the LPAR where the suspended Linux instance
was running.

What you should know about suspend and resume

Before suspending a Linux instance, you must be aware of the prerequisites and of
activities that can cause resume to fail.

Prerequisites for suspending a Linux instance

Suspend and resume support checks for conditions that might prevent resuming a
suspended Linux instance. You cannot suspend a Linux instance unless all
prerequisites are fulfilled.

The following prerequisites must be fulfilled regardless of whether a Linux

instance runs directly in an LPAR or as a z/VM guest:
v All tape device nodes must be closed and online tape drives must be unloaded.
v There must be no configured Common Link Access to Workstation (CLAW)
devices.
The CLAW device driver does not support suspend and resume. You must
ungroup all CLAW devices before you can suspend a Linux instance.
v The Linux instance must not have used any hotplug memory since it was last
booted.
v No program must be in a prolonged uninterruptible sleep state.
Programs can assume this state while they are waiting for an outstanding I/O
request to complete. Most I/O requests complete in a very short time and do not
compromise suspend processing. An example of an I/O request that can take too
long to complete is rewinding a tape.

For Linux on z/VM, the following additional prerequisites must be fulfilled:

v No discontiguous saved segment (DCSS) device must be accessed in
exclusive-writable mode.
You must remove all DCSSs of segment types EW, SW, and EN by writing the
DCSS name to the sysfs remove attribute.
You must remove all DCSSs of segment types SR and ER that are accessed in
exclusive-writable mode or change their access mode to shared.

Copyright IBM Corp. 2000, 2016 75

 For more information, see Removing a DCSS device on page 413 and Setting
the access mode on page 410.
v All device nodes of the z/VM recording device driver must be closed.
v All device nodes of the z/VM unit record device driver must be closed.
v No watchdog timer must run and the watchdog device node must be closed.

Precautions while a Linux instance is suspended

There are conditions outside the control of the suspended Linux instance that can
cause resume to fail.
v The CPU configuration must remain unchanged between suspend and resume.
v The data that is written to the swap partition when the Linux instance is
suspended must not be compromised.
In particular, be sure that the swap partition is not used if another operating
system instance runs in the LPAR or z/VM guest virtual machine while the
initial Linux instance is suspended.
v If the Linux instance uses expanded storage (XPRAM), this expanded storage
must remain unchanged until the Linux instance is resumed.
If the size or content of the expanded memory is changed before the Linux
instance is resumed or if the expanded memory is unavailable when the Linux
instance is resumed, resuming fails with a kernel panic.
v If an instance of Linux on z/VM uses one or more DCSSs these DCSSs must
remain unchanged until the Linux instance is resumed.
If the size, location, or content of a DCSS is changed before the Linux instance is
resumed, resuming fails with a kernel panic.
v Take special care when replacing a DASD and, thus, making a different device
available at a particular device bus-ID.
You might intentionally replace a device with a backup device. Changing the
device also changes its UID-based device nodes. Expect problems if you run an
application that depends on UID-based device nodes and you exchange one of
the DASD the application uses. In particular, you cannot use multipath tools
when the UID changes.
v Generally, avoid changes to the real or virtual hardware configuration between
suspending and resuming a Linux instance.
v Disks that hold swap partitions or the root file system must be present when
resuming the Linux instance.

Handling of devices that are unavailable when resuming

Devices that were available when the Linux instance was suspended might be
unavailable when resuming.

If such unavailable devices were offline when the Linux instance was suspended,
they are de-registered and the device name can be assigned to other devices.

If unavailable devices where online when the Linux instance was suspended,
handling depends on the respective device driver. DASD and FCP devices remain
registered as disconnected devices. The device name and the device configuration
are preserved. Devices that are controlled by other device drivers are de-registered.

76 Device Drivers, Features, and Commands on SLES 12 SP2

 Handling of devices that become available at a different
subchannel
The mapping between subchannels and device bus-IDs can change if the real or
virtual hardware is restarted between suspending and resuming Linux.

If the subchannel changes for a DASD or FCP device, the device configuration is
changed to reflect the new subchannel. This change is accomplished without
de-registration. Thus, device name and device configuration are preserved.

If the subchannel changes for any other device, the device is de-registered and
registered again as a new device.

Setting up Linux for suspend and resume

Configure suspend and resume support through kernel parameters, set up a
suitable swap partition for suspending and resuming a Linux instance, and update
your boot configuration.

Kernel parameters
You configure the suspend and resume support by adding parameters to the kernel
parameter line.

suspend and resume kernel parameter syntax

resume=<device_node>
no_console_suspend noresume

where:
resume=<device_node>
specifies the standard device node of the swap partition with the data that is
required for resuming the Linux instance.
This swap partition must be available during the boot process (see Updating
the boot configuration on page 78).
no_console_suspend
prevents Linux consoles from being suspended early in the suspend process.
Without this parameter, you cannot see the kernel messages that are issued by
the suspend process.
noresume
boots the kernel without resuming a previously suspended Linux instance.
Add this parameter to circumvent the resume process, for example, if the data
written by the previous suspend process is damaged.

Example

To use a partition /dev/disk/by-path/ccw-0.0.b100-part2 as the swap partition

and prevent Linux consoles from being suspended early in the suspend process
specify:
resume=/dev/disk/by-path/ccw-0.0.b100-part2 no_console_suspend

Chapter 6. Suspending and resuming 77

 Setting up a swap partition
During the suspend process, Linux writes data to a swap partition. This data is
required later to resume Linux.

Set up a swap partition that is at least the size of the available LPAR memory or
the memory of the z/VM guest virtual machine.

Do not use this swap partition for any other operating system that might run in
the LPAR or z/VM guest virtual machine while the Linux instance is suspended.

You cannot suspend a Linux instance while most of the memory and most of the
swap space are in use. If there is not sufficient remaining swap space to hold the
data for resuming the Linux instance, suspending the Linux instance fails.

To assure sufficient swap space you might have to configure two swap partitions,
one partition for regular swapping and another for suspending the Linux instance.
Configure the swap partition for suspending the Linux instance with a lower
priority than the regular swap partition.

Use the pri= parameter to specify the swap partitions in /etc/fstab with different
priorities. See the swapon man page for details.

The following example shows two swap partitions with different priorities:
# cat /etc/fstab
...
/dev/disk/by-path/ccw-0.0.b101-part1 swap swap pri=-1 0 0
/dev/disk/by-path/ccw-0.0.b100-part2 swap swap pri=-2 0 0

In the example, the partition to be used for the resume data is

/dev/disk/by-path/ccw-0.0.b100-part2.

You can check your current swap configuration by reading /proc/swaps.

# cat /proc/swaps
Filename Type Size Used Priority
/dev/disk/by-path/ccw-0.0.b101-part1 partition 7212136 71056 -1
/dev/disk/by-path/ccw-0.0.b100-part2 partition 7212136 0 -2

Updating the boot configuration

You have to update your boot configuration to include the kernel parameters that
are required for resuming Linux.

Procedure
Perform these steps to create a boot configuration that supports resuming your
Linux instance:
1. Run dracut -f to create an initial RAM disk with the module parameter that
identifies your device with the swap partition and with the device driver
required for this device.
2. Reboot your Linux instance.

78 Device Drivers, Features, and Commands on SLES 12 SP2

 Configuring for fast resume
The more devices are available to a Linux instance, the longer it takes to resume a
suspended instance.

With a thousand or more available devices, the resume process can take longer
than an IPL. If the duration of the resume process is critical for a Linux instance
with many devices, include unused devices in the exclusion list (see cio_ignore -
List devices to be ignored on page 670 and cio_ignore - Manage the I/O
exclusion list on page 514).

Suspending a Linux instance

Suspend a Linux instance by writing to the /sys/power/state sysfs attribute.

Before you begin

Attention: Only suspend a Linux instance for which you have specified the
resume= kernel parameter. Without this parameter, you cannot resume the
suspended Linux instance.

Procedure

Enter the following command to suspend a Linux instance:

# echo disk > /sys/power/state

Results

On the Linux console you might see progress indications until the console itself is
suspended. Most of these messages require log level 7 or higher to be printed. See
Using the magic sysrequest feature on page 47 about setting the log level. You
cannot see such progress messages if you suspend the Linux instance from an ssh
session.

Resuming a suspended Linux instance

Boot Linux to resume a suspended Linux instance.

About this task

Use the same kernel, initial RAM disk, and kernel parameters that you used to first
boot the suspended Linux instance.

You must reestablish any terminal session for HVC terminal devices and for
terminals that are provided by the iucvtty program. You also must reestablish all
ssh sessions that have timed out while the Linux instance was suspended.

If resuming the Linux instance fails, boot Linux again with the noresume kernel
parameter. The boot process then ignores the data that was written to the swap
partition and starts Linux without resuming the suspended instance.

Chapter 6. Suspending and resuming 79

80 Device Drivers, Features, and Commands on SLES 12 SP2
Chapter 7. Shutdown actions
Several triggers can cause Linux to shut down. For each shutdown trigger, you can
configure a specific shutdown action to be taken as a response.
Table 13. Shutdown triggers and default action overview
Default
Trigger Command or condition shutdown action
halt Linux chshut command stop
poff Linux poweroff or chshut command stop
reboot Linux reboot or chshut command reipl
restart v PSW restart on the HMC for Linux in LPAR mode stop
v z/VM CP system restart command for Linux on
z/VM
panic Linux dumpconf command stop

The available shutdown actions are summarized in Table 14.

Table 14. Shutdown actions
Action Explanation See also
stop For panic or restart, enters a disabled n/a
wait state. For all other shutdown
triggers, stops all CPUs.
ipl Performs an IPL according to the Displaying current IPL
specifications in /sys/firmware/ipl. parameters on page 68
reipl Performs an IPL according to the Rebooting from an alternative
specifications in /sys/firmware/reipl. source on page 70
dump Creates a dump according to the Using the Dump Tools on SUSE
specifications in /sys/firmware/dump. Linux Enterprise Server 12 SP1,
SC34-2746
dump_reipl Performs the dump action followed by the Using the Dump Tools on SUSE
reipl action. Linux Enterprise Server 12 SP1,
SC34-2746
vmcmd For Linux on z/VM, issues one or more Configuring z/VM CP
z/VM CP commands according to the commands as a shutdown
specifications in /sys/firmware/vmcmd. action on page 83

Use lsshut to find out which shutdown action is configured for each shutdown
trigger, see lsshut - List the current system shutdown actions on page 596.

Use the applicable command for setting the actions to be taken on shutdown:
v For halt, power off, and reboot use chshut, see chshut - Control the system
shutdown actions on page 503.
v For panic use dumpconf, see Using the Dump Tools on SUSE Linux Enterprise Server
12 SP1, SC34-2746

Copyright IBM Corp. 2000, 2016 81

 kdump for restart and panic

If kdump is set up for a Linux instance, kdump is started to create a dump,

regardless of the shutdown actions that are specified for restart and panic. With
kdump, these settings act as a backup that is used only if kdump fails.

Note: kdump is not a shutdown action that you can set as a sysfs attribute value
for a shutdown trigger. See Using the Dump Tools on SUSE Linux Enterprise Server 12
SP1, SC34-2746 about how to set up kdump.

The shutdown configuration in sysfs

The configured shutdown action for each shutdown trigger is stored in a sysfs
attribute /sys/firmware/shutdown_actions/on_<trigger>.

on_halt
on_poff
/sys/firmware shutdown_actions on_reboot
on_restart
on_panic

Figure 21. sysfs branch with shutdown action settings

The preferred way to read or change these settings is using the lsshut, chshut, and
dumpconf commands. Alternatively, you can read and write to the
/sys/firmware/shutdown_actions/on_<trigger> attributes.

Examples
v This command reads the shutdown setting for the poff shutdown trigger.
# cat /sys/firmware/shutdown_actions/on_poff
stop

v This command changes the setting for the restart shutdown trigger to ipl:
# echo ipl > /sys/firmware/shutdown_actions/on_restart

Details for the ipl, reipl, dump, and vmcmd shutdown actions are contained in the
corresponding subdirectories in /sys/firmware. For example, /sys/firmware/ipl
contains specifications for an IPL device and other IPL parameters.

82 Device Drivers, Features, and Commands on SLES 12 SP2

| Configuring z/VM CP commands as a shutdown action
| Use chshut and dumpconf to configure a CP command as a shutdown action, or
| directly write to the relevant sysfs attributes.

| Before you start: This information applies to Linux on z/VM only.

| In sysfs, two attributes are required to set a z/VM CP command as a shutdown

| action for a trigger <trigger>:
| v /sys/firmware/shutdown_actions/on_<trigger> must be set to vmcmd.
| v /sys/firmware/vmcmd/on_<trigger> specifies the z/VM CP command.

| The values of the attributes in the /sys/firmware/vmcmd directory must conform to

| these rules:
| v The value must be a valid z/VM CP command.
| v The commands, including any z/VM user IDs or device numbers, must be
| specified with uppercase characters.
| v Commands that include blanks must be delimited by double quotation marks
| (").
| v The value must not exceed 127 characters.

| You can specify multiple z/VM CP commands that are separated by the newline
| character \n. Each newline is counted as one character. When writing values
| with multiple commands, use this syntax to ensure that the newline character is
| processed correctly:
| # echo -e <cmd1>\n<cmd2>... | cat > /sys/firmware/vmcmd/on_<trigger>
|
|

| where <cmd1>\n<cmd2>... are two or more z/VM CP commands and on_<trigger>

| is one of the attributes in the vmcmd directory.

| The -e echo option and redirect through cat are required because of the newline
| character.

| Example for a single z/VM CP command

| Issue the following command to configure the z/VM CP LOGOFF command as the
| shutdown action for the poff shutdown trigger.
| # chshut poff vmcmd "LOGOFF"
|
|

| Alternatively, you can issue the following commands to directly write the
| shutdown configuration to sysfs:
| # echo vmcmd > /sys/firmware/shutdown_actions/on_poff
| # echo LOGOFF > /sys/firmware/vmcmd/on_poff
|
|

| Figure 22 on page 84 illustrates the relationship of the sysfs attributes for this
| example.
|

Chapter 7. Shutdown actions 83

|
on_halt
on_poff vmcmd
shutdown_actions on_reboot
on_restart
on_panic
/sys/firmware
on_halt
on_poff LOGOFF
vmcmd
devices on_reboot
on_restart
on_panic
|
| Figure 22. sysfs branch with shutdown action settings
|
| Example for multiple z/VM CP commands

| Issue the following command to configure two z/VM CP commands as the

| shutdown action for the poff shutdown trigger. First a message is sent to user
| OPERATOR, and then the LOGOFF command is issued.
| # chshut poff vmcmd "MSG OPERATOR Going down" vmcmd "LOGOFF"
|
|

| Alternatively, you can issue the following commands to directly write the
| shutdown configuration to sysfs:
| # echo vmcmd > /sys/firmware/shutdown_actions/on_poff
| # echo -e "MSG OPERATOR Going down\nLOGOFF" | cat > /sys/firmware/vmcmd/on_poff
|
|

84 Device Drivers, Features, and Commands on SLES 12 SP2

Chapter 8. Remotely controlling virtual hardware - snipl
snipl is a command line tool for remotely controlling virtual z Systems hardware.

This information applies to simple network IPL (snipl) version 2.3.0. A snipl
package is provided with SUSE Linux Enterprise Server 12 SP2.

You can use snipl to activate and deactivate virtual z Systems hardware with
Linux instances. You can set up a Linux instance on a mainframe system or on a
different hardware platform for running snipl.

snipl helps you to automate tasks that are typically performed by human
operators, for example, through the graphical interfaces of the HMC or SE.
Automation is required, for example, for failover setups within Linux clusters.

snipl can run in one of two modes, LPAR mode or z/VM mode.

Attention: snipl is intended for use by experienced system programmers and

administrators. Incautious use of snipl can result in unplanned downtime and loss
of data.

LPAR mode
In LPAR mode, snipl provides basic z Systems support element (SE) functions.

With snipl in LPAR mode, you can perform the following tasks:
v Activate, reset, or deactivate an LPAR.
v Load (IPL) an LPAR from a disk device, for example, a DASD device or a SCSI
device.
v Create a dump on a DASD or SCSI dump device.
v Send commands to the operating system and retrieve operating system
messages.

Setting up snipl for LPAR mode

The Linux instance where snipl runs requires access to all SEs that control LPARs
you want to work with.

snipl uses the hwmcaapi network management application programming

interfaces (API) provided by the SE. The API establishes an SNMP network
connection and uses the SNMP protocol to send and retrieve data. The libraries
that implement the API are available from IBM Resource Link at
www.ibm.com/servers/resourcelink.

Customize the API settings on the HMC or SE you want to connect to:
v Configure SNMP support.
v Add the IP address of the Linux instance where snipl runs and set the SNMP
community.
If the communication is through IPv6, an IPv6 community string must be set.
v In the firewall settings, ensure that UDP port 161 and TCP port 3161 are
enabled.

Copyright IBM Corp. 2000, 2016 85

 If snipl in LPAR mode repeatedly reports a timeout, the specified SE is most likely
inaccessible or not configured properly. For details about configuring the HMC or
SE, see the following publications:
v The Support Element Operations Guide for your mainframe system.
v The applicable Hardware Management Console Operations Guide.
v System z Application Programming Interfaces, SB10-7030
v S/390 Application Programming Interfaces, SC28-8141

You can obtain these publications from IBM Resource Link at www.ibm.com/
servers/resourcelink.

Command line syntax (LPAR mode)

There is a generic syntax with main options. Each main option has a specific set of
parameters.

Overview for LPAR mode summarizes snipl command in LPAR mode. Details
for each option are provided in context in the sections that follow.

Overview for LPAR mode

On the command line, a snipl command in LPAR mode always requires a main
option, access data, and , with one exception, specifications for one or more LPARs.

LPAR mode: overview

snipl

<image_name> lpar-access-data -a activate parameters

-d
-F
-r
-F
-o
-g
-l load parameters
-s SCSI parameters
-D
lpar-access-data -x list parameters
<image_name>
<image_name> lpar-access-data -i dialog parameters

Where:
<image_name>
specifies an LPAR. If snipl directly accesses the SE, this is the LPAR name as
defined in the hardware setup.
If snipl accesses the SE through an HMC, the specification has the format
<mainframe_system>-<lpar_name> where <mainframe_system> is the name that
identifies the mainframe on the HMC. If you are using a snipl configuration
file that defines an alias for an LPAR, you can specify the alias.

SE Example: lpar204

86 Device Drivers, Features, and Commands on SLES 12 SP2

 HMC Example: z02-lpar204
A snipl command applies to one or more LPARs that are controlled by the
same HMC or SE. If multiple LPARs are specified, it is assumed that all LPARs
are controlled by the same HMC or SE as the first LPAR. Other LPARs are
ignored.
|lpar-access-data|
is described in Specifying access data for LPAR mode.
-a, -d, -r, -o, -g
are described in Activate, deactivate, reset, stop, or get status information on
page 88.
-l is described in Perform an IPL operation from a CCW device on page 90.
-s, -D
are described in Perform an IPL or dump operation from a SCSI device on
page 91.
-x is described in List LPARs on page 93.
-i is described in Emulate the Operating Systems Messages applet on page 94.
-F or --force
unconditionally forces the operation.
-v or --version
displays the version of snipl and exits.
-h or --help
displays a short usage description and exits. To view the man page enter
man snipl.

Specifying access data for LPAR mode

The snipl command requires access data for the HMC or SE that controls a
particular LPAR.

lpar-access-data:

(1) -p public -f <defaultfile>

-L <ip_address>
-p <community> -f <filename>
-P

--timeout 60000

--timeout <timeout>

Notes:
1 -L can be omitted if the required information is specified through a
configuration file.

-L <ip_address> or --lparserver <ip_address>

specifies the IP address or host name of the HMC or SE that controls the LPAR
| or LPARs you want to work with. You can use IPv6 or IPv4 connections.

Chapter 8. snipl - control virtual hardware 87

 You can omit this parameter if the IP address or host name is specified through
a configuration file.
-p <community> or --password <community>
specifies the password in the SNMP configuration settings on the SE that
controls the LPAR or LPARs you want to work with. This parameter can also
be specified through a configuration file. The default password is public.

Note: The default password feature is deprecated and will be removed in a

subsequent release.
-P or --promptpassword
prompts for a password in protected entry mode.
-f <filename> or --configfilename <filename>
specifies the name of a configuration file that maps LPARs to the
corresponding specifications for the HMC or SE address and password
(community).
If no configuration file is specified, the user-specific default file ~/.snipl.conf
is used. If this file does not exist, the system default file /etc/snipl.conf is
used.
Be sure that the command-line parameters you provide uniquely identify the
configuration-file section you want to work with. If you specify multiple
LPARs on the command line, only the first specification is used to identify the
section. If your specifications map to multiple sections, the first match is
processed.
If conflicting specifications are provided through the command line and the
configuration file, the command-line specification is used.
If a configuration file is neither specified nor available at the default locations,
all required parameters must be specified on the command line.
For more information about the configuration file, see The snipl configuration
file on page 99.
--timeout <timeout>
specifies the timeout in milliseconds for general management API calls. The
default is 60000 ms.

Activate, deactivate, reset, stop, or get status information

Several main options follow a simple command syntax that requires specifications
for one or more LPARs and the corresponding access data.

88 Device Drivers, Features, and Commands on SLES 12 SP2

 LPAR mode: -a, -d, -r, -o, -g options

snipl <image_name>

(1)
--profilename <defaultprofile>
lpar-access-data -a
-F --profilename <filename>
-d
-F
-r
-F
-o
-g

Notes:
1 If not specified, the HMC or SE default profile for the specified LPAR is
used.

Where:
<image_name>
see Overview for LPAR mode on page 86.
|lpar-access-data|
see Specifying access data for LPAR mode on page 87.
-a or --activate
activates the specified LPARs.
--profilename <filename>
specifies an activation profile. If omitted, the SE or an HMC default profile for
the specified LPAR is used.
-d or --deactivate
deactivates the specified LPARs.
-r or --reset
resets the specified LPARs.
-o or --stop
stops all CPUs for the specified LPARs.
-g or --getstatus
returns the status for the specified LPARs.
-F or --force
unconditionally forces the operation.

Examples
v The following command deactivates an LPAR SZ01LP02 with the force option:
# snipl SZ01LP02 -L 192.0.2.4 -P -d -F
Enter password:
Warning : No default configuration file could be found/opened.
processing......
SZ01LP02: acknowledged.

v The following command retrieves the status for an LPAR SZ01LP03:

Chapter 8. snipl - control virtual hardware 89

 # snipl SZ01LP03 -L 192.0.2.4 -P -g
Enter password:
Warning : No default configuration file could be found/opened.
status of sz01lp03: operating

| v The following command retrieves the status for an LPAR SZ02LP03 on a

| mainframe system that is identified as SZ02 on an HMC with an IP address
| 2001:0db8::11a0:
| # snipl SZ02-SZ02LP03 -L 2001:0db8::11a0 -P -g
| Enter password:
| Warning : No default configuration file could be found/opened.
| status of SZ02-SZ02LP033: operating
|
|

Perform an IPL operation from a CCW device

To IPL an LPAR from a CCW device, snipl requires specifications for the LPAR,
the corresponding access data, and the IPL device. There are also several optional
parameters.

For IPL from a SCSI device, see Perform an IPL or dump operation from a SCSI
device on page 91.

LPAR mode: IPL from CCW

snipl <image_name> lpar-access-data -l

-F

-A <load_address> --parameters_load <string>

--load_timeout 60

--load_timeout <timeout> --noclear --storestatus

Where:
<image_name>
specifies the LPARs for which to perform the IPL. If multiple LPARs are
specified, the same IPL device and IPL parameters are used for all of them. See
also Overview for LPAR mode on page 86.
|lpar-access-data|
see Specifying access data for LPAR mode on page 87.
-l or --load
performs an IPL for the specified LPARs.
-F or --force
unconditionally forces the IPL operation.

90 Device Drivers, Features, and Commands on SLES 12 SP2

 -A <loadaddress> or --address_load <loadaddress>
| specifies the hexadecimal four-digit device number of the IPL device. To use a
| device from a subchannel set other than 0, specify five digits: The subchannel
| set ID followed by the device number, for example 15199. The default is
| subchannel set 0. If the - A parameter is omitted, the IPL device of the most
| recent IPL of the LPAR is used.
--parameters_load <string>
specifies a parameter string for IPL. If this parameter is omitted, the string of
the most recent IPL of the LPAR is used.
--load_timeout <timeout>
specifies the maximum time for load completion in seconds. The timeout must
be in the range of 60 - 600 seconds. The default timeout is 60 seconds.
If the timeout expires, control is returned without an indication about the
success of the IPL operation.
--noclear
prevents the memory from being cleared before loading.
--storestatus
stores status before performing the IPL. This option implies --noclear and also
prevents the main memory from being cleared before loading.

Examples:
| v The following command performs an IPL from a CCW device with bus ID
| 0.0.5119 for an LPAR SZ02LP03 on a mainframe system that is identified as SZ02
| on an HMC with an IP address 2001:0db8::11a0:
| # snipl SZ02-SZ02LP03 -L 2001:0db8::11a0 -P -l -A 5119
| Enter password:
| Warning : No default configuration file could be found/opened.
| processing......
| SZ02-SZ02LP03: acknowledged.
|
|
| v To perform an IPL from a CCW device in subchannel set 1 with the bus ID
| 0.1.5119 for an LPAR SZ03LP00:
| % snipl SZ03LP00 -L 192.0.2.4 -P -l -A 15119
|
|

Perform an IPL or dump operation from a SCSI device

To IPL an LPAR from a SCSI device, snipl requires specifications for the LPAR, the
corresponding access data, the IPL device, target WWPN, and LUN. There are also
several optional parameters.

For IPL from a CCW device, see Perform an IPL operation from a CCW device
on page 90.

Chapter 8. snipl - control virtual hardware 91

 LPAR mode: SCSI IPL or dump

snipl <image_name> lpar-access-data -s

-D -F

-A <load_address> --parameters_load <string>

--wwpn_scsiload <portname> --lun_scsiload <unitnumber>

--bps_scsiload <selector> --ossparms_scsiload <string>

--bootrecord_scsiload <hexaddress>

Where:
<image_name>
specifies the LPARs for which to perform the IPL or dump operation. If
multiple LPARs are specified, the same command parameters apply to all of
them. See also Overview for LPAR mode on page 86.
|lpar-access-data|
see Specifying access data for LPAR mode on page 87.
-s or --scsiload
performs an IPL from a SCSI device for the specified LPARs.
-D or --scsidump
creates a dump for the specified LPAR to a SCSI device.
-F or --force
unconditionally forces the operation.
-A <loadaddress> or --address_load <loadaddress>
specifies the hexadecimal four-digit device number of the IPL device. If this
parameter is omitted, the IPL device of the most recent SCSI IPL of the LPAR
is used.

| Note: The IPL device must be on subchannel set 0.

--parameters_load <string>
specifies a parameter string for IPL. If this parameter is omitted, the string of
the most recent SCSI IPL of the LPAR is used.
--wwpn_scsiload <portname>
specifies the worldwide port name (WWPN) for the SCSI IPL device. If fewer

92 Device Drivers, Features, and Commands on SLES 12 SP2

 than 16 characters are specified, the WWPN is padded with zeroes at the end.
If this parameter is omitted, the WWPN of the most recent SCSI IPL of the
LPAR is used.
--lun_scsiload <unitnumber>
specifies the logical unit number (LUN) for the SCSI IPL device. If fewer than
16 characters are specified, the LUN is padded with zeroes at the end. If this
parameter is omitted, the LUN of the most recent SCSI IPL of the LPAR is
used.
--bps_scsiload <selector>
specifies the boot program that is required for the SCSI IPL device. Selector
values are in the range 0 - 30. If this parameter is omitted, the boot program of
the most recent SCSI IPL of the LPAR is used.
--ossparms_scsiload <string>
specifies an operating system-specific parameter string for IPL from a SCSI
device. If this parameter is omitted, the string of the most recent SCSI IPL of
the LPAR is used. This parameter string is ignored by the boot program and
passed to the operating system or dump program to be loaded. For example,
you can specify additional kernel parameters for Linux (see Adding kernel
parameters when booting Linux on page 24).
--bootrecord_scsiload <hexaddress>
specifies the boot record logical block address for the SCSI IPL device. If fewer
than 16 characters are specified, the address is padded with zeroes at the end.
If this parameter is omitted, the address of the most recent SCSI IPL of the
LPAR is used.

Example: The following command performs a SCSI IPL for an LPAR SZ01LP00:
# snipl SZ01LP00 -L 192.0.2.4 -P -s -A 3d0f --wwpn_scsiload 500507630303c562 \
--lun_scsiload 4010404900000000
Enter password:
Warning : No default configuration file could be found/opened.
processing...
SZ01LP00: acknowledged.

Note: Instead of using the continuation sign (\) at the end of the first line, you can
specify the complete command on a single line.

List LPARs
To list all LPARs that are controlled by an HMC or SE, snipl requires specifications
for the HMC or SE and the corresponding access data.

Use the -x option to list all LPARs of a z Systems mainframe.

LPAR mode: list

snipl lpar-access-data -x
<image_name>

Where:

Chapter 8. snipl - control virtual hardware 93

 <image_name>
specifies an LPAR to identify a section in the snipl configuration file. Omit this
parameter if an HMC or SE is specified with the -L option (see Overview for
LPAR mode on page 86).
|lpar-access-data|
see Specifying access data for LPAR mode on page 87.
-x or --listimages
retrieves a list of all LPARs from the specified HMC or SE. If an HMC is
specified, all LPARs for all managed mainframe systems are listed.

Example: The following command lists the LPARs for an SE with IP address
192.0.2.4:
# snipl -L 192.0.2.4 -P -x
Enter password:
Warning : No default configuration file could be found/opened.

available images for server 192.0.2.4 :

SZ01LP00 SZ01LP01 SZ01LP02 SZ01LP03

Emulate the Operating Systems Messages applet

To emulate the HMC or SE Operating Systems Messages applet, snipl requires
specifications for the LPAR and the corresponding access data. There are also
optional parameters.

Use the -i option to start an emulation of the HMC or SE Operating Systems

Messages applet for a specified LPAR. End the emulation with CTRL+D.

LPAR mode: dialog

snipl <image_name> lpar-access-data -i

--msgtimeout 5000

--msgtimeout <interval> --msgfilename <name>

Where:
<image_name>
specifies the LPAR for which you want to emulate the HMC or SE Operating
Systems Messages applet (see also Overview for LPAR mode on page 86).
|lpar-access-data|
see Specifying access data for LPAR mode on page 87.
-i or --dialog
starts an emulation of the HMC or SE Operating System Message applet for
the specified LPAR.
--msgtimeout <interval>
specifies the timeout for retrieving operating system messages in milliseconds.
The default value is 5000 ms.

94 Device Drivers, Features, and Commands on SLES 12 SP2

 -M <name> or --msgfilename <name>
specifies a file to which the operating system messages are written in addition
to stdout. If no file is specified, the operating system messages are written to
stdout only.

Example: The following command opens an emulation of the SE Operating

Systems Messages applet with the operating system instance that runs on LPAR
SZ01LP02. During the emulation session, the operating system messages are written
to a file, SZ01LP02.transcript.
# snipl SZ01LP02 -L 192.0.2.4 -P -i -M SZ01LP02.transcript
Enter password:
Warning : No default configuration file could be found/opened.
processing......
...

z/VM mode
With snipl in z/VM mode, you can log on, reset, or log off a z/VM guest virtual
machine.

Setting up snipl for z/VM mode

The Linux instance where snipl runs requires access to the systems management
API of all z/VM systems that host z/VM guest virtual machines you want to work
with.

snipl in z/VM mode uses the systems management application programming

interfaces (APIs) of z/VM. How snipl communicates with the API on the z/VM
system depends on your z/VM system version and on your system setup.

If snipl in z/VM mode repeatedly reports RPC: Port mapper failure - RPC timed
out, it is most likely that the z/VM system is inaccessible, or not set up correctly.
Although only one of the communication methods uses RPC, this method is the
fallback method that is tried if the other method fails.

Using a SMAPI request server

snipl can access the systems management API through a SMAPI request server.
The following configuration is required for the z/VM systems you want to work
with:
v An AF_INET based SMAPI request server must be configured.
v A port on which the request server listens must be set up.
v A z/VM user ID to be specified with the snipl command must be set up. This
user ID must be authorized for the request server.

For more information, see z/VM Systems Management Application Programming,

SC24-6234.

Using a VSMSERVE service machine

snipl can access the systems management API through a VSMSERVE service
machine on your z/VM system. The following configuration is required for the
z/VM systems you want to work with:
v The VSMSERVE service machine must be configured and authorized for the
directory manager.

Chapter 8. snipl - control virtual hardware 95

 v The vsmapi service must be registered.
v A z/VM user ID to be specified with the snipl command must be set up. This
user ID must be authorized for VSMSERVE.

For more information, see z/VM Systems Management Application Programming,

SC24-6122-02 or earlier.

Command line syntax (z/VM mode)

In z/VM mode, the snipl command requires specification for a guest virtual
machine, credentials, and other access data for the systems management API.
There are also several optional parameters.

96 Device Drivers, Features, and Commands on SLES 12 SP2

 snipl command syntax ( z/VM mode)

snipl <guest_id>

zvm-access-data -a
-X 300
-d
-X <maxperiod>
-F
-r
-g
-x

zvm-access-data:

-V <ip_address>
(1)
-z <portnumber>

(2)
-f <defaultfile>
-u <user_id> -p <password>
-P -f <filename>

--timeout 60000

--timeout <timeout>

Notes:
1 Required for connections through a SMAPI request server, unless the port
is specified through a configuration file.
2 -V, -u, and -p can be omitted if the required data is specified through a
configuration file.

Where:
<guest_id>
specifies the z/VM guest virtual machine you want to work with. Specify
multiple z/VM user IDs to perform the same action for multiple z/VM guest
virtual machines.
If you are using a snipl configuration file that defines an alias for a z/VM
guest virtual machine, you can specify the alias.
You can omit this parameter for the -x option if other specifications on the
command line identify a section in the configuration file.
-V <ip_address> or --vmserver <ip_address>
specifies the IP address or host name of the SMAPI request server or

Chapter 8. snipl - control virtual hardware 97

 VSMSERVE service machine through which the specified z/VM guest virtual
| machines are controlled. You can use IPv6 or IPv4 connections.
This option can be omitted if defined in the configuration file.
-z <portnumber> or --port <portnumber>
specifies the port at which the SMAPI request server listens.
-u <user_id> or --userid <user_id>
specifies a z/VM user ID that is authorized to access the SMAPI request server
or VSMSERVE service machine. This option can be omitted if defined in the
configuration file.
-p <password> or --password <password>
specifies the password for the z/VM user ID specified with --userid. This
option can be omitted if defined in the configuration file.
-P or --promptpassword
prompts for a password in protected entry mode.
-f <filename> or --configfilename <filename>
specifies the name of a configuration file that maps z/VM guest virtual
machines to the corresponding specifications for the SMAPI request server or
VSMSERVE service machine, the authorized z/VM user ID, and the password.
If no configuration file is specified, the user-specific default file ~/.snipl.conf
is used. If this file does not exist, the system default file /etc/snipl.conf is
used.
Be sure that the command line parameters you provide uniquely identify the
configuration-file section you want to work with. If you specify multiple z/VM
guest virtual machines on the command line, only the first specification is used
to identify the section. If your specifications map to multiple sections, the first
match is processed.
If conflicting specifications are provided through the command line and the
configuration file, the command line specification is used. If no configuration
file is used, all required parameters must be specified on the command line.
For more information about the configuration file, see The snipl configuration
file on page 99.
--timeout <timeout>
specifies the timeout in milliseconds for general management API calls. The
default is 60000 ms.
-a or --activate
logs on the specified z/VM guest virtual machines.
-d or --deactivate
logs off the specified z/VM guest virtual machines.
-X <maxperiod> or --shutdowntime <maxperiod>
specifies the maximum period, in seconds, granted for graceful completion
before CP FORCE commands are issued against the specified z/VM guest
virtual machines. By default, the maximum period is 300 s.
-F or --force
immediately issues CP FORCE commands to log off the specified z/VM guest
virtual machines. This parameter is equivalent to -X 0.
-r or --reset
logs off the specified z/VM guest virtual machines and then logs them back
on.

98 Device Drivers, Features, and Commands on SLES 12 SP2

 -g or --getstatus
returns the status for the specified z/VM guest virtual machines.
-x or --listimages
lists the z/VM guest virtual machines as specified in a configuration-file
section (see The snipl configuration file). You can identify the configuration
file section with the -V parameter, by specifying a z/VM guest virtual
machine, or by specifying a z/VM guest virtual machine and the -u parameter.
-v or --version
displays the version of snipl and exits.
-h or --help
displays a short usage description and exits. To view the man page enter
man snipl.

Examples
v The following command logs on two z/VM guest virtual machines:
# snipl sndlnx04 sndlnx05 -V sandbox.www.example.com -z 44444 -u sndadm01 -p pw42play -a
Warning : No default configuration file could be found/opened.
* ImageActivate : Image sndlnx04 Request Successful
* ImageActivate : Image sndlnx05 Request Successful

| v The following command logs off a z/VM guest virtual machine:

| # snipl vm04lnxd -V 2001:0db8::1a:0015 -z 77899 -u vm04main -p mainpw -a
| Warning : No default configuration file could be found/opened.
| processing......
| * ImageActivate : Image vm04lnxd Request Successful
|
|

The snipl configuration file

Use the snipl configuration file to provide parameter values to snipl instead of
specifying all values on the command line.

See Specifying access data for LPAR mode on page 87 or Command line syntax
(z/VM mode) on page 96 about how to include a configuration file when issuing
a snipl command.

A snipl configuration file contains one or more sections. Each section consists of
multiple lines with specifications of the form <keyword>=<value> for either a z/VM
system or an SE.

The following rules apply to the configuration file:

v Lines that begin with a number sign (#) are comment lines. A number sign in
the middle of a line makes the remaining line a comment.
v Empty lines are permitted.
v The specifications are not case-sensitive.
v The same configuration file can contain sections for snipl in both LPAR mode
and z/VM mode.
v In a <keyword>=<value> pair, one or more blanks are allowed before or after the
equal sign (=).

Table 15 on page 100 summarizes the keywords for the configuration file and the
command -line equivalents for LPAR mode and z/VM mode.

Chapter 8. snipl - control virtual hardware 99

 Table 15. snipl configuration file keywords
Command-line
Keyword Value for LPAR mode Value for z/VM mode equivalent
server Starts a configuration file section by Starts a configuration file section by (See note 1)
specifying the IP address or host specifying the IP address or host name of
(required) name of an HMC or SE. a SMAPI request server or VSMSERVE
service machine.
| You can use IPv6 or IPv4 connections.
| You can use IPv6 or IPv4 connections.
type LPAR VM (See note 1)

(required)
user n/a A z/VM user ID that is authorized for -u or --user
the SMAPI request server or VSMSERVE
(See note 2) service machine.
password The value for community in the SNMP The password for the z/VM user ID -p or
settings of the SE. specified with the user keyword. --password
(See note 3)
If not specified through either the (See note 2)
configuration file or the command,
the default, public, is used.
port n/a Required if the server keyword specifies -z or --port
the IP address or host name of a SMAPI
request server.
image An LPAR name as defined in the A z/VM user ID that specifies a target A list of one or
mainframe hardware configuration. z/VM guest virtual machine. more items that
A valid section are separated by
must have one If the server keyword specifies an You can define an alias name for the blanks and
or more lines HMC, the specification begins with z/VM user ID by appending a forward specified
with this the name that identifies the slash (/) to the ID and specifying the without a
keyword. mainframe on the HMC, followed by alias after the slash. switch.
a hyphen (-), followed by the LPAR
name.

You can define an alias name for the

LPAR by appending a forward slash
(/) to the LPAR name and specifying
the alias after the slash.

Note:
1. Jointly, the server and type keywords are equivalent to the command-line
option -L for LPAR mode or to -V for z/VM mode.
2. Can be omitted and specified on the command line instead.
3. Do not include passwords in the snipl configuration file unless the security
policy at your installation permits you to do so.

Figure 23 on page 101 shows a configuration file example with multiple sections,
including sections for LPAR mode and for z/VM mode.

100 Device Drivers, Features, and Commands on SLES 12 SP2

 # z/VM system for Linux training sessions
server = sandbox.www.example.com
type = VM
password = pw42play
port = 44444
user = sndadm01
image = sndlnx01
image = sndlnx02
image = sndlnx03/tutor
image = sndlnx04
image = sndlnx05
image = sndcms01/c1

# SE for production SZ01

Server=192.0.2.4
type=LPAR
image=SZ01LP00
image=SZ01LP01
image=SZ01LP02
image=SZ01LP03

# HMC for test SZ02

| Server = 2001:0db8::11a0
type=LPAR
image=Z02-SZ02LP00/Z0200
image=Z02-SZ02LP01
image=Z02-SZ02LP02
image=Z02-SZ02LP03

| # Production VM 04 - uses SMAPI

| server = 2001:0db8::1a:0015
| type = VM
| port = 77899
| user = VM04MAIN
| image = VM04LNXA
| image = VM04LNXC
| image = VM04LNXD

# Production VM 05 - uses VSMSERVE so no port

server = 192.0.2.20
type = VM
user = VM05MAIN
image = VM05G001
image = VM05G002
image = VM05G003
image = VM05G004

Figure 23. Example of a snipl configuration file

Examples
The examples that follow assume that the configuration file of Figure 23 is used.
v The following command logs on two z/VM guest virtual machines, sndlnx01
and sndlnx03 (with alias tutor). In the example, the command output shows
that sndlnx03 is already logged on.
# snipl sndlnx01 sndlnx03 -V sandbox.www.example.com -z 44444 -u sndadm01 -p pw42play -a
Warning : No default configuration file could be found/opened.
* ImageActivate : Image sndlnx01 Request Successful
* ImageActivate : Image sndlnx03 Image Already Active

Assuming that the configuration file of Figure 23 is available at /etc/xcfg, an

equivalent command would be:

Chapter 8. snipl - control virtual hardware 101

 # snipl sndlnx01 tutor -a -f /etc/xcfg
Server sandbox.www.example.com from config file /etc/xcfg is used
* ImageActivate : Image sndlnx01 Request Successful
* ImageActivate : Image sndlnx03 Image Already Active

Assuming that the configuration file of Figure 23 on page 101 is used by default,
an equivalent command would be:
# snipl sndlnx01 tutor -a
Server sandbox.www.example.com from config file /etc/snipl.conf is used
* ImageActivate : Image sndlnx01 Request Successful
* ImageActivate : Image sndlnx03 Image Already Active

v The following command performs an IPL for an LPAR SZ01LP03:

# snipl SZ01LP03 -L 192.0.2.4 -l -P -A 5000
Enter password:
Warning : No default configuration file could be found/opened.
processing......
SZ01LP03: acknowledged.

Assuming that the configuration file of Figure 23 on page 101 is available at

/etc/xcfg, an equivalent command would be:
# snipl SZ01LP03 -l -P -A 5000 -f /etc/xcfg
Enter password:
Server 192.0.2.4 from config file /etc/xcfg is used
SZ01LP03: acknowledged.

Assuming that the configuration file of Figure 23 on page 101 is used by default,
an equivalent command would be:
# snipl SZ01LP03 -l -P -A 5000
Enter password:
Server 192.0.2.4 from config file /etc/snipl.conf is used
SZ01LP03: acknowledged.

v Assuming that the configuration file of Figure 23 on page 101 is available at

/etc/xcfg, the following command lists the z/VM guest virtual machines as
specified in the section for sandbox.www.example.com:
# snipl -V sandbox.www.example.com -f /etc/xcfg -x
available images for server sandbox.www.example.com and userid SNDADM01 :

sndlnx01 sndlnx02 sndlnx03 sndlnx04

sndlnx05 sndcms01

| v The following command logs off a z/VM guest virtual machine:

| # snipl vm04lnxd -V 2001:0db8::1a:0015 -z 77899 -u vm04main -p mainpw -a
| Warning : No default configuration file could be found/opened.
| processing......
| * ImageActivate : Image vm04lnxd Request Successful
|
|
| Assuming that the configuration file of Figure 23 on page 101 is used by default,
| an equivalent command would be:
| # snipl vm04lnxd -d
| Enter password:
| Server 2001:0db8::1a:0015 from config file /etc/snipl.conf is used
| processing......
| vm04lnxd: acknowledged.
|
|

102 Device Drivers, Features, and Commands on SLES 12 SP2

Connection errors and return codes
You might receive error indications from snipl or from the SE.

snipl return codes

Successful snipl commands return 0. If an error occurs, snipl writes a short
message to stderr and completes with a return code other than 0.

The following return codes indicate snipl syntax errors or specifications that are
not valid:
1 An unknown command option was specified.
2 A command option with an invalid value was specified.
3 A command option was specified more than once.
4 Conflicting command options was specified.
5 No command option was specified.
6 No SE, HMC, SMAPI request server or VSMSERVE service machine was
specified on the command line or through a configuration file.
7 No LPAR or z/VM guest virtual machine was specified.
8 No z/VM user ID was specified on the command line or through a
configuration file.
9 No password was specified on the command line or through a
configuration file.
10 A specified LPAR or z/VM guest virtual machine does not exist on the
specified SE or z/VM system.
22 More than one LPAR was specified for option --dialog.

The following return codes indicate setup errors or program errors:

30 An error occurred while loading one of the systems management API
libraries libhwmcaapi.so or libvmsmapi.so.
40 Operation --dialog encounters a problem while starting another process.
41 Operation --dialog encounters a problem with stdin attribute setting.
50 A response from the HMC or SE could not be interpreted.
60 The response buffer is too small for a response from the HMC or SE.
90 A storage allocation failure occurred.
99 A program error occurred.

Connection errors
If a connection error occurs (for example, a timeout), snipl sends a message to
stderr.

To recover connection errors, try again to issue the command. If the problem
persists, a networking failure is most likely. In this case, increase the timeout value.

Chapter 8. snipl - control virtual hardware 103

 Return codes from the SE
Error messages from the SE have the format
<LPAR_name>: <message> - rc is <rc>

In the message, <rc> is a return code from the network management application
programming interfaces (HWMCAAPI) on the SE.

Example
LPARLNX1: not acknowledged command was not successful rc is 135921664

To interpret these return codes, see System z Application Programming Interfaces,

SB10-7030. You can obtain this publication from IBM Resource Link at
www.ibm.com/servers/resourcelink.

STONITH support (snipl for STONITH)

The STONITH implementation is part of the Heartbeat framework of the High
Availability Project.

STONITH is usually used as part of this framework but can also be used
independently. snipl provides a plug-in to STONITH.

For a general description of the STONITH technology go to linux-ha.org.

Before you begin

v STONITH requires a configuration file that maps LPARs and z/VM guest virtual
machines to the specifications for the corresponding SE, HMC or z/VM system.
The snipl for STONITH configuration file has the same syntax as the snipl
configuration file, see The snipl configuration file on page 99.
v The SEs, HMCs and z/VM systems you want to work with must be set up as
described in Setting up snipl for LPAR mode on page 85 and Setting up snipl
for z/VM mode on page 95.

Using stonith

When using stonith commands for Linux on z/VM or for Linux in LPAR mode
you must provide <keyword>=<value> pairs as described in The snipl
configuration file on page 99. There are two ways to specify this information:
v On the command line with the stonith command, using the -p option and the
snipl_parm keyword.
v Through a configuration file, using the -p option and the snipl_file keyword.

Unlike snipl, you must specify all parameters in the same way; all parameters on
the command line or all parameters in the configuration file.

104 Device Drivers, Features, and Commands on SLES 12 SP2

 stonith syntax (simplified)

stonith -t lic_vps -p "snipl_param <parameters>"

"snipl_file <file>"

-T on <image>
off
reset

Where:
-t lic_vps
specifies the server type. For STONITH with snipl, the server type is always
lic_vps.
-p specifies parameters.
snipl_param <parameters>
specifies comma-separated <keyword>=<value> pairs with the same keywords
as used in the configuration file (see The snipl configuration file on page 99).
For LPAR mode the following keywords are required:
v server
v type
v password
v image
For z/VM mode the following keywords are required:
v server
v port (required if the z/VM system is configured with a SMAPI request
server rather than a VSMSERVE service machine)
v type
v user
v password
v image
snipl_file <parameters>
specifies a configuration file (see The snipl configuration file on page 99).
The configuration file must contain all required keywords, including the
password. The configuration file must always be specified explicitly. No file is
used by default.
-T specifies the action to be performed.
-on
activates the specified LPAR or logs on the specified z/VM virtual machine.
-off
deactivates the specified LPAR or logs off the specified z/VM virtual machine.
-reset
resets the specified LPAR or z/VM virtual machine.

Chapter 8. snipl - control virtual hardware 105

 <image>
specifies the LPAR or z/VM virtual machine you want to work with. If you
use the snipl_param parameter, the contained image keyword must specify the
same LPAR or z/VM virtual machine.

For more information, see the stonith man page.

Examples
v This example command resets the z/VM guest virtual machine sndlnx04:
# stonith -t lic_vps -p "snipl_param server=sandbox.www.example.com,type=vm\
,user=sndadm01,password=pw42play,image=sndlnx04" -T reset sndlnx04

Note: Instead of using the continuation sign (\) at the end of the first line, you
can specify the complete command on a single line.
v With /etc/xcfg as shown in Example of a snipl configuration file, the following
command is equivalent:
# stonith -t lic_vps -p "snipl_file /etc/xcfg" -T reset sndlnx04

106 Device Drivers, Features, and Commands on SLES 12 SP2

Part 3. Storage
Chapter 9. DASD device driver . . . . . . . 109 What you should know about storage-class
Features . . . . . . . . . . . . . . . 109 memory . . . . . . . . . . . . . . . 187
What you should know about DASD . . . . . 110 Setting up the storage-class memory device driver 188
Setting up the DASD device driver . . . . . . 119 Working with storage-class memory increments 188
Working with DASDs . . . . . . . . . . 122
Chapter 12. Channel-attached tape device driver 191
Chapter 10. SCSI-over-Fibre Channel device Features . . . . . . . . . . . . . . . 191
driver. . . . . . . . . . . . . . . . 145 What you should know about channel-attached
Features . . . . . . . . . . . . . . .145 tape devices . . . . . . . . . . . . . . 191
What you should know about zfcp . . . . . .145 Loading the tape device driver . . . . . . . 194
Setting up the zfcp device driver . . . . . . .151 Working with tape devices . . . . . . . . . 194
Working with FCP devices . . . . . . . . .153
Working with target ports . . . . . . . . .160 Chapter 13. XPRAM device driver . . . . . . 201
Working with SCSI devices . . . . . . . . .167 XPRAM features . . . . . . . . . . . . 201
Confirming end-to-end data consistency checking 180 What you should know about XPRAM . . . . . 201
Scenario for finding available LUNs . . . . . . 182 Setting up the XPRAM device driver . . . . . 202
zfcp HBA API support . . . . . . . . . . 183

Chapter 11. Storage-class memory device driver

supporting Flash Express . . . . . . . . 187

There are several z Systems specific storage device drivers for SUSE Linux
Enterprise Server 12 SP2 for z Systems.

Newest version

You can find the newest version of this publication on IBM Knowledge Center at
www.ibm.com/support/knowledgecenter/linuxonibm/liaaf/lnz_r_suse.html

Restrictions

For prerequisites and restrictions see the z Systems architecture specific

information in the SUSE Linux Enterprise Server 12 SP2 release notes at
www.suse.com/releasenotes

Copyright IBM Corp. 2000, 2016 107

108 Device Drivers, Features, and Commands on SLES 12 SP2
Chapter 9. DASD device driver
The DASD device driver provides access to all real or emulated direct access
storage devices (DASD) that can be attached to the channel subsystem of an IBM
mainframe.

DASD devices include various physical media on which data is organized in

blocks or records or both. The blocks or records in a DASD can be accessed for
read or write in random order.

Traditional DASD devices are attached to a control unit that is connected to a

mainframe I/O channel. Today, these real DASD have been largely replaced by
emulated DASDs. For example, such emulated DASDs can be the volumes of the
IBM System Storage DS8000 Turbo, or the volumes of the IBM System Storage
DS6000. These emulated DASD are completely virtual and the identity of the
physical device is hidden.

SCSI disks that are attached through an FCP channel are not classified as DASD.
They are handled by the zfcp driver (see Chapter 10, SCSI-over-Fibre Channel
device driver, on page 145).

Features
The DASD device driver supports a wide range of disk devices and disk functions.
v The DASD device driver has no dependencies on the adapter hardware that is
used to physically connect the DASDs to the z Systems hardware. You can use
any adapter that is supported by the z Systems hardware (see
www.ibm.com/systems/z/connectivity for more information).
v The DASD device driver supports ESS virtual ECKD type disks
v The DASD device driver supports the control unit attached physical ECKD
(Extended Count Key Data) and FBA (Fixed Block Access) devices as
summarized in Table 16:
Table 16. Supported control unit attached DASD
Device format Control unit type Device type
ECKD 1750 3380 and 3390
ECKD 2107 3380 and 3390
ECKD 2105 3380 and 3390
ECKD 3990 3380 and 3390
ECKD 9343 9345
ECKD 3880 3390
FBA 6310 9336
FBA 3880 3370

All models of the specified control units and device types can be used with the
DASD device driver. This includes large devices with more than 65520 cylinders,
for example, 3390 Model A. Check the storage support statement to find out
what works for SUSE Linux Enterprise Server 12 SP2.

Copyright IBM Corp. 2000, 2016 109

 v The DASD device driver provides a disk format with up to three partitions per
disk. See z Systems compatible disk layout on page 111 for details.
v The DASD device driver provides an option for extended error reporting for
ECKD devices. Extended error reporting can support high availability setups.
v The DASD device driver supports parallel access volume (PAV) and HyperPAV
on storage devices that provide this feature. The DASD device driver handles
dynamic PAV alias changes on storage devices. For more information about PAV
and HyperPAV, see How to Improve Performance with PAV, SC33-8414. Use the
dasdstat command to check whether a DASD uses PAV, see Scenario: Verifying
that PAV and HPF are used on page 136.
v The DASD device driver supports High Performance FICON, including
multitrack requests, on storage devices that provide this feature. Use the
dasdstat command to check whether a DASD uses High Performance FICON,
see Scenario: Verifying that PAV and HPF are used on page 136.

What you should know about DASD

The DASD device driver supports various disk layouts with different partitioning
capabilities. The DASD device naming scheme helps you to keep track of your
DASDs and DASD device nodes.

The IBM label partitioning scheme

Linux on z Systems supports the same standard DASD format that is also used by
traditional mainframe operating systems, but it also supports any other Linux
partition table.

The DASD device driver is embedded into the Linux generic support for
partitioned disks. As a result, you can use any partition table format that is
supported by Linux for your DASDs.

Traditional mainframe operating systems (such as, z/OS, z/VM, and z/VSE)
expect a standard DASD format. In particular, the format of the first two tracks of
a DASD is defined by this standard. These tracks include the z Systems IPL, label,
and for some layouts VTOC records. Partitioning schemes for platforms other than
z Systems generally do not preserve these mainframe specific records.

SUSE Linux Enterprise Server 12 SP2 for z Systems includes the IBM label
partitioning scheme that preserves the z Systems IPL, label, and VTOC records.
With this partitioning scheme, Linux can share a disk with other mainframe
operating systems. For example, a traditional mainframe operating system can
handle backup and restore for a partition that is used by Linux.

The following sections describe the layouts that are supported by the IBM label
partitioning scheme:
v z Systems compatible disk layout on page 111
v Linux disk layout on page 114
v CMS disk layout on page 114

DASD partitions
Partitioning DASD has the same advantages as for other disk types, but there are
some prerequisites and a special tool, fdasd.

110 Device Drivers, Features, and Commands on SLES 12 SP2

 A DASD partition is a contiguous set of DASD blocks that is treated by Linux as
an independent disk and by the traditional mainframe operating systems as a data
set.

With the Linux disk layout (LDL) and the CMS disk layout, you always have a
single partition only. This partition is defined by the LDL or CMS formatted area
of the disk. With the compatible disk layout, you can have up to three partitions.

There are several reasons why you might want to have multiple partitions on a
DASD, for example:
Limit data growth
Runaway processes or undisciplined users can consume disk space to an
extend that the operating system runs short of space for essential
operations. Partitions can help to isolate the space that is available to
particular processes.
Encapsulate your data
If a file system gets damaged, this damage is likely to be restricted to a
single partition. Partitioning can reduce the scope of data damage.

Recommendations
v Use fdasd to create or alter partitions on ECKD type DASD that are formatted
with the compatible disk layout. If you use another partition editor, it is your
responsibility to ensure that partitions do not overlap. If they do, data damage
occurs.
v Leave no gaps between adjacent partitions to avoid wasting space. Gaps are not
reported as errors, and can be reclaimed only by deleting and re-creating one or
more of the surrounding partitions and rebuilding the file system on them.

A disk need not be partitioned completely. You can begin by creating only one or
two partitions at the start of your disk and convert the remaining space to a
partition later.

There is no facility for moving, enlarging, or reducing partitions, because fdasd has
no control over the file system on the partition. You can only delete and re-create
them. Changing the partition table results in loss of data in all altered partitions. It
is up to you to preserve the data by copying it to another medium.

z Systems compatible disk layout

With the compatible disk layout, a DASD can have up to three partitions that can
be accessed by traditional mainframe operating systems.

You can format only ECKD type DASD with the compatible disk layout.

Figure 24 illustrates a DASD with the compatible disk layout.

Figure 24. Compatible disk layout

Chapter 9. DASD 111

 The IPL records, volume label (VOL1), and VTOC of disks with the compatible
disk layout are on the first two tracks of the disks. These tracks are not intended
for use by Linux applications. Using the tracks can result in data loss.

Linux can address the device as a whole as /dev/dasd<x>, where <x> can be one to
four letters that identify the individual DASD (see DASD naming scheme on
page 115). See DASD device nodes on page 116 for alternative addressing
possibilities.

Disks with the compatible disk layout can have one to three partitions. Linux
addresses the first partition as /dev/dasd<x>1, the second as /dev/dasd<x>2, and
the third as /dev/dasd<x>3.

You use the dasdfmt command (see dasdfmt - Format a DASD on page 535) to
format a disk with the compatible disk layout. You use the fdasd command (see
fdasd Partition a DASD on page 552) to create and modify partitions.

Volume label
The volume label includes information about the disk layout, the VOLSER, and a
pointer to the VTOC.

The DASD volume label is located in the third block of the first track of the device
(cylinder 0, track 0, block 2). This block has a 4-byte key, and an 80-byte data area
with the following content:
key for disks with the compatible disk layout, contains the four EBCDIC
characters VOL1 to identify the block as a volume label.
label identifier
is identical to the key field.
VOLSER
is a name that you can use to identify the DASD device. A volume serial
number (VOLSER) can be one to six EBCDIC characters. If you want to use
VOLSERs as identifiers for your DASD, be sure to assign unique VOLSERs.
You can assign VOLSERs from Linux by using the dasdfmt or fdasd
command. These commands enforce that VOLSERs:
v Are alphanumeric
v Are uppercase (by uppercase conversion)
v Contain no embedded blanks
v Contain no special characters other than $, #, @, and %

Tip: Avoid special characters altogether.

Note: The VOLSER values SCRTCH, PRIVAT, MIGRAT, or Lnnnnn (An L

followed by 5 digits) are reserved for special purposes by other mainframe
operating systems and should not be used by Linux.
These rules are more restrictive than the VOLSERs that are allowed by the
traditional mainframe operating systems. For compatibility, Linux tolerates
existing VOLSERs with lowercase letters and special characters other than
$, #, @, and %. Enclose VOLSERs with special characters in single
quotation marks if you must specify it, for example, as a command
parameter.

112 Device Drivers, Features, and Commands on SLES 12 SP2

VTOC address
contains the address of a standard IBM format 4 data set control block
(DSCB). The format is: cylinder (2 bytes) track (2 bytes) block (1 byte).

All other fields of the volume label contain EBCDIC space characters (code 0x40).

VTOC
Instead of a regular Linux partition table, Linux on z Systems, like other
mainframe operating systems, uses a Volume Table Of Contents (VTOC).

The VTOC contains pointers to the location of every data set on the volume. These
data sets form the Linux partitions.

The VTOC is on the second track (cylinder 0, track 1). It contains a number of
labels, each written in a separate block:
v One format 4 DSCB that describes the VTOC itself
v One format 5 DSCB
The format 5 DSCB is required by other operating systems but is not used by
Linux. fdasd sets it to zeros.
v For volumes with more than 65636 tracks, 1 format 7 DSCB following the format
5 DSCB
v For volumes with more than 65520 cylinders (982800 tracks), 1 format 8 DSCB
following the format 5 DSCB
v A format 1 DSCB for each partition
The key of the format 1 DSCB contains the data set name, which identifies the
partition to z/OS, z/VM or z/VSE.

The VTOC can be displayed with standard z Systems tools such as VM/DITTO. A
Linux DASD with physical device number 0x0193, volume label LNX001, and
three partitions might be displayed like this example:
VM/DITTO DISPLAY VTOC LINE 1 OF 5
===> SCROLL ===> PAGE

CUU,193 ,VOLSER,LNX001 3390, WITH 100 CYLS, 15 TRKS/CYL, 58786 BYTES/TRK

--- FILE NAME --- (SORTED BY =,NAME ,) ---- EXT BEGIN-END RELTRK,
1...5...10...15...20...25...30...35...40.... SQ CYL-HD CYL-HD NUMTRKS
*** VTOC EXTENT *** 0 0 1 0 1 1,1
LINUX.VLNX001.PART0001.NATIVE 0 0 2 46 11 2,700
LINUX.VLNX001.PART0002.NATIVE 0 46 12 66 11 702,300
LINUX.VLNX001.PART0003.NATIVE 0 66 12 99 14 1002,498
*** THIS VOLUME IS CURRENTLY 100 PER CENT FULL WITH 0 TRACKS AVAILABLE

PF 1=HELP 2=TOP 3=END 4=BROWSE 5=BOTTOM 6=LOCATE

PF 7=UP 8=DOWN 9=PRINT 10=RGT/LEFT 11=UPDATE 12=RETRIEVE

The ls command on Linux might list this DASD and its partitions like this
example:
# ls -l /dev/dasda*
brw-rw---- 1 root disk 94, 0 Jan 27 09:04 /dev/dasda
brw-rw---- 1 root disk 94, 1 Jan 27 09:04 /dev/dasda1
brw-rw---- 1 root disk 94, 2 Jan 27 09:04 /dev/dasda2
brw-rw---- 1 root disk 94, 3 Jan 27 09:04 /dev/dasda3

where dasda represent the whole DASD and dasda1, dasda2, and dasda3 represent
the individual partitions.

Chapter 9. DASD 113

 Linux disk layout
The Linux disk layout does not have a VTOC, and DASD partitions that are
formatted with this layout cannot be accessed by traditional mainframe operating
systems.

You can format only ECKD type DASD with the Linux disk layout. Apart from
accessing the disks as ECKD devices, you can also access them using the DASD
DIAG access method. See Enabling the DASD device driver to use the DIAG
access method on page 126 for how to enable DIAG.

Figure 25 illustrates a disk with the Linux disk layout.

Figure 25. Linux disk layout

DASDs with the Linux disk layout either have an LNX1 label or are not labeled.
The first records of the device are reserved for IPL records and the volume label,
and are not intended for use by Linux applications. All remaining records are
grouped into a single partition. You cannot have more than a single partition on a
DASD that is formatted in the Linux disk layout.

Linux can address the device as a whole as /dev/dasd<x>, where <x> can be one to
four letters that identify the individual DASD (see DASD naming scheme on
page 115). Linux can access the partition as /dev/dasd<x>1.

You use the dasdfmt command (see dasdfmt - Format a DASD on page 535) to
format a disk with the Linux disk layout.

CMS disk layout

The CMS disk layout applies only to Linux on z/VM. The disks are formatted with
z/VM tools.

Both ECKD or FBA type DASD can have the CMS disk layout. DASD partitions
that are formatted with this layout cannot be accessed by traditional mainframe
operating systems. Apart from accessing the disks as ECKD or FBA devices, you
can also access them using the DASD DIAG access method.

Figure 26 on page 115 illustrates two variants of the CMS disk layout.

114 Device Drivers, Features, and Commands on SLES 12 SP2

 Figure 26. CMS disk layout

The first variant contains IPL records, a volume label (CMS1), and a CMS data
area. Linux treats DASD like this equivalent to a DASD with the Linux disk layout,
where the CMS data area serves as the Linux partition.

The second variant is a CMS reserved volume. In this variant, the DASD was
reserved by a CMS RESERVE fn ft fm command. In addition to the IPL records and
the volume label, DASD with the CMS disk layout also have CMS metadata. The
CMS reserved file serves as the Linux partition.

For both variants of the CMS disk layout, you can have only a single Linux
partition. The IPL record, volume label and (where applicable) the CMS metadata,
are not intended for use by Linux applications.

Addressing the device and partition is the same for both variants. Linux can
address the device as a whole as /dev/dasd<x>, where <x> can be one to four
letters that identify the individual DASD (see DASD naming scheme). Linux can
access the partition as /dev/dasd<x>1.

Enabling the DASD device driver to use the DIAG access method on page 126
describes how to enable DIAG.

Disk layout summary

The available disk layouts differ in their support of device formats, the DASD
DIAG access method, and the maximum number of partitions.
Table 17. Disk layout summary
Disk layout ECKD FBA DIAG access Maximum Formatting
device device method number of tool
format format support partitions
(z/VM only)
Compatible disk Yes No No 3 dasdfmt
layout
Linux disk layout Yes No Yes 1 dasdfmt
CMS (z/VM only) Yes Yes Yes 1 z/VM tools

DASD naming scheme

The DASD naming scheme maps device names and minor numbers to whole
DASDs and to partitions.

Chapter 9. DASD 115

 The DASD device driver uses the major number 94. For each configured device it
uses four minor numbers:
v The first minor number always represents the device as a whole, including IPL,
VTOC, and label records.
v The remaining three minor numbers represent the up to three partitions.

With 1,048,576 (20-bit) available minor numbers, the DASD device driver can
address 262,144 devices.

The DASD device driver uses a device name of the form dasd<x> for each DASD.
In the name, <x> is one to four lowercase letters. Table 18 shows how the device
names map to the available minor numbers.
Table 18. Mapping of DASD names to minor numbers
Name for device as a whole Minor number for device as a Number of
whole devices
From To From To
dasda dasdz 0 100 26
dasdaa dasdzz 104 2804 676
dasdaaa dasdzzz 2808 73108 17,576
dasdaaaa dasdnwtl 73112 1048572 243,866
Total number of devices: 262,144

The DASD device driver also uses a device name for each partition. The name of
the partition is the name of the device as a whole with a 1, 2, or 3 appended to
identify the first, second, or third partition. The three minor numbers that follow
the minor number of the device as a whole are the minor number for the first,
second, and third partition.

Examples
v dasda refers to the whole of the first disk in the system and dasda1,
dasda2, and dasda3 to the three partitions. The minor number for the whole
device is 0. The minor numbers of the partitions are 1, 2, and 3.
v dasdz refers to the whole of the 101st disk in the system and dasdz1,
dasdz2, and dasdz3 to the three partitions. The minor number for the whole
device is 100. The minor numbers of the partitions are 101, 102, and 103.
v dasdaa refers to the whole of the 102nd disk in the system and dasdaa1,
dasdaa2, and dasdaa3 to the three partitions. The minor number for the
whole device is 104. The minor numbers of the partitions are 105, 106, and 107.

DASD device nodes

SUSE Linux Enterprise Server 12 SP2 uses udev to create multiple device nodes for
each DASD that is online.
Device nodes that are based on device names
udev creates device nodes that match the device names that are used by
the kernel. These standard device nodes have the form /dev/<name>.

The mapping between standard device nodes and the associated physical disk
space can change, for example, when you reboot Linux. To ensure that you access
the intended physical disk space, you need device nodes that are based on
properties that identify a particular DASD.

116 Device Drivers, Features, and Commands on SLES 12 SP2

udev creates additional devices nodes that are based on the following information:
v The bus ID of the disk
v The disk label (VOLSER)
v The universally unique identifier (UUID) of the file system on the disk
v If available: The label of the file system on the disk
Device nodes that are based on bus IDs
udev creates device nodes of the form
/dev/disk/by-path/ccw-<device_bus_id>

for whole DASD and

/dev/disk/by-path/ccw-<device_bus_id>-part<n>

for the <n>th partition.

Device nodes that are based on VOLSERs
udev creates device nodes of the form
/dev/disk/by-id/ccw-<volser>

for whole DASD and

/dev/disk/by-id/ccw-<volser>-part<n>

for the <n>th partition.

If you want to use device nodes that are based on VOLSER, be sure that
the VOLSERs in your environment are unique (see Volume label on page
112).
If you assign the same VOLSER to multiple devices, Linux can still access
each device through its standard device node. However, only one of the
devices can be accessed through the VOLSER-based device node. Thus, the
node is ambiguous and might lead to unintentional data access.
Furthermore, if the VOLSER on the device that is addressed by the node is
changed, the previously hidden device is not automatically addressed
instead. To reassign the node, you must reboot Linux or force the kernel to
reread the partition tables from disks, for example, by issuing:
# blockdev --rereadpt /dev/dasdzzz

You can assign VOLSERs to ECKD type devices with dasdfmt when
formatting or later with fdasd when creating partitions.
Device nodes that are based on file system information
udev creates device nodes of the form
/dev/disk/by-uuid/<uuid>

where <uuid> is the UUID for the file system in a partition.

If a file system label exists, udev also creates a node of the form:
/dev/disk/by-label/<label>
There are no device nodes for the whole DASD that are based on file
system information.
If you want to use device nodes that are based on file system labels, be
sure that the labels in your environment are unique.

Chapter 9. DASD 117

 Additional device nodes
/dev/disk/by-id contains additional device nodes for the DASD and
partitions, that are all based on a device identifier as contained in the uid
attribute of the DASD.

Note: If you want to use device nodes that are based on file system information
and VOLSER, be sure that they are unique for the scope of your Linux instance.
This information can be changed by a user or it can be copied, for example when
backup disks are created. If two disks with the same VOLSER or UUID are online
to the same Linux instance, the matching device node can point to either of these
disks.

Example

For a DASD that is assigned the device name dasdzzz, has two partitions, a device
bus-ID 0.0.b100 (device number 0xb100), VOLSER LNX001, and a UUID
6dd6c43d-a792-412f-a651-0031e631caed for the first and f45e955d-741a-4cf3-86b1-
380ee5177ac3 for the second partition, udev creates the following device nodes:

For the whole DASD:

v /dev/dasdzzz (standard device node according to the DASD naming scheme)
v /dev/disk/by-path/ccw-0.0.b100
v /dev/disk/by-id/ccw-LNX001

For the first partition:

v /dev/dasdzzz1 (standard device node according to the DASD naming scheme)
v /dev/disk/by-path/ccw-0.0.b100-part1
v /dev/disk/by-id/ccw-LNX001-part1
v /dev/disk/by-uuid/6dd6c43d-a792-412f-a651-0031e631caed

For the second partition:

v /dev/dasdzzz2 (standard device node according to the DASD naming scheme)
v /dev/disk/by-path/ccw-0.0.b100-part2
v /dev/disk/by-id/ccw-LNX001-part2
v /dev/disk/by-uuid/f45e955d-741a-4cf3-86b1-380ee5177ac3

Accessing DASD by udev-created device nodes

Use udev-created device nodes to access a particular physical disk space,
regardless of the device name that is assigned to it.

Example

The following example is based on these assumptions:

v A DASD with bus ID 0.0.b100 has two partitions.
v The standard device node of the DASD is dasdzzz.
v udev creates the following device nodes for a DASD and its partitions:
/dev/disk/by-path/ccw-0.0.b100
/dev/disk/by-path/ccw-0.0.b100-part1
/dev/disk/by-path/ccw-0.0.b100-part2

Instead of issuing:

118 Device Drivers, Features, and Commands on SLES 12 SP2

 # fdasd /dev/dasdzzz

issue:
# fdasd /dev/disk/by-path/ccw-0.0.b100

In the file system information in /etc/fstab replace the following specifications:

/dev/dasdzzz1 /temp1 btrfs defaults 0 0
/dev/dasdzzz2 /temp2 btrfs defaults 0 0

with these specifications:

/dev/disk/by-path/ccw-0.0.b100-part1 /temp1 btrfs defaults 0 0
/dev/disk/by-path/ccw-0.0.b100-part2 /temp2 btrfs defaults 0 0

You can make similar substitutions with other device nodes that udev provides for
you (see DASD device nodes on page 116).

Setting up the DASD device driver

Unless the DASD device driver modules are loaded for you during the boot
process, load and configure them with the modprobe command.

In most cases, SUSE Linux Enterprise Server 12 SP2 loads the DASD device driver
for you during the boot process. You can then use YaST to set the diag attribute. If
the DASD device driver is loaded for you and you must set attributes other than
diag, see Module parameters on page 26.

DASD module parameter syntax

eer_pages=5
modprobe dasd_mod
, eer_pages=<pages>

dasd= device-spec
autodetect
probeonly
nopav
nofcx
dasd_eckd_mod
dasd_fba_mod
dasd_diag_mod

device-spec:

<device_bus_id>
<from_device_bus_id>-<to_device_bus_id> :

( ro )
diag
erplog
failfast

Where:

Chapter 9. DASD 119

 dasd_mod
loads the device driver base module.
When you are loading the base module, you can specify the dasd=
parameter.
You can use the eer_pages parameter to determine the number of pages
that are used for internal buffering of error records.
autodetect
causes the DASD device driver to allocate device names and the
corresponding minor numbers to all DASD devices and set them online
during the boot process. See DASD naming scheme on page 115 for the
naming scheme.
The device names are assigned in order of ascending subchannel numbers.
Auto-detection can yield confusing results if you change your I/O
configuration and reboot, or if your Linux instance runs as a z/VM guest
because the devices might appear with different names and minor
numbers after rebooting.
probeonly
causes the DASD device driver to reject any open syscall with EPERM.
autodetect,probeonly
causes the DASD device driver to assign device names and minor numbers
as for auto-detect. All devices regardless of whether they are accessible as
DASD return EPERM to any open requests.
nopav suppresses parallel access volume (PAV and HyperPAV) enablement for
Linux instances that run in LPAR mode. The nopav keyword has no effect
for Linux on z/VM.
nofcx suppresses accessing the storage server with the I/O subsystem in
transport mode (also known as High Performance FICON).
<device_bus_id>
specifies a single DASD.
<from_device_bus_id>-<to_device_bus_id>
specifies the first and last DASD in a range. All DASD devices with bus
IDs in the range are selected. The device bus-IDs <from_device_bus_id> and
<to_device_bus_id> need not correspond to actual DASD.
(ro) accesses the specified device or device range in read-only mode.
(diag) forces the device driver to access the device (range) with the DIAG access
method.
(erplog)
enables enhanced error recovery processing (ERP) related logging through
syslogd. If erplog is specified for a range of devices, the logging is
switched on during device initialization.
(failfast)
immediately returns failed for an I/O operation when the last path to a
DASD is lost.

Attention: Enable immediate failure of I/O requests only in setups where

a failed I/O request can be recovered outside the scope of a single DASD
(see Enabling and disabling immediate failure of I/O requests on page
130).

120 Device Drivers, Features, and Commands on SLES 12 SP2

dasd_eckd_mod
loads the ECKD module.
dasd_fba_mod
loads the FBA module.
dasd_diag_mod
loads the DIAG module.

If you supply a DASD module parameter with device specifications

dasd=<device-list1>,<device-list2> ..., the device names and minor numbers
are assigned in the order in which the devices are specified. The names and
corresponding minor numbers are always assigned, even if the device is not
present, or not accessible. For information about including device specifications in
a boot configuration, see Including module parameters in a boot configuration
on page 27.

If you use autodetect in addition to explicit device specifications, device names

are assigned to the specified devices first and device-specific parameters, like ro,
are honored. The remaining devices are handled as described for autodetect.

The DASD base component is required by the other modules. Be sure that it is
loaded first. modprobe takes care of this dependency for you and ensures that the
base module is loaded automatically, if necessary.

Hint: modprobe might return before udev has created all device nodes for the
specified DASDs. If you must assure that all nodes are present, for example in
scripts, follow the modprobe command with:
# udevadm settle

For command details see the modprobe man page.

Example
modprobe dasd_mod dasd=0.0.7000-0.0.7002,0.0.7005(ro),0.0.7006

Table 19 shows the resulting allocation of device names:

Table 19. Example mapping of device names to devices
Name To access

dasda device 0.0.7000 as a whole

dasda1 the first partition on 0.0.7000
dasda2 the second partition on 0.0.7000
dasda3 the third partition on 0.0.7000

dasdb device 0.0.7001 as a whole

dasdb1 the first partition on 0.0.7001
dasdb2 the second partition on 0.0.7001
dasdb3 the third partition on 0.0.7001

dasdc device 0.0.7002 as a whole

dasdc1 the first partition on 0.0.7002
dasdc2 the second partition on 0.0.7002
dasdc3 the third partition on 0.0.7002

dasdd device 0.0.7005 as a whole

Chapter 9. DASD 121

 Table 19. Example mapping of device names to devices (continued)
Name To access

dasdd1 the first partition on 0.0.7005 (read-only)

dasdd2 the second partition on 0.0.7005 (read-only)
dasdd3 the third partition on 0.0.7005 (read-only)

dasde device 0.0.7006 as a whole

dasde1 the first partition on 0.0.7006
dasde2 the second partition on 0.0.7006
dasde3 the third partition on 0.0.7006

Including the nofcx parameter suppresses High Performance FICON for all DASD:
modprobe dasd_mod dasd=nofcx,0.0.7000-0.0.7002,0.0.7005(ro),0.0.7006

Working with DASDs

You might have to prepare DASDs for use, configure troubleshooting functions, or
configure special device features for your DASDs.

See Working with newly available devices on page 10 to avoid errors when you
are working with devices that have become available to a running Linux instance.
v Preparing an ECKD type DASD for use
v Preparing an FBA-type DASD for use on page 124
v Accessing DASD by force on page 125
v Enabling the DASD device driver to use the DIAG access method on page 126
v Using extended error reporting for ECKD type DASD on page 127
v Setting a DASD online or offline on page 128
v Enabling and disabling logging on page 130
v Enabling and disabling immediate failure of I/O requests on page 130
v Setting the timeout for I/O requests on page 131
v Working with DASD statistics in debugfs on page 132
v Accessing full ECKD tracks on page 137
v Handling lost device reservations on page 139
v Reading and resetting the reservation state on page 140
v Displaying DASD information on page 141

Preparing an ECKD type DASD for use

Before you can use an ECKD type DASD as a Linux on z Systems disk, you must
format it with a suitable disk layout and create a file system or define a swap
space.

Before you begin

v The modules for the base component and the ECKD component of the DASD
device driver must have been loaded.
v The DASD device driver must have recognized the device as an ECKD type
device.
v You must know the device bus-ID for your DASD.

122 Device Drivers, Features, and Commands on SLES 12 SP2

About this task

If you format the DASD with the compatible disk layout, you need to create one,
two, or three partitions. You can then use your partitions as swap areas or to create
a Linux file system.

Procedure
Perform these steps to prepare the DASD:
1. Issue lsdasd (see lsdasd - List DASD devices on page 584) to find out if the
device is online. If necessary, set the device online using chccwdev (see
chccwdev - Set CCW device attributes on page 492).

Example:
# chccwdev -e 0.0.b100

2. Format the device with the dasdfmt command (see dasdfmt - Format a DASD
on page 535 for details). The formatting process can take hours for large
DASDs. If you want to use the CMS disk layout, and your DASD is already
formatted with the CMS disk layout, skip this step.

Tips:
v Use the largest possible block size, ideally 4096; the net capacity of an ECKD
DASD decreases for smaller block sizes. For example, a DASD formatted
with a block size of 512 byte has only half of the net capacity of the same
DASD formatted with a block size of 4096 byte.
v Use the -p option to display a progress bar.

Example: Assuming that /dev/dasdzzz is a valid device node for 0.0.b100:

# dasdfmt -b 4096 -p /dev/dasdzzz

3. Proceed according to your chosen disk layout:

v If you have formatted your DASD with the Linux disk layout or the CMS
disk layout, skip this step and continue with step 4. You already have one
partition and cannot add further partitions on your DASD.
v If you have formatted your DASD with the compatible disk layout use the
fdasd command to create up to three partitions (see fdasd Partition a
DASD on page 552 for details).

Example: To start the partitioning tool in interactive mode for partitioning a

device /dev/dasdzzz issue:
# fdasd /dev/dasdzzz

If you create three partitions for a DASD /dev/dasdzzz, the device nodes for
the partitions are /dev/dasdzzz1, /dev/dasdzzz2, and /dev/dasdzzz3.

Result: fdasd creates the partitions and updates the partition table (see
VTOC on page 113).
4. Depending on the intended use of each partition, create a file system on the
partition or define it as a swap space.

Chapter 9. DASD 123

 v Either create a file system of your choice, for example, with the Linux mke2fs
command (see the man page for details).

Restriction: You must not make the block size of the file system smaller than
the block size that was used for formatting the disk with the dasdfmt
command.

Tip: Use the same block size for the file system that was used for formatting.

Example:
# mke2fs -j -b 4096 /dev/dasdzzz1

v Or define the partition as a swap space with the mkswap command (see the
man page for details).
5. Mount each file system to the mount point of your choice in Linux and enable
your swap partitions.

Example: To mount a file system in a partition /dev/dasdzzz1 to a mount point

/mnt and to enable a swap partition /dev/dasdzzz2 issue:
# mount /dev/dasdzzz1 /mnt
# swapon /dev/dasdzzz2

If a block device supports barrier requests, journaling file systems like ext3 or
raiser-fs can use this feature to achieve better performance and data integrity.
Barrier requests are supported for the DASD device driver and apply to ECKD,
FBA, and the DIAG discipline.
Write barriers are used by file systems and are enabled as a file-system specific
option. For example, barrier support can be enabled for an ext3 file system by
mounting it with the option -o barrier=1:
# mount -o barrier=1 /dev/dasdzzz1 /mnt

Preparing an FBA-type DASD for use

Before you can use an FBA-type DASD as a Linux on z Systems disk, you must
create a file system or define a swap space.

Before you begin

v The modules for the base component and the FBA component of the DASD
device driver must have been loaded.
v The DASD device driver must have recognized the device as an FBA device.
v You need to know the device bus-ID or the device node through which the
DASD can be addressed.

About this task

Note: To access FBA devices, use the DIAG access method (see Enabling the
DASD device driver to use the DIAG access method on page 126 for more
information).

Perform these steps to prepare the DASD:

124 Device Drivers, Features, and Commands on SLES 12 SP2

 Procedure
1. Depending on the intended use of the partition, create a file system on it or
define it as a swap space.
v Either create a file system, for example, with the Linux mke2fs command (see
the man page for details).

Example:
# mke2fs -b 4096 /dev/dasdzzy1

v Or define the partition as a swap space with the mkswap command (see the
man page for details).
2. Mount the file system to the mount point of your choice in Linux or enable
your swap partition.

Example: To mount a file system in a partition /dev/dasdzzy1 issue:

# mount /dev/dasdzzy1 /mnt

Accessing DASD by force

A Linux instance can encounter DASDs that are locked by another system.

Such a DASD is referred to as externally locked or boxed. The Linux instance

cannot analyze a DASD while it is externally locked.

About this task

To check whether a DASD has been externally locked, read its availability attribute.
This attribute should be good. If it is boxed, the DASD has been externally
locked. Because a boxed DASD might not be recognized as DASD, it might not
show up in the device driver view in sysfs. If necessary, use the device category
view instead (see Device views in sysfs on page 11).

CAUTION:
Breaking an external lock can have unpredictable effects on the system that
holds the lock.

Procedure
1. Optional: To read the availability attribute of a DASD, issue a command of this
form:
# cat /sys/bus/ccw/devices/<device_bus_id>/availability

Example: This example shows that a DASD with device bus-ID 0.0.b110 (device
number 0xb110) has been externally locked.
# cat /sys/bus/ccw/devices/0.0.b110/availability
boxed

If the DASD is an ECKD type DASD and if you know the device bus-ID, you
can break the external lock and set the device online. This means that the lock
of the external system is broken with the unconditional reserve channel
command.

Chapter 9. DASD 125

 2. To force a boxed DASD online, write force to the online device attribute. Issue
a command of this form:
# echo force > /sys/bus/ccw/devices/<device_bus_id>/online

Example: To force a DASD with device number 0xb110 online issue:

# echo force > /sys/bus/ccw/devices/0.0.b110/online

Results

If the external lock is successfully broken or if the lock has been surrendered by
the time the command is processed, the device is analyzed and set online. If it is
not possible to break the external lock (for example, because of a timeout, or
because it is an FBA-type DASD), the device remains in the boxed state. This
command might take some time to complete.

For information about breaking the look of a DASD that has already been analyzed
see tunedasd - Adjust low-level DASD settings on page 647.

Enabling the DASD device driver to use the DIAG access

method
Linux on z/VM can use the DIAG access method to access DASDs with the help
of z/VM functions.

Before you begin

This section only applies to Linux instances and DASD for which all of the
following are true:
v The Linux instance runs as a z/VM guest.
v The device can be of type ECKD with either LDL or CMS disk layout, or it can
be a device of type FBA.
v The module for the DIAG component must be loaded.
v The module for the component that corresponds to the DASD type
(dasd_eckd_mod or dasd_fba_mod) must be loaded.
v The DASD is offline.
v The DASD does not represent a parallel access volume alias device.

About this task

You can use the DIAG access method to access both ECKD and FBA-type DASD.
You use the device's use_diag sysfs attribute to enable or switch off the DIAG
access method in a system that is online. Set the use_diag attribute to 1 to enable
the DIAG access method. Set the use_diag attribute to 0 to switch off the DIAG
access method (this is the default).

Alternatively, you can specify diag on the command line, for example during IPL,
to force the device driver to access the device (range) with the DIAG access
method.

126 Device Drivers, Features, and Commands on SLES 12 SP2

 Procedure

Issue a command of this form:

# echo <flag> > /sys/bus/ccw/devices/<device_bus_id>/use_diag

where <device_bus_id> identifies the DASD.

If the DIAG access method is not available and you set the use_diag attribute to 1,
you cannot set the device online (see Setting a DASD online or offline on page
128).

Note: When switching between an enabled and a disabled DIAG access method on
FBA-type DASD, first reinitialize the DASD, for example, with CMS format or by
overwriting any previous content. Switching without initialization might cause
data-integrity problems.
For more details about DIAG see z/VM CP Programming Services, SC24-6179.

Example

In this example, the DIAG access method is enabled for a DASD with device
number 0xb100.
1. Ensure that the driver is loaded:
# modprobe dasd_diag_mod

2. Identify the sysfs CCW-device directory for the device in question and change
to that directory:
# cd /sys/bus/ccw/devices/0.0.b100/

3. Ensure that the device is offline:

# echo 0 > online

4. Enable the DIAG access method for this device by writing '1' to the use_diag
sysfs attribute:
# echo 1 > use_diag

5. Use the online attribute to set the device online:

# echo 1 > online

Using extended error reporting for ECKD type DASD

Control the extended error reporting feature for individual ECKD type DASD
through the eer_enabled sysfs attribute. Use the character device of the extended
error reporting module to obtain error records.

Before you begin

To use the extended error reporting feature, you need ECKD type DASD.

About this task

The extended error reporting feature is turned off by default.

Chapter 9. DASD 127

 Procedure

To enable extended error reporting, issue a command of this form:

# echo 1 > /sys/bus/ccw/devices/<device_bus_id>/eer_enabled

where /sys/bus/ccw/devices/<device_bus_id> represents the device in sysfs.

When it is enabled on a device, a specific set of errors generates records and might
have further side effects.
To disable extended error reporting, issue a command of this form:
# echo 0 > /sys/bus/ccw/devices/<device_bus_id>/eer_enabled

What to do next

You can obtain error records for all DASD for which extended error reporting is
enabled from the character device of the extended error reporting module,
/dev/dasd_eer. The device supports these file operations:
open
Multiple processes can open the node concurrently. Each process that opens the
node has access to the records that are created from the time the node is
opened. A process cannot access records that were created before the process
opened the node.
close
You can close the node as usual.
read
Blocking read and non-blocking read are supported. When a record is partially
read and then purged, the next read returns an I/O error -EIO.
poll
The poll operation is typically used with non-blocking read.

Setting a DASD online or offline

Use the chccwdev command or the online sysfs attribute of the device to set
DASDs online or offline.

About this task

When Linux boots, it senses your DASD. Depending on your specification for the
dasd= parameter, it automatically sets devices online.

Procedure
Use the chccwdev command (chccwdev - Set CCW device attributes on page 492)
to set a DASD online or offline.
Alternatively, you can write 1 to the device's online attribute to set it online or 0 to
set it offline. In contrast to the sysfs attribute, the chccwdev command triggers a
cio_settle for you and waits for the cio_settle to complete.
Outstanding I/O requests are canceled when you set a device offline. To wait
indefinitely for outstanding I/O requests to complete before setting the device
offline, use the chccwdev option --safeoffline or the sysfs attribute safe_offline.

128 Device Drivers, Features, and Commands on SLES 12 SP2

When you set a DASD offline, the deregistration process is synchronous, unless the
device is disconnected. For disconnected devices, the deregistration process is
asynchronous.

Examples
v To set a DASD with device bus-ID 0.0.b100 online, issue:
# chccwdev -e 0.0.b100

or
# echo 1 > /sys/bus/ccw/devices/0.0.b100/online

v To set a DASD with device bus-ID 0.0.b100 offline, issue:

# chccwdev -d 0.0.b100

or
# echo 0 > /sys/bus/ccw/devices/0.0.b100/online

v To complete outstanding I/O requests and then set a DASD with device bus-ID
0.0.4711 offline, issue:
# chccwdev -s 0.0.4711

or
# echo 1 > /sys/bus/ccw/devices/0.0.4711/safe_offline

If an outstanding I/O request is blocked, the command might wait forever.

Reasons for blocked I/O requests include reserved devices that can be released
or disconnected devices that can be reconnected.
1. Try to resolve the problem that blocks the I/O request and wait for the
command to complete.
2. If you cannot resolve the problem, issue chccwdev -d to cancel the
outstanding I/O requests. The data is lost.

Dynamic attach and detach

You can dynamically attach devices to a running SUSE Linux Enterprise Server 12
SP2 for z Systems instance, for example, from z/VM.

When a DASD is attached, Linux attempts to initialize it according to the DASD

device driver configuration. You can then set the device online. You can automate
setting dynamically attached devices online by using CCW hotplug events (see
CCW hotplug events on page 19).

Attention: Do not detach a device that is still being used by Linux. Detaching
devices might cause the system to hang or crash. Ensure that you unmount a
device and set it offline before you detach it.

See Working with newly available devices on page 10 to avoid errors when
working with devices that have become available to a running Linux instance.

Chapter 9. DASD 129

 Be careful to avoid errors when working with devices that have become available
to a running Linux instance.

Enabling and disabling logging

Use the dasd= module parameter or use the erplog sysfs attribute to enable or
disable error recovery processing (ERP) logging.

Procedure

You can enable and disable error recovery processing (ERP) logging on a running
system. There are two methods:
v Use the dasd= parameter when you load the base module of the DASD device
driver.

Example:

To define a device range (0.0.7000-0.0.7005) and enable logging, change the

parameter line to contain:
dasd=0.0.7000-0.0.7005(erplog)
v Use the sysfs attribute erplog to disable ERP-related logging.
Logging can be enabled for a specific device by writing 1 to the erplog attribute

Example:
echo 1 > /sys/bus/ccw/devices/<device_bus_id>/erplog

To disable logging, write 0 to the erplog attribute, for example:

Example:
echo 0 > /sys/bus/ccw/devices/<device_bus_id>/erplog

Enabling and disabling immediate failure of I/O requests

Prevent devices in mirror setups from being blocked while paths are unavailable
by making I/O requests fail immediately.

About this task

By default, a DASD that has lost all paths waits for one of the paths to recover.
I/O requests are blocked while the DASD is waiting.

If the DASD is part of a mirror setup, this blocking might cause the entire virtual
device to be blocked. You can use the failfast attribute to immediately return I/O
requests as failed while no path to the device is available.

Attention: Use this attribute with caution and only in setups where a failed I/O
request can be recovered outside the scope of a single DASD.

Procedure

Use one of these methods:

v You can enable immediate failure of I/O requests when you load the base
module of the DASD device driver.

130 Device Drivers, Features, and Commands on SLES 12 SP2

 Example:

To define a device range (0.0.7000-0.0.7005) and enable immediate failure of I/O

requests specify:
dasd=0.0.7000-0.0.7005(failfast)
v You can use the sysfs attribute failfast of a DASD to enable or disable
immediate failure of I/O requests.
To enable immediate failure of I/O requests, write 1 to the failfast attribute.

Example:
echo 1 > /sys/bus/ccw/devices/<device_bus_id>/failfast

To disable immediate failure of I/O requests, write 0 to the failfast attribute.

Example:
echo 0 > /sys/bus/ccw/devices/<device_bus_id>/failfast

Setting the timeout for I/O requests

DASD I/O requests can time out at two levels in the software stack.

About this task

When the DASD device driver receives an I/O request from an application, it
issues one or more low-level I/O requests to the affected storage system. Both the
initial I/O request from the application and the resulting low-level requests to the
storage system can time out. You set the timeout values through two sysfs
attributes of the DASD.
expires
specifies the maximum time, in seconds, that the DASD device driver waits
for a response to a low-level I/O request from a storage server.
The default for the maximum response time depends on the type of DASD:
ECKD uses the default that is provided by the storage server.
FBA 300 s
DIAG 50 s
If the maximum response time is exceeded, the DASD device driver
cancels the request. Depending on your setup, the DASD device driver
might then try the request again, possibly in combination with other
recovery actions.
timeout
specifies the time interval, in seconds, within which the DASD device
driver must respond to an I/O request from a software layer above it. If
the specified time expires before the request is completed, the DASD
device driver cancels all related low-level I/O requests to storage systems
and reports the request as failed.
This setting is useful in setups where the software layer above the DASD
device driver requires an absolute upper limit for I/O requests.
A value of 0 means that there is no time limit. This value is the default.

Chapter 9. DASD 131

 Procedure

You can use the expires and timeout attributes of a DASD to change the timeout
values for that DASD.
1. To find out the current timeout values, issue commands of this form:
# cat /sys/bus/ccw/devices/<device_bus_id>/expires
# cat /sys/bus/ccw/devices/<device_bus_id>/timeout

Example:
# cat /sys/bus/ccw/devices/0.0.7008/expires
30
# cat /sys/bus/ccw/devices/0.0.7008/timeout
0

In the example, a maximum response time of 30 seconds applies to the storage

server for a DASD with bus ID 0.0.7008. No total time limit is set for I/O
requests to this DASD.
2. To set different timeout values, issue commands of this form:
# echo <max_wait> > /sys/bus/ccw/devices/<device_bus_id>/expires
# echo <total_max> > /sys/bus/ccw/devices/<device_bus_id>/timeout

where:
<max_wait>
is the new maximum response time, in seconds, for the storage server. The
value must be a positive integer.
<total_max>
is the new maximum total time in seconds. The value must be a positive
integer or 0. 0 disables this timeout setting.
<device_bus_id>
is the device bus-ID of the DASD.

Example:
# echo 60 > /sys/bus/ccw/devices/0.0.7008/expires
# echo 120 > /sys/bus/ccw/devices/0.0.7008/timeout

This example sets timeout values for a DASD with bus ID 0.0.7008. The
maximum response time for the storage server is set to 60 seconds and the
overall time limit for I/O requests is set to 120 seconds.

Working with DASD statistics in debugfs

Gather DASD statistics and display the data with the dasdstat command.

Before you begin

v debugfs is required, but is mounted by default. If you unmounted the file
system, remount it before continuing. See debugfs on page xii.
v Instead of accessing raw DASD performance data in debugfs, you can use the
dasdstat command to obtain more structured data (see dasdstat - Display
DASD performance statistics on page 538).

132 Device Drivers, Features, and Commands on SLES 12 SP2

About this task

The DASD performance data is contained in the following subdirectories of

<mountpoint>/dasd, where <mountpoint> is the mount point of debugfs:
v A directory global that represents all available DASDs taken together.
v For each DASD, one directory with the name of the DASD block device with
which the DASD is known to the DASD device driver (for example, dasda,
dasdb, and dasdc).
v For each CCW device that corresponds to a DASD, a directory with the bus ID
as the name.
Block devices that are not set up for PAV or HyperPAV map to exactly one CCW
device and the corresponding directories contain the same statistics.
With PAV or HyperPAV, a bus ID can represent a base device or an alias device.
Each base device is associated with a particular block device. The alias devices
are not permanently associated with the same block device. At any one time, a
DASD block device is associated with one or more CCW devices. Statistics that
are based on bus ID, therefore, show more detail for PAV and HyperPAV setups.

Each of these directories contains a file statistics that you can use to perform
these tasks:
v Start and stop data gathering.
v Reset statistics counters.
v Read statistics.

To control data gathering at the scope of a directory in <mountpoint>/dasd, issue a

command of this form:
# echo <keyword> > <mountpoint>/dasd/<directory>/statistics

Where:
<directory>
is one of the directories in <mountpoint>/dasd.
<keyword>
specifies the action to be taken:
on to start data gathering.
off
to stop data gathering.
reset
to reset the statistics counters.

To read performance data, issue a command of this form:

# cat <mountpoint>/dasd/<directory>/statistics

Examples for gathering and reading DASD statistics in debugfs

Use the echo command to start and stop data gathering for individual devices or
across all DASDs. Use the cat command to access the raw performance data.

The following examples assume that debugfs is mounted at /sys/kernel/debug.

v To start data gathering for summary data across all available DASDs:

Chapter 9. DASD 133

 # echo on > /sys/kernel/debug/dasd/global/statistics

v To stop data gathering for block device dasdb:

# echo off > /sys/kernel/debug/dasd/dasdb/statistics

v To reset the counters for CCW device 0.0.b301:

# echo reset > /sys/kernel/debug/dasd/0.0.b301/statistics

v To read performance data for dasda, assuming that the degbugfs mount point is
/sys/kernel/debug, issue:
# cat /sys/kernel/debug/dasd/dasda/statistics
start_time 1283518578.085869197
total_requests 0
total_sectors 0
total_pav 0
total_hpf 0
histogram_sectors 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
histogram_io_times 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
histogram_io_times_weighted 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
histogram_time_build_to_ssch 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
histogram_time_ssch_to_irq 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
histogram_time_ssch_to_irq_weighted 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
histogram_time_irq_to_end 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
histogram_ccw_queue_length 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
total_read_requests 0
total_read_sectors 0
total_read_pav 0
total_read_hpf 0
histogram_read_sectors 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
histogram_read_times 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
histogram_read_time_build_to_ssch 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
histogram_read_time_ssch_to_irq 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
histogram_read_time_irq_to_end 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
histogram_read_ccw_queue_length 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

Interpreting the data rows

The raw DASD performance data in the statistics directories in debugfs is
organized into labeled data rows.

This section explains the raw data in the individual data rows of the statistics. Use
the dasdstat command to obtain more structured data.
start_time
is the UNIX epoch time stamp when data gathering was started or when
the counters were last reset.

Tip: Use the date tool to convert the time stamp to a more readily
human-readable format. See the date man page for details.
Single counters
have a single integer as the statistics data. All rows with labels that begin
with total_ are of this data type.
The following rows show data for the sum of all requests, read and write:
total_requests
is the number of requests that have been processed.
total_sectors
is the sum of the sizes of all requests, in units of 512-byte sectors.

134 Device Drivers, Features, and Commands on SLES 12 SP2

 total_pav
is the number of requests that were processed through a PAV alias
device.
total_hpf
is the number of requests that used High Performance FICON.
The following rows show data for read requests only:
total_read_requests
is the number of read requests that have been processed.
total_read_sectors
is the sum of the sizes of all read requests, in units of 512-byte
sectors.
total_read_pav
is the number of read requests that were processed through a PAV
alias device.
total_read_hpf
is the number of read requests that used High Performance
FICON.
Linear histograms
have a series of 32 integers as the statistics data. The integers represent a
histogram, with a linear scale, of the number of requests in the request
queue each time a request has been queued. The first integer shows how
often the request queue contained zero requests, the second integer shows
how often the queue contained one request, and the n-th value shows how
often the queue contained n-1 requests.
histogram_ccw_queue_length
is the histogram data for all requests, read and write.
histogram_read_ccw_queue_length
is the histogram data for read requests only.
Logarithmic histograms
have a series of 32 integers as the statistics data. The integers represent a
histogram with a logarithmic scale:
v The first integer always represents all measures of fewer than 4 units
v The second integer represents measures of 4 or more but less than 8
units
v The third integer represents measures of 8 or more but less than 16 units
v The n-th integer (1 < n < 32) represents measures of 2n or more but less
than 2n+1 units
v The 32nd integer represents measures of 232 (= 4G = 4,294,967,296) units
or more.
The following rows show data for the sum of all requests, read and write:
histogram_sectors
is the histogram data for request sizes. A unit is a 512-byte sector.
histogram_io_times
is the histogram data for the total time that is needed from creating
the cqr to its completion in the DASD device driver and return to
the block layer. A unit is a microsecond.

Chapter 9. DASD 135

 histogram_io_times_weighted
is the histogram data of the total time, as measured for
histogram_io_times, devided by the requests size in sectors. A unit
is a microsecond per sector.
This metric is deprecated and there is no corresponding histogram
data for read requests.
histogram_time_build_to_ssch
is the histogram data of the time that is needed from creating the
cqr to submitting the request to the subchannel. A unit is a
microsecond.
histogram_time_ssch_to_irq
is the histogram data of the time that is needed from submitting
the request to the subchannel until an interrupt indicates that the
request has been completed. A unit is a microsecond.
histogram_time_ssch_to_irq_weighted
is the histogram data of the time that is needed from submitting
the request to the subchannel until an interrupt indicates that the
request has been completed, divided by the request size in 512-byte
sectors. A unit is a microsecond per sector.
This metric is deprecated and there is no corresponding histogram
data for read requests.
histogram_time_irq_to_end
is the histogram data of the time that is needed from return of the
request from the channel subsystem, until the request is returned
to the block layer. A unit is a microsecond.
The following rows show data for read requests only:
histogram_read_sectors
is the histogram data for read request sizes. A unit is a 512-byte
sector.
histogram_read_io_times
is the histogram data, for read requests, for the total time that is
needed from creating the cqr to its completion in the DASD device
driver and return to the block layer. A unit is a microsecond.
histogram_read_time_build_to_ssch
is the histogram data, for read requests, of the time that is needed
from creating the cqr to submitting the request to the subchannel.
A unit is a microsecond.
histogram_read_time_ssch_to_irq
is the histogram data, for read requests, of the time that is needed
from submitting the request to the subchannel until an interrupt
indicates that the request has been completed. A unit is a
microsecond.
histogram_read_time_irq_to_end
is the histogram data, for read requests, of the time that is needed
from return of the request from the channel subsystem, until the
request is returned to the blocklayer. A unit is a microsecond.

Scenario: Verifying that PAV and HPF are used

Use the dasdstat command to display DASD performance statistics, including
statistics about Parallel Access Volume (PAV) and High Performance FICON (HPF).

136 Device Drivers, Features, and Commands on SLES 12 SP2

 Procedure
1. Enable DASD statistics for the device of interest.

Example:
# dasdstat -e dasdc
enable statistic "/sys/kernel/debug/dasd/dasdc/statistics"

2. Assure that I/O requests are directed to the device.

Hints:
v Access a partition, rather than the whole device, to avoid directing the I/O
request towards the first 2 tracks of a CDL formatted DASD. Requests to the
first 2 tracks of a CDL formatted DASD are exceptional in that they never
use High Performance FICON.
v Assure that a significant I/O load is applied to the device. PAV aliases are
used only if multiple I/O requests for the device are processed
simultaneously.

Example:
# dd if=/dev/dasdc1 of=/dev/null bs=4k count=256

3. Look for PAV and HPF in the statistics.

Example:
# dasdstat dasdc
--------------------------------------------------------------------------------
statistics data for statistic: dasdc
start time of data collection: Fri Dec 11 14:22:18 CET 2015

7 dasd I/O requests

with 4000 sectors(512B each)
3 requests used a PAV alias device
7 requests used HPF

In the example, dasdc uses both Parallel Access Volume and High Performance
FICON.

Accessing full ECKD tracks

In raw-track access mode, the DASD device driver accesses full ECKD tracks,
including record zero and the count and key data fields.

Before you begin

v This section applies to ECKD type DASD only.
v The DASD has to be offline when you change the access mode.
v The DIAG access method must not be enabled for the device.

About this task

With this mode, Linux can access an ECKD device regardless of the track layout.
In particular, the device does not need to be formatted for Linux.

Chapter 9. DASD 137

 For example, with raw-track access mode Linux can create a backup copy of any
ECKD device. Full-track access can also enable a special program that runs on
Linux to access and process data on an ECKD device that is not formatted for
Linux.

By default, the DASD device driver accesses only the data fields of ECKD devices.
In default access mode, you can work with partitions, file systems, and files in the
file systems on the DASD.

When using a DASD in raw-track access mode be aware that:

v In memory, each track is represented by 64 KB of data, even if the track
occupies less physical disk space. Therefore, a disk in raw-track access mode
appears bigger than in default mode.
v Programs must read or write data in multiples of complete 64 KB tracks. The
minimum is a single track. The maximum is eight tracks by default but can be
extended to up to 16 tracks.
The maximum number of tracks depends on the maximum number of sectors as
specified in the max_sectors_kb sysfs attribute of the DASD. This attribute is
located in the block device branch of sysfs at /sys/block/dasd<x>/queue/
max_sectors_kb. In the path, dasd<x> is the device name that is assigned by the
DASD device driver.
To extend the maximum beyond eight tracks, set the max_sectors_kb to the
maximum amount of data to be processed in a single read or write operation.
For example, to extend the maximum to reading or writing 16 tracks at a time,
set max_sectors_kb to 1024 (16 x 64).
v Programs must write only valid ECKD tracks of 64 KB.
v Programs must use direct I/O to prevent the Linux block layer from splitting
tracks into fragments. Open the block device with option O_DIRECT or work
with programs that use direct I/O.
For example, the options iflag=direct and oflag=direct cause dd to use direct
I/O. When using dd, also specify the block size with the bs= option. The block
size determines the number of tracks that are processed in a single I/O
operation. The block size must be a multiple of 64 KB and can be up to
1024 KB. Specifying a larger block size often results in better performance. If
you receive disk image data from a pipe, also use the option iflag=fullblock to
ensure that full blocks are written to the DASD device.
Tools cannot directly work with partitions, file systems, or files within a file
system. For example, fdasd and dasdfmt cannot be used.

Procedure

To change the access mode, issue a command of this form:

# echo <switch> > /sys/bus/ccw/devices/<device_bus_id>/raw_track_access

where:
<switch>
is 1 to activate raw data access and 0 to deactivate raw data access.
<device_bus_id>
identifies the DASD.

138 Device Drivers, Features, and Commands on SLES 12 SP2

 Example

The following example creates a backup of a DASD 0.0.7009 on a DASD 0.0.70a1.

The initial commands ensure that both devices are offline and that the DIAG access
method is not enabled for either of them. The subsequent commands activate the
raw-track access mode for the two devices and set them both online. The lsdasd
command that follows shows the mapping between device bus-IDs and device
names.

The dd command for the copy operation specifies direct I/O for both the input and
output device and the block size of 1024 KB. After the copy operation is
completed, both devices are set offline. The access mode for the original device
then is set back to the default and the device is set back online.
#cat /sys/bus/ccw/devices/0.0.7009/online
1
# chccwdev -d 0.0.7009
# cat /sys/bus/ccw/devices/0.0.7009/use_diag
0
# cat /sys/bus/ccw/devices/0.0.70a1/online
0
# cat /sys/bus/ccw/devices/0.0.70a1/use_diag
0
# echo 1 > /sys/bus/ccw/devices/0.0.7009/raw_track_access
# echo 1 > /sys/bus/ccw/devices/0.0.70a1/raw_track_access
# chccwdev -e 0.0.7009,0.0.70a1
# lsdasd 0.0.7009 0.0.70a1
Bus-ID Status Name Device Type BlkSz Size Blocks
==============================================================================
0.0.7009 active dasdf 94:20 ECKD 4096 7043MB 1803060
0.0.70a1 active dasdj 94:36 ECKD 4096 7043MB 1803060
# echo 1024 > /sys/block/dasdf/queue/max_sectors_kb
# echo 1024 > /sys/block/dasdj/queue/max_sectors_kb
# dd if=/dev/dasdf of=/dev/dasdj bs=1024k iflag=direct oflag=direct
# chccwdev -d 0.0.7009,0.0.70a1
# echo 0 > /sys/bus/ccw/devices/0.0.7009/raw_track_access
# chccwdev -e 0.0.7009

Handling lost device reservations

A DASD reservation by your Linux instance can be lost if another system
unconditionally reserves this DASD.

About this task

This other system then has exclusive I/O access to the DASD for the duration of
the unconditional reservation. Such unconditional reservations can be useful for
handling error situations where:
v Your Linux instance cannot gracefully release the DASD.
v Another system requires access to the DASD, for example, to perform recovery
actions.

After the DASD is released by the other system, your Linux instance might process
pending I/O requests and write faulty data to the DASD. How to prevent pending
I/O requests from being processed depends on the reservation policy. There are
two reservation policies:
ignore All I/O operations for the DASD are blocked until the DASD is released
by the second system. When using this policy, reboot your Linux instance
before the other system releases the DASD. This policy is the default.

Chapter 9. DASD 139

 fail All I/O operations are returned as failed until the DASD is set offline or
until the reservation state is reset. When using this policy, set the DASD
offline and back online after the problem has been resolved. See Reading
and resetting the reservation state about resetting the reservation state to
resume operations.

Procedure
Set the reservation policy with a command of this form:
# echo <policy> > /sys/bus/ccw/devices/<device_bus_id>/reservation_policy

where:
<device_bus_id>
specifies the DASD.
<policy>
is one of the available policies, ignore or fail.

Examples
v The command of this example sets the reservation policy for a DASD with bus
ID 0.0.7009 to fail.
# echo fail > /sys/bus/ccw/devices/0.0.7009/reservation_policy

v This example shows a small scenario. The first two commands confirm that the
reservation policy of the DASD is fail and that the reservation has been lost to
another system. Assuming that the error that had occurred has already been
resolved and that the other system has released the DASD, operations with the
DASD are resumed by setting it offline and back online.
# cat /sys/bus/ccw/devices/0.0.7009/reservation_policy
fail
# cat /sys/bus/ccw/devices/0.0.7009/last_known_reservation_state
lost
# chccwdev -d 0.0.7009
# chccwdev -e 0.0.7009

Reading and resetting the reservation state

How the DASD device driver handles I/O requests depends on the
last_known_reservation_state sysfs attribute of the DASD.

About this task

The last_known_reservation_state attribute reflects the reservation state as held
by the DASD device driver and can differ from the actual reservation state. Use the
tunedasd -Q command to find out the actual reservation state. The
last_known_reservation_state sysfs attribute can have the following values:
none The DASD device driver has no information about the device reservation
state. I/O requests are processed as usual. If the DASD is reserved by
another system, the I/O requests remain in the queue until they time out,
or until the reservation is released.
reserved
The DASD device driver holds a valid reservation for the DASD and I/O

140 Device Drivers, Features, and Commands on SLES 12 SP2

 requests are processed as usual. The DASD device driver changes this state
if notified that the DASD is no longer reserved to this system. The new
state depends on the reservation policy (see Handling lost device
reservations on page 139).
ignore The state is changed to none.
fail The state is changed to lost.
lost The DASD device driver had reserved the DASD, but subsequently
another system has unconditionally reserved the DASD (see Handling lost
device reservations on page 139). The device driver processes only
requests that query the actual device reservation state. All other I/O
requests for the device are returned as failed.
When the error that led another system to unconditionally reserve the
DASD is resolved and the DASD has been released by this other system
there are two methods for resuming operations:
v Setting the DASD offline and back online.
v Resetting the reservation state of the DASD.

Attention: Do not resume operations by resetting the reservation state

unless your system setup maintains data integrity on the DASD despite:
v The I/O errors that are caused by the unconditional reservation
v Any changes to the DASD through the other system
You reset the reservation state by writing reset to the
last_known_reservation_state sysfs attribute of the DASD. Resetting is
possible only for the fail reservation policy (see Handling lost device
reservations on page 139) and only while the value of the
last_known_reservation_state attribute is lost.

To find out the reservation state of a DASD issue a command of this form:
# cat /sys/bus/ccw/devices/<device_bus_id>/last_known_reservation_state

where <device_bus_id> specifies the DASD.

Example

The command in this example queries the reservation state of a DASD with bus ID
0.0.7009.
# cat /sys/bus/ccw/devices/0.0.7009/last_known_reservation_state
reserved

Displaying DASD information

Use tools to display information about your DASDs, or read the attributes of the
devices in sysfs.

About this task

There are several methods to display DASD information:

v Use lsdasd -l (see lsdasd - List DASD devices on page 584) to display
summary information about the device settings and the device geometry of
multiple DASDs.

Chapter 9. DASD 141

 v Use dasdview (see dasdview - Display DASD structure on page 541) to display
details about the contents of a particular DASD.
v Read information about a particular DASD from sysfs, as described in this
section.

The sysfs representation of a DASD is a directory of the form /sys/bus/ccw/

devices/<device_bus_id>, where <device_bus_id> is the bus ID of the DASD. This
sysfs directory contains a number of attributes with information about the DASD.
Table 20. Attributes with DASD information
Attribute Explanation
alias 1 if the DASD is a parallel access volume (PAV) alias device. 0 if the DASD is a
PAV base device or has not been set up as a PAV device.

For an example of how to use PAV see How to Improve Performance with PAV,
SC33-8414 on developerWorks at
www.ibm.com/developerworks/linux/linux390/documentation_suse.html

This attribute is read-only.

discipline Indicates the base discipline, ECKD or FBA, that is used to access the DASD. If
DIAG is enabled, this attribute might read DIAG instead of the base discipline.

This attribute is read-only.

eer_enabled 1 if the DASD is enabled for extended error reporting, 0 if it is not enabled (see
Using extended error reporting for ECKD type DASD on page 127).
erplog 1 if error recovery processing (ERP) logging is enabled, 0 if ERP logging is not
enabled (see Enabling and disabling logging on page 130).
expires Indicates the time, in seconds, that the DASD device driver waits for a response to
an I/O request from a storage server. If this time expires, the device driver
considers a request as failed and cancels it (see Setting the timeout for I/O
requests on page 131).
failfast 1 if I/O operations are returned as failed immediately when the last path to the
DASD is lost. 0 if a wait period for a path to return expires before an I/O
operation is returned as failed. (see Enabling and disabling immediate failure of
I/O requests on page 130).
last_known_reservation_state
The reservation state as held by the DASD device driver. Values can be:
none The DASD device driver has no information about the device reservation
state.
reserved
The DASD device driver holds a valid reservation for the DASD.
lost The DASD device driver had reserved the device, but this reservation has
been lost to another system.

See Reading and resetting the reservation state on page 140 for details.
online 1 if the DASD is online, 0 if it is offline (see Setting a DASD online or offline on
page 128).
raw_track_access 1 if the DASD is in raw-track access mode, 0 if it is in default access mode (see
Accessing full ECKD tracks on page 137).
readonly 1 if the DASD is read-only, 0 if it can be written to. This attribute is a device driver
setting and does not reflect any restrictions that are imposed by the device itself.
This attribute is ignored for PAV alias devices.
reservation_policy Shows the reservation policy of the DASD. Possible values are ignore and fail.
See Handling lost device reservations on page 139 for details.

142 Device Drivers, Features, and Commands on SLES 12 SP2

Table 20. Attributes with DASD information (continued)
Attribute Explanation
status Reflects the internal state of a DASD device. Values can be:
unknown
Device detection has not started yet.
new
Detection of basic device attributes is in progress.
detected
Detection of basic device attributes has finished.
basic
The device is ready for detecting the disk layout. Low-level tools can set a
device to this state when changing the disk layout, for example, when
formatting the device.
unformatted
The disk layout detection found no valid disk layout. The device is ready for
use with low-level tools like dasdfmt.
ready
The device is in an intermediate state.
online
The device is ready for use.
timeout Indicates the time, in seconds, within which the DASD device driver must respond
to an I/O request from a software layer above it. If the specified time expires
before the request is completed, the DASD device driver cancels all related
low-level I/O requests to storage systems and reports the request as failed (see
Setting the timeout for I/O requests on page 131).
uid A device identifier of the form
<vendor>.<serial>.<subsystem_id>.<unit_address>.<minidisk_identifier> where
<vendor>
is the specification from the vendor attribute.
<serial>
is the serial number of the storage system.
<subsystem_id>
is the ID of the logical subsystem to which the DASD belongs on the storage
system.
<unit_address>
is the address that is used within the storage system to identify the DASD.
<minidisk_identifier>
is an identifier that the z/VM system assigns to distinguish between minidisks
on the DASD. This part of the uid is only present for Linux on z/VM and if
the z/VM version and service level support this identifier.

This attribute is read-only.

use_diag 1 if the DIAG access method is enabled, 0 if the DIAG access method is not
enabled (see Enabling the DASD device driver to use the DIAG access method
on page 126). Do not enable the DIAG access method is for PAV alias devices.
vendor Identifies the manufacturer of the storage system that contains the DASD.

This attribute is read-only.

There are some more attributes that are common to all CCW devices (see Device
attributes on page 9).

Chapter 9. DASD 143

 Procedure

Issue a command of this form to read an attribute:

# cat /sys/bus/ccw/devices/<device_bus_id>/<attribute>

where <attribute> is one of the attributes of Table 20 on page 142.

Example

The following sequence of commands reads the attributes for a DASD with a
device bus-ID 0.0.b100:
# cat /sys/bus/ccw/devices/0.0.b100/alias
0
# cat /sys/bus/ccw/devices/0.0.b100/discipline
ECKD
# cat /sys/bus/ccw/devices/0.0.b100/eer_enabled
0
# cat /sys/bus/ccw/devices/0.0.b100/erplog
0
# cat /sys/bus/ccw/devices/0.0.b100/expires
30
# cat /sys/bus/ccw/devices/0.0.b100/failfast
0
# cat /sys/bus/ccw/devices/0.0.b100/last_known_reservation_state
reserved
# cat /sys/bus/ccw/devices/0.0.b100/online
1
# cat /sys/bus/ccw/devices/0.0.b100/raw_track_access
0
# cat /sys/bus/ccw/devices/0.0.b100/readonly
1
# cat /sys/bus/ccw/devices/0.0.b100/reservation_policy
ignore
# cat /sys/bus/ccw/devices/0.0.b100/status
online
# cat /sys/bus/ccw/devices/0.0.b100/timeout
120
# cat /sys/bus/ccw/devices/0.0.b100/uid
IBM.75000000092461.e900.8a
# cat /sys/bus/ccw/devices/0.0.b100/use_diag
1
# cat /sys/bus/ccw/devices/0.0.b100/vendor
IBM

144 Device Drivers, Features, and Commands on SLES 12 SP2

Chapter 10. SCSI-over-Fibre Channel device driver
The SCSI-over-Fibre Channel device driver for Linux on z Systems (zfcp device
driver) supports virtual QDIO-based z Systems SCSI-over-Fibre Channel adapters
(FCP devices) and attached SCSI devices (LUNs).

z Systems adapter hardware typically provides multiple channels, with one port
each. You can configure a channel to use the Fibre Channel Protocol (FCP). This
FCP channel is then virtualized into multiple FCP devices. Thus, an FCP device is a
virtual QDIO-based z Systems SCSI-over-Fibre Channel adapter with a single port.

A single physical port supports multiple FCP devices. Using N_Port ID

virtualization (NPIV) you can define virtual ports and establish a one-to-one
mapping between your FCP devices and virtual ports (see N_Port ID
Virtualization for FCP channels on page 150).

On Linux, an FCP device is represented by a CCW device that is listed under

/sys/bus/ccw/drivers/zfcp. Do not confuse FCP devices with SCSI devices. A
SCSI device is identified by a LUN.

Features
The zfcp device driver supports a wide range of SCSI devices, various hardware
adapters, specific topologies, and specific features that depend on the z Systems
hardware.
v Linux on z Systems can use various SAN-attached SCSI device types, including
SCSI disks, tapes, CD-ROMs, and DVDs. For a list of supported SCSI devices,
see
www.ibm.com/systems/z/connectivity
v SAN access through the following hardware adapters:
FICON Express16S (as of z13)
FICON Express8S
FICON Express8
FICON Express4
You can order hardware adapters as features for mainframe systems.
See Fibre Channel Protocol for Linux and z/VM on IBM System z, SG24-7266 for
more details about using FCP with Linux on z Systems.
v The zfcp device driver supports switched fabric and point-to-point topologies.
v As of zEnterprise, the zfcp device driver supports end-to-end data consistency
checking.
v As of FICON Express8S, the zfcp device driver supports the data router
hardware feature to improve performance by reducing the path length.

For information about SCSI-3, the Fibre Channel Protocol, and fiber channel related
information, see www.t10.org and www.t11.org

What you should know about zfcp

The zfcp device driver is a low-level driver or host-bus adapter driver that
supplements the Linux SCSI stack.

Copyright IBM Corp. 2000, 2016 145

 Figure 27 illustrates how the device drivers work together.

Figure 27. Device drivers that support the FCP environment

sysfs structures for FCP devices and SCSI devices

FCP devices are CCW devices. In the sysfs device driver view, remote target ports
with their LUNs are nested below the FCP devices.

When Linux is booted, it senses the available FCP devices and creates directories of
the form:
/sys/bus/ccw/drivers/zfcp/<device_bus_id>

where <device_bus_id> is the device bus-ID that corresponds to an FCP device. You
use the attributes in this directory to work with the FCP device.

Example: /sys/bus/ccw/drivers/zfcp/0.0.3d0c

The zfcp device driver automatically adds port information when the FCP device is
set online and when remote storage ports (target ports) are added. Each added
target port extends this structure with a directory of the form:
/sys/bus/ccw/drivers/zfcp/<device_bus_id>/<wwpn>

where <wwpn> is the worldwide port name (WWPN) of the target port. You use
the attributes of this directory to work with the port.

Example: /sys/bus/ccw/drivers/zfcp/0.0.3d0c/0x500507630300c562

With NPIV-enabled FCP devices, SUSE Linux Enterprise Server 12 SP2 uses
automatic LUN scanning by default. The zfcp sysfs branch ends with the target

146 Device Drivers, Features, and Commands on SLES 12 SP2

 port entries. For FCP devices that are not NPIV-enabled, or if automatic LUN
scanning is disabled, see Configuring SCSI devices on page 168.

Information about zfcp objects and their associated objects in the SCSI stack is
distributed over the sysfs tree. To ease the burden of collecting information about
zfcp devices, ports, units, and their associated SCSI stack objects, a command that
is called lszfcp is provided with s390-tools. See lszfcp - List zfcp devices on
page 609 for more details about the command.

See also Mapping the representations of SCSI devices in sysfs on page 170.

SCSI device nodes

User space programs access SCSI devices through device nodes.

SCSI device names are assigned in the order in which the devices are detected. In a
typical SAN environment, this can mean a seemingly arbitrary mapping of names
to actual devices that can change between boots. Therefore, using standard device
nodes of the form /dev/<device_name> where <device_name> is the device name
that the SCSI stack assigns to a device, can be a challenge.

SUSE Linux Enterprise Server 12 SP2 provides udev to create device nodes for you.
Use the device nodes to identify the corresponding actual device.
Device nodes that are based on device names
udev creates device nodes that match the device names that are used by
the kernel. These standard device nodes have the form /dev/<name>.

The examples in this section use standard device nodes as assigned by the SCSI
stack. These nodes have the form /dev/sd<x> for entire disks and /dev/sd<x><n>
for partitions. In these node names <x> represents one or more letters and <n> is
an integer. See Documentation/devices.txt in the Linux source tree for more
information about the SCSI device naming scheme.

To help you identify a particular device, udev creates additional device nodes that
are based on the device's bus ID, the device label, and information about the file
system on the device. The file system information can be a universally unique
identifier (UUID) and, if available, the file system label.
| Device nodes that are based on bus IDs
| udev creates device nodes of the form
| /dev/disk/by-path/ccw-<device_bus_id>-fc-<wwpn>-lun-<lun>

| for whole SCSI device and

| /dev/disk/by-path/ccw-<device_bus_id>-fc-<wwpn>-lun-<lun>-part<n>

| for the <n>th partition, where <wwpn> is the worldwide port number of
| the target port and <lun> is the logical unit number that represents the
| target SCSI device.

| Note: The format of these udev-created device nodes has changed and
| now matches the common code format. Device nodes of the prior form,
| ccw-<device_bus_id>-zfcp-<wwpn>:<lun> or ccw-<device_bus_id>-zfcp-
| <wwpn>:<lun>-part<n>, are no longer created.
Device nodes that are based on file system information
udev creates device nodes of the form

Chapter 10. SCSI-over-Fibre Channel 147

 /dev/disk/by-uuid/<uuid>

where <uuid> is a unique file-system identifier (UUID) for the file system
in a partition.

If a file system label was assigned, udev also creates a node of the form:
/dev/disk/by-label/<label>
There are no device nodes for the whole SCSI device that are based on file
system information.
Additional device nodes
/dev/disk/by-id contains additional device nodes for the SCSI device and
partitions, that are all based on a unique SCSI identifier generated by
querying the device.

Example

For a SCSI device that is assigned the device name sda, has two partitions labeled
boot and SWAP-sda2 respectively, a device bus-ID 0.0.3c1b (device number
0x3c1b), and a UUID 7eaf9c95-55ac-4e5e-8f18-065b313e63ca for the first and
b4a818c8-747c-40a2-bfa2-acaa3ef70ead for the second partition, udev creates the
following device nodes:

For the whole SCSI device:

v /dev/sda (standard device node according to the SCSI device naming scheme)
v /dev/disk/by-path/ccw-0.0.3c1b-fc-0x500507630300c562-lun-
0x401040ea00000000
v /dev/disk/by-id/scsi-36005076303ffc56200000000000010ea
v /dev/disk/by-id/wwn-0x6005076303ffc56200000000000010ea

For the first partition:

v /dev/sda1 (standard device node according to the SCSI device naming scheme)
v /dev/disk/by-path/ccw-0.0.3c1b-fc-0x500507630300c562-lun-
0x401040ea00000000-part1
v /dev/disk/by-uuid/7eaf9c95-55ac-4e5e-8f18-065b313e63ca
v /dev/disk/by-label/boot
v /dev/disk/by-id/scsi-36005076303ffc56200000000000010ea-part1
v /dev/disk/by-id/wwn-0x6005076303ffc56200000000000010ea-part1

For the second partition:

v /dev/sda2 (standard device node according to the SCSI device naming scheme)
v /dev/disk/by-path/ccw-0.0.3c1b-fc-0x500507630300c562-lun-
0x401040ea00000000-part2
v /dev/disk/by-uuid/b4a818c8-747c-40a2-bfa2-acaa3ef70ead
v /dev/disk/by-label/SWAP-sda2
v /dev/disk/by-id/scsi-36005076303ffc56200000000000010ea-part2
v /dev/disk/by-id/wwn-0x6005076303ffc56200000000000010ea-part2
Device nodes by-uuid use a unique file-system identifier that does not relate to the
partition number.

148 Device Drivers, Features, and Commands on SLES 12 SP2

 Multipath

Users of SCSI-over-Fibre Channel attached devices should always consider setting

up and using redundant paths through their Fibre Channel storage area network.

Path redundancy improves the availability of the LUNs. In Linux, you can set up
path redundancy with the device-mapper multipath tool. For information about
multipath devices and multipath partitions, see the chapter about multipathing in
How to use FC-attached SCSI devices with Linux on z Systems, SC33-8413.

Partitioning a SCSI device

You can partition SCSI devices that are attached through an FCP channel in the
same way that you can partition SCSI attached devices on other platforms.

About this task

Use the fdisk command to partition a SCSI disk, not fdasd.

udev creates device nodes for partitions automatically. For the SCSI disk /dev/sda,
the partition device nodes are called /dev/sda1, /dev/sda2, /dev/sda3, and so on.

Example

To partition a SCSI disk with a device node /dev/sda issue:

# fdisk /dev/sda

zfcp HBA API (FC-HBA) support

The zfcp host bus adapter API (HBA API) provides an interface for HBA
management clients that run on z Systems.

As shown in Figure 28 on page 150, the zfcp HBA API support includes a user
space library.

Chapter 10. SCSI-over-Fibre Channel 149

 Figure 28. zfcp HBA API support modules

The zFCP HBA API library is part of SUSE Linux Enterprise Server 12 SP2. It is
available as software package libzfcphbaapi0, see Getting ready to run
applications on page 184.

The default method in SUSE Linux Enterprise Server 12 SP2 is for applications to
use the zFCP HBA API library. If you develop applications yourself, see
Developing applications on page 183.

In a Linux on z Systems environment, HBAs are usually virtualized and are shown
as FCP devices.

For information about setting up the HBA API support, see zfcp HBA API
support on page 183.

N_Port ID Virtualization for FCP channels

Through N_Port ID Virtualization (NPIV), the sole port of an FCP channel appears
as multiple, distinct ports with separate port identification.

NPIV support can be configured on the SE per CHPID and LPAR for an FCP
channel. The zfcp device driver supports NPIV error messages and adapter
attributes. See Displaying FCP channel and device information on page 155 for
the Fibre Channel adapter attributes.

For more information, see the connectivity page at www.ibm.com/systems/z/

connectivity.

See also the chapter on NPIV in How to use FC-attached SCSI devices with Linux on z
Systems, SC33-8413.

150 Device Drivers, Features, and Commands on SLES 12 SP2

Setting up the zfcp device driver
SUSE Linux Enterprise Server 12 SP2 loads the zfcp device driver for you when an
FCP channel becomes available. Use YaST to configure the zfcp device driver.

You have the following options for configuring FCP:

v Use the YaST GUI yast2 zfcp. If cio_ignore is enabled, you might need to free
blacklisted FCP devices beforehand by using yast2 cio.
v Use the text-based interface yast zfcp. If cio_ignore is enabled, you might need
to free blacklisted FCP devices beforehand by using yast cio
v Use the command line, use zfcp_host_configure. It transparently frees the FCP
device specified on the command line from cio_ignore. cio_ignore does not
apply to zfcp_disk_configure.
See the section about hard disk configuration in the SUSE Linux Enterprise Server 12
SP2 Deployment Guide, and the procedure about configuring a zFCP disk in SUSE
Linux Enterprise Server 12 SP2 Administration Guide. The command-line tools
described work not only inside the rescue environment but also in a regularly
installed Linux instance.

Important: Configuration changes can directly or indirectly affect information that

is required to mount the root file system. Such changes require an update of the
initrd of both the auxiliary kernel and the target kernel, followed by a re-write of
the boot record (see Rebuilding the initial RAM disk image on page 56).

The parameters are described in the context of the modprobe command. For details
about specifying kernel and module parameters, see Chapter 3, Kernel and
module parameters, on page 23.

zfcp module parameter syntax

allow_lun_scan=1 datarouter=1
modprobe zfcp
allow_lun_scan=<value> datarouter=0

dbflevel=3 dbfsize=4 dif=0

dbflevel=<level> dbfsize=<pages> dif=<value>

port_scan_ratelimit=60000 port_scan_backoff=500

port_scan_ratelimit=<limit> port_scan_backoff=<delay>

no_auto_port_rescan=0 queue_depth=32

no_auto_port_rescan=1 queue_depth=<depth>

where:
allow_lun_scan=<value>
disables the automatic LUN scan for FCP devices that run in NPIV mode if set
to 0, n, or N. To enable the LUN scanning set the parameter to 1, y, or Y. When

Chapter 10. SCSI-over-Fibre Channel 151

 the LUN scan is disabled, all LUNs must be configured through the unit_add
zfcp attribute in sysfs. LUN scan is enabled by default.
datarouter=
enables (if set to 1, y, or Y) or disables (if set to 0, n, or N) support for the
hardware data routing feature. The default is 1.

Note: The hardware data routing feature becomes active only for FCP devices
that are based on adapter hardware with hardware data routing support.
dbflevel=<level>
sets the initial log level of the debug feature. The value is an integer in the
range 0 - 6, where greater numbers generate more detailed information. The
default is 3.
dbfsize=<pages>
specifies the number of pages to be used for the debug feature.
The debug feature is available for each FCP device and the following areas:
hba FCP device
san Storage Area Network
rec Error Recovery Process
scsi SCSI
pay Payloads for entries in the hba, san, rec, or scsi areas. The default is 8
pages.
The value that is given is used for all areas. The default for hba, san, rec, and
scsi is 4, that is, four pages are used for each area and FCP device. In the
following example the dbsfsize is increased to 6 pages:
zfcp.dbfsize=6

This results in six pages being used for each area and FCP device. The payload
is doubled to use 12 pages.
dif=<value>
turns on end-to-end data consistency checking if set to 1, y, or Y and off if set
to 0, n, or N. The default is 0.
port_scan_ratelimit=<limit>
sets the minimum delay, in milliseconds, between automatic port scans of your
Linux instance. The default value is 60000 milliseconds. To turn off the rate
limit, specify 0. Use this parameter to avoid frequent scans, while you still
ensure that a scan is conducted eventually.
port_scan_backoff=<delay>
sets additional random delay, in milliseconds, in which the port scans of your
Linux instance are spread. The default value is 500 milliseconds. To turn off the
random delay, specify 0. In an installation with multiple Linux instances, use
this attribute for every Linux instance to spread scans to avoid potential
multiple simultaneous scans.
no_auto_port_rescan=
turns the automatic port rescan feature off ( if set to 1, y, or Y) or on (if set to
0, n, or N). The default is 0. Automatic rescan is always run when an adapter
is set online and when user-triggered writes to the sysfs attribute port_rescan
occur.
queue_depth=<depth>
specifies the number of commands that can be issued simultaneously to a SCSI
device. The default is 32. The value that you set here is used as the default
queue depth for new SCSI devices. You can change the queue depth for each
SCSI device with the queue_depth sysfs attribute, see Setting the queue
depth on page 174.
device=<device_bus_id>, <wwpn>, <fcp_lun>

152 Device Drivers, Features, and Commands on SLES 12 SP2

 Attention: The device= module parameter is reserved for internal use. Do not
use.
<device_bus_id>
specifies the FCP device through which the SCSI device is attached.
<wwpn>
specifies the target port through which the SCSI device is attached.
<fcp_lun>
specifies the LUN of the SCSI device.

Working with FCP devices

Set an FCP device online before you attempt to perform any other tasks.

Working with FCP devices comprises the following tasks:

v Setting an FCP device online or offline
v Displaying FCP channel and device information on page 155
v Recovering a failed FCP device on page 158
v Finding out whether NPIV is in use on page 159
v Logging I/O subchannel status information on page 160

Setting an FCP device online or offline

By default, FCP devices are offline. Set an FCP device online before you perform
any other tasks.

About this task

Attention: Use the procedure described here for dynamic testing of configuration
settings. For persistent configuration in a production system, use one of the
following options:
v Use the YaST GUI yast2 zfcp. If cio_ignore is enabled, you might need to free
blacklisted FCP devices before by using yast2 cio.
v Use the text-based interface yast zfcp. If cio_ignore is enabled, you might need
to free blacklisted FCP devices before by using yast cio.
v Use the command line, use zfcp_host_configure. It transparently frees the FCP
device given on the command line from cio_ignore.

See the section about IBM z Systems hard disk configuration in the SUSE Linux
Enterprise Server 12 SP2 Deployment Guide, and the procedure about configuring a
zFCP disk in SUSE Linux Enterprise Server 12 SP2 Administration Guide. The
command line tools described work not only inside the rescue environment but
also in a regularly installed Linux instance.

Important: Configuration changes can directly or indirectly affect information that

is required to mount the root file system. Such changes require an update of the
initrd of both the auxiliary kernel and the target kernel, followed by a re-write of
the boot record (see Rebuilding the initial RAM disk image on page 56).

See Working with newly available devices on page 10 to avoid errors when you
work with devices that have become available to a running Linux instance.

Chapter 10. SCSI-over-Fibre Channel 153

 Setting an FCP device online registers it with the Linux SCSI stack and updates the
symbolic port name for the device on the FC name server. For FCP setups that use
NPIV mode, the device bus-ID and the host name of the Linux instance are added
to the symbolic port name.

Setting an FCP device online also automatically runs the scan for ports in the SAN
and waits for this port scan to complete.

To check if setting the FCP device online was successful, you can use a script that
first sets the FCP device online and after this operation completes checks if the
WWPN of a target port has appeared in sysfs.

When you set an FCP device offline, the port and LUN subdirectories are
preserved. Setting an FCP device offline in sysfs interrupts the communication
between Linux and the FCP channel. After a timeout has expired, the port and
LUN attributes indicate that the ports and LUNs are no longer accessible. The
transition of the FCP device to the offline state is synchronous, unless the device is
disconnected.

For disconnected devices, writing 0 to the online sysfs attribute triggers an

asynchronous deregistration process. When this process is completed, the device
with its ports and LUNs is no longer represented in sysfs.

When the FCP device is set back online, the SCSI device names and minor
numbers are freshly assigned. The mapping of devices to names and numbers
might be different from what they were before the FCP device was set offline.

Procedure

There are two methods for setting an FCP device online or offline:
v Use the chccwdev command (see chccwdev - Set CCW device attributes on
page 492). This is the preferred method.
v Alternatively, you can write 1 to an FCP device's online attribute to set it online,
or 0 to set it offline.

Examples
v To set an FCP device with bus ID 0.0.3d0c online issue:
# chccwdev -e 0.0.3d0c

or
# echo 1 > /sys/bus/ccw/drivers/zfcp/0.0.3d0c/online

v To set an FCP device with bus ID 0.0.3d0c offline issue:

# chccwdev -d 0.0.3d0c

or
# echo 0 > /sys/bus/ccw/drivers/zfcp/0.0.3d0c/online

154 Device Drivers, Features, and Commands on SLES 12 SP2

Displaying FCP channel and device information
For each online FCP device, there is a number of read-only attributes in sysfs that
provide information about the corresponding FCP channel and FCP device.

Before you begin

The FCP device must be online for the FCP channel information to be valid.

About this task

The following tables summarize the relevant attributes.

Table 21. Attributes with FCP channel information
Attribute Explanation
card_version Version number that identifies a particular hardware feature.
hardware_version Number that identifies a hardware version for a particular
feature. The initial hardware version of a feature is zero.
This version indicator is increased only for hardware
modifications of the same feature. Appending
hardware_version to card_version results in a hierarchical
version indication for a physical adapter.
lic_version Microcode level.
peer_wwnn WWNN of peer for a point-to-point connection.
peer_wwpn WWPN of peer for a point-to-point connection.
peer_d_id Destination ID of the peer for a point-to-point connection.

Table 22. Attributes with FCP device information

Attribute Explanation
in_recovery Shows if the FCP channel is in recovery (0 or 1).

For the attributes availability, cmb_enable, and cutype, see Device attributes on
page 9. The status attribute is reserved.
Table 23. Relevant transport class attributes, fc_host attributes
Attribute Explanation
maxframe_size Maximum frame size of adapter.
node_name Worldwide node name (WWNN) of adapter.
permanent_port_name WWPN associated with the physical port of the FCP
channel.
port_id A unique ID (N_Port_ID) assigned by the fabric. In an NPIV
setup, each virtual port is assigned a different port_id.
port_name WWPN associated with the FCP device. If N_Port ID
Virtualization is not available, the WWPN of the physical
port (see permanent_port_name).
port_type The port type indicates the topology of the port.
serial_number The 32-byte serial number of the adapter hardware that
provides the FCP channel.
speed Speed of FC link.
supported_classes Supported FC service class.

Chapter 10. SCSI-over-Fibre Channel 155

 Table 23. Relevant transport class attributes, fc_host attributes (continued)
Attribute Explanation
symbolic_name The symbolic port name that is registered with the FC name
server.
supported_speeds Supported speeds.
tgid_bind_type Target binding type.

Table 24. Relevant transport class attributes, fc_host statistics

Attribute Explanation
reset_statistics Writeable attribute to reset statistic counters.
seconds_since_last_reset Seconds since last reset of statistic counters.
tx_frames Transmitted FC frames.
tx_words Transmitted FC words.
rx_frames Received FC frames.
rx_words Received FC words.
lip_count Number of LIP sequences.
nos_count Number of NOS sequences.
error_frames Number of frames that are received in error.
dumped_frames Number of frames that are lost because of lack of host
resources.
link_failure_count Link failure count.
loss_of_sync_count Loss of synchronization count.
loss_of_signal_count Loss of signal count.
prim_seq_protocol_err_count Primitive sequence protocol error count.
invalid_tx_word_count Invalid transmission word count.
invalid_crc_count Invalid CRC count.
fcp_input_requests Number of FCP operations with data input.
fcp_output_requests Number of FCP operations with data output.
fcp_control_requests Number of FCP operations without data movement.
fcp_input_megabytes Megabytes of FCP data input.
fcp_output_megabytes Megabytes of FCP data output.

Procedure

Use the cat command to read an attribute.

v Issue a command of this form to read an attribute:
# cat /sys/bus/ccw/drivers/zfcp/<device_bus_id>/<attribute>

where:
<device_bus_id>
specifies an FCP device that corresponds to the FCP channel.
<attribute>
is one of the attributes in Table 21 on page 155 or Table 22 on page 155.
v To read attributes of the associated Fibre Channel host use:

156 Device Drivers, Features, and Commands on SLES 12 SP2

 # cat /sys/class/fc_host/<host_name>/<attribute>

where:
<host_name>
is the ID of the Fibre Channel host.
<attribute>
is one of the attributes in Table 23 on page 155.
v To read statistics attributes of the FCP channel associated with this Fibre
Channel host, use:
# cat /sys/class/fc_host/<host_name>/statistics/<attribute>

where:
<host_name>
is the ID of the Fibre Channel host.
<attribute>
is one of the attributes in Table 24 on page 156.

Examples
v In this example, information is displayed about an FCP channel that corresponds
to an FCP device with bus ID 0.0.3d0c:
# cat /sys/bus/ccw/drivers/zfcp/0.0.3d0c/hardware_version
0x00000000
# cat /sys/bus/ccw/drivers/zfcp/0.0.3d0c/lic_version
0x00009111

v Alternatively you can use lszfcp (see lszfcp - List zfcp devices on page 609) to
display attributes of an FCP channel:

Chapter 10. SCSI-over-Fibre Channel 157

 # lszfcp -b 0.0.3d0c -a
0.0.3d0c host0
Bus = "ccw"
availability = "good"
card_version = "0x0005"
cmb_enable = "0"
cutype = "1731/03"
devtype = "1732/03"
failed = "0"
hardware_version = "0x00000000"
in_recovery = "0"
lic_version = "0x00009111"
modalias = "ccw:t1731m03dt1732dm03"
online = "1"
peer_d_id = "0x000000"
peer_wwnn = "0x0000000000000000"
peer_wwpn = "0x0000000000000000"
status = "0x5400000a"
uevent = "DRIVER=zfcp"
Class = "fc_host"
active_fc4s = "0x00 0x00 ... 0x00"
dev_loss_tmo = "60"
maxframe_size = "2112 bytes"
node_name = "0x5005076400c89f25"
permanent_port_name = "0xc05076ffe5005611"
port_id = "0x656e00"
port_name = "0xc05076ffe5005611"
port_state = "Online"
port_type = "NPort (fabric via point-to-point)"
serial_number = "IBM02000000089F25"
speed = "8 Gbit"
supported_classes = "Class 2, Class 3"
supported_fc4s = "0x00 0x00 ... 0x00"
supported_speeds = "1 Gbit, 4 Gbit"
symbolic_name = "IBM 2817 020000000EAA14 PCHID: 0391"
tgtid_bind_type = "wwpn (World Wide Port Name)"
Class = "scsi_host"
active_mode = "Initiator"
can_queue = "4096"
cmd_per_lun = "1"
host_busy = "0"
megabytes = "28 0"
proc_name = "zfcp"
prot_capabilities = "0"
prot_guard_type = "0"
queue_full = "0 33333510"
requests = "184085 4 302"
seconds_active = "143"
sg_tablesize = "0"
state = "running"
supported_mode = "Initiator"
unchecked_isa_dma = "0"
unique_id = "5906"
utilization = "6 0 0"

Recovering a failed FCP device

Failed FCP devices are automatically recovered by the zfcp device driver. You can
read the in_recovery attribute to check whether recovery is under way.

Before you begin

The FCP device must be online.

Procedure

Perform these steps to find out the recovery status of an FCP device and, if
needed, start or restart recovery:

158 Device Drivers, Features, and Commands on SLES 12 SP2

 1. Issue a command of this form:
# cat /sys/bus/ccw/drivers/zfcp/<device_bus_id>/in_recovery

The value is 1 if recovery is under way and 0 otherwise. If the value is 0 for a
non-operational FCP device, recovery might have failed. Alternatively, the
device driver might have failed to detect that the FCP device is malfunctioning.
2. To find out whether recovery failed, read the failed attribute. Issue a
command of this form:
# cat /sys/bus/ccw/drivers/zfcp/<device_bus_id>/failed

The value is 1 if recovery failed and 0 otherwise.

3. You can start or restart the recovery process for the FCP device by writing 0 to
the failed attribute. Issue a command of this form:
# echo 0 > /sys/bus/ccw/drivers/zfcp/<device_bus_id>/failed

Example
In the following example, an FCP device with a device bus-ID 0.0.3d0c is
malfunctioning. The first command reveals that recovery is not already under way.
The second command manually starts recovery for the FCP device:
# cat /sys/bus/ccw/drivers/zfcp/0.0.3d0c/in_recovery
0
# echo 0 > /sys/bus/ccw/drivers/zfcp/0.0.3d0c/failed

Finding out whether NPIV is in use

An FCP device runs in NPIV mode if the port_type attribute of the FCP device
attribute contains the string "NPIV". Alternatively, if the applicable
permanent_port_name and port_name are not the same and are not NULL.

Procedure

Read the port_type attribute of the FCP device.

For example:
# cat /sys/bus/ccw/drivers/zfcp/0.0.1940/host0/fc_host/host0/port_type
NPIV VPORT

Alternatively, compare the values of the permanent_port_name attribute and the

port_name.

Tip: You can use lszfcp (see lszfcp - List zfcp devices on page 609) to list the
FCP device attributes.

Chapter 10. SCSI-over-Fibre Channel 159

 Example
# lszfcp -b 0.0.1940 -a
0.0.1940 host0
Bus = "ccw"
availability = "good"
...
Class = "fc_host"
active_fc4s = "0x00 0x00 ... 0x00"
dev_loss_tmo = "60"
maxframe_size = "2112 bytes"
node_name = "0x5005076400c1ebae"
permanent_port_name = "0x50050764016219a0"
port_id = "0x65ee01"
port_name = "0xc05076ffef805388"
port_state = "Online"
port_type = "NPIV VPORT"
...
symbolic_name = "DEVNO: 0.0.1940 NAME: mylinux"
...

The port_type attribute directly indicates that NPIV is used. The example also
shows that permanent_port_name is different from port_name and neither is NULL.
The example also shows the symbolic_name attribute that shows the symbolic port
name that was registered on the FC name server.

Logging I/O subchannel status information

When severe errors occur for an FCP device, the FCP device driver triggers a set of
log entries with I/O subchannel status information.

The log entries are available through the SE Console Actions Work Area with the
View Console Logs function. In the list of logs, these entries have the prefix 1F00.
The content of the entries is intended for support specialists.

Working with target ports

You can scan for ports, display port information, recover a port, or remove a port.

Working with target ports comprises the following tasks:

v Scanning for ports
v Controlling automatic port scanning on page 161
v Displaying port information on page 164
v Recovering a failed port on page 165
v Removing ports on page 166

Scanning for ports

Newly available target ports are discovered. However, you might want to trigger a
port scan to re-create accidentally removed port information or to assure that all
ports are present.

Before you begin

The FCP device must be online.

160 Device Drivers, Features, and Commands on SLES 12 SP2

 About this task

The zfcp device driver automatically adds port information to sysfs when:
v The FCP device is set online
v Target ports are added to the Fibre Channel fabric, unless the module parameter
no_auto_port_rescan is set to 1. See Setting up the zfcp device driver on page
151.
Scanning for ports might take some time to complete. Commands that you issue
against ports or LUNs while scanning is in progress are delayed and processed
when port scanning is completed.

Use the port_rescan attribute if a remote storage port was accidentally deleted
from the adapter configuration or if you are unsure whether all ports were added
to sysfs.

Procedure

Issue a command of this form:

# echo 1 > /sys/bus/ccw/drivers/zfcp/<device_bus_id>/port_rescan

where <device_bus_id> specifies the FCP device through which the target ports are
attached.

Tip: List the contents of /sys/bus/ccw/drivers/zfcp/<device_bus_id> to find out

which ports are currently configured for the FCP device.

Example
In this example, a port with WWPN 0x500507630303c562 is already configured for
an FCP device with bus ID 0.0.3d0c. An additional target port with WWPN
0x500507630300c562 is automatically configured by triggering a port scan.
# ls /sys/bus/ccw/drivers/zfcp/0.0.3d0c/0x*
0x500507630303c562
# echo 1 > /sys/bus/ccw/drivers/zfcp/0.0.3d0c/port_rescan
# ls /sys/bus/ccw/drivers/zfcp/0.0.3d0c/0x*
0x500507630303c562
0x500507630300c562

Controlling automatic port scanning

Automatic port scanning includes two zfcp parameters that improve the behaviour
of Linux instances in SANs. These zfcp parameters are set to default values that
work well for most installations.

If needed, you can fine-tune the frequency and timing of automatic port scans with
the zfcp parameters port_scan_backoff and port_scan_ratelimit.

You can enable automatic port scanning with the zfcp parameter
no_auto_port_rescan=0. This value is the default.

About this task

In a large installation, where many Linux instances receive the same notifications
of SAN changes, multiple instances might trigger scans simultaneously and too
Chapter 10. SCSI-over-Fibre Channel 161
 frequently. See Figure 29

Figure 29. Numerous port scans in a Linux installation

These scans might put unnecessary load on the name server function of fabric
switches and potentially result in late or inconclusive results.

You can avoid excessive scanning, yet still ensure that a port scan is eventually
conducted. You can control port scanning with the zfcp parameters:
port_scan_ratelimit
sets the minimum delay, in milliseconds, between automatic port scans of
your Linux instance. The default value is 60000 milliseconds. To turn off
the rate limit, specify 0.
port_scan_backoff
sets an additional random delay, in milliseconds, in which the port scans of
your Linux instance are spread. In an installation with multiple Linux
instances, use this zfcp parameter for every Linux instance to spread scans
to avoid potential multiple simultaneous scans. The default value is 500
milliseconds. To turn off the random delay, specify 0.

Use module parameters to set values for port scanning. See Setting up the zfcp
device driver on page 151 for zfcp attributes. On a running Linux system, you can
also query or set these values by using the sysfs attributes with the same names.

Using port_scan_ratelimit reduces the number of scans, as shown in Figure 30

Figure 30. Port scan behavior with scan rate limit.

However, if the rate limit is set to the same value, the scans can still occur almost
simultaneously, as for FCP device A and B in Linux 1.

Using port_scan_backoff and port_scan_ratelimit together delays port scans

even further and avoids simultaneous scans, as shown in Figure 31 on page 163. In

162 Device Drivers, Features, and Commands on SLES 12 SP2

the figure, FCP devices A and B in Linux 1 have the same rate limit and the same
backoff values. The random element in the backoff value causes the scans to occur
at slightly different times.

Figure 31. Port scan behavior with backoff and scan rate limit.

Procedure

Use port_scan_backoff and port_scan_ratelimit together or separately to tune the

behavior of port scanning:
v To avoid too frequent scanning, set a minimum wait time between two
consecutive scans for the same Linux instance. Use the port_scan_ratelimit
sysfs attribute. By default, port_scan_ratelimit is turned on and has a value of
60000 milliseconds. For example, to specify an attribute value of 12 seconds,
issue:
# echo 12000 > /sys/module/zfcp/parameters/port_scan_ratelimit

v To further spread scans over a certain time and thus avoid multiple
simultaneous scans, set the port_scan_backoff sysfs attribute. By default,
port_scan_backoff is turned on and has a value of 500 milliseconds. For
example, to query the setting, issue a command of this form:
# cat /sys/module/zfcp/parameters/port_scan_backoff
500

To set the attribute to 1 second, issue:

# echo 1000 > /sys/module/zfcp/parameters/port_scan_backoff

Results

The automatic port scans are delayed by the values specified. If a SAN notification
is received during the rate limit time, a port scan is conducted immediately after
the delay time passed.

Depending on the port event, one or more of the three zfcp parameters are
evaluated to schedule a port scan. For example, port scans that are triggered
manually through sysfs are not delayed. Table 25 on page 164 shows which events
evaluate which zfcp parameters.

Chapter 10. SCSI-over-Fibre Channel 163

 Table 25. Port events and their use of port scanning zfcp parameters.
zfcp parameter no_auto_port_rescan port_scan_backoff port_scan_ratelimit
Event
FCP device resume Yes Yes No
User sets FCP device No Yes No
online
User initiates a port No No No
scan
User starts FCP Yes Yes Yes
device recovery
Automatic FCP Yes Yes Yes
device recovery
SAN change Yes Yes Yes
notification

Displaying port information

For each target port, there is a number of read-only sysfs attributes with port
information.

About this task

Table 26 and Table 27 summarize the relevant attributes.

Table 26. zfcp-specific attributes with port information within the FCP device sysfs tree
Attribute Explanation
access_denied This attribute is obsolete. The value is always 0.
in_recovery Shows if port is in recovery (0 or 1)

Table 27. Transport class attributes with port information

Attribute Explanation
node_name WWNN of the remote port.
port_name WWPN of remote port.
port_id Destination ID of remote port
port_state State of remote port.
roles Role of remote port (usually FCP target).
scsi_target_id Linux SCSI ID of remote port.
supported_classes Supported classes of service.

Procedure

Use the cat command to read an attribute.

v Issue a command of this form to read a zfcp-specific attribute:
# cat /sys/bus/ccw/drivers/zfcp/<device_bus_id>/<wwpn>/<attribute>

where:

164 Device Drivers, Features, and Commands on SLES 12 SP2

 <device_bus_id>
specifies the FCP device.
<wwpn>
is the WWPN of the target port.
<attribute>
is one of the attributes in Table 26 on page 164.
v To read transport class attributes of the associated target port, use a command of
this form:
# cat /sys/class/fc_remote_port/<rport_name>/<attribute>

where:
<rport_name>
is the name of the target port.
<attribute>
is one of the attributes in Table 27 on page 164.

Tip: With the HBA API package installed, you can also use the zfcp_ping and
zfcp_show commands to find out more about your ports. See Tools for
investigating your SAN configuration on page 185.

Examples
v In this example, information is displayed for a target port 0x500507630300c562
that is attached through an FCP device with bus ID 0.0.3d0c:
# cat /sys/bus/ccw/drivers/zfcp/0.0.3d0c/0x500507630300c562/in_recovery
0

v To display transport class attributes of a target port you can use lszfcp:
# lszfcp -p 0x500507630300c562 -a
0.0.3d0c/0x500507630300c562 rport-0:0-0
...
Class = "fc_remote_ports"
active_fc4s = "0x00 0x00 0x01 ...
dev_loss_tmo = "60"
fast_io_fail_tmo = "off"
maxframe_size = "2048 bytes"
node_name = "0x5005076303ffc562"
port_id = "0x652113"
port_name = "0x500507630300c562"
port_state = "Online"
roles = "FCP Target"
scsi_target_id = "0"
supported_classes = "Class 2, Class 3"
...

Recovering a failed port

Failed target ports are automatically recovered by the zfcp device driver. You can
read the in_recovery attribute to check whether recovery is under way.

Before you begin

The FCP device must be online.

Chapter 10. SCSI-over-Fibre Channel 165

 Procedure

Perform these steps to find out the recovery status of a port and, if needed, start or
restart recovery:
1. Issue a command of this form:
# cat /sys/bus/ccw/drivers/zfcp/<device_bus_id>/<wwpn>/in_recovery

where:
<device_bus_id>
specifies the FCP device.
<wwpn>
is the WWPN of the target port.
The value is 1 if recovery is under way, and 0 otherwise. If the value is 0 for a
non-operational port, recovery might have failed, or the device driver might
have failed to detect that the port is malfunctioning.
2. To find out whether recovery failed, read the failed attribute. Issue a
command of this form:
# cat /sys/bus/ccw/drivers/zfcp/<device_bus_id>/<wwpn>/failed

The value is 1 if recovery failed, and 0 otherwise.

3. You can start or restart the recovery process for the port by writing 0 to the
failed attribute. Issue a command of this form:
# echo 0 > /sys/bus/ccw/drivers/zfcp/<device_bus_id>/<wwpn>/failed

Example

In the following example, a port with WWPN 0x500507630300c562 that is attached

through an FCP device with bus ID 0.0.3d0c is malfunctioning. The first command
reveals that recovery is not already under way. The second command manually
starts recovery for the port:
# cat /sys/bus/ccw/drivers/zfcp/0.0.3d0c/0x500507630300c562/in_recovery
0
# echo 0 > /sys/bus/ccw/drivers/zfcp/0.0.3d0c/0x500507630300c562/failed

Removing ports
Removing unused ports can save FCP channel resources. Additionally setting the
no_auto_port_rescan attribute avoids unnecessary attempts to recover unused
remote ports.

Before you begin

The FCP device must be online.

About this task

List the contents of /sys/bus/ccw/drivers/zfcp/<device_bus_id> to find out which

ports are currently configured for the FCP device.

166 Device Drivers, Features, and Commands on SLES 12 SP2

 You cannot remove a port while SCSI devices are configured for it (see
Configuring SCSI devices on page 168) or if the port is in use, for example, by
error recovery.

Note: The next port scan will attach all available ports, including any previously
removed ports. To prevent removed ports from being reattached automatically, use
zoning or the no_auto_port_rescan module parameter, see Setting up the zfcp
device driver on page 151.

Procedure

Issue a command of this form:

# echo <wwpn> > /sys/bus/ccw/drivers/zfcp/<device_bus_id>/port_remove

where:
<device_bus_id>
specifies the FCP device.
<wwpn>
is the WWPN of the port to be removed.

Example

In this example, two ports with WWPN 0x500507630303c562 and

0x500507630300c562 are configured for an FCP device with bus ID 0.0.3d0c. The
port with WWPN 0x500507630303c562 is then removed.
# ls /sys/bus/ccw/drivers/zfcp/0.0.3d0c/0x*
0x500507630303c562
0x500507630300c562
# echo 0x500507630303c562 > /sys/bus/ccw/drivers/zfcp/0.0.3d0c/port_remove
# ls /sys/bus/ccw/drivers/zfcp/0.0.3d0c/0x*
0x500507630300c562

Working with SCSI devices

In an NPIV setup with auto lun scan, the SCSI devices are configured
automatically. Otherwise, you must configure FCP LUNs to obtain SCSI devices. In
both cases, you can configure SCSI devices, display information, and remove SCSI
devices.

Working with SCSI devices comprises the following tasks:

v Configuring SCSI devices on page 168
v Mapping the representations of SCSI devices in sysfs on page 170
v Displaying information about SCSI devices on page 171
v Setting the queue depth on page 174
v Recovering failed SCSI devices on page 175
v Updating the information about SCSI devices on page 176
v Setting the SCSI command timeout on page 176
v Controlling the SCSI device state on page 177
v Removing SCSI devices on page 178

Chapter 10. SCSI-over-Fibre Channel 167

 Configuring SCSI devices
FCP devices that use NPIV mode detect the LUNs automatically and no
configuring is necessary. If needed, write the LUN to be configured to the sysfs
unit_add attribute of the applicable target port.

For each FCP device that uses NPIV mode and if you did not disable automatic
LUN scanning (see Setting up the zfcp device driver on page 151), the LUNs are
configured for you. In this case, no FCP LUN entries are created under
/sys/bus/ccw/drivers/zfcp/<device_bus_id>/<wwpn>.

To find out whether an FCP device is using NPIV mode, check the port_type
attribute, for example:
# cat /sys/bus/ccw/drivers/zfcp/0.0.1901/host0/fc_host/host0/port_type
NPIV VPORT

To find out whether automatic LUN scanning is enabled, check the current setting
of the module parameter zfcp.allow_lun_scan. The example below shows
automatic LUN scanning as turned on.
# cat /sys/module/zfcp/parameters/allow_lun_scan
Y

Important: Configuration changes can directly or indirectly affect information that

is required to mount the root file system. Such changes require an update of the
initrd of both the auxiliary kernel and the target kernel, followed by a re-write of
the boot record (see Rebuilding the initial RAM disk image on page 56).

Automatically attached SCSI devices

FCP devices that use NPIV mode detect the LUNs automatically and no
configuring is necessary.

In this case, no FCP LUN entries are created under /sys/bus/ccw/drivers/zfcp/

<device_bus_id>/<wwpn>.

What to do next

To check whether a SCSI device is registered, check for a directory with the name
of the LUN in /sys/bus/scsi/devices. If there is no SCSI device for this LUN, the
LUN is not valid in the storage system, or the FCP device is offline in Linux.

Manually configured FCP LUNs and their SCSI devices

For FCP devices that do not use NPIV mode, or if automatic LUN scanning is
disabled, FCP LUNs must be configured manually to obtain SCSI devices.

Before you begin

Attention: Use this procedure only to dynamically test configuration settings.

To configure persistent setting in a production system, use one of the following

options:
v The YaST GUI yast2 zfcp. If cio_ignore is enabled, you might need to free
blacklisted FCP devices beforehand by using yast2 cio. If cio_ignore is enabled,
you might need to free blacklisted FCP devices beforehand by using yast cio

168 Device Drivers, Features, and Commands on SLES 12 SP2

v The text-based interface yast zfcp
v The command line, use zfcp_disk_configure. Cio_ignore does not apply here.
See the section about IBM z Systems hard disk configuration in the SUSE Linux
Enterprise Server 12 SP2 Deployment Guide, and the procedure about configuring a
zFCP disk in SUSE Linux Enterprise Server 12 SP2 Administration Guide. The
command-line tools described work not only inside the rescue environment but
also in a regularly installed Linux instance.

You can always specify additional zfcp module parameters as explained in

Chapter 3, Kernel and module parameters, on page 23

Procedure

If your FCP device does not use NPIV mode, or if you have disabled automatic
LUN scanning, proceed as follows:

To configure a SCSI device for a target port, write the device's LUN to the port's
unit_add attribute. Issue a command of this form:
# echo <fcp_lun> > /sys/bus/ccw/drivers/zfcp/<device_bus_id>/<wwpn>/unit_add

where:
<fcp_lun>
is the LUN of the SCSI device to be configured. The LUN is a 16 digit
hexadecimal value padded with zeros, for example 0x4010403300000000.
<device_bus_id>
specifies the FCP device.
<wwpn>
is the WWPN of the target port.

This command starts a process with multiple steps:

1. It creates a directory in /sys/bus/ccw/drivers/zfcp/<device_bus_id>/<wwpn>
with the LUN as the directory name. The directory is part of the list of all
LUNs to configure. Without NPIV or with auto LUN scanning disabled, zfcp
registers only FCP LUNs contained in this list with the Linux SCSI stack in the
next step.
2. It initiates the registration of the SCSI device with the Linux SCSI stack. The
FCP device must be online for this step.
3. It waits until the Linux SCSI stack registration completes successfully or returns
an error. It then returns control to the shell. A successful registration creates a
sysfs entry in the SCSI branch (see Mapping the representations of SCSI
devices in sysfs on page 170).

Example

In this example, a target port with WWPN 0x500507630300c562 is attached through

an FCP device with bus ID 0.0.3d0c. A SCSI device with LUN 0x4010403200000000
is already configured for the port. An additional SCSI device with LUN
0x4010403300000000 is added to the port.

Chapter 10. SCSI-over-Fibre Channel 169

 # ls /sys/bus/ccw/drivers/zfcp/0.0.3d0c/0x500507630300c562/0x*
0x4010403200000000
# echo 0x4010403300000000 > /sys/bus/ccw/drivers/zfcp/0.0.3d0c/0x500507630300c562/unit_add
# ls /sys/bus/ccw/drivers/zfcp/0.0.3d0c/0x500507630300c562/0x*
0x4010403200000000
0x4010403300000000

What to do next

To check whether a SCSI device is registered for the configured LUN, check for a
directory with the name of the LUN in /sys/bus/scsi/devices. If there is no SCSI
device for this LUN, the LUN is not valid in the storage system, or the FCP device
is offline in Linux.

To see which LUNs are currently configured for the port, list the contents of
/sys/bus/ccw/drivers/zfcp/<device_bus_id>/<wwpn>.

Mapping the representations of SCSI devices in sysfs

Each SCSI device that is configured is represented by multiple directories in sysfs,
in particular, within the SCSI branch. Only manually configured LUNs are also
represented within the zfcp branch.

About this task

The directory in the sysfs SCSI branch has the following form:
/sys/bus/scsi/devices/<scsi_host_no>:0:<scsi_id>:<scsi_lun>

where:
<scsi_host_no>
is the SCSI host number that corresponds to the FCP device.
<scsi_id>
is the SCSI ID of the target port.
<scsi_lun>
is the LUN of the SCSI device.

The values for <scsi_id> and <scsi_lun> depend on the storage device. Often, they
are single-digit numbers but for some storage devices they have numerous digits.

For manually configured FCP LUNs, see Manually configured FCP LUNs and
their SCSI devices on page 168 for details about the directory in the zfcp branch.

Figure 32 on page 171 shows how the directory name is composed in the sysfs
SCSI branch. The sysfs zfcp branch only exists for manually configured FCP LUNs.
For manually configured FCP LUNs, the directory name is composed of attributes
of consecutive directories and you can find the name of the directory in the sysfs
SCSI branch by reading the corresponding attributes in the zfcp branch.

170 Device Drivers, Features, and Commands on SLES 12 SP2

 Figure 32. SCSI devices in sysfs

The hba_id, wwpn, and fcp_lun attributes of the SCSI device in the SCSI branch
match the names of the <device_bus_id>, <wwpn> and <fcp_lun> directories for the
same SCSI device in the zfcp branch.

Procedure

Use lszfcp (see lszfcp - List zfcp devices on page 609) to map the two
representations of a SCSI device.

Example

This example shows how to use lszfcp to display the name of the SCSI device that
corresponds to a zfcp unit, for example:
# lszfcp -l 0x4010403200000000
0.0.3d0c/0x500507630300c562/0x4010403200000000 0:0:0:0

In the example, the output informs you that the unit with the LUN
0x4010403200000000, which is configured on a port with the WWPN
0x500507630300c562 for an FCP device with bus ID 0.0.3d0c, maps to SCSI device
"0:0:0:0".

To confirm that the SCSI device belongs to the zfcp unit:

# cat /sys/bus/scsi/devices/0:0:0:0/hba_id
0.0.3d0c
# cat /sys/bus/scsi/devices/0:0:0:0/wwpn
0x500507630300c562
# cat /sys/bus/scsi/devices/0:0:0:0/fcp_lun
0x4010403200000000

Displaying information about SCSI devices

For each SCSI device, there is a number of read-only attributes in sysfs that
provide information for the device.

About this task

Table 28 on page 172 summarizes the read-only attributes for manually configured
FCP LUNs, including those attributes that indicate whether the device access is
restricted by access control software on the FCP channel. These attributes can be
found in the zfcp branch of sysfs. The path has the form:
/sys/bus/ccw/drivers/zfcp/<device_bus_id>/<wwpn>/<fcp_lun>/<attribute>

Chapter 10. SCSI-over-Fibre Channel 171

 Table 28. Attributes of manually configured FCP LUNs with device access information
Attribute Explanation
access_denied Flag that indicates whether access to the device is restricted by the
FCP channel.

The value is 1 if access is denied and 0 if access is permitted.

If access is denied to your Linux instance, confirm that your SCSI

devices are configured as intended. Also, be sure that you really want
to share a SCSI device. For shared access to a SCSI device, preferably
use NPIV (see N_Port ID Virtualization for FCP channels on page
150). You might also use different FCP channels or target ports.
access_shared This attribute is obsolete. The value is always 0.
access_readonly This attribute is obsolete. The value is always 0.
in_recovery Shows if unit is in recovery (0 or 1)

Table 29 lists further read-only attributes with information about the SCSI device.
These attributes can be found in the SCSI branch of sysfs. The path has the form:
/sys/class/scsi_device/<device_name>/device/<attribute>

Table 29. SCSI device class attributes

Attribute Explanation
device_blocked Flag that indicates whether the device is in blocked state (0 or 1).
iocounterbits The number of bits used for I/O counters.
iodone_cnt The number of completed or rejected SCSI commands.
ioerr_cnt The number of SCSI commands that completed with an error.
iorequest_cnt The number of issued SCSI commands.
queue_type The type of queue for the SCSI device. The value can be one of the
following types:
v none
v simple
v ordered
model The model of the SCSI device, received from inquiry data.
rev The revision of the SCSI device, received from inquiry data.
scsi_level The SCSI revision level, received from inquiry data.
type The type of the SCSI device, received from inquiry data.
vendor The vendor of the SCSI device, received from inquiry data.
fcp_lun The LUN of the SCSI device in 64-bit format.
hba_id The bus ID of the SCSI device.
wwpn The WWPN of the remote port.
zfcp_access_denied Flag that indicates whether access to the device is restricted by the
FCP channel.

The value is 1 if access is denied and 0 if access is permitted.

If access is denied to your Linux instance, confirm that your SCSI

devices are configured as intended. Also, be sure that you really want
to share a SCSI device. For shared access to a SCSI device, preferably
use NPIV (see N_Port ID Virtualization for FCP channels on page
150). You might also use different FCP channels or target ports.

172 Device Drivers, Features, and Commands on SLES 12 SP2

Table 29. SCSI device class attributes (continued)
Attribute Explanation
zfcp_in_recovery Shows if unit is in recovery (0 or 1).

Procedure

Issue a command of this form to read an attribute of a manually configured FCP

LUN:
# cat /sys/bus/ccw/drivers/zfcp/<device_bus_id>/<wwpn>/<fcp_lun>/<attribute>

where:
<device_bus_id>
specifies the FCP device.
<wwpn>
is the WWPN of the target port.
<fcp_lun>
is the FCP LUN of the SCSI device.
<attribute>
is one of the attributes in Table 28 on page 172.

Use the lszfcp command (see lszfcp - List zfcp devices on page 609) to display
information about the associated SCSI device.
Alternatively, you can use sysfs to read the information. To read attributes of the
associated SCSI device, use a command of this form:
# cat /sys/class/scsi_device/<device_name>/device/<attribute>

where:
<device_name>
is the name of the associated SCSI device.
<attribute>
is one of the attributes in Table 29 on page 172.

Tip: For SCSI tape devices, you can display a summary of this information by
using the lstape command (see lstape - List tape devices on page 597).

Examples
v In this example, information is displayed for a manually configured FCP LUN
with LUN 0x4010403200000000 that is accessed through a target port with
WWPN 0x500507630300c562 and is attached through an FCP device 0.0.3d0c. For
the device, access is permitted.
# cat /sys/bus/ccw/drivers/zfcp/0.0.3d0c/0x500507630300c562/0x4010403200000000/access_denied
0

For the device to be accessible, the access_denied attribute of the target port,
0x500507630300c562, must also be 0 (see Displaying port information on page
164).
v You can use lszfcp to display attributes of a SCSI device:

Chapter 10. SCSI-over-Fibre Channel 173

 # lszfcp -l 0x4010403200000000 -a
0.0.3d0c/0x500507630300c562/0x4010403200000000 0:0:0:0
Class = "scsi_device"
...
device_blocked = "0"
...
fcp_lun = "0x4010403200000000"
hba_id = "0.0.3d0c"
iocounterbits = "32"
iodone_cnt = "0xbe"
ioerr_cnt = "0x2"
iorequest_cnt = "0xbe"
...
model = "2107900"
queue_depth = "32"
queue_ramp_up_period = "120000"
queue_type = "simple"
...
rev = ".166"
scsi_level = "6"
state = "running"
timeout = "30"
type = "0"
uevent = "DEVTYPE=scsi_device"
vendor = "IBM"
...
wwpn = "0x500507630300c562"
zfcp_access_denied = "0"
zfcp_failed = "0"
zfcp_in_recovery = "0"
zfcp_status = "0x54000000"

Setting the queue depth

The Linux SCSI code automatically adjusts the queue depth as necessary. Changing
the queue depth is usually a storage server requirement.

Before you begin

Check the documentation of the storage server used or contact your storage server
support group to establish if there is a need to change this setting.

About this task

The value of the queue_depth kernel parameter (see Setting up the zfcp device
driver on page 151) is used as the default queue depth of new SCSI devices. You
can query the queue depth by issuing a command of this form:
# cat /sys/bus/scsi/devices/<SCSI device>/queue_depth

Example:
# cat /sys/bus/scsi/devices/0:0:19:1086537744/queue_depth
16

You can change the queue depth of each SCSI device by writing to the
queue_depth attribute, for example:
# echo 8 > /sys/bus/scsi/devices/0:0:19:1086537744/queue_depth
# cat /sys/bus/scsi/devices/0:0:19:1086537744/queue_depth
8

174 Device Drivers, Features, and Commands on SLES 12 SP2

 This method is useful on a running system where you want to make dynamic
changes. If you want to make the changes persistent across IPLs, you can:
v Use the kernel or module parameter.
v Write a udev rule to change the setting for each new SCSI device.

Linux forwards SCSI commands to the storage server until the number of pending
commands exceeds the queue depth. If the server lacks the resources to process a
SCSI command, Linux queues the command for a later retry and decreases the
queue depth counter. Linux then waits for a defined ramp-up period. If no
indications of resource problems occur within this period, Linux increases the
queue depth counter until reaching the previously set maximum value. To query
the current value for the queue ramp-up period in milliseconds:
# cat /sys/bus/scsi/devices/0:0:13:1086537744/queue_ramp_up_period
120000

To set a new value for the queue ramp-up period in milliseconds:

# echo 1000 > /sys/bus/scsi/devices/0:0:13:1086537744/queue_ramp_up_period

Recovering failed SCSI devices

Failed SCSI devices are automatically recovered by the zfcp device driver. You can
read the zfcp_in_recovery attribute to check whether recovery is under way.

Before you begin

The FCP device must be online.

Procedure

Perform the following steps to check the recovery status of a failed SCSI device:
1. Check the value of the zfcp_in_recovery attribute. Issue the lszfcp command:
# lszfcp -l <LUN> -a

where <LUN> is the LUN of the associated SCSI device.

Alternatively, you can issue a command of this form:
# cat /sys/class/scsi_device/<device_name>/device/zfcp_in_recovery

The value is 1 if recovery is under way and 0 otherwise. If the value is 0 for a
non-operational SCSI device, recovery might have failed. Alternatively, the
device driver might have failed to detect that the SCSI device is
malfunctioning.
2. To find out whether recovery failed, read the zfcp_failed attribute. Either use
the lszfcp command again, or issue a command of this form:
# cat /sys/class/scsi_device/<device_name>/device/zfcp_failed

The value is 1 if recovery failed, and 0 otherwise.

3. You can start or restart the recovery process for the SCSI device by writing 0 to
the zfcp_failed attribute. Issue a command of this form:

Chapter 10. SCSI-over-Fibre Channel 175

 # echo 0 > /sys/class/scsi_device/<device_name>/device/zfcp_failed

Example

In the following example, SCSI device 0:0:0:0 is malfunctioning. The first command
reveals that recovery is not already under way. The second command manually
starts recovery for the SCSI device:
# cat /sys/class/scsi_device/0:0:0:0/device/zfcp_in_recovery
0
# echo 0 > /sys/class/scsi_device/0:0:0:0/device/zfcp_failed

What to do next

If you manually configured an FCP LUN (see Manually configured FCP LUNs
and their SCSI devices on page 168), but did not get a corresponding SCSI device,
you can also use the corresponding FCP LUN sysfs attributes, in_recovery and
failed, to check on recovery. See Table 28 on page 172.

Updating the information about SCSI devices

Use the rescan attribute of the SCSI device to detect changes to a storage device on
the storage server that are made after the device was discovered.

Before you begin

The FCP device must be online.

About this task

The initial information about the available SCSI devices is discovered automatically
when LUNs first become available.

Procedure

To update the information about a SCSI device issue a command of this form:
# echo <string> > /sys/bus/scsi/devices/<scsi_host_no>:0:<scsi_id>:<scsi_lun>/rescan

where <string> is any alphanumeric string and the other variables have the same
meaning as in Mapping the representations of SCSI devices in sysfs on page 170.

Example

In the following example, the information about a SCSI device 1:0:18:1086537744 is

updated:
# echo 1 > /sys/bus/scsi/devices/1:0:18:1086537744/rescan

Setting the SCSI command timeout

You can change the timeout if the default is not suitable for your storage system.

176 Device Drivers, Features, and Commands on SLES 12 SP2

 Before you begin

The FCP device must be online.

About this task

There is a timeout for SCSI commands. If the timeout expires before a SCSI
command completes, error recovery starts. The default timeout is 30 seconds.

To find out the current timeout, read the timeout attribute of the SCSI device:
# cat /sys/bus/scsi/devices/<scsi_host_no>:0:<scsi_id>:<scsi_lun>/timeout

where the variables have the same meaning as in Mapping the representations of
SCSI devices in sysfs on page 170.

The attribute value specifies the timeout in seconds.

Procedure

To set a different timeout, enter a command of this form:

# echo <timeout> > /sys/bus/scsi/devices/<scsi_host_no>:0:<scsi_id>:<scsi_lun>/timeout

where <timeout> is the new timeout in seconds.

Example

In the following example, the timeout of a SCSI device 1:0:18:1086537744 is first

read and then set to 45 seconds:
# cat /sys/bus/scsi/devices/1:0:18:1086537744/timeout
30
# echo 45 > /sys/bus/scsi/devices/1:0:18:1086537744/timeout

Controlling the SCSI device state

You can use the state attribute of the SCSI device to set a SCSI device back online
if it was set offline by error recovery.

Before you begin

The FCP device must be online.

About this task

If the connection to a storage system is working but the storage system has a
problem, the error recovery might set the SCSI device offline. This condition is
indicated by a message like Device offlined - not ready after error recovery.

To find out the current state of the device, read the state attribute:
# cat /sys/bus/scsi/devices/<scsi_host_no>:0:<scsi_id>:<scsi_lun>/state

Chapter 10. SCSI-over-Fibre Channel 177

 where the variables have the same meaning as in Mapping the representations of
SCSI devices in sysfs on page 170. The state can be:
running
The SCSI device can be used for running regular I/O requests.
cancel The data structure for the device is being removed.
deleted
Follows the cancel state when the data structure for the device is being
removed.
quiesce
No I/O requests are sent to the device, only special requests for managing
the device. This state is used when the system is suspended.
offline
Error recovery for the SCSI device has failed.
blocked
Error recovery is in progress and the device cannot be used until the
recovery process is completed.

Procedure
To set an offline device online again, write running to the state attribute.

Issue a command of this form:

# echo running > /sys/bus/scsi/devices/<scsi_host_no>:0:<scsi_id>:<scsi_lun>/state

Example

In the following example, SCSI device 1:0:18:1086537744 is offline and is then set
online again:
# cat /sys/bus/scsi/devices/1:0:18:1086537744/state
offline
# echo running > /sys/bus/scsi/devices/1:0:18:1086537744/state

Removing SCSI devices

How to remove a SCSI device depends on whether your environment is set up to
use NPIV.

Important: Configuration changes can directly or indirectly affect information that

is required to mount the root file system. Such changes require an update of the
initrd of both the auxiliary kernel and the target kernel, followed by a re-write of
the boot record (see Rebuilding the initial RAM disk image on page 56).

Removing automatically attached SCSI devices

When running with NPIV and the automatic LUN scan, you can temporarily
delete a SCSI device by writing 1 to the delete attribute of the directory that
represents the device in the sysfs SCSI branch.

About this task

See Mapping the representations of SCSI devices in sysfs on page 170 about how
to find this directory.

Note: These steps delete the SCSI device only temporarily, until the next automatic
or user triggered Linux SCSI target scan. The scan automatically adds the SCSI

178 Device Drivers, Features, and Commands on SLES 12 SP2

devices again, unless the LUNs were deconfigured on the storage target. To
permanently delete SCSI devices, you must disable automatic LUN scannning and
configure all LUNs manually, see Manually configured FCP LUNs and their SCSI
devices on page 168.

Procedure

Issue a command of this form:

# echo 1 > /sys/bus/scsi/devices/<device>/delete

Example

In this example, a SCSI device with LUN 0x4010403700000000 is to be removed.

Before the device is deleted, the corresponding device in the sysfs SCSI branch is
found with an lszfcp command.
# lszfcp -l 0x4010403700000000
0.0.3d0f/0x500507630300c567/0x4010403700000000 0:0:3:1
# echo 1 > /sys/bus/scsi/devices/0:0:3:1/delete

Removing manually configured FCP LUNs and their SCSI device

Use the unit_remove attribute of the appropriate target port to remove a SCSI
device if your environment is not set up to use NPIV or if you disabled automatic
LUN scan.

For details about disabling automatic LUN scan, see Setting up the zfcp device
driver on page 151.

Before you begin

Attention: Use this procedure only to dynamically test configuration settings.

To configure persistent setting in a production system, use one of the following

options:
v The YaST GUI yast2 zfcp
v The text-based interface yast zfcp
v The command line, use zfcp_disk_configure
See the section about IBM z Systems hard disk configuration in the SUSE Linux
Enterprise Server 12 SP2 Deployment Guide, and the procedure about configuring a
zFCP disk in SUSE Linux Enterprise Server 12 SP2 Administration Guide. The
command-line tools described work not only inside the rescue environment but
also in a regularly installed Linux instance.

Procedure

Follow these steps to remove a manually configured FCP LUN and its SCSI device:
1. Optional: To manually unregister the SCSI device, write 1 to the delete
attribute of the directory that represents the device in the sysfs SCSI branch.
See Mapping the representations of SCSI devices in sysfs on page 170 for
information about how to find this directory. Issue a command of this form:
# echo 1 > /sys/bus/scsi/devices/<device>/delete

Chapter 10. SCSI-over-Fibre Channel 179

 2. Remove the SCSI device from the target port by writing the LUN of the device
to the unit_remove attribute of the port. Issue a command of this form:
# echo <fcp_lun> > /sys/bus/ccw/drivers/zfcp/<device_bus_id>/<wwpn>/unit_remove

where:
<fcp_lun>
is the LUN of the SCSI device to be configured. The LUN is a 16 digit
hexadecimal value padded with zeros, for example
0x4010403300000000.
<device_bus_id>
specifies the FCP device.
<wwpn>
is the WWPN of the target port.

Removing a LUN with unit_remove automatically unregisters the SCSI device

first.

Example

The following example removes a SCSI device with LUN 0x4010403200000000,

accessed through a target port with WWPN 0x500507630300c562 and is attached
through an FCP device with bus ID 0.0.3d0c. The corresponding directory in the
sysfs SCSI branch is assumed to be /sys/bus/scsi/devices/0:0:1:1.
1. Optionally, unregister the device:
# echo 1 > /sys/bus/scsi/devices/0:0:1:1/delete

2. Remove the device (if not done in previous step) and the LUN:
# echo 0x4010403200000000 > /sys/bus/ccw/drivers/zfcp/0.0.3d0c/0x500507630300c562/unit_remove

Confirming end-to-end data consistency checking

There are different types of end-to-end data consistency checking, with
dependencies on hardware and software.

About this task

End-to-end data consistency checking is based on a data integrity field (DIF) that is
added to transferred data blocks. DIF data is used to confirm that a data block
originates from the expected source and was not modified during the transfer
between the storage system and the FCP device. The SCSI standard defines several
types of DIF. Data integrity extension (DIX) builds on DIF to extend consistency
checking, for example, to the operating system, middleware, or an application.

If the zfcp device driver is loaded with the dif=1 module parameter, Linux
automatically discovers which FCP devices and which SCSI devices support
end-to-end data consistency checking. No further setup is required.

Note: SCSI devices for which end-to-end data consistency checking is enabled
must be accessed with direct I/O. Direct I/O requires direct access through the
block device or through a file system that fully supports end-to-end data
consistency checking. For example, XFS provides this support. Expect error
messages about invalid checksums when you use other access methods.

180 Device Drivers, Features, and Commands on SLES 12 SP2

The zfcp device driver supports the following modes:
v The FCP device calculates and checks a DIF checksum (DIF type 1)
v The Linux block integrity layer calculates and checks a TCP/IP checksum, which
the FCP device then translates to a DIF checksum (DIX type 1 with DIF type 1)

For SCSI devices for which end-to-end data consistency checking is used, there is a
sysfs directory
/sys/block/sd<x>/integrity

In the path, sd<x> is the standard name of the SCSI device.

End-to-end data consistency checking is used only if all of the following

components support end-to-end data consistency checking:
SCSI disk
Check your storage server documentation about T10 DIF support and any
restrictions.
z Systems hardware
z Systems FCP adapter hardware supports end-to-end data consistency
checking as of FICON Express8.
Hypervisor
For Linux on z/VM, you require a z/VM version with guest support for
end-to-end data consistency checking.
FCP device
Check your FCP adapter hardware documentation about the support and
any restrictions. For example, end-to-end data consistency checking might
be supported only for disks with 512-byte block size.

Read the prot_capabilities sysfs attribute of the SCSI host that is associated with
an FCP device to find out about its end-to-end data consistency checking support.
The following values are possible:
0 The FCP device does not support end-to-end data consistency checking.
1 The FCP device supports DIF type 1.
16 The FCP device supports DIX type 1.
17 The FCP device supports DIX type 1 with DIF type 1.

Procedure

Issue a command of this form:

# cat /sys/bus/ccw/devices/<device_bus_id>/host<n>/scsi_host/host<n>/prot_capabilities

where <device_bus_id> identifies the FCP device and <n> is an integer that
identifies the corresponding SCSI host.

Example
# cat /sys/bus/ccw/devices/0.0.1940/host0/scsi_host/host0/prot_capabilities
17

Chapter 10. SCSI-over-Fibre Channel 181

Scenario for finding available LUNs
There are several steps from setting an FCP device online to listing the available
LUNs.

Before you begin

Alternatively to this procedure, you can use one of the following options to
discover FCP devices, remote ports, and available LUNs:
v The YaST GUI yast2 zfcp
v The text-based interface yast zfcp
v The command-line tool zfcp_san_disc (does not list FCP devices)

See the section about IBM z Systems hard disk configuration in the SUSE Linux
Enterprise Server 12 SP2 Deployment Guide.

Procedure
1. Check for available FCP devices of type 1732/03:
# lscss -t 1732/03
Device Subchan. DevType CU Type Use PIM PAM POM CHPIDs
----------------------------------------------------------------------
0.0.3c02 0.0.0015 1732/03 1731/03 80 80 ff 36000000 00000000

Another possible type would be, for example, 1732/04.

2. Set the FCP device online:
# chccwdev 0.0.3c02 --online

A port scan is performed automatically when the FCP device is set online.
3. Optional: Confirm that the FCP device is available and online:
# lszfcp -b 0.0.3c02 -a
0.0.3c02 host0
Bus = "ccw"
availability = "good"
...
failed = "0"
...
in_recovery = "0"
...
online = "1"
...

4. Optional: List the available ports:

# lszfcp -P
0.0.3c02/0x50050763030bc562 rport-0:0-0
0.0.3c02/0x500507630310c562 rport-0:0-1
0.0.3c02/0x500507630040727b rport-0:0-10
0.0.3c02/0x500507630e060521 rport-0:0-11
...

5. Scan for available LUNs on FCP device 0.0.3c02, port 0x50050763030bc562:

182 Device Drivers, Features, and Commands on SLES 12 SP2

 # lsluns -c 0.0.3c02 -p 0x50050763030bc562
Scanning for LUNs on adapter 0.0.3c02
at port 0x50050763030bc562:
0x4010400000000000
0x4010400100000000
0x4010400200000000
0x4010400300000000
0x4010400400000000
0x4010400500000000
0x4010400600000000
...

zfcp HBA API support

You require different libraries for developing and running HBA management client
applications. To develop applications, you need the development version of the
SNIA HBA API library. To run applications, you need the zFCP HBA API library.

Developing applications
To develop applications, you must install the development version of the SNIA
HBA API provided by the libHBAAPI2-devel RPM, link your application against
the library, and load the library.

Procedure
1. Install the development RPM for the SNIA HBA API. Use, for example, zypper:
# zypper install libHBAAPI2-devel

The development RPM libHBAAPI2-devel provides the necessary header files

and .so symbolic links needed to program against the SNIA HBA API.
2. Add the command-line option -lHBAAPI during the linker step of the build
process to link your application against the SNIA HBA API library.
3. In the application, issue the HBA_LoadLibrary() call as the first call to load the
library. The vendor-specific library libzfcphbaapi0, in turn, supplies the
function HBA_RegisterLibrary that returns all function pointers to the common
library and thus makes them available to the application.

Functions provided
The zfcp HBA API implements Fibre Channel - HBA API (FC-HBA) functions as
defined in the FC-HBA specification.

You can find the FC-HBA specification at www.t11.org. The following functions are
available:
v HBA_CloseAdapter()
v HBA_FreeLibrary()
v HBA_GetAdapterAttributes()
v HBA_GetAdapterName()
v HBA_GetAdapterPortAttributes()
v HBA_GetDiscoveredPortAttributes()
v HBA_GetEventBuffer()
v HBA_GetFcpTargetMapping()
v HBA_GetFcpTargetMappingV2()
v HBA_GetNumberOfAdapters()
v HBA_GetRNIDMgmtInfo()
v HBA_GetVersion()
v HBA_LoadLibrary()

Chapter 10. SCSI-over-Fibre Channel 183

 v HBA_OpenAdapter()
v HBA_RefreshAdapterConfiguration()
v HBA_RefreshInformation()
v HBA_RegisterForAdapterAddEvents()
v HBA_RegisterForAdapterEvents()
v HBA_RegisterForAdapterPortEvents()
v HBA_RegisterForAdapterPortStatEvents()
v HBA_RegisterForLinkEvents()
v HBA_RegisterForTargetEvents()
v HBA_RegisterLibrary()
v HBA_RegisterLibraryV2()
v HBA_RemoveCallback()
v HBA_SendCTPassThru()
v HBA_SendCTPassThruV2()
v HBA_SendLIRR()
v HBA_SendReadCapacity()
v HBA_SendReportLUNs()
v HBA_SendReportLUNsV2()
v HBA_SendRNID()
v HBA_SendRNIDV()
v HBA_SendRPL()
v HBA_SendRPS()
v HBA_SendScsiInquiry()
v HBA_SendSRL()
v HBA_SetRNIDMgmtInfo()

All other FC-HBA functions return status code

HBA_STATUS_ERROR_NOT_SUPPORTED where possible.

Note: zFCP HBA API for Linux 3.12 can access only FCP devices, ports, and units
that are configured in the operating system.

Getting ready to run applications

To run an application, you must install the zFCP HBA API library that is provided
by the libzfcphbaapi0 RPM. You can set environment variables to log any errors
in the library, and use tools to investigate the SAN configuration.

Before you begin

To use the HBA API support, you need the following packages:
v The zFCP HBA API library RPM, libzfcphbaapi0
v The SNIA HBA API library RPM, libHBAAPI2

Installing libzfcphbaapi0 automatically installs libHBAAPI2 as a dependency.

The application must be developed to use the SNIA HBA API library, see
Developing applications on page 183.

Procedure

Follow these steps to access the library from a client application:

1. Install the libzfcphbaapi0 RPM with zypper. Zypper automatically installs all
dependent packages. For example:

184 Device Drivers, Features, and Commands on SLES 12 SP2

 # zypper install libzfcphbaapi0

2. Ensure that the /etc/hba.conf file exists and contains a line of the form:
<library name> <library pathname>

For example:
com.ibm.libzfcphbaapi /usr/lib64/libzfcphbaapi.so.0

The SNIA library requires a configuration file called /etc/hba.conf that

contains the path to the vendor-specific library libzfcphbaapi.so.
3. Optional: Set the environment variables for logging errors. The zfcp HBA API
support uses the following environment variables to log errors in the zfcp HBA
API library:
LIB_ZFCP_HBAAPI_LOG_LEVEL
specifies the log level. If not set or set to zero, there is no logging
(default). If set to an integer value greater than 1, logging is enabled.
LIB_ZFCP_HBAAPI_LOG_FILE
specifies a file for the logging output. If not specified, stderr is used.

What to do next

You can use the zfcp_ping and zfcp_show commands to investigate your SAN
configuration.

Tools for investigating your SAN configuration

As of version 2.1, the HBA API package includes the following tools that can help
you to investigate your SAN configuration and to solve configuration problems.
zfcp_ping
to probe a port in the SAN.
zfcp_show
to retrieve information about the SAN topology and details about the SAN
components.

See How to use FC-attached SCSI devices with Linux on z Systems, SC33-8413 for
details.

Chapter 10. SCSI-over-Fibre Channel 185

186 Device Drivers, Features, and Commands on SLES 12 SP2
Chapter 11. Storage-class memory device driver supporting
Flash Express
The storage-class memory device driver provides support of Flash Express.

The Flash Express memory is accessed as storage-class memory increments through

extended asynchronous data mover (EADM) subchannels. Each increment is
represented in Linux by a block device.

What you should know about storage-class memory

Storage-class memory (SCM) is a class of data storage devices that combines
properties of both storage and memory.

To access storage-class memory from within an LPAR, one or more increments

must be added to the I/O configuration of the LPAR. At least one EADM
subchannel must be available to this LPAR. Because SCM supports multiple
concurrent I/O requests, it is advantageous to configure multiple EADM
subchannels. A typical number of EADM subchannels is 64.

Each increment is available for use through a device node as a block device. You
can use the block device with standard Linux tools as you would use any other
block device. Commonly used tools that work with block devices include: fdisk,
mkfs, and mount.

Storage-class memory is useful for workloads with large write operations, that is,
with a block size of 256 KB or more of data. Write operations with a block size of
less than 256 KB of data might not perform optimally. Read operations can be of
any size.

Storage-class memory device nodes

Applications access storage-class memory devices by device nodes. SUSE Linux
Enterprise Server creates a device node for each storage increment. Alternatively,
use the mknod command to create one.

The device driver uses a device name of the form /dev/scm<x> for an entire block
device. In the name, <x> is one or two lowercase letters.

You can partition a block device into up to seven partitions. If you use partitions,
the device driver numbers them from 1 - 7. The partitions then have device nodes
of the form /dev/scm<x><n>, where <n> is a number in the range 1 - 7, for
example /dev/scma1.

The following example shows two block devices, scma and scmb, where scma has
one partition, scma1.
# lsblk
NAME MAJ:MIN RM SIZE RO MOUNTPOINT
scma 252:0 0 16G 0
`-scma1 252:1 0 16G 0
scmb 252:8 0 16G 0

Be sure to load the scm_block before you check for the device node.

Copyright IBM Corp. 2000, 2016 187

 To check whether there already is a node, use, for example, lsblk to list all block
devices and look for "scm" entries.

To create storage-class memory device nodes, issue commands of the form:

# mknod /dev/scma1 b <major> 1
# mknod /dev/scma2 b <major> 2
# mknod /dev/scma3 b <major> 3
...

Setting up the storage-class memory device driver

Configure the storage-class memory device driver by using the module parameters.

Storage-class memory module parameter syntax

nr_requests=64 write_cluster_size=64
modprobe scm_block
nr_requests=<num> write_cluster_size=<num>

| nr_request_per_io=8

nr_request_per_io=<num>

where
nr_requests
specifies the number of parallel I/O requests. Set this number to the number of
EADM subchannels. The default is 64.
write_cluster_size
specifies the number of pages that are used by the read-modify-write
algorithm. The default is 64, resulting in that all write requests smaller than
256 KiB are translated to 256 KiB writes. 1 KiB is 1024 bytes.
| nr_request_per_io
| submits more concurrent I/O requests than the current limit, which is based
| on the number of available EADM subchannels (64). Valid values are in the
| range 1 to 64. Increasing the requests increases the number of I/O requests per
| second, especially for requests with a small block size. The default number of
| requests is 8. Depending on the workload, this setting might improve the
| throughput of the scm_block driver.

Working with storage-class memory increments

You can list storage-class memory increments and EADM subchannels.
v Show EADM subchannels
v Listing storage-class memory increments on page 189
v Combining SCM devices with LVM on page 189

Show EADM subchannels

Use the lscss command to list EADM subchannels.

188 Device Drivers, Features, and Commands on SLES 12 SP2

 About this task

The extended asynchronous data mover (EADM) subchannels are used to transfer
data to and from the storage-class memory. At least one EADM subchannel must
be available to the LPAR.

Procedure
To list EADM subchannels, issue:
# lscss --eadm
Device Subchan.
-----------------
n/a 0.0.ff00
n/a 0.0.ff01
n/a 0.0.ff02
n/a 0.0.ff03
n/a 0.0.ff04
n/a 0.0.ff05
n/a 0.0.ff06
n/a 0.0.ff07

For more information about the lscss command, see lscss - List subchannels on
page 580.

Listing storage-class memory increments

Use the lsscm command to see the status and attributes of storage-class memory
increments.

About this task

Each storage-class memory increment can be accessed as a block device through a

device node /dev/scm<x>. Optionally, you can partition a storage-class memory
increment in up to seven partitions.

You can also use the lsblk command to list all block devices.

Procedure

To list all storage-class memory increments, their status, and attributes, issue:
# lsscm
SCM Increment Size Name Rank D_state O_state Pers ResID
--------------------------------------------------------------
0000000000000000 16384MB scma 1 2 1 2 1
0000000400000000 16384MB scmb 1 2 1 2 1

See lsscm - List storage-class memory increments on page 594 for details about
the lsscm command.

Combining SCM devices with LVM

You can use LVM to combine multiple SCM block devices into an arbitrary sized
LVM device.

Chapter 11. Storage-class memory 189

 Example

Configure SCM as any other block devices in LVM. If your version of LVM does
not accept SCM devices as valid LVM device types and issues an error message,
add the SCM devices to the LVM configuration file /etc/lvm/lvm.conf. Add the
following line to the section labeled devices:
types = [ "scm", 8 ]

190 Device Drivers, Features, and Commands on SLES 12 SP2

Chapter 12. Channel-attached tape device driver
The tape device driver supports channel-attached tape devices on SUSE Linux
Enterprise Server 12 SP2 for z Systems.

SCSI tape devices that are attached through an FCP channel are handled by the
zfcp device driver (see Chapter 10, SCSI-over-Fibre Channel device driver, on
page 145).

Features
The tape device driver supports a range of channel-attached tape devices and
functions of these devices.
v The tape device driver supports channel-attached tape drives that are compatible
with IBM 3480, 3490, 3590, and 3592 magnetic tape subsystems. Various models
of these device types are handled (for example, the 3490/10).
3592 devices that emulate 3590 devices are recognized and treated as 3590
devices.
v Logical character devices for non-rewinding and rewinding modes of operation
(see Tape device modes and logical devices).
v Control operations through mt (see Using the mt command on page 193)
v Message display support (see tape390_display - display messages on tape
devices and load tapes on page 645)
v Encryption support (see tape390_crypt - Manage tape encryption on page 641)
v Up to 128 physical tape devices

What you should know about channel-attached tape devices

A naming scheme helps you to keep track of your tape devices, their modes of
operation, and the corresponding device nodes.

Tape device modes and logical devices

The tape device driver supports up to 128 physical tape devices. Each physical
tape device can be used as a character device in non-rewinding or in rewinding
mode.

In non-rewinding mode, the tape remains at the current position when the device
is closed. In rewinding mode, the tape is rewound when the device is closed. The
tape device driver treats each mode as a separate logical device.

Both modes provide sequential (traditional) tape access without any caching done
in the kernel.

You can use a channel-attached tape device in the same way as any other Linux
tape device. You can write to it and read from it using standard Linux facilities
such as GNU tar. You can perform control operations (such as rewinding the tape
or skipping a file) with the standard tool mt.

Copyright IBM Corp. 2000, 2016 191

 Tape naming scheme
The tape device driver assigns minor numbers along with an index number when
a physical tape device comes online.

The naming scheme for tape devices is summarized in Table 30.

Table 30. Tape device names and minor numbers
Device Names Minor numbers
Non-rewinding character ntibm<n> 2<n>
devices
Rewinding character devices rtibm<n> 2<n>+1

where <n> is the index number that is assigned by the device driver. The index
starts from 0 for the first physical tape device, 1 for the second, and so on. The
name space is restricted to 128 physical tape devices, so the maximum index
number is 127 for the 128th physical tape device.

The index number and corresponding minor numbers and device names are not
permanently associated with a specific physical tape device. When a tape device
goes offline, it surrenders its index number. The device driver assigns the lowest
free index number when a physical tape device comes online. An index number
with its corresponding device names and minor numbers can be reassigned to
different physical tape devices as devices go offline and come online.

Tip: Use the lstape command (see lstape - List tape devices on page 597) to
determine the current mapping of index numbers to physical tape devices.

When the tape device driver is loaded, it dynamically allocates a major number to
channel-attached character tape devices. A different major number might be used
when the device driver is reloaded, for example when Linux is rebooted.

For online tape devices, directories provide information about the major/minor
assignments. The directories have the form:
v /sys/class/tape390/ntibm<n>
v /sys/class/tape390/rtibm<n>

Each of these directories has a dev attribute. The value of the dev attribute has the
form <major>:<minor>, where <major> is the major number for the device and
<minor> is the minor number specific to the logical device.

Example

In this example, four physical tape devices are present, with three of them online.
The TapeNo column shows the index number and the BusID column indicates the
associated physical tape device. In the example, no index number is allocated to
the tape device in the last row. The device is offline and, currently, no names and
minor numbers are assigned to it.
# lstape --ccw-only
TapeNo BusID CuType/Model DevType/DevMod BlkSize State Op MedState
0 0.0.01a1 3490/10 3490/40 auto UNUSED --- UNLOADED
1 0.0.01a0 3480/01 3480/04 auto UNUSED --- UNLOADED
2 0.0.0172 3590/50 3590/11 auto IN_USE --- LOADED
N/A 0.0.01ac 3490/10 3490/40 N/A OFFLINE --- N/A

Table 31 on page 193 summarizes the resulting names and minor numbers.

192 Device Drivers, Features, and Commands on SLES 12 SP2

 Table 31. Example names and minor numbers
Bus ID Index (TapeNo) Device Device name Minor number
0.0.01a1 0 non-rewind ntibm0 0
rewind rtibm0 1
0.0.01a0 1 non-rewind ntibm1 2
rewind rtibm1 3
0.0.0172 2 non-rewind ntibm2 4
rewind rtibm2 5
0.0.01ac not assigned n/a n/a not assigned

For the online devices, the major/minor assignments can be read from their
respective representations in /sys/class:
# cat /sys/class/tape390/ntibm0/dev
254:0
# cat /sys/class/tape390/rtibm0/dev
254:1
# cat /sys/class/tape390/ntibm1/dev
254:2
# cat /sys/class/tape390/rtibm1/dev
254:3
# cat /sys/class/tape390/ntibm2/dev
254:4
# cat /sys/class/tape390/rtibm2/dev
254:5

In the example, the major number is 254. The minor numbers are as expected for
the respective device names.

Tape device nodes

Applications access tape devices by device nodes. SUSE Linux Enterprise Server 12
SP2 uses udev to create two device nodes for each tape device.

The device nodes have the form /dev/<name>, where <name> is the device name
according to Tape naming scheme on page 192.

For example, if you have two tape devices, udev creates the device nodes that are
shown in Table 32:
Table 32. Tape device nodes
Node for non-rewind device rewind device
First tape device /dev/ntibm0 /dev/rtibm0
Second tape device /dev/ntibm1 /dev/rtibm1

Using the mt command

There are differences between the MTIO interface for channel-attached tapes and
other tape drives. Correspondingly, some operations of the mt command are
different for channel-attached tapes.

The mt command handles basic tape control in Linux. See the man page for general
information about mt.

Chapter 12. Channel-attached tape 193

 setdensity
has no effect because the recording density is automatically detected on
channel-attached tape hardware.
drvbuffer
has no effect because channel-attached tape hardware automatically
switches to unbuffered mode if buffering is unavailable.
lock and unlock
have no effect because channel-attached tape hardware does not support
media locking.
setpartition and mkpartition
have no effect because channel-attached tape hardware does not support
partitioning.
status returns a structure that, aside from the block number, contains mostly
SCSI-related data that does not apply to the tape device driver.
load does not automatically load a tape but waits for a tape to be loaded
manually.
offline and rewoffl and eject
all include expelling the currently loaded tape. Depending on the stacker
mode, it might attempt to load the next tape (see Loading and unloading
tapes on page 198 for details).

Loading the tape device driver

There are no module parameters for the tape device driver. SUSE Linux Enterprise
Server 12 SP2 loads the required device driver module for you when a device
becomes available.

You can also load the modules with the modprobe command.

Tape module syntax

modprobe tape_34xx
tape_3590

See the modprobe man page for details on modprobe.

Working with tape devices

Typical tasks for working with tape devices include displaying tape information,
controlling compression, and loading and unloading tapes.

For information about working with the channel measurement facility, see
Chapter 44, Channel measurement facility, on page 455.

For information about displaying messages on a tape device's display unit, see
tape390_display - display messages on tape devices and load tapes on page 645.

194 Device Drivers, Features, and Commands on SLES 12 SP2

 See Working with newly available devices on page 10 to avoid errors when
working with devices that have become available to a running Linux instance.
v Setting a tape device online or offline
v Displaying tape information on page 196
v Enabling compression on page 198
v Loading and unloading tapes on page 198

Setting a tape device online or offline

Set a tape device online or offline with the chccwdev command or through the
online sysfs attribute of the device.

About this task

Setting a physical tape device online makes both corresponding logical devices
accessible:
v The non-rewind character device
v The rewind character device

At any time, the device can be online to a single Linux instance only. You must set
the tape device offline to make it accessible to other Linux instances in a shared
environment.

Procedure

Use the chccwdev command (see chccwdev - Set CCW device attributes on page
492) to set a tape online or offline.
Alternatively, you can write 1 to the online attribute of the device to set it online;
or write 0 to set it offline.

Results

When a physical tape device is set online, the device driver assigns an index
number to it. This index number is used in the standard device nodes (see Tape
device nodes on page 193) to identify the corresponding logical devices. The
index number is in the range 0 - 127. A maximum of 128 physical tape devices can
be online concurrently.

If you are using the standard device nodes, you must find out the index number
that the tape device driver assigned to your tape device. This index number, and
consequently the associated standard device node, can change after a tape device
was set offline and back online.

If you need to know the index number, issue a command of this form:
# lstape --ccw-only <device_bus_id>

where <device_bus_id> is the device bus-ID that corresponds to the physical tape
device. The index number is the value in the TapeNo column of the command
output. For more information about the lstape command, see lstape - List tape
devices on page 597.

Chapter 12. Channel-attached tape 195

 Examples
v To set a physical tape device with device bus-ID 0.0.015f online, issue:
# chccwdev -e 0.0.015f

or
# echo 1 > /sys/bus/ccw/devices/0.0.015f/online

To find the index number that the tape device driver assigned to the device,
issue:
# lstape 0.0.015f --ccw-only
TapeNo BusID CuType/Model DevType/Model BlkSize State Op MedState
2 0.0.015f 3480/01 3480/04 auto UNUSED --- LOADED

In the example, the assigned index number is 2. The standard device nodes for
working with the device until it is set offline are then:
/dev/ntibm2 for the non-rewinding device
/dev/rtibm2 for the rewinding device
v To set a physical tape device with device bus-ID 0.0.015f offline, issue:
# chccwdev -d 0.0.015f

or
# echo 0 > /sys/bus/ccw/devices/0.0.015f/online

Displaying tape information

Use the lstape command to display summary information about your tape
devices, or read tape information from sysfs.

Alternatively, you can read tape information from sysfs. Each physical tape device
is represented in a sysfs directory of the form
/sys/bus/ccw/devices/<device_bus_id>

where <device_bus_id> is the device bus-ID that corresponds to the physical tape
device. This directory contains a number of attributes with information about the
physical device. The attributes: blocksize, state, operation, and medium_state,
might not show the current values if the device is offline.
Table 33. Tape device attributes
Attribute Explanation
online 1 if the device is online or 0 if it is offline (see Setting a tape
device online or offline on page 195)
cmb_enable 1 if channel measurement block is enabled for the physical
device or 0 if it is not enabled (see Chapter 44, Channel
measurement facility, on page 455)
cutype Type and model of the control unit
devtype Type and model of the physical tape device

196 Device Drivers, Features, and Commands on SLES 12 SP2

Table 33. Tape device attributes (continued)
Attribute Explanation
blocksize Currently used block size in bytes or 0 for auto
state State of the physical tape device, either of:
UNUSED
Device is not in use and is available to any operating
system image in a shared environment
IN_USE
Device is being used as a character device by a process
on this Linux image
OFFLINE
The device is offline.
NOT_OP
Device is not operational
operation The current tape operation, for example:
--- No operation
WRI Write operation
RFO Read operation
MSN Medium sense
Several other operation codes exist, for example, for rewind and
seek.
medium_state The current state of the tape cartridge:
1 Cartridge is loaded into the tape device
2 No cartridge is loaded
0 The tape device driver does not have information about
the current cartridge state

Procedure

Issue a command of this form to read an attribute:

# cat /sys/bus/ccw/devices/<device_bus_id>/<attribute>

where <attribute> is one of the attributes of Table 33 on page 196.

Example

The following lstape command displays information about a tape device with bus
ID 0.0.015f:
# lstape 0.0.015f --ccw-only
TapeNo BusID CuType/Model DevType/Model BlkSize State Op MedState
2 0.0.015f 3480/01 3480/04 auto UNUSED --- LOADED

This sequence of commands reads the same information from sysfs:

Chapter 12. Channel-attached tape 197

 # cat /sys/bus/ccw/devices/0.0.015f/online
1
# cat /sys/bus/ccw/devices/0.0.015f/cmb_enable
0
# cat /sys/bus/ccw/devices/0.0.015f/cutype
3480/01
# cat /sys/bus/ccw/devices/0.0.015f/devtype
3480/04
# cat /sys/bus/ccw/devices/0.0.015f/blocksize
0
# cat /sys/bus/ccw/devices/0.0.015f/state
UNUSED
# cat /sys/bus/ccw/devices/0.0.015f/operation
---
# cat /sys/bus/ccw/devices/0.0.015f/medium_state
1

Enabling compression
Control Improved Data Recording Capability (IDRC) compression with the mt
command provided by the RPM mt_st.

About this task

Compression is off after the tape device driver is loaded.

Procedure

To enable compression, issue:

# mt -f <node> compression

or
# mt -f <node> compression 1

where <node> is the device node for a character device, for example, /dev/ntibm0.
To disable compression, issue:
# mt -f <tape> compression 0

Any other numeric value has no effect, and any other argument disables
compression.

Example

To enable compression for a tape device with a device node /dev/ntibm0 issue:
# mt -f /dev/ntibm0 compression 1

Loading and unloading tapes

Unload tapes with the mt command. How to load tapes depends on the stacker
mode of your tape hardware.

198 Device Drivers, Features, and Commands on SLES 12 SP2

Procedure

Unload tapes by issuing a command of this form:

# mt -f <node> unload

where <node> is one of the character device nodes.

Whether you can load tapes from your Linux instance depends on the stacker
mode of your tape hardware. There are three possible modes:
manual
Tapes must always be loaded manually by an operator. You can use the
tape390_display command (see tape390_display - display messages on
tape devices and load tapes on page 645) to display a short message on
the tape device's display unit when a new tape is required.
automatic
If there is another tape present in the stacker, the tape device automatically
loads a new tape when the current tape is expelled. You can load a new
tape from Linux by expelling the current tape with the mt command.
system
The tape device loads a tape when instructed from the operating system.
From Linux, you can load a tape with the tape390_display command (see
tape390_display - display messages on tape devices and load tapes on
page 645). You cannot use the mt command to load a tape.

Example
To expel a tape from a tape device that can be accessed through a device node
/dev/ntibm0, issue:
# mt -f /dev/ntibm0 unload

Assuming that the stacker mode of the tape device is system and that a tape is
present in the stacker, you can load a new tape by issuing:
# tape390_display -l "NEW TAPE" /dev/ntibm0

NEW TAPE is a message that is displayed on the tape devices display unit until
the tape device receives the next tape movement command.

Chapter 12. Channel-attached tape 199

200 Device Drivers, Features, and Commands on SLES 12 SP2
Chapter 13. XPRAM device driver
With the XPRAM block device driver SUSE Linux Enterprise Server 12 SP2 for z
Systems can access expanded storage. XPRAM can be used as a basis for fast swap
devices or for fast file systems.

Expanded storage can be swapped in or out of the main storage in 4 KB blocks. All
XPRAM devices provide a block size of 4096 bytes.

XPRAM features
The XPRAM device driver automatically detects expanded storage and supports
expanded storage partitions.
v If expanded storage is not available, XPRAM fails gracefully with a log message
that reports the absence of expanded storage.
v The expanded storage can be divided into up to 32 partitions.

What you should know about XPRAM

There is a device node for each XPRAM partition. Expanded storage persists across
reboots and, with suitable boot parameters, the stored data can be accessed by the
rebooted Linux instance.

XPRAM partitions and device nodes

The XPRAM device driver uses major number 35. The standard device names are
of the form slram<n>, where <n> is the corresponding minor number.

You can use the entire available expanded storage as a single XPRAM device or
divide it into up to 32 partitions. Each partition is treated as a separate XPRAM
device.

If the entire expanded storage is used a single device, the device name is slram0.
For partitioned expanded storage, the <n> in the device name denotes the (n+1)th
partition. For example, the first partition is called slram0, the second slram1, and
the 32nd partition is called slram31.
Table 34. XPRAM device names, minor numbers, and partitions
Minor Name To access
0 slram0 the first partition or the entire expanded storage if there are no
partitions
1 slram1 the second partition
2 slram2 the third partition
... ... ...
<n> slram<n> the (<n>+1)th partition
... ... ...
31 slram31 the 32nd partition

The device nodes that you need to access these partitions are created by udev
when you load the XPRAM device driver module. The nodes are of the form

Copyright IBM Corp. 2000, 2016 201

 /dev/slram<n>, where <n> is the index number of the partition. In addition, to the
device nodes udev creates a symbolic link of the form /dev/xpram<n> that points to
the respective device node.

XPRAM use for diagnosis

Expanded storage persists across reboots, which makes it suitable for storing
diagnostic information.

Issuing an IPL command to reboot Linux does not reset expanded storage.
Expanded storage is persistent across IPLs and can be used, for example, to store
diagnostic information. The expanded storage is reset when the z/VM guest
virtual machine is logged off or when the LPAR is deactivated.

Reusing XPRAM partitions

You might be able to reuse existing file systems or swap devices on an XPRAM
device or partition after reloading the XPRAM device driver (for example, after
rebooting Linux).

For file systems or swap devices to be reusable, the XPRAM kernel or module
parameters for the new device or partition must match the parameters of the
previous use of XPRAM.

If you change the XPRAM parameters, you must create a new file system or a new
swap device for each changed partition. A device or partition is considered
changed if its size has changed. All partitions that follow a changed partition are
also considered changed even if their sizes are unchanged.

Setting up the XPRAM device driver

The XPRAM device driver is loaded automatically after extended memory has
been configured with YaST. You can also configure extended memory and load the
XPRAM device driver independently of YaST.

You can optionally partition the available expanded storage by using the devs and
sizes module parameters when you load the xpram module.

XPRAM module parameter syntax

modprobe xpram
devs=<number_of_partitions>
,

sizes= <partition_size>

where:
<number_of_partitions>
is an integer in the range 1 - 32 that defines how many partitions the expanded
storage is split into.
<partition_size>
specifies the size of a partition. The i-th value defines the size of the i-th
partition.

202 Device Drivers, Features, and Commands on SLES 12 SP2

 Each size is a non-negative integer that defines the size of the partition in KB
or a blank. Only decimal values are allowed and no magnitudes are accepted.
You can specify up to <number_of_partitions> values. If you specify fewer
values than <number_of_partitions>, the missing values are interpreted as
blanks. Blanks are treated like zeros.

Any partition that is defined with a non-zero size is allocated the amount of
memory that is specified by its size parameter.

Any remaining memory is divided as equally as possible among any partitions

with a zero or blank size parameter. Dividing the remaining memory is subject to
the following constraints:
v Blocks must be allocated in multiples of 4 K.
v Addressing constraints might leave un-allocated areas of memory between
partitions.

See the modprobe man page for details about modprobe.

Examples
v The following specification allocates the extended storage into four partitions.
Partition 1 has 2 GB (2097152 KB), partition 4 has 4 GB (4194304 KB), and
partitions 2 and 3 use equal parts of the remaining storage. If the total amount
of extended storage was 16 GB, then partitions 3 and 4 would each have
approximately 5 GB.
# modprobe xpram devs=4 sizes=2097152,0,0,4194304

v The following specification allocates the extended storage into three partitions.
The partition 2 has 512 KB and the partitions 1 and 3 use equal parts of the
remaining extended storage.
# modprobe xpram devs=3 sizes=,512

v The following specification allocates the extended storage into two partitions of
equal size.
# modprobe xpram devs=2

Chapter 13. XPRAM 203

204 Device Drivers, Features, and Commands on SLES 12 SP2
 Part 4. Networking
Chapter 14. qeth device driver for OSA-Express Working with LCS devices . . . . . . . . . 288
(QDIO) and HiperSockets . . . . . . . . . 209
Device driver functions . . . . . . . . . . 212 Chapter 17. CTCM device driver . . . . . . 293
What you should know about the qeth device Features . . . . . . . . . . . . . . . 293
driver . . . . . . . . . . . . . . . . 215 What you should know about CTCM . . . . . 293
Setting up the qeth device driver . . . . . . . 223 Setting up the CTCM device driver . . . . . . 295
Working with qeth devices . . . . . . . . . 224 Working with CTCM devices . . . . . . . . 295
Working with qeth devices in layer 3 mode . . . 245 Scenarios . . . . . . . . . . . . . . . 301
| Working with qeth devices in layer 2 mode . . . 255
Scenario: VIPA minimize outage due to adapter Chapter 18. NETIUCV device driver . . . . . 305
failure . . . . . . . . . . . . . . . . 257 What you should know about IUCV . . . . . 305
Scenario: Virtual LAN (VLAN) support. . . . . 263 Setting up the NETIUCV device driver . . . . . 306
HiperSockets Network Concentrator. . . . . . 267 Working with IUCV devices . . . . . . . . 307
Setting up for DHCP with IPv4 . . . . . . . 272 Scenario: Setting up an IUCV connection to a
Setting up Linux as a LAN sniffer . . . . . . 273 TCP/IP service machine . . . . . . . . . . 311

Chapter 15. OSA-Express SNMP subagent Chapter 19. AF_IUCV address family support 315
support . . . . . . . . . . . . . . . 277 Features . . . . . . . . . . . . . . . 315
What you should know about osasnmpd . . . . 277 Setting up the AF_IUCV address family support 316
Setting up osasnmpd . . . . . . . . . . . 278 Addressing AF_IUCV sockets in applications . . . 317
Working with the osasnmpd subagent . . . . . 282
Chapter 20. RDMA over Converged Ethernet 319
Chapter 16. LAN channel station device driver 287 Working with the RoCE support . . . . . . . 319
What you should know about LCS . . . . . . 287 Enabling debugging . . . . . . . . . . . 319
Setting up the LCS device driver . . . . . . . 288

There are several z Systems specific network device drivers for SUSE Linux
Enterprise Server 12 SP2 for z Systems.

Newest version

You can find the newest version of this publication on IBM Knowledge Center at
www.ibm.com/support/knowledgecenter/linuxonibm/liaaf/lnz_r_suse.html

Restrictions

For prerequisites and restrictions see the z Systems architecture specific

information in the SUSE Linux Enterprise Server 12 SP2 release notes at
www.suse.com/releasenotes

Example

An example network setup that uses some available network setup types is shown
in Figure 33 on page 206.

Copyright IBM Corp. 2000, 2016 205

 Figure 33. Networking example

In the example there are three Linux instances; two of them run as z/VM guests in
one LPAR and a third Linux instance runs in another LPAR. Within z/VM, Linux
instances can be connected through a guest LAN or VSWITCH. Within and
between LPARs, you can connect Linux instances through HiperSockets.
OSA-Express cards running in either non-QDIO mode (called LCS here) or in
QDIO mode can connect the mainframe to an external network.

206 Device Drivers, Features, and Commands on SLES 12 SP2

Table 35 lists which control units and device type combinations are supported by
the network device drivers.
Table 35. Supported device types, control units, and corresponding device drivers
Device
type Control unit Device driver Comment
1732/01 1731/01 qeth OSA configured as OSD
1732/02 1731/02 qeth OSA configured as OSX
1732/03 1731/02 qeth OSA configured as OSM
1732/05 1731/05 qeth HiperSockets
0000/00 3088/01 lcs P/390
0000/00 3088/08 ctcm Virtual CTC under z/VM
0000/00 3088/1e ctcm FICON channel
0000/00 3088/1f lcs 2216 Nways Multiaccess Connector
0000/00 3088/1f ctcm ESCON channel
0000/00 3088/60 lcs OSA configured as OSE (non-QDIO)

Part 4. Networking 207

208 Device Drivers, Features, and Commands on SLES 12 SP2
Chapter 14. qeth device driver for OSA-Express (QDIO) and
HiperSockets
The qeth device driver supports a multitude of network connections, for example,
connections through Open Systems Adapters (OSA), HiperSockets, guest LANs,
and virtual switches.
Real connections that use OSA-Express features
An IBM mainframe uses OSA-Express features, which are real
LAN-adapter hardware, see Figure 34. These adapters provide connections
to the outside world, but can also connect virtual systems (between LPARs
or between z/VM guest virtual machines) within the mainframe. The qeth
driver supports these adapters if they are defined to run in queued direct
I/O (QDIO) mode (defined as OSD in the hardware configuration).
OSD-devices are the standard z Systems LAN-adapters. For details about
OSA-Express in QDIO mode, see OSA-Express Customer's Guide and
Reference, SA22-7935.

Figure 34. OSA-Express adapters are real LAN-adapter hardware

The qeth device driver supports OSA-Express features for the z Systems
mainframes that are relevant to SUSE Linux Enterprise Server 12 SP2 as
shown in Table 36.:
Table 36. The qeth device driver support for OSA-Express features
Feature z13 zEC12 and zBC12 z196 and z114
OSA-Express5S Gigabit Ethernet Gigabit Ethernet Not supported
10 Gigabit Ethernet 10 Gigabit Ethernet
1000Base-T Ethernet 1000Base-T Ethernet

Copyright IBM Corp. 2000, 2016 209

 Table 36. The qeth device driver support for OSA-Express features (continued)
Feature z13 zEC12 and zBC12 z196 and z114
OSA-Express4S Gigabit Ethernet Gigabit Ethernet Gigabit Ethernet
10 Gigabit Ethernet 10 Gigabit Ethernet 10 Gigabit Ethernet
1000Base-T Ethernet 1000Base-T Ethernet
OSA-Express3 Not supported Gigabit Ethernet Gigabit Ethernet
10 Gigabit Ethernet 10 Gigabit Ethernet
1000Base-T Ethernet 1000Base-T Ethernet
OSA-Express2 Not supported Not supported Gigabit Ethernet
1000Base-T Ethernet

Note: Unless otherwise indicated, OSA-Express refers to the OSA-express

features as shown in Table 36 on page 209.
The qeth device driver supports CHPIDs of type OSM and OSX:
OSM provides connectivity to the intranode management network
(INMN) from Unified Resource Manager functions to a zEnterprise
CPC.
OSX provides connectivity to and access control for the intraensemble
data network (IEDN), which is managed by Unified Resource
Manager functions. A zEnterprise CPC and zBX within an
ensemble are connected through the IEDN. See zEnterprise System
Introduction to Ensembles, GC27-2609 and zEnterprise System
Ensemble Planning and Configuring Guide, GC27-2608 for more
details.
HiperSockets
An IBM mainframe uses internal connections that are called HiperSockets.
These simulate QDIO network adapters and provide high-speed TCP/IP
communication for operating system instances within and across LPARs.
For details about HiperSockets, see HiperSockets Implementation Guide,
SG24-6816.
The qeth device driver supports HiperSockets for all z Systems mainframes
on which you can run SUSE Linux Enterprise Server 12 SP2.
Virtual connections for Linux on z/VM
z/VM offers virtualized LAN-adapters that enable connections between
z/VM guest virtual machines and the outside world. It allows definitions
of simulated network interface cards (NICs) attached to certain z/VM
guest virtual machines. The NICs can be connected to a simulated LAN
segment called guest LAN for z/VM internal communication between
z/VM guest virtual machines, or they can be connected to a virtual switch
called VSWITCH for external LAN connectivity.
Guest LAN
Guest LANs represent a simulated LAN segment that can be
connected to simulated network interface cards. There are three
types of guest LANs:
v Simulated OSA-Express in layer 3 mode
v Simulated HiperSockets(layer 3) mode
v Simulated Ethernet in layer 2 mode
Each guest LAN is isolated from other guest LANs on the same
system (unless some member of one LAN group acts as a router to

210 Device Drivers, Features, and Commands on SLES 12 SP2

 other groups). See Figure 35.

Figure 35. Guest LAN

Virtual switch
A virtual switch (VSWITCH) is a special-purpose guest LAN that
provides external LAN connectivity through an additional
OSA-Express device served by z/VM without the need for a
routing virtual machine, see Figure 36 on page 212.

Chapter 14. qeth: OSA-Express (QDIO) and HiperSockets 211

 Figure 36. Virtual switch

A dedicated OSA adapter can be an option, but is not required for

a VSWITCH.
The qeth device driver distinguishes between virtual NICs in QDIO mode
or HiperSockets mode. It cannot detect whether the virtual network is a
guest LAN or a VSWITCH.
For information about guest LANs, virtual switches, and virtual
HiperSockets, see z/VM Connectivity, SC24-6174.

Device driver functions

The qeth device driver supports many networking transport protocol functions, as
well as offload functions and problem determination functions.

The qeth device driver supports functions listed in Table 37 and Table 38 on page
214.
Table 37. Real connections
HiperSockets HiperSockets
Layer 2 Layer 3
Function OSA Layer 2 OSA Layer 3 Ethernet Ethernet
Basic device or protocol functions
IPv4/multicast/broadcast Yes/Yes/Yes Yes/Yes/Yes Yes/Yes/Yes Yes/Yes/Yes
IPv6/multicast Yes/Yes Yes/Yes Yes/Yes Yes/Yes
Non-IP traffic Yes Yes Yes No
VLAN IPv4/IPv6/non IP sw/sw/sw hw/sw/sw sw/sw/sw hw/sw/No
Linux ARP Yes No (hw ARP) Yes No
Linux neighbor Yes Yes Yes No
solicitation
Unique MAC address Yes (random) No Yes Yes

212 Device Drivers, Features, and Commands on SLES 12 SP2

 Table 37. Real connections (continued)
HiperSockets HiperSockets
Layer 2 Layer 3
Function OSA Layer 2 OSA Layer 3 Ethernet Ethernet
Change MAC address Yes No Yes No
Promiscuous mode No No No v Yes (for
sniffer=1)
v No (for
sniffer=0)
MAC headers Yes/Yes faked/faked Yes/Yes faked/faked
send/receive
ethtool support Yes Yes Yes Yes
Bonding Yes No Yes No
Priority queueing Yes Yes Yes Yes
| Bridge port No No Yes No
Offload features
TCP segmentation No Yes No No
offload (TSO)
| Inbound (rx) checksum Yes Yes No No
| Outbound (tx) checksum Yes Yes No No
OSA/QETH specific features
Special device driver No required No Yes
setup for VIPA
Special device driver No required No Yes
setup for proxy ARP
Special device driver No required No Yes
setup for IP takeover
Special device driver No/No required/ No/No Yes/Yes
setup for routing required
IPv4/IPv6
Receive buffer count Yes Yes Yes Yes
Direct connectivity to Yes by HW Yes No Yes
z/OS
SNMP support Yes Yes No No
Multiport support Yes Yes No No
Data connection isolation Yes Yes No No
Problem determination
Hardware trace No Yes No No
Legend:
No Function not supported or not required.
Yes Function supported.
hw Function performed by hardware.
sw Function performed by software.
faked Function will be simulated.
required
Function requires special setup.

Chapter 14. qeth: OSA-Express (QDIO) and HiperSockets 213

 Table 38. z/VM VSWITCH or Guest LAN connections
Emulated
Emulated OSA Emulated OSA HiperSockets
Function Layer 2 Layer 3 Layer 3
Basic device or protocol features
IPv4/multicast/broadcast Yes/Yes/Yes Yes/Yes/Yes Yes/Yes/Yes
IPv6/multicast Yes/Yes Yes/Yes No/No
Non-IP traffic Yes No No
VLAN IPv4/IPv6/non IP sw/sw/sw hw/sw/No hw/No/No
Linux ARP Yes No (hw ARP) No
Linux neighbor solicitation Yes Yes No
Unique MAC address Yes No Yes
Change MAC address Yes No No
Promiscuous mode Yes Yes No
MAC headers send/receive Yes/Yes faked/faked faked/faked
ethtool support Yes Yes Yes
Bonding Yes No No
Priority queueing Yes Yes Yes
Offload features
TSO No No No
rx HW checksum No No No
OSA/QETH specific features
Special device driver setup for No required required
VIPA
Special device driver setup for No required required
proxy ARP
Special device driver setup for No required required
IP takeover
Special device driver setup for No/No required/required required/required
routing IPv4/IPv6
Receive buffer count Yes Yes Yes
Direct connectivity to z/OS No Yes Yes
SNMP support No No No
Multiport support No No No
Data connection isolation No No No
Problem determination
Hardware trace No No No
Legend:
No Function not supported or not required.
Yes Function supported.
hw Function performed by hardware.
sw Function performed by software.
faked Function will be simulated.
required
Function requires special setup.

214 Device Drivers, Features, and Commands on SLES 12 SP2

What you should know about the qeth device driver
Interface names are assigned to qeth group devices, which map to subchannels and
their corresponding device numbers and device bus-IDs. An OSA-Express adapter
can handle both IPv4 and IPv6 packets.

Layer 2 and layer 3

The qeth device driver consists of a common core and two device disciplines: layer
2 and layer 3.

In layer 2 mode, OSA routing to the destination Linux instance is based on MAC
addresses. A local MAC address is assigned to each interface of a Linux instance
and registered in the OSA Address Table. These MAC addresses are unique and
different from the MAC address of the OSA adapter. See MAC headers in layer 2
mode on page 218 for details.

In layer 3 mode, all interfaces of all Linux instances share the MAC address of the
OSA adapter. OSA routing to the destination Linux instance is based on IP
addresses. See MAC headers in layer 3 mode on page 219 for details.
The layer 2 discipline (qeth_l2)
The layer 2 discipline supports:
v OSA devices and z/VM virtual NICs that couple to VSWITCHes or
QDIO guest LANs
v OSA devices for NCP
v HiperSockets devices
v OSM (OSA-Express for Unified Resource Manager) devices
v OSX (OSA-Express for zBX) devices for IEDN
The layer 2 discipline is the default setup for OSA. On HiperSockets the
default continues to be layer 3. OSA guest LANs are layer 2 by default,
while HiperSockets guest LANs are always layer 3. See Setting the layer2
attribute on page 229 for details.
The layer 3 discipline (qeth_l3)
The layer 3 discipline supports:
v OSA devices and z/VM virtual NICs that couple to VSWITCHes or
QDIO guest LANs that are running in layer 3 mode (with faked link
layer headers)
v HiperSockets and HiperSockets guest LAN devices that are running in
layer 3 mode (with faked link layer headers)
v OSX (OSA-Express for zBX) devices for IEDN
This discipline supports those devices that are not capable of running in
layer 2 mode. Not all Linux networking features are supported and others
need special setup or configuration. See Table 44 on page 226. Some
performance-critical applications might benefit from being layer 3.

Layer 2 and layer 3 interfaces cannot communicate within a HiperSockets LAN or

within a VSWITCH or guest LAN. However, a shared OSA adapter can convert
traffic between layer 2 and layer 3 networks.

Chapter 14. qeth: OSA-Express (QDIO) and HiperSockets 215

 qeth group devices
The qeth device driver requires three I/O subchannels for each HiperSockets
CHPID or OSA-Express CHPID in QDIO mode. One subchannel is for control
reads, one for control writes, and the third is for data.

The qeth device driver uses the QDIO protocol to communicate with the
HiperSockets and OSA-Express adapter.

Figure 37. I/O subchannel interface

The three device bus-IDs that correspond to the subchannel triplet are grouped as
one qeth group device. The following rules apply for the device bus-IDs:
read no specific rules.
write must be the device bus-ID of the read subchannel plus one.
data can be any free device bus-ID on the same CHPID.

You can configure different triplets of device bus-IDs on the same CHPID
differently. For example, if you have two triplets on the same CHPID they can
have different attribute values for priority queueing.

Overview of the steps for setting up a qeth group device

You need to perform several steps before user-space applications on your Linux
instance can use a qeth group device.

Before you begin

Find out how the hardware is configured and which qeth device bus-IDs are on
which CHPID, for example by looking at the IOCDS. Identify the device bus-IDs
that you want to group into a qeth group device. The three device bus-IDs must be
on the same CHPID.

Procedure
Perform these steps to allow user-space applications on your Linux instance to use
a qeth group device:
1. Create the qeth group device.
After booting Linux, each qeth device bus-ID is represented by a subdirectory
in /sys/bus/ccw/devices/. These subdirectories are then named with the bus
IDs of the devices. For example, a qeth device with bus IDs 0.0.fc00, 0.0.fc01,
and 0.0.fc02 is represented as /sys/bus/ccw/drivers/qeth/0.0.fc00
2. Configure the device.
3. Set the device online.

216 Device Drivers, Features, and Commands on SLES 12 SP2

 4. Activate the device and assign an IP address to it.

What to do next

These tasks and the configuration options are described in detail in Working with
qeth devices on page 224.

qeth interface names and device directories

The qeth device driver automatically assigns interface names to the qeth group
devices and creates the corresponding sysfs structures.

According to the type of CHPID and feature used, the naming scheme uses the
following base names:
eth<n>
for Ethernet features.
hsi<n>
for HiperSockets devices.

where <n> is an integer that uniquely identifies the device. When the first device
for a base name is set online it is assigned 0, the second is assigned 1, the third 2,
and so on. Each base name is counted separately.

For example, the interface name of the first Ethernet feature that is set online is
eth0, the second eth1. When the first HiperSockets device is set online, it is
assigned the interface name hsi0.

While an interface is online, it is represented in sysfs as:

/sys/class/net/<interface>

The qeth device driver shares the name space for Ethernet interfaces with the LCS
device driver. Each driver uses the name with the lowest free identifier <n>,
regardless of which device driver occupies the other names. For example, assume
that the first qeth Ethernet feature is set online and there already is one LCS
Ethernet feature online. Then the LCS feature is named eth0 and the qeth feature
is named eth1. See also LCS interface names on page 287.

The mapping between interface names and the device bus-ID that represents the
qeth group device in sysfs is preserved when a device is set offline and back
online. However, it can change when rebooting, when devices are ungrouped, or
when devices appear or disappear with a machine check.

Finding out the interface name of a qeth group device on page 235 and Finding
out the bus ID of a qeth interface on page 235 provide information about
mapping device bus-IDs and interface names.

Support for IP Version 6 (IPv6)

The qeth device driver supports IPv6 in many network setups.

IPv6 is supported on:

v Ethernet interfaces of the OSA-Express adapter that runs in QDIO mode.
v HiperSockets layer 2 and layer 3 interfaces.
v z/VM guest LAN interfaces running in QDIO or HiperSockets layer 3 mode.
v z/VM guest LAN and VSWITCH interfaces in layer 2.

Chapter 14. qeth: OSA-Express (QDIO) and HiperSockets 217

 There are noticeable differences between the IP stacks for versions 4 and 6. Some
concepts in IPv6 are different from IPv4, such as neighbor discovery, broadcast,
and Internet Protocol security (IPsec). IPv6 uses a 16-byte address field, while the
addresses under IPv4 are 4 bytes in length.

Stateless autoconfiguration generates unique IP addresses for all Linux instances,

even if they share an OSA-Express adapter with other operating systems.

Be aware of the IP version when you specify IP addresses and when you use
commands that return IP-version specific output (such as qetharp).

MAC headers in layer 2 mode

In LAN environments, data packets find their destination through Media Access
Control (MAC) addresses in their MAC header.

MAC addr. } MAC header MAC addr. MAC addr.

IP addr. } IP header IP addr. IP addr.

Datagram Datagram Datagram

Linux

App.
Network
LAN device stack
LAN adapter driver

Hardware
Figure 38. Standard IPv4 processing

MAC address handling as shown in Figure 38) applies to non-mainframe

environments and a mainframe environment with an OSA-Express adapter where
the layer2 option is enabled.

The layer2 option keeps the MAC addresses on incoming packets. Incoming and
outgoing packets are complete with a MAC header at all stages between the Linux
network stack and the LAN as shown in Figure 38. This layer2-based forwarding
requires unique MAC addresses for all concerned Linux instances.

In layer 2 mode, the Linux TCP/IP stack has full control over the MAC headers
and the neighbor lookup. The Linux TCP/IP stack does not configure IPv4 or IPv6
addresses into the hardware, but requires a unique MAC address for the card.
Users working with a directly attached OSA-card should assign a unique
MAC-address themselves.

For Linux instances that are directly attached to an OSA-Express adapter in QDIO
mode, you should assign the MAC addresses yourself. You can add a line
LLADDR=<MAC address> to the configuration file /etc/sysconfig/network/ifcfg-
<if-name>. Alternatively, you can change the MAC address by issuing the
command:
ip link set addr <MAC address> dev <interface>

218 Device Drivers, Features, and Commands on SLES 12 SP2

 Note: Be sure not to assign the MAC address of the OSA-Express adapter to your
Linux instance.

For OSX and OSM CHPIDs, you cannot set your own MAC addresses. Linux uses
the MAC addresses defined by the Unified Resource Manager.

For HiperSockets connections, a MAC address is generated.

For connections within a QDIO-based z/VM guest LAN environment, z/VM

assigns the necessary MAC addresses to its guests.

MAC headers in layer 3 mode

A qeth layer 3 mode device driver is an Ethernet offload engine for IPv4 and a
partial Ethernet offload engine for IPv6. Hence, there are some special things to
understand about the layer 3 mode.

To support IPv6 and protocols other than IPv4, the device driver registers a layer 3
card as an Ethernet device to the Linux TCP/IP stack.

In layer 3 mode, the OSA-Express adapter in QDIO mode removes the MAC
header with the MAC address from incoming IPv4 packets. It uses the registered
IP addresses to forward a packet to the recipient TCP/IP stack. See Figure 39. Thus
the OSA-Express adapter is able to deliver IPv4 packets to the correct Linux
images. Apart from broadcast packets, a Linux image can get packets only for IP
addresses it configured in the stack and registered with the OSA-Express adapter.

(faked)
MAC addr. } MAC header MAC addr.
IP addr. } IP header IP addr. IP addr.

Datagram Datagram Datagram

Linux

App.
Network
LAN device stack
LAN adapter driver

Hardware
Figure 39. MAC address handling in layer3 mode

The OSA-Express QDIO microcode builds MAC headers for outgoing IPv4 packets
and removes them from incoming IPv4 packets. Hence, the operating systems'
network stacks send and receive only IPv4 packets without MAC headers.

This lack of MAC headers can be a problem for applications that expect MAC
headers. For examples of how such problems can be resolved, see Setting up for
DHCP with IPv4 on page 272.

Chapter 14. qeth: OSA-Express (QDIO) and HiperSockets 219

 Outgoing frames
The qeth device driver registers the layer 3 card as an Ethernet device. Therefore,
the Linux TCP/IP stack will provide complete Ethernet frames to the device driver.

If the hardware does not require the Ethernet frame (for example, for IPv4) the
driver removes the Ethernet header prior to sending the frame to the hardware. If
necessary information like the Ethernet target address is not available (because of
the offload functionality) the value is filled with the hardcoded address FAKELL.
Table 39. Ethernet addresses of outgoing frames
Frame Destination address Source address
IPv4 FAKELL Real device address
IPv6 Real destination address Real device address
Other packets Real destination address Real device address

Incoming frames
The device driver provides Ethernet headers for all incoming frames.

If necessary information like the Ethernet source address is not available (because
of the offload functionality) the value is filled with the hardcoded address
FAKELL.
Table 40. Ethernet addresses of incoming frames
Frame Destination address Source address
IPv4 Real device address FAKELL
IPv6 Real device address FAKELL
Other packets Real device address Real source address

Note that if a source or destination address is a multicast or broadcast address the

device driver can provide the corresponding (real) Ethernet multicast or broadcast
address even when the packet was delivered or sent through the offload engine.
Always providing the link layer headers enables packet socket applications like
tcpdump to work properly on a qeth layer 3 device without any changes in the
application itself (the patch for libpcap is no longer required).

While the faked headers are syntactically correct, the addresses are not authentic,
and hence applications requiring authentic addresses will not work. Some
examples are given in Table 41.
Table 41. Applications that react differently to faked headers
Application Support Reason
tcpdump Yes Displays only frames, fake Ethernet information is
displayed.
iptables Partially As long as the rule does not deal with Ethernet
information of an IPv4 frame.
dhcp Yes Is non-IPv4 traffic.

IP addresses
The network stack of each operating system that shares an OSA-Express adapter in
QDIO mode registers all its IP addresses with the adapter.

220 Device Drivers, Features, and Commands on SLES 12 SP2

 Whenever IP addresses are deleted from or added to a network stack, the device
drivers download the resulting IP address list changes to the OSA-Express adapter.

For the registered IP addresses, the OSA-Express adapter off-loads various

functions, in particular also:
v Handling MAC addresses and MAC headers
v ARP processing

ARP:

The OSA-Express adapter in QDIO mode responds to Address Resolution Protocol

(ARP) requests for all registered IPv4 addresses.

ARP is a TCP/IP protocol that translates 32-bit IPv4 addresses into the
corresponding hardware addresses. For example, for an Ethernet device, the
hardware addresses are 48-bit Ethernet Media Access Control (MAC) addresses.
The mapping of IPv4 addresses to the corresponding hardware addresses is
defined in the ARP cache. When it needs to send a packet, a host consults the ARP
cache of its network adapter to find the MAC address of the target host.

If there is an entry for the destination IPv4 address, the corresponding MAC
address is copied into the MAC header and the packet is added to the appropriate
interface's output queue. If the entry is not found, the ARP functions retain the
IPv4 packet, and broadcast an ARP request asking the destination host for its MAC
address. When a reply is received, the packet is sent to its destination.

Note:
1. On an OSA-Express adapter in QDIO mode, do not set the NO_ARP flag on
the Linux Ethernet device. The device driver disables the ARP resolution for
IPv4. Because the hardware requires no neighbor lookup for IPv4, but neighbor
solicitation for IPv6, the NO_ARP flag is not allowed on the Linux Ethernet
device.
2. On HiperSockets, which is a full Ethernet offload engine for IPv4 and IPv6 and
supports no other traffic, the device driver sets the NO_ARP flag on the Linux
Ethernet interface. Do not remove this flag from the interface.

| Layer 2 bridge port function

| HiperSockets ports that operate in layer 2 mode can be set up to receive all frames
| that are addressed to unknown MAC addresses.

| Other architectures

| Non z Systems networks use Ethernet Network Interface Controllers (NICs) to pass
| traffic between the operating system and the network. Normally, a NIC filters
| incoming traffic to admit only frames with destination MAC addresses that match
| addresses that are registered with the NIC.

| However, a NIC can also be configured to receive and pass to the operating system
| all Ethernet frames that reach it, regardless of the destination MAC address. This
| mode of operation is known as promiscuous mode. For example, promiscuous
| mode is a prerequisite for configuring a NIC as a member of a Linux software
| bridge.

| For more information about how to set up a software bridge, see the
| documentation that is provided by your distributor, or the bridging how-to
Chapter 14. qeth: OSA-Express (QDIO) and HiperSockets 221
| available at http://www.tldp.org/HOWTO/BRIDGE-STP-HOWTO

| z/Architecture

| OSA and HiperSockets adapters on z Systems do not have a direct equivalent of

| promiscuous mode of operation. Instead, OSA and HiperSockets hardware support
| bridge port functions. The operating system can assign a bridge port role to a
| logical port, and the adapter assigns an active state to one of the logical ports to
| which a role was assigned.

| A local port in active bridge port state receives all Ethernet frames with unknown
| destination MAC addresses. Figure 40 shows a setup with a HiperSockets bridge
| port and an OSA bridge port.
|
|

|
| Figure 40. HiperSockets and OSA bridge port in Linux
|
| HiperSockets only: Permission to configure ports as bridge ports must be granted
| in IBM zEnterprise Unified Resource Manager (zManager).

| Differences between promiscuous mode and bridge-port roles

| Making a logical port of an OSA or HiperSockets adapter an active bridge port is

| similar to enabling promiscuous mode on a non-mainframe NIC that is connected
| to a real Ethernet switch. However, there are important differences:
| Number of ports in promiscuous mode
| v Real switches: Any number of interfaces that are connected to a real
| switch can be turned to promiscuous mode, and all of them then receive
| frames with unknown destination addresses.
| v Bridge ports (on z): Although you can assign the bridge-port role to
| multiple ports of a single OSA or HiperSockets adapter, only one port is
| active and receives traffic to unknown destinations.
| Interception of traffic to other systems
|

222 Device Drivers, Features, and Commands on SLES 12 SP2

| v Real switches: A port of a real switch can be configured to receive
| frames with both known and unknown destinations. If a NIC in
| promiscuous mode is connected to the port, the corresponding host
| receives a copy of all traffic that passes through the switch. This includes
| traffic that is destined to other hosts connected to this switch.
| v Bridge ports (on z): Only frames with unknown destinations are passed
| to the operating system. It is not possible to intercept traffic addressed to
| systems connected to other ports of the same OSA or HiperSockets
| adapter.
| Limitation by the source of traffic (OSA bridge port only)
| v Real switches and HiperSockets bridge-port LAN: Frames with unknown
| destination MAC addresses are delivered to the promiscuous interfaces
| regardless of the port through which the frames enter the switch or
| HiperSockets adapter.
| v OSA bridge port only: An active bridge port learns which MAC
| addresses need to be routed to the owning system by analyzing ARP
| and other traffic. Incoming frames are routed to the active bridge port if
| their destination MAC address:
| Matches an address that is learned or registered with the bridge port
| Is not learned or registered with any of the local ports of the OSA
| adapter, but arrived from the physical Ethernet port

| Bridge port roles

| Linux can assign a primary or secondary role to a logical port of a HiperSockets

| adapter. Only one logical port of such an adapter can be assigned the primary role,
| but multiple other logical ports can be assigned secondary role. When one or more
| logical ports of an adapter are assigned primary or secondary role, the hardware
| ensures that exactly one of these ports is active. The active port receives frames
| with unknown destination. When a port with primary role is present, it always
| becomes active. When only ports with secondary role are present, the hardware
| decides which one becomes active. Changes in the ports' state are reported to
| Linux user space through udev events.

| See Configuring a network device as a member of a Linux bridge on page 255.

Setting up the qeth device driver

No module parameters exist for the qeth device driver. qeth devices are set up
using sysfs.

Loading the qeth device driver modules

There are no module parameters for the qeth device driver. SUSE Linux Enterprise
Server 12 SP2 loads the required device driver modules for you when a device
becomes available.

You can also load the module with the modprobe command:

Chapter 14. qeth: OSA-Express (QDIO) and HiperSockets 223

 qeth module syntax

modprobe qeth
qeth_l2
qeth_l3

where:
qeth is the core module that contains common functions that are used for both
the layer 2 and layer 3 disciplines.
qeth_l2
is the module that contains layer 2 discipline-specific code.
qeth_l3
is the module that contains layer 3 discipline-specific code.

When a qeth device is configured for a particular discipline, the driver tries to
automatically load the corresponding discipline module.

Switching the discipline of a qeth device

To switch the discipline of a device the network interface must be shut down and
the device must be offline.

If the new discipline is accepted by the device driver the old network interface will
be deleted. When the new discipline is set online the first time the new network
interface is created.

Removing the modules

Removing a module is not possible if there are cross dependencies between the
discipline modules and the core module.

To release the dependencies from the core module to the discipline module, all
devices of this discipline must be ungrouped. Now the discipline module can be
removed. If all discipline modules are removed, the core module can be removed.

Working with qeth devices

Typical tasks that you need to perform when working with qeth devices include
creating group devices, finding out the type of a network adapter, and setting a
device online or offline.

About this task

Attention: Use the procedures described here for dynamic testing of

configuration settings. For persistent configuration in a production system, use one
of the SUSE-provided tools YaST, yast2, or the qeth_configure command. For more
details about the qeth_configure command, see the man page.

YaST creates a udev configuration file called /etc/udev/rules.d/xx-qeth-

0.0.xxxx.rules. Additionally, cross-platform network configuration parameters are
defined in /etc/sysconfig/network/ifcfg-<if_name>

224 Device Drivers, Features, and Commands on SLES 12 SP2

 Table 42 and Table 44 on page 226 serve as both a task overview and a summary of
the attributes and the possible values you can write to them. Underlined values are
defaults.

Not all attributes are applicable to each device. Some attributes apply only to
HiperSockets or only to OSA-Express CHPIDs in QDIO mode, other attributes are
applicable to IPv4 interfaces only. See the task descriptions for the applicability of
each attribute.

OSA for NCP handles NCP-related packets. Most of the attributes do not apply to
OSA for NCP devices. The attributes that apply are:
v if_name
v card_type
v buffer_count
v recover
Table 42. qeth tasks and attributes common to layer2 and layer3
Task Corresponding attributes Possible attribute values
Creating a qeth group device on page 227 group n/a
Removing a qeth group device on page 228 ungroup 0 or 1
Setting the layer2 attribute on page 229 layer2 0 or 1, see Layer 2 and
layer 3 on page 215
| Using priority queueing on page 230 priority_queueing prio_queueing_vlan
| prio_queueing_skb
prio_queueing_prec
prio_queueing_tos
no_prio_queueing
no_prio_queueing:0
no_prio_queueing:1
no_prio_queueing:2
no_prio_queueing:3
Specifying the number of inbound buffers on page 232 buffer_count integer in the range 8
-128. The default is 64 for
OSA devices and 128 for
HiperSockets devices
Specifying the relative port number on page 233 portno integer, either 0 or 1, the
default is 0
Finding out the type of your network adapter on page 233 card_type n/a, read-only
Setting a device online or offline on page 234 online 0 or 1
Finding out the interface name of a qeth group device on if_name n/a, read-only
page 235
Finding out the bus ID of a qeth interface on page 235 none n/a
Activating an interface on page 235 none n/a
Deactivating an interface on page 238 none n/a
Recovering a device on page 238 recover 1
Turning inbound checksum calculations on and off on none n/a
page 239
Turning outbound checksum calculations on and off on none n/a
page 239
Isolating data connections on page 240 isolation none, drop, forward

Chapter 14. qeth: OSA-Express (QDIO) and HiperSockets 225

Table 42. qeth tasks and attributes common to layer2 and layer3 (continued)
Task Corresponding attributes Possible attribute values
Starting and stopping collection of QETH performance performance_stats 0 or 1
statistics on page 243
Capturing a hardware trace on page 244 hw_trap
arm
disarm
A value of -1 means that the layer has not been set and that the default layer setting is used when the device is set
online.

Table 43. qeth functions and attributes in layer 2 mode

Function Corresponding attributes Possible attribute values
Configuring a network device as a member of a Linux bridge_role primary, secondary, none
bridge on page 255 bridge_state active, standby, inactive
bridge_hostnotify 0 or 1

Table 44. qeth tasks and attributes in layer 3 mode

Task Corresponding attributes Possible attribute values
Setting up a Linux router on page 246 route4 primary_router
route6 secondary_router
primary_connector
secondary_connector
multicast_router
no_router
Enabling and disabling TCP segmentation offload on page none n/a
248
Faking broadcast capability on page 249 fake_broadcast 0 or 1
Taking over IP addresses on page 249 ipa_takeover/enable 0 or 1 or toggle
ipa_takeover/add4 IPv4 or IPv6 IP address
ipa_takeover/add6 and mask bits
ipa_takeover/del4
ipa_takeover/del6
ipa_takeover/invert4 0 or 1 or toggle
ipa_takeover/invert6
Configuring a device for proxy ARP on page 253 rxip/add4 IPv4 or IPv6 IP address
rxip/add6
rxip/del4
rxip/del6
Configuring a device for virtual IP address (VIPA) on page vipa/add4 IPv4 or IPv6 IP address
254 vipa/add6
vipa/del4
vipa/del6
Configuring a HiperSockets device for AF_IUCV hsuid 1 to 8 characters
addressing on page 255
Setting up a HiperSockets network traffic analyzer on page sniffer 0 or 1
273
not valid for HiperSockets

226 Device Drivers, Features, and Commands on SLES 12 SP2

 Tip: Use the qethconf command instead of using the attributes for IPA, proxy
ARP, and VIPA directly (see qethconf - Configure qeth devices on page 625). In
YaST, you can use "IPA Takeover".

sysfs provides multiple paths through which you can access the qeth group device
attributes. For example, if a device with bus ID 0.0.a100 corresponds to interface
eth0:
/sys/bus/ccwgroup/drivers/qeth/0.0.a100
/sys/bus/ccwgroup/devices/0.0.a100
/sys/devices/qeth/0.0.a100
/sys/class/net/eth0/device

all lead to the attributes for the same device. For example, the following
commands are all equivalent and return the same value:
# cat /sys/bus/ccwgroup/drivers/qeth/0.0.a100/if_name
eth0
# cat /sys/bus/ccwgroup/devices/0.0.a100/if_name
eth0
# cat /sys/devices/qeth/0.0.a100/if_name
eth0
# cat /sys/class/net/eth0/device/if_name
eth0

However, the path through /sys/class/net is available only while the device is
online. Furthermore, it might lead to a different device if the assignment of
interface names changes after rebooting or when devices are ungrouped and new
group devices created.

Tips:
v Work through one of the paths that are based on the device bus-ID.
v Using SUSE Linux Enterprise Server 12 SP2, you set qeth attributes in YaST.
YaST, in turn, creates a udev configuration file called /etc/udev/rules.d/xx-
qeth-0.0.xxxx.rules. Additionally, cross-platform network configuration
parameters are defined in /etc/sysconfig/network/ifcfg-<if_name>.

The following sections describe the tasks in detail.

Creating a qeth group device

Use the znetconf command to configure network devices. Alternatively, you can
use sysfs.

Before you begin

You need to know the device bus-IDs that correspond to the read, write, and data
subchannel of your OSA-Express CHPID in QDIO mode or HiperSockets CHPID
as defined in the IOCDS of your mainframe.

Procedure

To create a qeth group device, either:

v Issue the znetconf command to create and configure a group device. The
command groups the correct bus-IDs for you and sets the device online. For
information about the znetconf command, see znetconf - List and configure
network devices on page 665.

Chapter 14. qeth: OSA-Express (QDIO) and HiperSockets 227

 v Write the device numbers of the subchannel triplet to the sysfs group attribute to
only define a group device. Issue a command of the form:
# echo <read_device_bus_id>,<write_device_bus_id>,<data_device_bus_id> > /sys/bus/ccwgroup/drivers/qeth/group

Results

The qeth device driver uses the device bus-ID of the read subchannel to create a
directory for a group device:
/sys/bus/ccwgroup/drivers/qeth/<read_device_bus_id>

This directory contains a number of attributes that determine the settings of the
qeth group device. The following sections describe how to use these attributes to
configure a qeth group device.

Example

In this example (see Figure 41), a single OSA-Express CHPID in QDIO mode is
used to connect a Linux instance to a network.

Mainframe configuration:

Figure 41. Mainframe configuration

Linux configuration:

Assuming that 0.0.aa00 is the device bus-ID that corresponds to the read
subchannel:
# echo 0.0.aa00,0.0.aa01,0.0.aa02 > /sys/bus/ccwgroup/drivers/qeth/group

This command results in the creation of the following directories in sysfs:

v /sys/bus/ccwgroup/drivers/qeth/0.0.aa00
v /sys/bus/ccwgroup/devices/0.0.aa00
v /sys/devices/qeth/0.0.aa00

Both the command and the resulting directories would be the same for a
HiperSockets CHPID.

Removing a qeth group device

Use the ungroup sysfs attribute to remove a qeth group device.

228 Device Drivers, Features, and Commands on SLES 12 SP2

 Before you begin

The device must be set offline before you can remove it.

Procedure

To remove a qeth group device, write 1 to the ungroup attribute. Issue a command
of the form:
echo 1 > /sys/bus/ccwgroup/drivers/qeth/<device_bus_id>/ungroup

Example

This command removes device 0.0.aa00:

echo 1 > /sys/bus/ccwgroup/drivers/qeth/0.0.aa00/ungroup

Setting the layer2 attribute

If the detected hardware is known to be exclusively run in a discipline, the
corresponding discipline module is automatically requested.

Before you begin

v To change a configured layer2 attribute, the network interface must be shut
down and the device must be set offline.
v If you are using the layer2 option within a QDIO-based guest LAN environment,
you cannot define a VLAN with ID 1, because ID 1 is reserved for z/VM use.

About this task

The qeth device driver attempts to load the layer 3 discipline for HiperSockets
devices and layer 2 for non-HiperSockets devices.

You can use the layer 2 mode for almost all device types. However, note the
following about layer 2 to layer 3 conversion:
real OSA-Express
Hardware is able to convert layer 2 to layer 3 traffic and vice versa and
thus there are no restrictions.
HiperSockets
There is no support for layer 2 to layer 3 conversion and, thus, no
communication is possible between HiperSockets layer 2 interfaces and
HiperSockets layer 3 interfaces. Do not include HiperSockets layer 2
interfaces and HiperSockets layer 3 interfaces in the same LAN.
z/VM guest LAN
Linux has to configure the same mode as the underlying z/VM virtual
LAN definition. The z/VM definition "Ethernet mode" is available for
VSWITCHes and for guest LANs of type QDIO.

Chapter 14. qeth: OSA-Express (QDIO) and HiperSockets 229

 Procedure

The qeth device driver separates the configuration options in sysfs according to the
device discipline. Hence the first configuration action after you group the device
must be the configuration of the discipline. To set the discipline, issue a command
of the form:
echo <integer> > /sys/devices/qeth/<device_bus_id>/layer2

where <integer> is
v 0 to turn off the layer2 attribute; this results in the layer 3 discipline.
v 1 to turn on the layer2 attribute; this results in the layer 2 discipline (default).
If the layer2 attribute has a value of -1, the layer was not set. The default layer
setting is used when the device is set online.

Results

If you configured the discipline successfully, more configuration attributes are

shown (for example route4 for the layer 3 discipline) and can be configured. If an
OSA device is not configured for a discipline but is set online, the device driver
assumes that it is a layer 2 device. It then tries to load the layer 2 discipline.

For information about layer2, see:

v OSA-Express Customer's Guide and Reference, SA22-7935
v OSA-Express Implementation Guide, SG25-5848
v Networking Overview for Linux on zSeries, REDP-3901
v z/VM Connectivity, SC24-6174

Using priority queueing

An OSA-Express CHPID in QDIO mode has up to four output queues (queues 0 -
3) in central storage. The priority queueing feature gives these queues different
priorities (queue 0 having the highest priority). The four output queues are
available only if multiple priority is enabled for queues on the OSA-Express
CHPID in QDIO mode.

Before you begin

v Priority queueing applies to OSA-Express CHPIDs in QDIO mode only.
v If more than 160 TCP/IP stacks per OSA-Express CHPID are defined in the
IOCDS, priority queueing is disabled.
v The device must be offline while you set the queueing options.

About this task

Queueing is relevant mainly to high-traffic situations. When there is little traffic,

queueing has no impact on processing. The qeth device driver can put data on one
or more of the queues. By default, the driver uses queue 2 for all data.

Procedure

You can determine how outgoing IP packages are assigned to queues by setting a
value for the priority_queueing attribute of your qeth device.
Issue a command of the form:

230 Device Drivers, Features, and Commands on SLES 12 SP2

 # echo <method> > /sys/bus/ccwgroup/drivers/qeth/<device_bus_id>/priority_queueing

where <method> can be any of these values:

| prio_queueing_vlan
| to base the queue assignment on the two most significant bits in the priority
| code point in the IEEE 802.1Q header as used in VLANs. This value affects
| only traffic with VLAN headers, and hence works only with qeth devices in
| layer 2 mode.
| You can set the priority code point in the IEEE 802.1Q headers of the traffic
| based on skb->priority by using a command of the form:
| ip link add link <link> name <name> type vlan id <vlan-id> egress-qos-map <mapping>
|
|

| Note: Enabling this option makes all traffic default to queue 3.

| prio_queueing_skb
| to base the queue assignment on the priority flag of the skbs. An skb, or socket
| buffer, is a Linux kernel-internal structure that represents network data. The
| mapping to the priority queues is as follows:
| Table 45. Mapping of flag value to priority queues
| Priority flag of the skb Priority queue
| 0-1 3
| 2-3 2
| 4-5 1
| 6 0
|
| You can use prio_queueing_skb for any network setups, including conventional
| LANs.
| Use either sockopt SO_PRIORITY or an appropriate iptables command to adjust
| the priority flag of the skb (skb->priority).
| Note: The priority flag of the skbs defaults to 0, hence enabling this option
| makes all traffic default to queue 3.
prio_queueing_prec
to base the queue assignment on the two most significant bits of each packet's
| IP header precedence field. To set the precedence field, use sockopt IP_TOS (for
| IPv4) or IPV6_TCLASS (for IPv6).
| Note: Enabling this option makes all traffic default to queue 3.
prio_queueing_tos
Deprecated; do not use for new setups.
no_prio_queueing
causes the qeth device driver to use queue 2 for all packets. This value is the
default.
no_prio_queueing:0
causes the qeth device driver to use queue 0 for all packets.
no_prio_queueing:1
causes the qeth device driver to use queue 1 for all packets.

Chapter 14. qeth: OSA-Express (QDIO) and HiperSockets 231

 no_prio_queueing:2
causes the qeth device driver to use queue 2 for all packets. This value is
equivalent to the default.
no_prio_queueing:3
causes the qeth device driver to use queue 3 for all packets.

Example
To read the current value of priority queueing for device 0.0.a110, issue:
# cat /sys/bus/ccwgroup/drivers/qeth/0.0.a110/priority_queueing

Possible results are:

| by VLAN headers
| if prio_queueing_vlan is set.
| by skb-priority
| if prio_queueing_skb is set.
by precedence
if prio_queueing_prec is set.
by type of service
if prio_queuing_tos is set.
always queue <x>
otherwise.

| To configure queueing by skb->priority setting for device 0.0.a110 issue:

| # echo prio_queueing_skb > /sys/bus/ccwgroup/drivers/qeth/0.0.a110/priority_queueing
|
|

| Specifying the number of inbound buffers

Depending on the amount of available storage and the amount of traffic, you can
assign 8 - 128 inbound buffers for each qeth group device.

Before you begin

The device must be offline while you specify the number of buffers for inbound
traffic.

About this task

By default, the qeth device driver assigns 64 inbound buffers to OSA devices and
128 to HiperSockets devices.

The Linux memory usage for inbound data buffers for the devices is (number of
buffers) (buffer size).

The buffer size is equivalent to the frame size, which depends on the type of
CHPID:
| v For an OSA-Express CHPID in QDIO mode: 64 KB
v For HiperSockets: depending on the HiperSockets CHPID definition, 16 KB,
24 KB, 40 KB, or 64 KB

232 Device Drivers, Features, and Commands on SLES 12 SP2

 Procedure

Set the buffer_count attribute to the number of inbound buffers you want to
assign. Issue a command of the form:
# echo <number> > /sys/bus/ccwgroup/drivers/qeth/<device_bus_id>/buffer_count

Example

In this example, 64 inbound buffers are assigned to device 0.0.a000.

# echo 64 > /sys/bus/ccwgroup/drivers/qeth/0.0.a000/buffer_count

Specifying the relative port number

Use the portno sysfs attribute to specify the relative port number.

Before you begin

v This description applies to adapters that, per CHPID, show more than one port
to Linux.
v The device must be offline while you specify the relative port number.

Procedure

By default, the qeth group device uses port 0. To use a different port, issue a
command of the form:
# echo <integer> > /sys/bus/ccwgroup/drivers/qeth/<device_bus_id>/portno

Where <integer> is either 0 or 1.

Example

In this example, port 1 is assigned to the qeth group device.

# echo 1 > /sys/bus/ccwgroup/drivers/qeth/0.0.a000/portno

Finding out the type of your network adapter

Use the card_type attribute to find out the type of the network adapter through
which your device is connected.

Procedure

You can find out the type of the network adapter through which your device is
connected. To find out the type, read the card_type attribute of the device. Issue a
command of the form:
# cat /sys/bus/ccwgroup/drivers/qeth/<device_bus_id>/card_type

The card_type attribute gives information about both the type of network adapter
and the type of network link (if applicable) available at the card's ports. See
Table 46 on page 234 for details.

Chapter 14. qeth: OSA-Express (QDIO) and HiperSockets 233

 Table 46. Possible values of card_type and what they mean
Value of card_type Adapter type Link type
OSD_10GIG OSA card in OSD mode 10 Gigabit Ethernet
OSD_1000 Gigabit Ethernet, 1000BASE-T
OSD_GbE_LANE Gigabit Ethernet, LAN Emulation
OSD_FE_LANE LAN Emulation
OSD_Express Unknown
OSA for NCP ESCON/CDLC bridge or N/A
OSM OSA-Express for Unified 1000BASE-T
Resource Manager
OSX OSA-Express for zBX 10 Gigabit Ethernet
HiperSockets HiperSockets, CHPID type IQD N/A
Virtual NIC QDIO VSWITCH or guest LAN based N/A
on OSA
Virtual NIC Hiper Guest LAN based on N/A
HiperSockets
Unknown Other

Example

To find the card_type of a device 0.0.a100 issue:

# cat /sys/bus/ccwgroup/drivers/qeth/0.0.a100/card_type
OSD_100

Setting a device online or offline

Use the online device group attribute to set a device online or offline.

Procedure

To set a qeth group device online, set the online device group attribute to 1. To set
a qeth group device offline, set the online device group attribute to 0. Issue a
command of the form:
# echo <flag> > /sys/bus/ccwgroup/drivers/qeth/<device_bus_id>/online

Setting a device online associates it with an interface name (see Finding out the
interface name of a qeth group device on page 235).
Setting a device offline closes this network device. If IPv6 is active, you lose any
IPv6 addresses set for this device. After you set the device online, you can restore
lost IPv6 addresses only by issuing the ip or ifconfig commands again.

Example

To set a qeth device with bus ID 0.0.a100 online issue:

# echo 1 > /sys/bus/ccwgroup/drivers/qeth/0.0.a100/online

To set the same device offline issue:

234 Device Drivers, Features, and Commands on SLES 12 SP2

 # echo 0 > /sys/bus/ccwgroup/drivers/qeth/0.0.a100/online

Finding out the interface name of a qeth group device

When a qeth group device is set online, an interface name is assigned to it.

Procedure

To find the interface name of a qeth group device, either:

v Obtain a mapping for all qeth interfaces and devices by issuing the lsqeth -p
command.
v Find out the interface name of a qeth group device for which you know the
device bus-ID by reading the group device's if_name attribute. Issue a command
of the form:
# cat /sys/bus/ccwgroup/drivers/qeth/<device_bus_id>/if_name

Example
# cat /sys/bus/ccwgroup/drivers/qeth/0.0.a100/if_name
eth0

Finding out the bus ID of a qeth interface

Use the lsqeth -p command to obtain a mapping for all qeth interfaces and
devices. Alternatively, you can use sysfs.

Procedure

To find the device bus-ID that corresponds to an interface, either:

v Use the lsqeth -p command.
v Use the readlink command. For each network interface, there is a directory in
sysfs under /sys/class/net/, for example, /sys/class/net/eth0 for interface
eth0. This directory contains a symbolic link device to the corresponding
device in /sys/devices. Read this link to find the device bus-ID of the device
that corresponds to the interface.

Example

To find out which device bus-ID corresponds to an interface eth0 issue, for
example:
# readlink /sys/class/net/eth0/device
../../../0.0.a100

In this example, eth0 corresponds to the device bus-ID 0.0.a100.

Activating an interface
Use the ip command or equivalent to activate an interface.

Chapter 14. qeth: OSA-Express (QDIO) and HiperSockets 235

 Before you begin
v You must know the interface name of the qeth group device (see Finding out
the interface name of a qeth group device on page 235).
v You must know the IP address that you want to assign to the device.

About this task

The MTU size defaults to the correct settings for HiperSockets devices. For
OSA-Express CHPIDs in QDIO mode, the default MTU size depends on the device
mode, layer 2 or layer 3.
v For layer 2, the default MTU is 1500 bytes.
v For layer 3, the default MTU is 1492 bytes.

In most cases, the default MTU sizes are well suited for OSA-Express CHPIDs in
QDIO mode. If your network is laid out for jumbo frames, increase the MTU size
to a maximum of 9000 bytes for layer 2, or to 8992 bytes for layer 3. Exceeding the
defaults for regular frames or the maximum frame sizes for jumbo frames might
cause performance degradation. See OSA-Express Customer's Guide and Reference,
SA22-7935 for more details about MTU size.

For HiperSockets, the maximum MTU size is restricted by the maximum frame
size as announced by the Licensed Internal Code (LIC). The maximum MTU is
equal to the frame size minus 8 KB. Hence, the possible frame sizes of 16 KB,
24 KB, 40 KB, or 64 KB result in maximum corresponding MTU sizes of 8 KB,
16 KB, 32 KB, or 56 KB.

The MTU size defaults to the correct settings for both HiperSockets and
OSA-Express CHPIDs in QDIO mode. As a result, you do not need to specify the
MTU size when you activate the interface.

On heavily loaded systems, MTU sizes that exceed 8 KB can lead to memory
allocation failures for packets due to memory fragmentation. A symptom of this
problem are messages of the form "order-N allocation failed" in the system log. In
addition, network connections drop packets, possibly so frequently as to make the
network interface unusable.

As a workaround, use MTU sizes at most of 8 KB (minus header size), even if the
network hardware allows larger sizes. For example, HiperSockets or 10 Gigabit
Ethernet allow larger sizes.

Procedure

You activate or deactivate network devices with ip or an equivalent command. For

details of the ip command, see the ip man page.

Examples
v This example activates a HiperSockets CHPID with broadcast address
192.168.100.255:
# ip addr add 192.168.100.10/24 dev hsi0
# ip link set dev hsi0 up

v This example activates an OSA-Express CHPID in QDIO mode with broadcast

address 192.168.100.255:

236 Device Drivers, Features, and Commands on SLES 12 SP2

 # ip addr add 192.168.100.11/24 dev eth0
# ip link set dev eth0 up

v This example reactivates an interface that was already activated and

subsequently deactivated:
# ip link set dev eth0 up

Confirming that an IP address has been set under layer 3

There may be circumstances that prevent an IP address from being set, most
commonly if another system in the network has set that IP address already.

About this task

The Linux network stack design does not allow feedback about IP address
changes. If ip or an equivalent command fails to set an IP address on an
OSA-Express network CHPID, a query with ip shows the address as being set on
the interface although the address is not actually set on the CHPID.

There are usually failure messages about not being able to set the IP address or
duplicate IP addresses in the kernel messages. You can find these messages in the
output of the dmesg command. In SUSE Linux Enterprise Server 12 SP2, you can
also find the messages in /var/log/messages.

If you are not sure whether an IP address was set properly or experience a
networking problem, check the messages or logs to see if an error was encountered
when setting the address. This also applies in the context of HiperSockets and to
both IPv4 and IPv6 addresses. It also applies to whether an IP address has been set
for IP takeover, for VIPA, or for proxy ARP.

Duplicate IP addresses
The OSA-Express adapter in QDIO mode recognizes duplicate IP addresses on the
same OSA-Express adapter or in the network using ARP and prevents duplicates.

About this task

Several setups require duplicate addresses:

v To perform IP takeover you need to be able to set the IP address to be taken
over. This address exists prior to the takeover. See Taking over IP addresses on
page 249 for details.
v For proxy ARP you need to register an IP address for ARP that belongs to
another Linux instance. See Configuring a device for proxy ARP on page 253
for details.
v For VIPA you need to assign the same virtual IP address to multiple devices. See
Configuring a device for virtual IP address (VIPA) on page 254 for details.

You can use the qethconf command (see qethconf - Configure qeth devices on
page 625) to maintain a list of IP addresses that your device can take over, a list of
IP addresses for which your device can handle ARP, and a list of IP addresses that
can be used as virtual IP addresses, regardless of any duplicates on the same
OSA-Express adapter or in the LAN.

Chapter 14. qeth: OSA-Express (QDIO) and HiperSockets 237

 Deactivating an interface
You can deactivate an interface with ip or an equivalent command or by setting
the network device offline.

About this task

Setting a device offline involves actions on the attached device, but deactivating a
device only stops the interface logically within Linux.

Procedure

To deactivate an interface with ip. Issue a command of the form:

# ip link set dev <interface_name> down

Example

To deactivate eth0 issue:

# ip link set dev eth0 down

Recovering a device
You can use the recover attribute of a qeth group device to recover it in case of
failure.

About this task

For example, error messages in /var/log/messages from the qeth, qdio, or cio
kernel modules might inform you of a malfunctioning device.

Procedure

Issue a command of the form:

# echo 1 > /sys/bus/ccwgroup/drivers/qeth/<device_bus_id>/recover

Example
# echo 1 > /sys/bus/ccwgroup/drivers/qeth/0.0.a100/recover

Configuring offload operations

Some operations can be offloaded to the OSA adapter, thus relieving the burden on
the host CPU.

The qeth device driver supports offloading the following operations:

v Inbound (receive) checksum calculations
v Outbound (send) checksum calculations

The qeth device driver also supports offloading TSO segmentation, see Enabling
and disabling TCP segmentation offload on page 248.

VLAN interfaces inherit offload settings from their base interface.

238 Device Drivers, Features, and Commands on SLES 12 SP2

Offload operations are supported for OSA connections on layer 3 only. VLAN
interfaces inherit offload settings from their base interface.

The offload operations can be set with the Linux ethtool command, version 6 or
later. See the ethtool man page for details. The following abbreviated example
shows some offload settings:
# ethtool -k eth0
Features for eth0:
rx-checksumming: on
tx-checksumming: on
tx-checksum-ipv4: on
tx-checksum-ip-generic: off [fixed]
tx-checksum-ipv6: off [fixed]
tx-checksum-fcoe-crc: off [fixed]
tx-checksum-sctp: off [fixed]
scatter-gather: on
tx-scatter-gather: on
tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: on
tx-tcp-segmentation: on
tx-tcp-ecn-segmentation: off [fixed]
tx-tcp6-segmentation: off [fixed]
udp-fragmentation-offload: off [fixed]
generic-segmentation-offload: off [requested on]
generic-receive-offload: on
large-receive-offload: off [fixed]
...

Turning inbound checksum calculations on and off

A checksum calculation is a form of redundancy check to protect the integrity of
data. In general, checksum calculations are used for network data.

Procedure

The qeth device driver supports offloading checksum calculations on inbound

packets to the OSA feature.

To enable or disable checksum calculations by the OSA feature, issue a command

of this form:
# ethtool -K <interface_name> rx <value>

where <value> is on or off.

Examples
v To let the OSA feature calculate the inbound checksum for network device eth0,
issue
# ethtool -K eth0 rx on

v To let the host CPU calculate the inbound checksum for network device eth0,
issue
# ethtool -K eth0 rx off

Turning outbound checksum calculations on and off

The qeth device driver supports offloading outbound (send) checksum calculations
to the OSA feature.

Chapter 14. qeth: OSA-Express (QDIO) and HiperSockets 239

 About this task

You can enable or disable the OSA feature calculating the outbound checksums by
using the ethtool command.

Attention: For OSA-Express3 and earlier: When outbound checksum calculations

are offloaded, the OSA feature performs the checksum calculations. Offloaded
checksum calculations only applies to packets that go out to the LAN or come in
from the LAN. Linux instances that share an OSA port exchange packages directly.
The packages are forwarded by the OSA adapter but do not go out on the LAN
and no checksum offload is performed. The qeth device driver cannot detect this,
and so cannot issue any warning about it.

Procedure

Issue a command of the form:

# ethtool -K <interface_name> tx <value>

where <value> is on or off.

Example
v To let the OSA feature calculate the outbound checksum for network device eth0,
issue
# ethtool -K eth0 tx on

v To let the host CPU calculate the outbound checksum for network device eth0,
issue
# ethtool -K eth0 tx off

Isolating data connections

You can restrict communications between operating system instances that share an
OSA port on an OSA adapter.

About this task

A Linux instance can configure the OSA adapter to prevent any direct package
exchange between itself and other operating system instances that share an OSA
adapter. This configuration ensures a higher degree of isolation than VLANs.

QDIO data connection isolation is configured as a policy. The policy is

implemented as a sysfs attribute called isolation. Note that the attribute appears in
sysfs regardless of whether the hardware supports the feature. The policy can take
the following values:
none No isolation. This is the default.
drop Specifies the ISOLATION_DROP policy. All packets from guests that share
an OSA adapter to the guest that has this policy configured are dropped
automatically. The same holds for all packets that are sent by the guest that
has this policy configured to guests on the same OSA card. All packets to

240 Device Drivers, Features, and Commands on SLES 12 SP2

 or from the isolated guest must have a target that is not hosted on the
OSA card. You can accomplish this by a router hosted on a separate
machine or a separate OSA adapter.
For example, assume that three Linux instances share an OSA adapter, but
only one instance (Linux A) must be isolated. Then Linux A declares its
OSA adapter (QDIO Data Connection to the OSA adapter) to be isolated.
Any packet being sent to or from Linux A must pass at least the physical
switch to which the shared OSA adapter is connected. Linux A cannot
communicate with other instances that share the OSA adapter, here B or C.
The two other instances could still communicate directly through the OSA
adapter without the external switch in the network path (see Figure 42).

Figure 42. Linux instance A is isolated from instances B and C

forward
Specifies the ISOLATION_FORWARD policy. All packets are passed
through a switch. The ISOLATION_FORWARD policy requires a network
adapter in Virtual Ethernet Port Aggregator (VEPA) mode with an adjacent
switch port configured for reflective relay mode.
To check whether the switch of the adapter is in reflective relay mode, read
the sysfs attribute switch_attrs. The attribute lists all supported
forwarding modes, with the currently active mode enclosed in square
brackets. For example:
cat /sys/devices/qeth/0.0.f5f0/switch_attrs
802.1 [rr]

The example indicates that the adapter supports both 802.1 forwarding
mode and reflective relay mode, and reflective relay mode (rr) is active.
Using a network adapter in VEPA mode achieves further isolation. VEPA
mode forces traffic from the Linux guests to be handled by the external
switch. For example, Figure 43 on page 242 shows instances A and B with
ISOLATION_FORWARD specified for the policy. All traffic between A and
B goes through the external switch. The rule set of the switch now
determines which connections are possible. The graphic assumes that A can

Chapter 14. qeth: OSA-Express (QDIO) and HiperSockets 241

 communicate with B, but not with C.

Figure 43. Traffic from Linux instance A and B is forced through an external switch

If the ISOLATION_FORWARD policy was enforced successfully, but the

switch port later loses the reflective-relay capability, the device is set offline
to prevent damage.

You can configure the policy regardless of whether the device is online. If the
device is online, the policy is configured immediately. If the device is offline, the
policy is configured when the device comes online.

Examples
v To check the current isolation policy:
# cat /sys/devices/qeth/0.0.f5f0/isolation

v To set the isolation policy to ISOLATION_DROP:

# echo drop > /sys/devices/qeth/0.0.f5f0/isolation

v To set the isolation policy to ISOLATION_FORWARD:

# echo "forward" > /sys/devices/qeth/0.0.f5f0/isolation

If the switch is not capable of VEPA support, or VEPA support is not configured
on the switch, then you cannot set the isolation attribute value to 'forward' while
the device is online. If the switch does not support VEPA and you set the
isolation value 'forward' while the device is offline, then the device cannot be set
online until the isolation value is set back to 'drop' or 'none'.
v To set the isolation policy to none:
# echo "none" > /sys/devices/qeth/0.0.f5f0/isolation

When you use vNICs, VEPA mode must be enabled on the respective VSWITCH.
See z/VM Connectivity, SC24-6174 for information about setting up data connection
isolation on a VSWITCH.

242 Device Drivers, Features, and Commands on SLES 12 SP2

Starting and stopping collection of QETH performance
statistics
Use the performance_stats attribute to start and stop collection of QETH
performance statistics.

About this task

For QETH performance statistics, there is a device group attribute called

/sys/bus/ccwgroup/drivers/qeth/<device_bus_id>/performance_stats.

This attribute is initially set to 0, that is, QETH performance data is not collected.

Procedure

To start collection for a specific QETH device, write 1 to the attribute. For example:
echo 1 > /sys/bus/ccwgroup/drivers/qeth/<device_bus_id>/performance_stats

To stop collection write 0 to the attribute, for example:

echo 0 > /sys/bus/ccwgroup/drivers/qeth/<device_bus_id>/performance_stats

Stopping QETH performance data collection for a specific QETH device is

accompanied by a reset of current statistic values to zero.
To display QETH performance statistics, use the ethtool command. See the
ethtool man page for details.

Chapter 14. qeth: OSA-Express (QDIO) and HiperSockets 243

 Example

The following example shows statistic and device driver information:

# ethtool -S eth0
NIC statistics:
rx skbs: 86
rx buffers: 85
tx skbs: 86
tx buffers: 86
tx skbs no packing: 86
tx buffers no packing: 86
tx skbs packing: 0
tx buffers packing: 0
tx sg skbs: 0
tx sg frags: 0
rx sg skbs: 0
rx sg frags: 0
rx sg page allocs: 0
tx large kbytes: 0
tx large count: 0
tx pk state ch n->p: 0
tx pk state ch p->n: 0
tx pk watermark low: 2
tx pk watermark high: 5
queue 0 buffer usage: 0
queue 1 buffer usage: 0
queue 2 buffer usage: 0
queue 3 buffer usage: 0
rx handler time: 856
rx handler count: 84
rx do_QDIO time: 16
rx do_QDIO count: 11
tx handler time: 330
tx handler count: 87
tx time: 1236
tx count: 86
tx do_QDIO time: 997
tx do_QDIO count: 86
tx csum: 0
tx lin: 0
cq handler count: 0
cq handler time: 0
# ethtool -i eth0
driver: qeth_l3
version: 1.0
firmware-version: 087a
bus-info: 0.0.f5f0/0.0.f5f1/0.0.f5f2
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no

Capturing a hardware trace

Hardware traces are intended for use by the IBM service organization. Hardware
tracing is turned off by default. Turn on the hardware-tracing feature only when
instructed to do so by IBM service.

Before you begin

v The OSA-Express adapter must support the hardware-tracing feature.
v The qeth device must be online to return valid values of the hw_trap attribute.

244 Device Drivers, Features, and Commands on SLES 12 SP2

 About this task

When errors occur on an OSA-Express adapter, both software and hardware traces
must be collected. The hardware-tracing feature requests a hardware trace if an
error is detected. This feature makes it possible to correlate the hardware trace
with the device driver trace. If the hardware-tracing feature is activated, traces are
captured automatically, but you can also start the capturing yourself.

Procedure

To activate or deactivate the hardware-tracing feature, issue a command of the

form:
# echo <value> > /sys/devices/qeth/<device_bus_id>/hw_trap

Where <value> can be:

arm If the hardware-tracing feature is supported, write arm to the hw_trap sysfs
attribute to activate it. If the hardware-tracing feature is present and
activated, the hw_trap sysfs attribute has the value arm.
disarm
Write disarm to the hw_trap sysfs attribute to turn off the hardware-tracing
feature. If the hardware-tracing feature is not present or is turned off, the
hw_trap sysfs attribute has the value disarm. This setting is the default.
trap (Write only) Capture a hardware trace. Hardware traces are captured
automatically, but if asked to do so by IBM service, you can start the
capturing yourself by writing trap to the hw_trap sysfs attribute. The
hardware trap function must be set to arm.

Examples
In this example the hardware-tracing feature is activated for qeth device 0.0.a000:
# echo arm > /sys/devices/qeth/0.0.a000/hw_trap

In this example a trace capture is started on qeth device 0.0.a000:

1. Check that the hw_trap sysfs attribute is set to arm:
# cat /sys/devices/qeth/0.0.a000/hw_trap
arm

2. Start the capture:

# echo trap > /sys/devices/qeth/0.0.a000/hw_trap

Working with qeth devices in layer 3 mode

Tasks you can perform on qeth devices in layer 3 mode include setting up a router,
configuring offload operations, and taking over IP addresses.

Use the layer 2 attribute to set the mode. See Setting the layer2 attribute on page
229 about setting the mode. See Layer 2 and layer 3 on page 215 for general
information about the layer 2 and layer 3 disciplines.

Chapter 14. qeth: OSA-Express (QDIO) and HiperSockets 245

 Setting up a Linux router
By default, your Linux instance is not a router. Depending on your IP version, IPv4
or IPv6 you can use the route4 or route6 attribute of your qeth device to define it
as a router.

Before you begin

v A suitable hardware setup must be in place that enables your Linux instance to
act as a router.
v The Linux instance is set up as a router. To configure Linux running as a z/VM
guest or in an LPAR as a router, IP forwarding must be enabled in addition to
setting the route4 or route6 attribute.
For IPv4, enable IP forwarding by issuing:
# sysctl -w net.ipv4.conf.all.forwarding=1

For IPv6, enable IP forwarding by issuing:

# sysctl -w net.ipv6.conf.all.forwarding=1

About this task

You can set the route4 or route6 attribute dynamically, while the qeth device is
online.

The same values are possible for route4 and route6 but depend on the type of
CHPID, as shown in Table 47.
Table 47. Summary of router setup values
Router specification OSA-Express CHPID in HiperSockets CHPID
QDIO mode
primary_router Yes No
secondary_router Yes No
primary_connector No Yes
secondary_connector No Yes
multicast_router Yes Yes
no_router Yes Yes

Both types of CHPIDs accept:

multicast_router
causes the qeth driver to receive all multicast packets of the CHPID. For a
unicast function for HiperSockets see HiperSockets Network Concentrator on
page 267.
no_router
is the default. You can use this value to reset a router setting to the default.

An OSA-Express CHPID in QDIO mode accepts the following values:

primary_router
to make your Linux instance the principal connection between two networks.

246 Device Drivers, Features, and Commands on SLES 12 SP2

secondary_router
to make your Linux instance a backup connection between two networks.

A HiperSockets CHPID accepts the following values, provided the microcode level
supports the feature:
primary_connector
to make your Linux instance the principal connection between a HiperSockets
network and an external network (see HiperSockets Network Concentrator
on page 267).
secondary_connector
to make your Linux instance a backup connection between a HiperSockets
network and an external network (see HiperSockets Network Concentrator
on page 267).

Example

In this example, two Linux instances, Linux P and Linux S, running on an IBM
mainframe use OSA-Express to act as primary and secondary routers between two
networks. IP forwarding must be enabled for Linux in an LPAR or as a z/VM
guest to act as a router. In SUSE Linux Enterprise Server 12 SP2 you can set IP
forwarding permanently in /etc/sysctl.conf or dynamically with the sysctl
command.
Mainframe configuration:

Figure 44. Mainframe configuration

It is assumed that both Linux instances are configured as routers in the

LPARs or in z/VM.
Linux P configuration:
To create the qeth group devices:
# echo 0.0.0400,0.0.0401,0.0.0402 > /sys/bus/ccwgroup/drivers/qeth/group
# echo 0.0.0200,0.0.0201,0.0.0202 > /sys/bus/ccwgroup/drivers/qeth/group

To make Linux P a primary router for IPv4:

# echo primary_router > /sys/bus/ccwgroup/drivers/qeth/0.0.0400/route4
# echo primary_router > /sys/bus/ccwgroup/drivers/qeth/0.0.0200/route4

Chapter 14. qeth: OSA-Express (QDIO) and HiperSockets 247

 Linux S configuration:
To create the qeth group devices:
# echo 0.0.0404,0.0.0405,0.0.0406 > /sys/bus/ccwgroup/drivers/qeth/group
# echo 0.0.0204,0.0.0205,0.0.0206 > /sys/bus/ccwgroup/drivers/qeth/group

To make Linux S a secondary router for IPv4:

# echo secondary_router > /sys/bus/ccwgroup/drivers/qeth/0.0.0404/route4
# echo secondary_router > /sys/bus/ccwgroup/drivers/qeth/0.0.0204/route4

In this example, qeth device 0.0.1510 is defined as a primary router for IPv6:
/sys/bus/ccwgroup/drivers/qeth # cd 0.0.1510
# echo 1 > online
# echo primary_router > route6
# cat route6
primary router

See HiperSockets Network Concentrator on page 267 for further examples.

Enabling and disabling TCP segmentation offload

Offloading the TCP segmentation operation from the Linux network stack to the
adapter can lead to enhanced performance for interfaces with predominately large
outgoing packets.

Large send (TCP segmentation offload) is supported for OSA connections on layer
3 only. VLAN interfaces inherit offload settings from their base interface.

Procedure

To support TCP segmentation offload (TSO), a network device must support

outbound (TX) checksumming and scatter gather. For this reason, you must turn
on scatter gather and outbound checksumming prior to configuring TSO. All three
options can be turned on or off with a single ethtool command of the form:
# ethtool -K <interface_name> tx <value> sg <value> tso <value>

where <value> is either on or off.

For more information about TX checksumming, see Turning outbound checksum
calculations on and off on page 239.

Attention: When TCP segmentation is offloaded, the OSA feature performs the
calculations. Offloaded calculations apply only to packets that go out to the LAN
or come in from the LAN. Linux instances that share an OSA port exchange
packages directly. The packages are forwarded by the OSA adapter but do not go
out on the LAN and no TCP segmentation calculation is performed. The qeth
device driver cannot detect this, and so cannot issue any warning about it.

Examples
v To enable TSO for a network device eth0 issue:
# ethtool -K eth0 tx on sg on tso on

v To disable TSO for a network device eth0 issue:

248 Device Drivers, Features, and Commands on SLES 12 SP2
 # ethtool -K eth0 tx off sg off tso off

Faking broadcast capability

It is possible to fake the broadcast capability for devices that do not support
broadcasting.

Before you begin

v You can fake the broadcast capability only on devices that do not support
broadcast.
v The device must be offline while you enable faking broadcasts.

About this task

For devices that support broadcast, the broadcast capability is enabled

automatically.

To find out whether a device supports broadcasting, use the ip command. If the
resulting list shows the BROADCAST flag, the device supports broadcast. This
example shows that the device eth0 supports broadcast:
# ip -s link show dev eth0
3: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1492 qdisc pfifo_fast qlen 1000
link/ether 00:11:25:bd:da:66 brd ff:ff:ff:ff:ff:ff
RX: bytes packets errors dropped overrun mcast
236350 2974 0 0 0 9
TX: bytes packets errors dropped carrier collsns
374443 1791 0 0 0 0

Some processes, for example, the gated routing daemon, require the devices'
broadcast capable flag to be set in the Linux network stack.

Procedure

To set the broadcast capable flag for devices that do not support broadcast, set the
fake_broadcast attribute of the qeth group device to 1. To reset the flag, set it to 0.
Issue a command of the form:
# echo <flag> > /sys/bus/ccwgroup/drivers/qeth/<device_bus_id>/fake_broadcast

Example

In this example, a device 0.0.a100 is instructed to pretend that it can broadcast.

# echo 1 > /sys/bus/ccwgroup/drivers/qeth/0.0.a100/fake_broadcast

Taking over IP addresses

You can configure IP takeover if the layer2 option is not enabled. If you enabled
the layer2 option, you can configure for IP takeover as you would in a distributed
server environment.

Chapter 14. qeth: OSA-Express (QDIO) and HiperSockets 249

 About this task

For information about the layer2 option, see MAC headers in layer 2 mode on
page 218.

Taking over an IP address overrides any previous allocation of this address to

another LPAR. If another LPAR on the same CHPID already registered for that IP
address, this association is removed.

An OSA-Express CHPID in QDIO mode can take over IP addresses from any z
Systems operating system. IP takeover for HiperSockets CHPIDs is restricted to
taking over addresses from other Linux instances in the same Central Electronics
Complex (CEC).

IP address takeover between multiple CHPIDs requires ARP for IPv4 and
Neighbor Discovery for IPv6. OSA-Express handles ARP transparently, but not
Neighbor Discovery.

There are three stages to taking over an IP address:

Stage 1: Ensure that your qeth group device is enabled for IP takeover
Stage 2: Activate the address to be taken over for IP takeover
Stage 3: Issue a command to take over the address

Stage 1: Enabling a qeth group device for IP takeover

For OSA-Express and HiperSockets CHPIDs, both the qeth group device that is to
take over an IP address and the device that surrenders the address must be
enabled for IP takeover.

Procedure

By default, qeth devices are not enabled for IP takeover. To enable a qeth group
device for IP address takeover set the enable device group attribute to 1. To switch
off the takeover capability set the enable device group attribute to 0. In sysfs, the
enable attribute is located in a subdirectory ipa_takeover. Issue a command of the
form:
# echo <flag> > /sys/bus/ccwgroup/drivers/qeth/<device_bus_id>/ipa_takeover/enable

Example

In this example, a device 0.0.a500 is enabled for IP takeover:

# echo 1 > /sys/bus/ccwgroup/drivers/qeth/0.0.a500/ipa_takeover/enable

Stage 2: Activating and deactivating IP addresses for takeover

The qeth device driver maintains a list of IP addresses that qeth group devices can
take over or surrender. To enable Linux to take over an IP-address or to surrender
an address, the address must be added to this list.

Procedure

Use the qethconf command to add IP addresses to the list.

v To display the list of IP addresses that are activated for IP takeover issue:

250 Device Drivers, Features, and Commands on SLES 12 SP2

 # qethconf ipa list

v To activate an IP address for IP takeover, add it to the list. Issue a command of

the form:
# qethconf ipa add <ip_address>/<mask_bits> <interface_name>

v To deactivate an IP address delete it from the list. Issue a command of the form:
# qethconf ipa del <ip_address>/<mask_bits> <interface_name>

In these commands, <ip_address>/<mask_bits> is the range of IP addresses to be

activated or deactivated. See qethconf - Configure qeth devices on page 625
for more details about the qethconf command.

IPv4 example:

In this example, there is only one range of IP addresses (192.168.10.0 to

192.168.10.255) that can be taken over by device hsi0.

List the range of IP addresses (192.168.10.0 to 192.168.10.255) that can be taken over
by device hsi0.
# qethconf ipa list
ipa add 192.168.10.0/24 hsi0

The following command adds a range of IP addresses that can be taken over by
device eth0.
# qethconf ipa add 192.168.11.0/24 eth0
qethconf: Added 192.168.11.0/24 to /sys/class/net/eth0/device/ipa_takeover/add4.
qethconf: Use "qethconf ipa list" to check for the result

Listing the activated IP addresses now shows both ranges of addresses.

# qethconf ipa list
ipa add 192.168.10.0/24 hsi0
ipa add 192.168.11.0/24 eth0

The following command deletes the range of IP addresses that can be taken over
by device eth0.
# qethconf ipa del 192.168.11.0/24 eth0
qethconf: Deleted 192.168.11.0/24 from /sys/class/net/eth0/device/ipa_takeover/del4.
qethconf: Use "qethconf ipa list" to check for the result

IPv6 example:

The following command adds one range of IPv6 addresses,

fec0:0000:0000:0000:0000:0000:0000:0000 to fec0:0000:0000:0000:FFFF:FFFF:FFFF:FFFF,
that can be taken over by device eth2.

Add a range of IP addresses:

Chapter 14. qeth: OSA-Express (QDIO) and HiperSockets 251

 qethconf ipa add fec0::/64 eth2
qethconf: Added fec0:0000:0000:0000:0000:0000:0000:0000/64 to
sysfs entry /sys/class/net/eth2/device/ipa_takeover/add6.
qethconf: For verification please use "qethconf ipa list"

Listing the activated IP addresses now shows the range of addresses:

qethconf ipa list
...
ipa add fec0:0000:0000:0000:0000:0000:0000:0000/64 eth2

The following command deletes the IPv6 address range that can be taken over by
eth2:
qethconf ipa del fec0:0000:0000:0000:0000:0000:0000:0000/64 eth2:
qethconf: Deleted fec0:0000:0000:0000:0000:0000:0000:0000/64 from
sysfs entry /sys/class/net/eth2/device/ipa_takeover/del6.
qethconf: For verification please use "qethconf ipa list"

Stage 3: Issuing a command to take over the address

To complete taking over a specific IP address and re