SPARC T7 Series Servers Product Notes

Part No: E54981-14

July 2019
SPARC T7 Series Servers Product Notes
Part No: E54981-14
Copyright © 2015, 2019, Oracle and/or its affiliates. All rights reserved.
This software and related documentation are provided under a license agreement containing restrictions on use and disclosure and are protected by intellectual property laws. Except
as expressly permitted in your license agreement or allowed by law, you may not use, copy, reproduce, translate, broadcast, modify, license, transmit, distribute, exhibit, perform,
publish, or display any part, in any form, or by any means. Reverse engineering, disassembly, or decompilation of this software, unless required by law for interoperability, is
prohibited.
The information contained herein is subject to change without notice and is not warranted to be error-free. If you find any errors, please report them to us in writing.
If this is software or related documentation that is delivered to the U.S. Government or anyone licensing it on behalf of the U.S. Government, then the following notice is applicable:
U.S. GOVERNMENT END USERS: Oracle programs, including any operating system, integrated software, any programs installed on the hardware, and/or documentation,
delivered to U.S. Government end users are "commercial computer software" pursuant to the applicable Federal Acquisition Regulation and agency-specific supplemental
regulations. As such, use, duplication, disclosure, modification, and adaptation of the programs, including any operating system, integrated software, any programs installed on the
hardware, and/or documentation, shall be subject to license terms and license restrictions applicable to the programs. No other rights are granted to the U.S. Government.
This software or hardware is developed for general use in a variety of information management applications. It is not developed or intended for use in any inherently dangerous
applications, including applications that may create a risk of personal injury. If you use this software or hardware in dangerous applications, then you shall be responsible to take all
appropriate fail-safe, backup, redundancy, and other measures to ensure its safe use. Oracle Corporation and its affiliates disclaim any liability for any damages caused by use of this
software or hardware in dangerous applications.
Oracle and Java are registered trademarks of Oracle and/or its affiliates. Other names may be trademarks of their respective owners.
Intel and Intel Xeon are trademarks or registered trademarks of Intel Corporation. All SPARC trademarks are used under license and are trademarks or registered trademarks of
SPARC International, Inc. AMD, Opteron, the AMD logo, and the AMD Opteron logo are trademarks or registered trademarks of Advanced Micro Devices. UNIX is a registered
trademark of The Open Group.
This software or hardware and documentation may provide access to or information about content, products, and services from third parties. Oracle Corporation and its affiliates are
not responsible for and expressly disclaim all warranties of any kind with respect to third-party content, products, and services unless otherwise set forth in an applicable agreement
between you and Oracle. Oracle Corporation and its affiliates will not be responsible for any loss, costs, or damages incurred due to your access to or use of third-party content,
products, or services, except as set forth in an applicable agreement between you and Oracle.
Access to Oracle Support

Oracle customers that have purchased support have access to electronic support through My Oracle Support. For information, visit http://www.oracle.com/pls/topic/lookup?
ctx=acc&id=info or visit http://www.oracle.com/pls/topic/lookup?ctx=acc&id=trs if you are hearing impaired.
Référence: E54981-14
Copyright © 2015, 2019, Oracle et/ou ses affiliés. Tous droits réservés.
Ce logiciel et la documentation qui l'accompagne sont protégés par les lois sur la propriété intellectuelle. Ils sont concédés sous licence et soumis à des restrictions d'utilisation et
de divulgation. Sauf stipulation expresse de votre contrat de licence ou de la loi, vous ne pouvez pas copier, reproduire, traduire, diffuser, modifier, accorder de licence, transmettre,
distribuer, exposer, exécuter, publier ou afficher le logiciel, même partiellement, sous quelque forme et par quelque procédé que ce soit. Par ailleurs, il est interdit de procéder à toute
ingénierie inverse du logiciel, de le désassembler ou de le décompiler, excepté à des fins d'interopérabilité avec des logiciels tiers ou tel que prescrit par la loi.
Les informations fournies dans ce document sont susceptibles de modification sans préavis. Par ailleurs, Oracle Corporation ne garantit pas qu'elles soient exemptes d'erreurs et vous
invite, le cas échéant, à lui en faire part par écrit.
Si ce logiciel, ou la documentation qui l'accompagne, est livré sous licence au Gouvernement des Etats-Unis, ou à quiconque qui aurait souscrit la licence de ce logiciel pour le
compte du Gouvernement des Etats-Unis, la notice suivante s'applique :
U.S. GOVERNMENT END USERS: Oracle programs, including any operating system, integrated software, any programs installed on the hardware, and/or documentation,
delivered to U.S. Government end users are "commercial computer software" pursuant to the applicable Federal Acquisition Regulation and agency-specific supplemental
regulations. As such, use, duplication, disclosure, modification, and adaptation of the programs, including any operating system, integrated software, any programs installed on the
hardware, and/or documentation, shall be subject to license terms and license restrictions applicable to the programs. No other rights are granted to the U.S. Government.
Ce logiciel ou matériel a été développé pour un usage général dans le cadre d'applications de gestion des informations. Ce logiciel ou matériel n'est pas conçu ni n'est destiné à être
utilisé dans des applications à risque, notamment dans des applications pouvant causer un risque de dommages corporels. Si vous utilisez ce logiciel ou ce matériel dans le cadre
d'applications dangereuses, il est de votre responsabilité de prendre toutes les mesures de secours, de sauvegarde, de redondance et autres mesures nécessaires à son utilisation dans
des conditions optimales de sécurité. Oracle Corporation et ses affiliés déclinent toute responsabilité quant aux dommages causés par l'utilisation de ce logiciel ou matériel pour des
applications dangereuses.
Oracle et Java sont des marques déposées d'Oracle Corporation et/ou de ses affiliés. Tout autre nom mentionné peut correspondre à des marques appartenant à d'autres propriétaires
qu'Oracle.
Intel et Intel Xeon sont des marques ou des marques déposées d'Intel Corporation. Toutes les marques SPARC sont utilisées sous licence et sont des marques ou des marques
déposées de SPARC International, Inc. AMD, Opteron, le logo AMD et le logo AMD Opteron sont des marques ou des marques déposées d'Advanced Micro Devices. UNIX est une
marque déposée de The Open Group.
Ce logiciel ou matériel et la documentation qui l'accompagne peuvent fournir des informations ou des liens donnant accès à des contenus, des produits et des services émanant de
tiers. Oracle Corporation et ses affiliés déclinent toute responsabilité ou garantie expresse quant aux contenus, produits ou services émanant de tiers, sauf mention contraire stipulée
dans un contrat entre vous et Oracle. En aucun cas, Oracle Corporation et ses affiliés ne sauraient être tenus pour responsables des pertes subies, des coûts occasionnés ou des
dommages causés par l'accès à des contenus, produits ou services tiers, ou à leur utilisation, sauf mention contraire stipulée dans un contrat entre vous et Oracle.
Accès aux services de support Oracle

Les clients Oracle qui ont souscrit un contrat de support ont accès au support électronique via My Oracle Support. Pour plus d'informations, visitez le site http://www.oracle.com/
pls/topic/lookup?ctx=acc&id=info ou le site http://www.oracle.com/pls/topic/lookup?ctx=acc&id=trs si vous êtes malentendant.
Contents

Using This Documentation ................................................................................. 9

Product Documentation Library ......................................................................... 9
Feedback ....................................................................................................... 9

Late-Breaking Information ................................................................................. 11

Preinstalled Software ..................................................................................... 11
IMPORTANT - Install Latest OS Updates, Patches, and Firmware .......................... 12
Minimum Supported Versions of the Firmware, OS, and Software .......................... 13
Mandatory Oracle Solaris 11 OS Package Updates .............................................. 14
Mandatory Oracle Solaris 10 OS Patches ........................................................... 15
Oracle Solaris 10 1/13 Mandatory Patches ................................................. 15
Oracle Solaris 10 8/11 Mandatory Patches ................................................. 16
Oracle Solaris 10 9/10 Mandatory Patches ................................................. 16
▼ Obtain Oracle Solaris 10 Patches ........................................................ 17
Updating the System Firmware from a Version Earlier than 9.5.2.g ......................... 18
▼ Check the Current Firmware Version ................................................... 19
▼ Power Off the Server ........................................................................ 20
▼ Record SP Network Properties ........................................................... 21
▼ Back Up the Current SP Configuration ................................................. 22
▼ Load the Initial System Firmware Image .............................................. 22
▼ Restore the SP Network Properties ...................................................... 26
▼ Load the Firmware Image a Second Time ............................................. 27
▼ Restore the SP Configuration ............................................................. 30
▼ Load the Oracle Solaris Fallback Image ............................................... 31
▼ Load the Final System Firmware Image ............................................... 32
Updating SP Firmware from the HOST ............................................................. 34
Required Firmware Updates for X-Options ........................................................ 35
Determine Correct Firmware for IO Devices ...................................................... 35

5
Contents

Installing and Booting Oracle Solaris 11 From Devices Connected to a USB Port ....... 35
PCIe Slot Configuration Requirements for Aura7 PCIe Cards ................................ 36
Oracle VM Server for SPARC Guidelines .......................................................... 36
Oracle Software in Silicon Features .................................................................. 37
Oracle Solaris Fallback Miniroot Image Must Be Installed on a New SPM ............... 37
Connecting a Terminal Device to the SER MGT Port ........................................... 38
Known Issues ............................................................................................... 38
Some SPARC T7-2 Memory Risers Include Flat-Head Screws ....................... 39
When You Create Logical Domains, Some Memory Is Reserved By the
Server .................................................................................................. 40
javac Fails During Builds on T7 Platforms (19503356) ................................ 40
Processes Hang in nanosleep() in an Oracle Solaris 10 Guest Domain
(19690481) ........................................................................................... 40
Oracle VTS System Exerciser Disk Test Fails On eUSB Device Due to
Delayed taskq (18154963) ..................................................................... 41
Excessive Messages Generated for a Power Supply Fan Fault (19951780) ........ 41
Servers With Missing Miniroot Indicate Cooling in the List of Affected
Subsystems (20922954) .......................................................................... 42
Oracle Solaris Should Process os-root-device Only Upon Boot
(21077998) ........................................................................................... 43
Logical Domains Virtual Disk Access Failed - Oracle Solaris Cluster Not
Coming Up Following Single-Node Reboot (21421237) ............................... 45
Fault Repair From the SPM Does Not Work In a Multi-Hop Between a Control
Domain and the Root Domain (21459393) ................................................. 46
sun4v_pcbe_enable Warning Appears When Starting a Guest Domain Running
Oracle Solaris 10 (21466955) .................................................................. 46
ixgbevf Driver Does Not Properly Report a Link State Change to the MAC
Layer (21629053) .................................................................................. 47
Flash Logging Is Broken in Hypervisor (21646012) ..................................... 48
Solaris OS Fails to Drop to OpenBoot Upon Retirement of SCC Cores and
L2DS (21644300, 21772653) .................................................................. 48
Sun Dual 10GbE SFP+ PCIe Low Profile Adapter Card Faulted With
Virtual_TTE_invalid Error (21694361, 21848425) .................................... 51
The Solaris OS Sometimes Fails to Recover Following a QRAP Event
(22022572) ........................................................................................... 53
Virtual_TTE_invalid Error On Assigned IOV Device (22138210) ................... 55
Saved LDOMs Configuration Fails To Boot After Upgrading the Server With
an Additional Processor Module (22012359, 22161099) ............................... 56
T7-x Observes Dropped Characters Running SysFW 9.7.4 (25506535) ............ 57

6 SPARC T7 Series Servers Product Notes • July 2019

 Contents

Do Not Use Certain Hynix 32GB DIMMs on T7 Servers (23284277,

23284255, 23222472) ............................................................................. 58
Net Install of 11.3SRU21b4 and b5 Panics With NVME Devices Installed
(26241738) ........................................................................................... 58

7
8 SPARC T7 Series Servers Product Notes • July 2019
 Using This Documentation

■ Overview – Contains late-breaking information for the server.

■ Audience – Technicians, system administrators, and authorized service providers.
■ Required knowledge – Experience with the Oracle Solaris operating system,
troubleshooting, and replacing hardware.

Product Documentation Library

Documentation and resources for this product and related products are available at http://
www.oracle.com/goto/t7-1/docs, http://www.oracle.com/goto/t7-2/docs, http://www.
oracle.com/goto/t7-4/docs.

Feedback
Provide feedback about this documentation at http://www.oracle.com/goto/docfeedback.

Using This Documentation 9

10 SPARC T7 Series Servers Product Notes • July 2019
 Late-Breaking Information

These topics provide important information and late-breaking news about the server:
■ “Preinstalled Software” on page 11
■ “IMPORTANT - Install Latest OS Updates, Patches, and Firmware” on page 12
■ “Minimum Supported Versions of the Firmware, OS, and Software” on page 13
■ “Mandatory Oracle Solaris 11 OS Package Updates” on page 14
■ “Mandatory Oracle Solaris 10 OS Patches” on page 15
■ “Updating the System Firmware from a Version Earlier than 9.5.2.g” on page 18
■ “Updating SP Firmware from the HOST” on page 34
■ “Required Firmware Updates for X-Options” on page 35
■ “Determine Correct Firmware for IO Devices” on page 35
■ “Installing and Booting Oracle Solaris 11 From Devices Connected to a USB
Port” on page 35
■ “PCIe Slot Configuration Requirements for Aura7 PCIe Cards” on page 36
■ “Oracle VM Server for SPARC Guidelines” on page 36
■ “Oracle Software in Silicon Features” on page 37
■ “Oracle Solaris Fallback Miniroot Image Must Be Installed on a New SPM” on page 37
■ “Connecting a Terminal Device to the SER MGT Port” on page 38
■ “Known Issues” on page 38

Preinstalled Software

Software Location Description

Oracle Solaris 11.3 OS The OS is installed on drive 0, using a Host OS.
ZFS file system.
Oracle VM Server for SPARC† /opt/SUNWldm Manages logical domains.
Oracle VTS† /usr/sunvts Provides hardware validation tests.
†
These software components are part of the Oracle Solaris 11 OS distribution.

Late-Breaking Information 11
IMPORTANT - Install Latest OS Updates, Patches, and Firmware

The preinstalled OS is ready to be configured at the appropriate point when you first apply
power to the server.

Mandatory package updates might not be preinstalled. Ensure that you obtain and install all
mandatory updates before you put the server into production. See “Mandatory Oracle Solaris 11
OS Package Updates” on page 14.

Refer to the Oracle Solaris documentation for instructions on installing and configuring the
Oracle Solaris OS.

You can reinstall the OS along with mandatory package updates or patches instead of
using the preinstalled OS. See “Minimum Supported Versions of the Firmware, OS, and
Software” on page 13.

IMPORTANT - Install Latest OS Updates, Patches, and

Firmware
Some product features are enabled only when the latest versions of patches or firmware are
installed. In order to retain optimal performance, security, and stability, installing the latest
available patches or firmware is required.

Verify that the server firmware version is at a minimum 9.4.3 or higher.

1. Check the server firmware:

From the ILOM web interface, click System Information → Summary, then view the
property value for the System Firmware Version in the General Information table.
From the command prompt, type:

-> show /HOST

2. Ensure the server firmware version is at the minimum required version, shown above, or a
subsequent release, if available.
3. If required, download the latest available software release version from My Oracle Support
at:
https://support.oracle.com
4. If required, update the server firmware.
Refer to the information about performing firmware updates in the Oracle ILOM
Administrator's Guide for Configuration and Maintenance. Ensure that you perform the
preparatory steps described in that document before updating the firmware.

12 SPARC T7 Series Servers Product Notes • July 2019

 Minimum Supported Versions of the Firmware, OS, and Software

Minimum Supported Versions of the Firmware, OS, and

Software
You are required to install the latest available and supported versions of the system firmware,
OS, and patches, for optimal performance, security, and stability. See “IMPORTANT - Install
Latest OS Updates, Patches, and Firmware” on page 12.

Oracle Solaris 11 is the recommended OS for the SPARC T7 series servers. Oracle Solaris
11 provides simplified installation and maintenance, enhanced virtualization capabilities, and
performance improvements. A more detailed list of Oracle Solaris 11 advantages is available at
http://www.oracle.com/technetwork/server-storage/solaris11/overview/index.html.

Note - If you configure the server with Oracle VM Server for SPARC, you can install various
combinations of the minimum (or later) versions of the OS. For example, use Oracle Solaris
11.3 for the control domain, and Oracle Solaris 10 9/10 in a guest domain.

Software Minimum Supported Versions

Oracle System Firmware 9.4.3.c or later.

9.5.4.c or later to support 64 GByte DIMMs.

9.7.1.c or later to support Oracle 3.2 TB NVMe SSDs.

(Sun System Firmware 9.7.1.c includes Oracle ILOM 3.2.6.)

9.8.3 or later to support Aura 7 PCIe cards, with the Hardware_Programmables-1.0.14-SPARC_T7-1+T7-

2+T7-4+S7-2+S7-2L.pkg file also installed.
Note - The server might require more recent firmware to support Oracle Solaris Kernel Zones. For specific
firmware requirements, refer to Creating and Using Oracle Solaris Kernel Zones.
Oracle Solaris 11 OS Oracle Solaris 11.3 for the control domain, guest domains, and non-virtualized configurations.

Includes these software components:

■ Oracle VM Server for SPARC

■ Oracle VTS

Oracle Solaris 11.3 SRU 10.7 or later to support Oracle 3.2 TB NVMe SSDs.

(Oracle Solaris 11.3 SRU 10.7 includes Oracle VM Server for SPARC 3.4.)

Also see “Mandatory Oracle Solaris 11 OS Package Updates” on page 14.

Oracle Solaris 10 OS Oracle Solaris 10 9/10 or Oracle Solaris 10 8/11, plus Oracle Solaris 10 1/13 SPARC Bundle and patches.

For virtual guest domains only:

Includes Oracle VTS 7 PS15.

Also see “Mandatory Oracle Solaris 10 OS Patches” on page 15.

Late-Breaking Information 13
Mandatory Oracle Solaris 11 OS Package Updates

Software Minimum Supported Versions

Java Development Kit
Java applications running on this product require one of these minimum JDK versions:

■ JDK 11 or higher
■ JDK 1.8.0_60 b27 or higher
■ JDK 1.7.0_85 b33 or higher

For production use, Oracle recommends using the latest available JDK and JRE versions, and does not
recommend using JDK versions that are no longer updated and are in sustaining support. For details, refer to
https://www.oracle.com/technetwork/java/javase/overview/index.html

For information about running older versions of the JDK on SPARC servers, refer to Doc ID 2564392.1 on
My Oracle Support.
Oracle Database Enterprise 12.1.0.2, plus required bundle patch (BP13).
Edition
Required on Oracle Solaris 11.3 for in-memory features.

Note - Some PCIe cards and devices have other minimum requirements. For details, including
what is required for a device to be bootable, refer to the product notes and other documentation
for that device.

Mandatory Oracle Solaris 11 OS Package Updates

No package update is required at this time to use the preinstalled Oracle Solaris 11.3 OS with
this server.

If you reinstall the OS, you might need to install certain package updates before you put the
server and optional hardware or software components into production.

Install the most recent Oracle Solaris 11.3 Support Repository Update (SRU). Taking this action
ensures that your server has the latest software for the best performance, security, and stability.

Use the pkg info entire command to display which SRU is currently installed on your server.

Use the pkg command or the package manager GUI to download any available SRUs from:
https://pkg.oracle.com/solaris/support.

Note - To access the Oracle Solaris 11 package update repository, you must have an Oracle
support agreement that enables you to install a required SSL certificate and support key.

See Installing the OS in your servers Installation Guide.

14 SPARC T7 Series Servers Product Notes • July 2019

 Mandatory Oracle Solaris 10 OS Patches

Mandatory Oracle Solaris 10 OS Patches

If you choose to install the Oracle Solaris 10 OS in a guest domain, you must also install
additional patches and in some cases, a patch bundle.

Oracle Solaris 10 1/13 Mandatory Patches

A supported version of the OS is only supported in guest domains.

Installation Order OS and Patches

1 Oracle Solaris 10 1/13.
2 These mandatory patches:

■ All patches up to October 2015 or later.

■ KU150400-29.

To install Oracle Solaris 10 1/13, you must use an image

with the miniroot patched with KU150400-29, or higher.

The correct image is provided in MOS patch 26032848.

This image contains the original Oracle Solaris 10 1/13
software, and a miniroot image which includes kernel
patch 150400-48. Downloading and using this image
enables the installation of Oracle Solaris 10 1/13 on guest
domains via jumpstart without the need to manually
patch the packed miniroot.

(optional) If, for some reason, you do not want to use

the image provided in the MOS patch, you can use the
prior method to manually patch the packed miniroot.
You must also use this method to patch versions of
Oracle Solaris older than Oracle Solaris 10 1/13. Refer
to MOS article 1501320.1 for instructions on how to
patch a packed miniroot at My Oracle Support (https://
myoraclesupport.com).

You can use the Oracle Solaris 10 JumpStart technology

and finish scripts to patch the installed image.

If you are unfamiliar with the Oracle Solaris

10 JumpStart technology, contact your support
representative or Oracle Support for documentation that
describes how to configure an Oracle Solaris 11 zone as
an Oracle Solaris 10 1/13 JumpStart server.

Late-Breaking Information 15
Mandatory Oracle Solaris 10 OS Patches

Oracle Solaris 10 8/11 Mandatory Patches

This version of the OS is supported only in guest domains.

Installation Order OS and Patches

1 Oracle Solaris 10 8/11.
Note - Until the Oracle Solaris 10 1/13 SPARC Bundle
has been installed, Oracle Solaris 10 8/11 supports only
512 CPUs and 3840 Gbytes of memory. For servers
exceeding those resources, reduce the resources until the
Oracle Solaris 10 1/13 SPARC Bundle has been installed,
or install a later version of Oracle Solaris.
2 Oracle Solaris 10 1/13 SPARC Bundle.

This bundle causes the OS to operate as the Oracle

Solaris 10 1/13 OS, but the version number in the /etc/
release file remains as Oracle Solaris 10 8/11.
3 These mandatory patches:

■ All patches up to October 2015.

■ KU150400-29.

To install Oracle Solaris 10, you must patch the miniroot

of the installation media with KU150400-29, or higher.
You can use the Oracle Solaris 10 JumpStart technology
and finish scripts to patch the installed image. Refer
to MOS article 1501320.1 for instructions on how to
patch a packed miniroot at My Oracle Support (https://
myoraclesupport.com).

If you are unfamiliar with the Oracle Solaris

10 JumpStart technology, contact your support
representative or Oracle Support for documentation that
describes how to configure an Oracle Solaris 11 zone as
an Oracle Solaris 10 1/13 JumpStart server.

Note - Until the Oracle Solaris 10 1/13 SPARC Bundle is installed, you might encounter these
bugs—15712380, 15704520, 15665037. The first two bugs are resolved when the Oracle
Solaris 10 1/13 SPARC bundle is installed. These bugs do not impact the patch installations.

Oracle Solaris 10 9/10 Mandatory Patches

This version of the OS is supported only in guest domains.

16 SPARC T7 Series Servers Product Notes • July 2019

 Obtain Oracle Solaris 10 Patches

Installation Order OS, Patches, and Packages

1 Oracle Solaris 10 9/10
Note - Until the Oracle Solaris 10 1/13 SPARC Bundle
has been installed, Oracle Solaris 10 9/10 only supports
512 CPUs and 1023 GBytes of memory. For servers
exceeding those resources, reduce the resources until the
Oracle Solaris 10 1/13 SPARC Bundle has been installed,
or install a later version of Oracle Solaris.
2 Oracle Solaris 10 1/13 SPARC Bundle.

This bundle causes the OS to operate as the Oracle

Solaris 10 1/13 OS, but the version number in the /etc/
release file remains as Oracle Solaris 10 9/10.
3 These mandatory patches:

■ All patches up to October 2015.

■ KU150400-29.

To install Oracle Solaris 10, you must patch the miniroot

of the installation media with KU150400-29, or higher.
You can use the Oracle Solaris 10 JumpStart technology
and finish scripts to patch the installed image. Refer
to MOS article 1501320.1 for instructions on how to
patch a packed miniroot at My Oracle Support (https://
myoraclesupport.com).

If you are unfamiliar with the Oracle Solaris

10 JumpStart technology, contact your support
representative or Oracle Support for documentation that
describes how to configure an Oracle Solaris 11 zone as
an Oracle Solaris 10 1/13 JumpStart server.
4 Use the pkgadd command to install the SUNWust1 and
SUNWust2 packages.

Note - Until the Oracle Solaris 10 1/13 SPARC Bundle is installed, you might encounter these
bugs—15712380, 15704520, 15665037. The first two bugs are resolved when the Oracle
Solaris 10 1/13 SPARC Bundle is installed. These bugs do not impact the patch installations.

Obtain Oracle Solaris 10 Patches

1. Sign in to My Oracle Support:
https://support.oracle.com

2. Select the Patches & Updates tab.

Late-Breaking Information 17
Updating the System Firmware from a Version Earlier than 9.5.2.g

3. Search for a patch using the Patch Search panel.

When searching for a patch using the Patch Name or Number field, you must specify the
complete name or number of the patch. For example:
■ Solaris 10 1/13 SPARC Bundle
■ 13058415
■ 147159-03
To search using a patch number without the revision number (last two digits), type % in place of
the revision number. For example:
147159-%

4. After you locate the patch, access the README and download the patch from
the site.
The patch README provides the patch installation instructions.

Updating the System Firmware from a Version Earlier than

9.5.2.g
Upgrading the SPARC T7 server from a system firmware version earlier than 9.5.2.g requires a
different procedure than the usual system firmware upgrade process, because the NAND flash
memory on the SP is reformatted during the update. You must perform additional steps in order
to retain configuration information.

If you need to use this procedure, read it thoroughly before attempting to proceed. Failure to
accurately follow these steps can result in significant additional downtime before the server is
restored to a fully functional state. Allow at least 40 minutes to perform this upgrade procedure
on a SPARC T7 server.

Briefly, the procedure is as follows:

1. Determine which firmware version is currently on the server.
See “Check the Current Firmware Version” on page 19.
2. Power off the server.
See “Power Off the Server” on page 20
3. Record your SP network settings.
See “Record SP Network Properties” on page 21.
4. Back up the current SP configuration.
See “Back Up the Current SP Configuration” on page 22

18 SPARC T7 Series Servers Product Notes • July 2019

 Check the Current Firmware Version

5. Load the initial firmware image.

See “Load the Initial System Firmware Image” on page 22.
6. Restore the SP network connection.
See “Restore the SP Network Properties” on page 26.
7. Load the system firmware a second time.
See “Load the Firmware Image a Second Time” on page 27.
8. Restore the previously backed up SP configuration.
See “Restore the SP Configuration” on page 30.
9. Load the Oracle Solaris fallback image.
See “Load the Oracle Solaris Fallback Image” on page 31.
10. Load the final system firmware image.
See “Load the Final System Firmware Image” on page 32.

The examples throughout this procedure show use of the CLI on a new Oracle SPARC T7-4
server to upgrade from system firmware 9.5.2.c to 9.5.2.g. These examples show a syntax that
you can use on any Oracle SPARC T7 series server, but customer-specific values will vary.
Variables are in italics and should not be entered directly.

Unless otherwise noted, you can perform the steps in this procedure using any of these methods:
■ The server's local serial management connection
■ The web browser-based network management connection
■ The command-line SSH network management connection
■ Oracle Enterprise Manager OpsCenter

Refer to the Oracle ILOM documentation for additional details about performing operations
using the ILOM CLI or web interface.

Refer to the Oracle Enterprise Manager OpsCenter documentation for details on performing
steps with that program.

Check the Current Firmware Version

The upgrade to system firmware 9.5.2.g or later only requires additional steps if you are
upgrading from system firmware 9.5.2.c or earlier. Check which version of the firmware is
loaded on the server before continuing.

Note - You must have full user role privileges (i.e. aucro).

Late-Breaking Information 19
Power Off the Server

1. Log in to the SP. Type:

ORACLESP-<XXXXXXXX> login: root

Password:
Detecting screen size; please wait...done

Oracle(R) Integrated Lights Out Manager

Version X.X.X.X.x rXXXXXX

Copyright (c) 2017, Oracle and/or its affiliates. All rights reserved.

Hostname: ORACLESP-<XXXXXXXX>

->

2. Display the current system firmware version. Type:

-> show /HOST sysfw_version

/HOST
Properties:
sysfw_version = Sun System Firmware 9.5.2.c YYYY/MM/DD HH:MM

->

3. Determine your next steps:

■ If your server firmware version is 9.5.2.g or later, load the latest firmware
image.
See “Load the Final System Firmware Image” on page 32.

■ If your firmware version is earlier than 9.5.2.g, power off the server.
See “Power Off the Server” on page 20.

Power Off the Server

1. Power off the server. Type:

-> stop /System

Are you sure you want to stop /System (y/n)? y
Stopping /System

20 SPARC T7 Series Servers Product Notes • July 2019

 Record SP Network Properties

->

2. Verify that the server is powered off. Type:

-> show /System power_state

/System
Properties:
power_state = Off

->

3. Record your SP network properties.

See “Record SP Network Properties” on page 21.

Record SP Network Properties

During this firmware upgrade process, the SP network configuration information will be reset to
factory defaults. You must note your current SP network properties before continuing with the
firmware upgrade.

You will need this information later to restore SP network connection.

1. Display the current SP network configuration information. Type:

-> show /SP/network ipaddress ipgateway ipnetmask ipdiscovery

/SP/network
Properties:
ipaddress = XX.XX.XX.XX
ipgateway = YY.YY.YY.YY
ipnetmask = ZZ.ZZ.ZZ.ZZ
ipdiscovery = static

->

2. If your SP uses a static connection (i.e., if the ipdiscovery value is static), note
the current network configuration properties.

Note - If your SP uses a DHCP connection, the network configuration will be restored
automatically.

Late-Breaking Information 21
Back Up the Current SP Configuration

3. Back up the current SP configuration.

See “Back Up the Current SP Configuration” on page 22.

Back Up the Current SP Configuration

Note - You cannot use Oracle Enterprise Ops Center for this portion of the upgrade procedure.

1. Set a 16-character passphrase for encryption of the SP configuration

information. Type:
-> set /SP/config passphrase=<XXXXXXXXXXXXXXXX>
Set 'passphrase' to 'XXXXXXXXXXXXXXXX'

->

Note the passphrase for later use.

2. Create the backup of the SP configuration information. Type:

-> set /SP/config dump_uri=scp://<username>@<ip_address_or_hostname>/backup_XXXXXXXX.xml

Enter remote user password: *********

Dump successful.

->

The backup file might take several minutes to write, depending on the amount of platform-
specific data in your configuration.
Note the name and location of the SP backup file.

Note - If you are upgrading the system firmware on more than one server, Oracle recommends
using a different file name for each SP backup.

3. Load the initial firmware image.

See “Load the Initial System Firmware Image” on page 22.

Load the Initial System Firmware Image

Note - Loading the system firmware image might take up to 15 minutes.

22 SPARC T7 Series Servers Product Notes • July 2019

 Load the Initial System Firmware Image

1. Load the initial system firmware image. Type:

-> load -source http://<ip_address_or_hostname>/Sun_System_Firmware-9_5_2_g-

SPARC_T7-4.pkg /SP/firmware

NOTE: An upgrade takes several minutes to complete. ILOM

will enter a special mode to load new firmware. No
other tasks can be performed in ILOM until the
firmware upgrade is complete and ILOM is reset.

Are you sure you want to load the specified file (y/n)? y
Preserve existing configuration (y/n)? y
.............................................................................................................

Firmware update is complete.

ILOM will now be restarted with the new firmware.

After loading the initial image of the new system firmware, the SP automatically reboots from
primary flash. During reboot, the SP will use factory defaults even if you answered y to the
question, Preserve existing configuration (y/n)?

-> /sbin/reboot
Stopping coredump capture...done.
Watchdogd: Stopping patting ... Done
Network Interface Plugging Daemon...stop eth0...done.
Stopping kernel log daemon....
...
Mounting root filesystem read-only...done.
Will now restart.
Restarting system.
+****
Primary Bootstrap.
SYSRST = 0x1
Reading RO_Hw_Straps
Searching for packages
Good uboot CRC for data=[*000c0000, len=553b4] in pkg *0013f800 (crc: 0x44f46b65)
Good uboot CRC for data=[*000c0000, len=80000] in pkg *00140000 (crc: 0xb0b8c9ac)
Good kernel CRC for data=[*00142000, len=1e0de8] in pkg *00140000 (crc:
0xbf324077)
Good root CRC for data=[*00340000, len=ef1000] in pkg *00140000 (crc: 0x5d3b8e31)
Good vbscdir CRC for data=[*01740000, len=4c000] in pkg *00140000 (crc:
0x7a62c323)
Good uboot CRC for data=[*018c0000, len=553b4] in pkg *0193f800 (crc: 0x2f78db1d)
Good uboot CRC for data=[*018c0000, len=80000] in pkg *01940000 (crc: 0x97d92980)
Good kernel CRC for data=[*01942000, len=1df2e0] in pkg *01940000 (crc:
0xf7697623)
Good root CRC for data=[*01b40000, len=ef1000] in pkg *01940000 (crc: 0xb19fd0e5)

Late-Breaking Information 23
Load the Initial System Firmware Image

Good vbscdir CRC for data=[*02f40000, len=4c000] in pkg *01940000 (crc:

0xda3cef04)
Starting U-Boot at 0x000c0000\ufffd

U-Boot 2010.03

Custom Pilot3 U-Boot 0.1 (Dec 4 2015 - 11:41:49) r105871

VGA buffer reserved 0x800000 bytes at 0x80000000

sptrace allocated 0x800000 bytes at 0x9F800000
DRAM: 496 MB
Flash: 48 MB
In: serial
Out: serial
Err: serial

ARM restart caused by: reboot

hardware interface reset by: none
Galactic1: 0x4000c900, Galactic2: 0x0
The host is OFF(S5) (hostWantsPwr=0, powerGood=0,
allowPwrOn=0|0, outOfReset=0, fatalError=0).

...

The process continues. The section in italics in this example indicates output that is different
from typical firmware upgrade. In this case, this output is normal during a firmware upgrade
from a version earlier than 9.5.2.g to a firmware version 9.5.2.g or later.

Loading kernel module video.

Loading kernel module Timer.
Mounted vbsc
Erasing the nand ...
Erasing 512 Kibyte @ 40000000 -- 100 % complete.
no UBI volumes found, initializing media:
erasing media..
formatting media..
UBI device number 0, total 2048 LEBs (1056964608 bytes, 1008.0 MiB), available 2024 LEBs
(1044578304 bytes, 996.2 MiB), LEB size 516096 bytes (504.0 KiB)
UBI volumes (0):
params: (20.2 MB) created mounted
persist: (106.3 MB) created mounted
coredump: (70.4 MB) created mounted
large: (64.5 MB) created mounted
extra: (36.4 MB) created mounted
package: (230.3 MB) created mounted
miniroot: (468.1 MB) created mounted
Available freespace: 0 MB

24 SPARC T7 Series Servers Product Notes • July 2019

 Load the Initial System Firmware Image

Checking configuration files state ...

... not preserving config
... Restoring configuration files ...
params: (20.2 MB) exists was_mounted
persist: (106.3 MB) exists was_mounted
coredump: (70.4 MB) exists was_mounted
large: (64.5 MB) exists was_mounted
extra: (36.4 MB) exists was_mounted
package: (230.3 MB) exists was_mounted
miniroot: (468.1 MB) exists was_mounted
Available freespace: 0 MB
Saving TLI data...FAILED
Removing old configuration files
Removing platform specific files
Restoring default configuration files...Done
Restoring TLI data...FAILED
Configuration files state good

1970-01-01 00:02:42 ** Error: Did not find any saved ilom package file
sp_trace_write failed: -1
Setting the system clock.
System Clock set to: Mon Dec 7 18:51:52 UTC 2015.
Initializing SP Trace Buffers
Setting up networking....
Will now mount local filesystems:.
Will now activate swapfile swap:done.
Cleaning up temporary files...Cleaning /var/run...done.
Cleaning /var/lock...done.
...
Starting ILOM Watchdog daemon.
Done
Running plat...Preparsing sensor.xml... ( took 12 seconds ) done
Probing frus ...done
Done running plat
psnc_file_read: cannot access file '/persist/psnc_backup1.xml'
Identifying Product Data...Backup2
Setting poweron delay...
Starting Dynamic FRUID Daemon early init: dynafrud_early_init Done (0)
INIT: Entering runlevel: 3
Preparsing sensor.xml... ( took 1 seconds ) done
Starting Event Manager: eventmgr . Done
...
waiting for GM ready ....GM is ready
Starting Sppostadm: -a Done

ORACLESP-<XXXXXXXX> login:

2. Restore the SP network properties.

Late-Breaking Information 25
Restore the SP Network Properties

See “Restore the SP Network Properties” on page 26.

Restore the SP Network Properties

SP network connectivity is required in order to continue the firmware upgrade process. If your
SP is configured with a static network connection (i.e., the ipdiscovery parameter is set to
static), you must type in the network configuration properties before continuing.

Note - This part of the upgrade can only be performed using a console connection to the serial
management port.

Note - If your SP network configuration is set to use DHCP, continue to “Restore the SP
Configuration” on page 30.

1. Log in to the service processor:

Note - You must have full user role privileges (i.e. aucro) for this step.

ORACLESP-<XXXXXXXX> login: root

Password:
Detecting screen size; please wait...done

Oracle(R) Integrated Lights Out Manager

Version X.X.X.X.x rXXXXXX

Copyright (c) 2015, Oracle and/or its affiliates. All rights reserved.

Warning: The system appears to be in manufacturing test mode.

Contact Service immediately.

Warning: password is set to factory default.

Warning: HTTPS certificate is set to factory default.

Warning: Product identification data missing. System may not function properly.
Service must update product identification data. Contact Service immediately.

Unrecognized Chassis: This module is installed in an unknown or unsupported

chassis. You must upgrade the firmware to a newer version that supports
this chassis.

26 SPARC T7 Series Servers Product Notes • July 2019

 Load the Firmware Image a Second Time

Hostname: ORACLESP-<XXXXXXXX>

-> cd /SP/network
/SP/network

Unrecognized Chassis: This module is installed in an unknown or unsupported

chassis. You must upgrade the firmware to a newer version that supports
this chassis.

->

2. Configure the SP network properties. Type:

-> set /SP/network pendingipaddress=<XX.XX.XX.XX> pendingipgateway=<YY.YY.YY.YY>
pendingipnetmask=<ZZ.ZZ.ZZ.ZZ> commitpending=true

Set 'pendingipaddress' to 'XX.XX.XX.XX'

Set 'pendingipgateway' to 'YY.YY.YY.YY'
Set 'pendingipnetmask' to 'ZZ.ZZ.ZZ.ZZ'
Set 'commitpending' to 'true'

->

3. Load the second system firmware image.

See “Load the Firmware Image a Second Time” on page 27.

Load the Firmware Image a Second Time

After restoring the Oracle Solaris miniroot, you must load the new system firmware image
a second time. This updates the secondary flash in the SP. The SP then reboots, makes the
secondary flash the primary flash, and loads the new firmware image from the primary flash.
This process takes about six minutes.

1. Load the second image of system firmware 9.5.2.g or later. Type:

-> load -source http://<ip_address_or_hostname>/Sun_System_Firmware-9_5_2_g-
SPARC_T7-4.pkg /SP/firmware

NOTE: An upgrade takes several minutes to complete. ILOM

will enter a special mode to load new firmware. No
other tasks can be performed in ILOM until the
firmware upgrade is complete and ILOM is reset.

Late-Breaking Information 27
Load the Firmware Image a Second Time

Are you sure you want to load the specified file (y/n)? y
Preserve existing configuration (y/n)? y
..........................................................................
..........................................................................
..........................................................................
..........................................................................

Firmware update is complete.

ILOM will now be restarted with the new firmware.

-> /sbin/reboot
Stopping coredump capture...done.
Watchdogd: Stopping patting ... Done
Network Interface Plugging Daemon...stop eth0...done.
Stopping kernel log daemon....
...
Mounting root filesystem read-only...done.
Restarting system.
Will now restart+****
Primary Bootstrap.
SYSRST = 0x1
Reading RO_Hw_Straps
Searching for packages
Good uboot CRC for data=[*000c0000, len=553b4] in pkg *0013f800 (crc: 0x44f46b65)
Good uboot CRC for data=[*000c0000, len=80000] in pkg *00140000 (crc: 0xb0b8c9ac)
Good kernel CRC for data=[*00142000, len=1e0de8] in pkg *00140000 (crc:
0xbf324077)
Good root CRC for data=[*00340000, len=ef1000] in pkg *00140000 (crc: 0x5d3b8e31)
Good vbscdir CRC for data=[*01740000, len=4c000] in pkg *00140000 (crc:
0x7a62c323)
Good uboot CRC for data=[*018c0000, len=553b4] in pkg *0193f800 (crc: 0x44f46b65)
Good uboot CRC for data=[*018c0000, len=80000] in pkg *01940000 (crc: 0xb0b8c9ac)
Good kernel CRC for data=[*01942000, len=1e0de8] in pkg *01940000 (crc:
0xbf324077)
Good root CRC for data=[*01b40000, len=ef1000] in pkg *01940000 (crc: 0x5d3b8e31)
Good vbscdir CRC for data=[*02f40000, len=4c000] in pkg *01940000 (crc:
0x7a62c323)
Starting U-Boot at 0x000c0000\ufffd

U-Boot 2010.03

Custom Pilot3 U-Boot 0.1 (Dec 4 2015 - 11:41:49) r105871

VGA buffer reserved 0x800000 bytes at 0x80000000

sptrace allocated 0x800000 bytes at 0x9F800000
DRAM: 496 MB
Flash: 48 MB

28 SPARC T7 Series Servers Product Notes • July 2019

 Load the Firmware Image a Second Time

In: serial
Out: serial
Err: serial

ARM restart caused by: reboot

hardware interface reset by: none
Galactic1: 0x4000c900, Galactic2: 0x0
The host is OFF(S5) (hostWantsPwr=0, powerGood=0,
allowPwrOn=0|0, outOfReset=0, fatalError=0).

pilot3 chip revision : 0x5

SP Debug Jumper: Inserted
SP CLR_PASSD Jumper: Empty

...
Loading kernel module video.
Loading kernel module Timer.
Mounted vbsc
params: (20.2 MB) exists mounted
persist: (106.3 MB) exists mounted
coredump: (70.4 MB) exists mounted
large: (64.5 MB) exists mounted
extra: (36.4 MB) exists mounted
package: (230.3 MB) exists mounted
miniroot: (468.1 MB) exists mounted
Available freespace: 0 MB

Checking configuration files state ...

Image date: Fri Dec 4 12:18:44 PST 2015 Image revision: 105871
Conf date: Fri Dec 4 12:18:44 PST 2015 Conf revision: 105871
Configuration files state good after upgrade.
Setting the system clock.
System Clock set to: Mon Dec 7 19:31:08 UTC 2015.
...
Probing frus ...done
Done running plat
Identifying Product Data...Done
Setting poweron delay...
Starting Dynamic FRUID Daemon early init: dynafrud_early_init Done (0)
INIT: Entering runlevel: 3
Preparsing sensor.xml... ( took 1 seconds ) done
Starting Event Manager: eventmgr . Done
Starting ipmi log manager daemon: logmgr . Done
...
waiting for GM ready ...GM is ready
Starting Sppostadm: -a Done

ORACLESP-XXXXXXXX login:

Late-Breaking Information 29
Restore the SP Configuration

2. Restore the SP configuration.

See “Restore the SP Configuration” on page 30.

Restore the SP Configuration

Restoring the SP configuration you previously backed up might take three minutes or longer,
depending on the amount of platform-specific data.

Note - You cannot use Oracle Enterprise Ops Center for this portion of the upgrade procedure.

1. Log in to the SP. Type:

Note - You must have full user role privileges (i.e. aucro) to perform this task.

ORACLESP-<XXXXXXXX> login: root

Password:
Detecting screen size; please wait...done

Oracle(R) Integrated Lights Out Manager

Version X.X.X.X.x rXXXXXX

Copyright (c) 2017, Oracle and/or its affiliates. All rights reserved.

Hostname: ORACLESP-<XXXXXXXX>

->

2. Set the passphrase for restoring the SP configuration information.

Use the passphrase you noted earlier. Type:

-> set /SP/config passphrase=<XXXXXXXXXXXXXXXX>

Set 'passphrase' to 'XXXXXXXXXXXXXXXX'

->

3. Restore the backed up SP configuration. Type:

-> set /SP/config load_uri=scp://<username>@<ip_address_or_hostname>/backup_XXXXXXXX.xml

Enter remote user password: *********
Dump successful.
->

30 SPARC T7 Series Servers Product Notes • July 2019

 Load the Oracle Solaris Fallback Image

Note - If you are restoring the SP configuration from a local serial port connection, the screen
output includes the following:

set: Load partially successful, please view the event log

The /SP/logs/event/list log includes the following line:

Config restore: Unable to restore property '/SP/serial/external/commitpending'

(Can not change serial settings - the serial console is in use.)

4. Load the Oracle Solaris fallback image.

See “Load the Oracle Solaris Fallback Image” on page 31.

Load the Oracle Solaris Fallback Image

Now that you have loaded the initial image of system firmware 9.5.2.g or later and restored
the network and other configuration parameters to the SP, you must reload the Oracle Solaris
fallback image.

Note - Loading the Oracle Solaris fallback image takes about three minutes.

1. Load the Oracle Solaris miniroot package. Type:

-> load -source http://<ip_address_or_hostname>/sol-11_2_10_5_0-fallback_boot-sparc.pkg /SP/

firmware/host/miniroot

Are you sure you want to load the specified file (y/n)? y
This installs a new copy of miniroot. Do you want to continue (y/n)? y

Firmware update is complete.

->

2. Load the final system firmware image.

See “Load the Final System Firmware Image” on page 32.

Late-Breaking Information 31
Load the Final System Firmware Image

Load the Final System Firmware Image

This final firmware image load takes about six minutes.

1. Load the final system firmware image. Type:

-> load -source http://<ip_address_or_hostname>/Sun_System_Firmware-9_5_2_g-

SPARC_T7-4.pkg /SP/firmware

NOTE: An upgrade takes several minutes to complete. ILOM

will enter a special mode to load new firmware. No
other tasks can be performed in ILOM until the
firmware upgrade is complete and ILOM is reset.

Are you sure you want to load the specified file (y/n)? y
Preserve existing configuration (y/n)? y
..........................................................................................................

Firmware update is complete.

ILOM will now be restarted with the new firmware.

-> /sbin/reboot
Stopping coredump capture...done.
Watchdogd: Stopping patting ... Done
Network Interface Plugging Daemon...stop eth0...done.
Stopping kernel log daemon....
...
Mounting root filesystem read-only...done.
Restarting system.
Will now restart+****
Primary Bootstrap.
SYSRST = 0x1
Reading RO_Hw_Straps
Searching for packages
Good uboot CRC for data=[*000c0000, len=553b4] in pkg *0013f800 (crc: 0x44f46b65)
Good uboot CRC for data=[*000c0000, len=80000] in pkg *00140000 (crc: 0xb0b8c9ac)
Good kernel CRC for data=[*00142000, len=1e0de8] in pkg *00140000 (crc:
0xbf324077)
Good root CRC for data=[*00340000, len=ef1000] in pkg *00140000 (crc: 0x5d3b8e31)
Good vbscdir CRC for data=[*01740000, len=4c000] in pkg *00140000 (crc:
0x7a62c323)
Good uboot CRC for data=[*018c0000, len=553b4] in pkg *0193f800 (crc: 0x44f46b65)
Good uboot CRC for data=[*018c0000, len=80000] in pkg *01940000 (crc: 0xb0b8c9ac)
Good kernel CRC for data=[*01942000, len=1e0de8] in pkg *01940000 (crc:
0xbf324077)
Good root CRC for data=[*01b40000, len=ef1000] in pkg *01940000 (crc: 0x5d3b8e31)

32 SPARC T7 Series Servers Product Notes • July 2019

 Load the Final System Firmware Image

Good vbscdir CRC for data=[*02f40000, len=4c000] in pkg *01940000 (crc:

0x7a62c323)
Starting U-Boot at 0x000c0000\ufffd

U-Boot 2010.03

Custom Pilot3 U-Boot 0.1 (Dec 4 2015 - 11:41:49) r105871

VGA buffer reserved 0x800000 bytes at 0x80000000

sptrace allocated 0x800000 bytes at 0x9F800000
DRAM: 496 MB
Flash: 48 MB
In: serial
Out: serial
Err: serial

ARM restart caused by: reboot

hardware interface reset by: none
Galactic1: 0x4000c900, Galactic2: 0x0
The host is OFF(S5) (hostWantsPwr=0, powerGood=0,
allowPwrOn=0|0, outOfReset=0, fatalError=0).

pilot3 chip revision : 0x5

SP Debug Jumper: Inserted
SP CLR_PASSD Jumper: Empty

...
Loading kernel module video.
Loading kernel module Timer.
Mounted vbsc
params: (20.2 MB) exists mounted
persist: (106.3 MB) exists mounted
coredump: (70.4 MB) exists mounted
large: (64.5 MB) exists mounted
extra: (36.4 MB) exists mounted
package: (230.3 MB) exists mounted
miniroot: (468.1 MB) exists mounted
Available freespace: 0 MB

Checking configuration files state ...

Image date: Fri Dec 4 12:18:44 PST 2015 Image revision: 105871
Conf date: Fri Dec 4 12:18:44 PST 2015 Conf revision: 105871
Configuration files state good after upgrade.
Setting the system clock.
System Clock set to: Mon Dec 7 19:31:08 UTC 2015.
...
Probing frus ...done
Done running plat

Late-Breaking Information 33
Updating SP Firmware from the HOST

Identifying Product Data...Done

Setting poweron delay...
Starting Dynamic FRUID Daemon early init: dynafrud_early_init Done (0)
INIT: Entering runlevel: 3
Preparsing sensor.xml... ( took 1 seconds ) done
Starting Event Manager: eventmgr . Done
Starting ipmi log manager daemon: logmgr . Done
...
waiting for GM ready ...GM is ready
Starting Sppostadm: -a Done

ORACLESP-XXXXXXXX login:

The firmware update process is complete.

2. If the server does not power on automatically, power on the server. Type:

-> start /System

Are you sure you want to start /System (y/n)? y
Starting /System

->

Updating SP Firmware from the HOST

To update SP firmware from the HOST, Oracle is standardizing on the fwupdate utility. This
utility is included in Oracle Solaris as part of the Oracle Hardware Management Pack (OHMP).

Note - Beginning with SPARC T7 servers, the sysfwdownload utility does not ship with SP
firmware releases, and that utility is not supported.

For more information on using the fwpudate utility that is included in Oracle Solaris 11.3, refer
to "Update an Oracle ILOM Service Processor Using fwupdate" in the Oracle Server CLI Tools
for Oracle Solaris 11.3 User Guide (http://docs.oracle.com/cd/E64576_01/html/E64582/
gltkm.html#scrolltoc).

When you follow the steps in that procedure, and have obtained a desired firmware package,
this is the command that will update the SP from the HOST:

fwupdate update sp-bios-firmware -n sp_bios -f firmware-package-file.pkg

firmware-package-file.pkg is the path to the firmware package for your server's SP.

34 SPARC T7 Series Servers Product Notes • July 2019

 Required Firmware Updates for X-Options

Required Firmware Updates for X-Options

If you install the Oracle Flash Accelerator F160 PCIe Card or the Oracle 1.6TB NVMe SSD
as an x-option, you must update the firmware to RA13, or a subsequent release if available.
Servers ordered with this option installed already have the updated firmware.

For firmware upgrade instructions, refer to the NVMe card and SSD documentation at: http://
www.oracle.com/goto/oracleflashf160/docs.

Determine Correct Firmware for IO Devices

Some product features are enabled only when the latest versions of patches or firmware are
installed. In order to retain optimal performance, security, and stability, installing the latest
available patches or firmware is required. Confirm that the latest server firmware version is
installed.

To easily determine the latest released version of supported firmware for available IO devices,
locate the "IO Options Firmware 1.0" patch (patch number 25393974) on My Oracle Support
(MOS).

This patch can be found in a “Product or Family (Advanced)” search, under the “Patches &
Update” tab on MOS for any SPARC T7, S7 or M7 platform from Oracle. For example, a query
for SPARC S7-2L will include SPARC S7-2L IO Options Firmware 1.0.

The patch only contains a README, with a table of available IO devices and a pointer to the
latest supported firmware patch on MOS for each device.

Installing and Booting Oracle Solaris 11 From Devices

Connected to a USB Port
To install Oracle Solaris without using an IPS AutoInstall server on the network, you can use
Oracle Solaris media in a DVD drive. The DVD drive can be built into the server or attached to
a USB port. You also can boot from an ISO image copied to a DVD disk, hard drive, or SSD.

You can install Oracle Solaris 11.3 OS on this server from an image copied to a USB flash
drive. That USB image is available for download at the same location as the ISO images http:
//www.oracle.com/technetwork/server-storage/solaris11/downloads/index.html.

You also can create a persistent device alias for a device connected to a USB port. The simpler
alias name remains available for future installation or boot operations.

Late-Breaking Information 35
PCIe Slot Configuration Requirements for Aura7 PCIe Cards

For more information about installing Oracle Solaris and creating persistent device aliases, refer
to Installing Oracle Solaris 11.3 Systems at:

http://docs.oracle.com/cd/E53394_01.

You can boot Oracle Solaris 11.3 from drives installed in the server (hard drive, SSD, or DVD
drive) or from devices connected to a USB port.

For the path to identify a USB port in a boot command, refer to the list of devices output by the
show-dev OpenBoot command.

PCIe Slot Configuration Requirements for Aura7 PCIe Cards

The following PCIe slot configuration requirements exist for these servers when Aura7 PCIe
cards are installed:
■ In SPARC T7-4 servers that have Aura7 PCIe cards installed, you must install cards in slots
1/2/3/4/13/14/15/16 to avoid thermal concerns in the system.
■ In SPARC T7-1 servers that have Aura7 PCIe cards installed, you can populate no more
than four slots, which is two fewer than systems that use Aura6 PCIe cards.

Oracle VM Server for SPARC Guidelines

If you plan to configure Oracle VM Server for SPARC on this server, follow these guidelines:
■ Do not configure the Oracle Solaris 10 OS in the control domain, root domains, I/O
domains, or service domains.
■ You can configure Oracle Solaris 10 OS with the required patches and packages in guest
domains.
■ Be aware that instances of Oracle Solaris 10 with required patches and packages in a single
guest domain are limited to 1024 virtual CPUs and 4 Tbytes of memory.

The majority of a physical domain's memory can be assigned to logical domains. However, a
small portion of the PDomain's memory is preassigned to software components, the hypervisor,
and certain I/O devices. To determine which portions of the memory are unavailable for logical
domains, log in to the PDomain and type this command:
# ldm ls-devices -a mem

In the command output, look for the rows that include _sys_ in the BOUND column. These
portions of memory are not available for logical domains.

36 SPARC T7 Series Servers Product Notes • July 2019

 Oracle Software in Silicon Features

For more information about Oracle VM Server for SPARC, refer to the Oracle VM Server for
SPARC documentation.

Oracle Software in Silicon Features

The microprocessors for the SPARC T7 series servers offer co-engineered hardware and
software capabilities that enable applications to run with the highest levels of security,
reliability, and speed. This functionality is called as Oracle Software in Silicon.

Software in Silicon features include:

■ Silicon Secured Memory – detects common memory access errors including:

■ Buffer overflows
■ Unallocated or freed memory access errors
■ "Double free" memory access errors
■ Stale pointer memory access errors

With Silicon Secured Memory enabled, an error is likely to be raised if an application tries
to access memory to which it should not have access. This feature enhances application data
integrity (ADI). (The Silicon Secured Memory feature is not supported for use with Kernel
Zones at this time.)
■ Data Analytics Accelerator (DAX) – Coprocessors perform query-related operations
directly through the hardware, which improves Oracle Database performance. You can use
DAX hardware acceleration for Oracle Database 12c in-memory database operations. (The
DAX feature is not supported for use with Kernel Zones at this time.)

For more details on Silicon Secured Memory, refer to the Oracle Solaris 11.3 documentation.

To use DAX, you must configure the Oracle Database 12c in-memory feature. For instructions,
refer to "Using the In-Memory Column Store" at:

http://docs.oracle.com/database/121/ADMIN/memory.htm#ADMIN14257

Oracle Solaris Fallback Miniroot Image Must Be Installed on

a New SPM
When you replace the SPM in your server, you must install a fallback miniroot image that
corresponds to the version of Oracle Solaris that are using. The miniroot image is not part of

Late-Breaking Information 37
Connecting a Terminal Device to the SER MGT Port

the OS or firmware image that you install. Also, you must update the miniroot image when you
install a new version of Oracle Solaris on the server, so the fallback image matches that version
and SRU of Oracle Solaris.

You must download the fallback miniroot image for your hardware and your Oracle Solaris
version from My Oracle Support at https://support.oracle.com. Then follow the
procedure "Uploading a New Solaris Miniroot Package From SP to Host" in the Oracle ILOM
Administrator's Guide for Configuration and Maintenance.

You will see the defect.ilom.fs.miniroot-missing error if you fail to install a fallback
miniroot image. For example:

-> show faulty

Target | Property |Value

-------------------------+--------------------------
+---------------------------------------
/SP/faultmgmt/0 | fru | /SYS
/SP/faultmgmt/0/faults/0 | class | defect.ilom.fs.miniroot-
missing

Connecting a Terminal Device to the SER MGT Port

Crossover adapters are no longer provided in the shipping kit. To connect the server's
SER MGT port to a terminal device, use an RJ-45 cable that is set up for a null modem
configuration, in which the transmit and receive signals cross over.

Known Issues
These are the known issues for the SPARC T7 servers:

■ “Some SPARC T7-2 Memory Risers Include Flat-Head Screws” on page 39

■ “When You Create Logical Domains, Some Memory Is Reserved By the
Server” on page 40
■ “javac Fails During Builds on T7 Platforms (19503356)” on page 40
■ “Processes Hang in nanosleep() in an Oracle Solaris 10 Guest Domain
(19690481)” on page 40
■ “Oracle VTS System Exerciser Disk Test Fails On eUSB Device Due to Delayed taskq
(18154963)” on page 41

38 SPARC T7 Series Servers Product Notes • July 2019

 Known Issues

■ “Servers With Missing Miniroot Indicate Cooling in the List of Affected Subsystems
(20922954)” on page 42
■ “Oracle Solaris Should Process os-root-device Only Upon Boot
(21077998)” on page 43
■ “Logical Domains Virtual Disk Access Failed - Oracle Solaris Cluster Not Coming Up
Following Single-Node Reboot (21421237)” on page 45
■ “Fault Repair From the SPM Does Not Work In a Multi-Hop Between a Control Domain
and the Root Domain (21459393)” on page 46
■ “sun4v_pcbe_enable Warning Appears When Starting a Guest Domain Running Oracle
Solaris 10 (21466955)” on page 46
■ “ixgbevf Driver Does Not Properly Report a Link State Change to the MAC Layer
(21629053)” on page 47
■ “Flash Logging Is Broken in Hypervisor (21646012)” on page 48
■ “Solaris OS Fails to Drop to OpenBoot Upon Retirement of SCC Cores and L2DS
(21644300, 21772653)” on page 48
■ “Sun Dual 10GbE SFP+ PCIe Low Profile Adapter Card Faulted With
Virtual_TTE_invalid Error (21694361, 21848425)” on page 51
■ “The Solaris OS Sometimes Fails to Recover Following a QRAP Event
(22022572)” on page 53
■ “Virtual_TTE_invalid Error On Assigned IOV Device (22138210)” on page 55
■ “Saved LDOMs Configuration Fails To Boot After Upgrading the Server With an
Additional Processor Module (22012359, 22161099)” on page 56
■ “T7-x Observes Dropped Characters Running SysFW 9.7.4 (25506535)” on page 57
■ “Do Not Use Certain Hynix 32GB DIMMs on T7 Servers (23284277, 23284255,
23222472)” on page 58
■ “Net Install of 11.3SRU21b4 and b5 Panics With NVME Devices Installed
(26241738)” on page 58

Some SPARC T7-2 Memory Risers Include Flat-

Head Screws
Your SPARC T7-2 server could include a memory riser that is secured with a flat-head screw
instead of the standard hex-head screw. If that is the case, use a No. 1 flat-blade screwdriver to
service that memory riser.

Late-Breaking Information 39
Known Issues

When You Create Logical Domains, Some Memory

Is Reserved By the Server
When you use Oracle VM Server for SPARC, you can assign most of the memory in a SPARC
T7 server to logical domains. However, a small portion of the server's memory is preassigned to
software components, the hypervisor, and certain I/O devices. Also, if DIMM sparing is active,
a portion of memory is reserved to enable the server to operate normally after a DIMM has
failed.

To determine which portions of memory are unavailable for logical domains, type:

# ldm ls-devices -a mem

In the output, look for rows that include _sys_ in the Bound column. Each of those portions of
memory is not available for logical domains.

For more information about DIMM sparing, refer to “DIMM Sparing Overview” in SPARC T7
Series Servers Administration Guide.

javac Fails During Builds on T7 Platforms

(19503356)
If you are developing Java applications on SPARC T7 servers, use JDK version 7u72 or later.

Processes Hang in nanosleep() in an Oracle

Solaris 10 Guest Domain (19690481)
To install Oracle Solaris 10, you must patch the miniroot of the installation media with
KU150400-25, at minimum. You can use the Oracle Solaris 10 JumpStart technology and finish
scripts to patch the installed image. Refer to MOS article 1501320.1 for instructions on how to
patch a packed miniroot at https://support.oracle.com.

If you are unfamiliar with the Oracle Solaris 10 JumpStart technology, contact your support
representative or Oracle Support for documentation that describes how to configure an Oracle
Solaris 11 zone as an Oracle Solaris 10 1/13 JumpStart server.

40 SPARC T7 Series Servers Product Notes • July 2019

 Known Issues

Oracle VTS System Exerciser Disk Test Fails On

eUSB Device Due to Delayed taskq (18154963)
Under heavy load, one or more of the embedded USB devices used on SPARC T7 and SPARC
M7 servers to accommodate boot archives (not intended for direct customer use) might
experience disconnect and reconnect events, with messages similar to this one appearing on the
console or in syslog:
scsi: WARNING: /pci@309/pci@2/usb@0/storage@1/disk@0,0
(sd22): Command failed to complete...Device is gone
WARNING: /pci@309/pci@2/usb@0/storage@1 (scsa2usb13):
Reinserted device is accessible again.

If you are using Oracle VTS to validate the server during these disconnect/reconnect events,
the events might cause the VTS Disk test to fail, with an error logged in /var/sunvts/logs/
sunvts.err file such as the following:
SunVTS7.0ps19.2: VTSID 8009
Disk.diskmediatestmfg.FATAL rdsk/c7t0d0: Failed to open the device node,
Error Message : (No such device or address).
Suggestions :
(1) Verify if the device node exists in /dev/rdsk directory
(2) Run format(1M), rmformat(1)>
(3) Run '# devfsadm -C ' to clean-up dangling /dev links" SunVTS7.0ps19.2: VTSID 6427
vtsk.
ERROR : Disk.diskmediatest.0[c7t0d0] (pid=5575) exited with exit code: 1 during task
431404.

Recovery: The server recovers the failed device automatically.

Excessive Messages Generated for a Power

Supply Fan Fault (19951780)
When the fan on a power supply fails, fault messages are repeatedly generated and cleared until
you replace the power supply.

On the Oracle Solaris console, the repeated messages are similar to this:
Nov 4 09:58:41 system-name SC Alert: [ID 821027 daemon.alert] Fault
critical: Fault detected at time = Tue Nov 4 14:01:16 2014. The
suspect component: /SYS/PS0 has fault.chassis.device.psu.fail with
probability=100.

In the Oracle ILOM event log, the multiple entries are similar to this:

Late-Breaking Information 41
Known Issues

53 Tue Nov 4 14:01:16 2014 Fault Fault critical

Fault detected at time = Tue Nov 4 14:01:16 2014. The suspect
component: /SYS/PS0 has fault.chassis.device.psu.fail with
probability=100.
352 Tue Nov 4 14:01:06 2014 Sensor Log minor
Power Supply : /SYS/PS0/STATE : Predictive failure : Asserted
351 Tue Nov 4 14:01:06 2014 Sensor Log minor
Power Supply : /SYS/PS0/STATE : Failure detected : Deasserted
350 Tue Nov 4 14:01:00 2014 Fault UUID_Repaired minor
Fault with UUID 9e6ba288-ec5a-48bf-d129-e24a90055231 repaired
349 Tue Nov 4 14:01:00 2014 Fault Repair minor
Component /SYS/PS0 repaired
348 Tue Nov 4 14:01:00 2014 Fault Repair minor
Fault fault.chassis.device.psu.fail on component /SYS/PS0 cleared
347 Tue Nov 4 14:01:00 2014 Sensor Log minor
Power Supply : /SYS/PS0/STATE : Predictive failure : Deasserted
346 Tue Nov 4 14:01:00 2014 Sensor Log minor
Power Supply : /SYS/PS0/STATE : Failure detected : Asserted>

Workaround: The messages will stop repeating when you replace the failing power supply.

Servers With Missing Miniroot Indicate Cooling in

the List of Affected Subsystems (20922954)
This issue was fixed in system firmware 9.5.2.g.

If the miniroot volume is missing in a system, the property /System/Open_Problems shows

Cooling in the list of affected subsystems:

-> show /System/Open_Problems

Open Problems (1)

Date/Time Subsystems Component
------------------------ ------------------ ------------
Sat Aug 29 22:29:17 2015 System, Cooling /System (Host System)
The ILOM Mini-Root system is missing. (Probability:100,
UUID:ce6e7e97-523c-e55b-a6fc-96dd2dd09187, Resource:/SYS/SP, Part
Number:9999999294, Serial Number:465769T+1520BUR294, Reference
Document:http://support.oracle.com/msg/ILOM-8000-9W)

Cooling should not appear in the Subsystems column because the miniroot volume has nothing
to do with the cooling subsytem.

If the miniroot volume is missing from the system, this Cooling message can be safely ignored.
In addition, there is no interruption in service.

42 SPARC T7 Series Servers Product Notes • July 2019

 Known Issues

Oracle Solaris Should Process os-root-device Only

Upon Boot (21077998)
After Oracle Solaris is installed on the server with its root file system stored on an iSCSI
device that is accessed using IP over Infiniband (IPoIB), if Oracle Solaris is subsequently
reinstalled onto other storage devices, booting the newly installed Oracle Solaris instance
attempts to mount the root file system from the previously installed iSCSI devices using IPoIB.
This situation is due to the continued presence of the os-root-device NVRAM variable. For
example:

Note - This example shows output from an M7 server.

(1) os-root-device refers to a working IB HCA and iSCSI target configuration

SPARC M7-8, No Keyboard
Copyright (c) 1998, 2015, Oracle and/or its affiliates. All rights reserved.
OpenBoot 4.37.3, 1.4257 TB memory available, Serial #105354904.
Ethernet address 0:10:e0:47:96:a4, Host ID: 8765678.

Boot device: /pci@315/pci@1/nvme@0/disk@1 File and args:

SunOS Release 5.11 Version 11.3 64-bit
Copyright (c) 1983, 2015, Oracle and/or its affiliates. All rights reserved.
@ NOTICE: Configuring iSCSI to access the root filesystem...
Hostname: xxxxx
cannot mount 'rpool/export' on '/export': directory is not empty
cannot mount 'rpool/export' on '/export': directory is not empty
cannot mount 'rpool/export/home' on '/export/home': failure mounting parent
dataset
cannot mount 'rpool1/VARSHARE/zones' on '/system/zones': mountpoint or
dataset is busy
svc:/system/filesystem/local:default: WARNING: /usr/sbin/zfs mount -a failed:
one or more file systems failed to mount
Sep 29 02:03:56 svc.startd[13]: svc:/system/filesystem/local:default: Method
"/lib/svc/method/fs-local" failed with exit status 95.
Sep 29 02:03:56 svc.startd[13]: system/filesystem/local:default failed
fatally: transitioned to maintenance (see 'svcs -xv' for details)

xxxxx console login:

(2) os-root-device is invalid (bad device path to an IB device):

...

SPARC M7-8, No Keyboard

Copyright (c) 1998, 2015, Oracle and/or its affiliates. All rights reserved.
OpenBoot 4.37.3, 1.4257 TB memory available, Serial #105354904.

Late-Breaking Information 43
Known Issues

Ethernet address 0:10:e0:47:96:a4, Host ID: 8765678.

Boot device: /pci@315/pci@1/nvme@0/disk@1 File and args:

SunOS Release 5.11 Version 11.3 64-bit
Copyright (c) 1983, 2015, Oracle and/or its affiliates. All rights reserved.
resolve_ib_path_one: unable to hold /pci@30d/pci@1/pciex15b3,1003@1
WARNING: Cannot plumb network device 6

panic[cpu0]/thread=20012000: vfs_mountroot: cannot mount root

Warning - stack not written to the dumpbuf

000000002000fa00 genunix:main+1dc (208a1000, 20122e40, 202f6640, 0, 0, 1)
%l0-3: 0000000000000000 00000000203bdc00 0000000000000000 0000000010070800
%l4-7: 0000000020122c00 0000000010070800 0000000000000000 0000000000000000

Deferred dump not available.

skipping system dump - no dump device configured and deferred dump is
disabled
rebooting...
Resetting...

...

(3) os-root-device refers to an invalid or inaccessible iSCSI target:

SPARC M7-8, No Keyboard
Copyright (c) 1998, 2015, Oracle and/or its affiliates. All rights reserved.
OpenBoot 4.37.3, 1.4257 TB memory available, Serial #105354904.
Ethernet address 0:10:e0:47:96:a4, Host ID: 8765678.

Boot device: /pci@315/pci@1/nvme@0/disk@1 File and args:

SunOS Release 5.11 Version 11.3 64-bit
Copyright (c) 1983, 2015, Oracle and/or its affiliates. All rights reserved.
@ NOTICE: Configuring iSCSI to access the root filesystem...
@ WARNING: iscsi connection(5) login failed - Requested ITN does not exist at
this address. (0x02/0x03) Target:
oqn.1986-03.com.sun:02:sca-m78-168-pd0-s11u3-30-sparc, TPGT: -1
@ WARNING: iscsi connection(5) login failed - Requested ITN does not exist at
this address. (0x02/0x03) Target:
oqn.1986-03.com.sun:02:sca-m78-168-pd0-s11u3-30-sparc, TPGT: -1
@ WARNING: iscsi connection(5) login failed - Requested ITN does not exist at
this address. (0x02/0x03) Target:
oqn.1986-03.com.sun:02:sca-m78-168-pd0-s11u3-30-sparc, TPGT: -1
@ WARNING: iscsi connection(5) login failed - Requested ITN does not exist at
this address. (0x02/0x03) Target:
oqn.1986-03.com.sun:02:sca-m78-168-pd0-s11u3-30-sparc, TPGT: -1
@ WARNING: iscsi connection(5) login failed - Requested ITN does not exist at
this address. (0x02/0x03) Target:
oqn.1986-03.com.sun:02:sca-m78-168-pd0-s11u3-30-sparc, TPGT: -1

44 SPARC T7 Series Servers Product Notes • July 2019

 Known Issues

@ WARNING: Failed to configure iSCSI boot session

@ WARNING: Failed to get iscsi boot path

panic[cpu0]/thread=20012000: vfs_mountroot: cannot mount root

Warning - stack not written to the dumpbuf

000000002000fa00 genunix:main+1dc (208a1000, 20122e40, 202f6640, 0, 0, 1)
%l0-3: 0000000000000000 00000000203bdc00 0000000000000000 0000000010070800
%l4-7: 0000000020122c00 0000000010070800 0000000000000000 0000000000000000

Deferred dump not available.

skipping system dump - no dump device configured and deferred dump is
disabled
rebooting...
Resetting...

Workaround:
1. Clear os-root-device at the ok prompt.

ok set-default os-root-device
2. Boot Oracle Solaris.
ok boot

Logical Domains Virtual Disk Access Failed -

Oracle Solaris Cluster Not Coming Up Following
Single-Node Reboot (21421237)
This issue appears on the configuration of a two-node Oracle Solaris 3.3 3/13 cluster plus the
Oracle Solaris 10 1/13 recommended patch set and quorum devices using the SCSI-2 protocol,
configured as Oracle VM Server for SPARC guest domains serviced by Oracle Solaris 11.3 I/O
domains.

If one node leaves the cluster, either due to a node halt, panic, reboot event, or loss of cluster
interconnect (that is, split brain), delayed SCSI-2 access to a quorum device might lead to a
panic on the other cluster node, producing an error similar to this one:
panic[cpu13]/thread=30011df80e0: CMM: Unable to acquire the quorum device.

Workaround: On guest domains running Oracle Solaris 10 1/13 plus recommended patches,
configure the quorum device to use the SCSI-3 protocol.

Recovery: If the workaround is unsuccessful, reboot the server. Collect any panic core dumps
and contact your service representative.

Late-Breaking Information 45
Known Issues

Note - For more information about administering SCSI protocol settings for storage devices,
refer to the Oracle Solaris Cluster System Administration Guide.

Fault Repair From the SPM Does Not Work In a

Multi-Hop Between a Control Domain and the Root
Domain (21459393)
This issue was fixed in Oracle Solaris 11.3.2.4.

If an I/O domain diagnoses a fault against a device to which it has direct access, the fault
information is visible in the primary (that is, control) domain and in Oracle ILOM, as well as in
the I/O domain that diagnosed the fault. However, if you attempt to repair the fault from Oracle
ILOM, the repair action fails. No error message is displayed, but the fault is still present if you
execute subsequent fmadm commands from the I/O domain.

Note - This issue applies to all the repair-related variants of the fmadm command fmadm acquit,
fmadm repair, fmadm repaired, and fmadm replaced.

Recovery: If you encounter this bug, execute the fmadm repair command in the affected I/O
domain and the repair operation works as intended.

sun4v_pcbe_enable Warning Appears When Starting

a Guest Domain Running Oracle Solaris 10
(21466955)
When running Oracle Solaris 10 in a guest domain, this message appears on the guest domain
console upon start up:
Boot device: disk File and args: -k
Loading kmdb...
SunOS Release 5.10 Version Generic_150400-20 64-bit
Copyright (c) 1983, 2014, Oracle and/or its affiliates. All rights reserved.
WARNING: sun4v_pcbe_enable: no HV API found
Hostname: ...

This message appears because the Oracle Solaris 10 version of cpustat(1M) cannot monitor
the performance counters in the SPARC T7 and M7 processors. This message may be safely
ignored.

46 SPARC T7 Series Servers Product Notes • July 2019

 Known Issues

Starting with Oracle Solaris 11.3, the cpustat(1M) command can monitor the performance
counters in SPARC T7 and M7 processors.

ixgbevf Driver Does Not Properly Report a Link

State Change to the MAC Layer (21629053)
This issue was fixed in Oracle Solaris 11.3.2.4.

If you reboot the root domain with I/O Resiliency (IOR) enabled (that is, logical domains
failure policy set to ignore) the I/O domain might fail to initiate an ixbgevf network interface.

For example, before rebooting non primary root domain (NPRD), type:

Note - This example shows output from a SPARC T7-2 server.

# ipmpstat -i
INTERFACE ACTIVE GROUP FLAGS LINK PROBE STATE
net1 yes ipmp0 --mbM-- up disabled ok
net7 yes ipmp0 ------- up disabled ok

# dladm show-phys
LINK MEDIA STATE SPEED DUPLEX DEVICE

net0 Ethernet up 0 unknown vnet0

net7 Ethernet up 10000 full ixgbevf7
net1 Ethernet up 10000 full ixgbevf4

After rebooting the NPRD, type:

# ipmpstat -i
INTERFACE ACTIVE GROUP FLAGS LINK PROBE STATE
net1 no ipmp0 ------- down disabled failed
net7 yes ipmp0 --mbM-- up disabled ok
# dladm show-phys
LINK MEDIA STATE SPEED DUPLEX DEVICE

net0 Ethernet up 0 unknown vnet0

net7 Ethernet up 10000 full ixgbevf7
net1 Ethernet down 10000 full ixgbevf4

Recovery: Perform these steps.

1. Unplumb and plumb the failed interface.

Late-Breaking Information 47
Known Issues

# ipadm delete-ip net1

# ipadm create-ip net1
2. Add the device into the ipmp group again.

# ipadm add-ipmp -i net1 ipmp0

Flash Logging Is Broken in Hypervisor (21646012)

The flash file used to log the console before the SPM is available (saved to /HOST/console/
bootlog) does not work properly.

In some rare cases, the boot flash output stops when the hypervisor initiates during start up. The
console log is still available once the SPM is started, in /HOST/console/history.

If the server encounters an error when the SPM is down or before the SMP boot sequence
finishes, the error might not be logged in the system flash log for later recovery and diagnosis.
In addition, the error does not appear in the fault management shell or in show faulty output.

If the system flash log write action collides with a saved LDoms configuration, the LDoms
configuration might be overwritten. On the next power-on event, the server boots into factory
default mode, and the saved XML configuration file is available on the boot disk.

Recovery: To recover the configuration, enable recovery mode and reboot the server.
1. Enable recover mode.

primary# svccfg -s ldmd setprop ldmd/recovery_mode = astring: auto

primary# svcadm refresh ldmd
2. Reboot the server.

primary# reboot

Solaris OS Fails to Drop to OpenBoot Upon

Retirement of SCC Cores and L2DS (21644300,
21772653)
This issue was fixed in Sun System Firmware 9.5.2.

If the primary domain is configured without enough resources (two SCCs or fewer) and
correctable errors trigger an FMA retirement action affecting both these SCCs, then the domain

48 SPARC T7 Series Servers Product Notes • July 2019

 Known Issues

hangs upon reboot. Other domains are not affected, and continue to run normally as long as
their own network cards and drives are still available. If an error triggers a domain retirement,
you can view the fault using the fmadm faulty command.

Note - This example shows output from a SPARC T7-2 server.

SUNW-MSG-ID: SPSUN4V-8001-YA, TYPE: Problem, VER: 1, SEVERITY: Major

EVENT-TIME: Tue Oct 6 18:50:50 EDT 2015
PLATFORM: SPARC T7-2, CSN: 12345678, HOSTNAME: bur-t72-303-sp
SOURCE: fdd, REV: 1.0
EVENT-ID: f78853a2-87cf-e147-efb3-ecc370ef147e
DESC: An event was received indicating a fault was diagnosed by another fault manager.
AUTO-RESPONSE: Refer to the document at http://support.oracle.com/msg/SPSUN4V-8001-YA.
IMPACT: Refer to the document at http://support.oracle.com/msg/SPSUN4V-8001-YA.
REC-ACTION: Use 'fmadm faulty' to provide a more detailed view of this event. Please
refer to the associated reference document at http://support.oracle.com/msg/SPSUN4V-
8001-YA for the latest service procedures and policies regarding this diagnosis.

-> fmadm faulty

Time UUID msgid Severity

------------------- ------------------------------------ -------------- --------
2015-10-06/22:51:00 abea80bd-6d18-46a4-e9cc-fda7df765748 SPSUN4V-8001-YA Major

Problem Status : open [injected]

Diag Engine : fdd 1.0
System
Manufacturer : Oracle Corporation
Name : SPARC T7-2
Part_Number : 87654321
Serial_Number : 12345678

----------------------------------------
Suspect 1 of 1
Fault class : fault.cpu.generic-sparc.l2d-uc
Certainty : 100%
Affects : /SYS/MB/CM0/CMP/SCC3/L2D1
Status : faulted

FRU
Status : faulty
Location : /SYS/MB
Manufacturer : Oracle Corporation
Name : ASY,MB,T7-2
Part_Number : 7093274
Revision : 02
Serial_Number : 465769T+1434NH00JJ

Late-Breaking Information 49
Known Issues

Chassis
Manufacturer : Oracle Corporation
Name : SPARC T7-2
Part_Number : 87654321
Serial_Number : 12345678

Description : A cpu has experienced an uncorrectable level 2 data cache

error (UE).

Response : Cpu cores associated with the cache will be deconfigured.

Impact : Some services may be lost and performance may be impacted.

Action : Use 'fmadm faulty' to provide a more detailed view of this

event. Please refer to the associated reference document at
http://support.oracle.com/msg/SPSUN4V-8001-YA for the latest
service procedures and policies regarding this diagnosis.

------------------- ------------------------------------ -------------- --------

Time UUID msgid Severity
------------------- ------------------------------------ -------------- --------
2015-10-06/22:50:50 f78853a2-87cf-e147-efb3-ecc370ef147e SPSUN4V-8001-YA Major

Problem Status : open [injected]

Diag Engine : fdd 1.0
System
Manufacturer : Oracle Corporation
Name : SPARC T7-2
Part_Number : 87654321
Serial_Number : 12345678

----------------------------------------
Suspect 1 of 1
Fault class : fault.cpu.generic-sparc.l2d-uc
Certainty : 100%
Affects : /SYS/MB/CM0/CMP/SCC3/L2D0
Status : faulted

FRU
Status : faulty
Location : /SYS/MB
Manufacturer : Oracle Corporation
Name : ASY,MB,T7-2
Part_Number : 7093274
Revision : 02
Serial_Number : 465769T+1434NH00JJ
Chassis
Manufacturer : Oracle Corporation

50 SPARC T7 Series Servers Product Notes • July 2019

 Known Issues

Name : SPARC T7-2

Part_Number : 87654321
Serial_Number : 12345678

Description : A cpu has experienced an uncorrectable level 2 data cache

error (UE).

Response : Cpu cores associated with the cache will be deconfigured.

Impact : Some services may be lost and performance may be impacted.

Action : Use 'fmadm faulty' to provide a more detailed view of this

event. Please refer to the associated reference document at
http://support.oracle.com/msg/SPSUN4V-8001-YA for the latest
service procedures and policies regarding this diagnosis.

This issue is the root cause of a domain retirement if the fault is reported on the same cores
running the primary domain, and the primary domain hangs upon reboot.

Workaround: Ensure that the primary guest domain is assigned two SCCs or more (that is, a
minimum of two SCCs and a few additional cores) on the same node.

Recovery: Force reset the domain (reset -f /HOST) to regain access. Upon reboot the server
is unable to access the most recently saved SPM configuration, and reverts to the factory default
configuration instead.

Sun Dual 10GbE SFP+ PCIe Low Profile Adapter

Card Faulted With Virtual_TTE_invalid Error
(21694361, 21848425)
CR 21694361 was fixed in Sun System Firmware 9.5.2.

This issue occurs when guest domains configured using PCIe virtual functions are rebooted. In
that case, if the guest domain owning the virtual function reboots, the system might generate
TTE_Invalid errors during the reboot process. These errors are harmless and can be ignored, but
they will trigger system faults.

For example, type:

Note - This example shows output from an M7 server.

# fmdump -eV
2015-08-18/04:57:48 ereport.io.pciex.rc.epkt@/SYS/CMIOU0/IOH/IOS0
...

Late-Breaking Information 51
Known Issues

event_name = Virtual_TTE_invalid

# fmadm faulty

---------------- ------------------------------------ -------------- --------

Time UUID msgid Severity
------------------- ------------------------------------ ----------------------
2015-09-22/14:23:19 eeb0aefa-2af6-4cd4-9a7a-e873266f82a2 PCIEX-8000-0A Unknown

Problem Status : open

Diag Engine : eft 1.16
System
Manufacturer : Oracle Corporation
Name : SPARC M7-8
Part_Number : 32973358+5+1
Serial_Number : AK00246629

System Component
Manufacturer : Oracle Corporation
Name : SPARC M7-8
Part_Number : 7092780
Serial_Number : AK00254527

----------------------------------------
Suspect 1 of 1
Fault class : fault.io.pciex.device-interr
Certainty : 100%
Affects : /SYS/CMIOU2/PCIE3/CAR/CARD
Status : faulted

FRU
Status : faulty
Location : /SYS/CMIOU2/PCIE3/CAR
Manufacturer : Oracle Corporation
Name : TLA,CAR,X16
Part_Number : 7089613
Revision : 03
Serial_Number : 465769T+14329C07F5
Chassis
Manufacturer : Oracle Corporation
Name : SPARC M7-8
Part_Number : 7092780
Serial_Number : AK00254527

Description : A fault was diagnosed by the Host Operating System.

Action : Please refer to the associated reference document at

http://support.oracle.com/msg/PCIEX-8000-0A for a complete,

52 SPARC T7 Series Servers Product Notes • July 2019

 Known Issues

detailed description and the latest service procedures and

policies regarding this diagnosis.

Recovery: Clear the fault. For example:

# fmadm acquit eeb0aefa-2af6-4cd4-9a7a-e873266f82a2

The Solaris OS Sometimes Fails to Recover

Following a QRAP Event (22022572)
On rare occasions, the Solaris OS might fail to reboot successfully following a panic caused by
a SPARC processor queue array parity error (QRAP).

This issue is present if mreg-uc ereports with the error-condition field is set to QRAP.

From the fault management shell, type:

fmdump -eV
2015-10-14/09:11:42
ereport.cpu.generic-sparc.mreg-uc@/SYS/CMIOU15/CM/CMP/SCC7/CORE3
__tod-0 = 0x561e7ead
__tod-1 = 0x1cee5cc0
tstate = 0x4400000402
htstate = 0x4
ehdl = 0x1fdc100000000007
tpc = 0x60000f022fac
tl = 0x1
tt = 0x29
diagnose = 0x1
error-condition = QRAP
reported-by = Hypervisor
ps-pesr = 0x600
ps-res-err-qhead = 0x0
ps-res-err-qtail = 0x0
ps-nres-err-qhead = 0x0
ps-nres-err-qtail = 0x0
ps-cpu-mondo-qhead = 0x0
ps-cpu-mondo-qtail = 0x800000000011b340
ps-dev-mondo-qhead = 0x0
ps-dev-mondo-qtail = 0x0

In addition, output from the fmadm faulty command indicates that a core has been disabled by
the fault management software:
-> fmadm faulty
------------------- ------------------------------------ ----------------------
Time UUID msgid Severity

Late-Breaking Information 53
Known Issues

------------------- ------------------------------------ ----------------------

2015-11-13/05:59:06 ad7bc3bf-c0d7-657b-89e2-cc68f888c312 SPSUN4V-8000-AC Critical

Problem Status : open

Diag Engine : fdd 1.0
System
Manufacturer : Oracle Corporation
Name : SPARC M7-16
Part_Number : 32863269+3+1
Serial_Number : AK00247538

----------------------------------------
Suspect 1 of 1
Fault class : fault.cpu.generic-sparc.core-uc
Certainty : 100%
Affects : /SYS/CMIOU15/CM/CMP/SCC7/CORE3
Status : faulted

FRU
Status : faulty
Location : /SYS/CMIOU15
Manufacturer : Oracle Corporation
Name : CMIOU Module
Part_Number : 7090830
Revision : 04
Serial_Number : 465769T+14426C01MU
Chassis
Manufacturer : Oracle Corporation
Name : SPARC M7-16
Part_Number : 32863269+3+1
Serial_Number : AK00247538

Description : This core has encountered an uncorrectable error.

Response : The fault manager will attempt to remove all strands

associated with this resource from service.

Impact : System performance may be affected.

Action : Use 'fmadm faulty' to provide a more detailed view of this

event. Please refer to the associated reference document at
http://support.oracle.com/msg/SPSUN4V-8000-AC for the latest
service procedures and policies regarding this diagnosis.

Recovery

From ILOM, stop and then start the affected host. For example:
-> stop /HOST2

54 SPARC T7 Series Servers Product Notes • July 2019

 Known Issues

-> start /HOST2

Virtual_TTE_invalid Error On Assigned IOV Device

(22138210)
When the primary or root domain is rebooted, the I/O domain is notified to suspend the
assigned virtual functions from domain being rebooted, and resume them when the reboot is
complete. However, in some cases the resume notification might be issued prematurely, causing
the I/O domain to fail to resume one or more of its assigned virtual functions. This issue occurs
only on M7/T7 platforms when the LDOMS failure policy on the I/O domain is not set, or is set
to ignore.

Two symptoms indicate this failure:

■ A warning on the console in the I/O domain (also logged in /var/adm/messages)

■ An FMA fault in the primary or root domain to which the physical function is assigned

The warning from the I/O domain console is as follows:

WARNING: pxsoft_msi_resume: retry limit exceeded.

The FMA fault in the primary or root domain is fault.io.pciex.device-invreq (PCIEX-

8000-8R). It includes an ereport.io.pciex.rc.epkt error report with the following string:

event_name = Virtual_TTE_invalid

To list any logged FMA error reports, type:

# fmdump -e

For a verbose listing which might include the event_name = Virtual_TTE_invalid string, type:

# fmdump -eV

The warning from the I/O domain console is as follows:

WARNING: pxsoft_msi_resume: retry limit exceeded.

If this issue occurs, one or more assigned virtual functions will no longer work properly in the I/
O domain following a primary or root domain reboot. The device drivers for the affected virtual
functions cannot process any interrupt signals from the underlying hardware devices.

Recovery

Late-Breaking Information 55
Known Issues

To recover from this issue and regain the affected virtual functions, reboot the affected I/O
domain.

To make the I/O domain more resilient against this issue, configure the following setting in its
/etc/system file:

set pxsoft:pxsoft_resume_max_retries=1024

This setting only impacts the resume operations of virtual functions in the I/O domain. You
must reboot I/O domain for this new setting to take effect.

Mitigation

This issue occurs if an I/O domain is assigned multiple virtual functions coming from multiple
physical functions in the same PCIe bus. You can avoid this issue if you assign virtual functions
in the I/O domain only from a single physical function of that PCIe bus.

Saved LDOMs Configuration Fails To Boot

After Upgrading the Server With an Additional
Processor Module (22012359, 22161099)

Note - Oracle does not recommend using named device bindings in LDOMs configurations.

This issue occurs on a SPARC T7-4 server when you attempt to reboot into an existing LDOMs
configuration that uses named resource bindings after you have upgraded the server with
an additional processor module. Factory default mode is incorrectly selected during reboot.
The server boots into recovery mode, and named resource bindings specified in the previous
LDOMs configuration are ignored.

Recovery

If you made a record of your LDOMs configuration before adding the processor module,
regenerate the LDOMs configuration as described in the Workaround.

If you need immediate access to the old configuration, follow these steps:
1. Remove the processor module from PM1 and boot the server.
Refer to the SPARC T7-4 Server Service Manual for removal instructions.
The server reverts to its previous configuration, including the named resource bindings.
2. Make a record specific CPU and memory bindings of the LDOMs configuration.
3. Install the processor module into PM1 and boot the server.

56 SPARC T7 Series Servers Product Notes • July 2019

 Known Issues

Refer to the SPARC T7-4 Server Service Manual for installation instructions.
4. Regenerate the LDOMs configuration.
5. Save the LDOMs configuration and make it the default boot configuration.

Subsequent reboots will use the saved LDOMs configuration.

Workaround

Regenerate the LDOMs configuration:

1. In ILOM, set the server boot configuration to factory-default.

2. Install the processor module into PM1.
Refer to the SPARC T7-4 Server Service Manual for installation instructions.
3. Reboot the server into the factory default configuration.
4. Create your desired LDOMs configuration.
5. Save the LDOMs configuration and make it the default boot configuration.

Subsequent reboots will use the saved LDOMs configuration.

T7-x Observes Dropped Characters Running

SysFW 9.7.4 (25506535)
If you copy a large amount of text on OBP or Solaris while running SysFW 9.7.4, some
the characters might be dropped. In the example below show at OBP, when the text
select /pci@301/pci@1/scsi@0 is copied from line 2 to line 5, the scsi@0 text is unexpectedly
dropped.

{0} ok
{0} ok select /pci@301/pci@1/scsi@0
{0} ok show-sas-wwid
SAS World Wide ID is 50800200 0218f9d0
{0} ok
{0} ok unselect-dev
{0} ok select /pci@303/pci@1/ <-- missing the "scsi@0"
{0} ok show-sas-wwid
show-sas-wwid ?

Another possible scenario to encounter the issue is to use script to enter a long OBP or Solaris
command.

Workaround: If you encounter this issue, manually type the commands instead of copying a
large amount of text.

Late-Breaking Information 57
Known Issues

Workaround: Another option is to disable console logging from Oracle ILOM, however
choosing this option means you will not get a log of console output when the SP is degraded for
later retrieval. To use this workaround, type this command at the Oracle ILOM prompt:

-> set /HOST/console logging=disabled

For more information about how to work with Oracle ILOM features, refer to the Oracle ILOM
documentation at:

http://www.oracle.com/goto/ilom/docs

Do Not Use Certain Hynix 32GB DIMMs on T7

Servers (23284277, 23284255, 23222472)
Do not use Hynix 32GB DIMMs with date codes WW1532 to WW1617 in SPARC T7 series
servers.

Net Install of 11.3SRU21b4 and b5 Panics With

NVME Devices Installed (26241738)
SPARC T7 and SPARC S7 servers with NVMe disk drives may experience Oracle Solaris OS
panic during Oracle Solaris 11.3 SRU21 OS net install or boot.

For example:

Service discovery finished successfully

Process of obtaining install manifest initiated

panic[cpu111]/thread=2a122fd1b80: Deadlock: cycle in blocking chain

000002a122fd12f0 genunix:turnstile_will_prio+218 (2a122fd1b80, 2a12062fb80,

0, 3c, 1, 2087f000)
%l0-3: 000002a122fd1b80 0000000000000001 0000000000000000 000000002087f108
%l4-7: 00000000100b5fc0 0000000020121c08 000000001013b400 0000000000000000
000002a122fd13a0 genunix:turnstile_block+170 (c040b04238c38, 0,
c04090aa684c0, 20121c08, 0, 0)
%l0-3: 000c0408ea1a79a8 0000000000002000 00000000202ffd40 0000000000000001
%l4-7: 0000000000000000 000002a122fd1b80 0000000000000000 00000000208c0800
000002a122fd1450 unix:mutex_vector_enter+43c (2a12062fb80, 2a12062fb80,
20121c50, c04090aa684c0, 208ae708, 0)
%l0-3: 0000000000000000 0000115bb3b38222 0000000020121c38 000002a12062fb80

58 SPARC T7 Series Servers Product Notes • July 2019

 Known Issues

%l4-7: 0000000000000001 0000000000000000 0000000000000000 0000000000000004

000002a122fd1500 nvme:nvmex_resume+148 (40073a1d0b8, ffffffffffffffff,
c04090aa68000, 1, c04090aa684c0, 704dfc00)
%l0-3: 0000000000000002 0000000000000050 0000000000000001 000c040b04f86250
%l4-7: 0000000000000008 000c04090e636520 000c040b0f648690 0000000000000001
000002a122fd15b0 nvme:nvmex_intr_adjust+36c (c04090aa68000, 704df768, 100,
110de748, c04090aa684f0, c04090aa684c0)
%l0-3: 00000000704df768 0000000000000021 0000000000000021 000c0409cfdf1b80
%l4-7: 0000000000000021 0000000000000000 00000000110de400 00000000704df400
000002a122fd1680 nvme:nvmex_cbfunc+9c (40073a1d0b8, 0, 4, 0, 704dfe28,
704dfc00)
%l0-3: 00000000f7499a40 00000000f9861600 0000000000000001 0000000000000001
%l4-7: 0000000000000000 0000000000000000 0000000006cf7d40 0000000000000001
000002a122fd1730 genunix:i_ddi_irm_notify+d0 (c0409cd01d1d0, 0,
c04090aa9d458, 110e129c, 2, 0)
%l0-3: 0000040073a1d0b8 000c04090a4182a8 0000000000000021 0000000020846db8
%l4-7: 0000000000000021 0000000000000002 0000000000000000 0000000000000001
000002a122fd17f0 genunix:i_ddi_irm_balance+1fc (c0409cd01d1d0, 2, 20000, 0,
fffffffffffffffe, c0409cd01d238)
%l0-3: 000c04090aa9d458 0000000000000000 000c0409cd01d1f8 0000000000000004
%l4-7: 0000000000000000 0000000000000000 000c0409cd01d218 0000000000010000
000002a122fd18a0 genunix:irm_balance_thread+24 (c0409cd01d1d0, 0, 202ffd40,
89, c0409cd01d1f0, c0409cd01d202)
%l0-3: 0000040002f3a0d0 0000000000000000 0000000020847800 000002a113b73b80
%l4-7: 0000000000000000 000c0409c98e0408 0000000000000200 000000002051a4f0

syncing file systems... 41 40 done

Deferred dump not available.
skipping system dump - no dump device configured and deferred dump not
available
rebooting...

Workaround: Do not use Oracle Solaris S11.3 SRU 21 or SRU 22 if your server requires
NVMe support. Use a different, supported SRU for NMVe support.

Late-Breaking Information 59
60 SPARC T7 Series Servers Product Notes • July 2019