IBM Technology

Quick-install of the PowerHA Full System Thursday, April 27, 2023

Expert Labs
Flashcopy Manager 5.1

Christian Aasland
aasland@[Link]

IBM Technology Expert Labs 1

 IBM Technology
Expert Labs

❑ What is Flash Copy?

Overview ❑ Storage and HMC configuration
❑ Production LPAR setup
❑ Controller LPAR setup
❑ First Flash Copy
❑ Add BRMS Integrations
❑ Additional Topics

2
 IBM Technology
What the heck is this document for? Expert Labs

o This is a quick-install guide for configuring the Full System Flashcopy Manager for the following storage products:

o SVC family (FlashSystems, Storwize)

o DS8K family

o IBM PowerVS [Cloud]

o Customers can have it, but it is designed to be performed by a Lab Services consultant

o It does not explain details or how to handle errors or special/complex situations

o Primary documentation is the FSFC Manager Webpage:

o [Link]

3
 IBM Technology
Overview of Full System Flash Copy concepts Expert Labs

3 1 2
Partition returns to Write contents of memory to Flashcopy the disks
production duties disk

6
Copy BRMS data
back to the source

Flashcopy
4
Source
IPL the target partition

Tape
Target
5
Backup the target
partition to tape
Controller

7 LPARs
Deactivate target
LPAR
4
 IBM Technology
Flashcopy at DR site Expert Labs

It is possible to perform the

flashcopy at the DR site.

PowerHA Cluster
The toolkit will manage the Replication
hardware replication so that the Controller Controller
DR site can create a flashcopy
for backup purposes.
Hardware
This is supported for: Preferred Replication Preferred
Source Target
• DS8K GMIR
• SVC GMCV and MMIR

Flashcopy
This is not supported for: Target
• PowerVS
• DS8K MMIR
• SVC GMIR

5
 IBM Technology
Shared Flash Copy LPAR Expert Labs

The controlling LPAR can manage the host

connections of the Backup LPAR to its disks. This
allows one backup LPAR to sequentially service the
backups of multiple production LPARs, reducing the
footprint of the total solution.
Our toolkit’s controlling LPAR can coordinate the Controller
flashcopy processes among multiple production
LPARs. This includes when to perform the
flashcopy and managing the locks of multiple Preferred
processes when they share resources (LPARs, Source

Flashcopy
2am
disks, etc).
Preferred
For example, both flashcopies can occur at 2am. Source

Flashcopy
2am
Then the toolkit will attach the target LPAR to the
first flashcopy disks and run it’s backup. When that
finishes, the toolkit will attach the target LPAR to the Target
second flashcopy disks, IPL and run backups from
that.
The number of production LPARs sharing a target
LPAR is limited by how long the backups take.

6
 IBM Technology
Multi-Flash Expert Labs

When one production LPAR hosts multiple

applications which quiesce at different times, or an
application that has both pre-batch and post-batch Controller
processing, our toolkit can manage concurrent
flashcopy operations. In these scenarios each Preferred
backup LPAR will typically backup only a portion of Source

Flashcopy
the data, i.e. the data associated with a specific

2am
application.
Note that a quiesce or memory flush affects the
entire production LPAR, not just one application. Target 1
Target 2

7
 IBM Technology
Firewall Access (on-prem) Expert Labs
Workstation access required for
installation and configuration only

Ping, Telnet (23)

Source

(dft=55920)
Toolkit
Ping
Telnet 2300
Ping, ssh port 22

Toolkit communications default = 55920

BRMS Xfer port *DFT = 55066
Controllers
PowerHA ports 5550,
Ping, ssh port 22
5551 between
DSCLI = port 1751
controllers

Toolkit
Ping
(dft=55920)
Target
8
 IBM Technology
Firewall Access (PowerVS) Workstation access requiredExpert
for Labs
installation and configuration
only

Source

Toolkit (dft=55920)
Ping
55920
Toolkit communications default =
BRMS Xfer port *DFT = 55066

Controllers
Flashcopy

PowerHA ports
5550, 5551 between
controllers

IBM PowerVS URLs:

[Link]
resource-
Target [Link]
[Link]

9
 IBM Technology
SVC setup prior to our engagement Expert Labs

o Configure the storage unit for Primary, Secondary and Controlling LPAR.
o Firmware level [Link] or newer
o If using FS9100 with GMCV and the change volumes are in a data reduction pool (DRP), the SVC must be at
firmware level [Link] or higher
o Create or select user profile
o Must be assigned to CopyOperator (or better) user group
o If changing host connections, must be Administrator
o LUNs
o For source and target LPARs
o Host connections
o Licenses (Replication, Thin-provision, etc)
o If using replication:
o Partnerships
o We can remotely help you set this up (also ensures you have communication between the SVC’s before we
arrive)
o Start replication
o Replication should be completed before we’re onsite so that won’t have to wait for it to catch up
10
 IBM Technology
DS8K setup prior to our engagement Expert Labs

o Create a user profile on the DS8K

o Can be other than QLPAR, make a member of the admin group
o Remember the password, set to not expire
o chpass –expire 0
o Configure the storage unit for Primary, Secondary and Controlling LPAR.
o Recent firmware level
o Install DSCLI on the IBM i from the DS8K CD
o Bundle [Link] or newer (required for creating GMIR D-Copy)
o Create fixed block volumes (requires ranks, arrays, extent pools, space efficient repositories, etc)
o For source and target LPARs
o Volume groups, ports and host connections
o Licenses (Replication, Space Efficient, etc)
o PPRC Paths
o We can remotely help you set this up (also ensures you have communication between the DS’s before we arrive)
o Start replication
o Replication should be completed before we’re onsite so that won’t have to wait for it to catch up

11
 IBM Technology
Cloud setup prior to our engagement Expert Labs

o Create the Cloud environment including

o One or two persistent controlling PVM instances (IBM i)
o Access to the PowerVS APIs (i.e. [Link] etc)
o Production PVM instances
o Network access (IP addresses etc)
o Storage with OS loaded and configured
o BRMS (optional)
o Flashcopy PVM instances
o Network access (IP addresses etc)
o Backup device connectivity
o Serial numbers for all the instances
o DSPSYSVAL QSRLNBR
o Include potential LPM serial numbers, if known
o Service ID
o Access to the resources (instances, storage, etc)
o API Key (the API Key must be retained when created)

12
 IBM Technology
HMC Configuration Expert Labs

o Create a user on the LPAR HMCs

o Any user name will do (as long as you remember it)

o Password is required

o Hmcsuperadmin with AllSystemResources

o Additional considerations (these are enabled by default):

o Enable remote command execution

o Enable ssh through the HMC’s firewall

13
Restoring toolkit library, setup on Production LPARs IBM Technology
Expert Labs

o Place the toolkit savefile in QGPL (FTP, scp etc)

o Restore the toolkit library:
o RSTLIB SAVLIB(QZRDHASM) DEV(*SAVF) SAVF(QZRDHASM51)
o The ‘51' refers to the release and may change
o ADDLIBLE QZRDHASM
o Run the setup program
o SETUPFSFC NODEROLE(*SRC) ACSCODE('??')
o Press PF4 and specify the line description, TCPIP interface and subnet mask to create on the controller for the target
to use
o The port number is used for toolkit communications from the controllers, *DFT is 55920
o The line description and IP interfaces will be created
o Will create user profile QLPAR without a password, initialize files etc.
o If additional line descriptions and IP addresses are needed, for example for iSCSI VTL’s, create those manually
o If the target LPAR is on a different serial number:
o A license key for the target LPAR must be entered.
o Use ADDPRDACS on the Production LPAR to enter the serial number and license for the target LPAR.

14
 IBM Technology
Enter the controller information on the Production Expert Labs
LPARs
o If using multiple controllers, set up the toolkit so STRFSFLASH can be issued on the production LPAR and connect to the first
available controller.

o Use WRKSTRPRSC *CMN and enter the controller information:

Work with Communications Startup Resources

Type options, press Enter.

1=Add 2=Change 4=Remove

Opt Usage IP Interface Line Desc Hardware Resource Location Port

_ _____ ______________
_ *CTL1 [Link] N/A PRIMARY CONTROLLER *DFT
_ *CTL2 [Link] N/A SECONDARY CONTROLLER *DFT

o STRFSFLASH CTLR(*AUTO) will use this information to connect to the first available controller.

o STRFSFLASH can still be run from the controller with CTLR(*LOCAL)

15
 IBM Technology
Modifying the Startup Program on Production Expert Labs
LPARs
o Modify startup program on each node to prevent QSTRUPPGM from running on the target.
o Not necessary if using CFGSTRPSRC (for Full System Replication)
o This is optional but adds a layer to safety.
o QZRDHASM/RUNLPARCMD SRLN(xxxxxxx) LPAR(xx) CMD(CALL + PGM(QZRDHASM/QZRDENDSBS))
MONMSG MSGID(CPF0000)
o At the very beginning of the startup program
o Specify the target LPAR serial and LPAR numbers
o Review QZRDHASM/QCLSRC QZRDENDSBS for changes
o Include MONMSG CPF0000 after RUNLPARCMD

o Modify startup program on each node to start the subsystem:

o Not necessary if using CFGSTRPSRC (for Full System Replication) (CFGSTRPRSC will start the subsystem)
o After IP and QSYSWRK start, before applications,
o STRSBS QZRDHASM/QZRDFSR
MONMSG MSGID(CPF0000)

16
 IBM Technology
Creating the cluster on the controllers Expert Labs

o If there is only one controller, you must create a single-node cluster. Perform the following steps on the single node only.
o If multiple controllers are to be configured, issue these messages on all of them:
o STRTCPSVR *INETD
o CHGTCPSVR *INETD AUTOSTART(*YES)
o CHGNETA ALWADDCLU(*ANY)
o On the Master controller
o CRTCLU CLUSTER(FSFC) START(*YES) DEVDMN(*GEN)
o PF4, fill in Primary and Secondary Controlling node names and IP addresses

o On Auxiliary controller:
o WRKCLU, validate cluster is started

17
 IBM Technology
Restoring toolkit library, setup on both Controllers Expert Labs

o Place the toolkit savefile in QGPL (FTP, scp etc)

o Restore the toolkit library:
o RSTLIB SAVLIB(QZRDHASM) DEV(*SAVF) SAVF(QZRDHASM51)
o The ‘51' refers to the release and may change
o ADDLIBLE QZRDHASM
o Run the setup program
o SETUPFSFC NODEROLE(*CTL) PORT(*DFT) ACSCODE('??')
o The port is used to receive communications from the production LPARs, *DFT is 55920
o Will create user profile QLPAR without a password, initialize files etc.

Set up IBM Pwr HA tools - FSFC (SETUPFSFC)

Type choices, press Enter.

Node role . . . . . . . . . . . > *CTL *CTL, *PRD

FSFC communications port . . . . *DFT 1-65535, *SAME, *DFT
Toolkit access code . . . . . . 12345
18
 IBM Technology
Update the startup program on the controllers Expert Labs

o Modify the startup program (after IP has been started) on each controller to:
o Start the subsystem if any process will be initiated from the production LPAR:
o STRSBS QZRDHASM/QZRDFSR
o Start the cluster if there are multiple nodes using the DDD:
o STRCLUNOD CLUSTER(*) NODE(*ALL)
o This requires *IOSYSCFG so QSTRUPJD should specify a profile like QLPAR so after compiling the
startup program issue this command:
o CHGJOBD JOBD(QSTRUPJD) USER(QLPAR)

19
 IBM Technology
Download the Java Secure Channel code (on the Expert Labs
Controllers)
o Not necessary for PowerVS Operations
o Download Java Secure Channel to /QIBM/qzrdhasm/ssh from
o [Link]
o Use the latest version, ensure the file /QIBM/Qzrdhasm/ssh/[Link] links to what you downloaded.
o The Java Secure Channel is an open-source implementation of ssh
o Because it is open-source, IBM Legal requires that you download it yourself (i.e. we can't bundle it with our toolkit)
o Download to desktop, FTP to both IBM i controllers, place it into directory /QIBM/qzrdhasm/ssh/

ftp> bin
200 Representation type is binary IMAGE.
ftp> put [Link] /QIBM/qzrdhasm/ssh/[Link]
local: [Link] remote: /QIBM/qzrdhasm/ssh/[Link]
227 Entering Passive Mode (9,5,168,177,167,46).
150-NAMEFMT set to 1.
150 Sending file to /QIBM/qzrdhasm/ssh/[Link]
226 File transfer completed successfully.
249282 bytes sent in 0.742 secs (336.12 Kbytes/sec)
ftp> 20
 IBM Technology
Create the credentials on either controller Expert Labs

o FSFC uses userid/password to log into the HMCs, DS8Ks and SVCs. Use WRKCSECRDL or ADDCSECRDE to manage these
credentials.
o The ‘Role’ should be *USER if the host is not a CSM server
o Enter the IP address, user ID, password and a description of the host for:
o SVCs
o DS8Ks
o HMCs
o This information is encrypted and placed into the device data domain and is kept consistent on both of the controllers.
o WRKCSECRDL uses PowerHA to keep the controllers in sync
o Use option 6 to validate the credentials

Work with CSE Credentials List

Type options, press Enter.

1=Add 2=Change 4=Remove

Opt IP Address Role User ID Description

[Link] *USER qlpar CTCHAHMC2

[Link] *USER qlpar IBM.2107-75XA511
21
 IBM Technology
Create or identify a Cloud Service ID Expert Labs

o FSFC uses an API Key to authenticate cloud resource usage.

o An API Key is associated with a Service ID. Use the Cloud IAM web GUI to create or identify a service ID.

22
 IBM Technology
Create Service ID API Key Expert Labs

o After identifying the Service ID identify or create an API Key

o When the API Key is created it will be displayed or downloaded in a file.
o You must record this API Key as there will not be an opportunity to retrieve it later. If you have a Service ID but no API Key
then create a new key.

23
 IBM Technology
Enter the API Key into WRKCSECRDL Expert Labs

o Use the command WRKCSECRDL TYPE(*CLOUD) to work with Cloud credentials.

o Use option 1 to add a new set of credentials. Give the credentials a name which will later be used to refer to this cloud instance.
o The API Key can be entered but not extracted. It is stored in an encrypted space.
o The URNs and URLs will depend on the specific cloud implementation.
o The team which set up your PowerVS cloud account may have this information
o For Cloud Instance ID enter *SELECT and the API will provide a list based on the resources the API Key is authorized to.

Add Cloud Credential Entry (ADDCLDCRDE)

Add Cloud Credential Entry (ADDCLDCRDE)

Type choices, press Enter.

Key name . . . . . . . . . . . . > fsfckey Character value

IAM Identity Services URL . . . [Link]

Resource controller URL . . . . [Link]

API Key . . . . . . . . . . . . Your API KEY goes here

URN Endpoint . . . . . . . . . . [Link]

Cloud Instance ID . . . . . . . *SELECT

Description . . . . . . . . . . What does the smell of Purple sound like?

24
 IBM Technology
Storage Selector Expert Labs

SVC Environment Configuration

DS8K Environment Configuration

Cloud Environment Configuration

25
 IBM Technology
Create the SVC environments on the controller Expert Labs

o An FSFC Environment describes the storage to the toolkit. Use WRKCSE to manage the environments.
o Option 1 creates a new environment
o Enter *NONE when prompted for ASP Copy Descriptions
o The environments are stored in the device data domain and is kept in sync with both controllers.
o On the SVC, flashcopy consistency groups define background copy rates, full or incremental etc. The toolkit just manages the
consistency groups.
Change a FLASH Environment
Type choices, press Enter.

Environment name . . . . . . . . . : TEST

Storage Type . . . . . . . . . . . : SVC

FlashCopy SVC information:

Flash SVC IP Address . . . . . . . [Link] IPv4
FlashCopy consistency group Id . . 2 Id
GMCV Source SVC IP Address . . . . IPv4
Remote copy consistency group Id Id

Comment:
Text . . . . . . . . . . . . . . . Something meaningful to humans

Bottom
F1=Help F3=Exit F6=Validate F12=Cancel 26
 IBM Technology
Finding the Flashcopy consistency group Id Expert Labs

o The environment requires the Flashcopy consistency group Id.

o To find it, view the flash copy consistency groups and enable the Id column

Click here to continue with CSE Data

27
 IBM Technology
Create the DS environments on the controller Expert Labs

o An FSFC Environment describes the storage to the toolkit. Use WRKCSE to manage the environments.
o Option 1 creates a new environment
o The environments are stored in the device data domain and is kept in sync with both controllers.
o Enter the requested information then PAGE DOWN

Change a FLASH Environment

Type choices, press Enter.

Environment name . . . . . . . . . : TEST

Storage Type . . . . . . . . . . . : DS8K

FlashCopy Power HA, ASP information:

Device name . . . . . . . . . . . *SYSTEM *SYSTEM, Name
Source Copy Description . . . . . *NONE *NONE, Name
Target Copy Description . . . . . *NONE *NONE, Name

FlashCopy DS unit information:

Device . . . . . . . . . . . . . . IBM.1234-1234565 Name

More . . .
28
 IBM Technology
Enter the DS information Expert Labs

o Enter the flashcopy details

o Enter the DS unit details
o If the IP address isn’t in WRKCSECRDL yet, pressing enter will take you there to add it.

Change a FLASH Environment

Type choices, press Enter.

FlashCopy IASP Manager options:

Full FlashCopy . . . . . . . . . . *NO *YES, *NO
Resync FlashCopy . . . . . . . . . *NO *YES, *NO
Multi incremental resync . . . . . *YES *YES, *NO
Space Efficient FlashCopy . . . . *NO *YES, *NO
Target PPRC . . . . . . . . . . . *NO *YES, *NO
GMIR D-Copy target flash . . . . . *NO *YES, *NO

DS unit SMC information:

Flash hmc1 . . . . . . . . . . . . IPv4
Flash hmc2 . . . . . . . . . . . . IPv4
Port . . . . . . . . . . . . . . . 1751 1750, 1751

Comment:
Text . . . . . . . . . . . . . . .

Press Enter to add DS8K credentials for [Link], press F12 to cancel.

29
 IBM Technology
Enter the DS information Expert Labs

o Enter the volume details

Add, Change or Delete Volumes

Environment . : TEST Source device : IBM.123

Type . . . . . : FLASH Target device : IBM.123
Volume sets . : 0

Type Volume options; 1=Add, 2=Change, 4=Delete, press Enter.

Source Flash
Opt Volumes Volumes
1 0100 0200

30
Enter the DS information – host connections IBM Technology
Expert Labs

o From WRKCSE, use option 16 to manage the target LPAR host connections
o Note the use of F4 to prompt for hosts, and F6 to import LUNs

Work with Host Connections

Environment . : FSFC137 Device . . . . : XBM.2107-75XA511

Type . . . . . : FLASH

Type options, press Enter.

1=Add 2=Change 4=Delete 6=Work with Volumes

Number
Opt Host name Volumes

CTCCSM1 1

Click here to continue with CSE Data

Bottom
F1=Help F3=Exit F4=Prompt F12=Cancel
31
Enter the Cloud information – PVM Instance IBM Technology
Expert Labs
Information
o From WRKCSE, use option 1 to create a new Cloud environment.
o Enter the API Key name you created in WRKCSECRDL
o On the ‘PVM instance’ parameters press F4 to retrieve a list of the instances associated with the API Key

Change a FLASH environment

Type choices, press Enter.

Environment name . . . . . : CLOUD

Storage Type . . . . . . . : CLOUD

Primary ASP . . . . . . . . *SYSTEM *SYSTEM

Cloud information:
API Key name . . . . . . . us-east
Source LPAR PVM instance 47094614-322c-4c90-9006-f833b2676613
Target LPAR PVM instance 1786c35b-715a-475d-8f40-2c947b864c37

Comment:
Text . . . . . . . . . . .

Bottom
F1=Help F3=Exit F4=Prompt F12=Cancel
32
 IBM Technology
Create Copy Services Environment (CSE) Data on Expert Labs
either Controller
o The CSE Data describes the non-storage elements of an environment.
o This data is also stored in CRG’s. The toolkit will create the CRG. It will always remain inactive when viewed in WRKCLU opt 9.
o CRTCSEDTA, CHGCSEDTA, DLTCSEDTA and DSPCSEDTA can be used to work with this information.
o Stored in the CRG so the data is synchronized between the controllers
o WRKCSEDTA displays all the data created.

Work with CSE Data

Type options, press Enter.

1=Create 2=Change 3=Copy 4=Delete 5=Display

CRG
Opt CSE Data type Text

FSR9M2 FSR FSR from 9M to 9O

HA8FSR2 FSR DS8K FSR from HA8FSR1 to HA8FSR2
FSFC9J9K FSFC
FSFC9M9N FSFC GMCV Flash
FSFC9M9P1 FSFC
FSR9J2 FSR
FSFC9M9P2 FSFC
FSFC9F9G FSFC
More...
Parameters or command:
===>
33
Enter the Copy Services Environment (CSE) Data on IBM Technology
Expert Labs
either Controller
o Use CRTCSEDTA or WRKCSEDTA opt 1 to enter the flashcopy operational details
o The command will prompt for details depending on what you enter

Create Full System FlashCopy CSE Data

Supply all required values, press Enter.

CSE Data Name . . . . . . . : TEST

Use . . . . . . . . . . . . : *SYSTEM
Copy type . . . . . . . . . : *FLASH

Environment . . . . . . . . . TEST Name

HyperSwap environment . . . . *NONE *NONE, Name
Primary controlling node . . . NODE1 Name
Secondary controlling node . . node2 Name
Communications port . . . . . 55920

Physical-Virtual Isolation
FlashCopy . . . . . . . . . *NO *YES, *NO

More...
F1=Help F3=Exit F4=Prompt F6=Query HMC F12=Cancel
34
 IBM Technology
Memory flush method on either Controller Expert Labs

o Start with *FRCWRT

Change Full System FlashCopy CSE Data

Supply all required values, press Enter.

Source LPAR IP address . . . . [Link] IPv4 address

Source host alias . . . . . . PROD Name
Target host alias . . . . . . FC Name

Method to flush memory . . . . *FRCWRT *QUIESCE, *FRCWRT, *IPL,

*NONE

More...
F1=Help F3=Exit F4=Prompt F6=Query HMC F12=Cancel

35
 IBM Technology
HMC LPAR inforomation on either Controller Expert Labs

o If the source or target LPARs participate in LPM or LUN switches, specify *SEARCH on the HMC Managed System Name parameter.
o If this is a PowerVS Cloud environment specify *CLOUDENV on the Primary HMC IP parameter and leave the rest blank.
o Note that you can prompt on the system, LPAR and Profile names using F6
o Do not specify BRMS integration (yet)
Create Full System FlashCopy CSE Data

Supply all required values, press Enter.

Target LPAR Information:

Primary HMC IP . . . . . . . [Link] IPv4 address
Secondary HMC IP . . . . . . IPv4 address
HMC managed system . . . . . *SEARCH
HMC LPAR name . . . . . . . lparName
HMC Profile name . . . . . . lparProfile
Shutdown target before
FlashCopy . . . . . . . . *YES *YES, *NO
Restart target after
FlashCopy . . . . . . . . *YES *YES, *NO, *INQ, *FRCINQ,
*PAUSE, *COPIED

Use BRMS integration . . . . . *NO *YES, *NO

More...
F1=Help F3=Exit F4=Prompt F6=Query HMC F12=Cancel 36
 IBM Technology
Communications information on either Controller Expert Labs

o If it is available, enter:
o Target LPAR IO Card location code, line description and IP interfaces (include iSCSI interfaces)
o Use *LPAR- to have toolkit resolve type-model-serial-id on the target, i.e. *LPAR-C2-T1
o Routing details

Change Full System FlashCopy CSE Data

Supply all required values, press Enter.

Target Comm Interfaces:

IO card location code . . . *LPAR-C2-T1

*NONE, identifier
Line description . . . . . . FSFCLINE line name, *VIRTUALIP
IP interface . . . . . . . . [Link] IPv4 address

+ for more values . . .

Target LPAR Default Route:

Binding interface . . . . . *NOCHANGE IPv4 address
Next hop . . . . . . . . . . IPv4 address

37
 IBM Technology
Backup device information on either Controller Expert Labs

o If the Flash Copy (target) LPAR is going to use a VTL via iSCSI it needs to know the new initiators for the target LPAR to use.
o Any existing source initiators are removed when the LPAR/VM UUID changes, i.e. when:
o Flash Copied
o Replicated and switched
o Restored on new hardware
o The IQN information is not saved (not even *SAVCFG)
o The new initiators for the target to use.
o Add the existing tape device description name and the serial number

Change Full System FlashCopy CSE Data

Supply all required values, press Enter.

Target LPAR iSCSI Information:

Device IP address . . . . . [Link] *NONE, IP address
Device port number . . . . . *DFT *DFT, 0-65535
Device IQN . . . . . . . . . vtl-tgt-for-fclparname

Client IQN . . . . . . . . . [Link]:[Link]-i0

Target LPAR Device Setup:

Backup device description VTL_PS_DEV *NONE, device name
Device serial number . . . . YTC634303828
+ for more values . . .
38
 IBM Technology
Finding the iSCSI information Expert Labs

o Where to find the IQN’s in the VTL Console

Client IQN:

39
 IBM Technology
Entering backup information on either Controller Expert Labs

o No backup command (yet)

o “Wait for ENDFSFLASH” set to *YES
o Target keylock position set to *MANUAL if not a Cloud environment
o Cloud environments must use the system value QIPLTYPE

Change Full System FlashCopy CSE Data

Supply all required values, press Enter.

Target LPAR backup command #1 *NONE

+ for more values . . .

Wait for ENDFSFLASH . . . . . *YES *YES, *NO

FlashCopy Exit program . . . . *NONE *NONE, program name
Library . . . . . . . . . . library name
Hold scheduled jobs . . . . . *YES *YES, *NO
Target keylock position . . . *MANUAL *PANEL, *AUTO, *MANUAL
Stop target after backups . . *NO *YES, *NO, *RMV
Source lock wait time out . . 3600 300 - 108000 (seconds)
Target lock wait time out . . 86400 300 - 108000 (seconds)

Auto start cluster . . . . . . *YES *YES, *NO

Message Queue . . . . . . . . *SYSOPR name, *SYSOPR
Library . . . . . . . . . . library name
More...
F1=Help F3=Exit F4=Prompt F6=Query HMC F12=Cancel

40
Save the Copy Services Environment (CSE) Data on IBM Technology
Expert Labs
both Controllers
o The command SAVDDD will put all the DDD and CRG information into an IFS directly.
o SAVDDD *GEN
o Will save to /QIBM/QZRDHASM/QZ_DDDBKU_<timestamp>
o BRMS *LINK will save /QIBM/QZRDHASM
o Consider the following two control group exits prior to *LINK:
o RMVDIR DIR('/QIBM/QZRDHASM/QZ_DDDBKU_*') SUBTREE(*ALL)
o SAVDD
o In the even of a restore, create the directory and issue RSTDDD
o Add the restore command to the end of the recovery report. It should be done after re-establishing the cluster and DDD.

41
 IBM Technology
Test the configuration on either Controller Expert Labs

o Use CHKFSFLASH to verify communications configurations first

o Resolve issues until it is successful

CHKFSFLASH CSEDTA(FSFC9M9P1)
Acquired lock on LPAR CTCIHA9M.
Validating flashcopy consistency group 8
Validating flashcopy mappings
Consistency group 8 validated.
Successfully performed local verifications.
Performing Full System FlashCopy verifications on CTCIHA9M.
Released lock on LPAR CTCIHA9M.
Log file used : /QIBM/Qzrdhasm/fsfc/FSFC9M9P1/[Link].
CHKFSFLASH validation for FSFC9M9P1 completed successfully.

42
 IBM Technology
Before you test the Flash Copy Expert Labs

o On the production LPAR, do QCTL and QSYSWRK have prestart or autostart jobs?
o DSPSBSD SBSD(QSYS/QCTL) options 3 and 10
o DSPSBSD SBSD(QSYS/QSYSWRK) options 3 and 10
o If YES then you’ll need to use the exit program QZRDIAFFEX
o Remove them at *QUIESCE and add them back at *POSTFLASH (include MONMSG CPF0000)
o RMVAJE SBSD(QSYS/QCTL) JOB(jobname)
o RMVPJE
o ADDAJE SBSD(QSYS/QCTL) JOB(jobname) JOBD(job description)
o ADDPJE
o Add them to the BRMS Recovery report (we’ll do this later when we edit QO1AUSRRCY)
o Update the CSE Data to call the exit program
o If this is a Cloud environment, ensure the target will IPL in manual mode the first time:
o CHGSYSVAL QIPLTYPE VALUE('1')
o Did you add RUNLPARCMD to the startup program?
o On the target LPAR, do any comm adapters (virtual and physical) have the same slot numbers (Cxx) as the comm adapters on the
source LPAR?
o If yes, move them to other slots
o This will prevent the OS from using them with the existing line descriptions. 43
 IBM Technology
Test the configuration on either Controller Expert Labs

o Did you read the previous slide?

o Use STRFSFLASH to perform a flashcopy
o The target will IPL into manual mode
o Sign on to the target LPAR console
o Continue to IPL the LPAR to restricted state.
o When you’ve got a command line, verify the startup program is QZRDHASM/QZRDIASTRP
o DSPSYSVAL QSTRUPPGM
o Continue the IPL
o STRSBS QCTL
o Get the information needed for the communications interface
o DSPHDWRSC *CMN
o Get the location code
o Get the information needed for the tape devices
o DSPHDWRSC *STG
o Get the serial number
o On the target, execute command QZRDHASM/ENDFSFLASH to finish the process
o On the production LPAR change, if a cloud environment: CHGSYSVAL QIPLTYPE VALUE('0')
44
 IBM Technology
Update the communication information on either Expert Labs
Controller
o Using CHGCSEDTA, update:
o Communications interface location code
o *LPAR will results type, model, serial and virtual bus on the target
o Cxx = slot number xx
o Line description
o IP Address

Change Full System FlashCopy CSE Data

Supply all required values, press Enter.

Target Comm Interfaces:

Identifier Type . . . . . *LOC *SRLN, *LOC, *NONE

IO card identifier . . . . *LPAR-C2-T1

Line Description . . . . . FSFCLINE line name, *VIRTUALIP

IO card IP interface . . . [Link] IPv4 address

+ for more values . .

Target LPAR default route:

Binding interface . . . . *NOCHANGE IPv4 address
Next hop . . . . . . . . . IPv4 address

45
 IBM Technology
Update the backup devices on either Controller Expert Labs

o Using CHGCSEDTA, update:

o Device descriptions that the backups will use
o Serial numbers of the devices
o If using logical libraries, use the tape drive serial numbers

Change Full System FlashCopy CSE Data

Supply all required values, press Enter.

Target LPAR Device Setup:

Backup device description TS3400PROD *NONE, device name

Device serial number . . . 78-78F1101

+ for more values . .

46
 IBM Technology
Update the BRMS information on either Controller: Expert Labs
BRMS
o Using CHGCSEDTA, update:
o BRMS Integration = *YES
o Change the defaults if necessary
o Custom SYSBAS Timestamp is recommended if incremental backups are performed on the FlashCopy LPAR
o BRMS Advanced Feature is required to use Custom SYSBAS Timestamp

Use BRMS integration . . . . . *YES *YES, *NO

BRMS information:
BRMS transfer method . . . . *ALL *ALL, *CHGONLY
BRMS transfer port . . . . . *DFT *DFT, 1024-65535
Encrypt BRMS transfer . . . *NO *YES, *NO
Custom SYSBAS Timestamp . . *NONE *NONE, *BOTH, *QSYS *IFS
Lock BRMS . . . . . . . . . *SRCONLY *BOTH, *NO, *SRCONLY,
*TGTONLY
Lock type . . . . . . . . . *FCNUSG *ALL, *FCNUSG, *HOLD
Base media class . . . . . . *NONE class, *NONE
Base media class suffix . . *NONE suffix, *NONE
Restricted media class(es) *NONE *NONE, class
+ for more values . . .

47
Update the configuration on either Controller: BRMS IBM Technology
Expert Labs

o Specify a BRMS command

o If SBMJOB(*YES) then specify a job description that ensures it will run (i.e. if QBATCH isn’t started don’t send it to QBATCH)
o JOBD(QLPARJOBD) sends it to job queue QSYSNOMAX which sends to QSYSWRK
o The command is called from QSTRUPJD which usually runs under QPGMR.
Target LPAR backup command STRBKUBRM CTLGRP(BACKUPS) SBMJOB(*YES or *CTLSBS)

o Prompting (F4) is available on the command

But … it is prompted on the local (controlling) LPAR, not where the command will run (on the source LPAR). Select a default
control group (like *BKUGRP) then change it to the desired control group after pressing enter.

o At 7.5 BRMS changed the functional usage to be more restrictive. As a result, QPGMR does not have the authority to run BRMS
commands. Therefore, one of the following must be done:
o On the source LPAR, issue SETUSRBRM USER(QPGMR) USAGE(*ADMIN)
o Change the ‘Target LPAR backup command’ to:

SBMJOB CMD(STRBKUBRM CTLGRP(BACKUPS) SBMJOB(*YES or *CTLSBS)) JOB(BRMBACKUP) JOBQ(QSYS/QSYSNOMAX)

USER(QLPAR)
48
 IBM Technology
Update the configuration on either Controller: Expert Labs
Keylock
o Using CHGCSEDTA, update:
o Target keylock position = *AUTO
o Note: This parameter is not available in Cloud environments

Change Full System FlashCopy CSE Data

Supply all required values, press Enter.

Wait for ENDFSFLASH . . . . *YES *YES, *NO

FlashCopy Exit program . . . *NONE
Library . . . . . . . . . *LIBL *LIBL, library
Hold scheduled jobs . . . . *YES *YES, *NO
Target keylock position . . *AUTO *PANEL, *AUTO, *MANUAL
Stop target after backups *NO *YES, *NO, *RMV

Request type . . . . . . . . 0 Number

Auto start cluster . . . . . *YES *YES, *NO
Wait time . . . . . . . . . 0 Number of seconds
Message Queue . . . . . . . *SYSOPR name, *SYSOPR
Library . . . . . . . . . library name

Text . . . . . . . . . . . .

49
 IBM Technology
BRMS Changes on the Source LPAR Expert Labs

o Change the system policy to allow backups in batch:

o WRKPCYBRM *SYS, Option 1, Page down
V7R3M0 Change System Policy CTCIHA9L

Type choices, press Enter.

End all subsystems options

Controlled end delay time . . . . . . 1200 1-99999 seconds,*IMMED
Abnormal end delay time . . . . . . . *NOLIMIT 10-999 minutes,*NOLIMIT
End servers wait time . . . . . . . . 0 0-9999 seconds
Controlling subsystem:
Allow backups in batch . . . . . . . . *YES *NO, *YES
Restricted state time limit . . . . . *NOMAX 5-9999 minutes, *NOMAX
Allow alternate input media. . . . . . . *YES *NO, *YES
Volume prefix . . . . . . . . . . . . . Prefix
Enable Full System FlashCopy . . . . . . *YES *NO, *YES
BRMS submitted jobs:
Job description. . . . . . . . . . . . *USRPRF Name, *USRPRF
Library. . . . . . . . . . . . . . . Name, *LIBL, *CURLIB
Job queue. . . . . . . . . . . . . . . *JOBD Name, *JOBD
Library. . . . . . . . . . . . . . . Name, *LIBL, *CURLIB
BRMS flight recorder size . . . . . . . 050 001-999 megabytes 50
 IBM Technology
Specify an Output Queue Expert Labs

o Change the system policy to specify output for joblogs from the backups:
o WRKPCYBRM *SYS, Option 1, Page down
o If no output queue is specified the toolkit will create and use QUSRBRM/QZOUTQ
o Use ENDFSFLASH *TAGJOBLOG on the target in a job to bring back its joblog to the source LPAR

V7R4M0 Change System Policy CTCHADV1

Type choices, press Enter.

Media policy . . . . . . . . . . . . . . ULTRIUM3 Name, F4 for list

Devices . . . . . . . . . . . . . . . . TAPMLB01 Name, F4 for list

Home location for media . . . . . . . . LOST Name, F4 for list

Media class . . . . . . . . . . . . . . ULTRIUM3 Name, F4 for list
Sign off interactive users . . . . . . . *NO *YES, *NO
Sign off limit . . . . . . . . . . . . . 30 0-999 minutes
Output queue . . . . . . . . . . . . . . *PRTF Name, *PRTF
Library . . . . . . . . . . . . . . . Name, *LIBL
Day start time . . . . . . . . . . . . . [Link] Time
Media monitor . . . . . . . . . . . . . *YES *YES, *NO
Shared inventory delay . . . . . . . . . 60 30-9999 seconds
Auto enroll media . . . . . . . . . . . *NO *NO, *YES
Default usage. . . . . . . . . . . . . . *YES *NO, *YES
More...
F3=Exit F4=Prompt F5=Refresh F12=Cancel

51
 IBM Technology
BRMS Changes on the Source LPAR Expert Labs

o Modify the control group to call the toolkit exit program

o WRKCTLGBRM, Opt 8, page down to Backup item exit program
o Set the exit program to QZRDHASM/QZBRMSEXIT format BKUI0100
o IBM i OS releases prior to V7R5M0 must use format BKUI0100

Change Backup Control Group Attributes

Group . . . . . . . . . . . . . . . . : TESTEXIT

Type information, press Enter.

Backup item exit program . . . . . . . . QZBRMSEXIT Name, *NONE, *BKUPCY

Exit program library . . . . . . . . QZRDHASM Name
Exit program format . . . . . . . . . *BOTH BKUI0100, BKUI0200, *BOTH

o Note this information: [Link]

52
 IBM Technology
BRMS Changes on the Source LPAR - Subsystems Expert Labs

o If processing subsystems, subsystems should NOT be set to start

o WRKCTLGBRM, Opt 9

Subsystems to Process

Use . . . . . . . . . : *BKU
Control group . . . . : SAVSYSALL

Type choices, press Enter.

End
Seq Subsystem Library Option Delay Restart

10 *ALL *ALL *CNTRLD 30 *NO

53
 IBM Technology
BRMS Changes on the Source LPAR - Attributes Expert Labs

o Do not run STRMNTBRM or manage servers after control group

o WRKCTLGBRM, Opt 8, page down all the way
o Editable on V7R4M0, use OpsNav or API’s on prior releases

Additional Backup Policy Properties

Client backup policy . . . . . . . . . : SAVSYSALL

Type information, press Enter.

Allow activity overrides . . . . . . . . *YES *NO, *YES

Allow retention overrides . . . . . . . *YES *NO, *YES
Additional management:
TCP/IP servers . . . . . . . . . . . . *NO *NO, *END, *RESTART, *BOTH
Lotus servers . . . . . . . . . . . . *NO *NO, *END, *RESTART, *BOTH
Integrated Windows servers . . . . . . *NO *NO, *VARYOFF, *VARYON ...
Guest partitions . . . . . . . . . . . *NO *NO, *VARYOFF, *VARYON ...
Unmount user-defined file systems . . . *NO *NO, *YES
Run maintenance after backup . . . . . . *NO *NO, *YES

54
 IBM Technology
Modify BRMS recovery report user-added steps Expert Labs

o Insert custom message into the recovery reports to change system settings to start IP etc.
o STRSEU SRCFILE(QUSRBRM/QO1AUSRRCY) SRCMBR(STEP014)
o Insert the following text:
If restoring configuration settings from a FSFC backup, run the following commands:
CHGSYSVAL SYSVAL(QSTRUPPGM) VALUE('QSTRUP QSYS ')
CHGLINETH LIND(ETHLINE) ONLINE(*YES)
CHGTCPIFC INTNETADR('[Link]') AUTOSTART(*YES)
CHGIPLA STRTCP(*YES)  skip this if also using FSR
Check the job scheduler entries

o Modify the recovery report creation to include the user info

o Add the parameter USRRCYINF(*ADD)
o If STRMNTBRM is used to generate the reports
o Modify the STRMNTBRM call with PRTRCYRPT(*NONE)
o Add STRRCYBRM USRRCYINF(*ADD) to the job scheduler, to run 15 minutes (or so) after STRMNTBRM
o Consider using the Flashcopy Exit Program
o WRKMBRPDM QZRDHASM/QCLSRC member QZRDIAFFEX
o Copy the source files to utility libraries
o Compile a blank program for the controller, and one that calls STRMNTBRM and STRRCYBRM at exit
*FINISH on the source
55
 IBM Technology
Create an exit program to run BRMS Maintenance Expert Labs
and generate reports
o On the source:
o Copy the source from QZRDHASM/QCLSRC member QZRDIAFFEX to your tools library.
o WRKMBRPDM to edit it. At the *FINISH section add this code:
IF COND(%SST(&EXTRAS 1 8) *EQ '*SUCCESS') THEN(DO)
STRMNTBRM PRTRCYRPT(*NONE)
STRRCYBRM USRRCYINF(*ADD)
ENDDO

o Compile the program

o On the Controller create an empty version of that program:

CRTCLPGM PGM(QGPL/QZRDIAFFEX) SRCFILE(QZRDHASM/QCLSRC) SRCMBR(QZRDIAFFEX)

o Use WRKCSEDTA opt 2 and set the exit program name

56
 IBM Technology
Test the configuration on either Controller Expert Labs

o Use CHKFSFLASH to verify communications configurations first

o Resolve issues until it is successful

CHKFSFLASH CSEDTA(FSFC9M9P1)
Acquired lock on LPAR CTCIHA9M.
Validating flashcopy consistency group 8
Validating flashcopy mappings
Consistency group 8 validated.
Successfully performed local verifications.
Performing Full System FlashCopy verifications on CTCIHA9M.
Released lock on LPAR CTCIHA9M.
Log file used : /QIBM/Qzrdhasm/fsfc/FSFC9M9P1/[Link].
CHKFSFLASH validation for FSFC9M9P1 completed successfully.

57
 IBM Technology
Test the configuration on either Controller Expert Labs

o Use STRFSFLASH to perform a flashcopy

o Flashcopy target LPAR IPL etc will occur
o If this is a SAVSYS backup then the HMC SRC will be A900 3C70 while in Batch Restricted State
o After backups, BRMS will be transferred to the source LPAR
o If not, check /tmp/[Link] on the target
o On the source LPAR, verify backups are complete
o DSPLOGBRM
o WRKMEDIBRM
o BRMS Recovery reports
o Look for the customer recovery steps after RSTCFG.
o This is usually step 14 – if not, find the correct step and move the text in member QUSRBRM/QO1AUSRRCY STEP014 to
the correct member.

58
 IBM Technology
Schedule Log Cleanup on all Controlling LPARs Expert Labs

o CLEANLOGS will prune toolkit logs to save on space

o Tell it how many days of log entries to retain
o ADDJOBSCDE JOB(CLEANLOGS) FRQ(*WEEKLY) CMD(QZRDHASM/CLEANLOGS RETAIN(120)) SCDDATE(*NONE) SCDDAY(*ALL)
SCDTIME('22:30')

Clean Toolkit Logs (CLEANLOGS)

Type choices, press Enter.

Days of information to retain . *NONE, days

Additional Parameters

FSFC environment . . . . . . . . *NONE Name, *NONE, *ALL

59
 IBM Technology
How to reset after failure Expert Labs

o Failures can happen, you need to know how to set things back to normal.
o To abandon the backups:
o On the target: QZRDHASM/ENDFSFLASH *FAILBKU
o On the source: QZRDHASM/ENDFSFLASH *RSTFCNUSG
o PowerVS Volumes-clones can be displayed, canceled and deleted with these commands:
o DSPCLDCLNL
o CNLCLDCLN
o DLTCLDCLN
o WRKCLDVOL
o The toolkit webpage contains additional recovery steps

60
 IBM Technology
Where can I find the logs for troubleshooting? Expert Labs

o Logs are in the following place:

o /QIBM/Qzrdhasm/[Link]
o /QIBM/Qzrdhasm/fsfc/<CSE Data name>/*
o /QIBM/Qzrdhasm/[Link]
o /QIBM/Qzrdhasm/[Link]/*
o /QIBM/Qzrdhasm/joblogs/*
o DMPINF ENV(*ALL) EXTDLOGS(*YES) will grab all these files and put them in a zip file.
o Specify the failing job information on Job Name:

Dump ICSM Information (DMPINF)

Type choices, press Enter.

Environment name . . . . . . . . *ALL Name, *ALL

Type . . . . . . . . . . . . . . *ALL *ALL, *FLASH, *GMIR, *LUN...
Extended logging . . . . . . . . *YES *YES, *NO
Job name . . . . . . . . . . . . *NONE Name, *CURRENT, *NONE, *LAST
User . . . . . . . . . . . . . Name
Number . . . . . . . . . . . . 000000-999999
Days of logs to keep . . . . . . 90 days, *NONE, *NOMAX 61
 IBM Technology
Contacting support if you have problems Expert Labs

o Support for the FSFC Toolkit is to customers who meet the following criteria:
o Current System i Software Maintenance Agreement
o Current FSFC Toolkit Software Maintenance Agreement
o For non-urgent issues or questions contact the consultant who installed the Toolkit. To reach a Toolkit developer for non-urgent issues
and questions, or to report a bug, send an email to iessspt@[Link]

o For immediate 24x7 assistance, reach out to IBM Support:

o US: [Link]
o Worldwide: [Link]
o To assist IBM personnel in correctly routing your problem, request support for the iSeries
o Lab Services “Copy Services Toolkit – Full System Flashcopy” using component
o identifier 5798CST00.

62
IBM Technology
Expert Labs

63