https://emc--c.na5.visual.force.com/apex/KB_HowTo?
id=kA0700000004Xbm
Goal
Environment
Resolution
This KB is designed to Understand and Troubleshoot VBA Backup Failures and provide a Resolution Path to fix all
issues with NetWorker VMware Backup Protection Policies.
VBA 1.0.x
VBA 1.1.x
AVE 7.0.60-5
AVE 7.1.60-12
NetWorker 8.1.x
NetWorker 8.2.x
Document each step performed during the flow of troubleshooting for a quicker understanding of the issue by SME or Engineering,
if the issue needs an escalation.
The VBA Backup Failures should be tracked as per the sequence of events as discussed below:
1. Check NetWorker Management Console (NMC) for any specific Client failures. Points to be noted at this stage:
Names of the VM clients failed
Proxy names associated with each failed VM client
Device associated with each failed VM client
Any other VM client successful to same proxy / or same device? (Yes/No - details)
Figure 1
2> A backup job initiates NetWorker Policy that initiated the below processes from NetWorker:
Nsrd
nsrpolicy
nsrjobd
nsrexecd
nsrvba_save (libnsrvba library is used)
All the above processes are logged in daemon.raw logs on NetWorker server
For Windows server, it is present in <Installation Directory>\EMC NetWorker\nsr\logs\daemon.raw
For Linux/AIX server, is present in /nsr/logs/daemon.raw
To render the daemon.raw logs, follow KB 163411
For a complete list of relevant logs associated with VBA, see KB 199256.
nsrvba_save pulls information from Jobs Database of NetWorker server. This is logged in Policy Logs on NetWorker Server.
For Windows server, it is present in <Installation Directory>\EMC NetWorker\nsr\logs\<policyname>\<nsrvba_save_job-id>
For Linux/AIX server, is present in /nsr/logs/<policy-name>\<nsrvba_save_job-id>
For a complete list of relevant logs associated with VBA, see KB 199256.
Any failures at this stage need further review. Collect the information from above logs and proceed to next step. DO NOT escalate at this
point.
The nsrpolicy invokes the NSR Web Service(nsrvmwsd)to start a Backup Job
Figure 2
nsrvmwsd sends a call to VBA Web Service (managed by tomcat instance on VBA) on port 8543 (can be seen using
keywork 'nsrvmwsd' indaemon.raw logs)
VBA confirms the call in return to NSR Web Service on port 8080
This is logged on VBA appliance at /usr/local/avamar-tomcat/logs/catalina.<date>.log
For a complete list of relevant logs associated with VBA, see KB 199256.
VBA Web Service calls MC SDK (Avamar MCS takes over here) internally on VBA appliance
This is logged on VBA appliance at /usr/local/avamar/var/mc/server_log/mcserver.log.* [0-49]
If any errors found at this stage, collect VBA Logs using KB 199026 and immediately escalate to Avamar Support
using KB 199024 (Client/Proxy logs can be skipped).
For a complete list of relevant logs associated with VBA, see KB 199256.
MC SDK invokes 'mccli group start' that has now know which VM client to backup (stored in MCDB). At this time, the actual backup starts
and Avamar has full control of the Backup Job
This is logged on VBA appliance at /usr/local/avamar/var/ebr/server_log/ebr-server.log
If any errors found at this stage, collect VBA Logs using KB 199026 and immediately escalate to Avamar Support
using KB 199024 (Client/Proxy logs can be skipped).
For a complete list of relevant logs associated with VBA, see KB 199256.
4. Create, Assign and Perform Backup Task
NwBackupJobBusiness: Started backup now for group creates a work order which is sent to the avagent on VBA proxy
(responsible to receive work orders). It is logged in the log: /usr/local/avamarclient/var/avagent.log
VBA Proxy is selected by the VBA node depending on the available sessions. By default, the VBA node has one proxy server within itself. One proxy
server can handle a maximum of eight parallel sessions. If there are more than eight sessions, a different free proxy is assigned.
If any errors found at this stage, collect VBA Logs using KB 199026 and immediately escalate to Avamar Support using KB 199024 (Client/Proxy
logs REQUIRED).
For a complete list of relevant logs associated with VBA, see KB 199256.
avagent invokes avnwcomm process, responsible to give process updates to NetWorker Server. This is logged on proxy server at:
[Also called as Plugin Logs] /usr/local/avamarclient/var/<VPP-name>-<WO-ID>-<CID>-<Plugin-ID>.log, where
VPP-name = VMware Protection Policy Name
WO-ID = WorkOrder ID generated for each backup/restore job by mccli
CID = Unique Client ID registered with VBA Appliance (See KB 199312 to find CID for troubleshooting)
DO NOT escalate on errors at this stage and continue with next steps to review the details.
Further, avnwcomm invokes the VBA plugin avvcbimage that performs all Snapshot related tasks on the desired VM Client. This is logged on proxy
server at:
[Also called as REAL Plugin Logs] /usr/local/avamarclient/var/<VPP-name>-<WO-ID>-<CID>-<Plugin-ID>_real.log,
where
VPP-name = VMware Protection Policy Name
WO-ID = WorkOrder ID generated for each backup/restore job by mccli
CID = Unique Client ID registered with VBA Appliance (See KB 199312 to find CID for troubleshooting)
Note that, this log have an extra _real keyword as post-fix.
Backup session and save job is created now on NetWorker
The Plugin logs are still updated simultaneously while avnwcomm keeps updating NetWorker server.
To enable debug on avmwcomm logs, follow KB 195367.
DO NOT escalate on errors at this stage and continue with next steps to review the details.
Figure 3
avvcbimage has the full control of the Backup at this stage
avvcbimage contacts VMware and issues commands to perform a Snapshot. This is logged in REAL Plugin logs.
Once the snapshot is created, avvcbimage passes the control to avtar to perform a backup to device.
avtar is the Avamar process responsible to perform all read/write operations.
It logs the acvities in Avtar Logs:
/usr/local/avamarclient/var/<VPP-name>-<WO-ID>-<CID>-<Plugin-ID>_avtar.log, where
VPP-name = VMWare Protection Policy Name
WO-ID = WorkOrder ID generated for each backup/restore job by mccli
CID = Unique Client ID registered with VBA Appliance (See KB 199312 to find CID for troubleshooting)
Note that, this log have an extra _avtar keyword as post-fix.
If any errors found at this stage, collect VBA Logs using KB 199026 and immediately escalate to Avamar Support
using KB 199024 (Client/Proxy logs REQUIRED).
For a complete list of relevant logs associated with VBA, see KB 199256
5> Data Domain Integration
Most VM Backups are backed up on Data Domain devices using VBA NetWorker VMware Protection.
Client Backup metadata is stored on VBA
Client Backup Data is directly written on Data Domain via DDBoost
Snapshot of VM is copied to:/avamar-mtree/STAGING/<CID>/BACKUP-<timestamp>/
Once copy is finished, backup files are moved to permanent location: /avamar-mtree/cur/<CID>/<timestamp_of_backup>/
These can be seen in the Avtar logs on proxy server: /usr/local/avamarclient/var/<VPP-name>-<WO-ID>-<CID>-<PluginID>_avtar.log
VBA stores the files in Avamar MTREE in "RAW" format, the real VMDK files with a different name given by DDOS internally. See Figure 3.
The Backup Process further copies these "RAW" format files from Avamar MTREE to a CDSF package. This CDSF package is exported to NetWorker
MTree. See Figure 3.
The CDSF export job is logged on the proxy server at:
/usr/local/avamarclient/var/<VPP-name>-<WO-ID>-<CID>-<Plugin-ID>_cdsf_export.log, where
VPP-name = VMware Protection Policy Name
WO-ID = WorkOrder ID generated for each backup/restore job by mccli
CID = Unique Client ID registered with VBA Appliance (See KB 199312 to find CID for troubleshooting)
Note that, this log have an extra _cdsf_export keyword as post-fix.
If any errors found at this stage, collect VBA Logs using KB 199026 and immediately escalate to Avamar Support
using KB 199024 (Client/Proxy logs REQUIRED).
For a complete list of relevant logs associated with VBA, see KB 199256.
Note that, the errors in cdsf_export logs may need Data Domain Team's assistance, This will be initiated by Avamar Team.
