Technical White Paper

Dell EMC SRM: Automating Incident

Management in ServiceNow
Abstract
This white paper describes the Dell EMC™ SRM integration with ServiceNow®
IT Service Management application.

July 2021

H18840
Revisions

Revisions
Date Description
July 2021 Initial release

Acknowledgments
Author: Dejan Stojanovic

The information in this publication is provided “as is.” Dell Inc. makes no representations or warranties of any kind with respect to the information in this
publication, and specifically disclaims implied warranties of merchantability or fitness for a particular purpose.

Use, copying, and distribution of any software described in this publication requires an applicable software license.

Copyright © 2021 Dell Inc. or its subsidiaries. All Rights Reserved. Dell Technologies, Dell, EMC, Dell EMC and other trademarks are trademarks of Dell
Inc. or its subsidiaries. Other trademarks may be trademarks of their respective owners. [19-Jul-21] [Technical White Paper] [H18840]

2 Dell EMC SRM: Automating Incident Management in ServiceNow | H18840

Table of contents

Table of contents
Revisions.............................................................................................................................................................................2
Acknowledgments ...............................................................................................................................................................2
Table of contents ................................................................................................................................................................3
Executive summary .............................................................................................................................................................4
1 Introduction ...................................................................................................................................................................5
1.1 Dell EMC SRM....................................................................................................................................................5
1.2 ServiceNow ITSM ...............................................................................................................................................6
2 SRM alerting .................................................................................................................................................................8
2.1 Alerting concept ..................................................................................................................................................8
2.2 What is an alert? .................................................................................................................................................9
2.3 Alert consolidation ..............................................................................................................................................9
2.4 Types of events ................................................................................................................................................10
3 ServiceNow scripted REST API configuration ...........................................................................................................11
3.1 Create incident test ...........................................................................................................................................17
3.2 Close incident test ............................................................................................................................................22
4 SRM alert notification configuration............................................................................................................................26
4.1 SRM alert definition with external process action .............................................................................................26
4.2 SRM alert definition with webhook action .........................................................................................................30
5 Examples ....................................................................................................................................................................34
5.1 SRM new alert ..................................................................................................................................................34
5.2 SRM close alert ................................................................................................................................................37
5.3 SRM update alert ..............................................................................................................................................39
6 Technical support and resources ...............................................................................................................................40

3 Dell EMC SRM: Automating Incident Management in ServiceNow | H18840

Executive summary

Executive summary

This document describes details of Dell EMC Storage Resource Manager (SRM) application integration with ServiceNow
incident management process. Integration enables ServiceNow to automatically open, update, close and re-open incidents
based on alerts originated by SRM.

4 Dell EMC SRM: Automating Incident Management in ServiceNow | H18840

Introduction

1 Introduction

1.1 Dell EMC SRM

Dell EMC Storage Resource Manager (SRM) is a comprehensive on-prem storage infrastructure monitoring and reporting
solution helping IT organizations to visualize, analyze and optimize their end-to-end heterogeneous storage environments
from a single pane of glass.

SRM monitors storage inventory, health, capacity, performance and workload, data protection and configuration details,
along with the connected resources like hosts and fabric switches, to generate proactive alerts and actionable reports.

SRM data collection and reporting are completely automated, allowing you to focus on business needs and spend less time
manually collecting, normalizing, and aggregating key operational metrics.

The five key use cases in SRM are workload analysis, configuration compliance, capacity planning, chargeback reporting
and performance troubleshooting.

SRM use cases

5 Dell EMC SRM: Automating Incident Management in ServiceNow | H18840

Introduction

Workload Analysis helps you decide where to place a storage workload, for the best capacity / price / performance
combination. This also helps storage teams to identify underutilized and overutilized resources, allowing them to rebalance
existing workload to maintain performance and sustain their storage growth.

Configuration Compliance ensures that the customer’s environment is properly configured according to their own stack,
industry best practices, and with the Dell EMC Support Matrix. It also tracks configuration changes and determines if recent
changes have potentially placed data or business at risk.

Global Capacity Planning dashboards saves you hours by eliminating manual data collection and reporting. Capacity
Usage Trends and Forecasts provides information on when additional storage will be needed. This visibility ultimately helps
control storage consumption and improves ROI.

With its Chargeback capability, SRM tracks the true storage costs, supporting service levels, replication, and applications.
This data helps storage teams communicate the value of the storage services provided to users.

Performance Troubleshooting helps you troubleshoot performance issues and identify bottlenecks.

To achieve the above listed use cases, SRM is consisted of variety of modules, and the one responsible for the integration
with ServiceNow is Alerting.

1.2 ServiceNow ITSM

ServiceNow ITSM solution helps infrastructure and operations organizations manage the consumption of IT services, the
infrastructure that supports the IT services and the IT organization’s responsibility in delivering business value with these
services. These are most heavily used by IT service desks and IT service delivery functions to support the tasks and
workflows for processes including incident, request, problem, change, service level, knowledge, and configuration
management.

We will focus on ServiceNow Incident Management process, which has embedded workflows to identify, track, and resolve
high‑impact incidents.

Incident Management is responsible for managing the life cycle of incidents, from creation to closure.

It has several states which can be represented in a diagram as follows:

Incident management state model flow

6 Dell EMC SRM: Automating Incident Management in ServiceNow | H18840

Introduction

Incident states:

State Description
New Incident is logged but not yet investigated.
In Incident is assigned and is being investigated.
Progress
On Hold The responsibility for the incident shifts temporarily to another entity to provide further information,
evidence, or a resolution. When you select the On Hold option, the On hold reason list appears. If
the On hold reason is Awaiting Caller, the Additional comments becomes mandatory.

Note: If the caller updates the incident, the On hold reason field is cleared and the state of the
incident is changed to In Progress. An email notification is sent to the user whose name is mentioned
in the Assigned to field as well as to the users in the Watch list. An incident can be placed in the On
hold state one or more times prior to being closed.
Resolved A satisfactory fix is provided for the incident to ensure that it does not occur again.
Closed Incident is marked Closed after it is in the Resolved state for a specific duration and it is confirmed
that the incident is satisfactorily resolved.
Canceled Incident was triaged but found to be a duplicate incident, an unnecessary incident, or not an incident at
all.

Purpose of the SRM’s integration with ServiceNow Incident Management process is to automate the incident state change:

• Create incident or reopen closed incident: set incident state to New (state=1)
• Close incident: set incident state to Closed (state=7)

All the other incident states from the above table are managed from within ServiceNow and are not affected by SRM.

7 Dell EMC SRM: Automating Incident Management in ServiceNow | H18840

SRM alerting

2 SRM alerting

2.1 Alerting concept

Dell EMC SRM alerting module provides consolidated and customizable view of all alerts regarding infrastructure health
and availability, storage capacity threshold crossings for all storage types and assets, and compliance to service level
agreements.

Alerts are available in the global alerting reports as well as in the device-specific Report Library.

SRM All Alerts report

Every SolutionPack comes with appropriate normalization of alerts that are collected from its respective devices. Below is
the view of some Alert definitions within SRM Admin UI.

8 Dell EMC SRM: Automating Incident Management in ServiceNow | H18840

SRM alerting

SRM solution pack alert definitions

2.2 What is an alert?

In Dell EMC SRM, an event refers to a raw occurrence of any sort that is received or generated in any way. An alert is an
occurrence that is normalized into Dell EMC SRM and has a valid alert definition in the Alerting Frontend. The alert
definitions add the information to the Events database, making the alert available in the alerting reports.

Specialized alert definitions are provided for notifications, such as adding messages to logs, sending SNMP traps to fault
management system, sending email notifications or creating/closing incident in ITSM application.

Only the alerts that get written to the SRM events database appear in the Dell EMC SRM alerting reports and are candidate
to be sent as notification to external systems.

2.3 Alert consolidation

Alert consolidation is the Dell EMC SRM feature that collects events from various sources and normalizes them into the
same format so they can appear together in the same global alerting reports in the Dell EMC SRM UI.

Regardless of the device type, the event source, the type of event, or whether the alert was predefined or is a custom alert,
all normalized events are handled the same in Dell EMC SRM. All events appear together in the All Alerts report. SRM web
portal users can filter, sort, and manage all of them from the same interface and same report.

9 Dell EMC SRM: Automating Incident Management in ServiceNow | H18840

SRM alerting

2.4 Types of events

Alert consolidation collects and normalizes four types of events. The out-of-the-box supported event types are:

1. Metric-based threshold events

Metric-based threshold events are generated based on metrics that are collected by Dell EMC SRM collectors. If the
collected data matches conditions in an alert definition, the event becomes an alert.

2. Events sent to a listener port

Events that are sent to a listener port are events that originate outside of Dell EMC SRM, such as on a device or a network.
The source sends the event to a Dell EMC SRM Event Listener. The event data is normalized based on conversion rules.
If the resulting data matches conditions in an alert definition, it becomes an alert.

3. SNMP Traps

SNMP traps originate outside of Dell EMC SRM, such as on a device or network. The source sends the trap to a Dell EMC
SRM Trap Receiver. The trap data is normalized based on conversion rules. If the resulting data matches conditions in an
alert definition, it becomes an alert.

4. Events collected from a scheduled report

SRM web portal user can create an alert that is based on the reported or computed values in reports. For example, suppose
that a user wants to receive an email whenever storage capacity for a particular service level falls below a threshold. With
features on the reporting user interface, a user can configure an alert that does exactly that. If the resulting data matches
conditions in an alert definition, it becomes an alert.

Further details about the Dell EMC SRM alerting can be found in document Dell EMC SRM Alerting Guide, on dell.com
support web site.

10 Dell EMC SRM: Automating Incident Management in ServiceNow | H18840

ServiceNow scripted REST API configuration

3 ServiceNow scripted REST API configuration

To be able to trigger creation, update, closure and reopening of ServiceNow incidents from SRM alerting, you have to create
Scripted REST API resource, which will listen to HTTP POST requests from SRM and respond accordingly by creating,
updating, closing and reopening an incident.

Login to ServiceNow instance, type scripted rest api in the navigation tree search box and click on Scripted REST APIs
node in navigation tree on the left. Then click on New button to create new scripted rest api resource:

ServiceNow Scripted REST API - new

Type the name for the newly created scripted rest api service. Then click on Submit button:

ServiceNow Scripted REST API – name and API ID

11 Dell EMC SRM: Automating Incident Management in ServiceNow | H18840

ServiceNow scripted REST API configuration

Type the name of the newly created scripted REST API service SRM alerts in the search box. Newly created REST API
service will appear in the list below. Click on it:

ServiceNow Scripted REST API - created

Scripted REST API service configuration will appear with its namespace and base API path. Click on New button under the
Resources tab to create new resource for REST API:

ServiceNow Scripted REST API – new resource

12 Dell EMC SRM: Automating Incident Management in ServiceNow | H18840

ServiceNow scripted REST API configuration

Scripted REST API resource configuration will appear - type its name and select HTTP method (POST). Paste the javascript
content from below into the Script field and save the resource:

ServiceNow Scripted REST API – resource configuration

JavaScript:

// SRM alert to ServiceNow incident Scripted REST API script

// Incident is OPENED when SRM active=true & severity=[1-5] alert is received, and if
there is no existing incident. SNOW short_description field is set to SRM alertId, which is unique alert
id in SRM
// Incident is CLOSED when SRM active=false alert is received for the existing
short_description=alertId. No further updates to incident happen if same SRM close alert is received
repeatedly
// Incident is REOPENED when SRM active=true & severity=[1-5] alert is received for
the existing short_description=alertId. close_code, close_notes, reopen_count, reopened_by,
reopened_time are automatically updated
// Incident is UPDATED when SRM active=true & severity="N/A" or null alert is
received for the existing short_description=alertId. This is alert SRM update by the admin user in SRM
UI. SNOW user_input prop set with values of owner and ack

/* SRM alerts sent via webhook. Accepted values: null, "N/A", "", "string", "number". Not accepted:
empty value or any value except null without quotes

OPEN incident alert:

{

13 Dell EMC SRM: Automating Incident Management in ServiceNow | H18840

ServiceNow scripted REST API configuration

"alertId": "Unity-A007_Pool_Test-Pool93_Threshold Exceeded",

"severity": "1",
"eventType": "DURABLE",
"timestamp": "1623403895",
"systemName": "Unity-A007",
"systemType": "UnifiedArray",
"componentType": "Pool",
"componentName": "Test-Pool93",
"source": "Unity-GenericEvent",
"category": "Capacity",
"description": "Storage pool Test-Pool93 has exceeded its critical threshold of 95%",
"active":"true",
"acknowledged":"N/A",
"owner":"N/A"
}
CLOSE incident alert:
{
"alertId": "Unity-A007_Pool_Test-Pool93_Threshold Exceeded",
"severity": "1",
"eventType": "DURABLE",
"timestamp": "1623403895",
"systemName": "Unity-A007",
"systemType": "UnifiedArray",
"componentType": "Pool",
"componentName": "Test-Pool93",
"source": "Unity-GenericEvent",
"category": "Capacity",
"description": "Storage pool Test-Pool93 has exceeded its critical threshold of 95%",
"active":"false",
"acknowledged":"N/A",
"owner":"N/A"
}
REOPEN incident alert:
{
"alertId": "Unity-A007_Pool_Test-Pool93_Threshold Exceeded",
"severity": "1",
"eventType": "DURABLE",
"timestamp": "1623403895",
"systemName": "Unity-A007",
"systemType": "UnifiedArray",
"componentType": "Pool",
"componentName": "Test-Pool93",
"source": "Unity-GenericEvent",
"category": "Capacity",
"description": "Storage pool Test-Pool93 has exceeded its critical threshold of 95%",
"active":"true",
"acknowledged":"N/A",
"owner":"N/A"
}
UPDATE incident alert:
{
"alertId": "Unity-A007_Pool_Test-Pool93_Threshold Exceeded",
"severity":"N/A",
"eventType": "DURABLE",
"timestamp": "1623403895",
"systemName": "Unity-A007",
"systemType": "UnifiedArray",
"componentType": "Pool",
"componentName": "Test-Pool93",
"source": "Unity-GenericEvent",
"category": "Capacity",
"description": "Storage pool Test-Pool93 has exceeded its critical threshold of 95%",
"active":"true",

14 Dell EMC SRM: Automating Incident Management in ServiceNow | H18840

ServiceNow scripted REST API configuration

"acknowledged":"yes",
"owner":"Dejan"
}
*/
(function process( /*RESTAPIRequest*/ request, /*RESTAPIResponse*/ response) {
var requestBody = request.body;
var requestData = requestBody.data; // variable holds JSON data
var eventType = request.getHeader('x-srm-event');
var processedData = false;
var message = '';

var gr = new GlideRecord('incident');

switch (requestData.active) {

// Close incident - if SRM active=false and incident exists (get SNOW incident where
"short_description == alertId" --> assign close_code, close_notes, and set SNOW state property to 7
(closed))
// If SRM active=false alert with same alertId is sent repeatedly to SNOW after its
initial closure, no further updates will happen on incident - it will remain in "closed" state, with
initial "closedAt" timestamp
// If SRM active=false alert is not matched with SNOW incident, nothing happens
case "false":
gr.addQuery('short_description', requestData.alertId);
gr.query();
if (gr.getRowCount() == 1) { //there is a match, so update incident
while (gr.next()) {
gr.close_code = 'Solved (Permanently)';
gr.close_notes = 'SRM clear alert received, issue is resolved';
gr.state = 7; //closed state = 7
gr.comments = 'SRM alert closed';
gr.user_input = 'SRM alert closed';
gr.update();
processedData = true; //indicates that SNOW incident was updated
message = "Incident closed";
}
}
break;

// Open incident - SRM active=true & severity="1-5" and no existing incident with
"short_description == alertId"
case "true":
gr.addQuery('short_description', requestData.alertId);
gr.query();
if (gr.getRowCount() == 0 && requestData.severity != "N/A" &&
requestData.severity) { //there is no incident match, severity has value 1-5, so create incident
gr.initialize();

gr.active = true;
gr.short_description = requestData.alertId; //map SRM unique alertId to
SNOW short_description property
gr.description = requestData.description; //map SRM alert fullsmg
(description) to SNOW description property
gr.urgency = requestData.severity; //map SRM alert severity to SNOW
urgency
gr.impact = requestData.severity; //set impact to be the same as severity
gr.subcategory = 'SRM'; //tag incident with SRM string value
gr.cmdb_ci = requestData.systemName; //assign SRM device name to cmdb_ci
SNOW property
gr.comments = 'SRM alert created';
gr.user_input = 'SRM alert created';

gr.insert(); // Creates a new record

15 Dell EMC SRM: Automating Incident Management in ServiceNow | H18840

ServiceNow scripted REST API configuration

processedData = true; //indicates that SNOW incident was created

message = "Incident created";

// Update incident - SRM active=true & severity == "N/A" or null and incident
exists (new or closed state), then it is existing ACTIVE SRM alert update (ack or owner assigned from
SRM user in UI by right-click action)
} else if (gr.getRowCount() == 1 && (requestData.severity == "N/A" ||
requestData.severity === null)){

while (gr.next()) {
gr.active = true; //in case SRM alertId matches closed SNOW
incident, make it active again (can happen if SNOW closed incident which is still active on SRM)
gr.state = 1; //in case SRM alertId matches closed SNOW
incident, make its state=1 again
gr.comments = 'SRM user updated alert: acknowledged and/or
owner assigned'; //update SNOW comments property
gr.user_input = '';
if (requestData.owner){
gr.user_input = 'SRM alert owner assigned: ' +
requestData.owner + ' | '; //if owner property is set by SRM user, copy it to SNOW user_input property,
otherwise skip
}
if (requestData.acknowledged){
gr.user_input = gr.user_input + ' SRM alert
acknowledged: ' + requestData.acknowledged; //if acknowledged/unacknowledged is set by SRM user, set
user_input SNOW property, otherwise skip
}
gr.update();
processedData = true; //indicates that SNOW incident was
updated
message = "Incident updated";
}

}
// Reopen incident - SRM active=true & severity="1-5" and incident matched
(new or closed state). close_code, close_notes, reopen_count, reopened_by, reopened_time are
automatically updated
else {
while (gr.next()) {
gr.active = true;
gr.state = 1;
gr.comments = 'SRM alert reopened';
gr.user_input = 'SRM alert reopened';

gr.update();
processedData = true; //indicates that SNOW incident was
updated
message = "Incident reopened";
}
}
break;
}
if (processedData) {
response.setStatus(200);
return message;
} else {
response.setStatus(412);
message = 'No incident created/updated';
return message;
}
})(request, response);

16 Dell EMC SRM: Automating Incident Management in ServiceNow | H18840

ServiceNow scripted REST API configuration

3.1 Create incident test

Scripted REST API service configuration has now newly created resource highlighted below. Click on it to begin with explore
api test:

Create incident test - resource

17 Dell EMC SRM: Automating Incident Management in ServiceNow | H18840

ServiceNow scripted REST API configuration

While within the selected REST API resource, click on Explore REST API link highlighted:

Create incident test – explore REST API

Left side of REST API explorer will list selected resource details: namespace, API name and version. On the right side there
is URI of HTTP POST method listed, together with http headers:

Create incident test – request configuration

18 Dell EMC SRM: Automating Incident Management in ServiceNow | H18840

ServiceNow scripted REST API configuration

Paste the below json content into Request Body Raw field and click on Send to perform HTTP POST operation to selected
REST API resource – specific alert will trigger incident creation due to property active=true and non-existence of the same
alertId in incident database:

Create incident test – request body

{
"alertId": "Unity-A007_Pool_Test-Pool93_Threshold Exceeded",
"severity": "1",
"eventType": "DURABLE",
"timestamp": "1623403895",
"systemName": "Unity-A007",
"systemType": "UnifiedArray",
"componentType": "Pool",
"componentName": "Test-Pool93",
"source": "Unity-GenericEvent",
"category": "Capacity",
"description": "Storage pool Test-Pool93 has exceeded its critical threshold of 95%",
"active":"true",
"acknowledged":"N/A",
"owner":"N/A"
}

19 Dell EMC SRM: Automating Incident Management in ServiceNow | H18840

ServiceNow scripted REST API configuration

Explore REST API webapp will respond with below output which summarizes the request:

Create incident test – request summary

Scroll down and see complete response message. Status code must be 200 OK. Response Body message reflects the
specific action being performed within ServiceNow (Incident created):

Create incident test – response

20 Dell EMC SRM: Automating Incident Management in ServiceNow | H18840

ServiceNow scripted REST API configuration

Navigate to Incident/All in navigation tree and type in the incidents search field srm. Newly created incident will appear.
Click on it to open detailed view:

ServiceNow incident created

Details of the incident are provided below with state New, subcategory SRM, and short & detailed descriptions:

ServiceNow incident details

21 Dell EMC SRM: Automating Incident Management in ServiceNow | H18840

ServiceNow scripted REST API configuration

3.2 Close incident test

While within the selected REST API resource, click on Explore REST API link highlighted:

Close incident test – explore REST API

Paste the below json content into Request Body Raw field and click on Send to perform HTTP POST operation to selected
REST API resource – specific alert will trigger incident closure due to property active=false and existing alertId in incident
database:

22 Dell EMC SRM: Automating Incident Management in ServiceNow | H18840

ServiceNow scripted REST API configuration

Close incident test – request body

{
"alertId": "Unity-A007_Pool_Test-Pool93_Threshold Exceeded",
"severity": "1",
"eventType": "DURABLE",
"timestamp": "1623403895",
"systemName": "Unity-A007",
"systemType": "UnifiedArray",
"componentType": "Pool",
"componentName": "Test-Pool93",
"source": "Unity-GenericEvent",
"category": "Capacity",
"description": "Storage pool Test-Pool93 has exceeded its critical threshold of 95%",
"active":"false",
"acknowledged":"N/A",
"owner":"N/A"
}

23 Dell EMC SRM: Automating Incident Management in ServiceNow | H18840

ServiceNow scripted REST API configuration

Explore REST API webapp will respond with below output which summarizes the request:

Close incident test – request summary

Scroll down and see complete response message. Status code must be 200 OK. Response Body message reflects the
specific action being performed within ServiceNow (Incident closed):

Close incident test – response

24 Dell EMC SRM: Automating Incident Management in ServiceNow | H18840

ServiceNow scripted REST API configuration

Navigate to Incident/All in navigation tree and type in the incidents search field srm. Existing incident will be updated with
Closed state. Click on it to open detailed view:

ServiceNow incident closed

Details of the incident are provided below with state Closed, and relevant resolution code and notes populated:

ServiceNow Closed incident details

25 Dell EMC SRM: Automating Incident Management in ServiceNow | H18840

SRM alert notification configuration

4 SRM alert notification configuration

Following two paragraphs are explaining the custom alert definitions used for integration with ServiceNow.

4.1 SRM alert definition with external process action

Alert definition ServiceNow notification via external process contains filtered entry and external process action
components:

Alert definition with external process action

Filtered entry component filters on alerts received by Mail Forwarder Adapter on SRM PBE server, which is listening
socket for consolidated alerts. These alerts are written to events database on SRM PBE server and are candidates to be
sent as notification to external systems.

By default, all alerts are captured:

Alert definition filtered entry component

26 Dell EMC SRM: Automating Incident Management in ServiceNow | H18840

SRM alert notification configuration

External process component invokes script stored on SRM PBE server (/opt/APG/bin/srm-servicenow-https-post.sh).

Script arguments are processed by the script and posted to ServiceNow as request body in https post:

Alert definition external process action component

Script (Command parameter from above action configuration): mandatory field. Specify full path of the script
(/opt/APG/bin/srm-servicenow-https-post.sh). As apg user, copy below script content into file and store it on SRM PBE
server.

Make it executable:

srm-pbe> chmod +x /opt/APG/bin/srm-servicenow-https-post.sh

#!/bin/bash
# get script arguments to construct curl https post

# example open alert: https://dev78342.service-now.com/api/653807/srm_alerts,Unity-A007_Pool_Test-

Pool93_Threshold Exceeded,1,DURABLE,1623841270,Unity-A007,UnifiedArray,Pool,Test-Pool93,Unity-
GenericEvent,Capacity,Storage pool Test-Pool93 has exceeded its critical threshold of 95%,true,N/A,N/A
# example close alert: https://dev78342.service-now.com/api/653807/srm_alerts,Unity-A007_Pool_Test-
Pool93_Threshold Exceeded,1,DURABLE,1623841270,Unity-A007,UnifiedArray,Pool,Test-Pool93,Unity-
GenericEvent,Capacity,Storage pool Test-Pool93 has exceeded its critical threshold of 95%,false,N/A,N/A

snow_uri=$(echo ${1} | sed 's/"//g') # https://dev78342.service-now.com/api/653807/srm_alerts

alertId=$(echo ${2} | sed 's/"//g') # Unity-A007_Pool_Test-Pool93_Threshold Exceeded
severity=$(echo ${3} | sed 's/"//g') # 1,2,3,4,5 or N/A
eventType=$(echo ${4} | sed 's/"//g') # DURABLE
timestamp=$(echo ${5} | sed 's/"//g' | awk '{print substr($0,1,10)}') # convert ms to s:
1623841270000 to 1623841270
systemName=$(echo ${6} | sed 's/"//g') # Unity-A007
systemType=$(echo ${7} | sed 's/"//g') # UnifiedArray

27 Dell EMC SRM: Automating Incident Management in ServiceNow | H18840

SRM alert notification configuration

componentType=$(echo ${8} | sed 's/"//g') # Pool

componentName=$(echo ${9} | sed 's/"//g') # Test-Pool93
src=$(echo ${10} | sed 's/"//g') # Unity-GenericEvent
category=$(echo ${11} | sed 's/"//g') # Capacity
description=$(echo ${12} | sed 's/"//g') # Storage pool Test-Pool93 has exceeded its critical
threshold of 95%.
active=$(echo ${13} | sed 's/"//g') # true or false
acknowledged=$(echo ${14} | sed 's/"//g') # yes or no or N/A
owner=$(echo ${15} | sed 's/"//g') # string or N/A

curl "${snow_uri}" \
--request POST \
--header "Accept:application/json" \
--header "Content-Type:application/json" \
--data "{
\"alertId\":\"${alertId}\",
\"severity\":\"${severity}\",
\"eventType\":\"${eventType}\",
\"timestamp\":\"${timestamp}\",
\"systemName\":\"${systemName}\",
\"systemType\":\"${systemType}\",
\"componentType\":\"${componentType}\",
\"componentName\":\"${componentName}\",
\"source\":\"${src}\",
\"category\":\"${category}\",
\"description\":\"${description}\",
\"active\":\"${active}\",
\"acknowledged\":\"${acknowledged}\",
\"owner\":\"${owner}\"
}" > /opt/APG/srm-servicenow.log 2>&1

Script arguments (Command parameters from above action configuration): comma separated script arguments:

<snow_uri>,PROP.'Name',PROP.'severity',PROP.'eventtype',PROP.'OpenedAt',PROP.'device',PROP.'devtype',PRO
P.'parttype',PROP.'part',PROP.'Source',PROP.'category',PROP.'fullmsg',PROP.'active',PROP.'acknowledged',
PROP.'owner'

ServiceNow URI needs to be submitted, other arguments (PROP.*) are automatically loaded from SRM alert properties:

<snow_uri>: ServiceNow URI (example: https://dev78342.service-now.com/api/653807/srm_alerts)

PROP.'Name': unique alert name (example: Unity-A007_Pool_Test-Pool93_Threshold Exceeded)

PROP.'severity': alert severity (1,2,3,4,5 or N/A)
PROP.'eventtype': DURABLE or MOMENTARY
PROP.'OpenedAt': timestamp in seconds
PROP.'device': device name (example: Unity-A007) or N/A
PROP.'devtype': device type (example: Array, FabricSwitch, Host, VirtualMachine) or N/A
PROP.'parttype': part type (example: LUN, Disk, Port, FileSystem) or N/A
PROP.'part': part name (example: LUN123, Disk567, Port323, /storage/log) or N/A
PROP.'Source': collector name (example: VxRail-Collector, Huawei-Oceanstor) or N/A
PROP.'category': Capacity, Performance, Availability, Error, Configuration, Miscellaneous, Environment
or N/A
PROP.'fullmsg': alert description (example: Storage pool Test-Pool93 has exceeded its critical threshold
of 95%) or N/A
PROP.'active': true or false
PROP.'acknowledged': yes or no or N/A
PROP.'owner': string or N/A

28 Dell EMC SRM: Automating Incident Management in ServiceNow | H18840

SRM alert notification configuration

Copy below content into the file and save it as alerting-external-process.xml on a PC. Go to SRM Admin UI under
Config/Alerts/Manage Alert Definitions and import alerting-external-process.xml file under Import Alerting
Configuration into desired folder (here Examples folder):

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>

<AlertingConfig xmlns="http://www.watch4net.com/Alerting">
<definition-list name="Examples/ServiceNow notification via external process" enabled="false"
hidden="false" id="2688140534343879">
<description>Alert notification definition for sending https post</description>
<entry-point-list>2688140191502579</entry-point-list>
</definition-list>
<entry-point-list id="2688140191502579">
<name>ServiceNow notification filter</name>
<class>com.watch4net.alerting.operation.FilteredEntryPoint</class>
<filter>adapterName=='Mail Forwarder Adapter'</filter>
<description>ServiceNow notification entry point</description>
<action-list from="output" to="entry">2688140191724393</action-list>
</entry-point-list>
<action-list id="2688140191724393">
<name>SRM https post via curl to ServiceNow</name>
<class>com.watch4net.alerting.action.ExternalProcessAction</class>
<description>External script contains curl cmd with http post</description>
<param-list name="command">/opt/APG/bin/srm-servicenow-https-post.sh</param-list>
<param-list
name="parameters"><snow_uri>,PROP.'Name',PROP.'severity',PROP.'eventtype',PROP.'OpenedAt',PROP.'device',
PROP.'devtype',PROP.'parttype',PROP.'part',PROP.'Source',PROP.'category',PROP.'fullmsg',PROP.'active',PR
OP.'acknowledged',PROP.'owner'</param-list>
</action-list>
</AlertingConfig>

Alert definition ServiceNow notification via external process will appear under the selected folder as follows. Right-click
on it and select Enable. From the moment of alert definition enablement, incidents are created/updated/closed/reopened in
ServiceNow based on SRM alerts:

Alert definition with external process action in folder

29 Dell EMC SRM: Automating Incident Management in ServiceNow | H18840

SRM alert notification configuration

4.2 SRM alert definition with webhook action

Alert definition ServiceNow notification via WebHook contains filtered entry and webhook action components:

Alert definition with webhook action

Filtered entry component filters on alerts received by Mail Forwarder Adapter on SRM PBE server, which is listening
socket for consolidated alerts. These alerts are written to events database on SRM PBE server and are candidates to be
sent as notification to external systems.

By default, all alerts are captured:

Webhook alert definition filtered entry component

30 Dell EMC SRM: Automating Incident Management in ServiceNow | H18840

SRM alert notification configuration

SRM Webhook HTTPS POST to ServiceNow Scripted REST API: URL and Secret string are mandatory parameters.
WebHook Content: https post request body, mandatory parameter:

Alert definition webhook action component

ServiceNow REST API URL example: https://dev78342.service-now.com/api/653807/srm_alerts

Secret: an arbitrary string.

WebHook Content: JSON payload with parameters automatically loaded from alert properties:

alertId:PROP.'Name',
severity:PROP.'severity',
eventType:PROP.'eventtype',
timestamp:PROP.'OpenedAt',
systemName:PROP.'device',
systemType:PROP.'devtype',
componentType:PROP.'parttype',
componentName:PROP.'part',
source:PROP.'Source',
category:PROP.'category',
description:PROP.'fullmsg',
active:PROP.'active',
acknowledged:PROP.'acknowledged',
owner:PROP.'owner'

31 Dell EMC SRM: Automating Incident Management in ServiceNow | H18840

SRM alert notification configuration

Copy below content into the file and save it as alerting-webhook.xml on a PC. Go to SRM Admin UI under
Config/Alerts/Manage Alert Definitions and import alerting-webhook.xml into desired folder (here Examples folder):

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>

<AlertingConfig xmlns="http://www.watch4net.com/Alerting">
<definition-list name="Examples/ServiceNow notification via Webhook" enabled="false" hidden="false"
id="1563481087565022">
<description>Alert notification definition for sending webhook</description>
<entry-point-list>1563480754565796</entry-point-list>
</definition-list>
<entry-point-list id="1563480754565796">
<name>ServiceNow WebHook notification filter</name>
<class>com.watch4net.alerting.operation.FilteredEntryPoint</class>
<filter>adapterName=='Mail Forwarder Adapter'</filter>
<description>ServiceNow WebHook notification entry point</description>
<action-list from="output" to="entry">1563480754749057</action-list>
</entry-point-list>
<action-list id="1563480754749057">
<name>SRM webhook to ServiceNow</name>
<class>com.watch4net.alerting.action.WebHookAction</class>
<param-list name="WebHook Content">alertId:PROP.'Name',
severity:PROP.'severity',
eventType:PROP.'eventtype',
timestamp:PROP.'OpenedAt',
systemName:PROP.'device',
systemType:PROP.'devtype',
componentType:PROP.'parttype',
componentName:PROP.'part',
source:PROP.'Source',
category:PROP.'category',
description:PROP.'fullmsg',
active:PROP.'active',
acknowledged:PROP.'acknowledged',
owner:PROP.'owner'</param-list>
</action-list>
</AlertingConfig>

Alert definition ServiceNow notification via WebHook will appear under the selected folder as follows. Make sure it has
valid URL value and arbitrary Secret parameter. Right-click on it and select Enable.

From the moment of alert definition enablement, incidents are created/updated/closed/reopened in ServiceNow based on
SRM alerts:

32 Dell EMC SRM: Automating Incident Management in ServiceNow | H18840

SRM alert notification configuration

Alert definition with webhook action in folder

NOTE:

WebHook alert notification action is a new feature available from SRM 4.6. It currently supports HTTP protocol in
URL parameter, and if ServiceNow REST API resource cannot be configured to listen to http but only to https, alert
definition ServiceNow notification via WebHook will not work.

In such case, use first option to integrate SRM alerts into ServiceNow, via external process action.

33 Dell EMC SRM: Automating Incident Management in ServiceNow | H18840

Examples

5 Examples

Following three paragraphs are providing details on how incidents are created, closed, and updated from SRM in
ServiceNow.

5.1 SRM new alert

Alert definition Data Collection Issue checks if all the storage systems are collecting data within a 24-hour time span.

If the total time since last collection from device is > 24h, alert is generated and sent to ServiceNow for incident creation:

SRM data collection issue alert definition

Report Explore/Storage/Storage Systems shows time since last collection per storage array. Two devices are affected –
time since last collection is greater than 1day as highlighted:

Storage Systems report

34 Dell EMC SRM: Automating Incident Management in ServiceNow | H18840

Examples

Alert definition Data Collection Issue monitors time value and generates the alert shown in Operations/Alerts/All Alerts
report. Two critical alerts are created for the highlighted devices:

All Alerts report

Details of the alert related to one of the affected devices are highlighted on below alert summary:

• alias, severity, active, message, owner, acknowledged, timestamp and event definition are some of the alert
properties that are pushed to ServiceNow to be part of incident properties

Alert details

35 Dell EMC SRM: Automating Incident Management in ServiceNow | H18840

Examples

ServiceNow incidents are created from two SRM originated alerts. Short description incident property is mapped to SRM
unique alert name (event definition from above alert summary). Description incident property provides details about the
issue and is mapped to SRM alert message (message from above alert summary). User input incident property reflects
the action performed by the SRM alerting, and subsequently on incident (defined in scripted REST API javascript), and
Subcategory incident property is hardcoded to SRM in scripted REST API javascript, so that SRM originated incidents can
be searched from within Incidents search box:

ServiceNow new incident

Details of ServiceNow incident:

• priority value depends on urgency (SRM alert severity, here having value 1) and impact (hardcoded in the
scripted REST API javascript)
• impact value is hardcoded to urgency (here 1), so that final incident priority is always equal to SRM alert severity
value (here: priority = SRM severity = 1)

ServiceNow new incident details

36 Dell EMC SRM: Automating Incident Management in ServiceNow | H18840

Examples

5.2 SRM close alert

SRM alert can be closed in following cases:

• SRM admin user invokes Force Close action on selected alert in All Alerts report, or
• Clear alert is produced by SRM alert definition (here Data Collection Issue alert definition, if device’s status is
up again)

Below screenshot depicts Force Close action invoked by SRM admin user on selected alert:

Alert force close action on report

Related ServiceNow incident is closed:

ServiceNow closed incident

37 Dell EMC SRM: Automating Incident Management in ServiceNow | H18840

Examples

Details of closed ServiceNow incident with resolution code and notes:

ServiceNow closed incident details

38 Dell EMC SRM: Automating Incident Management in ServiceNow | H18840

Examples

5.3 SRM update alert

SRM alert can be updated in following cases:

• SRM admin user invokes one of the Acknowledge, Assign or Ticket ID actions on selected alert

SRM admin user invokes Acknowledge action on selected alert:

Alert acknowledge action on report

Alert is updated with Ack and Owner properties:

Acknowledged alert report

Related ServiceNow incident property User input is updated with acknowledgment and owner info:

ServiceNow updated incident

39 Dell EMC SRM: Automating Incident Management in ServiceNow | H18840

Technical support and resources

6 Technical support and resources

Dell.com/support is focused on meeting customer needs with proven support.

Dell can provide professional services to help build a complete, integrated SRM and ServiceNow solution. Contact your
Dell account team or your authorized partner and request them to engage with DT Specialty Deployment – Custom
Solutions.

Storage and data protection technical white papers and videos provide expertise that helps to ensure customer success
with Dell EMC storage and data protection products.

40 Dell EMC SRM: Automating Incident Management in ServiceNow | H18840