Troubleshooting
Methodologies and
Resources
This lesson describes engineering troubleshooting activities, DIReC
approach, and general troubleshooting flow. The lesson also
describes various support resources and services available (such as
Unit 42 and Managed Threat Hunting) to assist with
troubleshooting.
Troubleshooting Activities
Troubleshooting is an engineering discipline that requires certain skillsets in
the following areas.
Methodologies
Use multiple methodologies in problem determination and troubleshooting
provide guidance to help find the root cause of the problem.
Knowledge Base, Documentation, and Resources
Bookmark useful websites including the administrator’s guide and release
notes. Carefully examine release notes to be informed about known issues
and incompatibilities.
Troubleshooting Tools and Utilities
Use these tools to test various system components. For the Cortex XDR
agent, the primary app is Cytool, which comes with the Cortex XDR agent.
Logs and Traces
System components record their activities in the log files, and you should be
able to retrieve those records down to a point in time prior to the problem’s
occurrence. Traces are different. You enable the components to record their
activities only after the problem’s occurrence.
Working with Support
You should be able to generate enough data at one time to minimize back
and forth communication with support.
DIReC Approach
The DIReC approach is one of many popular methodologies for problem
solving. This mental framework for solving complex technical problems is
based on a four-step approach.
D
Define
The first step is to define the issue in specific quantifiable terms.
I
Isolate
The second step involves using tools and techniques to isolate and verify
the root cause of an issue.
Re
Resolve
In the third step, you resolve the problem. In the IT world, this step involves
careful planning and sequencing of corrective actions, sometimes with the
help of a maintenance window.
C
Confirm
In the fourth step, you confirm the effectiveness and permanence of the
resolution with an understanding that you may have to repeat the entire
process if you did not accurately identify the root cause(s).
There are many other frameworks and approaches to problem solving, and
many organizations have their own in-house variation based on leadership
philosophy, team experiences, tools available, and the architecture of the
production environment.
General Troubleshooting Flow
Regardless of the troubleshooting methodology, a general troubleshooting
flow consists of the following steps. The step names vary depending on the
methodology.
Click the tabs for more information about each step.
Verify the Problem
Gather Data
Extract Hypothesis
Reduce Hypothesis
Find the Root Cause
Note that these steps are for the post-problem period. There also is the
proactive approach to minimize the effects of problems. Before you can
effectively troubleshoot, you must understand what is “normal” for your
domain, particularly about throughput, latency, and average processor
burden. The OSI Reference Model or the more consolidated TCP/IP Reference
Model provides a useful framework for sequencing troubleshooting steps.
Cortex XDR Documentation
You can view and download Cortex XDR Administrator’s Guides, Cortex XDR
Agent Administrator’s Guide, release notes, and other reference resources
from the Cortex XDR Documentation site.
Click the arrows for more information about the administrator's guides,
release notes, and other resources available. Click the images to enlarge
them.
Cortex XDR Administrator’s Guides, Release Notes, and
Reference Docs
The Cortex XDR Administrator’s Guides provide comprehensive information
for setting up and managing the advanced endpoint infrastructure. They
also provide best practices and instructions for installing the agent on
endpoints, how to use the Cortex XDR management console to manage
endpoint security policies and agent settings, and to investigate the
incidents and to respond to suspicious activities on your endpoints and
network.
Note that the two different administrator’s guides depend on licenses.
Cortex XDR Agent Administrator’s Guide, Release Notes, and
Other Resources
The Cortex XDR Agent Administrator’s Guide provides comprehensive
information for managing the Cortex XDR agents on the endpoints. You can
review the Agent Release Notes for information about what has changed in
the new version, limitations, and known issues.
The Compatibility Matrix answers the question, “Where can I install the
Cortex XDR agent?” by providing a matrix of endpoint operating systems
supported by the Cortex XDR agent.
Cortex XDR Release Notes
The Cortex XDR Release Notes contain information about what has changed
in the new version. You can use the release notes to determine whether
unexpected behavior was caused by factors specific to a certain software
version.
Check these sections:
Features Introduced
Associated Software and Content Versions
Limitations
Known Issues
Cortex XDR Community Resources
The following are other resources that you can access:
Customer Support Portal (support.paloaltonetworks.com)*
Cortex XDR Documentation (docs-
cortex.paloaltonetworks.com/p/XDR)
LIVEcommunity Portal (live.paloaltonetworks.com)*
Palo Alto Networks Learning
Center (beacon.paloaltonetworks.com)*
YouTube Tutorials (www.youtube.com/user/paloaltonetworks)
*Requires a support login account for access
Cortex XDR Managed Threat Hunting
Cortex XDR Managed Threat Hunting (MTH) is the industry’s first threat
hunting service that leverages already-collected data to find sophisticated,
hidden, or unnoticed threats using the expertise of our threat hunters and
advanced hunting tools and methodologies.
Cortex XDR Managed Threat Hunting include the following capabilities:
Expertise of Unit 42
Our renowned threat hunters continuously monitor your environment for
stealthy indicators of attack.
Built on Cortex XDR
Find every attack with high-fidelity detections based on holistic visibility
from the industry’s first XDR category product.
Enriched By Pervasive Context
Augment experts with high-fidelity threat intelligence from a massive
customer base.
Unit 42: Global Response Capability
Unit 42 combines an elite group of cyber researchers and incident
responders. Grounded with a reputation for developing industry-leading
threat intelligence, Unit 42 consultants provide global incident response and
cyber risk management services.
Click the tabs for more information about Unit 42.
Data Breach Response
Data breach response minimizes the impact of an incident and exposure
time to attackers. Unit 42 can help identify, contain, and eradicate threats to
enable your organization to get back to business quickly.
Cyber Risk and Resilience Management
Unit 42 works with you to proactively detect and assess cyberthreats and to
mature your information security program.
Digital Investigations
Digital investigations collect, recover, and interpret information gleaned
from digital media by applying Unit 42 state-of-the-art digital forensics tools.
Data Analytics and Intelligence
Data analytics and intelligence find sensitive information at risk in a data
breach faster, more reliably, and at a lower cost using machine learning.
Expert Witness and Litigation Support
The expert witness and litigation support get unassailable opinions for
cybersecurity-related disputes. Unit 42 works with legal teams to review
evidence and provide expert-witness briefing in reports, depositions, and
open court testimony.
Agent Data
Stores
This lesson describes the Cortex XDR Cytool application that you
can use to query and manage the agent's basic and advanced
functions at the endpoint.
Cortex XDR Cytool App
The Cytool application is the command-line interface for administrators
when working with the Cortex XDR agent. Cytool's functions go far beyond
the agent console. It is part of the Cortex XDR agent installation package
and is automatically installed when an agent is installed.
You can use Cytool to query and manage the agent's basic and advanced
functions at the endpoint. Be aware that configuration changes you make
using Cytool may be overridden by settings in the Cortex XDR management
console during the next heartbeat.
Usage: cytool <option> [<option parameters>]
The Cytool app is in the agent installation folder on endpoints:%Program
Files%\Palo Alto Networks\Traps.
Cytool can display usage information for a specific option when the
switch /? is used after the option.
Cytool Options
An important note when using Cytool is that some Cytool options require the
supervisor (uninstall) password. For example, this command: cytool
protect disable needs the password, whereas cytool protect enable does
not.
Also, pay attention to the performance impact of running Cytool with a few
options, the cytool scan command does not consume more than 25
percent CPU load, because it runs as a low-priority thread. However,
the cytool imageprep scan command can use all available resources.
The Cytool command options is available for Windows endpoints. Click the
image to enlarge it.
Agent Configuration Data Locations
The Cortex XDR agent configuration data is distributed to several locations
and files on the endpoint.
Click the tabs for various locations where different files and data are stored.
Binary Files and Databases for Configuration
The folder %PROGRAMDATA%\Cyvera\LocalSystem contains binary files
and databases for the configuration. For example, an agent’s immediate
hash-verdict cache (originally named the WildFire cache) is stored in the file
wfcache. This cache contains file hash versus verdict information, and it
grows over time.
Agent-Related Settings
Some other agent-related settings are maintained through the Windows
registry under the subtree HKLM\SYSTEM\Cyvera. For example, you can
view the list of the trusted signers in HKLM\SYSTEM\Cyvera\Policy\
Organization\Settings
\TrustedPublishers.
The image displays the Windows registry for subtree HKLM\SYSTEM\
Cyvera.
Agent Persistent Databases
The agent components use on-disk key-value data for persistence of their
private data, which are stored in files in %PROGRAMDATA%\Cyvera\
LocalSystem\Persistence.
The following is a partial list of the persistent databases in LocalSystem\
Persistence.
agent_actions.db
agent_settings.db
cloud_frontend.db
hash_containers.db
hash_paths.db
post_detection.db
remediation_events.db
security_events.db
Cytool Options for Persistent Databases
You can use Cytool to list the names of the databases, dump information
from a given database, import, and export databases in JSON format. The
command cytool persist export creates JSON files in Documents\
PaloAltoNetworks\Traps\cytool for Windows.
Examples of the Cytool commands include:
cytool persist list
cytool persist print security_events.db
cytool persist export cloud_frontend.db
cytool persist import agent_settings.db <file>
Getting Information from Agent Databases
In this example, the command cytool persist print
cloud_frontend.db will display important communication settings,
including network addresses of the Cortex XDR servers in the cloud.
Note that the output is modified to fit. For example, the Authentication ID is
much longer than shown, the subdomain is specific for the instance, and the
domain is traps.paloaltonetworks.com.
For a complete list of URLs to which a Cortex XDR agent connects, refer to
the Cortex XDR Pro Administrator's Guide.
Click the image to enlarge it.
Agent
Identification
This lesson describes the two unique identifications (IDs) of a
Cortex XDR agent and how to use the advanced shortcut menu in
the Cortex XDR management console.
Unique Agent IDs
Cortex XDR uses two IDs, known as Endpoint ID and Machine ID, to uniquely
identify the software and hardware of a Cortex XDR agent.
The IDs are stored in agent.id and hardware.id in the %PROGRAMDATA%\
Cyvera\LocalSystem\OsPersistence folder. Click the tabs for more
information about the Endpoint ID and Machine ID.
Endpoint ID
The Endpoint ID is a unique identifier for the Cortex XDR agent software.
This ID is generated by Cortex XDR for the agent during the agent
installation.
Machine ID
The Machine ID is a unique identifier for the hardware (physical or virtual)
on which the Cortex XDR agent runs. Its generation is based on machine
characteristics including the hard disk ID. This ID is generated by the agent
during the agent installation.
Agent Identification Considerations
Consider the files agent.id and hardware.id during uninstall and reinstall
operations. Uninstallation of the Cortex XDR agent does not delete the two
files. When the agent is reinstalled, the IDs in these files are reused, which
eliminates duplicate endpoint entries in the Cortex XDR management
console.
Advanced Shortcut Menu
You can open an advanced shortcut menu for an endpoint in the
management console; press Alt + Right-Click on an endpoint in the
Endpoint Administration table.
Note that the additional actions Force Check-in and Change Managing
Server are only visible based upon certain conditions.
Click the image to enlarge it.
Change Managing Server
You can change the Cortex XDR instance of an existing agent directly from
the Cortex XDR management console. After the agent registers with the new
instance, the agent can no longer connect to the previous instance.
To change the Cortex XDR instance of an existing agent, perform
the Change Managing Server action by accessing the installation ID
from Endpoints > Agent Installations and copying the installation
package ID in the new instance to the new server.
Note: For more detailed information on this topic, refer to the Move Cortex
XDR Agents Between Managing XDR Servers section in the Cortex XDR Pro
Administrator's Guide.
Agent
Log
This lesson describes the Cortex XDR agent log and how to set the
log level using Cytool.
Cortex XDR Agent Log
The Cortex XDR agent components write to trapsd.log to record their
actions, such as service and status connection changes, policy updates, and
creation of alerts.
Log-Formatted Text File
You can directly open trapsd.log from %PROGRAMDATA%\Cyvera\Logs\
trapsd.log. The trapsd.log is a log-formatted text file, where the logs are
listed chronologically with the most recent log at the bottom. The amount of
information varies depending on the log severity level of the components.
Setting Log Level
You can use Cytool to change logging levels of the Cortex XDR agent
components.
The levels ranges from Trace to Fatal. The setting of a level enables all the
levels up to that level. You also can completely disable the logging.
The cytool log set_level <log_level> <components> command requires
the component name you want to set the log level for. For the components,
you can specify cyserver or all. The components vary by platform. For
Windows: [cyserver | all].
The default log level is 6, corresponding to Info. It is important to return the
log level to default when the collection is complete.
Working with
Technical Support
This lesson describes various methods to export or generate a
support file needed to create support cases.
Support File
A support file is a compressed, archived file that aggregates all the logs for
an endpoint. It is generated on-demand to help Palo Alto Networks Technical
Support troubleshoot and diagnose system issues. You usually attach this
file to your support cases.
You can obtain the file from the Cortex XDR management console; the
instance requests the file from the agent. Or you can use the Cortex XDR
agent console directly on the endpoint or Cytool.
The image shows a sample content of a support zip file.
Retrieving or Generating a Support File Methods
Click the arrows for more information about how to retrieve or generate a
support file via Cortex XDR management console, agent console, and
Cytool. Click the images to enlarge them.
Retrieving Support File in Management Console
You can initiate retrieval of a support file from an endpoint using
the Endpoint Control > Retrieve Support File action on the endpoint’s
shortcut menu in the management console. On reception of the request, the
Cortex XDR agent packages all available logs as a compressed file and
uploads the package to the Cortex XDR instance.
You can track the status of the action in the Action Center. When the action
status shows "Completed Successfully," you can download the file by taking
the Download files action from the Detailed Results dialog.
Generating Support File From Agent Console
If the endpoint is disconnected from your Cortex XDR management console,
you have other options to generate a support file directly from the endpoint.
Click the Generate Support File link on the agent console. When the
support file generation is complete, the agent automatically opens the
folder that contains the zip file in File Explorer.
You also can locate the generated support file in %APPDATA%\
PaloAltoNetworks\Traps\support\logs_<GUID>, where GUID is a
randomly created string of characters.
Generating Support File Via Cytool
You can also use Cytool to generate a support file using the
command cytool log collect.
The output folder for this command is the same folder as when you
click Generate Support File in the agent console: %APPDATA%\
PaloAltoNetworks\Traps\support.
Agent Tokens
Cortex XDR now supports the ability to create, retrieve, and use agent
tokens.
The agent token is useful for situations where an admin does not need to
know the agent password but still needs to access and perform agent
operations.
You can access the agent token via Endpoint Control > View Token.
Creating a Support Case
You can create a support case at the Customer Support Portal. From Home
> Case List, click Get Help to create a support case, and then append all
the gathered information such as the support file to the case.
