Module 8

Network Troubleshooting

ITNET04
WAN Connectivity
Module Objectives

Module Title: Network Troubleshooting

Module Objectives:
• Explain how network documentation is essential in aiding troubleshooting
• Explain troubleshooting approaches for various network problems.
• Troubleshoot end-to-end connectivity in a small to medium-sized business network, using a
systematic approach.

Module References:
 CCNAv7 ENSA– Module 12

© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 2
8.1 Troubleshooting Methodology

© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 3
Network Documentation
Documenting the Network
 To efficiently diagnose and correct network problems, a network engineer needs to know:

• How the network has been designed.

• The network’s expected performance.
 This information is captured in the network documentation. Network administrators must
have a complete set of accurate and current network documentation which includes:
• Configuration details, including network device and end-system configuration
• Topology diagrams keep track of the location, function, and status of devices on the
network.
• Baseline performance levels

© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 4
Network Documentation
Infrastructure Device Documentation
 Infrastructure device documentation focuses on the details (IP settings, connections, platform
information, location) of routers, switches, access points, firewalls, etc
Router Device
Documentation

Switch Device
Documentation

© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 5
Network Documentation
End System Documentation
 End-system documentation focuses on the IP configuration and services of end-system
devices such as servers, network management consoles, and user workstations.

End-System
Documentation

© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 6
Network Documentation
Network Topology
Diagrams
 Physical Topology network
diagrams show the physical
layout of the devices
connected to the network and
typically include:
• Device type
• Model and manufacturer
• Operating System version
• Cable type and identifier
• Cable specification
• Connector type
• Cabling endpoints
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 7
 Network Documentation
Network Topology
Diagrams (Cont.)
 Logical network topology diagrams
illustrate how devices are logically
connected to the network and might
include:
• Device identifiers
• IP address and prefix lengths
• Interface identifiers
• Connection type
• Frame Relay DLCI for virtual circuits
• Site-to-site VPNs
• Routing protocols and static routes
• WAN technologies used
• Data-link protocols
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 8
Network Documentation
Establishing a Network Baseline
 The purpose of network monitoring is to watch
network performance in comparison to a
predetermined baseline. A network baseline determines the
“personality” of the network under normal
conditions
 A network performance baseline
How does a network perform on an average day?
• Establishes normal network or system performance
Which part of the network is most heavily used?
• Requires collecting performance data from the ports
and devices that are essential to operation What part of the network is least used?
• Allows the network administrator to determine the
Where are the most errors occurring?
difference between abnormal behavior and proper
network performance Which devices should be monitored and at what
alert threshold?
 Analysis after an initial baseline also tends to reveal
Can the network meet the identified policies?
hidden problems. The collected data can show the
true nature of congestion or potential congestion in a
network.
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 9
Network Documentation
Steps to Establish a Network Baseline
 Step 1: Determine what types of data to
collect.
• Start out with a few variables that represent
the defined policy.
• Capturing too many data points can be
overwhelming and make analysis difficult.
• Start out simply, and fine-tune along the way.
 Step 2: Identify devices and ports of
interest.
• Use the network topology to identify key
devices where performance data should be
measured.
• Devices and ports of interest include network
device ports that connect to other network
devices, servers, and key users.
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 10
Network Documentation
Steps to Establish a Network Baseline (Cont.)
 Step 3: Determine the baseline duration

• The length of time and baseline information being gathered must be sufficient for establishing a typical
picture of the network.
• Daily trends of network traffic should be measured.
• Monitor for trends that occur over a longer period of time such as weekly or monthly.

 Capture data trends and include:

• Screenshots of CPU utilization

trends captured over a daily,
weekly, monthly, and yearly
period
 Note: Baseline measurements
should not be performed during
times of unique traffic patterns.

© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 11
Network Documentation
 When documenting the network, it is
Measuring Data necessary to gather information directly
from routers and switches.
 Ping, traceroute, and telnet are useful
commands to document.
 The figure to the left lists some of the most
common Cisco IOS show commands used
for data collection.
 Manual data collection using show
commands on individual network devices is
very time consuming and is not a scalable
solution. This should be reserved for
smaller networks or mission critical devices.
 Sophisticated network management
software is typically used to baseline large
and complex networks.
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 12
Network Documentation
Measuring Data
 For documentation, it is necessary to gather information directly from routers and switches.

 Ping, traceroute, telnet and show are useful commands to document.

 Manual data collection using show commands on individual network devices is usually reserved
for smaller networks or mission critical devices because it is time consuming.
Command Description

Show version Show uptime, hardware info and OS version of device

Show ip interface [brief] Show status and address configuration on interfaces

Show interface Show detailed output (protocol, status, statistics, addressing) for an interface

Show ip / ipv6 route Shows contents of routing table

Show arp / show ipv6 neighbors Shows ip address to mac address mappings

Show vlan Shows status of VLANs on a switch

Show port Shows the status of ports on a switch © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 13
Troubleshooting Process
General Troubleshooting Approaches
• For network engineers, administrators, and support personnel, troubleshooting is
a process that takes the greatest percentage their time.
• Using efficient troubleshooting techniques shortens overall troubleshooting time.
• Two extreme approaches to troubleshooting almost always result in
disappointment, delay, or failure.

The caveman The rocket scientist

- Makes random - Deeply analyzes the
changes to the situation until the root
network until it cause of the problem
miraculously starts has been identified
working again, - Takes too long to
- May not have found resolve an issue.
and fixed the root
cause of the issue
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 14
Troubleshooting Process
Troubleshooting Procedures
 Use efficient troubleshooting Stage 1: Gather symptoms
techniques to shorten overall
troubleshooting time.
 Stage 1. Gather symptoms – Stage 2: Isolate the problem

• Determines which network components

have been affected and how the Stage 3: Implement Corrective
functionality of the network has Action
changed in comparison to the baseline.
• Symptoms may come from the network
management system, console Problem
messages, and user complaints. Fixed?
• Question users and investigate the
issue in order to localize the problem to Undo corrective Document
a smaller range of possibilities. action and restart solution and save
process changes

© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 15
Troubleshooting Process
Troubleshooting Procedures
 Use efficient troubleshooting Stage 1: Gather symptoms
techniques to shorten overall
troubleshooting time.
 Stage 2. Isolate the problem – Stage 2: Isolate the problem

• Isolating is the process of eliminating

variables until a single problem, or a Stage 3: Implement Corrective
set of related problems has been Action
identified as the cause.
• Examine the problems at the logical
layer of the network so that the most Problem
likely cause can be detected. Fixed?

Undo corrective Document

action and restart solution and save
process changes

© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 16
Troubleshooting Process
Troubleshooting Procedures
 Use efficient troubleshooting Stage 1: Gather symptoms
techniques to shorten overall
troubleshooting time.
 Stage 3. Implement corrective action – Stage 2: Isolate the problem

• Correct the problem by implementing,

testing, and documenting possible Stage 3: Implement Corrective
solutions. Action
• Can the solution be implemented
immediately, or does it need to be
postponed? Problem
Fixed?
• The severity of the problem should be
weighed against the impact of the
solution. Undo corrective Document
action and restart solution and save
process changes

© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 17
Troubleshooting Process
Gathering Symptoms
Gather Information  It is important to gather facts and evidence that will allow you to
progressively eliminate possible causes, and eventually identify the
root cause of the issue.
Determine Ownership  There are five information gathering steps:

 Step 1. Gather Information

Narrow the Scope • Gather information from the trouble ticket, users, or end systems
affected by the problem to form a definition of the problem.
 Step 2. Determine ownership
Gather symptoms
from suspected • If the problem is within the control of the organization, move onto the
devices next stage. If it is outside of the boundary of organizational control,
contact an administrator for the external system.
Document symptoms

© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 18
Troubleshooting Process
Gathering Symptoms (Cont.)
 Step 3. Narrow the scope
Gather Information
• Determine if the problem is at the core, distribution, or access layer.
• At the identified layer, analyze the existing symptoms and try to
Determine Ownership determine which piece of equipment is most likely the cause.
 Step 4. Gather symptoms from suspect devices

• Using a layered troubleshooting approach, gather hardware and

Narrow the Scope software symptoms from the suspect devices.
• Is it a hardware of software configuration problem?
Gather symptoms  Step 5. Document symptoms
from suspected
devices • If the problem cannot be solved using the documented symptoms,
begin the isolating stage of the general troubleshooting process.

Document symptoms • Gather symptoms from devices using commands/tools, device logs
and packet captures.

© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 19
Troubleshooting Process
Questioning End Users
 In many cases, the problem is reported by an end user. This information may often be
misleading or vague. Use effective questioning techniques when asking the end users about a
network problem they may be experiencing.
Guidelines Example Open Ended End-User Questions

• What does not work?

Ask pertinent questions. • What exactly is the problem?
• What are you trying to accomplish?

• Who does this issue affect? Is it just you or others?

Determine the scope of the problem.
• What device is this happening on?

• When exactly does the problem occur?

Determine when the problem occurred / occurs. • When was the problem first noticed?
• Were there any error message(s) displayed?

• Can you reproduce the problem?

Determine if the problem is constant or intermittent.
• Can you send me a screenshot or video of the problem?

Determine if anything has changed. • What has changed since the last time it did work?

• What works?
Use questions to eliminate or discover possible problems. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 20
• What does not work?
Isolating the Issue Using Layered Models
Using Layered Models for Troubleshooting
 If no solution is identified, the network
administrator compares the
characteristics of the problem to the
logical layers of the network to isolate
and solve the issue.
 Logical networking models, such as the
OSI and TCP/IP models, separate
network functionality into modular
layers.
 When troubleshooting, these layered
models can be applied to the physical
network to isolate network problems.

© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 21
Isolating the Issue Using Layered Models
Using Layered Models for Troubleshooting
 Logical networking models, such as the OSI and TCP/IP models, separate network functionality
into modular layers.

Application Issues.
Implemented in software.

Software
Data Transport
Issues

Hardware

© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 22
Isolating the Issue Using Layered Models
Troubleshooting Methods
Application  Bottom-up Troubleshooting Method

• Start with the physical components of the

network and move up through the layers of the
Transport OSI model until the cause of the problem is
identified
• This is a good approach to use when the
Network problem is suspected to be a physical one.
• Most networking problems reside at the lower
levels, so using this method is often effective
Data Link • The disadvantage with this method is it
requires that you check every device and
interface on the network until the cause of the
problem is found.
Physical Start here

© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 23
Isolating the Issue Using Layered Models
Troubleshooting Methods
Application Start here  Top-Down Troubleshooting Method

• This method starts with troubleshooting the

end-user applications and moves down
Transport through the layers of the OSI model until the
cause of the problem has been identified.
• End-user applications are tested before
Network tackling the more specific networking pieces.
• Use this approach for simpler problems.
• The disadvantage is that it requires checking
Data Link every network application until the problem is
found.

Physical

© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 24
Isolating the Issue Using Layered Models
Troubleshooting Methods
Application  Divide-and-Conquer Troubleshooting
Method
• The network administrator selects a layer and
Transport Start here tests in both directions from that layer.
• Start by collecting user experiences of the
or problem, document the symptoms, and then,
Network Start here using that information, make an informed
guess as to which OSI layer to start your
or investigation.
• If a layer is functioning properly, all layers
Data Link Start here below can be assumed to be functioning.

Physical

© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 25
Isolating the Issue Using Layered Models
Guidelines for Selecting a Troubleshooting Method
 To quickly resolve network problems, take the
time to select the most effective network
troubleshooting method. An example:
• Two IP routers are not exchanging routing
information.
• The last time this type of problem occurred, it
was a protocol issue.
• Therefore, choose the divide-and-conquer
troubleshooting method.
• Analysis reveals that there is connectivity
between the routers.
• Start the troubleshooting process at the
physical or data link layer.
• Confirm connectivity and begin testing the
TCP/IP-related functions at the next layer up in
the OSI model, the network layer.
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 26
Isolating the Issue Using Layered Models
Other Troubleshooting Methods
• Educated guess by the network administrator
• Guess is based on the symptoms of the problem
• This is more successful when implemented by seasoned network
administrators who can rely on their extensive knowledge and experience
• Comparing a working and non-working situation
• Look for differences between configurations, software versions, and
hardware and other device properties.
• This method can be helpful when the network administrator is lacking an
area of expertise or when the problem needs to be resolved quickly.
• Substitution
• Involves swapping the problematic devices with known, working ones.
• If the problem remains, the network administrator knows to look
elsewhere.
• Follow the Path
• Used to discover the actual traffic path from source to destination to
reduce the scope of troubleshooting
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 27
8.2 Troubleshooting Scenarios

© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 28
Troubleshooting Tools  Common software troubleshooting tools
Software Troubleshooting Tools include these:
• Network Management System Tools
• NMS tools include device-level monitoring,
configuration, and fault-management tools.
• These graphical tools can be used to
investigate and correct network problems.
• Knowledge Bases
• On-line network device vendor knowledge
bases are very useful.
• When combined with Internet search engines,
a network administrator has access to a vast
pool of experience-based information.
• Baselining Tools
• Many tools for automating the network
documentation and baselining process are
available. For example:
• SolarWinds Network Performance Monitor

© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 29
Troubleshooting Tools
Protocol Analyzers  Protocol analyzers are useful to investigate
packet content while the content is flowing
through the network.
 A protocol analyzer decodes the various
protocol layers in a recorded frame and
presents it in an easy to use format.
 The figure to the left shows a screen
capture of the Wireshark protocol analyzer.
 Most protocol analyzers can filter traffic that
meets certain criteria. For example, all
traffic to and from a particular device can be
captured.
 Protocol analyzers are very helpful in
troubleshooting network performance
problems.
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 30
Troubleshooting Tools
Hardware Troubleshooting Tools  There are multiple types of hardware
troubleshooting tools including:
• Digital Multimeters are test instruments that
are used to directly measure electrical values
of voltage, current, and resistance.
• Cable Testers are specialized handheld
devices designed for testing the various types
of data communication cabling. They can be
used to detect broken wires, crossed-over
wiring, shorted connections, and improperly
paired connections. More expensive time-
domain reflectometers (TDRs) are used to
pinpoint the distance to a break in a cable.
• Cable Analyzers are multifunctional handheld
devices that are used to test and certify copper
and fiber cables for different services and
standards.

© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 31
Troubleshooting Tools
Hardware Troubleshooting Tools (Cont.)
• Portable Network Analyzers are used for
troubleshooting switched networks and VLANs.
• By plugging the network analyzer in anywhere on
the network, a network engineer can see the
switch port to which the devices is connected.
• They can also see the average and peak
utilization as well as the VLAN configuration.
• Network Analysis Module – The Cisco NAM
is a device or software.
• It provides an embedded browser-based interface
that generates reports on the traffic that consumes
critical network resources.
• The NAM can capture and decode packets and
track response times to pinpoint an application
problem to a particular network or server.

© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 32
Troubleshooting Tools
Using a Syslog Server for Troubleshooting
 Recall: Syslog is used by an IP device
known as a syslog client to send text-
based log messages to another IP
device known as the syslog server.
 Implementing a logging facility is a
very important part of network security
and also for network troubleshooting.
 Network devices can log various types
of information including configuration
changes, ACL violations, interface
 Cisco devices can send log messages to several different facilities:
status, and many other types of
• Console and Terminal lines events.
• Buffered logging
 Syslog messages fall into one of eight
• SNMP traps
levels. The lower the level number, the
• External Syslog server
higher the severity level.
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 33
Symptoms and Causes of Network Troubleshooting
Physical Layer Troubleshooting
 The physical layer is the only layer with physically tangible
properties, such as wires, cards, and antennas.
Application
 Because the upper layers of the OSI model depend on the
Presentation physical layer to function, a network administrator must
Session have the ability to effectively isolate and correct problems at
this layer.
Transport
 Common symptoms of network problems at the physical
Network layer include:
Data Link • Performance lower than baseline
• Loss of connectivity
Physical
• Network bottlenecks or congestion
• High CPU utilization rates
• Console error messages

© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 34
Symptoms and Causes of Network Troubleshooting
Physical Layer Troubleshooting (Cont.)
 Issues that commonly cause network problems at the physical layer include:

Problem Cause Description

Power-related Check the operation of the fans and ensure that the chassis intake and exhaust vents are clear.
Faulty or corrupt NIC driver files, bad cabling, or grounding problems can cause network transmission
Hardware faults
errors such as late collisions, short frames, and jabber.
Look for damaged cables, improper cable, and poorly crimped connectors.
Cabling faults
Suspect cables should be tested or exchanged with a known functioning cable.
Attenuation can be caused if a cable length exceeds the design limit for the media, or when there is a
Attenuation
poor connection resulting from a loose cable, or dirty or oxidized contacts.
Local electromagnetic interference (EMI) can be generated by many sources, such as crosstalk,
Noise
nearby electric cables, large electric motors, FM radio stations, police radio, and more.
Interface configuration Causes can include incorrect clock rate, incorrect clock source, and interface not being turned on.
errors This causes a loss of connectivity with attached network segments.
Exceeding design
A component could operate sub-optimally if it is being utilized beyond specifications.
limits
Symptoms include processes with high CPU utilization percentages, input queue drops, slow
CPU overload performance, SNMP timeouts, no remote access, no DHCP services, Telnet, and pings are slow or fail
to respond.
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 35
Symptoms and Causes of Network Troubleshooting
Data Link Layer Troubleshooting
 Troubleshooting Layer 2 problems can be a challenging
process.
Application
 Layer 2 problems cause specific symptoms that, when
Presentation recognized, will help identify the problem quickly:
Session • No functionality or connectivity at the network layer or above
Transport • Network is operating below baseline performance levels
• Excessive broadcasts
Network
• Most common Layer 2 console message is: “line protocol
Data Link down”

Physical

© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 36
Symptoms and Causes of Network Troubleshooting
Data Link Layer Troubleshooting (Cont.)
 Issues at the data link layer that commonly result in network
Application connectivity or performance problems include these:

Presentation • Encapsulation errors

• Encapsulation at one end of a WAN link is configured differently from that on
Session the other end.
• Address mapping errors
Transport
• In a point-to-multipoint or broadcast Ethernet topology, it is essential that an
Network appropriate Layer 2 destination address be given to the frame.
• Framing errors
Data Link
• A framing error occurs when a frame does not end on an 8-bit byte boundary.
Physical • Spanning Tree Protocol (STP) failures or loops.
• Most STP problems are related to forwarding loops that occur when no ports in
a redundant topology are blocked and traffic is forwarded in circles indefinitely.

© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 37
Symptoms and Causes of Network Troubleshooting
Network Layer Troubleshooting
 Network layer problems include any problem that involves a
Application Layer 3 protocol (routed or routing protocols)

Presentation  Common symptoms of network layer problems:

Session • Network failure

• Suboptimal performance
Transport
 Areas to explore when diagnosing a possible problem
Network involving routing protocols:
Data Link • General network issues
• Connectivity issues – Also check for Layer 1 or power issues
Physical
• Routing table issues – use debug
• Neighbor issues – check for adjacencies if using routing
protocols
• Check the routing table topology database
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 38
Symptoms and Causes of Network Troubleshooting
Transport Layer Troubleshooting - ACLs
 Network problems can arise from transport layer problems on
Application the router. Improper ACL configuration issues might include:

Presentation • Wrong selection of traffic flow (inbound/outbound)

• Incorrect order of access control entries
Session
• Implicit deny any
Transport • Misconfiguration of addresses and IPv4 wildcard masks
Network • Selecting both UDP and TCP protocols when unsure
• Incorrect source and destination ports
Data Link
• Incorrect use of the established keyword
Physical • Misconfiguration of uncommon protocols such as VPN and
encryption protocols

© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 39
Symptoms and Causes of Network Troubleshooting
Transport Layer Troubleshooting – NAT for IPv4

Application  There are a number of problems with NAT such as not

interacting with services like DHCP and tunneling.
Presentation
 These can include misconfigured NAT inside, NAT
Session outside, or a misconfigured ACL.
Transport  Other issues include interoperability with other network
Network technologies including:
• BOOTP and DHCP
Data Link
• DNS
Physical • SNMP
• Tunneling and encryption protocols

© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 40
Symptoms and Causes of Network Troubleshooting
Application Layer Troubleshooting
 Most of the application layer protocols provide user services
Application for network management, file transfer, distributed file
Presentation services, terminal emulation, and email.
 The most widely known and implemented TCP/IP application
Session
layer protocols include:
Transport • SSH/Telnet, HTTP, FTP, TFTP
Network • SMTP, POP, SNMP, DNS, NFS
Data Link  Application layer problems prevent services from being
provided to application programs.
Physical
 A problem at the application layer can result in unreachable
or unusable resources when the physical, data link, network,
and transport layers are functional.

© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 41
Troubleshooting IP Connectivity
Components of Troubleshooting End-to-End Connectivity
 By employing a structured
approach to the troubleshooting
process, an administrator can
reduce the time it takes to
diagnose and solve a problem.
 Sample scenario:

• The client host PC1 is unable to

access applications on server
SRV1 or server SRV2.
• PC1 uses SLAAC with EUI-64 to
create its IPv6 global unicast
address

© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 42
Troubleshooting IP Connectivity
Components of Troubleshooting End-to-End Connectivity (Cont.)
 Step 1. Check physical connectivity at the
point where network communication stops.
 Step 2. Check for duplex mismatches.

 Step 3. Check data link and network layer

addressing on the local network.
 Step 4. Verify that the default gateway is
correct.
 Step 5. Ensure that devices are determining
the correct path from the source to the
destination. Manipulate the routing
information if necessary.
 Step 6. Verify that the transport layer is
functioning properly (Telnet can be used).
 Step 7. Verify that there are no ACLs
blocking traffic.
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 43
 Step 8. Ensure that DNS settings are correct.
Troubleshooting IP Connectivity
End-to-End Connectivity Problem Initiates Troubleshooting
 Ping and traceroute are the two most common
utilities to test end-to-end connectivity.
 The ping command uses a Layer 3 protocol
that is a part of the TCP/IP suite called ICMP.
• ping uses the ICMP echo request and ICMP
echo reply packets.
• ping can be used for IPv4 and IPv6
 The traceroute command illustrates the path
the IPv4 packets take to reach their destination.

• The Cisco IOS traceroute command can be

used for both IPv4 and IPv6
• The tracert command can be used on Windows
 The traceroute command is commonly
performed when the
© 2016 Cisco and/orping command
its affiliates. All fails.
rights reserved. Cisco Confidential 44
Troubleshooting IP Connectivity
Step 1 – Verify the Router#show interface G0/0

Physical Layer GigabitEthernet0/0 is up, line protocol is up

Hardware is CN Gigabit Ethernet, address is d48c.b5ce.a0c0
 The following IOS commands may be
(bia d48c.b5ce.a0c0)
used to verify suspected physical Internet address is 10.1.10.1/24
issues: ...

• show processes cpu Input queue: 0/75/0 (size/max/drops); Total output drops:0
Queueing strategy: fifo
• show memory Output queue :0/40 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
• show interfaces 5 minute output rate 0 bits/sec, 0 packets/sec
0 packets input, 0 bytes, 0 no buffer
 If device exhibits performance issues Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
and hardware is suspected to be at fault, 0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0
abort
use the show interfaces command and 0 watchdog, 1017 multicast, 0 pause input
pay attention to the following: 0 input packets with dribble condition detected
0 packets output, 0 bytes, 0 underruns
• Input queue drops 0 output errors, 0 collisions, 1 interface resets
0 unknown protocol drops
• Output queue drops 0 babbles, 0 late collision, 0 deferred
• Input errors 0 lost carrier, 0 no carrier
0 output buffer failures, 0 output buffers swapped out
• Output errors © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 45
Troubleshooting IP Connectivity
Step 2 – Check for Duplex Mismatches
S1#show interface Fa0/20  Duplex mismatch between two ends of
FastEthernet0/20 is up, line protocol is up
Hardware is CN Fast Ethernet, address is an Ethernet link is another common
0010.11c4.7801 (bia 0010.11c4.7801) cause for interface errors.
MTU 1500 bytes, BW 100000 Kbit, DLY 1000 usec,
reliability 255/255, txload 1/255, rxload 1/255  Interfaces use autonegotation to
Encapsulation ARPA, loopback not set
Keepalive set (10 sec) determine link speed and duplex
Full-duplex, Auto-speed, media type is 10/100BaseTX
... • The IEEE 802.3ab Gigabit Ethernet
standard mandates the use of
S2#show interface Fa0/20 autonegotiation for speed and duplex.
FastEthernet0/20 is up, line protocol is up
Hardware is CN Fast Ethernet, address is • Most Fast Ethernet NICs also use
0010.11c4.7801 (bia 0010.11c4.7801) autonegotiation by default.
MTU 1500 bytes, BW 100000 Kbit, DLY 1000 usec,
reliability 255/255, txload 1/255, rxload 1/255  Set the speed and duplex manually on
Encapsulation ARPA, loopback not set
Keepalive set (10 sec) both ends if autonegortiation fails.
Half-duplex, Auto-speed, media type is 10/100BaseTX
... • Point-to-point Ethernet links should
always run in full-duplex mode.
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 46
Troubleshooting IP Connectivity
Step 3 – Verify Layer 2 and 3 Addressing on the Local Network
 Look for VLAN assignment issues when
troubleshooting end-to-end connectivity
S1>show mac address-table issues (show vlan)
Mac Address Table
 The output of the show mac address-
-------------------------------------------
table command can also be helpful when
Vlan Mac Address Type Ports looking for VLAN assignment issues.
All 0100.0ccc.ccc STATIC
CPU
All 0100.0ccc.ccd STATIC R1
CPU
1 d48c.b5ce.a0c0 DYNAMIC PC1
Fa0/1
10 000f.34f9.9201 DYNAMIC
Fa0/5
10 5475.d08e.9ad8 DYNAMIC
Fa0/13
---- ----------- -------- -----
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 47
Total MAC Addresses for this criterion: 5
Troubleshooting IP Connectivity
Step 3 – Verify Layer 2 and 3 Addressing on the Local Network
 The arp Windows command can C:\WINDOWS\system32>arp -a
be used to help verify mappings Interface: 192.168.2.62 --- 0xe
between destination IP addresses Internet Address Physical Address Type
and Layer 2 Ethernet addresses. 10.1.10.1 d48c.b5ce.a0c0
dynamic
• The arp –d command can be 10.1.10.255 ff-ff-ff-ff-ff-ff static
used to clear the arp cache and 224.0.0.22 01-00-5e-00-00-16 static
allow it to repopulate with updated 224.0.0.251 01-00-5e-00-00-fb static
224.0.0.252 01-00-5e-00-00-fc static
info.
239.255.255.250 01-00-5e-7f-ff-fa static
255.255.255.255 ff-ff-ff-ff-ff-ff static
 The netsh interface ipv6 show neighbor Windows command will list all devices that are
currently in the neighbor table.
• By examining the neighbor table, the network administrator can verify that the destination IPv6
addresses map to correct Ethernet addresses.
 The show ipv6 neighbors command can be used on a Cisco IOS router.
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 48
Troubleshooting IP Connectivity
Step 4 – Verify Default Gateway  If there is no default route on the router
or if the host is configured with the
wrong default gateway, then
R1>show ip route communication between two endpoints
... on different networks will not work.
Gateway of last resort is 192.168.1.2 to network 0.0.0.0  Use the following commands to verify
S* 0.0.0.0/0 [1/0] via 192.168.1.2 • show ip route / show ipv6 route to
check for the router default route on R1
C:\WINDOWS\system32>route print
• ipconfig Windows command to verify if
IPv4 Route Table a PC has a default gateway
=============================================================
Active Routes: • Route print Windows command to
Network Destination Netmask Gateway Interface Metric check the PC routing table for a default
0.0.0.0 0.0.0.0 10.1.10.2 10.1.10.100 50 gateway
• show ipv6 interface GigabitEthernet
0/0 command to verify if router is a
member of the correct multicast group.
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 49
Troubleshooting IP Connectivity  When troubleshooting a connectivity issue,
Step 5 – Verify Correct Path verify the path to the destination network.
 Use either the show ip route or show ipv6
route command to verify that the route exists to
the destination device/network.
 Forwarding packets is based on the longest bit
match or longest prefix match. If the destination
address in a packet:
• Does not match any entry in the routing table,
then the default route is used; otherwise drop
• Matches a single entry in the routing table,
forward through the interface that is defined in
this route.
• Matches more than one entry in the routing table
and the routing entries have the same prefix
length, load-balance among the routes that are
defined in the routing table.

© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 50
Troubleshooting IP Connectivity
Step 6 – Verify the Transport Layer
 If the network layer appears to be functioning as expected, but users are still unable to
access resources, then troubleshoot the upper layers.
 Most common issues that affect transport: ACL and NAT configuration problems.

 A common tool for testing transport layer functionality is the Telnet utility.

 If a ping is successful to a server, then all layers below the network layer, between the user
and the server are operational. Issue is likely to be with Layer 4 or up.
 For example: R1# telnet 2001:db8:acad:3::2

R1>telnet 2001:db8:acad:3::2 80
Trying 2001:db8:acad:3::2, 80...
% Connection refused by remote host

R1#

© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 51
Troubleshooting IP Connectivity
Step 7 – Verify ACLs
 ACLs may prohibit protocols
from passing through the
interface in the inbound or
R3#show access-lists
Extended IP access list 100
outbound direction.
deny ip 172.16.1.0 0.0.0.255 any (3 match(es))  Use the following commands
permit ip any any
to display the contents of all
R3#show ip interface Serial0/0/1 | include access list ACLs:
Outgoing access list is not set
Inbound access list is not set • show ip access-lists
• show ipv6 access-list
R3#show ip interface gigabitethernet0/0 | include access list
Outgoing access list is not set  Use the following commands
Inbound access list 100
to see if there are ACLs set
on a particular interface:
• show ip interfaces
• show ipv6 interfaces

© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 52
Troubleshooting IP Connectivity
Step 7 – Verify DNS

C:\WINDOWS\system32>nslookup  When DNS is used in the network

Default Server: router.xyz.com and the DNS server is configured on
Address: 10.1.10.1 a device, you can substitute the
hostname for the IP address for all IP
> SRV1 commands including ping and
Server: router.xyz.com
telnet.
Address: 192.168.2.1
 On a PC, use the nslookup
Name: SRV1.xyz.com command to check for availability of
Addresses: 172.16.1.100 the DNS server

© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 53
Questions?
Module Summary
What Did You Learn In This Module?
• Common network documentation includes physical and logical network topologies, network
device documentation, and network performance baseline documentation.
• Baselining allows an administrator to document what is considered normal behavior / traffic
characteristics of the network
• The troubleshooting process should be guided by structured methods which involve
gathering symptoms, isolating the issue, implementing corrective action and documenting the
solution
• Several troubleshooting methods may be used depending on the nature of the problem
• Use bottom-up approach when encountering a new problem or if the problem appears to
be a physical issue
• Use top-down approach when a problem appears to be software in nature
• Use divide and conquer when a problem seems to be similar to a previously diagnosed
issue.

© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 55
Module Practice and Quiz
What did I learn in this module? (Cont.)
• Troubleshooting tools include
• Hardware tools: digital multimeters, cable testers, cable analyzers, portable network analyzers,
Cisco Prime NAM,
• Software tools: NMS tools, knowledge bases, baselining tools, protocol analyzer, and syslog
servers.
• When identifying the cause of an issue, it is important to recognize the probable network
layer where it lies
• Physical layer problems cause failures and suboptimal conditions.
• Data link layer problems are typically caused by encapsulation errors, address mapping errors,
framing errors, and STP failures or loops.
• Network layer problems include IPv4, IPv6, routing protocols (such as EIGRP, OSPF, etc.).
• Transport layer problems can be misconfigured NAT or ACLs.
• Application layer problems can result in unreachable or unusable resources.

© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 56