Diagnose commands for FortiOS
3.0 (Some apply for FortiOS 2.8)
David Ramírez Joya, CISSP, FCNSP
Agenda
• Everyday troubleshooting tools
• FortiOS Debug route commands
• FortiOS Sniffer Demystified
• FortiOS Debug Flow
• FortiOS Application troubleshooting
• FortiOS system troubleshooting
• Best Practices to Escalate problems to TAC.
Everyday troubleshooting tips
Everyday Troubleshooting tips
• These are everyday commands used on Windows and Linux.
These might be something you already know, please bear with us
while we discuss these commands.
• In any case, the basis for this topic is common sense, and a
little of intuition in regard of the operative systems, network
and applications.
• A good tip to go further in regard or getting a better
understanding on how stuff work are:
Try to get a deep knowledge on the architecture of any operative
system.
Everyday Troubleshooting tips
• Operative system useful commands:
Netstat. This command shows the connection table of the operative
system along with other useful statistics.
Netstat –ona, vb
• This is the most useful one, it shows the following information per
column:
– Protocol
– Local Address and port
– Foreign address and port
– State
– Process ID.PID!
» The process ID can be matched with the task manager.
Everyday Troubleshooting tips
This is a “Daemon” or
C:\Documents and Settings\dramirez>netstat -onaservice listening for request
Active Connections
Proto Local Address Foreign Address State PID
TCP 0.0.0.0:80 0.0.0.0:0 LISTENING 3604
TCP 0.0.0.0:135 0.0.0.0:0 LISTENING 1660
TCP 0.0.0.0:443 0.0.0.0:0 LISTENING 3604
TCP 0.0.0.0:445 0.0.0.0:0 LISTENING 4
TCP 0.0.0.0:24913 0.0.0.0:0 This is an established
LISTENING 3604
TCP 127.0.0.1:1027 0.0.0.0:0 LISTENING 3032
TCP 127.0.0.1:1304 127.0.0.1:1305 connection to
ESTABLISHED 3956 a remote
TCP 127.0.0.1:1305 127.0.0.1:1304 destination
ESTABLISHED 3956
TCP 169.254.218.201:139 0.0.0.0:0 LISTENING 4
TCP 192.168.239.1:139 0.0.0.0:0 LISTENING 4
TCP 192.168.254.38:139 0.0.0.0:0 LISTENING 4
TCP 192.168.254.38:1104 216.59.240.50:4653 ESTABLISHED 3604
TCP 192.168.254.38:1307 64.233.167.99:80 ESTABLISHED 3956
TCP 192.168.254.38:1308 64.233.167.147:80 ESTABLISHED 3956
Everyday Troubleshooting
A TCP connectiontips
that is
trying to be established
Proto Local Address Foreign Address State PID
TCP 192.168.254.38:1322 4.2.2.1:25 SYN_SENT 168
UDP 0.0.0.0:1213 *:* 1540
UDP 0.0.0.0:6022 *:* 700
UDP 0.0.0.0:24913 *:* 3604
UDP 127.0.0.1:123 *:* 1008
UDP 127.0.0.1:1025 *:* 1008
UDP 127.0.0.1:1095 *:* 1008
UDP 127.0.0.1:1099 *:* 3604
A UDP daemon listening
connections
Everyday Troubleshooting tips
• Operative System Useful commands:
telnet <host> <port>
• This is a very useful command to test if a certain port is receiving
connections and with some protocols even to check if the service is
actually working.
The most common tests are for the following protocols:
• SMTP, POP3, IMAP, HTTP.
You can actually test any port for connection.
Everyday Troubleshooting tips
• An example for SMTP is:
C:\Documents and Settings\dramirez>telnet gama.fime.uanl.mx 25
This means the SMTP server
is answering connections, so
it’s working correctly
TIP: sometimes the SMTP server that is answering is NOT the
SMTP server you are expecting too, so be careful with this!
Everyday Troubleshooting tips
• An Example for HTTP is:
C:\>telnet
Welcome to Microsoft Telnet Client
Escape Character is 'CTRL+}'
Microsoft Telnet> set ?
bsasdel Backspace will be sent as delete
crlf New line mode - Causes return key to send CR & LF
delasbs Delete will be sent as backspace
escape x x is an escape charater to enter telnet client prompt
localecho Turn on localecho.
logfile x x is current client log file
logging Turn on logging
mode x x is console or stream
ntlm Turn on NTLM authentication.
term x x is ansi, vt100, vt52, or vtnt
Microsoft Telnet> set localecho
Local echo on
Microsoft Telnet>
Everyday Troubleshooting tips
The web server answered the
• Microsoft Telnet> open gama.fime.uanl.mx 80 command correctly with a
• Connecting To gama.fime.uanl.mx... webpage!
• GET / HTTP/1.0
Try the following
commands:
•OPTIONS / HTTP/1.0
•BLABLA
Everyday Troubleshooting tips
• Traceroute
This commands exists in windows and linux.
It is widely recommended to use without the DNS resolution find the
parameter for your favorite OS platform.
• Arp, get sys arp for FortiOS
This command is VERY USEFUL for troubleshooting “dead units”
or simply to check if we are sending information to the unit we are
TRYING to contact.
• Pathping IP
Detects latency on the links
Enabling Debug Output
• The following commands should be used when enabling
debugging output in the FortiOS.
diag debug info
• This command shows the current debug configuration status.
diag debug enable
• This commans enables the actual output of debug information.
diag debug console timestamp enable
• This should be enabled when sending debug output to the TAC.
FortiOS Debug route commands
• The debugging for routing has been already described but
there are more commands that we should try.
• diag ip route ?
This command allows you to debug the router daemons, along with
the restarting of the processes of the routing daemon.
• get router ?
This command allows you to get routing information from all the
routing daemons of the FortiOS.
FortiOS Debug route commands
The options shown below in BOLD, are the command options
that you should ONLY USE. The usage of the other options
might cause product instability that COULD ONLY be
corrected by rebooting the unit. In the worst case, you could
be forced to restore a configuration or reinstall the firmware.
You’ve been warned.
• diag ip
address IP addresses
arp ARP table
multicast multicast information
route routing table
router router
rtcache routing cache
tcp TCP sockets
udp UDP sockets
FortiOS Debug route commands
• To show the IPs as seen by the FortiOS Kernel, use:
diag ip address
• add add IP address
• delete delete IP address
• flush flush IP addresses
• list list IP addresses
• To show the arp table, as seen by the FortiOS Kernel, use:
diag ip arp
• add add an ARP entry
• delete delete an ARP entry
• flush flush ARP table
• list show ARP table
FortiOS Debug route commands
The following command branch will show you the routing table
exactly as it is in the FortiOS Kernel. If a route is not here, it
does not matter if it is configured or not, please check if there
is no interface connected.
• diag ip route
add add static route
delete delete static route
flush flush routing table
list list routing table
verify verify static route
• You SHOULD ONLY USE the “list” command. Using any of
the other commands, specially flush, can cause FortiOS
instability that COULD ONLY be corrected by rebooting the
unit.
FortiOS Debug route commands
This command branch allows you to get all the routing information
packets as they are being received by the FortiOS. This output will
help you out to see any problem that might be with dynamic routing
or multicast, but you should understand this protocol’s informative
output.
• diag ip router
bfd BFD debug
bgp BGP protocol
command Send command to routing daemon
igmp IGMP debug
ospf OSPF protocol
pim-dm PIM dense-mode
pim-sm PIM sparse-mode
rip RIP protocol
FortiOS Debug route commands
• This is an example configuration for the rip daemon.
The same applies for each daemon process.
• diag ip router rip
all Enable all debugging
events RIP events
level debug level
packet-receive RIP receive events
packet-send RIP send events
show show status of rip debugging
• The command to enable ALL logging to console for rip
events is:
diag ip router rip all enable
FortiOS Debug route commands
The following command branch helps you out to show the actual
routeing table in “friendly” format. Any similaties with another
vendor’s output is merely a coincidence ;-)
• get router
access-list access list configuration
aspath-list AS path list configuration
bgp router bgp configuration
community-list community list configuration
info show routing infomation
key-chain Key-chain configuration
multicast router multicast configuration
ospf router ospf configuration
Policy policy routing configuration
prefix-list prefix list configuration
rip router rip configuration
route-map route map configuration
static routing table configuration
static6 routing table configuration
FortiOS Debug route commands
• The output of the “get router” command, mostly shows the
configuration of the given routing protocol. We will be
focusing with the command branch of “get router info” here.
• get router info
routing-table show routing table information
protocols show routing protocols information
rip show rip information
ospf show ospf information
bgp show router info bgp information
multicast show routing multicast information
bfd show BFD information
FortiOS Debug route commands
• The “info” command, shows the actual information that the
FortiOS has for each routing protocol in question. The
“routing-table” option has a very friendly way to show the
FortiOS routing table.
• get router info
routing-table show routing table information
protocols show routing protocols information
rip show rip information
ospf show ospf information
bgp show router info bgp information
multicast show routing multicast information
bfd show BFD information
FortiOS Debug route commands
This command will show the portion of the routing table
requested. The “details” option will ask for an specific route
or host to check.
• get router info routing-table
details show routing table details information
all show all routing table entries
rip show rip routing table
ospf show ospf routing table
bgp show bgp routing table
static show static routing table
connected show connected routing table
database show routing information base
FortiOS Debug route commands
The output of the “get router info routing-table all” command is:
• get router info routing-table all
Codes: K - kernel, C - connected, S - static, R - RIP, B - BGP
O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external
type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, ia - IS-IS inter
area
* - candidate default
S* 0.0.0.0/0 [1/0] via 172.168.0.1, wan1
C 172.168.0.0/24 is directly connected, wan1
FortiOS Debug route commands
• The debug and informational routing commands mentioned
in the past slides should be everything you need to
troubleshoot a problem with dynamic or static routing
problems.
• A protocol not updating what it should, a routing ACL not
being applied correctly, everything can be seen as the
routing daemons show it by using the branches of:
diag ip router
• To check if a route has been correctly added to the FortiOS
routing table, in the correct format and information, in a
friendly way, can be seen by using the commands of:
get router
FortiOS Sniffer Demystified
• A regular IP header looks like: ToS bits explained
FortiOS Sniffer Demystified
• A Regular TCP Header looks like:
• A Regular UDP Header looks like:
FortiOS Sniffer Demystified
• diag sniffer packet <interface> <filter> <verbose>
<interface> the network interface to sniff (or "any")
<filter> flexible logical filters for sniffer (or "none")
<verbose>
• 1: print header of packets
• 2: print header and data from ip of packets
• 3: print header and data from ethernet of packets (if available)
• 4: print header of packets with interface name
• 5: print header and data from ip of packets with interface name
• 6: print header and data from ethernet of packets (if available) with intf
name
FortiOS Sniffer Demystified
• How to create advanced filters?
• The TCPDUMP manual page is your best friend!
http://www.tcpdump.org/tcpdump_man.html
• We will, create SEVERAL filters today.
FortiOS Sniffer Demystified
• Let’s sniff!
Sniff a Web session.
Sniff an email session.
Sniff an udp session.
Sniff messenger!
Sniff a several hosts.
Sniff specific hosts and ports
Sniff specific source and destination hosts with ports!
Sniff some non regular protocols!
FortiOS Debug Flow, 3.0 Only
• Diag debug flow
This is the best command we currently have to track down session
creation flow and policy Matching.
FortiOS Debug Flow, 3.0 Only
FGT-602803031526 # diag debug flow
filter trace packet with filter
show whether to display trace on console
trace start/stop trace
FGT-602803031526 # diag debug flow filter
addr ip address
clear clear filter
daddr dest ip address
dport destination port
negate inverse filter
port port
proto protocol number
saddr source ip address
sport source port
vd index of virtual domain, -1 matches all
FortiOS Debug Flow, 3.0 Only
FGT-602803031526 # diag debug flow show
console whether to display trace on console
function-name whether to show function name
FGT-602803031526 # diag debug flow trace
start start trace
stop stop trace
FortiOS Application Control
• The following command branch is very helpful to control
some of the FortiOS processes. Like stopping and restarting
a given process. Process statistics and functional output, etc.
• BE ADVISED: The use of this commands is only
recommended for situations where you actually need the
troubleshooting or the debug output as the output of this
commands MIGHT CAUSE FortiOS instability.
FortiOS Application Control
dia test application <APP> <TEST_LEVEL>
• Where APP could be:
ftpd ftp proxy
http http proxy
im im proxy
imap imap proxy
ipldbd ipldbd daemon
ipsengine ips sensor
ipsmonitor ips monitor
nntp nntp proxy
pop3 pop3 proxy
scanunit scanning unit
smtp smtp proxy
urlfilter urlfilter daemon
FortiOS Application Control
• And the <TEST_LEVEL> COULD BE:
This TEST_LEVEL is “could be” because not all the processes have the
same test levels. Again, use them with care.
This example is for the:
• FTP Proxy Test Usage
1: Dump Memory Usage
2: Drop all connections
4: Display connection stat
44: Display info per connection
444: Display connections per state
5: Toggle AV Bypass mode
6: Toggle Print Stat mode every ~40 seconds
7: Toggle Backlog Drop
8: Clear stats
88: Toggle statistic recording - stats cleared
9: Toggle Accounting info for display
99: Restart proxy
FortiOS Application Control
• The most commonly requested output by the TAC is:
1: Dump Memory Usage
4: Display connection stat
44: Display info per connection
444: Display connections per state
• The applications that have this output options are:
http, ftp, smtp, pop3, imap nntp
• The other option available to most of the commands is:
99: Restart proxy
This is the best command to use when having a process that might be using
too much resources without having to reboot the complete unit.
FortiOS Application Control
• Another useful command that you already used these days
is:
• dia test authserver <auth_proto> <options>
Which can be used to test the credentials of a user when enabling a
particular protocol authentication.
Each protocol has its own sintax, please use the inline help of the
FortiOS to provide the required parameters.
FortiOS Application Troubleshooting
• Diag debug application <process> <debug level>
To enable ALL DEBUG output from a given process, you should
specify “-1” as the debug level. This debug level works for all the
processes that can be monitored with debugging.
• BE ADVISED: The use of this commands is only
recommended for situations where you actually need the
troubleshooting or the debug output as the output of this
commands MIGHT CAUSE FortiOS instability.
FortiOS Application Troubleshooting
• DNS application troubleshooting.
• diag debug haproxy ?
clear clear haproxy cache
dump dump haproxy cache
fqdndump dump fqdn data
fqdnflush flush ip info of fqdn
reset reset statistics
stats show statistics
• The FortiOS calls the DNS process as haproxy.
FortiOS Application Troubleshooting
• This is the Authd daemon troubleshooting.
dia debug authd fsae
• clear-logons clear logon information
• list list current logons
• refresh-groups refresh group mappings
• refresh-logons resync logon database
• server-status show FSAE server connection status
• summary summary of current logons
dia debug authd
• clear clear internal data structures and keepalive sessions
• fsae FSAE client module
FortiOS FortiGuard Troubleshooting
There are some specific commands for troubleshooting
FortiGuard Services.
• dia deb rating
This commands outputs the actual FGD servers that the FG unit is
contacting.
• dia spamfilter fortishield statistics list
This command outputs statistics to the given fortiguard service
• dia webfilter fortiguard statistics list
This command outputs statistics to the given fortiguard service
• These commands should be used for statistical purporses
only, per the request of the TAC team.
FortiOS Application Troubleshooting
You can be creative.
• You can enable the debugging of any of the processes on the
FortiOS, and sometimes there are more than 2 processes that
are being used in a given FortiOS functionality.
• Careful, with too much debug enabled you can easily get too
much debug output in your console and most of the times,
unreadable.
• For example, when troubleshooting a SSL VPN tunnel mode,
you could enable the sslvpn troubleshooting. In tunnel mode,
the FortiOS assigns an IP to the actual sslvpn connection so
you should enable the ppp daemon also.
Diag debug app sslvpn -1
Diag debug app ppp -1
FortiOS Application Troubleshooting
• Sometimes, there is a problem with some FortiOS process
that might be crashing after a failure. When this happens,
there is a way to read this crashlog. With all the console
output logged to a local file, you can send this information to
the TAC.
• With debugging output enabled:
Diag debug crashlog read
• If the unit is crashing or freezing, you should connect a
computer directly to the console connection and enable the
following in the FortiOS:
diag debug en
diag debug console timestamp en
diag debug kernel level 5
FortiOS Application Troubleshooting
• Anatomy of the Crashlog output:
• Line 01: Build 0316 is a MR2 Interim build and no customer
should be running it in production. Bare minimum customers
should be using the Latest MR3 Patch build. If they require
HTTPS Web Filtering, AV, Web content archiving, IM
inspection etc they should be running the latest MR4 Patch
build.
MKTFG300A # dia deb crashlog read
1: 2007-06-13 12:03:17 <00065> firmware Fortigate-300A
3.00,build316,060613
FortiOS Application Troubleshooting
• Line 02: scanunit is the parent process and session allocator for all
AV proxies
• Line 03: signal 7 is usually quite rare to see, this would be most
likely something related to a memory leak
• Line 05: Antivirus Database signature/checksum (CPRL)
information
2: 2007-06-13 12:03:19 <00065> application scanunit
3: 2007-06-13 12:03:19 <00065> *** signal 7 (Bus error) received ***
4: 2007-06-13 12:03:19 <00065>
03000000AVEN00100010000606131944
5: 2007-06-13 12:03:19 <00065> AVDB
03000000AVDB00050075840705250507
6: 2007-06-13 12:03:19 <00065> AVSO
03000000AVEN00200020020601261145
FortiOS Application Troubleshooting
7: 2007-06-13 12:03:19 <00065> 13: 2007-06-13 12:03:19 <00065>
Register dump: ESP/signal: bfffc7cc CR2: 40a03d1d
8: 2007-06-13 12:03:19 <00065> 14: 2007-06-13 12:03:19 <00065>
• Line 18: scanunit
EAX: 409fd000 EBX: has a number of children processes (varies
Backtrace:
on diff.
bfffc844 ECX:hw models)EDX:
00006d1e the child process died and
15: 2007-06-13 was <00065>
12:03:19 restarted.
Code 7 is a kill signal 7 bus error
00000000 see line 3.
[0x08053931] => /bin/scanunitd
9:• 2007-06-13
Line 19: 12:03:19
Signal 11 is a seg
<00065> fault.
ESI: 16:This would 12:03:19
2007-06-13 indicate<00065>
there is
a hardware/memory
00006d1d EDI: leak. Total[0x08053245]
is the amount of memory
=> /bin/scanunitd
bfffc844 EBP: bfffc7d4
reserved ESP: bfffc7cc
for the process (depends on the memory
17: 2007-06-13 12:03:19available).
<00065>
10: 2007-06-13
Total=503MB12:03:19
and<00065>
free=223MB. [0x0804dba4]
The numbers=> /bin/scanunitd
are in
EIP: megabytes.
08053931 EFLAGS: 00010206 18: 2007-06-13 12:03:18
11: 2007-06-13 12:03:19 <00065> scanunit=child pid=66 exittype=exit
CS: 0023 DS: 002b ES: code=7 total=503 free=113
002b FS: 0000 GS: 0007 SS: 002b 19: 2007-06-13 12:03:20
12: 2007-06-13 12:03:19 <00065> scanunit=child pid=65 exittype=exit
Trap: 0000000e Error: code=11 total=503 free=223
00000006 OldMask: 00000000
FortiOS Application Troubleshooting
Line 20: IMD (User level IM proxy) exited conserve mode total
memory 224MB, free 110MB, margin to enter conserve mode is
10MB and it would leave conserve mode when 20MB of memory are
available.
Line 21: SMTP daemon failure mode is to deactivate, if fail
open or bypass is on then the traffic would flow. Otherwise
by default most proxies go into blocking mode when AV is not
available.
Line 22: thttp is the HTTP AV proxy. Signal 5 is trace,
breakpoint, range error, divide by zero, or overflow.
Line 23: Process ID 54, total memory 503MB, free 321mb
FortiOS Application Troubleshooting
20: 2007-06-13 12:03:18 <00056> proxy=imd conserve=exited
total=224 free=110 marginenter=10 marginexit=20
21: 2007-06-13 12:03:18 <00047> proxy=smtp session fail
mode=deactivated
22: 2007-06-13 12:03:18 <00054> proxy=thttp
subprocess=scanunit crashed=crashed code=5
23: 2007-06-13 12:03:20 <00054> proxy=thttp pid=54 total=503
free=321
24: 2007-06-13 12:03:20 <00054> dump current
connection(0x8775508) information
25: 2007-06-13 12:03:20 <00054> clt=1286(r=0, w=0)
srv=1287(r=0, w=0)
FortiOS Application Troubleshooting
26: 2007-06-13 12:03:20 <00054> 10.3.20.53:3705 >64.62.216.75:80s=RESPONSE_SCANUNIT_STATE
27: 2007-06-13 12:03:20 <00054> server -> client: 35177 = (0x41ffa969 - 0x41ff2000)
28: 2007-06-13 12:03:21 <00054> HTTP/1.1 200 OK.
29: 2007-06-13 12:03:21 <00054> Content-Length: 34976.
30: 2007-06-13 12:03:21 <00054> Content-Type: image/png.
31: 2007-06-13 12:03:21 <00054> Cache-Control:max-age=30060983.
32: 2007-06-13 12:03:21 <00054> Expires: Mon, 26 May 2008 17:19:40 GMT.
33: 2007-06-13 12:03:19 <00048> proxy=pop3 session fail mode=deactivated
34: 2007-06-13 12:03:20 <00056> proxy=imd session fail mode=deactivated
35: 2007-06-13 12:03:21 <00054> Date: Wed, 13 Jun 2007 19:03:17 GMT.
36: 2007-06-13 12:03:21 <00054>Connection: keep-alive.
37: 2007-06-13 12:03:22 <00054> .
38: 2007-06-13 12:03:22 <00054> .PNG.
39: 2007-06-13 12:03:22 <00054> .
40: 2007-06-13 12:03:22 <00054> ....IHDR..............O.P...gIDATx.....m.u....(.9-
41: 2007-06-13 12:03:22 <00054>
[email protected]....`..... T.#..b..iE.b],....1.E..uhS..
%^dF.DE......={.c...k..!..L..ns.5.....y.]..K.z.v.|...o.....s.........
42: 2007-06-13 12:03:22 <00054> request hostname: us.maps2.yimg.com
43: 2007-06-13 12:03:22 <00054> request url: /us.png.maps.yimg.com/png?v=3.52&t=m&x=1200y=509&z=6
44: 2007-06-13 12:03:22 <00054> proxy=thttp session fail mode=deactivated
FortiOS Application Troubleshooting
• Line 45-55 : the IPS monitor is restarted every time the rules are
updated, by default we check FDN every hour for new definitions.
45: 2007-06-13 12:06:13 the killed daemon is /bin/ipsmonitor
46: 2007-06-13 13:06:12 the killed daemon is /bin/ipsmonitor
47: 2007-06-13 14:06:12 the killed daemon is /bin/ipsmonitor
48: 2007-06-13 15:06:14 the killed daemon is /bin/ipsmonitor
49: 2007-06-13 16:06:11 the killed daemon is /bin/ipsmonitor
50: 2007-06-13 17:06:13 the killed daemon is /bin/ipsmonitor
51: 2007-06-13 18:06:12 the killed daemon is /bin/ipsmonitor
52: 2007-06-13 19:06:11 the killed daemon is /bin/ipsmonitor
53: 2007-06-13 20:06:11 the killed daemon is /bin/ipsmonitor
54: 2007-06-13 21:06:11 the killed daemon is /bin/ipsmonitor
55: 2007-06-13 22:06:10 the killed daemon is /bin/ipsmonitor
Debugging FortiOS HA
• The following commands SHOULD ONLY BE USED when
there is a problem with the FortiGate HA and you are sending
this information per the TAC request.
• Most of this information it is not in a easily understandable
format. Its output should be handled with care.
• The commands are:
dia sys ha status
• This command shows a simple status statistics of the HA in the
FortiOS.
Debugging FortiOS HA
• dia sys ha mac
Shows the MAC addresses being used by the FortiOS while in HA.
• diag sys ha dump <#>
This command dumpt the HA information. You can send different
kind of debug output from number 1 to number 5.
• diag sys ha showcsum <#>
This command will show the actual checksum of a configuration
object from the FortiOS. You can specify a level from level 1 to
level 7. This command would be most of the time useless unless
there is a real problem with synchronization and the TAC has
requested the information.
Debugging FortiOS HA
• When having to debug the HA daemons in real-time, you
could use the commands:
• dia deb application hatalk -1
This command will enable ALL debug output from the hatalk
daemon.
• dia deb application hasync -1
This command will enable ALL debug output from the hasync,
daemon.
• BE ADVISED: This information should be for TAC escalation
purposes ONLY.
Debugging FortiOS HA
Useful Links for TAC escalation.
• How to diagnose FortiOS v3.0 HA out of sync messages
http://kc.fortinet.com/default.asp?SID=&Lang=1&id=3001
• FortiOS v2.80 and v3.0 HA out of sync messages and the
objects that they reference
http://kc.fortinet.com/default.asp?SID=&Lang=1&id=2997
FortiOS Firewall Troubleshooting
The following commands show the actual status of the firewall state.
Most of the times this information is not usable for troubleshooting
purposes, but there are some command output that is sometimes
useful.
• diag firewall
fqdn fqdn
iplist ip list
ipmac ipmac
iprope iprope
iprope6 iprope6
ldb server load balance
proute Policy route
schedule schedule
statistic traffic statistics
FortiOS Firewall Troubleshooting
• The following command shows the properties of the firewall
connection table, as seen by the FortiOS. This branch has also
some self-explanatory useful commands.
dia firewall iprope
• authuser list authenticated users
• clear clear policy statistic
• flush flush
• list list
• resetauth resetauth
• resetfsae resetfsae
• show show policy statistic
• state state
• AGAIN, these FortiOS commands COULD CREATE INSTABILITY in
The FortiOS. USE THEM WITH CARE.
FortiOS System Troubleshooting
Basic Status Information.
• get sys status
When troubleshooting FortiGates, in FortiOS 3.0. Always use the “full-
configuration” option when showing the FortiGate Configuration.
• This can be enabled by adding the “full” word while showing:
• show full dnsconfig system dns
• show full sys fortiguard
• show full sys global
• Etc.
FortiOS System Troubleshooting
• Diag sys top <refresh_time> <# of processes>
Shows the process with their memory consumption and their CPU usage
The meaning of the letters in the second line is:
• U: user cpu usage (%)
• S: system cpu usage (%)
• I: idle cpu usage (%)
• T: total memory (MB)
• F: free memory (MB)
• KF: kernel free memory (MB)
• Diag sys kill <PID> <signal>
The PID can be obtained by the use of diag sys top
The signals most used are:
• 15 –TERM signal, ask the process to quit nicely
• 9 – KILL signal, forcefully shut down the process.
FortiOS System Troubleshooting
• dia sys session stat
Session Table status output.
• dia sys session ttl list
Lists the actual session TTL that is being applied by the FortiOS.
• Diag sys session
This command branch allows us to show and optionally, filter the
sessions from the FortiOS session table. Alternatively, you could
clear all the sessions from the table, USE THIS WITH CARE.
• clear clear the sessions defined by filter
• filter list session with filters
• help session help
• list list session
FortiOS Hardware Troubleshooting
This command branch allows us to get information from the hardware
attached to the FortiOS. This should be use mostly for informational
purposes with the exception of some commands that could be
helpful for troubleshooting.
• dia hardware
deviceinfo get device information
ioport read/write data via IO port
lspci list PCI parameters
pciconfig get PCI information
setpci set PCI parameters
sysinfo get system information
• Diag hardware deviceinfo nic <interface name>
This commands shows the physical state of an interface per its kernel driver
on the FortiOS
FortiOS Hardware Troubleshooting
• dia hardware sysinfo mem
This command shows the memory status per the FortiOS.
• dia hardware sysinfo slab
The output of this command, is the amount of memory objects
occupied by image name. The actual description of the fields on the
output is:
• Name, # Curr active objects, Total # objects, sizeof Obj Bytes, # Pages
1 active object, # alloc pages, # pages per Slab.
This command can be very useful to pointing processes that are
using abnormal amounts of memory.
• The rest of the commands are left for the students to test and
try. Use them wisely, please.
Best Practices to Escalate problems to
TAC.
• The KC includes a very useful troubleshooting guide with
most of the commands explained here, with usage examples
and recommendations.
• Troubleshooting guide.
http://kc.fortinet.com/default.asp?SID=&Lang=1&id=2094
Best Practices to Escalate problems to
TAC.
• When creating a ticket on the FortiCARE System once you
have determined there is an issue or bug on the FortiOS,
there are very well documented steps to escalate a ticket on
the following KC article:
http://kc.forticare.com/browsefile.asp?id=1475&SID=
The PDF file with the actual information can be found at:
• http://kc.forticare.com/redirfile.asp?id=1475&SID=
• This information, along with the Troubleshooting guide
mentioned before, will help you out to fully document a ticket
request, this will speed up the troubleshooting part for the
TAC engineers and we will also help us on escalating the
ticket easier in case it is needed
THANKS!
Questions?
David Ramírez Joya, CISSP, FCNSP
SE LATAM
