Capture System Data Usign Native Commands
Capture System Data Usign Native Commands
Performance Professionals
The Computer Measurement Group, commonly called CMG, is a not for profit, worldwide organization of data processing professionals committed to the
measurement and management of computer systems. CMG members are primarily concerned with performance evaluation of existing systems to maximize
performance (eg. response time, throughput, etc.) and with capacity management where planned enhancements to existing systems or the design of new
systems are evaluated to find the necessary resources required to provide adequate performance at a reasonable cost.
This paper was originally published in the Proceedings of the Computer Measurement Group’s 2005 International Conference.
Copyright 2005 by The Computer Measurement Group, Inc. All Rights Reserved. Published by The Computer Measurement Group, Inc. (CMG), a non-profit
Illinois membership corporation. Permission to reprint in whole or in any part may be granted for educational and scientific purposes upon written application to
the Editor, CMG Headquarters, 151 Fries Mill Road, Suite 104, Turnersville , NJ 08012.
BY DOWNLOADING THIS PUBLICATION, YOU ACKNOWLEDGE THAT YOU HAVE READ, UNDERSTOOD AND AGREE TO BE BOUND BY THE
FOLLOWING TERMS AND CONDITIONS:
License: CMG hereby grants you a nonexclusive, nontransferable right to download this publication from the CMG Web site for personal use on a single
computer owned, leased or otherwise controlled by you. In the event that the computer becomes dysfunctional, such that you are unable to access the
publication, you may transfer the publication to another single computer, provided that it is removed from the computer from which it is transferred and its use
on the replacement computer otherwise complies with the terms of this Copyright Notice and License.
Copyright: No part of this publication or electronic file may be reproduced or transmitted in any form to anyone else, including transmittal by e-mail, by file
transfer protocol (FTP), or by being made part of a network-accessible system, without the prior written permission of CMG. You may not merge, adapt,
translate, modify, rent, lease, sell, sublicense, assign or otherwise transfer the publication, or remove any proprietary notice or label appearing on the
publication.
Disclaimer; Limitation of Liability: The ideas and concepts set forth in this publication are solely those of the respective authors, and not of CMG, and CMG
does not endorse, approve, guarantee or otherwise certify any such ideas or concepts in any application or usage. CMG assumes no responsibility or liability
in connection with the use or misuse of the publication or electronic file. CMG makes no warranty or representation that the electronic file will be free from
errors, viruses, worms or other elements or codes that manifest contaminating or destructive properties, and it expressly disclaims liability arising from such
errors, elements or codes.
General: CMG reserves the right to terminate this Agreement immediately upon discovery of violation of any of its terms.
Learn the basics and latest aspects of IT Service Management at CMG's Annual Conference - www.cmg.org/conference
Buy the Latest Conference Proceedings and Find Latest Computer Performance Management 'How To' for All Platforms at www.cmg.org
Our company was paying large licensing fees for third-party products for gathering
systems data for capacity planning and performance management. The focus of
our effort was to save cost and at the same time obtain data for more mid-range
Join over 14,000 peers - subscribe to free CMG publication, MeasureIT(tm), at www.cmg.org/subscribe
systems (Unix, Linux, and Windows servers). We wrote scripts to gather our own
data using native commands.
This paper compares the various native commands and metrics capturing data in
AIX, Solaris, HP-UX, and Linux, and presents an overview of the capture
methodology. It also presents a high-level overview of how the data is used.
Windows data is captured in a similar manner, but is not covered in this paper.
Buy the Latest Conference Proceedings and Find Latest Computer Performance Management 'How To' for All Platforms at www.cmg.org
zipped files containing all of the detail data for • Data can be accessed for multi-server and multi-
a single day (see Table C). day reporting.
• SA data in the archive directory is converted to • Utilities are provided for converting the data to .csv
SAR reports at end of day. format.
• Configuration information is captured at end of
day. Since the data is now in .csv format, graphs can be
Join over 14,000 peers - subscribe to free CMG publication, MeasureIT(tm), at www.cmg.org/subscribe
• The archive files are retained on the agent servers created using Excel or any other analysis software that
for 7 days, and then removed. This means that 7 accepts .csv files.
full days of data are retained on the agent server,
even when data is transferred to the datahub. Excel is broadly used for 3 reasons:
Table D shows the actual native commands we are • It is universally available on every laptop.
using. In some cases, minor editing of data occurs on • We don’t have to educate the end user.
the agent server as the data is collected (no specific • The data resides in flat files on multiple hubs and
command output is presented). must be copied down to the user laptop prior to
analysis.
Managing Data If users want to use some other approach after they
copy it to their workstation (such as SAS or
We established six Windows datahubs to collect and GNUPLOT) they can, but we don’t provide support with
store our data. Activities on the datahubs include: the staff we have. Most other tools require the data to
be in a database on the hubs; we chose not to provide
• Data is transferred to the datahubs on a daily that type of support.
basis.
• Transfer is controlled by limiting transfers to four at Using the Data – Capacity Planning
a time; normally data transfer is finished shortly
after midnight. Most of our capacity planning analysts use Excel to
• For any given agent server, scripts on the datahub generate their analysis charts. CPU utilization is the
check what is already on the datahub and only variable most often used.
transfer files that are needed from the agent
server (remember that the agent server keeps We use Specints for comparison of computer speed
data for the last 7 days). (integer operation/sec measure of raw CPU speed)
• The data initially is transferred to a working area across different models and even vendors. We also
where it is “massaged”: frequently use transactions/second parameters that
− White space is compressed. are defined empirically for specific hardware and
− Inactive disk data records are removed from applications by our testing labs.
SAR reports.
− Zero length files are removed. Examples of Excel generated capacity planning charts
are reflected in Diagram E.
• Data is then moved to permanent directories.
• Data will be removed after one year.
• Granularity is maintained for the entire life of the
data; no peak/valley summarizations are
performed. Using the Data – Dashboards
Buy the Latest Conference Proceedings and Find Latest Computer Performance Management 'How To' for All Platforms at www.cmg.org
static content for the business partners to peruse at
their convenience. Examples are reflected in the
charts in Diagram F.
Troubleshooting
Results
Buy the Latest Conference Proceedings and Find Latest Computer Performance Management 'How To' for All Platforms at www.cmg.org
Metric Groups Interval Metric Unit Description
once per operating system name(sun,hp,aix,
config day Os linux,etc.)
os level operating system level
os version operating system Version
word operating system Word Length
Join over 14,000 peers - subscribe to free CMG publication, MeasureIT(tm), at www.cmg.org/subscribe
length (64/32)
memory KB physical memory
num of
cpu's number of cpu's
kernel parameters (important)
research
cpu speed (important)
hostname hostname
ip address ip address
block device model (nice to have)
nic type, full duplex y/n?, nic speed
(important)
physical swap configed kb
file once/day filesystem name name of filesystem
used kb used kb of filesystem
available kb available kb of filesystem
capacity kb capacity of filesystem
mounted
on mount path
used % % percentage used
network 5 min. name interface name
maximum transmission unit (packet
mtu size)
net/dest attached network
address address for each interface
ipkts pkts input packets per interface
ierrs errs input errors per interface
opkts pkts output packets per interface
oerrs errs output errors per interface
colls collisions collisions per interface
queue bytes send
bytes in / bytes out
% tcp resend
process 5 min. pid name process id
user name user name
cpu % cpu used by process
rss kb resident set size of process in KB
size of process in virtual memory in
vsz kb KB
command name command being executed
cpu 5 min. user % cpu in user mode
sys % cpu in system mode
Buy the Latest Conference Proceedings and Find Latest Computer Performance Management 'How To' for All Platforms at www.cmg.org
by individual cpu
average total cpu
load avg. 1,5,15 minutes (or 5,10,15
min - AIX)
portion of time device was busy
block device info. 5 min. busy % servicing a transfer request.
Join over 14,000 peers - subscribe to free CMG publication, MeasureIT(tm), at www.cmg.org/subscribe
system swapping/
switching 5 min. swpin/s transfers/sec swap ins
bswin/s blocks/sec blocks (512) swapped in
Buy the Latest Conference Proceedings and Find Latest Computer Performance Management 'How To' for All Platforms at www.cmg.org
Native Command Grouping Metric Unit Description Sun HP AIX
df filesystem name name of filesystem x x x
used kb used kb of filesystem x x x
Join over 14,000 peers - subscribe to free CMG publication, MeasureIT(tm), at www.cmg.org/subscribe
block device busy % portion of time device was busy servicing a transfer request x x x
avg queue requests average number of requests outstanding during that time. x x x
r+w/s reads+writes/sec. number of read/write transfers from or to device x x x
blks/sec blocks/sec. number of bytes transferred in 512-byte units. x x x
avwait ms average wait time in milliseconds. x x x
avserv ms average service time in milliseconds. x x x
average queue length of processes in memory and
queues runq-sz number of procs. runnable. x x x
runq-occ % percent of time run queue occupied x x x
swpq-sz number of procs. average length of swap queue of runnable processes n/a x x
percent of time swap queue of runnable processes was
swpocc % occupied n/a x x
buffer activity bread/s transfers per second between system buffers and disk or other block devices. x x x
bwrite/s transfers per second between system buffers and disk or other block devices. x x x
lread/s transfers per second access of system buffers x x x
lwrite/s transfers per second access of system buffers x x x
%rcache % cache hit ratios (1-bread/lread) x x x
%wcache % cache hit ratios (1-bread/lread) x x x
pread/s transfers per second transfers using raw (physical) device mechanism. x x x
pwrite/s transfers per second transfers using raw (physical) device mechanism. x x x
-
-
-
Buy the Latest Conference Proceedings and Find Latest Computer Performance Management 'How To' for All Platforms at www.cmg.org
atch/s flts/sec. page currently in memory (attaches per second).
pgin/s req/sec. page-in requests per second. x n/a n/a
ppgin/s pgs/sec. pages paged-in per second. x n/a n/a
page faults from protection errors per second (illegal access
pflt/s flts/sec. to page) or "copy-on-writes". x n/a n/a
address translation page faults per second (valid page not in
x n/a n/a
Join over 14,000 peers - subscribe to free CMG publication, MeasureIT(tm), at www.cmg.org/subscribe
pgscan/s pgs/sec. pages per second scanned by the page stealing daemon. x n/a n/a
the percentage of UFS inodes taken off the freelist by iget
which had reusable pages asso-
ciated with them.These pages are flushed and cannot be
reclaimed by processes.Thus, this is the percentage of igets
%ufs_ipf % with page flushes. x n/a n/a
/banktools/dnt/archive
Buy the Latest Conference Proceedings and Find Latest Computer Performance Management 'How To' for All Platforms at www.cmg.org
outfile AIX Sun HP Linux
cf uname -a uname -a syslog.log cpuinfo
oslevel isainfo -b vmstat -n meminfo
date date
ls -las /unix swap -s
Join over 14,000 peers - subscribe to free CMG publication, MeasureIT(tm), at www.cmg.org/subscribe
mp mpstat
pd prtdiag
sw lsps -a swap -s
up uptime uptime
E1
Buy the Latest Conference Proceedings and Find Latest Computer Performance Management 'How To' for All Platforms at www.cmg.org
Dot Com Web
250000
200000
Join over 14,000 peers - subscribe to free CMG publication, MeasureIT(tm), at www.cmg.org/subscribe
150000
Spec
100000
50000
0
Apr-04
Jun-04
Aug-04
Oct-04
Dec-04
Feb-05
Apr-05
Jun-05
Aug-05
Oct-05
Dec-05
Feb-06
Apr-06
Jun-06
Aug-06
Oct-06
Dec-06
Feb-07
Apr-07
Jun-07
Aug-07
Oct-07
Dec-07
Feb-08
Apr-08
Jun-08
Month
.COM Utilization (baseline) .COM Utilization (added) ABC Project Total .COM Capacity (spec)
Contingency Capacity (spec) 90% Contingency Capacity 80% Contingency Capacity 70% Contingency Capacity
.COM Actuals .COM Previous Forecast
E2
100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
Apr-04
May-04
Jun-04
Jul-04
Aug-04
Sep-04
Oct-04
Nov-04
Dec-04
Jan-05
Feb-05
Mar-05
Apr-05
May-05
Jun-05
Jul-05
Aug-05
Sep-05
Oct-05
Nov-05
Dec-05
Jan-06
Feb-06
Mar-06
Apr-06
May-06
Jun-06
Jul-06
Aug-06
Sep-06
Oct-06
Nov-06
Dec-06
Jan-07
Feb-07
Mar-07
Apr-07
E3
Buy the Latest Conference Proceedings and Find Latest Computer Performance Management 'How To' for All Platforms at www.cmg.org
160000
140000
120000
Join over 14,000 peers - subscribe to free CMG publication, MeasureIT(tm), at www.cmg.org/subscribe
100000
80000
60000
40000
20000
0
Apr-04
May-04
Jun-04
Jul-04
Aug-04
Sep-04
Oct-04
Nov-04
Dec-04
Jan-05
Feb-05
Mar-05
Apr-05
May-05
Jun-05
Jul-05
Aug-05
Sep-05
Oct-05
Nov-05
Dec-05
Jan-06
Feb-06
Mar-06
Apr-06
May-06
Jun-06
Jul-06
Aug-06
Sep-06
Oct-06
Nov-06
Dec-06
Jan-07
Feb-07
Mar-07
Apr-07
User Count Forecast Added Customers Keep Alive Total Model Capacity
Contingency Model Capacity 90% Capacity 80% Capacity 70% Capacity
User Count Actuals Model Session Previous Forecast
Buy the Latest Conference Proceedings and Find Latest Computer Performance Management 'How To' for All Platforms at www.cmg.org
Join over 14,000 peers - subscribe to free CMG publication, MeasureIT(tm), at www.cmg.org/subscribe
Buy the Latest Conference Proceedings and Find Latest Computer Performance Management 'How To' for All Platforms at www.cmg.org
Join over 14,000 peers - subscribe to free CMG publication, MeasureIT(tm), at www.cmg.org/subscribe
If we go back to Diagram F2 and click on the chart for ‘Week of 2005-08-25 to 2005-08-31, and ‘RV’, we get
Buy the Latest Conference Proceedings and Find Latest Computer Performance Management 'How To' for All Platforms at www.cmg.org
Join over 14,000 peers - subscribe to free CMG publication, MeasureIT(tm), at www.cmg.org/subscribe