Table of Contents

Copyright................................................................................................................................ 1
Preface................................................................................................................................... 2
Acknowledgments................................................................................................................. 10
Part 1: Basics......................................................................................................................... 11
Chapter 1. Input/Output Basics..................................................................................................................................................................................................... 12
Perl and Networking.................................................................................................................................................................................................................. 12
Networking Made Easy.............................................................................................................................................................................................................. 13
Filehandles................................................................................................................................................................................................................................. 15
Using Object-Oriented Syntax with the IO::Handle and IO::File Modules............................................................................................................................. 30
Summary.................................................................................................................................................................................................................................... 35
Chapter 2. Processes, Pipes, and Signals...................................................................................................................................................................................... 36
Processes.................................................................................................................................................................................................................................... 36
Pipes........................................................................................................................................................................................................................................... 39
Signals........................................................................................................................................................................................................................................ 48
Summary.................................................................................................................................................................................................................................... 55
Chapter 3. Introduction to Berkeley Sockets................................................................................................................................................................................ 56
Clients, Servers, and Protocols.................................................................................................................................................................................................. 56
Berkeley Sockets........................................................................................................................................................................................................................ 59
Socket Addressing...................................................................................................................................................................................................................... 63
A Simple Network Client........................................................................................................................................................................................................... 67
Network Names and Services.................................................................................................................................................................................................... 69
Network Analysis Tools............................................................................................................................................................................................................. 73
Summary.................................................................................................................................................................................................................................... 76
Chapter 4. The TCP Protocol......................................................................................................................................................................................................... 77
A TCP Echo Client...................................................................................................................................................................................................................... 77
Socket Functions Related to Outgoing Connections................................................................................................................................................................. 79
A TCP Echo Server..................................................................................................................................................................................................................... 80
Adjusting Socket Options.......................................................................................................................................................................................................... 85
Other Socket-Related Functions............................................................................................................................................................................................... 88
Exceptional Conditions during TCP Communications............................................................................................................................................................. 89
Summary.................................................................................................................................................................................................................................... 91
Chapter 5. The IO::Socket API...................................................................................................................................................................................................... 92
Using IO::Socket........................................................................................................................................................................................................................ 92
IO::Socket Methods................................................................................................................................................................................................................... 94
More Practical Examples........................................................................................................................................................................................................... 99
Concurrent Clients................................................................................................................................................................................................................... 105
Summary................................................................................................................................................................................................................................... 111
Part 2: Developing Clients for Common Services................................................................. 112
Chapter 6. FTP and Telnet........................................................................................................................................................................................................... 113
Net::FTP.................................................................................................................................................................................................................................... 113
Net::Telnet................................................................................................................................................................................................................................ 123
Summary.................................................................................................................................................................................................................................. 136
Chapter 7. SMTP: Sending Mail................................................................................................................................................................................................... 137
Introduction to the Mail Modules............................................................................................................................................................................................ 137
Net::SMTP................................................................................................................................................................................................................................ 137
MailTools.................................................................................................................................................................................................................................. 143
MIME-Tools.............................................................................................................................................................................................................................. 151
Summary.................................................................................................................................................................................................................................. 170
Chapter 8. POP, IMAP, and NNTP.............................................................................................................................................................................................. 171
The Post Office Protocol........................................................................................................................................................................................................... 171
The IMAP Protocol................................................................................................................................................................................................................... 184
Internet News Clients............................................................................................................................................................................................................... 187
A News-to-Mail Gateway.......................................................................................................................................................................................................... 197
Summary.................................................................................................................................................................................................................................. 202
Chapter 9. Web Clients............................................................................................................................................................................................................... 203
Installing LWP......................................................................................................................................................................................................................... 203
LWP Basics.............................................................................................................................................................................................................................. 204
LWP Examples......................................................................................................................................................................................................................... 215
Part 3: Developing TCP Client/Server Systems.................................................................... 247
Chapter 10. Forking Servers and the inetd Daemon.................................................................................................................................................................. 248
Standard Techniques for Concurrency................................................................................................................................................................................... 248
Running Example: A Psychotherapist Server......................................................................................................................................................................... 250
The Psychotherapist as a Forking Server................................................................................................................................................................................. 251
A Client Script for the Psychotherapist Server........................................................................................................................................................................ 256
Daemonization on UNIX Systems........................................................................................................................................................................................... 259
Starting Network Servers Automatically................................................................................................................................................................................. 264
Using the inetd Super Daemon............................................................................................................................................................................................... 267
Summary.................................................................................................................................................................................................................................. 272
Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Chapter 11. Multithreaded Applications..................................................................................................................................................................................... 273
About Threads.......................................................................................................................................................................................................................... 273
A Multithreaded Psychiatrist Server....................................................................................................................................................................................... 279
A Multithreaded Client............................................................................................................................................................................................................ 281
Summary.................................................................................................................................................................................................................................. 282
Chapter 12. Multiplexed Applications........................................................................................................................................................................................ 284
A Multiplexed Client................................................................................................................................................................................................................ 284
The IO::Select Module............................................................................................................................................................................................................. 286
A Multiplexed Psychiatrist Server........................................................................................................................................................................................... 289
Summary.................................................................................................................................................................................................................................. 295
Chapter 13. Nonblocking I/O...................................................................................................................................................................................................... 296
Creating Nonblocking I/O Handles........................................................................................................................................................................................ 296
Using Nonblocking Handles.................................................................................................................................................................................................... 298
Using Nonblocking Handles with Line-Oriented I/O............................................................................................................................................................ 300
A Generic Nonblocking I/O Module....................................................................................................................................................................................... 305
Nonblocking Connects and Accepts........................................................................................................................................................................................ 322
Summary.................................................................................................................................................................................................................................. 332
Chapter 14. Bulletproofing Servers............................................................................................................................................................................................. 333
Using the System Log.............................................................................................................................................................................................................. 333
Setting User Privileges............................................................................................................................................................................................................. 346
Taint Mode............................................................................................................................................................................................................................... 350
Using chroot().......................................................................................................................................................................................................................... 353
Handling HUP and Other Signals........................................................................................................................................................................................... 355
Summary.................................................................................................................................................................................................................................. 361
Chapter 15. Preforking and Prethreading................................................................................................................................................................................... 363
Preforking................................................................................................................................................................................................................................ 363
Prethreading............................................................................................................................................................................................................................ 387
Performance Measures............................................................................................................................................................................................................ 395
Summary.................................................................................................................................................................................................................................. 395
Chapter 16. IO::Poll..................................................................................................................................................................................................................... 396
Using IO::Poll.......................................................................................................................................................................................................................... 396
IO::Poll Events......................................................................................................................................................................................................................... 397
IO::Poll Methods..................................................................................................................................................................................................................... 399
A Nonblocking TCP Client Using IO::Poll.............................................................................................................................................................................. 399
Summary.................................................................................................................................................................................................................................. 402
Part 4: Advanced Topics..................................................................................................... 403
Chapter 17. TCP Urgent Data...................................................................................................................................................................................................... 404
"Out-of-Band" Data and the Urgent Pointer.......................................................................................................................................................................... 404
Using TCP Urgent Data........................................................................................................................................................................................................... 406
The sockatmark() Function...................................................................................................................................................................................................... 411
A Travesty Server..................................................................................................................................................................................................................... 413
Summary.................................................................................................................................................................................................................................. 423
Chapter 18. The UDP Protocol.................................................................................................................................................................................................... 424
A Time of Day Client................................................................................................................................................................................................................ 424
Creating and Using UDP Sockets............................................................................................................................................................................................ 426
UDP Errors.............................................................................................................................................................................................................................. 428
Using UDP Sockets with IO::Socket........................................................................................................................................................................................ 429
Sending to Multiple Hosts....................................................................................................................................................................................................... 431
UDP Servers............................................................................................................................................................................................................................. 433
Increasing the Robustness of UDP Applications.................................................................................................................................................................... 435
Summary.................................................................................................................................................................................................................................. 442
Chapter 19. UDP Servers............................................................................................................................................................................................................. 443
An Internet Chat System......................................................................................................................................................................................................... 443
The Chat Client........................................................................................................................................................................................................................ 446
The Chat Server........................................................................................................................................................................................................................ 453
Detecting Dead Clients............................................................................................................................................................................................................ 463
Summary.................................................................................................................................................................................................................................. 468
Chapter 20. Broadcasting............................................................................................................................................................................................................ 470
Unicasting versus Broadcasting.............................................................................................................................................................................................. 470
Broadcasting Explained........................................................................................................................................................................................................... 471
Sending and Receiving Broadcasts.......................................................................................................................................................................................... 472
Broadcasting Without the Broadcast Address........................................................................................................................................................................ 475
Enhancing the Chat Client to Support Resource Discovery................................................................................................................................................... 484
Summary.................................................................................................................................................................................................................................. 486
Chapter 21. Multicasting............................................................................................................................................................................................................. 488
Multicast Basics....................................................................................................................................................................................................................... 488
Using Multicast........................................................................................................................................................................................................................ 493
Sample Multicast Applications............................................................................................................................................................................................... 500
Summary.................................................................................................................................................................................................................................. 513
Chapter 22. UNIX-Domain Sockets............................................................................................................................................................................................ 514
Using UNIX-Domain Sockets.................................................................................................................................................................................................. 514
A "Wrap" Server....................................................................................................................................................................................................................... 518
Using UNIX-Domain Sockets for Datagrams.......................................................................................................................................................................... 521

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Summary.................................................................................................................................................................................................................................. 525
Appendix A. Additonal Source Code.................................................................................... 526
Net::NetmaskLite (Chapter 3).................................................................................................................................................................................................... 526
PromptUtil.pm (Chapters 8 and 9)............................................................................................................................................................................................. 528
IO::LineBufferedSet (Chapter 13)............................................................................................................................................................................................... 530
IO::LineBufferedSessionData (Chapter 13)................................................................................................................................................................................ 532
DaemonDebug (Chapter 14)........................................................................................................................................................................................................ 536
Text::Travesty (Chapter 17)......................................................................................................................................................................................................... 538
mchat_client.pl (Chapter 21)...................................................................................................................................................................................................... 540
Appendix B. Perl Error Codes and Special Variables........................................................... 544
System Error Constants.............................................................................................................................................................................................................. 544
Magic Variables Affecting I/O..................................................................................................................................................................................................... 547
Other Perl Globals....................................................................................................................................................................................................................... 548
Appendix C. Internet Reference Tables............................................................................... 549
Assigned Port Numbers............................................................................................................................................................................................................... 549
Registered Port Numbers............................................................................................................................................................................................................ 574
Internet Multicast Addresses...................................................................................................................................................................................................... 588
Appendix D. Bibliography................................................................................................... 590
Perl Programming....................................................................................................................................................................................................................... 590
TCP/IP and Berkeley Sockets...................................................................................................................................................................................................... 591
Network Server Design................................................................................................................................................................................................................ 592
Multicasting................................................................................................................................................................................................................................. 592
Application-Level Protocols........................................................................................................................................................................................................ 593

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 xi

Copyright

Network Programming with Perl

Many of the designations used by manufacturers and sellers to distinguish their products are claimed
as trademarks. Where those designations appear in this book and we were aware of a trademark
claim, the designations have been printed in initial capital letters or all capitals.
The author and publisher have taken care in the preparation of this book, but make no expressed
or implied warranty of any kind and assume no responsibility for errors or omissions. No liability is
assumed for incidental or consequential damages in connection with or arising out of the use of the
information or programs contained herein.

Copyright Information
Copyright © 2001 by Addison-Wesley
All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or
transmitted, in any form or by any means, electronic, mechanical, photocopying, recording, or oth-
erwise, without the prior consent of the publisher. Printed in the United States of America. Published
simultaneously in Canada.
The publisher offers discounts on this book when ordered in quantity for special sales. For more
information, please contact:
Visit us on the Web at http://www.awl. com/cseng/ [http://www.awl.com/cseng/]
Library of Congress Cataloging-in-Publication Data
Stein, Lincoln D.
Network programming with Perl / Lincoln Stein.
p. cm.
1. Perl (Computer program language). 2. Internet programming. I. Title.
QA76.73.P22 S73 2000
005.2'762--dc21
00-067574
Text printed on recycled paper.
1 2 3 4 5 6 7 8 9 10 - MA - 04 03 02 01 00
First printing, December 2000

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 xii

Preface

The network is everywhere. At the office, machines are wired together into local area networks, and
the local networks are interconnected via the Internet. At home, personal computers are intermit-
tently connected to the Internet or, increasingly, via "always-on" cable and DSL modems. New

Licensed by
wireless technologies, such as Bluetooth, promise to vastly expand the network realm, embracing
everything from cell phones to kitchen appliances.
Such an environment creates tremendous opportunities for innovation. Whole new classes of ap-
plications are now predicated on the availability of high-bandwidth, always-on connectivity. Inter-

Stjepan Maric
active games allow players from around the globe to compete on virtual playing fields and the instant
messaging protocols let them broadcast news of their triumphs to their friends. New peer-to-peer
systems, such as Napster and Gnutella, allow people to directly exchange MP3 audio files and other
types of digital content. The SETI@Home project takes advantage of idle time on the millions of
personal computers around the world to search for signs of extraterrestrial life in a vast collection

4218908
of cosmic noise.
The ubiquity of the network allows for more earthbound applications as well. With the right knowl-
edge, you can write a robot that will fetch and summarize prices from competitors' Web sites; a
script to page you when a certain stock drops below a specified level; a program to generate daily
management reports and send them off via e-mail; a server that centralizes some number-crunching
task on a single high-powered machine, or alternatively distributes that task among the multiple
nodes of a computer cluster.
Whether you are searching for the best price on a futon or for life in a distant galaxy, you'll need to
understand how network applications work in order to take full advantage of these opportunities.
You'll need a working understanding of the TCP/IP protocol—the common denominator for all In-
ternet-based communications and the most common protocol in use in local area networks as well.
You'll need to know how to connect to a remote program, to exchange data with that program, and
what to do when something goes wrong. To work with existing applications, such as Web servers,
you'll have to understand how the application-level protocols are built on top of TCP/IP, and how to
deal with common data exchange formats such as XML and MIME.
This book uses the Perl programming language to illustrate how to design and implement practical
network applications. Perl is an ideal language for network programming for a number of reasons.
First, like the rest of the language, Perl's networking facilities were designed to make the easy things
easy. It takes just two lines of code to open a network connection to a server somewhere on the
Internet and send it a message. A fully capable Web server can be written in a few dozen lines of
code.
Second, Perl's open architecture has encouraged many talented programmers to contribute to an
ever-expanding library of useful third-party modules. Many of these modules provide powerful in-
terfaces to common network applications. For example, after loading the LWP::Simple module, a
single function call allows you to fetch the contents of a remote Web page and store it in a variable.
Other third-party modules provide intuitive interfaces to e-mail, FTP, net news, and a variety of
network databases.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Preface xiii

Perl also provides impressive portability. Most of the applications developed in this book will run
without modification on UNIX machines, Windows boxes, Macintoshes, VMS systems, and OS/2.
However, the most compelling reason to choose Perl for network application development is that it
allows you to fully exploit the power of TCP/IP. Perl provides you with full access to the same low-
level networking calls that are available to C programs and other natively compiled languages. You
can create multicast applications, implement multiplexed servers, and design peer-to-peer systems.
Using Perl, you can rapidly prototype new networking applications and develop interfaces to existing
ones. Should you ever need to write a networking application in C or Java, you'll be delighted to
discover how much of the Perl API carries over into these languages.

This Book's Audience

Network Programming with Perl is written for novice and intermediate Perl programmers. I assume
you know the basics of Perl programming, including how to write loops, how to construct if-else
statements, how to write regular expression pattern matches, the concept of the automatic $_ var-
iable, and the basics of arrays and hashes.
You should have access to a Perl interpreter and some experience writing, running, and debugging
scripts. Just as important, you should have access to a computer that is connected both to a local
area network and to the Internet! Although the recipes in Chapter 10 on setting Perl-based network
servers to start automatically when a machine is booted do require superuser (administrative) ac-
cess, none of the other examples require privileged access to a machine.
This book does take advantage of the object-oriented features in Perl version 5 and higher, but most
chapters do not assume a deep knowledge of this system. Chapter 1 addresses all the details you
will need as a casual user of Perl objects.
This book is not a thorough review of the TCP/IP protocol at the lowest level, or a guide to installing
and configuring network hubs, routers, and name servers. Many good books on the mechanics of
the TCP/IP protocol and network administration are listed in Appendix D.

Roadmap
This book is organized into four main parts: Basics, Developing Clients for Common Services, De-
veloping TCP Client/Server Systems, and Advanced Topics.
Part I, Basics, introduces the fundamentals of TCP/IP network communications.
• Chapters 1 and 2, Networking Basics and Processes, Pipes, and Signals, review Perl's functions
and variables for input and output, discusses the exceptions that can occur during I/O operations,
and uses the piped filehandle as the basis for introducing sockets. These chapters also review
Perl's process model, including signals and forking, and introduces Perl's object-oriented ex-
tensions.
• Chapter 3, Introduction to Berkeley Sockets, discusses the basics of Internet networking and
discusses IP addresses, network ports, and the principles of client/server applications. It then
turns to the Berkeley Socket API, which provides the programmer's interface to TCP/IP.
• Chapters 4 and 5, The TCP Protocol and The IO::Socket API and Simple TCP Applications,
show the basics of TCP, the networking protocol that provides reliable stream-oriented com-
munications. These chapters demonstrate how to create client and server applications and then
introduce examples that show the power of technique as well as some common roadblocks.
Part II, Developing Clients for Common Services, looks at a collection of the best third-party modules
that developers have contributed to the Comprehensive Perl Archive Network (CPAN).

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Preface xiv

• Chapter 6, FTP and Telnet, introduces modules that provide access to the FTP file-sharing
service, as well as to the flexible Net::Telnet module which allows you to create clients to access
all sorts of network services.
• E-mail is still the dominant application on the Internet, and Chapter 7, SMTP: Sending Mail,
introduces half of the equation. This chapter shows you how to create e-mail messages on the
fly, including binary attachments, and send them to their destinations.
• Chapter 8, POP, IMAP, and NNTP: Processing Mail and Netnews, covers the other half of e-
mail, explaining modules that make it possible to receive mail from mail drop systems and proc-
ess their contents, including binary attachments.
• Chapter 9, Web Clients, discusses the LWP module, which provides everything you need to talk
to Web servers, download and process HTML documents, and parse XML.
Part III, Developing TCP Client/Server Systems—the longest part of the book—discusses the al-
ternatives for designing TCP-based client/server systems. The major example used in these chap-
ters is an interactive psychotherapist server, based on Joseph Weizenbaum's classic Eliza program.
• Chapter 10, Forking Servers and the inetd Daemon, covers the common type of TCP server that
forks a new process to handle each incoming connection. This chapter also covers the UNIX
and Windows inetd daemons, which allow programs not specifically designed for networking to
act as servers.
• Chapter 11, Multithreaded Applications, explains Perl's experimental multithreaded API, and
shows how it can greatly simplify the design of TCP clients and servers.
• Chapters 12 and 13, Multiplexed Operations and Nonblocking I/O, discuss the select() call,
which enables an application to process multiple I/O streams concurrently without using multi-
processing or multithreading.
• Chapter 14, Bulletproofing Servers, discusses techniques for enhancing the reliability and main-
tainability of network servers. Among the topics are logging, signal handling, and exceptions, as
well as the important topic of network security.
• Chapter 15, Preforking and Prethreading, presents the forking and threading models discussed
in earlier chapters. These enhancements increase a server's ability to perform well under heavy
loads.
• Chapter 16, IO::Poll, discusses an alternative to select() available on UNIX platforms. This
module allows applications to multiplex multiple I/O streams using an API that some people find
more natural than select()'s.
Part IV, Advanced Topics, addresses techniques that are useful for specialized applications.
• Chapter 17, TCP Urgent Data, is devoted to TCP urgent or "out of band" data. This technique
is often used in highly interactive applications in which the user urgently needs to signal the
remote server.
• Chapters 18 and 19, The UDP Protocol and UDP Servers, introduce the User Datagram Pro-
tocol, which provides a lightweight, message-oriented communications service. Chapter 18 in-
troduces the protocol, and Chapter 19 shows how to design UDP servers. The major example
in this and the next two chapters contain a live online chat and messaging system written entirely
in Perl.
• Chapters 20 and 21, Broadcasting and Multicasting, extend the UDP discussion by showing how
to build one-to-all and one-to-many message broadcasting systems. In these chapters we ex-
tend the chat system to take advantage of automatic server discovery and multicasting.
• Chapter 22, UNIX-Domain Sockets, shows how to create lightweight communications channels
between processes on the same machine. This can be useful for specialized applications such
as loggers.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Preface xv

The Many Versions of Perl

All good things evolve to meet changing conditions, and Perl has gone through several major
changes in the course of its short life. This book was written for version of Perl in the 5.X series
(5.003 and higher recommended). At the time I wrote this preface (August 2000), the most recent
version of Perl was 5.6, with the release of 5.7 expected imminently. I expect that Perl versions 5.8
and 5.9 (assuming there will be such versions) will be compatible with the code examples given
here as well.
Over the horizon, however, is Perl version 6. Version 6, which is expected to be in early alpha form
by the summer of 2001, will fix many of the idiosyncrasies and misfeatures of earlier versions of
Perl. In so doing, however, it is expected to break most existing scripts. Fortunately, the Perl lan-
guage developers are committed to developing tools to automatically port existing scripts to version
6. With an eye to this, I have tried to make the examples in this book generic, avoiding the more
obscure Perl constructions.

Cross-Platform Compatibility
More serious are the differences between implementations of Perl on various operating systems.
Perl started out on UNIX (and Linux) systems, but has been ported to many different operating
systems, including Microsoft Windows, the Macintosh, VMS, OS/2, Plan9, and others. A script writ-
ten for the Windows platform will run on UNIX or Macintosh without modifications.
The problem is that the I/O subsystem (the part of the system that manages input and output op-
erations) is the part that differs most dramatically from operating system to operating system. This
restricts the ability of Perl to make its I/O system completely portable. While Perl's basic I/O func-
tionality is identical from port to port, some of the more sophisticated operations are either missing
or behave significantly differently on non-UNIX platforms. This affects network programming, of
course, because networking is fundamentally about input and output.
In this book, Chapters 1 through 9 use generic networking calls that will run on all platforms. The
exception to this rule is the last example in Chapter 5, which calls a function that isn't implemented
on the Macintosh, fork(), and some of the introductory discussion in Chapter 2 of process man-
agement on UNIX systems. The techniques discussed in these chapters are all you need for the
vast majority of client programs, and are sufficient to get a simple server up and running. Chapters
10 through 22 deal with more advanced topics in server design. The table here shows whether the
features in the chapters are supported by UNIX, Windows, or the Macintosh ports of Perl.

Chapter Subject UNIX/Linux Windows Macintosh

1–9 Basic network programming + + +
10 Forking servers + P −
11 Multithreaded servers + + −
12 Multiplexing + + +
13 Nonblocking I/O + + +
14 Server bulletproofing + P −
15 Preforking and prethreading + − −
16 IO::Poll + − −
17 TCP urgent data + − −
18 UDP + + +
19 UDP servers + + +
20 Broadcasting + + +
21 Multicasting + − −
Key: + features supported; − features unsupported; P partial support

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Preface xvi

Chapter Subject UNIX/Linux Windows Macintosh

22 UNIX-domain sockets + − −
Key: + features supported; − features unsupported; P partial support

The nice thing is that the non-UNIX ports of Perl are improving rapidly, and there is a good chance
that new features will be available at the time you read this.

Getting the Code for the Code Examples

All the sample scripts and modules discussed in this book are available on the Web in ZIP and TAR/
GZIP formats. The URL for downloading the source is http://www.modperl.com/perl_networking.
This page also includes instructions for unpacking and installing the source code.

Installing Modules
Many of Perl's networking modules are preinstalled in the standard distribution. Others are third-
party modules that you must download and install from the Web. Most third-party modules are written
in pure Perl, but some, including several that are mentioned in this book, are written partly in C and
must be compiled before they can be used.
CPAN is a large Web-based collection of contributed Perl modules. You can get access to it via a
Web or FTP browser, or by using a command-line application built into Perl itself.

Installing from the Web

To find a CPAN site near you, point your Web browser at http://www.cpan.org/. This will present a
page that allows you to search for specific modules, or to browse the entire list of contributed mod-
ules sorted in various ways. When you find the module you want, download it to disk.
Perl modules are distributed as gzipped tar archives. You can unpack them like this:
% gunzip -c Digest-MD5-2.00.tar.gz | tar xvf -
Digest-MD5-2.00/
Digest-MD5-2.00/typemap
Digest-MD5-2.00/MD2/
Digest-MD5-2.00/MD2/MD2.pm
...

Once the archives are unpacked, you'll enter the newly created directory and give the perl Make-
file.PL, make, make test, and make install commands. These will build, test, and install the module.
% cd Digest-MD5-2.00
% perl Makefile.PL
Testing alignment requirements for U32...
Checking if your kit is complete...
Looks good
Writing Makefile for Digest::MD2
Writing Makefile for Digest::MD5
% make
mkdir ./blib
mkdir ./blib/lib
mkdir ./blib/lib/Digest
...
% make test
make[1]: Entering directory '/home/lstein/Digest-MD5-2.00/MD2'
make[1]: Leaving directory '/home/lstein/Digest-MD5-2.00/MD2'
PERL_DL_NONLAZY=1 /usr/local/bin/perl -I./blib/arch -I./blib/lib...
t/digest............ok
t/files.............ok
t/md5-aaa...........ok

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Preface xvii

t/md5...............ok
t/rfc2202...........ok
t/sha1..............skipping test on this platform
All tests successful.
Files=6, Tests=291, 1 secs ( 1.37 cusr 0.08 csys = 1.45 cpu)
% make install
make[1]: Entering directory '/home/lstein/Digest-MD5-2.00/MD2'
make[1]: Leaving directory '/home/lstein/Digest-MD5-2.00/MD2'
Installing /usr/local/lib/perl5/site_perl/i586-linux/./auto/Digest/MD5/MD5.so
Installing /usr/local/lib/perl5/site_perl/i586-linux/./auto/Digest/MD5/MD5.bs
...

On UNIX systems, you may need superuser privileges to perform the final step. If you don't have
such privileges, you can install the modules in your home directory. At the perl Makefile.PL step,
provide a PREFIX= argument with the path of your home directory. For example, assuming your
home directory can be found at /home/jdoe, you would type:
% perl Makefile.PL PREFIX=/home/jdoe

The rest of the install procedure is identical to what was shown earlier.
If you are using a custom install directory, you must tell Perl to look in this directory for installed
modules. One way to do this is to add the name of the directory to the environment variable
PERL5LIB. For example:
setenv PERL5LIB /home/jdoe # C shell
PERL5LIB=/home/jdoe; export PERL5LIB # bourne shell

Another way is to place the following line at the top of each script that uses an installed module.
use lib '/home/jdoe';

Installing from the Command Line

A simpler way to do the same thing is to use Andreas Koenig's wonderful CPAN shell. With it, you
can search, download, build, and install Perl modules from a simple command-line shell. The in-
stall command does it all:
% perl -MCPAN -e shell
cpan shell -- CPAN exploration and modules installation (v1.40)
ReadLine support enabled
cpan> install MD5
Running make for GAAS/Digest-MD5-2.00.tar.gz
Fetching with LWP:
ftp://ftp.cis.ufl.edu/pub/perl/CPAN/authors/id/GAAS/Digest-MD5-2.00.tar.gz
CPAN: MD5 loaded ok
Fetching with LWP:
ftp://ftp.cis.ufl.edu/pub/perl/CPAN/authors/id/GAAS/CHECKSUMS
...
Checksum for /home/lstein/.cpan/sources/authors/id/GAAS/Digest-MD5-2.00.tar.gz
ok
Digest-MD5-2.00/
Digest-MD5-2.00/typemap
...
Installing /usr/local/lib/perl5/site_perl/i586-linux/./auto/Digest/MD5/MD5.so
Installing /usr/local/lib/perl5/site_perl/i586-linux/./auto/Digest/MD5/MD5.bs
Installing /usr/local/lib/perl5/site_perl/i586-linux/./auto/MD5/MD5.so
...
Writing /usr/local/lib/perl5/site_perl/i586-linux/auto/MD5/.packlist
Appending installation info to /usr/local/lib/perl5/i586-linux/
5.00404/perllocal.pod
cpan> exit

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Preface xviii

Installing Modules with the Perl Package Manager

These examples all assume that you have UNIX-compatible versions of the gzip, tar and make
commands. Virgin Windows systems do not have these utilities. The Cygwin package, available
from http://www.cygnus.com/cygwin/, provides these utilities as part of a complete set of UNIX-
compatible tools.
It is easier, however, to use the ActiveState Perl Package Manager (PPM). This Perl script is in-
stalled by default in the ActiveState distribution of Perl, available at http://www.activestate.com. Its
interface is similar to the command-line CPAN interface shown in the previous section, except that
it can install precompiled binaries as well as pure-Perl scripts. For example:
C:\WINDOWS>ppm
PPM interactive shell (1.1.3) - type 'help' for available commands.
PPM> install MD5
Install package 'MD5?' (y/N) : Y
Retrieving package 'MD5' ...
Installing C:\Perl\site\lib\auto\MD5\MD5.bs
Installing C:\Perl\site\lib\auto\MD5\MD5.dll
Installing C:\Perl\site\lib\auto\MD5\MD5.exp
Installing C:\Perl\site\lib\auto\MD5\MD5.lib
Installing C:\Perl\site\lib\MD5.pm
Installing C:\Perl\site\lib\auto\MD5\autosplit.ix
Writing C:\Perl\site]lib\auto\MD5\.packlist
PPM> exit
Quit!
C:\WINDOWS>

Installing Modules from MacPerl

The MacPerl Module Porters site, http://pudge.net/cgi-bin/mmp.plx, contains a series of modules
that have been ported for use in MacPerl. A variety of helper programs have been developed to
make module installation easier on the Macintosh. The packages are described at http://pudge.net/
macperl/macperlmodinstall.html, which also gives instructions on downloading and installling them.

Online Documentation
In addition to books and Web sites, Network Programming with Perl refers to two major sources of
online information, Internet RFCs and Perl POD documentation.

Internet RFCs
The specifications of all the fundamental protocols of the Internet are described in a series of Re-
quests for Comment (RFC) submitted to the Internet Engineering Task Force (IETF). These docu-
ments are numbered sequentially. For example RFC 1927—"Suggested Additional MIME Types for
Associating Documents"—was the 1927th RFC submitted. Some of these RFCs eventually become
Internet Standards, in which case they are given sequentially numbered STD names. However,
most of them remain RFCs. Even though the RFCs are unofficial, they are the references that people
use to learn the details of networking protocols and to validate that a particular implementation is
correct.
The RFC archives are mirrored at many locations on the Internet, and maintained in searchable
form by several organizations. One of the best archives is maintained at http://www.faqs.org/rfcs/.
To retrieve an RFC from this site, go to the indicated page and type the number of the desired RFC
in the text field labeled "Display the document by number." The document will be delivered in a

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Preface xix

minimally HTMLized form. This page also allows you to search for standards documents, and to
search the archive by keywords and phrases. If you prefer a text-only form, the http://
www.faqs.org site contains a link to their FTP site, where you can find and download the RFCs in
their original form.

Plain Old Documentation

Much of Perl's internal documentation comes in Plain Old Documentation (POD) format. These are
mostly plain text, with a few markup elements inserted to indicate headings, subheadings, and
itemized lists.
When you installed Perl, the POD documentation was installed as well. The POD files are located
in the pod subdirectory of the Perl library directory. You can either read them directly, or use the
perldoc script to format and display them in a text pager such as more.
To use perldoc type the command and the name of the POD file you wish to view. The best place
to start is the Perl table of contents, perltoc:
% perldoc perltoc

This will give you a list of other POD pages that you can display.
For a quick summary of a particular Perl function, perldoc accepts the -f flag. For example, to see
a summary of the socket() function, type:
% perldoc -f socket

For Macintosh user's the MacPerl distribution comes with a "helper" application called shuck. This
adds POD viewing facilities to the MacPerl Help menu.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 xx

Acknowledgments

They say that the first skill an editor learns on the job is patience, but I think that Karen Gettman
was born with an excess of it. She must have caught on after the second or third time that when I
said "it should be done in just another week," I really was talking about months. Yet she never
betrayed any sign of dismay, even though I'm sure she was fighting an increasingly restive produc-

Licensed by
tion and marketing staff. To Karen, all I can say is "thank you!"
Thanks also to Mary Hart, the assistant editor responsible for my book. I have worked with Mary on
other projects, and I know that it is her tireless effort that makes publishing with Addison-Wesley
seem so frictionless.

Stjepan Maric
I am extremely grateful to the technical reviewers who worked so diligently to keep me honest: Jon
Orwant, James Lee, Harry Hochheiser, Robert Kolstad, Sander Wahls, and Megan Conklin. The
book is very much better because of your efforts.
I owe a debt of gratitude to the long-suffering members of my laboratory—Ravi, David, Marco, Hong,

4218908
Guanming, Nathalie, and Peter; they have somehow managed to keep things moving forward even
during the last months of manuscript preparation, when my morning absences became increasingly
extended.
And of course I wish to thank my wife, Jean, who has stuck with me through several of these projects
already, and has never, ever, asked for the dining room table back.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 1

Part 1. Basics

The four chapters that follow will provide the fundamental knowledge you need to write networking
applications in Perl using Berkeley sockets. They set the stage for later parts of the book that delve
more deeply into specific network problems and their solutions.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 2

Chapter 1. Input/Output Basics

This chapter provides you with the background information you'll need to write TCP/IP applications
in Perl. We review Perl's input/output (I/O) system using the language's built-in function calls, and
then using the object-oriented (OO) extensions of Perl5. This will prepare you to use the object-
oriented constructions in later chapters.

Perl and Networking

Why would you want to write networking applications in Perl?
The Internet is based on Transmission Control Protocol/Internet Protocol (TCP/IP), and most net-
working applications are based on a straightforward application programming interface (API) to the
protocol known as Berkeley sockets. The success of TCP/IP is due partly to the ubiquity of the
sockets API, which is available for all major languages including C, C++, Java, BASIC, Python,
COBOL, Pascal, FORTRAN, and, of course, Perl. The sockets API is similar in all these languages.
There may be a lot of work involved porting a networking application from one computer language
to another, but porting the part that does the socket communications is usually the least of your
problems.
For dedicated Perl programmers, the answer to the question that starts this chapter is clear—be-
cause you can! But for those who are not already members of the choir, one can make a convincing
argument that not only is networking good for Perl, but Perl is good for networking.

A Language Built for Interprocess Communication

Perl was built from the ground up to make it easy to do interprocess communication (the thing that
happens when one program talks to another). As we shall see later in this chapter, in Perl there is
very little difference between opening up a local file for reading and opening up a communications
channel to read data from another local program. With only a little more work, you can open up a
socket to read data from a program running remotely on another machine somewhere on the In-
ternet. Once the communications channel is open, it matters little whether the thing at the other end
is a file, a program running on the same machine, or a program running on a remote machine. Perl's
input/output functions work in the same way for all three types of connections.

A Language Built for Text Processing

Another Perl feature that makes it good for network applications is its powerful integrated regular
expression-matching and text-processing facilities. Much of the data on the Internet is text based
(the Web, for instance), and a good portion of that is unpredictable, line-oriented data. Perl excels
at manipulating this type of data, and is not vulnerable to the type of buffer overflow and memory
overrun errors that make networking applications difficult to write (and possibly insecure) in lan-
guages like C and C++.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Input/Output Basics 3

An Open Source Project

Perl is an Open Source project, one of the earliest. Examining other people's source code is the
best way to figure out how to do something. Not only is the source code for all of Perl's networking
modules available, but the whole source tree for the interpreter itself is available for your perusal.
Another benefit of Perl's openness is that the project is open to any developer who wishes to con-
tribute to the library modules or to the interpreter source code. This means that Perl adds features
very rapidly, yet is stable and relatively bug free.
The universe of third-party Perl modules is available via a distributed Web-based archive called
CPAN, for Comprehensive Perl Archive Network. You can search CPAN for modules of interest,
download and install them, and contribute your own modules to the archive. The preface to this book
describes CPAN and how to reach it.

Object-Oriented Networking Extensions

Perl5 has object-oriented extensions, and although OO purists may express dismay over the fast
and loose way in which Perl has implemented these features, it is inarguable that the OO syntax
can dramatically increase the readability and maintainability of certain applications. Nowhere is this
more evident than in the library modules that provide a high-level interface to networking protocols.
Among many others, the IO::Socket modules provide a clean and elegant interface to Berkeley
sockets; Mail::Internet provides cross-platform access to Internet mail; LWP gives you everything
you need to write Web clients; and the Net::FTP and Net::Telnet modules let you write interfaces to
these important protocols.

Security
Security is an important aspect of network application development, because by definition a network
application allows a process running on a remote machine to affect its execution. Perl has some
features that increase the security of network applications relative to other languages. Because of
its dynamic memory management, Perl avoids the buffer overflows that lead to most of thesecurity
holes in C and other compiled languages. Of equal importance, Perl implements a powerful "taint"
check system that prevents tainted data obtained from the network from being used in operations
such as opening files for writing and executing system commands, which could be dangerous.

Performance
A last issue is performance. As an interpreted language, Perl applications run several times more
slowly than C and other compiled languages, and about par with Java and Python. In most net-
working applications, however, raw performance is not the issue; the I/O bottleneck is. On I/O-bound
applications Perl runs just as fast (or as slowly) as a compiled program. In fact, it's possible for the
performance of a Perl script to exceed that of a compiled program. Benchmarks of a simple Perl-
based Web server that we develop in Chapter 12 are several times better than the C-based Apache
Web server.
If execution speed does become an issue, Perl provides a facility for rewriting time-critical portions
of your application in C, using the XS extension system. Or you can treat Perl as a prototyping
language, and implement the real application in C or C++ after you've worked out the architectural
and protocol details.

Networking Made Easy

Before we get into details, let's look at two simple programs.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Input/Output Basics 4

The lgetl.pl script (for "line get local," Figure 1.1) reads the first line of a local file. Call it with the path
to the file you want to read, and it will print out the top line. For example, here's what I see when I
run the script on a file that contains a quote from James Hogan's "Giants Star":

Figure 1.1. lgetl.pl—Read the first line of a local file

% lgetl.pl giants_star.txt
"Reintegration complete," ZORAC advised. "We're back in the universe."

This snippet illustrates the typographic conventions this book uses for terminal (command-line in-
terpreter) sessions. The "%" character is the prompt printed out by my command-line interpreter.
Bold-faced text is what I (the user) typed. Everything else is regular monospaced font.
The script itself is straightforward:
Lines 1–2: Load modules—We use() the IO::File module, which wraps an object-oriented in-
terface around Perl file operations.
Line 3: Process the command line argument—We shift() the filename off the command line
and store it in a variable named $file.
Line 4: Open the file—We call the IO::File->new() method to open the file, returning a
filehandle, which we store in $fh. Don't worry if the OO syntax is unfamiliar to you; we discuss
it more later in this chapter.
Lines 5–6: Read a line from the filehandle and print it—We use the <> operator to read a line of
text from the filehandle into the variable $line, which we immediately print.
Now we'll look at a very similar script named lgetr.pl (for "line get remote," Figure 1.2). It too fetches
and prints a line of text, but instead of reading from a local file, this one reads from a remote server.
Its command-line argument is the name of a remote host followed by a colon and the name of the
network service you want to access.

Figure 1.2. lgetr.pl—Read the first line from a remote server

To read a line of text from the "daytime" service running on the FTP server wuarchive.wustl.edu,
we use an argument of "wuarchive.wustl.edu:daytime." This retrieves the current time of day at the
remote site:
% lgetr.pl wuarchive.wustl.edu:daytime
Tue Aug 8 06:49:20 2000

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Input/Output Basics 5

To read the welcome banner from the FTP service at the same site, we ask for "wuarch-
ive.wustl.edu:ftp":
% lgetr.pl wuarchive.wustl.edu:ftp
2:220 wuarchive.wustl.edu FTP server (Version wu-2.6.1(1) Thu Jul 13
21:24:09 CDT 2000) ready.

Or for a change of hosts, we can read the welcome banner from the SMTP (Internet mail) server
running at mail.hotmail.com like this:
% lgetr.pl mail.hotmail.com:smtp
2:220-HotMail (NO UCE) ESMTP server ready at Tue Aug 08 05:24:40 2000

Let's turn to the code for the lgetr.pl script in Figure 1.2.
Lines 1–2: Load modules—We use() the IO::Socket module, which provides an object-oriented
interface for network socket operations.
Line 3: Process the command line argument—We shift() the host and service name off the
command line and store it in a variable named server
Line 4: Open a socket—We call the IO::Socket::INET->new() method to create a "socket"
connected to the designated service running on the remote machine. IO::Socket::INET is a fil-
ehandle class that is adapted for Internet-based communications. A socket is just a specialized
form of filehandle, and can be used interchangeably with other types of filehandles in I/O oper-
ations.
Lines 5–6: Read a line from the socket and print it—We use the <> operator to read a line of
text from the socket into the variable $line, which we immediately print.
Feel free to try the lgetr.pl script on your favorite servers. In addition to the services used in the
examples above, other services to try include "nntp," the Netnews transfer protocol, "chargen," a
test character generator, and "pop3," a protocol for retrieving mail messages. If the script appears
to hang indefinitely, you've probably contacted a service that requires the client to send the first line
of text, such as an HTTP (Web) server. Just interrupt the script and try a different service name.
Although lgetr.pl doesn't do all that much, it is useful in its own right. You can use it to check the
time on a remote machine, or wrap it in a shell script to check the time synchronization of all the
servers on your network. You could use it to generate a summary of the machines on your network
that are running an SMTP mail server and the software they're using.
Notice the similarity between the two scripts. Simply by changing IO::File->new() to
IO::Socket::INET->new(), we have created a fully functional network client. Such is the power
of Perl.

Filehandles
Filehandles are the foundation of networked applications. In this section we review the ins and outs
of filehandles. Even if you're an experienced Perl programmer, you might want to scan this section
to refresh your memory on some of the more obscure aspects of Perl I/O.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Input/Output Basics 6

Standard Filehandles
A filehandle connects a Perl script to the outside world. Reading from a filehandle brings in outside
data, and writing to one exports data. Depending on how it was created, a filehandle may be con-
nected to a disk file, to a hardware device such as a serial port, to a local process such as a com-
mand-line window in a windowing system, or to a remote process such as a network server. It's also
possible for a filehandle to be connected to a "bit bucket" device that just sucks up data and ignores
it.
A filehandle is any valid Perl identifier that consists of uppercase and lowercase letters, digits, and
the underscore character. Unlike other variables, a filehandle does not have a distinctive prefix (like
"$"). So to make them distinct, Perl programmers often represent them in all capital letters, or caps.
When a Perl script starts, exactly three filehandles are open by default: STDOUT, STDIN, and
STDERR. The STDOUT filehandle, for "standard output," is the default filehandle for output. Data sent
to this filehandle appears on the user's preferred output device, usually the command-line window
from which the script was launched. STDIN, for "standard input," is the default input filehandle. Data
read from this filehandle is taken from the user's preferred input device, usually the keyboard.
STDERR ("standard error") is used for error messages, diagnostics, debugging, and other such in-
cidental output. By default STDERR uses the same output device as STDOUT, but this can be changed
at the user's discretion. The reason that there are separate filehandles for normal and abnormal
output is so that the user can divert them independently; for example, to send normal output to a
file and error output to the screen.
This code fragment will read a line of input from STDIN, remove the terminating end-of-line character
with the chomp() function, and echo it to standard output:
$input = <STDIN>;
chomp($input);
print STDOUT "If I heard you correctly, you said: $input\n";

By taking advantage of the fact that STDIN and STDOUT are the defaults for many I/O operations,
and by combining chomp() with the input operation, the same code could be written more succinctly
like this:
chomp($input = <>);
print "If I heard you correctly, you said: $input\n";

We review the <> and print() functions in the next section. Similarly, STDERR is the default des-
tination for the warn() and die() functions.
The user can change the attachment of the three standard filehandles before launching the script.
On UNIX and Windows systems, this is done using the redirect metacharacters "<" and ">". For
example, given a script named muncher.pl this command will change the script's standard input so
that it comes from the file data.txt, and its standard output so that processed data ends up in
crunched.txt:
% muncher.pl <data.txt >crunched.txt

Standard error isn't changed, so diagnostic messages (e.g., from the built-in warn() and die()
functions) appear on the screen.
On Macintosh systems, users can change the source of the three standard filehandles by selecting
filenames from a dialog box within the MacPerl development environment.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Input/Output Basics 7

Input and Output Operations

Perl gives you the option of reading from a filehandle one line at a time, suitable for text files, or
reading from it in chunks of arbitrary length, suitable for binary byte streams like image files.
For input, the <> operator is used to read from a filehandle in a line-oriented fashion, and read()
or sysread() to read in a byte-stream fashion. For output, print() and syswrite() are used
for both text and binary data (you decide whether to make the output line-oriented by printing new-
lines).

$line = <FILEHANDLE>
@lines = <FILEHANDLE>
$line <>
@lines <>
The <> ("angle bracket") operator is sensitive to the context in which it is called. If it is used to assign to a scalar
variable, a so-called scalar context, it reads a line of text from the indicated filehandle, returning the data along
with its terminating end-of-line character. After reading the last line of the filehandle, <> will return undef,
signaling the end-of-file (EOF) condition.
When <> is assigned to an array or used in another place where Perl ordinarily expects a list, it reads all lines
from the filehandle through to EOF, returning them as one (potentially gigantic) list. This is called a list context.
If called in a "void context" (i.e., without being assigned to a variable),<> copies a line into the $_ global variable.
This is commonly seen in while() loops, and often combined with pattern matches and other operations that
use $_ implicitly:
while (<>) {
print "Found a gnu\n" if /GNU/i;
}

The <FILEHANDLE> form of this function explicitly gives the filehandle to read from. However, the <> form is
"magical." If the script was called with a set of file names as command-line arguments, <> will attempt to
open() each argument in turn and will then return lines from them as if they were concatenated into one large
pseudofile.
If no files are given on the command line, or if a single file named "-" is given, then <> reads from standard
input and is equivalent to <STDIN>. See the perlfunc POD documentation for an explanation of how this works
(pod perlfunc, as explained in the Preface).
$bytes = read (FILEHANDLE,$buffer,$length [,$offset])
$bytes = sysread (FILEHANDLE,$buffer,$length [,$offset])
The read() and sysread() functions read data of arbitrary length from the indicated filehandle. Up to
$length bytes of data will be read, and placed in the $buffer scalar variable. Both functions return the
number of bytes actually read, numeric 0 on the end of file, or undef on an error.
This code fragment will attempt to read 50 bytes of data from STDIN, placing the information in $buffer, and
assigning the number of bytes read to $bytes:
my $buffer;
$bytes = read (STDIN,$buffer,50);

By default, the read data will be placed at the beginning of $buffer, overwriting whatever was already there.
You can change this behavior by providing the optional numeric $offset argument, to specify that read data
should be written into the variable starting at the specified position.
The main difference between read() and sysread() is that read() uses standard I/O buffering, and
sysread() does not. This means that read() will not return until either it can fetch the exact number of bytes
requested or it hits the end of file. The sysread() function, in contrast, can return partial reads. It is guaranteed
to return at least 1 byte, but if it cannot immediately read the number of bytes requested from the filehandle, it
will return what it can. This behavior is discussed in more detail later in the Buffering and Blocking section.
$result = print FILEHANDLE $data1,$data2,$data3...

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Input/Output Basics 8

$result = print $data1,$data2,$data3...

The print() function prints a list of data items to a filehandle. In the first form, the filehandle is given explicitly.
Notice that there is no comma between the filehandle name and the first data item. In the second form,
print() uses the current default filehandle, usually STDOUT. The default filehandle can be changed using
the one-argument form of select() (discussed below). If no data arguments are provided, then print()
prints the contents of $_.
If output was successful, print() returns a true value. Otherwise it returns false and leaves an error message
in the variable named $!.
Perl is a parentheses-optional language. Although I prefer using parentheses around function arguments, most
Perl scripts drop them with print(), and this book follows that convention as well.
$result = printf $format,$data1,$data2,$data3...
The printf() function is a formatted print. The indicated data items are formatted and printed according to
the $format format string. The formatting language is quite rich, and is explained in detail in Perl's POD
documentation for the related sprintf() (string formatting) function.
$bytes = syswrite (FILEHANDLE,$data [,$length [,$offset]])
The syswrite() function is an alternative way to write to a filehandle that gives you more control over the
process. Its arguments are a filehandle and a scalar value (avariable or string literal). It writes the data to the
filehandle, and returns the number of bytes successfully written.
By default, syswrite() attempts to write the entire contents of $data, beginning at the start of the string.
You can alter this behavior by providing an optional $length and $offset, in which case syswrite() will
write $length bytes beginning at the position specified by $offset.
Aside from familiarity, the main difference between print() and syswrite() is that the former uses stand-
ard I/O buffering, while the latter does not. We discuss this later in the Buffering and Blocking section.
Don't confuse syswrite() with Perl's unfortunately named write() function. The latter is part of Perl's report
formatting package, which we won't discuss further.
$previous = select(FILEHANDLE)
The select() function changes the default output filehandle used by print print (). It takes the name of
the filehandle to set as the default, and returns the name of the previous default. There is also a version of
select() that takes four arguments, which is used for I/O multiplexing. We introduce the four-argument
version in Chapter 8.

When reading data as a byte stream with read() or sysread(), a common idiom is to pass
length($buffer) as the offset into the buffer. This will make read() append the new data to
the end of data that was already in the buffer. For example:
my $buffer;
while (1) {
$bytes = read (STDIN,$buffer,50,length($buffer));
last unless $bytes > 0;
}

Detecting the End of File

The end-of-file condition occurs when there's no more data to be read from a file or device. When
reading from files this happens at the literal end of the file, but the EOF condition applies as well
when reading from other devices. When reading from the terminal (command-line window), for ex-
ample, EOF occurs when the user presses a special key: control-D on UNIX, control-Z on Windows/
DOS, and command-. on Macintosh. When reading from a network-attached socket, EOF occurs
when the remote machine closes its end of the connection.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Input/Output Basics 9

The EOF condition is signaled differently depending on whether you are reading from the filehandle
one line at a time or as a byte stream. For byte-stream operations with read() or sysread(),
EOF is indicated when the function returns numeric 0. Other I/O errors return undef and set $! to
the appropriate error message. To distinguish between an error and a normal end of file, you can
test the return value with defined():
while (1) {
my $bytes = read(STDIN,$buffer,100);
die "read error" unless defined ($bytes);
last unless $bytes > 0;
}

In contrast, the <> operator doesn't distinguish between EOF and abnormal conditions, and returns
undef in either case. To distinguish them, you can set $! to undef before performing a series of
reads, and check whether it is defined afterward:
undef $!;
while (defined(my $line = <STDIN>)) {
$data .= $line;
}
die "Abnormal read error: $!" if defined ($!);

When you are using <> inside the conditional of a while() loop, as shown in the most recent code
fragment, you can dispense with the explicit defined() test. This makes the loop easier on the
eyes:
while (my $line = <STDIN>) {
$data .= $line;
}

This will work even if the line consists of a single 0 or an empty string, which Perl would ordinarily
treat as false. Outside while() loops, be careful to use defined() to test the returned value for
EOF.
Finally, there is the eof() function, which explicitly tests a filehandle for the EOF condition:

$eof = eof(FILEHANDLE)
The eof() function returns true if the next read on FILEHANDLE will return an EOF. Called without arguments
or parentheses, as in eof, the function tests the last filehandle read from.
When using while(<>) to read from the command-line arguments as a single pseudofile, eof() has "mag-
ical"—or at least confusing—properties. Called with empty parentheses, as in eof(), the function returns true
at the end of the very last file. Called without parentheses or arguments, as in eof, the function returns true
at the end of each of the individual files on the command line. See the Perl POD documentation for examples
of the circumstances in which this behavior is useful.

In practice, you do not have to use eof() except in very special circumstances, and a reliance on
it is often a signal that something is amiss in the structure of your program.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Input/Output Basics 10

Anarchy at the End of the Line

When performing line-oriented I/O, you have to watch for different interpretations of the end-of-line
character. No two operating system designers can seem to agree on how lines should end in text
files. On UNIX systems, lines end with the linefeed character (LF, octal \012 in the ASCII table);
on Macintosh systems, they end with the carriage return character (CR, octal \015); and the Win-
dows/DOS designers decided to end each line of text with two characters, a carriage return/linefeed
pair (CRLF, or octal \015\012). Most line-oriented network servers also use CRLF to terminate
lines.
This leads to endless confusion when moving text files between machines. Fortunately, Perl pro-
vides a way to examine and change the end-of-line character. The global variable $/ contains the
current character, or sequence of characters, used to signal the end of line. By default, it is set to
\012 on Unix systems, \015 on Macintoshes, and \015\012 on Windows and DOS systems.
The line-oriented <> input function will read from the specified handle until it encounters the end-

Licensed by
of-line character(s) contained in $/, and return the line of text with the end-of-line sequence still
attached. The chomp() function looks for the end-of-line sequence at the end of a text string and
removes it, respecting the current value of $/.
The string escape \n is the logical newline character, and means different things on different plat-

Stjepan Maric
forms. For example, \n is equivalent to \012 on UNIX systems, and to \015 on Macintoshes. (On
Windows systems, \n is usually \012, but see the later discussion of DOS text mode.) In a similar
vein, \r is the logical carriage return character, which also varies from system to system.
When communicating with a line-oriented network server that uses CRLF to terminate lines, it won't

4218908
be portable to set $/ to \r\n. Use the explicit string \015\012 instead. To make this less obscure,
the Socket and IO::Socket modules, which we discuss in great detail later, have an option to export
globals named $CRLF and CRLF() that return the correct values.
There is an additional complication when performing line-oriented I/O on Microsoft Windows and
DOS machines. For historical reasons, Windows/DOS distinguishes between filehandles in "text
mode" and those in "binary mode." In binary mode, what you see is exactly what you get. When you
print to a binary filehandle, the data is output exactly as you specified. Similarly, read operations
return the data exactly as it was stored in the file.
In text mode, however, the standard I/O library automatically translates LF into CRLF pairs on the
way out, and CRLF pairs into LF on the way in. The virtue of this is that it makes text operations on
Windows and UNIX Perls look the same—from the programmer's point of view, the DOS text files
end in a single \n character, just as they do in UNIX. The problem one runs into is when reading or
writing binary files—such as images or indexed databases—and the files become mysteriously cor-
rupted on input or output. This is due to the default line-end translation. Should this happen to you,
you should turn off character translation by calling binmode() on the filehandle.

binmode (FILEHANDLE [$discipline])

The binmode() function turns on binary mode for a filehandle, disabling character translation. It should be
called after the filehandle is opened, but before doing any I/O with it. The single-argument form turns on binary
mode. The two-argument form, available only with Perl 5.6 or higher, allows you to turn binary mode on by
providing :raw as the value of $discipline, or restore the default text mode using :crlf as the value.
binmode() only has an effect on systems like Windows and VMS, where the end-of-line sequence is more
than one character. On UNIX and Macintosh systems, it has no effect.

Another way to avoid confusion over text and binary mode is to use the sysread() and sys-
write() functions, which bypass the character translation routines in the standard I/O library.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Input/Output Basics 11

A whole bevy of special global variables control other aspects of line-oriented I/O, such as whether
to append an end-of-line character automatically to data output with the print() statement, and
whether multiple data values should be separated by a delimiter. See Appendix B for a brief sum-
mary.

Opening and Closing Files

In addition to the three standard filehandles, Perl allows you to open any number of additional file-
handles. To open a file for reading or writing, use the built-in Perl function open() If successful,
open() gives you a filehandle to use for the read and/or write operations themselves. Once you
are finished with the filehandle, call close() to close it. This code fragment illustrates how to open
the file message.txt for writing, write two lines of text to it, and close it:
open (FH,">message.txt") or die "Can't open file: $!";
print FH "This is the first line.\n";
print FH "And this is the second.\n";
close (FH) or die "Can't close file: $!";

We call open() with two arguments: a filehandle name and the name of the file we wish to open.
The filehandle name is any valid Perl identifier consisting of any combination of uppercase and
lowercase letters, digits, and the underscore character. To make them distinct, most Perl program-
mers choose all uppercase letters for filehandles. The " > " symbol in front of the filename tells Perl
to overwrite the file's contents if it already exists, or to create the file if it doesn't. The file will then
be opened for writing.
If open() succeeds, it returns a true value. Otherwise, it returns false, causing Perl to evaluate the
expression to the right of the or operator. This expression simply dies with an error message, using
Perl's $! global variable to retrieve the last system error message encountered.
We call print() twice to write some text to the filehandle. The first argument to print() is the
filehandle, and the second and subsequent arguments are strings to write to the filehandle. Again,
notice that there is no comma between the filehandle and the strings to print. Whatever is printed
to a filehandle shows up in its corresponding file. If the filehandle argument to print() is omitted,
it defaults to STDOUT.
After we have finished printing, we call close() to close the filehandle. close() returns a true
value if the filehandle was closed uneventfully, or false if some untoward event, such as a disk filling
up, occurred. We check this result code using the same type of or test we used earlier.
Let's look at open() and close() in more detail.

$success = open(FILEHANDLE,$path)
$success = open(FILEHANDLE,$mode,$path)
The open() call opens the file given in $path, associating it with a designated FILEHANDLE. There are both
two- and three-argument versions of open(). In the three-argument version, which is available in Perl versions
5.6 and higher, a $mode argument specifies how the file is to be opened. $mode is a one- or two-character
string chosen to be reminiscent of the I/O redirection operators in the UNIX and DOS shells. Choices are shown
here.

Mode Description
< Open file for reading
> Truncate file to zero length and open for writing
>> Open file for appending, do not truncate
+> Truncate file and then open for read/write

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Input/Output Basics 12

Mode Description
<+ Open file for read/write, do not truncate

We can open the file named darkstar.txt for reading and associate it with the filehandle DARKFH like
this:
open(DARKFH,'<','darkstar.txt');

In the two-argument form of open(), the mode is appended directly to the filename, as in:
open(DARKFH,'<darkstar.txt');

For readability, you can put any amount of whitespace between the mode symbol and the filename;
it will be ignored. If you leave out the mode symbol, the file will be opened for reading. Hence the
above examples are all equivalent to this:
open(DARKFH,'darkstar.txt');

If successful, open() will return a true value. Otherwise it returns false. In the latter case, the $!
global will contain a human-readable message indicating thecause of the error.

$success = close(FH);
The close() function closes a previously opened file, returning true if successful, or false otherwise. In the
case of an error, the error message can again be found in $!.
When your program exits, any filehandles that are still open will be closed automatically.

The three-argument form of open() is used only rarely. However, it has the virtue of not scanning
the filename for special characters the way that the two-argument form does. This lets you open
files whose names contain leading or trailing whitespace, ">" characters, and other weird and arbi-
trary data. The filename "-" is special. When opened for reading, it tells Perl to open standard input.
When opened for writing, it tells Perl to open standard output.
If you call open() on a filehandle that is already open, it will be automatically closed and then
reopened on the file that you specify. Among other things, this call can be used to reopen one of
the three standard filehandles on the file of your choice, changing the default source or destination
of the <>, print(), and warn() functions. We will see an example of this shortly.
As with the print() function, many programmers drop the parentheses around open() and
close(). For example, this is the most common idiom for opening a file:
open DARKSTAR,"darkstar.txt" or die "Couldn't open darkstar.txt: $!"

I don't like this style much because it leads to visual ambiguity (does the or associate with the string
"darkstar.txt" or with the open() function?). However, I do use this style with close(),
print(), and return() because of their ubiquity.
The two-argument form of open() has a lot of magic associated with it (too much magic, some
would say). The full list of magic behavior can be found in the perlfunc and perlopentut POD docu-
mentation. However, one trick is worth noting because we use it in later chapters. You can dupli-
cate a filehandle by using it as the second argument to open() with the sequence >& or <& pre-
pended to the beginning. >& duplicates filehandles used for writing, and <& duplicates those used
for reading:

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Input/Output Basics 13

open (OUTCOPY,">&STDOUT");
open (INCOPY,"<&STDOUT");

This example creates a new filehandle named OUTCOPY that is attached to the same device as
STDOUT. You can now write to OUTCOPY and it will have the same effect as writing to STDOUT. This
is useful when you want to replace one or more of the three standard filehandles temporarily, and
restore them later. For example, this code fragment will temporarily reopen STDOUT onto a file,
invoke the system date command (using the system() function, which we discuss in more detail
in Chapter 2), and then restore the previous value of STDOUT. When date runs, its standard output
is opened on the file, and its output appears there rather than in the command window:
#!/usr/bin/perl
# file: redirect.pl

print "Redirecting STDOUT\n";

open (SAVEOUT,">&STDOUT");
open (STDOUT,">test.txt") or die "Can't open test.txt: $!";

print "STDOUT is redirected\n";

system "date";

open (STDOUT,">&SAVEOUT");
print "STDOUT restored\n";

When this script runs, its output looks like this:

% redirect.pl
Redirecting STDOUT
STDOUT restored

and the file test.txt contains these lines:

STDOUT is redirected
Thu Aug 10 09:19:24 EDT 2000

Notice how the second print() statement and the output of the date system command went to
the file rather than to the screen because we had reopened STDOUT at that point. When we restored
STDOUT from the copy saved in SAVEOUT, our ability to print to the terminal was restored.
Perl also provides an alternative API for opening files that avoids the magic and obscure syntax of
open() altogether. The sysopen() function allows you to open files using the same syntax as the
C library's open() function.

$result = sysopen (FILEHANDLE,$filename,$mode [,$perms])

The sysopen() function opens the file indicated by $filename, using the I/O mode indicated by $mode. If
the file doesn't exist, and $mode indicates that the file should be created, then the optional $perms value
specifies the permission bits for the newly created file. We discuss I/O modes and permissions in more detail
below.
If successful, sysopen() returns a true result and associates the opened file with FILEHANDLE. Otherwise
it returns a false result and leaves the error message in $!.

The $mode argument used in sysopen() is different from the mode used in ordinary open().
Instead of being a set of characters, it is a numeric bitmask formed by ORing together one or more
constants using the bitwise OR operator " | ". For example, the following snippet opens up a file for
writing using a mode that causes it to be created if it doesn't exist, and truncated to length zero if it
does (equivalent to open()'s " > " mode):

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Input/Output Basics 14

sysopen(FH, "darkstar.txt",O_WRONLY|O_CREAT|O_TRUNC)
or die "Can't open: $!"

The standard Fcntl module exports the constants recognized by sysopen(), all of which begin
with the prefix O_. Just use Fcntl at the top of your script togain access to them.
The mode constants useful for sysopen() are listed in Table 1.1. Each call to sysopen() must
have one (and only one) of O_RDONLY, O_WRONLY, and O_RDWR. The O_WRONLY and O_RDWR
constants may be ORed with one or more of O_CREAT, O_EXCL, O_TRUNC, or O_APPEND.
O_CREAT causes the file to be created if it doesn't already exist. If it isn't specified, and the file
doesn't exist when you try to open it for writing, then sysopen() will fail.
Combining O_CREAT with O_EXCL leads to the useful behavior of creating the file if it doesn't already
exist, but failing if it does. This can be used to avoid accidentally clobbering an existing file.
If O_TRUNC is specified, then the file is truncated to zero length before the first write, effectively
overwriting the previous contents. O_APPEND has the opposite effect, positioning the write pointer
to the end of the file so that everything written to the file is appended to its existing contents.
Table 1.1. sysopen() Mode Constants
Constant Description
O_RDONLY Open read only.
O_WRONLY Open write only.
O_RDWR Open read/write.
O_CREAT Create file if it doesn't exist.
O_EXCL When combined with O_CREAT, create file if it doesn't exist and fail if it does.
O_TRUNC If file already exists, truncate it to zero.
O_APPEND Open file in append mode (equivalent to open()'s " >> ").
O_NOCTTY If the file is a terminal device, open it without allowing it to become the process's controlling terminal.
O_NONBLOCK Open file in nonblockingmode.
O_SYNC Open file for synchronous mode, so that all writes block until the data is physically written.

The O_NOCTTY, O_NONBLOCK, and O_SYNC modes all have specialized uses that are discussed in
later chapters.
If sysopen() needs to create a file, the $perm argument specifies the permissions mode of the
resulting file. File permissions is a UNIX concept that maps imperfectly onto the Windows/DOS
world, and not at all onto the Macintosh world. It is an octal value, such as 0644 (which happens to
specify read/write permissions for the owner of the file, and read-only permissions for others).
If $perm is not provided, sysopen() defaults to 0666, which grants read/write access to everyone.
However, whether you specify the permissions or accept the default, the actual permissions of the
created file are determined by performing the bitwise AND between the $perm argument and the
current contents of the user's umask (another UNIX concept). This is often set, at the user's dis-
cretion, to forbid access to the file from outside the user's account or group.
In most circumstances, it is best to omit the permissions argument and let the user adjust the
umask. This also increases the portability of the program. See the umask() entry in the perlfunc
POD documentation for information on how you can examine and set the umask programatically.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Input/Output Basics 15

Buffering and Blocking

When you print() or syswrite() to a filehandle, the actual output operation does not occur
immediately. If you are writing to a file, the system has to wait for the write head to reach the proper
location on the disk drive, and for the spinning platter to bring the proper location under the head.
This is usually an insignificant length of time (although it may be quite noticeable on a laptop that
intermittently spins down its disk to save battery power), but other output operations can take much
more time. In particular, network operations may take a considerable length of time to complete.
The same applies to input.
There is a fundamental mismatch between computational speed and I/O speed. A program can
execute the contents of a tight loop a million times a second, but a single I/O operation may take
several seconds to complete. To overcome this mismatch, modern operating systems use the tech-
niques of buffering and blocking.
The idea behind buffering is shown in Figure 1.3. Buffers decouple the I/O calls invoked by the
program from the actual I/O operations that occur at the hardware level. A call to print(), for
example, doesn't send data directly to the terminal, network card, or disk drive, but instead it results
in data being written to a memory area. This occurs quickly, because writes to memory are fast.
Meanwhile, in an asynchronous fashion, the operating system reads from data previously written to
the buffer and performs the actions necessary to write the information to the hardware device.

Figure 1.3. Buffers help solve the mismatch between computation speed and I/O speed

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Input/Output Basics 16

Similarly, for input operations, the operating system receives data from active input devices (the
keyboard, disk drive, network card) and writes that data to an input buffer somewhere in memory.
The data remains in the input buffer until your program calls <> or read(), at which point the data
is copied from the operating system's buffer into the memory space corresponding to a variable in
your program.
The advantage of buffering is significant, particularly if your program performs I/O in a "bursty" way;
that is, it performs numerous reads and writes of unpredictable size and timing. Instead of waiting
for each operation to complete at the hardware level, the data is safely buffered in the operating
system and "flushed"—passed on to the output device—whenever the downstream hardware can
accept it.
The buffers in Figure 1.3 are conceptually circular FIFO (first in first out) data structures. When data
is written beyond the end of the buffer memory area, the operating system merely begins writing
new data at the beginning. The operating system maintains two pointers in each of its I/O buffers.
The write pointer is the place where new data enters the buffer. The read pointer is the position from
which stored data is moved out of the buffer to its next destination. For example, on write operations,
each print() you perform adds some data to the output buffer and advances the write pointer
forward. The operating system reads older data starting at the read pointer and copies it to the low-
level hardware device.
The size of the I/O buffer minus the distance between the write pointer and the read pointer is the
amount of free space remaining. If your program is writing faster than the output hardware can
receive it, then the buffer will eventually fill up and the buffer's free space will be zero. What happens
then?
Because there is no room for new data in the buffer, the output operation cannot succeed immedi-
ately. As a result, the write operation blocks. Your program will be suspended at the blocked
print() or syswrite() for an indefinite period of time. When the backlog in the output buffer
clears and there is again room to receive new data, the output operation will unblock and print() or
syswrite() will return.
In a similar fashion, reads will block when the input buffer is empty; that is, it blocks when the amount
of free space is equal to the size of the buffer. In this case, calls to read() or sysread() will block
until some new data has entered the buffer and there is something to be read.
Blocking is often the behavior you want, but sometimes you need more control over I/O. There are
several techniques to manage blocking. One technique, discussed in Chapter 2 under Timing Out
System Calls, uses signals to abort an I/O operation prematurely if it takes too long. Another tech-
nique, discussed in Chapter 12, uses the four-argument select() system call to test a filehandle
for its readiness to perform I/O before actually making the read or write call. A third technique,
discussed in Chapter 13, is to mark the filehandle as nonblocking, which causes the read or write
operation to return immediately with an error code if the operation would block.

Standard I/O Buffering

Although we have spoken of a single buffer for I/O operations on a filehandle, there may in fact be
several buffers at different layers of the operating system. For example, when writing to a disk file,
there is a very low-level buffer on the disk hardware itself, another one in the SCSI or IDE driver
that controls the disk, a third in the driver for the filesystem, and a fourth in the standard C library
used by Perl. I may have missed a few.
You cannot control or even access most of these buffers directly, but there is one class of buffer
that you should be aware of. Many of Perl's I/O operations flow through "stdio," a standard C-lan-
guage library which maintains its own I/O buffers independent of the operating system's.1

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Input/Output Basics 17

Perl's <> operator, read(), and print() all use stdio. When you call print(), the data is trans-
ferred to an output buffer in the stdio layer before being sent to the operating system itself. Likewise,
<> and read() both read data from an stdio buffer rather than directly from the OS. Each filehandle
has its own set of buffers for input and output. For efficiency reasons, stdio waits until its output
buffers reach a certain size before flushing their contents to the OS.
Normally, the presence of the stdio buffering is not a problem, but it can run you into trouble when
doing more sophisticated types of I/O such as network operations. Consider the common case of
an application that requires you to write a small amount of data to a remote server, wait for a re-
sponse, and then send more data. You may think that the data has been sent across the network,
but in fact it may not have been. The output data may still be sitting in its local stdio buffer, waiting
for more data to come before flushing the buffer. The remote server will never receive the data, and
so will never return a response. Your program will never receive a response and so will never send
additional data. Deadlock ensues.
In contrast, the lower-level buffering performed by the operating system doesn't have this property.
The OS will always attempt to deliver whatever data is in its output buffers as soon as the hardware
can accept it.
There are two ways to work around stdio buffering. One is to turn on "autoflush" mode for the file-
handle. Autoflush mode applies only to output operations. When active, Perl tells stdio to flush the
filehandle's buffer every time print() is called.
To turn on autoflush mode, set the special variable $| to a true value. Autoflush mode affects the
currently selected filehandle, so to change autoflush mode for a specific filehandle, one must first
select() it, set $| to true, and then select() the previously selected filehandle. For example,
to turn on autoflush mode for filehandle FH:
my $previous = select(FH);
$| = 1;
select($previous);

You will sometimes see this motif abbreviated using the following mind-blowing idiom:
select((select(FH),$|=1)[0]);

However, it is much cleaner to bring in the IO::Handle module, which adds an autoflush()
method to filehandles. With IO::Handle loaded, FH can be put into autoflush mode like this:
use IO::Handle;
FH->autoflush(1);

If the OO syntax confuses you, see the Objects and References section later in this chapter.
The other way to avoid stdio buffering problems is to use the sysread()i and syswrite() calls.
These calls bypass the stdio library and go directly to the operating system I/O calls. An important
advantage of these calls is that they interoperate well with other low-level I/O calls, such as the four-
argument select() call, and with advanced techniques such as nonblocking I/O.
Another ramification of the fact that the sys*() functions bypass stdio is the difference in behavior
between read() and sysread() when asked to fetch a larger chunk of data than is available. In
the case of read(), the function will block indefinitely until it can fetch exactly the amount of data
requested. The only exception to this is when the filehandle encounters the end of file before the
full request has been satisfied, in which case read() will return everything to the end of the file. In

1You can think of the stdio library as a layersitting on top of the OS, making the OS look more C-like to programs; similarly, a Pascal standard
I/O library makes the OS look as if it were written in Pascal.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Input/Output Basics 18

contrast, sysread() can and will return partial reads. If it can't immediately read the full amount
of data requested, it will return the data that is available. If no data is available, sysread() will
block until it can return at least 1 byte. This behavior makes sysread() invaluable for use in net-
work communications, where data frequently arrives in chunks of unpredictable size.
For these reasons, sysread() and syswrite() are preferred for many network applications.

Passing and Storing Filehandles

Network applications frequently must open multiple filehandles simultaneously, pass them to sub-
routines, and keep track of them in hashes and other data structures. Perl allows you to treat file-
handles as strings, store them into variables, and pass them around to subroutines. For example,
this functional but flawed code fragment stores the MY_FH filehandle into a variable named $fh,
and then passes it to a subroutine named hello() to use in printing a friendly message:
# Flawed technique
$fh = MY_FH;
hello($fh);
sub hello {
my $handle = shift;
print $handle "Howdy folks!\n";
}

This technique often works; however, it will run you into problems as soon asyou try to pass file-
handles to subroutines in other packages, such as functions exported by modules. The reason is
that passing filehandles as strings loses the filehandle package information. If we pass the filehandle
MY_FH from the main script (package main) to a subroutine defined in the MyUtils module, the
subroutine will try to access a filehandle named MyUtils::MY_FH rather than the true filehandle,
which is main::MY_FH. The same problem also occurs, of course, when a subroutine from one
package tries to return a filehandle to a caller from another package.
The correct way to move filehandles around is as a typeglob or a typeglob reference. Typeglobs
are symbol table entries, but you don't need to know much more about them than that in order to
use them (see the perlref POD documentation for the full details). To turn a filehandle into a glob
put an asterisk ("*") in front of its name:
$fh = *MY_FH;

To turn a filehandle into a typeglob reference, put "\*" in front of its name:
$fh = \*MY_FH;

In either case, $fh can now be used to pass the filehandle back and forth between subroutines and
to store filehandles in data structures. Of the two forms, the glob reference (\*HANDLE) is the safer,
because there's less risk of accidentally writing to the variable and altering the symbol table. This
is the form we use throughout this book, and the one used by Perl's I/O-related modules, such as
IO::Socket.
Typeglob references can be passed directly to subroutines:
hello(\*MY_FH):

They can also be returned directly by subroutines:

my $fh = get_fh();
sub get_fh {
open (FOO,"foo.txt") or die "foo: $!";
return \*FOO;
}

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Input/Output Basics 19

Typeglob refs can be used anywhere a bare filehandle is accepted, including as the first argument
to print(), read(), sysread(), syswrite(), or any of the socket-related calls that we discuss
in later chapters.
Sometimes you will need to examine a scalar variable to determine whether it contains a valid
filehandle. The fileno() function makes this possible:

$integer = fileno (FILEHANDLE)

The fileno() function accepts a filehandle in the form of a bare string, a typeglob, or a typeglob reference.
If the filehandle is valid, fileno() returns the file descriptor for the filehandle. This is a small integer that
uniquely identifies the filehandle to the operating system. STDIN, STDOUT, and STDERR generally correspond
to descriptors 0, 1, and 2, respectively (but this can change if you close and reopen them). Other filehandles
have descriptors greater than 3.
If the argument passed to fileno() does not correspond to a valid filehandle (including former filehandles
that have been closed), fileno() returns undef. Here is the idiom for checking whether a scalar variable
contains a filehandle:
die "not a filehandle" unless defined fileno($fh);

Detecting Errors
Because of the vicissitudes of Internet communications, I/O errors are common in network appli-
cations. As a rule, each of the Perl functions that performs I/O returns undef, a false value, on
failure. More specific information can be found by examining the special global variable $!.
$! has an interesting dual nature. Treated as a string, it will return a human-readable error message
such as Permission denied. Treated as a number, however, it will return the numeric constant
for the error, as defined by the operating system (e.g., EACCES). It is generally more reliable to use
these numeric error constants to distinguish particular errors, because they are standardized across
operating systems.
You can obtain the values of specific error message constants by importing them from the Errno
module. In the use statement, you can import individual constants by name, or all of them at once.
To bring in individual constants, list them in the use() statement, as shown here:
use Errno qw(EACCES ENOENT);
my $result = open (FH,">/etc/passwd");
if (!$result) { # oops, something went wrong
if ($! == EACCESS) {
warn "You do not have permission to open this file.";
} elsif ($! == ENOENT) {
warn "File or directory not found.";
} else {
warn "Some other error occurred: $!";
}
}

The qw() operator is used to split a text string into a list of words. The first line above is equivalent
to:
use Errno ('EACCESS','ENOENT');

and brings in the EACCESS and ENOENT constants. Notice that we use the numeric comparison
operator " == " when comparing $! to numeric constants.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Input/Output Basics 20

To bring in all the common error constants, import the tag :POSIX. Thisbrings in the error constants
that are defined by the POSIX standard, across-platform API that UNIX, Windows NT/2000, and
many other operating systems are largely compliant with. For example:
use Errno qw(:POSIX);

Do not get into the habit of testing $! to see if an error occurred during the last operation. $! is set
when an operation fails, but is not unset when an operation succeeds. The value of $! should be
relied on only immediately after a function has indicated failure.

Using Object-Oriented Syntax with the IO::Handle and IO::File

Modules
We use Perl5's object-oriented facilities extensively later in this book. Although you won't need to

Licensed by
know much about creating object-oriented modules, you will need a basic understanding of how to
use OO modules and their syntax. This section illustrates the basics of Perl's OO syntax with ref-
erence to the IO::Handle and IO::File module, which together form the basis of Perl's object-oriented
I/O facilities.

Objects and References

Stjepan Maric
In Perl, references are pointers to data structures. You can create a reference to an existing data
structure using the backslash operator. For example:

4218908
$a = 'hi there';
$a_ref = \$a; # reference to a scalar
@b = ('this','is','an','array');
$b_ref = \@b; # reference to an array
%c = ( first_name => 'Fred', last_name => 'Jones');
$c_ref = \%c; # reference to a hash

Once a reference has been created, you can make copies of it, as you would any regular scalar, or
stuff it into arrays or hashes. When you want to get to the data contained inside a reference, you
dereference it using the prefix appropriate for its contents:
$a = $$a_ref;
@b = @$b_ref;
%c = %$c_ref;

You can index into array and hash references without dereferencing the whole thing by using the
-> syntax:
$b_ref->[2]; # yields "an"
$c_ref->{last_name}; # yields "Jones"

It is also possible to create references to anonymous, unnamed arrays and hashes, using the fol-
lowing syntax:
$anonymous_array = ['this','is','an','anonymous','array'];
$anonymous_hash = { first_name =>'Jane', last_name => 'Doe' };

If you try to print out a reference, you'll get a string like HASH(0x82ab0e0), which indicates the
type of reference and the memory location where it can be found (which is just short of useless).

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Input/Output Basics 21

An object is a reference with just a little bit extra. It is "blessed" into a particular module's package
in such a way that it carries information about what module created it.2 The blessed reference will
continue to work just like other references. For example, if the object named $object is a blessed
hash reference, you can index into it like this:
$object->{last_name};

What makes objects different from plain references is that they have methods. A method call uses
the -> notation, but followed by a subroutine name and optional subroutine-style arguments:
$object->print_record(); # invoke the print_record() method

You may sometimes see a method called with arguments, like this:
$object->print_record(encoding => 'EBCDIC');

The "=>" symbol is accepted by Perl as a synonym for ','. It makes the relationship between the two
arguments more distinct, and has the added virtue of automatically quoting the argument on the left.
This allows us to write encoding rather than "encoding". If a method takes no arguments, it's often
written with the parentheses omitted, as in:
$object->print_record;

In many cases, print_record() will be a subroutine defined in the object's package. Assuming
that the object was created by a module named BigDatabase, the above is just a fancy way of saying
this:
BigDatabase::print_record($object);

However, Perl is more subtle than this, and the print_record(), method definition might actually
reside in another module, which the current module inherits from. How this works is beyond the
scope of this introduction, and can be found in the perltoot, perlobj, and perlref POD pages, as well
as in [Wall 2000] and the other general Perl reference works listed in Appendix D.
To create an object, you must invoke one of its constructors. A constructor is a method call that is
invoked from the module's name. For example, to create a new BigDatabase object:
$object = BigDatabase->new(); # call the new() constructor

Constructors, which are a special case of a class method, are frequently named new(). However,
any subroutine name is possible. Again, this syntax is part trickery. In most cases an equivalent call
would be:
$object = BigDatabase::new('BigDatabase');

This is not quite the same thing, however, because class methods can also be inherited.

The IO::Handle and IO::File Modules

The IO::Handle and IO::File modules, standard components of Perl, together provide object-orien-
ted interface to filehandles. IO::Handle provides generic methods that are shared by all filehandle
objects, including pipes and sockets. The more specialized class, IO::File, adds functionality for
opening and manipulating files. Together, these classes smooth out some of the bumps and irreg-
ularities in Perl's built-in filehandles, and make larger programs easier to understand and maintain.

2The function responsible for turning ordinary references into blessed ones is, naturally enough, called bless().

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Input/Output Basics 22

IO::File's elegance does not by itself provide any very compelling reason to choose the object-
oriented syntax over native filehandles. Its main significance is that IO::Socket, IO::Pipe, and other
I/O-related modules also inherit their behavior from IO::Handle. This means that programs that read
and write from local files and those that read and write to remote network servers share a common,
easy-to-use interface.
We'll get a feel for the module by looking at a tiny example of a program that opens a file, counts
the number of lines, and reports its findings (Figure 1.4).

Figure 1.4. The count_lines.pl program

Lines 1–3: Load modules—We turn on strict syntax checking, and load the IO::File module.
Lines 4–5: Initialize variables—We recover from the command line the name of the file to perform
the line count on, and initialize the $counter variable to zero.
Line 6: Create a new IO::File object—We call the IO::File::new() method, using the syntax
IO::File->new(). The argument is the name of the file to open. If successful, new() returns
a new IO::File object that we can use for I/O. Otherwise it returns undef, and we die with an
error message.
Lines 7–9: Main loop—We call the IO::File object's getline() method in the test portion of a
while() loop. This method returns the next line of text, or undef on end of file—just like <>.
Each time through the loop we bump up $counter. The loop continues until getline() returns
undef.
Line 10: Print results—We print out our results by calling STDOUT->print(). We'll discuss why
this surprising syntax works in a moment.
When I ran count_lines.pl on the unfinished manuscript for this chapter, I got the following result:
% count_lines.pl ch1.pod
Counted 2428 lines

IO::File objects are actually blessed typeglob references (see the Passing and Storing Filehandles
section earlier in this chapter). This means that you can use them in an object-oriented fashion, as
in:
$fh->print("Function calls are for the birds.\n");

or with the familiar built-in function calls:

print $fh "Object methods are too effete.\n";

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Input/Output Basics 23

Many of IO::File's methods are simple wrappers around Perl's built-in functions. In addition to
print() and getline() methods, there are read(), syswrite(), and close() methods,
among others. We discuss the pros and cons of using object-oriented method calls and function
calls in Chapter 5, where we introduce IO:: Socket.
When you load IO::File (technically, when IO::File loads IO::Handle, which it inherits from), it adds
methods to ordinary filehandles. This means that any of the methods in IO::File can also be used
with STDIN, STDOUT, STDERR, or even with any conventional filehandles that you happen to create.
This is why line 10 of Figure 1.4 allows us to print to standard output by calling print().
Of the method listings that follow, only the new() and new_tmpfile() methods are actually de-
fined by IO::File. The rest are inherited from IO::Handle and can be used with other descendents of
IO::Handle, such as IO::Socket. This list is not complete. I've omitted some of the more obscure
methods, including those that allow you to move around inside a file in a record-oriented fashion,
because we won't need them for network communications.

$fh = IO::File->new($filename [,$mode [,$perms]])

The new() method is the main constructor for IO::File. It is a unified replacement for both open() and
sysopen(). Called with a single argument, new() acts like the two-argument form of open(), taking a file-
name optionally preceded by a mode string. For example, this will open the indicated file for appending:
$fh = IO::File->new(">darkstar.txt");

If called with two or three arguments, IO::File treats the second argument as the open mode, and the third
argument as the file creation permissions. $mode may be a Perl-style mode string, such as " +< ", or an octal
numeric mode, such as those used by sysopen(). As a convenience, IO::File automatically imports the Fcntl
O_* constants when it loads. In addition, open() allows for an alternative type of symbolic mode string that
is used in the C fopen() call; for example, it allows " w " to open the file for writing. We won't discuss those
modes further here, because they do not add functionality.
The permission agreement given by $perms is an octal number, and has the same significance as the corre-
sponding parameter passed to sysopen().
If new() cannot open the indicated file, it will return undef and set $! to the appropriate system error message.
$fh = IO::File->new_tmpfile
The new_tmpfile() constructor, which is called without arguments, creates a temporary file opened for
reading and writing. On UNIX systems, this file is anonymous, meaning that it cannot be seen on the file system.
When the IO::File object is destroyed, the file and all its contents will be deleted automatically.
This constructor is useful for storing large amounts of temporary data.
$result = $fh->close
The close() method closes the IO::File object, returning a true result if successful. If you do not call
close() explicitly, it will be called automatically when the object is destroyed. This happens when the script
exits, if you happen to undef() the object, or if the object goes out of scope such as when a my variable
reaches the end of its enclosing block.
$result = $fh->open($filename [,$mode [,$perms]])
You can reopen a filehandle object on the indicated file by using its open() method. The input arguments are
identical to new(). The method result indicates whether the open was successful.
This is chiefly used for reopening the standard filehandles STDOUT, STDIN, and STDERR. For example:
STDOUT->open("log.txt") or die "Can't reopen STDOUT: $!";

Calls to print() will now write to the file log.txt .

$result = $fh->print(@args)
$result=$fh->printf($fmt,@args)
$bytes=$fh->write($data [,$length [,$offset]])

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Input/Output Basics 24

$bytes = $fh->syswrite($data [,$length [,$offset]])

The print(), printf(), and syswrite() methods work exactly like their built-in counterparts. For exam-
ple, print() takes a list of data items, writes them to the filehandle object, and returns true if successful.
The write() method is the opposite of read(), writing a stream of bytes to the filehandle object and returning
the number successfully written. It is similar to syswrite(), except that it uses stdio buffering. This method
corrects the inconsistent naming of the built-in write() function, which creates formatted reports. The IO::File
object method that corresponds to the built-in write() goes by the name of format_write().
$line = $fh->getline
@lines=$fh->getlines
$bytes = $fh->read($buffer,$length ",$offset")
$bytes = $fh->ysread($buffer,$length ",$offsetr")
The getline() and getlines() methods together replace the <> operator. getline() reads one line
from the filehandle object and returns it, behaving in the same way in both scalar and list contexts. The
getlines() method acts like <> in a list context, returning a list of all the available lines. getline() will
return undef at the end of file.
The read() and sysread() methods act just like their built-in function counterparts.
$previous = $fh->autoflush([$boolean])
The autoflush() method gets or sets the autoflush() mode for the filehandle object. Called without
arguments, it turns on autoflush. Called with a single boolean argument, it sets autoflush to the indicated status.
In either case, autoflush()i returns the previous value of the autoflush state.
$boolean = $fh->opened
The opened() method returns true if the filehandle object is currently valid. It is equivalent to:
defined fileno($fh);

$boolean = $fh->eof
Returns true if the next read on the filehandle object will return EOF.
$fh->flush
The flush() method immediately flushes any data that is buffered in the filehandle object. If the filehandle
is being used for writing, then its buffered data is written to disk (or to the pipe, or network, as we'll see when
we get to IO::Socket objects). If the filehandle is being used for reading, any data in the buffer is discar-
ded, forcing the next read to come from disk.
$boolean = $fh->blocking("$boolean")
The blocking() method turns on and off blocking mode for the filehandle. We discuss how to use this at
length in Chapter 13.
$fh->clearerr
$boolean = $fh->error
These two methods are handy if you wish to perform a series of I/O operations and check the error status only
after you're finished. The error()i method will return true if any errors have occurred on the filehandle since
it was created, or since the last call to clearerr(). The clearerr() method clears this flag.

In addition to the methods listed here, IO::File has a constructor named new_from_fd(), and a
method named fdopen(), both inherited from IO::Handle. These methods can be used to save
and restore objects in much the way that the >&FILEHANDLE does with standard filehandles.

$fh = IO::File->new_from_fd($fd,$mode)

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Input/Output Basics 25

The new_from_fd() method opens up a copy of the filehandle object indicated by $fd using the read/write
mode given by $mode. The object may be an IO::Handle object, an IO::File object, a regular filehandle, or a
numeric file descriptor returned byfileno(). $mode must match the mode with which $fd was originally
opened. For example:
$saveout = IO::File->new_from_fd(STDOUT,">");

$result = $fh->fdopen($fd,$mode)
The fdopen() method is used to reopen an existing filehandle object, making it a copy of another one. The
$fd argument may be an IO::Handle object or a regular filehandle, or a numeric file descriptor $mode must
match the mode with which $fd was originally opened.
This is typically used in conjunction with new_from_fd() to restore a saved filehandle:
$saveout = IO::File->new_from_fd(STDOUT,">"); # save STDOUT
STDOUT->open('>log.txt'); # reopen on a file
STDOUT->print("Yippy yie yay!\n"); # print something
STDOUT->fdopen($saveout,">"); # reopen on saved value

See the POD documentation for IO::Handle and IO::File for information about the more obscure
features that these modules provide.

Summary
Perl and network programming were made for each other. Perl's strong text-processing abilities
combine with a flexible I/O subsystem to create an environment that is ideal for interprocess com-
munication.
This, combined with its native support for the Berkeley Sockets protocol, make Perl an excellent
choice for network applications.
In this chapter we reviewed the essential components of Perl's I/O API. Filehandles are the funda-
mental object used for Perl input/output operations, and offer both line-oriented and byte-stream-
oriented modes.
The STDIN, STDOUT, and STDERR filehandles are available when a program is started, and corre-
spond to the standard input, output, and error devices. A script may open up additional filehandles,
or reopen the standard ones on different files.
The standard I/O library, used by the <>, read(), and print() functions, improves I/O efficiency
by adding a layer of buffering to input and output operations. However, this buffering can sometimes
get in the way. One way to avoid buffering problems is to put the filehandle into autoflush mode.
Another way is to use the lower-level syswrite() and sysread() functions.
The IO::File and IO::Handle modules add object-oriented methods to filehandles. They smooth out
some of the inconsistencies in Perl's original design, and pave the way to a smooth transition to
IO::Socket.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 26

Chapter 2. Processes, Pipes, and Signals

This chapter discusses three key Perl features: processes, pipes, and signals. By creating new
processes, a Perl program can run another program or even clone copies of itself in order to divide
the work. Pipes allow Perl scripts to exchange data with other processes, and signals make it pos-
sible for Perl scripts to monitor and control other processes.

Processes
UNIX, VMS, Windows NT/2000, and most other modern operating systems are multitasking. They
can run multiple programs simultaneously, each one running in a separate thread of execution
known as a process. On machines with multiple CPUs, the processes running on different CPUs
really are executing simultaneously. Processes that are running on single-CPU machines only ap-
pear to be running simultaneously because the operating system switches rapidly between them,
giving each a small slice of time in which to execute.
Network applications often need to do two or more things at once. For example, a server may need
to process requests from several clients at once, or handle a request at the same time that it is
watching for new requests. Multitasking simplifies writing such programs tremendously, because it
allows you to launch new processes to deal with each of the things that the application needs to do.
Each process is more or less independent, allowing one process to get on with its work without
worrying about what the others are up to.
Perl supports two types of multitasking. One type, based on the traditional UNIX multiprocessing
model, allows the current process to clone itself by making a call to the fork() function. After
fork() executes, there are two processes, each identical to the other in almost every respect. One
goes off to do one task, and the other does another task.
Another type, based on the more modern concept of a lightweight "thread," keeps all the tasks within
a single process. However, a single program can have multiple threads of execution running through
it, each of which runs independently of the others.
In this section, we introduce fork() and the variables and functions that are relevant to processes.
We discuss multithreading in Chapter 11.

The fork() Function

The fork() function is available on all UNIX versions of Perl, as well as the VMS and OS/2 ports.
Version 5.6 of Perl supports fork() on Microsoft Windows platforms, but not, unfortunately, on the
Macintosh.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Processes, Pipes, and Signals 27

The Perl fork() function takes no arguments and returns a numeric result code. When fork()
is called, it spawns an exact duplicate of the current process. The duplicate, called the child, shares
the current values of all variables, filehandles (including data in the standard I/O buffers), and other
data structures. In fact, the duplicate process even has the memory of calling fork(). It is like a
man walking into the cloning booth of a science fiction movie. The copy wakes up in the other booth
having all the memories of the original up to and including walking into the cloning booth, but thinking
wait, didn't I start out over there? And who is the handsome gentleman in that other booth?
To ensure peaceful coexistence, it is vital that the parent and child processes know which one is
which. Each process on the system is associated with a unique positive integer, known as its process
ID, or PID.
After a call to fork(), the parent and child examine the function's return value. In the parent proc-
ess, fork() returns the PID of the child. In the child process, fork() returns numeric 0. The code
will go off and do one thing if it discovers it's the parent, and do another if it's the child.

$pid = fork()
Forks a new process. Returns the child's PID in the parent process, and 0 in the child process. In case of error
(such as insufficient memory to fork), returns undef, and sets $! to the appropriate error message.

If the parent and child wish to communicate with each other following the fork, they can do so with
a pipe (discussed later in this chapter in the Pipes section), or via shared memory (discussed in
Chapter 14 in the An Adaptive Preforking Server Using Shared Memory section). For simple mes-
sages, parent and child can send signals to each others' PIDs using the kill() function. The parent
gets the child's PID from fork()'s result code, and the child can get the parent's PID by calling
getppid(). A process can get its own PID by examining the $$ special variable.

$pid = getppi()
Returns the PID of the parent process. Every Perl script has a parent, even those launched directly from the
command line (their parent is the shell process).
$$
The $$ variable holds the current PID for the process. It can be read, but not changed.

We discuss the kill() function later in this chapter, in the Signals section.
If it wishes, a child process can itself fork(), creating a grandchild. The original parent can also
fork() again, as can its children and grandchildren. In this way, Perl scripts can create a whole
tribe of (friendly, we hope) processes. Unless specific action is taken, each member of this tribe
belongs to the same process group.
Each process group has a unique ID, which is usually the same as the process ID of the shared
ancestor. This value can be obtained by calling getpgrp():

$processid = getpgrp([$pid])
For the process specified by $pid, the getpgrp() function returns its process group ID. If no PID is specified,
then the process group of the current process is returned.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Processes, Pipes, and Signals 28

Each member of a process group shares whatever filehandles were open at the time its parent
forked. In particular, they share the same STDIN, STDOUT, and STDERR. This can be modified by
any of the children by closing a filehandle, or reopening it on some other source. However, the
system keeps track of which children have filehandles open, and will not close the file until the last
child has closed its copy of the filehandle.
Figure 2.1 illustrates the typical idiom for forking. Before forking, we print out the PID stored in $$.
We then call fork() and store the result in a variable named $child. If the result is undefined,
then fork() has failed and we die with an error message.

Figure 2.1. This script forks a single child

We now examine $child to see whether we are running in the parent or the child. If $child is
nonzero, we are in the parent process. We print out our PID and the contents of $child, which
contains the child process's PID.
If $child is zero, then we are running in the child process. We recover the parent's PID by calling
ppid() and print it and our own PID.
Here's what happens when you run fork.pl:
% fork.pl
PID=372
Parent process: PID=372, child=373
Child process: PID=373, parent=372

The system() and exec() Functions

Another way for Perl to launch a subprocess is with system(). system() runs another program
as a subprocess, waits for it to complete, and then returns. If successful, system() returns a result
code of 0 (notice that this is different from the usual Perl convention!). Otherwise it returns -1 if the
program couldn't be started, or the program's exit status if it exited with some error. See the perl-
var POD entry for the $? variable for details on how to interpret the exit status fully.

$status = system ('command and arguments')

$status = system ('command', 'and', 'arguments')

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Processes, Pipes, and Signals 29

The system() function executes a command as a subprocess and waits for it to exit. The command and its
arguments can be specified as a single string, or as a list containing the command and its arguments as
separate elements. In the former case, the string will be passed intact to the shell for interpretation. This allows
you to execute commands that contain shell metacharacters (such as input/output re-directs), but opens up
the possibility of executing shell commands you didn't anticipate. The latter form allows you to execute com-
mands with arguments that contain whitespace, shell metacharacters, and other special characters, but it
doesn't interpret metacharacters at all.

The exec() function is like system(), but replaces the current process with the indicated com-
mand. If successful, it never returns because the process is gone. The new process will have the
same PID as the old one, and will share the same STDIN, STDOUT, and STDERR filehandles. How-
ever, other opened filehandles will be closed automatically.1

$status = exec ('command and arguments')

$status = exec ('command', 'and', 'arguments')
exec() executes a command, replacing the current process. It will return a status code only on failure. Oth-
erwise it does not return. The single-value and list forms have the same significance as system().

exec() is often used in combination with fork() to run a command as a subprocess after doing
some special setup. For example, after this code fragment forks, the child reopens STDOUT onto a
file and then calls exec() to run the ls -l command. On UNIX systems, this command generates a
long directory listing. The effect is to run ls -l in the background, and to write its output to the indicated
file.
my $child = fork();
die "Can't fork: $!" unless defined $child;
if ($child == 0) { # we are in the child now
open (STDOUT,">log.txt") or die "open() error: $!";
exec ('ls','-l');
die "exec error(): $!"; # shouldn't get here
}

We use exec() in this way in Chapter 10, in the section titled The Inetd Super Daemon.

Pipes
Network programming is all about interprocess communication (IPC). One process exchanges data
with another. Depending on the application, the two processes may be running on the same ma-
chine, may be running on two machines on the same segment of a local area network, or may be
halfway across the world from each other. The two processes may be related to each other—for
example, one may have been launched under the control of the other—or they may have been
written decades apart by different authors for different operating systems.
The simplest form of IPC that Perl offers is the pipe. A pipe is a filehandle that connects the current
script to the standard input or standard output of another process. Pipes are fully implemented on
UNIX, VMS, and Microsoft Windows ports of Perl, and implemented on the Macintosh only in the
MPW environment.

1You can arrange for some filehandles to remain open across exec() by changing the value of the $~F special variable. See the perlvar
POD document for details.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Processes, Pipes, and Signals 30

Opening a Pipe
The two-argument form of open() is used to open pipes. As before, the first argument is the name
of a filehandle chosen by you. The second argument, however, is a program and all its arguments,
either preceded or followed by the pipe " | " symbol. The command should be entered exactly as
you would type it in the operating system's default shell, which for UNIX machines is the Bourne
shell ("sh") and the DOS/NT command shell on Microsoft Windows systems. You may specify the
full path to the command, for example /usr/bin/ls, or rely on the PATH environment variable to find
the command for you.
If the pipe symbol precedes the program name, then the filehandle is opened for writing and ev-
erything written to the filehandle is sent to the standard input of the program. If the pipe symbol
follows the program, then the filehandle is opened for reading, and everything read from the file-
handle is taken from the program's standard output.
For example, in UNIX the command ls -l will return a listing of the files in the current directory. By

Licensed by
passing an argument of " ls -l | " to open(), we can open a pipe to read from the command:
open (LSFH,"ls -l |") or die "Can't open ls -l: $!";
while (my $line = <LSFH>) {
print "I saw: $line\n";
}

Stjepan Maric
close LSFH;

This fragment simply echoes each line produced by the ls -l command. In a real application, you'd
want to do something more interesting with the information.
As an example of an output pipe, the UNIX wc -lw command will count the lines (option " -l ") and

4218908
words (option " -w ") of a text file sent to it on standard input. This code fragment opens a pipe to
the command, writes a few lines of text to it, and then closes the pipe. When the program runs, the
word and line counts produced by wc are printed in the command window:
open (WC,"| wc -lw") or die "Can't open wordcount: $!";
print WC "This is the first line.\n";
print WC "This is the another line.\n";
print WC "This is the last line.\n";
print WC "Oops. I lied.\n";
close WC;

IO::Filehandle supports pipes through its open() method:

$wc = IO::Filehandle->open("| wc - lw") or die "Can't open wordcount:
$!";

Using Pipes
Let's look at a complete functional example (Figure 2.2). The program whos_there.pl opens up a
pipe to the UNIX who command and counts the number of times each user is logged in. It produces
a report like this one:

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Processes, Pipes, and Signals 31

Figure 2.2. A script to open a pipe to the who command

% whos_there.pl
jsmith 9
abu 5
lstein 1
palumbo 1

This indicates that users "jsmith" and "abu" are logged in 9 and 5 times, respectively, while "lstein"
and "palumbo" are each logged in once. The users are sorted in descending order of the number
of times they are logged in. This is the sort of script that might be used by an administrator of a busy
system to watch usage.
Lines 1–3: Initialize script—We turn on strict syntax checking with use strict. This catches
mistyped variables, inappropriate use of globals, failure to quote strings, and other potential
errors. We create a local hash %who to hold the set of logged-in users and the number of times
they are logged in.
Line 4: Open pipe to who command—We call open() on a filehandle named WHOFH, using who
| as the second argument. If the open() call fails, die with an error message.
Lines 5–8: Read the output of the who command —We read and process the output of who one
line at a time. Each line of who looks like this:
jsmith pts/23 Aug 12 10:26 (cranshaw.cshl.org)

The fields are the username, the name of the terminal he's using, the date he logged in, and the
address of the remote machine he logged in from (this format will vary slightly from one dialect
of UNIX to another). We use a pattern match to extract the username, and we tally the names
into the %who hash in such a way that the usernames become the keys, and the number of times
each user is logged in becomes the value.
The <WHOFH> loop will terminate at the EOF, which in the case of pipes occurs when the program
at the other end of the pipe exits or closes its standard output.
Lines 9–11: Print out the results—We sort the keys of %who based on the number of times each
user has logged in, and print out each username and login count. The printf() format used
here, " %10s %d\n ", tells printf() to format its first argument as a string that is right justified
on a field 10 spaces long, to print a space, and then to print the second argument as a decimal
integer.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Processes, Pipes, and Signals 32

Line 12: Close the pipe—We are done with the pipe now, so we close() it. If an error is detected
during close, we print out a warning.
With pipes, the open() and close() functions are enhanced slightly to provide additional infor-
mation about the subprocess. When opening a pipe, open() returns the process ID (PID) of the
command at the other end of the pipe. This is a unique nonzero integer that can be used to monitor
and control the subprocess with signals (which we discuss in detail later in the Handling Signals
section). You can store this PID, or you can ignore its special meaning and treat the return value
from open() as a Boolean flag.
When closing a pipe, the close() call is enhanced to place the exit code from the subprocess in
the special global variable $?. Contrary to most Perl conventions, $? is zero if the command suc-
ceeded, and nonzero on an error. The perlvar POD page has more to say about the exit code, as
does the section Handling Child Termination in Chapter 10.
Another aspect of close() is that when closing a write pipe, the close() call will block until the
process at the other end has finished all its work and exited. If you close a read pipe before reading
to the EOF, the program at the other end will get a PIPE signal (see The PIPE Signal) the next time
it tries to write to standard output.

Pipes Made Easy: The Backtick Operator

Perl's backtick operator, (`), is an easy way to create a one-shot pipe for reading a program's output.
The backtick acts like the double-quote operator, except that whatever is contained between the
backticks is interpreted as a command to run. For example:
$ls_output = `ls`;

This will run the ls (directory listing) command, capture its output, and assign the output to the
$ls_output scalar.
Internally, Perl opens a pipe to the indicated command, reads everything it prints to standard output,
closes the pipe, and returns the command output as the operator result. Typically at the end of the
result there is a new line, which can be removed with chomp().
Just like double quotes, backticks interpolate scalar variables and arrays. For example, we can
create a variable containing the arguments to pass to ls like this:
$arguments = '-l -F';
$ls_output = `ls $arguments`;

The command's standard error is not redirected by backticks. If the subprocess writes any diagnostic
or error messages, they will be intermingled with your program's diagnostics. On UNIX systems,
you can use the Bourne shell's output redirection system to combine the subprocess's standard
error with its standard output like this:
$ls_output = `ls 2>&1`;

Now $ls_output will contain both the standard error and the standard output of the command.

Pipes Made Powerful: The pipe() Function

A powerful but slightly involved way to create a pipe is with Perl's built-in pipe() function.
pipe() creates a pair of filehandles: one for reading and one for writing. Everything written to the
one filehandle can be read from the other.

$result = pipe (READHANDLE,WRITEHANDLE)

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Processes, Pipes, and Signals 33

Open a pair of filehandles connected by a pipe. The first argument is the name of a filehandle to read from,
and the second is a filehandle to write to. If successful, pipe() returns a true result code.

Why is pipe() useful? It is commonly used in conjunction with the fork() function in order to
create a parent-child pair that can exchange data. The parent process keeps one filehandle and
closes the other, while the child does the opposite. The parent and child process can now commu-
nicate across the pipe as they work in parallel.
A short example will illustrate the power of this technique. Given a positive integer, the facfib.pl script
calculates its factorial and the value of its position in the Fibonacci series. To take advantage of
modern multiprocessing machines, these calculations are performed in two subprocesses so that
both calculations proceed in parallel. The script uses pipe() to create filehandles that the child
processes can use to communicate their findings to the parent process that launched them. When
we run this program, we may see results like this:
% facfib.pl 8
factorial(1) => 1
factorial(2) => 2
factorial(3) => 6
factorial(4) => 24
factorial(5) => 120
fibonacci(1) => 1
factorial(6) => 720
fibonacci(2) => 1
factorial(7) => 5040
fibonacci(3) => 2
factorial(8) => 40320
fibonacci(4) => 3
fibonacci(5) => 5
fibonacci(6) => 8
fibonacci(7) => 13
fibonacci(8) => 21

The results from the factorial and Fibonacci calculation overlap because they are occurring in par-
allel.
Figure 2.3 shows how this program works.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Processes, Pipes, and Signals 34

Figure 2.3. Using pipe() to create linked filehandles

Lines 1–3: Initialize module—We turn on strict syntax checking and recover the command-line
argument. If no argument is given, we default to 10.
Line 4: Create linked pipes—We create linked pipes with pipe(). READER will be used by the
main (parent) process to read results from the children, which will use WRITER to write their
results.
Lines 5–10: Create first child process—We call fork() to clone the current process. In the
parent process, fork() returns the nonzero PID of the child process. In the child process,
fork() returns numeric 0. If we see that the result of fork() is 0, we know we are the child
process. We close the READER filehandle because we don't need it. We select() WRITER,
making it the default filehandle for output, and turn on autoflush mode by setting $| to a true
value. This is necessary to ensure that the parent process gets our messages as soon as we
write them.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Processes, Pipes, and Signals 35

We now call the factorial() subroutine with the integer argument from the command line.
After this, the child process is done with its work, so we exit(). Our copy of WRITER is closed
automatically.
Lines 11–16: Create the second child process—Back in the parent process, we invoke
fork() again to create a second child process. This one, however, calls the fibonacci()
subroutine rather than factorial().
Lines 17–19: Process messages from children—In the parent process, we close WRITER be-
cause we no longer need it. We read from READER one line at a time, and print out the results.
This will contain lines issued by both children. READER returns undef when the last child has
finished and closed its WRITER filehandle, sending us an EOF. We could close() READER
and check the result code, or let Perl close the filehandle when we exit, as we do here.
Lines 20–25: The factorial() subroutine—We calculate the factorial of the subroutine ar-
gument in a straightforward iterative way. For each step of the calculation, we print out the
intermediate result. Because WRITER has been made the default filehandle with select(),
each print() statement enters the pipe, where it is ultimately read by the parent process.
Lines 26–34: The fibonacci() subroutine—This is identical to factorial() except for the
calculation itself.
Instead of merely echoing its children's output, we could have the parent do something more useful
with the information. We use a variant of this technique in Chapter 14 to implement a preforked Web
server. The parent Web server manages possibly hundreds of children, each of which is responsible
for processing incoming Web requests. To tune the number of child processes to the incoming load,
the parent monitors the status of the children via messages that they send via a pipe launching more
children under conditions of high load, and killing excess children when the load is low.
The pipe() function can also be used to create a filehandle connected to another program in much
the way that piped open() does. We don't use this technique elsewhere, but the general idea is
for the parent process to fork(), and for the child process to reopen either STDIN or STDOUT onto
one of the paired filehandles, and then exec() the desired program with arguments. Here's the
idiom:
pipe(READER,WRITER) or die "pipe no good: $!";
my $child = fork();
die "Can't fork: $!" unless defined $child;
if ($child == 0) { # child process
close READER; # child doesn't need this
open (STDOUT,">&WRITER"); # STDOUT now goes to writer
exec $cmd,$args;
die "exec failed: $!";
}
close WRITER; # parent doesn't need this

At the end of this code, READER will be attached to the standard output of the command named
$cmd, and the effect is almost exactly identical to this code:
open (READER,"$cmd $args |") or die "pipe no good: $!";

Bidirectional Pipes
Both piped open() and pipe() create unidirectional filehandles. If you want to both read and write
to another process, you're out of luck. In particular, this sensible-looking syntax does not work:
open(FH,"| $cmd |");

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Processes, Pipes, and Signals 36

One way around this is to call pipe() twice, creating two pairs of linked filehandles. One pair is
used for writing from parent to child, and the other for child to parent, rather like a two-lane highway.
We won't go into this technique, but it's what the standard IPC::Open2 and IPC::Open3 modules do
to create a set of filehandles attached to the STDIN, STDOUT, and STDERR of a subprocess.
A more elegant way to create a bidirectional pipe is with the socketpair() function. This creates
two linked filehandles like pipe() does, but instead of being a one-way connection, both filehandles
are read/write. Data written into one filehandle comes out the other one, and vice versa. Because
the socketpair() function involves the same concepts as the socket() function used for net-
work communications, we defer our discussion of it until Chapter 4.

Distinguishing Pipes from Plain Filehandles

You will occasionally need to test a filehandle to see if it is opened on a file or a pipe. Perl's filehandle
tests make this possible (Table 2.1).
Table 2.1. Perl's Filehandle Tests
Test Description
-p Filehandle is a pipe.
-t Filehandle is opened on a terminal.
-s Filehandle is a socket.

If a filehandle is opened on a pipe, the -p test will return true:

print "I've got a pipe!\n" if -p FILEHANDLE;

The -t and -S file tests can distinguish other special types of filehandle. If a filehandle is opened
on a terminal (the command-line window), then -t will return true. Programs can use this to test
STDIN to see if the program is being run interactively or has its standard input redirected from a file:
print "Running in batch mode, confirmation prompts disabled.\n"
unless -t STDIN;

The -S test detects whether a filehandle is opened on a network socket (introduced in Chapter 3):
print "Network active.\n" if -S FH

There are more than a dozen other file test functions that can give you a file's size, modification
date, ownership, and other information. See the perlfunc POD page for details.

The Dreaded PIPE Error

When your script is reading from a filehandle opened on a pipe, and the program at the other end
either exits or simply closes its end of the pipe, your program will receive an EOF on the filehandle.
What happens in the opposite case, when your script is writing to a pipe and the program at the
other end terminates prematurely or closes its end of the connection?
To find out, we can write two short Perl scripts. One, named write_ten.pl, opens up a pipe to the
second program and attempts to write ten lines of text to it. The script checks the result code from
print(), and bumps up a variable named $count whenever print() returns a true result. When
write_ten.pl is done, it displays the contents of $count, indicating the number of lines that were
successfully written to the pipe. The second program, named read_three.pl, reads three lines of text
from standard input and then exits.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Processes, Pipes, and Signals 37

The two scripts are shown in Figures 2.4 and 2.5. Of note is that write_ten.pl puts the pipe into
autoflush mode so that each line of text is sent down the pipe immediately, rather than being buffered
locally. write_ten.pl also sleep()s for one second after writing each line of text, giving
read_three.pl a chance to report that the text was received. Together, these steps make it easier
for us to see what is happening. When we run write_ten.pl we see the following:

Figure 2.4. The write_ten.pl script writes ten lines of text to a pipe

Figure 2.5. The read_three.pl script reads three lines of text from standard input

% write_ten.pl
Writing line 1
Read_three got: This is line number 1
Writing line 2
Read_three got: This is line number 2
Writing line 3
Read_three got: This is line number 3
Writing line 4
Broken pipe
%

Everything works as expected through line three, at which point read_three.pl exits. When
write_ten.pl attempts to write the fourth line of text, the script crashes with a Broken pipe error.
The statement that prints out the number of lines successfully passed to the pipe is never executed.
When a program attempts to write to a pipe and no program is reading at the other end, this results
in a PIPE exception. This exception, in turn, results in a PIPE signal being delivered to the writer.
By default this signal results in the immediate termination of the offending program. The same error
occurs in network applications when the sender attempts to transmit data to a remote program that
has exited or has stopped receiving.
To deal effectively with PIPE, you must install a signal handler, and this brings us to the next major
topic.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Processes, Pipes, and Signals 38

Signals
As with filehandles, understanding signals is fundamental to network programming. A signal is a
message sent to your program by the operating system to tell it that something important has oc-
curred. A signal can indicate an error in the program itself such as an attempt to divide by zero, an
event that requires immediate attention such as an attempt by the user to interrupt the program, or
a noncritical informational event such as the termination of a subprocess that your program has
launched.
In addition to signals sent by the operating system, processes can signal each other. For example,
when the user presses control-C (^C) on the keyboard to send an interrupt signal to the currently
running program, that signal is sent not by the operating system, but by the command shell that pro
cesses and interprets keystrokes. It is also possible for a process to send signals to itself.

Common Signals
The POSIX standard defines nineteen signals. Each has a small integer value and a symbolic name.
We list them in Table 2.2 (the gaps in the integer sequence represent nonstandard signals used by
some systems).
The third column of the table indicates what happens when a process receives the signal. Some
signals do nothing. Others cause the process to terminate immediately, and still others terminate
the process and cause a core dump. Most signals can be "caught." That is, the program can install
a handler for the signal and take special action when the signal is received. A few signals, however,
cannot be intercepted in this way.
You don't need to understand all of the signals listed in Table 2.2 because either they won't occur
during the execution of a Perl script, or their generation indicates a low-level bug in Perl itself that
you can't do anything about. However, a handful of signals are relatively common, and we'll look at
them in more detail now.
HUP signals a hangup event. This typically occurs when a user is running a program from the com-
mand line, and then closes the command-line window or exits the interpreter shell. The default action
for this signal is to terminate the program.
INT signals a user-initiated interruption. It is generated when the user presses the interrupt key,
typically ^C. The default behavior of this signal is to terminate the program. QUIT is similar to
INT, but also causes the program to generate a core file (on UNIX systems). This signal is issued
when the user presses the "quit" key, ordinarily ^\.
Table 2.2. POSIX Signals
Signal Name Value Notes Comment
HUP 1 A Hangup detected
INT 2 A Interrupt from keyboard
QUIT 3 A Quit from keyboard
ILL 4 A Illegal Instruction
ABRT 6 C Abort
Notes:
A—Default action is to terminate process.
B—Default action is to ignore the signal.
C—Default action is to terminate process and dump core.
D—Default action is to stop the process.
E—Default action is to resume the process.
F—Signal cannot be caught or ignored.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Processes, Pipes, and Signals 39

Signal Name Value Notes Comment

FPE 8 C Floating point exception
KILL 9 AF Termination signal
USR1 10 A User-defined signal 1
SEGV 11 C Invalid memory reference
USR2 12 A User-defined signal 2
PIPE 13 A Write to pipe with no readers
ALRM 14 A Timer signal from alarm clock
TERM 15 A Termination signal
CHLD 17 B Child terminated
CONT 18 E Continue if stopped
STOP 19 DF Stop process
TSTP 20 D Stop typed at tty
TTIN 21 D tty input for background process
TTOU 22 D tty output for background process
Notes:
A—Default action is to terminate process.
B—Default action is to ignore the signal.
C—Default action is to terminate process and dump core.
D—Default action is to stop the process.
E—Default action is to resume the process.
F—Signal cannot be caught or ignored.

By convention, TERM and KILL are used by one process to terminate another. By default, TERM
causes immediate termination of the program, but a program can install a signal handler for TERM
to intercept the terminate request and possibly perform some cleanup actions before quitting. The
KILL signal, in contrast, is uncatchable. It causes an immediate shutdown of the process without
chance of appeal. For example, when a UNIX system is shutting down, the script that handles the
shutdown process first sends a TERM to each running process in turn, giving it a chance to clean
up. If the process is still running a few tens of seconds later, then the shutdown script sends a KILL.
The PIPE signal is sent when a program writes to a pipe or socket but the process at the remote
end has either exited or closed the pipe. This signal is so common in networking applications that
we will look at it closely in the Handling PIPE Exceptions section.
ALRM is used in conjunction with the alarm() function to send the program a prearranged signal
after a certain amount of time has elapsed. Among other things, ALRM can be used to time out
blocked I/O calls. We will see examples of this in the Timing Out Long-Running Operations section.
CHLD occurs when your process has launched a subprocess, and the status of the child has changed
in some way. Typically the change in status is that the child has exited, but CHLD is also generated
whenever the child is stopped or continued. We discuss how to deal with CHLD in much greater
detail in Chapters 4 and 9.
STOP and TSTP both have the effect of stopping the current process. The process is put into sus-
pended animation indefinitely; it can be resumed by sending it a CONT signal. STOP is generally
used by one program to stop another. TSTP is issued by the interpreter shell when the user presses
the stop key (^Z on UNIX systems). The other difference between the two is that TSTP can be caught,
but STOP cannot be caught or ignored.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Processes, Pipes, and Signals 40

Catching Signals
You can catch a signal by adding a signal handler to the %SIG global hash. Use the name of the
signal you wish to catch as the hash key. For example, use $SIG{INT} to get or set the INT signal
handler. As the value, use a code reference: either an anonymous subroutine or a reference to a
named subroutine. For example, Figure 2.6 shows a tiny script that installs an INT handler. Instead
of terminating when we press the interrupt key, it prints out a short message and bumps up a counter.
This goes on until the script counts three interruptions, at which point it finally terminates. In the
transcript that follows, the "Don't interrupt me!" message was triggered each time I typed ^C:

Licensed by
Stjepan Maric
% interrupt.pl
I'm sleeping.
4218908 Figure 2.6. Catching the INT signal

I'm sleeping.
Don't interrupt me! You've already interrupted me 1x.
I'm sleeping.
I'm sleeping.
Don't interrupt me! You've already interrupted me 2x.
I'm sleeping.
Don't interrupt me! You've already interrupted me 3x.

Let's look at the script in detail.

Lines 1–3: Initialize script—We turn on strict syntax checking, and declare a global counter
named $interruptions. This counter will keep track of the number of times the script has
received INT.
Line 4: Install INT handler—We install a handler for the INT signal by setting $SIG{INT} to a
reference to the subroutine handle_interruptions().
Lines 5–8: Main loop—The main loop of the program simply prints a message and calls sleep
with an argument of 5. This puts the program to sleep for 5 seconds, or until a signal is received.
This continues until the $interruptions counter becomes 3 or greater.
Lines 9–12: The handle_interruptions() subroutine—The handle_interrup-
tions() subroutine is called whenever the INT signal arrives, even if the program is busy doing
something else at that moment. In this case, our signal handler bumps up $interruptions
and prints out a warning.
For short signal handlers, you can use an anonymous subroutine as the handler. For example, this
code fragment is equivalent to that in Figure 2.6, but we don't have to formally name the handler
subroutine:

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Processes, Pipes, and Signals 41

$SIG{INT} = sub {
$interruptions++;
warn "Don't interrupt me! You've already interrupted
me ${interruptions}x.\n";
};

In addition to code references, %SIG recognizes two special cases. The string " DEFAULT " restores
the default behavior of the signal. For example, setting $SIG{INT} to " DEFAULT " will cause the
INT signal to terminate the script once again. The string " IGNORE " will cause the signal to be
ignored altogether.
As previously mentioned, don't bother installing a handler for either KILL or STOP. These signals
can be neither caught nor ignored, and their default actions will always be performed.
If you wish to use the same routine to catch several different signals, and it is important for the
subroutine to distinguish one signal from another, it can do so by looking at its first argument, which
will contain the name of the signal. For example, for INT signals, the handler will be called with the
string " INT ":
$SIG{TERM} = $SIG{HUP} = $SIG{INT} = \&handler
sub handler {
my $sig = shift;
warn "Handling a $sig signal.\n";
}

Handling PIPE Exceptions

We now have what we need to deal with PIPE exceptions. Recall the write_ ten.pl and
read_three.pl examples from Figures 2.4 and 2.5 in The Dreaded PIPE Error section. write_ten.pl
opens a pipe to read_three.pl and tries to write ten lines of text to it, but read_three.pl is only prepared
to accept three lines, after which it exits and closes its end of the pipe. write_ten.pl, not knowing that
the other end of the connection has exited, attempts to write a fourth line of text, generating a
PIPE signal.
We will now modify write_ten.pl so that it detects the PIPE error and handles it more gracefully. We
will use variants on this technique in later chapters that deal with common issues in network com-
munications.
The first technique is shown in Figure 2.7, the write_ten_ph.pl script. Here we set a global flag,
$ok, which starts out true. We then install a PIPE handler using this code:

Figure 2.7. The write_ten_ph.pl script

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Processes, Pipes, and Signals 42

$SIG{PIPE} = sub { undef $ok };

When a PIPE signal is received, the handler will undefine the $ok flag, making it false.
The other modification is to replace the simple for() loop in the original version with a more so-
phisticated version that checks the status of $ok. If the flag becomes false, the loop exits. When
we run the modified script, we see that the program runs to completion, and correctly reports the
number of lines successfully written:
% write_ten_ph.pl
Writing line 1
Read_three got: This is line number 1
Writing line 2
Read_three got: This is line number 2
Writing line 3
Read_three got: This is line number 3
Writing line 4
Wrote 3 lines of text

Another general technique is to set $SIG{PIPE} to 'IGNORE', in order to ignore the PIPE signal
entirely. It is now our responsibility to detect that something is amiss, which we can do by examining
the result code from print(). If print() returns false, we exit the loop.
Figure 2.8 shows the code for write_ten_i.pl, which illustrates this technique. This script begins by
setting $SIG{PIPE} to the string 'IGNORE', suppressing PIPE signals. In addition, we modify the
print loop so that if print() is successful, we bump up $count as before, but if it fails, we issue
a warning and exit the loop via last.

Figure 2.8. The write_ten_i.pl script

When we run write_ten_i.pl we get this output:

% write_ten_i.pl
Writing line 1
Read_three got: This is line number 1
Writing line 2
Read_three got: This is line number 2
Writing line 3

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Processes, Pipes, and Signals 43

Read_three got: This is line number 3

Writing line 4
An error occurred during writing: Broken pipe
Wrote 3 lines of text

Notice that the error message that appears in $! after the unsuccessful print is "Broken pipe." If we
wanted to treat this error separately from other I/O errors, we could explicitly test its value via a
pattern match, or, better still, check its numeric value against the numeric error constant EPIPE. For
example:
use Errno ':POSIX';
...
unless (print PIPE "This is line number $_\n") { # handle write error
last if $! == EPIPE; # on PIPE, just terminate the loop
die "I/O error: $!"; # otherwise die with an error message
}

Sending Signals
A Perl script can send a signal to another process using the kill() function:

$count = kill($signal,@processes)
The kill() function sends signal $signal to one or more processes. You may specify the signal numerically,
for example 2, or symbolically as in " INT ". @processes is a list of one or more process IDs to deliver the
signal to. The number of processes successfully signaled is returned as the kill() function result.

One process can only signal another if it has sufficient privileges to do so. In general, a process
running under a normal user's privileges can signal only other processes that are running under the
same user's privileges. A process running with root or superuser privileges, however, can signal
any other process.
The kill() function provides a few tricks. If you use the special signal number 0, then kill()
will return the number of processes that could have been signaled, without actually delivering the
signal. If you use a negative number for the process ID, then kill() will treat the absolute value
of the number as a process group ID and deliver the signal to all members of the group.
A script can send a signal to itself by calling kill() on the variable $$, which holds the current
process ID. For example, here's a fancy way for a script to commit suicide:
kill INT => $$; # same as kill('INT',$$)

Signal Handler Caveats

Because a signal can arrive at any time during a program's execution, it can arrive while the Perl
interpreter is doing something important, like updating one of its memory structures, or even while
inside a system call. If the signal handler does something that rearranges memory, such as allo-
cating or dis posing of a big data structure, then on return from the handler Perl may find its world
changed from underneath it, and get confused, which occasionally results in an ugly crash.
To avoid this possibility, signal handlers should do as little as possible. The safest course is to set
a global variable and return, as we did in the PIPE handler in Figure 2.7. In addition to memory-
changing operations, I/O operations within signal handlers should also be avoided. Although we
liberally pepper our signal handlers with diagnostic warn() statements throughout this book, these
operations should be stripped out in production programs.
It's generally OK to call die() and exit() within signal handlers. The exception is on Microsoft
Windows systems, where due to limitations in the signal library, these two calls may cause "Dr.
Watson" errors if invoked from within a signal handler.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Processes, Pipes, and Signals 44

Indeed, the implementation of signals on Windows systems is currently extremely limited. Simple
things, such as an INT handler to catch the interrupt key, will work. More complex things, such as
CHLD handlers to catch the death of a subprocess, do not work. This is an area of active development
so be sure to check the release notes before trying to write or adapt any code that depends heavily
on signals.
Signal handling is not implemented in MacPerl.

Timing Out Slow System Calls

A signal may occur while Perl is executing a system call. In most cases, Perl automatically restarts
the call and it takes up exactly where it left off.
A few system calls, however, are exceptions to this rule. One is sleep(), which suspends the script
for the indicated number of seconds. If a signal interrupts sleep(), however, it will exit early, re-
turning the number of seconds it slept before being awakened. This property of sleep() is quite
useful because it can be used to put the script to sleep until some expected event occurs.

$slept = sleep([$seconds])
Sleep for the indicated number of seconds or until a signal is received. If no argument is provided, this function
will sleep forever. On return, sleep() will return the number of seconds it actually slept.

Another exception is the four-argument version of select(), which can be used to perform a timed
wait until one or more of a set of filehandles are ready for I/O. This function is described in detail in
Chapter 12.
Sometimes the automatic restarting of system calls is not what you want. For example, consider an
application that prompts a user to type her password and tries to read the response from standard
input. You might want the read to time out after some period of time in case the user has wandered
off and left the terminal unattended. This fragment of code might at first seem to do the trick:
my $timed_out = 0;

$SIG{ALRM} = sub { $timed_out = 1 };

print STDERR "type your password: ";

alarm (5); # five second timeout
my $password = <STDIN>;
alarm (0);

print STDERR "you timed out\n" if $timed_out;

Here we use the alarm() function to set a timer. When the timer expires, the operating system
generates an ALRM signal, which we intercept with a handler that sets the $timed_out global to
true. In this code we call alarm() with a five-second timeout, and then read a line of input from
standard input. After the read completes, we call alarm() again with an argument of zero, turning
the timer off. The idea is that the user will have five seconds in which to type a password. If she
doesn't, the alarm clock goes off and we fall through to the rest of the program.

$seconds_left = alarm($seconds)
Arrange for an ALRM signal to be delivered to the process after $seconds. The function result is the number
of seconds left from the previous timer, if any. An argument of zero disables the timer.

The problem is that Perl automatically restarts slow system calls, including <>. Even though the
alarm clock has gone off, we remain in the <> call, waiting for the user's keyboard input.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Processes, Pipes, and Signals 45

The solution to this problem is to use eval{} and a local ALRM handler to abort the read. The
general idiom is this:
print STDERR "type your password: ";
my $password =
eval {
local $SIG{ALRM} = sub { die "timeout\n" };
alarm (5); # five second timeout
return <STDIN>;
};
alarm (0);
print STDERR "you timed out\n" if $@ =~ /timeout/;

Instead of having an ALRM handler in the main body of the program, we localize it within an
eval{} block. The eval{} block sets the alarm, as before, and attempts to read from STDIN. If
<> returns before the timer goes off, then the line of input is returned from the eval{} block, and
assigned to $password.
However, if the timer goes off before the input is complete, the ALRM handler executes, dying with
the error message "timeout." However, since we are dying within an eval{} block, the effect of this
is for eval{} to return undef, setting the variable $@ to the last error message. We pattern match
$@ for the timeout message, and print a warning if found.
In either case, we turn off the timer immediately after returning from the eval{} block in order to
avoid having the timer go off at an inconvenient moment.
We will use this technique several times in later chapters when we need to time out slow network
calls.

Summary
This chapter introduced three topics that we will use throughout this book.
Processes correspond to an instance of a running program. Perl can create new processes via its
system() and fork() commands, or replace the current process with a different one with exec().
Pipes are I/O connections between two processes. A pipe looks and acts like a filehandle, but it is
connected to another process rather than to a file. If a pipe is opened for reading, data read from it
is taken from the standard output of the process at the other end. If a pipe is opened for writing, data
printed to it is received by the other process on its standard input.
Signals provide programs with notification of exceptional conditions, among which are PIPE errors
and other I/O-related problems. Signals are also useful for timing out long-running operations and
catching urgent requests from the user. You can manage incoming signals by installing signal han-
dlers in the %SIG hash, and you can send signals to other processes (or your own) using the
kill() function.
The next chapter goes into the particulars of Berkeley sockets before leading into a full discussion
of TCP networking.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 46

Chapter 3. Introduction to Berkeley Sockets

This chapter introduces Perl's version of the Berkeley socket API, the low-level network API that
underlies all of Perl's networking modules. It explains the different kinds of sockets, demonstrates
basic Berkeley-based clients and servers, and discusses some of the common traps programmers
encounter when first working with the API.

Clients, Servers, and Protocols

Network communication occurs when two programs exchange data across the net. With rare ex-
ceptions, the two programs are not equal. One, the client, initiates the connection and is usually,
but not always, connected to a single server at a time. The other partner in the connection, the
server, is passive, waiting quietly until it is contacted by a client seeking a connection. In contrast
to clients, it is common for a server to service incoming connections from multiple clients at the same
time.
Although it is often true that the computer ("host") that runs the server is larger and more powerful
than the client machine, this is not a rule by any means. In fact, in some popular applications, such
as the X Windows System, the situation is reversed. The server is usually a personal computer or
other desktop machine, while the client is run on a more powerful "server class" machine.
Most of the network protocols that we are familiar with are client-server applications. This includes
the HTTP used on the Web, the SMTP used for Internet e-mail, and all the database access pro-
tocols. However, a small but growing class of network applications is peer-to-peer. In peer-to-peer
scenarios, the two ends of the connection are equivalent, each making multiple connections to other
copies of the same program. The controversial Napster file-sharing protocol is peer-to-peer, as are
its spiritual heirs Gnutella and Freenet.

Protocols
We've thrown around the word protocol, but what is it, exactly? A protocol is simply an agreed-upon
set of standards whereby two software components interoperate. There are protocols at every level
of the networking stack (Figure 3.1).

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Introduction to Berkeley Sockets 47

Figure 3.1. The layers of the TCP/IP stack

At the lowest level is the hardware or datalink layer, where, for example, the drivers built into Ethernet
network interface cards have a common understanding of how to interpret the pulses of electric
potential on the network wire in terms of Ethernet frames, how to detect when the wire is in use by
another card, and how to detect and resolve collisions between two cards trying to transmit at the
same time.
One level up is the network layer. At this layer, information is grouped into packets that consist of
a header containing the sender and recipient's address, and a payload that consists of the actual
data to send. Payloads are typically in the range of 500 bytes to 1500 bytes. Internet routers act at
the IP layer by reading packet headers and figuring out how to route them to their destinations. The
main protocol at this layer is the Internet Protocol, or IP.
The transport layer is concerned with creating data packets and ensuring the integrity of their con-
tents. The two important protocols at this layer are the Transmission Control Protocol (TCP), which
provides reliable connection-oriented communications, and the User Datagram Protocol (UDP),
which provides an unreliable message-oriented service. These protocols are responsible for getting
the data to its destination. They don't care what is actually inside the data stream.
At the top of the stack is the application layer, where the content of the data stream does matter.
There is an abundance of protocols at this level, including such familiar and unfamiliar names as
HTTP, FTP, SMTP, POP3, IMAP, SNMP, XDMCP, and NNTP. These protocols specify, sometimes
in excruciating detail, how a client should contact a server, what messages are allowed, and what
information to exchange with each message.
The combination of the network layer and the transport layer is known as TCP/IP, named after the
two major protocols that operate at those layers.

Binary versus Text-Oriented Protocols

Before they can exchange information across the network, hosts have a fundamental choice to
make. They can exchange data either in binary form or as human-readable text. The choice has
far-reaching ramifications.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Introduction to Berkeley Sockets 48

To understand this, consider exchanging the number 1984. To exchange it as text, one host sends
the other the string 1984, which, in the common ASCII character set, corresponds to the four hex-
adecimal bytes 0x31 0x39 0x38 0x34. These four bytes will be transferred in order across the
network, and (provided the other host also speaks ASCII) will appear at the other end as "1984".
However, 1984 can also be treated as a number, in which case it can fit into the two-byte integer
represented in hexadecimal as 0x7C0. If this number is already stored in the local host as a number,
it seems sensible to transfer it across the network in its native two-byte form rather than convert it
into its four-byte text representation, transfer it, and convert it back into a two-byte number at the
other end. Not only does this save some computation, but it uses only half as much network capacity.
Unfortunately, there's a hitch. Different computer architectures have different ways of storing inte-
gers and floating point numbers. Some machines use two-byte integers, others four-byte integers,
and still others use eight-byte integers. This is called word size. Furthermore, computer architectures
have two different conventions for storing integers in memory. In some systems, called big-endian
architectures, the most significant part of the integer is stored in the first byte of a two-byte integer.
On such systems, reading from low to high, 1984 is represented in memory as the two bytes:
0x07 0xC0
low -> high

On little-endian architectures, this convention is reversed, and 1984 is stored in the opposite ori-
entation:
0xC0 0x07
low -> high

These architectures are a matter of convention, and neither has a significant advantage over the
other. The problem comes when transferring such data across the network, because this byte pair
has to be transferred serially as two bytes. Data in memory is sent across the network from low to
high, so for big-endian machines the number 1984 will be transferred as 0x07 0xC0, while for little-
endian machines the numbers will be sent in the reverse order. As long as the machine at the other
end has the same native word size and byte order, these bytes will be correctly interpreted as 1984
when they arrive. However, if the recipient uses a different byte order, then the two bytes will be
interpreted in the wrong order, yielding hexadecimal 0xC007, or decimal 49,159. Even worse, if the
recipient interprets these bytes as the top half of a four-byte integer, it will end up as
0xC0070000, or 3,221,684,224. Someone's anniversary party is going to be very late.
Because of the potential for such binary chaos, text-based protocols are the norm on the Internet.
All the common protocols convert numeric information into text prior to transferring them, even
though this can result in more data being transferred across the net. Some protocols even convert
data that doesn't have a sensible text representation, such as audio files, into a form that uses the
ASCII character set, because this is generally easier to work with. By the same token, a great many
protocols are line-oriented, meaning that they accept commands and transmit data in the form of
discrete lines, each terminated by a commonly agreed-upon newline sequence.
A few protocols, however, are binary. Examples include Sun's Remote Procedure Call (RPC) sys-
tem, and the Napster peer-to-peer file exchange protocol. Such protocols have to be exceptionally
careful to represent binary data in a common format. For integer numbers, there is a commonly
recognized network format. In network format, a "short" integer is represented in two big-endian
bytes, while a "long" integer is represented with four big-endian bytes. As we will see in Chapter
19, Perl's pack() and unpack () functions provide the ability to convert numbers into network
format and back again.
Floating point numbers and more complicated things like data structures have no commonly ac-
cepted network representation. When exchanging binary data, each protocol has to work out its own
way of representing such data in a platform-neutral fashion.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Introduction to Berkeley Sockets 49

We will stick to text-based protocols for much of this book. However, to give you a taste for what it's
like to use a binary protocol, the UDP-based real-time chat system in Chapters 19 and 20 exchanges
platform-neutral binary messages.

Berkeley Sockets
Berkeley sockets are part of an application programming interface (API) that specifies the data
structures and function calls that interact with the operating system's network subsystem. The name
derives from the origins of the API in release 4.2 of the Berkeley Standard Distribution (4.2BSD) of
UNIX. Berkeley sockets act at the transport layer: They help get the data where it's going, but have
nothing to say about the content of the data.
Berkeley sockets are part of an API, not a specific protocol, which defines how the programmer
interacts with an idealized network. Although strongly associated with the TCP/IP network protocol
for which the API was first designed, the Berkeley sockets API is generic enough to support other
types of network, such as Novell Netware, Windows NT networking, and Appletalk.
Perl provides full support for Berkeley sockets on most of the platforms it runs on. On certain plat-
forms such as the Windows and Macintosh ports, extension modules also give you access to the
non-Berkeley APIs native to those machines. However, if you're interested in writing portable ap-
plications you'll probably want to stick to the Berkeley API.

The Anatomy of a Socket

A socket is an endpoint for communications, a portal to the outside world that we can use to send
outgoing messages to other processes, and to receive incoming traffic from processes interested
in sending messages to us.
To create a socket, we need to provide the system with a minimum of three pieces of information.

The Socket's Domain

The domain defines the family of networking protocols and addressing schemes that the socket will
support. The domain is selected from a small number of integer constants defined by the operating
system and exported by Perl's Socket module. There are only two common domains (see Table 3.1).
AF_INET is used for TCP/IP networking. Sockets in this domain use IP addresses and port numbers
as their addressing scheme (more on this later). AF_UNIX is used only for interprocess communi-
cation within a single host. Addresses in this domain are file pathnames. The name AF_UNIX is
unfairly UNIX-specific; it's possible for non-UNIX systems to implement it. For this reason, POSIX
has tried to rename this constant AF_LOCAL, although few systems have followed suit.

Table 3.1. Common Socket Domains

Constant Description
AF_INET The Internet protocols
AF_UNIX Networking within a single host

In addition to these domains, there are many others including AF_APPLETALK, AF_IPX, and
AF_X25, each corresponding to a particular addressing scheme. AF_INET6, corresponding to the
long addresses of TCP/IP version 6, will become important in the future, but is not yet supported by
Perl.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Introduction to Berkeley Sockets 50

The AF_ prefix stands for "address family." In addition, there is a series of "protocol family" constants
starting with the PF_ prefix. For example, there is a PF_INET constant that corresponds to
AF_INET. These constants evaluate to the same value and can, in fact, be used interchangeably.
The distinction between them is a historical artifact, and you'll find that published code sometimes
uses the one and sometimes the other.

The Socket's Type

The socket type identifies the basic properties of socket communications. As explained more fully
in the next section, sockets can be either a "stream" type, in which case data is sent through a socket
in a continuous stream (like reading or writing to a file, for example), or a "datagram" type, in which
case data is sent and received in discrete packages.
The type is controlled by an operating-system-defined constant that evaluates to a small integer.
Common constants exported by Socket are shown in Table 3.2.

Licensed by
Table 3.2. Constants Exported by Socket
Constant Description
SOCK_STREAM A continuous stream of data
SOCK_DGRAM Individual packets of data

Stjepan Maric
SOCK_RAW Access to internal protocols and interfaces

Perl fully supports the SOCK_STREAM and SOCK_DGRAM socket types. SOCK_RAW is supported
through an add-on module named Net::Raw.

The Socket's Protocol

4218908
For a given socket domain and type, there may be one or several protocols that implement the
desired behavior. Like the domain and socket type, the protocol is a small integer. However, the
protocol numbers are not available as constants, but instead must be looked up at run time using
the Perl getprotobyname() function. Some of the protocols are listed in Table 3.3.
Table 3.3. Some Socket Protocols
Protocol Description
tcp Transmission Control Protocol for stream sockets
udp User Datagram Protocol for datagram sockets
icmp Internet Control Message Protocol
raw Creates IP packets manually

The TCP and UDP protocols are supported directly by the Perl sockets API. You can get access to
the ICMP and raw protocols via the Net::ICMP and Net::Raw third-party modules, but we do not
discuss them in this book (it would be possible, but probably not sensible, to reimplement TCP in
Perl using raw packets).
Generally there exists a single protocol to support a given domain and type. When creating a socket,
you must be careful to set the domain and socket type to match the protocol you've selected. The
possible combinations are summarized in Table 3.4.
Table 3.4. Allowed Combinations of Socket Type and Protocol in the INET and UNIX Domains
Domain Type Protocol
AF_INET SOCK_STREAM tcp
AF_INET SOCK_DGRAM udp
AF_UNIX SOCK_STREAM PF_UNSPEC

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Introduction to Berkeley Sockets 51

Domain Type Protocol

AF_UNIX SOCK_DGRAM PF_UNSPEC

The allowed combinations of socket domain, type, and protocol are few. SOCK_STREAM goes with
TCP, and SOCK_DGRAM goes with UDP. Also notice that the AF_UNIX address family doesn't use
a named protocol, but a pseudoprotocol named PF_UNSPEC (for "unspecified").
The Perl object-oriented IO::Socket module (discussed in Chapter 5) can fill in the correct socket
type and protocol automatically when provided with partial information.

Datagram Sockets
Datagram-type sockets provide for the transmission of connectionless, unreliable, unsequenced
messages. The UDP is the chief datagram-style protocol used by the Internet protocol family.
As the diagram in Figure 3.2 shows, datagram services resemble the postal system. Like a letter or
a telegram, each datagram in the system carries its destination address, its return address, and a
certain amount of data. The Internet protocols will make the best effort to get the datagram delivered
to its destination.

Figure 3.2. Datagram sockets provide connectionless, unreliable, unsequenced transmission of message

There is no long-term relationship between the sending socket and the recipient socket: A client can
send a datagram off to one server, then immediately turn around and send a datagram to another
server using the same socket. But the connectionless nature of UDP comes at a price. Like certain
countries' postal systems, it is very possible for a datagram to get "lost in the mail." A client cannot
know whether a server has received its message until it receives an acknowledgment in reply. Even
then, it can't know for sure that a message was lost, because the server might have received the
original message and the acknowledgment got lost!

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Introduction to Berkeley Sockets 52

Datagrams are neither synchronized nor flow controlled. If you send a set of datagrams out in a
particular order, they might not arrive in that order. Because of the vagaries of the Internet, the first
datagram may go by one route, and the second one may take a different path. If the second route
is faster than the first one, the two datagrams may arrive in the opposite order from which they were
sent. It is also possible for a datagram to get duplicated in transit, resulting in the same message
being received twice.
Because of the connectionless nature of datagrams, there is no flow control between the sender
and the recipient. If the sender transmits datagrams faster than the recipient can process them, the
recipient has no way to signal the sender to slow down, and will eventually start to discard packets.
Although a datagram's delivery is not reliable, its contents are. Modern implementations of UDP
provide each datagram with a checksum that ensures that its data portion is not corrupted in transit.

Stream Sockets
The other major paradigm is stream sockets, implemented in the Internet domain as the TCP pro-
tocol. Stream sockets provide sequenced, reliable bidirectional communications via byte-oriented
streams. As depicted in Figure 3.3, stream sockets resemble a telephone conversation. Clients
connect to servers using their address, the two exchange data for a period of time, and then one of
the pair breaks off the connection.

Figure 3.3. Stream sockets provide sequenced, reliable, bidirectional communications

Reading and writing to stream sockets is a lot like reading and writing to a file. There are no arbitrary
size limits or record boundaries, although you can impose a record-oriented structure on the stream
if you like. Because stream sockets are sequenced and reliable, you can write a series of bytes into
a socket secure in the knowledge that they will emerge at the other end in the correct order, provided
that they emerge at all ("reliable" does not mean immune to network errors).
TCP also implements flow control. Unlike UDP, where the danger of filling the data-receiving buffer
is very real, TCP automatically signals the sending host to suspend transmission temporarily when
the reading host is falling behind, and to resume sending data when the reading host is again ready.
This flow control happens behind the scenes and is ordinarily invisible.
If you have ever used the Perl open(FH, " | command ") syntax to open up a pipe to an external
program, you will find that working with stream sockets is not much different. The major visible
difference is that, unlike pipes, stream sockets are bidirectional.
Although it looks and acts like a continuous byte stream, the TCP protocol is actually implemented
on top of a datagram-style service, in this case the low-level IP protocol. IP packets are just as
unreliable as UDP datagrams, so behind the scenes TCP is responsible for keeping track of packet
sequence numbers, acknowledging received packets, and retransmitting lost packets.

Datagram versus Stream Sockets

With all its reliability problems, you might wonder why anyone uses UDP. The answer is that most
client/server programs on the Internet use TCP stream sockets instead. In most cases, TCP is the
right solution for you, too.
There are some circumstances, however, in which UDP might be a better choice. For example, time
servers use UDP datagrams to transmit the time of day to clients who use the information for clock
synchronization. If a datagram disappears in transit, it's neither necessary nor desirable to retransmit
it because by the time it arrives it will no longer be relevant.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Introduction to Berkeley Sockets 53

UDP is also preferred when the interaction between one host and the other is very short. The length
of time to set up and take down a TCP connection is about eightfold greater than the exchange of
a single byte of data via UDP (for details, see [Stevens 1996]). If relatively small amounts of data
are being exchanged, the TCP setup time will dominate performance. Even after a TCP connection
is established, each transmitted byte consumes more bandwidth than UDP because of the additional
overhead for ensuring reliability.
Another common scenario occurs when a host must send the same data to many places; for ex-
ample, it wants to transmit a video stream to multiple viewers. The overhead to set up and manage
a large number of TCP connections can quickly exhaust operating system resources, because a
different socket must be used for each connection. In contrast, sending a series of UDP datagrams
is much more sparing of resources. The same socket can be reused to send datagrams to many
hosts.
Whereas TCP is always a one-to-one connection, UDP also allows one-to-many and many-to-many
transmissions. At one end of the spectrum, you can address a UDP datagram to the "broadcast
address," broadcasting a message to all listening hosts on the local area network. At the other end
of the spectrum, you can target a message to a predefined group of hosts using the "multicast"
facility of modern IP implementations. These advanced features are covered in Chapters 20 and 21.
The Internet's DNS is a common example of a UDP-based service. It is responsible for translating
hostnames into IP addresses, and vice versa, using a loose-knit network of DNS servers. If a client
does not get a response from a DNS server, it just retransmits its request. The overhead of an
occasional lost datagram outweighs the overhead of setting up a new TCP connection for each
request. Other common examples of UDP services include Sun's Network File System (NFS) and
the Trivial File Transfer Protocol (TFTP). The latter is used by diskless workstations during boot in
order to load their operating system over the network. UDP was originally chosen for this purpose
because its implementation is relatively small. Therefore, UDP fit more easily into the limited ROM
space available to workstations at the time the protocol was designed.
The TCP/IP protocol suite is described well in [Stevens 1994, Wright and Stevens 1995, and Stevens
1996], as well as in RFC 1180, A TCP/IP Tutorial .

Socket Addressing
In order for one process to talk to another, each of them has to know the other's address. Each
networking domain has a different concept of what an address is. For the UNIX domain, which can
be used only between two processes on the same host machine, addresses are simply paths on
the host's filesystem, such as /usr/tmp/log. For the Internet domain, each socket address has
three parts: the IP address, the port, and the protocol.

IP Addresses
In the currently deployed version of TCP/IP, IPv4, the IP address is a 32-bit number used to identify
a network interface on the host machine. A series of subnetworks and routing tables enables any
machine on the Internet to send packets to any other machine based on its IP address.
For readability, the four bytes of a machine's IP address are usually spelled out as a series of four
decimal digits separated by dots to create the "dotted quad address" that network administrators
have come to know and love. For example, 143.48.7.1 is the IP address of one of the servers at my
workplace. Expressed as a 32-bit number in the hexadecimal system, this address is 0x8f3071.
Many of Perl's networking calls require you to work with IP addresses in the form of packed binary
strings. IP addresses can be converted manually to binary format and back again using pack()
and unpack() with a template of "C4" (four unsigned characters). For example, here's how to
convert 18.157.0.125 into its packed form and then reverse the process:

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Introduction to Berkeley Sockets 54

($a,$b,$c,$d) = split /\./, '18.157.0.125';

$packed_ip_address = pack 'C4',$a,$b,$c,$d;
($a,$b,$c,$d) = unpack 'C4',$packed_ip_address;
$dotted_ip_address = join '.', $a,$b,$c,$d;

You usually won't have to do this, however, because Perl provides convenient high-level functions
to handle this conversion for you.
Most hosts have two addresses, the "loopback" address 127.0.0.1 (often known by its symbolic
name "localhost") and its public Internet address. The loopback address is associated with a device
that loops transmissions back onto itself, allowing a client on the host to make an outgoing con-
nection to a server running on the same host. Although this sounds a bit pointless, it is a powerful
technique for application development, because it means that you can develop and test software
on the local machine without access to the network.
The public Internet address is associated with the host's network interface card, such as an Ethernet
card. The address is either assigned to the host by the network administrator or, in systems with
dynamic host addressing, by a Boot Protocol (BOOTP) or Dynamic Host Configuration Protocol
(DHCP) server. If a host has multiple network interfaces installed, each one can have a distinct IP
address. It's also possible for a single interface to be configured to use several addresses. Chapter
21 discusses IO::Interface, a third-party Perl module that allows a Perl script to examine and alter
the IP addresses assigned to its interface cards.

Reserved IP Addresses, Subnets, and Netmasks

In order for a packet of information to travel from one location to another across the Internet, it must
hop across a series of physical networks. For example, a packet leaving your desktop computer
must travel across your LAN (local area network) to a modem or router, then across your Internet
service provider's (ISP) regional network, then across a backbone to another ISP's regional network,
and finally to its destination machine.
Network routers keep track of how the networks interconnect, and are responsible for determining
the most efficient route to get a packet from point A to point B. However, if IP addresses were
allocated ad hoc, this task would not be feasible because each router would have to maintain a map
showing the locations of all IP addresses. Instead, IP addresses are allocated in contiguous chunks
for use in organizational and regional networks.
For example, my employer, the Cold Spring Harbor Laboratory (CSHL), owns the block of IP ad-
dresses that range from 143.48.0.0 through 143.48.255.255 (this is a so-called class B address).
When a backbone router sees a packet addressed to an IP address in this range, it needs only to
determine how to get the packet into CSHL's network. It is then the responsibility of CSHL's routers
to get the packet to its destination. In practice, CSHL and other large organizations split their allo-
cated address ranges into several subnets and use routers to interconnect the parts.
A computer that is sending out an IP packet must determine whether the destination machine is
directly reachable (e.g., over the Ethernet) or whether the packet must be directed to a router that
interconnects the local network to more distant locations. The basic decision is whether the packet
is part of the local network or part of a distant network.
To make this decision possible, IP addresses are arbitrarily split into a host part and a network part.
For example, in CSHL's network, the split occurs after the second byte: the network part is 143.48.
and the host part is the rest. So 143.48.0.0 is the first address in CSHL's network, and
143.48.255.255 is the last.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Introduction to Berkeley Sockets 55

To describe where the network/host split occurs for routing purposes, networks use a netmask,
which is a bitmask with 1s in the positions of the network part of the IP address. Like the IP address
itself, the netmask is usually written in dotted-quad form. Continuing with our example, CSHL has
a netmask of 255.255.0.0, which, when written in binary, is
11111111,11111111,00000000,00000000.
Historically, IP networks were divided into three classes on the basis of their netmasks (Table
3.5). Class A networks have a netmask of 255.0.0.0 and approximately 16 million hosts. Class B
networks have a netmask of 255.255.0.0 and some 65,000 hosts, and class C networks use the
netmask 255.255.255.0 and support 254 hosts (as we will see, the first and last host numbers in a
network range are unavailable for use as a normal host address).
Table 3.5. Address Classes and Their Netmasks
Class Netmask Example Address Network Park Host Part
A 255.0.0.0 120.155.32.5 120. 155.32.5
B 255.255.0.0 128.157.32.5 128.157. 32.5
C 255.255.255.0 192.66.12.56 192.66.12. 56

As the Internet has become more crowded, however, networks have had to be split up in more
flexible ways. It's common now to see netmasks that don't end at byte boundaries. For example,
the netmask 255.255.255.128 (binary 11111111,11111111,11111111,10000000) splits the last
byte in half, creating a set of 126-host networks. The modern Internet routes packets based on this
more flexible scheme, called Classless Inter-Domain Routing (CIDR). CIDR uses a concise con-
vention to describe networks in which the network address is followed by a slash and an integer
containing the number of 1s in the mask. For example, CSHL's network is described by the CIDR
address 143.48.0.0/16. CIDR is described in detail in RFCs 1517 through 1520, and in the FAQs
listed in Appendix D.
Figuring out the network and broadcast addresses can be confusing when you work with netmasks
that do not end at byte boundaries. The Net::Netmask module, available on CPAN, provides facilities
for calculating these values in an intuitive way. You'll also find a short module that I wrote, Net::Net-
maskLite, in Appendix A. You might want to peruse this code in order to learn the relationships
among the network address, broadcast address, and netmask.
The first and last addresses in a subnet have special significance and cannot be used as ordinary
host addresses. The first address, sometimes known as the all-zeroes address, is reserved for use
in routing tables to denote the network as a whole. The last address in the range, known as the all-
ones address, is reserved for use as the broadcast address. IP packets sent to this address will be
received by all hosts on the subnet. For example, for the network 192.18.4.x (a class C address or
192.18.4.0/24 in CIDR format), the network address is 192.18.4.0 and the broadcast address is
192.18.4.255. We will discuss broadcasting in detail in Chapter 20.
In addition, several IP address ranges have been set aside for special purposes (Table 3.6). The
class A network 10.x.x.x, the 16 class B networks 172.16.x.x through 172.31.x.x, and the 255 class
C addresses 192.168.0.x through 192.168.255.x are reserved for use as internal networks. An or-
ganization may use any of these networks internally, but must not connect the network directly to
the Internet. The 192.168.x.x networks are used frequently in testing, or placed behind firewall sys-
tems that translate all the internal network addresses into a single public IP address. The network
addresses 224.x.x.x through 239.x.x.x are reserved for multicasting applications (Chapter 21), and
everything above 240.x.x.x is reserved for future expansion.
Table 3.6. Reserved IP Addresses
Address Description
127.0.0.x Loopback interface
10.x.x.x Private class A address

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Introduction to Berkeley Sockets 56

Address Description
172.16.x.x–172.32.x.x Private class B addresses
192.168.0.x–172.168.255.x Private class C addresses

Finally, IP address 127.0.0.x is reserved for use as the loopback network. Anything sent to an ad-
dress in this range is received by the local host.

IPv6
Although there are more than 4 billion possible IPv4 addresses, the presence of several large re-
served ranges and the way the addresses are allocated into subnetworks reduces the effective
number of addresses considerably. This, coupled with the recent explosion in network-connected
devices, means that the Internet is rapidly running out of IP addresses. The crisis has been fore-
stalled for now by various dynamic host-addressing and address-translation techniques that share
a pool of IP addresses among a larger set of hosts. However, the new drive to put toaster ovens,
television set-top boxes, and cell phones on the Internet is again threatening to exhaust the address
space.
This is one of the major justifications for the new version of TCP/IP, known as IPv6, which expands
the IP address space from 4 to 16 bytes. IPv6 is being deployed on the Internet backbones now,
but this change will not immediately affect local area networks, which will continue to use addresses
backwardly compatible with IPv4. Perl has not yet been updated to support IPv6, but will undoubtedly
do so by the time that IPv6 is widely implemented.
More information about IPv6 can be found in [Stevens 1996] and [Hunt 1998 ].

Network Ports
Once a message reaches its destination IP address, there's still the matter of finding the correct
program to deliver it to. It's common for a host to be running multiple network servers, and it would
be impractical, not to say confusing, to deliver the same message to them all. That's where the port
number comes in. The port number part of the socket address is an unsigned 16-bit number ranging
from 1 to 65535. In addition to its IP address, each active socket on a host is identified by a unique
port number; this allows messages to be delivered unambiguously to the correct program. When a
program creates a socket, it may ask the operating system to associate a port with the socket. If the
port is not being used, the operating system will grant this request, and will refuse other programs
access to the port until the port is no longer in use. If the program doesn't specifically request a port,
one will be assigned to it from the pool of unused port numbers.
There are actually two sets of port numbers, one for use by TCP sockets, and the other for use by
UDP-based programs. It is perfectly all right for two programs to be using the same port number
provided that one is using it for TCP and the other for UDP.
Not all port numbers are created equal. The ports in the range 0 through 1023 are reserved for the
use of "well-known" services, which are assigned and maintained by ICANN, the Internet Corpora-
tion for Assigned Names and Numbers. For example, TCP port 80 is reserved for use for the HTTP
used by Web servers, TCP port 25 is used for the SMTP used by e-mail transport agents, and UDP
port 53 is used for the domain name service (DNS). Because these ports are well known, you can
be pretty certain that a Web server running on a remote machine will be listening on port 80. On
UNIX systems, only the root user (i.e., the superuser) is allowed to create a socket using a reserved
port. This is partly to prevent unprivileged users on the system inadvertently running code that will
interfere with the operations of the host's network services.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Introduction to Berkeley Sockets 57

A list of reserved ports and their associated well-known services is given in Appendix C. Most serv-
ices are either TCP- or UDP-based, but some can communicate with both protocols. In the interest
of future compatibility, ICANN usually reserves both the UDP and TCP ports for each service. How-
ever, there are many exceptions to this rule. For example, TCP port 514 is used on UNIX systems
for remote shell (login) services, while UDP port 514 is used for the system logging daemon.
In some versions of UNIX, the high-numbered ports in the range 49152 through 65535 are reserved
by the operating system for use as "ephemeral" ports to be assigned automatically to outgoing TCP/
IP connections when a port number hasn't been explicitly requested. The remaining ports, those in
the range 1024 through 49151, are free for use in your own applications, provided that some other
service has not already claimed them. It is a good idea to check the ports in use on your machine
by using one of the network tools introduced later in this chapter (Network Analysis Tools) before
claiming one.

The sockaddr_in Structure

A socket address is the combination of the host address and the port, packed together in a binary
structure called a sockaddr_in. This corresponds to a C structure of the same name that is used
internally to call the system networking routines. (By analogy, UNIX domain sockets use a packed
structure called a sockaddr_un.) Functions provided by the standard Perl Socket module allow you
to create and manipulate sockaddr_in structures easily:

$packed_address = inet_aton($dotted_quad)
Given an IP address in dotted-quad form, this function packs it into binary form suitable for use by sock-
addr_in(). The function will also operate on symbolic hostnames. If the hostname cannot be looked up, it
returns undef.
$dotted_quad = inet_ntoa($packed_address)
This function takes a packed IP address and converts it into human-readable dotted-quad form. It does not
attempt to translate IP addresses into hostnames. You can achieve this effect by using gethostbyaddr(),
discussed later.
$socket_addr = sockaddr_in($port,$address)
($port,$address) = sockaddr_in($socket_addr)
When called in a scalar context, sockaddr_in() takes a port number and a binary IP address and packs
them together into a socket address, suitable for use by socket(). When called in a list context, sock-
addr_in() does the opposite, translating a socket address into the port and IP address. The IP address must
still be passed through inet_ntoa() to obtain a human-readable string.
$socket_addr = pack_sockaddr_in($port,$address)
($port,$address) = unpack_sockaddr_in($socket_addr)
If you don't like the confusing behavior of sockaddr_in(), you can use these two functions to pack and
unpack socket addresses in a context-insensitive manner.

We'll use several of these functions in the example that follows in the next section.
In some references, you'll see a socket's address referred to as its "name." Don't let this confuse
you. A socket's address and its name are one and the same.

A Simple Network Client

To put this information into context, we'll now write a client for the daytime service. This service,
which runs on many UNIX hosts, listens for incoming connections on TCP port 13. When it sees
such a connection, it outputs a single line of text with the current time and date.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Introduction to Berkeley Sockets 58

The script gets the dotted IP address of the daytime server from the command line. When we run
it on 64.7.3.43, which is the IP address of the wuarchive.wustl.edu software archive, we get
output like this:
% daytime_cli.pl 64.7.3.43
Sat Jan 6 19:11:22 2001

Figure 3.4 shows the code for the daytime client.

Figure 3.4. A daytime client

Lines 1–3: Load modules—We turn on strict type checking with use strict. This will avoid bugs
introduced by mistyped variable names, the automatic interpretation of bare words as strings,
and other common mistakes. We then load the Socket module, which imports several functions
useful for socket programming.
Lines 4–6: Define constants—We now define several constants. DEFAULT_ADDR is the IP ad-
dress of a remote host to contact when an address isn't explicitly provided on the command line.
We use 127.0.0.1, the loopback address.
PORT is the well-known port for the daytime service, port 13, while IPPROTO_TCP is the numeric
protocol for the TCP protocol, needed to construct the socket.
It's inelegant to hard code these constants, and in the case of IPPROTO_TCP may impede port-
ability. The next section shows how to look up these values at run time using symbolic names.
Lines 7–9: Construct the destination address—The next part of the code constructs a destination
address for the socket using the IP address of the daytime host and the port number of the
daytime service. We begin by recovering the dotted-quad address from the command line or, if
no address was specified, we default to the loopback address.
We now have an IP address in string form, but we need to convert it into packed binary form
before passing it to the socket creation function. We do this by using the inet_aton() function,
described earlier.
The last step in constructing the destination address is to create the sockaddr_in structure,
which combines the IP address with the port number. We do this by calling the sock-
addr_in() function with the port number and packed IP address.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Introduction to Berkeley Sockets 59

Line 10: Create the socket—The socket() function creates the communications endpoint. The
function takes four arguments. The first argument, SOCK, is the filehandle name to use for the
socket. Sockets look and act like filehandles, and like them are capitalized by convention. The
remaining arguments are the address family, the socket type, and the protocol number. With the
exception of the protocol, which we have hard-coded, all constants are taken from the socket
module.
This call to socket() creates a stream-style Internet-domain socket that uses the TCP protocol.
Line 11: Connect to the remote host— SOCK corresponds to the local communications endpoint.
We now need to connect it to the remote socket. We do this using the built-in Perl function
connect(), passing it the socket handle and the remote address that we built earlier. If con-
nect() is successful, it will return a true value. Otherwise, we again die with an error message.
Line 12: Read data from remote host and print it—We can now treat SOCK like a read/write
filehandle, either reading data transmitted from the remote host with the read() or <> functions,
or sending data to the host using print(). In this case, we use the angle bracket operator
(<>) to read all lines transmitted by the remote host and immediately echo the data to standard
output.
We discuss the socket() and connect () calls in more detail in Chapter 4.

Network Names and Services

The example in the last section used the remote host's dotted-quad IP address and numeric port
and protocol numbers to construct the socket. Usually, however, you'll want to use symbolic names
instead of numbers. Not only does this make it easier to write the program, but it makes it easier for
end users to use it, allowing them to enter the host address as wuarchive.wustl.edu rather than
128.252.120.8.
The domain name system (DNS) is an Internet-wide database that translates hostnames into dotted
IP addresses and back again. Various local database services translate from service names into
numbers. This section introduces the functions that allow you to translate names into numbers, and
vice versa.

Translating Hostnames into IP Addresses

Perl's gethostbyname() and gethostbyaddr() functions translate a symbolic hostname into
a packed IP address, and vice versa. They are front ends to system library calls of the same name.
Depending on your system's name-resolution setup, the call will consult one or more static text files
such as /etc/hosts, local area network databases such as NIS, or the Internet-wide DNS.

($name,$aliases,$type,$len,$packed_addr) = gethostbyname($name);
$packed_addr = gethostbyname($name);
If the hostname doesn't exist, gethostbyname() returns undef. Otherwise, in a scalar context it returns the
IP address of the host in packed binary form, while in a list context it returns a five-element list.
The first element, $name, is the canonical hostname—the official name—for the requested host. This is fol-
lowed by a list of hostname aliases, the address type and address length, usually AF_INET and 4, and the
address itself in packed form. The aliases field is a space-delimited list of alternative names for the host. If
there are no alternative names, the list is empty.
You can pass the packed address returned by gethostbyname() directly to the socket functions, or translate
it back a human-readable dotted-quad form using inet_ntoa(). If you pass gethostbyname() a dotted-
quad IP address, it will detect that fact and return the packed version of the address, just as inet_aton()
does.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Introduction to Berkeley Sockets 60

$name = gethostbyaddr($packed_addr,$family);
($name,$aliases,$type,$len,$packed_addr) = gethostbyaddr($packed_addr, $family);
The gethostbyaddr() call performs the reverse lookup, taking a packed IP address and returning the host-
name corresponding to it.
gethostbyaddr() takes two arguments, the packed address and the address family (usually AF_INET). In
a scalar context, it returns the hostname of the indicated address. In a list context, it returns a five-element list
consisting of the canonical hostname, a list of aliases, the address type, address length, and packed address.
This is the same list returned by gethostbyname(literal).
If the call is unsuccessful in looking up the address, it returns undef or an empty list.

The inet_aton() function can also translate hostnames into packed IP addresses. In general,
you can replace scalar context calls to gethostbyname() with calls to inet_aton():

Licensed by
$packed_address = inet_aton($host);
$packed_address = gethostbyname($host);

Which of these two functions you use is a matter of taste, but be aware that gethostbyname() is
built into Perl, but inet_aton() is available only after you load the socket module. I prefer

Stjepan Maric
inet_aton() because, unlike gethostbyname(), it isn't sensitive to list context.

Hostname Translation Examples

Using these functions, we can write a simple program to translate a list of hostnames into dotted-

4218908
quad IP addresses (Figure 3.5). Given a file of hostnames stored in hostnames.txt, the script's output
looks like this:

Figure 3.5. Translating hostnames into IP addresses

% ip_trans.pl < hostnames.txt

pesto.cshl.org => 143.48.7.220
w3.org => 18.23.0.20
foo.bar.com => ?
ntp.css.gov => 140.162.3

Figure 3.6 shows a short program for performing the reverse operation, translating a list of dotted-
quad IP addresses into hostnames. For each line of input, the program checks that it looks like a
valid IP address (line 6) and, if so, packs the address using inet_aton(). The packed address
is then passed to gethostbyaddr(), specifying AF_INET for the address family (line 7). If suc-
cessful, the translated hostname is printed out.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Introduction to Berkeley Sockets 61

Figure 3.6. Translating IP addresses into hostnames

Getting Information about Protocols and Services

In the same way that gethostbyname() and gethostbyaddr() interconvert hostnames and IP
addresses, Perl provides functions to translate the symbolic names of protocols and services into
protocol and port numbers.

$number = getprotobyname($protocol)
($name,$aliases,$number) = getprotobyname($protocol)
getprotobyname() takes a symbolic protocol name, such as "udp", and converts it into its corresponding
numeric value. In a scalar context, just the protocol number is returned. In a list context, the function returns
the protocol name, a list of aliases, and the number. Multiple aliases are separated by spaces. If the named
protocol is unknown, the function returns undef or an empty list.
$name = getprotobynumber($protocol_number)
($name,$aliases,$number) = getprotobyname($protocol_number)
The rarely used getprotobynumber() function reverses the previous operation, translating a protocol num-
ber into a protocol name. In a scalar context, it returns the protocol number's name as a string. In a list context,
it returns the same list as getprotobyname(). If passed an invalid protocol number, this function returns
undef or an empty list.
$port = getservbyname($service,$protocol)
($name,$aliases,$port,$protocol) = getservbyname($service,$protocol)
The getservbyname() function converts a symbolic service name, such as "echo," into a port number suit-
able for passing to sockaddr_in(). The function takes two arguments corresponding to the service name
and the desired protocol. The reason for the additional protocol argument is that some services, echo included,
come in both UDP and TCP versions, and there's no guarantee that the two versions of the service use the
same port number, although this is almost always the case.
In a scalar context, getservbyname() returns the port number of the service, or undef if unknown. In a list
context, the function returns a four-element list consisting of the canonical name for the service, a space-
delimited list of aliases, if any, the port number, and the protocol number. If the service is unknown, the function
returns an empty list.
$name = getservbyport($port,$protocol)
($name,$aliases,$port,$protocol) = getservbyport($port,$protocol)
The getservbyport() function reverses the previous operation by translating a port number into the cor-
responding service name. Its behavior in scalar and list contexts is exactly the same as getservbyname().

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Introduction to Berkeley Sockets 62

Daytime Client, Revisited

We now have the tools to eliminate the hard-coded port and protocol numbers from the daytime
client that we developed earlier. In addition, we can make the program more user friendly by ac-
cepting the name of the daytime host as either a DNS name or a dotted-IP address. Figure 3.7 gives
the code for the revised client. The differences from the earlier version are as follows:

Figure 3.7. Daytime client, using symbolic hostnames and service names

Line 5: Look up IP address of the daytime host—We call gethostbyname() to translate the
hostname given on the command line into a packed IP address. If no command-line argument
is given, we default to using the loopback address. If the user provided a dotted-IP address
instead of a hostname, then gethostbyname() simply packs it in the manner of
inet_aton(). If gethostbyname() fails because the provided hostname is invalid, it returns
undef and we exit with an error message.
Line 6: Look up the TCP protocol number—We call getprotobyname() to retrieve the value
of the TCP protocol number, rather than hard coding it as before.
Line 7: Look up the daytime service port number—Instead of hard coding the port number for
the daytime service, we call getservbyname() to look it up in the system service database. If
for some reason this doesn't work, we exit with an error message.
Lines 8–11: Connect and read data—The remainder of the program is as before, with the ex-
ception that we use the TCP protocol number retrieved from getprotobyname() in the call to
socket().
You can test the script on the host http://www.modperl. com [http://www.modperl.com]

Other Sources of Network Information

In addition to gethostbyname() and its brethren, specialized add-on modules offer direct access
to a number of other common databases of network information. I won't cover them in detail here,
but you should know that they exist if you need them. All the modules can be obtained from CPAN.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Introduction to Berkeley Sockets 63

Net::DNS
This module offers you much greater control over how hostnames are resolved using the domain
name system. In addition to the functionality offered by gethostbyname() and gethost-
byaddr(), Net::DNS allows you to fetch and iterate over all hosts in a domain; get the e-mail
address of the network administrator responsible for the domain; and look up the machine respon-
sible for accepting e-mail for a domain (the "mail exchanger," or MX).

Net::NIS
Many traditional UNIX networks use Sun's Network Information System (NIS) to distribute such
things as hostnames, IP addresses, usernames, passwords, automount tables, and cryptographic
keys. This allows a user to have the same login username, password, and home directories on all
the machines on the network. Perl's built-in network information access functions, such as
gethostbyname() and getpwnam(), provide transparent access to much, but not all, of NIS's
functionality. The Net::NIS module provides access to more esoteric information, such as automount
tables.

Net::LDAP
NIS is slowly being displaced by the Lightweight Directory Access Protocol (LDAP), which provides
greater flexibility and scalability. In addition to the types of information stored by NIS, LDAP is often
used to store users' e-mail addresses, telephone numbers, and other "white pages"–type informa-
tion. Net::LDAP gives you access to these databases.

Win32::NetResource
On Windows networks, the NT domain controller provides directory information on hosts, printers,
shared file systems, and other network information. Win32::NetResources provides functions for
reading and writing this information.

Network Analysis Tools

This section lists some of the basic tools for analyzing networks and diagnosing network-related
problems. Some of these tools come preinstalled on many systems; others must be downloaded
and installed. For more information on using these tools, please see the comprehensive discussion
of network configuration and troubleshooting in [Hunt 1998].

ping
The ping utility, available as a preinstalled utility on all UNIX and Windows machines, is the single
most useful network utility. It sends a series of ICMP "ping" messages to the remote IP address of
your choice, and reports the number of responses the remote machine returns.
ping can be used to test if a remote machine is up and reachable across the network. It can also be
used to test network conditions by looking at the length of time between the outgoing ping and the
incoming response, and the number of pings that have no response (due to either loss of the out-
going message or the incoming response).
For example, this is how ping can be used to test connectivity to the machine at IP address
216.32.74.55 (which happens to be www.yahoo.com):
% ping 216.32.74.55
PING 216.32.74.55: 56 data bytes
64 bytes from 216.32.74.55: icmp_seq=0 ttl=245 time=41.1 ms
64 bytes from 216.32.74.55: icmp_seq=1 ttl=245 time=16.4 ms
64 bytes from 216.32.74.55: icmp_seq=2 ttl=245 time=16.3 ms

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Introduction to Berkeley Sockets 64

^C

--- 216.32.74.55 ping statistics ---

4 packets transmitted, 3 packets received, 25% packet loss
round-trip min/avg/max = 16.3/24.6/41.1 ms

This session shows good connectivity. The average response time is a snappy 24 ms, and no
packets were lost. You can also give ping a DNS name, in which case it will attempt to resolve the
name before pinging the host.
One thing to watch for is that some firewall systems are configured to block ping. In this case, the
destination machine may be unpingable, although you can reach it via telnet or other means.
There are many variants of ping, each with a different overlapping set of features.

nslookup
The nslookup utility, available on most UNIX systems, can be used to test and verify the DNS. It
can be used interactively or as a one-shot command-line tool. To use it from the command line, call
it with the DNS name of the host or domain you wish to look up. It will perform the DNS search, and
return IP addresses and other DNS information corresponding to the name. For example:
% nslookup www.yahoo.com
Server: presto.lsjs.org
Address: 64.7.3.44
Non-authoritative answer:
Name: www.yahoo.akadns.net
Addresses: 204.71.200.67, 204.71.200.68, 204.71.202.160, 204.71.200.74, 204.71.200.75
Aliases: www.yahoo.com

This tells us that the host www.yahoo.com has a canonical name of www.yahoo.akadns.net,
and has five IP addresses assigned to it. This is typical of a heavily loaded Web server, where
multiple physical machines balance incoming requests by servicing them in a round-robin fashion.

traceroute
While ping tells you only whether a packet can get from A to B, the traceroute program
displays the exact path a network packet takes to get there. Call it with the IP address of the des-
tination. Each line of the response gives the address of a router along the way. For example:
% traceroute www.yahoo.com
traceroute to www.yahoo.akadns.net (216.32.74.52), 30 hops max, 40 byte packets
1 gw.lsjs.org (192.168.3.1) 2.52 ms 8.78 ms 4.85 ms
2 64.7.3.46 (64.7.3.46) 9.7 ms 9.656 ms 3.415 ms
3 mgp-gw.nyc.megapath.net (64.7.2.1) 19.118 ms 23.619 ms 16.601 ms
4 216.35.48.242 (216.35.48.242) 10.532 ms 10.515 ms 11.368 ms
5 dcr03-g2-0.jrcy01.exodus.net (216.32.222.121) 9.068 ms 9.369 ms 9.08 ms
6 bbr02-g4-0.jrcy01.exodus.net (209.67.45.126) 9.522 ms 11.091 ms 10.212 ms
7 bbr01-p5-0.stng01.exodus.net (209.185.9.98) 15.516 ms 15.118 ms 15.227 ms
8 dcr03-g9-0.stng01.exodus.net (216.33.96.145) 15.497 ms 15.448 ms 15.462 ms
9 csr22-ve242.stng01.exodus.net (216.33.98.19) 16.044 ms 15.724 ms 16.454 ms
10 216.35.210.126 (216.35.210.126) 15.954 ms 15.537 ms 15.644 ms
11 www3.dcx.yahoo.com (216.32.74.52) 15.644 ms 15.582 ms 15.577 ms

traceroute can be invaluable for locating a network outage when a host can no longer be pinged.
The listing will stop without reaching the desired destination, and the last item on the list indicates
the point beyond which the breakage is occurring.
As with ping, some firewalls can interfere with traceroute. Traceroute is preinstalled on most UNIX
systems.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Introduction to Berkeley Sockets 65

netstat
The netstat utility, preinstalled on UNIX and Windows NT/2000 systems, prints a snapshot of all
active network services and connections. For example, running netstat on an active Web and FTP
server produces the following display (abbreviated for space):
% netstat -t
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 brie.cshl.org:www writer.loci.wisc.e:1402 ESTABLISHED
tcp 0 0 brie.cshl.org:www 157-238-71-168.il.:1215 FIN_WAIT2
tcp 0 0 brie.cshl.org:www 157-238-71-168.il.:1214 FIN_WAIT2
tcp 0 0 brie.cshl.org:www 157-238-71-168.il.:1213 TIME_WAIT
tcp 0 0 brie.cshl.org:6010 brie.cshl.org:2225 ESTABLISHED
tcp 0 0 brie.cshl.org:2225 brie.cshl.org:6010 ESTABLISHED
tcp 0 2660 brie.cshl.org:ssh presto.lsjs.org:64080 ESTABLISHED
tcp 0 0 brie.cshl.org:www 206.169.243.7:1724 TIME_WAIT
tcp 0 20 brie.cshl.org:ftp usr25-wok.cableine:2173 ESTABLISHED
tcp 0 891 brie.cshl.org:www usr25-wok.cableine:2117 FIN_WAIT1
tcp 0 80 brie.cshl.org:ftp soa.sanger.ac.uk:49596 CLOSE

The -t argument restricts the display to TCP connections. The Recv-Q and Send-Q columns
show the number of bytes in the sockets' read and write buffers, respectively. The Local and
Foreign Address columns show the name and port numbers of the local and remote peers,
respectively, and the State column shows the current state of the connection.
netstat can also be used to show services that are waiting for incoming connections, as well as
UDP and UNIX-domain sockets. The netstat syntax on Windows systems is slightly different. To
get a list of TCP connections similar to the one shown above, use the command netstat -p tcp.

tcpdump
The tcpdump utility, available preinstalled on many versions of UNIX, is a packet sniffer. It can be
used to dump the contents of every packet passing by your network card, including those not di-
rected to your machine. It features a powerful filter language that can be used to detect and display
just those packets you are interested in, such as those using a particular protocol or directed toward
a specific port.

MacTCP Watcher
MacTCP Watcher for the Macintosh combines the functionality of ping, dnslookup, and netstat
into one user-friendly application. It can be found by searching the large shareware collection located
at http://www.shareware.com. [http://www.shareware.com]

scanner.exe
For Windows 98/NT/2000 developers, the small scanner.exe utility, also available from http://
www.shareware.com , combines the functionality of ping and dnslookup with the ability to scan
a remote host for open ports. It can be used to determine the services a remote host provides.

net-toolbox.exe
This is a comprehensive set of Windows network utilities that include ping, dnslookup, tcpdump,
and netstat functionality. It can be found by anonymous FTP to gatekeeper.dec.com in the di-
rectory /pub/micro/pc/winsite/win95/netutil/.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Introduction to Berkeley Sockets 66

Summary
A socket is an endpoint for communications. In Perl, a socket looks and acts much like a filehandle.
There are several species of socket distinguished by their address families, type, and communica-
tions protocol. The most frequently used sockets belong to the AF_INET (Internet) address families,
and use either the stream-oriented TCP protocol or the datagram-oriented UDP protocol.
UNIX-domain sockets use addresses based on local filenames, whereas Internet-domain sockets
use a combination of IP address and port number. Addresses must be packed into binary form
before passing them to any of Perl's built-in network functions.
Perl provides a complete set of functions for interconverting the numeric and symbolic forms of host
addresses, protocols, and services. Using the symbolic names makes programs easier to use and
maintain, and promotes portability.
We closed this chapter with a brief list of the utilities that are commonly used for detecting and
diagnosing network configuration problems.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 67

Chapter 4. The TCP Protocol

In this chapter we look at TCP, a reliable, connection-oriented byte-stream protocol. These features
make working with TCP sockets similar to working with familiar filehandles and pipes. After opening
a TCP socket, you can send data through it using print() or syswrite(), and read from it using
<>, read(), or sysread().

A TCP Echo Client

We'll start by developing a small TCP client that is more sophisticated than the examples we looked
at in the previous chapter. By and large, clients are responsible for actively initiating their connection
to a remote service. We have already seen the outlines of this process in Chapter 3. To review, a
TCP client takes the following steps:
1. Call socket() to create a socket—Using the socket() function, the client creates a stream-
type socket in the INET (Internet) domain using the TCP protocol.
2. Call connect() to connect to the peer—Using connect(), the client constructs the desired
destination address and connects the socket to it.
3. Perform I/O on the socket—The client calls Perl's various input and output operations to com-
municate over the socket.
4. Close the socket—When the client is done with I/O, it can close the socket with close().
Our example application is a simple client for the TCP "echo" service. This service, run by default
on many UNIX hosts, is simple. It waits for an incoming connection, accepts it, and then echoes
every byte that it receives, rather like an annoying child. This continues until the client closes the
connection.
If you want to try the example script and have no echo server of your own handy, there is an echo
server running on wuarchive.wustl.edu, a large public FTP site.
Figure 4.1 shows tcp_echo_cli1.pl. We'll step through the code first, and then explain what we did
and why we did it.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 The TCP Protocol 68

Figure 4.1. A TCP echo client

Lines 1–6: Load modules—We turn on strict syntax checking, and load the Socket and IO::Han-
dle modules. We use Socket for its socket-related constants, and IO::Handle for its auto-
flush() method.
Line 7: Declare globals—We create two global variables for keeping track of the number of bytes
we send and receive.
Lines 8–9: Process command-line arguments—We read the destination host and port number
from the command line. If the host isn't provided, we default to "localhost". If the port number
isn't provided, we use getservbyname() to look up the port number for the echo service.
Lines 10–11: Look up protocol and create packed IP address—We use getprotobyname()
to find the TCP protocol number for use with socket(). We then use inet_aton() to turn
the hostname into a packed IP address for use with sockaddr_in().
Line 12: Create the socket—We call socket() to create a socket filehandle named SOCK in
the same way that we did in Figure 3.4 (in Chapter 3). We pass arguments specifying the
AF_INET Internet address family, a stream-based socket type of SOCK_STREAM, and the TCP
protocol number looked up earlier.
Lines 13–14: Create destination address and connect socket to it—We use sockaddr_in()
to create a packed address containing the destination's IP address and port number. This ad-
dress is now used as the destination for a connect() call. If successful, connect() returns
true. Otherwise, we die with an error message.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 The TCP Protocol 69

Line 15: Turn on autoflush mode for the socket—We want data written to the socket to be flushed
immediately, rather than hang around in a local buffer. We call the socket's autoflush()
method to turn on autoflush mode. This method is available courtesy of IO::Handle.
Lines 16–22: Enter main loop—We now enter a small loop. Each time through the loop we read
a line of text from standard input, and print it verbatim to SOCK, sending it to the remote host.
We then use the <> operator to read a line of response from the server, and print it to standard
output.
Each time through the loop we tally the number of bytes sent and received. This continues until
we reach EOF on standard input.
Lines 23–24: Close the socket and print statistics—After the loop is done, we close the socket
and print to standard error our summary statistics on the number of bytes sent and received.
A session with tcp_echo_cli1.pl looks like this:
% tcp_echo_cli1.pl www.modperl.com
How now brown cow?
How now brown cow?
There's an echo in here.
There's an echo in here.
Yo-de-lay-ee-oo!
Yo-de-lay-ee-oo!
^D
bytes_sent = 61, bytes_received = 61

The ^D on the second-to-last line of the transcript shows where I got tired of this game and pressed
the end-of-input key. (On Windows systems, this would be ^Z.)

Socket Functions Related to Outgoing Connections

We'll now look at the functions related to creating sockets and establishing outgoing TCP connec-
tions in more detail.

$boolean = socket (SOCKET,$domain,$type,$protocol)

Given a filehandle name, a domain, a type, and a protocol, socket() creates a new socket and associates
it with the named filehandle. On success, the function returns a true value. On error, socket() returns
undef and leaves the error message in $!.
The domain, type, and protocol are all small integers. Appropriate values for the first two are constants defined
in the Socket module, but the protocol value must be determined at run time by calling getprotoby-
name(). For creating TCP sockets, the idiom is typically
socket(SOCK,AF_INET,SOCK_STREAM, scalar getprotobyname("tcp"))
Here we force getprotobyname() into a scalar context in order to return a single function result containing
the protocol number.
$boolean = connect (SOCK,$dest_addr)
The connect() function attempts to connect a connection-oriented socket to the indicated destination ad-
dress. The socket must already have been created with socket(), and the packed destination address cre-
ated by sockaddr_in() or equivalent. The system will automatically choose an ephemeral port to use for
the socket's local address.
On success, connect() returns a true value. Otherwise, connect() returns false and $! is set to the system
error code explaining the problem. It is illegal for a connection-oriented socket to call connect() more than
once; if another call is attempted, it results in an EISCONN ("Transport endpoint is already connected") error.
$boolean = close (SOCK)

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 The TCP Protocol 70

The close() call works with sockets just like it does with ordinary filehandles. The socket is closed for busi-
ness. Once closed, the socket can no longer be read from or written to. On success, close() returns a true
value. Otherwise, it returns undef and leaves an error message in $!.
The effect of close() on the other end of the connection is similar to the effect of closing a pipe. After the
socket is closed, any further reads on the socket at the other end return an end of file (EOF). Any further writes
result in a PIPE exception.
$boolean = shutdown (SOCK, $how)
shutdown() is a more precise version of close() that allows you to control which part of the bidirectional
connection to shut down. The first argument is a connected socket. The second argument, $how, is a small
integer that indicates which direction to shut down. As summarized in Table 4.1, a $how of 0 closes the socket
for further reads, a value of 1 closes the socket for writes, and a value of 2 closes the socket for both reading
and writing (like close()). A nonzero return value indicates that the shutdown was successful.

Table 4.1. Shutdown() Values

Licensed by
Value of HOW Description
0 Closes socket for reading
1 Closes socket for writing
2 Closes socket completely

Stjepan Maric
In addition to its ability to half-close a socket, shutdown() has one other advantage over
close(). If the process has called fork() at any point, there may be copies of the socket file-
handle in the original process's children. A conventional close() on any of the copies does not

4218908
actually close the socket until all other copies are closed as well (filehandles have the same be-
havior). In consequence, the client at the other end of the connection won't receive an EOF until
both the parent and its child process(es) have closed their copies. In contrast, shutdown() closes
all copies of the socket, sending the EOF immediately. We'll take advantage of this feature several
times in the course of this book.

A TCP Echo Server

Now we'll look at a simple TCP server. In contrast to a TCP client, a server does not usually call
connect(). Instead, a TCP server follows the following outline:
1. Create the socket. This is the same as the corresponding step in a client.
2. Bind the socket to a local address. A client program can let the operating system choose an
IP address and port number to use when it calls connect(), but a server must have a well-
known address so that clients can rendezvous with it. For this reason it must explicitly asso-
ciate the socket with a local IP address and port number, a process known as binding. The
function that does this is called bind().
3. Mark the socket as listening. The server calls the listen() function to warn the operating
system that the socket will be used to accept incoming connections. This function also speci-
fies the number of incoming connections that can be queued up while waiting for the server
to accept them.
A socket that is marked in this way as ready to accept incoming connections is called a listening
socket.
4. Accept incoming connections. The server now calls the accept() function, usually in a loop.
Each time accept() is called, it waits for an incoming connection, and then returns a new
connected socket attached to the peer (Figure 4.2). The listening socket is unaffected by this
operation.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 The TCP Protocol 71

Figure 4.2. When accept() receives an incoming connection, it returns a new socket connected to the client
5. Perform I/O on the connected socket. The server uses the connected socket to talk to the
peer. When the server is done, it closes the connected socket.
6. Accept more connections. —Using the listening socket, the server can accept() as many
connections as it likes. When it is done, it will close() the listening socket and exit.
Our example server is named tcp_echo_serv.pl. This server is a slightly warped version of the
standard echo server. It echoes everything sent to it, but rather than send it back verbatim, it re-
verses each line right to left (but preserves the newline at the end). So if one sends it "Hello world!,"
the line echoed back will be "!dlrow olleH." (There's no reason to do this except that it adds some
spice to an otherwise boring example.)
This server can be used by the client of Figure 4.1, or with the standard Telnet program. Figure
4.3 lists the server code.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 The TCP Protocol 72

Figure 4.3. tcp_echo_serv1.pl provides a TCP echo service

Lines 1–9: Load modules, initialize constants and variables—We start out as in the client by
bringing in the Socket and IO::Handle modules. We also define a private echo port of 2007 that
won't conflict with any existing echo server. We set up the $port and $protocol variables as
before (lines 8–9) and initialize the counters.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 The TCP Protocol 73

Lines 10–13: Install INT interrupt handler—There has to be a way to interrupt the server, so we
install a signal handler for the INT (interrupt) signal, which is sent from the terminal when the
user presses ^C. This handler simply prints out the accumulated byte counters' statistics and
exits.
Line 14: Create the socket—Using arguments identical to those used by the TCP client in Figure
4.1, we call socket() to create a stream TCP socket.
Line 15: Set the socket's SO_REUSEADDR option—The next step is to set the socket's
SO_REUSEADDR option to true by calling setsockopt(). This option is commonly used to allow
us to kill and restart the server immediately. Otherwise, there are conditions under which the
system will not allow us to rebind the local address until old connections have timed out.
Lines 16–17: Bind the socket to a local address—We now call bind() to assign a local address
to a socket. We create a local address using sockaddr_in(), passing it our private echo port
for the port, and INADDR_ANY as the IP address. INADDR_ANY acts as a wildcard. It allows the
operating system to accept connections on any of the host's IP addresses (including the loop-
back address and any network interface card addresses it might have).
Line 18: Call listen() to make socket ready to accept incoming connections—We call lis-
ten() to alert the operating system that the socket will be used for incoming connections.
The listen() function takes two arguments. The first is the socket, and the second is an
integer indicating the number of incoming connections that can be queued to wait for processing.
It's common for multiple clients to try to connect at roughly the same time; this argument deter-
mines how large the backlog of pending connections can get. In this case, we use a constant
defined in the Socket module, SOMAXCONN, to take the maximum number of queued connections
that the system allows.
Lines 19–34: Enter main loop—The bulk of the code is the server's main loop, in which it waits
for, and services, incoming connections.
Line 21: accept() an incoming connection—Each time through the loop we call accept(),
using the name of the listening socket as the second argument, and a name for the new socket
(SESSION) as the first. (Yes, the order of arguments is a little odd.) If the call to accept() is
successful, it returns the packed address of the remote socket as its function result, and returzns
the connected socket in SESSION.
Lines 22–23: Unpack client's address —We call sockaddr_in() in a list context to unpack the
client address returned by accept() into its port and IP address components, and print the
address to standard error. In a real application, we might write this information to a time-stamped
log file.
Lines 24–33: Handle the connection —This section handles communications with the client us-
ing the connected socket. We first put the SESSION socket into autoflush mode to prevent buf-
fering problems. We now read one line at a time from the socket using the <> operator, reverse
the text of the line, and send it back to the client using print(>).
This continues until <SESSION> returns undef, which indicates that the peer has closed its end
of the connection. We close the SESSION socket, print a status message, and go back to
accept() to wait for the next incoming connection.
Line 35: Clean up —After the main loop is done we close the listening socket. This part of the
code is never reached because the server is designed to be terminated by the interrupt key.
When we run the example server from the command line, it prints out the "waiting for incoming
connections" message and then pauses until it receives an incoming connection. In the session
shown here, there are two connections, one from a local client at the 127.0.0.1 loopback address,
and the other from a client at address 192.168.3.2. After interrupting the server, we see the statistics
printed out by the INT handler.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 The TCP Protocol 74

% tcp_echo_serv1.p1
waiting for incoming connections on port 2007...
Connection from [127.0.0.1,2865]
Connection from [127.0.0.1,2865] finished
Connection from [192.168.3.2,2901]
Connection from [192.168.3.2,2901] finished
bytes_sent = 26, bytes_received = 26

The INT handler in this server violates the recommendation from Chapter 2 that signal handlers not
do any I/O. In addition, by calling exit() from within the handler, it risks raising a fatal exception
on Windows machines as they shut down. We will see a more graceful way of shutting down a server
in Chapter 10.

Socket Functions Related to Incoming Connections

Three new functions are related to the need of servers to handle incoming connections.

$boolean = bind (SOCK,$my_addr)

Bind a local address to a socket, returning a true value if successful, false otherwise. The socket must already
have been created with socket(), and the packed address created by sockaddr_in() or equivalent. The
port part of the address can be any unused port on the system. The IP address part may be the address of
one of the host's network interfaces, the loopback address, or the wildcard INADDR_ANY.
On UNIX systems, it requires superuser (root) privileges to bind to the reserved ports below 1024. Such an
attempt will return undef and set $! to an error of EACCES ("Permission denied").
The bind() function is usually called in servers in order to associate a newly created socket with a well-known
port; however, a client can call it as well if it wishes to specify its local port and/or network interface.
$boolean = listen (SOCK,$max_queue)
The listen() function tells the operating system that a socket will be used to accept incoming connections.
The call's two arguments are a socket filehandle, which must already have been created with socket(), and
an integer value indicating the number of incoming connections that can be queued.
The maximum size of the queue is system-specific. If you specify a higher value than the system allows,
listen() silently truncates it to its maximum value. The Socket module exports the constant SOMAXCONN to
determine this maximum value.
If successful, listen() returns a true value and marks SOCK as listening. Otherwise it returns undef and
sets $! to the appropriate error.
$remote_addr = accept (CONNECTED_SOCKET,LISTEN_SOCKET)
Once a socket has been marked listening, call accept() to accept incoming connections. The accept()
function takes two arguments, CONNECTED_SOCKET, the name of a filehandle to receive the newly connected
socket, and LISTEN_SOCKET, the name of the listening socket. If successful, the packed address of the remote
host will be returned as the function result, and CONNECTED_SOCKET will be associated with the incoming
connection.
Following accept(), you will use CONNECTED_SOCKET to communicate with the peer. You do not need to
create CONNECTED_SOCKET beforehand. In case you're confused by this, think of accept() as a special form
of open() in which LISTEN_SOCKET replaces the file name.
If no incoming connection is waiting to be accepted, accept() will block until there is one. If multiple clients
connect faster than your script calls accept(), they will be queued to the limit specified in the listen() call.
The accept() function returns undef if any of a number of error conditions occur, and sets $! to an appro-
priate error message.
$my_addr = getsockname (SOCK)
$remote_addr = getpeername (SOCK)
Should you wish to recover the local or remote address associated with a socket, you can do so with get-
sockname() or getpeername().

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 The TCP Protocol 75

The getsockname() function returns the packed binary address of the socket at the local side, or undef if
the socket is unbound. The getpeername() function behaves in the same way, but returns the address of
the socket at the remote side, or undef if the socket is unconnected.
In either case, the returned address must be unpacked with sockaddr_in(), as illustrated in this short ex-
ample:
if ($remote_addr = getpeername(SOCK)) {
my ($port,$ip) = sockaddr_in($remote_addr);
my $host = gethostbyaddr($ip,AF_INET);
print "Socket is connected to $host at port $port\n";
}

Limitations of tcp_echo_serv1.pl
Although tcp_echo_serv1.pl works as written, it has a number of drawbacks that are addressed in
later chapters. The drawbacks include the following:
1. No support for multiple incoming connections. This is the biggest problem. The server can
accept only one incoming connection at a time. During the period that it is busy servicing an
existing connection, other requests will be queued up until the current connection terminates
and the main loop again calls accept(). If the number of queued clients exceeds the value
specified by listen(), new incoming connections will be rejected.
To avoid this problem, the server would have to perform some concurrent processing with
threads or processes, or cleverly multiplex its input/output operations. These techniques are
discussed in Part III of this book.
2. Server remains in foreground. After it is launched, the server remains in the foreground, where
any signal from the keyboard (such as a ^C) can interrupt its operations. Long-running servers
will want to dissociate themselves from the keyboard and put themselves in the background.
Techniques for doing this are described in Chapter 10, Forking Servers and the inetd Daemon.
3. Server logging is primitive. The server logs status information to the standard error output
stream. However, a robust server will run as a background process and shouldn't have any
standard error to write to. The server should append log entries to a file or use the operating
system's own logging facilities. Logging techniques are described in Chapter 16, Bulletproof-
ing Servers.

Adjusting Socket Options

Sockets have options that control various aspects of their operation. Among the things you can
adjust are the sizes of the buffers used to send and receive data, the values of send and receive
timeouts, and whether the socket can be used to receive broadcast transmissions.
The default options are fine for most purposes; however, occasionally you may want to adjust some
options to optimize your application, or to enable optional features of the TCP/IP protocol. The most
commonly used option is SO_REUSEADDR, which is frequently activated in server applications.
Socket options can be examined or changed with the Perl built-in functions getsockopt() and
setsockopt().

$value = getsockopt (SOCK,$level,$option_name);

$boolean = setsockopt (SOCK,$level,$option_name,$option_value);

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 The TCP Protocol 76

The getsockopt() and setsockopt() functions allow you to examine and change a socket's options. The
first argument is the filehandle for a previously created socket. The second argument, $level, indicates the
level of the networking stack you wish to operate on. You will usually use the constant SO_SOCKET, indicating
that you are operating on the socket itself. However, a getsockopt() and setsockopt() are occasionally
used to adjust options in the TCP or UDP protocols, in which case you use the protocol number returned by
getprotobyname().
The third argument, $option_name, is an integer value selected from a large list of possible constants. The
last argument, $option_value, is the value to set the option to, or it returns undef if inapplicable. On suc-
cess, getsockopt() returns the value of the requested option, or undef if the call failed. On success,
setsockopt() returns a true value if the option was successfully set; otherwise, it returns undef.

The value of an option is often a Boolean flag indicating whether the option should be turned on and
off. In this case, no special code is needed to set or examine the value. For example, here is how
to set the value of SO_BROADCAST to true (broadcasting is discussed in Chapter 20):
setsockopt(SOCK,SO_SOCKET,SO_BROADCAST,1);

And here is how to retrieve the current value of the flag:

my $reuse = getsockopt(SOCK,SO_SOCKET,SO_BROADCAST);

However, a few options act on integers or more esoteric data types, such as C timeval structures.
In this case, you must pack the values into binary form before passing them to setsockopt() and
unpack them after calling getsockopt(). To illustrate, here is the way to recover the maximum
size of the buffer that a socket uses to store outgoing data. The SO_SNDBUF option acts on a packed
integer (the "I" format):
$send_buffer_size = unpack("I",getsockopt($sock,SOL_SOCKET,SO_SNDBUF));

Common Socket Options

Table 4.2 lists the common socket options used in network programming. These constants are
imported by default when you load the Socket module.
SO_REUSEADDR allows a TCP socket to rebind to a local address that is in use. It takes a Boolean
argument indicating whether to activate address reuse. See the section The SO_REUSEADDR
Socket Option later in this chapter.
SO_KEEPALIVE, when set to a true value, tells a connected socket to send messages to the peer
intermittently. If the remote host fails to respond to a message, your process will receive a PIPE
signal the next time it tries to write to the socket. The interval of the keepalive messages cannot be
changed in a portable way and varies from OS to OS (it is 45 seconds on my Linux system).
SO_LINGER controls what happens when you try to close a TCP socket that still has unsent queued
data. Normally, close() returns immediately, and the operating system continues to try to send
the unsent data in the background. If you prefer, you can make the close() call block until all the
data has been sent by setting SO_LINGER, allowing you to inspect the return value returned by
close() for successful completion.
Unlike other socket options, SO_LINGER operates on a packed data type known as the linger struc-
ture. The linger structure consists of two integers: a flag indicating whether SO_LINGER should be
active, and a timeout value indicating the maximum number of seconds that close() should linger
before returning. The linger structure should be packed and unpacked using the II format:
$linger = pack("II",$flag,$timeout);

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 The TCP Protocol 77

For example, to make a socket linger for up to 120 seconds you would call:
setsockopt(SOCK,SOL_SOCKET, SO_LINGER,pack("II",1,120))
or die "Can't set SO_LINGER: $!";

SO_BROADCAST is valid for UDP sockets only. If set to a true value, it allows send() to be used
to send packets to the broadcast address for delivery to all hosts on the local subnet. We discuss
broadcasting in Chapter 20.
The SO_OOBINLINE flag controls how out-of-band information is handled. This feature allows the
peer to be alerted of the presence of high-priority data. We describe how this works in Chapter 17.
SO-SNDLOWAT and SO_RCVLOWAT set the low-water marks for the output and input buffers,
respectively. The significance of these options is discussed in Chapter 13, Nonblocking I/O. These
options are integers, and must be packed and unpacked using the I pack format.
SO_TYPE is a read-only option. It returns the type of the socket, such as SOCK_STREAM. You will
need to unpack this value with the I format before using it. The IO::Socket sockopt() method
discussed in Chapter 5 does the conversion automatically.
Table 4.2. Common Socket Options
Option Description
SO_REUSEADDR Enable reuse of the local address.
SO_KEEPALIVE Enable the transmission of periodic "keepalive" messages.
SO_LINGER Linger on close if data present.
SO_BROADCAST Allow socket to send messages to the broadcast address.
SO_OOBINLINE
Keep out-of-band data inline.
SO_SNDLOWAT Get or set the size of the output buffer "low water mark."
SO_RECVLOWAT Get or set the size of the input buffer "low water mark."
SO_TYPE Get the socket type (read only).
SO_ERROR Get and clear the last error on the socket (read only).

Last, the read-only SO_ERROR option returns the error code, if any, for the last operation on the
socket. It is used for certain asynchronous operations, such as nonblocking connects (Chapter
13). The error is cleared after it is read. As before, users of getsockopt() need to unpack the
value with the I format before using it, but this is taken care of automatically by IO::Socket.

The SO_REUSEADDR Socket Option

Many developers will want to activate the SO_REUSEADDR flag in server applications. This flags
allows a server to rebind to an address that is already in use, allowing the server to restart imme-
diately after it has crashed or been killed. Without this option, the bind() call will fail until all old
connections have timed out, a process that may take several minutes.
The idiom is to insert the following line of code after the call to socket() and before the call to
bind():
setsockopt(SOCK,SOL_SOCKET,SO_REUSEADDR,1) o r die "setsockopt: $!";

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 The TCP Protocol 78

The downside of setting SO_REUSEADDR is that it allows you to launch your server twice. Both
processes will be able to bind to the same address without triggering an error, and they will then
compete for incoming connections, leading to confusing results. Servers that we develop later (e.g.,
Chapters 10, 14, and 16) avoid this possibility by creating a file when the program starts up and
deleting it on exit. The server refuses to launch if it sees that this file already exists.
The operating system does not allow a socket address bound by one user's process to be bound
by another user's process, regardless of the setting of SO_REUSEADDR.

The fcntl() and ioctl() Functions

In addition to socket options, a number of attributes are adjusted using the fcntl() and
ioctl() functions. We discuss fcntl() in Chapter 13, where we use it to turn on nonblocking I/
O, and again in Chapter 17, where we use fcntl() to set the owner of a socket so that we receive
a URG signal when the socket receives TCP urgent data.
The ioctl() function appears in Chapter 17 as well, where we use it to implement the sockat-
mark() function for handling urgent data, and again in Chapter 21, where we create a variety of
functions for examining and modifying the IP addresses assigned to network interfaces.

Other Socket-Related Functions

In addition to the functions we have already seen, there are three Perl built-in functions related to
sockets: send(), recv(), and socketpair(). We will use send() and recv() in later chapters
of this book when we discuss TCP urgent data (Chapter 17), and the UDP protocol (Chapters 17–
20).

$bytes = send (SOCK,$data,$flags[,$destination])

The send() function uses the socket indicated by the first argument to deliver the data indicated by $data,
to the destination address indicated by $destination. If the data was successfully queued for transmission,
send() returns the number of bytes sent; otherwise, it returns undef. The third argument, $flags, is the
bitwise OR of zero or more of the two options listed in Table 4.3.
We discuss the MSG_OOB flag in detail in Chapter 17. MSG_DONTROUTE is used in routing and diagnostic
programs and is not discussed in this book. In general, you should pass 0 as the value of $flags in order to
accept the default behavior.
If the socket is a connected TCP socket, then $destination should not be specified and send() is roughly
equivalent to syswrite(). With UDP sockets, the destination can be changed with every call to send().
$address = recv (SOCK,$buffer,$length,$flags)
The recv() function accepts up to $length bytes from the indicated socket and places them into the scalar
variable $buffer. The variable is grown or shrunk to the length actually read. The $flags argument has the
same significance as the corresponding argument in $send and should usually be set to 0.
If successful, recv() returns the packed socket address of the message sender. In case of error, the function
returns undef and sets $! appropriately.
When called on a connected TCP socket, recv() acts much like sysread(), except that it returns the address
of the peer. The real usefulness of recv() is for receiving datagrams for UDP transmissions.
$boolean = socketpair (SOCK_A,SOCK_B,$domain,$type,$protocol)
The socketpair() function creates two unnamed sockets connected end to end. $domain, $type, and
$protocol have the same significance as in the socket() function. If successful, socketpair() returns
true and opens sockets on SOCK_A and SOCK_B.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 The TCP Protocol 79

Table 4.3. send() Flags

Option Description
MSG_OOB Transmit a byte of urgent data on a TCP socket.
MSG_DONTROUTE Bypass routing tables.

The socketpair() function is similar to the pipe() function that we saw in Chapter 2, except
that the connection is bidirectional. Typically a script creates a pair of sockets and then fork(),
with the parent closing one socket and the child closing the other. The two sockets can then be used
for bidirectional communication between parent and child.
While in principle socketpair() can used for the INET protocol, in practice most systems only
support socketpair() for creating UNIX-domain sockets. Here is the idiom:
socketpair(SOCK1,SOCK2,AF_UNIX,SOCK_STREAM,PF_UNSPEC) or die $!;

We show examples of using UNIX-domain sockets in Chapter 22.

End-of-Line Constants Exported by the Socket Module

In addition to the constants used for constructing sockets and establishing outgoing connections,
the Socket module optionally exports constants and variables that are useful for dealing with text-
oriented network servers.
As we saw in Chapter 2, operating systems have different ideas of what constitutes the end of line
in a text file, with various OSs using carriage return (CR), linefeed (LF), or carriage return/linefeed
(CRLF). Adding to the confusion is the fact that Perl's "\r" and "\n" string escapes translate into
different ASCII characters depending on the local OS's idea of the newline.
Although this is not a hard and fast rule, most text-oriented network services terminate their lines
with the CRLF sequence, octal " \015\012 ". When performing line-oriented reads from such serv-
ers, you should set the input record separator, global $/, to " \015\012 " (not to " \r\n ", because
this is nonportable). To make this simpler, the Socket module optionally exports several constants
defining the common line endings (see Table 4. 4). In addition, to make it easier to interpolate these
sequences into strings, Socket exports the variables $CRLF, $CR, and $LF.

Table 4.4. Constants Exported by Socket Module

Name Description
CRLF A constant that contains the CRLF sequence
CR A constant that contains the CR character
LF A constant that contains the LF character

These symbols are not exported by default, but must be brought in with use either individually or
by importing the ":crlf" tag. In the latter case, you probably also want to import
the ":DEFAULT" tag in order to get the default socket-related constants as well:
use Socket qw(:DEFAULT :crlf);

Exceptional Conditions during TCP Communications

The TCP protocol is tremendously robust in the face of poor network conditions. It can survive slow
connections, flaky routers, intermittent network outages, and a variety of misconfigurations, and still
manage to deliver a consistent, error-free data stream.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 The TCP Protocol 80

TCP can't overcome all problems. This section briefly discusses the common exceptions, as well
as some common programming errors.

Exceptions during connect()

Various errors are common during calls to connect().
1. The remote host is up, but no server is listening when the client tries to connect. —A client
tries to connect to a remote host, but no server is listening to the indicated port. The con-
nect() function aborts with a "Connection refused" (ECONNREFUSED) error.
2. The remote host is down when the client tries to connect. A client tries to connect to a remote
host, but the host is not running (it is crashed or unreachable). In this case, connect() blocks
until it times out with a "Connection timed out" (ETIMEDOUT) error. TCP is forgiving of slow
network connections, so the timeout might not occur for many minutes.
3. The network is misconfigured. —A client tries to connect to a remote host, but the operating

Licensed by
system can't figure out how to route the message to the desired destination, because of a local
misconfiguration or the failure of a router somewhere along the way. In this case, con-
nect() fails with a "Network is unreachable" (ENETUNREACH) error.
4. There is a programmer error. —Various errors are due to common programming mistakes.

Stjepan Maric
For example, an attempt to call connect() with a filehandle rather than a socket results in a
"Socket operation on non-socket" (ENOTSOCK) error. An attempt to call connect() on a
socket that is already connected results in "Transport endpoint is already connected" (EIS-
CONN) error.

4218908
The ENOTSOCK error can also be returned by other socket calls, including bind(), listen(),
accept(), and the sockopt() calls.

Exceptions during Read and Write Operations

Once a connection is established, errors are still possible. You are almost sure to encounter the
following errors during your work with networked programs.
1. The server program crashes while you are connected to it. —If the server program crashes
during a communications session, the operating system will close the socket. From your per-
spective, the situation is identical to the remote program closing its end of the socket deliber-
ately.
On reads, this results in an EOF the next time read() or sysread() is called. On writes,
this results in a PIPE exception, exactly as in the pipe examples in Chapter 2. If you intercept
and handle PIPE, print() or syswrite() returns false and $! is set to "Broken pipe"
(EPIPE). Otherwise, your program is terminated by the PIPE signal.
2. The server host crashes while a connection is established. —On the other hand, if the host
crashes while a TCP connection is active, the operating system has no chance to terminate
the connection gracefully. At your end, the operating system has no way to distinguish between
a dead host and one that is simply experiencing a very long network outage. Your host will
continue to retransmit IP packets in hopes that the remote host will reappear. From your per-
spective, the current read or write call will block indefinitely.
At some later time, when the remote host comes back on line, it will receive one of the local
host's retransmitted packets. Not knowing what to do with it, the host will transmit a low-level
reset message, telling the local host that the connection has been rejected. At this point, the
connection is broken, and your program receives either an EOF or a pipe error, depending on
the operation.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 The TCP Protocol 81

One way to avoid blocking indefinitely is to set the SO_KEEPALIVE option on the socket. In
this case, the connection times out after some period of unresponsiveness, and the socket is
closed. The keepalive timeout is relatively long (minutes in some cases) and cannot be
changed.
3. The network goes down while a connection is established. —If a router or network segment
goes down while a connection is established, making the remote host unreachable, the current
I/O operation blocks until connectivity is restored. In this case, however, when the network is
restored the connection usually continues as if nothing happened, and the I/O operation com-
pletes successfully.
There are several exceptions to this, however. If, instead of simply going down, one of the routers
along the way starts issuing error messages, such as "host unreachable," then the connection is
terminated and the effect is similar to scenario (1). Another common situation is that the remote
server has its own timeout system. In this case, it times out and closes the connection as soon as
network connectivity is restored.

Summary
TCP sockets are created using socket(). Clients call connect() to establish an outgoing con-
nection to a remote host. Servers call bind() to assign an address to the socket, listen() to
notify the operating system that the socket is ready to accept connections, and accept() to accept
an incoming connection. Once a TCP socket is connected, it can be used like a filehandle to read
and write stream-oriented data.
Sockets have a number of options that can be set and examined with setsockopt() and get-
sockopt(), respectively. Unlike filehandles, which when closed allow no further reading or writing,
sockets can be half-closed using shutdown(), which allows the socket to be closed for reading,
writing, or both.
You've now seen the entire Berkeley socket API. The next chapter introduces Perl's object-oriented
extensions, which greatly simplify the task of working with sockets.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 82

Chapter 5. The IO::Socket API

The last chapter walked through Perl's built-in interface to Berkeley sockets, which closely mirrors
the underlying C-library calls. Some of the built-in functions, however, are awkward because of the
need to convert addresses and other data structures into the binary forms needed by the C library.
The advent of the object-oriented extensions to Perl5 made it possible to create a more intuitive
interface based on the IO::Handle module. IO::Socket and related modules simplify code and make
it easier to read by eliminating the noisy C-language-related calls and allowing you to focus on the
core of the application.
This chapter introduces the IO::Socket API, and then walks through some basic TCP applications.

Using IO::Socket
Before we discuss IO::Socket in detail, let's get a feel for it by reimplementing some of the examples
from Chapters 3 and 4.

A Daytime Client
The first example is a rewritten version of the daytime service client developed in Chapter 3 (Figure
3.7). As you recall, this client establishes an outgoing connection to a daytime server, reads a line,
and exits.
This is a good opportunity to fix a minor bug in the original examples (left there in the interests of
not unnecessarily complicating the code). Like many Internet servers, the daytime service termi-
nates its lines with CRLF rather than a single LF as Perl does. Before reading from daytime, we set
the end-of-line character to CRLF. Otherwise, the line we read will contain an extraneous CR at the
end.
Figure 5.1 lists the code.

Figure 5.1. Time of day client using IO::Socket

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 The IO::Socket API 83

Lines 1–4: Initialize module—We load IO::Socket and import the ":crlf" constants as well as
the default constants. These constants are conveniently reexported from Socket. We recover
the name of the remote host from the command line.
Line 5: Set the end-of-line separator—To read lines from the daytime server, we set the $/ end-
of-line global to CRLF. Note that this global option affects all filehandles, not just the socket.
Lines 6–7: Create socket—We create a new IO::Socket object by calling IO::Socket::INET's new
method, specifying the destination address in the form $host:service. We will see other ways to
specify the destination address later.
Lines 8–9: Read the time of day and print it—We read a single line from the server by calling
getline(), and remove the CRLF from the end with chomp(). We print this line to STDOUT.
When we run this program, we get the expected result:
% time_of_day_tcp.pl wuarchive.wustl.edu
Tue Aug 15 07:39:49 2000

TCP Echo Client

Now we'll look at an object-oriented version of the echo service client from Chapter 4. Figure 5.2
lists the code.

Figure 5.2. TCP echo client using IO::Socket

Lines 1–8: Initialize script—We load IO::Socket, initialize constants and globals, and process
the command-line arguments.
Line 9: Create socket—We call the IO::Socket::INET->new() method using the $host:
$port argument. If new() is successful, it returns a socket object connected to the remote host.
Lines 10–16: Enter main loop—We now enter the main loop. Each time through the loop we call
getline() on the STDIN filehandle to retrieve a line of input from the user. We send this line
of text to the remote host using print() on the socket, and read the server's response using
the <> operator. We print the response to standard output, and update the statistics.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 The IO::Socket API 84

Lines 17–18: Clean up—The main loop will end when STDIN is closed by the user. We close
the socket and print the accumulated statistics to STDERR.
Did you notice that line 10 uses the object-oriented getline() method with STDIN? This is a
consequence of bringing in IO::Socket, which internally loads IO::Handle. As discussed in Chapter
2, a side effect of IO::Handle is to add I/O object methods to all filehandles used by your program,
including the standard ones.
Unlike with the daytime client, we don't need to worry about what kind of line ends the echo service
uses, because it echoes back to us exactly what we send it.
Note also that we did not need to set autoflush mode on the IO::Socket object, as we did in examples
in Chapter 4. Since IO::Socket version 1.18, autoflush is turned on by default in all sockets created
by the module. This is the version that comes with Perl 5.00503 and higher.

IO::Socket Methods
We'll now look at IO::Socket in greater depth.

The IO::Handle Class Hierarchy

Figure 5.3 diagrams the IO::Handle class hierarchy. The patriarch of the family tree, IO::Handle,
provides object-oriented syntax for all of Perl's various input/output methods. Its immediate de-
scendent, IO::Socket, defines additional methods that are suitable for Berkeley sockets. IO::Socket
has two descendents. IO::Socket::INET defines behaviors that are specific for Internet domain
sockets; IO::Socket::UNIX defines appropriate behaviors for AF_UNIX (a.k.a., AF_LOCAL) sockets.

Figure 5.3. The IO::Handle class hierarchy

One never creates an IO::Socket object directly, but creates either an IO::Socket::INET or an
IO::Socket::UNIX object. We use the IO::Socket::INET subclass both in this chapter and in much of
the rest of this book. Future versions of the I/O library may support other addressing domains.
Other important descendents of IO::Handle include IO::File, which we discussed in Chapter 2, and
IO::Pipe, which provides an object-oriented interface to Perl's pipe() call. From IO::Socket::INET
descends Net::Cmd, which is the parent of a whole family of third-party modules that provide inter-
faces to specific command-oriented network services, including FTP and Post Office Protocol. We
discuss these modules beginning in Chapter 6.
Although not directly descended from IO::Handle, other modules in the IO::* namespace include
IO::Dir for object-oriented methods for reading and manipulating directories, IO::Select for testing
sets of filehandles for their readiness to perform I/O, and IO::Seekable for performing random access
on a disk file. We introduce IO::Select in Chapter 12, where we use it to implement network servers
using I/O multiplexing.

Creating IO::Socket::INET Objects

As with other members of the IO::Handle family, you create new IO::Socket::INET objects by in-
voking its new() method, as in:

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 The IO::Socket API 85

$sock = IO::Socket::INET->new('wuarchive.wustl.edu:daytime');

This object is then used for all I/O related to the socket. Because IO::Socket::INET descends from
IO::Handle, its objects inherit all the methods for reading, writing, and managing error conditions
introduced in Chapter 2. To these inherited methods, IO::Socket::INET adds socket-oriented meth-
ods such as accept(), connect(), bind(), and sockopt().
As with IO::File, once you have created an IO::Socket option you have the option of either using the
object with a method call, as in:
$sock->print('Here comes the data.');

or using the object as a regular filehandle:

print $sock 'Ready or not, here it comes.';

Which syntax you use is largely a matter of preference. For performance reasons discussed at the
end of this chapter, I prefer the function-oriented style whenever there is no substantive difference
between the two.
The IO::Socket::INET->new() constructor is extremely powerful, and is in fact the most com-
pelling reason for using the object-oriented socket interface.

$socket = IO::Socket::INET->new (@args);

The new() class method attempts to create an IO::Socket::INET object. It returns the new object, or if an error
was encountered, undef. In the latter case, $! contains the system error, and $@ contains a more verbose
description of the error generated by the module itself.

IO::Socket::INET->new() accepts two styles of argument. In the simple "shortcut" style,

new() accepts a single argument consisting of the name of the host to connect to, a colon, and the
port number or service name. IO::Socket::INET creates a TCP socket, looks up the host and service
name, constructs the correct sockaddr_in structure, and automatically attempts to connect()
to the remote host. The shortcut style is very flexible. Any of these arguments is valid:
wuarchive.wustl.edu:echo
wuarchive.wustl.edu:7
128.252.120.8:echo
128.252.120.8:7

In addition to specifying the service by name or port number, you can combine the two so that
IO::Socket::INET will attempt to look up the service name first, and if that isn't successful, fall back
to using the hard-coded port number. The format is hostname:service(port). For instance, to connect
to the wuarchive echo service, even on machines that for some reason don't have the echo service
listed in the network information database, we can call:
my $echo = IO::Socket::INET->new('wuarchive.wustl.edu:echo(7)')
or die "Can't connect: $!\n";

The new() method can also be used to construct sockets suitable for incoming connections, UDP
communications, broadcasting, and so forth. For these more general uses, new() accepts a named
argument style that looks like this:
my $echo = IO::Socket::INET->new(PeerAddr => 'wuarchive.wustl.edu',
PeerPort => 'echo(7)',
Type => SOCK_STREAM,
Proto => 'tcp')
or die "Can't connect: $!\n";

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 The IO::Socket API 86

Recall from Chapter 1 that the " => " symbol is accepted by Perl as a synonym for ",". The newlines
between the argument pairs are for readability only. In shorter examples, we put all the name/
argument pairs on a single line.
The list of arguments that you can pass to IO::Socket::INET is extensive. They are summarized in
Table 5.1
Table 5.1. Arguments to IO::Socket::INET->new()
Argument Description Value
PeerAddr Remote host address <hostname or address>[:<port>]
PeerHost Synonym for PeerAddr
PeerPort Remote port or service <service name or number>
LocalAddr Local host bind address <hostname or address>[:port]
LocalHost Synonym for LocalAddr
LocalPort Local host bind port <service name or port number>
Proto Protocol name (or number) <protocol name or number>
Type Socket type SOCK_STREAM | SOCK_DGRAM | ...
Listen Queue size for listen <integer>
Reuse Set SO_REUSEADDR before binding <boolean>
Timeout Timeout value <integer>
MultiHomed Try all adresses on multihomed hosts <boolean>

The PeerAddr and PeerHost arguments are synonyms which are used to specify a host to connect
to. When IO::Socket::INET is passed either of these arguments, it will attempt to connect() to the
indicated host. These arguments accept a hostname, an IP address, or a combined hostname and
port number in the format that we discussed earlier for the simple form of new(). If the port number
is not embedded in the argument, it must be provided by PeerPort.
PeerPort indicates the port to connect to, and is used when the port number is not embedded in the
hostname. The argument can be a numeric port number, a symbolic service name, or the combined
form, such as "ftp(22)."
The LocalAddr, LocalHost, and LocalPort arguments are used by programs that are acting as serv-
ers and wish to accept incoming connections. LocalAddr and LocalHost are synonymous, and
specify the IP address of a local network interface. LocalPort specifies a local port number. If
IO::Socket::INET sees any of these arguments, it constructs a local address and attempts to
bind() to it.
The network interface can be specified as an IP address in dotted-quad form, as a DNS hostname,
or as a packed IP address. The port number can be given as a port number, as a service name, or
using the "service(port)" combination. Itis also possible to combine the local IP address with the port
number, as in "127.0.0.1:http(80)." In this case, IO::Socket::INET will take the port number from
LocalAddr, ignoring the LocalPort argument.
If you specify LocalPort but not LocalAddr, then IO::Socket::INET binds to the INADDR_ANY wild-
card, allowing the socket to accept connections from any of the host's network interfaces. This is
usually the behavior that you want.
Stream-oriented programs that wish to accept incoming connections should also specify the Lis-
ten and possibly Reuse arguments. Listen gives the size of the listen queue. If the argument is
present, IO::Socket will call listen() after creating the new socket, using the argument as its
queue length. This argument is mandatory if you wish to call accept() later.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 The IO::Socket API 87

Reuse, if a true value, tells IO::Socket::INET to set the SO_REUSEADDR option on the new socket.
This is useful for connection-oriented servers that need to be restarted from time to time. Without
this option, the server has to wait a few minutes between exiting and restarting in order to avoid
"address in use" errors during the call to bind().
Proto and Type specify the protocol and socket type. The protocol may be symbolic (e.g., "tcp") or
numeric, using the value returned by getprotoby name(). Type must be one of the SOCK_*
constants, such as SOCK_STREAM. If one or more of these options is not provided, IO::Socket::INET
guesses at the correct values from context. For example, if Type is absent, IO::Socket:: INET infers
the correct type from the protocol. If Proto is absent but a service name was given for the port, then
IO::Socket::INET attempts to infer the correct protocol to use from the service name. As a last resort,
IO::Socket::INET defaults to "tcp."
Timeout sets a timeout value, in seconds, for use with certain operations. Currently, timeouts are
used for the internal call to connect() and in the accept() method. This can be handy to prevent
a client program from hanging indefinitely if the remote host is unreachable.
The MultiHomed option is useful in the uncommon case of a TCP client that wants to connect to a
host with multiple IP addresses and doesn't know which IP address to use. If this argument is set
to a true value, the new(), method uses gethostbyname() to look up all the IP addresses for the
hostname specified by PeerAddr. It then attempts a connection to each of the host's IP addresses
in turn until one succeeds.
To summarize, TCP clients that wish to make outgoing connections should call new() with a
Proto argument of tcp, and either a PeerAddr with an appended port number, or a PeerAddr/Peer-
Port pair. For example:
my $sock = IO::Socket::INET->new(Proto => 'tcp',
PeerAddr => 'www.yahoo.com',
PeerPort => 'http(80)');

TCP servers that wish to accept incoming connections should call new(), with a Proto of " tcp ",
a LocalPort argument indicating the port they wish to bind with, and a Listen argument indicating
the desired listen queue length:
my $listen = IO::Socket::INET->new(Proto => 'tcp',
LocalPort => 2007,
Listen => 128);

As we will discuss in Chapter 19, UDP applications need provide only a Proto argument of " udp "
or a Type argument of SOCK_DGRAM. The idiom is the same for both clients and servers:
my $udp = IO::Socket::INET->new(Proto => 'udp');

IO::Socket Object Methods

Once a socket is created, you can use it as the target for all the standard I/O functions, including
print(), read(), <>, sysread(), and so forth. The object-oriented method calls discussed in
Chapter 2 in the context of IO::File are also available. In addition, IO::Socket::INET adds the fol-
lowing socket-specific methods to its objects:

$connected_socket = $listen_socket->accept()
($connected_socket,$remote_addr) = $listen_socket->accept()
The accept() method performs the same task as the like-named call in the function-oriented API. Valid only
when called on a listening socket object, accept() retrieves the next incoming connection from the queue,
and returns a connected session socket that can be used to communicate with the remote host. The new socket
inherits all the attributes of its parent, except that it is connected.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 The IO::Socket API 88

When called in a scalar context, accept() returns the connected socket. When called in an array context,
accept() returns a two-element list, the first of which is the connected socket, and the second of which is the
packed address of the remote host. You can also recover this address at a later time by calling the connected
socket's peername() method.
$return_val = $sock->connect ($dest_addr)
$return_val = $sock->bind ($my_addr)
$return_val = $sock->listen ($max_queue)
These three TCP-related methods are rarely used because they are usually called automatically by new().
However, if you wish to invoke them manually, you can do so by creating a new TCP socket without providing
either a PeerAddr or a Listen argument:
$sock = IO::Socket::INET->new(Proto=>'tcp');
$dest_addr = sockaddr_in(...) # etc.
$sock->connect($dest_addr);

$return_val = $sock->connect ($port, $host)

$return_val = $sock->bind ($port, $host)
For your convenience, connect() and bind() both have alternative two-argument forms that take unpacked
port and host addresses rather than a packed address. The host address can be given in dotted-IP form or as
a symbolic hostname.
$return_val = $socket->shutdown($how)
As in the function-oriented call, shutdown() is a more forceful way of closing a socket. It will close the socket
even if there are open copies in forked children. $how controls which half of the bidirectional socket will be
closed, using the codes shown in Table 3.1.
$my_addr = $sock->sockname()
$her_addr = $sock->peername()
The sockname() and peername() methods are simple wrappers around their function-oriented equivalents.
As with the built-in functions, they return packed socket addresses that must be unpacked using sock-
addr_in().
$result = $sock->sockport()
$result = $sock->peerport()
$result = $sock->sockaddr()
result = $sock->peeraddr()
These four methods are convenience functions that unpack the values returned by sockname() and peer-
name(). sockport() and peerport() return the port numbers of the local and remote endpoints of the
socket, respectively. sockaddr(), and peeraddr() return the IP address of the local and remote endpoints
of the socket as binary structures suitable for passing to gethostbyaddr(). To convert theresult into dotted-
quad form, you still need to invoke inet_ntoa().
$my_name = $sock->sockhost()
$her_name = $sock->peerhost()
These methods go one step further, and return the local and remote IP addresses in full dotted-quad form
("aa.bb.cc.dd"). If you wish to recover the DNS name of the peer, falling back to the dotted-quad form in case
of a DNS failure, here is the idiom:
$peer = gethostbyaddr($sock->peeraddr,AF_INET) || $sock->peerhost;

$result = $sock->connected()
The connected() method returns true if the socket is connected to a remote host, false otherwise. It works
by calling peername().
$protocol = $sock->protocol()
$type = $sock->socktype()

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 The IO::Socket API 89

$domain = $sock->sockdomain()
These three methods return basic information about the socket, including its numeric protocol, its type, and its
domain. These methods can be used only to get the attributes of a socket object. They can't be used to change
the nature of an already-created object.
$value = $sock->sockopt($option [,$value])
The sockopt() method can be used to get and/or set a socket option. It is a front end for both getsock-
opt() and setsockopt(). Called with a single numeric argument, sockopt() retrieves the current value
of the option. Called with an option and a new value, sockopt() sets the option to the indicated value, and
returns a result code indicating success or failure. There is no need to specify an option level, as you do with
getsockopt(), because the SOL_SOCKET argument is assumed.
Unlike the built-in getsockopt(), the object method automatically converts the packed argument returned
by the underlying system call into an integer, so you do not need to unpack the option values returned by
sockopt(). As we discussed earlier, the most frequent exception to this is the SO_LINGER option, which
operates on an 8-byte linger structure as its argument.
$val = timeout([$timeout])
timeout() gets or sets the timeout value that IO::Socket uses for its connect(), and accept() methods.
Called with a numeric argument, it sets the timeout value and returns the new setting. Otherwise, it returns the
current setting. The timeout value is not currently used for calls that send or receive data. The eval{} trick,
described in Chapter 2, can be used to achieve that result.
$bytes = $sock->send ($data [, $flags ,$destination])
$address = $sock-> recv ($buffer,$length [,$flags])
These are front ends for the send() and recv() functions, and are discussed in more detail when we discuss
UDP communications in Chapter 19.

An interesting side effect of the timeout implementation is that setting theIO::Socket::INET timeout
makes the connect() and accept() calls interruptable by signals. This allows a signal handler
to gracefully interrupt a program that is hung waiting on a connect() or accept(). We will see
an example of this in the next section.

More Practical Examples

We'll now look at additional examples that illustrate important aspects of the IO::Socket API. The
first is a rewritten and improved version of the reverse echo server from Chapter 4. The second is
a simple Web client.

Reverse Echo Server Revisited

We'll now rewrite the reverse echo server in Figure 4.2. In addition to being more elegant than the
earlier version (and easier to follow, in my opinion), this version makes several improvements on
the original. It replaces the dangerous signal handler with a handler that simply sets a flag and
returns. This avoids problems stemming from making I/O calls within the handler and problems with
exit() on Windows platforms. The second improvement is that the server resolves the names of
incoming connections, printing the remote hostname and port number to standard error. Finally, we
will observe the Internet convention that servers use CRLF sequences for line endings. This means
that we will set $/ to CRLF and append CRLF to the end of all our writes. Figure 5.4 lists the code
for the improved server.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 The IO::Socket API 90

Licensed by
Stjepan Maric
4218908

Figure 5.4. The reverse echo server, using IO::Socket

Lines 1–7: Initialize script—We turn on strict syntax checking and load the IO::Socket module.
We import the default constants and the newline-related constants by importing the
tags :DEFAULT and :crlf.
We define our local port as a constant, and initialize the byte counters for tracking statistics. We
also set the global $/ variable to CRLF in accordance with the network convention.
Lines 8–9: Install INT signal handler—We install a signal handler for the INT signal, so that the
server will shut down gracefully when the user presses the interrupt key. The improved handler
simply sets the flag named $quit to true.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 The IO::Socket API 91

Lines 10–15: Create the socket object—We recover the port number from the command line or,
if no port number is provided, we default to the hard-coded constant. We now call
IO::Socket::INET->new() with arguments that cause it to create a listening socket bound
to the specified local port. Other arguments set the SO_REUSEADDR option to true, and specify
a 1-hour timeout (60*60 seconds) for the accept() operation.
The Timeout parameter makes each call to the accept() method return undef if an incoming
connection has not been received within the specified time. However, our motivation for acti-
vating this feature is not for its own sake, but for the fact that it changes the behavior of the
method so that it is not automatically restarted after being interrupted by a signal. This allows
us to interrupt the server with the ^C key without bothering to wrap accept() in an eval{}
block (see Chapter 2, Timing Out Slow System Calls).
Lines 16–31: Enter main loop—After printing a status message, we enter a loop that continues
until the INT interrupt handler has set $quit to true. Each time through the loop, we call the
socket's accept() method. If the accept() method completes without being interrupted by a
signal or timing out on its own, it returns a new connected socket object, which we store in a
variable named $session. Otherwise, accept() returns undef, in which case we go back to
the beginning of the loop. This gives us a chance to test whether the interrupt handler has set
$quit to true.
Lines 19–21: Get remote host's name and port—We call the connected socket's peeraddr()
method to get the packed IP address at the other end of the connection, and attempt to translate
it into a hostname using gethostbyaddr(). If this fails, it returns undef, and we call the
peerhost() method to give us the remote host's address in dotted-quad form.
We get the remote host's port number using peerport(), and print the address and port num-
ber to standard error.
Lines 22–30: Handle the connection—We read lines from the connected socket, reverse them,
and print them back out to the socket, keeping track of the number of bytes sent and received
while we do so. The only change from the earlier example is that we now terminate each line
with CRLF.
When the remote host is done, we get an EOF when we try to read from the connected socket.
We print out a warning, close the connected socket, and go back to the top of the loop to
accept() again.
When we run the script, it acts like the earlier version did, but the status messages give hostnames
rather than dotted-IP addresses.
% tcp_echo_serv2.pl
waiting for incoming connections on port 2007...
Connection from [localhost,2895]
Connection from [localhost,2895] finished
Connection from [formaggio.cshl.org,12833]
Connection from [formaggio.cshl.org,12833] finished
^C
bytes_sent = 50, bytes_received = 50

A Web Client
In this section, we develop a tiny Web client named web_fetch.pl. It reads a Universal Resource
Locator (URL) from the command line, parses it, makes the request, and prints the Web server's
response to standard output. Because it returns the raw response from the Web server without
processing it, it is very useful for debugging misbehaving CGI (Common Gateway Interface) scripts
and other types of Web-server dynamic content.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 The IO::Socket API 92

The Hypertext Transfer Protocol (HTTP) is the main protocol for Web servers. Part of the power
and appeal of the protocol is its simplicity. A client wishing to fetch a document makes a TCP con-
nection to the desired Web server, sends a brief request, and then receives the response from the
server. After the document is delivered, the Web server breaks the connection. The hardest part is
parsing the URL. HTTP URLs have the following general format:
http://hostname:port/path/to/document#fragment
All HTTP URLs begin with the prefix http://. This is followed by a hostname such as www.ya-
hoo.com, a colon, and the port number that the Web server is listening on. The colon and port may
be omitted, in which case the standard server port 80 is assumed. The hostname and port are
followed by the path to the desired document using UNIX-like file path conventions. The path may
be followed by a "#" sign and a fragment name, which indicate a subsection in the document that
the Web browser should scroll to.
Our client will parse the components of this URL into the hostname:port combination and the path.
We ignore the fragment name. We then connect to the designated server using a TCP socket and
send an HTTP request of this form:
GET /path/to/document HTTP/1.0 CRLF CRLF

The request consists of the request method "GET" followed by a single space and the designated
path, copied verbatim from the URL. This is followed by another space, the protocol version number
HTTP/1.0, and two CRLF pairs. After the request is sent, we wait for a response from the server. A
typical response looks like this:
HTTP/1.1 200 OK
Date: Wed, 01 Mar 2000 17:00:41 GMT
Server: Apache/1.3.6 (UNIX)
Last-Modified: Mon, 31 Jan 2000 04:28:15 GMT
Connection: close
Content-Type: text/html

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML//EN">

<html> <head> <title> Presto Home Page </title> </head>
<body>
<h1>Welcome to Presto</h1>
...

The response is divided into two parts: a header containing information about the returned docu-
ment, and the requested document itself. The two parts are separated by a blank line formed by
two CRLF pairs.
We will delve into the structure of HTTP responses in more detail in Chapter 9, where we discuss
the LWP library, and Chapter 13, where we develop a more sophisticated Web client capable of
retrieving multiple documents simultaneously. The only issue to worry about here is that, whereas
the header is guaranteed by the HTTP protocol to be nice human-readable lines of text, each ter-
minated by a CRLF pair, the document itself can have any format. In particular, we must be prepared
to receive binary data, such as the contents of a GIF or MP3 file.
Figure 5.5 shows the web_fetch.pl script in its entirety.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 The IO::Socket API 93

Figure 5.5. The web_fetch.pl script

Lines 1–5: Initialize module—We turn on strict syntax checking and load the IO::Socket module,
importing the default and newline-related constants. As in previous examples, we are dealing
with CRLF-delimited data. However, in this case, we set $/ to be a pair of CRLF sequences.
Later, when we call the <> operator, it will read the entire header down through the CRLF pair
that terminates it.
Lines 6–8: Parse URL—We read the requested URL from the command line and parse it using
a pattern match. The match returns the hostname, possibly followed by a colon and port number,
and the path up through, but not including, the fragment.
Lines 9–10: Open socket—We open a socket connected to the remote Web server. If the URL
contained the port number, it is included in the hostname passed to PeerAddr, and the Peer-
Port argument is ignored. Otherwise, PeerPort specifies that we should connect to the standard
"http" service, port 80.
Line 11: Send the request—We send an HTTP request to the server using the format described
earlier.
Lines 12–14: Read and print the header—Our first read is line-oriented. We read from the socket
using the <> operator. Because $/ is set to a pair of CRLF sequences, this read grabs the entire
header up through the blank line. We now print the header, but since we don't particularly want
extraneous CRs to mess up the output, we first replace all occurrence of $CRLF with a logical
newline ("\n", which will evaluate to whatever is the appropriate newline character for the current
platform.
Line 15: Read and print the document—Our subsequent reads are byte-oriented. We call
read() in a tight loop, reading up to 1024 bytes with each operation, and immediately printing
them out with print(). We exit when read() hits the EOF and returns 0.
Here is an example of what web_fetch.pl looks like when it is asked to fetch the home page for
www.cshl.org:

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 The IO::Socket API 94

% web_fetch.pl http://www.cshl.org/
HTTP/1.1 200 OK
Server: Netscape-Enterprise/3.5.1C
Date: Wed, 16 Aug 2000 00:46:12 GMT
Content-type: text/html
Last-modified: Fri, 05 May 2000 13:19:29 GMT
Content-length: 5962
Accept-ranges: bytes
Connection: close

<HTML>
<HEAD>
<TITLE>Cold Spring Harbor Laboratory</TITLE>

<META NAME="GENERATOR" CONTENT="Adobe PageMill 2.0 Mac"> <META

Name="keywords" Content="DNA, genes, genetics, genome, genome
sequencing, molecular biology, biological science, cell biology,
James D. Watson, Jim Watson, plant genetics, plant biology,
bioinformatics, neuroscience, neurobiology, cancer, drosophila,
Arabidopsis, double-helix, oncogenesis, Cold Spring Harbor
Laboratory, CSHL">
...

Although it seems like an accomplishment to fetch a Web page in a mere 15 lines of code, client
scripts that use the LWP module can do the same thing—and more—with just a single line. We
discuss how to use LWP to access the Web in Chapter 9.

Performance and Style

Although the IO::Socket API simplifies programming, and is generally a big win in terms of devel-
opment effort and code maintenance, it has some drawbacks over the built-in function-oriented
interface.
If memory usage is an issue, you should be aware that IO::Socket adds to the Perl process's memory
"footprint" by a significant amount: approximately 800 K on my Linux/Intel-based system, and more
than double that on a Solaris system.
The object-oriented API also slows program loading slightly. On my laptop, programs that use
IO::Socket take about half a second longer to load than those that use the function-oriented Socket
interface. Fortunately, the execution speed of programs using IO::Socket are not significantly dif-
ferent from the speeds of classical interface. Network programs are generally limited by the speed
of the network rather than the speed of processing.
Nevertheless, for the many IO::Socket methods that are thin wrappers around the corresponding
system calls and do not add significant functionality, I prefer to use the IO::Socket as plain filehan-
dles rather than use the object-oriented syntax. For example, rather than writing:
$socket->syswrite("A man, a plan, a canal, panama!");

I will write:
syswrite ($socket,"A man, a plan, a canal, panama!");

This has exactly the same effect as the method call, but avoids its overhead.
For methods that do improve on the function call, don't hesitate to use them. For example, the
accept() method is an improvement over the built-in function because it returns a connected
IO::Socket object rather than a plain filehandle. The method also has a syntax that is more Perl-like
than the built-in.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 The IO::Socket API 95

Concurrent Clients
This chapter concludes by introducing a topic that is one of the central issues of Part III, the problem
of concurrency.

A Gab Client, First Try

To motivate this discussion, let's write a simple client that can be used for interactive conversations
with line-oriented servers. This program will connect to a designated host and port, and simply act
as a direct conduit between the remote machine and the user. Everything the user types is trans-
mitted across the wire to the remote host, and everything we receive from the remote host is echoed
to standard output.
We can use this client to talk to the echo servers from this chapter and the last, or to talk with other
line-oriented servers on the Internet. Common examples include FTP, SMTP, and POP3 servers.
We'll call this client gab1.pl, because it is the first in a series of such clients. A simple—but incorrect
—implementation of the gab client looks like Figure 5.6.

Figure 5.6. An incorrect implementation of a gab client

Lines 1–6: Initialize module—We load IO::Socket as before and recover the desired remote
hostname and port number from the command-line arguments.
Lines 7–8: Create socket—We create the socket using IO::Socket::INET->new() or, if
unsuccessful, die with an error message.
Lines 9–21: Enter main loop—We enter a loop. Each time through the loop we read one line
from the socket and print it to standard output. Then we read a line of input from the user and
print it to the socket.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 The IO::Socket API 96

Because the remote server uses CRLF pairs to end its lines, but the user types conventional
newlines, we need to keep setting and resetting $/. The easiest way to do this is to place the
code that reads a line from the socket in a little block, and to localize (with local) the $/ variable,
so that its current value is saved on entry into the block, and restored on exit. Within the block,
we set $/ to CRLF.
If we get an EOF from either the user or the server, we leave the loop by calling last.
At first, this straightforward script seems to work. For example, this transcript illustrates a session
with an FTP server. The first thing we see on connecting with the server is its welcome banner
(message code 220). We type in the FTP USER command, giving the name "anonymous," and get
an acknowledgment. We then provide a password with PASS, and get another acknowledgment.
Everything seems to be going smoothly.
% gab1.pl phage.cshl.org ftp
220 phage.cshl.org FTP server ready.
USER anonymous
331 Guest login ok, send your complete e-mail address as password.
PASS [email protected]
230 Guest login ok, access restrictions apply.

Unfortunately, things don't last that way for long. The next thing we try is the HELP command, which
is supposed to print a multiline summary of FTP commands. This doesn't go well. We get the first
line of the expected output, and then the script stops, waiting for us to type the next command. We
type another HELP, and get the second line of the output from the first HELP command. We type
QUIT, and get the third line of the HELP command.
HELP
214-The following commands are recognized (* =>'s unimplemented).
HELP
USER PORT STOR MSAM* RNTO NLST MKD CDUP
QUIT
PASS PASV APPE MRSQ* ABOR SITE XMKD XCUP
QUIT
ACCT* TYPE MLFL* MRCP* DELE SYST RMD STOU
QUIT
...

Clearly the script has gotten out of synch. As it is written, it can deal with only the situation in which
a single line of input from the user results in a single line of output from the server. Having no way
of dealing with multiline output, it can't catch up with the response to the HELP command.
What if we changed the line that reads from the server to something like this?
while ($from_server = <$socket>) {
chomp $from_server;
print $from_server,"\n";
}

Unfortunately, this just makes matters worse. Now the script hangs after it reads the first line from
the server. The FTP server is waiting for us to send it a command, but the script is waiting for another
line from the server and hasn't even yet asked us for input, a situation known as deadlock.
In fact, none of the straightforward rearrangements of the read and print orders fix this problem. We
either get out of synch or get hopelessly deadlocked.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 The IO::Socket API 97

A Gab Client, Second Try

What we need to do is decouple the process reading from the remote host from the process of
reading from the socket. In fact, we need to isolate the tasks in two concurrent but independent
processes that won't keep blocking each other the way the naive implementation of the gab client
did.
On UNIX and Windows systems, the easiest way to accomplish this task is using the fork() com-
mand to create two copies of the script. The parent process will be responsible for copying data
from standard input to the remote host, while the child will be responsible for the flow of data in the
other direction. Unfortunately, Macintosh users do not have access to this call. A good but somewhat
more complex solution that avoids the call to fork() is discussed in Chapter 12, Multiplexed Op-
erations.
As it turns out, the simple part of the script is connecting to the server, forking a child, and having
each process copy data across the network. The hard part is to synchronize the two processes so
that they both quit gracefully when the session is done. Otherwise, there is a chance that one proc-
ess may continue to run after the other has exited.
There are two scenarios for terminating the connection. In the first scenario, the remote server
initiates the process by closing its end of the socket. In this case, the child process receives an EOF
when it next tries to read from the server and calls exit(). It somehow has to signal to the parent
that it is done. In the second scenario, the user closes standard input. The parent process detects
EOF when it reads from STDIN and has to inform the child that the session is done.
On UNIX systems, there is a built-in way for children to signal parents that they have exited. The
CHLD signal is sent automatically to a parent whenever one of its subprocesses has died (or have
either stopped or resumed; we discuss this in more detail in Chapter 10). For the parent process to
detect that the remote server has closed the connection it merely has to install a CHLD handler that
calls exit(). When the child process detects that the server has closed the connection, the child
will exit, generating a CHLD signal. The parent's signal handler is invoked, and the process now
exits too.
The second scenario, in which the user closes STDIN, is a bit more complicated. One easy way is
for the parent just to kill() its child after standard input has closed. There is, however, a problem
with this. Just because the user has closed standard input doesn't mean that the server has finished
sending output back to us. If we kill the child before it has received and processed all the pending
information from the server, we may lose some information.
The cleaner way to do this is shown in Figure 5.7. When the parent process gets an EOF from
standard input, it closes its end of the socket, thereby sending the server an end-of-file condition.
The server detects the EOF, and closes its end of the connection, thereby propagating the EOF
back to the child process. The child process exits, generating a CHLD signal. The parent intercepts
this signal, and exits itself.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 The IO::Socket API 98

Figure 5.7. Closing a connection in a forked client

The beauty of this is that the child doesn't see the EOF until after it has finished processing any
queued server data. This guarantees that no data is lost. In addition, the scheme works equally well
when the termination of the connection is initiated by the server. The risk of this scheme is that the
server may not cooperate and close its end of the connection when it receives an EOF. However,
most servers are well behaved in this respect. If you encounter one that isn't, you can always kill
both the parent and the child by pressing the interrupt key.
There is one subtle aspect to this scheme. The parent process can't simply close() its copy of the
socket in order to send an EOF to the remote host. There is a second copy of the socket in the child
process, and the operating system won't actually close a filehandle until its last copy is closed. The
solution is for the parent to call shutdown(1) on the socket, forcefully closing it for writing. This
sends EOF to the server without interfering with the socket's ability to continue to read data coming
in the other direction. This strategy is implemented in Figure 5.8, in a script named gab2.pl.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 The IO::Socket API 99

Figure 5.8. A working implementation of a gab client

Lines 1–7: Initialize module—We turn on strict syntax checking, load IO::Socket, and fetch the
host and port from the command line.
Line 8: Create the socket—We create the connected socket in exactly the same way as before.
Lines 9–10: Call fork—We call fork(), storing the result in the variable $child. Recall that if
successful, fork() duplicates the current process. In the parent process, fork() returns the
PID of the child; in the child process, fork() returns numeric 0.
In case of error, fork() returns undef. We check for this and exit with an error message.
Lines 11–15: Parent process copies from standard input to socket —The rest of the script is
divided into halves. One half is the parent process, and is responsible for reading lines from
standard input and writing to the server; the other half is the child, which is responsible for reading
lines from the server and writing them to standard output.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 The IO::Socket API 100

In the parent process, $child is nonzero. For the reasons described earlier, we set up a signal
handler for the CHLD signal. This handler simply calls exit(). We then call the
user_to_host() subroutine, which copies user data from standard input to the socket.
When standard input is closed, user_to_host() returns. We call the socket's shutdown()
method, closing it for writing. Now we go to sleep indefinitely, awaiting the expected CHLD signal
that will terminate the process.
Lines 16–19: Child process copies from socket to standard output—In the child process, we call
host_to_user() to copy data from the socket to standard output. This subroutine will return
when the remote host closes the socket. We don't do anything special after that except to warn
that the remote host has closed the connection. We allow the script to exit normally and let the
operating system generate the CHLD message.
Lines 20–26: The user_to_host() subroutine—This subroutine is responsible for copying
lines from standard input to the socket. Our loop reads a line from standard input, removes the

Licensed by
newline, and then prints to the socket, appending a CRLF to the end. We return when standard
input is closed.
Lines 27–34: The host_to_user() subroutine—This subroutine is almost the mirror image of
the previous one. The only difference is that we set the $/ input record separator global to

Stjepan Maric
CRLF before reading from the socket. Notice that there's no reason to localize $/ in this case
because changes made in the child process won't affect the parent. When we've read the last
line from the socket, we return.
You may wonder why the parent goes to sleep rather than simply exit after it has shutdown() its

4218908
copy of the socket. The answer is simply esthetic. As soon as the parent exits, the user will see the
command-line prompt reappear. However, the child may still be actively reading from the socket
and writing to standard output. The child's output will intermingle in an ugly way with whatever the
user is doing at the command line. By sleeping until the child exits, the parent avoids this behavior.
You may also wonder about the call to exit() in the CHLD signal handler. While this is a problematic
construction on Windows platforms because it causes crashes, the sad fact is that the Windows
port of Perl does not generate or receive CHLD signals when a child process dies, so this issue is
moot. To terminate gab2.pl on Windows platforms, press the interrupt key.
When we try to connect to an FTP server using the revised script, the results are much more sat-
isfactory. Multiline results now display properly, and there is no problem of synchronization or dead-
locking.
% gab2.pl phage.cshl.org ftp
220 phage.cshl.org FTP server ready.
USER anonymous
331 Guest login ok, send your complete e-mail address as password.
PASS [email protected]
230 Guest login ok, access restrictions apply.
HELP
214-The following commands are recognized (* =>'s unimplemented).
USER PORT STO RMSAM* RNTO NLST MKD CDUP
PASS PASV APP EMRSQ* ABOR SITE XMKD XCUP
ACCT* TYPE MLFL* MRCP* DELE SYST RMD STOU
SMNT* STRU MAIL* ALLO CWD STAT XRMD SIZE
REIN* MODE MSND* REST XCWD HELP PWD MDTM
QUIT RETR MSOM* RNFR LIST NOOP XPWD
214 Direct comments to [email protected]
QUIT
221 Goodbye.
Connection closed by foreign host.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 The IO::Socket API 101

This client is suitable for talking to many line-oriented servers, but there is one Internet service that
you cannot successfully access via this client—the Telnet remote login service itself. This is because
Telnet servers initially exchange some binary protocol information with the client before starting the
conversation. If you attempt to use this client to connect to a Telnet port (port 23), you will just see
some funny characters and then a pause as the server waits for the client to complete the protocol
handshake. The Net::Telnet module (Chapter 6) provides a way to talk to Telnet servers.

Summary
The object-oriented IO::Socket library significantly simplifies network programming by eliminating
much of the "noise code" handed down from the socket API's C heritage and replacing it with a
clean and convenient interface. Throughout the remainder of this book we create IO::Socket objects
using the class's flexible new() method, and use its object methods whenever they offer a clear
syntactic advantage (e.g., $sock->accept()). In cases where there is no clear syntactic advant-
age, for example between read ($sock,$data,1024) and $sock->read ($data,1024), we
use the Perl built-ins in order to benefit from the performance gains.
With the IO::Socket module it is easy to write simple client and server programs, such as the Web
client and the reverse echo server demonstrated in this chapter. However, things became unex-
pectedly complicated when we tried to write a seemingly trivial tool to echo user commands to a
server and back again. We solved this by creating two processes using fork(). The struggle
against deadlock is a recurring theme in this book, which we deal with again in later chapters.
You have now learned how to build network applications on top of the low-level socket libraries. The
next part of this book moves up a level, showing you how to interact with application-level protocols.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 102

Part 2. Developing Clients for Common

Services

Perl's ease of use in writing network applications has spawned a large number of high-level modules
for everything from sending e-mail to accessing Web servers. There are many dozens of user-
contributed network client modules, ranging in complexity from a few to a few thousand lines of
code. Part II covers some of the popular client modules and shows how you can use them to solve
typical problems. They are built on top of the Berkeley socket API that we discussed in earlier
chapters.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 103

Chapter 6. FTP and Telnet

Two of the oldest Internet protocols are the File Transfer Protocol, FTP, and Telnet, for remote login.
They illustrate the two extremes of network protocols: An FTP session is a highly structured and
predictable set of transactions; a Telnet session is unpredictable and highly interactive. Perl has
modules that can tame them both.

Net::FTP
There's a directory on a remote FTP server that changes every few weeks. You want to mirror a
copy of the directory on your local machine and update your copy every time it changes. You can't
use one of the many "mirror" scripts to do this because the directory name contains a timestamp,
and you need to do a pattern match to identify the right directory. Net::FTP to the rescue.
Net::FTP is part of the libnet utilities by Graham Barr. In addition to Net::FTP, libnet includes
Net::SMTP, Net::NNTP, and Net::POP3 discussed in later chapters. When you install the libnet
modules, the install script prompts you for various default configuration parameters used by the
Net::* modules. This includes such things as an FTP firewall proxy and the default mail exchanger
for your domain. See the documentation for Net::Config (also part of the libnet utilities) for informa-
tion on how to override the defaults later.
Net::FTP, like many of the client modules, uses an object-oriented interface. When you first log in
to an FTP server, the module returns a Net::FTP object to you. You then use this object to get
directory listings from the server, to transfer files, and to send other commands.

A Net::FTP Example
Figure 6.1 is a simple example that uses Net::FTP to connect to ftp.perl.org and download the file
named RECENT from the directory /pub/CPAN/. If the program runs successfully, it creates a file
named RECENT in the current directory. This file contains the names of all files recently uploaded
to CPAN.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 FTP and Telnet 104

Figure 6.1. Downloading a single file with Net::FTP

Lines 1–5: Initialize—We load the Net::FTP module and define constants for the host to connect
to and the file to download.
Line 6: Connect to remote host—We connect to the FTP host by invoking the Net::FTP new()
method with the name of the host to connect to. If successful, new() returns a Net::FTP object
connected to the remote server. Otherwise, it returns undef, and we die with an error message.
In case of failure, new() leaves a diagnostic error message in $@.
Line 7: Log in to the server—After connecting to the server, we still need to log in by calling the
Net::FTP object's login() method with a username and password. In this case, we are using
anonymous FTP, so we provide the username "anonymous" and let Net::FTP fill in a reasonable
default password. If login is successful, login() returns a true value. Otherwise, it returns false
and we die, using the FTP object's message() method to retrieve the text of the server's last
message.
Line 8: Change to remote directory—We invoke the FTP object's cwd() ("change working di-
rectory") method to enter the desired directory. If this call fails, we again die with the server's
last message.
Line 9: Retrieve the file—We call the FTP object's get() method to retrieve the desired file. If
successful, Net::FTP copies the remote file to a local one of the same name in the current
directory. Otherwise we die with an error message.
Lines 10–11: Quit—We call the FTP object's quit() method to close the connection.

FTP and Command-Based Protocols

FTP is an example of a common paradigm for Internet services: the command-based protocol. The
interaction between client and server is constrained by a well-defined protocol in which the client
issues a single-line command and the server returns a line-oriented response.
Each of the client commands is a short case-insensitive word, possibly followed by one or more
arguments. The command is terminated by a CRLF pair. As we saw in Chapter 5, when we used
the gab2.pl script to communicate with an FTP server, the client commands in the FTP protocol
include user and PASS, which together are used to log into the server; HELP, to get usage informa-
tion; and QUIT, to quit the server. Other commands are used to send and retrieve files, obtain
directory listings, and so forth. For example, when the client wishes to log in under the user name
"anonymous," it will send this command to the server:
USER anonymous

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 FTP and Telnet 105

Each response from the server to the client consists of one or more CRLF-delimited lines. The first
line always begin with a three-digit numeric result code indicating the outcome of the command.
This is usually followed by a human-readable message. For example, a successful USER command
will result in the following server response:
331 Guest login ok, send your complete e-mail address as password.

Sometimes a server response will stretch over several lines. In this case, the numeric result code
on the first line will end in a "-", and the result code will be repeated (without the dash) on the last
line. The FTP protocol's response to the HELP command illustrates this:
HELP
214-The following commands are recognized (* =>'s unimplemented).
USER PORT STOR MSAM* RNTO NLST MKD CDUP
PASS PASV APPE MRSQ* ABOR SITE XMKD XCUP
ACCT* TYPE MLFL* MRCP* DELE SYST RMD STOU
SMNT* STRU MAIL* ALLO CWD STAT XRMD SIZE
REIN* MODE MSND* REST XCWD HELP PWD MDTM
QUIT RETR MSOM* RNFR LIST NOOP XPWD
214 Direct comments to [email protected]

Commonly the client and server need to exchange large amounts of non-command data. To do this,
the client sends a command to warn the server that the data is coming, sends the data, and then
terminates the information by sending a lone dot (".") on a line by itself. We will see an example of
this in the next chapter when we examine the interaction between an e-mail client and an SMTP
server.
Server result codes are arbitrary but generally follow a simple convention. Result codes between
100 and 199 are used for informational messages, while those in the 200–299 range are used to
indicate successful completion of a command. Codes in the 300–399 range are used to indicate
that the client must provide more information, such as the password that accompanies a username.
Result codes of 400 or greater indicate various errors: the 400–499 codes are used for client errors,
such as an invalid command, while 500 and greater are used for server-side errors, such as an out
of memory condition.
Because command-based servers are so common, the libnet package comes with a generic building
block module called Net::Cmd. The module doesn't actually do anything by itself, but adds func-
tionality to descendents of the IO::Socket module that allow them to easily communicate with this
type of network server. Net::FTP, Net::SMTP, Net::NNTP, and Net::POP3 are all derived from
Net::Cmd.
The two major methods provided by Net::Cmd objects are command() and response():

$success = $obj->command($command [,@args])

Send the command indicated by $command to the server, optionally followed by one or more arguments.
command() automatically inserts spaces between arguments and appends a CRLF to the end of the com-
mand. If the command was delivered successfully, the method returns true.
$status = $obj->response
Fetches and parses the server's response to the last command, returning the most significant digit as the
method result. For example, if the server's result code is 331, response() will return 3. It returns undef in
case of failure.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 FTP and Telnet 106

Subclasses of Net::Cmd build more sophisticated methods on top of the command() and
response(). For example, the Net::FTP login() method calls command() twice: once to issue
the USER command and again to issue the PASS command. You will not ordinarily call com-
mand() and response(), yourself, but use the more specialized (and convenient) methods pro-
vided by the subclass. However, command() and response() are available should you need
access to functionality that isn't provided by the module.
Several methods provided by Net::Cmd are commonly used by end-user applications. These are
code(), message(), and ok():

$code = $obj->code
Returns the three-digit numeric result code from the last response.
$message = $obj->message
Returns the text of the last message from the server. This is particularly useful for diagnosing errors.
$ok = $obj->ok
The ok() method returns true if the last server response indicated success, false otherwise. It returns true if
the result code is greater than 0 but less than 400.

The Net::FTP API

We'll now look at the Net::FTP API in greater detail. Net::FTP is a descendent of both IO::Socket
and Net::Cmd. As a descendent of IO::Socket, it can be used as a filehandle to communicate directly
with the server. For example, you canread and write to a Net::FTP object with syswrite() and
sysread(), although you would probably not want to. As a descendent of Net::Cmd, Net::FTP
supports the code(), message(), and ok() methods discussed in the previous section. The FTP
protocol's status codes are listed in RFC 959 (see Appendix D).
To the generic methods inherited from its ancestors, Net::FTP adds a large number of specialized
methods that support the special features of the FTP protocol. Only the common methods are listed
here. See the Net::FTP documentation for the full API.

$ftp = Net::FTP->new($host [,%options])

The new() method creates a Net::FTP object. The mandatory first argument is the domain name of the FTP
server you wish to contact. Additional optional arguments are a set of key/value pairs that set options for the
session, as shown in Table 6.1. For example, to connect to ftp.perl.org with hash marks enabled and a timeout
of 30 seconds, we could use this statement:
$ftp = Net::FTP('ftp.perl.org', Timeout=>30, Hash=>1);

Table 6.1. Net::FTP->new() Options

Option Description
Firewall Name of the FTP proxy to use when your machine is behind certain types of firewalls
BlockSize Block size of transfers (default 10240)
Port FTP port to connect to (default 21)
Timeout Timeout value, in seconds, for various operations (default 120 seconds)
Debug Debug level; set to greater than zero for verbose debug messages
Passive Use FTP passive mode for all file transfers; required by some firewalls
Hash Prints a hash mark to STDERR for each 1024 bytes of data transferred

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 FTP and Telnet 107

$success = $ftp->login([$username [,$password [,$account]]])

The login() method attempts to log in to the server using the provided authentication information. If no
username is provided, then Net::FTP assumes "anonymous". Ifno username or password is provided, then
Net::FTP looks up the authentication information in the user's .netrc file. If this is still not found, it generates
a password of the form "$user@", where $USER is your login name.
The optional $account argument is for use with some FTP servers that require an additional authentication
password to gain access to the filesystem after logging into the server itself. login() returns true if the login
was successful, and false otherwise.
See the Net::Netrc manual pages for more information on the .netrc file.
$type = $ftp->ascii
Puts the FTP object into ASCII mode. The server automatically performs newline translation during file transfers
(ending lines with CRLF on Windows machines, LF on UNIX machines, and CR on Macintoshes). This is
suitable for transferring text files.
The return value is the previous value of the transfer type, such as "binary." Note: ASCII mode is the default.
$type = $ftp->binary
Puts the FTP object into binary mode. The server will not perform translation. This is suitable for transferring
binary files such as images.
$success = $ftp->delete($file)
Deletes the file $file on the server, provided you have sufficient privileges to do this.
$success = $ftp->cwd([$directory])
Attempts to change the current working directory on the remote end to the specified path. If no directory is
provided, will attempt to change to the root directory " / ". Relative directories are understood, and you can
provide a pathname of ".." to move up one level.
$directory = $ftp->pwd
Returns the full pathname of the current working directory on the remote end.
$success = $ftp->rmdir($directory)
Remove the specified directory, provided you have sufficient privileges to do so.
$success = $ftp->mkdir($directory [,$parents])
Creates a new directory at the indicated path, provided you have sufficient privileges to do so. If $parents is
true, Net::FTP attempts to create all missing intermediate directories as well.
@items = $ftp->ls([$directory])
Gets a short-format directory list of all the files and subdirectories in the indicated directory or, if not specified,
in the current working directory. In a scalar context, ls() returns a reference to an array rather than the list
itself.
By default, each member of the returned list consists of just the bare file or directory name. However, since
the FTP daemon just passes the argument to the ls command, you are free to pass command-line arguments
to ls. For example, this returns a long listing:
@items = $ftp->ls('-lF');

@items = $ftp->dir([$directory])
Gets a long-format directory list of all the files and subdirectories in the indicated directory or, if not specified,
in the current working directory. In a scalar context, dir() returns a reference to an array rather than the list
itself.
In contrast to ls(), each member of the returned list is a line of a directory listing that provides the file modes,
ownerships, and sizes. It is equivalent to calling the ls command with the -lg options.
$success = $ftp->get($remote [,$local [, $offset]])
The get() method retrieves the file named $remote from the FTP server. You may provide a full pathname
or one relative to the current working directory.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 FTP and Telnet 108

The $local argument specifies the local pathname to store the retrieved file to. If not provided, Net::FTP
creates a file with the same name as the remote file in the current directory. You may also pass a filehandle
in $local, in which case the contents of the retrieved file are written to that handle. This is handy for sending
files to STDOUT:
$ftp->get('RECENT',\*STDOUT)

The $offset argument can be used to restart an interrupted transmission. It gives a position in the file that
the FTP server should seek before transmitting. Here's an idiom for using it to restart an interrupted transmis-
sion:
my $offset = (stat($file))[7] || 0;
$ftp->get($file,$file,$offset);

The call to stat() fetches the current size of the local file or, if none exists, 0. This is then used as the offset
to get().
$fh = $ftp->retr($filename)
Like get(), the retr() method can be used to retrieve a remote file. However, rather than writing the file to
a filehandle or disk file, it returns a filehandle that can be read from to retrieve the file directly. For example,
here is how to read the file named RECENT located on a remote FTP server without creating a temporary local
file:
$fh = $ftp->retr('REMOTE') or die "can't get file ",$ftp->
message;
print while <$fh>;

$success = $ftp->put($local [,$remote])

The put() method transfers a file from the local host to the remote host. The naming rules for $local and
$remote are identical to get(), including the ability to use a filehandle for $local.
$fh = $ftp->stor($filename)
$fh = $ftp->appe($filename)
These two methods initiate file uploads. The file will be stored on the remote server under the name $file-
name. If the remote server allows the transfer, the method returns a filehandle that can be used to transmit the
file contents. The methods differ in how they handle the case of an existing file with the specified name. The
stor(), method overwrites the existing file, and appe() appends to it.
$modtime = $ftp->mdtm($file)
The mdtm() method returns the modification time of the specified file, expressed as seconds since the epoch
(the same format returned by the stat() function). If the file does not exist or is not a plain file, then this
method returns undef. Also be aware that some older FTP servers (such as those from Sun) do not support
retrieval of modification times. For these servers mdtm() will return undef.
$size = $ftp->size($file)
Returns the size of the specified file in bytes. If the file does not exist or is not a plain file, then this method
returns undef. Also be aware that older FTP servers that do not support the SIZE command also return undef.

A Directory Mirror Script

Using Net::FTP, we can write a simple FTP mirroring script. It recursively compares a local directory
against a remote one and copies new or updated files to the local machine, preserving the directory
structure. The program preserves file modes in the local copy (but not ownerships) and also makes
an attempt to preserve symbolic links.
The script, called ftp_mirror.pl, is listed in Figure 6.2. To mirror a file or directory from a remote
server, invoke the script with a command-line argument consisting of the remote server's DNS name,
a colon, and the path of the file or directory to mirror. This example mirrors the file RECENT, copying
it to the local directory only if it has changed since the last time the file was mirrored:

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 FTP and Telnet 109

Figure 6.2. The ftp_mirror.pl script

%ftp_mirror.pl ftp.perl.org:/pub/CPAN/RECENT

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 FTP and Telnet 110

The next example mirrors the entire contents of the CPAN modules directory, recursively copying
the remote directory structure into the current local working directory (don't try this verbatim unless
you have a fast network connection and a lot of free disk space):
%ftp_mirror.pl ftp.perl.org:/pub/CPAN/

The script's command-line options include --user and --pass, to provide a username and password
for non-anonymous FTP, --verbose for verbose status reports, and --hash to print out hash marks
during file transfers.
Lines 1–5: Load modules—We load the Net::FTP module, as well as File::Path and Ge-
topt::Long. File::Path provides the mkpath() routine for creating a subdirectory with all its in-
termediate parents. Getopt::Long provides functions for managing command-line arguments.
Lines 6–19: Process command-line arguments—We process the command-line arguments, us-
ing them to set various global variables. The FTP host and the directory or file to mirror are
stored into the variables $HOST and $PATH, respectively.

Licensed by
Lines 20–23: Initialize the FTP connection—We call Net::FTP->new() to connect to the de-
sired host, and login() to log in. If no username and password were provided as command-
line arguments, we attempt an anonymous login. Otherwise, we attempt to use the authentica-
tion information to log in.

Stjepan Maric
After successfully logging in, we set the file transfer type to binary, which is necessary if we want
to mirror exactly the remote site, and we turn on hashing if requested.
Lines 24–26: Initiate mirroring—If all has gone well, we begin the mirroring process by calling
an internal subroutine do_mirror() with the requested path. When do_mirror() is done,

4218908
we close the connection politely by calling the FTP object's quit() method and exit.
Lines 27–36: do_mirror() subroutine—The do_mirror() subroutine is the main entry point
for mirroring a file or directory. When first called, we do not know whether the path requested
by the user is a file or directory, so the first thing we do is invoke a utility subroutine to make that
determination. Given a path on a remote FTP server, find_type() returns a single-character
code indicating the type of object the path points to, a "-" for an ordinary file, or a "d" for a
directory.
Having determined the type of the object, we split the path into the directory part (the prefix) and
the last component of the path (the leaf; either the desired file or directory). We invoke the FTP
object's cwd() method to change into the parent of the file or directory to mirror.
If the find_type() subroutine indicated that the path is a file, we invoke get_file() to mirror
the file. Otherwise, we invoke get_dir().
Lines 37–53: get_file() subroutine—This subroutine is responsible for fetching a file, but
only if it is newer than the local copy, if any. After fetching the file, we try to change its mode to
match the mode on the remote site. The mode may be provided by the caller; if not, we determine
the mode from within the subroutine.
We begin by fetching the modification time and the size of the remote file using the FTP object's
mdtm() and size() methods. Remember that these methods might return undef if we are
talking to an older server that doesn't support these calls. If the mode hasn't been provided by
the caller, we invoke the FTP object's dir() method to generate a directory listing of the re-
quested file, and pass the result to parse_listing(), which splits the directory listing line into
a three-element list consisting of the file type, name, and mode.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 FTP and Telnet 111

We now look for a file on the local machine with the same relative path and stat() it, capturing
the local file's size and modification time information. We then compare the size and modification
time of the remote file to the local copy. If the files are the same size, and the remote file is as
old or older than the local one, then we don't need to freshen our copy. Otherwise, we invoke
the FTP object's get() method to fetch the remote file. After the file transfer is successfully
completed, we change the file's mode to match the remote version.
Lines 54–73: get_dir() subroutine, recursive directory mirroring—The get_dir(), subrou-
tine is more complicated than get_file() because it must call itself recursively in order to
make copies of directories nested within it. Like get_file(), this subroutine is called with the
path of the directory and, optionally, the directory mode.
We begin by creating a local copy of the directory in the current working directory if there isn't
one already, using mkpath() to create intermediate directories if necessary. We then enter the
newly created directory with the chdir() Perl built-in, and change the directory mode if re-
quested.
We retrieve the current working directory at the remote end by calling the FTP object's pwd()
method. This path gets stored into a local variable for safekeeping. We now enter the remote
copy of the mirror directory using cwd().
We need to copy the contents of the mirrored directory to the local server. We invoke the FTP
object's dir() method to generate a full directory listing. We parse each line of the listing into
its type, pathname, and mode using the parse_listing() subroutine. Plain files are passed
to get_file(), symbolic_links() to make_link(), and subdirectories are passed re-
cursively to get_dir().
Having dealt with each member of the directory listing, we put things back the way they were
before we entered the subroutine. We call the FTP object's cwd(), routine to make the saved
remote working directory current, and chdir('..') to move up a level in the local directory
structure as well.
Lines 74–84: find_type() subroutine — find_type() is a not-entirely-satisfactory subrou-
tine for guessing the type of a file or directory given only its path. We would prefer to use the
FTP dir() method for this purpose, as in the preceding get_dir() call, but this is unreliable
because of slight differences in the way that the directory command works on different servers
when you pass it the path to a file versus the path to a directory.
Instead, we test whether the remote path is a directory by trying to cwd() into it. If cwd() fails,
we assume that the path is a file. Otherwise, we assume that the path is a directory. Note that
by this criterion, a symbolic link to a file is treated as a file, and a symbolic link to a directory is
treated as a directory. This is the desired behavior.
Lines 85–92: make_link() subroutine—The make_link() subroutine tries to create a local
symbolic link that mirrors a remote link. It works by assuming that the entry in the remote directory
listing denotes the source and target of a symbolic link, like this:
README.html -> index.html

We split the entry into its two components and pass them to the symlink(), built-in. Only
symbolic links that point to relative targets are created. We don't attempt to link to absolute paths
(such as "/CPAN") because this will probably not be valid on the local machine. Besides, it's a
security issue.
Lines 93–106: parse_listing() subroutine—The parse_listing() subroutine is invoked
by get_dir() to process one line of the directory listing retrieved by Net::FTP->dir(). This
subroutine is necessitated by the fact that the vanilla FTP protocol doesn't provide any other
way to determine the type or mode of an element in a directory listing. The subroutine parses

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 FTP and Telnet 112

the directory entry using a regular expression that allows variants of common directory listings.
The file's type code is derived from the first character of the symbolic mode field (e.g., the "d" in
drwxr-xr-x), and its mode from the remainder of the field. The filename is whatever follows
the date field.
The type, name, and mode are returned to the caller, after first converting the symbolic file mode
into its numeric form.
Lines 107–122: filemode() subroutine—This subroutine is responsible for converting a sym-
bolic file mode into its numeric equivalent. For example, the symbolic mode rw-r--r-- be-
comes octal 0644. We treat the setuid or setgid bits as if they were execute bits. It would be a
security risk to create a set-id file locally.
When we run the mirror script in verbose mode on CPAN, the beginning of the output looks like the
following:
% ftp_mirror.pl --verbose ftp.perl.org:/pub/CPAN
Getting directory CPAN/
Symlinking CPAN.html -> authors/Jon_Orwant/CPAN.html
Symlinking ENDINGS -> .cpan/ENDINGS
Getting file MIRRORED.BY
Getting file MIRRORING.FROM
Getting file README
Symlinking README.html -> index.html
Symlinking RECENT -> indices/RECENT-print
Getting file RECENT.html
Getting file ROADMAP
Getting file ROADMAP.html
Getting file SITES
Getting file SITES.html
Getting directory authors/
Getting file 00.Directory.Is.Not.Maintained.Anymore
Getting file 00upload.howto
Getting file 00whois.html
Getting file 01mailrc.txt.gz
Symlinking Aaron_Sherman -> id/ASHER
Symlinking Abigail -> id/ABIGAIL
Symlinking Achim_Bohnet -> id/ACH
Symlinking Alan_Burlison -> id/ABURLISON
...

When we run it again a few minutes later, we see messages indicating that most of the files are
current and don't need to be updated:
% ftp_mirror.pl --verbose ftp.perl.org:/pub/CPAN
Getting directory CPAN/
Symlinking CPAN.html -> authors/Jon_Orwant/CPAN.html
Symlinking ENDINGS -> .cpan/ENDINGS
Getting file MIRRORED.BY: not newer than local copy.
Getting file MIRRORING.FROM: not newer than local copy.
Getting file README: not newer than local copy.
...

The major weak point of this script is the parse_listing() routine. Because the FTP directory
listing format is not standardized, server implementations vary slightly. During development, I tested
this script on a variety of UNIX FTP daemons as well as on the Microsoft IIS FTP server. However,
this script may well fail with other servers. In addition, the regular expression used to parse directory
entries will probably fail on filenames that begin with whitespace.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 FTP and Telnet 113

Net::Telnet
FTP is the quintessential line-oriented server application. Every command issued by the client takes
the form of a single, easily parsed line, and each response from the server to the client follows a
predictable format. Many of the server applications that we discuss in later chapters, including POP,
SMTP, and HTTP, are similarly simple. This is because the applications were designed to interact
primarily with software, not with people.
Telnet is almost exactly the opposite. It was designed to interact directly with people, not software.
The output from a Telnet session is completely unpredictable, depending on the remote host's con-
figuration, the shell the user has installed, and the setup of the user's environment.
Telnet does some things that make it easy for human beings to use: It puts its output stream into a
mode that echoes back all commands that are sent to it, allowing people to see what they type, and
it puts its input stream into a mode that allows it to read and respond to one character at a time.
This allows command-line editing and full-screen text applications to work.
While these features make it easy for humans to use Telnet-based applications, it makes scripting
such applications a challenge. Because the Telnet protocol is more complex than sending com-
mands and receiving responses, you can't simply connect a socket to port 23 (Telnet's default) on
a remote machine and start exchanging messages. Before the Telnet client and server can talk,
they must engage in a handshake procedure to negotiate communications session parameters. Nor
is it possible for a Perl script to open a pipe to the Telnet client program because the Telnet, like
many interactive programs, expects to be opened on a terminal device and tries to change the
characteristics of the device using various ioctl() calls.
Given these factors, it is best not to write clients for interactive applications. Sometimes, though, it's
unavoidable. You may need to automate a legacy application that is available only as an interactive
terminal application. Or you may need to remotely drive a system utility that is only accessible in
interactive form. A classic example of the latter is the UNIX passwd program for changing users'
login passwords. Like Telnet, passwd expects to talk directly to a terminal device, and you must do
special work to drive it from a Perl script.
The Net::Telnet module provides access to Telnet-based services. With its facilities, you can log
into a remote host via the Telnet protocol, run commands, and act on the results using a straight-
forward pattern-matching idiom. When combined with the IO::Pty module, you can also use Net::Tel-
net to control local interactive programs.
Net::Telnet was written by Jay Rogers and is available on CPAN. It is a pure Perl module, and will
run unmodified on Windows and Macintosh systems. Although it was designed to interoperate with
UNIX Telnet daemons, it is known to work with the Windows NT Telnet daemon available on the
Windows NT Network Resource Kit CD and several of the freeware daemons.

A Simple Net::Telnet Example

Figure 6.3 shows a simple script that uses Net::Telnet. It logs into a host, runs the command ps -
ef to list all running processes, and then echoes the information to standard output.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 FTP and Telnet 114

Figure 6.3. remoteps.pl logs into a remote host and runs the "ps" command

Lines 1–3: Load modules—We load the Net::Telnet module. Because it is entirely object-orien-
ted, there are no symbols to import.
Lines 4–6: Define constants—We hard-code constants for the host to connect to, and the user
and password to log in as (no, this isn't my real password!). You'll need to change these as
appropriate for your system.
Line 7: Create a new Net::Telnet object—We call Net::Telnet->new() with the name of the
host. Net::Telnet attempts to connect to the host, returning a new Net::Telnet object if successful
or, if a connection could not be established, undef.
Line 8: Log in to remote host—We call the Telnet object's login() method with the username
and password. login() will attempt to log in to the remote system, and will return true if suc-
cessful.
Lines 9–10: Run the "ps" command—We invoke the cmd() method with the command to run,
in this case ps -ef. If successful, this method returns an array of lines containing the output of
the command (including the newlines). We print the result to standard output.
When we run the remoteps.pl script, there is a brief pause while the script logs into the remote host,
and then the output of the ps command appears, as follows:
% remoteps1.pl
UID PID PPID C STIME TTY TIME CMD
root 1 0 0 Jun26 ? 00:00:04 init
root 2 1 0 Jun26 ? 00:00:15 [kswapd]
root 3 1 0 Jun26 ? 00:00:00 [kflushd]
root 4 1 0 Jun26 ? 00:00:01 [kupdate]
root 34 1 0 Jun26 ? 00:00:01 /sbin/cardmgr
root 114 1 30 Jun26 ? 19:18:46 [kapmd]
root 117 1 0 Jun26 ? 00:00:00 [khubd]
bin 130 1 0 Jun26 ? 00:00:00 /usr/sbin/rpc.portmap
root 134 1 0 Jun26 ? 00:00:25 /usr/sbin/syslogd
...

Net::Telnet API
To accommodate the many differences between Telnet implementations and shells among oper-
ating systems, the Net::Telnet module has a large array of options. We only consider the most
frequently used of them here. See the Net::Telnet documentation for the full details.
Net::Telnet methods generally have both a named-argument form and a "shortcut" form that takes
a single argument only. For example, new() can be called either this way:
my $telnet = Net::Telnet->new('phage.cshl.org');

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 FTP and Telnet 115

or like this:
my $telnet = Net::Telnet->new(Host=>'phage.cshl.org', Timeout=>5);

We show both forms when appropriate.

The new() method is the constructor for Net::Telnet objects:

$telnet = Net::Telnet->new($host)
$telnet = Net::Telnet->new(Option1=>$value1,Option2=>$value2 ..)
The new() method creates a new Net::Telnet object. It may be called with a single argument containing the
name of the host to connect to, or with a series of option/ value pairs that provide finer control over the object.
new() recognizes many options, the most common of which are shown in Table 6.2.

Table 6.2. Net::Telnet->new() Arguments

Option Description Default Value
Host Host to connect to "localhost"
Port Port to connect to 23
Timeout Timeout for pattern matches, in seconds 10
Binmode Suppress CRLF translation false
Cmd_remove_mode Remove echoed command from input "auto"
Errmode Set the error mode "die"
Input_log Log file to write input to none
Fhopen Filehandle to communicate over none
Prompt Command-line prompt to match "/[\$%#>]$/"

The Host and Port options are the host and port to connect to, and Timeout is the period in seconds
that Net::Telnet will wait for an expected pattern before declaring a timeout.
Binmode controls whether Net::Telnet will perform CRLF translation. By default (Binmode=>0), ev-
ery newline sent from the script to the remote host is translated into a CRLF pair, just as the Telnet
client does it. Likewise, every CRLF received from the remote host is translated into a newline. With
Binmode set to a true value, this translation is suppressed and data is transmitted verbatim.
Cmd_remove_mode controls the removal of echoed commands. Most implementations of the Telnet
server echo back all user input. As a result, text you send to the server reappears in the data read
back from the remote host. If CMD_REMOVE_MODE is set to true, the first line of all data received
from the server will be stripped. A false value prevents stripping, and a value of "auto" allows
Net::Telnet to decide for itself whether to strip based on the "echo" setting during the initial Telnet
handshake.
Errmode determines what happens when an error occurs, typically an expected pattern not being
seen before the timeout. The value of Errmode can be one of the strings "die" (the default) or "return".
When set to "die", Net::Telnet dies on anerror, aborting your program. A value of "return" modifies
this behavior, so that instead of dying the failed method returns undef. You can then recover the
specific error message using errmsg(). In addition to these two strings, Errmode accepts either a
code reference or an array reference. Both of these forms are used to install custom handlers that
are invoked when an error occurs. The Net::Telnet documentation provides further information.
The value for Input_log should be a filename or a filehandle. All data received from the server is
echoed to this file or filehandle. Since the received data usually contains the echoed command, this
is a way to capture a transcript of the Net::Telnet session and is invaluable for debugging. If the
argument is a previously opened filehandle, then the log is written to that filehandle. Otherwise, the
argument is treated as the name of a file to open or create.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 FTP and Telnet 116

The Fhopen argument can be used to pass a previously opened filehandle to Net::Telnet for it to
use in communication. Net::Telnet will use this filehandle instead oftrying to open its own connection.
We use this later to coerce Net::Telnet into working across a Secure Shell link.
Prompt sets the regular expression that Net::Telnet uses to identify the shell command-line prompt.
This is used by the login() and cmd() methods to determine that the command ran to completion.
By default, Prompt is set to a pattern that matches the default sh, csh, ksh, and tcsh prompts.
Once a Net::Telnet object is opened you control it with several object modules:

$result = $telnet->login($username,$password)
$result = $telnet->login(Name => $username,
Password => $password,
[Prompt => $prompt,]
[Timeout=> $timeout])
The login() method attempts to log into the remote host using the provided username and password. In the
named-parameter form of the method call, you may override the values of Prompt and Timeout provided to
new().
If the Errmode is "die" and the login method encounters an error, the call aborts your script with an error
message. Otherwise, login() returns false.
$result = $telnet->print(@values)
Print a value or list of values to the remote host. A newline is automatically added for you unless you explicitly
disable this feature (see the Net::Telnet documentation for details). The method returns true if all of the data
was successfully written.
It is also possible to bypass Net::Telnet's character translation routines and write directly to the remote host
by using the Net::Telnet object as a filehandle:
print $telnet "ls -lF\015\012";

$result = $telnet->waitfor($pattern)
($before,$match) = $telnet->waitfor($pattern)
($before,$match) = $telnet->waitfor([Match=>$pattern,]
[String=>$string,]
[Timeout=>$timeout])
The waitfor() method is the workhorse of Net::Telnet. It waits up to Timeout seconds for the specified string
or pattern to appear on the data stream coming from the remote host. In a scalar context, waitfor() returns
a true value if the desired pattern was seen. In a list context, the method returns a two-element list consisting
of the data seen before the match and the matched string itself.

You can give waitfor() a regular expression to pattern match or a simple string, in which case
Net::Telnet uses index() to scan for it in incoming data. In the method's named-argument form,
use the Match argument for a pattern match, and String for a simple string match. You can specify
multiple alternative patterns or strings to match simply by providing more than one Match and/or
String arguments.
The strings used for MATCH must be correctly delimited Perl pattern match operators. For example,
"/bash> $/" and "m(bash> $)" will both work, but "bash> $" won't because of the absence
of pattern match delimiters.
In the single-argument form of waitfor(), the argument is a pattern match. The Timeout argument
may be used to override the default timeout value.
This code fragment will issue an ls -lF command, wait for the command line prompt to appear, and
print out what came before the prompt, which ought to be the output of the ls command:

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 FTP and Telnet 117

$telnet->print('ls -lF');
($before,$match) = $telnet->waitfor('/[$%#>] $/');
print $before;

To issue a command to the remote server and wait for a response, you can use one of several
versions of cmd():

$result = $telnet->cmd($command)
@lines = $telnet->cmd($command)
@lines = $telnet->cmd(String=>$command,
[Output=>$ref,] [Prompt=>$pattern,]
[Timeout=>$timeout,] [Cmd_remove_mode=>$mode]
The cmd() method is used to send a command to the remote host and return its output, if any. It is equivalent
to a print() of the command, followed by a waitfor() using the default shell prompt pattern.

In a scalar context, cmd() returns true if the command executed successfully, false if the method
timed out before the shell prompt was seen. In a list context, this method returns all the lines received
prior to matching the prompt.
In the named-argument form of the call, the Output argument designates either a scalar reference
or an array reference to receive the lines that preceded the match. The Prompt, Timeout, and
Cmd_remove_mode arguments allow you to override the corresponding settings.
Note that a true result from cmd() does not mean that the command executed successfully. It only
means that the command completed in the time allotted for it.
To receive data from the server without scanning for patterns, use get(), getline(), or get-
lines():

$data = $telnet->get([Timeout=>$timeout])
The get() method performs a timed read on the Telnet session, returning any data that is available. If no data
is received within the allotted time, the method dies if Errmode is set to "die" or returns undef otherwise. The
get() method also returns undef on end-of-file (indicating that the remote host has closed the Telnet ses-
sion). You can use eof() and timed_out() to distinguish these two possibilities.
$line = $telnet->getline([Timeout=>$timeout])
The getline() method reads the next line of text from the Telnet session. Like get(), it returns undef on
either a timeout or an end-of-file. You may change the module's notion of the input record separator using the
input_record_separator() method, described below.
@lines = $telnet->getlines([Timeout=>$timeout])
Return all available lines of text, or an empty list on timeout or end-of-file.

Finally, several methods are useful for debugging and for tweaking the communications session:

$msg = $telnet->errmsg
This method returns the error message associated with a failed method call. For example, after a timeout on
a waitfor(), errmsg() returns "pattern match timed-out."
$line = $telnet->lastline
This method returns the last line read from the object. It's useful to examine this value after the remote host
has unexpectedly terminated the connection because it might contain clues to the cause of this event.
$value = $telnet->input_record_separator([$newvalue])

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 FTP and Telnet 118

$value = $telnet->output_record_separator([$newvalue])
These two methods get and/or set the input and output record separators. The input record separator is used
to split input into lines, and is used by the getline(), getlines(), and cmd() methods. The output record
separator is printed at the end of each line output by the print() method. Both values default to \n.
$value = $telnet->prompt([$newvalue])
$value = $telnet->timeout([$newvalue])
$value = $telnet->binmode([$newvalue])
$value = $telnet->errmode([$newvalue])
These methods get and/or set the corresponding settings, and can be used to examine or change the defaults
after the Telnet object is created.
$telnet->close
The close() method severs the connection to the remote host.

A Remote Password-Changing Program

As a practical example of Net::Telnet, we'll develop a remote password-changing script named
change_passwd.pl. This script will contact each of the hosts named on the command line in turn
and change the user's login password. This might be useful for someone who has accounts on
several machines that don't share the same authentication database. The script is used like this:
% change_passwd.pl --old=mothergOOse --new=bopEEp chiron masdorf sceptre

This command line requests the script to change the current user's password on the three machines
chiron, masdorf, and sceptre. The script reports success or failure to change the password on each
of the indicated machines.
The script uses the UNIX passwd program to do its work. In order to drive passwd, we need to
anticipate its various prompts and errors. Here's a sample of a successful interaction:
% passwd
Changing password for lstein
Old password: xyzzy
Enter the new password (minimum of 5, maximum of 8 characters)
Please use a combination of upper and lower case letters and numbers.
New password: plugn
Re-enter new password: plugn
Password changed.

At the three password: prompts I typed my current and new passwords. However, the passwd
program turns off terminal echo so that the passwords don't actually display on the screen.
A number of errors may occur during execution of passwd. In order to be robust, the password-
changing script must detect them. One error occurs when the original password is typed incorrectly:
% passwd
Changing password for lstein
Old password: xyzyy
Incorrect password for lstein.
The password for lstein is unchanged.

Another error occurs when the new password doesn't satisfy the passwd program's criteria for a
secure, hard-to-guess password:
% passwd
Changing password for lstein
Old password: xyzzy
Enter the new password (minimum of 5, maximum of 8 characters)

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 FTP and Telnet 119

Please use a combination of upper and lower case letters and numbers.
New password: hi
Bad password: too short. Try again.
New password: aaaaaaaaaa
Bad password: a palindrome. Try again.
New password: 12345
Bad password: too simple. Try again.

This example shows several attempts to set the password, each one rejected for a different reason.
The common part of the error message is "Bad password." We don't have to worry about a third
common error in running passwd, which is failing to retype the password correctly at the confirmation
prompt.
The change_passwd.pl script is listed in Figure 6.4.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 FTP and Telnet 120

Licensed by
Stjepan Maric
4218908

Figure 6.4. Remote password-changing script

Lines 1–4: Load modules—We load Net::Telnet and the Getopt::Long module for command-line
option parsing.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 FTP and Telnet 121

Lines 5–12: Define constants—We create a DEBUG flag. If this is true, then we instruct the
Net::Telnet module to log all its input to a file named passwd.log. This file contains password
information, so be sure to delete it promptly. The USAGE constant contains the usage statement
printed when the user fails to provide the correct command-line options.
Lines 13–19: Parse command line options—We call GetOptions() to parse the command-
line options. We default to the current user's login name if none is provided explicitly using the
LOGNAME environment variable. The old and new password options are mandatory.
Line 20: Invoke change_passwd() subroutine—For each of the machines named on the com-
mand line, we invoke an internal subroutine named change_passwd(), passing it the name
of the machine, the user login name, and the old and new passwords.
Lines 21–41: change_passwd() subroutine—Most of the work happens in change_
passwd(). We begin by opening up a new Net::Telnet object on the indicated host, and then
store the object in a variable named $shell. If DEBUG is set, we turn on logging to a hard-coded
file. We also set errmode() to "return" so that Net::Telnet calls will return false rather than dying
on an error.
We now call login() to attempt to log in with the user's account name and password. If this fails,
we return with a warning constructed from the Telnet object's errmsg() routine.
Otherwise we are at the login prompt of the user's shell. We invoke the passwd command and wait
for the expected "Old password:" prompt. If the prompt appears within the timeout limit, we send
the old password to the server. Otherwise, we return with an error message.
Two outcomes are possible at this point. The passwd program may accept the password and prompt
us for the new password, or it may reject the password for some reason. We wait for either of the
prompts to appear, and then examine the match string returned by waitfor() to determine which
of the two patterns we matched. In the former case, we proceed to provide the new password. In
the latter, we return with an error message.
After the new desired password is printed (line 33), there are again two possibilities: passwd may
reject the proposed password because it is too simple, or it may accept it and prompt us to confirm
the new password. We handle this in the same way as before.
The last step is to print the new password again, confirming the change. We do not expect any errors
at this point, but we do wait for the "Password changed" confirmation before reporting success.
Because there is little standardization among passwd programs, this script is likely to work only with
those variants of UNIX that use a passwd program closely derived from the BSD version. To handle
other passwd variants, you will need to modify the pattern matches appropriately by including other
Match patterns in the calls to waitfor().
Running change_passwd.pl on a network of Linux systems gives output like this:
% change_passwd.pl --user=george --old=m00nd0g --new=swampH0und \
localhost pesto prego romano
Password changed for george on localhost.
Password changed for george on pesto.
Password changed for george on prego.
Password changed for george on romano.

While change_passwd.pl is running, the old and new passwords are visible to anyone who runs a
ps command to view the command lines of running programs. If you wish to use this script in pro-
duction, you will probably want to modify it so as to accept this sensitive information from standard
input. Another consideration is that the password information is passed in the clear, and therefore
vulnerable to network sniffers. The SSH-enabled password-changing script in the next section
overcomes this difficulty.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 FTP and Telnet 122

Using Net::Telnet for Non-Telnet Protocols

Net::Telnet can be used to automate interactions with other network servers. Often it is as simple
as providing the appropriate Port argument to the new() call. The Net::Telnet manual page provides
an example of this with the POP3 protocol, which we discuss in Chapter 8.
With help from the IO::Pty module, Net::Telnet can be used to automate more complicated network
services or to script local interactive programs. Like the standard Telnet client, the problem with
local interactive programs is that they expect access to a terminal device (a TTY) in order to change
screen characteristics, control the cursor, and so forth. What the IO::Pty module does is to create
a "pseudoterminal device" for these programs to use. The pseudoterminal is basically a bidirectional
pipe. One end of the pipe is attached to the interactive program; from the program's point of view,
it looks and acts like a TTY. The other end of the pipe is attached to your script, and can be used
to send data to the program and read its output.
Because the use of pseudoterminals is a powerful technique that is not well documented, we will
show a practical example. Many security-conscious sites have replaced Telnet and FTP with the
Secure Shell (SSH), a remote login protocol that authenticates and encrypts login sessions using
a combination of public key and symmetric cryptography. The change_passwd.pl script does not
work with sites that have disabled Telnet in favor of SSH, and we would like to use the ssh client to
establish the connection to the remote host in order to run the passwd command.
The ssh client emits a slightly different login prompt than Telnet. A typical session looks like this:
% ssh -l george prego
george@prego's password: *******
Last login: Mon Jul 3 08:20:28 2000 from localhost
Linux 2.4.01.
%

The ssh client takes an optional -l command-line switch to set the name of the user to log in as, and
the name of the remote host (we use the short name rather than the fully qualified DNS name in
this case). ssh prompts for the password on the remote host, and then attempts to log in.
To work with ssh, we have to make two changes to change_passwd.pl: (1) we open a pseudoter-
minal on the ssh client and pass the controlling filehandle to Net::Telnet->new() as the
Fhopen argument and (2) we replace the call to login() with our own pattern matching routine so
as to handle ssh's login prompt.
The IO::Pty module, available on CPAN, has a simple API:

$pty = IO::Pty->new
The new() method takes no arguments and returns a new IO::Pty pseudoterminal object. The returned object
is a filehandle corresponding to the controlling end of the pipe. Your script will ordinarily use this filehandle to
send commands and read results from the program you're driving.
$tty = $pty->slave
Given a pseudoterminal created with a call to IO::Pty->new(), the slave(), method returns the TTY half
of the pipe. You will ordinarily pass this filehandle to the program you want to control.

Figure 6.5 shows the idiom for launching a program under the control of a pseudoterminal. The
do_cmd() subroutine accepts the name of a local command to run and a list of arguments to pass
it. We begin by creating a pseudoterminal filehandle with IO::Pty->new() (line 3). If successful,
we fork(), and the parent process returns the pseudoterminal to the caller. The child process,

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 FTP and Telnet 123

however, has a little more work to do. We first detach from the current controlling TTY by calling
POSIX::setsid() (see Chapter 10 for details). The next step is to recover the TTY half of the
pipe by calling the IO::Pty object's slave(), method, and then close the pseudoterminal half (lines
7–8).

Figure 6.5. Launching a program in a pseudo-tty

We now reopen STDIN, STDOUT, and STDERR on the new TTY object using fdopen(), and close
the now-unneeded copy of the filehandle (lines 9–12). We make STDOUT unbuffered and invoke
exec() to run the desired command and arguments. When the command runs, its standard input
and output will be attached to thenew TTY, which in turn will be attached to the pseudo-tty controlled
by the parent process.
With do_cmd() written, the other changes to change_passwd.pl are relatively minor. Figure
6.6 shows the revised script written to use the ssh client, change_passwd_ssh.pl.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 FTP and Telnet 124

Figure 6.6. Changing passwords over a Secure Shell connection

Lines 1–6: Load modules—We load IO::Pty and the setsid() routine from the POSIX module.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 FTP and Telnet 125

Lines 7–23: Process command-line arguments and call change_passwd()—The only change
here is a new constant, PROMPT, that contains the pattern match that we will expect from the
user's shell command prompt.
Lines 24–27: Launch ssh subprocess—We invoke do_cmd() to run the ssh program using the
requested username and host. If do_cmd() is successful, it returns a filehandle connected to
the pseudoterminal driving the ssh subprocess.
Lines 28–31: Create and initialize Net:: Telnet object —In the change_passwd() routine, we
create a new Net::Telnet object, but now instead of allowing Net::Telnet to open a connection
to the remote host directly, we pass it the ssh filehandle using the Fhopen argument. After
creating the Net::Telnet object, we configure it by putting it into binary mode with binmode(),
setting the input log for debugging, and setting the error mode to "return". The use of binary
mode is a small but important modification of the original script. Since the SSH protocol termi-
nates its lines with a single LF character rather than CRLF pairs, the default Net::Telnet CRLF
translation is inappropriate.
Lines 32–34: Log in—Instead of calling Net::Telnet's built-in login() method, which expects
Telnet-specific prompts, we roll our own by waiting for the ssh "password:" prompt and then
providing the appropriate response. We then wait for the user's command prompt. If, for some
reason, this fails, we return with an error message.
Lines 35–49: Change password—The remainder of the change_passwd() subroutine is iden-
tical to the earlier version.
Lines 50–65: do_cmd() subroutine—This is the same subroutine that we examined earlier.
The change_passwd_ssh.pl program now uses the Secure Shell to establish connections to the
indicated machines and change the user's password. This is a big advantage over the earlier ver-
sion, which was prone to network eavesdroppers who could intercept the new password as it passed
over the wire in unencrypted form. On multiuser systems you will still probably want to modify the
script to read the passwords from standard input rather than from the command line.
For completeness, Figure 6.7 lists a routine, prompt_for_passwd(i), that uses the UNIX stty program
to disable command-line echo temporarily while the user is typing the password. You can use it like
this:
$old = get_password('old password');
$new = get_password('new password');

Figure 6.7. Disabling echo while prompting for a password

A slightly more sophisticated version of this subroutine, which takes advantage of the Term::Read-
Key module, if available, appears in Chapter 20.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 FTP and Telnet 126

The Expect Module

An alternative to Net::Telnet is the Expect module, which provides similar services for talking to local
and remote processes that expect human interaction. Expect implements a rich command language,
which among other things can pause the script and prompt the user for information, such as pass-
words. Expect can be found on CPAN.

Summary
This chapter covered Perl client modules for two of the most widespread application-level protocols,
FTP and Telnet. Together they illustrate the extremes of application protocols, from a rigidly defined
command language designed to interact with client programs to a loose interactive environment
designed for people.
The Net::FTP module allows you to write scripts to automatically connect to FTP sites, explore their
holdings, and selectively download or upload files. Net::Telnet's flexible pattern matching facilities
give you the ability to write scripts to automate processes that were designed primarily for the con-
venience of people rather than software.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 127

Chapter 7. SMTP: Sending Mail

E-mail is one of the oldest Internet applications, and it should come as no surprise that many client-
side modules have been written to enable Perl to interoperate with the mail system. Various modules
allow you to send and receive mail, manipulate various mailbox formats, and work with MIME at-
tachments.

Introduction to the Mail Modules

If you examine the "Internet Mail and Utilities" section of CPAN, you'll find a bewildering array of
similarly named modules. This is a quick guide to the major components.
Net::SMTP—This allows you to interact directly with mail transport daemons in order to send Internet
mail via the Simple Mail Transport Protocol (SMTP). The module also provides access to some of
the other functions of these daemons, such as expanding e-mail aliases.
MailTools—This is a higher-level way to create outgoing e-mail. It uses a variety of local mailer
packages to do the grunt work.
MIME-Tools—This is a package of modules for creating, decoding, and manipulating Multipurpose
Internet Mail Extensions (MIME), commonly known as attachments.
Net::POP3—This is a client for the Post Office Protocol version 3 (POP3). It provides a way to
retrieve a user's stored mail messages from a central maildrop.
Net::IMAP—This is a client module for the Internet Message Access Protocol (IMAP), a sophisti-
cated protocol for storing and synchronizing e-mail messages between mail drops and clients.
This chapter discusses tools involved in creating outgoing mail, including Net::SMTP and MIME-
Tools. Chapter 8 covers the Net::POP3 and Net::IMAP modules, both of which are involved in pro-
cessing incoming mail.

Net::SMTP
Net::SMTP operates at the lowest level of the e-mail access modules. It interacts directly with the
SMTP daemons to transmit e-mail across the Internet. To use it effectively, you must know a bit
about the innards of SMTP. The payoff for this added complexity is that Net::SMTP is completely
portable, and works as well from Macintoshes and Windows machines as from UNIX systems.

The SMTP Protocol

When a client e-mail program wants to send mail, it opens a network connection to a mail server
somewhere using the standard SMTP port, number 25. The client conducts a brief conversation
with the server, during which time it establishes its identity, announces that it wishes to send mail
to a certain party, and transmits the e-mail message. The server then takes care of seeing that the
message gets where it needs to go, whether by delivering it to a local user or by transmitting the
message to another server somewhere else in the world.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 SMTP: Sending Mail 128

The language spoken by SMTP servers is a simple human-readable line-oriented protocol. Figure
7.1 shows the interaction needed to send a complete e-mail manually using Telnet as the client (the
client's input is in bold).

Figure 7.1. A chat with an SMTP daemon

After connecting to the SMTP port, the server sends us a code "220" message containing a banner
and greeting. We issue a HELO command, identifying the hostname of the client machine, and the
server responds with a "250" message, which essentially means "OK."
After this handshake, we are ready to send some mail. We issue a MAIL command with the argu-
ment <From: sender's address>, to designate the sender. If the sender is OK, the server
responds with another "250" reply. We now issue a RCPT ("recipient") command with the argument
<To: recipient's address> to indicate the recipient. The server again acknowledges the
command. Some SMTP servers have restrictions on the senders and recipients they will service;
for example, they may refuse to relay e-mail to remote domains. Inthis case, they respond with a
variety of error codes in the 500 to 599 range. It is possible to issue multiple RCPT commands for
e-mail that has several recipients at the site(s) served by the SMTP server.
Having established that the sender and recipient(s) are OK, we send the DATA command. The server
responds with a message prompting us for the e-mail message. The server will accept lines of input
until it sees a line containing just a ".".
Internet mail has a standard format consisting of a set of header lines, ablank line, and the body of
the message. Even though we have already specified the sender and recipient, we must do so again
in order to create a valid e-mailmessage. A minimal mail header has a From: field, indicating the
sender, a To: field, indicating the recipient, and a Subject: field. Other standard fields, such as the
date, are filled in automatically by the mail daemon.
We add a blank line to separate the header from the body, enter the e-mail message text, and
terminate the message with a dot. The server's code 250 acknowledgment indicates that the mes-
sage was queued successfully for delivery.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 SMTP: Sending Mail 129

We could now send additional messages by issuing further MAIL commands, but instead we dis-
connect politely by issuing the QUIT command. The full specification of the SMTP protocol can be
found in RFC 821. The standard format for Internet mail headers is described in RFC 822.

The Net::SMTP API

Net::SMTP mirrors the SMTP protocol very closely. Net::SMTP is part of the libnet utilities and is
available on CPAN. Like the other Net::* modules, it uses an object-oriented interface in which you
establish a connection with a particular mailer daemon, yielding a Net::SMTP object. You then call
the SMTP object's methods to send commands to the server. Like Net::FTP (but unlike Net::Telnet),
Net::SMTP inherits from Net::Cmd and IO::Socket::INET, allowing you to use the Net::Cmd mes-
sage() and code() methods to retrieve the most recent message and numeric status code from
the server. All the low-level IO::Socket and IO::Socket::INET methods are also inherited.
To create a new Net::SMTPobject, use the new() constructor:

Table 7.1. Net::SMTP->new() Arguments

Option Description Default
Hello The domain name to use in the HELO command. Name of local host
Timeout Seconds to wait for response from server. 120
Debug Turn on verbose debugging information. undef
Port Numeric or symbolic name of port to connect to. 25

$smtp = Net::SMTP->new([$host] [,$opt1=>$val1, $opt2=>$val2...])

The new() method establishes a connection to an SMTP server and returns a new Net::SMTP object. The
first optional argument is the name of the host to contact, and will default to the mail exchanger configured into
Net::Config when libnet was first installed. The options are a series of named arguments. In addition to the
options recognized by the IO::Socket::INET superclass, the arguments shown in Table 7.1 are possible.

If the connection is refused (or times out), new() returns false. Here's an example of contacting the
mail server for the cshl.org domain with a timeout of 60 seconds.
$smtp = Net::SMTP->new('mail.cshl.org',Timeout=>60);

Once the object is created, you can send or retrieve information to the server by calling object
methods. Some are quite simple:

$banner = $smtp->banner()
$domain = $smtp->domain()
Immediately after connecting to an SMTP server, you can retrieve the banner and/or domain name with which
it identified by calling these two methods.

To send mail, you will first call the mail() and recipient() methods to set up the exchange:

$success = $smtp->mail($address [,\%options])

The mail() method issues a MAIL command to the server. The required first argument is the address of the
sender. The optional second argument is a hash reference containing various options to be passed to servers
that support the Extended Simple Mail Transport Protocol, or ESMTP. These are rarely needed; see the
Net::SMTP documentation for details.
The address may be in any of the forms accepted by e-mail clients, including [email protected],
<[email protected]>, John Doe <[email protected]>, and [email protected] (John Doe).

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 SMTP: Sending Mail 130

If successful, this method returns a true value. Otherwise, it returns undef, and the inherited message()
method can be used to return the text of the error message.
$success = $smtp->recipient($address1,$address2,$address3,...)
@ok_addr = $smtp-> recipient($addr1,$addr2,$addr3,...,{SkipBad=>1})
The recipient() method issues an RCPT command to the server. The arguments are a list of valid e-mail
addresses to which the mail is to be delivered. The list of addresses may be followed by a hash reference
containing various options.

The addresses passed to recipient() must all be acceptable to the server, or the entire call will
return false. To modify this behavior, pass the option SkipBad in the options hash. The module now
ignores addresses rejected by the server, and returns the list of accepted addresses as its result.
For example:
@ok=$smtp->recipient('[email protected]','[email protected]',{SkipBad=>1})

Licensed by
Provided that the server has accepted the sender and recipient, you may now commence sending
the message text using the data(), datasend(), and dataend() methods.

Stjepan Maric
$success = $smtp->data([$text])
The data() method issues a DATA command to the server. If called with a scalar argument, it transmits the
value of the argument as the content (header and body) of the e-mail message. If you wish to send the message
one chunk at a time, call data without an argument and make a series of calls to the datasend() method.
This method returns a value indicating success or failure of the command.

4218908
$success = $smtp->datasend(@data)
After calling data() without an argument, you may call datasend() one or more times to send lines of e-
mail text to the server. Lines starting with a dot are automatically escaped so as not to terminate the trans-
mission prematurely.
You may call datasend() with an array reference, if you prefer. This method and dataend() are both
inherited from the Net::Cmd base class.
$success = $smtp->dataend
When your e-mail message is sent, you should call dataend() to transmit the terminal dot. If the message
was accepted for delivery, the return value is true.

Two methods are useful for more complex interactions with SMTPservers:

$smtp->reset
This sends an RSET command to the server, aborting mail transmission operations in progress. You might call
this if one of the desired recipients is rejected by the server; it resets the server so you can try again.
$valid = $smtp->verify($address)
@recipients = $smtp->expand($address)
The expand() and verify() methods can be used to check that a recipient address is valid prior to trying
to send mail. verify() returns true if the specified address is accepted.
expand() does something more interesting. If the address is valid, it expands it into one or more aliases, if
any exist. This can be used to identify forwarding addresses and mailing list recipients. The method returns a
list of aliases or, if the specified address is invalid, an empty list. For security reasons, many mail administrators
disable this feature, in which case, the method returns an empty list.

Finally, when you are done with the server, you will call the quit() method:

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 SMTP: Sending Mail 131

$smtp->quit
This method politely breaks the connection with the server.

Using Net::SMTP
With Net::SMTP we can write a one-shot subroutine for sending e-mail. The mail() subroutine
takes two arguments: the text of an e-mail message to send (required), and the name of the SMTP
host to use (optional). Call it like this:
$msg = ≪ 'END';
From: John Doe <[email protected]>
To: L Stein <[email protected]>
Cc: [email protected], [email protected]
Subject: hello there

This is just a simple e-mail message.

Nothing to get excited about.

Regards, JD
END

mail($msg,'presto.lsjs.org') or die "arggggh!";

We create the text of the e-mail message using the here-is (≪) syntax and store it in the variable
$msg. The message must contain an e-mail header with (at a minimum) the From: and To: fields.
We pass the message to the mail() subroutine, which extracts the sender and recipient fields and
invokes Net::SMTP to do the dirty work. Figure 7.2 shows how mail() works.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 SMTP: Sending Mail 132

Figure 7.2. A simple subroutine for sending e-mail

Lines 1–9: Parse the mail message—We split the message into the header and the body by
splitting on the first blank line. Header fields frequently contain continuation lines that begin with
a blank, so we fold those into a single line.
We parse the header into a hash using a simple pattern match, and store the From: and To:
fields in local variables. The To: field can contain multiple recipients, so we isolate the individual
addressees by splitting on the comma character (this will fail in the unlikely case that any of the
addresses contain commas). We do likewise if the header contained a Cc: field.
Lines 10–16: Send messages—We create a new Net::SMTP object and call its mail(), and
recipient() methods to initiate the message. The call to recipient() uses the SkipBad
option so that the method will try to deliver the mail even if the server rejects some of the recip-
ients. We compare the number of recipients the server accepted to the number we attempted,
returning from the subroutine if none were accepted, or just printing a warning if only some were
rejected.
We call data() to send the complete e-mail message to the server, and quit() to terminate
the connection.
Although this subroutine does its job, it lacks some features. For example, it doesn't handle the Bcc:
field, which causes mail to be delivered to a recipient without that recipient appearing in the header.
The MailTools module, described next, corrects the deficiencies.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 SMTP: Sending Mail 133

MailTools
The MailTools module, also written by Graham Barr, is a high-level object-oriented interface to the
Internet e-mail system. MailTools, available on CPAN, provides a flexible way to create and ma-
nipulate RFC 822-compliant e-mail messages. Once the message is composed, you can send it off
using SMTP or use one of several UNIX command-line mailer programs to do the dirty work. This
might be necessary on a local network that does not have direct access to an SMTP server.

Using MailTools
A quick example of sending an e-mail from within a script will give you the flavor of the MailTools
interface (Figure 7.3).

Figure 7.3. Sending e-mail with Mail::Internet

Lines 1–2: Load modules—We bring in the Mail::Internet module. It brings in other modules that
it needs, including Mail::Header, which knows how to format RFC 822 headers, and Mail::Mailer,
which knows how to send mail by a variety of methods.
Lines 3–8: Create header—We call Mail::Header->new to create a new header object, which
we will use to build the RFC 822 header. After creating the object, we call its add() method
several times to add the From:, To:, Cc:, and Subject: lines. Notice that we can add the same
header multiple times, as we do with the Cc: line. Mail::Header will also insert other required
RFC 822 headers on its own.
Lines 9–13: Create body—We create the body text, which is just a block of text.
Lines 14–16: Create the Mail::Internet object—We now create a new Mail::Internet object by
calling the package's new() method. The named arguments include Header, to which we pass
the header object that we just created, and Body, which receives the body text. The Body ar-
gument expects an array reference containing discrete lines of body text, so we wrap $body
into an anonymous array reference. Modify, the third argument to new(), flags Mail::Internet
that it is OK to reformat the header lines to meet restrictions on line length that some SMTP
mailers impose.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 SMTP: Sending Mail 134

Line 17: Send mail—We call the newly created Mail::Internet object's send() method with an
argument indicating the sending method to use. The "sendmail" argument indicates that Mail::In-
ternet should try to use the UNIX sendmail program to deliver the mail.
Although at first glance Mail::Internet does not hold much advantage over the Net::SMTP-based
mail() subroutine we wrote in the previous section, the ability to examine and manipulate
Mail::Header objects gives MailTools its power. Mail::Header is also the base class for MIME::Head,
which manipulates MIME-compliant e-mail headers that are too complex to be handled manually.

Mail::Header
E-mail headers are more complex than they might seem at first. Some fields occur just once, others
occur multiple times, and some allow multiple values to be strung together by commas or another
delimiter. A field may occupy a single line, or may be folded across multiple lines with leading white-
space to indicate the presence of continuation lines. The mail system also places an arbitrary limit
on the length of a header line. Because of these considerations, you should be cautious of con-
structing e-mail headers by hand for anything much more complicated than the simple examples
shown earlier.
The Mail::Header module simplifies the task of constructing, examining, and modifying RFC 822
headers. Once constructed, a Mail::Header object can be passed to Internet::Mail for sending.
Mail::Header controls the syntax but not the content of the header, which means that you can con-
struct a header with fields that are not recognized by the mail subsystem. Depending on the mailer,
a message with invalid headers might make it through to its destination, or it might get bounced. To
avoid this, be careful to limit headers to the fields listed in the SMTP and MIME RFCs (RFC 822
and RFC 2045, respectively). Table 7.2 gives some of the common headers in e-mail messages.
Fields that begin with X- are meant to be used as extensions. You can safely build a header con-
taining any number of X- fields, and the fields will be passed through unmodified by the mail system.
For example:
$header = Mail::Header->new(Modify=>1);
$header->add('X-Mailer' => "Fido's mailer v1.0");
$header->add('X-HiMom' => 'Hi mom!');

Mail::Header supports a large number of methods. The following list gives the key methods. To
create a new object, call the Mail::Header new() method.

Table 7.2. Mail::Header Fields

Bcc Date Received Sender
Cc From References Subject
Comments Keywords Reply-To To
Content-Type Message-ID Resent-From X-*
Content-Transfer-Encoding MIME-Version Resent-To
Content-Disposition Organization Return-Path

$head = Mail::Header->new([$arg] [,@options])

The new() method is the constructor for the Mail::Header class. Called with no arguments, it creates a new
Mail::Header object containing an empty set of headers.
The first argument, if provided, is used to initialize the object. Two types of arguments are accepted. You may
provide an open filehandle, in which case the headers are read from the indicated file, or you may provide an
array reference, in which case the headers are read from the array. In either case, each line must be a correctly
formatted e-mail header, such as "Subject: this is a subject."

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 SMTP: Sending Mail 135

@options, if provided, is a list of named arguments that control various header options. The one used most
frequently is Modify, which if set true allows Mail::Header to reformat header lines to make them fully RFC 822-
compliant. For example:
open HEADERS,"./mail.msg";
$head = Mail::Header(\*HEADERS, Modify=>1);

Once a Mail::Header object is created, you may manipulate its contents in several ways:

$head->read(FILEHANDLE)
As an alternative way to populate a header object, you can create an empty object by calling new() with no
arguments, and then read in the headers from a filehandle using read().
$head->add($name,$value [,$index])
$head->replace($name,$value [,$index])
$head->delete($name [,$index])
The add(), replace(), and delete() methods allow you to modify the Mail::Header object. Each takes
the name of the field to operate on, the value for the field, and optionally an index that selects a member of a
multivalued field.
The add() method appends a field to the header. If $index is provided, it inserts the field into the indicated
position; otherwise, it appends the field to the end of the list.
The replace() method replaces the named field with the indicated value. If the field is multivalued, then
$index is used to select which value to replace; otherwise, the first field is replaced.
Delete() removes the indicated field.

All three of these methods accept a shortcut form that allows you to specify the field name and value
in a single line. This shortcut allows you to replace the Subject line like this:
$head->replace('Subject: returned to sender')

rather than like this:

$head->replace(Subject => 'returned to sender')

To retrieve information about a header object, you use get() to get the value of a single field, or
tags() and commit() to get information about all the available fields.

$line = $head->get($name [,$index])

@lines = $head->get($name)
The get() method retrieves the named field. In a scalar context, it returns the text form of the first indicated
field; in a list context it returns all such fields. You may provide an index in order to select a single member of
a multivalued field.
A slightly annoying feature of get() is that the retrieved field values contain the terminating newlines. These
must be removed manually with chomp().
@fields = $head->tags
Returns the list of field names (which the Mail::Header documentation calls "tags").
$count = $head->count($tag)
Returns the number of times the given tag appears in the header.

Finally, three methods are useful for exporting the header in various forms:

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 SMTP: Sending Mail 136

$string = $head->as_string
Returns the entire header as a string in the form that will appear in the message.
$hashref = $head->header_hashref([\%headers])
The header_hashref() method returns the headers as a hash reference. Each key is the unique name of
a field, and each value is an array reference containing the header's contents. This form is suitable for passing
to Mail::Mailer->open(), as described later in this chapter.
You may also use this method to set the header by passing it a hash reference of your own devising. The
composition of \%headers is similar to header_hashref()'s result, but the hash values can be simple
scalars if they are not multivalued.
$head->print([FILEHANDLE])
Prints header to indicated filehandle or, if not specified, to STDOUT. Equivalent to:
print FILEHANDLE $head->as_string

Mail::Internet
The Mail::Internet class is a high-level interface to e-mail. It allows you to create messages, ma-
nipulate them in various ways, and send them out. It was designed to make it easy to write autor-
esponders and other mail-processing utilities.
As usual, you create a new object using the Mail::Internet new() method:

$mail = Mail::Internet->new([$arg] [,@options])

The new() method constructs a new Mail::Internet object. Called with no arguments, it creates an empty object,
which is ordinarily not particularly useful. Otherwise, it initializes itself from its arguments in much the same
way as Mail::Header. The first argument, if provided, may be either a filehandle or an array reference. In the
former case, Mail::Internet tries to read the headers and body of the message from the filehandle. If the first
argument is an array reference, then the new object initializes itself from the lines of text contained in the array.
@options is a list of named-argument pairs. Several arguments are recognized. Header designates a
Mail::Header object to use with the e-mail message. If present, this header is used, ignoring any header in-
formation provided in $arg. Similarly, Body points to an array reference containing the lines of the e-mail body.
Any body text provided by the $arg input is ignored.

Once the object is created, several methods allow you to examine and modify its contents:

$arrayref = $mail->body
The body() method returns the body of the e-mail message as a reference to an array of lines of text. You
may manipulate these lines to modify the body of the message.
$header = $mail->head
The head() method returns the message's Mail::Header object. Modifying this object changes the message
header.
$string = $mail->as_string
$string = $mail->as_mbox_string
The as_string() and as_mbox_string() methods both return the message (both header and body) as
a single string. The as_mbox_string() function returns the message in a format suitable for appending to
UNIX mbox-format mailbox files.
$mail->print([FILEHANDLE})
$mail->print_header([FILEHANDLE})

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 SMTP: Sending Mail 137

$mail->print_body([FILEHANDLE})
These three methods print all or part of the message to the designated filehandle or, if not otherwise specified,
STDOUT.

Several utility methods perform common transformations on the message's contents:

$mail->add_signature([$file])
$mail->remove_sig([$nlines])
These two methods manipulate the signatures that are often appended to the e-mail messages. The
add_signature() function appends the signature contained in $file to the bottom of the e-mail message.
If $file is not provided, then the method looks for the file $ENV{HOME}/.signature.
remove_sig() scans the last $nlines of the message body looking for a line consisting of the characters
"--", which often sets the body off from the signature. The line and everything below it is removed. If not speci-
fied, $nlines defaults to 10.
$reply = $mail->reply
The reply() method creates a new Mail::Internet object with the header initialized to reply to the original
message, and the body text indented. This is suitable for autoreply applications.

Finally, the send() method sends the message via the e-mail system:

$result = $mail->send([$method] [,@args])

The send() method converts message into a string and sends it using Mail::Mailer. The $method and
@args arguments select and configure the mailing method. The next section describes the available methods
—mail, sendmail, smtp, and test.
If no method is specified, send() chooses a default that should work on your system.

A Mail Autoreply Program

With Mail::Internet, we can easily write a simple autoreply program for received e-mail (Figure
7.4). The autoreply.pl script is similar to the venerable UNIX vacation program. When it receives
mail, it checks your home directory for the existence of a file named .vacation. If the file exists, the
script replies to the sender using the contents of the file. Otherwise, the program does nothing.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 SMTP: Sending Mail 138

Figure 7.4. An autoreply program

This autoreply script takes advantage of a feature of the UNIX mail system that allows incoming e-
mail to be piped to a program. Provided that you're using such a system, you may activate the script
by creating a .forward file in your home directory that contains lines like the following:
lstein
| /usr/local/bin/autoreply.pl

Replace the first line with your login name, and the second with the path to the autoreply script. This
tells the mail subsystem to place one copy of the incoming mail in the user-specific inbox, and to
send another copy to the standard input of the autoreply.pl script.
Let's step through autoreply.pl.
Lines 1–3: Load modules—We turn on strict type checking and load the Mail::Internet module.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 SMTP: Sending Mail 139

Lines 4–7: Define constants—One problem with working with programs run by the mailer dae-
mon is that the standard user environment isn't necessarily set up. This means that
$ENV{HOME} and other standard environment variables may not exist. Our first action, therefore,
is to look up the user's home directory and login name and store them in appropriate constants.
Lines 4 and 5 use the getpwuid() function to retrieve this information. We then use the
HOME constant to find the locations of the .vacation and .signature files.
Lines 8–9: Create a Mail::Internet object—We check that the .vacation file is present and, if
it is not, exit. Otherwise, we create a new Mail::Internet object initialized from the message sent
us on STDIN.
Lines 10–19: Check that the message should be replied to—We shouldn't autoreply to certain
messages, such as those sent to us in the Cc: line, or those distributed to a mailing list. Another
type of message we should be very careful not to reply to are bounced messages; replying to
those has the potential to set up a nasty infinite loop. The next section of the code tries to catch
these situations.
We recover the header by calling the Mail::Internet object's head() method, and perform a
series of pattern matches on its fields. First we check that our username is mentioned on the
To: line. If not, we may be receiving this message as a Cc: or as a member of a mailing list. We
next check the Precedence: field. If it's "bulk," then this message is probably part of a mass
mailing. If the Subject: line contains the strings "returned mail" or "bounced mail", or if the sender
is the mail system itself (identified variously as "mailer daemon," "mail subsystem," or "post-
master"), then we are likely dealing with returned mail and we shouldn't reply or risk setting up
a loop. In each of these cases, we just exit normally.
Lines 20–21: Generate reply—To create a new message initialized as a reply to the original, we
call the mail message object's reply() method.
Lines 22–26: Prepend vacation message to text—The reply() method will have created body
text consisting of the original message quoted and indented. We prepend the contents of
the .vacation file to this. We open the contents of .vacation, call the mail message's
body() method to return a reference to the array of body lines, and then use unshift() to
insert the contents of .vacation in front of the body. We could replace the body entirely, if we
preferred.
Lines 27–28: Add signature—We call the reply's add_signature() method to append the
contents of the user's signature file, if any, to the bottom of the message body.
Lines 29–30: Send message—We call the reply's send() method to send the message by the
most expedient means.
Here is an example of a reply issued by the autoreply.pl script in response to the sample mes-
sage we composed with Net::SMTP in the previous section. The text at the top came from
~/.vacation and the signature at the bottom from ~/.signature. The remainder is quoted from
the original message.
To: John Doe <[email protected]>
From: L Stein <[email protected]>
Subject: Re: hello there
Date: Fri, 7 Jul 2000 08:12:17 -0400
Message-Id: <200007071212.IAA12128@pesto>

Hello,

I am on vacation from July 6-July 12, and will not be reading my

e-mail. I will respond to this message when I return.

Lincoln

John Doe <[email protected]> writes:

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 SMTP: Sending Mail 140

> This is just a simple e-mail message.

> Nothing to get excited about.

> Regards, JD
--

======================================================================
Lincoln D. Stein Cold Spring Harbor Laboratory
======================================================================

If you adapt this autoreply program to your own use, you might want to check the size of the quoted
body and delete it if it is unusually large. Otherwise, you might inadvertently echo back a large binary
enclosure.
For complex e-mail-processing applications, you should be sure to check out the procmail program,
which uses a special-purpose programming language to parse and manipulate e-mail. A number of
sophisticated applications have been written on top of procmail, including autoresponders, mailing
list generators, and filters for spam mail.

Mail::Mailer Licensed by
The last component of MailTools that we consider is Mail::Mailer, which is used internally by
Mail::Internet to deliver mail. Mail::Mailer provides yet another interface for sending Internet mail.

Stjepan Maric
Although it doesn't provide Mail::Internet's header- and body-handling facilities, I find it simpler and
more elegant to use in most circumstances.
Unlike Net::SMTP and Mail::Internet, which use object methods to compose and send mail, the
Mail::Mailer object acts like a filehandle. This short code fragment shows the idiom:
use Mail::Mailer;
my $mailer = Mail::Mailer->new;
$mailer->open( {To 4218908
=> '[email protected]',
From => '[email protected]',
CC => ['[email protected]','[email protected]'],
Subject => 'hello there'});
print $mailer "This is just a simple e-mail message.\n";
print $mailer "Nothing to get excited about.\n\n";
print $mailer "Regards, JD\n";
$mailer->close;

After creating the object with new(), we initialize it by calling open() with a hash reference con-
taining the contents of the e-mailer header. We then use the mailer object as a filehandle to print
several lines of the body text. Then we call the object's close() method to finish processing the
message and send it out.
The complete list of Mail::Mailer methods is relatively short.

$mailer = Mail::Mailer->new([$method] [,@args])

The new() method creates a new Mail::Mailer object. The optional $method argument specifies how the mail
will be sent out, and @args passes additional arguments to the mailer. Table 7.3 shows the currently recog-
nized mail methods.

The contents of @args depends on the method. In the "mail" and "sendmail" methods, whatever
you provide in @args is appended to the command line used to invoke the mail and sendmail
programs. For the "smtp" method, you can pass the named argument Server to specify the SMTP
server to use. For example:
$mailer = Mail::Mailer->new('smtp',Server => 'mail.lsjs.org')

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 SMTP: Sending Mail 141

Internally, Mail::Mailer opens up a pipe to the indicated mailer program unless "smtp" is specified,
in which case it uses Net::SMTP to send the message. If no method is explicitly provided, then
Mail::Mailer scans the command PATH looking for the appropriate executables and chooses the
first method it finds, beginning with "mail." The Mail::Mailer documentation describes how you can
alter this search order by setting the PERL_MAILERS environment variable.

Table 7.3. Mail:Mailer Mailing Methods

Method Description
mail Use the UNIX mail or mailx programs.
sendmail Use the UNIX sendmail program.
smtp Use Net::SMTP to send the mail.
test A debug mode that prints the contents of the message rather than mailing it.

Once created, you initialize the Mail::Mailer object with a set of header fields:

$fh = $mailer->open(\%headers)
The open() method begins a new mail message with the specified headers. For the "mail", "sendmail", and
"test" mailing methods, this call forks and execs the mailer program and then returns a pipe opened on the
mailer. For the "smtp" method, open(), returns a tied filehandle that intercepts calls to print() and passes
them to the datasend() method of Net::SMTP. The returned filehandle is identical to the original Mail::Mailer
object, so you are free to use it as a Boolean indicating success or failure of the open() call.

The argument to open() is a hash reference whose keys are the fields of the mail header, and
whose values can be scalars containing the contents of the corresponding field, or array references
containing the values for multivalued fields such as Cc: or To:. This format is compatible with the
header_hashref(), method of the Mail::Header class. For example:
$mailer->open({To => ['[email protected]','[email protected]'],
From => '[email protected]'}) or die "can't open: $!";

Once the object is initialized, you will print the body of the message to it using it as a filehandle:
print $mailer "This is the first line of the mail message.\n";

When the body is done, you should call the object's close() method:

$mailer->close
close() tidies up and sends the message. You should not use the close() Perl built-in for this purpose,
because some of the Mail::Mailer methods need to do postprocessing on the message before sending it.

MIME-Tools
Net::SMTP and MailTools provide the basic functionality to create simple text-only e-mail messages.
The MIME-Tools package takes this a step further by allowing you to compose multipart messages
that contain text and nontext attachments. You can also parse MIME-encoded messages to extract
the attachments, add or remove attachments, and resend the modified messages.

A Brief Introduction to MIME

The Multipurpose Internet Mail Extensions, or MIME, are described in detail in RFCs 1521, 2045,
2046, and 2049. Essentially, MIME adds three major extensions to standard Internet mail:

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 SMTP: Sending Mail 142

1. Every message body has a type. —In the MIME world, the body of every message has a type
that describes its nature; this type is given in the Content-Type: header field. MIME uses a
type/subtype nomenclature in which type indicates the category of document, and subtype
gives its specific format. Table 7.4 lists some common types and subtypes. The major media
categories are "audio," "video," "text," and "image." The "message" category is used for e-mail
enclosures, such as when you forward an e-mail onward to someone else, and the "applica-
tion" category is a hodgepodge of things that could not be classified otherwise. We'll talk about
"multipart" momentarily.
2. Every message body has an encoding. —Internet e-mail was originally designed to handle
messages consisting entirely of 7-bit ASCII text broken into relatively short lines; some parts
of the e-mail system are still limited to this type of message. However, as the Internet became
global, it became necessary to accommodate non-English character sets that have 8- or even
16-bit characters. Another problem was binary attachments such as image files, which are not
even text-oriented.
To accommodate the full range of messages that people want to send without rewriting the
SMTP protocol and all supporting software, MIME provides several standard encoding algo-
rithms that can encapsulate binary data in a text form that conventional mailers can handle.
Each header has a Content-Transfer-Encoding: field that describes the message body's en-
coding. Table 7.5 lists the five standard encodings.
If you are dealing with 8-bit data, only the quoted-printable and base64 encodings are guar-
anteed to make it through e-mail gateways.
3. Any message may have multiple parts. —The multipart/* MIME types designate messages
that have multiple parts. Each part has its own content type andMIME headers. It's even pos-
sible for a part to have its own subparts. The multipart/alternativec MIME type is used when
the various subparts correspond to the same document repeated in different formats. For
example, some browser-based mailers send their messages in both text-only and HTML form.
multipart/mixed is used when the parts are not directly related to each other, for example an
e-mail message and a JPEG enclosure.
Table 7.4. Common MIME Types
Type Description
audio/* A sound
audio/basic Sun microsystem's audio "au" format
audio/mpeg An MP3 file
audio/midi An MIDI file
audio/x-aiff AIFF sound format
audio/x-wav Microsoft's "wav" format
image/* An image
image/gif Compuserve GIF format
image/jpeg JPEG format
image/png Portable network graphics format
image/tiff TIFF format
message/* An e-mail message
message/news Usenet news message format
message/rfc822 Internet e-mail message format
multipart/* A message containing multiple parts
multipart/alternative The same information in alternative forms
multipart/mixed Unrelated pieces of information mixed together
text/* Human-readable text
text/html Hypertext Markup Language

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 SMTP: Sending Mail 143

Type Description
text/plain Plain text
text/richtext Enriched text in RFC 1523 format
text/tab-separated-values Tables
video/* Moving video or animation
video/mpeg MPEG movie format
video/quicktime Quicktime movie format
video/msvideo Microsoft "avi" movie format
application/* None of the above
application/msword Microsoft Word Format
application/news-message-id News posting format
application/octet-stream A raw binary stream
application/postscript PostScript
application/rtf Microsoft rich text format
application/wordperfect5.1 Word Perfect 5.1 format
application/gzip Gzip file compression format
application/zip PKZip file compression format

Table 7.5. MIME Encodings

Encoding Description
7bi The body is not actually encoded. This value simply asserts that text is 7-bit ASCII, with no line longer than
1,000 characters.
8bit The body is not actually encoded. This value asserts that the text may contain 8-bit characters, but has no line
longer than 1,000 characters.
binary The body is not actually encoded. This value asserts that the text may contain 8-bit characters and may have lines
longer than 1,000 characters.
quoted-printable This encoding is used for text-oriented messages that may contain 8-bit characters (such as messages in non-
English character sets). All 8-bit characters are encoded into 7-bit escape sequences, and long lines are folded
at 72 characters.
base64 This encoding is used for arbitrary binary data such as audio and images. Every 8-bit character is encoded as a
7-bit string using the uuencode algorithm. The resulting text is then folded into 72-character lines.

Any part of a multipart MIME message may contain a Content-Disposition: header, which is a hint
to the mail reader as to how to handle the part. Possible dispositions include attachment, which tells
the reader to treat the part's body as an enclosure to be saved to disk, and inline, which tells the
reader to try to display the part as a component of the document. For example, a mail reader ap-
plication may beable to display an inline image in the same window as the textual part of the mes-
sage. The Content-Disposition: field can also suggest a filename to store attachments under. An-
other field, Content-Description:, provides an optional human-readable description of the part.
Notice that an e-mail message with a JPEG attachment is really a multipart MIME message con-
taining two parts, one for the text of the message and the other for the JPEG image.
Without going into the format of a MIME message in detail, Figure 7. 5 shows a sample multipart
message to give you a feel for the way they work. This message has four parts: a 7-bit text message
that appears at the top of the message, a base64-encoded audio file that uses the Microsoft WAV
format, a base64-encoded JPEG file, and a final 7-bit part that contains some parting words and
the e-mail signature. (The binary enclosures have been truncated to save space.)

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 SMTP: Sending Mail 144

Figure 7.5. A sample multipart MIME message

Notice that each part of the message has its own header and body, and that the parts are delimited
by a short unique boundary string beginning with a pair of hyphens. The message as a whole has
its own header, which is a superset of the RFC 822 Internet mail header, and includes a Content-
Type: field of multipart/mixed.
This is pretty much all you need to know about MIME. The MIME modules will do all the rest of the
work for you.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 SMTP: Sending Mail 145

Organization of the MIME::* Modules

MIME-Tools has four major parts.
MIME::Entity MIME::Entity is a MIME message. It contains a MIME::Head (the
message header) and a MIME::Body (the message body). In mul-
tipart messages, the body may contain other MIME::Entities, and
any of these may contain their own MIME::Entities, ad infinitum.
Among other things, MIME::Entity has methods for turning the
message into a text string and for mailing the message.
MIME::Head MIME::Head is the header part of a MIME message. It has meth-
ods for getting and setting the various fields.
MIME::Body MIME::Body represents the body part of a message. Because
MIME bodies can get quite large (e.g., audio files), MIME::Body
has methods for storing data to disk and reading and writing it in a
filehandle-like fashion.
MIME::Parser The MIME::Parser recursively parses a MIME-encoded message
from a file, a filehandle, or in-memory data, and returns a
MIME::Entity. You can then extract the parts, or modify and remail
the message.
Figure 7.6 is a short example of using MIME::Entity to build a simple message that consists of a text
greeting and an audio enclosure.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 SMTP: Sending Mail 146

Figure 7.6. Sending an audio attachment with MIME tools

Lines 1–3: Load modules—We turn on strict type checking and load the MIME::Entity module.
It brings in the other modules it needs, including MIME::Head and MIME::Body.
Lines 4–8: Create top-level MIME::Entity—Using the MIME::Entity->build(), method, we
create a "top-level" multipart MIME message that contains the two subparts. The arguments to
build() include the From: and To: fields, the Subject: line, and a MIME Type of multipart/
mixed. This returns a MIME::Entity object.
Lines 9–18: Attach the text of the message—We create the text of the message and store it in
a scalar variable. Then, using the top-level MIME entity's attach() method, we incorporate
the text data into the growing multipart message, specifying a MIME Type of text/plain, an
Encoding of 7bit, and the message text as the Data.
Lines 19–23: Attach the audio file—We again call attach(), but this time specify a Type of
audio/wav and an Encoding of base64. We don't want to read the whole audio file into memory,
so we use the Path argument to direct MIME::Entity to the file where the audio data can be found.
The Description argument adds a human-readable description of the attachment to the outgoing
message.
Lines 24–25: Sign the message—We call the MIME entity object's sign() utility to append our
signature file to the text of the message.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 SMTP: Sending Mail 147

Lines 26–27: Send the message—We call the send() method to format and mail the completed
message using the smtp method.
That's all there is to it. In the next sections we will look at the MIME modules more closely.

MIME::Entity
MIME::Entity is a subclass of Mail::Internet and, like it, represents an entire e-mail message. How-
ever, there are some important differences between Mail::Internet and MIME::Entity. Whereas
Mail::Internet contains just a single header and body, the body of a MIME::Entity can be composed
of multiple parts, each of which may be composed of subparts. Each part and subpart is itself a
MIME::Entity (Figure 7.7). Because of these differences, MIME:: Entity adds several methods for
manipulating the message's body in an object-oriented fashion.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 SMTP: Sending Mail 148

Figure 7.7. A MIME message can contain an unlimited number of nested attachments

This summary omits some obscure methods. See the MIME::Entity POD documentation for the full
details.
The main constructor for MIME::Entity is build(): build() negotiates a large number of con-
structors. These are the most common:

$entity = MIME::Entity->build(arg1 => $val1, arg2 => $val2, ...)

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 SMTP: Sending Mail 149

The build() method is the main constructor for MIME::Entity. It takes a series of named arguments and
returns an initialized MIME::Entity object. The following arguments are the most common.

Field name. Any of the RFC 822 or MIME-specific fields can be used as arguments, and the pro-
vided value will be incorporated into the message header. As in Mail::Header, you can use an array
reference to pass a multivalued field. You should probably confine yourself to using RFC 822 fields,
such as From: and To:, because any MIME fields that you provide will override those generated by
MIME::Entity.
Data. For single-part entities only, the data to use as the message body. This can be a scalar or
an array reference containing lines to be joined to form the body.
Path. For single-part entities only, the path to a file where the data for the body can be found. This
can be used to attach to the outgoing message a file that is larger than you could store in main
memory.
Boundary. The boundary string to place between parts of a multipart message. MIME::Entity will
choose a good default for you; ordinarily you won't want to use this argument.
Description. A human-readable description of the body used as the value of the Content-Descrip-
tion: field.
Disposition. This argument becomes the value of the header's Content-Disposition: field. It may be
either attachment or inline, defaulting to inline if the argument is not specified.
Encoding. The value of this argument becomes the Content-Encoding: field. Youshould provide
one of 7bit, 8bit, binary, quoted-printable, or base64. Include this argument even if you are sending
a simple text message because, if you don't, MIME::Entity defaults to binary. You may also provide
a special value of-SUGGEST to have MIME::Entity make a guess based on a byte-by-byte inspec-
tion of the entire body.
Filename. The recommended filename for the mail reader to use when saving this entity to disk. If
not provided, the recommended filename will be derived from the value of Path.
Type. —The MIME type of the entity, text/plain by default. MIME::Entity makes no attempt to guess
the MIME type from the file name indicated by the Path argument or from the contents of the Data
argument.
Here's the idiom for creating a single-part entity (which may later be attached to a multipart entity):
$part = MIME::Entity->build(To => '[email protected]',
Type => 'image/jpeg',
Encoding => 'base64',
Path => '/tmp/pictures/oranges.jpg');

And here's the idiom for creating a multipart entity, to which subparts will be added:
$multipart = MIME::Entity->build(To => '[email protected]',
Type => 'multipart/mixed');

Notice that single-part entities should have a body specified using either the Data or the Path ar-
guments. Multipart entities should not.
Once the MIME::Entity is created, you will attach new components to it using add-part() or
attach():

$part = $entity->add_part($part [,$offset])

The add_part() method adds a subpart to the multipart MIME::Entity contained in $entity. The $part
argument must be a MIME::Entity object. Each multipart MIME::Entity object maintains an array of its subparts,
and by default, the new part is appended to the end of the current array. You can modify this by providing an
offset argument. The method returns the newly added part.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 SMTP: Sending Mail 150

If you attempt to add a part to a single-part entity, MIME::Entity automagically converts the entity into type
multipart/mixed, and reattaches the original contents as a subpart. The entity you are adding then becomes
the second subpart on the list. This feature allows you to begin to compose a single-part message and later
add attachments without having to start anew.
$part = $entity->attach(arg1 => $val1, arg2 => $val2, ...)
The attach() method is a convenience function that first creates a new MIME::Entity object using
build(), and then calls $entity->add_part() to insert the newly created part into the message. The
arguments are identical to those of build(). If successful, the method returns the new MIME::Entity.

Several methods provide access to the contents of the entity:

$head = $entity->head([$newhead])

Licensed by
The head() method returns the MIME::Head object associated with the entity. You can then call methods in
the head object to examine and change fields. The optional $newhead argument, if provided, can be used to
replace the header with a different MIME::Body object.
$body = $entity->bodyhandle([$newbody])
The bodyhandle() method gets or sets the MIME::Body object associated with the entity. You can then use

Stjepan Maric
this object to retrieve or modify the unencoded contents of the body. The optional $newbody argument can
be used to replace the body with a different MIME::Body object. Don't confuse this method with body(), which
returns an array ref containing the text representation of the encoded body.
If the entity is multipart, then there will be no body, in which case bodyhandle(), returns undef. Before trying

4218908
to fetch the body, you can use the is_multipart(), method to check for this possibility.
$pseudohandle = $entity->open($mode)
The open() method opens the body of the entity for reading or writing, and returns a MIME pseudohandle.
As described later in the section on the MIME::Body class, MIME pseudohandles have object methods similar
to those in the IO::Handle class (e.g., read(), getline(), and print()), but they are not handles in the
true sense of the word. The pseudohandle can be used to retrieve or change the contents of the entity's body.
$mode is one of "r" for reading, or "w" for writing.

@parts = $entity->parts($index)
$parts = $entity->parts($index)
@parst= $entity->parts(\@parts)
The parts() method returns the list of MIME::Entity parts in a multipart entity. If called with no arguments,
the method returns the entire list of parts; if called with an integer index, it returns the designated part. If passed
the reference to an array of parts, the method replaces the current parts with the contents of the array. This
allows you delete parts or rearrange their order.
For example, this code fragment reverses the order of the parts in the entity:
$entity->parts([reverse $entity->parts])

If the entity is not multipart, parts() returns an empty list.

A variety of methods return information about the Entity:

$type = $entity->mime_type
$type = $entity->effective_type

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 SMTP: Sending Mail 151

The mime_type() and effective_type() methods both return the MIME type of the entity's body. Al-
though the two methods usually return the same value, there are some error conditions in which MIME::Parser
cannot decode the entity and is therefore unable to return the body in its native form. In this case,
mime_type(), returns the type that the body is supposed to be, and effective_type() returns the type
that actually returns when you retrieve or save the body data (most probably application/octet-stream). To be
safe, use effective_type() when retrieving the body of an entity created by MIME::Parser. For entities
you created yourself with MIME::Entity->build(), there's no difference.
$boolean = $entity->is_multipart
The is_multipart() method is a convenience routine that returns true if the entity is multipart, false if it
contains a single part only.
$entity->sign(arg1 => $val1, arg2=> $val2, ...)
The sign() method attaches a signature to the message. If the message contains multiple parts, MIME::Entity
searches for the first text entity and attaches the signature to that.
The method adds some improvements to the version implemented in Mail::Internet, however you must provide
at least one set of named arguments. Possibilities include:
File. This argument allows you to use the signature text contained in a file. Its value should be the path to
a local file.
Signature. This argument uses the indicated text as the signature. Its value can be a scalar or a reference
to an array of lines.
Force. Sign the entity even if its content type isn't text/*. The value is treated as a Boolean.
Remove. Call remove_sig() to scan for an existing signature and remove it before adding the new
signature. The value of this argument is passed to remove_sig(). Provide 0 to disable signature removal
entirely.
For example, here's how to add a signature using a scalar value:
$entity->sign(Signature => "That's all folks!");

$entity->remove_sig([$nlines])
Remove_sig() scans the last $nlines of the message body as it looks for a line consisting of the characters
"--". The line and everything below it is removed. $nlines defaults to 10.
$entity->dump_skeleton([FILEHANDLE])
Dump_skeleton() is a debugging utility. It dumps a text representation of the structure of the entity and its
subparts to the indicated filehandle, or, if no filehandle is provided, to standard output.

Finally, several methods are involved in exporting the entity as text and mailing it:

$entity->print([FILEHANDLE])
$entity->print_header([FILEHANDLE])
$entity->print_body([FILEHANDLE])
These three methods, inherited from Mail::Internet, print the encoded text representations of the whole mes-
sage, the header, or the body, respectively. The parts of a multipart entity are also printed. If no filehandle is
provided, it prints to STDOUT.
$arrayref = $entity->header
The header() method, which is inherited from Mail::Internet, returns the text representation of the header as
a reference to an array of lines. Don't confuse this with the head() method, which returns a MIME::Head object.
$arrayref = $entity->body
This method, which is inherited from Mail::Internet, returns the body of the message as a reference to an array
of lines. The lines are encoded in a form suitable for passing to a mailer. Don't confuse this method with
bodyhandle() (discussed next), which returns a MIME::Body object.
$string = $entity->as_string $string

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 SMTP: Sending Mail 152

$string = $entity->stringify_body
$string $entity->stringify_header
The as_string() method converts the message into a string, encoding any parts that need to be. The
stringify_body() and stringify_header() methods respectively operate on the body and header
only.
$result = $entity->send([$method])
The send() method, which is inherited from Mail::Internet, sends off the message using the selected method.
I have noticed that some versions of the UNIX mail program have problems with MIME headers, and so it's
best to set $method explicitly to either "sendmail" or "smtp".
$entity->purge
If you have received the MIME::Entity object from MIME::Parser, it is likely that the body of the entity or one of
its subparts is stored in a temporary file on disk. After you are finished using the object, you should call
purge() to remove these temporary files, reclaiming the disk space. This does not happen automatically when
the object is destroyed.

MIME::Head
The MIME::Head class contains information about a MIME entity's header. It is returned by the
MIME::Entity head() method.
MIME::Head is a class of Mail::Header and inherits most of its methods from there. It is a historical
oddity that one module is called "Head" and the other "Header." MIME::Head adds a few utility
methods to Mail::Header, the most useful of which are read() and from_file():

$head = MIME::Head->read(FILEHANDLE)
In addition to creating a MIME::Head object manually by calling add() for each header field, you can create
a fully initialized header from an open filehandle by calling the read() method. This supplements Mail::Head-
er's read() method, which allows you to read a file only into a previously created object.
$head = MIME::Head->from_file($file)
The from_file() constructor creates a MIME::Head object from the indicated file by opening it and passing
the resulting filehandle to read().

All other functions behave as they do in Mail::Header. For example, here is one way to retrieve and
change the subject line in a MIME::Entity object:
$old_subject = $entity->head->get('Subject');
$new_subject = "Re: $old_subject";
$entity->head->replace(Subject => $new_subject);

Like Mail::Header, MIME::Head->get() also returns newlines at the ends of removed field values.

MIME::Body
The MIME::Body class contains information on the body part of a MIME::Entity. MIME::Body objects
are returned by the MIME::Entity bodyhandle() method, and are created as needed by the
MIME::Entity build() and attach() methods. You will need to interact with MIME::Body objects
when parsing incoming MIME-encoded messages.
Because MIME-encoded data can be quite large, an important feature of MIME::Body is its ability
to store the data on disk or in memory ("in core" as the MIME-Tools documentation calls it). The
methods available in MIME::Body allow you to control where the body data is stored, to read and
write it, and to create new MIME::Body objects.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 SMTP: Sending Mail 153

MIME::Body has three subclasses, each specialized for storing data in a different manner:
MIME::Body::File: This subclass stores its body data in a disk file. This is suitable for large binary
objects that wouldn't easily fit into main memory.
MIME::Body::Scalar: This subclass stores its body data in a scalar variable in main memory.
It's suitable for small pieces of data such as the text part of ane-mail message.
MIME::Body::InCore: This subclass stores its body data in an array reference kept in main
memory. It's suitable for larger amounts of text on which you will perform multiple reads or writes.
Normally MIME::Parser creates MIME::Body::File objects to store body data on disk while it is pars-
ing.

$body = MIME::Body::File->new($path)
To create a new MIME::Body object that stores its data to a file, call the MIME::, Body::File->new()
method with the path to the file. The file doesn't have to exist, but will be created when you open the body for
writing.
$body = MIME::Body::Scalar->new(\$string)
The MIME::Body::Scalar->new() method returns a body object that stores its data in a scalar reference.
$body = MIME::Body::InCore->new($string)
$body = MIME::Body::InCore->new(\$string)
$body = MIME::Body::InCore->new(\@string)
The MIME::Body::InCore class has the most flexible constructor. Internally it stores its data in an array refer-
ence, but it can be initialized from a scalar, a reference to a scalar, or a reference to an array.

Once you have a MIME::Body object, you can access its contents by opening it with the open()
method.

$pseudohandle = $body->open($mode)
This method takes a single argument that indicates whether to open the body for reading ("r") or writing ("w").
The returned object is a pseudohandle that implements the IO::Handle methods read(), print(), and
getline(). However, it is not a true filehandle, so be careful not to pass the returned pseudohandle to any
of the built-in procedures such as <> or read().

The following code fragment illustrates how to read the contents of a large MIME::Body stored in a
MIME::Entity object and print it to STDOUT. The contents recovered in this way are in their native
form, free of any MIME encoding:
$body = $entity->body handle or die "no body";
$handle = $body->open("r");
print $data while $handle->read($data,1024);

For line-oriented data, we would have used the getline() method instead.
Another code fragment illustrates how to write a MIME::Body's contents using its print() method.
If the body is attached to a file, the data is written there. Otherwise, it is written to an in-memory data
structure:
$body = $entity->body handle or die "no body";
$handle = $body->open("w");
$handle->print($_) while <>;

MIME::Body provides a number of convenience methods:

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 SMTP: Sending Mail 154

@lines = $body->as_lines
$string = $body->as_string
as_lines() and as_string() are convenience functions that return the entire contents of the body in a
single operation. as_lines() opens the body and calls get_line() repeatedly, returning an array of new-
line-terminated lines. as_string() reads the entire body into a scalar. Because either method can read a
large amount of data into memory, you should exercise some caution before calling them.
$path = $body->path([$newpath])
If the body object is attached to a file, as in MIME::Body::File, then path() returns the path to the file or sets
it if the optional $newpath argument is provided. If the data is kept in memory, then path() returns undef.
$body->print([FILEHANDLE])
The print() method prints the unencoded body to the indicated filehandle, or, if none is provided, to the
currently selected filehandle. Do not confuse this with the print() method provided by the pseudohandles
returned by the open() method, which is used to write data into the body object.
$body->purge
Purge unlinks the file associated with the body object, if any. It is not called automatically when the object is
destroyed.

MIME::Parser
The last major component of MIME-Tools is the MIME::Parser class, which parses the text repre-
sentation of a MIME message into its various components. The class is simple enough to use, but
has a large number of options that control various aspects of its operation. The short example in
Figure 7.8 will give you the general idea.

Figure 7.8. Using MIME::Parser

Lines 1–3: Load modules—We turn on strict type checking and load the MIME::Parser module.
It brings in the other modules it needs, including MIME::Entity.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 SMTP: Sending Mail 155

Lines 4–5: Open a message—We recover the name of a file from the command line, which
contains a MIME-encoded message, and open it. This filehandle will be passed to the parser
later.
Lines 6–8: Create and configure the parser—We create a new parser object by calling
MIME::Parser->new(). We then call the newly created object's output_dir(), method to
set the directory where the parser will write the body data of extracted enclosures.
Lines 9–10: Parse the file—We pass the open filehandle to the parser's parse(), method. The
value returned from the method is a MIME::Entity object corresponding to the top level of the
message.
Lines 11–14: Print information about the top-level entity—To demonstrate that we parsed the
message, we recover and print the From: and Subject: lines of the header, calling the entity's
head() method to get the MIME::Head object each time. We also print the MIME type of the
whole message, and the number of subparts, which we derive from the entity's parts()
method.
Lines 15–17: Print information about the parts—We loop through each part of the message. For
each, we call its mime_type() method to retrieve the MIME type, and the path() method of
the corresponding MIME::Body to get the name of the file that contains the data.
Line 18: Clean up—When we are finished, we call purge() to remove all the parsed body data
files.
When I ran the program on a MIME message stored in the file mime.test, this is was the result:
% simple_parse.pl ~/mime.test
From = Lincoln Stein <[email protected]>
Subject = testing mime parser
MIME type = multipart/mixed
Parts = 5
text/plain /tmp/msg-1857-1.dat
audio/wav /tmp/assimilated.wav
image/jpeg /tmp/aw-2-19.jpg
audio/mpeg /tmp/NorthwestPassage.mp3
text/plain /tmp/msg-1857-2.dat

This multipart message contains five parts. The first and last parts contain text data and correspond
to the salutation and the signature. The remaining parts are enclosures, consisting of an audio/wav
sound file, a JPEG image, and a ripped MP3 track.
We will walk through a more complex example of MIME::Parser in Chapter 8, where we deal with
writing Post Office Protocol clients. The example developed there will spawn external viewers to
view image and audio attachments.
Because MIME files can be quite large, MIME::Parser's default is to store the parsed MIME::Body
parts as files using the MIME::Body::File class. You can control where these files are stored using
either the output_dir() or the output_under() methods. The output_dir() method tells
MIME::Parser to store the parts directly inside a designated directory. output_under(), on the
other hand, creates a two-tier directory. For each parsed e-mail message, MIME::Parser creates a
subdirectory under the base directory specified by output_under(), and then writes the
MIME::Body::File data there.
In either case, all the temporary files are cleared when you call the top-level MIME::Entity's
purge() method. You can instead keep some or all of the parts. To keep some parts, step through
the message parts and call purge() selectively on those that you don't want to keep. You can
either leave the other parts where they are or move them to a different location for safekeeping. To
keep all parsed parts, don't call purge() at all.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 SMTP: Sending Mail 156

Parsing is complex, and the parse() method may die if it encounters any of a number of exceptions.
You can catch such exceptions and attempt to perform some error recovery by wrapping the call to
parse() in an eval{} block:
$entity = eval { $parser->parse(\*F) };
warn $@ if $@;

Here is a brief list of the major functions in MIME::Parser, starting with the constructor.

$parser = MIME::Parser->new
The new() method creates a new parser object with default settings. It takes no arguments.
$dir = $parser->output_dir
$previous = $parser->output_dir($newdir)
The output_dir() method gets or sets the output directory for the parse. This is the directory in which the
various parts and enclosures of the parsed message are (temporarily) stored.
If called with no arguments, it returns the current value of the output directory. If called with a directory path, it
sets the output directory and returns its previous value. The default setting is ".", the current directory.
$dir = $parser->output_under
$parser->output_under($basedir [,DirName=>$dir [,Purge=>$purge]])
output_under() changes the temporary file strategy to use a two-tier directory. MIME::Parser creates a
subdirectory inside the specified base directory and then places the parsed MIME::Body::File data in the newly
created subdirectory.

In addition to $basedir, output_under() accepts two optional named arguments:

DirName. By default, the subdirectory is named by concatenating the current time, process ID,
and a sequence number. If you would like a more predictable directory name, you can use
DirName to provide a subdirectory name explicitly.
Purge. If you use the same subdirectory name each time you run the program, you might want
to set Purge to a true value, in which case output_under() will remove anything in the sub-
directory before beginning the parse.
Called with no arguments, output_under() returns the current base directory name. Here are
two examples:
# store enclosures in ~/mime_enclosures
$parser->output_under("$ENV{HOME}/mime_enclosures");

# store enclosures under /tmp in subdirectory "my_mime"

$parser->output_under("/tmp", DirName=>'my_mime', Purge=>1);

The main methods are parse(), parse_data(), and parse_open():

$entity = $parser->parse(\*FILEHANDLE)
The parse() method parses a MIME message by reading its text from an open filehandle. If successful, it
returns a MIME::Entity object. Otherwise, parse() can throw any number of run-time exceptions. To catch
those exceptions, wrap parse() in an eval{} block as described earlier.
$entity = $parser->parse_data($data)
The parse_data() method parses a MIME message that is contained in memory. $data can be a scalar
holding the text of the message, a reference to a scalar, or a reference to an array of scalars. The latter is
intended to be used on an array of the message's lines, but can be any array which, when concatenated, yields
the text of the message. If successful, parse_data() returns a MIME::Entity object. Otherwise, it generates
a number of run-time exceptions.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 SMTP: Sending Mail 157

$entity = $parser->parse_open($file)
The parse_open() method is a convenience function. It opens the file provided, and then passes the resulting
filehandle to parse(). It is equivalent to:
open (F,$file);
$entity = $parser->parse(\*F);

Because parse_open() uses Perl's open() function, you can play the usual tricks with pipes. For
example:
$entity = $parser->parse_open("zcat ./mailbox.gz |");

This uncompresses the compressed mailbox using the zcat program and pipes the result to
parse().
Several other methods control the way the parse operates:

$flag = $parser->output_to_core
$parser->output_to_core($flag)
The output_to_core() method controls whether MIME::Parser creates files to hold the decoded body data
of MIME::Entity parts, or attempts to keep the data in memory. If $flag is false (the default), then the parts
are parsed into disk files. If $flag is true, then MIME::Parser stores the body parts in main memory as
MIME::Body::InCore objects.
Since enclosures can be quite large, you should be cautious about doing this. With no arguments, this method
returns the current setting of the flag.
$flag = $parser->ignore_errors
$parser->ignore_errors($flag)
The ignore_errors() method controls whether MIME::Parser tolerates certain syntax errors in the MIME
message during parsing. If true (the default), then errors generate warnings, but if not, they cause a fatal
exception during parse().
$error = $parser->last_error
$head = $parser->last_head
These two methods are useful for dealing with unparseable MIME messages. last_error() returns the last
error message generated during the most recent parse. It is set when an error was encountered, and either
ignore_errors() is true, or the call to parse() was wrapped in an eval{}.
last_head() returns the top-level MIME::Head object from the last stream we attempted to parse. Even
though the body of the message wasn't successfully parsed, we can use the header returned by this method
to salvage some information, such as the subject line and the name of the sender.

MIME Example: Mailing Recent CPAN Entries

In this section, we develop an application that combines the Net::FTP module from Chapter 19 with
the Mail and MIME modules from this chapter. The program will log into the CPAN FTP site at
ftp.perl.org, read the RECENT file that contains a list of modules and packages recently contributed
to the site, download them, and incorporate them as attachments into an outgoing e-mail message.
The idea is to run the script at weekly intervals to get automatic notification of new CPAN uploads.
Figure 7.9 shows the listing for the application, called mail_recent.pl.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 SMTP: Sending Mail 158

Figure 7.9. The mail_recent.pl script

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 SMTP: Sending Mail 159

Lines 1–4: Load modules—We turn on strict syntax checking and load the Net::FTP and
MIME::Entity modules.
Lines 5–9: Define constants—We set constants corresponding to the FTP site to connect to, the
CPAN directory, and the name of the RECENT file itself. We also declare a constant with the e-
mail address of the recipient of the message (in this case, my local username), and a DEBUG
constant to turn on verbose progress messages.
Lines 10–11: Declare globals—The %RETRIEVE global contains the list of files to retrieve from
CPAN. $TMPDIR contains the path of a directory in which to store the downloaded files tempo-
rarily before mailing them. This is derived from the TMPDIR environment variable, or, if not oth-
erwise specified, from /usr/tmp. Windows and Macintosh users have to check and modify this
for their systems.
Lines 12–15: Log into CPAN and fetch the RECENT file—We create a new Net::FTP object and
log into the CPAN mirror. If successful, we change to the directory that contains the archive and
call the FTP object's retr() method to return a filehandle from which we can read the RE-
CENT file.
Lines 17–23: Parse the RECENT file— RECENT contains a list of all files on the CPAN archive
that are new or have changed recently, but we don't want to download them all. The files we're
interested in have lines that look like this:
modules/by-module/Apache/Apache-Filter-1.011.tar.gz
modules/by-module/Apache/Apache-iNcom-0.09.tar.gz
modules/by-module/Audio/Audio-Play-MPG123-0.04.tar.gz
modules/by-module/Bundle/Bundle-WWW-Search-ALL-1.09.tar.gz

We open the file for reading and scan through it one line at a time, looking for lines that match
the appropriate pattern. We store the filename and its CPAN path in %RETRIEVE.
After processing the filehandle, we close it.
Lines 24–32: Begin the mail message—We begin the outgoing mail message with ashort text
message that gives the number of enclosures. We create a new MIME::Entity object by calling
the build() constructor with the introduction as its initial contents.
Notice that the arguments we pass to build() create a single-part document of type text/
plain. Later, when we add the enclosures, we rely on MIME::Entity's ability to convert the mes-
sage into a multipart message when needed.
Lines 33–44: Retrieve modules and attach them to the mail—We loop through the filenames
stored in %RETRIEVE. For each one, we call the FTP object's get(), method to download the
file to the temporary directory. If successful, we use the Filename argument to attach the file to
the outgoing mail message by calling the top-level entity's attach() method. Other
attach() arguments set the encoding to base64, and the MIME type to application/x-gzip.
CPAN files are gzipped by convention. We also add a short description to the attachment; cur-
rently it is just a copy of the filename.
Line 45: Add signature to the outgoing mail —If there is a file named .signature in the current
user's home directory, we call the MIME entity's sign() method to attach it to the end of the
message.
Lines 46–49: Send the mail—We call the entity's send() method to MIME-encode the message
and send it via the SMTP protocol. When this is done, we call the entity's purge() method,
deleting the downloaded files in the temporary directory. This works because the files became
the basis for the MIME-entity bodies via the MIME::Body::File subclass when they were attached
to the outgoing message, and purge() recursively deletes these files.
Note that the send() method relies on libnet being correctly configured to find a working SMTP
server. If this is not the case, check and fix the Libnet.cfg file.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 SMTP: Sending Mail 160

Line 51: Close FTP connection —Our last step is to close the FTP connection by calling the FTP
object's quit() method.
Figure 7.10 shows a screenshot of Netscape Navigator displaying the resulting MIME message.
Clicking on one of the enclosures will prompt you to save it to disk so that you can unpack and build
the module.

Licensed by
Stjepan Maric
4218908

Figure 7.10. A mail message sent from mail_recent.pl

A deficiency in the program is that the CPAN filenames can be cryptic, and it isn't always obvious
what a package does. A nice enhancement to this script would be to unpack the package, scan
through its contents looking for the POD documentation, and extract the description line following
the NAME heading. This information could then be used as the MIME::Entity Description: field rather
than the filename itself. A simpler alternative would be to enclose the .readme file that frequently
(but not always) accompanies a package's .tar.gz file.

Summary
The Net::SMTP, Mail::Internet, and Mail::Mailer modules make it possible, and convenient, to send
properly formatted Internet mail. The MIME-Tools package builds on these classes to construct and
process complex messages that contain MIME attachments.
The next chapter shows the other side of the equation: how to receive and process incoming mes-
sages. In addition, it contains practical examples of processing message attachments using
MIME::Parser.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 161

Chapter 8. POP, IMAP, and NNTP

Processing Mail and Netnews

In the last chapter we looked at client modules for sending Internet mail. In the first part of this
chapter we'll look at modules for receiving mail and processing messages with enclosures (including
multimedia enclosures). In the second part, we'll look at clients for the closely related Netnews
protocol.

The Post Office Protocol

POP3 and IMAP are the two protocols used most to access Internet mail. Both were designed to
allow a user to access mail drops on remote machines, and provide methods to list the contents of
the user's mailbox, to download mail for viewing, and to delete messages the user is no longer
interested in.
POP3 (Post Office Protocol version 3) is the older and simpler of the two. Described in RFC 1725
and STD 53, it provides a straightforward interface for listing, retrieving, and deleting mail held on
a remote server. IMAP (Internet Message Access Protocol), described in RFC 2060, adds sophis-
ticated facilities for managing sets of remote and local mailboxes and synchronizing them when the
user connects.
We will consider fetching mail from a POP3 server in this section. There are at least two Perl modules
on CPAN for dealing with POP3 servers: Mail::POP3Client, written by Sean Dowd, and Net::POP3,
by Graham Barr. Both provide essentially the same functionality but they use different APIs. The
most important feature difference between the two is that Net::POP3 allows you to save the contents
of a mail message to a filehandle, while Mail::POP3Client reads the entire mail message into mem-
ory. Because the ability to save to a filehandle makes a big difference when dealing with large e-
mails (such as those containing MIME enclosures), I recommend Net::POP3.
Net::POP3 inherits from Net::Cmd, making it similar in style to Net::FTP and Net::SMTP. You begin
by creating a new Net::POP3 object connected to the mailbox host. If this is successful, you log in
using a username and password, and then invoke various methods to list the contents of the mail-
box, retrieve individual messages, and possibly delete the retrieved messages.

Summarizing a POP3 Mailbox

Figure 8.1 shows a small program that will access a user's mailbox on a maildrop machine and print
a brief summary of the senders and subject lines of all new messages. The username and mailhost
are specified on the command line using the format [email protected]. The program
prompts for the password. Appendix A contains the listing for the PromptUtil.pm package.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 POP, IMAP, and NNTP 162

Figure 8.1. List entries in a user's inbox

Lines 1–6: Load modules—We bring in the Net::POP3 module to contact the remote POP server,
and Mail::Header to parse the retrieved mail headers. We also bring in a new home-brewed
utility module, PromptUtil, which provides the get_passwd() function, along with a few other
user prompting functions.
Lines 6–8: Get username, host, and password—We get the username and host from the com-
mand line, and prompt the user to enter his or her password using the get_passwd() function.
The latter turns off terminal echo so that the password is not visible on the screen.
Line 9: Connect to mailbox host—We call the Net::POP3 new() method to connect to the indi-
cated host, giving the server 30 seconds in which to respond with the welcome banner. The
new() constructor returns a Net::POP3 object.
Lines 10–13: Log in and count messages—We call the POP3 object's login() method to log
in with the user's name and password. If the login is successful, it returns the total number of
messages in the user's mailbox; if there are no messages in the mailbox, it returns 0E0 ("zero
but true"). This value has a property of 1 if treated in a logical text to test whether login was
successful, and is equal to 0 when used to count the number of available messages.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 POP, IMAP, and NNTP 163

Next we call the POP3 object's last() method to return the number of the last message the
user read (0 if none read). We will use this to list the unread messages. Because the message
count retrieved by new() can be 0E0, we add zero to it to convert it into a more familiar number.
We then print the total number of old and new messages.
Lines 14–21: Summarize messages—Each message is numbered from 1 to the total of mes-
sages in the mailbox. For each one, we call the POP object's top() method to retrieve the
message header as a reference to an array of lines, and pass this to Mail::Header-
>new() for parsing. We call the parsed header's get() method twice to retrieve the Subject:
and From: lines, and pass the sender's address to the clean_from() utility subroutine to clean
it up a bit. We then print out the message number, sender's name, and subject.
Line 22: Log out—The POP object's quit() method logs out cleanly.
Lines 23–29: Clean up with the clean_from() subroutine—This subroutine cleans up sender
addresses a bit, by extracting the sender's name from these three common address formats:
"Lincoln Stein" <[email protected]>
Lincoln Stein <[email protected]>
[email protected] (Lincoln Stein)

When we run this program, we get output like this:

% pop_stats.pl lstein@localhost
inbox has 6 messages (6 new)
1 Geoff Winisky Re: total newbie question
2 Robin Lofving Server updates
3 James W Goldblum Comments part 2
4 Jessica Raymond Statistics on Transaction Security
5 James W Goldbum feedback access from each page
6 The Western Web The Western Web Newsletter

Net::POP3 API
The Net::POP3 API is simple. You can log in, log out, list messages, retrieve message headers,
retrieve the entire message, and delete messages.

$pop = Net::POP3->new([$host] [,$opt1=>$val1, $opt2=>$val2…])

The new() method constructs a new Net::POP3 object. The first, optional, argument is the name or IP address
of the mailbox host. This may be followed by a series of option/value pairs. If the host is not provided, it will be
retrieved from the Net::Config "POP3_hosts" value specified when the libnet module was installed. The options
are listed in Table 8.1.
The ResvPort option is used with some POP3 servers that require clients to connect from reserved ports.
If unsuccessful, new() returns undef and $! is set to some error code.
$messages = $pop->login([$username [,$password]])
The login() method attempts to log into the server using the provided username and password. If one or
both of the password and username are not given, then login() looks in the user's .netrc file for the authen-
tication information for the specified host.

If successful, login() returns the total number of messages in the user's mailbox. If there are no
messages, login() returns the following point number 0E0, which will be treated as true when
used in a logical context to test whether login was successful, but evaluate to zero when treated in
a numeric context to count the number of available messages. If an error occurs, login() returns
undef and $pop->message() contains an error message.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 POP, IMAP, and NNTP 164

If the login fails, you may try again or try to login using apop(). Some servers close the connection
after a number of unsuccessful login attempts. With the exception of quit(), none of the other
methods will be accepted until the server accepts the login.
Some POP servers support the APOP command.
$messages = $pop->apop($username,$password)

APOP is similar to a standard login, but instead of sending passwords across the network in the clear, it uses
a challenge/ response system to authenticate the user without processing cleartext passwords. Unlike
login(), .netrc is not consulted if the username and password are absent. The value returned from
apop() is the same as that from login().

Table 8.1. Net::POP3->new() Options

Option Description Default

Port Remote port to connect to POP3(110)
ResvPort Local port to bind to ephemeral port
Timeout Second to wait for a response 120
Debug Turn on verbose debugging undef

Many POP3 servers need special configuration before the APOP command will authenticate cor-
rectly. In particular, most UNIX servers need a password file distinct from the system password file.
Once login is successful, you can use a variety of methods to access the mailbox:

$last_msgnum = $pop->last
POP messages are numbered from 1 through the total number of messages in the inbox. At any time, the user
may have read one or more messages using the RETR command (see below), but not deleted them from the
inbox. Last() returns the highest number from the set of retrieved messages, or 0 if no messages have been
retrieved. New messages begin at $last_msgnum+1.
Many POP servers store the last-read information between connections; however, a few discard this informa-
tion.
$arrayref = $pop->get($msgnum [,FILEHANDLE])
Following a successful login, the get() method retrieves the message indicated by its message number, using
the POP3 RETR command. It can be called with a filehandle, in which case the contents of the message (both
header and body) are written to the filehandle. Otherwise, the get() method returns an array reference con-
taining the lines of the message.
$handle = $pop->getfh($msgnum)
This is similar to get(), but the return value is a tied filehandle. Reading from this handle returns the contents
of the message. When the handle returns end-of-file, it should be closed and discarded.
$flag = $pop->delete($msgnum)
delete() marks the indicated message for deletion. Marked messages are not removed until the quit()
method is called, and can be unmarked by calling reset().
$arrayref = $pop->top($msgnum[,$lines])
The top() method returns the header of the indicated message as a reference to an array of lines. This format
is suitable for passing to the Mail::Header->new() method. If the optional $lines argument is provided,
then the indicated number of lines of the message body are included.
$hashref = $pop->list
$size = $pop->list($msgnum)

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 POP, IMAP, and NNTP 165

The list() method returns information on the size of mailbox messages. Called without arguments, it returns
a hash reference in which the keys are message IDs, and the values are the sizes of the messages, in bytes.
Called with a message ID, the method returns the size of the indicated message, or if an invalid message
number was provided, it returns undef.
($msg_count,$size) = $pop->popstat
pop_stat() returns a two-element list that consists of the number of undeleted messages in the mailbox and
the size of the mailbox in bytes.
$uidl = $pop->uidl([$msgnum])
The uidl() method returns a unique identifier for the given message number. Called without an argument,
it returns a hash reference in which the keys are the message numbers for the entire mailbox, and the values
are their unique identifiers. This method is intended to help clients track messages across sessions, since the
message numbers change as the mailbox grows and shrinks.

When you call the quit() method, messages marked for deletion are removed unless you
reset() first.

$pop->reset
This method resets the mailbox, unmarking the messages marked for deletion.
$pop->quit
The quit() method quits the remote server and disconnects. Any messages marked for deletion are removed
from the mailbox.

Retrieving and Processing MIME Messages via POP

To show Net::POP3 in a real-world application, I developed a script called pop_fetch.pl that com-
bines Net::POP3 and MIME::Parse. Figure 8.2 shows a session with this program. After I invoke it
with the mailbox name in user@host form, the program prompts me for my login password. The
program reports the number of messages in my mailbox, and then displays the date, sender, and
subject line of the first, prompting me to read it or skip to the next.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 POP, IMAP, and NNTP 166

Figure 8.2. A session with pop_fetch.pl

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 POP, IMAP, and NNTP 167

I choose to read the message, causing the program to display the message header and the text
part of the body. It then reports that the message has two attachments (technically, two non-text/
plain MIME parts). For each one, the program prompts me for the disposition of the attachment. For
the first attachment, of type image/jpeg, I choose to view the attachment, causing my favorite image
viewer (the XV application, written by John Bradley) to pop up in a new window and show the picture.
After I quit the viewer, the script prompts me again for the disposition. This time I choose to save
the image under its default name.
The next attachment is a Microsoft Word document. No viewer is defined for this document type,
so the prompt only allows the attachment to be saved to disk.
After dealing with the last attachment, the program prompts me to keep or delete the entire message
from the inbox, or to quit. I quit. The program then moves on to the next unprocessed message.

The pop_fetch.pl Script

pop_fetch.pl is broken into two parts. The main part, listed in Figure 8.3, handles the user interface.
A smaller module named PopParser.pm subclasses Net::POP3 in such a way that messages re-
trieved from a POP3 mailbox are automatically parsed into MIME::Entities.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 POP, IMAP, and NNTP 168

Figure 8.3. The pop_fetch.pl script

We'll look at pop_fetch.pl first.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 POP, IMAP, and NNTP 169

Lines 1–6: Activate taint checking and load modules—Since we will be launching external ap-
plications (the viewers) based on information from untrusted sources, we need to be careful to
check for tainted variables. The -T switch turns on taint checking. (See Chapter 10 for more
information.)
We load PopParser and PromptUtil, two modules developed for this application.
Lines 7–11: Define viewers—We define constants for certain external viewers. For example,
HTML files are invoked with the command lynx %s, where %s is replaced by the name of the
HTML file to view. For variety, some of the viewers are implemented as pipes. For example, the
player for MP3 audio files is invoked as mpg123-, where the - symbol tells the player to take its
input from standard input.
At the end of the code walkthrough, we'll discuss replacing this section of code with the standard
mailcap facility.
Lines 12–13: Taint check precautions —As explained in more depth in Chapter 10, taint checking
will not let us run with an untrusted path or with several other environment variables set. We set
PATH to a known, trusted state, and delete four other environment variables that affect the way
that commands are processed.
Lines 14–20: Recover username and mailbox host—We process the command-line arguments
to recover the name of the user and the POP3 host.
The $entity global holds the most recent parsed MIME::Entity object. We make it global so
that the script's END{} block can detect it and call its purge(), method in case the user quits
the program prematurely. This will delete all temporary files from disk. For similar reasons, we
intercept the INT signal to exit gracefully if the user hits the interrupt key.
Lines 21–26: Log in to mailbox server—The PopParser.pm module defines a new subclass of
Net::POP3 that inherits all the behavior of the base class, but returns parsed MIME::Entity ob-
jects from the get() method rather than the raw text of the message. We create a new Pop-
Parser object connected to the mailbox host. If this is successful, we call get_passwd() (im-
ported from the PromptUtil module) to get the user's login password.
Next, we authenticate ourselves to the remote host. We don't know a priori whether the server
accepts APOP authentication or the less secure cleartext authentication method, so we try them
both. If the apop() method fails, then we try login(). If that also fails, we die with an error
message.
If login is successful, we print the number of messages returned by the apop(), or login()
methods. We add 0 to the message count to convert the 0E0 result code into a more user-friendly
integer.
Lines 27–38: Enter the main message-processing loop —We now enter the main message-
processing loop. For each message, we fetch its header by calling the PopParser object's
top() method (which is inherited without modification from Net::POP3). The header text is then
passed to our print_header() method to display it as a one-line message summary.
We ask the user if he or she wants to read the message, and if so, we call the PopParser object's
get() method, which fetches the indicated message, parses it, and returns a MIME::Entity
object. This object is passed to our display_entity(), subroutine in order to display it and
its subparts. When display_entity() is finished, we delete the entity's temporary files by
calling its purge() method.
The last step is to ask the user if he or she wants to delete the message from the remote mailbox,
and if the answer is affirmative, we call the PopParser's delete() method.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 POP, IMAP, and NNTP 170

Lines 39–45: print_header() subroutine —The print_header() subroutine takes an ar-

ray ref containing the header lines returned by $POP->top() and turns it into a one-line sum-
mary for display. Although we could have used the Mail::Header module for this purpose, it
turned out to be cleaner to parse the header into a hash ourselves using the idiom of the
Mail::SMTP mail client of Figure 7.2.
The output line contains the date, sender, and subject line, separated by tabs.
Lines 46–60: display_entity() subroutine—This subroutine is responsible for displaying a
MIME::Entity object. It is called recursively to process both the top-level object and each of its
subparts (and sub-subparts, if any).
We begin by retrieving the message's mail header as a MIME::Head object. If the header con-
tains a From: field, then we can conclude that it is the top-level entity. We print out the header
so that the user can see the sender's name and other fields.
Next we check whether the entity is multipart, by calling its is_multipart(), method. If this
method returns true, then we call handle_multipart() to prompt the user for each of the

Licensed by
parts. Otherwise, we invoke a subroutine called display_part() to display the contents of
the entity.
Lines 61–78: The handle_multipart() subroutine—The handle_multipart(), subrou-
tine loops through and processes each part of a multipart MIME::Entity object. We begin by

Stjepan Maric
calling the entity's parts() method to fetch each of the subparts as a MIME::Entity object. We
then call Perl's grep() built-in twice to sort the parts into those that we can display directly and
those that are to be treated as attachments that must be displayed using an external application.
Since we know how to display only plain text, we sort on the MIME type text/plain.
For each of the text/plain parts, we call the display_part() subroutine to print the message

4218908
body to the screen. If there are nontext attachments, we prompt the user for permission to display
them, and if so, invoke display_entity(), recursively on each attachment. This recursive
invocation of display_entity(), allows for attachments that are themselves multipart mes-
sages, such as forwarded e-mails.
Lines 79–99: The display_part() subroutine—The display_part() subroutine is invoked
to display a single-part MIME::Entity. Depending on the user's wishes, its job is to display, save,
or ignore the part.
We begin by retrieving the part's header, MIME type, description, and suggested filename for
saving (derived from the Content-Disposition: header, if present). We also recover the part's
MIME::Body object by calling its bodyhandle() method. This object gives us access to the
body's unencoded content.
If the part's MIME type is text/plain, we do not need an external viewer to display it. We simply
call the body object's print() method to print the contents to standard output. Otherwise, we
call get_viewer() to return the name of an external viewer that can display this MIME type.
We print a summary that contains the part's MIME type, description, and suggested filename,
and then prompt the user to view or save the part. Depending on the user's response, we invoke
save_body() to save the part's content to disk, or display_body() to launch the external
viewer to display it. This continues in a loop until the user chooses "n" to go to the next part.
If no viewer is defined for the part's MIME type, the user's only option is to save the content to
disk.
Lines 100–114: The save_body() subroutine—The save_body() subroutine accepts a
MIME::Body object and a default filename. It gives the user the opportunity to change the file-
name, opens the file, and writes the contents of the part to disk.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 POP, IMAP, and NNTP 171

The most interesting feature of this subroutine is the way that we treat the default filename for
the attachment. This filename is derived from the Content-Disposition: header, and as such is
untrusted data. Someone who wanted to spoil our day could choose a malicious pathname, such
as one that would overwrite a treasured configuration file. For this reason we forbid absolute
pathnames and those that contain the ".." relative path component. We also forbid filenames
that contain unusual characters such as shell metacharacters. Having satisfied these tests, we
extract the filename using a pattern match, thereby untainting it. Perl will now allow us to open
the file for writing. We do so and write the attachment's contents to it by calling the MIME::Body
object's print() method.
Lines 116–128: The display_body () subroutine—The display_body() subroutine is
called to launch an external viewer to display an attachment. It is passed a MIME::Body object,
and a command to launch an external viewer to display it.
To make this application a bit more interesting, we allow for two types of viewers: those that
read the body data from a file on disk and those that read from standard input. The former are
distinguished from the latter by containing the symbol %s, which will be replaced by the filename
before execution (this is a standard convention in the UNIX mailcap file).
We begin by calling the MIME::Body object's path() method to obtain the path to the temporary
file in which the object's data is stored. We then use this in a pattern substitution to replace any
occurrence of %s in the viewer command. If the substitution is successful, it returns a true value,
and we call system() to invoke the command.
Otherwise, we assume that the viewer will read the data from standard input. In this case, we
use open() to open a pipe to the viewer command, and invoke the body object's print()
method to print to the pipe filehandle. Before doing this, however, we set the PIPE handler to
IGNORE to avoid the program terminating unexpectedly because of a recalcitrant viewer.
This subroutine works correctly both for line-oriented applications, such as the Lynx HTML
viewer, and for windowing applications, such as XV.
Lines 129–137: The get_viewer() subroutine— get_viewer() is an extremely simple sub-
routine that uses a pattern match to examine the MIME type of the attachment and selects a
hard-coded viewer for it.
Lines 138–140: END{} block—This script's END{} block takes care of calling any leftover
MIME::Entity's purge() method. This deletes temporary files that might be left around if the
user interrupted the script's execution unexpectedly.

The PopParser Module

The other main component of the pop_fetch.pl script is the PopParser module, which subclasses
Net::POP3 in a way that enables it to parse MIME messages at the same time that it is fetching
them. Figure 8.4 shows the code for PopParser.pm.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 POP, IMAP, and NNTP 172

Figure 8.4. The PopParser module

Lines 1–6: Load modules—We turn on strict checking and load the Net::POP3 and MIME::Parser
modules. We use the global @ISA array to tell Perl that PopParser is a subclass of Net::POP3.
Lines 7–15: Override the new() method—We override the Net::POP3 new() method in order
to create and initialize a MIME::Parser for later use. We first invoke our parent's new() method
to create the basic object and connect to the remote host, create and configure a MIME::Parser
object, and store the parser for later use by invoking our parser() accessor method.
Lines 16–21: The parser() method—This method is an accessor for the MIME::Parser object
created during the call to new(). If we are called with a parser object on our subroutine stack,
we store it among our instance variables. Otherwise, we return the current parser object to the
caller.
The way we stash the parser object among our instance variables looks weird, but it is the
conventional way to store instance variables in filehandle objects:
${*$self}{'pp_parser'} = shift

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 POP, IMAP, and NNTP 173

What this is doing is referencing a hash in the symbol table that happens to have the same name
as our filehandle. We then index into that as if it were a conventionally created hash. We need
to store our instance variables this way because Net::POP3 ultimately descends from IO::Han-
dle, which creates and manipulates blessed filehandles, rather than more conventional blessed
hash references.
Lines 22–30: Override the get() method—The last part of this module overrides the Net::POP3
get() method. We are called with the number of the message to retrieve, which we pass to
getfh() to obtain a tied filehandle from which to read the desired message. The returned
filehandle is immediately passed to our stored MIME::Parser object to parse the message and
return a MIME::Entity object.
The nice thing about the design of the PopParser module is that message retrieval and message
parsing occur in tandem, rather than downloading the entire message and parsing it in two steps.
This saves considerable time for long messages.
There are a number of useful enhancements one could make to pop_fetch.pl. The one with the
greatest impact would be to expand the range and flexibility of the viewers for nontext attachments.
The best way to do this would be to provide support for the system /etc/mailcap and per-
user .mailcap files, which on UNIX systems map MIME types to external viewers. This would allow
the user to install and customize viewers without editing the code. Support for the mailcap system
can be found in the Mail::Cap module, which is part of Graham Barr's MailTools package. To use
Mail::Cap in the pop_fetch.pl script, replace lines 7 through 11 of Figure 8.3 with these lines:
use Mail::Cap;
my $mc = Mail::Cap-new;

This brings in the Mail::Cap module and creates a new Mail::Cap object that we can use to fetch
information from the mailcap configuration files.
Replace line 90, which invokes the get_viewer() subroutine, with the equivalent call from
Mail::Cap:
my $viewer = $mc->viewCmd($type);

This takes a MIME type and returns the command to invoke to view it if one is defined.
The last modification is to replace line 97, which invokes the display_ body() subroutine to
invoke the viewer on the body of an attachment, with the Mail::Cap equivalent:
$mc->view($type,$body->path);

This call looks up the appropriate view command for the specified MIME type, does any needed
string substitutions, and invokes the command using system().
We no longer need the get_viewer() and display_body() subroutines, because Mail::Cap
takes care of their functionality. You can delete them.
Other potential enhancements to this script include:
• the ability to reply to messages
• the ability to list old and new messages and jump directly to messages of interest
• a full windowing display using the text-mode Curses module or the graphical PerlTK package,
both available from CPAN
With a little work, you could turn this script into a full-featured e-mail client!

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 POP, IMAP, and NNTP 174

The IMAP Protocol

The POP3 protocol was designed to handle the case of a user who spends most of his or her time
working on a single machine. The mail client's job is to fetch the user's unread mail from time to
time from the remote mailbox server. The user then reads the mail and possibly sorts it into several
local mail folders.
Keeping track of mail becomes more complicated, however, when the user is moving around a lot:
working on a desktop in the office, a laptop while traveling, and another desktop at home. In this
case, the user wants to see the same set of mail files no matter where he or she happens to be
working. The Internet Message Access Protocol (IMAP) satisfies these needs by managing multiple
remote mail folders and transparently synchronizing them with local copies, providing the user with
a consistent view of stored e-mail. IMAP clients also provide the user with the ability to work off-line,
and with sophisticated server-side message search functions.
Unfortunately, the IMAP protocol is also rather complex and it does certain things that the simple
request/response model of Net::POP3 can't easily handle. Among other things, IMAP servers send
unsolicited messages to the client from time to time, for example to alert the client that new mail has
arrived. No fewer than three Perl modules on CPAN deal with IMAP: Mail::IMAPClient, Net::IMAP,
and Net::IMAP::Simple.
Mail::IMAPClient, written by David Kernen, provides the most functionality of the three, providing
methods for issuing all of the IMAP commands. However, Mail::IMAPClient does not do such a good
job at mapping the IMAP server's responses onto easily handled Perl objects. To use this module,
you'll need RFC 2060 on hand and be prepared to parse the server responses yourself.
Net::IMAP, written by Kevin Johnson, does a better job at handling the server's responses, and
provides a nifty callback interface that allows you to intercept and handle server events. Unfortu-
nately, the module is in alpha stage and the interfaces are changing. Also, at the time this book was
written, the module's documentation was incomplete.
Currently, the most usable interface to IMAP is Joao Fonseca's Net::IMAP::Simple, which provides
access to the subset of IMAP that is most like POP3. In fact, Net::IMAP::Simple shares much of
Net::POP3's method interface and is, to a large extent, plug compatible.
Like Net::POP3, you work with Net::IMAP::Simple by calling its new(), method to connect to an
IMAP server host, authenticate with login(), list messages with list() and top(), and retrieve
messages with get(). Unlike Net::POP3, Net::IMAP::Simple has no apop() method for authen-
ticating without plaintext passwords. To make up for this deficiency, it has the ability to work with
multiple remote mailboxes. Net::IMAP::Simple can list the user's mailboxes, create and delete them,
and copy messages from one folder to another.

Summarizing an IMAP Mailbox

The pop_stats.pl program from Figure 8.1 summarizes the contents of a POP3 mailbox. We'll now
enhance this program to summarize an IMAP mailbox. As an added feature, the new script, named
imap_stats.pl, indicates whether a message has been read. You call it like pop_stats.pl, but with an
additional optional command-line argument that indicates the name of the mailbox to summarize:
% pop_stats.pl lstein@localhoszt gd_bug_reports
lstein@localhost password:
gd has 6 messages (2 new)
1 Honza Pazdziora Re: ANNOUNCE: GD::Latin2 patch (fwd) read
2 Gurusamy Sarathy Re: patches for GD by Gurusamy Sarathy read
3 Honza Pazdziora Re: ANNOUNCE: GD::Latin2 patch (fwd) read
4 Erik Bertelsen GD-1.18, 2 minor typos read
5 Erik Bertelsen GD fails om some GIF's unread
6 Honza Pazdziora GDlib version 1.3 unread

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 POP, IMAP, and NNTP 175

Figure 8.5 lists imap_stats.pl.

Figure 8.5. Summarize an IMAP mailbox

Lines 1–5: Load modules—We load Net::IMAP::Simple, Mail::Header, and the Prompt Util mod-
ule used in earlier examples.
Lines 6–9: Process command-line arguments—We parse out the username and mailbox host
from the first command-line argument, and recover the mailbox name from the second. If no
mailbox name is provided, we default to INBOX, which is the default mailbox name on many
UNIX systems. We then prompt for the user's password.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 POP, IMAP, and NNTP 176

Lines 10–14: Connect to remote host—We call the Net::IMAP::Simple->new(), method to

connect to the designated host, and then call login() to authenticate. If these steps are suc-
cessful, we call the object's select() method to select the indicated mailbox. This call returns
the total number of messages in the mailbox, or if the mailbox is empty or missing, undef. We
fetch the number of the last message read by calling last().
Lines 15–24: List contents of the mailbox—We loop through each of the messages from first to
last. For each one, we fetch the header by calling top(), parse it into a Mail::Header object,
and retrieve the Subject: and From: fields. We also call the IMAP object's seen() method to
determine whether the message has been retrieved. We then print the message number,
sender, subject line, and read status.
Lines 26–32: clean_from() subroutine—This is the same subroutine we saw in the earlier
version of this program. It cleans up the sender addresses.

The Net::IMAP::Simple API

Although Net::IMAP::Simple is very similar to Net::POP3, there are some important differences. The
most dramatic difference is that Net::IMAP::Simple does not inherit from Net::Cmd and, therefore,
does not implement the message() or code() methods. Furthermore, Net::IMAP::Simple is not a
subclass of IO::Socket and, therefore, cannot be treated like a filehandle.
The new() and login() methods are similar to Net::POP3:

$imap = Net::IMAP::Simple->new($host [,$opt1=>$val1, $opt2=>$val2…])

The new() method constructs a new Net::IMAP::Simple object. The first argument is the name of the host,
and is not optional (unlike the Net::POP3 equivalent). This is followed by a series of options that are passed
directly to IO::Socket::INET.
If unsuccessful, new() returns undef and $! is set to some error code. Otherwise, it returns a Net::IMAP::Sim-
ple object connected to the server.
$messages = $imap->login($username,$password)
The login() method attempts to log into the server using the provided username and password. The user-
name and password are required, also a departure from Net::POP3. If successful, the method returns the
number of messages in the user's default mailbox, normally INBOX. Otherwise, login() returns undef.
Note that login() does not return 0E0 for a default mailbox that happens to be empty. The correct test for a
successful login is to test for a defined return value.

Several functions provide access to mailboxes.

@mailboxes = $imap->mailboxes
The mailboxes() method returns a list of all the user's mailboxes.
$messages = $imap->select($mailbox)
The select() method selects a mailbox by name, making it current. If the mailbox exists, select() returns
the number of messages it contains (0 for a mailbox that happens to be empty). If the mailbox does not exist,
the method returns undef and the current mailbox is not changed.
$success = $imap->create_mailbox($mailbox)
$success = $imap->delete_mailbox($mailbox)
$success = $imap->rename_mailbox($old_name,$new_name)
The create_mailbox(), delete_mailbox(), and rename_mailbox() methods attempt to create, de-
lete, and rename the named mailbox, respectively. They return true if successful, and false otherwise.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 POP, IMAP, and NNTP 177

Once you have selected a mailbox, you can examine and retrieve its contents.

$last_msgnum = $imap->last
The last() method returns the highest number of the read messages in the current mailbox, just as
Net::POP3 does. You can also get this information by calling the seen() method, as described below.
$arrayref = $imap->get($msgnum)
The get() method retrieves the message indicated by the provided message number from the current mail-
box. The return value is a reference to an array containing the message lines.
$handle = $imap->getfh($msgnum)
This is similar to get() but the return value is a filehandle that can be read from in order to retrieve the indicated
message. This method differs from the similarly named Net::POP3 method by returning a filehandle opened
on a temporary file, rather than a tied filehandle. This means that the entire message is transferred from the
remote server to the local machine behind the scenes before you can begin to ork with it.
$flag = $imap->delete($msgnum)
The delete() method marks the indicated message for deletion from the current mailbox. Marked messages
are not removed until the quit() method is called. However, there is no reset() call to undo a deletion.
$arrayref = $imap->top($msgnum)
The top() method returns the header of the indicated message as a reference to an array of lines. This format
is suitable for passing to the Mail::Header->new() method. There is no option for fetching a certain number
of lines from the body text.
$hashref = $imap->list
$size = $imap->list($msgnum)
The list() method returns information on the size of mailbox messages. Called without arguments, it returns
a hash reference in which the keys are message IDs, and the values are the sizes of the messages, in bytes.
Called with a message ID, the method returns the size of the indicated message, or if an invalid message
number was provided, it returns undef.
$flag = $imap->seen($msgnum)
The seen() method returns true if the indicated message has been read (by calling the get() method), or
false if it has not.
$success = $imap->copy($msgnum,$mailbox_destination)
The copy() method attempts to copy the indicated message from the current mailbox to the indicated desti-
nation mailbox. If successful, the method returns a true value and the indicated message is appended to the
end of its destination. You may wish to call delete() to remove the message from its original mailbox.

When you are finished, the quit() method will clean up:

$imap->quit()
quit() takes no arguments. It deletes all marked messages and logs off.

Internet News Clients

The Netnews system dates back to 1979, when researchers at Duke University and the University
of North Carolina designed a system to distribute discussion group postings that would overcome
the limitations of simple mailing lists [Spencer & Lawrence, 1998]. This rapidly grew into Usenet, a
global Internet-based bulletin-board system comprising thousands of named newsgroups.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 POP, IMAP, and NNTP 178

Because of its sheer size (more than 34,000 newsgroups and daily news flow rates measured in
the gigabytes), Usenet has been diminishing in favor among Internet users. However, there has
been a resurgence of interest recently in using Netnews for private discussion servers, helpdesk
applications, and other roles in corporate intranets.
Netnews is organized in a two-level hierarchy. At the upper level are the newsgroups. These have
long meaningful names like comp.graphics.rendering.raytracing. Each newsgroup, in turn, contains
zero or more articles. Users post articles to their local Netnews server, and the Netnews distribution
software takes care of distributing the article to other servers. Within a day or so, a copy of the article
appears on every Netnews server in the world. Articles live on Netnews for some period before they
are expired. Depending on each server's storage capacity, a message may be held for a few days
or a few weeks before expiring it. A few large Netnews servers, such as the one at
www.deja.com, hold news articles indefinitely.
Newsgroups are organized using a hierarchical namespace. For example, all newsgroups beginning
with comp. are supposed to have something to do with computers or computer science, and all
those beginning with soc.religion. are supposed to concern religion in society. The creation and
destruction of newsgroups, by and large, is controlled by a number of senior administrators. The
exception is the alt hierarchy, in which newsgroups can be created willy-nilly by anyone who desires
to do so. Some very interesting material resides in these groups.
Regardless of its position in the namespace hierarchy, a newsgroup can be moderated or unmod-
erated. Moderated groups are "closed." Only a small number of people (typically a single moderator)
have the right to post to the newsgroup. When others attempt to post to the newsgroup, their posting
is automatically forwarded to the moderator via e-mail. The moderator then posts the message at
his or her discretion. Anyone can post to unmoderated groups. The posted article is visible imme-
diately on the local server, and diffuses quickly throughout the system.
Articles are structured like e-mails, and in fact share the same RFC 822 specification. Figure 8.6
shows a news article recently posted to comp.lang.perl.modules. The article consists of a message
header and body. The header contains several fields that you will recognize from the standard e-
mail, such as the Subject: and From: lines, and some fields that are specific to news articles, such
as Article:, Path:, Message-ID:, Distribution:, and References:. Many of these fields are added au-
tomatically by the Netnews server.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 POP, IMAP, and NNTP 179

Figure 8.6. A typical Netnews article

To construct a valid Netnews article, you need only take a standard e-mail message and add a
Newsgroups: header containing a comma-delimited list of newsgroups to post to. Another frequently
used article header is Distribution:, which limits the distribution of an article. Valid values for Distri-
bution: depend on the setup of your local Netnews server, but they are typically organized geo-
graphically. For example, the usa distribution limits message propagation to the political boundaries
of the United States, and nj limits distribution to New Jersey. The most common distribution is
world, which allows the article to propagate globally.
Other article header fields have special meaning to the Netnews system, and can be used to create
control messages that cancel articles, add or delete newsgroups, and perform other special func-
tions. See [Spencer and Lawrence 1998] for information on constructing your own control mes-
sages.
Netnews interoperates well with MIME. An article can have any number of MIME-specific headers,
parts, and subparts, and MIME-savvy news readers are able to decode and display the parts.
Articles can be identified in either of two ways. Within a newsgroup, an article can be identified by
its message number within the group. For example, the article shown in Figure 8.6 is message
number 36,166 of the newsgroup comp.lang.perl.modules. Because articles are constantly expiring
and being replaced by new ones, the number of the first message in a group is usually not 1, but
more often a high number. The message number for an article is stable on any given news server.
On two subsequent days, you can retrieve the same article by entering a particular newsgroup and
retrieving the same message number. However, message numbers are not stable across servers.
An article's number on one news server may be quite different on another server.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 POP, IMAP, and NNTP 180

The other way to identify articles is by the message ID. The message ID of the sample article is
<[email protected]>, including the angle brackets at either
side. Message IDs are unique, global identifiers that remain the same from server to server.

Net::NNTP
Historically, Netnews has been distributed in a number of ways, but the dominant mode is now the
Net News Transfer Protocol, or NNTP, described in RFC 977. NNTP is used both by Netnews
servers to share articles among themselves and by client applications to scan and retrieve articles
of interest. Graham Barr's Net::NNTP module, part of the libnet utilities, provides access to NNTP
servers.
Like other members of the libnet clan, Net::NNTP descends from Net::Cmd and inherits that mod-
ule's methods. Its API is similar to Net::POP3 and Net::IMAP::Simple. You connect to a remote
Netnews server, creating a new Net::NNTP object, and use this object to communicate with the
server. You can list and filter newsgroups, make a particular newsgroup current, list articles, down-

Licensed by
load them, and post new articles.
newsgroup_stats.pl is a short script that uses Net::NNTP to find all newsgroups that match a pattern
and count the number of articles in each. For example, to find all the newsgroups that have some-
thing to do with Perl, we could search for the pattern "*.perl*" (the output has been edited slightly

Stjepan Maric
for space):
% newsgroup_stats.pl '*.perl*'
alt.comp.perlcgi.freelance 454 articles
alt.flame.marshal.perlman 3 articles

4218908
alt.music.perl-jam 11 articles
alt.perl.sockets 45 articles
comp.lang.perl.announce 43 articles
comp.lang.perl.misc 18940 articles
comp.lang.perl.moderated 622 articles
comp.lang.perl.modules 2240 articles
comp.lang.perl.tk 779 articles
cz.comp.lang.perl 63 articles
de.comp.lang.perl.cgi 1989 articles
han.comp.lang.perl 174 articles
it.comp.lang.perl 715 articles
japan.comp.lang.perl 53 articles

Notice that the pattern match wasn't perfect, and we matched alt.music.perl-jam as well as news-
groups that have to do with the language. Figure 8.7 lists the code.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 POP, IMAP, and NNTP 181

Figure 8.7. match_newsgroup.pl script

Lines 1–3: Load modules—We turn on strict checking and load the Net::NNTP module.
Line 4: Create new Net::NNTP object—We call Net::NNTP->new() to connect to a Netnews
host. If the host isn't specified explicitly, then Net::NNTP chooses a suitable host from environ-
ment variables or the default NNTP server specified when libnet was installed.
Lines 5–6: Print stats and quit—For each argument on the command line, we call the
print_stats() print_stats() subroutine to look up the pattern and print out matching news-
groups. We then call the NNTP object's quit() method.
Lines 7–17: print_stats() subroutine—In the print_stats() subroutine we invoke the
NNTP object's newsgroups() method to find newsgroups that match a pattern. If successful,
newsgroups() returns a hash reference in which the keys are newsgroup names and the
values are brief descriptions of the newsgroup.
If the value returned by newsgroups() is undef or empty, we return. Otherwise, we sort the
groups alphabetically by name, and loop through them. For each group, we call the NNTP ob-
ject's group() method to return a list containing information about the number of articles in the
group and the message numbers of the first and last articles. We print the newsgroup name and
the number of articles it contains.

The Net::NNTP API

The Net::NNTP API can be divided roughly into those methods that deal with the server as a whole,
those that affect entire newsgroups, and those that concern individual articles in a newsgroup.
Newsgroups can be referred to by name or, for some methods, by a wildcard pattern match. The
pattern-matching system used by most NNTP servers is similar to that used by the UNIX and DOS
shells. "*" matches zero or more of any characters, "?" matches exactly one character, and a set of
characters enclosed in square brackets, as in "[abc]", matches any member of the set. Bracketed
character sets can also contain character ranges, as in "[0–9]" to match the digits 0 through 9, and

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 POP, IMAP, and NNTP 182

the "^" character may be used to invert a set—for example, "[^A–Z]" to match any character that is
not in the range A through Z. Any other character matches itself exactly once. As in the shell (and
unlike Perl's regular expression operations), NNTP patterns are automatically anchored to the be-
ginning and end of the target string.
Articles can be referred to by their number in the current newsgroup, by their unique message IDs,
or, for some methods, by a range of numbers. In the latter case, the range is specified by providing
a reference to a two-element array containing the first and last message numbers of the range.
Some methods allow you to search for particular articles by looking for wildcard patterns in the
header or body of the message using the same syntax as newsgroup name wildcards.
Other methods accept times and dates, as for example, the newgroups() method that searches
for newsgroups created after a particular date. In all cases, the time is expressed in its native Perl
form as seconds since the epoch, the same as that returned by the time() built-in.
In addition to the basic NNTP functions, many servers implement a number of extension commands.
These extensions make it easier to search a server for articles that match certain criteria and to
summarize quickly the contents of a discussion group. Naturally, not all servers support all exten-
sions, and in such cases the corresponding method usually returns undef In the discussion that
follows, methods that depend on NNTP extensions are marked.
We look first at methods that affect the server itself.

$nntp = Net::NNTP->new([$host],[$option1=>$val1,$option2=>$val2…])
The new() method attempts to connect to an NNTP server. The $host argument is the DNS name or IP
address of the server. If not specified, Net::NNTP looks for the server name in the NNTPSERVER and NEWS-
HOSTS environment variables first, and then in the Net::Config nntp_hosts key. If none of these variables is
set, the Netnews host defaults to news.

In addition to the options accepted by IO::Socket::INET, Net::NNTP recognizes the name/value

pairs shown in Table 8.2.
By default, when Net::NNTP connects to a server, it announces that it is a news reader rather than
a news transport agent (a program chiefly responsible for bulk transfer of messages). If you want
to act like a news transfer agent and really know what you're doing, provide new() with the option
Reader=>0.

$success = $nntp->authinfo($user => $password)

Some NNTP servers require the user to log in before accessing any information. The authinfo() method
takes a username and password, and returns true if the credentials were accepted.
$ok = $nntp->postok()
postok() returns true if the server allows posting of new articles. Even though the server as a whole may
allow posting, individual moderated newsgroups may not.
$time = $nntp->date()
The date() method returns the time and date on the remote server, as the number of seconds since the
epoch. You can convert this into a human-readable time-date string using the localtime() or gmtime()
functions.

Table 8.2. Net::NNTP->new() Options

Option Description Default
Timeout Seconds to wait for response from server 120
Debug Turn on verbose debugging information undef

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 POP, IMAP, and NNTP 183

Option Description Default

Port Numeric or symbolic name of port to connect to 119
Reader Act like a news reader 1

$nntp->slave()
$nntp->reader() [extension]
The slave() method puts the NNTP server into a mode in which it expects to engage in bulk transfer with
the client. The reader() method engages a mode more suitable for the interactive transfer of individual
articles. Unless explicitly disabled, reader() is issued automatically by the new() method.
$nntp->quit()
The quit() method cleans up and severs the connection with the server. This is also issued automatically
when the NNTP object is destroyed.

Once created, you can query an NNTP object for information about newsgroups. The following
methods deal with newsgroup-level functions.

$group_info = $nntp->list()
The list() method returns information about all active newsgroups. The return value is a hash reference in
which each key is the name of a newsgroup, and each value is a reference to a three-element array that
contains group information. The elements of the array are [$first,$last,$postok], where $first and
$last are the message numbers of the first and last articles in the group, and $postok is "y" if the posting
is allowed to the group or "m" if the group is moderated.
$group = $nntp->group([$group])
($articles,$first,$last,$name) = $nntp->group([$group])
The group() method gets or sets the current group. Called with a group name as its argument, it sets the
current group used by the various article-retrieval methods.
Called without arguments, the method returns information about the current group. In a scalar context, the
method returns the group name. In a list context, the method returns a four-element list that contains the number
of articles in the group, the message numbers of the first and last articles, and the name of the group.
$group_info = $nntp->newgroups($since [,$distributions])
The newgroups() method works like list(), but returns only newsgroups that have been created more
recently than the date specified in $since. The date must be expressed in seconds since the epoch as returned
by time().
The $distributions argument, if provided, limits the returned list to those newsgroups that are restricted
to the specified distribution(s). You may provide a single distribution name as a string, such as nj, or a reference
to an array of distributions, such as ['nj','ct','ny'] for the New York tristate region.
$new_articles = $nntp->newnews($since [,$groups [,$distributions]])
The newnews() method returns a list of articles that have been posted since the time value indicated by
$since. You may optionally provide a group pattern or a reference to an array of patterns in $groups, and
a distribution pattern or reference to an array of distribution patterns in $distributions.
If successful, the method returns a reference to an array that contains the message IDs of all the matching
articles. You may then use the article() and/or articlefh() methods described below to fetch the con-
tents of the articles. This method is chiefly of use for mirroring an entire group or set of groups.
$group_info = $nntp->active([$pattern]) [extension]
The active() method works like list(), but limits retrieval to those newsgroup that match the wildcard
pattern $pattern. If no pattern is specified, active() is functionally equivalent to list().
This method and the ones that follow all use common extensions to the NTTP protocol, and are not guaranteed
to work with all NNTP servers.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 POP, IMAP, and NNTP 184

$group_descriptions = $nntp->newsgroups([$pattern]) [extension]

$group_descriptions = $nntp->xgtitle($pattern) [extension]
The newsgroups() method takes a newsgroup wildcard pattern and returns a hash reference in which the
keys are group names and the values are brief text descriptions of the group. Because many Netnews sites
have given up on keeping track of all the newsgroups (which appear and disappear very dynamically), de-
scriptions are not guaranteed to be available. In such cases, they appear as the string "No description", as "?",
or simply as an empty string.
xgtitle() is another extension method that is functionally equivalent to newsgroups(), with the exception
that the group pattern argument is required.
$group_times = $nntp->active_times()[extension]
This method returns a reference to a hash in which the keys are newsgroup names and the values are a
reference to a two-element list giving the time the group was created and the ID of its creator. The creator ID
may be something useful, like an e-mail address, but is more often something unhelpful, like "newsmaster."
$distributions = $nntp->distributions() [extension]
$subscriptions = $nntp->subscriptions() [extension]
These two methods return information about local server distribution and subscription lists. Local distributions
can be used to control the propagation of messages in the local area network; for example, a company that is
running multiple NNTP servers might define a distribution named engineering. Subscription lists are used to
recommend lists of suggested newsgroups to new users of the system.
distributions() returns a hash reference in which the keys are distribution names and the values are
human-readable descriptions of the distributions. subscriptions() returns a hash reference in which the
keys are subscription list names and the values are array references containing the newsgroups that belong
to the subscription list.

Once a group is selected using the group() method, you can list and retrieve articles. Net::NNTP
gives you the option of retrieving a specific article by specifying its ID or message number, or iter-
atively fetching articles in sequence, starting at the current message number and working upward.

$article_arrayref = $nntp->article ([$message] [,FILEHANDLE])

The article() method retrieves the indicated article. If $message is numeric, it is interpreted as a message
number in the current newsgroup. Net::NNTP returns the contents of the indicated message, and sets the
current message pointer to this article. An absent first argument or a value of undef retrieves the current article.
If the first argument is not numeric, Net::NNTP treats it as the article's unique message ID. Net::NNTP retrieves
the article, but does not change the position of the current message pointer. In fact, when referring to an article
by its message ID, it is not necessary for the indicated article to belong to the current group
The optional filehandle argument can be used to write the article to the specified destination. Otherwise, the
article's contents (header, blank separating line, and body) are returned as a reference to an array containing
the lines of the article.

Should something go wrong, article() returns undef and $nntp->message contains an error
message from the server. A common error is "no such article number in this group", which can be
issued even when the message number is in range because of articles that expire or are cancelled
while the NNTP session is active.
Other article-retrieval methods are more specialized.

$header_arrayref = $nntp->head([$message] [,FILEHANDLE])

$body_arrayref = $nntp->body([$message] [,FILEHANDLE])
The head() and body() methods work like article() but retrieve only the header or body of the article,
respectively.
$fh = $nntp->articlefh([$message])

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 POP, IMAP, and NNTP 185

$fh = $nntp->headfh([$message])
$fh = $nntp->bodyfh([$message])
These three methods act like article(), head(), and body(), but return a tied filehandle from which the
contents of the article can be retrieved. After using the filehandle, you should close it. For example, here is
one way to read message 10000 of the current newsgroup:
$fh = $nntp->articlefh(10000) or die $nntp->message;
while (<$fh>) {
print;
}

$msgid = $nntp->next()
$msgid = $nntp->last()
$msgid = $nntp->nntpstat($message)
The next(), last(), and nntpstat() methods control the current article pointer. next() advances the
current article pointer to the next article in the newsgroup, and last() moves the pointer to the previous entry.
The nntpstat() method moves the current article pointer to the position indicated by $message, which
should be a valid message number. After setting the current article pointer, all three methods return the mes-
sage ID of the current article.

Net::NMTP allows you to post new articles using the post(), postfh(), and ihave() methods.

$success = $nntp->post([$message])
The post() method posts an article to Netnews. The posted article does not have to be directed to the current
newsgroup; in fact, the news server ignores the current newsgroup when accepting an article and looks only
at the contents of its Newsgroups: header. The article may be provided as an array containing the lines of the
article or as a reference to such an array. Alternatively, you may call post() with no arguments and use the
datasend() and dataend() methods inherited from Net::Cmd to send the article one line at a time.
If successful, post() returns a true value. Otherwise, it returns undef and $nntp->message contains an
error message from the server.
$fh = $nntp->postfh()
The postfh() method provides an alternative interface for posting an article. If the server allows posting, this
method returns a tied filehandle to which you can print the contents of the article. After finishing, be sure to
close the filehandle. The result code from close() indicates whether the article was accepted by the server.
$wants_it = $nntp->ihave($messageID[,$message])
The ihave() method is chiefly of use for clients that are acting as news relays. The method asks the Netnews
server whether it wishes to accept the article whose ID is $messageID.
If the server indicates its assent, it returns a true result. The article must then be transferred to the server, either
by providing the article's contents in the $message argument or by sending the article one line at a time using
the Net::Cmd datasend() and dataend() methods. $message can be an array of article lines or a reference
to such an array.

Last, several methods allow you to search for particular articles of interest.

$header_hashref = $nntp->xhdr($header,$message_range) [extension]

$header_hashref = $nntp->xpat($header,$pattern,$message_range) [extension]
$references = $nntp->xrover($message_range) [extension]
The xhdr() method is an extension function that allows you to retrieve the value of a header field from multiple
articles. The $header article is the name of an article header field, such as "Subject". $message_range is
either a single message number or a reference to a two-element array containing the first and last messages
in the desired range. If successful, xhdr() returns a hash reference in which the keys are the message
numbers (not IDs) and the values are the requested header fields.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 POP, IMAP, and NNTP 186

The header field is case-insensitive. However, not all headers can be retrieved in this way because NNTP
servers typically index only that subset of the headers used to generate overview listings (see the next method).
The xpat() method is similar to xhdr(), but it filters the articles returned for those with $header fields that
match the wildcard pattern in $pattern. The xrover() method returns the cross-reference fields for articles
in the specified range. It is functionally identical to:
$xref = $nntp->xhdr('References',[$start,$end]);

The result of this call is a hash reference in which the keys are message numbers and the values are the
message IDs that the article refers to. These are typically used to reconstruct discussion threads.
$overview_hashref = $nntp->xover($message_range) [extension]
$format_arrayref = $nntp->overview_fmt() [extension]
The overview_fmt() and xover() methods return newsgroup "overview" information. The overview is a
summary of selected article header fields; it typically contains the Subject: line, References:, article Date:, and
article length. It is used by newsreaders to index, sort, and thread articles.
Pass the xover() method a message range (a single message number or a reference to an array containing
the extremes of the range). If successful, the method's return value is a hash reference in which each key is
a message number and each value is a reference to an array of the overview fields.
To discover what these fields are, call the overview_fmt() method. It returns an array reference containing
field names in the order in which they appear in the arrays returned by xover(). Each field is followed by a
colon and, occasionally, by a server-specific modifier. For example, my laboratory's Netnews server returns
the following overview fields:
('Subject:','From:','Date:','Message- ID:','References:',
'Bytes:','Lines:','Xref:full')

If you would prefer the values of the overview array to be a hash reference rather than an array
reference, you can use the small subroutine shown here to do the transformation. The trick is to use
the list of field names returned by overview_fmt() to create a hash slice to which we assign the
article overview array:
sub get_overview {
my ($nntp,$range) = @_;
my @fields = map {/(\w+):/&& $1} @{$nntp->overview_fmt};
my $over = $nntp->xover($range) || return;
foreach (keys %$over) {
my $h = {};
@{$h}{@fields}= @{$over->{$_}};
$over->{$_} = $h;
}
return $over;
}

Use the subroutine like this:

$over = get_overview($nntp,[30000,31000]);

The returned value will have a structure like this:

{
30000 => {
'Bytes' => 2704
'Date' => 'Sat, 27 May 2000 19:35:10 GMT'
'From' => '[email protected]'
'Lines' => 72
'Message-ID' => '<[email protected]>'
'References' => ''
'Subject' => 'mod_perl make test'
'Xref' => 'Xref: rQdQ comp.lang.perl.modules:34162'
},
30001 => {

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 POP, IMAP, and NNTP 187

'Bytes' => 1117

'Date' => 'Sat, 27 May 2000 20:28:22 GMT'
'From' => 'Robert Gasiorowski <[email protected]>'
'Lines' => 6
'Message-ID' => '<[email protected]>'
'References' => ''
'Subject' => 'installing module as non-root'
'Xref' => 'Xref: rQdQ comp.lang.perl.modules:34163'
},
....
}

A News-to-Mail Gateway
The last code example of this chapter is a custom news-to-mail gateway. It periodically scans Net-
news for articles of interest, bundles them into a MIME message, and mails them via Internet mail.
Each time the script is run it keeps track of the messages it has previously sent and only sends
messages that haven't been seen before.
You control the script's scope by specifying a list of newsgroups and, optionally, one or more patterns
to search for in the subject lines of the articles contained in the newsgroups. If you don't specify any
subject-line patterns, the script fetches the entire contents of the listed newsgroups.
The subject-line patterns take advantage of Perl's pattern-matching engine, and can be any regular
expression. For performance reasons, however, we use the built-in NNTP wildcard patterns for
newsgroup names.
The following command searches the comp.lang.perl.* newsgroups for articles that have the word
"Socket" or "socket" in the subject line. Matching articles will be mailed to the local e-mail address
lstein. Options include -subject, to specify the subject pattern match, -mail to set the mail recipient(s),
and -v to turn on verbose progress messages.
% scan_newsgroups.pl -v -mail lstein -subj '[sS]ocket' 'comp.lang.perl.*'
Searching comp.lang.perl.misc for matches
Fetching overview for comp.lang.perl.misc
found 39 matching articles
Searching comp.lang.perl.announce for matches
Fetching overview for comp.lang.perl.announce
found 0 matching articles
Searching comp.lang.perl.tk for matches
Fetching overview for comp.lang.perl.tk
found 1 matching articles
Searching comp.lang.perl.modules for matches
Fetching overview for comp.lang.perl.modules
found 4 matching articles
44 articles, 40 unseen
sending e-mail message to lstein

The received e-mail message contains a brief prologue that describes the search and newsgroup
patterns, followed by the matching articles. Each article is attached as an enclosure of MIME type
message/rfc822. Depending on the reader's mail-reading software, the enclosures are displayed
as either in-line components of the message or attachments. The result is particularly nice in the
Netscape mail reader (Figure 8.8) because each article is displayed using fancy fonts and hyper-
links.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 POP, IMAP, and NNTP 188

Figure 8.8. E-mail message sent from scan_newsgroups.pl

Figure 8.9 lists the code for scan_newsgroups.pl.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 POP, IMAP, and NNTP 189

Figure 8.9. The scan_newsgroup.pl script

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 POP, IMAP, and NNTP 190

Lines 1–7: Load modules—We load the Net::NNTP and MIME::Entity modules, as well as the
Getopt::Long module for argument processing. We need to keep track of all the messages that
we have found during previous runs of the script, and the easiest way to do that is to keep the
message IDs in an indexed DBM database. However, we don't know a priori what DBM library
is available, so we import the AnyDBM_File module, which chooses a library for us. The code
contained in the BEGIN{} block changes the DBM library search order, as described in the
AnyDBM_File documentation.
We also load the Fcntl module in order to have access to several constants needed to initialize
the DBM file.
Lines 9–22: Define constants—We choose a name for the DBM file, a file named .newscache
in the user's home directory, and create a usage message.
Lines 23–25: Declare globals—The first line of globals correspond to command-line options. The
second line of globals are various data structures manipulated by the script. The %Seen hash
will be tied to the DBM file. Its keys are the message IDs of articles that we have previously

Licensed by
retrieved. %Articles contains information about the articles recovered during the current
search. Its keys are message IDs, and its values are hash references of header fields derived
from the overview index. Last, @Fields contains the list of header fields returned by the
xover() method.

Stjepan Maric
Lines 26–34: Process command-line arguments—We call GetOptions() to process the com-
mand-line options, and then check consistency of the arguments. If the e-mail recipient isn't
explicitly given on the command line, we default to the user's login name.
Lines 35–36: Open connection to Netnews server—We open a connection to the Netnews server
by calling Net::NNTP->new(). If the server isn't explicitly given on the command line, the

4218908
$SERVER option is undefined and Net::NNTP picks a suitable default.
Lines 37–39: Open DBM file—We tie %Seen to the .newscache file using the AnyDBM_File
module. The options passed to tie() cause the file to be opened read/write and to be created
with file mode 0640 (-rw-r-----), if it doesn't already exist.
Lines 40–41: Compile the pattern match—For efficiency's sake, we compile the pattern matches
into an anonymous subroutine. This subroutine takes the text of a subject line and returns true
if all the patterns match, and false otherwise. The match_code() subroutine takes the list of
pattern matches, compiles them, and returns an appropriate code reference.
Lines 42–43: Expand newsgroup patterns—We pass the list of newsgroups to a subroutine
named expand_newsgroups(). It calls the NNTP server to expand the wildcards in the list of
newsgroups and returns the expanded list of newsgroup names.
Lines 44–45: Search for matching articles—We loop through the expanded list of newsgroups
and call grep_group() for each one. The arguments to grep_group() consist of the news-
group name and a code reference to filter them. Internally, grep_group() accumulates the
matched articles' message IDs into the %Articles hash. We do it this way because the same
article may be cross-posted to several related newsgroups; using the article IDs in a hash avoids
accumulating duplicates.
Lines 46–48: Filter out articles already seen—We use Perl's grep() function to filter out articles
whose message IDs are already present in the tied %Seen hash. New article IDs are added to
the hash so that on subsequent runs we will know that we've seen them. The unseen article IDs
are assigned to the @to_fetch array.
If the user ran the script with the -all option, we short-circuit the grep() operation so that all
articles are retrieved, including those we've seen before. This does not affect the updating of
the tied %Seen hash.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 POP, IMAP, and NNTP 191

Lines 49–52: Add articles to an outgoing mail message and quit—We pass the list of article IDs
to send_mail(), which retrieves their contents and adds them to an outgoing mail message.
We then call the NNTP object's quit() method to disconnect from the server, and exit our-
selves.
Lines 53–62: The match_code() subroutine—The match_code() subroutine takes a list of
zero or more patterns and constructs a code reference on the fly. The subroutine is built up line-
by-line in a scalar variable called $code. The subroutine is designed to return true only if all the
patterns match the passed subject line. If no patterns are specified, the subroutine returns true
by default. If the -insensitive option was passed to the script, we do case-insensitive pattern
matches with the i flag. Otherwise, we do case-sensitive matches.
After constructing the subroutine code, we eval() it and return the result to the caller. If the
eval() fails (presumably because of an error in one or more of the regular expressions), we
propagate the error message and die.
Lines 63–73: The expand_newsgroups() subroutine—The expand_newsgroups(), sub-
routine takes a list of newsgroup patterns and calls the NNTP object's newsgroups() method
on each of them in turn, expanding them to a list of valid newsgroup names. If a newsgroup
contains no wildcards, we just pass it back unchanged.
Lines 74–85: The grep_group() subroutine— grep_group() scans the specified news-
group for articles whose subject lines match a set of patterns. The patterns are provided in the
form of a code reference that returns true if the subject line matches.
We call the get_overview() subroutine to return the server's overview index for the news-
group. get_overview() returns a hash reference in which each key is a message number
and each value is a hash of indexed header fields. We step through each message, recover its
Subject: and Message-ID: fields, and pass the subject field to the pattern-matching code refer-
ence. If the code reference returns false, we go on to the next article. Otherwise, we add the
article's message ID and overview data to the %Articles global.
When all articles have been examined, we return to the caller the number of those that matched.
Lines 89–102: The get_overview() subroutine—The get_overview() subroutine used
here is a slight improvement over the version shown earlier. We start by calling the NNTP object's
group() method, recovering the newsgroup's first and last message numbers. We then call
the object's overview_fmt() method to retrieve the names of the fields in the overview index.
Since this information isn't going to change during the lifetime of the script, however, we cache
it in the @Fields global and call overview_fmt() only if the global is empty. Before assigning
to @Fields, we clean up the field names by removing the ":" and anything following it.
We recover the overview for the entire newsgroup by calling the xover() method for the range
spanning the first and last article numbers. We now loop through the keys of the returned over-
view hash, replacing its array reference values, which lists fields by position, with anonymous
hashes that list fields by name. In addition to recording the header fields that occur in the article
itself, we record a pseudofield named Message-Number: that contains the group name and
message number in the form group.name:number. We use this information during e-mail con-
struction to create the default name for the article enclosure.
Lines 103–124: The send_mail() subroutine— send_mail() is called with an array of article
IDs to fetch, and is responsible for constructing a multipart MIME message containing each
article as an attachment.
We create a short message prologue that summarizes the program's run-time options and create
a new MIME::Entity by calling the build() method. The message starts as a single-part mes-
sage of type text/plain, but is automatically promoted to a multipart message as soon as we start
attaching articles to it.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 POP, IMAP, and NNTP 192

We then call attach_article() for each article listed in $to_fetch. This array may be
empty, in which case we make no attachments. When all articles have been attached, we call
the MIME entity's smtpsend() method to send out the mail using the Mail::Mailer SMTP
method, and clean up any temporary files by calling the entity's purge() method.
Lines 125–134: The attach_article() subroutine—For the indicated message ID we fetch
the entire article's contents as an array of lines by calling the NNTP object's article() method.
We then attach the article to the outgoing mail message, specifying a MIME type of message/
rfc822, a description corresponding to the article's subject line, and a suggested filename derived
from the article's newsgroup and message number (taken from the global %Articles hash).
An interesting feature of this script is the fact that because we are storing unique global message
IDs in the .newscache hashed database, we can switch to a different NNTP server without worrying
about retrieving articles we have already seen.

Summary
Net::POP3 and Net::IMAP::Simple allow client programs to receive and process Internet mail.
Net::NNTP provides access to the Netnews system via the NNTP protocol. These modules can be
combined with MIME-Tools to perform sophisticated mail processing and sorting tasks.
The ease with which the Net::*, Mail::*, and MIME::* modules interoperate is a tribute to the design
skills of the authors of those modules as well as to the elegance of the Internet mail system itself.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 193

Chapter 9. Web Clients

In the previous chapters we reviewed client modules for sending and receiving Internet mail, trans-
ferring files via FTP, and interacting with Netnews servers. In this chapter we look at LWP, the Library
for Web access in Perl. LWP provides a unified API for interacting with Web, FTP, News and Mail
servers, as well as with more obscure services such as Gopher.
With LWP you can (1) request a document from a remote Web server using its URL; (2) POST data
to a Web server, emulating the submission of a fill-out form; (3) mirror a document on a remote Web
server in such a way that the document is transferred only if it is more recent than the local copy;
(4) parse HTML documents to recover links and other interesting features; (5) format HTML docu-
ments as text and postscript; and (6) handle cookies, HTTP redirects, proxy servers, and HTTP user
authentication. Indeed, LWP implements all the functionality one needs to write a Web browser in
Perl, and if you download and install the Perl-TK distribution, you'll find it contains a fully functional
graphical Web browser written on top of LWP.
The base LWP distribution contains 35 modules, and another dozen modules are required for HTML
parsing and formatting. Because of its size and scope, we will skim the surface of LWP. For an
exhaustive treatment, see LWP's POD documentation, or the excellent, but now somewhat dated
Web Client Programming with Perl [Wong 1999].

Installing LWP
The first version of LWP appeared in 1995, and was written by Martijn Koster and Gisle Aas. It has
since been maintained and extended by Gisle Aas, with help from many contributors.
The basic LWP library, distributed via CPAN in the file libwww-X.XX.tar.gz (where X.XX is the most
recent version number), provides supports for the HTTP, FTP, Gopher, SMTP, NNTP, and HTTPS
(HTTP over Secure Sockets Layer) protocols. However, before you can install it, you must install a
number of prerequisite modules:

URI URL parsing and manipulation

Net::FTP to support ftp://URLs
MIME::Base64 to support HTTP Basic authentication
Digest::MD5 to support HTTP Digest authentication
HTML::HeadParser for finding the <BASE> tag in HTML headers

You could download and install each of these modules separately, but the easiest way is to install
LWP and all its prerequisites in batch mode using the standard CPAN module. Here is how to do
this from the command line:
% perl -MCPAN -e 'install Bundle::LWP'

This loads the CPAN module and then calls the install() function to download, build, and install
LWP and all the ancillary modules that it needs to run.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Web Clients 194

The HTML-parsing and HTML-formatting modules were once bundled with LWP, but are now dis-
tributed as separate packages named HTML-Parser and HTML-Formatter, respectively. They each
have a number of prerequisites, and again, the easiest way to install them is via the CPAN module
using this command:
% perl -CPAN -e 'install HTML::Parser' -e 'install HTML::Formatter'

If you want to install these libraries manually, here is the list of the packages that you need to
download and install:

HTML-Parser HTML parsing

HTML-Tree HTML syntax-tree generation
Font-AFM Postscript font metrics
HTML-Format HTML formatting

To use the HTTPS (secure HTTP) protocol, you must install one of the Perl SSL modules,
IO::Socket::SSL, as well as OpenSSL, the open source SSL library that IO::Socket::SSL depends
on. OpenSSL is available from http://www.openssl.org/.
LWP is pure Perl. You don't need a C compiler to install it. In addition to the module files, when you
install LWP you get four scripts, which serve as examples of how to use the library, as well as useful
utilities in their own right. The scripts are:
• lwp-request—Fetch a URL and display it.
• lwp-download—Download a document to disk, suitable for files too large to hold in memory.
• lwp-mirror—Mirror a document on a remote server, updating only the local copy if the remote
one is more recent.
• lwp-rget—Copy an entire document hierarchy recursively.

LWP Basics
Figure 9.1 shows a script that downloads the URL given on the command line. If successful, the
document is printed to standard output. Otherwise, the script dies with an appropriate error mes-
sage. For example, to download the HTML source for Yahoo's weather page, located at http://
www.yahoo.com/r/wt, you would call the script like this:

Figure 9.1. Fetch a URL using LWP's object-oriented interface

% get_url.pl http://www.yahoo.com/r/wt > weather.html

The script can just as easily be used to download a file from an FTP server like this:

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Web Clients 195

% get_url.pl ftp://www.cpan.org/CPAN/RECENT

The script will even fetch news articles, provided you know the message ID:
% get_url.pl news:[email protected]

All this functionality is contained in a script just 10 lines long.

Lines 1–3: Load modules—We turn on strict syntax checking and load the LWP module.
Line 4: Read URL—We read the desired URL from the command line.
Line 5: Create an LWP::UserAgent—We create a new LWP::UserAgent object by calling its
new() method. The user agent knows how to make requests on remote servers and return their
responses.
Line 6: Create a new HTTP::Request—We call HTTP::Request->new(), passing it a request
method of "GET" and the desired URL. This returns a new HTTP::Request object.
Line 7: Make the request—We pass the newly created HTTP::Request to the user agent's
request() method. This issues a request on the remote server, returning an HTTP::Response.
Lines 8–9: Print response—We call the response object's is_success() method to determine
whether the request was successful. If not, we die with the server's error message, returned by
the response object's message() method. Otherwise, we retrieve and print the response con-
tents by calling the response object's content() method.
Short as it is, this script illustrates the major components of the LWP library. HTTP::Request contains
information about the outgoing request from the client to the server. Requests can be simple objects
containing little more than a URL, as shown here, or can be complex objects containing cookies,
authentication information, and arguments to be passed to server scripts.
HTTP::Response encapsulates the information returned from the server to the client. Response
objects contain status information, plus the document contents itself.
LWP::UserAgent intermediates between client and server, transmitting HTTP::Requests to the re-
mote server, and translating the server's response into an HTTP::Response to return to client code.
In addition to its object-oriented mode, LWP offers a simplified procedural interface called
LWP::Simple. Figure 9.2 shows the same script rewritten using this module. After loading the
LWP::Simple module, we fetch the desired URL from the command line and pass it to get-
print(). This function attempts to retrieve the indicated URL. If successful, it prints its content to
standard output. Otherwise, it prints a message describing the error to STDERR.

Figure 9.2. Fetch a URL using LWP::Simple procedural interface

In fact, we could reduce Figure 9.1 even further to this one-line command:
% perl -MLWP::Simple -e 'getprint shift' http://www.yahoo.com/r/wt

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Web Clients 196

The procedural interface is suitable for fetching and mirroring Web documents when you do not
need control over the outgoing request and you do not wish to examine the response in detail. The
object-oriented interface is there when you need to customize the outgoing request by providing
authentication information and data to post to a server script, or by changing other header informa-
tion passed to the server. The object-oriented interface also allows you to interrogate the response
to recover detailed information about the remote server and the returned document.

HTTP::Request
The Web paradigm generalizes all client/server interactions to a client request and a server re-
sponse. The client request consists of a Uniform Resource Locator (URL) and a request method.
The URL, which is known in the LWP documentation by its more general name, URI (for Uniform
Resource Identifier), contains information on the network protocol to use and the server to contact.
Each protocol uses different conventions in its URLs. The protocols supported by LWP include:
HTTP—The Hypertext Transfer Protocol, the "native" Web protocol described in RFCs 1945 and
2616, and the one used by all Web servers. HTTP URLs have this familiar form:
http://server.name:port/path/to/document

The http: at the beginning identifies the protocol. This is followed by the server DNS name, IP ad-
dress, and, optionally, the port the server is listening on. The remainder of the URL is the path to
the document.
FTP—A document stored on an FTP server. FTP URLs have this form:
ftp://server.name:port/path/to/document

GOPHER —A document stored on a server running the now rarely used gopher protocol. Gopher
URLs have this form:
gopher://server.name:port/path/to/document

SMTP—LWP can send mail messages via SMTP servers using mailto: URLs. These have the form:
mailto:[email protected]

where [email protected] is the recipient's e-mail address. Notice that the location of the SMTP server
isn't part of the URL. LWP uses local configuration information to identify the server.
NNTP—LWP can retrieve a news posting from an NNTP server given the ID of the message you
wish to retrieve. The URL format is:
news:message-id

As in mail: URLs, there is no way to specify the particular NNTP server. A suitable server is identified
automatically using Net::NNTP's rules (see Chapter 8).
In addition to the URL, each request has a method. The request method indicates the type of trans-
action that is requested. A number of methods are defined, but the most frequent ones are:
GET—Fetch a copy of the document indicated by the URL. This is the most common way of fetching
a Web page.
PUT—Replace or create the document indicated by the URL with the document contained in the
request. This is most commonly seen in the FTP protocol when uploading a file, but is also used by
some Web page editors.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Web Clients 197

POST—Send some information to the indicated URL. It was designed for posting e-mail messages
and news articles, but was long ago appropriated for use in sending fill-out forms to CGI scripts and
other server-side programs.
DELETE—Delete the document indicated by the URL. This is used to delete files from FTP servers
and by some Web-based editing systems.
HEAD—Return information about the indicated document without changing or downloading it.
HTTP protocol requests can also contain other information. Each request includes a header that
contains a set of RFC 822-like fields. Common fields include Accept:, indicating the MIME type(s)
the client is prepared to receive, User-agent:, containing the name and version of the client software,
and Content-type:, which describes the MIME type of the request content, if any. Other fields handle
user authentication for password-protected URLs.
For the PUT and POST methods, but not for GET, HEAD, and DELETE, the request also contains
content data. For PUT, the content is the document to upload to the location indicated by the URL.
For POST, the content is some data to send, such as the contents of a fill-out form to send to a CGI
script.
The LWP library uses a class named HTTP::Request to represent all requests, even those that do
not use the HTTP protocol. You construct a request by calling HTTP::Request->new() with the
name of the desired request method and the URL you wish to apply the request to. For HTTP
requests, you can then add or alter the outgoing headers to do such things as add authentication
information or HTTP cookies. If the request method expects content data, you'll normally add the
data to the request object using its content () method.
The API description that follows lists the most frequently used HTTP:: Request methods. Some of
them are defined in HTTP::Request directly, and others are inherited.
One begins by creating a new request object with HTTP::Request->new().

$request = HTTP::Request->new($method, $url [,$header [,$content]])

The new() method constructs a new HTTP::Request. It takes a minimum of two arguments. $method is the
name of the request method, such as GET, and $url is the URL to act on. The URL can be a simple string
or a reference to a URI object created using the URI module. We will not discuss the URI module in detail here,
but it provides functionality for dissecting out the various parts of URLs.

new() also accepts optional header and content arguments. $header should be a reference to an
HTTP::Headers object. However, we will not go over the HTTP::Headers API because it's easier to
allow HTTP::Request to create a default headers object and then customize it after the object is
created. $content is a string containing whatever content you wish to send to the server.
Once the request object is created, the header() method can be used to examine or change
header fields.

$request->header($field1 => $val1, $field2 => $val2 ...)

@values = $request->header($field)
Call header() with one or more field/value pairs to set the indicated fields, or with a single field name to
retrieve the current values. When called with a field name, header() returns the current value of the field. In
a list context, header() returns multivalued fields as a list; in a scalar context, it returns the values separated
by commas.

This example sets the Referer: field, which indicates the URL of the document that referred to the
one currently being requested:

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Web Clients 198

$request->header(Referer => 'http://www.yahoo.com/whats_cool.html')

An HTTP header field can be multivalued. For example, a client may have a Cookie: field for each
cookie assigned to it by the server. You can set multivalued field values by using an array reference
as the value, or by passing a string in which values are separated by commas. This example sets
the Accept: field, which is a multivalued list of the MIME types that the client is willing to accept:
$request->header(Accept => ['text/html','text/plain','text/rtf'])

Alternatively, you can use the push_header() method described later to set multivalued fields.

$request->push_header($field => $value)

The push_header() method appends the indicated value to the end of the field, creating it if it does not
already exist, and making it multivalued otherwise. $value can be a scalar or an array reference.
$request->remove_header(@fields)
The remove_header() method deletes the indicated fields.

A variety of methods provide shortcuts for dealing with header fields.

$request->scan(\&sub)
The scan() method iterates over each of the HTTP headers in turn, invoking the code reference provided in
\&sub. The subroutine you provide will be called with two arguments consisting of the field name and its value.
For multivalued fields, the subroutine is invoked once for each value.
$request->date()
$request->expires()
$request->last_modified()
$request->if_modified_since()
$request->content_type()
$request->content_length()
$request->referer()
$request->user_agent()
These methods belong to a family of 19 convenience methods that allow you to get and set a number of
common unique-valued fields. Called without an argument, they return the current value of the field. Called
with a single argument, they set it. The methods that deal with dates use system time format, as returned by
time().

Three methods allow you to set and examine one request's content.

$request->content([$content])
$request->content_ref
The content() method sets the content of the outgoing request. If no argument is provided, it returns the
current content value, if any. content_ref() returns a reference to the content, and can be used to manip-
ulate the content directly.
When POSTing a fill-out form query to a dynamic Web page, you use content() to set the query string, and
call content_type() to set the MIME type to either application/x-www-form-urlencoded or multipart/form-
data.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Web Clients 199

It is also possible to generate content dynamically by passing content() a reference to a piece of code that
returns the content. LWP invokes the subroutine repeatedly until it returns an empty string. This facility is useful
for PUT requests to FTP servers, and POST requests to mail and news servers. However, it's inconvenient to
use with HTTP servers because the Content-Length: field must be filled out before sending the request. If you
know the length of the dynamically generated content in advance, you can set it using the con-
tent_length() method.
$request->add_content($data)
This method appends some data to the end of the existing content, if any. It is useful when reading content
from a file.

Finally, several methods allow you to change the URL and method.

$request->uri([$uri])
This method gets or sets the outgoing request's URI.
$request->method([$method])
This method() gets or sets the outgoing request's method.
$string = $request->as_string
The as_string() method returns the outgoing request as a string, often used during debugging.

HTTP::Response
Once a request is issued, LWP returns the server's response in the form of an HTTP::Response
object. HTTP::Response is used even for non-HTTP protocols, such as FTP.
HTTP::Response objects contain status information that reports the outcome of the request, and
header information that provides meta-information about the transaction and the requested docu-
ment. For GET and POST requests, the HTTP::Response usually contains content data.
The status information is available both as a numeric status code and as a short human-readable
message. When using the HTTP protocol, there are more than a dozen status codes, the most
common of which are listed in Table 9.1. Although the text of the messages varies slightly from
server to server, the codes are standardized and fall into three general categories:
• Informational codes, in the range 100 through 199, are informational status codes issued before
the request is complete.
• Success codes, which occupy the 200 through 299 range, indicate successful outcomes.
• Redirection status codes, in the 300 through 399 range, indicate that the requested URL has
moved elsewhere. These are commonly encountered when a Web site has been reorganized
and the administrators have installed redirects to avoid breaking incoming external links.
• Errors in the 400 through 499 range indicate various client-side errors, and those 500 and up
are server-side errors.
When dealing with non-HTTP servers, LWP synthesizes appropriate status codes. For example,
when requesting a file from an FTP server, LWP generates a 200 ("OK") response if the file was
downloaded, and 404 ("Not Found") if the requested file does not exist.
The LWP library handles some status codes automatically. For example, if a Web server returns a
redirection response indicating that the requested URL can be found at a different location (codes
301 or 302), LWP automatically generates a new request directed at the indicated location. The
response that you receive corresponds to the new request, not the original. If the response requests
authorization (status code 401), and authorization information is available, LWP reissues the re-
quest with the appropriate authorization headers.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Web Clients 200

HTTP::Response headers describe the server, the transaction, and the enclosed content. The most
useful headers include Content-type: and Content-length:, which provide the MIME type and length
of the returned document, if any, Last-modified:, which indicates when the document was last modi-
fied, and Date:, which tells you the server's idea of the time (since client and server clocks are not
necessarily synchronized).
Table 9.1. Common HTTP Status Codes and Messages
Code Message Description
1XX codes: informational
100 Continue Continue with request.
101 Switching Protocols It is upgrading to newer version of HTTP.
2XX codes: success
200 OK The URL was found. Its contents follows.
201 Created A URL was created in response to a POST.

Licensed by
202 Accepted The request was accepted for processing at a later date.
204 No Response The request is successful, but there's no content.
3XX codes: redirection
301 Moved The URL has permanently moved to a new location.

Stjepan Maric
302 Found The URL can be temporarily found at a new location.
4XX codes: client errors
400 Bad Request There's a syntax error in the request.
401 Authorization Required Password authorization is required.

4218908
403 Forbidden This URL is forbidden, and authorization won't help.
404 Not Found It isn't here.
5XX codes: server errors
500 Internal Error The server encountered an unexpected error.
501 Not Implemented Used for unimplemented features.
502 Overloaded The server is temporarily overloaded.

Like the request object, HTTP::Response inherits from HTTP::Message, and delegates unknown
method calls to the HTTP::Headers object contained within it. To access header fields, you can call
header(), content_type(), expires(), and all the other header-manipulation methods de-
scribed earlier.
Similarly, the response content can be accessed using the content() and content_ref()
methods. Because some documents can be quite large, LWP also provides methods for saving the
content directly to disk files and spooling them to subroutines in pieces.
Although HTTP::Response has a constructor, you will not usually construct it yourself, so it isn't
listed here. For brevity, a number of other infrequently used methods are also omitted. See the
HTTP::Response documentation for full API.

$status_code = $response->code
$status_message = $response->message
The code() and message() methods return information about the outcome of the request. code() returns
a numeric status code, and message() returns its human-readable equivalent. You can also provide these
methods with an argument in order to set the corresponding field.
$text = $response->status_line
The status_line() method returns the status code followed by the message in the same format returned
by the Web server.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Web Clients 201

$boolean = $response->is_success
$boolean = $response->is_redirect
$boolean = $response->is_info
$boolean = $response->is_error
These four methods return true if the response was successful, is a redirection, is informational, or is an error,
respectively.
$html = $response->error_as_HTML
If is_error() returns true, you can call error_as_HTML() to return a nicely formatted HTML document
describing the error.
$base = $response->base
The base() method returns the base URL for the response. This is the URL to use to resolve relative links
contained in the returned document. The value returned by base() is actually a URI object, and can be used
to "absolutize" relative URLs. See the URI module documentation for details.
$request = $response->request
The request() method returns a copy of the HTTP::Request object that generated this response. This may
not be the same HTTP::Request that you constructed. If the server generated a redirect or authentication
request, then the request returned by this method is the object generated internally by LWP.
$request = $response->previous
previous() returns a copy of the HTTP::Request object that preceded the current object. This can be used
to follow a chain of redirect requests back to the original request. If there is no previous request, this method
returns undef.

Figure 9.3 shows a simple script named follow_chain.pl that uses the previous() method
to show all the intermediate redirects between the requested URL and the retrieved URL. It begins
just like the get_url.pl script of Figure 9.1, but uses the HEAD method to retrieve information about
the URL without fetching its content. After retrieving the HTTP::Response, we call previous()
repeatedly to retrieve all intermediate responses. Each response's URL and status line is prepended
to a growing list of URLs, forming a response chain. At the end, we format the response chain a bit
and print it out.

Figure 9.3. The follow_chain.pl script tracks redirects

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Web Clients 202

Here is the result of fetching a URL that has been moved around a bit in successive reorganizations
of my laboratory's Web site:
% follow_chain.pl http://stein.cshl.org/software/WWW
Response chain:
http://stein.cshl.org/software/WWW (302 Found)
-> http://stein.cshl.org/software/WWW/ (301 Moved Permanently)
-> http://stein.cshl.org/WWW/software/ (200 OK)

LWP::UserAgent
The LWP::UserAgent class is responsible for submitting HTTP::Request objects to remote servers,
and encapsulating the response in a suitable HTTP::Response. It is, in effect, a Web browser en-
gine.
In addition to retrieving remote documents, LWP::UserAgent knows how to mirror them so that the
remote document is transferred only if the local copy is not as recent. It handles Web pages that
require password authentication, stores and returns HTTP cookies, and knows how to negotiate
HTTP proxy servers and redirect responses.
Unlike HTTP::Response and HTTP::Request, LWP::UserAgent is frequently subclassed to cus-
tomize the way that it interacts with the remote server. We will see examples of this in a later section.

$agent = LWP::UserAgent->new
The new() method constructs a new LWP::UserAgent object. It takes no arguments. You can reuse one user
agent multiple times to fetch URLs.
$response = $agent->request ($request, [$dest [,$size]])
The request() method issues the provided HTTP::Request, returning an HTTP:: Response. A response is
returned even on failed requests. You should call the response's is_success() or code() methods to de-
termine the exact outcome.
The optional $dest argument controls where the response content goes. If it is omitted, the content is placed
in the response object, where it can be recovered with the content() and content_ref() methods.

If $dest is a scalar, it is treated as a filename. The file is opened for writing, and the retrieved
document is stored to it. Because LWP prepends a > symbol to the filename, you cannot use com-
mand pipes or other tricks. Because the content is stored to the file, the response object indicates
successful completion of the task, but content(), returns undef.
$dest can also be a reference to a callback subroutine. In this case, the content data is passed to
the indicated subroutine at regular intervals, giving you a chance to do something with the data, like
pass it to an HTML parser. The callback subroutine should look something like this:
sub handle_content {
my ($data,$response,$protocol) = @_;
...
}

The three arguments passed to the callback are the current chunk of content data, the current
HTTP::Response object, and an LWP::Protocol object. The response object is provided so that the
subroutine can make intelligent decisions about how to process the content, such as piping data of
type image/jpeg to an image viewer. The LWP::Protocol object implements protocol-specific access
methods that are used by LWP internally. It is unlikely that you will need it.
If you use a code reference for $dest, you can exercise some control over the content chunk size
by providing a $size argument. For example, if you pass 512 for $size, the callback will be called
repeatedly with 512-byte chunks of the content data.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Web Clients 203

Two variants of request() are useful in certain situations.

$response = $agent->simple_request($request, [$dest [,$size]])

simple_request() behaves like request(), but does not automatically reissue requests to handle redi-
rects or authentication requirements. Its arguments are identical to those of request().
$response = $agent->mirror($url,$file)
The mirror() method accepts a URL (a URI object or a string) and the path to a file in which to store the
remote document. If the local file doesn't already exist, then mirror() fetches the remote document. Other-
wise, mirror() compares the modification dates of the remote and local copies, and only fetches the docu-
ment if the local copy appears to be out of date. For HTTP URLs, mirror() constructs an HTTP::Request
object that has the correct If-Modified-Since: header field to perform a conditional fetch. For FTP URLs, LWP
uses the MDTM (modification time) command to fetch the modification date of the remote file.

Two methods allow you to set time and space limits on requests.

$timeout = $agent->timeout([$timeout])
timeout() gets or sets the timeout on requests, in seconds. The default is 180 seconds (3 minutes). If the
timeout expires before the request completes, the returned response has a status code of 500, and a message
indicating that the request timed out.
$bytes = $agent->max_size([$bytes])
The max_size() method gets or sets a maximum size on the response content returned by the remote server.
If the content exceeds this size, then the content is truncated and the response object contains an X-Content-
Range: header indicating the portion of the document returned. Typically, this header has the format bytes
start-end, where start and end are the start and endpoints of the document portion.
By default, the size is undef, meaning that the user agent will accept content of any length.

The agent() and form() methods add information to the request.

$id = $agent->agent([$id])
The agent() method gets or sets the User-Agent: field that LWP will send to HTTP servers. It has the form
name/x.xx (comment), where name is the client software name, x.xx is the version number, and (comment) is
an optional comment field. By default, LWP uses libwww-perl/x.xx, where x.xx is the current module version
number.
You may need to change the agent ID to trigger browser-specific behavior in the remote server. For example,
this line of code changes the agent ID to Mozilla/4.7, tricking the server into thinking it is dealing with a Netscape
version 4.X series browser running on a Palm Pilot:
$agent->agent('Mozilla/4.7 [en] (PalmOS)')

$address = $agent->from([$address])
The from() method gets or sets the e-mail address of the user responsible for the actions of the user agent.
It is incorporated into the From: field used in mail and news postings, and will be issued, along with other fields,
to HTTP servers. You do not need to provide this information when communicating with HTTP servers, but it
can be provided in Web crawling robots as a courtesy to the remote site.

A number of methods control how the agent interacts with proxies, which are commonly used when
the client is behind a firewall that doesn't allow direct Internet access, or in situations where band-
width is limited and the organization wishes to cache frequently used URLs locally.

$proxy = $agent->proxy($protocol => $proxyy)

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Web Clients 204

The proxy() method sets or gets the proxy servers used for requests. The first argument, $protocol, is
either a scalar containing the name of a protocol to proxy, such as "ftp", or an array reference that lists several
protocols to proxy, such as ['ftp','http','gopher']. The second argument, $proxy, is the URL of the
proxy server to use. For example:
$agent->proxy([qw(ftp http)] => 'http://proxy.cshl.org:8080')

You may call this method several times if you need to use a different proxy server for each protocol:
$agent->proxy(ftp => 'http://proxy1.cshl.org:8080');
$agent->proxy(http => 'http://proxy2.cshl.org:9000');

As this example shows, HTTP servers are commonly used to proxy FTP requests as well as HTTP requests.
$agent->no_proxy(@domain_list)
Call the no_proxy() method to deactivate proxying for one or more domains. You would typically use this to
turn off proxying for intranet servers that you can reach directly. This code fragment disables proxying for the
"localhost" server and all machines in the "cshl.org" domain:
$agent->no_proxy('localhost','cshl.org')

Calling no_proxy() with an empty argument list clears the list of proxyless domains. It cannot be used to
return the current list.
$agent->env_proxy
env_proxy() is an alternative way to set up proxies. Instead of taking proxy information from its argument
list, this method reads proxy settings from *_proxy environment variables. These are the same environment
variables used by UNIX and Windows versions of Netscape. For example, a C-shell initialization script might
set the FTP and HTTP proxies this way:
setenv ftp_proxy http://proxy1.cshl.org:8080
setenv http_proxy http://proxy2.cshl.org:9000
setenv no_proxy localhost,cshl.org

Lastly, the agent object offers several methods for controlling authentication and cookies.

($name,$pass) = $agent->get_basic_credentials($realm,$url [,$proxy])

When a remote HTTP server requires password authentication to access a URL, the user agent invokes its
get_basic_credentials() method to return the appropriate username and password. The arguments
consist of the authentication "realm name", the URL of the request, and an optional flag indicating that the
authentication was requested by an intermediate proxy server rather than the destination Web server. The
realm name is a string that the server sends to identify a group of documents that can be accessed using the
same username/password pair.
By default, get_basic_credentials() returns the username and password stored among the user agent's
instance variables by the credentials() method. However, it is often more convenient to subclass
LWP::UserAgent and override get_basic_credentials() in order to prompt the user to enter the required
information. We'll see an example of this later.
$agent->credentials($hostport,$realm,$name,$pass)
The credentials() method stores a username and password for use by get_basic_credentials().
The arguments are the server hostname and port in the format hostname:port, authentication realm, username,
and password.
$jar = $agent->cookie_jar([$cookie_jar])
By default, LWP::UserAgent ignores cookies that are sent to it by remote Web servers. You can make the
agent fully cookie-compatible by giving it an object of type HTTP::Cookies. The module will then stash incoming
cookies into this object, and later search it for stored cookies to return to the remote server. Called with an
HTTP::Cookies argument, cookie_jar() uses the indicated object to store its cookies. Called without ar-
guments, cookie_jar() returns the current cookie jar.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Web Clients 205

We won't go through the complete HTTP::Cookies API, which allows you to examine and manipulate
cookies, but here is the idiom to use if you wish to accept cookies for the current session, but not
save them between sessions:
$agent->cookie_jar(new HTTP::Cookies);

Here is the idiom to use if you wish to save cookies automatically in a file named .lwp-cookies for
use across multiple sessions:
my $file = "$ENV{HOME}/.lwp-cookies";
$agent->cookie_jar(HTTP::Cookies->new(file=>$file,autosave=>1));

Finally, here is how to tell LWP to use an existing Netscape-format cookies file, assuming that it is
stored in your home directory in the file ~/.netscape/cookies (Windows and Mac users must modify
this accordingly):
my $file = "$ENV{HOME}/.netscape/cookies";
$agent->cookie_jar(HTTP::Cookies::Netscape->new(file=>$file,
autosave=>1));

LWP Examples
Now that we've seen the LWP API, we'll look at some practical examples that use it.

Fetching a List of RFCs

The Internet FAQ Consortium (http://www.faqs.org) maintains a Web server that archives a large
number of useful Internet documents, including Usenet FAQs and IETF RFCs. Our first example is
a small command-line tool to fetch a list of RFCs by their numbers.
The RFC archive at http://www.faqs.org follows a predictable pattern. To view RFC 1028, for ex-
ample, we would fetch the URL http://www.faqs.org/rfcs/rfc1028.html. The returned HTML docu-
ment is a minimally marked-up version of the original text-only RFC. The FAQ Consortium adds an
image and a few links to the top and bottom. In addition, every reference to another RFC becomes
a link.
Figure 9.4 shows the get_rfc.pl script. It accepts one or more RFC numbers on the command
line, and prints their contents to standard output. For example, to fetch RFCs 1945 and 2616, which
describe HTTP versions 1.0 and 1.1, respectively, invoke get_rfc.pl like this:

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Web Clients 206

Figure 9.4. The get_rfc.pl script

% get_rfc.pl 1945 2616

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML//EN">
<HTML>
<HEAD>
<TITLE>rfc1945 - Hypertext Transfer Protocol -- HTTP/1.0</TITLE>
<LINK REV="made" HREF="mailto:[email protected]";>
<META name="description" content="Hypertext Transfer Protocol -- HTTP/1.0">
<META name="authors" content="T. Berners-Lee, R. Fielding & H. Frystyk">
...

The retrieved files can be saved to disk or viewed in a browser.

Lines 1–4: Load modules—We turn on strict syntax checking and load the LWP module. In
addition, we define a constant URL prefix to use for fetching the desired RFC.
Line 5: Process command-line arguments—We check that at least one RFC number is given
on the command line, or die with a usage message.
Lines 6–8: Create user agent—We create a new LWP::UserAgent and change its default User-
Agent: field to get_rfc/1.0. We follow this with the original default agent ID enclosed in paren-
theses.
Lines 9–18: Main loop—For each RFC listed on the command line, we construct the appropriate
URL and use it to create a new HTTP::Request GET request. We pass the request to the user
agent object's request() method and examine the response. If the response's is_suc-
cess() method indicates success, we print the retrieved content. Otherwise, we issue a warning
using the response's status message.

Mirroring a List of RFCs

The next example represents a slight modification. Instead of fetching the requested RFCs and
sending them to standard output, we'll mirror local copies of them as files stored in the current
working directory. LWP will perform the fetch conditionally so that the remote document will be
fetched only if it is more recent than the local copy. In either case, the script reports the outcome of
each attempt, as shown in this example:

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Web Clients 207

% mirror_rfc.pl 2616 1945 11

RFC 2616: OK
RFC 1945: Not Modified
RFC 11: Not Found

We ask the script to retrieve RFCs 2616, 1945 and 11. The status reports indicate that RFC 2616
was retrieved OK, RFC 1945 did not need to be retrieved because the local copy is current, and
that RFC 11 could not be retrieved because no such file exists on the remote server (there is, in
fact, no RFC 11).
The code, shown in Figure 9.5, is only 15 lines long.

Figure 9.5. The mirror_rfc.pl script

Lines 1–8: Load modules and create user agent—The setup of the LWP::UserAgent is identical
to the previous example, except that we modify the usage message and the user agent ID
appropriately.
Lines 9–15: Main loop—We read RFC numbers from the command line. For each RFC, we
construct a local filename of the form rfcXXXX.html, where XXXX is the number of the requested
document. We append this to the RFC server's base URL in order to obtain the full remote URL.
In contrast with the previous example, we don't need to create an HTTP::Request in order to do
mirroring. We simply pass the remote URL and local filename to the agent's mirror() method,
obtaining an HTTP::Response in return. We then print the status message returned by the re-
sponse object's message() method.

Simulating Fill-out Forms

The previous two examples fetched static documents from remote Web servers. However, much of
the interesting content on the Web is generated by dynamic server-side scripts such as search
pages, on-line catalogs, and news updates.
Server-side CGI scripts (as well as servlets and other types of dynamic content) are usually driven
by fill-out HTML forms. Forms consist of a series of fields to complete: typically a mixture of text
fields, pop-up menus, scrolling lists, and buttons. Each field has a name and a value. When the form
is submitted, usually by clicking on a button, the names and current values of the form are bundled
into a special format and sent to the server script.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Web Clients 208

You can simulate the submission of a fill-out form from within LWP provided that you know what
arguments the remote server is expecting and how it is expecting to receive them. Sometimes the
remote Web site documents how to call its server-side scripts, but more often you have to reverse
engineer the script by looking at the fill-out form's source code.
For example, the Internet FAQ Consortium provides a search page at http://www.faqs.org/rfcs/ that
includes, among other things, a form for searching the RFC archive with text search terms. By
navigating to the page in a conventional browser and selecting the "View Source" command, I ob-
tained the HTML source code for the page. Figure 9.6 shows an excerpt from this page, which
contains the definition for the search form (it's been edited slightly to remove extraneous formatting
tags).

Figure 9.6. Definition of the HTML form used by the FAQ Consortium's RFC seach script

In HTML, fill-out forms start with a <FORM> tag and end with </FORM>. Between the two tags are
one or more <INPUT> tags, which create simple fields like text entry fields and buttons,
<SELECT> tags, which define multiple-choice fields like scrolling lists and pop-up menus, and
<TEXTAREA> tags, which create large text entry fields with horizontal and vertical scrollbars.
Form elements have a NAME attribute, which assigns a name to the field when it is sent to the Web
server, and optionally a VALUE attribute, which assigns a default value to the field. <INPUT> tags
may also have a TYPE attribute that alters the appearance of the field. For example, TYPE="text"
creates a text field that the user can type in, TYPE="checkbox" creates an on/off checkbox, and
TYPE="hidden" creates an element that isn't visible in the rendered HTML, but nevertheless has
its name and value passed back to the server when the form is submitted.
The <FORM> tag itself has two required attributes. METHOD specifies how the contents of the fill-out
form are to be sent to the Web server, and may be one of GET and POST. We'll talk about the
implications of the method later. ACTION specifies the URL to which the form fields are to be sent.
It may be a full URL or an abbreviated form relative to the URL of the HTML page that contains the
form.
Occasionally, the ACTION attribute may be missing entirely, in which case the form fields should be
submitted to the URL of the page in which the form is located. Strictly speaking, this is not valid
HTML, but it is widely used.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Web Clients 209

In the example in Figure 9.6, the RFC search form consists of two elements. A text field named
"query" prompts the user for the text terms to search for, and a menu named "archive" specifies
which part of the archive to search in. The various menu choices are specified using a series of
<OPTION> tags, and include the values "rfcs", "rank", and "rfcindex". There is also a submission
button, created using an <INPUT> tag with a TYPE attribute of "submit". However, because it has
no NAME attribute, its contents are not included in the information to the server. Figure 9.7 shows
what this looks like when rendered by a browser.

Figure 9.7. The FAQ Consortium's fill-out form rendered by a browser

When the form is submitted, the browser bundles the current contents of the form into a "query
string" using a MIME format known as application/x-www-form-urlencoded. This format consists of
a series of name= value pairs, where the names and values are taken from the form elements and
their current values. Each pair is separated by an ampersand (&) or semicolon (;). For example, if
we typed "MIME types" into the RFC search form's text field and selected "Search RFC Index" from
the pop-up menu, the query string generated by the browser would be:
query=MIME%20types&archive=rfcindex

Notice that the space in "MIME types" has been turned into the string %20. This is a hexadecimal
escape for the space character (0x20 in ASCII). A number of characters are illegal in query strings,
and must be escaped in this way. As we shall see, the URI::Escape module makes it easy to create
escaped query strings.
The way the browser sends the query string to the Web server depends on whether the form sub-
mission method is GET or POST. In the case of GET, a " ? " followed by the query string is appended
directly to the end of the URL indicated by the <FORM> tag's ACTION attribute. For example:
http://www.faqs.org/cgi-bin/rfcsearch?query=MIME%20types&archive=rfcindex

In the case of a form that specifies the POST method, the correct action is to POST a request to
the URL indicated by ACTION, and pass the query string as the request content.
It is very important to send the query string to the remote server in the way specified by the
<FORM> tag. Some server-side scripts are sufficiently flexible to recognize and deal with both GET
and POST requests in a uniform way, but many do not.
In addition to query strings of type application/x-www-form-urlencoded, some fill-out forms use a
newer encoding system called multipart/form-data. We will talk about dealing with such forms in the
section File Uploads Using multipart/form-data.
Our next sample script is named search_rfc.pl. It invokes the server-side script located at http://
www.faqs.org/cgi-bin/rfcsearch to search the RFC index for documents having some relevance to
the search terms given on the command line. Here's how to search for the term "MIME types":

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Web Clients 210

% search_rfc.pl MIME types

RFC 2503 MIME Types for Use with the ISO ILL Protocol
RFC 1927 Suggested Additional MIME Types for Associating Documents

search_rfc.pl works by simulating a user submission of the fill-out form shown in Figures 9.6
and 9.7. We generate a query string containing the query and archive fields, and POST it to the
server-side search script. We then extract the desired information from the returned HTML docu-
ment and print it out.
To properly escape the query string, we use the uri_escape() function, provided by the LWP
module named URI::Escape. uri_escape() replaces disallowed characters in URLs with their
hexadecimal escapes. Its companion, uri_unescape(), reverses the process.
Figure 9.8 shows the code for the script.

Licensed by
Stjepan Maric
4218908

Figure 9.8. The search_rfc.pl script

Lines 1–4: Load modules—We turn on strict syntax checking and load the LWP and URI::Escape
modules. URI::Escape imports the uri_escape() and uri_unescape() functions automat-
ically.
Lines 5–7: Define constants—We define one constant for the URL of the remote search script,
and another for the page on which the fill-out form is located. The latter is needed to properly fill
out the Referer: field of the request, for reasons that we will explain momentarily.
Lines 8–10: Create user agent—This code is identical to the previous examples, except for the
user agent ID.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Web Clients 211

Lines 11–12: Construct query string—We interpolate the command-line arguments into a string
and use it as the value of the fill-out form's query field. We are interested in searching the ar-
chive's RFC index, so we use "rfcindex" as the value of the archive field. These are incorporated
into a properly formatted query string and escaped using uri_escape().
Lines 13–15: Construct request—We create a new POST request on the remote search script,
and use the returned request object's content() method to set the content to the query string.
We also alter the request object's Referer: header so that it contains the fill-out form's URL. This
is a precaution. For consistency, some server-side scripts check the Referer: field to confirm
that the request came from a fill-out form located on their own server, and refuse to service
requests that do not contain the proper value. Although the Internet FAQ Consortium's search
script does not seem to implement such checks, we set the Referer: field here in case they
decide to do so in the future.
As an aside, the ease with which we are able to defeat the Referer: check illustrates why this
type of check should never be relied on to protect server-side Web scripts from misuse.
Lines 16–17: Submit request—We pass the request to the LWP::UserAgent's request()
method, obtaining a response object. We check the response status with is_success(), and
die if the method indicates a failure of some sort.
Lines 18–21: Fetch and parse content—We retrieve the returned HTML document by calling the
response object's content() method and assign it to a scalar variable. We now need to extract
the RFC name and title from the document's HTML. This is easy to do because the document
has the predictable structure shown in Figures 9.9 (screenshot) and 9.10 (HTML source). Each
matching RFC is an item in an ordered list (HTML tag <OL>) in which the RFC number is con-
tained within an <A> tag that links to the text of the RFC, and the RFC title is contained between
a pair of <STRONG> tags.

Figure 9.9. RFC Index Search results

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Web Clients 212

Figure 9.10. HTML code for the RFC Index Search results

We use a simple global regular expression match to find and match all lines referring to RFCs,
extract the RFC name and title, and print the information to standard output.
An enhancement to this script would be to provide an option to fetch the text of each RFC returned
by the search. One way to do this would be to insert a call to $ua->request() for each matched
RFC. Another, and more elegant, way would be to modify get_rfc.pl from Figure 9.4 so as to
accept its list of RFC numbers from standard input. This would allow you to fetch the content of each
RFC returned by a search by combining the two commands in a pipeline:
% fetch_rfc.pl MIME type | get_rfc.pl

Because The Internet FAQ Consortium has not published the interface to its search script, there is
no guarantee that they will not change either the form of the query string or the format of the HTML
document returned in response to searches. If either of these things happen, search_rfc.pl will break.
This is a chronic problem for all such Web client scripts and a compelling reason to check at each
step of a complex script that the remote Web server is returning the results you expect.
This script contains a subtle bug in the way it constructs its query strings. Can you find it? The bug
is revealed in the next section.

Using HTTP::Request::Common to Post a Fill-out Form

Because submitting the field values from fill-out forms is so common, LWP provides a class named
HTTP::Request::Common to make this convenient to do. When you load HTTP::Request::Common,
it imports four functions named GET(), POST(), HEAD(), and PUT(), which build various types of
HTTP::Request objects.
We will look at the POST() function, which builds HTTP::Request objects suitable for simulating fill-
out form submissions. The other three are similar.

$request = POST($url [,$form_ref] [,$header1=>$val1....])

The POST() function returns an HTTP::Request object that uses the POST method. $url is the requested
URL, and may be a simple string or a URI object. The optional $form_ref argument is an array reference
containing the names and values of form fields to submit as content. If you wish to add additional headers to
the request, you can follow this with a list of header/value pairs.

Using POST() here's how we could construct a request to the Internet FAQ Consortium's RFC index
search engine:
my $request = POST('http://www.faqs.org/cgi-bin/rfcsearch',
[ query => 'MIME types',
archive => 'rfcindex' ]
);

And here's how to do the same thing but setting the Referer: header at the same time:

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Web Clients 213

my $request =POST('http://www.faqs.org/cgi-bin/rfcsearch',
[ query => 'MIME types',
archive => 'rfcindex' ],
Referer => 'http://www.faqs.org/rfcs');

Notice that the field/value pairs of the request content are contained in an array reference, but the
name/value pairs of the request headers are a simple list.
As an alternative, you may provide the form data as the argument to a pseudoheader field named
Content:. This looks a bit cleaner when setting both request headers and form content:
my $request = POST('http://www.faqs.org/cgi-bin/rfcsearch',
Content => [ query => 'MIME types',
archive => 'rfcindex' ],
Referer => 'http://www.faqs.org/rfcs');

POST() will take care of URI escaping the form fields and constructing the appropriate query string.
Using HTTP::Request::Common, we can rewrite search_rfc.pl as shown in Figure 9.11. The new
version is identical to the old except that it uses POST() to construct the fill-out form submission
and to set the Referer: field of the outgoing request (lines 12–17). Compared to the original version
of the search_rfc.pl script, the new script is easier to read. More significant, however, it is less prone
to bugs. The query-string generator from the earlier versions contains a bug that causes it to gen-
erate broken query strings when given a search term that contains either of the characters "&" or
"=". For example, given the query string "mime&types", the original version generates the string:

Figure 9.11. An improved version of search_rfc.pl

query=mime&types&archive=rfcindex

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Web Clients 214

The manual fix would be to replace " & " with " %26 " and " = " with " %3D " in the search terms
before constructing the query string and passing it to uri_escape(). However, the POST()-
based version handles this automatically, and generates the correct content:
query=mime%26types&archive=rfcindex

File Uploads Using multipart/form-data

In addition to form elements that allow users to type in text data, HTML version 4 and higher provides
an <INPUT> element of type "file". When compatible browsers render this tag, they generate a user
interface element that prompts the user for a file to upload. When the form is submitted, the browser
opens the file and sends it contents, allowing whole files to be uploaded to a server-side Web script.
However, this feature is not very compatible with the application/x-www-form-urlencoded encoding
of query strings because of the size and complexity of most uploaded files. Server scripts that sup-
port this feature use a different type of query encoding scheme called multipart/form-data. Forms
that support this encoding are enclosed in a <FORM> tag with an ENCTYPE attribute that specifies
this scheme. For instance:
<FORM METHOD=POST ACTION="/cgi-bin/upload" ENCTYPE="multipart/form-data">

The POST method is always used with this type of encoding. multipart/form-data uses an encoding
scheme that is extremely similar to the one used for multipart MIME enclosures. Each form element
is given its own subpart with a Content-Disposition: of "form-data", a name containing the field name,
and body data containing the value of the field. For uploaded files, the body data is the content of
the file.
Although conceptually simple, it's tricky to generate the multipart/form-data format correctly. Fortu-
nately, the POST() function provided by HTTP::Request:: Common can also generate requests
compatible with multipart/form-data. The key is to provide POST() with a Content_Type: header
argument of "form-data":
my $request = POST('http://www.faqs.org/cgi-bin/rfcsearch',
Content_Type => 'form-data',
Referer => 'http://www.faqs.org/rfcs',
Content => [ query => 'MIME types',
archive => 'rfcindex' ]
);

This generates a request to the RFC search engine using the multipart/form-data encoding scheme.
But don't try it: the RFC FAQ site doesn't know how to handle this scheme.
To tell LWP to upload a file, the value of the corresponding form field must be an array reference
containing at least one element:
$fieldname => [ $file, $filename, header1=>$value.... ]

The mandatory first element in the array, $file, is the path to the file to upload. The optional
$filename argument is the suggested name to use for the file, and is similar to the MIME::Entity
Filename argument. This is followed by any number of additional MIME headers. The one used
most frequently is Content_Type:, which gives the server script the MIME type of the uploaded file.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Web Clients 215

To illustrate how this works, we'll write a client for the CGI script located at http://stein.cshl. org/
WWW/software/CGI/examples/file_upload.cgi [http://stein.cshl.org/WWW/software/CGI/examples/
file_upload.cgi]. This is a script that I wrote some years ago to illustrate how CGI scripts accept and
process uploaded files. The form that drives the script (Figures 9.12 and 9.14) contains a single file
field named filename, and three checkboxes named count with values named "count lines",
"count words", and "count characters". There's also a hidden field named .cgifields with
a value of "count."

Figure 9.12. The form that drives the file_upload.cqi script

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Web Clients 216

Figure 9.14. Output from file_upload.cqi script

After form submission, the script reads the uploaded file and counts its lines, words, and/or char-
acters, depending on which checkboxes are selected. It prints these statistics, along with the name
of the file and its MIME type, if any (Figure 9.13).

Figure 9.13. HTML source for the file_upload.cqi form

We will now develop an LWP script to drive this CGI script. remote_wc.pl reads a file from the
command line or standard input and uploads it to file_upload.cgi. It parses the HTML result and
prints the word count returned by the remote server:
% remote_wc.pl ~/public_html/png.html
lines = 20; words = 47; characters = 362

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Web Clients 217

This is a pretty difficult way to perform a word count, but it does illustrate the technique! Figure
9.15 gives the code for remote_wc.pl.

Figure 9.15. The remote_wc.pl script

Lines 1–4: Load modules—We turn on strict syntax checking and load the LWP and HTTP::Re-
quest::Common modules.
Lines 5–7: Process arguments—We define a constant for the URL of the CGI script and recover
the name of the file to upload from the command line.
Lines 8–21: Create user agent and request—We create the LWP::UserAgent in the usual way.
We then create the request using the POST() function, passing the URL of the CGI script as
the first argument, a Content_Type argument of "form-data", and a Content argument containing
the various fields used by the upload form.
Notice that the count field appears three times in the Content array, once for each of the check-
boxes in the form. The value of the filename field is an anonymous array containing the file path
provided on the command line. We also provide values for the .cgifields hidden field and the
submit button, even though it isn't clear that they are necessary (they aren't, but unless you have
the documentation for the remote server script, you won't know this).

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Web Clients 218

Lines 22–23: Issue request—We call the user agent's request() method to issue the POST,
and get a response object in return. As in earlier scripts, we check the is_success() method
and die if an error occurs.
Lines 24–27: Extract results—We call the response's content() method to retrieve the HTML
document generated by the remote script, and perform a pattern match on it to extract the values
for the line, word, and character counts (this regular expression was generated after some ex-
perimentation with sample HTML output). Before exiting, we print the extracted values to stand-
ard output.

Fetching a Password-Protected Page

Some Web pages are protected by username and password using HTTP authentication. LWP can
handle the authentication protocol, but needs to know the username and password.
There are two ways to provide LWP with this information. One way is to store the username and
password in the user agent's instance variables using its credentials() method. As described
earlier, credentials() stores the authentication information in a hash table indexed by the Web
server's hostname, port, and realm. If you store a set of passwords before making the first request,
LWP::UserAgent consults this table to find a username and password to use when accessing a
protected page. This is the default behavior of the get_basic_credentials() method.
The other way is to ask the user for help at runtime. You do this by subclassing LWP::UserAgent
and overriding the get_basic_credentials() method. When invoked, the customized
get_basic_credentials() prompts the user to enter the required information.
The get_url2.pl script implements this latter scheme. For unprotected pages, it acts just like the
original get_url.pl script (Figure 9.1). However, when fetching a protected page, it prompts the user
to enter his or her username and password. If the name and password are accepted, the URL is
copied to standard output. Otherwise, the request fails with an "Authorization Required" error (status
code 401):
% get_url2.pl http://stein.cshl.org/private/
Enter username and password for realm "example".
username: perl
password: programmer
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML//EN">
<html> <head>
<title>Password Protected Page</title>
<link rel="stylesheet" href="/stylesheets/default.css">
</head>
...

If you wish to try this script with the URL given in the example, the username is "perl" and the
password is "programmer."
Figure 9.16 shows the code for get_url2.pl. Except for an odd little idiom, it's straightforward. We
are going to declare a subclass of LWP::UserAgent, but we don't want to create a whole module
file just to override a single method. Instead, we arrange for the script itself (package "main") to be
a subclass of LWP::UserAgent, and override the get_basic_credentials() method directly in
the main script file. This is a common, and handy, trick.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Web Clients 219

Figure 9.16. The get_url2.pl script

Lines 1–6: Load modules—We turn on strict syntax checking and load LWP. We also load the
PromptUtil module (listed in Appendix A), which provides us with the get_passwd() function
for prompting the user for a password without echoing it to the screen.
We set the @ISA array to make sure that the current package is a subclass of LWP::UserAgent.
Lines 7–12: Issue request, print content—The main section of the script is identical to the original
get_url.pl, with one exception. Instead of calling LWP::User Agent->new() to create a new
user agent object, we call _PACKAGE->new(). The Perl interpreter automatically replaces the
_PACKAGE_ token with the name of the current package ("main" in this case), creating the de-
sired LWP::UserAgent subclass.
Lines 13–20: Override get_basic_credentials() method—This section of the code overrides
get_basic_credentials() with a custom subroutine. The subclass behaves exactly like
LWP::UserAgent until it needs to fetch authentication information, at which point this subroutine
is invoked.
We are called with three arguments, consisting of the user agent object, the authentication realm,
and the URL that has been requested. We prompt the user for a username, and then call
get_passwd() to prompt and fetch the user's password. These are returned to the caller as a
two-element list.
An interesting characteristic of this script is that if the username and password aren't entered cor-
rectly the first time, LWP invokes the get_basic_credentials() once more and the user is
prompted to try again. If the credentials still aren't accepted, the request fails with an "Authorization
Required" status. This nice "second try" feature appears to be built into LWP.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Web Clients 220

Parsing HTML and XML

Much of the information on the Web is now stored in the form of HTML documents. So far we have
dealt with HTML documents in an ad hoc manner by writing regular expressions to parse out the
particular information we want from a Web page. However, LWP offers a more general solution to
this. The HTML::Parser class provides flexible parsing of HTML documents, and HTML::Formatter
can format HTML as text or PostScript.
An added benefit of HTML::Parser is that at the throw of a switch it can handle XML (eXtensible
Markup Language) as well. Because HTML was designed to display human-readable documents,
it doesn't lend itself easily to automated machine processing. XML provides structured, easily parsed
documents that are more software-friendly than traditional HTML. Over the next few years, HTML
will gradually be replaced by XHTML, a version of HTML that follows XML's more exacting stand-
ards. HTML::Parser can handle HTML, XML, and XHTML, and in fact can be used to parse much
of the more general SGML (Standard Generalized Markup Language) from which both HTML and
XML derive. The XML standard and a variety of tutorials can be found at [http://www.w3.org/XML/].

Licensed by
In this section, we demonstrate how to use HTML::Formatter to transform HTML into nicely format-
ted plain text or postscript. Then we show some examples of using HTML::Parser for the more
general task of extracting information from HTML files.

Stjepan Maric
Formatting HTML
The HTML::Formatter module is the base class for a family of HTML formatters. Only two members
of the family are currently implemented. HTML::FormatText takes an HTML document and produces
nicely formatted plain text, and HTML::FormatPS creates postscript output. Neither subclass of
HTML::Formatter handles inline images, forms, or tables. In some cases, this can be a big limitation.

4218908
There are two steps to formatting an HTML file. The first step is to parse the HTML into a parse tree,
using a specialized subclass of HTML::Parser named HTML::TreeBuilder. The second step is to
pass this parse tree to the desired subclass of HTML::Formatter to output the formatted text.
Figure 9.17 shows a script named format_html.pl that uses these modules to read an HTML file from
the command line or standard input and format it. If given the —postscript option, the script produces
postscript output suitable for printing. Otherwise, it produces plain text.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Web Clients 221

Figure 9.17. The format_html.pl script

Lines 1–4: Load modules—We turn on strict syntax checking and load the Getopt::Long and
HTML:TreeBuilder modules. The former processes the command-line arguments, if any. We
don't load any HTML::Formatter modules at this time because we don't know yet whether to
produce plain text or postscript.
Lines 5–7: Process command-line options—We call the GetOptions() function to parse the
command-line options. This sets the global variable $PS to true if the —postscript option is
specified.
Lines 8–15: Create appropriate formatter—If the user requested postscript output, we load the
HTML::FormatPS module and invoke the class's new() method to create a new formatter object.
Otherwise, we do the same thing with the HTML:: FormatText class. When creating an
HTML::FormatPS formatter, we pass the new() method a PaperSize argument of "Letter" in
order to create output compatible with the common 81/2 × 11" letter stock used in the United
States.
Lines 16–18: Parse HTML—We create a new HTML::TreeBuilder parser by calling the class's
new() method. We then read the input HTML one line at a time using the <> operator and pass
it to the parser object. When we are done, we tell the parser so by calling its eof() method.
This series of operations leaves the HTML parse tree in the parser object itself, in a variable
named $tree.
Line 19–20: Format and output the tree—We pass the parse tree to the formatter's format()
method, yielding a formatted string. We print this, and then clean up the parse tree by calling is
delete() method.

The HTML::Formatter API

The API for HTML::Formatter and its subclasses is extremely simple. You create a new formatter
with new() and perform the formatting with format(). A handful of arguments recognized by
new() adjust the formatting style.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Web Clients 222

$formatter = HTML::FormatText->new([leftmargin=>$left,rightmargin=>$right])
HTML::FormatText->new() takes two optional arguments, leftmargin and rightmargin, which set the left
and right page margins, respectively. The margins are measured in characters. If not specified, the left and
right margins default to 3 and 72, respectively. It returns a formatter object ready for use in converting HTML
to text.
$formatter = HTML::FormatPS->new([option1=>$val1, option2=>$val2...])
Similarly, HTML::FormatPS->new() creates a new formatter object suitable for rendering HTML into post-
script. It accepts a larger list of argument/value pairs, the most common of which are listed here:
• PaperSize sets the page height and width appropriately for printing. Acceptable values are A3, A4, A5, B4,
B5, Letter, Legal, Executive, Tabloid, Statement, Folio, 10x14, and Quarto. United States users take note!
The default PaperSize is the European A4. You should change this to Letter if you wish to print on common
81/2 x 11" paper.
• LeftMargin, RightMargin, TopMargin, and BottomMargin control the page margins. All are given in point
units.
• FontFamily sets the font family to use in the output. Recognized values are Courier, Helvetica, and
Times, the default.
• FontScale allows you to increase or decrease the font size by some factor. For example, a value of 1.5 will
scale the font size up by 50 percent.

Once a formatter is created, you can use it as many times as you like to format HTML::TreeBuilder
objects.

$text = $formatter->format($tree)
Pass an HTML parse tree to the format() method. The returned value is a scalar variable, which you can
then print, save to disk, or send to a print spooler.

The HTML::TreeBuilder API

The basic API for HTML::TreeBuilder is also straightforward. You create a new HTML::TreeBuilder
object by calling the class's new() method, then parse a document using parse() or
parse_file(), and when you're done, destroy the object using delete().

$tree = HTML::TreeBuilder->new
The new() method takes no arguments. It returns a new, empty HTML::TreeBuilder object.
$result = $tree->parse_file($file)
The parse_file() method accepts a filename or filehandle and parses its contents, storing the parse tree
directly in the HTML::TreeBuilder object. If the parse was successful, the result is a copy of the tree object; if
something went wrong (check $! for the error message), the result is undef.

For example, we can parse an HTML file directly like this:

$tree->parse_file('rfc2010.html') or die "Couldn't parse: $!";

and parse from a filehandle like this:

open (F,'rfc2010.html') or die "Couldn't open: $!";
$tree->parse_file(\*F);

$result = $tree->parse($data)

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Web Clients 223

With the parse() method, you can parse an HTML file in chunks of arbitrary size. $data is a scalar that
contains the HTML text to process. Typically you will call parse() multiple times, each time with the next
section of the document to process. We will see later how to take advantage of this feature to begin HTML
parsing while the file is downloading. If something goes wrong during parsing, parse() returns undef. If
parse() is successful, it will return a copy of the HTML::TreeBuilder object, undef otherwise.
$tree->eof
Call this method when using parse(). It tells HTML::TreeBuilder that no more data is coming and allows it to
finish the parse.

Figure 9.16 is a good example of using parse() and eof() to parse the HTML file on standard
input one line at a time.

$tree->delete
When you are finished with an HTML::TreeBuilder tree, call its delete() method to clean up. Unlike other
Perl objects, which are automatically destroyed when they go out of scope, you must be careful to call
delete() explicitly when working with HTML::TreeBuilder objects or risk memory leaks. The HTML::Element
POD documentation explains why this is so.

Many scripts combine HTML::TreeBuilder object creation with file parsing using this idom:
$tree = HTML::TreeBuilder->new->parse_file('rfc2010.html');

However, the HTML::TreeBuilder object created this way will never be deleted, and will leak mem-
ory. If you are parsing files in a loop, always create the HTML::TreeBuilder object, call its
parse_file() method, and then call its delete() method.
The parse tree returned by HTML::TreeBuilder is actually a very feature-rich object. You can re-
cursively descend through its nodes to extract information from the HTML file, extract hypertext
links, modify selected HTML elements, and then convert the whole thing back into printable HTML.
However, the same functionality is also available in a more flexible form in the HTML::Parser class,
which we cover later in this chapter. For details, see the HTML::TreeBuilder and HTML::Element
POD documentation.

Returning Formatted HTML from the get_url.pl Script

We'll now rewrite get_url.pl a third time in order to take advantage of the formatting features offered
by HTML::FormatText. When the new script, imaginatively christened get_url3.pl, detects an HTML
document, it automatically converts it into formatted text.
The interesting feature of this script is that we combine LWP::UserAgent's request callback mech-
anism with the HTML::TreeBuilder parse() method to begin the parse as the HTML document is
downloading. When we parallelize downloading and parsing, the script executes significantly faster.
Figure 9.18 shows the code.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Web Clients 224

Figure 9.18. The get_url3.pl script

Lines 1–6: Load modules—We bring in LWP, PromptUtil, HTML::FormatText, and the
HTML::TreeBuilder modules.
Lines 7–11: Set up request—We set up the HTTP::Request as we did in earlier iterations of this
script. Again, when required, we prompt the user for authentication information so the script is
made a subclass of LWP::UserAgent so that we can override the get_basic_creden-
tials() method.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Web Clients 225

Lines 12–14: Send the request—We send the request using the agent's request(), method.
However, instead of allowing LWP to leave the returned content in the HTTP::Response object
for retrieval, we give request() a second argument containing a reference to the proc-
ess_document() subroutine. This subroutine is responsible for parsing incoming HTML docu-
ments.
process_document() leaves the HTML parse tree, if any, in the global variable
$html_tree, which we declare here. After the request() is finished, we check the status of
the returned HTTP::Response object and die with an explanatory error message if the request
failed for some reason.
Lines 15–20: Format and print the HTML—If the requested document is HTML, then proc-
ess_document() has parsed it and left the tree in $html_tree. We check to see whether the
tree is nonempty. If so, we call its eof() method to tell the parser to finish, and pass the tree
to a newly created HTML::FormatText object to create a formatted string that we immediately
print. We are now done with the parse tree, so we call its delete() method.
As we shall see, process_document() prints all non-HTML documents immediately, so
there's no need to take further action for non-HTML documents.
Lines 21–29: The process_document () subroutine—LWP::UserAgent invokes call-backs
with three arguments consisting of the downloaded data, the current HTTP::Response object,
and an LWP::Protocol object.
We call the response object's content_type() method to get the MIME type of the incoming
document. If the type is text/html, then we pass the data to the parse tree's parse() method.
If necessary, we create the HTML::TreeBuilder first, using the ||= operator so that the call to
HTML::TreeBuilder->new() is executed only if the $html_tree variable is undefined.
If the content type is something other than text/html, then we immediately print the data. This is
a significant improvement to earlier versions of get_url.pl because it means that non-HTML data
starts to appear on standard output as soon as it arrives from the remote server.
Lines 30–38: The get_basic_credentials() subroutine—This is the same subroutine we
looked at in get_url2.pl.
This script does not check for the case in which the response does not provide a content type. Strictly
speaking it should do so, as the HTTP specification allows (but strongly discourages) Web servers
to omit this field. Run the script with the -w switch to detect and report this case. Useful enhance-
ments to get_url3.pl might include using HTML::FormatPS for printing support, or adapting the script
to use external viewers to display non-HTML MIME types the way we did in the pop_fetch.pl script
of Chapter 8.

The HTML::Parser Module

HTML::Parser is a powerful but complex module that allows you to parse HTML and XML docu-
ments. Part of the complexity is inherent in the structure of HTML itself, and part of it is due to the
fact that there are two distinct APIs for HTML::Parser, one used by version 2.2X of the module and
the other used in the current 3.X series.
HTML and XML are organized around a hierarchical series of markup tags. Tags are enclosed by
angle brackets and have a name and a series of attributes. For example, this tag
<img src="/icons/arrow.gif" alt="arrow">

has the name img and the two attributes src and alt.
In HTML, tags can be paired or unpaired. Paired tags enclose some content, which can be plain
text or can contain other tags. For example, this fragment of HTML

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Web Clients 226

<p>Oh dear, now the <strong>bird</strong> is gone!</p>

consists of a paragraph section, starting with the <p> tag and ending with its mate, the </p> tag.
Between the two is a line of text, a portion of which is itself enclosed in a pair of <strong> tags
(indicating strongly emphatic text). HTML and XML both constrain which tags can occur within oth-
ers. For example, a <title> section, which designates some text as the title of a document, can
occur only in the <head> section of an HTML document, which in turn must occur in an <html>
section. See Figure 9.19 for a very minimal HTML document.

Figure 9.19. A skeletal HTML document

In addition to tags, an HTML document may contain comments, which are ignored by rendering
programs. Comments begin with the characters <!-- and end with --> as in:
<!-- ignore this -->

HTML files may also contain markup declarations, contained within the characters <! and >. These
provide meta-information to validators and parsers. The only HTML declaration you are likely to see
is the <!DOCTYPE ...> declaration at the top of the file that indicates the version of HTML the
document is (or claims to be) using. See the top of Figure 9.19 for an example.
Because the "<" and ">" symbols have special significance, all occurrences of these characters in
proper HTML have to be escaped to the "character entities" &lt; and &gt;, respectively. The
ampersand has to be escaped as well, to &. Many other character entities are used to represent
nonstandard symbols such as the copyright sign or the German umlaut.
XML syntax is a stricter and regularized version of HTMLs. Instead of allowing both paired and
unpaired tags, XML requires all tags to be paired. Tag and attribute names are case sensitive
(HTML's are not), and all attribute values must be enclosed by double quotes. If an element is empty,
meaning that there is nothing between the start and end tags, XML allows you to abbreviate this as
an "empty element" tag. This is a start tag that begins with <tagname and ends with />. As an
illustration of this, consider these two XML fragments, both of which have exactly the same meaning:
<img src="/icons/arrow.gif" alt="arrow"></img>
<img src="/icons/arrow.gif" alt="arrow" />

Using HTML::Parser
HTML::Parser is event driven. It parses through an HTML document, starting at the top and tra-
versing the tags and subtags in order until it reaches the end. To use it, you install handlers for
events that you are interested in processing, such as encountering a start tag. Your handler will be
called each time the desired event occurs.
Before we get heavily into the HTML::Parser, we'll look at a basic example. The print_links.pl script
parses the HTML document presented to it on the command line or standard input, extracts all the
links and images, and prints out their URLs. In the following example, we use get_url2.pl to fetch
the Google search engine's home page and pipe its output to print_links.pl:

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Web Clients 227

% get_url2.pl http://www.google.com | print_links.pl

img: images/title_homepage2.gif
link: advanced_search.html
link: preferences.html
link: link_NPD.html
link: jobs.html
link: http://directory.google.com
link: adv/intro.html
link: websearch_programs.html
link: buttons.html
link: about.html

Figure 9.20 shows the code for print_links.pl.

Figure 9.20. The print_links.pl script

Lines 1–3: Load modules—After turning on strict syntax checking, we load HTML:: Parser. This
is the only module we need.
Lines 4–5: Create and initialize the parser object—We create a new HTML::Parser object by
calling its new() method. For reasons explained in the next section, we tell new() to use the
version 3 API by passing it the api_version argument.
After creating the parser, we configure it by calling its handler() method to install a handler
for start tag events. The start argument points to a reference to our print_link() subroutine;
this subroutine is invoked every time the parser encounters a start tag. The third argument to
handler() tells HTML:: Parser what arguments to pass to our handler when it is called. We
request that the parser pass print_link() the name of the tag (tagname) and a hash refer-
ence containing the tag's attributes (attr).
Lines 6–7: Parse standard input—We now call the parser's parse() method, passing it lines
read via the <> function. When we reach the end of file, we call the parser's eof() method to
tell it to finish up. The parse() and eof() methods behave identically to the HTML::TreeBuilder
methods we looked at earlier.
Lines 8–15: The print_link() callback—Most of the program logic occurs in
print_link(). This subroutine is called during the parse every time the parser encounters a
start tag. As we specified when we installed the handler, the parser passes the subroutine the
name of the tag and a hash reference containing the tag's attributes. Both the tag name and all
the attribute names are automatically transformed to lowercase letters, making it easier to deal
with the rampant variations in case used in most HTML.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Web Clients 228

We are interested only in hypertext links, the <a> tag, and inline images, the <img> tag. If the tag
name is "a", we print a line labeled "link:" followed by the contents of the href attribute. If, on the
other hand, the tag name is "img", we print "img:" followed by the contents of the src attribute. For
any other tag, we do nothing.

The HTML::Parser API

HTML::Parser has two APIs. In the earlier API, which was used through version 2 of the module,
you install handlers for various events by subclassing the module and overriding methods named
start(), end(), and text(). In the current API, introduced in version 3.0 of the module, you call
handler() to install event callbacks as we did in Figure 9.20.
You may still see code that uses the older API, and HTML::Parser goes to pains to maintain com-
patibility with the older API. In this section, however, we highlight only the most useful parts of the
version 3 API. See the HTML::Parser POD documentation for more information on how to control
the module's many options.
To create a new parser, call HTML::Parser->new().

$parser = HTML::Parser->new(@options)
The new() method creates a new HTML::Parser. @options is a series of option/value pairs that change
various parser settings. The most used option is api_version, which can be "2" to create a version 2 parser, or
"3" to create a version 3 parser. For backward compatibility, if you do not specify any options new() creates
a version 2 parser.

Once the parser is created, you will call handler() one or more times to install handlers.

$parser->handler($event => \&handler, $args)

The handler() method installs a handler for a parse event. $event is the name of the event, &handler
contains a reference to the callback subroutine to handle it, and $args is a string telling HTML::Parser what
information about the event the subroutine wishes to receive.

The event name is one of start, end, text, comment, declaration, process, or default. The first three
events are the most common. A start event is generated whenever the parser encounters a start
tag, such as <strong>. An end event is triggered when the parser encounters an end tag, such as
</strong>. text events are generated for the text between tags. The comment event is generated
for HTML comments. declaration and process events apply primarily to XML elements. Last, the
default event is a catchall for anything that is not explicitly handled elsewhere.
$args is a string containing a comma-delimited list of information that you want the parser to pass
to the handler. The information will be passed as subroutine arguments in the exact order that they
appear in the $args list. There are many possible arguments. Here are some of the most useful:
• tagname—the name of the tag
• text—the full text that triggered the event, including the markup delimiters
• dtext—decoded text, with markup removed and entities translated
• attr—a reference to a hash containing the tag attributes and values
• self—a copy of the HTML::Parser object itself
• "string" —the literal string (single or double quotes required!)
For example, this call causes the get_text() handler to be invoked every time the parser pro-
cesses some content text. The argument passed to the handler will be a three-element list that
contains the parser object, the literal string "TEXT", and the decoded content text:

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Web Clients 229

$parser->handler('text'=>\&get_text, "self,'TEXT',dtext");

• tagname is most useful in conjunction with start and end events. Tags are automatically down-
cased, so that <UL>, <ul>, and <Ul> are all given to the handler as "ul". In the case of end
tags, the "/" is suppressed, so that an end handler receives "ul" when a </ul> tag is encountered.
• dtext is used most often in conjunction with text events. It returns the nontag content of the
document, with all character entities translated to their proper values.
• The attr hash reference is useful only with start events. If requested for other events, the hash
reference will be empty.
Passing handler() a second argument of undef removes the handler for the specified event,
restoring the default behavior. An empty string causes the event to be ignored entirely.

$parser->handler($event =>\@array, $args)

Instead of having a subroutine invoked every time the parser triggers an event, you can have the parser fill an
array with the information that would have been passed to it, then examine the array at your leisure after the
parse is finished.
To do this, use an array reference as the second argument to handler(). When the parse is done, the array
will contain one element for each occurrence of the specified event, and each element will be an anonymous
array containing the information specified by $args.

Once initialized, you trigger the parse with parse_file() or parse().

$result = $parser->parse_file($file)
$result = $parser->parse($data)
$parser->eof
The parse_file(), parse(), and eof() methods work exactly as they do for HTML::TreeBuilder. A han-
dler that wishes to terminate parsing early can call the parser object's eof() method.

Two methods are commonly used to tweak the parser.

$bool = $parser->unbroken_text([$bool])
When processing chunks of content text, HTML::Parser ordinarily passes them to the text handler one chunk
at a time, breaking text at word boundaries. If unbroken_text() is set to a true value, this behavior changes
so that all the text between two tags is passed to the handler in a single operation. This can make some pattern
matches easier.
$bool = $parser->xml_mode([$bool])
The xml_mode() method puts the parser into a mode compatible with XML documents. This has two major
effects. First, it allows the empty element construct, <tagname/>. When the parser encounters a tag like this
one, it generates two events, a start event and an end event.
Second, XML mode disables the automatic conversion of tag and attribute names into lowercase. This is
because XML, unlike HTML, is case sensitive.

search_rfc.pl Using HTML::Parser

We'll now rewrite search_rfc.pl (Figures 9.8 and 9.10) to use HTML::Parser. Instead of using an ad
hoc pattern match to find the RFC names in the search response document, we'll install handlers
to detect the appropriate parts of the document, extract the needed information, and print the results.
Recall that the matching RFCs are in an ordered list (<OL>) section and have the following format:

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Web Clients 230

<OL>
<LI><A HREF="ref1">rfc name 1</A> - <STRONG>description 1</STRONG>
<LI><A HREF="ref2">rfc name 2</A> - <STRONG>description 2</STRONG>
...
</OL>

We want the parser to extract and print the text located within <A> and <STRONG> elements, but
only those located within an <OL> section. The text from other parts of the document, even those
in other <A> and <STRONG> elements, are to be ignored. The strategy that we will adopt is to have
the start handler detect when an <OL> tag has been encountered, and to install a text handler to
intercept and print the content of any subsequent <A> and <STRONG> elements. An end handler
will detect the </OL> tag, and remove the text handler, so that other text is not printed.
Figure 9.21 shows this new version, named search_rfc3.pl.

Licensed by
Stjepan Maric
4218908

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Web Clients 231

Figure 9.21. The search_rfc3.pl script

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Web Clients 232

Lines 1–5: Load modules—In addition to the LWP and HTTP::Request::Common modules, we
load HTML::Parser.
Lines 6–18: Set up search—We create an LWP::UserAgent and a new HTTP::Request in the
same way as in the previous incarnation of this script.
Lines 19–20: Create HTML::Parser—We create a new version 3 HTML::Parser object, and install
a handler for the start event. The handler will be the start() subroutine, and it will receive a
copy of the parser object and the name of the tag.
Lines 21–22: Issue request and parse—We call the user agent's request() method to process
the request. As in the print_links.pl script (Figure 9.20), we use a code reference as the second
argument to request() so that we can begin processing incoming data as soon as it arrives.
In this case, the code reference is an anonymous subroutine that invokes the parser's
parse() method.
After the request is finished, we call the parser's eof() method to have it finish up.
Line 23: Warn of error conditions—If the response object's is_success() method returns
false, we die with an error message. Otherwise, we do nothing: The parser callbacks are re-
sponsible for extracting and printing the relevant information from the document.
Lines 24–31: The start() subroutine—The start() subroutine is the callback for the start
event. It is called whenever the parser encounters a start tag. We begin by recovering the parser
object and the tag name from the stack. We need to remember the tag later when we are pro-
cessing text, so we stash it in the parser object under the key last-tag. (The HTML::Parser POD
documentation informs us that the parser is a blessed hash reference, and specifically invites
us to store information there in this manner.)
If the tag is anything other than "ol", we do nothing and just return. Otherwise, we install two new
handlers. One is a handler for the text event. It will be passed the parser object and the decoded
text. The other is a handler for the end event. Like start(), it will be passed the parser object
and the name of the end tag.
Lines 32–38: The end() subroutine—The end() subroutine is the handler for the end event.
It begins by resetting the last_tag key in the parser object. If the end tag isn't equal to "ol",
we just return, doing nothing. Otherwise, we set both the text and the end handlers to undef,
disabling them.
Lines 39–45: The extract() subroutine— extract() is the handler for the text event, and
is the place where the results from the search are extracted and printed. We get a copy of the
parser object and the decoded text on the subroutine call stack. After stripping whitespace from
the text, we examine the value of the last_tag key stored in the parser object. If the last tag is
"a", then we are in the <A> section that contains the name of the RFC. We print the text, followed
by a tab. If the last tag is "strong", then we are in the section of the document that contains the
title of the RFC. We print that, followed by a newline.
The new version of search_rfc.pl is more than twice as long as the original, but it adds no new
features, so what good is it? In this case, a full-blown parse of the search results document is overkill.
However, there will be cases when you need to parse a complex HTML document and regular
expressions will become too cumbersome to use. In these cases, HTML::Parser is a life saver.

Extracting Images from a Remote URL

To tie all the elements of this chapter together, our last example is an application that mirrors all the
images in an HTML document at a specified URL. Given a list of one or more URLs on the command
line, mirror_images.pl retrieves each document, parses it to find all inline images, and then fetches
the images to the current directory using the mirror() method. To keep the mirrored images up
to date, this script can be run repeatedly.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Web Clients 233

As the script runs, it prints the local name for the image. For example, here's what happened when
I pointed the script at http://www.yahoo.com:
% mirror_images.pl http://www.yahoo.com
m5v2.gif: OK
messengerpromo.gif: OK
sm.gif: OK

Running it again immediately gives three "Not Modified" messages. Figure 9.22 gives the complete
code listing for the script.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Web Clients 234

Figure 9.22. The mirror_images.pl script

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Web Clients 235

Lines 1–7: Load modules—We turn on strict syntax checking and load the LWP, PromptUtil,
HTTP::Cookies, HTML::Parser, and URI modules. The last module is used for its ability to re-
solve relative URLs into absolute URLs.
Lines 8–11: Create the user agent—We again use the trick of subclassing LWP::User Agent to
override the get_basic_credentials() method. The agent is stored in a variable named
$agent. Some of the remote sites we contact might require HTTP cookies, so we initialize an
HTTP::Cookies object on a file in our home directory and pass it to the agent's
cookie_jar() method. This allows the script to exchange cookies with the remote sites au-
tomatically.
Lines 12–15: Create the request and the parser—We enter a loop in which we shift URLs off the
command line and process them. For each URL, we create a new GET request using
HTTP::Request->new(), and an HTML::Parser object to parse the document as it comes in.
We install the subroutine start() as the parse handler for the start event. This handler will
receive a copy of the parser object, the name of the start tag, and a hash reference containing
the tag's attributes and their values.
Lines 16–24: Issue the request—We call the agent's request() method to issue the request,
returning a response object. As in the last example, we provide request() with a code refer-
ence as the second argument, causing the agent to pass the incoming data to this subroutine
as it arrives.
In this case, the code reference is an anonymous subroutine. We first check that the MIME type
of the response is text/html. If it isn't, we die with an error message. This doesn't cause the script
as a whole to die, but does abort processing of the current URL and leaves the error message
in a special X-Died: field of the response header.
Otherwise, the incoming document is parseable as an HTML file. Our handler is going to need
two pieces of extra information: the base URL of the current response for use in resolving relative
URLs, and the user agent object so that we can issue requests for inline images. We use the
same technique as in Figure 9.21, and stash this information into the parser's hash reference.
Lines 25–27: Warn of error conditions—After the request has finished, we check the response
for the existence of the X-Died: header and, if it exists, issue a warning. Likewise, we print the
response's status message if the is_success() method returns false.
Lines 28–37: The start() handler—The start() subroutine is invoked by the parser to han-
dle start tags. As called for by the argument list passed to handler(), the subroutine receives
a copy of the parser object, the name of the current tag, and a hash reference containing tag
attributes.
We check whether we are processing an <IMG> tag. If not, we return without taking further
action. We then check that the tag's src attribute is defined, and if so, copy it to a local variable.
The src attribute contains the URL of the inline image, and may be an absolute URL like
http://www. yahoo.com/images/messengerpromo.gif, or a relative one like images/
messengerpromo.gif. To fetch image source data, we must resolve relative URLs into absolute
URLs so that we can request them via the LWP user agent. We must also construct a local
filename for our copy of the image.
Absolutizing relative URLs is an easy task thanks to the URI module. The URI->new_abs()
method constructs a complete URL given a relative URL and a base. We obtain the base URL
of the document containing the image by retrieving the "base" key from the parser hash where
we stashed it earlier. This is passed to new_abs() along with the URL of the image (line 33),
obtaining an absolute URL. If the URL was already absolute, calling new_abs() doesn't hurt.
The method detects this fact and passes the URL through unchanged.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Web Clients 236

Constructing the local filename is a matter of extracting the filename part of the path (line 34),
using a pattern match to extract the rightmost component of the image URL.
We now call the user agent's mirror() method to copy the remote image to our local filesystem
and print the status message. Notice how we obtain a copy of the user agent from the parser
hash reference. This avoids having to create a new user agent.
Lines 38–46: The get_basic_credentials() method—This is identical to earlier versions.
There is a slight flaw in mirror_images.pl as it is now written. All images are mirrored to the same
directory, and no attempt is made to detect image name clashes between sites, or even within the
same site when the image paths are flattened (as might occur, for example, when mirroring remote
images named /images/whats_new.gif and /news/hot_news/whats_new.gif).
To make the script fully general, you might want to save each image in a separate subdirectory
named after the remote hostname and the path of the image within the site. We can do this relatively
painlessly by combining the URI host() and path() methods with the dirname() and
mkpath() functions imported from the File::Path and File::Basename modules. The relevant sec-
tion of start() would now look like this:
...
use File::Path 'mkpath';
use File::Basename 'dirname';
...
sub start {
...
my $remote_name = URI->new_abs($url,$parser->{base});
my $local_name = $remote_name->host . $remote_name->path;
mkpath(dirname($local_name),0,0711);
...
}

For the image URL http://www.yahoo. com/images/whats_new.gif, this will mirror the file
into the subdirectory http://www.yahoo.com/images.

Summary
The LWP module allows you to write scripts that act as World Wide Web clients. You can retrieve
Web pages, simulate the submission of fill-out forms, and easily negotiate more obscure aspects
of the HTTP protocol, such as cookies and user authentication.
The HTML-Formatter and HTML-Parser modules enhance LWP by giving you the ability to format
and parse HTML files. These modules allow you to transform HTML into text or postscript for printing,
and to extract interesting information from HTML files without resorting to error-prone regular ex-
pressions. As an added benefit, HTML::Parser can parse XML.
There's more to LWP than can be covered in a single book chapter. A good way to learn more about
the package is to examine the lwp-request, lwp-download, and lwp-rget scripts, and other examples
that come with the package.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 237

Part 3. Developing TCP Client/Server Systems

The next seven chapters lead you through the process of creating novel TCP-based network serv-
ices. We develop a variety of real-life applications and illustrate the tradeoffs to consider when
choosing server architectures.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 238

Chapter 10. Forking Servers and the inetd

Daemon

Although the simple TCP servers developed in Chapters 4 and 5 (Figures 4.2 and 5.4) are straight-
forward, they actually suffer a significant deficiency. Both of these servers work by servicing one
client at a time. While they are working on one client, other clients can't connect.1
Connection-oriented servers must overlap their I/O by providing some sort of concurrency among
the multiple sessions. This chapter discusses various techniques for doing so.

Standard Techniques for Concurrency

Over the years, network programmers have developed a number of standard techniques for main-
taining concurrent I/O. The techniques range from simple tricks that add only a couple of lines of
code to the basic server, to methods that more than double the size and complexity of the code.
Unfortunately, these techniques are hostage to the peculiarities of how the underlying operating
system handles I/O, and this is notoriously variable from one platform to another. As a result, some
of the techniques I describe here will only be available on UNIX systems.
Moving upward in complexity from the simplest to the most complex, the techniques are the forking
server, the multithreaded server, and the multiplexed server.

Forking Server
The server spends its time in an accept() loop. Each time a new incoming connection is accepted,
the server forks, creating an identical child process. The task of handling the child connection's I/O
is handed off to the child, and the parent goes back to listening for new connections (Figure 10.1).
When the child is finished handling the connection, it simply exits.

1Technically, they do connect, but the operating system queues them until the script calls accept (). No I/O can occur until the server has
finished servicing the previous connection.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Forking Servers and the inetd Daemon 239

Figure 10.1. A forking server

In a forking server, the multitasking nature of the operating system allows parent and child to run
simultaneously. At any point in time there is a single parent process and multiple child processes,
each child dedicated to handling a different client connection.
This technique is available on platforms that implement fork(), all UNIX versions of Perl, and
version 5.6 and higher on Win32 platforms. The Macintosh port of Perl does not currently support
fork().
A special case of the multitasked server is the Inetd "super daemon," which can be used to write
simple concurrent servers without worrying too much about the details. We look at Inetd at the end
of this chapter.

Multithreaded Server
Next in complexity is multithreading. Conceptually similar to the previous solution, the server calls
accept() in a tight loop. Each time accept() returns a connected socket, the server launches a
new thread of execution to handle the client session. Threads are similar to processes, but threads
share the same memory and other resources. When the thread is done, it exits. In this model, there
are multiple simultaneous threads of execution, one handling the main accept() loop, and the
others handling client sessions.
This technique is available in Perl versions 5.005 and higher, and only on platforms that support
threads. The Windows version of Perl supports threads, as do many (but not all) UNIX versions.
MacPerl does not currently support multithreading. We discuss multithreading in Chapter 11.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Forking Servers and the inetd Daemon 240

Multiplexed Server
The most complex technique uses the select() function to interweave communications sessions.
This technique takes advantage of the fact that the network is slower than the CPU, and that most
of the time in a network server is spent waiting for a socket to become ready for reading or writing.
In this technique, the server creates and maintains a pool of filehandles, one for the listen socket,
and one for each connected client. Each time through the loop, the server checks the sockets using
the select() function to ascertain whether any is ready for reading and writing. If so, the server
handles the I/O for that socket and then goes back to waiting with select().
select() is available on all major Perl platforms. Unfortunately, the technique is also the trickiest
to use correctly. We discuss multiplexing in Chapters 12 and 13.
Built on top of these basic techniques are a number of variations, including preforking, thread pools,
and nonblocking I/O. There are also more esoteric methods for achieving concurrency, including
signal-driven I/O, asynchronous I/O, and others. We don't cover those here; for further information

Licensed by
on these techniques, see [Stevens 1998].

Running Example: A Psychotherapist Server

Stjepan Maric
Our running example for this and the next two chapters uses the Chatbot::Eliza module, John No-
lan's marvelous pure-Perl clone of Joseph Weizenbaum's classic psychotherapist simulation. Chat-
bot::Eliza works very simply. It accepts a line of input from the user, transforms it according to a
clever set of transformational rules, and echoes it back to the user in the form of a question. The
effect is a lot like talking to a particularly robotlike Freudian psychoanalyst.

4218908
The Chatbot::Eliza module is available on CPAN. It is simple to use. You load it, create a new
Chatbot::Eliza object with the new() method, and then invoke the object's command_inter-
face() method. This method conducts a conversation with the user, reading from standard input
and writing to standard output. The lines of code in Figure 10.2 are all that are needed to run a
psychotherapist session.

Figure 10.2. Command-line Eliza program

Lines 1–2: Load modules—We load the Chatbot::Eliza module.

Line 3: Turn on autoflush—We turn on autoflush mode for STDOUT so that, rather than be buf-
fered, data is output immediately with each call to print(). Although this isn't necessary for
the command-line version of the program, it is necessary for the server version.
Line 4: Create a new Chatbot::Eliza object—We call Chatbot::Eliza->new() to create a
new object named $bot.
Line 5: Enter Chatbot object's IO loop—We now call the Chatbot object's command_inter-
face() method. This starts a simple I/O loop, which prints out a prompt, accepts user input,
transforms the input into a statement from the "psychiatrist," and then repeats.
Here's a transcript of a conversation with Eliza (yours will vary):

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Forking Servers and the inetd Daemon 241

% eliza.pl
Eliza: Hello, I am a computer program.
you: I am a human being.
Eliza: How long have you been a human being?
you: About 39 years.
Eliza: That is interesting. Please continue.
you: It started when I was born...
Eliza: Why do you tell me you were born now?
you: Because you asked.
Eliza: We were discussing you -- not me.
you: bye
Eliza: Goodbye. It was nice talking to you.

The psychotherapist application calls for a connection-oriented interface because the interaction
between client and server is long-lived and because there is state associated with each connection.
The Chatbot::Eliza object keeps track of the history of the conversation in order to construct its
utterances based on the user's previous statements.

The Psychotherapist as a Forking Server

We are going to reimplement the psychotherapist program as a forking network server, but before
we do so we must discuss issues surrounding the termination of child processes in UNIX-based
forking servers. This discussion does not apply to servers running under the Microsoft Windows
versions of Perl.

Zombies
We've already used fork(): In Chapter 2 we used it in a toy example to distribute the load of a
computation across two child processes (Figure 2.5), and in Chapter 5 we used it to avoid synchro-
nization and deadlock problems in the gab2.pl script (Figure 5.8).
One difference between those examples and the forking server examples in this chapter is the
relative longevity of the parent and the child processes. In the earlier examples, the parent process
does not survive the demise of its children for any significant length of time. The parent exits soon
after its children do.
In forking servers, however, the parent process is very long-lived. Web servers, for example, run
for months at a time. The children, however, live only as long as a client connection, and a server
may spawn thousands of children during its lifetime. Under this scenario, the issue of "zombie pro-
cesses" becomes important.
Once fork() is called, parent and child processes are almost, but not quite, free to go their own
ways. The UNIX system maintains a tenuous connection between the two processes. If the child
exits before the parent does, the child process does not disappear, but instead remains in the system
process table in a mummified form known as a "zombie." The zombie remains in the process table
for the sole purpose of being able to deliver its exit status code to the parent process when and if
the parent process asks for it using the wait() or waitpid() call, a process known as "reaping."
This is a limited form of IPC that allows the parent to find out whether a process it launched exited
successfully, and if not, why.
If a parent process forks a lot of children and does not reap them in a timely manner, zombie pro-
cesses accumulate in the process table, ultimately creating a virtual Night of the Living Dead, in
which the table fills up with defunct processes. Eventually, the parent process hits a system-imposed
limitation on the number of subprocesses it can launch, and subsequent calls to fork() fail.
To avoid this eventuality, any program that calls fork() must be prepared to reap its children by
calling wait() or waitpid() at regular intervals, preferably immediately after a child exits.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Forking Servers and the inetd Daemon 242

UNIX makes it convenient to call wait() or waitpid() at the appropriate time by providing the
CHLD signal. The CHLD signal is sent to a parent process whenever the state of any of its children
changes. Possible state changes include the child exiting (which is the event we're interested in)
and the child being suspended by a STOP signal. The CHLD signal does not provide information
beyond the bare-bones fact that some child's state changed. The parent must call wait() or
waitpid() to determine which child was affected, and if so, what happened to it.

$pid = wait ()
This function waits for any child process to exit and then returns the PID of the terminated child. If no child is
immediately ready for reaping, the call hangs (block) until there is one.
If you wish to determine whether the child exited normally or because of an error, you may examine the special
$? variable, which contains the child's exit status code. A code of 0 indicates that the child exited normally.
Anything else indicates an abnormal termination. See the perlvar POD page for information on how to interpret
the contents of $?.
$pid = waitpid ($pid, $flags)
This version waits for a particular child to exit and returns its PID, placing the exit status code in $?. If the child
named by $pid is not immediately available for reaping, waitpid() blocks until it is. To wait for any child to
be available as wait() does, use a $pid argument of -1.

The behavior of waitpid() can be modified by the $flags argument. There are a number of
handy constants defined in the :sys_wait_h group of the standard POSIX module. These con-
stants can be bitwise ORed together to combine them. The most frequently used flag is WNOHANG,
which, if present, puts waitpid() into nonblocking mode. waitpid() returns the PID of the child
process if available; if no children are available, it returns -1 and waitpid() blocks waiting for
them. Another occasionally useful flag is WUNTRACED, which tells waitpid() to return the PIDs of
stopped children as well as terminated ones.

Reaping Children in the CHLD Handler

The standard way for Perl servers to reap their children is to install a handler for the CHLD signal.
You'll see this fragment in many examples of server code:
$SIG{CHLD} = sub { wait(); }

The effect of this is to call wait() every time the server receives a CHLD signal, immediately reaping
the child and ignoring its result code. This code works most of the time, but there are a number of
unusual situations that will break it. One such event is when a child is stopped or restarted by a
signal. In this case, the parent gets a CHLD signal, but no child has actually exited. The wait() call
stalls indefinitely, bringing the server to a halt—not at all a desirable state of affairs.
Another event that can break this simple signal handler is the nearly simultaneous termination of
two or more children. The UNIX signal mechanism can deal with only one signal of a particular type
at a time. The two termination events are bundled into a single CHLD event and delivered to the
server. Although two children need to be reaped, the server calls wait() only once, leaving an
unreaped zombie. This "zombie leak" becomes noticeable after a sufficiently long period of time.
The last undesirable situation occurs when the parent process makes calls that spawn subpro-
cesses, including the backtick operator (`), the system() function, and piped open()s. For these
functions Perl takes care of calling wait() for you before returning to the main body of the code.
On some platforms, however, extraneous CHLD signals leak through even though there's no
unreaped child to wait for. The wait() call again hangs.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Forking Servers and the inetd Daemon 243

The solution to these three problems is to call waitpid() with a PID of -1 and a flag of WNO-
HANG. The first argument tells waitpid() to reap any available child. The second argument pre-
vents the call from hanging if no children are available for reaping. To avoid leaking zombies, you
should call waitpid() in a loop until it indicates, by returning a result code of -1, that there are no
more children to reap.
Here's the idiom:
use POSIX 'WNOHANG';
$SIG{CHLD} = \&reaper;
sub reaper {
while ((my $kid = waitpid(-1,WNOHANG)) > 0) {
warn "Reaped child with PID $kid\n";
}
}

In this case we print the PID of the reaped child for the purpose of debugging. In many cases you
will ignore the child PID, but in others you'll want to examine the child PID and status code and
perform some action in case of a child that exited abnormally. We'll see examples of this in later
sections.

Psychotherapist Server with fork

We're now ready to rewrite the psychotherapist example as a forking server (Figure 10.3).

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Forking Servers and the inetd Daemon 244

Figure 10.3. Psychotherapist as a forking server

Lines 1–5: Bring in modules—We begin by loading the Chatbot::Eliza and IO::Socket modules,
and importing the WNOHANG constant from the POSIX module. We also define the port our server
will listen to, in this case 12000.
Lines 6–7: Define constants and variables—We define the default port to bind to, and initialize
a global variable, $quit to false. When this variable becomes true, the main server loop exits.
Lines 8–11: Install signal handlers—We install a signal handler for CHLD events using a variant
of the waitpid() idiom previously discussed.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Forking Servers and the inetd Daemon 245

$SIG{CHLD} = sub { while ( waitpid(-1,WNOHANG)>0 ) { } };

We want the server to clean up gracefully after interruption from the command line, so we create
an INT handler. This handler just sets $quit to true and returns.
Lines 12–19: Create listening socket—We create a new listening socket by calling
IO::Socket::INET->new() with the LocalPort and Listen arguments. We also specify a
PROTO argument of "tcp" and a true value for Reuse, allowing this server to be killed and
relaunched without the otherwise mandatory wait for the port to be freed.
In addition to these standard arguments, we declare a Timeout of 1 hour. As we did in the reverse
echo server of Figure 5.4, this is done in order to make accept() interruptable by signals. We
want accept() to return prematurely when interrupted by INT so that we can check the status
of $quit.
Lines 20–21: Accept incoming connections—We now enter a while() loop. Each time through
the loop we call accept() to get an IO::Socket object connected to a new client.
Lines 22–27: Fork: child handles connection—Once accept() returns, instead of talking di-
rectly to the connected socket, we immediately call fork() and save the result code in the
variable $child. If $child is undefined, then the fork() failed for some reason and we die
with an error message.
Otherwise, if the value of $child is equal to numeric 0, then we know we are inside the child
process and will be responsible for handling the communications session. As the child, we will
not call accept() again, so we close our copy of the listening socket. This closing is not strictly
necessary, but it's always a good idea to tidy up unneeded resources, and it avoids the possibility
of the child inadvertently trying to perform operations on the listen socket.
We now call a subroutine named interact(), passing it the connected socket object. inter-
act() manages the Eliza conversation and returns when the user terminates the connection
(by typing "bye" for example). After interact() returns, the child terminates by calling
exit().
Lines 28–29: Parent cleans up—If $child was nonzero, then we are the parent process. In this
case, we just close our copy of the connected socket and go back to the top of the loop to
accept() another connection. While we are waiting for a new connection, the child is taking
care of the old one.
Lines 30–38: interact() subroutine—The interact() subroutine is called by the child
process to handle all interaction between the client and the server. The Chatbot::Eliza com-
mand_interface() method is hardwired to read from STDIN and write to STDOUT. But we
want it to read and write to the socket instead.
This is actually an easy problem to finesse. When we loaded IO::Socket, it also brought in methods
from its parent class, IO::Handle. Among these methods is fdopen(), which we looked at in
Chapter 1 (The IO::Handle and IO::File Modules). The fdopen() method closes an existing file-
handle and then reopens it using information from another filehandle that you give it. It works with
any filehandle, including the standard three. We call fdopen() three times, once each for STDIN,
STDOUT, and STDERR. Each time we call fdopen(), we pass it the socket object and a symbolic
access mode. STDIN is reopened for reading with a mode of <, while STDOUT and STDERR are each
reopened for writing with a mode of >. Now, almost as if by magic, writing to STDOUT and
STDERR send data flying down the connected socket, and reading from STDIN performs a read on
the socket.
The remainder of this subroutine is identical to the command-line psychotherapist from Figure
10.2. We turn on autoflush mode so that data sent to STDOUT by Chatbot::Eliza is immediately visible
at the remote end, create a new Chatbot::Eliza object, and call its command_interface() method.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Forking Servers and the inetd Daemon 246

With the server running, you may confirm that you can telnet to port 12000 and regale the psychiatrist
simulation with your personal hang-ups and pet peeves. Without closing the previous session, open
a new window and telnet to port 12000 again. You should be able to conduct multiple simultaneous
sessions. Run the ps program to confirm that each active session creates a new copy of the server,
and that when you close a session, the corresponding process disappears immediately without
leaving a zombie. When you are done, kill the server with ^C.
The following is an example of the server in action:
% telnet localhost 12000
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
Eliza: How do you do. Please tell me your problem.
you: hey this really works!
Eliza: I'm not sure I understand you fully.
you: you are a server
Eliza: Perhaps you would like to be a server.
you: no
Eliza: Why not?
you: quit
Eliza: Goodbye. It was nice talking to you.
Connection closed by foreign host.

Using the Psychotherapist Server on Windows Platforms

Although fork() works correctly on Windows systems, fdopen() on sockets does not. For Win-
dows systems, the interact() subroutine from Figure 10.3 must be modified to avoid the
fdopen() call. The easiest way to do this is to replace the call to command_interface() with a
new version that accepts the input and output filehandles to use instead of hardwired STDIN and
STDOUT. In the next chapter, Figure 11.2 develops a subclass of Chatbot::Eliza, called Chat-
bot::Eliza::Server, that does exactly that.
To run the forking server on Windows platforms, change the use Chatbot::Eliza line to:
use Chatbot::Eliza::Server;

and modify interact() to read like this:

sub interact {
my $sock = shift;
my $bot = Chatbot::Eliza::Server->>new;
$bot->>command_interface ($sock, $sock);
close $sock;
{

A Client Script for the Psychotherapist Server

Before we push onward with our discussion of forking servers, let's write a client that we can use to
talk to the psychotherapist server. After all, we shouldn't have to be stuck with musty old telnet when
we can use Perl! Seriously, though, this script illustrates the usefulness of sysread() and sys-
write() for working with unbuffered byte streams.
At first, the gab2.pl script developed in Chapter 5 (Figure 5.8) would seem to fit the bill for the client
side of the equation. But there's a problem. gab2.pl was designed for line-oriented communications,
in which the server transmits complete lines terminated in CRLF. The psychotherapist server, how-
ever, is not entirely line oriented. For one thing, it terminates its lines with what Chatbot::Eliza thinks

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Forking Servers and the inetd Daemon 247

is appropriate (which happens to be the logical newline " \n " character). For another, the "you:"
prompt that the server transmits after each utterance does not end with a newline. The combined
effect of these problems is that when we point gab2.pl at the psychotherapist server's port, we see
no output.
What we need is a more general bytestream-oriented client that reads and writes its data in arbitrary
chunks as they become ready, rather than waiting for complete lines. As it turns out, very few mod-
ifications to gab2.pl are needed to turn it into this type of client.
Figure 10.4 shows the revised script, gab3.pl. The significant changes are in the
user_to_host() and host_to_user() subroutines. Instead of the line-oriented read and write
calls of the earlier version, these subroutines now consist of tight loops using sysread() and
syswrite(). For example, here is the code fragment from host-to-user() that reads from the
socket and writes to STDOUT:

Figure 10.4. A bytestream-oriented gab client

syswrite(STDOUT,$data) while sysread($s,$data,BUFSIZE);

Similarly, the user_to_host() subroutine, which is responsible for copying user data to the
socket, is modified to look like this:

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Forking Servers and the inetd Daemon 248

syswrite($s,$data) while sysread(STDIN,$data,BUFSIZE)

The BUFSIZE parameter is relatively arbitrary. For performance it should be roughly as large as the
largest chunk of text that can be emitted by the psychotherapist server, but it will work just fine if it's
smaller or larger. In this case, I chose 1024 for the constant, which seems to work pretty well.
The significance of using sysread() here rather than read(), its buffered alternative, is that
sysread() allows partial reads. If there are no BUFSIZE bytes ready to be read, sysread()
returns whatever is available. read(), in contrast, blocks while waiting to satisfy the request, de-
laying the psychotherapist's responses indefinitely. The same argument doesn't apply to sys-
write(), versus print(), however. Since IO::Socket objects are autoflushed by default, sys-
write() and print() have exactly the same effect.

Notes on gab3.pl
After developing gab3.pl, I was interested in how it performed relative to the send-wait-read version
of Figure 5.6 (gab1.pl) and the forking line-oriented version of Figure 5.7 ( gab2.pl). To do this, I
timed the three scripts while transmitting a large text file to a conventional echo server. This test
allowed both the line-oriented scripts and the byte-stream script to function properly.
Relative to gab1.pl, I found an approximately 5-fold increase in speed, and relative to gab2.pl, a 1.5-
fold increase. The big efficiency gain when switching from the single to the multitasking design was
dramatic, and represents the fact that the multitasking design keeps the network pipe full and running
in both directions simultaneously, while the send-wait-read design uses only half the bandwidth at
any time, and waits to receive the entire transmission before sending a response.
Another interesting benchmark result is that when I tried replacing the built-in calls to sys-
write() and sysread() in gab3.pl with their object-oriented wrappers, I found a 20 percent
decrease in efficiency, reflecting Perl's method call overhead. This probably won't make a significant
difference in most networking applications, which are dominated by network speeds, but is worth
keeping in mind for those tight inner loops where efficiency is critical.
As an aside, while testing gab3.pl with eliza_server.pl, I discovered an apparent bug in the Eliza
module's command_interface() method. When it reads a line of input from STDIN, it never
checks for end of file. As a result, if you terminate the connection at the client side, com-
mand_interface() goes into a very unattractive infinite loop that wastes CPU time.
The easy solution is to replace the Chatbot::Eliza_testquit() method, which checks the
input string for words like "quit" and "bye." By checking whether the string is undefined, _test-
quit() can detect end of file. Insert this definition somewhere near the bottom of the Eliza server:
sub Chatbot::Eliza::_testquit {
my ($self,$string) = @_;
return 1 unless defined $string; # test for EOF
foreach (@{$self->{quit}}) { return 1 if $string =~ /\b$_\b/i };
}

The server will now detect and respond correctly to the end-of-file condition.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Forking Servers and the inetd Daemon 249

Daemonization on UNIX Systems

The forking psychotherapist server has a deficiency.2 When the server is launched, it doesn't au-
tomatically go into the background but instead ties up a terminal where it can be brought down by
an inadvertent tap on the interrupt key. Of course, the user launching it from the command line can
always background the server, but that is inconvenient and error prone, since the server might be
brought back into the foreground inadvertently.
Under UNIX, most network servers act as "daemons." When launched, they go into the background,
and keep running until they are deliberately killed or the system itself is shut down. They have no
access to a terminal or command window. Instead, if they want to issue status messages, they must
log them to a file. The word "daemon" was chosen to convey the image of a sorcerer's magical
servant who does his bidding invisibly. In this case, the server is the daemon, and network com-
munications is the magic.
On launch, a daemon should put itself into the background automatically and close its standard
input, output, and error handles. It should also completely dissociate itself from the "controlling
terminal" (the terminal window or console from which the daemon was launched). This has two
purposes. One is that the program (or a subprocess launched by it) will not be able to reopen the
terminal device and inadvertently intermix its output with that of other programs. The second effect
is that the daemon will not receive a HUP (hangup) signal when the user exits the command shell
after launching the server.
Network daemons should also:
1. Change their current working directory to the root directory. This normalizes the environment
and avoids problems with unmounting the filesystem from which the daemon was started.
2. Change their file creation mask to a known state (rather than inheriting it from the shell).
3. Normalize the PATH environment variable.
4. Write their process IDs to a file in /var/run or a similar location.
5. Optionally, use syslog (or the Windows event logger) to write diagnostic messages to the
system log file.
6. Optionally, handle the HUP signal by reinitializing themselves or reloading their configuration
files.
7. Optionally, use the chroot() call to place themselves in a restricted part of the filesystem,
and/or set their privileges to those of an unprivileged user on the system.

Autobackgrounding
In this section, we develop a routine for autobackgrounding network daemons and performing tasks
1 through 4. In Chapter 16 we discuss techniques for implementing items 5 through 7.
Figure 10.5 lists the become_daemon() subroutine, which a server process should call very early
during its initialization phase. This subroutine uses a standard UNIX trick for backgrounding and
dissociating from the controlling terminal. It forks itself (line 2) and the parent process exits, leaving
only the child in control.

2This discussion relies heavily on the UNIX process model, and will not translate to Macintosh or Windows systems. Windows NT and 2000
users can turn Perl scripts into background services using a utility called srvany.exe. See the section Background on Windows and Macintosh
Systems later in this chapter.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Forking Servers and the inetd Daemon 250

Figure 10.5. The become_daemon() subroutine

Licensed by
The child process now starts a new process session by calling the setsid() function, provided by
the POSIX module (line 4). A session is a set of processes that share the same terminal. At any
given time only one member of the set has the privilege of reading and writing to the terminal and
is said to be in the foreground, while other members of the group remain in the background (and if

Stjepan Maric
they try to do I/O to the terminal, are suspended until they are brought to the foreground). This
system is used by command shells to implement job control.
A session group is related, but not identical, to a process group. A process group is the set of
subprocesses that have been launched by a single parent, and is an integer corresponding to the

4218908
PID of the group's shared ancestor. You can use the Perl getpgrp() function to fetch the process
group for a particular process, and pass kill() the negative of a process group to send the same
signal simultaneously to all members of the group. This is how the shell does it when sending a
HUP signal to all its subprocesses just prior to exiting. A newly forked child belongs to the same
session group and process group as its parent.
setsid() does several things. It creates both a new session and a new process group, and makes
the current process the session leader. At the same time, it dissociates the current process from
the controlling terminal. The effect is to make the child process completely independent of the shell.
setsid() fails if the process is a session leader at the time the function is called (i.e., is in the
foreground), but the earlier fork ensures that this is not the case.
After calling setsid(), we reopen the STDIN and STDOUT filehandles onto the "do nothing" special
device, /dev/null, and make STDERR a copy of STDOUT (lines 5–7). This maneuver prevents output
from the daemon from appearing on the terminal. It then calls chdir() to change the current work-
ing directory to the root filesystem, resets the file creation mask to 0, and sets the PATH environment
variable to a small number of standard directories (line 10). We return the new process ID from the
$$ global. Because we forked, the process ID is now different from its value when the subroutine
was called, and returning the new PID explicitly in this way is a good way to remind ourselves of
that fact.
There are a number of variations on the become_daemon() subroutine. Stevens [1998] recom-
mends forking not once but twice, warning that otherwise it is possible for the first child to reacquire
a controlling terminal by deliberately reopening the /dev/tty device. However, this event is unlikely,
and few production servers do this.
Instead of reopening the standard filehandles onto /dev/null, you may want to simply close them:
close $_ foreach (\*STDIN,\*STDOUT,\*STDERR);

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Forking Servers and the inetd Daemon 251

However, this strategy may confuse subprocesses that expect the standard filehandles to be open,
so it is best avoided.
Finally, a few older UNIX systems, such as ULTRIX, do not have a working setsid(). On such
systems, the call to setsid() returns a run-time error. On such systems, you can use the
Proc::Daemon module, available on CPAN, which contains the appropriate workarounds.

PID Files
Another feature we can add at this time is a PID file for the psychotherapist server. By convention,
servers and other system daemons write their process IDs into a file named something like /var/run/
servername.pid. Before exiting, the server removes the file. This allows the system administrator
and other users to send signals to the daemon via this shortcut:
kill -TERM `cat /var/run/servername.pid`

A clever daemon checks for the existence of this file during startup, and refuses to run if the file
exists, which might indicate that the server is already running. Very clever daemons go one step
further, and check that the process referred to by the PID file is still running. It is possible that a
previous server crashed or was killed before it had a chance to remove the file. The
open_pid_file() subroutine listed in Figure 10.6 implements this strategy.

Figure 10.6. open_pid_file() routine

Lines 1–3: Check whether old PID file exists— open_pid_file() is called with the path to the
PID file. Our first action is to apply the -e file test to the file to determine whether it already exists.
Lines 4–6: Check whether old PID file is valid—If the PID file exists, we go on to check whether
the process it indicates is still running. We use IO::File to open the old PID file and read the
numeric PID from it. To determine if this process is still running, we use kill() to send signal
number 0 to the indicated process. This special signal number 0 doesn't actually send a signal,
but instead returns true if the indicated process (or process group) can receive signals. If
kill() returns true, we know that the process is still running and exit with an error message.
Otherwise, if kill() returns false, then we know that the previous server process either exited
uncleanly without cleaning up its PID file, or that it is running under a different user ID and the
current process lacks the privileges to send the signal. We ignore this latter case, assuming that
the server is always launched by the same user. If this assumption is false, then our attempt to
unlink the old PID file in the next step will fail and no harm will be done.
Lines 7–9: Unlink old PID file—We write a warning to standard error and attempt to unlink the
old PID file, first checking with the -w file test operator that it is writable. If either the -w test or
the unlink() fail, we abort.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Forking Servers and the inetd Daemon 252

Lines 10–12: Create new PID file—The last two steps are to create a new PID file and open it
for writing. We call IO::File->new() with a combination of flags that creates the file and
opens it, but only if it does not previously exist. This prevents the file from being clobbered in
the event that the server is launched twice in quick succession, both instances check for the PID
file and find it absent, and both try to create a new PID file at about the same time. If successful,
we return the open filehandle to the caller.
open_pid_file() should be invoked before autobackgrounding the server. This gives it a chance
to issue error messages before standard error is closed. The caller should then call become_dae-
mon() to get the new process ID, and write that PID to the PID file using the filehandle returned by
open_pid_file(). Here's the complete idiom:
use constant PID_FILE => '/var/run/servername.pid';
$SIG{TERM} = $SIG{INT} = sub { exit 0; }

my $fh = open_pid_file(PID_FILE);
my $pid = become_daemon();
print $fh $pid;
close $fh;

END { unlink PID_FILE if $pid == $$; }

By convention, the /var/run directory is used by many UNIX systems to write PID files for running
daemons. Solaris systems use /etc or /usr/local/etc.
The END{} block guarantees that the server will remove the PID file before it exits. The file is un-
linked only if the current process ID matches the process ID returned by become_daemon(). This
prevents any of the server's children from inadvertently unlinking the file.
The reason for installing signal handlers for the TERM and INT signals is to ensure that the program
exits normally when it receives these signals. Otherwise, the END{} block would not be executed
and the PID file would remain around after the server had exited.
Figure 10.7 puts all these techniques together in a new and improved forking server, eliza_dae-
mon.pl. There should be no surprises in this code, with the minor exception that instead of placing
the PID file inside the standard /var/run directory, the example uses /var/tmp/eliza.pid. /var/run is a
privileged directory, and to write into it we would have to be running with root privileges. However,
this carries security implications that are not discussed until Chapter 16. It is not a particularly good
idea for a root process to write into a world-writable directory such as /var/tmp for reasons discussed
in that chapter, but there's no problem doing so as an unprivileged user. This script also incorporates
the fix to the Chatbot::Eliza::_testquit() subroutine discussed earlier.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Forking Servers and the inetd Daemon 253

Figure 10.7. The Eliza server (forking version) with daemon code

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Forking Servers and the inetd Daemon 254

Another point is that we create the listen socket before calling become_daemon(). This gives us
a chance to die with an error message before become_daemon() closes standard error. Chapter
16 discusses how daemons can log errors to a file or via the syslog system.

Starting Network Servers Automatically

Once a network server has been written, tested, and debugged, you may want it to start up auto-
matically each time the machine is booted. On UNIX systems this is relatively simple once you learn
the specifics of your operating system.
At boot time, UNIX (and Linux) systems run a series of shell scripts. Each shell script performs an
aspect of system initialization, such as checking file systems, checking user quotas, mounting re-
mote directories, and starting network services. You need only find a suitable shell script, add the
command needed to start your Perl-based server, and you're done.
The only catch is that the location and organization of these shell scripts varies considerably among
UNIX dialects. There are two general organizational styles in use. One style, derived from the BSD
lineage, uses a series of scripts beginning with the characters rc, for example, rc.boot and rc.sin-
gle. These files are usually located in either /etc or /etc/rc.d. On such systems, there is generally a
boot script reserved for local customizations named rc.local. If you have such a system, then the
easiest way to start a Perl script at boot time is to add a section to the bottom of rc.local, using as
your example other sections in the script.
For example, on the BSD-based system that I use at home, the end of my rc.local script has several
sections like this one:
# start time server
if [ -x /usr/sbin/xntpd ]; then
echo "Starting time server..."
/usr/sbin/xntpd
fi

This is a bit of Bourne shell-scripting language which says that if the file /usr/sbin/xntpd exists and
is executable, then echo a message to the console and run the program.
To start our eliza_daemon.pl script at boot time, we would add a section like this one:
# start psychotherapist server
if [ -x /usr/local/bin/eliza_daemon.pl ]; then
echo "Starting psychotherapist server..."
/usr/local/bin/eliza_daemon.pl
fi

This assumes that eliza_daemon.pl has been installed into the /usr/local/bin directory. Before you
reboot your system, you should try executing this fragment of the shell script a few times to make
sure you've got it right.
The other boot script organizational style found on UNIX systems is derived from the AT&T family
of UNIX. In this style, startup scripts are sorted into subdirectories with names like rc0.d, rc1.d, and
so on. Depending on the operating system, these directories may be located in /etc, /etc/rc.d, or /
sbin. Each subdirectory is named after a runlevel, which controls the level of service the system will
provide. For example, in runlevel 1 (corresponding to directory rc1.d) the system may provide single-
user services only, blocking all network logins, and in runlevel 3 (rc3.d) it may allow full network
login, filesharing, and a host of other multiuser services.
You will need to determine what runlevel your system commonly runs at. This can be done by
examining /etc/inittab for the initdefault entry, or by running the runlevel command if your system
provides one.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Forking Servers and the inetd Daemon 255

The next step is to enter the rc*.d directory that corresponds to this runlevel. You will see a host of
scripts with names that begin with either "S" or "K"— for example, S15nfs.server and K20lp. The
scripts that begin with "S" correspond to services that are started when the system enters that
runlevel; those that begin with "K" are services that the system kills when it leaves the runlevel. On
Solaris systems, S15nfs.server starts up NFS filesharing services, and K20lp shuts down line print-
ing. The numbers in the script name are used to control the order in which the startup scripts are
executed, since the boot system sorts the scripts alphabetically before invoking them.
Frequently, the scripts are just symbolic links to general-purpose shell scripts that can start and stop
the service. The real script is located in a directory named init.d, located variously in /etc/init.d, /etc/
rc.d/init.d, or /sbin/init.d. When the boot system wants to launch or kill the service, it follows the link
to its location, and then invokes the script with the arguments 'start' or 'stop'.
On systems with AT&T-style boot scripts, the strategy again is to see how another service already
does it, clone and rename its startup script, and then modify it to invoke your script at startup time.
Here is an extremely simple script that can be used on many systems to start and stop the psycho-
therapist daemon. Name it eliza, make it executable, and store it in /etc/init.d (or whatever is the
proper location for such scripts on your system). Then create a link from /etc/r3.d/S20eliza to this
script, again modifying the exact path as appropriate for your operating system.
#!/bin/sh
# psychotherapist startup script
case "$1" in
'start')
if [ -x /usr/local/bin/eliza_daemon.pl ]; then
echo -n "Starting psychotherapist: "
/usr/local/bin/eliza_daemon.pl
fi
;;
'stop')
if [ -e /var/tmp/eliza.pid ]; then
echo -n "Shutting down psychotherapist"
kill -TERM `cat /var/tmp/eliza.pid`
fi
;;
*)
echo "usage: $0 {start|stop}"
;;
esac

Again, it's a good idea to test this script from the command line before committing it to your boot
scripts directory.
One thing to watch for is that the boot scripts run as the superuser, so your network application also
runs with superuser privileges. This is generally an undesirable feature. Chapter 14 describes how
scripts started with superuser privileges can relinquish those privileges to become an ordinary user.
Alternatively, you can use the su command to launch the script using the privileges of an ordinary
user. In the two shell scripts mentioned, replace the calls to /usr/local/bin/eliza_daemon.pl with:
su nobody -c /usr/local/bin/eliza_daemon.pl

This will run the server under the nobody account.

Nemeth [1995] has an excellent discussion of the boot process on a variety of popular UNIX sys-
tems.
The inetd daemon, discussed at length later in this chapter in the section Using the inetd Super
Daemon, provides a convenient way to automatically launch servers that are used only occasionally.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Forking Servers and the inetd Daemon 256

Backgrounding on Windows and Macintosh Systems

Neither the Macintosh nor the Microsoft Windows have the same concept of background processes
as UNIX. This section explains how to achieve daemonlike behavior for long-running network ap-
plications on these platforms.
On the Macintosh, the best that you can currently do is to have a network script started automatically
at boot time by placing the Perl script file into the Startup Items folder in the System folder. At boot
time, MacPerl will be launched and run the script. However, as soon as you exit MacPerl, the server
will be terminated, along with any other Perl scripts that are running.
You can improve this situation esthetically by loading the Mac::Apps::Launch module within the
script and immediately calling the Hide() function, using "MacPerl" as the name of the application
to hide. This code fragment illustrates the idiom:
use Mac::Apps::Launch;
Hide(MacPerl => 1) or warn $^E;

(Under MacPerl, the $^E global returns Macintosh-specific error information.) To show the appli-
cation again, launch MacPerl, bringing it to the foreground.
Microsoft Windows offers a more generic way to turn applications into background daemons, using
its system of "services." Services are available only on Windows NT and 2000 systems. To do this,
you need two utilities: instsrv.exe and srvany.exe. These utilities are not part of the standard Win-
dows NT/2000 distributions, but are add-ons provided by the Windows NT/2000 Resource Kits.
There are two steps to the process. In the first step, you use instsrv.exe to define the name of the
new service. In the second step, you use the registry editor to associate the newly defined service
with the name and command-line arguments of the Perl script.
The first step is to define the new service using instsrv.exe. From the DOS command window, type
the following:
% C:\rkit\instsrv.exe PSYCHOTHERAPIST C:\rkit\srvany.exe

Replace C:\rkit with the path of the actual instsrv.exe and srvany.exe files, and PSYCHOTHERA-
PIST with the name that you wish to use when referring to the network service. The next step is to
edit the registry using the Registry Editor. The usual caveats and dire warnings apply to this process.
Launch regedt32.exe and locate the following key:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\PSYCHOTHERAPIST

Modify this as appropriate for the service name you selected earlier. Now you add a key named
Parameters, and two subkeys named Application and AppParameters. Application contains the path
to the Perl executable, and AppParameters contains the arguments passed to Perl, including the
script name and any script arguments.
Click on the PSYCHOTHERAPIST key and choose Add Key from the Edit menu. When prompted,
enter a key name of Parameters and leave the class field blank. Now select the newly created
Parameters key and invoke Add Value from the Edit menu. When prompted, enter a value name of
Application, a data type of REG_SZ (a null-terminated string), and a string containing the correct
path to the Perl executable, such as C:\Perl\bin\perl5.6.0.exe.
Select the Parameters key once again and invoke Add Value. This time enter a value name of
AppParameters, a data type of REG_SZ, and a value containing the complete path of the script and
any arguments you wish to pass to it, for example C:\scripts\eliza_server.pl.3

3Don't use the version of the server that autobackgrounds itself and dissociates from the session group, because these tricks are UNIX specific.
Use the forking server from Figure 10.3 with the interact () subroutine modified for Windows systems.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Forking Servers and the inetd Daemon 257

Close the Registry Editor. You should now be able to go to the Services control panel and set it to
start automatically at system startup time. From the list of NT/2000 services, select the psycho-
therapist server, and press the button labeled Startup. When prompted, change the startup type to
Automatic, and set the LogOnAs field to the name of the user you wish the server to run as. A
common choice is "System Acount." Also clear the checkbox labeled "Allow service to interact with
users."
The Services control panel allows you to manually start and stop the server. If you prefer, you can
use the DOS commands NET START PSYCHOTHERAPIST and NET STOP PSYCHOTHERA-
PIST to the same effect.

Using the inetd Super Daemon

Let's go back to Figure 10.3 and take a second look at the interact() function:
sub interact {
my $sock = shift;
STDIN->fdopen($sock,"<") or die "Can't reopen STDIN: $!";
STDOUT->fdopen($sock,">") or die "Can't reopen STDOUT: $!";
STDERR->fdopen($sock,">") or die "Can't reopen STDERR: $!";
$|=1;
my $bot = Chatbot::Eliza->new;
$bot->command_interface();
}

The psychotherapist daemon is pretty generic in its handling of incoming connections and forking.
In fact, interact() is the only place where application-specific code appears.
Now consider this version of interact():
sub interact {
my $sock = shift;
STDIN->fdopen($sock,"<") or die "Can't reopen STDIN: $!";
STDOUT->fdopen($sock,">") or die "Can't reopen STDOUT: $!";
STDERR->fdopen($sock,">") or die "Can't reopen STDERR: $!";
exec "eliza.pl";
}

After reopening STDIN, STDOUT, and STDERR onto the socket, we simply exec() the original com-
mand-line eliza.pl script from Figure 10.2. Assuming that eliza.pl is on the command path, Perl
launches it and replaces the current process with the new one. The command-line version of
eliza.pl runs, reading user input from STDIN and sending the psychotherapist's responses to
STDOUT. But STDIN, STDOUT, and STDERR are inherited from the parent process, so the program
is actually reading and writing to the socket. We've converted a command-line program into a server
application without changing a line of source code!
In fact, we can make this even more general by adding arguments to interact() that contain the
name and command-line arguments of a command to execute:
sub interact {
my ($sock,@command) = @>_;
STDIN->fdopen($sock,"<") or die "Can't reopen STDIN: $!";
STDOUT->fdopen($sock,">") or die "Can't reopen STDOUT: $!";
STDERR->fdopen($sock,">") or die "Can't reopen STDERR: $!";
exec @command;
}

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Forking Servers and the inetd Daemon 258

Now any program that reads from STDIN and writes to STDOUT can be run as a server. For example,
on UNIX systems, you could rig up a simple echo server just by passing /bin/cat as the argument
to interact(). Since cat reads from STDIN and writes a copy to STDOUT, it will echo everything
it reads from the socket back to the peer.
This simple way of creating network servers has not gone unnoticed by operating system designers.
UNIX (and Linux) systems have a standard daemon called inetd, which is little more than a config-
urable version of this generic server capable of launching and running a variety of network services
on demand.
inetd is launched at boot time. It reads a configuration file named /etc/inetd.conf, which is essentially
a list of ports to monitor and programs to associate with each port. When a connection comes in to
one of its monitored ports, inetd calls accept() to get a new connected socket, forks, remaps the
three standard filehandles to the socket, and finally launches the appropriate program.
The advantage of this system is that instead of launching a dozen occasionally used services man-
ually or at boot time, inetd launches them only when they are needed. Another nice feature of
inetd is that it can be reconfigured on the fly by sending it a HUP signal. When such a signal arrives,
it rereads its configuration file and reconfigures itself if needed. This allows you to add services
without rebooting the machine.
Unfortunately, inetd is not standard on Win32 or Macintosh machines. For Windows, you can get
inetd lookalikes at the following locations:
• Cygnus Win32 tools
ftp://go.cygnus.com/pub/ftp.cygnus.com/gnu-win32
http://www.cygnus.com/misc/gnu-win32/ [http://www.cygnus.com/misc/gnu-win32]
• Ockham Technology inetd for Windows NT (commercial)
http://www.ockham.be/inetd.html
Many years ago I used an inetd lookalike for the Macintosh, which used Apple Events to simulate
a true inetd daemon, but it no longer seems to be available on the Web.

Using inetd
With inetd we can turn the command-line psychotherapist program of Figure 10.2 into a server
without changing a line of code. Just add the following line to the bottom of the /etc/inetd.conf con-
figuration file:
12000 stream tcp nowait nobody /usr/local/bin/eliza.pl eliza.pl

You must have superuser access to edit this file. If there is no account named nobody, replace it
with your login name (or another of your choosing). Adjust the path to the eliza.pl script to reflect its
actual location (I suggest you use a version of the script that includes the _testquit() patch
described earlier). When you're done editing the file, restart the inetd daemon by sending it a HUP
signal. You can do this by finding its process ID (PID) using the ps command and then using the
kill command to send the signal. For example:
% ps aux | grep inetd
root 657 0.0 0.8 1220 552 ? S 07:07 0:00 inetd
lstein 914 0.0 0.5 948 352 pts/1 S 08:07 0:00 grep inetd
% kill -HUP 657

Two shortcuts work on many Linux systems:

% kill -HUP `cat /var/run/inetd.pid`
% killall -HUP inetd

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Forking Servers and the inetd Daemon 259

Now you can use either the standard telnet program or the gab3.pl client developed earlier in this
chapter to talk to the psychotherapist server.
Let's look at the inetd.conf entry in more detail. It's divided into seven fields delimited by whitespace
(tabs or spaces):
12000 —This is the service name or port number that the server will listen to. Be sure to check that
your system isn't already using a port before you take it (you can use the netstat program for this
purpose).
Some versions of inetd require you to use a symbolic service name in this field, such as eliza rather
than 12000. On such systems, you must manually edit the file /etc/services, add the name and port
number you desire, and then use that symbolic name in inetd.conf. For the psychotherapist daemon,
an appropriate /etc/services line would be:
eliza 12000/tcp

We would then use eliza instead of the port number as the first field in inetd.conf.
stream —This field specifies the server type, and can be either stream for connection-oriented
services that send and receive data as continuous streams of data, or dgram for services that send
and receive connectionless messages. Any program that reads STDIN and writes STDOUT is a
stream-based service, so we use stream here.
tcp —This specifies the communications protocol, and may be either tcp or udp (many systems
also support more esoteric protocols, but we won't discuss them here). Stream-based services use
tcp.
nowait —This tells inetd what to do after launching the server program. It can be wait, to tell
inetd to wait until the server is done before launching the program again to handle a new incoming
connection, or nowait, which allows inetd to launch the program multiple times to handle several
incoming connections at once. The most typical value for stream-based services is nowait, which
makes inetd act as a forking server. If multiple clients connect simultaneously, inetd launches a copy
of the program to deal with each one. Some versions of inetd allow you to put a ceiling on the number
of processes that can run simultaneously.
/usr/local/bin/eliza.pl—This is the full path to the program.
eliza.pl—The seventh and subsequent fields are command-line arguments to pass to the script.
This can be any number of space-delimited command-line arguments and switches. By convention,
the first argument is the name of the program. You can use the actual script name, as shown here,
or make up a different name. This value shows up in the script in the $0 variable. Other command-
line arguments appear in the @ARGV array in the usual manner.
The main "gotcha" with inetd-launched programs is that stdio buffering may cause the data to flow
unpredictably. For example, you might not see the psychotherapist's initial greeting until the program
has output a few more lines of text. This is solved by turning on autoflush, as we did in Figure 10.2.

Using inetd in wait Mode

Using inetd in nowait mode is not as efficient as writing your own forking server. This is because
inetd must launch your program each time it forks, and the Perl interpreter can take a second or two
to launch, parse your script, and load and compile all the modules you require. In contrast, a forking
server has already been through the parsing and compiling phases; therefore, the overhead of
forking to handle a new connection is much less significant.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Forking Servers and the inetd Daemon 260

A nice compromise between convenience and performance is to use inetd in wait mode. In this
mode it launches your server when the first incoming connection arrives, and waits for the server
to finish. Your server will do everything an ordinary server does, including forking to handle new
connections. The only difference is that it will not create the listening socket itself, but inherit it from
one created by inetd. Since inetd duplicates the socket onto the three standard filehandles, you can
recover it from any one of them, typically STDIN.
inetd thus nicely relieves you of the responsibility of launching the server by hand without incurring
a performance penalty. In addition, you can write the server to exit under certain conditions—for
example, if it has been idle for a certain number of minutes, or after servicing a set number of
connections. After it exits, inetd will relaunch it when it is next needed. This means that you need
not keep an occasionally used server running all the time.
A new version of the psychotherapist server designed to be run from inetd in wait mode is given in
Figure 10.8. The corresponding entry in /etc/inetd.conf is almost identical to the original, except that
it uses wait in the fourth field and has a different script name in the sixth field:

Licensed by
Stjepan Maric
4218908

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Forking Servers and the inetd Daemon 261

Figure 10.8. inetd psychotherapist in wait mode

12000 stream tcp wait nobody /usr/local/bin/eliza_inetd.pl eliza_inetd.pl

In addition to inheriting its listening socket from inetd, this server differs from previous versions in
having a one-minute timeout on the call to accept(). If no new connections arrive within the timeout
period, the parent process exits. inetd will relaunch the server again if needed. The changes required
to the basic forking server are small.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Forking Servers and the inetd Daemon 262

Lines 1–7: Define timeout values—We recover the timeout from the command line, or default to
one minute if no value is supplied. Notice that we no longer read the port number from the
command line; it is supplied implicitly by inetd.
Lines 10–13: Recover the listening socket—We recover the listening socket from STDIN. First
we check that we are indeed running under inetd by testing that STDIN is a socket using the -S
file test. If STDIN passes this test, we turn it back into an IO::Socket object by calling IO::Socket's
new_from_fd() method. This method, inherited from IO::Handle, is similar to fdopen() ex-
cept that instead of reopening an existing handle on the specified filehandle, it creates a new
handle that is a copy of the old one. In this case, we create a new IO::Socket object that is a
copy of STDIN, opened for reading and writing with the "+<" mode.
Lines 15–21: Call accept() with a timeout—We now enter a standard accept() loop, except
that the call to accept() is wrapped in an eval{} block. Within the eval, we create a local
ALRM signal handler that calls die(), and use alarm() to set a timer that will go off after
$timeout minutes have expired. We then call the listen socket's accept() method. If an in-
coming connection is received before the timeout expires, then the result from the eval{} block
is the connected socket. Otherwise, the ALARM signal handler is called, and the eval{} block
is aborted, returning an undefined value. In the latter case, we call exit(), terminating the
whole server. Otherwise, we call alarm(0) to cancel the timeout.
Lines 22–43: The remainder of the server is unchanged. We also include the Chat-
bot::Eliza::_testquit() workaround in order to avoid problems when the user closes
the connection unexpectedly.
When I first wrote this program, I thought that I could simply use IO::Socket's built-in timeout mech-
anism, rather than roll my own ALRM-based timeout. However, there turned out to be a problem.
With the built-in timeout activated, accept() returned undef both when the legitimate timeout
occurred and when it was interrupted by the CHLD signal that accompanies every child process's
termination. After some trial and error, I decided there was no easy way to distinguish between the
two events, and went with the technique shown here.
inetd can also be used to launch UDP applications. In this case, when the program is launched, it
finds STDIN already opened on an appropriate UDP socket. recv() and send() can then be used
to communicate across the socket in the normal way. See Chapters 18 and 19 for more details.

Summary
Concurrent I/O is essential for connection-oriented servers to service multiple clients. Concurrency
can also be useful in client code to avoid deadlock situations.
This chapter introduced forking, the most common technique for achieving concurrency. Forking is
generally easy to use, but it has a few things to watch for, the most important being the need to wait
on exited child processes. It is also common for production servers to detach themselves from the
controlling terminal and autobackground themselves. In this chapter we developed a become_dae-
mon() subroutine to do this.
On UNIX systems, the inetd superdaemon provides a simple way to turn ordinary command-line
applications into forking servers. You can also use it as a handy way to launch a conventional forking
server when needed, thus avoiding having to start the daemon manually.
The next chapters look at other techniques for handling concurrent connections, beginning with
multithreading and continuing with multiplexing.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 263

Chapter 11. Multithreaded Applications

This chapter discusses network application development using Perl's lightweight thread API.
Threads provide a program architecture that in many ways is easier to use than multiprocessing.

About Threads
Multithreading is quite different from multiprocessing. Instead of there being two or more processes,
each with its own memory space, signal handlers, and global variables, multithreaded programs
have a single process in which run several "threads of execution." Each thread runs independently;
it can loop or perform I/O without worrying about the other threads that are running. However, all
threads share global variables, filehandles, signal handlers, and other resources.
While this sharing of resources enables threads to interact in a much more intimate way than the
separate processes created by fork(), it creates the possibility of resource contention. For ex-
ample, if two threads try to modify a variable at the same time, the result may not be what you expect.
For this reason, resource locking and control becomes an issue in threaded programs. Although
multithreaded programming simplifies your programming in some ways, it complicates it in others.
The Thread module was introduced in Perl 5.005. To use it, you must run an operating system that
supports threads (including most versions of UNIX and Microsoft Windows) and have compiled Perl
with threading enabled. In Perl 5.005, you do this by running the Configure installation program with
the option -Dusethreads. With Perl 5.6.0 and higher, the option becomes -
Duse5005threads. No precompiled Perl binaries come with threading support activated.

Threads Are Experimental

Perl threads are an experimental feature. The 5.005 thread implementation has known bugs that
can lead to mysterious crashes, particularly when running on machines with more than one CPU.
Not all Perl modules are thread-safe; that is, using these modules in a multithreaded program will
lead to crashes and/or incorrect results, and even some core Perl features are problematic. Although
the thread implementation has improved in Perl 5.6 and higher, some fundamental design flaws
remain in the system. In fact, the Perl thread documentation warns that multithreading should not
be used in production systems.
The Perl developers are developing a completely new threading design that will be known as inter-
preter threads (ithreads) that will be part of Perl version 6, expected to be available in the summer
of 2001. It promises to be more stable than the 5.005 implementation, but its API may be different
from what is described here.

The Thread API

The thread API described here is the 5.005 version of threads, and not the interpreter threads cur-
rently under development.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Multithreaded Applications 264

The API threads, which is described in the Thread, Thread::Queue, Thread:: Semaphore, and
attrs manual pages, seems simple but hides many complexities. Each program starts with a single
thread, called the main thread. The main thread starts at the beginning of the program and runs to
the end (or until exit() or die() is called).
To create a new thread, you call Thread->new(), passing it a reference to a subroutine to execute
and an optional set of arguments. This creates a new concurrent thread, which immediately exe-
cutes the indicated subroutine. When the subroutine is finished, the thread exits. For example, here's
how you might launch a new thread to perform a time-consuming calculation:
my $thread = Thread->new(\&calculate_pi, precision => 190);

The new thread executes calculate_pi(), passing it the two arguments " precision " and "
190." If successful, the call immediately returns with a new Thread object, which the calling thread
usually stashes somewhere. The Thread object can now call detach(), which frees the main
thread from any responsibility for dealing with it.
Alternatively, the thread can remain in its default attached state, in which case the main thread (or
any other thread) should at some point call the Thread object's join() method to retrieve the
subroutine's return value. This is sometimes done just before exiting the program, or at the time the
return value is needed. If the thread has not yet finished, join() blocks until it does. To continue
with the previous example, at some point the main thread may wish to retrieve the value of pi com-
puted by the calculate_pi() subroutine. It can do this by calling:
my $pi = $thread->join;

Unlike the case with parent and children processes where only a parent can wait() on its children,
there is no strict familial relationship between threads. Any thread can call join() on any other
thread (but a thread cannot join() itself).
For a thread to exit, it need only return() from its subroutine, or just let control fall naturally through
to the bottom of the subroutine block. Threads should never call Perl's exit() function, because
that would kill both the current thread and all other threads (usually not the intended effect!). Nor
should any thread other than the main one try to install a signal handler. There's no way to ensure
that a signal will be delivered to the thread you intend to receive it, and it's more than likely that Perl
will crash.
A thread can also exit abnormally by calling die() with an error message. However, the effect of
dying in a thread is not what you would expect. Instead of raising some sort of exception immediately,
the effect of die() is postponed until the main thread tries to join() the thread that died. At that
point, the die() takes effect, and the program terminates. If a non-main thread calls join() on a
thread that has died, the effect is postponed until that thread itself is joined.
You can catch this type of postponed death and handle using eval(). The error message passed
to die() will be available in the $@ global.
my $pi = eval {$thread->join} || warn "Got an error: $@";

A Simple Multithreaded Application

Here's a very simple multithreaded application. It spawns two new threads, each of which runs the
hello() subroutine. hello() loops a number of times, printing out a message specified by the
caller. The subroutine sleeps for a second each time through the loop (this is just for illustration
purposes and is not needed to obtain thread concurrency). After spawning the two threads, the main
thread waits for the two threads to terminate by calling join().

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Multithreaded Applications 265

#!/usr/bin/perl
use Thread;
my $thread1 = Thread->new(\&hello, "I am thread 1",3);
my $thread2 = Thread->new(\&hello, "I am thread 2",6);
$_->join foreach ($thread1,$thread2);

sub hello {
my ($message,$loop) =@_;
for (1..$loop) { print $message,"\n"; sleep 1; }
}

When you run this program, you'll see output like this:
% perl hello.pl
I am thread 1
I am thread 2
I am thread 1
I am thread 2
I am thread 1
I am thread 2
I am thread 2
I am thread 2
I am thread 2

Locking
The problem with threads appears as soon as two threads attempt to modify the same variable
simultaneously. To illustrate the problem, consider this deceptively simple bit of code:
my $bytes_sent = 0;
my $socket = IO::Socket->new(....);

sub send_data {
my $data = shift
my $bytes = $socket->syswrite($data);
$bytes_sent += $bytes;
}

The problem occurs in the last line of the subroutine, where the $bytes_sent variable is incre-
mented. If there are multiple simultaneous connections running, then the following scenario can
occur:
1. Thread 1 fetches the value of $bytes_sent and prepares to increment it.
2. A context switch occurs. Thread 1 is suspended and thread 2 takes control. It fetches the value
of $bytes_sent and increments it.
3. A context switch again occurs, suspending thread 2 and resuming thread 1. However, thread
1 is still holding the value of $bytes_sent it fetched from step 1. It increments the original
value and stores it back into $bytes_sent, overwriting the changes made by thread 2.
This chain of events won't happen every time but will happen in a rare, nondeterministic fashion,
leading to obscure bugs that are hard to track down.
The fix for this is to use the lock() call to lock the $bytes_sent variable before trying to use it.
With this small modification, the example now works properly:
my $bytes_sent = 0;
my $socket = IO::Socket->new(....);

sub send_data {
my $data = shift
my $bytes = $socket->syswrite($data);
lock($bytes_sent);
$bytes_sent += $bytes;
}

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Multithreaded Applications 266

lock() creates an "advisory" lock on a variable. An advisory lock prevents another thread from
calling lock() to lock the variable until the thread that currently holds the lock has relinquished it.
However, the lock doesn't prevent access to the variable, which can still be read and written even
if the thread doesn't hold a lock on it. Locks are generally used to prevent two threads from trying
to update the same variable at the same time.
If a variable is locked and another thread tries to lock it, that thread is suspended until such time as
the lock is available. A lock remains in force until the lock goes out of scope, just like a local variable.
In the preceding example, $bytes_sent is locked just before it's incremented, and the lock remains
in force throughout the scope of the subroutine.
If a number of variables are changed at the same time, it is common to create an independent
variable that does nothing but manage access to the variables. In the following example, the
$ok_to_update variable serves as the lock for two related variables, $bytes_sent and
$bytes_left:
my $ok_to_update;
sub send_data {
my $data = shift
my $bytes = $socket->syswrite($data);
lock($ok_to_update);
$bytes_sent += $bytes;
$bytes_left -= $bytes;
}

It is also possible to lock an entire subroutine using the notation lock(\&subroutine). When a
subroutine is locked, only one thread is allowed to run it at one time. This is recommended only for
subroutines that execute very quickly; otherwise, the multiple threads serialize on the subroutine
like cars backed up at a traffic light, obliterating most of the advantages of threads in the first place.
Variables that are not shared, such as the local variables $data and $bytes in the preceding
example, do not need to be locked. Nor do you need to lock object references, unless two or more
threads share the object.
When using threads in combination with Perl objects, object methods often need to lock the object
before changing it. Otherwise, two threads could try to modify the object simultaneously, leading to
chaos. This object method, for example, is not thread safe, because two threads might try to modify
the $self object simultaneously:
sub acknowledge { # NOT thread safe
my $self = shift;
print $self->{socket} "200 OK\n";
$self->{acknowledged}++;
}

You can lock objects within object methods explicitly, as in the previous example:
sub acknowledge { # thread safe
my $self = shift;
lock($self);
print $self->{socket} "200 OK\n";
$self->{acknowledged}++;
}

Since $self is a reference, you might wonder whether the call to lock() is locking the $self
reference or the thing that $self points to. The answer is that lock() automatically follows ref-
erences up one level (and one level only). The call to lock($self) is exactly equivalent to calling
lock(%$self), assuming that $self is a hash reference.
Threading versions of Perl provide a new syntax for adding attributes to subroutines. With this syn-
tax, the subroutine name is followed by a colon and a set of attributes:

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Multithreaded Applications 267

sub acknowledge: locked method { # thread safe

my $self = shift;
print $self->{socket} "200 OK\n";
$self->{acknowledged}++;
}

To create a locked method, use the attributes locked and method. If both attributes are present, as
in the preceding example, then the first argument to the subroutine (the object reference) is locked
on entry into the method and released on exit. If only locked is specified, then Perl locks the sub-
routine itself, as if you had specifically written lock(\&acknowledge). The key difference here is
that when the attributes are set to locked method, it's possible for multiple threads to run the sub-
routine simultaneously so long as they're working with different objects. When a subroutine is
marked locked only, then only one thread can gain access to the subroutine at a time, even if they're
working with different objects.

Thread Module Functions and Methods

The thread API has several other core parts, including ways for threads to signal each other when
a particular condition has become true. Here is a very brief synopsis of the thread API. More infor-
mation is available in the perlthread manual page, and other features are explained in depth later
when we use them.

$thread = Thread->new(\&subroutine [, @arguments]);

Creates a new thread of execution and returns a Thread object. The new thread immediately runs the sub-
routine given as the first argument, passing it the arguments listed in the optional second and subsequent
arguments.
$return_value = $thread->join()
join() waits for the given thread to terminate. The return value is the result (if any) returned by the subroutine
specified when the thread was created. If the thread is running, then join() blocks until it terminates there
is no way to do a nonblocking join on a particular thread.
$thread->detach()
If you aren't interested in a thread's return value, you can call its detach() method. This makes it impossible
to call join() later. The main advantage of detaching a thread is that it frees the main thread from the re-
sponsibility of joining the other threads later.
@threads = Thread->list()
This class method returns a list of Thread objects. The list includes those that are running as well as those that
have terminated but are waiting to be joined.
$thread = Thread->self()
This class method returns the Thread object corresponding to the current thread.
$tid = $thread->tid()
Each thread is associated with a numeric identifier known as the thread ID (tid). There's no particular use for
this identifier except perhaps as an index into an array or to incorporate into debugging messages. This tid can
be retrieved with the tid() method.
lock($variable)
The lock() function locks the scalar, array, or hash passed to it in such a way that no other thread can lock
the variable until the first thread's lock goes out of scope. For container variables, such as arrays, locking the
whole array (e.g., with lock(@connections)) is different from locking a component of the array (e.g.,
lock($connections[3]))).

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Multithreaded Applications 268

You do not need to explicitly import the Thread module to use lock(). It is built into the core of all
versions of Perl that support multithreading. On versions of Perl that don't support multithreading,
lock() has no effect. This allows you to write thread-safe modules that will work equally well on
threading and nonthreading versions of Perl.
The next five items are functions that must be imported explicitly from the Thread module:
use Thread qw(async yield cond_wait cond_signal cond_broadcast);

$thread = async {BLOCK}

The async() function is an alternative way to create a new Thread object. Instead of taking a code reference
and its arguments like new(), it accepts a code block, which becomes the body of the new thread. The Thread
object returned by async() can be join()ed, just like a thread created with new().
yield()
The yield() function is a way for a thread to hint to Perl that a particular spot might be a good place to do a
thread context shift. Because of the differences in thread implementations on different operating systems, this
may or may not have an effect. It is not usually necessary to call yield() to obtain concurrency, but it might
help in some circumstances to distribute the time slices of execution more equitably among the threads.
cond_wait($variable)
cond_wait() waits on a variable it is signaled. The function takes a locked variable, releases the lock, and
puts the thread to sleep until the variable is signalled by another thread calling cond_signal() or
cond_broadcast(). The variable is relocked before cond_wait() returns.
cond_signal($variable)
cond_signal() signals $variable, restarting any threads that are waiting on it. If no threads are waiting,
then the call does nothing. If multiple threads are waiting on the variable, one (and only one) of them is un-
blocked. Which one is awakened is indeterminate.
cond_broadcast($variable)
cond_broadcast() works like cond_signal(), except that all waiting threads are awakened. Each thread
reacquires the lock in turn and executes the code following the cond_wait(). The order in which the waiting
threads are awakened is indeterminate.

We will use cond_wait() and cond_broadcast() in Chapter 14, when we develop an adaptive
prethreaded server.

Threads and Signals

If you plan to mix threads with signals, you must be aware that the integration of signal handling
with threads is one of the more experimental parts of Perl's experimental threads implementation.
The issue is that signals arrive at unpredictable times and may be delivered to any currently exe-
cuting thread, leading to unpredictable results.
The Thread::Signal module is supposed to help with this by arranging for all signals to be delivered
to a special thread that runs in parallel with the main thread. You don't have to do anything special.
Just loading the module is sufficient to start the signal thread running:
use Thread::Signal;

However, you should be aware that Thread::Signal changes the semantics of signals so that they
can no longer be used to interrupt long-running system calls. Hence, this trick will no longer work:

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Multithreaded Applications 269

alarm (10);
my $bytes =
eval {
local $SIG{ALARM} = sub { die };
sysread($socket,$data,1024);
};

In some cases, you can work around this limitation by replacing the eval{} section with a call to
select(). We use this trick in Chapter 15.
In practice, Thread::Signal sometimes seems to make programs less stable rather than more so,
depending on which version of Perl and which threading libraries you are using. My advice for
experimenting with threading features is to first write the program without Thread::Signal and add
it later if unexpected crashes or other odd behavior occurs.

A Multithreaded Psychiatrist Server

Despite the long introduction to threads, an actual multithreaded server is quite short. Here we
develop a multithreaded version of the psychiatrist server (Figure 11.1).

Figure 11.1. Multithreaded psychiatrist server

Lines 1–5: Load modules—We begin by loading IO::Socket and the Thread module. We also
bring in a specialized version of Chatbot::Eliza in which the command_interface() method
has been rewritten to work well in a multithreaded environment (Figure 11.2).

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Multithreaded Applications 270

Licensed by
Stjepan Maric
4218908
Figure 11.2. The Chatbot::Eliza::Server class
Lines 6–12: Create listening socket—As in the previous examples, we create a new listening
socket with IO::Socket::INET->new(). If a listening socket can't be created, we die with
the error message IO::Socket leaves in $.
Lines 12–15: Accept loop—We now enter the server's main loop. Each time through the loop
we call accept(), yielding a new socket connected to the incoming client. We launch a new
thread to handle the connection by calling Thread->new() with a reference to the inter-
act() subroutine and the connected socket as its single argument. We then go back to waiting
on accept().
Notice that there is no need to close the listen or accept socket, as we did in the forking server
examples. This is because duplicate socket handles are never created.
Lines 16–31: The interact() subroutine—This subroutine handles the conversation with the
user and runs in a separate thread. Since the main server never checks the return value of the
connection threads it launches, there's no need to keep this status information; so we begin by
detaching ourselves from the main thread.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Multithreaded Applications 271

We next create a new Chatbot::Eliza::Server object and invoke its command_interface()

method. Unlike the previous examples, this subclass does not read and write to STDIN and
STDOUT but to a pair of filehandles passed in the argument list. The first argument is a filehandle
to read the user's remarks from, and the second is a filehandle to write the psychotherapist's
responses to. In this case, the two filehandles are both the connected socket. We let com-
mand_interface() do its thing, then close the socket.The thread terminates when the sub-
routine ends.

The Chatbot::Eliza::Server Class

The reason for subclassing Chatbot::Eliza is simple. The class's command_interface() method
is hardwired to use STDIN and STDOUT. In the forking server examples, we were able to trick Chat-
bot::Eliza into reading and writing to the connected socket by reopening standard input and output
on the socket filehandle. This worked because each child process had its own copies of STDIN and
STDOUT and we could alter them without affecting other children that might be running concurrently.
In the multithreaded example, however, this trick won't work because there is only a single copy of
the STDIN and STDOUT filehandles shared among all the threads. Therefore, command_inter-
face() must be rewritten to read and write to filehandles that are passed to it at runtime.
Figure 11.2 shows the code necessary to achieve this. The module inherits from Chatbot::Eliza via
the @ISA array. It then redefines the command_interface() method.
To create command_interface(), I simply duplicated the original Chatbot::Eliza code and added
the new filehandle argument to all print and read statements (I also removed some extraneous
debugging code). To remain compatible with the original version of the module, the filehandles used
for reading and writing will default to STDIN and STDOUT, if not otherwise specified.

A Multithreaded Client
To go along with the multithreaded psychiatrist server, this section develops a multithreaded client
named gab4.pl (Figure 11.3). It is similar to the byte stream-oriented forking client gab3.pl in Chapter
10 (Figure 10.4); but instead of forking a child process to read from the remote server, the read loop
is done inside a thread running the do_read() subroutine.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Multithreaded Applications 272

Figure 11.3. Threaded concurrent client

The other major difference between this client and the previous version is the termination process.
In both clients, when do_write() detects that standard input has been closed, the subroutine
closes the transmission half of the socket by calling shutdown(1). This sends an end of file to the
server, causing it to close its side of the socket, and this event propagates back to the
do_read() thread.
So far so good, but what happens when the server is the one to initiate the disconnection? The
do_read() thread detects the end-of-file condition and exits. However, the do_write() loop
running in the main thread is usually blocked waiting for data from standard input and will not be
notified of anything untoward until it tries to write a line of text to the socket and triggers a PIPE
signal. In the forking client, we finessed this by having the CHLD handler call exit(). In the thread-
ing example, there is no CHLD signal to catch, and so the easiest course of action is just to have
the host_to_user() thread call exit().

Summary
Multithreading provides an elegant way to achieve concurrency in connection-oriented network ap-
plications. It is unfortunate that the Perl implementation is not entirely reliable and that the API is
subject to change.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Multithreaded Applications 273

Perl 6 is expected to provide robust multithreading, and although the API will likely be different in
detail from what is presented here, the basic concepts of thread creation, destruction, and locking
will not change.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 274

Chapter 12. Multiplexed Applications

The forking and threading techniques discussed in the last two chapters allow a program to handle
multiple concurrent connections. The last general technique that we cover is I/O multiplexing. Mul-
tiplexing doesn't take advantage of any operating system tricks to achieve the illusion of concur-
rency. Instead, multiplexed applications handle all connections in one main loop. For example, a
server that is currently servicing ten clients reads from each connected socket in turn, handles the
request, and then services the next client.
The big problem with interleaving I/O in this way is the risk of blocking. If you try to read from a
socket that doesn't have data ready, the read() and sysread() calls will block until new data is
received. When you're serving multiple connections, this is unacceptable because it causes all the
connections to stall until the connection you're waiting on becomes ready. Another potential problem
is that if the client on the other end isn't ready to read, then calls to syswrite() or print() will
also block. The performance of the server is held hostage to the performance of the slowest client.
The key to multiplexing is a built-in function called select() and its object-oriented equivalent, the
IO::Select module. With select() you can check whether an I/O operation on a filehandle will
block before performing the operation. This chapter discusses how to use these facilities.

A Multiplexed Client
Before addressing the details of how select() works, let's rewrite our "gab" client to use multi-
plexing. gab5.pl, like its previous incarnations, accepts lines from standard input, transmits them to
a remote server, and then relays the response from the server to standard output. Figure 12.1 shows
the code.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Multiplexed Applications 275

Figure 12.1. A multiplexed client

Lines 1–9: Load modules and process command-line arguments—We turn on strict type check-
ing and load the IO::Select and IO::Socket modules. We read the host and port to connect to
from the command line, or if a host is not specified, we assume the echo service on the local host.
Line 10: Create a new connected socket—We create a socket connected to the specified peer
by calling IO::Socket::INET->new() with the one-argument shortcut form.
Lines 11–13: Create a new IO::Select set—We will multiplex our reads on standard input and
on the socket. This means that we will read from standard input only when the user has some
data for us and read from the socket only when there's server data to be read.
To do this, we create a new IO::Select object by calling IO::Select->new(). An IO::Select
object holds one or more filehandles that can be monitored for their readiness to do I/O. After
creating the select object, we add STDIN and the socket by calling the select object's add()
method.
Lines 14–17: Main I/O loop—We now enter a while() loop. Each time through, we call the
select object's can_read() method to return the list of handles ready for reading. This list may
contain the socket handle, STDIN, or both. Our task is to loop through the list of ready handles
and take the appropriate action for each. If STDIN is ready for reading, we copy data from it to
the socket. If the socket is ready, we copy data from it to STDOUT.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Multiplexed Applications 276

Lines 18–24: Handle data on STDIN—If STDIN is ready to be read, we use sysread() to read
up to 2K bytes of data into a string variable named $buffer. If sysread() returns a positive
value, we write a copy of what we received to the socket. Otherwise, we have encountered an
end of file on standard input. We shutdown() the write half of the socket, sending the remote
server an end of file.
Lines 25–32: Handle data on the socket—If there is data to be read from the connected socket,
then we call sysread() on the socket to read up to 2K bytes. If the read is successful, we
immediately print it to STDOUT. Otherwise, the remote host has closed the connection, so we
write a message to that effect and exit.
You can use gab5.pl to talk to a variety of network servers, including those that are line oriented
and those that produce less predictable output. Because this script doesn't rely on either forking or
threading, it runs on practically all operating systems where Perl is available, including the Macin-
tosh.

The IO::Select Module

Perl versions 5.003 and higher come with an object-oriented wrapper class called IO::Select. You
create an IO::Select object, add to it the handles you wish to monitor, and then call its can_read(),
can_write(), or has_exceptions() methods to wait for one or more of the object's handles to
become ready for I/O.

$select = IO::Select->new([@handles])
The IO::Select new() class method creates a new IO::Select object. It can be called with a list of handles, in
which case they will be added to the set that IO::Select monitors, or it can be called with an empty argument
list, in which case the monitor set will be initially empty.

The handle list can be composed of any type of filehandle including IO::Handle objects, globs, and
glob references. You may also add and remove handles after the object is created.

$select->add(@handles)
This adds the list of handles to the monitored set and returns the number of unique handles successfully added.
If you try to add the same filehandle multiple times, the redundant entries are ignored.
$select->remove(@handles)
This removes the list of filehandles from the monitored set. IO::Select indexes its handles by file number, so
you can refer to the handle one way when you add it (e.g., STDOUT) and another way when you remove it (e.g.,
\*STDOUT).
$value = $select->exists($handle)
$count = $select->count
These are utility routines. The exists() method returns a true value if the handle is currently a member of
the monitored set. The count() method returns the number of handles contained in the IO::Select set.

The can_read(), can_write(), and has_exceptions() methods monitor the handle list for
changes in status.

@readable = $select->can_read([$timeout])
@writable = $select->can_write([$timeout])
@exceptional = $select->has_exception([$timeout])

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Multiplexed Applications 277

The can_read(), can_write(), and has_exception() methods each call select() on your behalf,
returning an array of filehandles that are ready for reading or writing or have exceptional conditions pending.
The call blocks until one of the handles in the IO::Select object becomes ready, or until the optional timeout
given by $timeout is reached. In the latter case, the call returns an empty list. The timeout is expressed in
seconds, and can be fractional.
Any of these methods can return an empty list if the process is interrupted by a signal. Therefore, always check
the returned list for filehandles, even if you have provided no timeout.

If you wish to select for readers and writers simultaneously, use the select() method.

($rout,$wout,$eout) = IO::Select->select($readers, $writers, $except [,$timeout])

select() is a class method that waits on multiple IO::Select sets simultaneously. $readers, $writers,
and $except are IO::Select objects or undef, and $timeout is an optional time-out period in seconds. When
any handle in any of the sets becomes ready for I/O, select() returns a three-element list, each element an
array reference containing the handles that are ready for reading or writing, or that have exceptional conditions.
If select() times out or is interrupted by a signal before any handles become ready, the method returns an
empty list.

Exceptional conditions on sockets are not what this term may imply. An exceptional condition occurs
when a TCP socket receives urgent data (we talk about how to generate and handle urgent data in
Chapter 17). An I/O error on a socket does not generate an exceptional condition, but instead makes
the socket both readable and writable. The nature of the error can then be detected by performing
a read or write on the socket and checking the $! variable.
IO::Select->select() can be used to put the current process to sleep for a fractional period
of seconds. Simply call the method using undef for all three IO::Select sets and the number of
seconds you wish to sleep. This code fragment causes the program to pause for 0.25 seconds:
@dis3:IO::Select->select(undef,undef,undef,0.25);

As with sleep(), select() returns prematurely if it is interrupted by a signal. Also don't count on
getting a pause of exactly 250 milliseconds, because select() is limited by the underlying reso-
lution of the system clock, which might not provide millisecond resolution. To get a version of sleep
that has microsecond resolution, use the Time::HiRes module, available from CPAN.

The Built-in select() Function

The built-in select() function is the Perl primitive that IO::Select uses internally. It is called with
four arguments, like this:
$nready = select($readers,$writers,$exceptions,$timeout);

Sadly, select()'s argument-passing scheme is archaic and unPerlish, in volving complex manip-
ulation of bit vectors. You may see it in older scripts, but IO::Select is both easier to use and less
prone to error. However, you might want to use select() to achieve a fractional sleep without
importing IO::Select:
select(undef,undef,undef,0.25);

Don't confuse the four-argument version of select() with the one-argument version discussed in
Chapter 1. The latter is used to select the default filehandle used with print().
The perlfunc manual pages give full details on the built-in select() function.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Multiplexed Applications 278

When Is a Filehandle Ready for I/O?

To use select() to its best advantage, it's important to understand the rules for a handle's read-
iness for I/O. Some of the rules apply equally to ordinary filehandles, pipes, and sockets; others are
specific to sockets. Filehandles, pipes, and sockets are all ready for reading when:
1. The filehandle has pending data. If there is at least 1 byte of data in the file-handle's input
buffer (the receive buffer in the case of sockets), then select() indicates that the filehandle
is ready for reading. sysread() on the filehandle will not block and returns the number of
bytes read.
For sockets, this rule can be modified by setting the value of the receive buffer's "low water mark"
as described in the next section.
2. There is an end of file on the filehandle. select() indicates that a filehandle is ready for
reading if the next read returns an end of file. The next call to sysread() will return numeric
0 without blocking. This occurs in normal filehandles when the end of the file is reached, and
in TCP sockets when the remote host closes the connection.
3. There is a pending error on the filehandle. Any I/O error on the filehandle also causes
select() to indicate that it is ready for reading. sysread() returns undef and sets $! to
the error code.
In addition, select() indicates that a socket is ready for reading when:
1. There is an incoming connection on a listen socket. select() indicates that a listen socket
is ready for reading if there is a pending incoming connection. accept() returns the con-
nected socket without blocking.
2. There is an incoming datagram on a UDP socket. A socket is ready for reading if it uses the
UDP protocol and there is an incoming datagram waiting to be read. The recv() function
returns the datagram without blocking. We discuss UDP in Chapter 18.
Filehandles, pipes, and sockets are all ready for writing when:
1. There is room in the output buffer for new data. If there is at least 1 byte of free space in the
filehandle's output buffer (the send buffer in the case of sockets), then syswrite() of a single
byte will succeed without blocking. Calling syswrite() with more than 1 byte may still block,
however, if the amount of data to write exceeds the amount of free space. On sockets, this
behavior can be adjusted by setting the send buffer's low water mark.
If the filehandle is marked as nonblocking, then syswrite() always succeeds without blocking
and returns the number of bytes actually written. We discuss nonblocking I/O in Chapter 13.
2. There is a pending error on the filehandle. If there is an I/O error on the filehandle, then
select() indicates that it is ready for writing. syswrite() will not block, but returns
undef as its function result. $! contains the error code.
In addition, sockets are ready for writing when:
1. The peer has closed its side of the connection. If the socket is connected and the remote end
has closed or shutdown() its end of the connection, then select() indicates that the socket
is ready for writing. The next syswrite() attempt will generate a PIPE exception. If you
ignore or handle PIPE signals, then syswrite() returns undef and $! is equal to EPIPE.
Local pipes also behave in this way.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Multiplexed Applications 279

2. A nonblocking connect has been initiated, and the attempt completes. When a TCP socket is
nonblocking and you attempt to connect it, the call to connect() returns immediately and
the connection attempt continues in the background. When the connect attempt eventually
completes (either successfully or with an error), select() indicates that the socket is ready
for writing. This is discussed in more detail in Chapter 13.
Exceptional conditions apply only to sockets. There is only one common exception, which occurs
when a connected TCP socket has urgent data to be read. We discuss how urgent data works in
Chapter 17.

Combining select() with Standard I/O

When using select() to multiplex I/O, you must be extremely careful not to mix it with functions
that use stdio buffering. The reason is that the stdio library maintains its own I/O buffers independent
of the buffers used by the operating system. select() doesn't know about these buffers; as a
result it may indicate that there's no more data to be read from a filehandle, when in fact there is
data remaining in the buffer. This will lead to confusion and frustration.
In multiplexed programs you should avoid the <> operator, print(), and read(), as well as the
built-in getline() function and the IO::Handle->getline() method. Instead, you should use
the low-level sysread() and syswrite() calls exclusively. This makes it difficult to do line-ori-
ented I/O. However, Chapter 13 develops some modules that overcome this limitation.

Adjusting the Low Water Marks

On some versions of UNIX, you can adjust when a socket is ready for reading or writing by changing
the low water marks on the socket's send and/or receive buffers. The receive buffer's low water
mark indicates the amount of data that must be available before select() will indicate that it is
ready for reading and sysread() can be called without blocking. The low water mark is 1 byte by
default, but you can change this by calling setsockopt() with the SO_RECVLOWAT option. You
might want to adjust this if you are expecting transmissions of a fixed size from the peer and don't
want to bother processing partial messages.
When writing to a socket, the low water mark indicates how many bytes of data can be written to
the socket with syswrite() without blocking. The default is 1 byte, but it can be changed by calling
setsockopt() with the SO_SNDLOWAT option.
Changing a socket's low water marks is nonportable. It works on most versions of UNIX but fails on
Windows and Macintosh machines. In addition, the Linux kernel (up through version 2.4) allows the
receive buffer's low water mark to be set, but not the send buffer's.

A Multiplexed Psychiatrist Server

In this section we develop a version of the Chatbot::Eliza server that uses multiplexing. It illustrates
how a typical multiplexed server works. The basic strategy for a multiplexed server follows this
general outline:
1. Create a listen socket.
2. Create an IO::Select set and add the listen socket to the list of sockets to be monitored for
reading.
3. Enter a select() loop.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Multiplexed Applications 280

4. When the select() loop returns, examine the list of sockets ready for reading. If the listen
socket is among them, call accept() and add the resulting connected socket to the IO::Select
set.
5. If other sockets are ready for reading, perform I/O on them.
6. As client connections finish, remove them from the IO::Select set.
This version of the Eliza server illustrates how this works in practice.

The Main Server Program

The new server is called eliza_select.pl. It is broken into two parts, the main part, shown in Figure
12.2, and a module that subclasses Chatbot::Eliza called Chatbot::Eliza::Polite.

Licensed by
Stjepan Maric
4218908

Figure 12.2. Multiplexed psychiatrist server

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Multiplexed Applications 281

Compared to the previous versions of this server, the major design change is the need to break up
the Chatbot::Eliza object's command_interface() method. The reason is that command_inter-
face() has its own I/O loop, which doesn't relinquish connection until the conversation with the
client is done. We can't allow this, because it would lock out other clients.
Instead, we again subclass Chatbot::Eliza to create a "polite" version, which adds three new meth-
ods named welcome(), one_line(), and done(). The first method returns a string containing
the greeting that the user sees when he or she first connects. The second takes a line of user input,
transforms it, and returns the string containing the psychiatrist's response, along with a new prompt
for the user. done() returns a true value if the user's previous line consisted of one of the quit
phrases such as "bye," "goodbye," or "exit."
Another change is necessary to keep track of multiple Chatbot::Eliza instances. Because each ob-
ject maintains an internal record of the user's utterances, we have to associate each connected
socket with a unique Eliza object. We do this by creating a global hash named %SESSIONS in which
the indexes are the socket objects and the values are the associated Chatbot::Eliza objects. When
can_read() returns a socket that is ready for I/O, we use %SESSIONS to look up the corresponding
Chatbot::Eliza object.
We'll walk through the main part first.
Lines 1–6: Load modules—We bring in IO::Socket, IO::Select, and Chatbot::Eliza::Polite. We
also declare the %SESSIONS hash for mapping IO::Socket instances to Chatbot objects.
Lines 7–12: Create listen socket—We create a listen socket on our default port.
Lines 13–15: Add listen socket to IO:: Select object—We create a new IO::Select object and add
the listen socket to it.
Lines 16–18: Main select() loop—We now enter the main loop. Each time through the loop,
we call the IO::Select object's can_read() method. This blocks indefinitely until the listen
socket becomes ready for accept() or a connected socket (none of which have yet been
added) becomes ready for reading.
Line 18: Loop through ready handles—When can_read() returns, its result is a list of handles
that are ready for reading. It's now our job to loop through this list and figure out what to do with
each one.
Lines 19–24: Handle the listen socket—If the handle is the listen socket, then we call its
accept() method, returning a new connected socket. We create a new Chatbot::Eliza::Polite
object to handle the connection and add the socket and the Chatbot object to the %SESSIONS
hash. By indexing the hash with the unique name of the socket object, we can recover the
corresponding Chatbot object whenever we need to do I/O on that particular socket.
After creating the Chatbot object, we invoke its welcome() method. This returns a welcome
message that we syswrite() to the newly connected client. After this is done, we add the
connected socket to the IO::Select object by calling IO::Select->add(). The connected
socket will now be monitored for incoming data the next time through the loop.
Lines 25–27: Handle I/O on connected socket—If a handle is ready for reading, but it is not the
listen socket, then it must be a connected socket accepted during a previous iteration of the
loop. We recover the corresponding Chatbot object from the %SESSIONS hash. If the lookup is
unsuccessful (which logically shouldn't happen), we just ignore the socket and go on to the next
ready socket.
Otherwise, we want to read a line of input from the client. Reading a line of input from the user
is actually a bit of a nuisance because Perl's line-oriented reads, including the socket's get-
line() method, use stdio buffering and are thus incompatible with calls to select().

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Multiplexed Applications 282

We'll see in the next chapter how to roll our own line-reading function that is compatible with
select(), but in this case we punt on the issue by doing a byte-oriented sysread() of up to
1,024 characters and treating that as if it were a full line. This is usually the case if the user is
interactively typing messages, so it's good enough for this server.
Lines 28–32: Send response to client— sysread() returns either the number of bytes read, or
0 on end of file. Because the call is unbuffered, the number of bytes returned may be greater
than 0 but less than the number we requested.
If $bytes is positive, then we have data to process. We clean the data up and pass it to the
Eliza object's one_line() method, which takes a line of user input and returns a response.
We call syswrite() to send this response back to the client. If $bytes is 0 or undef, then
we treat it as an end of file and allow the next section of code to close the session.
Lines 33–38: Handle session termination—The last part of the loop is responsible for closing
down sessions.
A session should be closed when either of two things occur. First, a result code of 0 from
sysread() signifies that the client has closed its end of the connection. Second, the user may
enter one of several termination phrases that Eliza recognizes, such as "bye," "quit," or "good-
bye." In this case, Eliza's done() method returns true.
We check for both eventualities. In either case, we remove the socket from the list of handles
being monitored by the IO::Select object, close it, and remove it from the %SESSIONS hash.
Note that we treat a return code of undef from sysread(), which indicates an I/O error of some
sort, in the same way as an end of file. This is often sufficient, but a server that was processing
mission-critical data would want to distinguish between a client that deliberately shut down the con-
nection and an error. In this case, you could pass $bytes to defined() to distinguish the two
possibilities.

The Eliza::Chatbot::Polite Module

Figure 12.3 shows the code for Eliza::Chatbot::Polite. Like the earlier modification for threads, this
module was created by cutting and pasting from the original Eliza::Chatbot source code.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Multiplexed Applications 283

Figure 12.3. Multiplexed psychiatrist server

Lines 1–3: Module setup—We load the Chatbot::Eliza module and declare the current package
to be its subclass by placing the name of the parent in the @ISA array.
Lines 4–14: The welcome() method—The welcome() method is copied from the top part of
the old command_interface() method. It sets the two prompts (the one printed before the
psychiatrist's utterances and the one printed in front of the user's inputs) to reasonable defaults
and then returns a greeting randomly selected from an internally defined list. The user prompt
is appended to this string.
Lines 15–34: The one_line() method—The one_line() method takes a string as input and
returns a response. We start by checking the user input for one of the quit phrases. If there is a
quit phrase, then we generate an exiting remark from a random list of final phrases, set an
internal flag that the user is done, and return the reply. Otherwise, we invoke our inherited
transform() method to turn the user's input into a suitably cryptic utterance and return the
response along with the next prompt.
Lines 35–36: The done() method—In this method we simply check the internal exit flag set in
one_line() and return true if the user wants to exit.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Multiplexed Applications 284

Problems with the Psychiatrist Server

You can run this version of the server, telnet to it (or use one of the gab clients developed in this or
previous chapters), and have a nice conversation. While one session is open, you can open new
sessions and verify that they correctly maintain the thread of the conversation.
Unfortunately, this server is not quite correct. It starts to display problems as soon as you try to use
it noninteractively, for example, by running one of the clients in batch mode with standard input
redirected from a file. The responses may appear garbled or contain embedded newlines. The
reason for this is that we incorrectly assume that sysread() will return a full line of text each time
it's called. In fact, sysread() is not line oriented and just happens to behave that way when used
with a client that transmits data in line-length chunks. If the client is not behaving this way, then
sysread() may return a small chunk of data that's shorter than a line or may return data that
contains multiple newlines.
An obvious solution, since we must avoid the <> operator, is to write our own readline() routine.
When called, it buffers calls to sysread(), returning only the section of the buffer up to the first
newline. If a newline isn't seen at first, readline() calls sysread() as many times as needed.
However, this solution just moves the problem about because only the first call to sysread() is
guaranteed not to block. A poorly written or malicious client could hang the entire server by sending
a single byte of data that wasn't a newline. Our readline() routine would read the byte, call
sysread() again in an attempt to get to the newline, and block indefinitely. We could work around
this by calling select() again within readline() or by putting a timeout on the read using the
idiom described in Chapter 2's section Timing Out Slow System Calls.
However, there's a related problem in the multiplexed server that's not so easily fixed. What happens
if a client connects to the server, sends us a lot of data, but never reads what we send back?
Eventually the socket buffer at the client's side of the connection fills up, causing further TCP trans-
missions to stall. This in turn propagates back to us via TCP's flow control and eventually causes
our server to block in syswrite(). This makes all current sessions hang, and prevents the server
from accepting incoming connections.
To solve these problems we must use nonblocking I/O. The next chapter presents two useful mod-
ules. One, called IO::Getline, provides a wrapper around filehandles that allows you to perform line-
oriented reads safely in conjunction with select(). The other, IO::SessionSet, adds the ability to
buffer partial writes and solves the problem of the server stalling on a blocked write.

Win32 Issues
Another problem arises when using multiplexed applications on Microsoft Windows platforms. When
I originally developed the scripts in these chapters, I tested them on Windows98 using ActiveState
Perl, and everything seemed to work fine. Later, however, I discovered that the gab5.pl script (Figure
12.1) was consuming a large amount of CPU time, even when it was apparently doing nothing but
waiting for keyboard input.
When I tracked down this problem, I learned that the Win32 port of Perl does not support
select() on non-socket filehandles, including STDIN and STDOUT. So client-side scripts that
multiplex across STDIN do not wait for the filehandle to be ready, but just loop. This problem affects
the scripts in Figure 13.1 and 13.7. The IO: Poll call is affected as well (Chapter 16), and Figure
16.1 will exhibit the same problem.
Multiplexing across sockets works just fine on Win32 platforms, and so all the server examples work
as expected. The Macintosh port of Perl has no problem with select().

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Multiplexed Applications 285

Summary
This chapter completes our survey of all three of the major techniques for handling concurrent
stream-oriented connections. Each has advantages and disadvantages.
Multitasking using fork() is currently available only on UNIX and Win32 platforms (using Perl 5.6
or higher). It has more overhead than the other methods due to the necessity of launching a new
process to deal with each connection, but programs written using the technique tend to be simple
and reliable.
Multithreading is available on Win32 and many UNIX platforms and requires a version of Perl that
is built with threading support. Threaded applications must be careful to lock and release shared
variables and other resources, adding to the complexity of the code. We were able to get away
without using locking in the examples so far, but most real network applications have to deal with
this issue. Unfortunately, threading is not stable in current versions of Perl, and its API is likely to
change.
Multiplexing is available on all platforms on which Perl runs, including the Macintosh. Its drawback
is that it makes the program logic more difficult to follow due to the necessity of interleaving the I/O
from multiple sessions. Furthermore, it isn't bulletproof unless combined with nonblocking I/O, which
adds significant complexity to the code.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 286

Chapter 13. Nonblocking I/O

By default, on most operating systems I/O is blocking. If a request to read or write some data can't
be satisfied immediately, the operating system puts the program to sleep until the call completes or
generates an error. For most programming tasks, this does not cause a problem, because disk
drives, terminal windows, and other I/O devices are relatively fast, at least in human terms. As we
saw in the last chapter, however, blocking presents a problem for client/server programming be-
cause a single blocked network call can make the whole program hang while other requests wait.
To mitigate the effect of blocking in read or write calls, one can either have several concurrent
threads of execution, as with forking and multithreading servers, or use select() to determine
which filehandles are ready for I/O. The latter strategy presents a problem, however, because a
socket or other filehandle may still block on syswrite() if you attempt to write more data than it
is ready to accept. At this point the write attempt blocks and the program stalls. To avoid this, you
may use nonblocking I/O.
This chapter describes how to set up and use nonblocking I/O. In addition to avoiding blocking during
reads and writes, nonblocking I/O can also be used to avoid long waits during the connect() call.
As we will see, nonblocking I/O avoids the problems associated with managing threads and pro-
cesses but introduces its own complexities.

Creating Nonblocking I/O Handles

You can make a Perl filehandle nonblocking when you first open it, or change its nonblocking status
at any time thereafter. As implied by the name, a filehandle that is nonblocking never blocks on read
or write operations, but instead generates an error message. Nonblocking filehandles can be safely
operated only on using the sysread() and syswrite() functions. Because of the buffering is-
sues, combining nonblocking handles with the stdio routines used by the higher-level functions is
guaranteed to lead to tears of frustration.
A nonblocking handle always returns immediately from sysread() or syswrite(). If the call can
be satisfied without blocking, it returns the number of bytes read or written. If the call would block,
it returns undef and sets $! to the error code EWOULDBLOCK (also known as EAGAIN). The string
form of EWOULDBLOCK appears variously as "operation in progress" or "resource temporarily un-
available." sysread() and syswrite() can, as always, encounter other I/O errors as well. The
EWOULDBLOCK constant may be imported from the Errno module.
The other distinctive feature of a nonblocking handle is that it enables partial writes. With an ordinary
blocking handle, syswrite() does not return until the entire request has been satisfied. With
nonblocking handles, however, this behavior is changed such that if some but not all of a write
request can be satisfied immediately, syswrite() writes as much of the data as it can and then
returns the number of bytes sent. Recall that partial reads from sysread() are always possible,
regardless of the blocking status of the handle.

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Nonblocking I/O 287

Creating Nonblocking Handles: Function Interface

Perl provides both low-level and object-oriented interfaces for creating and working with nonblocking
handles. We will look at the low-level interface first, and then show how the API is cleaned up in the
object-oriented versions.

$result = sysopen (FILEHANDLE,$filename,$mode [,$perms])

The sysopen() call, introduced in Chapter 2, allows you to mark a filehandle as nonblocking when it is opened.
The idiom is to add the O_NONBLOCK flag to the flags passed in the $mode argument. For example, to open
device /dev/tape0 for nonblocking writes, you might call:
use Fcntl;
sysopen (TAPE,'/dev/tape0',O_WRONLY|O_NONBLOCK);

sysopen() works only with local files and cannot be used to open pipes or sockets. Therefore,
unless you are dealing with slow local devices such as tape drives, you'll probably never create a
nonblocking filehandle in this way. More typically, you'll take a handle that has been opened with
socket() or open() and mark it as nonblocking after the fact. This is what the fcntl() function
can do.

$result = fcntl($handle, $command, $operand)

To put a previously opened filehandle or socket into nonblocking mode, use fcntl(). The fcntl() function
is actually a catchall utility for many low-level handle operations. In addition to altering a handle's flags, you
can lock and unlock it, duplicate it, and perform more esoteric options. We will see some of these applications
in Chapter 14.

The call takes three arguments. The first two arguments are a previously opened handle and a numeric con-
stant specifying a command to perform on the handle. The third argument is a numeric parameter to pass to
the command. Some commands don't need additional data, in which case passing a third argument of 0 will
do. If successful, fcntl() returns a true value. Otherwise, it sets $! and returns undef.

The Fcntl module provides constants for all the fcntl() commands. The two commands relevant
to nonblocking handles are F_GETFL and F_SETFL, which are used to retrieve and modify a han-
dle's flags after creation. When you call fcntl() with a command of F_GETFL, it returns a bitmask
containing the handle's current flags. Call fcntl() with the F_SETFL command to change the
handle's flags to the value set in $operand. You will want $operand to include the O_NON-
BLOCK flag. The result code will indicate success or failure in changing the flags.
There is a subtlety to using fcntl() to set the nonblocking status of a filehandle. Because non-
blocking behavior is just one of several options that can be set in the flag bitmap, you should call
F_GETFL first to find out what options are already set, set the O_NONBLOCK bit using a bitwise OR,
and then call F_SETFL to apply the modified flags to the handle.
Here's a small subroutine named blocking() that illustrates this. The routine's first argument is
a handle, and its optional second argument is a Boolean value that can be used to turn blocking
behavior on or off. If called without a second argument, the subroutine returns true if the handle is
blocking; otherwise, it returns false:
use Fcntl;

sub blocking {
my ($handle,$blocking) = @_;
die "Can't fcntl(F_GETFL)" unless my $flags =

Network Programming with Perl. Network Programming with Perl, ISBN: 0-201-61571-1
Prepared for [email protected], Stjepan Maric
Copyright © 2001 by Addison-Wesley. This download file is made available for personal use only and is subject to the Terms of Service. Any other use requires prior written consent from
the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved.
 Nonblocking I/O 288

fcntl($handle,F_GETFL,0);
my $current = ($flags & O_NONBLOCK) == 0;
if (defined $blocking) {
$flags &= ~O_NONBLOCK if $nonblocking;
$flags |= O_NONBLOCK unless $blocking;
die "Can't fcntl(F_SETFL)" unless fcntl($handle,F_SETFL,$flags);
}
return $current;
}

Notice that sockets start blocking by default. To make them nonblocking, you need to call block-
ing() with a 0 argument.
warn "making socket nonblocking";
blocking($sock,0);

The Perl perlfunc POD pages contain more information on the fcntl() function.

Creating Nonblocking Handles: Object-Oriented Interfacye

If you are using the object-oriented IO::Socket or IO::File modules, then setting nonblocking mode
is as easy as calling its blocking() method.

$blocking_status = $handle->blocking([$boolean])
Called without an argument, blocking() returns the current status of blocking I/O for the handle. A true value
indicates that the handle is in normal blocking mode; a false value indicates that nonblocking I/O is active. You
can change the blocking status of a handle by providing a Boolean value to blocking(). A false value makes
the socket nonblocking; a true argument restores the normal blocking behavior.

Remember that socket objects start blocking by default. To make a socket nonblocking, you must
call blocking() with a false argument:
$socket->blocking(0);

Using Nonblocking Handles

As soon as you use nonblocking filehandles, things become a bit more complex because of the
several possible outcomes of calling sysread() and syswrite().

sysread() on Nonblocking Filehandles

When you use sysread() with a nonblocking filehandle, the following outcomes are possible:
1. If you request N bytes of data and at least N are available, then sysread() fills the scalar
buffer you provide with N bytes and returns the number of bytes read.
2. If you request N bytes of data and fewer bytes are available (but at least 1), then sys-
r