0. Why this page exists

I administer about forty Unix boxes (Solaris, a few Linux) and a handful of NT machines I try not to touch. In 1997 I got tired of writing shell scripts that fell over when a hostname had a space in it and picked up Perl. I learned it the hard way, which is to say by breaking things in production and then reading the manual to find out why. This page is the manual I wish somebody had handed me. It is not a tutorial for people who have never programmed. It is the notes of a person who was in a hurry.

Everything here is Perl 5. Specifically, 5.005_03, which is what ships on our Red Hat 6.0 boxes and what I built on the Solaris ones. If your perl -v says 4-point-anything, stop reading and upgrade; Perl 4 is dead and has been for years.

The single most important thing on this page, so I'll say it first:

#!/usr/bin/perl -w
use strict;

Every script. Every time. -w turns on warnings; use strict makes you declare your variables with my and refuses to run if you misspell one. Yes, it's more typing. It has caught more bugs for me than everything else on this page combined. Every one of the mistakes in section 6 would have been found by use strict and -w. I'm not going to say it again; just assume it's at the top of every example below.

1. Scalars, arrays, hashes

Perl has three kinds of variables and the sigil in front tells you which.

my $host  = "web1";          # a scalar: one thing (string or number, Perl doesn't care)
my $count = 42;
my $total = $count * 2.5;    # 105

my @hosts = ("web1", "web2", "db1");   # an array: an ordered list
print $hosts[0];             # "web1" -- note the $: ONE element is a scalar
print $hosts[-1];            # "db1"  -- negative index counts from the end
print $#hosts;               # 2      -- the LAST index, not the count
print scalar(@hosts);        # 3      -- the count

push @hosts, "web3";         # add to the end
my $last  = pop @hosts;      # take from the end
my $first = shift @hosts;    # take from the front
unshift @hosts, "lb1";       # add to the front

my %port = (                 # a hash: key => value pairs, no order
    http  => 80,
    https => 443,
    ssh   => 22,
);
print $port{ssh};            # 22 -- again, one element is a scalar, so $
$port{smtp} = 25;            # add one
delete $port{https};         # remove one
if (exists $port{ftp}) { }   # test for a key without creating it

foreach my $name (sort keys %port) {
    printf "%-6s %d\n", $name, $port{$name};
}

The thing that trips everyone: $hosts[0] and @hosts are the same variable, and the sigil changes depending on whether you're talking about one element or the whole list. Same with $port{ssh} and %port. Once that clicks, half of Perl clicks.

The hash is the workhorse. If you find yourself scanning an array to see whether something is in it, you wanted a hash. Counting things:

# how many hits from each host, out of an Apache access log
my %hits;
open(LOG, "</var/log/httpd/access_log") or die "can't open log: $!";
while (my $line = <LOG>) {
    my ($host) = split ' ', $line;     # first whitespace-separated field
    $hits{$host}++;                    # autovivifies: no need to set it to 0 first
}
close(LOG);

foreach my $host (sort { $hits{$b} <=> $hits{$a} } keys %hits) {
    printf "%6d %s\n", $hits{$host}, $host;
}

That sort block sorts numerically, descending, by value. Swap $a and $b for ascending. Use cmp instead of <=> for strings. This eight-line script replaced a 40-line awk pipeline I was not proud of.

2. Context, or why your array printed "3"

Perl expressions behave differently depending on whether they're being asked for a single value (scalar context) or a list (list context). An array in scalar context is its length. This is a feature, but it's a feature that will surprise you about once a week for the first year.

my @list = (10, 20, 30);
my $n    = @list;          # scalar context: $n is 3
my ($x)  = @list;          # LIST context (the parens): $x is 10
print "have @list\n";      # interpolates as "have 10 20 30"
print "have " . @list . "\n";   # the . forces scalar: "have 3"

my $line = <FH>;           # scalar context: one line
my @all  = <FH>;           # list context: every remaining line, slurped into memory

my $when = localtime;      # scalar: "Sun Sep 19 14:02:33 1999"
my @when = localtime;      # list: (sec, min, hour, mday, mon, year, wday, yday, isdst)

That last one is worth a whole paragraph. In list context, localtime gives you the month as 0 to 11 and the year as years since 1900. Right now the year comes back as 99. On January 1 it will come back as 100, not 0 and not 2000. If you have ever written "19$year" to print the year, you have a Y2K bug, and it will print 19100. The right way is $year + 1900, always. I found two of these in scripts on our own boxes last winter. Both were mine.

3. Regular expressions

Regexes are why you learn Perl. The basics:

if ($line =~ /error/)      { }   # matches if $line contains "error"
if ($line =~ /error/i)     { }   # case-insensitive
if ($line !~ /^#/)         { }   # does NOT start with #
$line =~ s/\s+$//;               # strip trailing whitespace (in place)
(my $copy = $line) =~ s/foo/bar/g;   # substitute in a copy, all occurrences

The pieces you use every day:

Pulling the fields out of an Apache common-log line (host, ident, user, date in brackets, the request in quotes, status, bytes):

while (my $line = <LOG>) {
    next unless $line =~ m{
        ^(\S+)\s+\S+\s+\S+       # host, ident, authuser
        \s+\[([^\]]+)\]          # [date]
        \s+"(\w+)\s+(\S+)[^"]*"  # "METHOD /path HTTP/1.x"
        \s+(\d{3})\s+(\d+|-)     # status, bytes (or -)
    }x;
    my ($host, $when, $method, $path, $status, $bytes) = ($1, $2, $3, $4, $5, $6);
    $bytes = 0 if $bytes eq '-';
    print "$status $path\n" if $status >= 400;
}

Things to notice. m{...} lets you use braces instead of slashes so you don't have to escape slashes inside. /x lets you put whitespace and comments in the pattern, and you should, because the version of that regex on one line is unreadable and I know because I wrote it that way first. [^\]]+ means "one or more characters that aren't a close bracket", which is how you match up to the next bracket without being greedy.

About greedy: .* grabs as much as it can. .*? grabs as little as it can. If you write /"(.*)"/ against a line with two quoted strings, $1 is everything from the first quote to the last one. You wanted /"(.*?)"/ or, better, /"([^"]*)"/. This one cost me a Saturday.

Don't test $1 unless the match succeeded. If it didn't, $1 still holds whatever the last successful match left in it, from anywhere in the program. Always write if ($x =~ /(..)/) { use $1 }, or assign the captures in list context: my ($lo, $hi) = $x =~ /(\d+)-(\d+)/;, which gives you an empty list on failure.

4. Files, and locking them

open(IN,  "<$path")  or die "can't read $path: $!";    # read
open(OUT, ">$path")  or die "can't write $path: $!";   # write (clobbers)
open(LOG, ">>$path") or die "can't append $path: $!";  # append
open(PS,  "ps -ef |") or die "can't run ps: $!";       # read from a command
close(IN);

Always check the return of open. $! is the system error ("No such file or directory", "Permission denied"), and printing it will save you from guessing. A script that opens a file that isn't there and carries on reading from the closed handle produces no output and no error and you will stare at it for an hour.

If more than one copy of your script can run at the same time and they all write to the same file — which is every CGI script ever written — you need a lock, or two writes will land on top of each other and you'll get half of one line and half of another. Perl's flock does it:

use Fcntl qw(:flock);      # gives you LOCK_SH, LOCK_EX, LOCK_UN

open(FH, ">>$path") or die "can't append $path: $!";
flock(FH, LOCK_EX)   or die "can't lock $path: $!";     # blocks until we have it
seek(FH, 0, 2);            # go to the real end: someone may have written while we waited
print FH "$entry\n";
close(FH);                 # closing releases the lock

LOCK_EX is exclusive (for writing), LOCK_SH is shared (for reading; many readers can hold it at once, but not while a writer has the exclusive one). The seek after the lock is the part people leave out. The lock is advisory — it only works if every program touching the file uses it — but since every program touching the file is yours, use it.

5. The CGI guestbook, in full

Here is the whole thing that runs the guestbook on my own (non-GeoCities) box. It's short on purpose. It uses CGI.pm, which comes with Perl 5.004 and later and does the form parsing so you never have to write that yourself again, and it runs under taint mode.

Taint mode (-T on the shebang line) means Perl marks every piece of data that came from outside the program — form input, environment variables, file contents — as "tainted", and refuses to let tainted data do anything dangerous (run a command, open a file for writing by that name, and so on) until you have explicitly checked it with a regex and extracted the safe part. It also insists you set $ENV{PATH} yourself. Every CGI script should run with -T. The ones on the Net that don't are how servers get broken into.

#!/usr/bin/perl -wT
# guestbook.cgi -- append an entry if one was posted, then show the book.
use strict;
use CGI qw(:standard);
use Fcntl qw(:flock);

$ENV{PATH} = '/bin:/usr/bin';          # taint mode requires a known PATH
my $file = '/home/dave/gbook/entries.txt';   # NOT under the document root

print header();                        # Content-type: text/html, blank line
print start_html('Guestbook');

my $name    = param('name')    || '';
my $comment = param('comment') || '';

if ($name ne '' && $comment ne '') {
    for ($name, $comment) {
        s/[\r\n]+/ /g;                 # keep each entry on one line
        s/\|/-/g;                      # pipe is my field separator
    }
    open(BOOK, ">>$file") or die "can't append to $file: $!";
    flock(BOOK, LOCK_EX)   or die "can't lock $file: $!";
    seek(BOOK, 0, 2);
    print BOOK join('|', scalar(localtime), $name, $comment), "\n";
    close(BOOK)            or die "can't close $file: $!";
    print p("Thanks, ", escapeHTML($name), ". You're in the book.");
}

open(BOOK, "<$file") or die "can't read $file: $!";
flock(BOOK, LOCK_SH) or die "can't lock $file: $!";
my @entries = <BOOK>;
close(BOOK);

print h2('The book so far');
foreach my $line (reverse @entries) {   # newest first
    chomp $line;
    my ($when, $who, $what) = split /\|/, $line, 3;
    print p(b(escapeHTML($who)), " wrote on $when:", br(), escapeHTML($what));
}

print end_html();

Notes, in the order you'll wonder about them:

• use CGI qw(:standard) gives you param(), header(), and a set of functions named after HTML tags (p, b, br, h2...) that build the tags for you. You can also just print HTML strings; I use the functions because I forget close tags.
• escapeHTML turns < into < and so on. Never print what a visitor typed back into a page without it. Otherwise the first joker who comes along types a <script> or a <marquee> into your guestbook and it runs, or scrolls, for everyone.
• The data file lives outside the web server's document root, so nobody can fetch the raw file, and the web server's user (nobody on our boxes) has to be able to write to it — that's a chmod on the file and the directory, and it's the number one reason guestbook scripts "don't work".
• Under -T, appending tainted text to a file is allowed (it's the filename that mustn't be tainted, and mine is a literal). If you ever build a filename from form data, you must untaint it with a regex first, like my ($safe) = $name =~ /^(\w+)$/;, and die if $safe is empty.
• The 3 in split /\|/, $line, 3 says "at most three fields", so a comment containing a pipe (which I replaced anyway, but belt and suspenders) doesn't get chopped.
• While you're debugging, add use CGI::Carp qw(fatalsToBrowser); near the top and your die messages will show up in the browser instead of buried in the server's error log. Take it out when you're done; it tells the world about your file paths.
• The HTML form that posts to this is two fields named name and comment and a submit button. That's it. (I'm not putting a form on this GeoCities page; it wouldn't run here anyway.)

6. Mistakes that cost me a weekend

Each of these is real, and each is documented here so that I don't do it a third time.

1. == versus eq. == compares numbers, eq compares strings. if ($host == "web1") is always true, because both sides are 0 as numbers. -w warns you. I didn't have -w. That was the first weekend.
2. Forgetting chomp. Lines read from a file end in a newline. If you compare $line eq "yes" it will never match, because $line is "yes\n". chomp($line) takes the newline off.
3. if ($x = 5). One equals sign assigns. That's always true and it also just set $x to 5. You meant ==. -w warns.
4. Not checking open. See section 4. The script ran for three weeks writing its report to nowhere.
5. Slurping a 400 MB log with my @all = <LOG> on a box with 128 MB. Read it a line at a time in a while loop. The box came back after a reboot.
6. "19$year". See section 2. I did this in a backup rotation script in 1997 and found it in February 1999 while looking for exactly this bug. It would have started naming tapes 19100-01-01 in fifteen weeks.
7. The greedy .*. See section 3.
8. Editing a script on the NT box and copying it to Unix. Windows line endings (\r\n) on the shebang line mean the kernel tries to run /usr/bin/perl\r, which doesn't exist, and the error message says "No such file or directory" about a file that's plainly right there. Twenty minutes, every time, until I learned to look for it first.
9. No use strict. A typo in a variable name ($totl for $total) silently made a new empty variable, and the report showed zero for every host for a week before anyone noticed. That was the second weekend. It was also the last one without use strict.

7. Books

• Learning Perl (O'Reilly, the one with the llama on the cover, 2nd edition). Start here. A week of evenings. Written by people who teach this for a living and it shows.
• Programming Perl (O'Reilly, the camel). The reference, by the man who wrote the language and two others. The 2nd edition covers Perl 5. You don't read it front to back; you keep it next to the keyboard and open it when the llama runs out. Everyone calls it "the Camel book" and after a month so will you.
• Perl Cookbook (O'Reilly, the ram). Recipes. "How do I..." with the answer. The file-locking section is where I finally understood section 4 above.
• The documentation that came with Perl. Type perldoc perl for the index, perldoc perlre for regexes, perldoc perlfunc for every built-in, perldoc perlfaq for the questions everyone asks. It's all there, it's free, and it's better than most of what's been printed.
• CPAN. The Comprehensive Perl Archive Network: the module library. Before you write anything that talks to a database, parses a date, or fetches a web page, somebody already wrote it and put it there. Your Perl came with instructions for getting at it (perldoc CPAN).
• comp.lang.perl.misc on Usenet, if you can stand it. Read for a month before posting. Read the FAQ before that.

Neighbors: Brian's PC Help Page · The Y2K Watch: 100 Days to Go (a man who has read more COBOL than I've read Perl) · Mom's Guide to the Family Computer · Quake II Clan [DUST] (what the other half of the office does after 6) · SiliconValley neighborhood

Visitors since 1998-11-02:   

Corrections welcome and expected: [email protected]. If you tell me a code sample is wrong, include the error message and your perl -v. If you ask me to debug your script, include the script. If you ask me why your CGI doesn't work, check the file permissions first, then the line endings, then ask.

Last updated 1999-09-19. © 1998-1999 Dave Ferreira. The code on this page is yours to use; if it breaks something, you were warned in section 0.