View
5.943
Download
3
Category
Tags:
Preview:
DESCRIPTION
Techniques for writing good code in perl running on Unix platform.
Citation preview
Unix Programming with Perl 2
Unix Programming with Perl 2
DeNA Co., Ltd.
Kazuho Oku
Writing correct codeWriting correct code
tests aren’t enoughtests don’t ensure that the code is correct
writing correct code requires…knowledge of perl and knowledge of the OS
Oct 15 2011 Unix Programming with Perl 2 2
Last Year’s TalkLast Year’s Talk
Covered these aspects of Unix programming using Perl$! and Errno
how to evaluate the errors
file handlestheir internalsinteraction w. fork(2)
Unix signalsvarious signals and how to handle themwriting cancellable code
Oct 15 2011 Unix Programming with Perl 2 3
Last Year’s SlidesLast Year’s Slides
http://www.slideshare.net/kazuho/unix-programming-with-perl
Oct 15 2011 Unix Programming with Perl 2 4
Today’s TalkToday’s Talk
will cover more advanced topicsinter-process communicationUnix signals and race condition
Oct 15 2011 Unix Programming with Perl 2 5
IPC::Open3
Oct 15 2011 Unix Programming with Perl 2 6
IPC::Open3 - Pipe and DeadlockIPC::Open3 - Pipe and Deadlock
# check syntax errors in perl scripts by using perl –cmy @cmd = ( $^X, (map { "-I$_" } grep { !ref $_ } @INC), '-c', $file,);my $pid = open3(my $cin, my $cout, my $cerr, @cmd);while (waitpid($pid, 0) != $pid) {}ok((WIFEXITED($?) && WEXITSTATUS($?) == 0), $file);
http://d.hatena.ne.jp/tokuhirom/20100813/1281666615
Oct 15 2011 Unix Programming with Perl 2 7
IPC::Open3 - Pipe and DeadlockIPC::Open3 - Pipe and Deadlock
Q. generally speaking, this code might block. But when?
my $pid = open3(my $cin, my $cout, my $cerr, @cmd);while (waitpid($pid, 0) != $pid) {}
Oct 15 2011 Unix Programming with Perl 2 8
IPC::Open3 - Pipe and DeadlockIPC::Open3 - Pipe and Deadlock
Q. generally speaking, this code might block. But when?
my $pid = open3(my $cin, my $cout, my $cerr, @cmd);while (waitpid($pid, 0) != $pid) {}
A1. blocks if the child process reads from STDIN
Oct 15 2011 Unix Programming with Perl 2 9
IPC::Open3 - Pipe and DeadlockIPC::Open3 - Pipe and Deadlock
Close STDIN of the child before calling waitpid
my $pid = open3(my $cin, my $cout, my $cerr, @cmd);close $cin;while (waitpid($pid, 0) != $pid) {}
Oct 15 2011 Unix Programming with Perl 2 10
IPC::Open3 - Pipe and DeadlockIPC::Open3 - Pipe and Deadlock
Q. generally speaking, this code might block. But when?
my $pid = open3(my $cin, my $cout, my $cerr, @cmd);while (waitpid($pid, 0) != $pid) {}
A1. blocks if the child process reads from STDIN
A2. blocks if the child process writes long data to STDOUT or STDERR
Oct 15 2011 Unix Programming with Perl 2 11
Size of the pipe bufferSize of the pipe buffer
Pipe has size limitcannot write infinitelyunless the other peer reads from pipe
But actually, how large is the size limit?
Oct 15 2011 Unix Programming with Perl 2 12
Size of the pipe buffer (2)Size of the pipe buffer (2)
Checking the size of the pipe buffer
for (my $sz = 1; ; $sz++) { my @cmd = ( $^X, '-e', qq(print "1"x$sz), ); my $pid = open3(my $cin, my $cout, 0, @cmd) or die $!; while (waitpid($pid, 0) != $pid) {} print "size: $sz\n";}
Oct 15 2011 Unix Programming with Perl 2 13
Size of the pipe buffer (3)Size of the pipe buffer (3)
Size of the pipe buffer was…Linux 2.6.32 (x86-64): 65,536 bytesMac OS X 10.6: 16,384 bytes
Size may varyold versions of Linux: 4,096 bytes
TCP streams and Unix sockets have configurable buffer size as well
Oct 15 2011 Unix Programming with Perl 2 14
IPC::Open3 - Pipe and DeadlockIPC::Open3 - Pipe and Deadlock
How should we avoid deadlocks?if we do not need the output of the child
process (ex. perl –c), could this be the right answer?
my $pid = open3(my $cin, my $cout, my $cerr, @cmd);close $cout;close $cerr;while (waitpid($pid, 0) != $pid) {}ok((WIFEXITED($?) && WEXITSTATUS($?) == 0), $file);
Oct 15 2011 Unix Programming with Perl 2 15
IPC::Open3 - Pipe and DeadlockIPC::Open3 - Pipe and Deadlock
No. The child process may get killed while trying to write to the output streams closed by the parent process
my $pid = open3(my $cin, my $cout, my $cerr, @cmd);close $cout;close $cerr;# if child process tries to write at this moment, it# will be killed by SIGPIPEwhile (waitpid($pid, 0) != $pid) {}# and as a result, $? may become differentok((WIFEXITED($?) && WEXITSTATUS($?) == 0), $file);
Oct 15 2011 Unix Programming with Perl 2 16
IPC::Open3 - Pipe and DeadlockIPC::Open3 - Pipe and Deadlock
So how about this? we read all data from the child process
my $pid = open3(my $cin, my $cout, my $cerr, @cmd);while (<$cout>) {}while (<$cerr>) {}while (waitpid($pid, 0) != $pid) {}
Oct 15 2011 Unix Programming with Perl 2 17
IPC::Open3 - Pipe and DeadlockIPC::Open3 - Pipe and Deadlock
No. Will deadlock if the child process writes more bytes than the pipe buffer size to STDERR
my $pid = open3(my $cin, my $cout, my $cerr, @cmd);while (<$cout>) {} # may enter deadlock at this pointwhile (<$cerr>) {}while (waitpid($pid, 0) != $pid) {}
Oct 15 2011 Unix Programming with Perl 2 18
IPC::Open3 - Pipe and DeadlockIPC::Open3 - Pipe and Deadlock
Tip: passing undef as CHLD_ERR will send all output to CHLD_OUT
my $pid = open3(my $cin, my $cout, undef, @cmd);while (<$cout>) {}while (waitpid($pid, 0) != $pid) {}
Oct 15 2011 Unix Programming with Perl 2 19
IPC::Open3 – Better to use temporary filesIPC::Open3 – Better to use temporary files
Advice: don’t use pipes, use temporary files (unless you need to read the output of the child process while it is running)
my $cout = File::Temp->new();my $pid = do { local *COUT = $cout; open3(my $cin, ’>&COUT’, 0, @cmd);} or die $!;while (waitpid($pid, 0) != $pid) {}seek($cout, 0, SEEK_SET) or die $!; # seek to the start...Oct 15 2011 Unix Programming with Perl 2 20
IPC::Open3 vs. open |-IPC::Open3 vs. open |-
Q. Why use IPC::Open3 instead of open ’-| cmd > tmpfile’?
A. to skip the shell invocationfor speed and security
Oct 15 2011 Unix Programming with Perl 2 21
IPC::Open3 vs. open |- (2)IPC::Open3 vs. open |- (2)
Escaping for open |- is difficult# using IPC::Open3my @cmd = ( $prog, ’-e’, $arg2,);my $pid = open3(my $cin, my $cout, 0, @cmd) or die $!;
# using open |-my $cmd = ”$prog –e $arg”; # need to escape $argopen my $fh, ’|-’,”$cmd > ” . $tempfh->filename or die $!;
Oct 15 2011 Unix Programming with Perl 2 22
Avoid shell invocationAvoid shell invocation
Shell invocation is evilex. iT○ns upgrade accidentally removes
user filesfailed to quote usernames with a whitespace
Direct invocation is safersystem($args) => system(@args)open | => IPC::Open2 or IPC::Open3
Oct 15 2011 Unix Programming with Perl 2 23
Rewriting open |- using IPC::Open3Rewriting open |- using IPC::Open3
# the originalopen(my $fh,’|-’, $cmd) or die $!;print $fh ”hello\n”;close $fh;# TODO: check $?
# using IPC::Open3my $pid = open3(my $fh,’>&STDOUT’, ’>&STDERR’, @cmd) or die $!;print $fh ”hello\n”;close $fh;while (waitpid($pid, 0) != $pid) {}# TODO: check $?
Oct 15 2011 Unix Programming with Perl 2 24
Rewriting open -| using IPC::Open3Rewriting open -| using IPC::Open3
# the originalopen(my $fh,’-|’, $cmd) or die $!;my $line = <$fh>;close $fh;# TODO: check $?
# using IPC::Open3my $pid = open3(my $cin, my $fh, ’>&STDERR’, @cmd) or die $!;close $cin;my $line = <$fh>;close $fh;while (waitpid($pid, 0) != $pid) {}# TODO: check $?
Oct 15 2011 Unix Programming with Perl 2 25
Signal and Race Condition
Oct 15 2011 Unix Programming with Perl 2 26
How to sleep until receiving a signal?How to sleep until receiving a signal?
Is the code correct?
my $gothup = 0;local $SIG{HUP} = sub { $gothup++ };while (! $gothup) { sleep(30 * 60); # sleep for 30 minutes do_cleanup(); # do some periodical tasks}print ”SIGHUP!\n”;
Oct 15 2011 Unix Programming with Perl 2 27
How to sleep until receiving a signal? (2)How to sleep until receiving a signal? (2)
a race condition exists
my $gothup = 0;local $SIG{HUP} = sub { $gothup++ };while (! $gothup) { # What happens if a signal is arrives here? # sleep() will sleep 30 minutes since it never # gets interrupted by the signal sleep(30 * 60); # sleep for 30 minutes do_cleanup(); # do some periodical tasks}print ”SIGHUP!\n”;
Oct 15 2011 Unix Programming with Perl 2 28
Use POSIX::pselectUse POSIX::pselect
SIGHUP is blocked outside of pselect (and thus no race conditions)
my $blocked = POSIX::SigSet->new(SIGHUP);my $unblocked = POSIX::SigSet->new();my $gothup = 0;local $SIG{HUP} = sub { $gothup++ };sigprocmask(SIG_BLOCK, $blocked, $unblocked);while (! $gothup) { pselect(undef, undef, undef, 30 * 60, $unblocked); do_cleanup(); # do some periodical tasks}print ”SIGHUP!\n”;
Oct 15 2011 Unix Programming with Perl 2 29
The problem of pselectThe problem of pselect
Pselect has race condition on many environments, implemented like…
sub select { my ($rset, $wset, $eset, $secs, $mask) = @_; my $oldmask = POSIX::SigSet->new(); sigprocmask(SIG_SETMASK, $mask, $oldmask); $oldmask); my $ret = select($rset, $wset, $eset, $secs); sigprocmask(SIG_SETMASK, $oldmask); $ret;}
osx has the problem, glibc on linux does not have the problem but bionic (android) has the problem, …
Oct 15 2011 Unix Programming with Perl 2 30
Using eval & die does not solve the problemUsing eval & die does not solve the problem
my $blocked = POSIX::SigSet->new(SIGHUP);my $unblocked = POSIX::SigSet->new();sigprocmask(SIG_BLOCK, $blocked, $unblocked);$SIG{HUP} = sub { die ’sighup:’ };while (! $gothup) { { local $@; eval { sigprocmask(SIG_SETMASK, $unblocked); # what if the signal is delivered at the very moment the # perl interpreter calls sleep(3)? sleep(30 * 60); # sleep for 30 minutes }; $gothup = $@ =~ /^sighup:/; sigprocmask(SIG_SETMASK, $blocked); } do_cleanup(); # do some periodical tasks}
Oct 15 2011 Unix Programming with Perl 2 31
The fix – call syswrite on signalThe fix – call syswrite on signal
# set unsafe signal handler (see perldoc perlipc) that converts a# signal to a message (Q. proper error handling as advised in this# year and last year’s slides are missing. Can you name them?)socketpair(my $sig_rdr, my $sig_wtr, AF_UNIX, SOCK_STREAM, PF_UNSPEC) or die $!;POSIX::sigaction(SIGHUP, POSIX::SigAction->new(sub { syswrite($sig_wtr, "1", 1) == 1 or die $!; }}));while (1) { my $rfds = ''; vec($rfds, fileno($sig_rdr), 1) = 1; if (select($rfds, undef, undef, 30 * 60) > 0) { sysread($sig_rdr, my $buf, 1) == 1 or die $!; last; } do_cleanup(); # do some periodical tasks}
Oct 15 2011 Unix Programming with Perl 2 32
Summary
Oct 15 2011 Unix Programming with Perl 2 33
SummarySummary
buffer size is not infinite, be aware of deadlocks on inter-process / network communication
avoid shell invocation, use system(@args) or IPC::Open3
be careful of race conditions when handling Unix signals
Oct 15 2011 Unix Programming with Perl 2 34
Recommended