develooper Front page | perl.fwp | Postings from March 2002

Re: perl job saving (was Re: Non-golf fun)

Chris Benson
March 20, 2002 12:49
Re: perl job saving (was Re: Non-golf fun)
Message ID:
On Tue, Mar 19, 2002 at 11:16:40PM -0500, Uri Guttman wrote: 
> this current thread on fwp on perl saving someone's job is interesting 
> and i think more on the lines of advocacy and even discussing perl 
> jobs. so i cc'ed to those lists to get more of those 'perl saved my A**'
> stories. we have success stories on and others on 
> and advocacy has some stories i think. but a collection of 
> these stories would be great to have.



I used to be development/1st-/2nd-/3rd-line tech. support for a software
house selling back-office systems to lawyers.

Lawyers accounts remain open for the life of a case -- you can't
transfer balances at the end of a period like (some) normal accounts.
Therefore you have to transfer data from any existing system -- even if
it's a 20 year-old mini from a defunct manufacturer, with no support,
no documentation, no system software and no file/record layouts.

We would get the user to print all the reports on the system ... and
capture the output to disk.  I used grep+sed+awk to process these text
files into the form our application needed.

The story:

Close of business one Friday afternoon: I visited a firm of lawyers
and collected their live data.  I took the data home and started
the conversion on the machine that I would be installing -- with the
converted data -- on Monday morning.  (I had already collected test
data and written and tested the scripts with a sub-set of the data).
The conversion should take until Saturday night, then I would spend
Sunday verifying and cross-checking the results.

Saturday morning: come down to find the machine only 10% through the
processing and the disk chattering away.   I eventually managed to log
in and found that the awk conversion process was using all RAM + 50%
swap ... and growing at 4kB every few records ...

A quick calculation showed that (a) it would run out of swap before
completion and (b) it was not going to be finished in time for a Monday
morning installation.


I was committed to a Monday install.  One thing you learn working for
lawyers -- you do not upset lawyers: it tends to be expensive.


I had been playing round with this new language that has been released
(in several dozen messages posted to comp.lang.misc (or something)):
Perl -- already at version 3.  I had managed to get it compiled on the
Altos Unix systems we were using then, but hadn't done much with it
-- the challenge had been getting it running on Altos Unix :-),  but I'd
noticed that (a) it was supposed to be at least 10x faster than awk and
(b) it had a awk-to-perl translator.  I happened to have a tape with
Perl compiled for Altos Unix with me.

Back to the plot:

* I cut several thousand lines of awk code out of the shell scripts they
were embedded in and saved them in separate files.

* I installed perl from the tape.

* I ran a2p on the files -- and saved the results.  I noticed a few
warnings telling me to check whether the correct character or numeric
test had been chosen: it had.

* a2p also pointed out a mis-placed comment  -- that turned out to
be causing the memory leak -- but didn't generate an error or warning
from awk.

Even though removing the comment had (apparently) solved to problem, 
I was running late, so having created the Perl scripts I decided to use
them:  the processing took under 1 hour.

		** The processing took under 1 hour. **

The expected processing time in awk was ~18 hours:  In spite of the
problems with awk,  because of Perl I was now ahead of schedule.

The system was installed on the Monday morning.

This story still brings tears of relief/joy to my eyes.

I have used Perl ever since.  
Chris Benson Perl Programming lists via nntp and http.
Comments to Ask Bjørn Hansen at | Group listing | About