develooper Front page | perl.perl5.porters | Postings from August 2010

Re: Patch to make string-append on win32 100 times faster

Thread Previous | Thread Next
Elizabeth Mattijsen
August 17, 2010 01:15
Re: Patch to make string-append on win32 100 times faster
Message ID:
On Aug 17, 2010, at 10:09 AM, Wolfram Humann wrote:
> -------- Original Message --------
> Subject: Re: Patch to make string-append on win32 100 times faster
> From: Jan Dubois <>
> To: 'David Golden' <>, 'Reini Urban' <>
> Date: 17.08.2010 01:09
>> On Mon, 16 Aug 2010, David Golden wrote:
>>> I think I'm with Reini on this one. I'd rather keep it Windows-only until a
>>> performance problem is documented elsewhere.
>> It's been documented that the patch improves performance on FreeBSD by a
>> *factor* of 160 for one particular test case, as well as a factor of 4
>> on Linux. In the meantime Wolfram has constructed another testcase where
>> the patch improves performance on Linux by a factor of 45 (my system) to
>> 230 (Wolfram's system).
>> Granted, these are worst-case samples, but at least the first test
>> program was derived from a real-world application: appending lots of
>> small strings to an ever-growing large buffer.
> I think that even my worst-case Linux test case is not totally out-of-the-world. It occurs whenever a large number of strings grows simultaneously by small increments. One application that comes to mind is pivoting (exchanging rows and columns of) a large text-based table: loop over the lines, splitting each row and appending the chunks to the respective column's string; finally print the column strings as new rows when the complete table is read.

FWIW, I have several pieces of important business logic that add packed integers to hash values for millions of iterations.  Seems like a worst case to me.

Having said that, I've been moving these cases to use a temporary file, basically using the file system as a hash.  This in the end, proved to work a lot faster and scaled much better, way beyond the physical RAM that was available.

Thread Previous | Thread Next Perl Programming lists via nntp and http.
Comments to Ask Bjørn Hansen at | Group listing | About