develooper Front page | perl.perl5.porters | Postings from June 2016

RE: Inefficient stat-ing of files?

Thread Previous | Thread Next
From:
bulk 88
Date:
June 24, 2016 16:21
Subject:
RE: Inefficient stat-ing of files?
Message ID:
BAY182-W56E0DF5FA309241E6DD38DDF2E0@phx.gbl
> From: davidnmfarrell@gmail.com 
> Date: Thu, 23 Jun 2016 22:16:08 -0400 
> Subject: Inefficient stat-ing of files? 
> To: perl5-porters@perl.org 
>  
> Hi Porters, 
>  
> I used strace on a Perl one liner and found that 12% of the time was  
> spent stat-ing files like this: 
>  
> stat("/home/dfarrell/.plenv/versions/5.22.0/lib/perl5/site_perl/5.22.0/strict.pmc",  
> 0x7ffecd546980) = -1 ENOENT (No such file or directory) 
> stat("/home/dfarrell/.plenv/versions/5.22.0/lib/perl5/site_perl/5.22.0/strict.pm<http://strict.pm>",  
> 0x7ffecd5468b0) = -1 ENOENT (No such file or directory) 
> stat("/home/dfarrell/.plenv/versions/5.22.0/lib/perl5/5.22.0/x86_64-linux/strict.pmc",  
> 0x7ffecd546980) = -1 ENOENT (No such file or directory) 
> stat("/home/dfarrell/.plenv/versions/5.22.0/lib/perl5/5.22.0/x86_64-linux/strict.pm<http://strict.pm>",  
> 0x7ffecd5468b0) = -1 ENOENT (No such file or directory) 
> stat("/home/dfarrell/.plenv/versions/5.22.0/lib/perl5/5.22.0/strict.pmc",  
> 0x7ffecd546980) = -1 ENOENT (No such file or directory) 
> stat("/home/dfarrell/.plenv/versions/5.22.0/lib/perl5/5.22.0/strict.pm<http://strict.pm>",  
> {st_mode=S_IFREG|0444, st_size=4433, ...}) = 0 
> stat("/home/dfarrell/.plenv/versions/5.22.0/lib/perl5/site_perl/5.22.0/x86_64-linux/warnings.pmc",  
> 0x7ffecd546980) = -1 ENOENT (No such file or directory) 
> stat("/home/dfarrell/.plenv/versions/5.22.0/lib/perl5/site_perl/5.22.0/x86_64-linux/warnings.pm<http://warnings.pm>",  
> 0x7ffecd5468b0) = -1 ENOENT (No such file or directory) 
> stat("/home/dfarrell/.plenv/versions/5.22.0/lib/perl5/site_perl/5.22.0/warnings.pmc",  
> 0x7ffecd546980) = -1 ENOENT (No such file or directory) 
> stat("/home/dfarrell/.plenv/versions/5.22.0/lib/perl5/site_perl/5.22.0/warnings.pm<http://warnings.pm>",  
> 0x7ffecd5468b0) = -1 ENOENT (No such file or directory) 
> stat("/home/dfarrell/.plenv/versions/5.22.0/lib/perl5/5.22.0/x86_64-linux/warnings.pmc",  
> 0x7ffecd546980) = -1 ENOENT (No such file or directory) 
> stat("/home/dfarrell/.plenv/versions/5.22.0/lib/perl5/5.22.0/x86_64-linux/warnings.pm<http://warnings.pm>",  
> 0x7ffecd5468b0) = -1 ENOENT (No such file or directory) 
> stat("/home/dfarrell/.plenv/versions/5.22.0/lib/perl5/5.22.0/warnings.pmc",  
> 0x7ffecd546980) = -1 ENOENT (No such file or directory) 
> stat("/home/dfarrell/.plenv/versions/5.22.0/lib/perl5/5.22.0/warnings.pm<http://warnings.pm>",  
> {st_mode=S_IFREG|0444, st_size=44835, ...}) = 0 
> stat("/home/dfarrell/.plenv/versions/5.22.0/lib/perl5/site_perl/5.22.0/x86_64-linux/XSLoader.pmc",  
> 0x7ffecd546980) = -1 ENOENT (No such file or directory) 
> stat("/home/dfarrell/.plenv/versions/5.22.0/lib/perl5/site_perl/5.22.0/x86_64-linux/XSLoader.pm",  
> 0x7ffecd5468b0) = -1 ENOENT (No such file or directory) 
> stat("/home/dfarrell/.plenv/versions/5.22.0/lib/perl5/site_perl/5.22.0/XSLoader.pmc",  
> 0x7ffecd546980) = -1 ENOENT (No such file or directory) 
> stat("/home/dfarrell/.plenv/versions/5.22.0/lib/perl5/site_perl/5.22.0/XSLoader.pm",  
> 0x7ffecd5468b0) = -1 ENOENT (No such file or directory) 
> stat("/home/dfarrell/.plenv/versions/5.22.0/lib/perl5/5.22.0/x86_64-linux/XSLoader.pmc",  
> 0x7ffecd546980) = -1 ENOENT (No such file or directory) 
> stat("/home/dfarrell/.plenv/versions/5.22.0/lib/perl5/5.22.0/x86_64-linux/XSLoader.pm",  
> 0x7ffecd5468b0) = -1 ENOENT (No such file or directory) 
> stat("/home/dfarrell/.plenv/versions/5.22.0/lib/perl5/5.22.0/XSLoader.pmc",  
> 0x7ffecd546980) = -1 ENOENT (No such file or directory) 
> stat("/home/dfarrell/.plenv/versions/5.22.0/lib/perl5/5.22.0/XSLoader.pm",  
> {st_mode=S_IFREG|0444, st_size=10305, ...}) = 0 
> stat("/home/dfarrell/.plenv/versions/5.22.0/lib/perl5/site_perl/5.22.0/x86_64-linux/Exporter.pmc",  
> 0x7ffecd546980) = -1 ENOENT (No such file or directory) 
> stat("/home/dfarrell/.plenv/versions/5.22.0/lib/perl5/site_perl/5.22.0/x86_64-linux/Exporter.pm",  
> 0x7ffecd5468b0) = -1 ENOENT (No such file or directory) 
> stat("/home/dfarrell/.plenv/versions/5.22.0/lib/perl5/site_perl/5.22.0/Exporter.pmc",  
> 0x7ffecd546980) = -1 ENOENT (No such file or directory) 
> stat("/home/dfarrell/.plenv/versions/5.22.0/lib/perl5/site_perl/5.22.0/Exporter.pm",  
> 0x7ffecd5468b0) = -1 ENOENT (No such file or directory) 
>  
> In these cases Perl was stat-ing 7 files for every 1 file that existed.  
> I'm no systems programmer but have you considered using the directory  
> of the most recent successful stat rather than cycling through  
> directories in the same order? I wonder if that could save startup  
> time, particularly for programs with hundreds of dependencies. 
>  
> This is with Perl 5.22 on Fedora, Linux kernel 4.3.5-300 
>  
> Interested to hear your thoughts, thanks! 
>  
> David 

The solution is less @INC dirs, but in Unix FHS, having a dozen lib or include or etc dirs is encouraged and not knowing where anythign is is enoucouraged instead of discouraged. 

One idea is, if the dir doesn't exist, have the require loader remove it from @INC.

 stat("/home/dfarrell/.plenv/versions/5.22.0/lib/perl5/site_perl/5.22.0/

Funny dir name, "5.22.0" appears twice in the path. Maybe it shouldn't. Also PMC can be disabled with -DPERL_DISABLE_PMC which I like to do personally. I've taken whacks at the .PM loader in perl with regards to IO before

http://perl5.git.perl.org/perl.git/commitdiff/d345f4877521ed3ad5ad337f7e87a95625df05c0

http://perl5.git.perl.org/perl.git/commitdiff/8ca2a5d6d84d93b456a8dfa622707ae9222a388a

http://perl5.git.perl.org/perl.git/commitdiff/1e777496fd51e7d05020c0f05a4f2e19f2a3148d

but there isn't much more that I think can be done without user observable changes, which I am not gonna bikeshed here or waste time trying to push through.

If you are crazy, you can write a patch that uses inotify and a ropendir/eaddir cache of the dir listing to avoid the system calls for .pm misses. Perhaps if the directory doesn't exist, no .pm'es in it can exist either, until a .pm "happens" to create/vivify the @INC dir that doesn't exist, when the .pm is loaded, so inotify can monitor the last (right most) valid directory in each @INC path to see if the next deep dir is created and therefore ther eis any chance of a .pm appearing in that previously non-existant dir.

strict warnings XSLoader and Exporter .pms were rewritten in XS and static linked into cperl binary in cperl, that eliminated the IO and startup overheads you are claiming. It is unimaginable any production perl app wouldn't use those 4 .pm'es EVERYTIME in EVERY process.

 		 	   		  
Thread Previous | Thread Next


nntp.perl.org: Perl Programming lists via nntp and http.
Comments to Ask Bjørn Hansen at ask@perl.org | Group listing | About