Re: [RfC] vtable->dump

On Fri, Sep 05, 2003 at 09:34:04AM -0400, Dan Sugalski wrote:
> On Fri, 5 Sep 2003, Peter Haworth wrote:

> > With the seen hash approach, I wouldn't expect the hash itself to take
> > nearly as much space as the frozen structure; 
> 
> My experience tracing variable graphs in Perl 5 indicates otherwise. 

I think it may be worth listening to Dan on this one.

Appended patch to Storable implements an option to count the size of
the seen hashes. (There seem to be two, one for objects, one for classes)
It's disabled by default - compile Storable with -DDEBUGME or change the
#if 0 at line 23 of Storable.xs to enable. You'll also need Devel::Size
installed.

Taking a relatively complex structure, such as the POSIX exports:

$ /usr/local/bin/perl5.8.0 -Mblib -MStorable -MPOSIX -MDevel::Size -le 'package POSIX; foreach (\@EXPORT, \@EXPORT_OK, \%EXPORT_TAGS) { print Devel::Size::total_size $_; print length Storable::freeze $_; print $Storable::SEENSIZE; print ""}'

reformatted, this give:

		size in memory	    size frozen		seen hash size
@EXPORT		22680		    5477		19068
@EXPORT_OK	1943		    422			1864
%EXPORT_TAGS	23686		    5963		20472


So it seems that the seen hash is about 3 to 4 times the size of the frozen
object, and comparable in size to the original structure. I was surprised
by these results - I had taken the "common sense" view that this hash
would be small

I'm wondering if a documented version of this patch should go into
Storable. It could be useful for experimenting.

Nicholas Clark

--- Storable.pm.orig	2003-07-29 08:06:35.000000000 +0100
+++ Storable.pm	2003-09-07 22:53:24.000000000 +0100
@@ -361,6 +361,19 @@ sub thaw {
 	return $self;
 }
 
+# We take advantage of autoload - this reference to SEENSIZE won't get noticed
+# by perl unless we're called, which doesn't happen unless the user has
+# already made reference to the variable.
+sub _add_seensize {
+  undef $Storable::SEENSIZE;
+  eval {
+    require Devel::Size;
+    foreach (@_) {
+      $SEENSIZE += Devel::Size::size ($_) if defined $_;
+    }
+  };
+  warn $@ if $@;
+}
 1;
 __END__
 
--- Storable.xs.orig	2003-09-05 19:42:41.000000000 +0100
+++ Storable.xs	2003-09-07 23:05:28.000000000 +0100
@@ -1181,10 +1181,6 @@ static void init_store_context(
 	 *
 	 * It is reported fixed in 5.005, hence the #if.
 	 */
-#if PERL_VERSION >= 5
-#define HBUCKETS	4096				/* Buckets for %hseen */
-	HvMAX(cxt->hseen) = HBUCKETS - 1;	/* keys %hseen = $HBUCKETS; */
-#endif
 
 	/*
 	 * The `hclass' hash uses the same settings as `hseen' above, but it is
@@ -1197,7 +1193,18 @@ static void init_store_context(
 	cxt->hclass = newHV();			/* Where seen classnames are stored */
 
 #if PERL_VERSION >= 5
-	HvMAX(cxt->hclass) = HBUCKETS - 1;	/* keys %hclass = $HBUCKETS; */
+#ifdef DEBUGME
+	/* Only pre-stretch the buckets if we're not interested in sizes.
+	   You may want to tweak this to still stretch the buckets even
+	   if we are interested in size.  */
+	if (!perl_get_sv("Storable::SEENSIZE", FALSE))
+#endif
+	  {
+#define HBUCKETS	4096			/* Buckets for %hseen */
+
+	    HvMAX(cxt->hseen) = HBUCKETS - 1;	/* keys %hseen = $HBUCKETS; */
+	    HvMAX(cxt->hclass) = HBUCKETS - 1;	/* keys %hclass = $HBUCKETS; */
+	  }
 #endif
 
 	/*
@@ -1250,6 +1257,28 @@ static void clean_store_context(stcxt_t 
 			HeVAL(he) = &PL_sv_undef;
 	}
 
+#ifdef DEBUGME
+	if (perl_get_sv("Storable::SEENSIZE", FALSE)) {
+	  /* User would like a count of the seen size.  */
+	  dSP;
+
+	  ENTER;
+	  SAVETMPS;
+
+	  PUSHMARK(SP);
+	  XPUSHs(cxt->hseen ? sv_2mortal(newRV_inc((SV *)cxt->hseen))
+		 : &PL_sv_undef);
+	  XPUSHs(cxt->hclass ? sv_2mortal(newRV_inc((SV *)cxt->hclass))
+					   : &PL_sv_undef);
+	  PUTBACK;
+
+	  call_pv("Storable::_add_seensize", G_DISCARD);
+
+	  FREETMPS;
+	  LEAVE;
+  	}
+#endif
+
 	/*
 	 * And now dispose of them...
 	 *

• Re: [RfC] vtable->dump by Nicholas Clark