From: Leon Timmermans On Thu, Jan 3, 2013 at 5:38 PM, Gabor Szabo <gabor@szabgab.com> wrote:
> John,
>
> just a misunderstanding.
>
> repository => 'git://github.com/jgamble/Math-Polynomial-Solve.git',
>
> works
>
> repository => {
> url => 'git://github.com/jgamble/Math-Polynomial-Solve.git',
> web => 'http://github.com/jgamble/Math-Polynomial-Solve',
> type => 'git',
> }
>
> does NOT. (but it would be nice if it did)
>
> Just as Leon wrote.

What we really need is a meta_merge2 key. Or maybe a repository key.
Or probably both would be nice to have.

Leon

From: Gabor Szabo John,

just a misunderstanding.

repository => 'git://github.com/jgamble/Math-Polynomial-Solve.git',

works

repository => {
url => 'git://github.com/jgamble/Math-Polynomial-Solve.git',
web => 'http://github.com/jgamble/Math-Polynomial-Solve',
type => 'git',
}

does NOT. (but it would be nice if it did)

Just as Leon wrote.

Gabor

From: John M. Gamble On Thu, January 3, 2013 9:44 am, Gabor Szabo wrote:
> On Thu, Jan 3, 2013 at 3:18 PM, Leon Timmermans <fawaka@gmail.com> wrote:
>> On Thu, Jan 3, 2013 at 12:04 PM, Gabor Szabo <gabor@szabgab.com> wrote:
>>> After getting a few comments I found out that beside the link to the
>>> repository one could also include the type of
>>> the repository (git/svn/etc...), but as far as I can see only
>>> META.json holds that information.
>>> At least in the distributions I checked only META.json had this.
>>> All these distributions were generated using Dist::Zilla.
>>>
>>> So I wonder how can ask Module::Build to generate META.json as well,
>>> and how can I tell it
>>> to include the other fields of the repository as well?
>>
>> Since version 0.3800 Module::Build generates valid META.json, but it's
>> actually upconverted meta 1.4, so it won't contain any of the new
>> fields. This is a known bug. I would welcome someone adding proper
>> meta 2.0 support, but it's fairly low on my own todo list to be
>> honest. MakeMaker has the same issue.
>>
>> Leon
>
> Thanks.
>
> I checked on a distribution that is using MB and it created the
> META.json for me too. ++!
>
> Interestingly Sam Graham posted a link to his article where he shows
> how he did this
> 2 years ago:
> http://www.illusori.co.uk/blog/2013/01/03/setting-repository-with-module-build.html

Looking at it, it looks okay to me (although it looks like a cut-and-paste
example, as the only keys he has unnecessarily quoted are the resource and
repository keys).

For what it's worth, here's a link to my Build.PL, which successfully adds
the repository link:
<https://github.com/jgamble/Math-Polynomial-Solve/blob/master/Build.PL>

>
> I tried the same using MB 0.4003 but got the following error:
>
> Error is: Invalid metadata structure. Errors: 'HASH(0xfcde50)' for
> 'repository' does not have a URL scheme (resources -> repository)
> [Validation: 1.4]

What URL did you have? Maybe there's a legitimate error there.

-john

From: Gabor Szabo On Thu, Jan 3, 2013 at 3:18 PM, Leon Timmermans <fawaka@gmail.com> wrote:
> On Thu, Jan 3, 2013 at 12:04 PM, Gabor Szabo <gabor@szabgab.com> wrote:
>> After getting a few comments I found out that beside the link to the
>> repository one could also include the type of
>> the repository (git/svn/etc...), but as far as I can see only
>> META.json holds that information.
>> At least in the distributions I checked only META.json had this.
>> All these distributions were generated using Dist::Zilla.
>>
>> So I wonder how can ask Module::Build to generate META.json as well,
>> and how can I tell it
>> to include the other fields of the repository as well?
>
> Since version 0.3800 Module::Build generates valid META.json, but it's
> actually upconverted meta 1.4, so it won't contain any of the new
> fields. This is a known bug. I would welcome someone adding proper
> meta 2.0 support, but it's fairly low on my own todo list to be
> honest. MakeMaker has the same issue.
>
> Leon

Thanks.

I checked on a distribution that is using MB and it created the
META.json for me too. ++!

Interestingly Sam Graham posted a link to his article where he shows
how he did this
2 years ago:
http://www.illusori.co.uk/blog/2013/01/03/setting-repository-with-module-build.html

I tried the same using MB 0.4003 but got the following error:

Error is: Invalid metadata structure. Errors: 'HASH(0xfcde50)' for
'repository' does not have a URL scheme (resources -> repository)
[Validation: 1.4]

Was the feature removed?

regards
Gabor

From: Leon Timmermans On Thu, Jan 3, 2013 at 12:04 PM, Gabor Szabo <gabor@szabgab.com> wrote:
> After getting a few comments I found out that beside the link to the
> repository one could also include the type of
> the repository (git/svn/etc...), but as far as I can see only
> META.json holds that information.
> At least in the distributions I checked only META.json had this.
> All these distributions were generated using Dist::Zilla.
>
> So I wonder how can ask Module::Build to generate META.json as well,
> and how can I tell it
> to include the other fields of the repository as well?

Since version 0.3800 Module::Build generates valid META.json, but it's
actually upconverted meta 1.4, so it won't contain any of the new
fields. This is a known bug. I would welcome someone adding proper
meta 2.0 support, but it's fairly low on my own todo list to be
honest. MakeMaker has the same issue.

Leon

From: Gabor Szabo Hi,

recently I wrote an article how people could include a link to their
version control system in the META.yml file.
http://perl5maven.com/how-to-add-link-to-version-control-system-of-a-cpan-distributions

After getting a few comments I found out that beside the link to the
repository one could also include the type of
the repository (git/svn/etc...), but as far as I can see only
META.json holds that information.
At least in the distributions I checked only META.json had this.
All these distributions were generated using Dist::Zilla.

So I wonder how can ask Module::Build to generate META.json as well,
and how can I tell it
to include the other fields of the repository as well?

regards
Gabor

From: jvromans [Quoting Philippe Bruhat (BooK), on January 2 2013, 01:23, in "Re: How To Build A P"]
> One issue I had when trying to store distributions with
> Git::CPAN::Hook is that if a file does not change between two
> versions of a distribution, then git won't "detect" it.

This doesn't look like a problem if the approach is to checkout a
particular tag corresponding to a specific install.

-- Johan
http://johan.vromans.org/seasons_greetings.html

From: demerphq On 31 December 2012 19:38, Leon Timmermans <fawaka@gmail.com> wrote:
> On Mon, Dec 31, 2012 at 6:50 PM, Jan Dubois <jand@activestate.com> wrote:
>> Mostly I would prohibit sharing of directories between Perl installations,
>> and even within a single installation, the sharing of directories between
>> install locations.
>>
>> E.g. the default configuration right now has $Config{installbin} and
>> $Config{installsitebin} pointing to the the same directory. This means that
>> if you install ExtUtils::ParseXS from CPAN, you end up with the new version
>> of the module in $Config{installsitelib}, but the xsubpp script installed
>> into $Config{installsitebin} will overwrite the core version already in
>> $Config{installbin} because they are the same directory.
>>
>> This means it is now impossible to remove the ExtUtils::ParseXS module from
>> the "site" install location and reverting to the core version.
>>
>> Even if you don't care about "delete" functionality in your package manager,
>> you may still want to preserve the integrity of core install. Otherwise it
>> is possible that the package manager updates a package it relies upon itself
>> that breaks the package manager. Then it is impossible to fix this situation
>> for a regular user without doing a complete reinstall of Perl itself.
>>
>> For this reason the ActivePerl package manager explicitly removes the "site"
>> directories from @INC and only uses the modules originally included in the
>> distribution.
>
> I think that would clash with most vendor distributed perls (or at
> least it does with both Debian and Red Hat). It would be nice if this
> system was instead able to integrate with them instead of them nuking
> it to prevent users from doing something stupid.

FWIW I think that Perl should use one install format and the distros
should not fight that.

IMO a lot of MakeMakers problems come from trying to please too many
people and ending up pleasing no one.

Yves

--
perl -Mre=debug -e "/just|another|perl|hacker/"

From: Philippe Bruhat On Thu, Dec 20, 2012 at 05:48:10PM +0100, Johan Vromans wrote:
> Tim Bunce <Tim.Bunce@pobox.com> writes:
>
> > A separate install database for each install location seems like the
> > only workable approach.
>
> Store the complete distribution in a git repository?
>

One issue I had when trying to store distributions with Git::CPAN::Hook is
that if a file does not change between two versions of a distribution,
then git won't "detect" it.

That was an issue for me, as I tried to create special tree objects for
each distribution, so that "install this version of the distribution"
would basically mean "apply the patch that creates the whole tree". This
does not work if the tree does not contain files that haven't changed
between versions.

Depending on what you meant, that could be an issue for you too.

--
Philippe Bruhat (BooK)

When you double-cross a friend, you triple-cross yourself.
(Moral from Groo The Wanderer #8 (Epic))

From: Lyle I submitted a bug to rjbs about one of his modules not building on
Windows. He's kicked back that it's actually a Module::Build error.
Looking at the build log, it appears he's correct:
http://ppm4.activestate.com/MSWin32-x86/5.16/1600/R/RJ/RJBS/CGI-Application-Server-0.062.d/log-20120819T121907.txt
Then I would presume that this is affecting a lot of modules?


Lyle

-------- Original Message --------
Subject: [rt.cpan.org #82306] Not building for Perl 5.16 on Windows
Date: Tue, 1 Jan 2013 15:13:41 -0500
From: Ricardo Signes via RT <bug-CGI-Application-Server@rt.cpan.org>
Reply-To: bug-CGI-Application-Server@rt.cpan.org
To: COSMICNET@cpan.org



<URL: https://rt.cpan.org/Ticket/Display.html?id=82306 >

This bug is a failure of Module::Build to compile.

It is related to CGI::Application::Server only in that its Build.PL loads Module::Build.

--
rjbs

From: Leon Timmermans On Mon, Dec 31, 2012 at 6:50 PM, Jan Dubois <jand@activestate.com> wrote:
> Mostly I would prohibit sharing of directories between Perl installations,
> and even within a single installation, the sharing of directories between
> install locations.
>
> E.g. the default configuration right now has $Config{installbin} and
> $Config{installsitebin} pointing to the the same directory. This means that
> if you install ExtUtils::ParseXS from CPAN, you end up with the new version
> of the module in $Config{installsitelib}, but the xsubpp script installed
> into $Config{installsitebin} will overwrite the core version already in
> $Config{installbin} because they are the same directory.
>
> This means it is now impossible to remove the ExtUtils::ParseXS module from
> the "site" install location and reverting to the core version.
>
> Even if you don't care about "delete" functionality in your package manager,
> you may still want to preserve the integrity of core install. Otherwise it
> is possible that the package manager updates a package it relies upon itself
> that breaks the package manager. Then it is impossible to fix this situation
> for a regular user without doing a complete reinstall of Perl itself.
>
> For this reason the ActivePerl package manager explicitly removes the "site"
> directories from @INC and only uses the modules originally included in the
> distribution.

I think that would clash with most vendor distributed perls (or at
least it does with both Debian and Red Hat). It would be nice if this
system was instead able to integrate with them instead of them nuking
it to prevent users from doing something stupid.

Leon

From: Jan Dubois On Fri, Dec 28, 2012 at 4:18 PM, Leon Timmermans <fawaka@gmail.com> wrote:

> > Anyways, I just wanted to say that without putting some restrictions on
> how
> > modules (and corresponding scripts) can be installed, creating a package
> > manager would seem to get even more complex than ExtUtils::Makemaker...
> :)
>
> What kind of restrictions are you thinking about exactly?
>

Mostly I would prohibit sharing of directories between Perl installations,
and even within a single installation, the sharing of directories between
install locations.

E.g. the default configuration right now has $Config{installbin} and
$Config{installsitebin} pointing to the the same directory. This means that
if you install ExtUtils::ParseXS from CPAN, you end up with the new version
of the module in $Config{installsitelib}, but the xsubpp script installed
into $Config{installsitebin} will overwrite the core version already in
$Config{installbin} because they are the same directory.

This means it is now impossible to remove the ExtUtils::ParseXS module from
the "site" install location and reverting to the core version.

Even if you don't care about "delete" functionality in your package
manager, you may still want to preserve the integrity of core install.
Otherwise it is possible that the package manager updates a package it
relies upon itself that breaks the package manager. Then it is impossible
to fix this situation for a regular user without doing a complete reinstall
of Perl itself.

For this reason the ActivePerl package manager explicitly removes the
"site" directories from @INC and only uses the modules originally included
in the distribution.

Cheers,
-Jan

From: Leon Timmermans On Fri, Dec 28, 2012 at 2:44 AM, Jan Dubois <jand@activestate.com> wrote:
> I wonder if it isn't time to deprecate all the complex install combinations
> that may have made sense when hard disks was rather limited.
>
> In ActivePerl we enforce a pretty simplified install layout to be able to
> create an intuitive package manager:
>
> no sharing of directories between different versions
> every install area has its own bin, etc, lib, and html subdirectories
> the etc subdirectory records packages installed in that location

I think that ship already sailed, if only because different
distributions have very different layouts.

> Anyways, I just wanted to say that without putting some restrictions on how
> modules (and corresponding scripts) can be installed, creating a package
> manager would seem to get even more complex than ExtUtils::Makemaker... :)

What kind of restrictions are you thinking about exactly?

Leon

From: Jan Dubois ---------- Forwarded message ----------
From: Jan Dubois <jand@activestate.com>
Date: Thu, Dec 27, 2012 at 5:40 PM
Subject: Re: How To Build A Perl Package Database
To: Leon Timmermans <fawaka@gmail.com>


On Thu, Dec 20, 2012 at 2:42 AM, Leon Timmermans <fawaka@gmail.com> wrote:

> On Mon, Dec 17, 2012 at 9:36 AM, Tim Bunce <Tim.Bunce@pobox.com> wrote:
> > A separate install database for each install location seems like the only
> > workable approach.
>
> One minor complication of that is the strictest sense an "install
> location" isn't all that well defined. Or perhaps I should say every
> dist has 8 install locations with unspecified relationship on the
> filesystem (lib, arch, bin, script, libdoc, bindoc, libhtml, binhtml).
> In practice you may want to use lib *and* arch (they are never used
> simultaneously, but stuff in lib may be shared over different perl
> installations, contrary to arch which should be for one specific
> install).
>

I wonder if it isn't time to deprecate all the complex install combinations
that may have made sense when hard disks was rather limited.

In ActivePerl we enforce a pretty simplified install layout to be able to
create an intuitive package manager:

- no sharing of directories between different versions
- every install area has its own bin, etc, lib, and html subdirectories
- the etc subdirectory records packages installed in that location

Here is how PPM looks (on my Mac, but it is rather similar on Windows or
Unix systems too):
$ ppm area
┌────────┬──────┬────────────────────────────────────────┐
│ name │ pkgs │ lib │
├────────┼──────┼────────────────────────────────────────┤
│ user* │ 38 │ /Users/jan/Library/ActivePerl-5.16/lib │
│ (site) │ n/a │ /usr/local/ActivePerl-5.16/site/lib │
│ (perl) │ 245 │ /usr/local/ActivePerl-5.16/lib │
└────────┴──────┴────────────────────────────────────────┘

$ ls /Users/jan/Library/ActivePerl-5.16
bin etc html lib

$ ppm query XML
┌──────────────────────┬─────────┬─────────────────────────────────────────────────────────┬──────┐
│ name │ version │
abstract │ area │
├──────────────────────┼─────────┼─────────────────────────────────────────────────────────┼──────┤
│ XML-NamespaceSupport │ 1.11 │ a simple generic namespace support
class │ user │
│ XML-Parser │ 2.41-r1 │ A perl module for parsing XML
documents │ perl │
│ XML-SAX │ 0.99 │ Simple API for
XML │ user │
│ XML-SAX-Base │ 1.08 │ Base class for SAX Drivers and
Filters │ user │
│ XML-Simple │ 2.20 │ Easily read/write XML (esp config
files) │ perl │
│ XML-Stream │ 1.23 │ Creates an XML Stream connection and
parses return data │ user │
└──────────────────────┴─────────┴─────────────────────────────────────────────────────────┴──────┘
(6 packages installed matching 'XML')

$ ppm files XML-Stream
/Users/jan/Library/ActivePerl-5.16/lib/XML/Stream.pm
/Users/jan/Library/ActivePerl-5.16/lib/XML/Stream/IO/Select/Win32.pm
/Users/jan/Library/ActivePerl-5.16/lib/XML/Stream/Namespace.pm
/Users/jan/Library/ActivePerl-5.16/lib/XML/Stream/Node.pm
/Users/jan/Library/ActivePerl-5.16/lib/XML/Stream/Parser.pm
/Users/jan/Library/ActivePerl-5.16/lib/XML/Stream/Parser/DTD.pm
/Users/jan/Library/ActivePerl-5.16/lib/XML/Stream/Tree.pm
/Users/jan/Library/ActivePerl-5.16/lib/XML/Stream/XPath.pm
/Users/jan/Library/ActivePerl-5.16/lib/XML/Stream/XPath/Op.pm
/Users/jan/Library/ActivePerl-5.16/lib/XML/Stream/XPath/Query.pm
/Users/jan/Library/ActivePerl-5.16/lib/XML/Stream/XPath/Value.pm
/Users/jan/Library/ActivePerl-5.16/lib/auto/XML/Stream/.packlist

$ ppm remove XML-NamespaceSupport
XML-NamespaceSupport: required by XML-SAX
ppm remove failed: No packages uninstalled

Having a separate perl/bin and perl/site/bin and perl/vendor/bin is
somewhat inconvenient for adding things to the $PATH, but it makes it
possible to install an updated core package into the site directory, and
later uninstall it without breaking the original core version. We don't use
perl/vendor but instead merge all pre-installed packages into the core
directories to keep $PATH a little shorter.

Anyways, I just wanted to say that without putting some restrictions on how
modules (and corresponding scripts) can be installed, creating a package
manager would seem to get even more complex than ExtUtils::Makemaker... :)

Cheers,
-Jan

From: Tim Bunce On Thu, Dec 20, 2012 at 11:42:06AM +0100, Leon Timmermans wrote:
> On Mon, Dec 17, 2012 at 9:36 AM, Tim Bunce <Tim.Bunce@pobox.com> wrote:
> > A separate install database for each install location seems like the only
> > workable approach.
>
> One minor complication of that is the strictest sense an "install
> location" isn't all that well defined. Or perhaps I should say every
> dist has 8 install locations with unspecified relationship on the
> filesystem (lib, arch, bin, script, libdoc, bindoc, libhtml, binhtml).
> In practice you may want to use lib *and* arch (they are never used
> simultaneously, but stuff in lib may be shared over different perl
> installations, contrary to arch which should be for one specific
> install).

Good point.
Strengthening the definition of an "install location" would be good.
Supporting only the common subset of possible layouts seems reasonable.

Tim.

From: Johan Vromans Tim Bunce <Tim.Bunce@pobox.com> writes:

> A separate install database for each install location seems like the
> only workable approach.

Store the complete distribution in a git repository?

-- Johan

From: Leon Timmermans On Mon, Dec 17, 2012 at 9:36 AM, Tim Bunce <Tim.Bunce@pobox.com> wrote:
> A separate install database for each install location seems like the only
> workable approach.

One minor complication of that is the strictest sense an "install
location" isn't all that well defined. Or perhaps I should say every
dist has 8 install locations with unspecified relationship on the
filesystem (lib, arch, bin, script, libdoc, bindoc, libhtml, binhtml).
In practice you may want to use lib *and* arch (they are never used
simultaneously, but stuff in lib may be shared over different perl
installations, contrary to arch which should be for one specific
install).

Leon

From: Ask Bjørn Hansen
On Dec 17, 2012, at 9:36, Tim Bunce <Tim.Bunce@pobox.com> wrote:

> A separate install database for each install location seems like the only
> workable approach.

It seems to me that the database indeed will have to be[1] "per library root" and the tools using the database will need to know to do lookups everywhere and merge the results.


Ask

[1] Where "will have to" means "to fit *my* universe".

From: Alberto Simoes
>
> Thoughts?
>

Seems a good idea.
Anybody wiling to submit a TPF grant to work on this? O:-)

From: Tim Bunce On Sun, Dec 16, 2012 at 04:53:49PM -0800, Michael G Schwern wrote:
> On 2012.12.16 11:57 AM, Leon Timmermans wrote:
>
> >> * Where to put the database? What about non-standard install locations?

> >> Another is to have a separate install database for non-standard install
> >> locations.

A separate install database for each install location seems like the only
workable approach.

> >> This makes sense to me, but it brings in the sticky problem
> >> of having to merge install databases. Sticky, but still a SMOP. Once you
> >> have to implement merging anyway, it now makes sense to have an install
> >> database for each install location. One for core. One for vendor. One for
> >> perl. And one for each custom location. This has a lot of advantages to
> >> better fit how Perl layers module installs.
> >>
> >> * allows separation of permissions
> >> * allows queries of what's installed based on what's in @INC

Perhaps that could be taken one step further: one per installed distribution.

Then, what's kept at each install location is a cached summary of what's
installed below it. One that can be cross-checked against the individual
distribution 'databases' and rebuilt from it.

That seems more robust against various kinds of 'damage'.

> >> That second one is important. When a normal user queries the database, they
> >> want to get what's installed in the standard library location. When a
> >> local::lib user queries the database, they want to get what's installed in the
> >> standard library locations AND their own local lib.

I.e., the default view is "what's installed in my @INC".

> > The combination of these is problematic. You might upgrade EU::Install
> > in your local module path, but not have write permissions on the
> > system paths. In practice, we might have to support all our older
> > versions :-|
>
> Erg, good point. That very likely scenario is definitely going to require
> some thought.

*nods*

Here's where "one install database per distribution with a cache
database at the install location" offers another benefit.
The "per distribution install database" can be kept in a very simple
plain text format that targets readability and future-proofing,
while the "cache database at the install location" can target
performance.

If an install location has an incompatible version of the db,
the per distribution dbs could be read instead. That's slow but workable
and seems reasonable for that presumably uncommon case.
I can think of a few further options as well.

Tim.

From: Lyle On 17/12/2012 13:52, Johan Vromans wrote:
> Lyle <webmaster@cosmicperl.com> writes:
>
>> Michael is right. I deal with setting up Perl driven software are a
>> wide variety of systems. This often means setting up Perl itself
>> because the system doesn't have one, or to have a non system instance.
>> It can be a real pain, it shouldn't be, it should be easy and straight
>> forward for all. TIMTOWTDI, relying on the OS for package management
>> can be far from idea, or not even an option.
> I think it is neccesary to make a distinction between developers, power
> users, and desktop/end users. Developers and power users usually know
> how to deal with CPAN/local installs and package management. 'Command
> line' and 'root' do not have real secrets for them.

True, but even for power users managing Perl modules and dependencies
can feel needlessly long winded, and not uncommonly fraught with
complications. Clashes between versions, system packages, cpan installed
modules, missing deps, etc.

When one can spend what feels like an eternity installing deps for a
Catalyst app, knowing full well that things like Drupal can be installed
in minutes, doesn't do the image of Perl an favours.

> Desktop users, on the other hand, just use point and click on desktops
> and web sites, and things like 'Perl' and 'modules' do not mean anything
> significant. For these users, an application is just a single piece of
> software that gets installed with the click of a button. If it is a Perl
> application, this means you have to provide Perl and all necessary
> libraries and modules with the install kit[1]. Installable packages can
> be created with PAR, althoug personally I prefer Cava Packager
> (especially in combination with Citrus Perl). Other people may wish to
> use PerlApp.
>
> -- Johan
>
> [1] Unless you create installers for each and every specific platform
> the customers use -- which is a huge maintenance burden.

I started a design a project I called PerlSI a few of years back (Perl
Scriptable Installer), which aimed to remedy this among other things.
Unfortunately I started a part time degree and didn't get far with it.
Maybe next year I'll have chance to look at it again.


Lyle

From: Adam Kennedy Packlist 2.0

MYMETA + installed file details

?
On Dec 17, 2012 12:36 AM, "Tim Bunce" <Tim.Bunce@pobox.com> wrote:

> On Sun, Dec 16, 2012 at 04:53:49PM -0800, Michael G Schwern wrote:
> > On 2012.12.16 11:57 AM, Leon Timmermans wrote:
> >
> > >> * Where to put the database? What about non-standard install
> locations?
>
> > >> Another is to have a separate install database for non-standard
> install
> > >> locations.
>
> A separate install database for each install location seems like the only
> workable approach.
>
> > >> This makes sense to me, but it brings in the sticky problem
> > >> of having to merge install databases. Sticky, but still a SMOP.
> Once you
> > >> have to implement merging anyway, it now makes sense to have an
> install
> > >> database for each install location. One for core. One for vendor.
> One for
> > >> perl. And one for each custom location. This has a lot of
> advantages to
> > >> better fit how Perl layers module installs.
> > >>
> > >> * allows separation of permissions
> > >> * allows queries of what's installed based on what's in @INC
>
> Perhaps that could be taken one step further: one per installed
> distribution.
>
> Then, what's kept at each install location is a cached summary of what's
> installed below it. One that can be cross-checked against the individual
> distribution 'databases' and rebuilt from it.
>
> That seems more robust against various kinds of 'damage'.
>
> > >> That second one is important. When a normal user queries the
> database, they
> > >> want to get what's installed in the standard library location. When a
> > >> local::lib user queries the database, they want to get what's
> installed in the
> > >> standard library locations AND their own local lib.
>
> I.e., the default view is "what's installed in my @INC".
>
> > > The combination of these is problematic. You might upgrade EU::Install
> > > in your local module path, but not have write permissions on the
> > > system paths. In practice, we might have to support all our older
> > > versions :-|
> >
> > Erg, good point. That very likely scenario is definitely going to
> require
> > some thought.
>
> *nods*
>
> Here's where "one install database per distribution with a cache
> database at the install location" offers another benefit.
> The "per distribution install database" can be kept in a very simple
> plain text format that targets readability and future-proofing,
> while the "cache database at the install location" can target
> performance.
>
> If an install location has an incompatible version of the db,
> the per distribution dbs could be read instead. That's slow but workable
> and seems reasonable for that presumably uncommon case.
> I can think of a few further options as well.
>
> Tim.
>

From: Johan Vromans Lyle <webmaster@cosmicperl.com> writes:

> Michael is right. I deal with setting up Perl driven software are a
> wide variety of systems. This often means setting up Perl itself
> because the system doesn't have one, or to have a non system instance.
> It can be a real pain, it shouldn't be, it should be easy and straight
> forward for all. TIMTOWTDI, relying on the OS for package management
> can be far from idea, or not even an option.

I think it is neccesary to make a distinction between developers, power
users, and desktop/end users. Developers and power users usually know
how to deal with CPAN/local installs and package management. 'Command
line' and 'root' do not have real secrets for them.

Desktop users, on the other hand, just use point and click on desktops
and web sites, and things like 'Perl' and 'modules' do not mean anything
significant. For these users, an application is just a single piece of
software that gets installed with the click of a button. If it is a Perl
application, this means you have to provide Perl and all necessary
libraries and modules with the install kit[1]. Installable packages can
be created with PAR, althoug personally I prefer Cava Packager
(especially in combination with Citrus Perl). Other people may wish to
use PerlApp.

-- Johan

[1] Unless you create installers for each and every specific platform
the customers use -- which is a huge maintenance burden.

From: demerphq On 17 December 2012 01:53, Michael G Schwern <schwern@pobox.com> wrote:
> On 2012.12.16 11:57 AM, Leon Timmermans wrote:
>> I can agree with all of that. Actually, starting a discussion about
>> this was on my todo-list for the last QA hackathon but I didn't get
>> around to it. Ideally, it should replace not only packlists but also
>> perllocal
>
> I was thinking about what you said about packlists, and I wonder how much
> information one could scrape out of them. Would it be enough to reconstruct
> at least that a group of files belongs to a release? That would be enough to
> be able to fully uninstall a requested module. For example, if the user asks
> to uninstall ExtUtils::MakeMaker the database could have seen that
> ExtUtils/MakeMaker.pm was in a packlist together with ExtUtils/MM_Unix.pm and
> so on and uninstall them. Probably given their original purpose was to
> provide an uninstaller.
>
> Also what's with this .meta directory I see popping up? I missed a memo.
>
>
>>> This is all totally doable, and efficient enough, with a small pile of DBM
>>> files and Storable. Where to put the database is a bit more complicated, see
>>> the list of open problems below.
>>
>> Given that Storable's format isn't forward-compatible, something more
>> stable such as JSON would be more appropriate.
>
> That's a good point about Storable. JSON requires a dependency.
> ExtUtils::Install could bundle JSON::PP, but it would be simpler to use
> Data::Dumper. It makes de/serialization faster and simpler. The main
> disadvantage is its only readable by Perl, but that's ok since this pile of
> DBM files will be opaque to everything but the Perl API. Too much of a mess
> to contemplate otherwise.

IMO the question is whether you want the data human readable or not.

If you dont care then use Sereal as a replacement for Storable. Same
feature set pretty much except it is faster and produces smaller
output.

Yves

--
perl -Mre=debug -e "/just|another|perl|hacker/"

From: Lyle On 17/12/2012 00:43, Michael G Schwern wrote:
> ... Perl is a cross platform, cross environment
> language. Not all the environments we work with have good package managers.
> Off the top of my head: Windows, a huge, hidden number of our users, has
> nothing usable and the Solaris package manger is a joke. Not all users in
> those environments have access to those package managers. Not everybody wants
> to use them to manage Perl. For example, cpan2rpm is a commendable effort but
> also a nightmare...
> ... more excellent points...
> Most shops that make it work have a Perl expert on hand. They should not need
> one.

Michael is right. I deal with setting up Perl driven software are a wide
variety of systems. This often means setting up Perl itself because the
system doesn't have one, or to have a non system instance. It can be a
real pain, it shouldn't be, it should be easy and straight forward for
all. TIMTOWTDI, relying on the OS for package management can be far from
idea, or not even an option.


Lyle

From: Leon Timmermans On Mon, Dec 17, 2012 at 1:53 AM, Michael G Schwern <schwern@pobox.com> wrote:
> I was thinking about what you said about packlists, and I wonder how much
> information one could scrape out of them. Would it be enough to reconstruct
> at least that a group of files belongs to a release? That would be enough to
> be able to fully uninstall a requested module. For example, if the user asks
> to uninstall ExtUtils::MakeMaker the database could have seen that
> ExtUtils/MakeMaker.pm was in a packlist together with ExtUtils/MM_Unix.pm and
> so on and uninstall them. Probably given their original purpose was to
> provide an uninstaller.

You can use them to uninstall (I assume that's the reason why Debian
disables them for vendor packages). It can get a little messy when
modules are split or some such, but that's relatively rare anyway.

> Also what's with this .meta directory I see popping up? I missed a memo.

AFAIK that's cpanminus specific. AFAIK it stores meta information so
that carton can use it. Ask Miyagawa for the details.

> That's a good point about Storable. JSON requires a dependency.
> ExtUtils::Install could bundle JSON::PP, but it would be simpler to use
> Data::Dumper. It makes de/serialization faster and simpler. The main
> disadvantage is its only readable by Perl, but that's ok since this pile of
> DBM files will be opaque to everything but the Perl API. Too much of a mess
> to contemplate otherwise.

JSON::PP is already in modern perl releases, so it only requires a
dependency on older perls.

Leon

From: Michael G Schwern On 2012.12.16 11:57 AM, Leon Timmermans wrote:
> I can agree with all of that. Actually, starting a discussion about
> this was on my todo-list for the last QA hackathon but I didn't get
> around to it. Ideally, it should replace not only packlists but also
> perllocal

I was thinking about what you said about packlists, and I wonder how much
information one could scrape out of them. Would it be enough to reconstruct
at least that a group of files belongs to a release? That would be enough to
be able to fully uninstall a requested module. For example, if the user asks
to uninstall ExtUtils::MakeMaker the database could have seen that
ExtUtils/MakeMaker.pm was in a packlist together with ExtUtils/MM_Unix.pm and
so on and uninstall them. Probably given their original purpose was to
provide an uninstaller.

Also what's with this .meta directory I see popping up? I missed a memo.


>> This is all totally doable, and efficient enough, with a small pile of DBM
>> files and Storable. Where to put the database is a bit more complicated, see
>> the list of open problems below.
>
> Given that Storable's format isn't forward-compatible, something more
> stable such as JSON would be more appropriate.

That's a good point about Storable. JSON requires a dependency.
ExtUtils::Install could bundle JSON::PP, but it would be simpler to use
Data::Dumper. It makes de/serialization faster and simpler. The main
disadvantage is its only readable by Perl, but that's ok since this pile of
DBM files will be opaque to everything but the Perl API. Too much of a mess
to contemplate otherwise.


>> * Auto-upgrade to SQLite if ExtUtils::Install::DB::SQLite is installed.
>>
>> If a special module is installed we can offer SQLite support (or whatever) for
>> a more advanced database. At install time it would copy the existing DBM
>> system into its own database.
>>
>> In general, more functionality can be added as more optional (or bundled)
>> dependencies are available to the system. Through it all the basic DBM
>> database would continue to be redundantly maintained to provide a fallback
>> should those optional modules break or go away.
>
> Having a proper database would be really nice, but I'm not sure if
> it's going to be worth the hassle if we have a robust system already.

That's the great thing about Open Source and possibilities. You just have to
make the possibilities available and let someone surprise you!


>> * Where to put the database? What about non-standard install locations?
>>
>> $Config{archlib} would seem the obvious location, but it presents a
>> permissions problem. If a non-root user installs into their home
>> directory, you don't want them needing root to write to the installation
>> database. There's several ways to deal with this.
>>
>> One is to simply not record non-standard install locations, but this loses
>> data and punishes all those local::lib users out there.
>>
>> Another is to have a separate install database for non-standard install
>> locations. This makes sense to me, but it brings in the sticky problem
>> of having to merge install databases. Sticky, but still a SMOP. Once you
>> have to implement merging anyway, it now makes sense to have an install
>> database for each install location. One for core. One for vendor. One for
>> perl. And one for each custom location. This has a lot of advantages to
>> better fit how Perl layers module installs.
>>
>> * allows separation of permissions
>> * allows queries of what's installed based on what's in @INC
>>
>> That second one is important. When a normal user queries the database, they
>> want to get what's installed in the standard library location. When a
>> local::lib user queries the database, they want to get what's installed in the
>> standard library locations AND their own local lib.
>
> The combination of these is problematic. You might upgrade EU::Install
> in your local module path, but not have write permissions on the
> system paths. In practice, we might have to support all our older
> versions :-|

Erg, good point. That very likely scenario is definitely going to require
some thought.


>> Not perfect, but gets us off the ground. Its not a great database, but it
>> does the important job of recording the critical install-time data for later
>> use. Its implementable within the current system. It doesn't require a bunch
>> of dependencies, just one upgrade. It works with most existing module
>> releases. It solves a major design problem with the Perl module system.
>>
>> I think it's a Simple(?!) Matter Of Programming in ExtUtils::Install to get it
>> off the ground. IMO the most important bit of coordination is putting some
>> thought into what the basic database should look like so we don't have to
>> worry about complicated upgrades later.
>
> I'm not sure it's as simple as you make it sound, but it is a good
> idea nonetheless.

Many devils in the details. Glad you like it. Thanks for looking it over.


--
60. "The Giant Space Ants" are not at the top of my chain of command.
-- The 213 Things Skippy Is No Longer Allowed To Do In The U.S. Army
http://skippyslist.com/list/

From: Michael G Schwern On 2012.12.16 1:10 PM, Leon Timmermans wrote:
> On Sun, Dec 16, 2012 at 6:34 PM, Johan Vromans <jvromans@squirrel.nl> wrote:
>> Debian, and most other systems have decent package- and install
>> managers. *They* maintain the database with installed distributions,
>> releases and files. The only good approach for us is to play with them.
>>
>> So, an enhanced META.yaml or whatsoever may be a good idea, but *only*
>> to generate a deb control file, or rpm spec file, or innosetup file and
>> so on.
>>
>> It is no longer neccessary to handle everything ourselves. We're not
>> alone anymore.
>
> There are many ways to deploy stuff, not everyone uses rpm/deb, there
> are good reasons not to do so: for starters it assumes you have root
> privileges.

I agree with Johan that there are better ways to do this. I also agree with
Leon that they are not available to all users and operating systems.

I would much, much rather use an existing, reliable system than build our own.
Really really really! But Perl is a cross platform, cross environment
language. Not all the environments we work with have good package managers.
Off the top of my head: Windows, a huge, hidden number of our users, has
nothing usable and the Solaris package manger is a joke. Not all users in
those environments have access to those package managers. Not everybody wants
to use them to manage Perl. For example, cpan2rpm is a commendable effort but
also a nightmare.

In addition, there are some people who live on one OS and are happy and
comfortable with how that OS does things. And there are people who happily
jump from OS to OS and want to do things the Perl way. Its nice to be able to
sit down on any given machine, grab cpanm, perlbrew and local::lib and get
everything running smoothly.

OS packages are oriented towards one install per machine. Perl is per project
and/or per user. Furthermore, Perl is part of the operating system. Making
the needs of your project not conflict with the needs of the operating system
is tricky. There are ways to shoehorn OS packages to do these things, but it
doesn't work well, many admins don't know how to do that, and its different
from OS to OS.

If none of that convinces you, I'll say this: we've been advancing the "build
OS packages" route for years now and its never really worked. There's tons of
Perl shops out there with tangled, messed up Perl/CPAN installations who find
it difficult to upgrade in part because they can't replicate their
installation. My current client is one of them. My best practice for dealing
with big Perl installs is to make one giant OS package of a non-system perl
and all the necessary CPAN modules rather than making individual CPAN module
packages and their interdependencies come out right. This is kind of gross,
but its manageable.

Most shops that make it work have a Perl expert on hand. They should not need
one.

This is in no way in conflict with advancing cpan2package tools. If done
right the package database will make packaging Perl modules *easier*. Those
same hooks and APIs I talked about to allow for enhanced package database
functionality could also be used by OS build systems. The needs of the
package database will force modules to become more normalized and provide more
and better meta data. At worst, it won't make it any more difficult.

Open Source is not a zero sum game.


--
The interface should be as clean as newly fallen snow and its behavior
as explicit as Japanese eel porn.

From: Alberto Simões

On 16/12/12 21:23, Johan Vromans wrote:
> [Quoting Leon Timmermans, on December 16 2012, 22:10, in "Re: How To Build A P"]
>> There are many ways to deploy stuff, not everyone uses rpm/deb,
>> there are good reasons not to do so: for starters it assumes you
>> have root privileges.
>
> Reality has overtaken these ancient views for a long time already.
>
> Sudo wrappers help nonroot users.

In fact there is a few share of users using shared servers. But in the
other hand, there are a lot of users installing locally a Perl version
for each one of their project, knowing this way updating a module for
one of their projects will not ruin other projects running on the same
machine.

Therefore, restringing this magic to system-wide modules doesn't seem
fair, nor really useful.

> On Android and iOS, users are not root yet they install apps all of
> the time.

iOS is single user AFAIK. Android at the moment is single user as well
(with some new tries for multi-user). So not a big example.

My 2 cents,
ambs

From: jvromans [Quoting Leon Timmermans, on December 16 2012, 22:10, in "Re: How To Build A P"]
> There are many ways to deploy stuff, not everyone uses rpm/deb,
> there are good reasons not to do so: for starters it assumes you
> have root privileges.

Reality has overtaken these ancient views for a long time already.

Sudo wrappers help nonroot users.

On Android and iOS, users are not root yet they install apps all of
the time.

And so on.

Time to change views.

-- Johan

From: Leon Timmermans On Sun, Dec 16, 2012 at 6:34 PM, Johan Vromans <jvromans@squirrel.nl> wrote:
> Debian, and most other systems have decent package- and install
> managers. *They* maintain the database with installed distributions,
> releases and files. The only good approach for us is to play with them.
>
> So, an enhanced META.yaml or whatsoever may be a good idea, but *only*
> to generate a deb control file, or rpm spec file, or innosetup file and
> so on.
>
> It is no longer neccessary to handle everything ourselves. We're not
> alone anymore.

There are many ways to deploy stuff, not everyone uses rpm/deb, there
are good reasons not to do so: for starters it assumes you have root
privileges.

Leon

From: Leon Timmermans On Sat, Dec 15, 2012 at 11:59 PM, Michael G Schwern <schwern@pobox.com> wrote:
> We have a lot of serious problems because we lack a database of installed
> distributions, releases and files. There are serious problems with
> implementing one given A) the limitations of the standard Perl install and B)
> wedging it into existing systems. But I think I have a solution. Its similar
> to how meta data was slipped into the ecosystem without requiring authors to
> rewrite their releases or install a bunch of extra modules. It just happens
> as part of the normal CPAN module upgrade process.
>
> I've been thinking that a minimal package database could be created by putting
> some hooks into ExtUtils::Install::install(), which every Perl build system
> ultimately uses, to record what gets installed. That way when
> ExtUtils::Install is upgraded, the user gets a build database without
> upgrading everything else.
>
> This would be a fairly straight forward process at install time...
>
> 1) Copy everything to a temp directory
> 2) Record everything in that temp directory
> 3) Copy everything from temp into the real location
>
> You could probably optimize this by skipping the copy to temp and just have
> install() record stuff as it goes by, but this is the dumb, simple, robust way
> to do it.
>
> Storage is a problem. The only reliable "database" Perl ships with is DBM, an
> on disk hash, so we can't get too fancy. It might take several DBM files, but
> this is enough to record information and do simple queries. What are those
> queries?
>
> * What version of the database is this?
> * What distributions are installed?
> * What release of a distribution is installed?
> * What files are in that release?
> * What version is that release?
> * What location was a release installed into? (core, vendor, site, custom)
> * What are the checksums of those files?
>
> And the basic operations we need to support.
>
> * Add a release (ie. install).
> * Delete a release (and its files).
> * Delete an older version of a release (as part of install).
> * Delete an older version of a release, only if its in the same release
> location. This is so CPAN installs don't delete vendor installed modules.
> * Verify the files of a release.
> * List distributions/releases installed.
>
> It would also store the MYMETA data which gives us a lot of information (such
> as dependencies) for free.

I can agree with all of that. Actually, starting a discussion about
this was on my todo-list for the last QA hackathon but I didn't get
around to it. Ideally, it should replace not only packlists but also
perllocal

> This is all totally doable, and efficient enough, with a small pile of DBM
> files and Storable. Where to put the database is a bit more complicated, see
> the list of open problems below.

Given that Storable's format isn't forward-compatible, something more
stable such as JSON would be more appropriate.

> There's lots and lots and lots of additional information which could be stored
> and queries and operations to allow, but if we can get the basics working
> it'll allow a heap of new solutions. And I think this is a SMOP.
>
>
> Future possibilities include...
>
> * Auto-upgrade to SQLite if ExtUtils::Install::DB::SQLite is installed.
>
> If a special module is installed we can offer SQLite support (or whatever) for
> a more advanced database. At install time it would copy the existing DBM
> system into its own database.
>
> In general, more functionality can be added as more optional (or bundled)
> dependencies are available to the system. Through it all the basic DBM
> database would continue to be redundantly maintained to provide a fallback
> should those optional modules break or go away.

Having a proper database would be really nice, but I'm not sure if
it's going to be worth the hassle if we have a robust system already.

> * Upgrading the database.
>
> I'd like to put some thought into how things are laid out initially to avoid a
> lot of major revisions, and thought into what information should be recorded
> so its available later, but eventually we're going to want to change the
> "schema", such as it is with DBM files.
>
> I figure this can happen as part of upgrading ExtUtils::Install. It checks
> what version of the database you have and performs the necessary transforms to
> bring it up to the current version. We know how to do this, just have to keep
> it in mind and remember to implement it.
>
> * Where to put the database? What about non-standard install locations?
>
> $Config{archlib} would seem the obvious location, but it presents a
> permissions problem. If a non-root user installs into their home
> directory, you don't want them needing root to write to the installation
> database. There's several ways to deal with this.
>
> One is to simply not record non-standard install locations, but this loses
> data and punishes all those local::lib users out there.
>
> Another is to have a separate install database for non-standard install
> locations. This makes sense to me, but it brings in the sticky problem
> of having to merge install databases. Sticky, but still a SMOP. Once you
> have to implement merging anyway, it now makes sense to have an install
> database for each install location. One for core. One for vendor. One for
> perl. And one for each custom location. This has a lot of advantages to
> better fit how Perl layers module installs.
>
> * allows separation of permissions
> * allows queries of what's installed based on what's in @INC
>
> That second one is important. When a normal user queries the database, they
> want to get what's installed in the standard library location. When a
> local::lib user queries the database, they want to get what's installed in the
> standard library locations AND their own local lib.

The combination of these is problematic. You might upgrade EU::Install
in your local module path, but not have write permissions on the
system paths. In practice, we might have to support all our older
versions :-|

> Not perfect, but gets us off the ground. Its not a great database, but it
> does the important job of recording the critical install-time data for later
> use. Its implementable within the current system. It doesn't require a bunch
> of dependencies, just one upgrade. It works with most existing module
> releases. It solves a major design problem with the Perl module system.
>
> I think it's a Simple(?!) Matter Of Programming in ExtUtils::Install to get it
> off the ground. IMO the most important bit of coordination is putting some
> thought into what the basic database should look like so we don't have to
> worry about complicated upgrades later.

I'm not sure it's as simple as you make it sound, but it is a good
idea nonetheless.

Leon

From: Alberto Simões Hello

On 16/12/12 17:34, Johan Vromans wrote:
> So, an enhanced META.yaml or whatsoever may be a good idea, but *only*
> to generate a deb control file, or rpm spec file, or innosetup file and
> so on.

That is the problem. There are too many options out there. Can we
control all Linux distributions packaging systems?

And Windows, or even Mac OS X? For those there is no real option...

Finally, we are here for ages, and ee can't even have the more relevant
Perl modules correctly installed on major distributions (my honest opinion).

Alberto

From: Johan Vromans Michael G Schwern <schwern@pobox.com> writes:

> We have a lot of serious problems because we lack a database of
> installed distributions, releases and files.

No, that is not our problem.

Our problem is that we want to handle it ourselves. This may have been a
good approach in the dark ages, but nowadays there are better solutions.

For example, on Sat, 15 Dec 2012 14:15:24 -0800 you wrote:

> When it comes to packaging, my first thought is always WDDD? What Does
> Debian Do? They usually do it right.

Debian, and most other systems have decent package- and install
managers. *They* maintain the database with installed distributions,
releases and files. The only good approach for us is to play with them.

So, an enhanced META.yaml or whatsoever may be a good idea, but *only*
to generate a deb control file, or rpm spec file, or innosetup file and
so on.

It is no longer neccessary to handle everything ourselves. We're not
alone anymore.

-- Johan

From: Alberto Simões

On 15/12/12 23:00, Michael G Schwern wrote:
> On 2012.12.15 2:25 PM, Leon Timmermans wrote:
>>> We don't have that, but we should. Imma let you finish and not hijack your
>>> post to talk about that. And IMO we should continue to assume we don't have a
>>> package database until we actually do.
>>
>> We have packlists, so we can easily map a distribution to its files.
>> The reverse mapping can be generated from this, though this is
>> relatively expensive.
>
> Packlists are unreliable and have design problems. Some perl distributions
> (Debian) disable them. I wouldn't build anything around them.
>
> PS Sorry, I didn't mean to change the thread subject. I wound up writing my
> own post.

Your proposal makes sense but is more difficult (takes more time).

But it solves the issue, and other issues.

Alberto