Re: Two further features, one definitely needed for survival, otherlikely needed.

Front page | perl.perl5.porters | Postings from May 2021

Re: Two further features, one definitely needed for survival, otherlikely needed.

From:

Nicholas Clark

Date:

May 24, 2021 10:21

Subject:

Re: Two further features, one definitely needed for survival, otherlikely needed.

Message ID:

20210524102137.GC16703@etla.org

On Sun, May 23, 2021 at 10:00:39PM +0200, Tomasz Konojacki wrote:
> On Sun, 23 May 2021 12:33:58 -0700
> L A Walsh <astara@tlinx.org> wrote:

This is a sensible suggestion, but

1) I'm not sure whether we could even get this right if we wanted to
2) The opportunity cost of attempting it is too high

We're not gong to do it.

> > 1) Making future perl-libs backward compatible so programs don't need
> > rebuilding of their binary parts.
> > 
> > Precedent: libc (glibc).  Apparently it has been the case for some time,
> > but as each new version of glibc is introduced, brought about by some
> > incompatibility in the new API, the old API's are listed as being
> > satisfied within the new library.  I.e. programs built against
> > glibc-X are guaranteed to run with glibc-X+1, +2, +3...
> > 
> > This would imply that new versions of libperl-5.x would be useable by
> > programs built against earlier versions.  This would be a forward
> > compatibility from some specific perl version, going forward.
> > 
> > It may be that it would start with 5.34, such that progs built
> > against 5.34 work with perl-lib 5.36.  Going back before 5.34 would
> > be a separate issue on a version/by/version basis.
> 
> This isn't happening. Maintaining binary compatibility between *minor*
> releases is already problematic, see [1] for a recent example, so we
> *definitely* aren't going to extend our guarantees even further.

We used to do (roughly) this two decades ago, but in a limited fashion.
You could build (say) 5.004 the default way, or you could opt to build it
compatible with 5.003. *But* your options for building 5.005 were only
"current" or "like 5.004" - there was no indefinite guarantee, and it was
only intended as a transition system.

Also both "perl" and "CPAN" were much smaller then (how much API was exposed,
and how much code was consuming it, and relying on it)

IIRC 5.8.0 didn't offer this because it was not possible to implement the
needed internal improvements in any way that could be wedged into 5.6-land,
so the feature was dropped. To my knowledge, no-one has asked for it since,
and no-one had really missed it.

As Tomasz says, Maintaining binary compatibility between *minor* releases is
hard. I was doing it for most of maint-5.8. You have to look at every change
not just with "is this correct?" "does this add regressions?" but also
"is this compatible with code compiled against the previous version(s)"

This means obvious things like:

* you can't re-order structs or change the types of any members

but subtle things:

* you can't add anything at the *end* of structs, if that struct is embedded
  into any other struct. (Strictly, "at all", if that struct is exposed to
  CPAN code that could do this)

Those sort of constrains would prevent all sorts of runtime optimisations
that we have made. For example the large amount of structure re-ordering and
rationalisation that I did between 5.8.0 and 5.10.0, which reduced both
memory usage and CPU cache lookup.

Thirdly, my observations both of how OS distributions seem to work, and
every firm that I've worked at that has built its own perl from source:

It seems that even the work of trying to maintain binary compatibility
*within* minor versions is mostly wasted. For any given version, OSes seem
to stick on the exact x.y.z version that they started on, and then
"backport" patches from our x.y.z+1 (etc) versions to their "x.y.z",
without changing its "x.y.z", or relying on adding earlier versions to @INC.

Firms I've observed are even more conservative - they don't tend to backport
*any* patches, treat an upgrade to x.y.z+1 or x.y+2.* with equal
trepidation.

Of comparable languages (dynamic programming languages implemented in C as
monolithic compilers/interpreters exposing a large C extension API) I
think that

1) only Python has any concept of a stable A*B*I
2) that's also a much more limited set of calls than the general API
3) Python's API generally is way better and way cleaner than ours

We're not in the situation that (say) glibc is - we are not supporting a
compiled language where folks *ship* binaries, and many end users don't
have the source code to the programs that they run.

Doing this well - well enough to be trustworthy - requires a lot of human
judgement calls, from folks with particular skills. We don't have many
people who can do that. If they are tied up doing this, they can't add
features (or fix bugs) - improvements which far more people benefit from.

Not doing this generally means that computers have to work harder,
(re)compiling more code. Computers scale better than humans, particularly
volunteers.

So we can't offer an stable ABI.

Nicholas Clark