develooper Front page | perl.perl5.porters | Postings from April 2019

We made a different security release and here's what we learned

Sawyer X
April 1, 2019 11:02
We made a different security release and here's what we learned
Message ID:
With a title like that, this must be an interesting read.

In late November 2018, we made two security-focused maintenance releases
(5.26.3, 5.28.1), addressing several CVEs. These maintenance releases
were done with a different release cycle than we usually do, and I would
like to explain why we did this and what we learned from it.

Back in April of 2018, we released security maintenance versions 5.26.2
and 5.24.4. We had been working on these for over a month before
releasing them. This included announcements to vendors and working with
them on merging and backporting the fixes to their vendor versions of
perl. (A lot of credit here goes to Tony Cook, who spearheaded these

Part of the work for a security-specific release is to determine a date
(by vendors and us) for releasing the security information. However, we
pushed our commits to blead before releasing the tarballs. This meant
that the code was out while everyone (us and the vendors) committed to
waiting on any announcements. The sum of these actions was that we have
effectively undermined our vendors.

We were approached by two vendors asking to understand this situation.
If they cannot release until we release a tarball, why are we pushing
fixes? We decided we need to make a definite change to the process of
release when the version is security-focused.

In the security discussion group, I raised these concerns by our vendors
in late March. We floated ideas for future handling of releases (when to
push commits and how), but we did not reach a firm conclusion.

When we were preparing 5.26.3 and 5.28.1, Tony Cook (the technical lead
on the release), Steve Hay (the release manager of maintenance
releases), and myself started the discussion again and selected what
looked like the best option to try this time, based on the input from
vendors and prior comments on the security discussion group.

We should have brought this discussion back to the security discussion
group instead of continuing it between us, but we did not. This was
wrong, and we apologize for that.

However, despite the discussion happening between us, we announced to
the security group on November 3rd the plan we decided upon. No
objections were raised between our announcement (Nov. 3rd) and RC1s
release on schedule (Nov. 8th).

The biggest problem with the plan was testing. This topic came up during
the first Perl 5 Core Hackathon, back in 2017. One idea that came up
from there is two-fold: Reach out to vendors and dedicated smokers.

In early November I reached out to vendors who raised concerns, and two
had stepped up. During that time, I also reached out to Andreas Koenig,
since he is (along with Slaven Rezic) Perl's biggest smoke-tester and my
go-to address on smoking. Andreas had offered to help in the past when
specific cases come up. Andreas then shared my request (and our
communication) publicly on the list in a thread I much regret having
behaved poorly in.

In this release cycle, we decided to withhold the commits from blead
until we make the release and only then push them.

What I maintain we did correctly:

The way we handled releases in the past were wrong and in simpler terms:
unfair toward our vendors. Trust and cooperation are hard to maintain if
we pull the rug from under the feet of our colleagues. I want to thank
Niko Tyni and the Debian team and Theo de Raadt from OpenBSD for
reaching out to us and holding us to a higher standard.

Changing our release cycle was the correct decision.

We had initially thought Andreas Koenig was not included in the security
discussion group. He plays a critical role in the Perl core (not to
mention the Perl community) and his support and insights are
instrumental in any such important discussion. However, despite our
initial trepidation, I double checked and confirmed that Andreas was
added to the list in Sat Feb 18 15:23:17 2017. Andreas confirms he
receives the emails from the list. That means that the ability to affect
this release was - at every point - possible to all members of the
security discussion group.

What I think we should have done differently:

The discussion in which Tony, Steve, and I decided on a plan should have
been held on the security discussion group. One could say that once
announced and no objections were raised, it was well received, but we
don't know that for sure. On the other hand, it is often more difficult
to also make a decision with *everyone*. Sometimes, a small group to
investigate and suggest a solution is a more practical approach. We
should have suggested that to the group.

Additionally, the risk that this release cycle introduces is reduced
testing. While we still believe this is a correct form of action, we
should have put more effort into enlisting smoke testers and providing
guidance and assistance in security testing.

Andreas has collected the statistics, and it shows the risk in a vivid
form: The total number of collected reports with 5.28.0 RC1 (which was a
standard release with no restricted access) was 61,444. The number of
different dists was 28,735. This means that it had 4x the number of
dists, 3x the number of tests. As Andreas put it, "We clearly can do
better." Our goal now is to understand how and to do this in a more
public format.

In conclusion, Andreas' concern about the seeming lack of a formal plan
and transparency regarding our actions and decision-making deserved an
extensive response. I hope this email provides the context, background,
and transparency that Andreas rightfully raised as necessary in this

I want to thank Steve Hay for keeping track of the record and helping me
form a proper timeline of events, tracking down threads and
conversations. I also want to thank Andreas for pushing all of us to do

Sawyer. Perl Programming lists via nntp and http.
Comments to Ask Bjørn Hansen at | Group listing | About