develooper Front page | perl.perl5.porters | Postings from August 2021

Re: concurrency list (was Re: Relinquishing Maintenance of CoreModules)

Thread Previous | Thread Next
David Christensen
August 2, 2021 17:40
Re: concurrency list (was Re: Relinquishing Maintenance of CoreModules)
Message ID:

Thank you everyone for your thoughts.  :-)

On 7/24/21 10:48 PM, Oodler 577 via perl5-porters wrote:> * David 
Christensen <> [2021-07-24 16:08:58 -0700]:

 >> Whatever is done, I have been wanting a Perl concurrency mailing list
 >> for many years.

 > Regarding the discussion of "threads" - I'd love to have one.

I'm fine talking about threading, concurrency, etc., here.  It is useful 
to have archives, so a cc: model does not appeal.

On 7/25/21 4:10 AM, Leon Timmermans wrote:
 > On Sun, Jul 25, 2021 at 1:09 AM David Christensen wrote:
 >> For example, I have discovered that threads is not fully compatible
 >> with Data::Dumper and Capture::Tiny.
 > The former sounds like a bug.

 > The latter deals with a process-global resource (standard IO), so
 > will inherently have some funkiness.

The following Capture::Tiny documentation led directly to my previous 
"self-fulfilling prophecy" statement:

2021-08-01 15:51:56 dpchrist@dipsy ~
$ perldoc Capture::Tiny | head -n 262 | tail -n 4
    Using threads
     Filehandles are global. Mixing up I/O and captures in different threads
     without coordination is going to cause problems. Besides, threads are
     officially discouraged.

Threading was a core Perl feature.  CPAN modules were supposed to 
support threading.  Then threads were declared "officially discouraged". 
  What was supposed to replace threads?

Now my choices are 1) make Capture::Tiny thread-safe, or 2) write a 
sawed-off thread-safe replacement.  Madness: =~ s/lisp/perl/ig

 > IMO what we need is to research other threading models on top of the
 > primitives that we have (and if needed extend those primitives).
 > threads::lite was my first attempt at that, though I've started a
 > successor.

AIUI threads and threads::shared were the canonical Perl threading 
primitives, as demonstrated by their inclusion in the Camel book.  I am 
aware of the following documentation, which describes the threading 
model they implement:

"Programming Perl" [1], pages 1035-1036, 1078.

"Network Programming with Perl" [2] identifies additional primitives and 
non-primitives that can relate to concurrency.

On 7/25/21 8:26 AM, Oodler 577 via perl5-porters wrote:
 > The ideas of "practical" concurrency is way more important to
 > perl/Perl's future than virtually any other issue.

Which feature is most important is debatable.

I would put "Conceptual Integrity" and "Programming Systems Product" at 
the top of the list, per Brooks [3].

AIUI threads were pushed to the forefront ~20 years ago, when multi-core 
and hyper-threading microprocessors became commodity items.  As always, 
user expectations have risen since then.  And, as always, developers are 
expected to create software that can make full use of the latest 
hardware.  So, if Perl wants developers, Perl must give developers the 
tools they need to meet user expectations.

But, threading is just one way to achieve concurrency on a single 
computer.  The traditional Unix way is fork/ exec.  I can simulate 
concurrency in a single thread by invoking short, non-blocking 
subroutines in a tight loop (e.g. polling, asynchronous I/O, select), 
but that approach is brittle; my application becomes a scheduler. 
Perhaps I could write a Perl program, launch it into the background with 
a 1-argument system() or qx(), and communicate with it over non-blocking 
pipes or sockets. Are there any other core Perl mechanisms to achieve 
concurrency within a single computer?

Are there any core Perl mechanisms to achieve real concurrency on a 
network of computers?

 > ... openmp ...

Perl modules/ distributions for these could be useful.

 > Off the top of my head there are many topics ripe for discussion:
 > * ithreads
 > * forking based things
 > * async
 > * "real" threads via OpenMP/pthreads bindings
 > * perl vs Perl memory models
 > * atomicity/sequential consistency

I'm happy to discuss any or all of them.  :-)

On 7/28/21 8:26 AM, Ovid wrote:
 > Give me:
 >     - Concurrency
 >     - OOP
 >     - Signatures
 >     - Some way of defining enforceable types
 > With that, Perl can come out of the gate swinging.

I would like:

- Conceptual Integrity

- Programming Systems Product -- functional specification, design/ 
construction specification, test/ validation specification, etc., with 
everything documented and under configuration management

- Multi-paradigm -- imperative, functional, object-oriented, 
declarative, meta-programming (includes signatures and types)

- Multi-Platform -- add Android and iOS

- Memory safe/ garbage collected

- Secure

- Concurrent/ Distributed/ Networked

- Macros

- Programs can be compiled and statically linked into stand-alone 

On 7/28/21 10:32 AM, shmem wrote:> Last time I looked there was MCE 
which somehow looks meaningful.

I saw MCE.  And POE.  And Coro.  And PDL.  I put my effort into threads 
and threads::shared years ago (pre-discouraged) because they looked like 
the foundation of threading in Perl, they were the most similar to my 
previous C/ assembly experiences, and I thought I could build useful 
stuff with them.  These reasons have proven to be mostly correct.  The 
problems I have now are dealing with non-thread-safe code throughout the 
rest of Perl and the knowledge that threads are on the chopping block.

On 7/28/21 11:05 AM, Oodler 577 via perl5-porters wrote:
 > What we need is a door into the sharing of memory; if not among
 > threads (not my goal), surely amoung fork'd processes. Coupled with
 > atomic synchronization primatives (which we have in some form via
 > sysopen), this would provide us a path to utilize forks+shared memory.

I read about shared memory on Unix via C years ago, but have not needed 
or used it.  STFW I see existing modules/ distributions that do shared 
memory.  You don't see anything you like?

On 7/28/21 8:59 PM, Yuki Kimoto wrote:
 > The word "concurrency" is always ambiguous when people talk about
 > "concurrency".

 > I hear the this conversation "Perl doesn't have concurrency, so Perl
 > can't use CPUs in parallel"
 > But, I know the pre-fork model Perl web applications are using CPUs in
 > parallel.
 > I think we need to properly classify concurrency(fork, thread, I/O,
 > CPU(OpenMP), GPU, coroutine, async/await syntax) before we say we want
 > to add concurrency.

Perl has several forms of concurrency.  Unfortunately, a 
high-visibility, core form -- threading -- has not worked out and is now 

On 7/28/21 10:12 PM, Oodler 577 wrote:
 > Typically, OpenMP is accomplished at low level with pthreads.

AIUI POSIX Threads (pthreads) are not supported on all of the platforms 
that support Perl.  So, Perl was required invented its own 
multi-platform threading model.

I suspect OpenMP will similarly lose to multi-platform.  Can OpenMP be 
put into a Perl module that works on the relevant subset of platforms?

For that matter, can POSIX Threads be put into a Perl module that works 
on the relevant subset of platforms?

Have these been tried already (see next)?

On 7/29/21 10:34 AM, Leon Timmermans wrote:
 > In practice, all ideas have either been already done, or are very
 > difficult (or even impossible) to do.

Please let us know if you see the discussion headed that way.

On 7/29/21 10:34 AM, Leon Timmermans wrote:
 > I agree [concurrency is] important, that's why I'm actually working on
 > the issue. To be precise, an ithreads based (so can actually use
 > multiple processors) continuation passing style (same model Go uses
 > for its channels) module.

STFW again, I see existing modules/ distributions with continuations. 
You do not see anything you like?

Can continuations be abstracted out, become a Perl primitive, and used 

On 7/29/21 10:47 AM, Oodler 577 wrote:

 > I am doing 2 things:
 > 1. continuing to develop the idea of the "perl" memory model
 > (uniprocess) and the Perl memory model (semantically, admits
 > concurrency). See Sub::Genius.
 > 2. approaching concurrency from the shared memory perspective by
 > exploring various ideas in order to determine how OpenMP can be most
 > naturally leveraged on an SMP machine. See OpenMP::Environment and
 > Alien::OpenMP.

I have been searching for a general model of concurrency for many years. 
  Flow-based programming (FBP) has caught my interest:

STFW I did not see anything I liked:

I have built a simplified FBP library in Perl using threads and a 
re-write of Thread::Queue (with multiple readers and writers, and 
coordinated open and close).  It can run FBP programs concurrently on 
multi-core and/or hyper-threaded processors.  It should be able to run a 
FBP program concurrently on multi-processor computers (when I have such 
a machine for testing).  I am now researching extensions so that it can 
run a FBP program concurrently on a network of multiple computers.



[1] Tom Christiansen, et al, 2012, "Programming Perl", 4 e,

[2] Lincoln D. Stein, 2000, "Network Programming with Perl",

[3] Frederick P. Brooks, 1996, "Mythical Man-Month, The: Essays on 
Software Engineering, Anniversary Edition," 2 e.,

Thread Previous | Thread Next Perl Programming lists via nntp and http.
Comments to Ask Bjørn Hansen at | Group listing | About