develooper Front page | perl.perl5.porters | Postings from April 2021

Re: threading and stuff (was Re: Perl’s leaky bucket)

B. Estrade
April 8, 2021 03:45
Re: threading and stuff (was Re: Perl’s leaky bucket)
Message ID:
Thank you, Dan. I will look into that.

In regards to the nature of why I am asking; here it is as a general 

We are looking for fundamental capabilities to drive Perl into the 
future. In regards to myself, I want to use Perl on large shared memory 
machines, not be relegated to single CPU VMs or docker images. I find 
the area of SMP highly fruitful for new compelling capabilities 
(particularly in regards to coherent (albeit incremental) semantic 
additions to Perl/perl itself) - and it keeps perl from being relegated 
to the hypervisor "matrix". I am also confident that it would drive a 
lot of interesting folks doing interesting things to Perl. But that's 
just my intuition speaking.

Without taking a look at *hard* or *interesting* (which are usually not 
easy) application areas that push Perl in uncomfortable ways or take it 
out of its comfort zone, what is driving the future? I hope it is not in 
becoming indistinguishable from JavaScript or PHP (my deepest, darkest 
fear tbh). What can be done? Keep thinking about what those 
uncomfortable or unpopular (hard) application areas are. We will need 
that list at some point.

I'll close with 2 inspirational quotes. One is a classic movie quote 
that also asks a very important question; with an even more important 

"Was it over when the Germans bombed Pearl Harbor?" ~ Senator John Bluto 

The answer, of course, is NO. Or, I guess pointing out the fact that the 
Germans did not do this, is actually more correct. But, that is not the 
point! Don't give up, we will find our way.

The second is one I found in the Internet just now, I think it's legit:

"It's not about the operating system, but it's about the operating 
system." ~ Abraham Lincoln

Keep being great, all of you. And thank you all for your efforts.


On 4/7/21 10:20 PM, Dan Book wrote:
> On Wed, Apr 7, 2021 at 8:50 PM B. Estrade < 
> <>> wrote:
>     0. upon reaching a "threaded block", lightweight threads (e.g.,
>     pthreads) are forked from the parent process
>     1. each thread is aware of it's id (0 .. N-1); parent is tid 0; all
>     threads sibling threads have access to parent process's memory
>     2. each thread has the ability to create private variables
>     3. communication is done so implicitly through the use of shared memory
>     and various forms of synchronization constructs
>     4. at the end of the block, child private memorys are destroyed
>     (including parent thread tid 0)
>     5. primary process memory reachable by child processes during the
>     threaded activity is available, potentially as affected during the
>     threaded fork (come what may)
>     I believe the essence of what characterizes what I am thinking comes
>     down to 2 things:
>     a. the exploitation of lightweight threads spawned from the parent
>     (perl) process
>     b. the availability of regions of directly shared memory to the child
>     threads
>     I don't know squat about internals, but I do believe I have a handle on
>     the ideal that a "perl" process is a singular line of execution. What I
>     am suggesting would take that process; and for a moment in the
>     execution
>     spawn it into threads whereby the threads may operate independently
>     over
>     the same process space (here's the 'shared memory' portion).
>     It is very important to know that I understand this is not a "fork".
>     The
>     process is not "cloned". There is a period of time where it is
>     executing
>     a very precise piece of code using multiple threads with access to the
>     parent process's memory.  The distinction in terms of it versus the
>     "fork" in Perl is that it's not a bunch of cloned perl processes that
>     operate in their own space as if there was no interruption. There is
>     only ever one process and the part that the threads operate over are
>     fully encapsulated inside of the original process. They communicate
>     through the developer carefully coordinating the reading and writing of
>     these areas of shared memory - most of which was initialized in the
>     main
>     process pre-spawn. In OpenMP lingo, it's a "fork/join"; but since we
>     use
>     that word "fork" to mean something very specific as it relates to how
>     the operating system treats it, it's probably better to call it a
>     "spawn/join" (not to be further confused with the "join" keyword).
> Yes, this clears up a lot of my confusion. I assumed by the use of the 
> word "fork" you were referring to forking, and using "thread" as a 
> design term, but it seems you were doing the opposite.
>     Note, while I keep saying "OpenMP"; I do so as it relates to the
>     simplified model it presents for those who use it. Most compilers that
>     implement it (pretty much all relevant ones) facilitate it by injecting
>     pthread calls during source transformation phases of the
>     compilation. If
>     I am worried about any unforeseen complication with what I am
>     suggesting, it's the concern that what I am asking for would
>     necessarily
>     require Perl to do something crazy like that. I hope that's not the
>     case. In my mind, what I am suggesting looks like:
>     0. perl hits spawn block
>     1. perl calls pthreads init to generate threads
>     2. threads do stuff (again, if this idea is "impossible" it's here)
>     3. threaded section ends, threads die; original parent process memory
>     remains as affect
>     All solutions I have seen have employed a combination of fork+provide
>     communication layer or are asynchronous "fire and forget" - none of
>     which are SMP.
>     I must also be careful to make sure it's clear I am not asking for
>     phtread "scripting" support. This is what Qore is, essentially - albeit
>     presented with a very perlish syntax.
>     My main objective is to ask, "what would it look like in perl to
>     provide
>     some regions of shared memory over which lightweight threads may
>     operate".
> I'd defer to others more familiar with ithreads such as Nicholas's 
> earlier post, but what you are describing is largely what 
> <> does, and its problems of being heavyweight to spawn 
> and causing bugs largely stem from trying to share memory and have the 
> perl interpreter available in each thread. So I'll just say, these are a 
> lot of nice wishes, but getting the tradeoffs right in the 
> implementation is a challenge, if it is possible at all.
> -Dan Perl Programming lists via nntp and http.
Comments to Ask Bjørn Hansen at | Group listing | About