develooper Front page | perl.perl5.porters | Postings from April 2021

Re: threading and stuff (was Re: Perl’s leaky bucket)

Thread Next
B. Estrade
April 8, 2021 00:50
Re: threading and stuff (was Re: Perl’s leaky bucket)
Message ID:

On 4/7/21 7:13 PM, Dan Book wrote:
> On Wed, Apr 7, 2021 at 7:55 PM B. Estrade < 
> <>> wrote:


>     Yes, thank you, Dan. I am aware of these. I also use fork a lot and am
>     happy exploiting machine CPUs that way. But I really (really) want
>     personally - and think it'd be a highly compelling capability for Perl
>     itself - is to be able to exploit SMP natively, even if "unsafely".
> I'm not sure I understand. How does "exploit SMP natively" with a 
> "fork/join model" differ from what these modules offer?
> -Dan

Sorry, I was not clear. I meant "natively" within Perl.

Just to provide contrast, MCE provides fork process management and 
communication among forked processes. Most of my experience with 
threading is from OpenMP, which is supported by every major compiler out 
there. Granted it's high level and declarative via the use of pragmas. 
So I reference this with the risk that that point gets misconstrued. 
It's the threading model (fork/join) or (spawn/join) that I am talking 

I am not sure how else to explain it without sending you a bunch of 
google links and risking the insulting of your knowledge.

So I can put it in terms of what I think would be great.

0. upon reaching a "threaded block", lightweight threads (e.g., 
pthreads) are forked from the parent process
1. each thread is aware of it's id (0 .. N-1); parent is tid 0; all 
threads sibling threads have access to parent process's memory
2. each thread has the ability to create private variables
3. communication is done so implicitly through the use of shared memory 
and various forms of synchronization constructs
4. at the end of the block, child private memorys are destroyed 
(including parent thread tid 0)
5. primary process memory reachable by child processes during the 
threaded activity is available, potentially as affected during the 
threaded fork (come what may)

I believe the essence of what characterizes what I am thinking comes 
down to 2 things:

a. the exploitation of lightweight threads spawned from the parent 
(perl) process
b. the availability of regions of directly shared memory to the child 

I don't know squat about internals, but I do believe I have a handle on 
the ideal that a "perl" process is a singular line of execution. What I 
am suggesting would take that process; and for a moment in the execution 
spawn it into threads whereby the threads may operate independently over 
the same process space (here's the 'shared memory' portion).

It is very important to know that I understand this is not a "fork". The 
process is not "cloned". There is a period of time where it is executing 
a very precise piece of code using multiple threads with access to the 
parent process's memory.  The distinction in terms of it versus the 
"fork" in Perl is that it's not a bunch of cloned perl processes that 
operate in their own space as if there was no interruption. There is 
only ever one process and the part that the threads operate over are 
fully encapsulated inside of the original process. They communicate 
through the developer carefully coordinating the reading and writing of 
these areas of shared memory - most of which was initialized in the main 
process pre-spawn. In OpenMP lingo, it's a "fork/join"; but since we use 
that word "fork" to mean something very specific as it relates to how 
the operating system treats it, it's probably better to call it a 
"spawn/join" (not to be further confused with the "join" keyword).

Note, while I keep saying "OpenMP"; I do so as it relates to the 
simplified model it presents for those who use it. Most compilers that 
implement it (pretty much all relevant ones) facilitate it by injecting 
pthread calls during source transformation phases of the compilation. If 
I am worried about any unforeseen complication with what I am 
suggesting, it's the concern that what I am asking for would necessarily 
require Perl to do something crazy like that. I hope that's not the 
case. In my mind, what I am suggesting looks like:

0. perl hits spawn block
1. perl calls pthreads init to generate threads
2. threads do stuff (again, if this idea is "impossible" it's here)
3. threaded section ends, threads die; original parent process memory 
remains as affect

All solutions I have seen have employed a combination of fork+provide 
communication layer or are asynchronous "fire and forget" - none of 
which are SMP.

I must also be careful to make sure it's clear I am not asking for 
phtread "scripting" support. This is what Qore is, essentially - albeit 
presented with a very perlish syntax.

My main objective is to ask, "what would it look like in perl to provide 
some regions of shared memory over which lightweight threads may operate".

Hope that helps.

Thread Next Perl Programming lists via nntp and http.
Comments to Ask Bjørn Hansen at | Group listing | About