Sterling has too many projects

Blogging about Raku programming, microcontrollers & electronics, 3D printing, and whatever else...

»

The react block in Raku is the primary means of re-synchronizing asynchronous coding activity. Using it, you can easily pull together promises, supplies, and channels to make a coherent whole of your program or a subsystem.

A react block itself can run any code you want plus one or more whenever blocks. The code in the block will run once and the block will exit either when a done subroutine is called or when all the objects associated with whenever blocks are finished (i.e., all promises are kept or broken, all supplies have quit or closed, and all channels have failed or closed).

Aside from the syntax, the key thing to note about a react block is that all code inside of the block will always run as if single-threaded (i.e., multiple threads might be employed, but never will any code within this block be run concurrently).

Let’s take a look at an example react block:

my $commands = Channel.new;
my $input = Supplier.new;
my $output = Supplier.new;
my $quit = Promise.new;
react {
    print '> ';

    start loop { $input.emit: $*IN.getc }

    whenever $input.Supply.lines.map({ .trim }) {
        when /^add \s+ (\d+) \s+ (\d+)$/ { $commands.send: ('add', +$0, +$1) }
        when /^sub \s+ (\d+) \s+ (\d+)$/ { $commands.send: ('sub', +$0, +$1) }
        when 'quit' | 'exit' { $quit.keep }
        default { $output.emit: 'syntax error' }
    }

    whenever $commands -> @command {
        multi doit('add', Int $a, Int $b) { $a + $b }
        multi doit('sub', Int $a, Int $b) { $a - $b }

        $output.emit: doit(|@command);
    }

    whenever $output.Supply { .say; print '> ' }

    whenever $quit {
        say 'Quitting.';
        done;
    }
}

This program provides a small interactive shell that can perform addition and subtraction. When run, you might use it as follows:

> add 4 5
9
> sub 10 7
3
> exit
Quitting.

As you can see the code operates with a couple Supply objects, a Channel, and a Promise. Each of them work with whenever in the expected way. All of the code within the run block runs as if in a single thread (though, there’s no particular guarantee that only a single thread is used, only that no code will run concurrently).

Running from a single task with no concurrency does, however, present a problem in this case. The $*IN file handle only performs blocking reads, even if you use the .Supply method to get the data asynchronously. Therefore, we must pull input in a task running in a background thread, which is why we put a start before the loop that reads character input. Without this concurrent task, we’d have to hit the Return key extra times to give the other whenever clauses a chance to run.

That said, we could move the work of the other whenever blocks into separate concurrent tasks and just pull each together in this react block and it would work just as well. The goal of the react block is to synchronize the asynchronous work in a straightforward syntax. I think it does the job pretty well.

Cheers.

»

A large number of concurrency-oriented coding in Raku depends on the use of a Scheduler. Many async operations depend on the default scheduler created by the VM at the start of runtime. You can access this via the dynamic variable named $*SCHEDULER.

The most important feature of a Scheduler is the .cue method. Calling that method with a code reference will schedule the work for execution. The type of scheduler will determine what exactly that means.

That said, this is a low-level interface and you probably shouldn’t be calling .cue in most code. The best practice is to rely on high-level tools like start blocks which due this and construct a Promise for you to monitor the work.

Every scheduler provides three methods:

  1. The .uncaught_handler is an accessor that returns a routine or can be set to a routine which is called whenever an exception is thrown by the scheduler and is not handled by the task code itself. If no handler is provided and a cued task throws an exception, the application will exit on that exception. If you use the high level concurrency tools, such as start blocks, the .uncaught_handler will never be used because they each provide their own exception handling.

  2. The .cue method is used to add a task to the schedule. The scheduler will perform that task as resources allow (depending on how the scheduler operates).

  3. The .loads method returns an integer indicating the current load on the scheduler. This is an indication of the size of the current job queue.

So, you could build a very simple scheduler like this:

class MyScheduler does Scheduler {
    method cue(&code, Instant :$at, :$in, :$every, :$times = 1; :&catch) {
        sleep $at - now if $at && $at > now;
        sleep $in if $in;


        for ^$times {
            code();
            CATCH { 
                default {
                    if &catch {
                        catch($_);
                    }
                    elsif self.uncaught_handler {
                        self.uncaught_handler.($_);
                    }
                    else {
                        .throw;
                    }
                }
            }
            sleep $every if $every;
        }

        class { method cancel() { } }
    }

    # We don't really queue jobs, so always return 0 for the load
    method loads(--> 0) { } 
}

This is somewhat similar to what the CurrentThreadScheduler does.

Rakudo has two schedulers built-in:

  • ThreadPoolScheduler is the usual default $*SCHEDULER. When it is constructed, you can set the number of threads it is permitted to use simultaneously. It then manages a pool of threads and will schedule cued tasks on those threads. As tasks complete, freeing up threads, the next tasks will be scheduled to run. Tasks may run concurrently with this scheduler. When .cue returns, the tasks may not have started yet. The .cancel method of the returned object may be used to request cancelling the work of a given task.

  • CurrentThreadScheduler is an alternate scheduler. It basically just executes the task immediately and returns after the task is complete. The returned cancellation object has a .cancel method, but it is a no op as the work will always have completed by the time the scheduler returns.

Many async methods, such as the .start method on Promise, take a named :scheduler argument where you can pass a custom scheduler. In general, you can stick to the default scheduler. Probably the most common adjustment to a scheduler would be to change the number of threads in the thread pool or switch to using the current thread scheduler under some circumstances. Chances are you will need to do neither of these. And if you need something exotic, it may be reasonable to define your own scheduler as well. Things to consider.

Cheers.

»

Warning! We are delving into the inner depths of Raku now. Threads are a low-level API and should be avoided by almost all applications. However, if your particular application needs direct Thread access, it is here for you.1

Use of the Thread class in Raku is straight-forward and looks very similar to what you would expect if you are familiar with threading tools in other languages:

my $t = Thread.start:
    name => 'Background task',
    :app_lifetime,
    sub {
        if rand > 0.5 { say 'weeble' }
        else { say 'wobble' }
    },
    ;

say "Starting $t.id(): $t.name()";
say "Main App Thread is $*THREAD.id(): $*THREAD.name()";

$t.finish; # wait for the thread to stop

Give the Thread.start method some code to run and you’re off. The name and app_lifetime options are optional. If app_lifetime is False (which is the default), the thread will be terminated when the main application thread terminates. If set to True, the application will continue to run as long as this thread is running. Under normal circumstances, only the main thread of your application has this privilege.

All code, runs within a thread. Your code can access the thread it is running in using the dynamic variable named $*THREAD. This can be helpful for pulling the .id when debugging to help understand which thread a task is running in at the moment.

When you want to pause the current thread to wait for another thread to finish, you do that with the .finish method (or you can use .join, which is a synonym for .finish).

Another way to run a thread is to use a combination of .new and .run. This is similar to .start, but code must be passed as a named argument to .new:

my $t2 = Thread.new:
    name => 'Another task',
    code => sub {
        loop {
            say 'stuff';
            sleep 1;
        }
    },
    ;

# The thread does not start until we...
$t2.run;

# And then we'd better wait for it or we'll exit immediately
$t2.finish;

For the most part, I make mention of threads in the advent calendar as a way of describing the “lanes” in which code runs. However, I will make use $*THREAD.id from time to time to help illustrate that code does run in different threads. Otherwise, I will generally ignore the Thread object directly.

Almost all Raku programs should stick to using start blocks or Promise.start to start tasks to run on another thread. You should only make use of Thread directly if you really need it, which is probably never or close to it for most Raku developers.

Cheers.


  1. As a point of clarification, a Thread object does not necessarily represent a specific OS thread, but it should get you as close as the implementation is able.  ↩

»

A Channel, in Raku, is an asynchronous queue of data. You can feed data in to one end of the queue and receive data at the other end safely, even when multiple threads are involved.

Let’s consider a variant of the Dining Philosopher Problem: We have five philosophers eating soup at the table. They do not talk to one another because they are too busy thinking about philosophy. However, there are only 2 spoons. Each time a philosopher wishes to take a sip of soup, she needs to acquire a spoon. Fortunately, each philosopher is willing to share spoons and sets her spoon at the center of the table after finishing each bite.

We can model this problem like this:

my $table = Channel.new;
my @philosophers = (^5).map: -> $p { 
    start {
        my $sips-left = 100;
        while $sips-left > 0 {
            my $spoon = $table.receive;
            say "Philosopher $p takes a sip with the $spoon.";
            $sips-left--;
            sleep rand;
            $table.send($spoon);
            sleep rand;
        }
    }
}
$table.send: 'wooden spoon';
$table.send: 'bamboo spoon';
await Promise.allof(@philosophers);

Here we have five tasks running in five threads, each contending for one of two resources. They will each take 100 sips of soup. Running this program will give you 500 lines of output similar to this:

...
Philosopher 0 takes a sip with the wooden spoon.
Philosopher 2 takes a sip with the bamboo spoon.
Philosopher 3 takes a sip with the wooden spoon.
Philosopher 1 takes a sip with the bamboo spoon.
Philosopher 4 takes a sip with the bamboo spoon.
Philosopher 2 takes a sip with the bamboo spoon.
Philosopher 0 takes a sip with the wooden spoon.
Philosopher 1 takes a sip with the wooden spoon.
Philosopher 0 takes a sip with the wooden spoon.
...

The code itself is very simple. We start five threads each representing a philosopher. Each philosopher calls to .receive the next available spoon. This method will block until a spoon becomes available. The philosopher takes a sip and then uses .send to return the spoon to the table for anyone else to use. Eventually, the philosopher finishes 100 sips and the Promise returned by start will be kept.

The main thread kicks off the process by placing two spoons on the table using .send. Then, it uses await to keep the program running until all the philosopher tasks are complete.

Channels have very low overhead as far as CPU is concerned. The senders will not wait on the receivers. The receivers can either block until something is available in the queue using .receive or they can use .poll to check for items without blocking. The cost is transferred to memory. The Channel must store all sent items internally until they are received and the queue will continue to grow until the program runs out of memory.

Therefore, a Channel is helpful when you have resources or messages you want to distribute, but not share between tasks. Or when you just need to communicate point to point. Job queues, resource pools, and peer-to-peer task communication are good examples of the sort of problems Channels are ideally suited for.

Cheers.

»

When you have a stream of data flowing through your Raku application that needs to be accessed safely among threads, you want a Supply. Today we’re going to discuss one particular way of working with supplies, the supply block. If you are familiar with sequences, aka Seq objects, a supply block works in a very similar fashion, but lets you pull work as it arrives and easily do something else in the meantime.

multi a(1) { 1 }
multi a(2) { 2 }
multi a($n where $n > 2) { a($n - a($n-1)) + a($n - a($n-2)) }

my $hofstadter-generator = supply {
    for (1 ... *).map(-> $n { a($n) }) -> $v {
        emit $v;
    }
}

react {
    whenever $hofstadter-generator -> $v {
        note $v;
    }
    whenever Supply.interval(1) {
        say "Waiting...";
    }
}

So we have three sections of code here:

  1. There is a function a() which generates values of the Hofstadter Q-sequence1 (in a particularly inefficient way).

  2. There is a Supply object generated using a supply block which outputs values of of the Hofstadter Q-sequence using the function we defined, starting at the beginning and continuing until the program quits.

  3. And then there is a react block, outputting the values whenever they are available and outputting a waiting message every second in between values. I will go into react blocks in more another day. For now, just know that react blocks allow us to synchronize asynchronous work by pulling the results back into a single thread.

Regarding our central subject, the supply block, it returns a Supply object that can be “tapped.” A Supply is tapped by calling either the .tap or .act methods or by using it in a whenever block as part of a react or another supply block.

In the case of a supply block, the kind of Supply object we get is called an “on demand Supply.” This means that each time the Supply is tapped, the code associated with the block is run. Each time an emit is encountered, the block passed to .tap or the whenever statement will be run and given the value emitted as the argument (named $v in the example code above). Execution continues until either block given to supply exits or the done statement is reached, which causes the supply block to exit (this operates very similar to last for loops).

For example, if we want our sequence to end automatically after 100 iterations, we could rewrite our supply like so:

my $stopping-hofstadter-generator = supply 
    for (1 ... *).map(-> $n { a($n) }) -> $v {
        emit $v;
        done if $n > 100;
    }
}

One very important factor to remember while using a supply block is that the emit command blocks until every tap has finished working.2 This means that that supply block “pays for” the compute time of its taps. This delay provides a form a back-pressure which prevents the generated Supply from running faster than the taps can process. Therefore, if you want your streams of events to run at light speed and don’t care if the tools processing can keep up, you need to either make sure that the taps immediately start new tasks to run to avoid blocking or you want a different mechanism from a plain Supply to handle the workload.

Cheers.


  1. For background on the following integer sequence, see https://oeis.org/A005185.  ↩

  2. This is a feature of all Supply objects, not just ones generated by supply blocks or on–demand supplies.  ↩

»

In Raku, Promises represent the simplest of the high-level language features for communicating between asynchronous tasks. They are very much like promises between people. For example, I might promise my son I will help him with his school work. I keep that promise when I help him. Or I break that promise if, for any reason, I fail to help him. The same is true of a Promise in Raku. A Promise to return a value is kept when that value arrives. A Promise to return a value is broken if an error occurs that prevents the value from arriving.

So, let’s see a basic promise in action in Raku:

my $promise = start {
    my $nth = 0;
    my $nth-prime;
    for 1..* -> $number {
        $nth++ if $number.is-prime;
        if $nth == 10_000 {
            $nth-prime = $number;
            last;
        }
    }
    $nth-prime;
}
await $promise.then({ say .result });

The code above uses a start block to begin finding the 10,000th prime. This block returns a Promise object. This object exists in one of three states (which you can check with the .status method). The initial state is Planned and then it enters either of two final states. The Broken state is entered if a result cannot be reached (usually because an exception occurred) and the Kept state is entered when the result becomes available. Once in the Kept status, the .result method will immediately return the value of the kept Promise.

Promises can be chained together using then .then method. This works by adding another block that starts as soon as the first is kept. The new block will be given the previous Promise object as an argument and the method returns a new Promise which will contain result from the following block.

The start block in the code above schedules the computation to run on the next unused thread in the default thread pool and returns a Promise object.1 We use .then() to output the .result of the computation as soon as it comes available.

Finally, we have the await statement, which causes the main thread to pause until the value becomes available. Without that statement, our program would end before the computation completes.

An await also allows a broken Promise to deliver exceptions. Consider this code:

my $promise = start { die 'bad stuff' }
sleep 1;
say 'something';

The code above will sleep for 1 second and print “something” to output. However, the exception will never be received. This is because while an exception causes the promise to be broken, we aren’t looking at the result of the Promise at all. We can add an await on the Promise where we are ready to receive the value and any exception thrown that causes the Promise to be broken will be received:

my $promise = start { die 'bad stuff' }
sleep 1;
say 'something';
await $promise;
CATCH {
    default {
        say "ERROR: $_.message()";
    }
}

This code does exactly the same thing as before, but also outputs “ERROR: bad stuff” after “something”. Always be sure to either handle your exceptions inside the start block or in another block that receives the Promise this way, or you may end up with bizarre and unexpected problems.

Those are the essential elements of Raku Promises.

Cheers.


  1. Note that this is the process for the usual default scheduler running in Rakudo under MoarVM. What actually happens might vary somewhat based on the current value in the $*SCHEDULER variable. I will discuss this further in a future post.  ↩

»

Before we get into the guts of this advent calendar for the next 23 days after today, I want to be sure to introduce the basic concepts I’m going to cover. I am calling this the Raku Async & Concurrency Advent Calendar, but what do those terms mean and how do they apply to Raku?

This advent calendar assumes a basic knowledge of Raku. If you don’t know Raku, but you are familiar with another language, I’m sure you will probably be able to follow along. However, this is aimed at intermediate to advanced developers, so I recommend one of these resources for learning Raku first:

Asynchronous Programming

Async is short for asynchronous programming because asynchronous is annoyingly painful to type. Asynchronous programming is a style of programming where the call to a function is disconnected from its return. Normally, we write a function like this:

sub double($x) { $x * 2 }
my $val = double(21);

If we want make this async, we might try something like this:

sub double($x) { sub () { $x * 2 } }
my $promise = double($x);
# more code
# ...
# more code
my $val = $promise.();

That’s a silly contrived example, but the point is that we don’t actually receive the value until later. This style of programming is useful primarily when you know you need some work done, but don’t need the results immediately. This is especially useful when you might need to do other work in the meantime.

When does it make sense?

A couple example situations where async could be helpful:

  • All socket-based client/server applications are necessarily async. For TCP, on a server, you establish a listening socket and then when an client connects, you receive a connected socket that you can exchange data on using read and write operations. A TCP client, meanwhile, requests a connection and then waits until the server accepts the connection. The client could continue to do work while waiting.

  • A program that processes log data in real-time typically has several processing steps to run each line through. It waits for data to arrive. It parses each line of data to get the interesting information out of it. It filters the data to decide if it’s useful for the current task or not. It runs calculations on the data. Each of these operations could be performed in an asynchronous programming chain where the operations are performed as soon as some amount data is ready for each step.

Async is just a style of programing and it fits wherever you, the software developer, decide it fits. Some problems are more inherently asynchronous than others, but you could build them synchronously too. Use async when it suits you.

What are the tools?

In Raku, asynchronous programming is primarily performed through high-level, “composable” interfaces, such as Promise, Supply, and Channel. These are high-level because they provide an interface aimed at someone without needing to rigorously prove the safety of how these tools are used. You still have to be careful, to be sure, but the pitfalls are much reduced and should be more obvious from their interface. They are composable because they are designed to let you take different libraries using these interfaces and use them together in predictable ways.

Low-level tools, on the other hand, like Lock and Semaphore can be used in ways that result in unpredictable behavior when different libraries using these are combined. These too are available in Raku because sometimes you need to build with them, but they aren’t the bread and butter of Raku’s async programming.

What does it look like?

So what does async look like in Raku? Here’s a basic counting program, but instead of using a loop, we’ll output each number every second.

react {
    whenever Supply.interval(1) -> $n {
        say $n;
    }
}

This is functionally equivalent to running:

for 1...* -> $n {
    say $n;
    sleep 1;
}

As you can see the syntax is very simple and hopefully easy to follow. You can even think of it as sort of being synchronous without getting into too much trouble. A goal of async programming in Raku is to make it accessible to those who don’t fully understand it to at least understand what is happening.

Concurrent Programming

I took a class on concurrent programming in college. I’m afraid I passed the class without really grasping the topic very well. However, I think that’s partly because it is a hard concept to really understand by lifting the hood and looking at all the moving parts. However, it’s also partly because the class taught concurrency in mathematical terms and I’ve found the practical terms much easier to follow.

Concurrent programming is merely the act of running parts of a program simultaneously. The parts of a program that can be run simultaneously are called threads. You can think of threads as being lanes in which a program can run its code. For example, each thread might be running the same program code with different starting data. Or a program could have a thread for updating the graphics interface and a thread for talking to other programs on the network and a thread for doing local processing of the data received.

Bare metal threads

Running in lanes can be achieved in various ways, including:

  • On a single CPU/single core computer, the CPU will interleave code execution that needs to happen simultaneously. Code does not actually run at the same time, but every thread of each program is given time to run some instructions on the CPU, then a context switch is performed and another thread runs.

  • On a multi-CPU/multi-core computer, this means both interleaving code on each CPU core and it means running various programs and threads on each core in separate hardware simultaneously.

In any case, you have a problem where the code in any given thread might be only fractionally complete with a task, so you need to make sure the threads interact with one another safely. This is called “thread safe code.”

Thread safety

There are a few basic programming styles for building thread safe code. I present them here from most preferred to least preferred style:

  1. Any code that is stateless or which only has changing state internal to itself is inherently thread safe. It is only when code needs to reach data that other code might use that it becomes unsafe.

  2. Often, code can be written such that it is stateless internally, but performs state changes when it finishes execution. This data transformations are thread safe so long as something synchronizes the state updates in a safe way. The code that performs the transformation does not itself have to make any special provision for safety.

  3. For situations where some changeable state must be shared between executing threads, special care must be taken every time that data is read from and written to in order to make sure the changes don’t corrupt the state.

Raku provides specialized tools for handling each of these situations. Whenever writing concurrent programs, you should always aim for the top two coding styles when you can. These are the easiest to understand and maintain. However, the third option is sometimes the best

The Raku way

In Raku, we don’t tend to work with threads directly. Instead, we schedule blocks of code to run asynchronously. How the code is actually run depends on the scheduler used. The default scheduler on Rakudo will schedule blocks to run on a separate thread from the main thread.1

The primary means of scheduling a block to run is using a start block similar to the following example.

start {
    # Subtask 1
}
start {
    # Subtask 2
}
# Main task

I, personally, refer to work scheduled this way or by other means as a “task” or a “subtask” depending on context. That’s not a Rakuism, though, so beware that the terminology of others may vary.

Anyway, the main point is that concurrency refers to running two different parts of your code simultaneously.

Parallel Programming

A third term we will frequently be using in this calendar is parallel programming. Parallel programming is very similar to concurrency, but primarily oriented around processing data in parallel. For example, if you have a large number of integers you want to run through a function, you can perform that action concurrently for every one of those integers. This is parallel programming.

The reason it is a separate term from async or concurrency is twofold. One, a parallel program is not necessarily concurrent or async and, two, there are some special optimizations related to parallel programming that are not necessarily a part of general concurrency or async.

For example, here is a small parallel program in Raku:

my @doubles = (1...10_000_000).hyper.map(* * 2);

When this program runs, Raku will schedule the tasks to run in multiple threads to iterate through all values in the range and double all 10 million values in batches. The work of dividing the work into batches and then choosing how to schedule them is what makes parallel programming a special topic of its own. This program is probably concurrent, but it is not async. We could do parallel programming asynchronously too, but we aren’t in this example.

Programming Programming

So, when we put the terms together we find that a program might have any of the three properties described. It might be concurrent and async and parallel. It might be none of those things. It might be just one or two of those things, in any combination. These are different styles of programming.

Raku provides tooling that aims to make it easier to read your code when it is being formed asynchronously, concurrently, and in parallel. It also aims to encourage you to write your code in an inherently thread safe way by allowing you to safely transform state while operating in multiple threads simultaneously.

It is my hope that over the following 23 days, I will better equip you to decide when to embrace each of these features of Raku and how to do so in a way that makes your code faster while remaining easy for humans to parse and understand.

Cheers.


  1. I will cover the details of schedulers later. When I do you will see the exact mechanism used by Raku to determine whether a task is scheduled to run immediately in a separate thread, run at some future time on a separate thread, run on the current thread whenever the current thread stops being busy, etc. However, I will generally assume that start will schedule a task on the next available thread as this is generally safe to assume in Raku.  ↩

»

I have never done a full advent calendar before. I have contributed to some other calendar blogs in the past, but this will be the first time I do a complete one of my own. This is my announcement: I am doing an advent calendar. It’s even mostly written already. So what’s the theme?

Raku! Concurrency! Async! Parallelism! All that stuff is my focus. I’ve spent some extra hours here and there trying to better understand the concurrency and async programming features of Raku and I’m going to share my knowledge with anyone who is interested. I will be covering a number of topics to including some details about how these objects in Raku work:

  • Supply
  • Promise
  • Channel
  • Lock
  • Lock::Async
  • Semaphore
  • Proc::Async
  • IO::Socket::Async

I am going to cover various low level topics like Thread and Scheduler and cas.

I am going to cover various high level topics like how to break down complex asynchronous coding problems, how to decide make use of concurrency, comparing react blocks with calls to tap, different ways to sleep, and so much more!

So, this holiday season, if you want to give the gift that gives in parallel, asynchronously, and at the same time, please tell your friends to read my Advent Calendar for 2019 on Raku. From December 1 to December 24, I will have an article every day right here!

Cheers.

»

  • Category: Blog

Quickly now, let’s consider the difference between a sub and a method. When programming Perl 6, the only significant difference between a sub and a method is that a method always takes at least one positional argument whereas a sub only takes what’s listed in the parameter list. In a method, the required first positional parameter is not passed as part of the parameter list, but assigned to self.

For example,

class Demo {
    has $.value;
    sub foo(Demo $val) is export { put $val }
    method bar() { put self }
    method Str() { ~$.value }
}

import Demo;

my $demo = Demo.new(value => 42);
foo($demo);  # OUTPUT: «42»
$demo.bar(); # OUTPUT: «42»

Ready for the trick? The subroutine can be called as a method too, like this:

$demo.&foo; # OUTPUT: «42»

That’s it. Any subroutine can be used as a method by using the .& operator to make the call. The object before the operation will be passed as the first argument.

My favorite usage of this feature is this one:

use JSON::Fast;
my %data = "config.json".IO.slurp.&from-json;

There’s more, but I’m just posting this quickly.

Cheers.

»

  • Category: Blog

I started this post as a rehash of Modules with some additional details. However, as I started running my examples, I found out that while the documentation on modules is good, it does not tell the full story regarding exports. As I do not want to write a manifesto on module exports, I’m going to assume you already read the above document and understand Perl 6 exports. If not, go read it and I’ll wait for you to return.

Ready? Okay, let’s go.

First, let me explain why I’m on this odyssey: I am writing a module, let’s call it Prometheus::Client, and that module really needs to export some symbols to be useful. However, due to how I’ve structured things, I would might prefer that the symbols I export actually be located in other compilation units, for example, the file declaring the Prometheus::Client::Metrics module. That means I need a way to re-export the exports of another module. I’ve done this before on a small scale for some things, but this is going to be much expanded. I wanted to make sure I knew what Perl 6 was doing before I started. In the process I discovered that the exports rabbit hole is deeper than I’d originally thought.

Let’s start with this simple statement from the Modules documentation I mentioned above:

Beneath the surface, is export is adding the symbols to a UNIT scoped package in the EXPORT namespace. For example, is export(:FOO) will add the target to the UNIT::EXPORT::FOO package. This is what Perl 6 is really using to decide what to import.

This is followed by the claim that this code:

unit module MyModule;
 
sub foo is export { ... }
sub bar is export(:other) { ... }

Is the same as:

unit module MyModule;
 
my package EXPORT::DEFAULT {
    our sub foo { ... }
}
 
my package EXPORT::other {
    our sub bar { ... }
}

If I were a “fact checker” I’d have to rate the quote “is the same as” regarding these two code snippets as “Half True” at best. It does, in fact, create these packages. However, that is not all it does.

This becomes clear if you understand the implications of the Introspection section of that same document. There it shows code like this:

use URI::Escape;
say URI::Escape::EXPORT::.keys;
# OUTPUT: «(DEFAULT ALL)␤»

And this:

say URI::Escape::EXPORT::DEFAULT::.keys;
# OUTPUT: «(&uri-escape &uri-unescape &uri_escape &uri_unescape)␤» 

If you aren’t careful, you won’t see it. I didn’t until I started playing around with code to see what works. Those is export lines in that MyModule example above do not only create the UNIT scoped package. They also create an OUR scoped package inside the current package namespace that can be used for introspection.

So, if you really want to replicate what is export does internally when it calls Rakudo::Interals.EXPORT_SYMBOL, you will have to do something like this for the complete MyModule-without-is export implementation:

unit module MyModule;

my package EXPORT::DEFAULT {
    our sub foo { ... }
}

my package EXPORT::other {
    our sub bar { ... }
}

{
    # Create a package object to be MyModule::EXPORT
    my $export = Metamodel::PackageHOW.new_type(:name('EXPORT'));
    $export.^compose;

    # Create a package object to be MyModule::EXPORT::DEFAULT
    my $default = Metamodel::PackageHOW.new_type(:name('DEFAULT'));
    $default.^compose;

    # Add the &foo symbol to the introspection package
    $default.WHO<&foo> := &UNIT::EXPORT::DEFAULT::foo;

    # Create a package object to be MyModule::EXPORT::other
    my $other = Metamodel::PackageHOW.new_type(:name('other'));
    $other.^compose;

    # Add the &bar symbol to the introspection package
    $other.WHO<&bar> := &UNIT::EXPORT::other::bar;

    # Add DEFAULT and other in EXPORT
    $export.WHO<DEFAULT> := $default;
    $export.WHO<other> := $other;

    # Add EXPORT in MyModule
    MyModule.WHO<EXPORT> := $export;
}

I haven’t yet found a cleaner way to do all that extra stuff at the bottom without doing all this introspective symbol table munging. However, code like this will get you pretty close to what is export does internally. I also haven’t even delved into what an EXPORT sub does by comparison. I’ll save that for another time.

I should also mention that I came across a module in the Perl 6 ecosystem named CompUnit::Util which might be useful to me on my Prometheus::Client problems and maybe even for setting up the two EXPORT modules too. However, I haven’t really dived into that any farther than noting the age of the code and that it makes use of undocumented-but-roasted methods of Stash to do whatever it does. I may look at it later when I’m less tired. Or maybe I will just decide to do what my new library in a completely different way. Whatev'.

Cheers.