Asynchronous Workflow Using Julia Tutorial

Date: January 7 2023

Summary: A thorough tutorial on how to use Julia for asynchronous workflows to do many things, at once!

Keywords: #blog #archive #julia #programming #literate #workflow #asynchronous #weave #browser #sync #node

Bibliography

Not Available

How To Cite
References:
Discussion:

Motivation

I sometimes find myself wanting to execute a Julia file I have just edited. This is different than what the excellent Revise.jl package (created by Tim Holy) does in that I want to execute the full file and not just update function definitions. Moreover, I may just want Julia to do something else entirely upon updating a file or directory. This can be very helpful in analysis workflows as well as literate programming. Creating an extended post on my process for this was inspired by a Julia Discourse post I provided an answer to.

Set-Up

Here is how I recommend setting up your Julia environment to follow this tutorial (I assume you have Julia installed correctly):

Create an empty directory (folder) somewhere on your computer. (NOTE: For sake of this tutorial, I will refer to it as the "test" folder).
Within your favorite text editor (like Notepad, Vim, VSCode, etc.), open this directory.
Open and create a file called "tmp.jmd" (NOTE: Keep the file and your editor open as we will be coming back to this file quite a lot).
Open your Julia REPL within the directory and activate a temporary environment within package mode:

pkg> activate --temp

Add into this temporary environment the following packages:

pkg> add FileWatching, Weave

Install node onto your machine.

I suggest using the great tool, nvm, which allows easy management of node versions.

Install the node package, browser-sync, which allows for syncing of webpages based on updates.

Here is the command you can use in your command line:

npm i -g browser-sync

NOTE: Steps 6 and 7 are not necessary if you are not interested in the section on literate programming

Simple Workflow Process

There are two ways to start this workflow. One blocking (i.e. you cannot use the Julia REPL while working) and another spawning an asynchronous background process to enable you to work with the same REPL. Here is gif showing these workflows in action:

Blocking Workflow

This creates a synchronous task in your Julia REPL that can then run whatever command you want while you monitor a specific file (or folder). In this case, I am making the task print a statement that says “Change Detected!” to my REPL whenever there is a change in the file called “tmp.jmd”.

using FileWatching

while true 
    watch_file("./tmp.jmd")
    println("Change Detected!")
end

This workflow is not optimal as this loop continues and locks your REPL from doing anything else.

Asynchronous Workflow

This creates an asynchronous task in the background of your Julia REPL that can then run whatever command you want while you monitor a specific file (or folder). In this case, I am making the task print a statement that says “Change Detected!” to my REPL whenever there is a change in the file called “tmp.jmd”.

using FileWatching

@async while true 
    watch_file("./tmp.jmd")
    println("Change Detected!")
end

From here, you could then still interact with this REPL while this task is running in the background. Furthermore, you can change the line that says println("Change Detected!") to whatever command (or commands) you want to trigger.

Asynchronous Workflow for Literate Programming

This workflow allows one to work with a Julia Markdown document with Weave.jl to preview your work in real time. There a few steps to start the workflow but here is how it looks when in operation:

First, execute the following snippet in your Julia REPL to start the asynchronous process to watch for changes and tell Julia to weave the file we will modify:

using FileWatching
using Weave

@async while true 
    watch_file("./tmp.jmd")
    weave("./tmp.jmd")
end

Next, run this in a separate terminal:

browser-sync start --server --start --file "./*.html"

or this command within your Julia REPL (NOTE: This will make your REPL session very cluttered but you can still use your REPL):

@async run(`browser-sync start --server --start --file "./*.html"`)

At this point, you should see browser-sync spit out a lot of information that looks something like this:

[Browsersync] Access URLs:
 --------------------------------------
       Local: http://localhost:3000
    External: http://192.168.1.186:3000
 --------------------------------------
          UI: http://localhost:3001
 UI External: http://localhost:3001
 --------------------------------------
[Browsersync] Serving files from: ./

You'll want to grab that URL that looks like http://localhost:3000 in the Local spot and then open that in your web browser. With the URL that you get (it could be on a different port), navigate to http://localhost:3000/tmp.html.

Finally, when you have this all in place, go ahead and add this code block into the tmp.jmd file that was created:

# Hello World 

This is a Julia Markdown file. 

```julia 
2 + 2
```

You should see a few things happen. You'll see a few messages from the Julia REPL stating that Weave.jl is weaving your document to your desired output. Then, you'll see browser-sync say something about updating or syncing. Finally, you will see your browser update to the latest version of your weaved document.

Feel free to play around with this more and see the dynamism of the workflow! I tend to do this when I am iteratively developing reports within Julia and want to tinker within the REPL at the same time I am creating documents. Having to wait for rendering Weave.jl documents and the like was a pain and this took the pain away.

Asynchronous Workflow Using Julia Tutorial

Bibliography

Table of Contents

Motivation

Set-Up

Simple Workflow Process

Blocking Workflow

Asynchronous Workflow

Asynchronous Workflow for Literate Programming

Conclusion

How To Cite

References:

Discussion: