RSH 8: Command line arguments and running tasks in parallel

Video

Collaborative notes taken during the session

Icebreaker

Where are you watching from and how is the weather at your place?

• Tromso, 5 C, heavy rain since days
• Helsinki, 15-20C, cloudy but not much rain (feels like +1)
• Metro between Kamppi and Rautatieasema, dark outside
• Stockholm, 13.3C, dark but dry

First time with command line

• Commodore 64 :D (yes I am that old) +1
• ZX spectrum
• FreeBSD vtty

Links mentioned during the stream

• http://us-rse.org/rse-stories/
• https://github.com/ResearchSoftwareHour/demo-parallel-tasks

Questions

• bash: "for" unfortunately does not end with "rof"

  • good point :-)
  • haha, great!

• I am not familiar with this expression. It get's parsed? What does it mean, please?

  • Intuitively: "parsed" as in "read and understood" by the script

• Have a look at fire which is like click, but less boilerplate (but maybe not so powerful as click) +1

  • Fairly used, but not as popular as click.

• +1 for reading from file (especially a struct like written in json)

• What are the underscores for, please?

  • with -I _ I instructed xargs to replace underscores by the files it got from find, this was my choice, I could have done -I x instead and then it would replace "x"

• Python:

from multiprocessing import Pool
from pathlib import Path

def task(filename):
    # computationally intensive task
    # or even shell process using os.system(..) or subprocess.run(..)
    print(filename)


with Pool() as pool:
    files = Path.cwd().glob("*.jpg")
    pool.map(task, files)

Ideas that were suggested via Twitch chat

• offer material both for novices and experts
• working with grib and netcdf files
• machine learning tutorial