An introduction to the Shell
Software Carpentry Nov 2011 Bootcamp
Cochlear Implants
A cochlear implant is a small electronic device that is surgically implanted in the inner ear to give deaf people a sense of hearing. More than a quarter of a million people have them, but there is still no widely-accepted benchmark to measure their effectiveness. In order to establish a baseline for such a benchmark, our supervisor got teenagers with CIs to listen to audio les on their computer and report:
the quietest sound they could hear the lowest and highest tones they could hear the narrowest range of frequencies they could
discriminate
[Link]
Cochlear Implants
To participate, subjects attended our laboratory and one of our lab techs played an audio sample, and recorded their data - when they rst heard the sound, or rst heard a difference in the sound. Each set of test results were written out to a text le, one set per le. Each participant has a unique subject ID, and a made-up subject name. Each experiment has a unique experiment ID.
[Link]
Our job is to do some preliminary analysis on that data. We need to: because the participant didn't complete all three tests);
Cochlear Implants
identify and label les that are missing data (for example, normalize the data (the rst version of the software
reported a score for each test in the range 0-9, but it was later "xed" to report scores in the range 1-10);
put the data into a database to make subsequent analysis
easier; and
calculate a few simple statistics, such as average scores for
each test by CI model and participant's age and sex.
[Link]
The experiment has collected 351 les so far, and we expect to get another 30-40 per week for the next couple of months, so we'd really like to automate the four steps above.
Data is a bit of a mess
data data Bert Frank_Richard
NOTES data_212 data_221 data_224 data_233 data_237 data_247 ...
THOMAS Lawrence gerdal jamesm
0213 0241 0244 0249 0274 0281 0287 ... Data0214 Data0225 Data0234 Data0248 Data0251 Data0253 ... Data0211 Data0218 Data0220 Data0227 Data0229 Data0230 .... NOTES data_217.txt data_219.txt data_226.txt data_228.txt data_231.txt data_236.txt ...
audioresult-00215 audioresult-00222 audioresult-00223 audioresult-00235 audioresult-00239 audioresult-00246 ....
Data is a bit of a mess
data
Inconsistent le names Some directories have Bert Frank_Richard THOMAS extraneous NOTES le multiple directories.
audioresult-00215 audioresult-00222 audioresult-00223 audioresult-00235 audioresult-00239 audioresult-00246 .... NOTES data_212 data_221 data_224 data_233 data_237 data_247 ... 0213 0241 0244 0249 0274 0281 0287 ...
data Lawrence gerdal jamesm
Data0214 Data0225 Data0234 Data0248 Data0251 Data0253 ... Data0211 Data0218 Data0220 Data0227 Data0229 Data0230 .... NOTES data_217.txt data_219.txt data_226.txt data_228.txt data_231.txt data_236.txt ...
Data is a bit of a mess
data job, by end of this Our session: Make data one directory (alldata) THOMAS Lawrence gerdal jamesm have all data les in there, all with .txt extension Get rid of NOTES les.
0213 0241 0244 0249 0274 0281 0287 ... Data0214 Data0225 Data0234 Data0248 Data0251 Data0253 ... Data0211 Data0218 Data0220 Data0227 Data0229 Data0230 ....
Bert
Frank_Richard
NOTES data_212 data_221 data_224 data_233 data_237 data_247 ...
audioresult-00215 audioresult-00222 audioresult-00223 audioresult-00235 audioresult-00239 audioresult-00246 ....
NOTES data_217.txt data_219.txt data_226.txt data_228.txt data_231.txt data_236.txt ...
Shell vs GUI
Presents a Command Line Interface (CLI, or CUI) vs GUI interface to your computer. Why on earth would you use a command line interface?
vs.
GUI: Operating
Very good at operating an existing system. Click on existing controls, use existing functionality.
GUI: Operating
Useful for basic computer operations, Operating existing software packages.
CLI - creating
For better or worse, a blank canvas Good for creating/ expressing new things. Programming in a GUI hard (but not impossible; Mac OSX Automator)
CLI - reproducable
Command lines can be cryptic to learn, but once you have the command, you can communicate it to others exactly.
GUIs - not as reproducable
Click on Filters, then Recent Then drag the green arrow down to the big grey box.. ... No, the other one.. ... Not there! Ok, lets start again...
Reproducable - stop wasting time rediscovering how to do things Automatable - can do the same thing hundreds of times easily without wasting time More time doing research
CLI makes you more productive
CLI makes you more productive
But theres a learning curve. Investment in future productivity.
GUI - Easy / Hard
Easy to learn/ discover Hard to use for big tasks productively.
CLI - Hard/Easy
Hard to learn/ discover Easy to use for big tasks productively.
GUI vs CLI
data GUI, we could With (painfully) do this one data at a time. le But in two gerdal jamesm THOMAS Lawrence months, when theres another 350 les, have to do it exactly again. No further ahead.
0213 0241 0244 0249 0274 0281 0287 ... Data0214 Data0225 Data0234 Data0248 Data0251 Data0253 ... Data0211 Data0218 Data0220 Data0227 Data0229 Data0230 ....
Bert
Frank_Richard
NOTES data_212 data_221 data_224 data_233 data_237 data_247 ...
audioresult-00215 audioresult-00222 audioresult-00223 audioresult-00235 audioresult-00239 audioresult-00246 ....
NOTES data_217.txt data_219.txt data_226.txt data_228.txt data_231.txt data_236.txt ...
GUI vs CLI
Were going to spend a data of time learning the lot shell today, towards THOMAS Lawrence gerdal jamesm doing this. But doing it the next time will be much faster.
0213 0241 0244 0249 0274 0281 0287 ... Data0214 Data0225 Data0234 Data0248 Data0251 Data0253 ... Data0211 Data0218 Data0220 Data0227 Data0229 Data0230 ....
data
Bert
Frank_Richard
NOTES data_212 data_221 data_224 data_233 data_237 data_247 ...
audioresult-00215 audioresult-00222 audioresult-00223 audioresult-00235 audioresult-00239 audioresult-00246 ....
NOTES data_217.txt data_219.txt data_226.txt data_228.txt data_231.txt data_236.txt ...
Open a Terminal
Mac: Applications/ Utilities/Terminal. (May as well drag this to the dock) Windows: Click on the cygwin icon. Linux:Various.
Terminal launches a shell
When you use a terminal, youre interacting with the shell A program provides access to les, network, other programs.
Shell
les
network other programs
Terminal launches a shell You type in
commands Shell interprets them Performs actions on its own, or (more often) launches other programs Like ipython Shell
les
network other programs
The shell
The shell most commonly used is bash (Bourne-Again SHell). There are others; mostly the same but some syntax is different. Type hello=world(no spaces). If you get an error about no command youre probably running tcsh. Type bash to start a bash shell and try again.
Basics - echo
Lets start by having the shell greet you:
segfault:~ ljdursi$ hello=world segfault:~ ljdursi$ echo Hello, world Hello, world segfault:~ ljdursi$ echo Hello, $hello Hello, world
Basics - File system
Now lets learn how to start moving around amongst our les and directories. This is easy to do in a GUI (click on folders), harder here, but you get very fast at it in the shell...
Basics - File system
Lets start poking around. Type pwd . Prints current working directory - where you are in the le structure. Type ls - that will list the les in that directory
Directories = folders
Often called folders because of how theyre represented in GUIs Directories are listings of les - can contain les or other directories
/Users/ljdursi
Desktop Documents
[Link]
Projects
Start at Home
When you launch a shell, it starts in your home directory /Users/[username] or /home/ [username] or something Top directory of all your stuff
/Users/ljdursi
Desktop Documents
[Link]
Projects
File types Would like to
know which entries are directories, which are plain les ls -F : labels directories with /, executables with *, etc.
segfault:~ ljdursi$ ls -F Applications/ [Link] Classes/ bin/ Codes/ configurationdata/ Desktop/ [Link] Documents/ drupal-7.9/ Downloads/ [Link]* ...
Changing Directories: cd
Choose one of the directories in your home directory and type cd [dir] And then ls -F Listing of contents of new directory cd without arguments will return to home dir
segfault:~ ljdursi$ cd Desktop segfault:Desktop ljdursi$ ls -F [Link] [Link] [Link] [Link] [Link] [Link] [Link] [Link] LFBP/ [Link] ... segfault:~ ljdursi$ cd segfault:~ ljdursi$ pwd /Users/ljdursi
Commands so far
A couple things to observe: Commands designed to be fast/easy to use. Pretty cryptic to learn.
echo pwd cd [directory] cd ls ls -F Prints output Print current directory Change directory Change directory to home Directory LiSting LiSting with Filetypes
Options: -something
ls (and it turns out lots of others) have options eg, -F or --help How do we know what the options are?
echo pwd cd [directory] cd ls ls -F Prints output Print current directory Change directory Change directory to home Directory LiSting LiSting with Filetypes
Manual: man pages
segfault:~ ljdursi$ man ls
Most programs have a manual page describing its use and the options. Good for nding out more about a command you already use; Less good for learning what a command does.
LS(1) NAME
BSD Gen
ls -- list directory content
SYNOPSIS ls [-ABCFGHLOPRSTUW@abcdefgh [file ...]
DESCRIPTION For each operand that names other than directory, ls dis well as any requested, assoc tion. For each operand that type directory, ls displays
Manual: man pages
Many programs have gazillions of options. No human being who has ever lived has known all the options to ls at same time. Over time you nd a few that you nd useful for your favourite commands.
segfault:~ ljdursi$ man ls LS(1) NAME
BSD Gen
ls -- list directory content
SYNOPSIS ls [-ABCFGHLOPRSTUW@abcdefgh [file ...]
DESCRIPTION For each operand that names other than directory, ls dis well as any requested, assoc tion. For each operand that type directory, ls displays
Using ls on other directories
segfault:~ ljdursi$ pwd /Users/ljdursi/Desktop segfault:Desktop Applications/ Classes/ Codes/ Desktop/ Documents/ Downloads/ ... ljdursi$ ls -F /Users/ljdursi [Link] bin/ configurationdata/ [Link] drupal-7.9/ [Link]*
If you give ls an argument, it will do the listing of that directory...
Using ls on other directories
segfault:~ ljdursi$ pwd /Users/ljdursi/Desktop segfault:Desktop ljdursi$ ls -F /Users/ljdursi/codes FLASH2.5/ athena3.1/ [Link] [Link] segfault:Desktop ljdursi$
If you give ls an argument, it will do the listing of that directory...
Using ls on other directories
segfault:~ ljdursi$ pwd /Users/ljdursi/Desktop segfault:Desktop ljdursi$ ls *.py [Link] [Link] [Link] segfault:Desktop ljdursi$ ls /Users/ljdursi/*.py /Users/ljdursi/[Link]
...or those les.
The shell interprets arguments
The shell takes my line ls *.py It looks for all les that are of the form [anything].py, and passes them as arguments to the ls command (/bin/ls).
segfault:~ ljdursi$ pwd /Users/ljdursi/Desktop segfault:Desktop ljdursi$ ls *.py [Link] [Link] [Link] segfault:Desktop ljdursi$ ls /Users/ljdursi/*.py /Users/ljdursi/[Link]
The shell interprets arguments
echo *.py works just as well; Shell generates list of .py les, puts them as arguments to echo echo echos them to screen.
segfault:~ ljdursi$ pwd /Users/ljdursi/Desktop segfault:Desktop ljdursi$ ls *.py [Link] [Link] [Link] segfault:Desktop ljdursi$ ls /Users/ljdursi/*.py /Users/ljdursi/[Link]
The shell interprets arguments
If the argument is a directory (or a le name), theres no processing to be done Passes it to ls
segfault:~ ljdursi$ pwd /Users/ljdursi/Desktop segfault:Desktop ljdursi$ ls -F /Users/ljdursi/codes FLASH2.5/ athena3.1/ [Link] [Link]
Directories in the shell
A couple things to observe: Directories in bash separated by /. (Windows by \). The top directory is /; under that, Users, under that, ljdursi, etc.
/ /Users ljdursi
Desktop Documents
[Link]
Projects
nt1d
mri
Directories in the shell
Can always specify a le by its full name, eg /Users/ ljdursi/ Projects/mri/ [Link] If you are in that directory, can just say [Link]
/ /Users ljdursi
Desktop Documents
[Link]
Projects
nt1d
mri
Directories in the shell
/ /Users
But can also specify relative paths; if youre in Projects, mri/[Link] is enough.
/Users/ljdursi
Desktop Documents
[Link]
Projects
nt1d
mri
Shortcuts for moving around directories:
A shortcut for one directory up is .. If Im in Desktop, ls .. does an ls of home directory; and ls ../Projects looks in my Projects directory.
/Users/ljdursi
Desktop Documents [Link] Projects
nt1d
mri
Shortcuts for moving around directories:
One dot means the current directory: . If Im in my home directory, ls ./ [Link] just lists the [Link] there.
/Users/ljdursi
Desktop Documents [Link] Projects
nt1d
mri
Shortcuts for moving around directories:
A shortcut for your home directory is ~ Wherever I am, ls ~ does a listing of /Users/ljdursi ls ~/Desktop does a listing of /Users/ ljdursi/Desktop.
/Users/ljdursi
Desktop Documents [Link] Projects
nt1d
mri
Looking at les
Desktop scbc-2011
Lets go into the data directory from svn:
branches your-branch data
Your svn location
segfault:data ljdursi$ ls -F data/ ex_data.txt segfault:data ljdursi$ cd data segfault:data ljdursi$ ls -F Bert/ Lawrence/ Frank_Richard/ THOMAS/ segfault:data ljdursi$ cd Bert segfault:Bert ljdursi$ ls audioresult-00215 audioresult-00332 audioresult-00451 audioresult-00222 audioresult-00350 audioresult-00453 audioresult-00223 audioresult-00353 audioresult-00460 audioresult-00235 audioresult-00355 audioresult-00466 ... segfault:Bert ljdursi$ alexander/ jamesm/ gerdal/
Your branch
segfault:~ ljdursi$ cd ~/Desktop/scbc-2011/branches/ljdursi-branch/data generate_data.py
Looking at a le
segfault:~ ljdursi$ file audioresult-00215 audioresult-00215: ASCII text
echo pwd cd [directory] cd ls ls -F man [cmd] file [filename]
Prints output Print current directory Change directory Change directory to home Directory LiSting LiSting with Filetypes MANual page for [cmd] What is in [lename]?
Looking at a le
segfault:Bert ljdursi$ file audioresult-00215 audioresult-00215: ASCII text segfault:Bert ljdursi$ file au<TAB> segfault:Bert ljdursi$ file audioresult-00
Tab completion! If you hit <TAB> when typing a lename, shell will complete what youre typing (as much as possible)
Looking at a le
segfault:Bert ljdursi$ file audioresult-00215 audioresult-00215: ASCII text segfault:Bert ljdursi$ file au<TAB> segfault:Bert ljdursi$ file audioresult-00
Other handy tip Up arrow lets you preview previous commands; can edit and/ or press <Return>
Looking at a le
segfault:Bert ljdursi$ file audioresult-00215 audioresult-00215: ASCII text segfault:Bert ljdursi$ more audioresult-00215 # Reported: Sun Jun 26 [Link] 2011 Subject: beyonceLennon177 Year/month of birth: 1993/09 Sex: N CI type: 20 Volume: 8 Range: 5 Discrimination: 7 segfault:Bert ljdursi$
Looking at a le
segfault:Bert ljdursi$ file audioresult-00215 audioresult-00215: ASCII text segfault:Bert ljdursi$ more audioresult-00215 # Reported: Sun Jun 26 [Link] 2011 Subject: beyonceLennon177 Year/month of birth: 1993/09 echo Prints output Sex: N pwd Print current directory CI type: 20 Volume: 8 cd [directory] Change directory Range: 5 cd Change directory to home Discrimination: 7 ls Directory LiSting segfault:Bert ljdursi$
ls -F man [cmd] file [filename] more [filename] cat [filename]
LiSting with Filetypes MANual page for [cmd] What is in [lename]? Prints out lename(s) by page Dumps out lename(s)
More vs Cat:
try more au* and then cat au* What's the difference?
cating les together
Dumping all the les together is how cat got its name - short for concatenate. Try cating all the les together into a new le:
segfault:~ ljdursi$ cat au* > all-results segfault:~ ljdursi$ more all-results
Redirection
[cmd] > [filename] takes what would have gone to the screen, creates a new le [filename], and redirects output to that le. Blows away previous contents of le if it had existed.
Redirection
[cmd] >> [filename] appends to [filename] if it exists. [cmd] < [filename] - programs input comes from le, as if you were typing.
cat - echos input
If cat isnt given lenames, it just dumps its input to the screen.
segfault:Bert ljdursi$ cat hello hello there there ^D
mv, cp
Weve created our rst le from the shell! We can make copies, or move the le around:
mv, cp
segfault:Bert ljdursi$ cp all-results all-results-2 segfault:Bert ljdursi$ ls all* ??? segfault:Bert ljdursi$ mv all-results all-results-3 segfault:Bert ljdursi$ ls all* ??? segfault:Bert ljdursi$ mv all-results3 .. ???
mv, cp - move, copy
segfault:Bert ljdursi$ cp all-results all-results-2 segfault:Bert ljdursi$ ls all* all-results all-results-2 segfault:Bert ljdursi$ mv all-results all-results-3 segfault:Bert ljdursi$ ls all* all-results all-results-3 segfault:Bert ljdursi$ mv all-results3 .. segfault:Bert ljdursi$ ls all* all-results segfault:Bert ljdursi$ ls .. Bert Lawrence alexander Frank_Richard THOMAS all-results-3 gerdal jamesm
rm - remove
Deletes (ReMoves) le. Does not move it to trash; deletes it. No safety net! (But - if the le is in version control, can recover it).
segfault:Bert ljdursi$ ls .. Bert Lawrence alexander Frank_Richard THOMAS all-results-3 gerdal jamesm
rm
segfault:Bert ljdursi$ ls -F .. Bert/ Lawrence/ alexander / Frank_Richard/ THOMAS/ all-results-3 gerdal/ jamesm/ segfault:Bert ljdursi$ rm ../all-results-3 segfault:Bert ljdursi$ ls -F .. Bert/ Lawrence/ alexander / Frank_Richard/ THOMAS/ jamesm/ gerdal/
mkdir, rmdir
To create and delete directories, use mkdir and rmdir. Uncharacteristically, rmdir protects you - you cant delete a directory with les in it Have to delete them rst
mkdir, rmdir
segfault:Bert ljdursi$ mkdir foo segfault:Bert ljdursi$ ls foo segfault:Bert ljdursi$ cp all-results foo segfault:Bert ljdursi$ ls foo all-results segfault:Bert ljdursi$ rmdir foo rmdir: foo: Directory not empty segfault:Bert ljdursi$ rm foo/all-results segfault:Bert ljdursi$ rmdir foo
wc - word count of text les
wc [filename] prints the lines, words, and characters (non-spaces) in a text le wc -l, wc -w, and wc -c print just the # of lines, words, and characters of the le try wc all-results (tab completion will work after the al)
wc
Weve just wced a cated le Should have same as totals of all les Lets try that: wc au*
wc
segfault:Bert ljdursi$ wc all-results 423 1124 6916 all-results segfault:Bert ljdursi$ wc au* ... 9 24 147 audioresult-00521 9 24 146 audioresult-00532 9 24 147 audioresult-00534 9 24 151 audioresult-00535 9 24 148 audioresult-00557 423 1124 6916 total
Dealing with too much output
wc au* printed out results for each le, and total - handy. But it provided too much output; couldnt see it all. How are we going to x that (using just what we know so far)?
wc, more
segfault:Bert ljdursi$ wc all-results 423 1124 6916 all-results segfault:Bert ljdursi$ wc au* > all-wcs segfault:Bert ljdursi$ more all-wcs
head, tail
segfault:Bert ljdursi$ head all-wcs ??? segfault:Bert ljdursi$ tail all-wcs ???
head, tail prints start, end of le
Useful options to head/tail: -n [number] : only rst/last n lines. (default = 10)
Pipeline of commands
This idea of chaining commands together the output from one becomes the input of another - is part of what makes the shell (and programming generally) so powerful.
Pipeline of commands
So far weve done
segfault:Bert ljdursi$ wc au* > all-wcs segfault:Bert ljdursi$ more all-wcs
Creates a temporary le we dont really care about; we just want to page through all the wc results.
Pipeline of commands
Interesting (honest, youll see) fact - like cat, if more isnt given a lename, it also reads from input: So this would also work:
segfault:Bert ljdursi$ wc au* > all-wcs segfault:Bert ljdursi$ more < all-wcs
Pipeline of commands
segfault:Bert ljdursi$ wc au* > all-wcs segfault:Bert ljdursi$ more < all-wcs
This combination of actions - output of one command goes straight into another - so common that shell has special facilities for this:
segfault:Bert ljdursi$ wc au* | more
Pipeline of commands
Allows you to chain together small pieces into a very powerful analysis pipeline. Lets look at another example:
sort sorts lines in a le
Lets create a short le and have sort sort it. Can write le in editor, but lets use our new cat-and-redirection skills:
segfault:Bert ljdursi$ cat > toBeSorted Ernie Bert Oscar Big Bird segfault:Bert ljdursi$
sort sorts lines in a le
segfault:Bert ljdursi$ cat toBeSorted Ernie Bert Oscar Big Bird segfault:Bert ljdursi$ sort toBeSorted Bert Big Bird Ernie Oscar
sort sorts lines in a le
Useful options to sort: -n : sort in numerical order (not lexicographic; eg, 101 < 30 without -n.) -k [number] : sort by the kth column. -r : reverses order (decreasing, not increasing)
sort the data les by size (in characters)
segfault:Bert ljdursi$ sort -n -k 3 all-wcs ... 9 24 151 audioresult-00535 9 24 152 audioresult-00286 9 24 152 audioresult-00353 423 1124 6916 total segfault:Bert .. 9 9 9 9 ljdursi$ sort -n -k 3 -r all-wcs 24 23 24 24 144 144 143 142 audioresult-00239 audioresult-00453 audioresult-00393 audioresult-00493
sort the data les by size (in characters)
segfault:Bert ljdursi$ wc au* | sort -n -k 3 ... 9 24 151 audioresult-00535 9 24 152 audioresult-00286 9 24 152 audioresult-00353 423 1124 6916 total segfault:Bert ljdursi$ wc au* | sort -n -k 3 ?? | more
Modify this to print only smallest, then only largest, data le.
segfault:Bert ljdursi$ wc au* | sort -n -k 3 ... 9 24 151 audioresult-00535 9 24 152 audioresult-00286 9 24 152 audioresult-00353 423 1124 6916 total segfault:Bert ljdursi$ wc au* | sort -n -k 3 ?? | more
Pop quiz!
Our rst shell script
So this is useful enough that we are going to write a script that contains this line. Will be a program that prints largest (say) data le in the directory. First, clean up:
segfault:Bert ljdursi$ rm all-wcs all-results toBeSorted
Our rst shell script
Create the following le, called biggest. More complex than toBeSorted: use an editor
#!/bin/bash wc * | sort -n -k 3 | tail -2 | head -1
Now run it with
segfault:Bert ljdursi$ source biggest
what do you get?
Our rst shell script
To make this into a real program, were going to tell the OS that this le is executable. Then the #!/bin/bash line will tell the OS to run this program with our shell, bash
segfault:Bert ljdursi$ chmod a+x biggest segfault:Bert ljdursi$ ./biggest
Largest range - grep
Largest number of characters in data le probably not super important for our analysis. How about experiment with largest range? Data les all have line Range: [Number]
segfault:Bert ljdursi$ grep Range audioresult-00557 Range: 2
grep outputs lines containing the rst input string in all of the les given.
segfault:Bert ljdursi$ grep Range * ???
Pop Quiz
Modify biggest to print out which experiment has the biggest Range. Quick tip - what column needs to be sorted? (And do we need the head/tail trick?)
Pop Quiz
Modify biggest to print out which experiment has the biggest Range. Quick tip - what column needs to be sorted? (And do we need the head/tail trick?)
segfault:Bert ljdursi$ more biggestRange #!/bin/bash grep Range * | sort -n -k 2 | tail -1
Arguments in bash scripts
Wed like to use this for each directory, but we dont want one copy in each directory. Lets move it up one level in directory, and modify it so it would work on any directorys les
segfault:data ljdursi$ more biggestRange #!/bin/bash grep Range $1/* | sort -n -k 2 | tail -1
Arguments in bash scripts
When you run a command in the shell, its name is put in argument 0 ($0) Any other arguments are $1, $2...
segfault:data ljdursi$ more biggestRange #!/bin/bash grep Range ${1}/* | sort -n -k 2 | tail -1
Arguments in bash scripts
segfault:data ljdursi$ ./biggestRange Bert Bert/audioresult-00384:Range: 10 segfault:data ljdursi$ ./biggestRange THOMAS THOMAS/0336:Range: 10
For loops in bash
Bash has for loops much like python does. We can use this to run our program on several directories:
For loops in bash
segfault:data ljdursi$ for dir in Bert gerdal jamesm > do > echo "The biggest range in directory " ${dir} " is:" > ./biggestRange ${dir} > done The biggest range in directory Bert is: Bert/audioresult-00384:Range: 10 The biggest range in directory gerdal is: gerdal/Data0559:Range: 10 The biggest range in directory jamesm is: jamesm/data_517.txt:Range: 10 segfault:data ljdursi$
nd
Wildcards are very powerful: From data/data directory, type: ls */*00* Finds les with 00 in name in any subdirectory Similarly: echo */*00* or
for i in */*00* ; do echo ${i}; done
nd
But can only match if you know the path (how many levels of dirs down) And can only match by lename. find is a tool which lets you nd les anywhere below a given directory, based on arbitrary criteria.
find: do the following
segfault:data ljdursi$ find . -print | more
directory to start
What to do to the le
find: can execute arbitrary commands
segfault:data ljdursi$ find . -exec echo {} \; | more
directory to start
What to do. {} gets lled in with lename; command ends with \;
find: can execute arbitrary commands
segfault:data ljdursi$ find . -exec echo {} \; | more
directory to start
What to do. {} gets lled in with lename; command ends with \;
find: can choose les by type
segfault:data ljdursi$ find . -type f -print | more
directory to start
Only les (type f) get printed; directories are excluded
find: can choose les by type, name
segfault:data ljdursi$ find . -type f -name *00* -print | more
directory to start
Only les with 00 in their names; can chain together conditions
find: can choose les by contents
find . -type f -exec grep Volume {} \; -print | more
Only search les
If grep returns true (eg, contains Volume), then matches
Assignment
Copy all of the data les from data/data/.. to a new directory, cleaneddata. All data les must end in .txt Get rid of the NOTES les. Need to have this done before databases section (next)
Assignment
Do it manually: that works. Try to nd a solution which will work next time it needs to be done, too. Play with things on the command line.. Many ways to do this! Bonus points: put it in a script!
echo Prints output pwd Print current directory cd [directory] Change directory cd Change directory to home ls Directory LiSting ls -F LiSting with Filetypes man [cmd] MANual page for [cmd] file [filename] What is in [lename]? more [filename] Prints out lename(s) by page cat [filename] Dumps out lename(s) wc [filename] Line/word/char count of le mv [src] [dest] Move le cp [src] [dest] Copy le rm [filename] Delete le head [filename] First lines of le tail [filename] Last lines of le sort [filename] Sort lines of le mkdir [filename]Create directory rmdir [filename]Remove directory grep Searches input for text for..do..done for loops in bash find Searches for les
