% echo $a
100

% echo $b
foo

% echo $c
ls -l

% echo $d
This is a complete sentence.

% echo $a $b $c $d
100 foo ls -l This is a complete sentence.

% $c
total 104
drwxr-xr-x 2 phys210d public  4096 Aug 29 14:54 Desktop
drwxr-xr-x 3 phys210d public  4096 Aug  1 16:02 Documents
drwxr-xr-x 2 phys210d public  4096 Aug  1 16:02 Downloads
drwxr-xr-x 2 phys210d public  4096 Aug  1 16:02 Music
drwxr-xr-x 2 phys210d public  4096 Aug  1 16:02 Pictures
drwxr-xr-x 2 phys210d public  4096 Aug 16  2012 Public
-rw-r--r-- 1 phys210d public  3433 Sep  3 12:11 README-EXERCISE2
                             .
                             .
                             .
drwx------ 2 phys210d public  4096 Oct  3  2013 maplepgm-soln
drwxr-xr-x 2 phys210d public  4096 Oct  3  2013 maplepgm2
drwxr-xr-x 2 phys210d public 16384 Nov 29  2013 matlab
drwxr-xr-x 2 phys210d public  4096 Sep 17  2013 numbers
drwxr-xr-x 3 phys210d public  4096 Oct  7  2013 octave-work

phys210d@cord ~]$ 

phys210d@cord ~]$ echo $PS1
[\u@\h \W]$

#-----------------------------------------------------------------------
# Set the command prompt to include the hostname (\h), the username (\u), 
# and the last component (i.e. the basename) of the current working 
# directory (\W).  See 'man bash' and search for 'PROMPTING' for more 
# information on configuring the prompt.
#
# If you want to restore the lab-default prompt which includes the *full*
# pathname, comment-out or delete the following line.
#-----------------------------------------------------------------------
PS1="[\u@\h \W]$ "

phys210d@cord ~]$ PS1='% '
%

% source ~/.bashrc
[phys210d@cord ~]$

Note:

You can safely ignore the error message ...

bash: TMOUT: readonly variable

% env

LC_PAPER=en_CA.UTF-8
LC_ADDRESS=en_CA.UTF-8
XDG_SESSION_ID=c229
LC_MONETARY=en_CA.UTF-8
HOSTNAME=cord
GPG_AGENT_INFO=/tmp/gpg-WDqNl5/S.gpg-agent:10992:1
TERM=xterm
SHELL=/bin/bash
                         .
                         .
                         .
TMP=/tmp
LOGNAME=phys210d
LC_CTYPE=en_CA.UTF-8
SSH_CONNECTION=142.103.234.164 34375 142.103.243.203 22
LESSOPEN=|/usr/bin/lesspipe.sh %s
META_CLASS=download
XDG_RUNTIME_DIR=/run/user/15008
DISPLAY=localhost:10.0
LC_TIME=en_CA.UTF-8
LC_NAME=en_CA.UTF-8
_=/bin/env

% echo $HOME
/home/phys210d

% printenv HOME
/home/phys210d

% printenv HOME USER LOGNAME
/home/phys210d
phys210d
phys210d

% FOO='foo'
% export FOO

% export BAR='bar'

% echo $FOO $BAR
foo bar

% printenv FOO BAR
foo
bar

% LOL='lol'
% export lol

% echo $FOO $LOL
foo lol

% printenv FOO LOL
foo

% printenv SHLVL
2

% bash
%

% printenv SHLVL
3

% echo $FOO $BAR
foo bar

% echo $a $b $c $d

%

% exit
exit
% 

% printenv SHLVL
2

Bash pattern matching

Recall that pattern matching allows us to select filenames that match a specified pattern (naturally!).

Let's do our work in a directory that contains a LOT of files (2350 to be exact)

cd ~phys210/words

You can expect the following ls command to take a bit of time! It will also generate so much output that it will overflow the terminal's buffer, and you won't be able to scroll back [using the mouse roller button] to see the first part of the output)

% ls

A                       emanate               polysyllabical
Acrania                 emblazer              pomacentroid
Adamic                  embroiler             ponderant
Adoxaceae               emetomorphine         pooa
Akamnik                 emulant               poppethead
Aladdinize              enarbor               porite
                          .
                          .
                          .

Now, let's select various sets of filenames using pattern matching.

Get listings of files whose names:

1. Begin with z

   % ls z*
   
   zabtie     zenithward   zoa            zoomantist  zygal
   zarabanda  zimentwater  zonoplacental  zoospore    zymologic
   

2. Have a z in them

   % ls *z*
   
   Aladdinize            epitomize             nonsympathizer      torporize
   automatize            Erethizontidae        Nowroze             tzaritza
   azotate               femalize              ozonoscope          uncriticizing
   bacterize             frizziness            paronymize          unminimized
   bedazzlingly          glycogenize           perfectionizement   unmunicipalized
                                   .
                                   .
                                   .
   disequalize           monzonitic            semihydrobenzoinic  zoomantist
   dizenment             muscularize           skeletonization     zoospore
   electrocauterization  Nazerini              squeezing           zygal
   electrolyzer          nomadize              subzygomatic        zymologic
   emblazer              nonanalyzable         sucivilized
   

3. End with z

   % ls *z
   
   berkowitz  bruzz  untz
   

4. Are three characters long

   % ls ???
   
   ask  bat  icy  jib  Ker  neo  nix  old  Ona  Ova  ree  she  vex  zoa
   

5. Begin and end with a vowel (here I won't include 'y' or 'Y' as vowels):

   % ls [aAeEiIoOuU]*[aAeEiIoOuU] 
   
   Acrania           aelurophobe       evocable         oxyaphia
   Adoxaceae         albumoscope       exflagellate     oxyphile
   Aladdinize        alemana           existence        ozonoscope
                             .
                             .
                             .
   aceanthrene       escarbuncle       overobedience    uranite
   acetylsalicylate  esophago          oversimple       urubu
   acolyte           estivage          overturnable     usucaptable
   addlepate         euryscope         oxane
   

6. Begin and end with a consonant, note consonant = not a vowel, and the caret/hat (^) reverses the sense of the match set

   % ls [^aAeEiIoOuU]*[^aAeEiIoOuU]
   
   Baldwin                 gastrohysterectomy   psorospermial
   Baltis                  gastrular            psycholepsy
   Bandhor                 gelatin              psychosurgery
                     .
                     .
                     .
   gallowglass             pseudocultivated     zoomantist
   gamont                  pseudoholoptic       zygal
   ganner                  pseudonarcotic       zymologic
   garrulously             pseudoracemism
   

7. Begin with an a, end with a y, and have 6 characters

   % ls a????y
   
   archly  assify  autecy
   

Pattern matching exercise 1

Now devise and execute some of your own pattern matches using the files in ~phys210/words.

Try to be creative, and be prepared to show me or one of the TAs your handiwork.

End exercise

Bash pattern matching can be used in conjunction with any command. For example, let's create a temporary directory, and copy all of the files that have the character sequence 'and' in their names to that directory

Execute the following---again, be very careful with your typing and note that you can type your actual login name rather than $LOGNAME should you wish:

% mkdir -p /tmp/$LOGNAME/mywords
% cp *and* /tmp/$LOGNAME/mywords

Provided that you have issued these commands correctly, you can cd to the temporary directory, and verify that the copying succeeded, as follows

% cd /tmp/$LOGNAME/mywords
% ls

Bandhor      androgenesis  handclap          ribbandry        toyland
Bulanda      cropland      heterandry        sandless         triandrian
Candlemas    distilland    landdrost         sleeveband       wanderluster
Jutlandish   expand        misunderstanding  stylomandibular  zarabanda
Maorilander  grandisonant  northland         substandard

Return to the original words directory ...

% cd ~phys210/words

% cp *and* /tmp/$LOGNAME/words not-a-dir
cp: target 'not-a-dir' is not a directory

Pattern matching exercise 2

End exercise

History/ Events

% history

Standard input & standard output

• standard input (stdin) is the default source of input
• standard output (stdout) is the default destination of output

% cat
isn't
isn't
this
this
fun
fun
?
?
^D

Output redirection & output append redirection

Output redirection

% cd
% mkdir redirect
% cd redirect
% pwd
/home/phys210d/redirect

% cp ~phys210/redirect/* .
% ls
fireice  night  pools  road  snowy

Look at snowy, for example, using the more command as necessary.

Note:

To control more, type

• space to go forward a page
• b to go back a page
• q to quit

% more snowy

Whose woods these are I think I know.
His house is in the village, though;
              .
              .
              .
The woods are lovely, dark and deep,
But I have promises to keep,
And miles to go before I sleep,
And miles to go before I sleep.

% ls -l > ls-output

% cat ls-output

total 20
-rw-r--r-- 1 phys210d public   0 Sep  6 22:35 ls-output
-rw-r--r-- 1 phys210d public 250 Sep  6 22:35 fireice
-rw-r--r-- 1 phys210d public 614 Sep  6 22:35 night
-rw-r--r-- 1 phys210d public 539 Sep  6 22:35 pools
-rw-r--r-- 1 phys210d public 752 Sep  6 22:35 road
-rw-r--r-- 1 phys210d public 555 Sep  6 22:35 snowy

Output append redirection

% cat fireice >> poems
% cat night >> poems
% cat pools >> poems
% cat road >> poems
% cat snowy >> poems

% more poems

Some say the world will end in fire,
Some say in ice.
From what I've tasted of desire
I hold with those who favor fire.
          .
          .
          .
The woods are lovely, dark and deep,
But I have promises to keep,
And miles to go before I sleep,
And miles to go before I sleep.

% RM poems
% cat fireice night pools road snowy > poems
% more poems

Some say the world will end in fire,
Some say in ice.
From what I've tasted of desire
I hold with those who favor fire.
          .
          .
          .
The woods are lovely, dark and deep,
But I have promises to keep,
And miles to go before I sleep,
And miles to go before I sleep.

End of Unix Lab 3: Work on HW 1 or vacate the premises as you wish ...

kate (gedit) & .bashrc exercise

I have also pointed out that the default bash prompt includes the trailing component of the full pathname of the working directory; e.g. execute the following (where I am now using the real prompt for user phys210d). Observe that the first cd command ensures that we start from our home directory ...

[phys210d@cord ~]$ cd 

[phys210d@cord ~]$ cd redirect

[phys210d@cord redirect]$ pwd

/home/phys210d/redirect

To eliminate the need to type pwd after most/all cd's, we can change the prompt variable, PS1 so that it displays the full, rather than truncated, pathname. You may find this useful, at least until you get more familiar with command line work.

The change is simple: we need to modify the line in ~/.bashrc that reads ...

PS1="[\u@\h \W]$ "

PS1="[\u@\h \w]$ "

Exercise per se

1. First, in a terminal window change to your home directory and make a backup copy of ~/.bashrc:

   % cd
   % cp .bashrc .bashrc.O
   

   If you are prompted to confirm overwrite ...

   cp: overwrite '.bashrc.O'? 
   

   ... answer y

2. Now, from within the same terminal window, edit ~/.bashrc with kate or gedit ...

   % kate .bashrc
   

   ... or ...

   % gedit .bashrc
   

3. Make the change to the setting of PS1, and save the file.

4. Caution! Do not exit the editor or the terminal window in which you started the editor yet!

5. Open a new terminal window, and test that the desired change has taken effect, e.g. ...

   [phys210d@cord ~]$ cd /phys210/$LOGNAME/public_html
   
   [phys210d@cord /phys210/phys210d/public_html]$ 
   

   ... or provided that you completed the defining aliases exercise ...

   [phys210d@cord ~]$ cdweb
   
   [phys210d@cord /phys210/phys210d/public_html]$
   

   ... where cdweb in the above is the alias that I defined to change to my web directory; the name of your corresponding alias will probably be different (again assuming that you did complete the exercise.

   Note that the prompt is, of course, considerably longer than it was previously when the working directory is, e.g., your web directory.

6. Once you are sure that you have made the change correctly, it is safe to exit the editor and, should you choose, exit the terminal window from which you were editing.

7. If the change was not successful, return to step 3 and try again, asking for assistance as necessary.

8. Again, the reason that we don't want to exit the original terminal window until we are sure that the changes to ~/.bashrc are correct is that there's the distinct possibility that incorrect changes could cripple our ability to start a new shell. In that case, and assuming that you can't debug the ~/.bashrc, you can always revert to the backup via ...

   % cd
   % cp ~/.bashrc.O ~/.bashrc
   

   ... and where you will have to confirm the overwrite. It's best to be doubly cautious and open a new terminal once you've restored the original, just to be sure that nothing else went awry. And, as always, ask for help if you run into difficulties.

9. Finally, if you don't like the change, just revert to the backup as has just been described.

Input redirection & pipes

% cd 
% mkdir numbers 
% cd numbers 
% pwd 
/home/phys210d/numbers

Copy the file ~phys210/numbers/random100 to the working directory (~/numbers) and view the contents of your copy of random100.

% cp ~phys210/numbers/random100 .
% more random100

15
13
6
2
12
4
6
5
17
8
5
2
18
--More--(12%)   

random100 contains a list of 100 randomly generated numbers each in the range 1 to 20, with one number per line.

Verify that there are indeed 100 numbers (lines) in the file by using the wc (word count) command with the -l (list number of lines only) option

% wc -l random100

100 random100

As mentioned briefly in the Unix notes, sort is a powerful command that can be used to (surprise, surprise) sort lines from files or standard input according to various criteria. By default (i.e. if executed with no arguments), sort will

1. Read lines from standard input until an end-of-file (EOF) is encountered. Recall that you can generate an EOF at the terminal by typing ^D (Control-d)
2. Sort the lines in alphabetical order, using the collating sequence, A, B, C, ... Y, Z, a, b, z, ..., y, z
3. Output the sorted lines to standard output

Recall that any command that by default reads and writes to stdin and stdout respectively is known as a filter.

In this exercise we first want to sort the numbers in random100 in ascending numerical order. To do this, we use the -n option (for numeric sort), and redirect the standard input from random100. By default sort outputs to standard output, so we will save the sorted values using output redirection. So let's do the sorting twice, the first with the output going to stdout, the second with output going to the file random100-sorted

% sort -n < random100
1
2
2
2
2
.
.
.
19
20
20
20
20
20

% sort -n < random100 | more
1
2
2
2
2
2
2
2
3
3
3
3
4
4
4
--More--   

% sort -n < random100 > random100-sorted

% more random100-sorted
1
2
2
2
2
2
2
2
3
3
3
3
4
4
4
--More--(11%)      

Question

Why does more show the percentage of the random100-sorted file that has been displayed when it fills the screen, but does not show a percentage when 'sort -n < random100 | more' is executed?

End question

There's another Unix/Linux command, uniq, which like sort, is a filter and which collapses any number of consecutive lines in its input which are identical to a single instance of the line. Thus, we can use sort, uniq and a pipe to generate an ordered list of which integers are contained in random100:

% sort -n < random100 | uniq
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

A quick scan indicates that random100 contains at least one instance of each integer between 1 and 20 inclusive, but we can make an even longer pipeline just to make sure!

% sort -n < random100 | uniq | wc -l
20

We can get sort to sort in descending order using the -r option

% sort -n -r < random100 | uniq

20
19
18
17
16
15
14
13
12
11
10
9
8
7
6
5
4
3
2
1

And, finally, note that the output from a pipeline (not just a single command) can be redirected to a file

% sort -n < random100 | uniq | wc -l > nuniq
% cat nuniq

20

You can construct pipelines that are as long as you wish, but for each pipeline, there can be

1. At most one input redirection; i.e. at most one < per pipeline.
2. At most one output redirection; i.e. at most one > per pipeline.

In terms of the garden hose analogy I made previously, no matter how many hoses I connect together, there is only one place where the water enters, and one where it leaves. In general, the input redirection will be associated with the first command in the pipeline, and the output redirection with the last.

This is an important point that often trips-up novices.

IMPORTANT!

If you haven't yet thought of this, note that in building up pipelines and other similar constructs (sequences that use input/output redirection), you can save a lot of typing by using the history mechanism (up/down arrows) to retrieve then modify previously issued command lines.

grep & regular expressions

Let's first use grep with the file 'poems' that we generated by concatenating the 5 Robert Frost poems in ~/redirect.

Just to ensure that everyone is working with the same file, execute the following

% cd 
% mkdir grep 
% cd grep 
% cp ~phys210/grep/poems .

% ls -l
-rw-r--r-- 1 phys210d public 2710 Aug 29 12:51 poems

1. All lines in poems that contain the word 'the' (case sensitive)

   % grep the poems
   
   Some say the world will end in fire,
   I have been one acquainted with the night.
   I have outwalked the furthest city light.
   I have looked down the saddest city lane.
                .
                .
                .
   Whose woods these are I think I know.
   His house is in the village, though;
   Between the woods and frozen lake
   The darkest evening of the year.
   To ask if there's some mistake.
   The only other sound's the sweep
   

   (Note how the matches are displayed in the terminal in a different color [red in my case]. This is a configurable feature of grep; i.e. you can control what color is used for highlighting. Use 'man grep' for full details.)

   How many lines are there that contain 'the'?

   % grep the poems | wc -l
   
   30
   

   And how many lines are there that contain 'the' (case insensitive), so includes The (and conceivably ThE, thE, etc.)? Here we use the -i option, for 'ignore case'

   % grep -i the poems | wc -l
   
   33
   

2. All lines in poems that start with 'I ' (so that 'I' is in the first column of the line and is followed by at least one space).

   Here we use the ^ (caret) to "anchor" the search pattern to the beginning of the line, and since the search pattern includes whitespace, we'll need to enclose the regular expression (regexp) in forward quotes. In fact, just to be safe, we'll quote all the regexps in the subsequent examples (this is a good habit to develop)

   % grep '^I ' poems
   
   I hold with those who favor fire.
   I think I know enough of hate
   I have been one acquainted with the night.
   I have walked out in rain-and back in rain.
   I have outwalked the furthest city light.
   I have looked down the saddest city lane.
   I have passed by the watchman on his beat
   I have stood still and stopped the sound of feet
   I have been one acquainted with the night.
   I doubted if I should ever come back.
   I shall be telling this with a sigh
   I took the one less traveled by,
   

   If we forget to anchor the search at the beginning of the line using the ^, the result is different:

   % grep 'I ' poems 
   
   I hold with those who favor fire.
   I think I know enough of hate
   I have been one acquainted with the night.
   I have walked out in rainâand back in rain.
   I have outwalked the furthest city light.
   I have looked down the saddest city lane.
   I have passed by the watchman on his beat
   I have stood still and stopped the sound of feet
   I have been one acquainted with the night. 
   And sorry I could not travel both
   And be one traveller, long I stood
   And looked down one as far as I could
   Oh, I kept the first for another day!
   I doubted if I should ever come back.
   I shall be telling this with a sigh
   I took the one less traveled by,
   Whose woods these are I think I know.
   But I have promises to keep,
   And miles to go before I sleep,
   And miles to go before I sleep. 
   

3. All lines in poems that start with either 'I' or 'F' (whitespace not required after either letter in this case):

   % grep '^[IF]' poems
   
   From what I've tasted of desire
   I hold with those who favor fire.
   I think I know enough of hate
   Is also great
   I have been one acquainted with the night.
   I have walked out in rain-and back in rain.
   I have outwalked the furthest city light.
   I have looked down the saddest city lane.
   I have passed by the watchman on his beat
   I have stood still and stopped the sound of feet
   I have been one acquainted with the night.
   From snow that melted only yesterday.
   In leaves no step had trodden black.
   I doubted if I should ever come back.
   I shall be telling this with a sigh
   I took the one less traveled by,
   

4. All lines in poems that start with any upper case letter in the inclusive range T through Z

   % grep '^[T-Z]' poems
   
   To know that for destruction ice
   When far away an interrupted cry
   These pools that, though in forests, still reflect
   The total sky almost without defect,
   Will like the flowers beside them, soon be gone,
   The trees that have it in their pent-up buds
   To darken nature and be summer woods
   To blot out and drink up and sweep away
   These flowery waters and these watery flowers
   Two roads diverged in a yellow wood,
   To where it bent in the undergrowth;
   Then took the other, as just as fair,
   Though as for that the passing there
   Yet knowing how way leads on to way,
   Two roads diverged in a wood, and I
   Whose woods these are I think I know.
   To watch his woods fill up with snow.
   To stop without a farmhouse near
   The darkest evening of the year.
   To ask if there's some mistake.
   The only other sound's the sweep
   The woods are lovely, dark and deep,
   

5. All lines in poems that do not start with an upper case letter or a space.

   This one is a bit tricky since we use ^ for two distinct purposes, one to anchor the search to the beginning of the line, the other to negate the sense of the range A-Z plus space, i.e. [^A-Z ] means 'not in the inclusive range A through Z nor a space.

   % grep '^[^A-Z ]' poems
   

   ... and there's no output, so true to form for this type of poetry, the first word of each line is capitalized (assuming every non-blank line starts in the first column).

6. All lines in poems that contain only one or more spaces.

   This one is also a bit tricky. To ensure that there is at least one space the regular expression needs to have an explicit space. We then follow that with a single character range [ ] that can occur zero or more times, i.e. [ ]*. Finally, we need to anchor the search to both the beginning of the line using ^ and the end of the line using $. Also note that I deliberately added spaces to empty lines in ~phys210/grep/poems to make this example generate some matches.

   % grep '^ [ ]*$' poems
   

   You should "see" many blank lines on the screen at this point. How many? Again, pipe the output into wc -l

   % grep '^ [ ]*$' poems | wc -l
   
   24
   

   Two more examples that may help you with the homework ...

7. All lines in poems that contain both 'and' and 'the' (case insensitive). We can do this easily by piping two greps together

   % grep -i 'and' poems | grep -i 'the' 
   
   I have stood still and stopped the sound of feet
   And further still at an unearthly height
   And like the flowers beside them, chill and shiver,
   These flowery waters and these watery flowers
   And having perhaps the better claim,
   And that has made all the difference.
   Between the woods and frozen lake
   The woods are lovely, dark and deep,
   

   Note carefully that the first grep in the pipeline is supplied with poems as a file argument, but that the second grep is reading from the standard output of the first one, and hence does not need, and should not be given, a file argument

8. All lines in poems that are exactly 30 characters long.

   Here you need to be careful to type precisely 30 periods (.) between the ^ and $ that match the beginning and end of a line, respectively. Recall that '.' matches any single character

   % grep '^..............................$' poems
   
   But if it had to perish twice,
   Somewhere ages and ages hence:
   

grep exercise

Again, note that it is not necessary to type 30 periods again: just recall and edit the previous command using the arrow keys as usual.

Try to predict what will happen, i.e. how you would state in words which lines will match, before you execute the command.

How many lines match this time?

End exercise

Now let's use grep to look for things in our history and list of environment vars. Here, grep will be used in "filter mode", since history or env will supply the input to grep via a pipe. The output, will, of course, be user-dependent

1. Look in the history for commands that include the text grep itself.

   % history | grep grep
                    .
   
                    .
                    .
    1132  mv grep numbers redirect words thedate Save
    1227  cd; mkdir grep; cd grep; cp ~phys210/grep/poems .
    1229  grep the poems
    1230  grep the poems | wc -l
    1231  grep -i the poems | wc -l
    1232  grep '^I ' poems
    1233  grep '^[IF]' poems
    1234  grep '^[T-Z]' poems
    1235  grep '^[^A-Z ]' poems
    1236  grep '^ [ ]*$' poems
    1237  grep '^ [ ]*$' poems | wc -l
    1238  grep -i 'and' poems | grep -i 'the'
    1239  grep '^..............................$' poems
    1240  history | grep grep
   

2. Look in the history for cd commands and only cd commands.

   The regular expression in this case is fairly complex, so take a few moments to study it to try to ensure that you understand what is going on.

   Note the constructions that match:

   1. The command-line numbers, which are all integers, so match "zero or more consecutive digits". Here we take advantage of the fact that every history entry must contain a number, so we don't have to use [0-9][0-9]* rather than [0-9]*.

   2. The spaces before and after the numbers: we require there to be at least one space or semicolon between the number and cd.

   3. At least one space after cd.

   % history | grep '^[ ]*[0-9]*[ ][ ]*cd[ ;]'
   
                       .
                       .
                       .
    1076  cd
    1079  cd; mkdir redirect; cd redirect; pwd
    1092  cd ../grep
    1129  cd ..
    1145  cd
    1185  cd ~phys210/words
    1197  cd; mkdir redirect; cd redirect; pwd
    1213  cd; mkdir numbers; cd numbers; pwd
    1227  cd; mkdir grep; cd grep; cp ~phys210/grep/poems .
   

   Question

   Why won't ...

   % history | grep '[ ][ ]*cd[ ][ ]*'
   

   ... extract the cd commands and only the cd commands?

   End question

3. Look for environment vars whose names begin with 'LC'

   % env | grep '^LC'
   
   LC_PAPER=en_CA.UTF-8
   LC_ADDRESS=en_CA.UTF-8
   LC_MONETARY=en_CA.UTF-8
   LC_SOURCED=1
   LC_NUMERIC=en_CA.UTF-8
   LC_ALL=POSIX
   LC_TELEPHONE=en_CA.UTF-8
   LC_MESSAGES=en_CA.UTF-8
   LC_COLLATE=en_CA.UTF-8
   LC_IDENTIFICATION=en_CA.UTF-8
   LC_MEASUREMENT=en_CA.UTF-8
   LC_CTYPE=en_CA.UTF-8
   LC_TIME=en_CA.UTF-8
   LC_NAME=en_CA.UTF-8
   

4. Look for environment vars or values that contain $LOGNAME. Note the in this example the shell evaluates $LOGNAME to your literal account name before executing the grep command:

   % env | grep $LOGNAME
   
   USER=phys210d
   SCREENDIR=/home/phys210d/tmp
   MAIL=/var/spool/mail/phys210d
   PATH=.:/home/phys210d/bin:/home/phys210/bin:/usr/local/bin:/usr/bin:/usr/games:/usr/lib64/qt4/bin
   PWD=/home/phys210d/grep
   HOME=/home/phys210d
   LOGNAME=phys210d
   OLDPWD=/home/phys210d
   

We won't do any more grep lab exercises since you'll get lots of grep practice completing Homework 1!

sed: a stream editor

Hint: sed may be useful in completing Homework 1

sed is one of the many Unix commands that has a large number of its own subcommands. Here we will illustrate only one of these, namely the substitute command. We will also use sed purely in filter mode, so that it gets its input from standard input, and puts its output to standard output. This is sed's default mode of operation, and, as with all filters, we can use input redirection to make the command get its input from some file, and output redirection to have it put its output in some other file.

Our basic usage of the sed substitution command will be ...

s/regexp/replacement/g

The slashes in the command are delimiters---i.e. they separate the regexp and replacement---and are used by convention. Provided neither regexp nor replacement contains one or more slashes they are a good choice: if one of more slashes do do appear then use some other character which does not, e.g.

s?regexp?replacement?g

The substitute incantation is a subcommand of the command sed. To have sed perform the subcommand we pass it to sed on the command-line as follows:

% sed 's/regexp/replacement/g'

Let's now consider a few examples. In your terminal window change to the redirect subdirectory of your home, and get a listing of its contents:

% cd ~/redirect
% ls
fireice  ls-output  night  poems  pools  road  snowy

% grep 'fire' fireice
Some say the world will end in fire,
I hold with those who favor fire.

% sed 's/fire/an inferno/g' < fireice

Some say the world will end in an inferno,
Some say in ice.
From what I've tasted of desire
I hold with those who favor an inferno.
But if it had to perish twice,
I think I know enough of hate
To know that for destruction ice
Is also great
And would suffice.

% sed 's/fire/an inferno/g' < fireice > inferno-ice

% cat inferno-ice

Some say the world will end in an inferno,
Some say in ice.
From what I've tasted of desire
I hold with those who favor an inferno.
But if it had to perish twice,
I think I know enough of hate
To know that for destruction ice
Is also great
And would suffice.

% sed 's/fire/an inferno/g' < fireice > fireice

-bash: fireice: cannot overwrite existing file

Now let's use sed twice via a pipeline to change all occurrences of fire to an inferno and all instances of ice to frigidity by executing ...

% sed 's/fire/an inferno/g' < fireice | sed 's/ice/frigidity/g'

Some say the world will end in an inferno,
Some say in frigidity.
From what I've tasted of desire
I hold with those who favor an inferno.
But if it had to perish twfrigidity,
I think I know enough of hate
To know that for destruction frigidity
Is also great
And would sufffrigidity.

Here's one last example (potentially useful for the first homework, hint, hint, nudge, nudge, wink, wink, ...) which, in contrast to the two above, uses a non-trivial regular expression as the search pattern. Let's use sed to filter all lines of the file poems so that only the first word of each line remains. An appropriate command is:

sed 's/ .*$//g' < poems

The output is

Some
Some
From
I
But
I
To
Is
And

I
I
I
 .
 .
 .
The
But
And
And

sed 's/ .*$//g' < poems | grep -v '^[ ]*$'

Some
Some
From
I
But
I
To
Is
And
I
I
I
 .
 .
 .
The
But
And
And

sed exercise

End exercise

Optional pipeline exercise

Add stages to the pipeline ...

sed 's/ .*$//g' < poems | grep -v '^[ ]*$'

End exercise

All those quotes

We'll keep this section brief since the various quoting mechanisms available, and the distinctions among them, are most relevant in the context of shell programming, which we will not be studying at this time. Thus, we'll simply work through the examples from the notes.

That said, be sure you understand how the backquotes operate since they may be very useful in completing what some of you may find a particularly challenging problem in Homework 1 (hint, hint!)

Forward quotes

Recall that these quotes inhibit shell evaluation of any and all special characters and/or constructs, as we have already seen in some of the grep examples:

% a=100
% echo $a
100

% b=$a
% echo $b
100

% b='$a'
% echo $b
$a

Note how the forward quotes stop the shell from replacing $a with its value prior to the assignment to b.

Double quotes

Recall that these work like single quotes, except that the shell "looks inside" them and evaluates

1. any references to the values of shell variables
2. anything within backquotes (see below)

% a=100
% echo $a
100

% string="The value of a is $a"
% echo $string
The value of a is 100

You should now understand how/why if you defined the alias cdhw1 using

% alias cdhw1="cd /phys210/$LOGNAME/hw1"

% alias cdhw1
alias cdhw1='cd /phys210/phys210d/hw1'

(If you used your actual account name rather than $LOGNAME in the definition of cdhw1, cdhw2, etc., you might want to try modifying the definitions to use $LOGNAME).

Backward quotes (backquotes, backticks)

Recall that these quotes (backticks) provide the useful and very powerful mechanism that allows you to capture the output from a command (or, in general, a pipeline) as a string, which can then, for example, be assigned to a shell variable, or supplied as an argument to another command:

% date
Thu Aug 29 12:57:18 PDT 2013

% thedate=`date`
% echo $thedate
Thu Aug 29 12:57:29 PDT 2013

% which true
/usr/bin/true

% file `which true`
/usr/bin/true: ELF 64-bit LSB executable, x86-64 ...

% file `which true` `which false`
/usr/bin/true:  ELF 64-bit LSB executable, x86-64 ..
/usr/bin/false: ELF 64-bit LSB executable, x86-64 ...

Backquotes exercise

Part a)

% cd ~phys210t
% your-command-here
% echo $mylist

cmd dir1/ dir2/

Part b) -- trickier

Hint: which ls is the "real" (i.e. non-aliased) ls?

End exercise

A brief introduction to gnuplot

With that harangue out of the way, let me illustrate just a couple of things that will get you going with gnuplot. Note that the application is command-line driven, has many distinct commands of its own, and that many of those commands (including fundamental ones such as plot) have a syntax that can become quite complicated. That said, it also quite straightforward to make simple plots.

First, start the application from the bash command-line:

% gnuplot

gnuplot>

gnuplot>

One of the nice things about gnuplot is that it is easy to plot quantities that are explicit functions of a single variable; i.e. expressions which use the usual arithmetic operations and most standard mathematical functions (sin, cos, tan, log, exp, ...)

Try the following ...

gnuplot> plot sin(x) 

gnuplot has an extensive built-in help facility, type help for an overview:

gnuplot> help

`Gnuplot` is a portable command-line driven graphing utility for Linux, OS/2,
 MS Windows, OSX, VMS, and many other platforms. The source code is copyrighted
 but freely distributed (i.e., you don't have to pay for it). It was originally
 created to allow scientists and students to visualize mathematical functions
 and data interactively, but has grown to support many non-interactive uses
 such as web scripting. It is also used as a plotting engine by third-party
 applications like Octave. Gnuplot has been supported and under active
 development since 1986.
                                    .
                                    .
                                    .
Press return for more:

gnuplot> help plot

 `plot` is the primary command for drawing plots with `gnuplot`.  It creates
 plots of functions and data in many, many ways.  `plot` is used to draw 2D
 functions and data; `splot` draws 2D projections of 3D surfaces and data.
 `plot` and `splot` offer many features in common; see `splot` for differences.
 Note specifically that although the `binary ` variation does
 work for both `plot` and `splot`, there are small differences between them.

 Syntax:
       plot {<ranges>
            {<iteration>
            {<function> | {"<datafile> {datafile-modifiers}}}
            {axes <axes> {<title-spec> {with <style>
            {, {definitions{,}} <function< ...}
                                .
                                .
                                .
Subtopics available for plot:
    acsplines         axes              bezier            binary
    cnormal           csplines          cumulative        data
    datafile          errorbars         errorlines        every
    example           frequency         functions         index
    iteration         kdensity          matrix            parametric
    ranges            sbezier           smooth            special-filenames
    style             thru              title             unique
    using             volatile          with
Subtopic of plot:

You can now type in one of the plot subtopics for help with it, or simply Enter to return to the top level prompt.

It's always good to learn a computational facility through examples, so ask gnuplot for some sample usages of plot

Subtopic of plot: example

 This example plots the data in the file "population.dat" and a theoretical
 curve:

       pop(x) = 103*exp((1965-x)/10)
       set xrange [1960:1990]
       plot 'population.dat', pop(x)
                    .
                    .
                    .

One command in gnuplot that is frequently problematic for newcomers, and that you will need to use in completing the first homework is

set terminal

As you can read in more detail via ...

gnuplot> help set terminal

In the "old days" we tended to use many different physical graphics devices, such as different types of hardcopy plotters and special-purpose graphics displays sold by various and sundry vendors. Click HERE to see an image of a Tektronix 4014 which is a device that I have fond memories of and which at $10,000 or so back in the 70's and 80's---perhaps $30k in today's dollars---was a bargain. Each of these devices effectively spoke its own language, and "high level" plotting programs such as gnuplot had to be able to translate their graphing instructions into those languages. You told a program like gnuplot which specific device you wanted to talk to and it generated the appropriate code to make the device perform the requisite plotting task.

In the current era, although there is still a proliferation of graphics devices (indeed, more so than in the "old days"), we tend to restrict use to the following "terminal types" (think devices, or places that the graphics output is directed):

• The "normal" one associated with plotting in a separate window on our desktop. The default for this is wxt, named after the "widgets library" (software) that is used to create the window. We see reference to this device, just before the prompt appears, when we fire up gnuplot:

  Terminal type set to 'wxt'
  gnuplot>
  

• Those associated with output to a file that is encoded in one of the many popular graphics formats such as JPEG, GIF, PNG, EPS (encapsulated Postscript) etc.

Thus, your solutions to the homework problems involving gnuplot will generally incorporate a sequence like ...

gnuplot> set terminal ...
gnuplot> set output ...

Also note that the for any particular terminal type there are a variety of options that can be set to control features such as line types, fonts used, font sizes, etc. Use, e.g.

gnuplot> help set terminal wxt

We now consider a feature of gnuplot which illustrates a general principle that is applicable to essentially all interactive computer work that involves typing text:

Whenever possible, type commands into a text file and then input them to the command-line driven program using its facility for reading commands from a file (assuming that one exists).

I will reemphasize the essential utility of the principle throughout the course.

I will reemphasize the essential utility of the principle throughout the course.

You get the point.

To see how this works in gnuplot, do the following:

1. Terminate your gnuplot session using the command quit (or exit)
2. Using the bash command-line, create a subdirectory of your home directory named plot, cd to it and restart gnuplot:

   % cd 
   % mkdir plot
   % cd plot
   % gnuplot
   

3. Open a new terminal window: Ctrl-Shift-N within the existing one will create a new one, or you can use a menu selection: File -> New Window
4. In that window enter the following commands

   % cd
   % cd plot
   % kate gnuplot-input
   

5. In the kate window, not at the gnuplot prompt, enter the following text

   plot cos(x)**2
   

   (** is the exponentiation operator in gnuplot) and save the file. Remember that since we started kate from the command-line with the name of a file (a new one in this case), we don't have to tell the editor what to call the file when it is saved. Also, since kate was started in the directory ~/plot, the file will be saved there as well.

   Important terminology:

   In computer science, a file that contains a list of commands for some normally interactively command-line-driven program is generically called a script.

   End terminology

   We will use the term script throughout the course, including in the contexts of Maple and MATLAB programming.

6. Return to the window running gnuplot and enter the following:

   load 'gnuplot-input'
   

   Ensure that you enclose the name of the file in quotes or gnuplot will emit an error message. Gnuplot doesn't care whether you use single or double quotes, so

   load "gnuplot-input"
   

   is equivalent.

Once you enter the load command, the gnuplot window should pop up with a plot of \(\cos(x^2)\).

After the commands in the script file gnuplot-input have been "loaded" (and executed), control of gnuplot returns to the command-line and you can continue to issue directives interactively. Better still, if you take the script-based approach to heart (and I recommend that you do!) you can:

1. Modify the script.
2. Save it.
3. Reload it into gnuplot (note that you can recall the previous load command using the up-arrow key).
4. Repeat as necessary until you're satisfied with your plot.

1. It is easier to edit gnuplot "code" using kate than via the command line.
2. You automatically maintain a record of what plotting commands you used to perform a particular graphing task, including all of the syntactic idiosyncracies that you had to deal with.
3. Crucially, the files of commands that you create can serve as templates for subsequent plotting chores. When doing programming of any sort it is easier to start from something that works and then modify than to start from scratch.

I reiterate that these benefits generally apply whenever one is working with a command-line oriented system, including the shell, or, as we shall see later in the course, maple and MATLAB.

I'll conclude this whirlwind introduction by pointing out that one very useful utility command is test

gnuplot> test

Note that when you set non-default terminal options via

gnuplot> set terminal which-terminal option1 option2 ...

gnuplot exercise

Modify the script gnuplot-input so that it plots \(\cos(x^2)\) using a green dashed line, and gives the plot a title

My plot of cos(x**2)

I suggest that you figure out how to plot the title first.

You may find

gnuplot> help set terminal wxt
gnuplot> help plot
gnuplot> help plot style
gnuplot> help title
gnuplot> help colors

Hint

You will need to use

gnuplot> set terminal wxt some-option

gnuplot> test

Note

As much as possible, please try to do this exercise by yourself and try to resist telling your classmates how you did it until they have done it as well.

Remember to always save the script file when you have changed it, before you reload it into gnuplot.

Also note that, as is the case when working with all programmable environments, there are many ways to complete the exercise, so don't be surprised if your solution is different from your classmates'.

End exercise

End of Unix Lab 4: Work on HW 1 or head out as you wish ...