Before proceeding to specific discussions of the Unix commands we will use for program development, let us consider the basic job of a compiler and work through some simple examples. A compiler translates (or compiles) "high-level" code (such as C or Fortran) into a form which the hardware can more or less run directly. In brief then, a compiler's job is to convert source code into executables. In the simplest case, the source code will reside in a single source file: on Unix systems, and by convention, Fortran source code is prepared in files whose names have a .f extension. Here is a simple example:
% ls *.f
first.f
% cat first.f
write(*,*) 'Hello World!'
stop
end
Here the Fortran source file first.f contains a complete
Fortran program. We can produce an executable using the pgf77
command as follows:
% pgf77 -g first.f -o firstThe Fortran compiler quietly does its work, producing an executable called first
% ls -l first -rwxr-xr-x 1 phys410 phys410 73610 Sep 24 16:16 firstwhich, provided that
. is in our current path, we can execute simply by typing
its name:
% first Hello World! FORTRAN STOPNote that the diagnostic message
FORTRAN STOPwill appear after the execution of any executable produced by pgf77, and is simply a convention which the vendor has chosen to implement. Also note that there is a leading space (blank) before the Hello World! text; this is also typical of the output from Fortran programs and dates back to the dark ages when the standard output device was a line printer, and the first character in any output line was reserved for "carriage control".
The pgf77 command issued above requires a little further explanation. In addition to the source file first.f, we supply as arguments the option -g (debug option) which tells the compiler to include information in the executable to facilitate debugging, and the option -o first which tells pgf77 to name the executable first. Thus, there are effectively 3 arguments to pgf77 in the above example: (1) -g, (2) first.f, and (3) -o first. These can appear in any order on the command line, so the following invocations (among others), are equivalent to our original form:
% pgf77 -o first -g first.f % pgf77 first.f -o first -gIf you don't specify a name for the executable using the -o option, pgf77 will call your executable a.out:
% pgf77 -g first.f Linking: % a.out Hello World! FORTRAN STOPHowever, I strongly recommend that you avoid using this default behaviour.
Let us consider a slightly more complex example, in which we introduce the concept of an "intermediate" level of code---known as object code---which is also compiled from source code, but which is not directly executable. In this example we have two Fortran source files, greeting.f which contains a Fortran main program and sayhello.f which contains a Fortran subroutine (or procedure) that the main program calls:
% cd ~phys410/f77/ex1
% ls *.f
greeting.f sayhello.f
% cat greeting.f
c This is the main program
program greeting
call sayhello()
stop
end
% cat sayhello.f
c This is the subroutine
subroutine sayhello()
write(*,*) 'Hello World!'
return
end
As in our previous example, we can generate an executable directly
using the pgf77 command; we simply pass both source files
as arguments:
% pgf77 -g greeting.f sayhello.f -o greeting greeting.f: sayhello.f: Linking:Note that this time, the pgf77 command echoes the name of each source file (followed by a colon) as it is processed. Also note that the name of the executable produced is greeting:
% greeting Hello World! FORTRAN STOPIf we check the contents of the directory:
% ls Makefile greeting* greeting.f greeting.o sayhello.f sayhello.owe notice that in addition to the executable, greeting, the pgf77 command created two files, greeting.o and sayhello.o, both having .o extensions. These are object files which, as mentioned above, we can view as an "intermediate" level between source code and executable code. Loosely speaking then, the process of translating source code into executable code in Unix can be separated into two phases:
Although there is a separate linking command in Unix (usually called ld), you don't have to invoke it directly---the pgf77 (or pgcc) command will do it for you, provided you have issued a command which calls for the creation of an executable. Such is the case in the two examples above where we generally used the -o option. However, we can also use pgf77 to create an executable in two phases. First, by supplying the -c option to the compiler we request that .f files only be compiled into .o files:
% RM greeting *.o % ls Makefile greeting.f sayhello.f % pgf77 -g -c greeting.f sayhello.f greeting.f: sayhello.f: % ls Makefile greeting.f greeting.o sayhello.f sayhello.oNote that this last pgf77 command did not create an executable. To make the executable, we supply pgf77 with the names of the .o files which contain the object code which we wish to be "linked" to create an executable, and, as in our early examples, we use the -o option to give the executable a specific name:
% pgf77 -g greeting.o sayhello.o -o greeting Linking: % ls Makefile greeting* greeting.f greeting.o sayhello.f sayhello.o % greeting Hello World! FORTRAN STOPHere, pgf77 basically passes all of its arguments to the actual loader command, ld, along with additional information for ld which is common to all Fortran programs. Just before it hands off control to ld, it prints the diagnostic message
Linking:to inform us that the loader is being invoked; the loader then creates the executable.
Although the two phase process of first creating object files using the -c option, and then linking them together to create an executable file may seem awkward, there are advantages to this technique. For example, if we are working with a program consisting of many thousands of lines of source code contained in many distinct source files, and make a change to one of the source files, then by using separate compilation and link phases, we need only recompile (using -c) the single source file which was modified, then relink all of the object files to produce a new executable. For large programs this can significantly decrease the development cycle-time and hence is recommended practice. However, for short programs, and in particular for programs which are entirely contained in a single source file, the first approach used above will suffice.
We end this survey with a brief discussion of libraries (or library archives) in Unix. Libraries are closely related to object code, and you can think of them as collections of routines (procedures, functions) which have been converted into object code and which are ready to be included (linked to) by any program which wants to call them. A simple example will illustrate the idea:
% cd ~phys410/f77/ex0
% cat tdmach.f
c-------------------------------------------------
c Simple illustration of use of libraries
c
c dmach is a function defined in the LINPACK
c library
c
c /usr/local/PGI/lib/liblinpack.a
c
c It computes and returns information
c concerning the invoking machine's floating
c point model.
c-------------------------------------------------
program tdmach
implicit none
real*8 dmach
write(0,*) dmach(1)
stop
end
This simple program invokes the function dmach to compute
machine-epsilon (a concept we will discuss at a later date),
and then outputs the value to standard error.
However, dmach is not a basic part of the f77 language.
Thus, if we naively try to build an
executable, we get an "undefined reference"" error message:
% pgf77 -g tdmach.f -o tdmach Linking: tdmach.o: In function `tdmach': tdmach.f:19: undefined reference to `dmach_'which tells us that the loader, ld, (again, automatically invoked here by the pgf77 command) was unable to locate object code for any routine named dmach_.
The fact that the loader tells us that it can't find a definition for dmach_ rather than dmach requires a little digression. One can write programs in a variety of "high-level" languages: C, C++, Fortran 77, Ada ..., and it is often convenient to be able to combine pieces of code written in different languages into a single executable. One problem which we wish to avoid in such "mixed mode" programming is the "collision" of the names of routines which are written using different languages, then subsequently linked together by the loader. For example, it should be possible for a programmer to write a subroutine named foo in Fortran , and a function also named foo in C, and have the loader be able to distinguish between them should they both be included in the same executable.
To make a longish story relatively short, the way most Unix compiler suites, including the PG compilers, deal with this naming conundrum is to append an underscore to the name of any Fortran symbol which is to be communicated to the "external world"; i.e. to the loader. C external names, on the other hand, retain their source-code-defined identity, which is only reasonable since the vast bulk of any self-respecting Unix implementation is coded in C.
Unfortunately, the folk who implemented the g77 compiler, which is part of the standard Linux distribution, introduced the following annoying little twist: if the Fortran name itself contains one or more underscores, then an ADDITIONAL underscore (for a total of two) is appended for communication with the outside world. Indeed, in order to allow us to mix in code which has been compiled with g77, pgf77 accepts a special option
-Msecond_underscorewhich makes pgf77 act in the same bizarre fashion as g77. In fact, in short order we will start to use this option as a matter of standard practice when compiling and linking programs using the PG compilers.
We now return to the failed pgf77 invocation. To fix the problem, we need to add some additional options and arguments which are used in the linking phase of executable generation:
% pgf77 -g -L/usr/local/PGI/lib tdmach.f -llinpack -o tdmach Linking:This time the link phase succeeds, so we can now execute tdmach:
% tdmach 1.0842021724855044E-019 FORTRAN STOPWe will discuss libraries in a little more detail below. Here I will only point out that the rather cryptic options -L/usr/local/PGI/lib and -llinpack, combined with the fact that a file with the name
liblinpack.aexists in the directory
/usr/local/PGI/libis sufficient to make things work. More specifically, -llinpack tells the loader that it should search for a file named
liblinpack.a or liblinpack.soin one of the "standard" directories where libraries are maintained on the system, while -L/usr/local/PGI/lib tells the loader to append the directory /usr/local/PGI/lib to the list of "standard" directories. Armed with this information, the loader finds the library archive /usr/local/PGI/lib/linpack.a, inspects it, detects that it contains object code for a routine named dmach_, and links that object code into the executable. The search that the loader performs for a specific library archive is very much analogous to the resolution of names of commands using the path.
USAGE EXAMPLES
The following command compiles and loads pgm.f with the debug option, creating the executable pgm.
% pgf77 -g pgm.f -o pgmThe first of the following commands compiles main.f and subs.f producing object files main.o and subs.o. The second command loads both object files creating the executable main.
% pgf77 -g -c main.f subs.f % pgf77 -g main.o subs.o -o mainThe following example is the same as the previous one, except that we now link to the library /usr/local/PGI/lib/libp410f.a using the -L and -l options
% pgf77 -g -c main.f subs.f % pgf77 -g -L/usr/local/PGI/lib main.o subs.o -lp410f -o mainNote that -L/usr/local/PGI/lib adds the directory /usr/local/PGI/lib to the default search path the loader uses when searching for library archives. The option -lp410f tells the loader which specific archive it is seeking. Observe that the strings lib and .a are automatically pre- and post-pended, respectively, to create the actual filename of the archive (libp410f.a in this case).
USEFUL pgf77 OPTIONS
See man pgf77 for additional information. Note that compiler options tend to be system-specific. Options similar to those described here should be available on most Unix Fortran implementations.
USAGE EXAMPLES
The following command compiles and loads pgm.c with the debug option, creating the executable pgm.
% pgcc -g pgm.c -o pgmThe first of the following commands compiles main.c and routines.c, producing object files main.o and routines.o. The second command loads both object files creating the executable main.
% pgcc -g -c main.c routines.c % pgcc -g main.o routines.o -o mainThe following example is the same as the previous one, except that we now link to the libraries /usr/local/PGI/lib/libp410f.a and /usr/lib/libmath.a using the -L and -l options
% pgcc -g -c main.c routines.c % pgcc -g -L/usr/local/PGI/lib routines.o subs.o -lp410f -lm -o mainNote that when multiple libraries are specified in the load phase, as they are above, the loader searches each library exactly once for unresolved symbols and searches in the order specified on the command line. Thus, if the program above calls a routine in the p410f library, and that routine calls a routine in the m library (standard C math support) then
% pgcc -g -L/usr/local/lib routines.o subs.o -lm -lp410f -o mainwill result in a load error.
#include'ing FILES FROM NON-STANDARD LOCATIONS
Statements of the form
#include "mytypes.h" #include <stdio.h>are C pre-processor directives which effectively include the contents of a file in-place in the C source. In the first form, where the filename is enclosed in double quotes ("), the specified file must reside in the working directory. In the second case, where the filename is enclosed in angle-brackets (<>), the preprocessor searches for the file in the ``standard include directory'', /usr/include. Additional directories which are to be searched for #include files may be specified with the -I option. Thus, assuming that mytypes.h lives in /home/matt/include and that the source code for myinclude.c contains the statement
#include <mytypes.h>then the pgcc command
% pgcc -I/home/matt/include myinclude.c -o myincludewill ensure that the file is properly included.
USEFUL pgcc OPTIONS
See man pgcc for additional information. Note that compiler options tend to be system-specific. Options similar to those described here should be available on most Unix C implementations.
% cd /usr/lib
% ls lib*.a
libAVLTree.a libglib.a libp2c.a
libBLT.a@ libgmodule.a libpanel.a
libBLT24.a libgmp.a libpanel_applet.a
libBrokenLocale.a libgncengine.a libpanel_g.a
.
.
.
Link to libraries located in standard locations (notably /lib and /usr/lib) using the -l option to either pgf77 or pgcc:
% pgcc -cpgm.o -lm -lX11 -o cpgm % pgf77 f77pgm.o -lblas -o f77pgm
Use the -L option to prepend a directory to the default search path for library archives. Thus, assuming that I have a library named /home/matt/lib/libvutil.a, the following pgf77 command will link (if necessary) to that archive:
% pgf77 -L/home/matt/lib pgm.o -lvutil -o pgm
CREATING LIBRARIES
Create and maintain library archives using the ar (archive) command.
Typically one creates an archive file from one or more object files. Thus assuming that the following object files reside in the working directory:
% ls *.o procs1.o procs2.o procsio.oThen the following ar command will create or overwrite a library archive file libmylib.a containing all routines defined by the 3 object files and will ensure that the archive has a ``table-of-contents'' as required by the loader:
% ar r libmylib.a procs1.o procs2.o procsio.oNote that the r immediately following ar in the above is an option (replace) to the ar command: i.e. ar options do not have to begin with a minus sign. Also note that on some systems, ar will not automatically add a table of contents. In such cases there is usually a command ranlib which will do the job.
See man ar for more information.
% pgdbg pgm
pgdbg 3.1-2, Copyright 1989-1999, The Portland Group, Incorporated.
.
.
.
Symbol table built: ./pgm
Loaded: ./pgm
pgdbg> help
pgdbg> help list
pgdbg> help run
pgdbg> help print
pgdbg> help stop
to get going. Use
pgdbg> quitto exit pgdbg.
Remember that if you want to use the debugger, you must compile and link with the -g option.
The basic idea behind make is to view executables (compiled from Fortran 77 source, e.g.) as targets which, in general, have one or more dependencies. Dependencies are typically files themselves, and may themselves have (further) dependencies. This view captures the notion, for example, that an executable foo is constructed from (depends on) the object file foo.o, which in turn is constructed from (depends on) the source file foo.f. A makefile, then, generically consists of definitions of one or more targets; equivalently, things to build, or things to 'make'. Each target definition usually consists of
foo: foo.f pgf77 -g foo.f -o foo ^ |-------- TAB charactermake presumes that once something is made, it doesn't have to be re-made until one of the dependencies changes: whether or not a particular dependency (file) has or has not been changed since a previous make can be deduced via the time of last modification of the file, a statistic which Unix maintains for all files.
(Observe that the idea here is to provide a mechanism to ensure that executables and the like are up to date, so that we are less likely to change some small piece of the code and forget to rebuild the application. We are less concerned about "wasted" work which results from re-making something which really didn't have to be re-made, which might happen, for example, if we edited a file, made a change, saved the file, then undid the change, and re-saved the file. In such an instance, the time of last modification of the file is the time of the last save, even though one might be inclined to view the file as not having been modified.)
Before proceeding to an examination of a Makefile typical of the sort which will be used in the course (and with which, therefore, you should become familiar!), we need to discuss three important preliminary points:
Important point 1: I re-emphasize that all command-lines which define how a particular target is made MUST begin with a TAB character: be especially careful if you "cut and paste" parts of makefiles (such as those reproduced below) into you own---the cut will generally "convert" the TAB to spaces, and you will NOT end up with a valid makefile. Whenever you see a message such as
Makefile:12: *** missing separator. Stop.it's a safe bet that at or near line 12 (in this example), there's an action (command) line which begins with something other than a TAB!
Important point 2: In order to keep our makefiles as portable as possible (so that, as much as possible, we can use the same makefile on different machines running different versions of Unix), we will use certain specific environment variables to communicate with make. One of the nice features of make is that it automatically inherits all environment variables defined at the instant the make command is issued as make "macros", which, for the purposes of this course we can view as make "local variables". We simply have to keep in mind the slightly peculiar syntax of evaluation of variables (macros) in make. So for example, to access the value of the HOME environment variable in a makefile, we use the construct
$(HOME)
rather than the form
$HOME
which we use in the shell.
To set, or define, a makefile macro, we use the following syntax
<macroname> = <macro definition>where <macroname> should adhere to the "usual" rules for an identifier (variable) name, and where <macro definition> is an arbitrary string (do not enclose the definition string in quotes, even if it contains white space.) Here is a simple example which defines a macro MYLIB in terms of the pre-defined (inherited from the environment) macro, HOME:
MYLIB = $(HOME)/libAssuming, for concreteness, that $(HOME) evaluates to /home/matt, $(MYLIB) will then evaluate to /home/matt/lib.
The specific environment variables which we will use (as pre-defined macros for make), along with their recommended settings on the lnx machines are as follows:
# 'F77' defines the name of the Fortran 77 compiler. setenv F77 'pgf77' # 'F77FLAGS' defines options which are used in BOTH the compilation and load phases setenv F77FLAGS '-g -Msecond_underscore' # 'F77CFLAGS' defines flags which are used ONLY in the compilation phase setenv F77CFLAGS '-c' # 'F77LFLAGS' defines flags which are used ONLY in the load phase setenv F77LFLAGS '-L/usr/local/PGI/lib' # 'LIBBLAS' defines the name of the BLAS (Basic Linear Algebra Software) library setenv LIBBLAS '-lblas'
Recommended settings of the above variables on a variety of machines used in the course can be found HERE.
Note that these variables should generally be set in your .cshrc (or equivalent) so that
On the lnx machines, I have modified the system-wide tcsh start-up file, /etc/csh.cshrc so that the variables are set as above; i.e. you do not have to add anything to your .cshrc.
Important point 3: If you want to "break" a long line in a makefile over two or more lines, you can use the "usual" shell "continuation construct" ("usual" since it also works in the csh, tcsh, bash etc.), which is simply a backslash (\) immediately followed by a new-line. Thus, in a makefile, the macro assignments
FILELIST = a.f b.f c.fand
FILELIST = a.f \
b.f \
c.f
are identical.
Note again that there can be NO white space following the \ character, or make will become confused.
Given the above preamble, we can now turn to an example Makefile which can be found in the directory ~phys410/f77/ex3 on the lnx machines. (Observe, however, that the on-line version has been stripped of the comments included in the version below).
############################################################
# Lines beginning with a '#' are comments
############################################################
############################################################
# The .IGNORE: directive tells 'make' to keep going if
# one or more commands executed as a result of the 'make'
# do not complete successfully. The default is to bail
# out.
############################################################
.IGNORE:
############################################################
# Note that this 'Makefile' assumes that the following
# environment variables are set:
#
# F77 -> name of f77 compiler
# F77FLAGS -> generic f77 flags
# F77CFLAGS -> f77 flags for compilation phase
# F77LFLAGS -> f77 flags for load phase
############################################################
# Define some macros for
#
# F77_COMPILE: The command which will be used to compile
# Fortran source
# F77_LOAD: The command which will be used to load
# Fortran object files, link to libraries
# and create executables.
#
# In this Makefile, macros are used almost precisely like shell
# or environment variables. Note, however, that macros are
# evaluated with the $(MACRONAME) construct: the () are CRUCIAL.
# Also note that ALL environment variables (HOME, DISPLAY,
# etc.) are automatically available as macros with the
# same name. Thus, for example, $(HOME) will evaluate to
# your home directory. We will use this feature to create
# Makefiles which are portable across systems provided that
# the appropriate environment variables are set properly
# (typically in your '~/.cshrc') on the various systems.
############################################################
F77_COMPILE = $(F77) $(F77FLAGS) $(F77CFLAGS)
F77_LOAD = $(F77) $(F77FLAGS) $(F77LFLAGS)
############################################################
# The following defines a GENERIC target (rule)
# which tells 'make' how to produce a '.o' file from
# a '.f' file. 'Make' will automatically use such a rule
# unless a specific target overrides it.
############################################################
.f.o:
$(F77_COMPILE) $*.f
############################################################
# Define a macro for all the executables in the directory
############################################################
EXECUTABLES = fdemo2 mysum tdvfrom tdvto
############################################################
# Since this is the first SPECIFIC target in the makefile,
# if 'make' is invoked with no arguments, this is the target
# which will be made. Since $(EXECUTABLES) evaluates to
# 'fdemo2 mysum tdvfrom tdvto', 'make' will make each of
# 'fdemo2', 'mysum', 'tdvfrom' and 'tdvto' in turn
############################################################
all: $(EXECUTABLES)
############################################################
# The target 'fdemo2' depends on the object file 'fdemo2.o'.
# When 'fdemo2' is being made, 'make' figures out that it
# first needs to make 'fdemo2.o' from 'fdemo2.f' using
# the generic rule above. Once the dependencies of any
# given target have been updated, the commands which
# follow the target are executed in turn. In this case,
# the object file is simply loaded and the executable
# 'fdemo2' is created. Again note that each command line
# MUST BEGIN WITH A TAB. Continue long lines with \
# (backslash, followed by carriage return, with no
# intervening spaces).
############################################################
fdemo2: fdemo2.o
$(F77_LOAD) fdemo2.o -o fdemo2
mysum: mysum.o
$(F77_LOAD) mysum.o -o mysum
############################################################
# A little more complicated example since there are 2
# dependencies ('tdvfrom.o' and 'dvfrom.o') and we
# link to the 'p410f' library
############################################################
tdvfrom: tdvfrom.o dvfrom.o
$(F77_LOAD) tdvfrom.o dvfrom.o -lp410f -o tdvfrom
tdvto: tdvto.o dvto.o
$(F77_LOAD) tdvto.o dvto.o -lp410f -o tdvto
############################################################
# Makefiles often have a 'clean' target which cleans
# up object files, executables and other files which
# tend to consume precious disk space, and which can
# always be reconstructed (via 'make' of course!)
############################################################
clean:
rm *.o
rm $(EXECUTABLES)
Here's the same example with the comments removed:
.IGNORE:
F77_COMPILE = $(F77) $(F77FLAGS) $(F77CFLAGS)
F77_LOAD = $(F77) $(F77FLAGS) $(F77LFLAGS)
.f.o:
$(F77_COMPILE) $*.f
EXECUTABLES = fdemo2 mysum tdvfrom tdvto
all: $(EXECUTABLES)
fdemo2: fdemo2.o
$(F77_LOAD) fdemo2.o -o fdemo2
mysum: mysum.o
$(F77_LOAD) mysum.o -o mysum
tdvfrom: tdvfrom.o dvfrom.o
$(F77_LOAD) tdvfrom.o dvfrom.o -lp410f -o tdvfrom
tdvto: tdvto.o dvto.o
$(F77_LOAD) tdvto.o dvto.o -lp410f -o tdvto
clean:
rm *.o
rm $(EXECUTABLES)
and here's some output from make generated using this makefile:
% make pgf77 -g -Msecond_underscore -c fdemo2.f pgf77 -g -Msecond_underscore -L/usr/local/PGI/lib fdemo2.o -o fdemo2 Linking: pgf77 -g -Msecond_underscore -c mysum.f pgf77 -g -Msecond_underscore -L/usr/local/PGI/lib mysum.o -o mysum Linking: pgf77 -g -Msecond_underscore -c tdvfrom.f pgf77 -g -Msecond_underscore -c dvfrom.f pgf77 -g -Msecond_underscore -L/usr/local/PGI/lib tdvfrom.o dvfrom.o -lp410f -o tdvfrom Linking: pgf77 -g -Msecond_underscore -c tdvto.f pgf77 -g -Msecond_underscore -c dvto.f pgf77 -g -Msecond_underscore -L/usr/local/PGI/lib tdvto.o dvto.o -lp410f -o tdvto Linking:Notice how make "echoes" each command (action) to standard output as it is executed.
Since the first specific target in the makefile is all, the above invocation is equivalent to
% make allThe Unix touch command simulates modification of its file arguments by setting the last-modified time of its arguments to the current time. Thus having previously made everything,
% touch dvto.f % make pgf77 -g -Msecond_underscore -c dvto.f pgf77 -g -Msecond_underscore -L/usr/local/PGI/lib tdvto.o dvto.o -lp410f -o tdvto Linking:we see that make re-makes only those targets which depend on the single modified file. Note that we can easily make a single target by supplying the target as the sole argument to make:
% make clean rm *.o rm fdemo2 mysum tdvfrom tdvto % make fdemo2 pgf77 -g -Msecond_underscore -c fdemo2.f pgf77 -g -Msecond_underscore -L/usr/local/PGI/lib fdemo2.o -o fdemo2 Linking:Finally if make deduces that a target is up to date, it will generally tell you so:
% make fdemo2 make: `fdemo2' is up to date.
See man make or the suggested Unix references for more information.