A Cluster is a collection of compatible computers connected by a network. The PGI CDK Cluster Development Kit supports parallel computation on clusters of Intel Pentium 2/3/4 or AMD Athlon/AthlonXP/AthlonMP compatible Linux workstations or servers interconnected by a TCP/IP-based network such as ethernet or fast ethernet. Linux networking must be properly installed and configured on the cluster prior to installation of the PGI CDK. Prior releases of the PGI CDK required that rsh work between all nodes in the cluster. With the 4.1 release, users can utilize ssh or rsh for MPI-CH, PBS, and pgdbg implementation.
You can view the online HTML interface to the PGI CDK using any web browser. Enter the following URL in a web browser:
file:/mnt/cdrom/index.htm
running on a Linux system with the PGI CDK CD-ROM inserted and mounted on the CD-ROM drive. If you do not know how to insert and mount a CD-ROM on your Linux system, see Step 3 below in section 1.2 or contact your system administrator.
Release 4.1 of the PGI CDK consists of the following PGI compilers and tools:
the following open source clustering utilities:
and the following documentation and tutorial materials:
Generally, clusters are configured with a "master" node from which jobs are launched and "slave" nodes that are used only for computation. Typically, the master node is accessible from the general-purpose or "public" network and shares a file system with the other computers on your network using NFS. The master node and all of the slave nodes are interconnected using a second "private" network that is only accessible from computers that are part of the cluster. There are two common cluster configurations:
1) The master node is used only for compilation and job submission, and only the slave nodes are used for computation.
2) All nodes are used for computation, including the master node
To use MPI-CH in the first configuration, PBS should be installed. Otherwise, the master node will be used as one of the computation nodes by the mpirun command by default; it is possible to exclude the master node in the second configuration if mpirun is invoked with the -nolocal option (see the man page for mpirun). If you are using the first configuration, it is possible to install MPI-CH and run parallel MPI or HPF jobs without installing any of the other components. However, if you will have multiple users running jobs on your cluster simultaneously, you will likely want to use PBS to ensure your cluster nodes are allocated and used efficiently.
Typically, a master node has two network cards to allow communication to the outside network as well as to the cluster nodes themselves, which may be on their own subnet. If this is the case on your cluster, then when the installation script prompts you for the name of the master node, you should use the name associated with the network card connected to the cluster nodes.
Also, it is important to note that in order for MPI-CH and PBS to run correctly, access from each node to every other node must be available via the `rsh' or `ssh' command. For example, if a 3-node cluster consists of a master and 2 slaves named master, node1, node2, then as a user you should be able to issue the commands:
% rsh master date
% rsh node2 date or % ssh master date
% ssh node2 date
From node1, and similarly from node2 and master.
By default, all of the PGI compilers and tools will be installed on your system. You will select which of the open source components to install. At this point, you should have determined:
* Which PGI CDK open source components you will install (see above for a list of these)
* The hostnames of all the nodes that will be included in your cluster - you will need a list of these during the installation
* Whether the master node will be strictly a front-end for compilation, job launching, etc or whether it will participate as a compute node
* If you are installing PBS, which users at your site will have PBS queue manager permissions (you will need their usernames)
* Whether the compute nodes can share files with the master node (this is strongly recommended)
Section 1.2 below describes how to install the PGI Fortran, C and C++ compilers and tools on Linux using the installcdk script from the PGI CDK CD-ROM. NOTE: you must have root permissions to successfully execute the installcdk script. You must install the software as specified in section 1.2 and then follow the instructions in section 1.3 for configuring and starting the FLEXlm license daemon.
The FLEXlm license daemon enables use of the PGI compilers and tools by any user on any system networked to the system on which the PGI software is installed. For example, users can compile, debug, and profile using the PGI CDK compilers and tools on any system on your general-purpose network, subject to the constraints on concurrent usage for the product you have purchased.
Sections 2.1 - 2.5 describe basic usage of the Open Source components of the PGI CDK, including MPI-CH, the PBS batch queuing system, ScaLAPACK libraries, and the example benchmark programs and tutorials.
For the first 60 days after your purchase, you may send technical questions about the PGI CDK compilers and tools to the e-mail address trs@pgroup.com. If you have purchased PGI's Subscription Service, you will have access to e-mail service for an additional 12 months and will be notified by e-mail when maintenance releases occur and are available for electronic download and installation. Contact PGI at sales@pgroup.com for information about the PGI Subscription Service for the products you have purchased.
MPI-CH, PBS, and ScaLAPACK, are all open source software packages that are not formally supported by PGI. All source code for these components is included on the PGI CDK CD-ROM in the cdk subdirectory. Each of these components has end-user and implementer documentation, generally in the form of printable postscript, along with the source code. Support for these products is generally provided by their respective user communities, which you can learn more about at the following URLs:
* MPI-CH - http://www-unix.mcs.anl.gov/mpi/mpich contains a wealth of information, including online documentation, tutorials, FAQ files, patch distributions, and information on how to submit bug reports to the MPI-CH developers.
* PBS - http://www.openpbs.org is the main site for the PBS product we integrate with the CDK.
* ScaLAPACK- http://www.netlib.org/scalapack contains FAQ files and current distributions of ScaLAPACK.
Once you've completed installation of the PGI CDK, the MAILUSERS file on the PGI CDK CD-ROM includes a simple text template for an introductory e-mail to end-users of your cluster system. It explains how to initialize one's environment, access the man pages and documentation, and execute a simple program on the cluster.
Step 1 - Create the directory in which you wish to install the PGI CDK Compilers and tools. NOTE: The installation directory you choose must be accessible from all nodes in the cluster using a uniform pathname. In the example below, we assume /usr/pgi which is the default installation directory. However, installation can occur in any directory where you have appropriate permissions. Please make sure that the installation directory has the necessary ownership and permissions appropriate for your site by using the chown and chmod commands.
Set the environment variable PGI to the name of the installation directory. Assuming csh:
% setenv PGI /usr/pgi
Or, assuming sh, ksh, or bash:
% export PGI=/usr/pgi
Step 2 - All software should fit into less than 250 MB of disk space. If you wish to install all of the source code for the open source components of the PGI CDK, about 350 MB of disk space is required. Verify that you have sufficient space on the disk where your installation directory will be located.
Step 3 - The installcdk script must be run to properly install the software, and you must have root permissions to execute this script. If you do not have root privileges, you will need to get help from your system administrator during the installation process.
If you are installing from a CD-ROM and you're not sure how to access the CD-ROM drive on your system, check with your system administrator. Typically, you must insert the CD-ROM into the CD-ROM drive on the master node and issue the following command:
% mount /mnt/cdrom
while logged in as root to make the data on the CD-ROM accessible. Next, issue the following command from your root window on the master node:
% /mnt/cdrom/installcdk
NOTE: If you have difficulty running this script, especially on a Slackware Linux system, check the permissions on /dev/null. Permissions should be set to crw-rw-rw-. If they are not, reset them using chmod and try running the install script again, or try `mount -o exec /mnt/cdrom'. Otherwise, check and see whether the CDROM was mounted with `noexec' set. If so, unmount it and then mount it with `mount -o exec /mnt/cdrom'
The install script will install all of the binaries for the PGI compilers and tools, MPI-CH, and ScaLAPACK in the $PGI directory tree in the appropriate bin, include, lib, and man subdirectories. It will also install PBS in the /usr/local/pbs and /var/spool/pbs directories, and start the PBS daemons running on the master node and all of the compute nodes in your cluster. You will be prompted for various information about how to configure your cluster as the script executes. Once the installation script has completed, exit the root login.
Step 4 - All of the PGI CDK compilers and development tools are license-managed. The other components of the PGI CDK, including MPI-CH, ScaLAPACK, and the PBS batch scheduler, are open source products that are not license-managed.
If you choose to create a temporary demo license for the PGI compilers and tools, the install script asks for your real name, your username, and your email address. It then creates a fifteen-day license and prints a message like this:
NOTE: your evaluation license will expire in
14 days, 23.6 hours. For a permanent license,
please read the order acknowledgement that you
received. Connect to https://www.pgroup.com/License
with the username and password in the order
acknowledgement.
Name: <your name>
User: <your username>
Email: <your e-mail address>
Hostid: PGI=9BF378E0131FF0C3CD37F6
FLEXlm hostid: 00a024a3dfe7
Hostname: yourhost.yourdomain.com
Installation: /usr/pgi
PGI Release: 4.1-2
The message above is also saved to the file $PGI/license.info for retrieval at a later time.
Once you have obtained your permanent license keys using your personalized account on the PGI web page, place them in the file $PGI/license.dat. Until the permanent license is obtained, the PGI CDK compilers will only be usable under the username specified above during generation of the temporary keys.
Step 5 - You can view the online HTML interface to the PGI CDK documentation and tutorial materials using any web browser. Assuming you use Netscape, issue the following command:
% netscape $PGI/index.htm
You can view the online HTML manuals for the PGI compilers and tools directly by issuing the following command:
% netscape $PGI/doc/index.htm
You may want to place a bookmark on these locations for easy future reference to the online manuals.
Step 6 - Once the temporary or permanent license file is in place, execute the following commands to make the PGI CDK compilers accessible from a normal user shell window.
Assuming csh:
% setenv PGI /usr/pgi
% set path = ( $PGI/linux86/bin /usr/local/pbs/bin \
$path )
% setenv MANPATH "$MANPATH":$PGI/cdk/man:\
/usr/local/pbs/man
Or, assuming sh or ksh:
%export PGI=/usr/pgi
% export
PATH=$PGI/linux86/bin:/usr/local/pbs/bin:$PATH
% export
MANPATH=$MANPATH:$PGI/man:/usr/local/pbs/man
Note: you may see a warning message when you set the MANPATH environment variable if you do not already have MANPATH defined.
Each person who will be using the PGI CDK Fortran, C, and C++ compilers and tools should add the above commands to his or her startup files (e.g. .cshrc) to enable access to the PGI compilers and tools by default upon future logins.
Step 7 - You can verify the release number of the compilers you have installed using the -dryrun -V options on any of the compiler commands. This will also show you the sequence of steps the compiler will use to compile and link programs for execution on your system.
* For Fortran 77, use "pgf77 -V x.f"
* For Fortran 90, use "pgf90 -V x.f"
* For HPF, use "pghpf -V x.f"
* For C++, use "pgCC -V x.c"
* For C, use "pgcc -V x.c"
Note that the files x.f or x.c need not exist in order for you to successfully execute these commands.
The base installation of the PGI CDK is now complete. Follow the directions in section 1.3 to enable floating license capability, and the following sections for basic usage instructions for MPI-CH, PBS, and ScaLAPACK.
The PGI CDK Fortran, C, and C++ compilers and tools are license-managed using the FLEXlm software license management system from Globetrotter Software. The steps below describe how to set up and run license daemons on the master node of your cluster. These daemons allow the PGI CDK compilers to be used on any machine on your public network, subject to the constraint on maximum number of concurrent users as specified in your license. The master node will function as the license "server", distributing seats as requested for use on other computers on your network.
Alternatively, if you already use other software products managed using FLEXlm, you can incorporate the PGI products into the configuration you use for license serving those products.
Step 1 - Install the PGI CDK compilers as described in section 1.2 above.
Step 2 - Once you have obtained permanent license keys (see section 1.2 above for how to obtain these), place them in a file named license.dat in the $PGI directory. For example, the license.dat file should look similar to the following:
SERVER <hostname> <hostid> 7496
DAEMON pgroupd <install_dir>/linux86/bin/pgroupd
FEATURE pghpf-linux86 pgroupd 4.100 31-dec-0 1 \
2B9CF0F163159E4ABE32 VENDOR_STRING=123456:16:cdk \
HOSTID=<hostid> ck=49 . . .
It will include features for each of the PGI CDK compilers and tools. <hostname> and <hostid> should match those you submitted to PGI and <install_dir> must be changed to match the directory in which the compilers are installed. In particular, <install_dir> should match the value of $PGI as defined above.
Step 4 - When the license file is in place, execute the following commands to make the PGI products you have purchased accessible. If you are not using other products managed by FLEXlm, and have not previously set the environment variable LM_LICENSE_FILE, issue the following command to do so (assuming csh):
% setenv LM_LICENSE_FILE $PGI/license.dat
Or, assuming sh, ksh, or bash:
% export LM_LICENSE_FILE=$PGI/license.dat
If you are using other products managed by FLEXlm, and have previously set the environment variable LM_LICENSE_FILE, either incorporate the PGI license keys into your existing license file or issue the following command to append the PGI license file to the definition of LM_LICENSE_FILE (assuming csh):
% setenv LM_LICENSE_FILE \ "$LM_LICENSE_FILE":$PGI/license.dat
Or, assuming sh or ksh or bash:
% export LM_LICENSE_FILE= \ $LM_LICENSE_FILE:$PGI/license.dat
Each person who will be using the PGI CDK compilers should add the above commands to his or her startup files to enable access to the PGI compilers and tools by default upon future logins.
NOTE: If LM_LICENSE_FILE is not set or exported, and the node-locked 15-day temporary license file $PGI/PGIinstall still exists, then $PGI/PGIinstall will be used for resolving compiler licenses.
Step 5 - You must now start the license manager daemon. Edit the shell script template $PGI/linux86/bin/lmgrd.rc. If you have installed the PGI CDK compilers in a directory other than /usr/pgi, substitute the correct installation directory into the definition of the PGI environment variable on line 3 of the script. Now exit the editor and issue the following command to start the license server and PGI license daemon running on your system:
% lmgrd.rc start
If you wish to stop the license server and license daemon at a later time, you can do so with the command:
% lmgrd.rc stop
To make sure that the license server and PGI daemon are started each time your system is booted, log in as root, set the PGI environment variable as above, and then execute the following two commands:
% cp $PGI/linux86/bin/lmgrd.rc /etc/rc.d/init.d/lmgrd
% ln -s /etc/rc.d/init.d/lmgrd /etc/rc.d/rc3.d/S90lmgrd
Note that your system's default runlevel may be something other than `3', and if it is, that number should be used above in setting the correct subdirectory. Run /sbin/runlevel to check the system's runlevel. Note also that if you're using a Linux distribution other than Red Hat, your rc files may be in a directory other than /etc/rc.d. Some Linux distributions, such as Red Hat and Mandrake, include the chkconfig(8) utility that manages the runlevel scripts. If your system has this tool and you wish to use it, then run the following commands:
% cp $PGI/linux86/bin/lmgrd.rc /etc/rc.d/init.d
% chkconfig -- add lmgrd.rc
The appropriate links will be created in the /etc/rc.d directory hierarchy. For more information on chkconfig, please see the manual page.
In addition to the installcdk installation script, there are two additional shell scripts that may be of use to you:
* uninstallcdk - uninstalls selected components of the PGI CDK. This is useful, for example, if you wish to download an updated version of one of the open source components and rebuild and reinstall it on your cluster.
* restartpbs - shuts down and restarts all of the PBS daemons on each node of your cluster. If you find that your default queue is not operating properly for some reason, try restarting PBS using this script to see if it fixes the problem.
As with the installcdk script, you must have root privileges in order to execute these scripts.
Installation of your PGI CDK Fortran, C, and C++ compilers and tools for Linux is now complete. If you have difficulties with the installation, send e-mail to trs@pgroup.com for assistance.
The following sections describe basic usage of the open source PGI CDK clustering utilities.