Phoenix Getting Started Guide

左手的ㄟ右手 2022-05-09 05:46 156阅读 0赞

This guide should help you getting acquainted with the Adelaide high performance computing (HPC) facility Phoenix. The workflow you will find here is slightly different to what you are probably used to on your personal computer. In summary, the workflow can be separated into a couple of steps:

1.  connecting to a login node (sometimes called head node, a computer whose sole purpose is to handle user traffic from and to phoenix);
2.  uploading/editing an input file, preparing a 'compute job';
3.  submitting a job to the job queue
4.  retrieving/downloading the results

## Contents ##

\[[hide][]\]

*  [1Connecting to Phoenix][]
 *  [2Transferring files to Phoenix][]
 *  [3Interacting with the Phoenix System: Mastering Linux Basics][3Interacting with the Phoenix System_ Mastering Linux Basics]
 *  [4Loading software packages][]
    
     *  [4.1For advance users: To install your own packages][4.1For advance users_ To install your own packages]
 *  [5Preparing to submit a job][]
    
     *  [5.1Creating a simple job script][]
     *  [5.2Creating a MPI Job Script][]
     *  [5.3Debiting compute time: Multiple associations][5.3Debiting compute time_ Multiple associations]
 *  [6HPC user guides - Selecting a Job Queue][]
 *  [7Submitting A Job][]
 *  [8Monitoring The Queue][]
 *  [9Canceling a Job][]

## Connecting to Phoenix\[[edit][] | [edit source][]\] ##

Please follow [this guide][] to connect to Phoenix.

After logging in, it is usually advisable to change the default tcsh to bash. To change the shell environment permanently, use the following command to assign /bin/bash to $SHELL.

rcshell -s /bin/bash

## Transferring files to Phoenix\[[edit][edit 1] | [edit source][edit source 1]\] ##

Before you run your application, you will need to upload your data and/or program code from your computer to your [`$FAST`][FAST] directory on the Phoenix system. Fortunately, this step is very easy and there are a number of ways to do this.

Follow our [Data transfer from/to Phoenix guide][Data transfer from_to Phoenix guide] to learn about the necessary steps.

## Interacting with the Phoenix System: Mastering Linux Basics\[[edit][edit 2] | [edit source][edit source 2]\] ##

Like most HPC systems, Phoenix uses Linux as its operating system. Interacting with the system is easier if you are familiar with the basics of the Linux command line interface. As a starting point to learn about Linux basics and discover the most useful commands for using Phoenix, you can refer to our [Linux for HPC][] guide.

## Loading software packages\[[edit][edit 3] | [edit source][edit source 3]\] ##

Software packages on Phoenix are organised in modules, some of the [currently available packages and documentations][] can be found [here][currently available packages and documentations].

In most cases, your required software is not loaded by default on the Phoenix system. After logging in, you will need to load your required software before you can perform any calculations. Phoenix uses the `module` system to manage the software environment. To see a list of the available software, use the `module avail` command as in the example below. If you can not find your required software under this list, there is a good chance we can make it available to you, contact us via [email][] to make a software installation request.

$ module avail
    
    ----------------------------------------------------- /usr/share/Modules/modulefiles ------------------------------------------------------
    dot         module-git  module-info modules     null        use.own
    
    ------------------------------------------------------------ /etc/modulefiles -------------------------------------------------------------
    cuda/6.0              cuda/7.0              gnu-parallel/20150322 intelmpi/5.0.3.048    openmpi/gnu/1.8.4     subread/1.4.6-p2
    cuda/6.5              gcc/4.8.4             intel/13.1.3.174      matlab/2014a          openmpi/intel/1.8.1

To load a software package, use the `module load` command:

$ module load cuda/6.5

To unload a software package, use the `module unload` command:

$ module unload gcc/4.8.4

To swap between software packages, use the `module swap` command:

$ module swap cuda/6.5 cuda/7.0

### For advance users: [To install your own packages][]\[[edit][edit 4] | [edit source][edit source 4]\] ###

## Preparing to submit a job\[[edit][edit 5] | [edit source][edit source 5]\] ##

There are two components required to submit a job for analysis in the Phoenix system.

1.  The software you wish to run (and any associated data files)
2.  A job script that requests system resources

To ensure a fair and optimized system for all Phoenix users, we use a resource management tool, SLURM, for job scheduling. In order to submit a job to Phoenix, you must create a SLURM job script and save this along with the program files, into your directory folder on Phoenix. Below are examples of sample job scripts called <my\_job.sh> for each of the two common job types, namely **simple jobs** and **parallel (MPI) jobs**. For each job type, a downloadable version is provided for you to use. Please configure your job script according to one of following that best suits your requirements.

### Creating a simple job script\[[edit][edit 6] | [edit source][edit source 6]\] ###

A job script is a text file that specifies the resources your code needs to run. The job scheduler then uses these resources to determine when to to run your job. Let's have a look at a simple job script example for some sequential code (that only runs on 1 CPU core):

#!/bin/bash
    #SBATCH -p batch        	                                # partition (this is the queue your job will be added to) 
    #SBATCH -N 1               	                                # number of nodes (use a single node)
    #SBATCH -n 1              	                                # number of cores (sequential job uses 1 core)
    #SBATCH --time=01:00:00    	                                # time allocation, which has the format (D-HH:MM:SS), here set to 1 hour
    #SBATCH --mem=4GB         	                                # specify memory required per node (here set to 4 GB)
    
    # Executing script (Example here is sequential script)
    ./my_sequential_program  	                                # your software with any arguments

We'll begin by explaining the purpose of each line of the script example:

*  The header line `#!/bin/bash` simply tells the scheduler which shell language is going to be used to interpret the script. The default shell on Phoenix is **bash**.
 *  The next set of lines all begin with the prefix `#SBATCH`. This prefix is used to indicate that we are specifying a resource request for the scheduler.
    
     *  The scheduler divides the cluster workload into [partitions (work queues)][partitions _work queues]. Different partitions are used for different types of compute job. Each compute job must select a partition with the **-p** option. To learn more about the different partitions available on Phoenix, see <reference>.
     *  The Phoenix cluster is a collection of compute nodes, where each node has multiple CPU cores. Each job must specify the CPU resources required by using the **-N** option to request nodes and the **-n** to request the number of cores per node required. See <reference>
     *  Each compute job needs to specify an estimate of the amount of time it needs to complete. This is commonly referred to as the walltime, specified with the **--time=HH:MM:SS** option. The estimated walltime needs to be larger than the actual time needed by the job, otherwise the scheduler will terminate the job for exceeding its requested time.
     *  Dedicated memory (RAM) is allocated for each job when it runs, and the amount of memory required *per node* must be specified with the **--mem** option.

A simple job is one in which the computational process is sequential and is carried out by a single node. (Note: If your program file does not use MPI or MPI enabled libraries, your job belongs to this category.) Depending on your computational needs, you may need to use either CPU or GPU-accelerated computing nodes. [This post][] provides some insights about the [differences between CPU and GPU-accelerated computing][This post]. If you need further support with this, please contact the team to discuss.

Below is a sample job script for simple CPU jobs. You will need to create an .sh file in your directory on Phoenix, and can copy and paste the script below into that file. Please remember you must then configure the job script to your needs. The most common fields that need modification are the number of nodes and cores you wish to use, the duration of time for which you wish to run the job, and the email address to which notifications should be sent (i.e. your email address).

#!/bin/bash
    #SBATCH -p batch            	                                # partition (this is the queue your job will be added to) 
    #SBATCH -N 1               	                                # number of nodes (due to the nature of sequential processing, here uses single node)
    #SBATCH -n 2              	                                # number of cores (here uses 2)
    #SBATCH --time=01:00:00    	                                # time allocation, which has the format (D-HH:MM), here set to 1 hour
    #SBATCH --mem=4GB         	                                # specify memory required per node (here set to 4 GB)
    
    # Notification configuration 
    #SBATCH --mail-type=END					    	# Type of email notifications will be sent (here set to END, which means an email will be sent when the job is done)
    #SBATCH --mail-type=FAIL   					# Type of email notifications will be sent (here set to FAIL, which means an email will be sent when the job is fail to complete)
    #SBATCH --mail-user=firstname.lastname@adelaide.edu.au  	# Email to which notification will be sent
    
    # Executing script (Example here is sequential script and you have to select suitable compiler for your case.)
    bash ./my_program.sh 	                                        # bash script used here for demonstration purpose, you should select proper compiler for your needs

For simple GPU jobs, the following example job script can be copied and pasted into a new .sh file in your Phoenix directory:

#!/bin/bash
    
    # Configure the resources required
    #SBATCH -p batch                                                # partition (this is the queue your job will be added to)
    #SBATCH -n 1              	                                # number of tasks (sequential job starts 1 task) (check this if your job unexpectedly uses 2 nodes)
    #SBATCH -c 8              	                                # number of cores (sequential job calls a multi-thread program that uses 8 cores)
    #SBATCH --time=01:00:00                                         # time allocation, which has the format (D-HH:MM), here set to 1 hour
    #SBATCH --gres=gpu:4                                            # generic resource required (here requires 4 GPUs)
    #SBATCH --mem=16GB                                              # specify memory required per node (here set to 16 GB)
    
    # Configure notifications 
    #SBATCH --mail-type=END                                         # Type of email notifications will be sent (here set to END, which means an email will be sent when the job is done)
    #SBATCH --mail-type=FAIL                                        # Type of email notifications will be sent (here set to FAIL, which means an email will be sent when the job is fail to complete)
    #SBATCH --mail-user=my_email@adelaide.edu.au                    # Email to which notification will be sent
    
    # Execute your script (due to sequential nature, please select proper compiler as your script corresponds to)
    bash ./my_program.sh                                            # bash script used here for demonstration purpose, you should select proper compiler for your needs

### Creating a MPI Job Script\[[edit][edit 7] | [edit source][edit source 7]\] ###

A parallel (MPI) job is one that harnesses the computational power of multiple nodes, which are networked together and will perform related calculations or processes simultaneously. This can allow highly complex computational processes to be performed in much shorter time-frames. To enable parallel computing, the program you use will need to be MPI enabled or incorporate MPI enabled library. If you do need to run a parallel job on CPUs, following job script is an example:

#!/bin/bash
    #SBATCH -p batch        # partition (this is the queue your job will be added to) 
    #SBATCH -N 2            # number of nodes (here uses 2)
    #SBATCH -n 64           # number of cores (here 64 cores requested)
    #SBATCH --time=01:00:00 # time allocation, which has the format (D-HH:MM), here set to 1 hour
    #SBATCH --mem=32GB      # specify memory required per node (here set to 32 GB)
    
    mpirun -np 64 ./my_program

For jobs that use GPU accelerators, <my\_job.sh> will look like something like the example below:

#!/bin/bash
    #SBATCH -p batch # partition (this is the queue your job will be added to)
    #SBATCH -n 2 # number of cores (here 2 cores requested)
    #SBATCH --time=01:00:00 # time allocation, which has the format (D-HH:MM), here set to 1 hour
    #SBATCH --gres=gpu:1 # generic resource required (here requires 1 GPU)
    #SBATCH --mem=8GB # specify memory required per node (here set to 8 GB)
    
    mpirun -np 8 ./my_program

More commonly used options, which can be added to \#SBATCH lines includes

#SBATCH --mail-type=END # Type of email notifications will be sent (here set to END, which means an email will be sent when the job is done)
    #SBATCH --mail-type=FAIL # Type of email notifications will be sent (here set to FAIL, which means an email will be sent when the job is fail to complete)
    #SBATCH --mail-user=my_email@adelaide.edu.au # Email to which notification will be sent

SLURM is very powerful and allows detailed tailoring to fit your specific needs. If you want to explore all available SLURM parameters available, simply type following in the command line:

man sbatch

### Debiting compute time: Multiple associations\[[edit][edit 8] | [edit source][edit source 8]\] ###

The `-A association` argument specifies the association from which you wish the compute time to be be debited. Note that you normally only need to specify an association if you have access to multiple allocations, otherwise the scheduler will debit the resources used from your default association.

## HPC user guides - Selecting a Job Queue\[[edit][edit 9] | [edit source][edit source 9]\] ##

Standard compute jobs should select the **batch** queue. For further details and examples, please refer to the [job requirement guide][].

## Submitting A Job\[[edit][edit 10] | [edit source][edit source 10]\] ##

To submit a job script to the queue use the sbatch command:

$ sbatch my_job.sh

If your job script requires additional variables you can define these with the --export option to sbatch:

$ sbatch --export=ALL,var1=val1,var2=val2 my_job.sh

Be sure to include the ALL option to --export to ensure your job runs correctly.

## Monitoring The Queue\[[edit][edit 11] | [edit source][edit source 11]\] ##

You can view your job's progress through the queue with the `squeue` command:

$ squeue
     JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
     2916 batch my_job. a1234567 PD 0:00 1 (Resources)
     2915 batch my_job. a1234567 R 01:03 2 phoenixcs[1-2]
     2914 batch my_job. a1234567 R 00:21 1 phoenixg1

The fifth column gives the status of the job as follows: R - running, PD - Pending, F - Failed, ST - Stopped, TO - Timeout

Running `squeue` without arguments will list all currently running jobs. However if the list displayed is too long for you to easily locate your job, you can limit the search to your own jobs by using `-u` argument like this

squeue -u aXXXXXXX

where aXXXXXXX is your UofA ID number

## Canceling a Job\[[edit][edit 12] | [edit source][edit source 12]\] ##

To cancel a job you own, use the sbatch command followed by the slurm job ID:

$ scancel 2914

To cancel all jobs you own in a particular queue, use the -p argument:

$ scancel -p batch

[hide]: https://wiki.adelaide.edu.au/hpc/Getting_Started_Guide#
[1Connecting to Phoenix]: https://wiki.adelaide.edu.au/hpc/Getting_Started_Guide#Connecting_to_Phoenix
[2Transferring files to Phoenix]: https://wiki.adelaide.edu.au/hpc/Getting_Started_Guide#Transferring_files_to_Phoenix
[3Interacting with the Phoenix System_ Mastering Linux Basics]: https://wiki.adelaide.edu.au/hpc/Getting_Started_Guide#Interacting_with_the_Phoenix_System:_Mastering_Linux_Basics
[4Loading software packages]: https://wiki.adelaide.edu.au/hpc/Getting_Started_Guide#Loading_software_packages
[4.1For advance users_ To install your own packages]: https://wiki.adelaide.edu.au/hpc/Getting_Started_Guide#For_advance_users:_To_install_your_own_packages
[5Preparing to submit a job]: https://wiki.adelaide.edu.au/hpc/Getting_Started_Guide#Preparing_to_submit_a_job
[5.1Creating a simple job script]: https://wiki.adelaide.edu.au/hpc/Getting_Started_Guide#Creating_a_simple_job_script
[5.2Creating a MPI Job Script]: https://wiki.adelaide.edu.au/hpc/Getting_Started_Guide#Creating.C2.A0a_MPI_Job_Script
[5.3Debiting compute time_ Multiple associations]: https://wiki.adelaide.edu.au/hpc/Getting_Started_Guide#Debiting_compute_time:_Multiple_associations
[6HPC user guides - Selecting a Job Queue]: https://wiki.adelaide.edu.au/hpc/Getting_Started_Guide#HPC_user_guides_-_Selecting_a_Job_Queue
[7Submitting A Job]: https://wiki.adelaide.edu.au/hpc/Getting_Started_Guide#Submitting_A_Job
[8Monitoring The Queue]: https://wiki.adelaide.edu.au/hpc/Getting_Started_Guide#Monitoring_The_Queue
[9Canceling a Job]: https://wiki.adelaide.edu.au/hpc/Getting_Started_Guide#Canceling_a_Job
[edit]: https://wiki.adelaide.edu.au/hpc/index.php?title=Getting_Started_Guide&veaction=edit&vesection=1
[edit source]: https://wiki.adelaide.edu.au/hpc/index.php?title=Getting_Started_Guide&action=edit&section=1
[this guide]: https://wiki.adelaide.edu.au/hpc/Connecting_to_Phoenix
[edit 1]: https://wiki.adelaide.edu.au/hpc/index.php?title=Getting_Started_Guide&veaction=edit&vesection=2
[edit source 1]: https://wiki.adelaide.edu.au/hpc/index.php?title=Getting_Started_Guide&action=edit&section=2
[FAST]: https://wiki.adelaide.edu.au/hpc/Phoenix_data_management#.2Ffast
[Data transfer from_to Phoenix guide]: https://wiki.adelaide.edu.au/hpc/Data_transfer_from/to_Phoenix
[edit 2]: https://wiki.adelaide.edu.au/hpc/index.php?title=Getting_Started_Guide&veaction=edit&vesection=3
[edit source 2]: https://wiki.adelaide.edu.au/hpc/index.php?title=Getting_Started_Guide&action=edit&section=3
[Linux for HPC]: https://wiki.adelaide.edu.au/hpc/Linux_for_HPC
[edit 3]: https://wiki.adelaide.edu.au/hpc/index.php?title=Getting_Started_Guide&veaction=edit&vesection=4
[edit source 3]: https://wiki.adelaide.edu.au/hpc/index.php?title=Getting_Started_Guide&action=edit&section=4
[currently available packages and documentations]: https://wiki.adelaide.edu.au/hpc/Phoenix/Software
[email]: mailto:hpcsupport@adelaide.edu.au
[To install your own packages]: https://wiki.adelaide.edu.au/hpc/For_advanced_users:_Install_software_packages_for_your_own_usage
[edit 4]: https://wiki.adelaide.edu.au/hpc/index.php?title=Getting_Started_Guide&veaction=edit&vesection=5
[edit source 4]: https://wiki.adelaide.edu.au/hpc/index.php?title=Getting_Started_Guide&action=edit&section=5
[edit 5]: https://wiki.adelaide.edu.au/hpc/index.php?title=Getting_Started_Guide&veaction=edit&vesection=6
[edit source 5]: https://wiki.adelaide.edu.au/hpc/index.php?title=Getting_Started_Guide&action=edit&section=6
[edit 6]: https://wiki.adelaide.edu.au/hpc/index.php?title=Getting_Started_Guide&veaction=edit&vesection=7
[edit source 6]: https://wiki.adelaide.edu.au/hpc/index.php?title=Getting_Started_Guide&action=edit&section=7
[partitions _work queues]: https://wiki.adelaide.edu.au/hpc/Partitions_and_QoSs
[This post]: https://wiki.adelaide.edu.au/hpc/CPU_tasks_VS._GPU_tasks
[edit 7]: https://wiki.adelaide.edu.au/hpc/index.php?title=Getting_Started_Guide&veaction=edit&vesection=8
[edit source 7]: https://wiki.adelaide.edu.au/hpc/index.php?title=Getting_Started_Guide&action=edit&section=8
[edit 8]: https://wiki.adelaide.edu.au/hpc/index.php?title=Getting_Started_Guide&veaction=edit&vesection=9
[edit source 8]: https://wiki.adelaide.edu.au/hpc/index.php?title=Getting_Started_Guide&action=edit&section=9
[edit 9]: https://wiki.adelaide.edu.au/hpc/index.php?title=Getting_Started_Guide&veaction=edit&vesection=10
[edit source 9]: https://wiki.adelaide.edu.au/hpc/index.php?title=Getting_Started_Guide&action=edit&section=10
[job requirement guide]: https://wiki.adelaide.edu.au/hpc/Selecting_a_Job_Queue
[edit 10]: https://wiki.adelaide.edu.au/hpc/index.php?title=Getting_Started_Guide&veaction=edit&vesection=11
[edit source 10]: https://wiki.adelaide.edu.au/hpc/index.php?title=Getting_Started_Guide&action=edit&section=11
[edit 11]: https://wiki.adelaide.edu.au/hpc/index.php?title=Getting_Started_Guide&veaction=edit&vesection=12
[edit source 11]: https://wiki.adelaide.edu.au/hpc/index.php?title=Getting_Started_Guide&action=edit&section=12
[edit 12]: https://wiki.adelaide.edu.au/hpc/index.php?title=Getting_Started_Guide&veaction=edit&vesection=13
[edit source 12]: https://wiki.adelaide.edu.au/hpc/index.php?title=Getting_Started_Guide&action=edit&section=13