Skip to content

Profiling

Perf-report

perf-report is a lightweight profiling tool that provides basic information about where the computing time is actually spent in a given application. It is developed by ARM (formerly Allinea) https://developer.arm.com/tools-and-software/server-and-hpc/debug-and-profile/arm-forge

Usage:

For example, analyzing a MPI job can be done with perf-report with the following script:

 #!/bin/bash
 #
 #SBATCH -J prflng        
 #SBATCH -N 2 
 #SBATCH --ntasks-per-node=16
 #SBATCH --ntasks-per-core=1

 module purge
 module load openmpi/4.1.4-gcc-12.2.0-x6sxxj3 arm/20.1_FORGE

 perf-report srun --jobid $SLURM_JOB_ID --mpi=pmi2 -n 32 ./a.out

This will result in the creation of two summary files in *.txt and *.html format providing an overview of the relative time spent in MPI, I/O, OpenMP etc.

forge = map + ddt

Synopsis:

DDT are ARM's (formerly Allinea's) advanced tools for performance analysis, see https://developer.arm.com/tools-and-software/server-and-hpc/debug-and-profile/arm-forge. Licenses for up to 512 parallel tasks are available. Of additional note,

Usage of map:

Profiling may be split into two steps, where the initial task is to create a *.map file from within a regular job script submitted to SLURM. In a subsequent step this *.map file can then be analyzed within an interactive session on the login node. For example, if we have mpicc -g -O3 ./my_prog.c -o a.out then we can profile it with the following submit script,

 #!/bin/bash
 #
 #SBATCH -J map       
 #SBATCH -N 4
 #SBATCH --ntasks-per-node 16
 #SBATCH --ntasks-per-core  1

 module purge
 module load  intel/18  intel-mpi/2018 arm/20.1_FORGE

 map --profile srun --jobid $SLURM_JOB_ID --mpi=pmi2 -n 64 ./a.out

which generates a *.map file (note the mention of #tasks and #nodes together with the date/time stamp in the filename) that may then be analyzed via the gui, ie

 ssh vsc4.vsc.ac.at -l my_uid -X
 cd wherever/the/map/file/may/be

 module purge
 module load allinea/20.1_FORGE

 map ./a_64p_4n_2020-09-24_11-42.map

Usage of ddt:

Debugging with ddt is currently limited to the Remote Launch option. There are two ways to launch this type of ddt-sessions: Best is to launch ddt-sessions on separate compute nodes.

ddt (semi-interactive via sbatch):

The following steps need to be carried out:

ssh vsc3.vsc.ac.at -l my_uid -X 
my_uid@l33$  module load allinea/18.2_FORGE 
my_uid@l33$  rm -rf ~/.allinea/  # to get rid of obsolete configurations from previous sessions 
my_uid@l33$ ddt &      # gui should open
                ... select 'Remote Launch - Configure'
                ... click  'Add' 
                ... set my_uid@l33.vsc.ac.at as 'Host Name' depending on which l3[1-5] node we currently are 
                ... set 'Remote Installation Directory' to /opt/sw/x86_64/glibc-2.17/ivybridge-ep/allinea/18.2_FORGE 
                ... check it with 'Test Remote Launch'     # will ask for a Password/OTP then monitor successful testing 
                ... click OK twice to close the dialogues 
                ... click Close to exit from the Configure menu
                ... next really select 'Remote Launch' by clicking the name tag that was auto-assigned   # (will again ask for a Password/OTP, 
                then the licence label should come up ok in the lower left corner and the connecting client should appear in the lower right corner )

ssh vsc3.vsc.ac.at -l my_uid   # a second terminal will be needed to actually start the debug session 
my_uid@l35$  cd wherever/my/app/may/be
my_uid@l35$  module load intel/16  intel-mpi/5  # or whatever else suite of MPI 
my_uid@l35$  mpiicc -g -O0 my_app.c
my_uid@l35$  vi  ./run.ddt.slrm.scrpt
                ... insert the usual '#SBATCH ...' commands
                ... don't forget to include 'module load allinea/18.2_FORGE' and '#SBATCH -L allinea@vsc'
                ... the actual program execution should be prefixed with 'ddt --connect   --np=32 ...' we better avoid express launch style 
my_uid@l35$  sbatch ./run.ddt.slrm.scrpt  # (in the other ddt-window sent into the background initially, a separate window pops up saying 'Reverse Connect Request' 
which needs to be accepted, then the usual ddt options will become available and the actual session may be launched by clicking 'Run')

ddt (fully interactive via salloc):

The following steps need to be carried out:

   ssh vsc3.vsc.ac.at -l my_uid -X
   my_uid@l33$  cd wherever/my/app/may/be
   my_uid@l33$  salloc -N 4 -L allinea@vsc
   my_uid@l33$  echo $SLURM_JOB_ID    # just to figure out the current job ID, say it's 8909346 
   my_uid@l33$  srun --jobid 8909346 -n 4 hostname | tee ./machines.txt # (this is important ! it looks like a redundant command but will actually fix a lot of the prerequisites usually taken care of in the SLURM prologue of regular submit scripts, one of them being provisioning of required licenses )
                ... let's assume we got n305-[044,057,073,074] which should now be listed inside file 'machines.txt' 
   my_uid@l33$  rm -rf ~/.allinea/   # to get rid of obsolete configurations from previous sessions 
   my_uid@l33$  module purge
   my_uid@l33$  module load  intel/18  intel-mpi/2018  allinea/20.1_FORGE   # or whatever else suite of MPI 
   my_uid@l33$  mpiicc -g -O0 my_app.c
   my_uid@l33$  ddt &     # gui should open 
                ... select 'Remote Launch - Configure'
                ... click  'Add'   
                ... set my_uid@n305-044 as 'Host Name' or any other node from the above list
                ... set 'Remote Installation Directory' to /opt/sw/x86_64/glibc-2.17/ivybridge-ep/allinea/20.1_FORGE
                ... keep auto-selected defaults for the rest, then check it with 'Test Remote Launch'     # should be ok 
                ... click OK twice to close the dialogues
                ... click Close to exit from the Configure menu
                ... next really select 'Remote Launch' by clicking the name tag that was auto-assigned above   # ( licence label should be ok in the lower left corner 
                and the hostname of the connecting client should appear in the lower right corner )

   ssh vsc3.vsc.ac.at -l my_uid   # a second terminal will be needed to actually start the debug session 
   my_uid@l34$  ssh n305-044      # log into that compute node that was selected/prepared above for remote launch
   my_uid@n305-044$  module purge
   my_uid@n305-044$  module load  intel/18  intel-mpi/2018  allinea/20.1_FORGE
   my_uid@n305-044$  cd wherever/my/app/may/be
   my_uid@n305-044$  srun --jobid 8909346 -n 16 hostname    # just a dummy check to see whether all is set up and working correctly 
   my_uid@n305-044$  ddt --connect srun --jobid 8909346 --mpi=pmi2 -n 64 ./a.out -arg1 -arg2   # (in the initial ddt-window a dialogue will pop up prompting 
   for a Reverse Connection request; accept it and click Run and the usual debug session will start )