Overview
Scaling up the computational resources is a big advantage for doing certain large scale calculations on OSG. Consider the extensive sampling for a multi-dimensional Monte Carlo integration or molecular dynamics simulation with several initial conditions. These type of calculations require submitting lot of jobs.
In the previous example, we submitted the job to a single worker machine. About a million CPU hours per day are available to OSG users on an opportunistic basis. Learning how to scale up and control large numbers of jobs to realizing the full potential of distributed high throughput computing on the OSG.
In this section, we will see how to scale up the calculations with simple example. Once we understand the basic HTCondor script, it is easy to scale up.
$ tutorial ScalingUp-R
$ cd tutorial-ScalingUp-R
As we discussed in the previous section on HTCondor scripts, we need to prepare the job execution and the job submission scripts. Here again is our job execution script:
Submitting jobs concurrently
If we want to submit several jobs, we need to track log, out and error files for each
job. An easy way to do this is to add the $(Cluster) and $(Process)
macros to the HTCondor submit file.
universe = vanilla
Executable = R-wrapper.sh
arguments = mcpi.R $(Process)
transfer_input_files = mcpi.R # mcpi.R is the R program we want to run
output = Log/job.out.$(Cluster).$(Process)
error = Log/job.error.$(Cluster).$(Process)
log = Log/job.log.$(Cluster).$(Process)
requirements = (HAS_CVMFS_oasis_opensciencegrid_org =?= TRUE) # Checks if OASIS available
queue 100
Note the Queue 100. This tells Condor to enqueue 100 copies of this job
as one cluster.
Let us take a look at the execution script, R-wrapper.sh
#!/bin/bash
source /cvmfs/oasis.opensciencegrid.org/osg/modules/lmod/current/init/bash
module load R
Rscript $1 > mcpi.$2.out
The wrapper loads the R module and then executes the script with Rscript utility. From the submit
file described above, the first argument is the name of the R program - mcpi.R and the second argument is the process number. The process number is a sequence of integers and used here to name the output
files.
You'll see something like the following upon submission:
$ condor_submit R.submit
Submitting job(s).........................
100 job(s) submitted to cluster 837.
Apply your condor_q and connect watch knowledge to see this job
progress. Execute the following bash script to compute the average from all the jobs.
Interlude: utilization plots
Before we continue, let's look at a URL: your OSG Connect home page. If you have not signed in, you'll be redirected back to the main site. Sign In as you did the first time you signed up, and then click again on the your OSG Connect home link.
You see a number of graphs and plots here showing things happening in OSG Connect. We'll go over these briefly, then return later.
Connect Histogram
We're waiting on 1,000 jobs. Let's use connect watch to
watch for job completions. As soon as you see some jobs enter R state
(running), press control-C, and let's introduce a new command:
$ connect histogram
Val |Ct (Pct) Histogram
unl.edu |46 (68.66%) ████████████████████████████████▏
bu.edu |13 (19.40%) █████████▏
uconn.edu |2 (2.99%) █▌
CRUSH-OSG-10-5-220-34 |1 (1.49%) ▊
ufhpc |1 (1.49%) ▊
LAW-D-SBA01-S2-its-c6-osg-20141013|1 (1.49%) ▊
CRUSH-OSG-10-5-10-33 |1 (1.49%) ▊
iu.edu |1 (1.49%) ▊
vt.edu |1 (1.49%) ▊
This command gives us a simple histogram of where on the grid our jobs are running. The column on the left is (for the most) a list of sites that OSG jobs run on. At times we don't correctly group job locations together. For example, the two rows for CRUSH-* above are really the same site, but histogram doesn't know about that site (yet) so it displays as two. But most of the big sites are mapped correctly. You see that in my case, 67 of my 100 jobs have begun running, and among them 69% (46 of 67) are running at University of Nebraska at Lincoln.
connect histogram gives metrics on current jobs. As jobs complete,
they no longer appear. How to see where jobs have already run? connect
histogram --last shows the run sites of your last job cluster.
$ connect histogram --last
Val |Ct (Pct) Histogram
uc3 |49 (49.00%) ████████████████████████████████▏
bu.edu |21 (21.00%) █████████████▊
uconn.edu |11 (11.00%) ███████▎
unl.edu |9 (9.00%) ██████
mwt2.org |3 (3.00%) ██
c5a-s22.ufhpc |3 (3.00%) ██
LCS-215-021-S2-its-c6-osg-20141013|3 (3.00%) ██
cinvestav.mx |1 (1.00%) ▊
Post process⋅
Once the jobs are completed, you might want to invoke the script⋅
~~~ $mcpi_ave.bash ~~~
to compute the average value of pi from all the available outputs.⋅⋅
Key Points
- [x] Scaling up the computational resources on OSG is crucial to taking full advantage of grid computing.
- [x] Changing the value of
Queueallows the user to scale up the resources. - [x]
Argumentsallows you to pass parameters to a job script. - [x]
$(Cluster)and$(Process)can be used to name log files uniquely. - [x]
connect histogramgives a nice plot of resource assignments.
Getting Help
For assistance or questions, please email the OSG User Support team at support@opensciencegrid.org or visit the help desk and community forums.
This page was updated on Jan 28, 2020 at 13:00 from tutorials/tutorial-ScalingUp-R/README.md.