ATLAS software with the Lancaster HEP Linux cluster

This page gives some brief instructions on how to run ATLAS (Athena) jobs on the Lancaster HEP cluster, how to submit jobs to the Grid from Lancaster, and how to copy Grid files to our local disks. It is recommended that you can work comfortably with the software on CERN's LXPLUS cluster before you start at Lancaster; further documentation can be found at the bottom of the page.


1. Location of ATLAS-related files on the filesystem

Data is stored in /data/atlas. If you have privately produced events then create a subdirectory here with your user ID name, and organize your files within this directory as you wish. On the other hand, if the data is officially produced ATLAS data from the Grid, then it should be organised as follows:
   (i) The release number of the software which created the data
   (ii) The channel identification number and description, i.e. number.description
For example:
    /data/atlas/11.0.41/017700.Bs_Jpsi_mu6mu3_phi_KplusKminus11041 for Bs->J/psi(mumu)phi(KK) events produced with release 11.0.41. The events go straight into this directory. The assumption is that only reconstructed (AOD) files will be copied to Lancaster - ESD and digits should remain on the Grid. Also, you should not store any code here - just data files.

Code is stored in /atlas/athena/release_<releaseNumber>. The code here is from the ATLAS repository and should not be modified - it will be used by everyone in the group.

Your own work should be stored in your home space.

2. Installing a new release

The assumption is that you wish to install release 12.0.3. First, log into the HEP cluster (use an SL4 machine). Then:

[id@lapi]$ cd /atlas/athena
[id@lapi]$ export release=12.0.3
[id@lapi]$ export installdir=/atlas/athena/release_12.0.3/
[id@lapi]$ mkdir $installdir
[id@lapi]$ chmod 777 $installdir
[id@lapi]$ export http_proxy=http://wwwcache.lancs.ac.uk:8080
[id@lapi]$ wget http://physics.bu.edu/pacman/sample_cache/tarballs/pacman-latest.tar.gz
[id@lapi]$ tar xzf pacman-latest.tar.gz
[id@lapi]$ rm -f pacman-latest.tar.gz
[id@lapi]$ cd pacman-*
[id@lapi]$ source setup.sh
[id@lapi]$ cd $installdir
[id@lapi]$ pacman -http-proxy http://wwwcache.lancs.ac.uk:8080 -get am-CERN:12.0.3

 Then go for a very long lunch; this will take several hours. Once it's finished, please inform the rest of the group that the new release is in place.

3. Setting up CMT

Before you can do anything, you'll need to create a directory called cmthome in your home directory, move into it, and copy in the following script. The file must be called requirements.

#---------------------------------------------------------------------
set CMTSITE STANDALONE
set SITEROOT /atlas/athena/release_12.0.3
macro ATLAS_DIST_AREA /atlas/athena/release_12.0.3
macro ATLAS_TEST_AREA ${HOME}
apply_tag setup
apply_tag simpleTest
use AtlasLogin AtlasLogin-* $(ATLAS_DIST_AREA)
set CMTCONFIG i686-slc3-gcc323-opt
set DBRELEASE_INSTALLED 2.5
#---------------------------------------------------------------------

 The assumption here is that you are working with 12.0.3 and have set your working area to be $HOME/12.0.3. If you decide otherwise or are using a different release, you should modify the script accordingly. Now, save and exit and establish your CMT setup by typing
[id@lapi]$ source /atlas/athena/release_12.0.3/CMT/*/mgr/setup.sh
[id@lapi]$ cmt config
 You only have to do this once per installation.

4. Running unmodified Athena

If you wish to run Athena without modifying the code, there is no need to check out any packages (even a TestRelease or TestProject). All you have to do is set up Athena, get the job options and run. The instructions below assume you are using release 12.0.3 and have a working directory in $HOME/12.0.3.

5. Checking out a package and modifiying the code

In order to to this you must have set up your account to access the CERN code repository; instructions for doing this are found at the bottom of this page. If you have not done this, please follow those instructions before continuing further.

6. Using the batch system

If you need to run long jobs, then you should use the batch system to avoid slowing down your and other people's machines. The batch system will also allow you to submit many jobs at once, and will automatically run them when resources become available. In order to use the system, you essentially need to provide a script listing the commands that you would issue to run "normally". An example (for running reconstruction) is shown below:
source /home/atlas/jc/setupAthena.sh
cd /home/atlas/jc/11.0.41
mkdir run
cd run
source /atlas/athena/release_11.0.41/dist/11.0.41/Control/AthenaRunTime/AthenaRunTime-00-00-06/cmt/setup.sh
get_files your_jobOptions.py
athena.py your_jobOptions.py

The important points to note are (i) The use of absolute paths at the start and (ii) The use of "athena.py" rather than "athena". Everything else should be the same. To submit the job, you need to issue this command:

[id@lapi]$ qsub <submission_script.sh> -l nodes=SL3 -l cput=48:00:00

 where the "nodes=SL3" orders the job to go to a Scientific Linux 3 machine, and "cput=48:00:00" sets the maximum time (you can put any time in here in the form HH:MM:SS). The job queue can be checked with

[id@lapi]$ qstat

 When your jobs are finished, they will simply disappear off the list. In the directory from where you submitted the job, a directory will have appeared which contains the log and error output of the job. This is useful for identifying problems.

Appendix: setting up an ssh token

This is necessary if you want to be able to check out ATLAS packages from CERN and use cmt. These instructions have been pilfered from the Manchester ATLAS group.
1. ssh id@lapi.lancs.ac.uk
2. [id@lapi]$ cd .ssh
3. Check if the following files are present: identity and identity.pub. If so, go to the next step. If not, then:
[id@lapi]$ ssh-keygen -t rsa1
This generates some output. Accept the default directory if it is /home/atlas/user_name/.ssh/identity and insert a passphrase when prompted:
4. Copy the public key (~/.ssh/identity.pub) to your AFS home directory at CERN:
[id@lapi]$ scp ~/.ssh/identity.pub CERN_USERNAME@lxplus.cern.ch:/afs/cern.ch/user/<n>/<user_name>/.
 5. Log on to LXPLUS and run set_ssh by doing the following
[id@lapi]$ ssh CERN_USERNAME@lxplus.cern.ch
[id@lxplus017]$ /afs/cern.ch/project/cvs/dist/bin/set_ssh
 6. Add the PUBLIC key you copied in 3. in your ~/.ssh/authorized_keys file with the following command:
[id@lxplus017]$ cat ~/identity.pub >> ~/.ssh/authorized_keys
 7. Set the permissions as follows:
[id@lxplus017]$ chmod 755 ~/.ssh ~/public ~/.ssh/authorized_keys
 8. Logoff lxplus
[id@lxplus017]$ logout
To stop Athena pestering you for a password every time you download a single file from CERN you have to setup the atlas-sw.cern.ch and isscvs.cern.ch servers properly in a file called config in the .ssh directory in your home area. Do it like this:
9. Log into the HEP group system (if you're not already)
10. From your home directory $HOME navigate to .ssh/ and modify (or create) a file called config containing the following:
Host *
ForwardAgent yes

Host lxplus.cern.ch lxplus
User <CERN_username>
Protocol 2
PubkeyAuthentication no
PasswordAuthentication yes

Host isscvs.cern.ch isscvs
User <CERN_username>
Protocol 1
ForwardX11 no
IdentityFile ~/.ssh/identity
PasswordAuthentication no

Host atlas-sw.cern.ch atlas-sw
HostName atlas-sw.cern.ch
User <CERN_username>
Protocol 1
IdentityFile ~/.ssh/identity
PasswordAuthentication no
ForwardX11 yes

3. Test the setup by making a small testSSH.sh in your home directory (not .ssh) containg the following:
#Sorts out ssh, asks for the passphrase used to make the rsa1 key
if [ -z "$SSH_AUTH_SOCK" ] ; then
eval ` ssh-agent -s`
ssh-add
fi

 11. Run the testSSH.sh script
[id@lapi]$ source testSSH.sh
This will prompt for the passphrase you entered when making the rsa1 key if all is well.

12. Try to logon to isscvs.cern.ch as shown below. This should not ask for a password. The command reads ssh -ONE not L.

ssh -1 <CERN_USERNAME>@isscvs.cern.ch>

If everything is OK you will see a CERN login screen, and then an error message about there being the wrong number of arguments. Ignore this. However, if you are prompted for a password check your config file and that retrace the CMT setup steps.

James Catmore, 23/10/2006
Instructions based partly on ATLAS WorkBook, partly on Manchester HEP group pages, and partly from our own experimentation.

* Technical reasons = some bizarre feature of ATLAS software that is too convoluted to explain. In other words, I have no idea why.