Chroma Tutorial 1: Measurement with chroma

Introduction

This part of the tutorial uses the pre-installed Chroma applications on a workstation. The purpose is to get you familiar with the model for making measurements so that you can go and run similar measurements yourself. Also, you will learn where to find example input files.

We will perform a spectroscopy calculation

We will use a simple 4x4x4x8 lattice configuration that comes bundled with the chroma in the tests/ directory. It is called test_purgaug.cfg1.

Setting up a working directory

Make a directory to work in. Let's call it tut1. You should then cd into it.

Getting the chroma program

The workhorse of the chroma system is a program called chroma. For this tutorial we will use a pre-installed copy of chroma located in the directory

/usr/local/chroma/scalar

Get the test Configuration

Copy the test configuration from the chroma source. Usually you will have checked out the chroma sources from the CVS, or you will have downloaded them as a tarball. For the purpose of this exercise I have put a copy of the source tree into

/usr/local/chroma/src

An overview of the computation

The chroma application allows you to perform a series of tasks on a configuration. It is a lot like setting up a pipeline that can be applied to every single configuration. The tasks that the chroma program executes are called (for historical reasons) InlineMeasurement tasks. We need to perform the following tasks to compute the spectroscopy on a single configuration.

• Read the gauge configuration
• Make a colour source
• Invert the fermion matrix on the colour source to compute a quark propagator
• Discard the colour source
• Potentially save the propagator
• Perform various contractions on the quark propagator

In chroma a colour source is the same size as a full propagator so that is why I have put in the step to explicitly discard it - to save memory. Also some people like to save propagators. This is not always necessary, for example a heavy quark prop may be computed faster on the QCDOC than it may be loaded depending on the size of the partition and the type of fermion used. In this example we will save the propagator just to show you how it is done.

You should also observe the data flow among the tasks. We read in the gauge configuration and it remains 'live' all through the tasks. The source is needed for the propagator calculation and not after. Likewise the propagator is needed by the contractions and for saving but not thereafter.

We need to specify all these tasks to Chroma. We do it in an XML file.

Anatomy of an XML file

Let us have a look at the input file for this job. You can find it in the example.ini.xml file here. As this is an XML file it is nice to view it as a tree from the Firefox browser (to do this, place the pointer over the hyperlink, press the right mouse button and, holding the right button down, select "Open Link in New Window"). Alternatively you can look at the document in an editor such as vi, or emacs. You should see a nested (indented) tag structure. (In firefox you can close up the tags) to give you a more or less verbose depiction of the tree) Closing all but the outer level tags the file looks in firefox like so:

• The outermost tag pair is called <chroma>
• Within this, at the first indentation, there are 4 tags:

  <annotation>

  This is an annotation tag. You can write whatever you want in it

  <Param>

  THis is the main parameter tag for chroma and we will look at it in greater detail next

  <RNG>

  This is a place to supply a random number seed to the program. This tag is optional, but you may like to have it if you have stochastic measurements such as Ritz eigenvalue/vector computations or noisy sources. - I will not focus on the contents of this tag anymore

  <Cfg>

  This tag details the default input configuration.

The main points of interest are in the <Param> tag. Let us look at this now. Expanding this tag in firefox I get:

There are 2 tags at this level

<InlineMeasurements>

This tag holds the list of measurement tasks. Each tag sits in an <elem> </elem> tag pair. You can see 5 tags corresponding to 5 measurement actions. We will look inside the measurement elements shortly

<nrow>

this tag contains the dimensions of the lattice. Now, some configurations formats contain this information in their metadata. Some however do not. Also occasionally it can happen that we work with unit gauge or other internal configurations. So we have to explicitly state the size of the lattice we use. You can see that these tags hold the value "4 4 4 8" which correspond to the dimensions of our lattice in the X, Y, Z and T directions respectively.

Before going into details of the measurements, lets look at the <Cfg> tag. Expanding it we get:

We see 2 subtags

<cfg_type>

Corresponds to the type of configuration we are reading. In this case we are reading the SZIN format, but we could well be configs in NERSC, SCIDAC (&ILDG), CPPACS and other formats too. Also there are some internal formats such as UNIT for the unit gauge, DISORDERED for a hot start and, WEAK_FIELD which is a slightly perturbed UNIT gauge.

<cfg_file>

Corresponds to the filename which holds the config. For non file based formats, the value of this field is ignored

Changes in the wind for <Cfg>

We note that times are changing. This is the old way of reading configs into chroma, where the input config was somehow special. In the new way, a configuration will be a NamedObject just like propagators etc. Already the mechanism for this is in place. Inline measurements support a gauge_id tag for a gauge field in a named object, and the <Cfg> tag is kept for backward compatibility. It places a gauge field into a default named object and measurements with no <gauge_id> tags use this default.

The measurement elements

Let us now look at the elements in the <InlineMeasurements> tag. I will expand them one by one.

The MAKE_SOURCE measurement

The first element corresponds to task of making a source (the gauge field already being handled with the <Cfg> tag.) A first level expansion of the <elem> tag is below:

There are some fairly univeral elements here:

<Name>

This is the name of the task. It is used as an ObjectId to the factory which creates the measurements. Each measurement has one of these. The name of this task is MAKE_SOURCE which is used for forward propagator source creation. (Sequential sources would be created with SEQSOURCE)

<Frequency>

This tag is useful in the context of using the measurement as an inline measurement The measurement is only measured if its frequency divides the current update number. Of course 1 divides into everything, so for the case when the measurement is being used in chroma we set the frequency to 1.

<Param>

This tag contains the parameters for the measurement and its contents are generally measurement specific

<NamedObject>

These are the tags that specify which objects the measurement creates. Most measurements have these. In this case, the measurement creates a source and we want to give it a name so we can refer to it elsewhere. The source in our program will be called pt_source as we are making a point source. Note Other measurements may contain more or fewer named objects with different IDs depending on their needs..

<xml_file>

This indicates that the output from this measurement should go to its own separate XML file rather than the global log XML file. This tag is common to measurements, however it is optional. If you do not have it in place, the output will go into the global log. I put it here to illustrate the general structure, but on the whole the output from the source creation is generally not considered exciting enough to merit a file on its own, and so in source creation this tag is often absent

The above tags are generic to the measurements and I shall not repeat there discussion unless there is something distinct to be said (Usually to do with the <NamedObject> tags). Now I will concentrate on what is in the measurement specific <Param> tag. Expanding it out we get

we see that there is are the following tags:

<version>

This indicates the version of the parameter input XML. Over time the information input can change and we try to keep track of it with version numbers.

<Source>

These are the parameters for the source creation, which is now done through factories. The <SourceType> subtag is the factory object ID. The <j_decay> subtag indicates the time direction and the <t_srce> tag specifies the coordinates of the point source

Of course the program can create more than just point sources, and it can create smeared sources with wavefunction and link smearing both at the source and the sink, however discussion of the markup for those measurements is beyond the scope of this short tutorial. The interested readers should look in the

chroma/tests/chroma/hadron/make_source

The PROPAGATOR measurement

The second <InlineMeasurement> element is used to compute a propagator. This is the conversion of the previous propagator program into an inline measurement. The propagator program itself has been deprecated.

It also has <Name>, <Frequency> and <NamedObject> tags which are as before except

• The name of this measurement if PROPAGATOR
• The <NamedObject> tag has now got two tags within it:

  <source_id>

  This is the source to invert on, it is the one we created in the previous step

  <prop_id>

  This is the ID of the propagator that will be created by this measurement

The measurement specific parameter tag now looks like:

Let us look at these tags:

<version>

Again we have a version type

<quarkSpinType>

This specifies the kind of propagator we make. In this instance FULL means we compute the full prop on all 12 spin colour components. We could also specify UPPER and LOWER for non relativistic projection using (1/2)(1 + gamma_4) or/and (1/2)(1 - gamma_4) projections respectively.

<obsvP>

This flag was put in to allow us to hook in some extra observables onto the ends of the propagator calculations. This option is really only useful in 5 dimenasional solves where the observables may be done with the 5D props, as the output of the propagator measurement will only be the effective 4D prop. In 5D enabling this flag to true for example could measure the correlators needed for 5D mres measurements. In 4D solves we should just keep this as false

<FermionAction>

This tag specifies the fermion action we use. It contains the subtags

<FermAct>

This is the factory Product ID that specifies the type of fermion action to be used

<FermionBC>

Specifies the fermion boundaries to be used. The <FermBC> tag is a factory key after which the parameters are specific to the type of boundary condition. Simple boundaries can be denoted with the <boundary> array which has a number for each dimension. In this case 1 means periodic, -1 means antiperiodic and 0 means Dirichlet boundaries. The example uses periodic boundaries in space and antiperiodic ones in time

<Kappa>

Speficies the Kappa parameter for Wilson fermions. Internally this is converted to the free field definition of the quark mass. One could also use a <Mass> tag here - which is more conventional in chroma.

<InvertParam;>

These are the parameters for the inversion. The subtags are as follows:

<invType>

The type of inverter to use.

<RsdCG>

The target relative residuum

<MaxCG>

The maximum number of iterations to perform before claiming nonconvergence.

We do not specify a a separate XML output file for this type.

The ERASE_NAMED_OBJECT element

The third elemen is used to discard the source which is no longer needed. Expanding out the XML looks like:

The structure should be familiar to you. The <NamedObject> section contains the ID of the named object to discard in the <object_id> tag. Apart from this the measurement has no <Param> tag.

The QIO_WRITE_NAMED_OBJECT element

We come to the 4th element of the inline measurements, which is the task where we save the propagator. The expanded XML looks like

The XML structure is as usual. The name of this task is QIO_WRITE_NAMED_OBJECT to signify that we will write the object in QIO LIME format. The interesting tags are as follows

<NamedObject>

We specify the details of the object to write.

<object_id>

The name of the named object

<object_type>

What kind of object we wish to write. Up until now this has been implicit, since sources could only create sources and propagators could only deal with sources and propagators. However now we have a routine that does not know up front what the type of the input object is. Now we could have got around this by either having a separate measurement to write each type of object, but we chose to have one general writing function with a string specifying the type. The type string is as one would have in the C++ code, with the exception that the < and > symbols due to templates are discarded. Ie a multi1d<LatticeColorMatrix> type would appear in the XML file as multi1dLatticeColorMatrix

<File>

Here we specify the details of the I/O.

<file_name>

The name of the output file

<file_volfmt>

The volume format of the output file. Here we choose SINGLEFILE to indicate the prop should be just dumped into a single file. We could also use MULTIFILE in which case every node would write its own file, or PARTFILE where only nominated I/O nodes would write files. In the latter cases the filename would have the string .volXXX appended to it where XXX is an ID pertaining to the node that wrote the file.

The SPECTRUM Measurement

We reach the last of our measurements, which measures simple degenerate spectroscopy. This is the conversion of the previous spectrum_w program into an inline measurement. The spectrum_w program itself has been deprecated. The expanded XML for the measurement is as below:

The <NamedObject> tag now contains a list of propagator object IDs to work on.since one may have computed several props in principle before reaching this measurement. Also this measurement will write its own spectrum.out.xml file. The tags in <Param;> are as follows:

Pt_snk

Whether to do spectroscopy with unsmeared (point) sink

Sl_snk

Whether to do spectroscopy with a smeared sink

Wl_snk

Whether to compute correlators with a Wall sink

MesonP

Whether to compute degenerate Meson Correlators

CurrentP

Whether to compute various current correlators

BaryonP

Whether to compute degenerate Baryon correlators

HybMesP

Whether to compute non-degenerate mesons

fact_sm

Smearing factor for the sink

numb_sm

The number of smearing steps to apply to the sink

time_rev

Whether to time reverse the prop before computing Baryon correlators

mom2_max

The maximum value of the norm of sink momenta

avg_equiv_mom

Whether to average over momenta of equal norm

wvf_kind

The kind of wavefunction for sink smearing. In this case we should use a Gauge invariant Gaussian wavefungion

wvf_param

The coefficient of the wavefunction smearing of the sink

wvfIntPar

An integer parameter for the sink smearing algorithm. Usually this is an iteration count

link_smear_fact

The factor to be used when performing link smearing at the sink

link_smear_num

The number of link smearing steps to perform at the sing

Running the program

Now that we know what the XML file does, let us run the chroma program

./chroma -i example.ini.xml -o example.out.xml

You should see lots of output on your screen as the code runs, telling you about its progress. I have captured the standard output in this file and you should see similar things. In particular, the program should

• echo back the XML for the inline measurements
• should display the name of each measurement, how long it took to run and whether it ran successfully
• You should be able to see the output from the solver as it works on each spin-color component

• example.out.xml
• spectrum.out.xml
• my_prop

Notes

• The command lines -i and - flags are used by chroma to select where to read the input XML from and write the output XML respectively. If you do not specify them chroma will try to read the input XML from a file called DATA and write the output XML to a file called XMLDAT in the working directory (these are left over from the old SZIN days. On operating systems that support the getcwd system call to get the path to the current working directory (eg under most UNIX operating systems) this all works fine. On the QCDOC the getcwd system call is not publicly exposed so the program will always look in the root directory from the mount point. These flags can be used to read input from and or place output to different working directories, and they are generally convenient if you run a long chain of tools and don't want to rename everything to DATA and XMLDAT between running the tools
• The XML file contains floating point numbers in variable width format even in double precision. Fixing this to always print an appropriate number of digits whatever the precision is on our to do list.
• In this exercise you have not been working on a parallel machine. Working on a parallel machine, you can add an extra option to chroma called -geom. This allows you to specify the geometry of your processors. We assume a logical 4D processor array, although underneath everything it may actually be a different topology (eg switched). On the QCDOC you of course need to remap the machine axes suitably prior to running. For example on a 128 node switched machine, you may choos to map the processor grid as a 1x2x8x8 grid. You would then use -geom 1 2 8 8. Alternatively you may have a 256 node QCDOC, which you have remapped (using qpartition_remap) as a 4x4x4x4 grid. You would then specify that on the command line as -geom 4 4 4 4

Output files

Have a look at the example.out.xml file. This will look awfully long and complicated. My one is here. However, the power of the firefox XML browser allows me to compress it. The toplevel view then looks as follows:

The tags containg the following information:

<Input>

This is an echo of the full input file. I have just spent considerable time going through this so I won't comment more on it

<ProgramInfo>

This is a rather useful tag.

Firstly it gives the version numbers of the code used (both for chroma and qdp). Secondly it tries to record providing the system sports a working date function. Finally it provides information on the geometry used in the problem

<latt_size>

The lattice size

<logical_size>

The logical processor grid size

<subgrid_size>

THe local lattice size per processor

total_volume

The number of sites in the total volume

subgrid_volume

The number of sites in the local volume

<RNG>

This just echoes back the stat of the random number generator

<ConfigInfo>

This is information that could be gleaned from the configuration headers if any

<Observables>

These are standard observables that are always measured on the default configuration.

They are:

<w_plaq>

The full plaquette, normalised that the value on the unit gauge is 1

<s_plaq>

The spatial plaquettes

<t_plaq>

The temporal plaquettes

<link>

The sum of the traces of the links (this is not gauge invariant)

<pollp;>

The Polyakov loop - each <elem></elem> contains the polyakov loop in a given direction. The list is ordered. THe first element corresponds to direction 0, the second to direction 1 etc, and the <re> and <im> tags refer to the real and imaginary components of the values respectively

These observables as a check that the configuration has been read in correctly.

<InlineObservables>

This element contains the actual output from each task. I will not go into this in glorious detail as it is quite complex. I draw your attention however to the second <elem></elem> pair which referes to the propagator, and when suitably collapsed in looks as follows:

Apart from repeating everything again about the input and the program etc we are also given the pion correlator constructed from the source in <Source_correlator> - we can see that we used a point source which had a Dirac Delta for each of the 12 components. We are also given the timesliced zero momentum pion correlator in the <prop_corr> tag within the <Prop_correlator> group. The total number of iterations is given in the <ncg_had> tag within the <Relaxation_iterations> group. Finally expanding the <QuarkProp4> tag we get the individual per component CG iteration counts:

The last element of the output from inline measurements is the output from the spectroscopy measurement. It is simply:

thus indicating that we redirected the output to the spectrum.out.xml file.

Output File from Spectroscopy

The output file spectrum.out.xml is now rather large, as all the spectrum computations have been dumped into it. Skipping the informational tags (such as the echoing of the input and the information gleaned from gauge configs we get to the results <Wilson_hadron_measurements> tag. This is now an array of the loop over propagators.

Each element of the loop throws up some information about the propagator it uses. This is followed by several tags:

• Point_Point_Wilson_Mesons
• Point_Shell_Wilson_Mesons
• Point_Point_Meson_Currents
• Point_Point_Wilson_Baryons
• Point_Shell_Wilson_Baryons

The meson tags are relatively easy to understand. The structure is as follows:

• Each <elem> deals with a particular gamma matrix which is inserted between the propagator and the anti-quark propagator. The spin basis is the same as the MILC code and also the Columbia Physics System. The basis is achiral one. The gamma matrices are labeled gamma_{1,2,3,4} for x,y,z,t in that order. The index of a gamma matrix is then mapped to a digit in a 4 digit binary number.

  # gamma( 1)    # ( 0001 ) --> 1
  # gamma( 2)    # ( 0010 ) --> 2
  # gamma( 3)    # ( 0100 ) --> 4
  # gamma( 4)    # ( 1000 ) --> 8
  

  The way we do mesons is enumerate the possible gamma matrix products

  Gamma = gamma_1^n1 * gamma_2^n2 * gamma_3^n3 * gamma_4^n4

  where n_i are single bit fields. Since gamma_1 comes first the bit for it must come first. So, then, for example,
  • gamma_5 is represented as  1111b = 15d .
  • gamma_1*gamma_2*gamma_4 is represented as  1011b = 11d (note the ordering).
• For each gamma value there are correlation functions for each momenta in the array with tag <momenta>
• Each element of the momenta array contains a linear index for the momenta, and an actual 3 vector of the momenta (or a representative momentum vector if equivalet momenta are summed over) and the correlation function in tag <mesprop>

Making use of the XML files from spectroscopy

Clearly at this stage of the game it becomes difficult to deal with the XML files just by looking at them in an editor, and some form of data binding would be useful. Or just like in the SZIN case, we can strip the data out of the XML file into individual files for each correlation function.

The task of stripping the data has been delegated to data analysis. There is a software package called ADAT which was written to sllow this kind of thing. It contains stripping programs for many of the interesting output formats in chroma such as spectroscopy and some 3 point functions. For this exercise we will use the strip_spectrum program.

Getting and compiling ADAT

In this section we will check out the adat software package from CVS and build the strip_spectrum application. If you don't have CVS installed or want to skip this section you can go on to the next bit using the copy of adat this copy of strip_spectrum (Linux FedoraCore3).

Check out the ADAT pacakge from the anonymous CVS by typing the following commands (I assume you are using the bash shell)

export CVSROOT=:pserver:anonymous@cvs.jlab.org:/group/lattice/cvsroot
export CVS_RSH=ssh
cvs login (hit return when asked for password)
cvs checkout adat

cd adat

./configure
make