Chroma Tutorial 1: Measurement with chroma

Introduction

This part of the tutorial uses the pre-installed Chroma applications on a workstation. The purpose is to get you familiar with the model for making measurements so that you can go and run similar measurements yourself. Also, you will learn where to find example input files.

We will perform a spectroscopy calculation

We will use a simple 4x4x4x8 lattice configuration that comes bundled with the chroma in the tests/ directory. It is called test_purgaug.cfg1.

Setting up a working directory

Make a directory to work in. Let's call it tut1. You should then cd into it.

Getting the chroma program

The workhorse of the chroma system is a program called chroma. You can get it various ways

On the NESC Lab Machines

Just go to ~/chroma/tutorials/Tutorial1! You should find the chroma program there.

Using a version of chroma you have just built

If you have just built and installed chroma yourself, you will have different source, build and installation directories. We will refer to the source directory as the place where the sources are and the install directory as the place where the executables are. The build directory is the place where you built chroma (the directory where you type make and make install). When you build your own codes, the executables get built in the

mainprogs/main

mainprogs/tests

If you have already typed make install you will find all the executables copied to prefix/bin. Where prefix is the installation directory you specified with the --prefix= configure option. If you didn't explicitly specify a prefix directory it defaults to /usr/local/

If you have just built using jlab-standard-chroma-build

If you have just built the code using the jlab-standard-chroma-build package, the chroma sources will be installed into

install-root/chroma/version/src

The binaries are in

install-root/chroma/version/target/bin

Suppose that your install-root is /dist/scidac and that you are building chroma version chroma3-28-4 for a scalar target. Then your source directory will be:

/dist/scidac/chroma/chroma3-28-4/src

/dist/scidac/chroma/chroma3-28-4/scalar/bin

Get the test Configuration

The gauge configuration for this tutorial is called test_purgaug.cfg1 and it lives in the tests/ subdirectory of the source directory. (eg: /opt/scidac/chroma/tests/test_purgaug.cfg1) You can also get a copy from here

An overview of the computation

The chroma application allows you to perform a series of tasks on a configuration. It is a lot like setting up a pipeline that can be applied to every single configuration. The tasks that the chroma program executes are called (for historical reasons) InlineMeasurement tasks. We need to perform the following tasks to compute the spectroscopy on a single configuration.

• Read the gauge configuration
• Make a colour source
• Invert the fermion matrix on the colour source to compute a quark propagator
• Discard the colour source
• Potentially save the propagator
• Potentially sink smear the propagator
• Perform various contractions on the quark propagator

In chroma a colour source is the same size as a full propagator so that is why I have put in the step to explicitly discard it - to save memory. Also some people like to save propagators. This is not always necessary, for example a heavy quark prop may be computed faster on the QCDOC than it may be loaded depending on the size of the partition and the type of fermion used. In this example we will save the propagator just to show you how it is done. Also not everyone likes to sink smear their props. However, the way the current spectroscopy chain is set up they have to go through the sink smearing measurement, even if that is trivial, as it attaches various information about the sinks to the propagators.

You should also observe the data flow among the tasks. We read in the gauge configuration and it remains 'live' all through the tasks. The source is needed for the propagator calculation and not after. Likewise the propagator is needed by the contractions and for saving but not thereafter.

We need to specify all these tasks to Chroma. We do it in an XML file.

Anatomy of an XML file

Let us have a look at the input file for this job. You can find it in the example.ini.xml file here. As this is an XML file it is nice to view it as a tree from the Firefox browser (to do this, place the pointer over the hyperlink, press the right mouse button and, holding the right button down, select "Open Link in New Window"). Alternatively you can look at the document in an editor such as vi, or emacs. You should see a nested (indented) tag structure. (In firefox you can close up the tags) to give you a more or less verbose depiction of the tree) Closing all but the outer level tags the file looks in firefox like so:

• The outermost tag pair is called <chroma>
• Within this, at the first indentation, there are 4 tags:

  <annotation>

  This is an annotation tag. You can write whatever you want in it

  <Param>

  THis is the main parameter tag for chroma and we will look at it in greater detail next

  <RNG>

  This is a place to supply a random number seed to the program. This tag is optional, but you may like to have it if you have stochastic measurements such as Ritz eigenvalue/vector computations or noisy sources. - I will not focus on the contents of this tag anymore

  <Cfg>

  This tag details the default input configuration.

The main points of interest are in the <Param> tag. Let us look at this now. Expanding this tag in firefox I get:

There are 2 tags at this level

<InlineMeasurements>

This tag holds the list of measurement tasks. Each tag sits in an <elem> </elem> tag pair. You can see 8 tags corresponding to 8 measurement actions. We will look inside the measurement elements shortly

<nrow>

this tag contains the dimensions of the lattice. Now, some configurations formats contain this information in their metadata. Some however do not. Also occasionally it can happen that we work with unit gauge or other internal configurations. So we have to explicitly state the size of the lattice we use. You can see that these tags hold the value "4 4 4 8" which correspond to the dimensions of our lattice in the X, Y, Z and T directions respectively.

Before going into details of the measurements, lets look at the <Cfg> tag. Expanding it we get:

We see 2 subtags

<cfg_type>

Corresponds to the type of configuration we are reading. In this case we are reading the SZIN format, but we could well be configs in NERSC, SCIDAC (&ILDG), CPPACS and other formats too. Also there are some internal formats such as UNIT for the unit gauge, DISORDERED for a hot start and, WEAK_FIELD which is a slightly perturbed UNIT gauge.

<cfg_file>

Corresponds to the filename which holds the config. For non file based formats, the value of this field is ignored

The <Cfg> task reads the gauge field into a NamedObject. NamedObjects are similar to a little internal file system, where we can put objects to share between tasks, and refer to the by names. Initially this system was added for machines where I/O was prohibitively slow. Since their inception, NamedObjects have become prevasive and are now used outside of just measurements such as a place to put monomials in HMC. The object_id of the config specified in the <Cfg> element is default_gauge_field.

The measurement elements

Let us now look at the elements in the <InlineMeasurements> tag. I will expand them one by one.

The MAKE_SOURCE measurement

The first element corresponds to task of making a source (the gauge field already being handled with the <Cfg> tag.) A first level expansion of the <elem> tag is below:

There are some fairly univeral elements here:

<Name>

This is the name of the task. It is used as an ObjectId to the factory which creates the measurements. Each measurement has one of these. The name of this task is MAKE_SOURCE which is used for forward propagator source creation. (Sequential sources would be created with SEQSOURCE)

<Frequency>

This tag is useful in the context of using the measurement as an inline measurement The measurement is only measured if its frequency divides the current update number. Of course 1 divides into everything, so for the case when the measurement is being used in chroma we set the frequency to 1.

<Param>

This tag contains the parameters for the measurement and its contents are generally measurement specific

<NamedObject>

These are the tags that specify which objects the measurement creates. Most measurements have these. In this case, the measurement creates a source and we want to give it a name so we can refer to it elsewhere. The source in our program will be called sh_source_1. The measurement also may make reference to a gauge field which we specify by the <gauge_id> element. In this case we will use the default_gauge_field that is specified by the <Cfg> tag (see earlier). Note Other measurements may contain more or fewer named objects with different IDs depending on their needs..

The above tags are generic to the measurements and I shall not repeat there discussion unless there is something distinct to be said (Usually to do with the <NamedObject> tags). Now I will concentrate on what is in the measurement specific <Param> tag. Expanding it out we get

we see that there is are the following tags:

<version>

This indicates the version of the parameter input XML. Over time the information input can change and we try to keep track of it with version numbers.

<Source>

These are the parameters for the source creation, which is now done through factories. The <SourceType> subtag is the factory object ID. The <j_decay> subtag indicates the time direction and the <t_srce> tag specifies the coordinates of the center of the source source. In addition there is a subgroups to the parameters of the source:

<SmearingParams>

This refers to the smearing of the source. In this case we use gauge invariant wafefunction smearing. The smearing factor is 2 and we apply 5 iterations. We omit direction 3 from smearing so that the smearing is 'spatial' only.

Of course the program can create more than just point sources, and it can create smeared sources with wavefunction and link smearing both at the source and the sink, however discussion of the markup for those measurements is beyond the scope of this short tutorial. The interested readers should look in the

chroma/tests/chroma/hadron/make_source

The PROPAGATOR measurement

The second <InlineMeasurement> element is used to compute a propagator. This is the conversion of the previous propagator program into an inline measurement. The propagator program itself has been deprecated.

It also has <Name>, <Frequency> and <NamedObject> tags which are as before except

• The name of this measurement if PROPAGATOR
• The <NamedObject> tag has now got three tags within it:

  <gauge_id>

  This is the ID of the gauge field to use in the inversions

  <source_id>

  This is the source to invert on, it is the one we created in the previous step

  <prop_id>

  This is the ID of the propagator that will be created by this measurement

The measurement specific parameter tag now looks like:

Let us look at these tags:

<version>

Again we have a version type

<quarkSpinType>

This specifies the kind of propagator we make. In this instance FULL means we compute the full prop on all 12 spin colour components. We could also specify UPPER and LOWER for non relativistic projection using (1/2)(1 + gamma_4) or/and (1/2)(1 - gamma_4) projections respectively.

<obsvP>

This flag was put in to allow us to hook in some extra observables onto the ends of the propagator calculations. This option is really only useful in 5 dimenasional solves where the observables may be done with the 5D props, as the output of the propagator measurement will only be the effective 4D prop. In 5D enabling this flag to true for example could measure the correlators needed for 5D mres measurements. In 4D solves we should just keep this as false

<FermionAction>

This tag specifies the fermion action we use. It contains the subtags

<FermAct>

This is the factory Product ID that specifies the type of fermion action to be used

<AnisoParam>

This group specifies anisotropy for the action. In this case since <anisoP> is set to false the action is not anisotropic (ie it is isotropic). If it were set to true the fields would have the following meaning:

<t_dir>

The time direction (direction of the fine dimension)

<xi_0>

The usual xi_0 term for (inverse) bare gauge anisotropy

<nu>

The fermion anisotropy

<FermState>

This is the element where we specify things about our fermion link field state. In this example, we don't stout smear or anything complex, so we just call it a SIMPLE_FERM_STATE. The only subtag to set here is to do with the fermionic boundaries.

<FermionBC>

Specifies the fermion boundaries to be used. The <FermBC> tag is a factory key after which the parameters are specific to the type of boundary condition. Simple boundaries can be denoted with the <boundary> array which has a number for each dimension. In this case 1 means periodic, -1 means antiperiodic and 0 means Dirichlet boundaries. The example uses periodic boundaries in space and antiperiodic ones in time

<Kappa>

Speficies the Kappa parameter for Wilson fermions. Internally this is converted to the free field definition of the quark mass. One could also use a <Mass> tag here - which is more conventional in chroma.

<InvertParam>

These are the parameters for the inversion. The subtags are as follows:

<invType>

The type of inverter to use - this is a factory key which is followed by parameters pertinent to the inversion

<RsdCG>

The target relative residuum

<MaxCG>

The maximum number of iterations to perform before claiming nonconvergence.

We do not specify a a separate XML output file for this type.

An ERASE_NAMED_OBJECT element

The third elemen is used to discard the source which is no longer needed. Expanding out the XML looks like:

The structure should be familiar to you. The <NamedObject> section contains the ID of the named object to discard in the <object_id> tag. Apart from this the measurement has no <Param> tag.

The QIO_WRITE_NAMED_OBJECT element

We come to the 4th element of the inline measurements, which is the task where we save the propagator. The expanded XML looks like

The XML structure is as usual. The name of this task is QIO_WRITE_NAMED_OBJECT to signify that we will write the object in QIO LIME format. The interesting tags are as follows

<NamedObject>

We specify the details of the object to write.

<object_id>

The name of the named object

<object_type>

What kind of object we wish to write. Up until now this has been implicit, since MAKE_SOURCE could only create sources and PROPAGATOR could only create propagators. However now we have a routine that does not know up front what the type of the object to write is. Now we could have got around this by either having a separate measurement to write each type of object, but we chose to have one general writing "measutement" with a string specifying the type. The type string is the same one we would have in the C++ code, with the exception that the < and > symbols due to templates are discarded. Ie a multi1d<LatticeColorMatrix> type would appear in the XML file as multi1dLatticeColorMatrix

<File>

Here we specify the details of the I/O.

<file_name>

The name of the output file

<file_volfmt>

The volume format of the output file. Here we choose SINGLEFILE to indicate the prop should be just dumped into a single file by QIO. We could also use MULTIFILE in which case every node would write its own file, or PARTFILE where only nominated I/O nodes would write files. In the latter cases the filename would have the string .volXXX appended to it where XXX is an ID pertaining to the node that wrote the file.

The SINK_SMEAR Measurement

We reach the 5th of our measurements, which performs sink smearing on a prop. The XML structure is as usual, with the name SINK_SMEAR. We perform two sink smearing measurements here - one is a simple point sink (ie no smearing at all). The XML looks like this:

Here the <SinkType> specifies that we have a point sink. The <j_decay> denotes that we treat dimension 3 as time. The <gauge_id> denotes the input gauge field (which is not used when making a point sink) and the <prop_id> denotes tha propagator on which we are performing this trivial sink smearing. The trivially sink smeared prop (which in reality is a source smeared point sink propagator) is given the name sh_pt_sink_1 in the <smeared_prop_id> tag.

The second SINK_SMEAR element

The second sink smearing task is a little more convoluted. This time we perform wavefunction smearing on the propagator to create a shell sink. The XML is like for the point sink but with an extra <SmearingParam> element which has parameters that are the same as in the MAKE_SOURCE element. The name we give to the sink smeared prop (which is actually both source and sink smeared prop) is sh_sh_sink_1. The XML is below:

The second ERASE_NAMED_OBJECT element

The 7th measurement is another ERASE_NAMED_OBJECT measurement that erases the un-sink-smeared prop to save some memory. It is very similar in all aspects to the first one.

The HADRON_SPECTRUM element

Our last measurement measures basic hadron spectroscopy observables. The XML is below:

version

The standard version parameter.

MesonP

Whether to compute Meson Correlators

CurrentP

Whether to compute various current correlators

BaryonP

Whether to compute Baryon correlators

time_rev

Whether to time reverse the prop before computing Baryon correlators

mom2_max

The maximum value of the norm of sink momenta

avg_equiv_mom

Whether to average over momenta of equal norm

The named object tag specifies the gauge field to use, and then a list of <sink_pairs>. The sink pairs contain a list of sink smeared prop pairs. These pairs refer to the sink smeared quark props to use in computing the spectroscopy. In this case, the spectroscopy correlators are computed twice. Once with smeared-point props and the second time with smeared smeared correlators.

Finally this measurement has an extra tag: <xml_file>. This tag specifies that the output from the data should be written to a separate XML file, rather than in the default output XML file. This option is available on several inline measurements (almost a de-facto standard). In this case the output is placed into a file called hadspec.dat.xml.

Running the program

Now that we know what the XML file does, let us run the chroma program

./chroma -i example.ini.xml -o example.out.xml

You should see lots of output on your screen as the code runs, telling you about its progress. I have captured the standard output in this file and you should see similar things. In particular, the program should

• echo back the XML for the inline measurements
• should display the name of each measurement, how long it took to run and whether it ran successfully
• You should be able to see the output from the solver as it works on each spin-color component

• hadspec.dat.xml
• example.out.xml
• sh_prop_1

Notes

• The command lines -i and -o flags are used by chroma to select where to read the input XML from and write the output XML respectively. In the gauge generation programs the -l provides a location to log the progress of the MD. If you do not specify them chroma will try to read the input XML from a file called DATA and write the output XML to a file called XMLDAT and any log file to XMLLOG in the working directory (these are left over from the old SZIN days. On operating systems that support the getcwd system call to get the path to the current working directory (eg under most UNIX operating systems) this all works fine. On the QCDOC the getcwd system call is not publicly exposed so the program will always look in the root directory from the mount point. These flags can be used to read input from and or place output to different working directories, and they are generally convenient if you run a long chain of tools and don't want to rename everything to DATA and XMLDAT between running the tools
• In this exercise you have not been working on a parallel machine. Working on a parallel machine, you can add an extra option to chroma called -geom. This allows you to specify the geometry of your processors. We assume a logical 4D processor array, although underneath everything it may actually be a different topology (eg switched). On the QCDOC you of course need to remap the machine axes suitably prior to running. For example on a 128 node switched machine, you may choos to map the processor grid as a 1x2x8x8 grid. You would then use -geom 1 2 8 8. Alternatively you may have a 256 node QCDOC, which you have remapped (using qpartition_remap) as a 4x4x4x4 grid. You would then specify that on the command line as -geom 4 4 4 4

Output files

Have a look at the example.out.xml file. This will look awfully long and complicated. My one is here. However, the power of the firefox XML browser allows me to compress it. The toplevel view then looks as follows:

The tags containg the following information:

<Input>

This is an echo of the full input file. I have just spent considerable time going through this so I won't comment more on it

<ProgramInfo>

This is a rather useful tag.

Firstly it gives the version numbers of the code used (both for chroma and qdp). Secondly it tries to record providing the system sports a working date function. Finally it provides information on the geometry used in the problem

<latt_size>

The lattice size

<logical_size>

The logical processor grid size

<subgrid_size>

The local lattice size per processor

<total_volume>

The number of sites in the total volume

<subgrid_volume>

The number of sites in the local volume

<RNG>

This just echoes back the stat of the random number generator

<ConfigInfo>

This is information that could be gleaned from the configuration headers if any

<Observables>

These are standard observables that are always measured on the default configuration.

They are:

<w_plaq>

The full plaquette, normalised that the value on the unit gauge is 1

<s_plaq>

The spatial plaquettes

<t_plaq>

The temporal plaquettes

<plane_01_plaq>

The plaquette in the 0-1 plane

<plane_02_plaq>

The plaquette in the 0-2 plane

<plane_12_plaq>

The plaquette in the 1-2 plane

<plane_03_plaq>

The plaquette in the 0-3 plane

<plane_13_plaq>

The plaquette in the 1-3 plane

<plane_23_plaq>

The plaquette in the 2-3 plane

<link>

The sum of the traces of the links (this is not gauge invariant)

<pollp>

The Polyakov loop - each <elem></elem> contains the polyakov loop in a given direction. The list is ordered. The first element corresponds to direction 0, the second to direction 1 etc, and the <re> and <im> tags refer to the real and imaginary components of the values respectively

These observables as a check that the configuration has been read in correctly.

<InlineObservables>

This element contains the actual output from each task. I will not go into this in glorious detail as it is quite complex. I draw your attention however to the second <elem></elem> pair which referes to the propagator, and when suitably collapsed in looks as follows:

Apart from repeating everything again about the input and the program etc we are also given the pion correlator constructed from the source in <Source_correlator> - we can see that we used a smeared source - had we used a Dirac Delta for each of the 12 components the first element of the <source_corr> would be 12 (one for each spin color component). We are also given the timesliced zero momentum pion correlator in the <prop_corr> tag within the <Prop_correlator> group. The total number of iterations is given in the <ncg_had> tag within the <Relaxation_iterations> group. Finally expanding the <QuarkProp4> tag we get the individual per component CG iteration counts and attained relative residua:

The last element of the output from inline measurements is the output from the spectroscopy measurement. It is simply:

thus indicating that we redirected the output to the hadspec.dat.xml file.

Output File from Spectroscopy

The output file hadspec.dat.xml is now rather large, as all the spectrum computations have been dumped into it. Skipping the informational tags (such as the echoing of the input and the information gleaned from gauge configs we get to the results <Wilson_hadron_measurements> tag.

This tag consists of a series of <elem> tags, one for each <sink_pair> we specified in the HADRON_SPECTRUM measurement. The <Mass_1> and <Mass_2> tags reveal the masses used for the propagators that are correlated and <t0> shows the time coordinate of the source. There is also information gleaned from the headers of the propagators as well as from the sink smearings. There after come the correlations themselves these in this case live in parent tags such as:

• Shell_Point_Wilson_Mesons
• Shell_Point_Wilson_Currents
• Shell_Point_Wilson_Baryons

The meson tags are relatively easy to understand. The structure is as follows:

• Each <elem> deals with a particular gamma matrix which is inserted between the propagator and the anti-quark propagator. The spin basis is the same as the MILC code and also the Columbia Physics System. The basis is achiral one. The gamma matrices are labeled gamma_{1,2,3,4} for x,y,z,t in that order. The index of a gamma matrix is then mapped to a digit in a 4 digit binary number.

  # gamma( 1)    # ( 0001 ) --> 1
  # gamma( 2)    # ( 0010 ) --> 2
  # gamma( 3)    # ( 0100 ) --> 4
  # gamma( 4)    # ( 1000 ) --> 8
  

  The way we do mesons is enumerate the possible gamma matrix products

  Gamma = gamma_1^n1 * gamma_2^n2 * gamma_3^n3 * gamma_4^n4

  where n_i are single bit fields. Since gamma_1 comes first the bit for it must come first. So, then, for example,
  • gamma_5 is represented as  1111b = 15d .
  • gamma_1*gamma_2*gamma_4 is represented as  1011b = 11d (note the ordering).
• For each gamma value there are correlation functions for each momenta in the array with tag <momenta>
• Each element of the momenta array contains a linear index for the momenta, and an actual 3 vector of the momenta (or a representative momentum vector if equivalet momenta are summed over) and the correlation function in tag <mesprop>

Making use of the XML files from spectroscopy

Clearly at this stage of the game it becomes difficult to deal with the XML files just by looking at them in an editor, and some form of data binding would be useful. Or just like in the SZIN case, we can strip the data out of the XML file into individual files for each correlation function.

The task of stripping the data has been delegated to data analysis. There is a software package called ADAT which was written to sllow this kind of thing. It contains stripping programs for many of the interesting output formats in chroma such as spectroscopy and some 3 point functions. For this exercise we will use the strip_hadspec program.

Getting and compiling ADAT

In this section we will check out the adat software package from CVS and build the strip_hadspec application. If you don't have CVS installed or want to skip this section you can go on to the next bit using the copy of adat this copy of strip_hadspec (Linux Ubuntu 6.06).

Check out the ADAT pacakge from the anonymous CVS by typing the following commands (I assume you are using the bash shell)

export CVSROOT=:pserver:anonymous@cvs.jlab.org:/group/lattice/cvsroot
export CVS_RSH=ssh
cvs login (hit return when asked for password)
cvs checkout adat

cd adat

./configure
make