Single file format

Single binary QDP files are composed of a series of one or more application records. A single application record encodes a single QDP field or an array of QDP fields of the same data type. Physics metadata, managed at the convenience of the applications programmer, is associated with the file itself and with each application record as well. Above the API the QDP file is viewed as follows:

• File physics metadata
• Record 1 physics metadata and data
• Record 2 physics metadata and data
• etc.

For example, a file might record a series of staggered fermion eigenvectors for a gauge field configuration. Each record would map to a single field of type LatticeColorVector. The file metadata might include information about the gauge field configuration and the record metadata might encode the eigenvalue and an index for the eigenvector.

For another example, the gauge field configuration in four dimensions is represented in QDP as an array of four color matrix fields. The configuration is conventionally written so that the four color matrices associated with each site appear together. A file containing a single gauge field configuration would then consist of a single record containing the array of four color matrices.

The API permits mixing records of different datatypes in the same file. While this practice may be convenient for managing projects, it may be forbidden for archival files.

Additional metadata is automatically managed by QIO (without requiring intervention by the applications programmer) to facilitate the implementation and to check data integrity. Thus the file actually begins with QIO metadata and physics metadata and each application record consists of five logical records. Within QIO the file is viewed as a series of logical records as follows:

• Private file QIO metadata
• User file physics metadata
• Record 1 private QIO metadata
• Record 1 user physics metadata
• Record 1 binary data
• Record 1 private checksum
• Record 2 private QIO metadata
• Record 2 user physics metadata
• Record 2 binary data
• Record 2 private checksum
• etc.

The site order of the binary data is lexicographic according to the site coordinate $r_i$ with the first coordinate $r_0$ varying most rapidly.

A new format called LIME (Lattice-QCD Interchange Message Encapsulation) is used for packaging the logical records. A feature of this format is the maximum record size is quite - $2^{64}-1$ bytes. However, file system limitations may require splitting a single file into multiple physical files. No provision is provided for such splitting. It is expected that in time file systems will evolve to allow much larger file sizes. In the interim, facilities like cpio and tar can be used for file splitting.