Back to the Basement Computing Page
 

Documentation
 

Index
 
ASCII Table
AudMat
AudSDK
Basement Blog
BC - Behaving Cat
COM Data Collection
COMA Data Analysis
Correcting GW for phone
CSF format
DAFLIB
Data Schemas
DCP - Introduction
DCP - Tech Notes
DCP - Users Guide
DCPLIB
DSS-III Notes
DSS-III Prototype
DSSLIB
DSS3LIB
DSS4LIB
DSS4FLIB
Effective SPL computation
EP - Evoked Potential
Error Codes
Falling Cost of Computing
FF - Free Field
Fortran and C/C++ mix
GWE - Waveform Editor
Local FAQs
Mathematical Formulae
Minicomps. & Neurophys.
MISLIB
Numerical Constants
NUMLIB
PIN - Edge/Notch
PIN - Summary
Probe Tube Correction
RA - Introduction
RA - Tech Notes
RA - Users Guide
RAP - Users Guide
Rayleigh Test
SER - Search Stimulus
SERCOM - COM Search
Signal Analysis Review
Simulating Free Space
STATPK - Introduction
STATPK - Tech Notes
STATPK - Users Guide
Status Table Formats
SysInfo
 
Disclaimer
 
Status Table Formats
Quick Links
Introduction
Type-1 Status Table
Type-2 Status Table
Type-3 Status Table
Routines for Type-3 Status Table storage
Acknowledgements
References
Feedback

Ravi Kochhar
Dept. of Neuroscience
University of Wisconsin - Madison
 
Nov. 21, 1999
rev. 1.009, Jan 14, 2004
Programming Note no. 18

 

Introduction

The Status Table (STATTB) is a data structure that is an integral part of most data formats used to store Auditory Neurophysiology Data at the University of Wisconsin. An overview of the data file storage format is available in ref. 1

As described there, data are stored as data sets within a data file on disk. A file many contain many data sets, and each data set may contain many "stimulus points" or "trials". The Status Table maintains pointers to the location of data for each stimulus point within a data set. It may also contain other information such as stimulus parameters. The Status Table is usually stored near the end of each data set, though this is not a requirement. At present, all pointers in a Status Table are stored as 32-bit integers, and they are relative to the start of the data set (not data file), and they are in units of 32-bit words. Thus, a Status Table pointer of 1 means the very first word in that data set, a pointer of 129 means the 129'th (32-bit) word, which really means the word starting at byte number 513 in that data set.

The function of the Status Table is to facilitate and speed access for data retrieval. In a typical analysis operation, the analysis program requests data for a specified stimulus point - the retrieval routine first reads and scans the Status Table (STATTB) for the exact location of the requested data withing the file, and then reads and returns the data to the analysis program. Since data sets can be quite large (several megabytes), this can result in considerable speed-up of data retrieval.

The STATTB data structure is created and stored by the relevant data collection program (see ref. 2 or 3). In some limited cases the STATTB can also be re-created by the EDITDF program (ref. 4). This is done most often where the data storage process was prematurely aborted, or in some cases to repair a damaged data file.

Three separate Status Table formats are supported at present. They are referred to as "Type-1", "Type-2" and "Type-3" Status Tables. Any data set may contain only one "type" of Status Table, though other data sets within the same data file may contain other "types". A variable (STFORM) in the header of each data set contains the STATTB-type for that data set. If STFORM is missing from the data set header then a Type-2 table should be assumed.

The following sections contain details about the structure of each of these three Status Table types.

Back to Top

Type-1 Status Table

To be filled in later.

Back to Top

Type-2 Status Table

Type-2 Status Tables are the simplest of all the Status Tables. This format was originally designed for cases where the stimulus variables (e.g. Frequency and SPL) are varied between two limits (high to low, or low to high) in equal linear increments. This made it possible to store a Status Table as a simple two-dimensional rectangular table of pointers (thus giving rise to the term "Status Table" itself). There was no need to store the stimulus variable values themselves, since the increments were linear and known.

Even today, a large portion of data collected uses this simplest form of Status Table (Type-2), but it is important to know that Type-2 tables also cover the more general cases of (a) Number of variables other than 2 (e.g. 1 or 3 or more) and (b) Log increments as well as linear. It is also possible for any or all variables to vary between limits in three different ways - (a) low to high (b) low to high and (c) random. All these cases are included and possible with Type-2 Status tables.

In a Type-2 Status Table, the values of stimulus variables are not stored as part of the table, only address pointers are stored. There can be more than one address pointer at each stimulus point. This is the case (for example) when both spike time data and digitized analog data are stored. (Note: For Harris data files, the analog data pointers were stored as a separate "Analog Status Table").

The following table summarizes the format of Type-2 Status Tables:

P11 P21 P31 ............................ Pm1
P12 P22 P32 ............................ Pm2
... ... ... ............................ ...
... ... ... ............................ ...
... ... ... ............................ ...
... ... ... ............................ ...
P1n-1 P2n-1 P3n-1 ............................ Pmn-1
P1n P2n P3n ............................ Pmn
Table - 1

where:

       Pji = the j'th pointer for the i'th stimulus point
       n = Number of stimulus points
       m = Number of pointers per stimulus point (typically 1 or 2)

As for all Status Tables, missing data is flagged by negative address pointers.

The Type-2 format imposes no restrictions on the number of stimulus parameters, but the range and increment (or number of steps per octave) must be separately stored in the data set header. The order in which (pointers for) stimulus points are stored in a Type-2 table is such that the last variable varies fastest, and the first variable varies the slowest. For example, if the X-variable is Frequency, Y-variable is SPL, and Z-variable is Duration, then the Status Table pointers are ordered such that Duration varies first, then the SPL, and Frequency the last.

The following table illustrates the order in which pointers are stored for a typical case. Here we assume that we have two stimulus variables, FREQ and SPL, and that FREQ varies from 1000 to 2000 Hz in linear steps of 200 Hz, and the SPL varies from 10 to 40 dB in steps of 10 dB. Further assume that there is only one pointer per stimulus point (i.e. we only collected spike time data). The order of storage of pointer in this case would be as follows:

Location Stimulus Point
1 Spon
2 FREQ=1000, SPL=10
3 FREQ=1000, SPL=20
4 FREQ=1000, SPL=30
5 FREQ=1000, SPL=40
6 Spon
7 FREQ=1200, SPL=10
8 FREQ=1200, SPL=20
9 FREQ=1200, SPL=30
10 FREQ=1200, SPL=40
11 Spon
12 FREQ=1400, SPL=10
.. ......
.. ......
29 FREQ=2000, SPL=30
30 FREQ=2000, SPL=40
Table - 2

The order in which the Status Table is stored (for this example) will be as shown above, for the case where the stimulus variables were varied low-to-high, in random or non-random order. Exception: if the stimulus variables were varied high-to-low then the order of storage will be reversed from the above example. The burden of sorting the Status Table is left to the data collection program, which simplifies data retrieval.

One new item introduced in the example above is the "Spon" stimulus point. "Spon" means "Spontaneous Activity", and as noted above, it is stored once in Type-2 table before each change in the X-variable (FREQ in the above example). The purpose of spontaneous activity recordings is to record the resting or baseline activity of units, and it is recorded with no stimulus being presented periodically during data collection. If Spon Activity was not recorded then the "Spon" pointer in the Status Table is set to a negative or zero value, but the space for Spon is always reserved in each Type-2 Status table, whether it was recorded for a particular data set or not. (In some data schemas, there is a variable named SDATA which is 1 or 0 to indicate whether Spontaneous Activity was recorded or not. It's value has no impact on Type-2 Status Tables, there is always space reserved in each Type-2 table for Spon activity, and the only way to know whether it was recorded is to check the value of the Spon pointers in the table.)

The above example shows a case where two stimulus variables were varied in linear steps. In general a Type-2 table may contain entries for stimulus variables being varied in log steps. It is also possible to vary one variable in log steps, and the other in linear steps (for example).

To properly decipher a Type-2 Status Table, one must extract and use the following values from the data set header:

  • STFORM - must be 2 (if not present, then assume 2).
  • NUMPT - number of pointers (per stim point) in Status Table (normally 1 or 2).
  • UDATA,ADATA,CDATA - Determines what data type was collected. Normally only UDATA=1 and the rest are zero, which is the case where only spike time data were collected.
  • LSTAT - Location of the Status Table within data set.
  • NUMV - Number of Stimulus Variables (typically 1,2 or 3)
  • XVAR - Information about Range, Step Size, Order and whether increments were Log are Linear, for the 1st (or X) variable.
  • YVAR - Same for the 2nd (or Y) variable.
  • ZVAR - Same for the 3rd (or Z) variable.

The values of NUMV, XVAR (and YVAR and ZVAR if NUMV > 1) can be used to calculate the number of times the X-,Y- and Z-variables were varied. Suppose these values are Xnum, Ynum and Znum. The size of the Type-2 Status Table can then be computed as follows:

Size of Status Table = NUMPT * Xnum * ((Ynum * Znum) + 1)

Note the "+1" to account for the space needed by the Spon Activity entries.

Back to Top

Type-3 Status Table

This is an extension of the Type-1 Status Table (STATTB). It is the most flexible of all three types. Like Type-1 tables, it allows stimulus-points to be recorded in any arbitary order. In addition, it allows each STATTB entry to be of different length, and to contain an arbitrarily different number and types of stimulus variables.

For example, the first STATTB entry might contain two variables, e.g. FREQ and SPL, along with their values, and the second entry may contain three different variables, e.g. DELAY, PHASE and DMOD, along with their values. This is achieved by adding two things to each STATTB (type-3) entry:

  • A param that contains the length of each STATTB entry, and
  • The name and type of each variable is stored along with their values.

Thus, in our example, we would store not only the values of FREQ and SPL, but also their names (FREQ, SPL) and their type (e.g. floating point).

Each variable within the STATUS table can be one of four types: (1) Integer, (2) Floating point, (3) Character string or (4) Repeating group. A Repeating Group (RG) is a collection of other data types, and RG's may be nested.

The basic structure of each entry of a type-3 status table can be represented as follows:

      01 NVSTAT           /* Number of variables in this entry */
      01 STVARS TYPE RG OCCURS NVSTAT TIMES
         02 STVNAM TYPE STRING 8     /* Name of variable */
         02 STVTYP TYPE INTEGER*2    /* Variable Type (1,2,3,4) */
         02 STVLEN TYPE INTEGER*2    /* Value Length (in 32-bit words) */
         02 STVVAL LENGTH STVLEN     /* Value for this variable */
      01 ADDRPT TYPE INTEGER OCCURS NUMPT TIMES   /* Pointers to data */

The adddress pointers (ADDRPT) are stored as 32-bit integers, and are always relative to the start of the data set (with the first word in the data set having an address=1), and not to the start of the data file. The address pointers are in units of 32-bit words. Units of data smaller than 32-bits cannot be addressed directly. The number of pointers (NUMPT) must be stored as part of the data set header, and cannot vary within one Status Table. It is typically 2 to represent pointers for UET and A/D data.

The Varibale names (STVNAM) are limited to 8 characters or less. If less than 8 chars. are used then the name must be padded with blanks on the right to make it exactly 8 chars. for storage.

The convention for "Variable Type" is as follows:

      STVTYP      Description
      ------      -----------
         1         32-bit Integer
         2         Floating Point
         3         Character string
         4         Repeating Group
         5         Vector String
         6         Vector Repeating Group

To study an example, let us create a STATTB entry with just two variables, FREQ and SPL, and two pointers (perhaps for UET and A/D data). Let the FREQ be 1050 Hz, and the SPL be 44 dB, and the two pointers be 12304 and 12655. Then the STATTB can be created schematically as follows:

2 FREQ 2 1 1050.0 SPL 2 1 44.0 12304 12655

In the next example, let there be four variables, NACH (an integer), SRATE (floating-pt.), PREVID (char. string), and STIMPARM (a Repeating Group), plus two pointers. Further let's assume that the STIMPARM RG has two variables within it, FREQ and SPL. The STATTB entry for this case can be represented as follows:

4 NACH 1 1 2 SRATE 2 1 1000.0 PREVID 3 3 1-275B STIMPARM 4 9 2 FREQ 2 1 1050.0 SPL 2 1 44.0 12304 12655

This example is more interesting, and we can note the following points from it: The value of NVSTAT is just 4, because the RG counts as only one variable even though it has two variables within it. The length of the STIMPARM RG is 9 (32-bit words) and is computed as follows: 1 word for the No. of vars. in RG + 2 words for the name of the first var (FREQ) + 1 word for type/len of that var + 1 word for its value (1050) + 2 words for the name of the second var (SPL) + 1 word for its type/len + 1 word for it's value (44) yields a total of 9. Note that this length does _not_ include the name of the RG itself, nor the type/len entry for the RG.

Another point of interest is that the character string (PREVID) occupies a multiple of 32-bit words (i.e. 4 chars). In our case the length of PREVID is 3 words = 3x4 = 12 chars. To store character strings that are not multiples of 4 bytes, you must pad them on the right with blanks.

A single STATTB entry may contain any number of variables, including repeating groups, in any order.

Of course, the actual data will be a binary representation, following the rules outlined above. It is also useful to note that the binary representation will (or can) be system dependent. For example, the representation of floating point data is different in VMS and MS-Windows.

Back to Top

Routines for Data Collection with a Type-3 Status Table

A number of routines are available (in DCPLIB) to ease the task of constructing and storing type-3 Status Tables. An overview is as follows:

A Status Table is normally at the end of a data set. This means that the Status Table cannot be merged into the data set until all data collection (for that data set) is completed, since data sets are of variable length. Status Tables are stored in disk files while under construction. At the end of data collection (for a data set) the Status Table is merged with the data set, and the Status Table File (STBF) is deleted.

(1) SUBROUTINE STBINI(NAMSTB,LUN,NTYP,NBLK,NV,NP,EDF,DSID,NY,NER)

This routine is called once for each data set, normally early in the data collection process. It creates the STBF on disk, and initializes it. See Ref. no. 5 for a description of the parameters. Note that the value of NTYP should be 3 for type-3 status tables.

(2) SUBROUTINE STBEN3A(NVSTAT,IVARN,IVNAM,IVTYP,IVLEN,IVVAL,NER)

This makes a sub-entry (i.e. the entry for one variable) in a type-3 Status Table. Normally this routine is called repeatedly (NVSTAT times) for each variable entry within the type-3 status table. Note that NVSTAT can be set to zero if it's value is not known initially. However, at least one of the calls to STBEN3A must contain a non-zero positive value for NVSTAT. If IVTYP=4, (i.e. a Repeating Group) then this must be followed by calls to STBEN3R to store each variable within the RG, and finally a call to STBEN3R_END to end the repeating group. If IVTYP=4, then the value of IVLEN should be set to 1. After all variables have been stored, the routine STBEN3B should be called to complete the STATTB entry.

(3) SUBROUTINE STBEN3R(IVARN,IVNAM,IVTYP,IVLEN,IVVAL,NER)

This makes a sub-entry (i.e. the entry for one variable) in the repeating group (RG) that is currently opened (by a call to STBEN3A) The parameters are as follows:

     IVARN   : Current variable number (1,2,3,... upto NVRG)
     IVNAM   : Variable name, upto 8 chars (Int. array)
     IVTYP   : Variable type (1=integer, 2= Flt. pt. etc.)
     IVLEN   : Length of variable value (in 32-bit words)
     IVVAL   : Variable value (of length IVLEN words)
     NER     : Error code, normally zero.

IVARN, IVNAM, IVTYP, IVLEN and IVVAL must be supplied, while NER will be returned. The parameters of this routine are similar to STBEN3A, but there is one fewer (NVSTAT is not needed).

(4) SUBROUTINE STBEN3R_END(NVRG,NER)

This completes the currently open RG variable entry. NVRG must be supplied, and contain the number of variables within the RG.

(5) SUBROUTINE STBEN3B(NPT,NP,NER)

This routine is called once for each entry of a type-3 status table. It completes the entry and stores the pointer values. It is normally called once after several calls to STBEN3A.

(6) SUBROUTINE STBTR(NAMSTB,LUN,LUNEDF,LOC,NSZW,NER)

This routine is normally called near the end of data collection for a data set. It merges the Status Table with the current data set, and deletes the STBF.

Examples:

The following summary shows the order in which the routines would be called to create a type-3 status table. The first example shows the simpler case (example 1) discussed above, with just two variables in the STATTB entry:

        call STBINI(...)
          :
        call STBEN3A(...)
        call STBEN3A(...)
        call STBEN3B(...)
          :
          more calls to STBEN3A followed by STBEN3B
          :
        call STBTR(...)

The next example is for the second case above, where we also store a Repeating Group containing the Stimulus Parameters:

        call STBINI(...)
          :
        call STBEN3A(...)       (stores the variable NACH)
        call STBEN3A(...)       (stores the variable SRATE)
        call STBEN3A(...)       (stores the variable PREVID)
        call STBEN3A(...)       (starts the STIMPARM repeating group)
        call STBEN3R(...)       (params within the RG)
        call STBEN3R(...)           "
        call STBEN3R_END(...)   (complete the RG)
        call STBEN3B(...)       (complete the entry)
          :
          more STATTB entries, if any
          :
        call STBTR(...)

Back to Top

Acknowledgments

This work was done under the direction of Dr. W.S. Rhode. Supported by a grant from NIH.

References

(1) "DAFLIB - Data File Structure and Storage and Retrieval Routines", R. Kochhar, Technical Report # 12, Dept. of Physiology, June 1993.

(2) "RA - Response Area Data Collection", R. Kochhar, Technical Report # 3, Dept. of Physiology, Oct. 1996.

(3) "COM - Cat Oculomotor Program" Jane Sekulski, Dept. of Physiology.

(4) "EDITDF - Data File Editor", R.Kochhar, Dept. of Physiology, Univ. of Wisconsin.

(5) "DCPLIB - Data Collection and Storage Routines", R. Kochhar, Technical Report # 13, Dept. of Physiology, April 1991.

(6) "Unit Data Storage - Part 1", R.Kochhar, Conversion to the PDP-11, Report no. 10, Sept., 1982.

(7) "Unit Data Storage - Part 2", R.Kochhar, Conversion to the PDP-11, Report no. 15, July, 1983.

Back to Top

Feedback

Please send feedback/suggestions/questions/complaints to the author via email, at rkochhar@wisc.edu

Back to Top

(This page last modified on May 20, 2015)

 
Back to Top
Back to The Basement Computing Page
Back to The Basement

Please send questions, complaints or comments to rkochhar@wisc.edu