---

table2frame.c

The table2frame tool

This tool is designed especially for translating e2e outputs into frames.

It assumes that you have available two files:

1. a data file in "columnar" format, namely containing data in equal length rows, each column referring to a different channel.
2. a optional description file containing a series of strings, to be used as names for the columns in the data file.

As an example, assume that the data file is named simulation.out and the description file simulation.dhx.

A conversion can be run with the following command

 table2frame -i simulation.out -description simulation.dhx
            -seconds_per_frame 2 -frames_per_file 2

The switches which can be used are the following:

• -detector <name> : give this "name" to the detector
• -i <file> : read data from "file", should be in tabular format
• -skip <int> : how many chars to skip at the beginning of the input file
• -comment <char> : which comment to assume for the input file
• -description <file> : read from "file" the channel names
• -sampling <float> : sampling frequency
• -channel <name> : set the current channel
• -type <adc|proc|sim> : the kind of data to store in current channel
• -gain <float> : the gains to apply to the current channel
• -bits <int> : the bits to encode the current channel
• -GTimeS <int> : set starting time in seconds
• -GTimeN <int> : set nanosecond portion of the starting time
• -seconds_per_frame <float> : seconds for each frame
• -frames_per_file <int> : how many frames per file
• -o <string> : output base file name, shall be completed with other informations, including the GPS time.
• -compress <int> : compression type
• -gzip <int> : gzip level
• -debug <int> : debug level

Remarks:

Compression can be

• 0 no compression
• 1 gzip
• 3 differentiation + gzip
• 5 differentiation + zero_suppression, only for short
• 6 differentiation + zero_suppression for short. gzip for others. The default is 6

Remarks:

The gzip level ranges from 0 (no compression) to 9 (maximum, slow). Default 1

Remarks:

The type can be adc, proc, sim. Default sim. Please use

• adc for raw data (adc counts)
• proc for processed data
• sim for simulated data.

Remarks:

The allowed number of bits depend on the type:

• proc or sim:
  • if bits < -32, double format is used
  • if bits >= -32 float format is used
• adc:
  • if bits > 16, long int are used
  • if bits > 8, short int
  • if bits > 0, char
  • if bits < -32, double
  • default float

Remarks:

The recognized names for the detectors (case unsensitive) are:

• TAMA_300 [Tokyo, JPN]
• VIRGO [Cascina, ITA]
• GEO_600 [Garching, GER]
• LIGO_HANFORD_2K [Hanford, WA, USA]
• LIGO_HANFORD_4K [Hanford, WA, USA]
• LIGO_LIVINGSTON [Livingston, LA, USA]
• PROTOTYPE [Caltech, 40m, Pasadena Ca]
• ALLEGRO [Baton Rouge, LA, USA]
• AURIGA [Padova, IT]
• EXPLORER [CERN, CH && FR]
• NIOBE [Perth, AU]
• NAUTILUS [Frascati, IT]

  If the detector name is recognized, some variables in the FrDetector structure are filled accordingly. These variables include the detector positions and orientations.

The error bits in the dataQuality flag of the frames are also set depending on the name of the detector: see specifications for the frame format.

If the name is not recognized the FrDetector structures are just left at default values and the dataQuality flag is not set.

Remarks:

The convention adopted for the output files where the frame are saved is the following:

ROOT-GPS_TIME-FRAMES_PER_FILE-TIME_PER_FRAME.F where

• ROOT is the root filename, provided by the user
• GPS_TIME is the starting time of the frame, in GPS seconds
• FRAMES_PER_FILE is the number of frames saved in this file
• TIME_PER_FRAME is length in seconds of each frame.

If both FRAMES_PER_FILE and TIME_PER_FRAME are 1, they are omitted

/******************************************************************************
 *
 * Copyright (C) 2001 by Andrea Vicere'.
 *
 * Permission to use, copy, modify, and distribute this software and its
 * documentation under the terms of the GNU General Public License is hereby 
 * granted. No representations are made about the suitability of this software 
 * for any purpose. It is provided "as is" without express or implied warranty.
 * See the GNU General Public License for more details.
 *
 */
#include <stdlib.h>
#include <string.h>
#include <math.h>
#include "fbe.h"
#include "table_read.h"

#define MAX_FILES 64
#define MAX_CHANNEL 256

void Help(FILE *stream, char * name)
{
  fprintf(stream,"\n"
          "This program converts one ASCII file in table format into frames\n"
          "and saves them into one or more frame files, with options on the\n"
          "size of the frames and on the frames requested per file.\n"
          "Columns should be terminated by a newline character.\n"
          "A comment char can be specified.\n"
          "-------------------------------------------------------------------- \n"
          "Usage: %s -i <filename> : input file\n"
          "          -description <filename> channel names description file\n"
          "          -o <filename> : output base file name\n"
          "          -detector <name> : name to assume for the detector\n"
          "          -compress <int> : compression type\n"
          "          -gzip <int> : gzip compression level>\n"
          "          -debug <int> : debug level\n"
          "          -seconds_per_frame <float> : seconds for each frame\n"
          "          -frames_per_file <int> : how many frames per file\n"
          "          -GTimeS <int> : starting time in seconds \n"
          "          -GTimeN <int> : starting time (nanosecond portion) \n"
          "          -channel <name> add this channel to the selected\n"
          "          -sampling <float> : sampling frequency\n"
          "          -type <string string string> : the kind of data to store\n"
          "          -gain <float> : the gains to apply to current channel\n"
          "          -bits <int> : the bits to encode the current channel\n"
          "          -skip <int> : how many chars to skip at the beginning of the input file\n"
          "          -comment <char> : which comment to assume for the input file\n"
          "-------------------------------------------------------------------- \n"
          "Remarks:\n"
          "         -i The input file may have as first column the time of the sample;\n"
          "            this is assumed, uness -sampling is explicitly given\n"
          "         -compress Compression types are 0 (no compression), 1 (gzip)\n"
          "            3 (differentiation+gzip), 5 (gzip for float+zero suppress for int)\n"
          "            6 (zero suppress for int). The default is 3\n"
          "         -gzip The gzip level ranges from 0 (no compression) to 9 (maximum, slow).\n"
          "            Default is 1\n"
          "         -description The description file is a list of channel names: if not\n"
          "            given, progressive names channel0-N are given\n"
          "         -sampling If given, the first column in the input file is ignored!\n"
          "                   It is assumed to be a integer, actually.\n"
          "         -type Can be adc, proc, sim. Default is sim.\n"
          "         -bits Allowed values depend if the type is adc, proc or sim\n"
          "               For proc and sim, if bits < -32, double format is used,\n"
          "                        otherwise float.\n"
          "               For adc, if bits > 16, long int are used\n"
          "                        if bits > 8, short int\n"
          "                        if bits > 0, char\n"
          "                        if bits < -32, double\n"
          "                        default float\n"
          "         -channel : the name shoud be in the description file or\n"
          "                        be channel0, channel1 ...\n"
          "         -skip skip some chars before start reading: default 0\n"
          "         -comment The default comment char is '#'\n",
          name);
  return;
}

struct Parameters {
  char * inputFile;
  char * outputFileBaseName;
  char * detector;
  int compress;
  int gzip_level;
  int debug;
  double seconds_per_frame;
  int frames_per_file;
  int GTimeS;
  int GTimeN;
  char * descriptionFile;
  char ** selectedChannels;
  long nChannel;
  double sampling;
  char ** type;
  double * gain;
  int * bits;
  int skip;
  char comment;
};

/* parsing of the command line and setting of the defaults */
struct Parameters * ParametersNew(int argc, char * argv[])
{
  struct Parameters * this;
  int i;
  int bitsCounter = 0;
  int gainCounter = 0;
  int typeCounter = 0;

  this = (struct Parameters*) malloc(sizeof(struct Parameters));
  
  /* default values */
  this->inputFile = NULL;
  this->outputFileBaseName = strdup("P");
  this->detector = strdup("unknown");
  this->compress = 3;
  this->gzip_level = 1;
  this->debug = 0;
  this->seconds_per_frame = 1.0; /* means ALL in a frame */
  this->frames_per_file = 1;

  this->GTimeS = -1;  /* means use the system clock */
  this->GTimeN = 0;
  this->descriptionFile = NULL;
  this->selectedChannels = 
    (char**) malloc(sizeof(char*)*MAX_CHANNEL);
  this->nChannel = 0;
  this->sampling = 0.0;
  this->type = (char**) malloc(sizeof(char*)*MAX_CHANNEL);
  this->gain = (double*) malloc(sizeof(double)*MAX_CHANNEL);
  this->bits = (int*) malloc(sizeof(int)*MAX_CHANNEL);
  for(i = 0; i < MAX_CHANNEL; i++)
    {
      this->gain[i] = 1.0;
      this->bits[i] = -32;
      this->type[i] = strdup("sim");
    }
  this->skip = 0;
  this->comment = '#';

  /* parse the command line */
  for(i=1; i<argc; i++)
    {
      if(strcmp(argv[i],"-i")==0)
        this->inputFile = strdup(argv[++i]);
      else if(strcmp(argv[i],"-o")==0)
        {
          free(this->outputFileBaseName);
          this->outputFileBaseName = strdup(argv[++i]);
        }
      else if(strcmp(argv[i],"-detector")==0)
        {
          free(this->detector);
          this->detector = strdup(argv[++i]);
        }
      else if(strcmp(argv[i],"-compress")==0)
        this->compress = atoi(argv[++i]);
      else if(strcmp(argv[i],"-debug")==0)
        this->debug = atoi(argv[++i]);
      else if(strcmp(argv[i],"-gzip")==0)
        this->gzip_level = atoi(argv[++i]);
      else if(strcmp(argv[i],"-seconds_per_frame")==0)
        this->seconds_per_frame=atof(argv[++i]);
      else if(strcmp(argv[i],"-frames_per_file")==0)
        this->frames_per_file=atoi(argv[++i]);
      else if(strcmp(argv[i],"-GTimeS")==0)
        this->GTimeS = atol(argv[++i]);
      else if(strcmp(argv[i],"-GTimeN")==0)
        this->GTimeN = atol(argv[++i]);
      else if(strcmp(argv[i],"-description")==0)
        this->descriptionFile = strdup(argv[++i]);
      else if(strcmp(argv[i],"-channel")==0)
        while(((i+1)<argc) && (argv[i+1][0] != '-'))
          {
            this->selectedChannels[this->nChannel] = strdup(argv[i+1]);
            this->nChannel++;
            i++;
          }
      else if(strcmp(argv[i],"-sampling")==0)
        this->sampling = atof(argv[++i]);
      else if(strcmp(argv[i],"-type")==0)
        {if(typeCounter < MAX_CHANNEL)
          this->type[typeCounter++] = strdup(argv[++i]);}
      else if(strcmp(argv[i],"-gain")==0)
        {if(gainCounter < MAX_CHANNEL)
          this->gain[gainCounter++] = atof(argv[++i]);}
      else if(strcmp(argv[i],"-bits")==0)
        {if(bitsCounter < MAX_CHANNEL)
          this->bits[bitsCounter++] = atof(argv[++i]);}
      else if(strcmp(argv[i],"-skip")==0)
        this->skip = atoi(argv[++i]);
      else if(strcmp(argv[i],"-comment")==0)
        this->comment = argv[++i][0];
      else if(strcmp(argv[i],"-h")==0)
        {
          Help(stdout,argv[0]);
          exit(0);
        }
      else
        {
          fprintf(stderr,"%s: unrecognized option %s\n",
                  argv[0], argv[i]);
          fprintf(stderr,"please issue %s -h | more for help\n",
                  argv[0]);
          exit(1);
        }
    }
  return this;
}

int main(int argc, char * argv[])
{
  /* declarations */
  int errorFlag = 0;
  char buffer[128];
  int i, c, c1, f;
  double sampling;
  long nFrames = 0;
  long nDataPerFrame = 0;
  long nChannel;
  double ** channel = NULL;
  struct Parameters * param;
  struct tableFile  * tableInput;
  struct FbeBase * fbeBuilder;
  struct FbeBase ** sampler;
  struct FbeBase * oFile;

  /* Read parameters, or give help if requested */
  if(argc == 1)
    {
      Help(stdout, argv[0]);
      exit(0);
    }
  
  param = ParametersNew(argc, argv);

  /* open input file and read its characteristics */
  if((tableInput = 
      tableFileNew(param->inputFile, param->descriptionFile))==NULL)
    exit(1);

  errorFlag |= tableFileIni(tableInput, param->comment, param->skip);
  if(errorFlag & FBE_FATAL)
    exit(errorFlag);

  /* --------------- allocations ---------------*/

  /* set the length of a single frame */
  if(param->seconds_per_frame < tableInput->dt)
    fbeBuilder = FbeBuilderNew(param->seconds_per_frame, NULL);
  else
    fbeBuilder = FbeBuilderNew(tableInput->dt, NULL);
  if(fbeBuilder == NULL)
    exit(1);

  /* set other characteristics of the frame */
  sprintf(buffer,"debug=%d detector=%s GTimeS=%d GTimeN=%d run=0 frame=0",
          param->debug, param->detector, param->GTimeS, 0);
  errorFlag |= FbeBuilderSet(fbeBuilder, buffer, NULL);

  if(errorFlag & FBE_FATAL)
    exit(errorFlag);

  /* space for the different channels */
  nChannel = (param->nChannel == 0 ? tableInput->nCol : param->nChannel);
  sampler = (struct FbeBase**) malloc(sizeof(struct FbeBase*)*nChannel);
  channel = (double**) malloc(sizeof(double)*nChannel);

  /* what is the sampling? */
  sampling = (param->sampling != 0.0 ? param->sampling : tableInput->sampling);

  /* which sampler samples which channel? */
  if(param->nChannel == 0)  /* the user did not select any channel */
    {
      /* did the user provide a sampling? Then the first column
         of the file contains useful data */
      if(param->sampling != 0.0)
        for(c = 0; c < nChannel; c++)
          {
            channel[c] = 
              tableFileChannel(tableInput, tableInput->columnName[c]);
            if((sampler[c] = 
                FbeSamplerNew(1.0/sampling, fbeBuilder->frameH)) == NULL)
              exit(1);
            sprintf(buffer, "debug=%d name=%s gain=%g bits=%d type=%s", 
                    param->debug, tableInput->columnName[c], 
                    param->gain[c], param->bits[c], param->type[c]);
            errorFlag |= FbeSamplerSet(sampler[c], buffer, NULL);
          }
      else /* no sampling provided: assume the first column is time */
        {
          sprintf(buffer,
                  "assuming column 0 (name \"%s\") of file \"%s\" as time\n"
                  "deducing sampling <- %g (Hz)",
                  tableInput->columnName[0], tableInput->nameInput, sampling);
          errorFlag |= fbeMessage(FBE_INFO,argv[0],buffer);
          nChannel--;
          for(c = 0; c < nChannel; c++)
            {
              channel[c] = 
                tableFileChannel(tableInput, tableInput->columnName[c+1]);
              if((sampler[c] = 
                  FbeSamplerNew(1.0/sampling, fbeBuilder->frameH))==NULL)
                exit(1);
              sprintf(buffer, "debug=%d name=%s gain=%g bits=%d type=%s", 
                      param->debug, tableInput->columnName[c+1], 
                      param->gain[c], param->bits[c], param->type[c]);
              errorFlag |= FbeSamplerSet(sampler[c], buffer, NULL);
            }
        }
    }
  else /* the user selected the wanted channels */
    for(c1 = c = 0; c < param->nChannel; c++)
      {
        channel[c1] = tableFileChannel(tableInput,param->selectedChannels[c]);
        if(channel[c1] == NULL)
          {
            fprintf(stderr,
                    "channel %s not in the description file: left out\n",
                    param->selectedChannels[c]);
            nChannel--;
          }
        else
          {
            if((sampler[c1] = 
                FbeSamplerNew(1.0/sampling, fbeBuilder->frameH))==NULL)
              exit(1);
            sprintf(buffer, "debug=%d name=%s gain=%g bits=%d type=%s", 
                    param->debug, param->selectedChannels[c1], 
                    param->gain[c], param->bits[c], param->type[c]);
            errorFlag |= FbeSamplerSet(sampler[c1], buffer, NULL);
            c1++;
          }
      }

  /* alloc the output file */
  if((oFile = FbeFilerNew(fbeBuilder->dt * param->frames_per_file,
                          fbeBuilder->frameH))==NULL)
    exit(1);

  sprintf(buffer,
          "debug=%d name=dummy fileName=%s compress=%d "
          "gzip_level=%d frames_per_file=%d",
          param->debug, param->outputFileBaseName, 
          param->compress,
          param->gzip_level, param->frames_per_file);
  errorFlag |= FbeFilerSet(oFile, buffer, NULL);

  /* any fatal errors? If no, reset the flag */
  if(errorFlag & FBE_FATAL)
    exit(errorFlag);

  errorFlag = 0;

  /* we know how many data we have to read; how many per frame? */
  if((nDataPerFrame = (long) (param->seconds_per_frame * sampling)) == 0)
    {
      sprintf(buffer,"less than 1 data point per frame\n"
              "because sampling * seconds_per_frame = %g * %g = 0\n"
              "issue %s -h for further help",
              sampling, param->seconds_per_frame, argv[0]);
      fbeMessage(FBE_FATAL,argv[0],buffer);
      exit(1);
    }

  /* how many frames do we need? */
  nFrames = 
    (long) (((double) tableInput->nRow)/((double) nDataPerFrame) + 0.5);

  if(nFrames * nDataPerFrame < tableInput->nRow)
    fbeMessage(FBE_WARNING,"table2frame","one frame shall be incomplete");
  
  /* cycle reading and filling the frames */
  for(f = 0; f < nFrames; f++)
    {
      /* create the frame */
      FbeBuilderSim(fbeBuilder, NULL);

      /* cycle and fill the channels */
      for(i = 0; i < nDataPerFrame; i++)
        {
          /* read one line of the file */
          errorFlag |= tableFileRead(tableInput);
          if( errorFlag & FBE_WARNING)
            fprintf(stderr,"warning: less columns than channels?\n");
          else if( errorFlag & FBE_FATAL)
            exit(errorFlag);

          /* use the selected channels */
          for(c = 0; c < nChannel; c++)
            {
              errorFlag |= FbeSamplerSim(sampler[c], channel[c]);
              if(errorFlag & FBE_FATAL)
                exit(errorFlag);
            }
        }

      /* close the frame and possibly write on file */
      errorFlag |= FbeFilerSim(oFile, NULL);
      if(errorFlag & FBE_FATAL)
        exit(errorFlag);
    }

  /* close the file */
  FbeFilerFree(oFile);

  /* free the Fbe memory */
  FbeBuilderFree(fbeBuilder);
  for(c=0; c<nChannel; c++)
    FbeSamplerFree(sampler[c]);

  /* free the table */
  tableFileFree(tableInput);

  /* any fatal errors? */
  if(errorFlag & FBE_FATAL)
    exit(errorFlag);
  else
    exit(0); /* don't bother for warnings */
}

---