History of Changes for LIGOtools Package: dataflow

Version 1.3

Version 1.4

• New "putMeta" utility.
• Fixed bug in getFrames which prevented proper parsing of a comma-separated channel list.
• Various minor bug fixes and robustness improvements.

Version 2.0

• Updated data flow manager to work with the new syntax of messages from the LDAS manager, as of LDAS release 0.0.14. Not backward compatible with earlier versions of LDAS!
• Reorganized the dfm client C library code into a "socket" layer (general- purpose utility routines, not really related to the dfm) and a "dfm client" layer.
• Incorporated Philip Charlton's "metaio" library for parsing LIGO_LW XML files containing tables from the LDAS metadata database.
• Created some new utilities based on the metaio library: lwtscan, lwtprint, lwtdiff.

Version 2.1

• Modified dfm to handle putMeta result message syntax for LDAS 0.0.15.
• In getFrames, putMeta, and ldaspw, replaced [exec hostname] and [exec logname] with references to built-in Tcl variables. This fixes a problem that was encountered on some Linux systems.
• Improved geturl to be able to send output to stdout, and to use the "geturlWhole" routine only if the "-w" flag is specified on the command line. Modified getFrames and getMeta to call geturl with the -w flag (except when downloading a long list of frame files).

Version 2.2

• Modified getMeta, putMeta, and dfm to support multiple databases at each LDAS installation.
• Modified metaio.c to handle empty stream, and to stop dumping core when there is an error opening an input file.
• Modified lwtprint to add '-u float' option.
• Modified dfm to work with LARS even through a firewall.
• New utility 'httpTextDisplay', which is basically a simple web browser that only displays text. Intended to display status info from a web page.
• Added "-g" option to getFrames, to pop up a job status window (using httpTextDisplay).
• Made some more modifications to dfm needed to communicate the job information back to getFrames, so it can pop up a job status window.
• New "readMeta" MEX-file to read LIGO metadata table into a Matlab structure. Also made readMeta.m file to allow one to get detailed usage information by typing "help readMeta" from within Matlab.
• Moved "enum Type" from metaio.c to metaio.h, and prepended "METAIO_" to all the enum elements.
• Modified dfm to truncate list of monitoring messages if it grows too large.
• Modified dfm to handle the (rare) situation of being told by LDAS that a job has finished before being told that it started.
• Modified dfm to use ldas.ligo-wa as authentication agent.
• Renamed geturl to getUrl to avoid conflict with an existing utility. Modified getFrames and getMeta accordingly.
• Modified dfm to retry getting list of servers, and to return an informative error message if it fails.
• Modified getFrames, getMeta and putMeta to allow the user to omit the space between a switch and its argument, if desired.
• Modified getFrames and getMeta to print a more informative error message if there is an error while retrieving a URL.
• In dfm, change client sockets (including "reply" and "monitor") to be non-blocking.

Version 2.2.1

• Fixed bug in dfm which could cause an infinite loop if certain errors occurred.
• Added 'SeqInsert' utility, for sequential insertion of metadata into LDAS database.
• New 'dfmstart' script to allow programs (getFrames, etc.) to start up a properly "daemonized" dfm process.

Version 2.2.2

• Added '-j' option to getMeta, to allow user to restrict query to the metadata (e.g. event candidates) produced by the wrapperAPI during a specific LDAS job.
• Changed authentication site (when using getFrames) from LHO to LLO.

Version 3.0

• New command-line utility 'tconvert' to convert to/from GPS time.
• New source files TConvert.h and TConvert.c, to create routines TConvertSysToGPS and TConvertGPSToSys for libdataflow.a.
• dfm can now web-patch itself from any of five different LDAS web servers. If all are unreachable, it will continue, using the cached patch file from disk.
• dfm can now check the user's password using any of three LDAS sites.
• dfm determines its own IP address from a client connection (if at all possible), to ensure that it is on the "external" network interface.
• dfm now opens the "client", monitor, and http listening sockets on the "localhost" network interface, rather than the Internet. Also modified dfmon, dfmpipe, dfmcheck, dfmkill, SeqInsert, getFrames, getMeta, putMeta, and DfmClient.c accordingly.
• Modified getFrames and dfm to allow a LARS request to specify the compression code to be used with FrCopy.
• URL of server-info file has changed (now servers.txt). Can find it on any of the LDAS web servers.
• New dfm agent type: 'HttpAgent', to handle UDNs which map to URLs.
• dfm now acts as a web server! (On localhost:11894) Among other things, it lets user find UDNs of a given type and/or matching a given glob-style pattern.
• dfm now checks that the time range in a LARS request has tstop > tstart.
• Various internal changes in dfm.
• A few internal modifications to getUrl.
• Modified metaio.h, metaio.c, and readMeta.c to handle INT_8S datatype.
• Bug fix to allow getFrames to retrieve multiple frame files.
• New file "dfm_protocol.txt" in the doc directory.

Version 3.0.1

• Bug fix in tconvert to work with systems which don't recognize time zone specifications of the form "US/Pacific". (On such systems, if an explicit U.S. standard or daylight-savings time zone was specified, tconvert would always claim that the time did not occur during that time zone.)
• Fixed getUrl to time out only if the data transfer remains stalled for more than one minute. Previously, it would time out one hour after the transfer started, even if the transfer was actively in progress.

Version 4.0

• Modified dfm, and all utilities that use it, to start a separate copy of dfm for each Unix username. This means that port numbers are no longer fixed, and have to be told to the client. This is done using a "register file" in the user's home directory, with a name of the form .dfmRegister.<hostname> .
• Modified "dfmcheck -n" to return the dfm's client port number (or 0 if the dfm is not running).
• Modified dfmstart to start up a copy of dfm ONLY if one is not already running for this user, and to block until the dfm has successfully started up and is ready for user input (or until 10 seconds have elapsed, in which case dfmstart exits with status=1).
• Added a new "agent" routine to dfm: LDASJobAgent. UDNs for this have the form: //ligo/ldas/lho .
• Modified dfm to allow the client to specify a pseudo-UDN of the form <agent>:<host>:<port>, e.g. "LDASMeta:myhost.caltech.edu:1000".
• Modified getMeta, putMeta, and SeqInsert to allow you to communicate with a "private" LDAS system. Instead of specifying a shorthand name ('lho', 'dev', 'cit', etc.) or a UDN ('//ligo/meta/lho', etc.), you can now specify an explicit server (hostname and port number), e.g. 'myhost:10001' or 'myhost.caltech.edu:10001'. (This uses the pseudo-UDN scheme mentioned above.)
• Modified tconvert to use the user's home directory to store the file of leap seconds, in case the LIGOTOOLS environment variable is undefined and the location of the config/public directory can't be determined from the script's location.
• Modified tconvert so that it will run under tclsh if tclshexe is not in the user's PATH. Thus, it can be used outside the LIGOtools environment.
• In dfm, increased timeout period for connecting to LDAS manager.
• New utility 'lwtselect' to select a single table out of a LIGO_LW file containing multiple tables.
• Modified the implementation of the '-j' option in getMeta to make use of the jobid value stored in the process table, instead of the one in the process_params table (which made for more complex queries)
• Modified getUrl so that if you specify a directory for the destination, it automatically determines the filename from the url.
• Updated dfm to use "!host!" in the LDAS email address for the dev system, which auto-detects the IP address of the client.
• Updated dfm to make use of the LDAS job proxy server if its IP address is in one of the ranges reserved for private networks, AND if the LDAS system it wants to talk to does not understand "!host!".
• Added some self-test code to tconvert.
• Updated dfm to be able to interpret the new format returned by the LDAS getFrameCache command.
• Modified getFrames to allow it to handle multiple output files, downloading each one as it arrives rather than all at once. (However, the dfm and LARS currently send all result URLs at the end of the whole job.)

Version 4.0mex (Linux only)

I am now able to compile Matlab MEX-files for Linux, so this is a re-release for Linux which includes the readMeta MEX-file. On Solaris, readMeta is already included in version 4.0.

Version 4.1

• Changes to communication protocol with LDAS.
• Modified SeqInsert so that you can specify a database when submitting a file to be inserted; this overrides the value given on the "SeqInsert server ..." command line. Also added "hold" and "purge" commands to SeqInsert.
• Changed behavior of getFrames in case it downloads a file with the same name as an existing file. Previously, it would tack a number on the end to avoid overwriting the existing file; now, it overwrites the existing file.
• Various bug fixes.

Version 4.2

• Modifications to SeqInsert to make it work better during the E7 run. Has been made rather "fault-tolerant", automatically retrying a few times when a database insertion job fails. This makes it much less susceptible to LDAS glitches.
• Increased various timeout intervals for communicating with LDAS, to try to minimize timing-related failures.
• New utility putSpectrum to read a spectrum from an ASCII file and insert it into the summ_spectrum table in the LDAS database.

Version 4.3

• Modifications to be able to read E7 data from the LDAS systems at the observatory sites using getFrames. getFrames now has a '-1' option which forces the output to be written as 1-second-long frame files in a subdirectory, which is the form required to be able to look at the data with DTT or Data Viewer. (Currently only available on Solaris.)
• New utility gpsclock to pop up a window to display the current GPS and UTC times, synchronized (with ~1 second precision) to the CDS framebuilder at one of the observatory sites if possible.
• Extracted the core time conversion routines from the tconvert command-line utility into a Tcl library package, which one can access from a Tcl script using "package require tconvert". This creates a Tcl procedure called tconvert which behaves the same as the command-line utility. In fact, the command-line tconvert utility and gpsclock now load the tconvert library.
• Upgraded dfmon with new command-line options to allow it to write to standard output or to a file, in addition to the original graphical display.

Version 4.4

• Enhanced getMeta utility with a new option '-a' to get ALL rows matching a query, even if that is more than the LDAS limit on the number of rows returned from a single LDAS job (currently 10000). There is also a new '-v' option which allows you to track the progress of getMeta.
• New utility concatMeta to concatenate LIGO_LW files containing rows from the same database table.
• Minor bug fixes in getFrames and metaio.c.

Version 4.5

• Re-instated code in dfm to use the LDAS job proxy server if the local IP address is part of the address space reserved for private networks.
• Fixed bug in dfm that could lead to an infinite loop when a job timed out. This seems to have caused a peculiar situation in which dfmkill claimed to have shut down the dfm, but the dfm did not actually shut down.
• Modified dfm to use hard-coded numeric IP addresses for LDAS and other servers that it must communicate with; this avoids problems if dynamic name resolution is slow for some reason.
• Made some additional changes to dfm to avoid long waits in certain circumstances.
• Fixed a bug in getFrames: previously, if a request returned multiple output files AND the user specified an explicit output filename using the "-o" option, then the output files were written on top of each other. Modified code to assign slightly different names in this case.
• Modified getMeta to add the '-t' option, allowing the user to restrict a query to entries generated by dataPipeline jobs with a given "userTag".
• Modified lwtprint, lwtdiff, and lwtscan so that option values can follow option flags without an intervening space (e.g. "-csearch" is now permitted as well as "-c search", and "-r1-10" is now permitted as well as "-r 1-10").
• Modified lwtprint, lwtdiff, lwtscan, and the readMeta MEX-file to exit promptly after printing/reading a limited number of rows from a large file.
• Fixed bugs in gpsclock: Time Converter didn't display times properly when converting from a GPS time; optional "sec" in relative time offset didn't work correctly in combination with the synchronization time offset.
• Various other minor improvements and bug fixes.

Version 4.6

• Code changes to be compatible with latest version of LDAS, in which some things have changed:
  • New frame cache index format
  • Get frame cache info via http rather than running an LDAS job
  • Use "complex" framequery syntax for raw data requests
  • Change in getMetaData user command syntax (option '-returnformat' has been changed to '-outputformat')
  • Handle nonstandard error message seen in some cases
• Added code to dfm to use an http proxy server and/or an ljproxy server, based on the settings of the environment variables HTTPPROXY, HTTPPROXYBYPASS, and LJPROXY. For usage details, contact Peter Shawhan.
• Fixed various bugs:
  • Could not use getFrames via the ljproxy server, because of a problem using LDAS to verify the password
  • Occasional corrupted queries when using 'getMeta -a' on Linux machines.
  • "File already exists" error when using 'getMeta -a' to overwrite an existing file; now, the file is overwritten silently
  • On Linux machines, lwtscan and lwtprint ignored the column exclusion list
• Added workaround for PSU LDAS system.

Version 4.7

• Various changes to dfm:
  • Added code to implement the MD5-based password protection protocol.
  • Added code so that LARS jobs will use the ljproxy server if the LJPROXY environment variable is set.
  • Accept an absolute URL as the name of an output file, as reported by LDAS when the job finishes.
  • Fixed some bugs in handling of error messages.

• Added 'error_exec' command to SeqInsert. This allows you to register a shell command to be executed when SeqInsert requires human intervention. You execute this command after you start the SeqInsert server process. For example, the following is a typical sequence to start up the server process:

    SeqInsert purge    # Otherwise SeqInsert will process any leftover queued files
    SeqInsert server dev:ldas_tst >& SeqInsert.log &
    SeqInsert error_exec "echo SeqInsert error on sand | mail shawhan_p@ligo.caltech.edu"
  

Version 4.8

• Made substantial additions to metaio.c and metaio.h to write output LIGO_LW files, by copying selected rows from input LIGO_LW files. (Code does not yet support constructing arbitrary tables for output.)
• New utility makeSegmentXML.
• Various modifications to communication protocol between dfm and LDAS (already in use prior to this release, via web-patching)
• Fixed bug in _getMetaLoopHelper which caused an error if the input file was empty, or possibly if it had just a few rows.
• Modified ldaspw utility for use by the new version of dtt.

Version 4.9

• Fixed lwtselect so that it can select based on table name, even if the table name field in the file ends in ":table" rather than the actual table name. (Fixed independently by Rauha)
• Fixed error handling in tconvert.tcl (for case in which tcleaps.txt is not writable), and improved error messages.
• Improved some error messages in tconvert command-line utility.
• Modified dfm to retrieve data from LDAS using createRDS instead of getFrameData.
• Modified dfm to automatically retry an LDAS job when it encounters certain "transient" job submission errors.
• Fixed code which converts an ftp URL into an http URL: made it less sensitive to the choice of ftp base directory. (Works if base directory is /, /ldas_outgoing, or /ldas_outgoing/jobs).
• Fixed a bug in gpsclock which showed up when LHO framebuilder was down.
• Modified dfm to time out if attempt to read from an http URL stalls for 30 seconds.
• Modified dfm to preferentially choose a local server if multiple servers can handle the request.
• Modified dfm to handle new LDAS message syntax for listing output files.
• Modified dfm to determine success/failure of an LDAS job by looking for the word "error" to appear ANYWHERE in the subject line.
• New utility FrameCacheQuery.
• Modified getFrames to change 'RDS_R_L1' to just 'RDS' in the names of files it downloads. Also fixed a minor bug.

Version 4.10

• Added code to dfm to automatically use LDAS job proxy server when dfm is running on an LLO machine and connecting to a machine not at LLO.
• Fix bug which caused dfm to hang when one requested a list of UDNs (reported by Mike Landry).
• Modified SeqInsert to be more robust if the dfm does not start promptly: now it will wait up to a minute after dfmstart returns with error status.
• Modified dfmstart to change timeout from 10 sec to 30 sec.
• Changed dfm idle shutdown time from 5 minutes to 1 hour.
• In SeqInsert, changed the regexp to detect "duplicate row" error since the error message from DB2 has changed.
• Fixed bugs in readMeta which prevented you from specifying particular rows/columns.
• Modified metaio.c and metaio.h to add a new function, MetaioOpenTable(), which lets the caller specify the name of a particular table to be read from a multi-table file. Modified readMeta, lwtscan, lwtprint, and lwtdiff to make use of this new functionality when the user specifies optional arguments.
• Modified readMeta MEX-file so that spectra with known mimetypes (in the summ_spectrum table) are automatically unpacked from BLOBs into numeric arrays. Modified readMeta.m too to note the new behavior.
• Improved usage message for FrameCacheQuery.
• Modified FrameCacheQuery to get frame cache information via an LDAS user command, since it is no longer available directly via http.
• In dfm, revised "local preference" logic to work for server IP addresses expressed in EITHER string OR numeric form.
• Updated default list of web-patch hosts to include ldas-cit.
• Put in code to check the WEBPATCH_HOST environment variable. This allows a user to specify a specific web-patch host to use instead of the default list (i.e. because it is local), or to skip web-patching altogether by setting it to "none".
• Increased EchoMyIP time from 3 sec per web-patch host to a variable number, depending on how many hosts there are in the list.

Version 4.11

• New version of the 'metaio' parsing code from Philip Charlton, to avoid using variable names reserved by some compilers.
• Modified concatMeta to handle input file(s) containing no events.
• New utilities FrameCacheDump and FrameCacheRanges.
• Modified Makefile to use -static-libgcc flag, and added a mexopts.sh to do this when building MEX-files (which are essentially shared-object libraries). This allows libraries and MEX-files built using gcc 3.x to be used on computers on which gcc 3.x has not been installed.
• dfm now uses ranges.txt file.
• Some bug fixes.

Version 4.12

• Philip Charlton provided new version of metaio.c to allow various utilities (lwtscan, lwtprint, lwtdiff, readMeta) to be able to read the slightly different LIGO_LW format produced by LDAS release 0.7.0. The code reads the pre-0.7.0 format as well.

Version 4.13

• Modified concatMeta, FrameCacheDump, FrameCacheQuery, and FrameCacheRanges to work properly with LDAS 0.7.0.

Version 5.0

• Various changes, mostly internal.

Version 5.1

• Incorporates new version of the metaio code from Philip Charlton to compile properly using gcc 3.x.
• Linux precompiled distribution is now built with gcc 3.x on a machine running Red Hat 9.
• New utility lwtcut.

Version 5.2

• Updated FrameCacheDump, FrameCacheQuery, and FrameCacheRanges to handle new format (as of LDAS 1.0) of output of getFrameCache user command.
• Fixed a bug in FrameCacheRanges which showed up when multiple directories contained overlapping time ranges.
• Modified getFrames to add new syntax to print out the list of UDNs, or the channel list or time interval list for a given UDN.
• Modified getFrames and dfm to allow getFrames to get trend frame data.

Version 5.3

• Fixed bugs in FrameCacheDump and FrameCacheQuery: there was a mistake in the code to parse the frame cache info, with the result that all frame files were assumed to have a duration of 1 second.
• Enhanced lwtcut to allow string comparisons.

Version 5.4

• Modify dfm to use mirfak.ligo.caltech.edu as default LDAS job proxy server, rather than sheratan.ligo.caltech.edu.
• Fix dfm error message in case it can't connect to the LDAS job proxy server. (It was reporting being unable to connect to the LDAS manager operator socket.)
• Fixed argument parsing in lwtscan, lwtcut, lwtprint, and lwtdiff so that the table name, output file, delimiter, and float option can be specified (with their corresponding option flags, of course) before the input file, if desired.

Web-patch version 5.4.1

• Modify dfm to use getFrameData instead of createRDS. Adjust end time of data to account for LDAS convention of including the final second.

Release version 5.5

• Modified readMeta so that INT_8S values read in are stored internally as 8-byte integers (Matlab class mxINT64) rather than doubles, to preserve precision.
• Copied latest version of metaio library code and related utilities from the metaio CVS repository. There are two new library functions, MetaioOpenFile() and MetaioOpenTableOnly(), contributed by Patrick Brady. The library has also had some bug fixes and internal improvements by Philip Charlton and Peter Shawhan. Note that header.h has been replaced by ligo_lw_header.h, and this is now in the 'include' directory.

Release version 6.0

• Major reorganization of the dataflow package! It had grown over time to include a wide range of unrelated functions. It has now been split into several packages:
  • The dataflow package now consists only of:
    the data flow manager (dfm);
    the dataflow C library, which now contains ONLY functions to interface to the dfm, i.e. it no longer includes the metaio parsing code or the TConvert functions; and
    the utilities which make use of the data flow manager: dfmstart, dfmcheck, dfmon, dfmkill, ldaspw, getFrames, getMeta, putMeta, putSpectrum, dfmpipe, SeqInsert .
  • The metaio LIGO_LW parsing library and utilities (lwtscan, lwtprint, lwtdiff, lwtcut, lwtselect, concatMeta) have been moved into a new 'metaio' package. Note that the _getMetaLoopHelper program has also been moved into the metaio package, even though it is only intended for use by the getMeta program here in the dataflow package.
  • The tconvert and gpsclock utilities, tconvert Tcl library, and GPS-UTC time conversion C functions have been moved into a new 'time' package.
  • The getUrl and httpTextDisplay utilities have been moved into a new 'httptools' package.
  • A few other things that were in the dataflow package are more appropriate elsewhere: the FrameCacheDump, FrameCacheQuery, and FrameCacheRanges utilities have been moved into the 'ldasjob' package, while the makeSegmentXML script has been moved into the 'segments' package.
• Modified SeqInsert to allow client functions to connect to a server running on a different host.
• Modified dfm to strip away domain name, if it is included in output from [info hostname], when constructing name of register file.

Release version 6.1

Removed C library from this package, since I'm sure no one is using it. Now it contains no compiled code, so it can be distributed as a "universal" distribution file.

Release version 6.2

Modified dfmcheck, dfmkill, dfmpipe, getFrames, and getMeta to strip away domain name, if it is included in output from [info hostname], when constructing name of register file.

Release version 7.0

Modified dfm to use persistent socket connections when communicating with LDAS, avoiding firewall issues.

Modified SeqInsert to use the ldasjob library instead of dfm.

Release version 7.1

Fixed a bug that was introduced into SeqInsert in the previous version.