LIGOtools Package: dataflow

• The dfm (Data Flow Manager) program, which acts as an intermediary between client programs and various information servers. The user normally does not run this program directly; it is automatically started when needed, and runs in the background. (You can manually start it in the background by typing dfmstart, though normally this is done only by other programs. For debugging purposes, it can also be run in the foreground by typing dfm -v.)
• Some utilities to start the DFM, check whether it is currently running, and shut it down gracefully (dfmstart, dfmcheck, and dfmkill).
• The utilities getMeta and getFrames, which allow the user to retrieve metadata and frame data, respectively, from the unix command line or from within a script.
• The putMeta utility, which allows the user to insert metadata into the LDAS database.
• The putSpectrum utility, which reads a frequency series from an ASCII file and inserts it into the summ_spectrum table in the LDAS database.
• The dfmpipe utility, which provides a stdin/stdout interface to the DFM (mostly intended for use by other programs which are not able to open a socket connection to the DFM).
• The SeqInsert utility, which queues metadata files for sequential insertion into the LDAS database.
• The ldaspw script, which securely stores one's LDAS username and password for later use by getMeta and getFrames.
• The dfmon client, which allows one to monitor the actions of the data flow manager.

Author: Peter Shawhan

Version 7.2

Frequently Asked Questions related to this package

• How can I retrieve data from the LDAS database?
• How can I work with a large amount of data from the LDAS database?
• How can I get LIGO frame data onto my local disk?
• How do I store my LDAS username/password so that LIGOtools utilities can use them?
• How can I use LIGOtools programs from behind a firewall?

Notes

• Versions 6.0 and earlier of the dataflow package included compiled C code, and thus were distributed in SOURCE and precompiled binary versions. The reorganization in October 2005 removed all compiled code from this package, so now it is distributed in a "universal" version.

• The programs in this package use the tclshexe and wishexe programs from the LIGOtools package 'tclexe', so that package must be installed and active.

• getFrames provides access to "recent" data at the LIGO observatory sites, as well as to data in the archive at Caltech. See this note about access to LIGO frame data for dataset names, etc. using getFrames. However, you may be better off using the alternative utility getFrameData in the 'ldasjob' package, which is simpler and generally more robust. The only disadvantage of getFrameData is that you have to specify what LDAS system to get the data from, which is usually easy anyway.

• getMeta, getFrames, and putMeta are "fairly simple" programs which basically just pass the parameters on to the data flow manager, which takes care of all of the communication with data servers. The data flow manager has the special ability to "patch" itself from a web site when it is run, so new capabilities (e.g. access to additional data servers or datasets which become available) should be immediately usable without having to install a new version of the dataflow package.

• Before you use getMeta, getFrames, etc., it is highly recommended that you run the ldaspw utility. This stores your LDAS username and password so that you don't ever have to type them in again when using the utilities in the dataflow package.

• Starting with version 4.0 of the dataflow package, one dfm process is started (when needed) per user per machine. (Previously, there was only one dfm process per machine, started by the first user who needed it and shared by all other users.)

• The dfmon client currently is mainly useful for debugging; eventually it should be revamped to present job status in a clear fashion, and to allow job control.

Documentation

• Typing getMeta, getFrames, putMeta, or putSpectrum without any arguments will print out usage information - in most cases, rather detailed.

• Available UDNs (Universal Dataset Names) are listed here.

• The utility dfmstart starts the DFM in the background (unless it is already running), and does not return until the DFM is ready to accept client connections (or until 10 seconds have elapsed, in which case dfmstart exits with code 1). However, you should not normally have to do this, since any program which makes use of the DFM will automatically start it up in the background. You should not run the DFM in the foreground (i.e. by typing dfm) except for debugging purposes.

  The dfmon utility can be used to display internal messages generated by the DFM while it is running. Depending on command-line arguments, it either pops up a simple GUI to display the dfm's internal messages, or dumps the messages to a file or to standard output. Type "dfmon -h" to see the command syntax.

• Use the dfmcheck utility to check whether the DFM is currently running. By default, this prints out a message; alternatively, you can use the -n option to print the DFM's client port number (or 0 if the DFM is not running), or the -s option to disable all output but set the exit status to 0 or 1 depending on whether the DFM is running. Type "dfmcheck -h" for more information.

• dfmkill shuts down the DFM even if it has active jobs, but is somewhat more graceful than kill.

• A description of the communication protocol used by the Data Flow Manager (of interest to people developing client programs, not to users of existing client programs) can be found here.

Distribution files