How to run DMT Test via SeqInsert

In an engineering run, the actual dmt triggers are sent via a ligotools program SeqInsert via an http like process. This test exercises the method to make sure ldas is compatible with actual dmt process; the test objectives are the same as simulated DMT:

Ability of ldas to ingest data from external xml formatted by dmt process by retrieving xml from url, converting it into ilwd format and inserting the data into the database.

Account for all the data sent to ldas from dmt: at times there maybe errors obtaining the file from url, or there maybe due foreign key errors if the data is not inserted in sequence. To ensure data integrity some fine tuning maybe necessary for the test e.g extending http time out in the ligolwAPI when the system is very busy.

The LDAS APIs affected by this test are:

managerAPI: directs processing of the insertion user commands to the ligolwAPI and metadataAPI and returns the user command status to the user.

ligolwAPI:
- Retrieves the data to be ingested from the specified url to ilwd object and sends the ilwd to the metadataAPI for insertion.
metadataAPI:
- insert the metadata ilwd data from ligolwAPI into the specified database.

Setup

Choose a directory to install the tests scripts.

The script files are:
insertTrigs - main driver script
directory of dmt files - download sample dmt files into a directory and untar and unzip the files
```
tar xvzf dmt-test.tar.gz
```
Remove the tar ball (.gz file) from the directory since only xml files can be used.
SeqInsert - from ligotools

Note: The above files should be installed on your local platform which should also have ligotools installed already.

To setup the environment for ligotools, login on a sun workstation and enter :

eval `/ligoapps/ligotools/bin/use_ligotools`

Running the test

The test should be run from your local platform, and run in the scripts directory. Before running the test, clear out any previous dmt data in the database you are going to do insertion by running the delall.sql script from ldas installation: e.g.


			db2 'connect to my_db'
			db2 -tf /ldas/doc/db2/doc/text/delall.sql 
			cd <scripts directory>
			tclsh insertTrigs cit ldas_tst ../triggers2 >& cit.log &

Usage: tclsh insertTrigs [options] >& insertTrigs.log &

Option	Description
`<site>`	ldas site [ dev \| test \| lho \| llo \| mit ] or <mycomputer>:10001
`<dsname>`	database name at a site e.g. cit_test
`<dmt triggers directory>`	directory holding dmt triggers xml files e.g. /home/john/triggers2

Test Verification:

The test should be monitored initially to make sure data is flowing in a steady pace and that the insertions are successful impling that data is inserted in the correct sequence.

Monitoring dmt
All output from script SeqInsert is captured to stdout which can be redirected to a file at startup e.g. cit.log.
A putMetaData user command is generated for each of the dmt files and sent to the managerAPI. The insertions can be monitored via cmonClient for the number of jobs and the database insertions for the specified database.

Analysis
When all the xmls are sent to LDAS, the logs can be extracted via cmonClient Log page; enter the gpstime ranges recorded in insertTrigs.log for logs in managerAPI, ligolwAPI and metadataAPI. Validate the following is true for successful insertions:
- Number of files submitted by SeqInsert = number of xml files via unix command
```
 grep -c submit < insertTrigs log file >  
```
- Jobs statistics page should report the number sent by SeqInsert.
- Database statistics page should report the total number of insertions sent by SeqInsert.
- Number of gds_triggers inserted in database = number of gds_triggers in xml files
  use guild query
```
select count(*) from gds_trigger where process_id in 
(select process_id from process where program
in ('glitchMon', 'PSLmon'))
```
  to confirm this is the case.
If there is a discrepancy search the logs for red balls to account for missing records.
Also examine the ldas database web page for the number of processes, segments and gds_triggers inserted to make sure they match the expected number.
If all the process and trigger data are inserted, the test passes. If not, the test fails and the errors in the logs should account for all data received at LDAS.