Summary of this page:
Updated livetime list to DLT 929. For some reason skdbntuple chokes when dumping the ntuple if the `-1' flag is given, so I had to make multiple ntuples and hmerge them together. Result is in /u/bocka/scratch/data/. This gives 784.9 days.
Just realized that I was not limiting the SKDB subruns to those for which I have Danka-flasher cuts. Because of this I overestimated the livetime below. It should be 667.2 live days. The corrected runcuts-final.kumac has replaced the wrong one in the link below.
The final subrun cuts are defined by runcuts-final.kumac. With up to DLT799 this gives 742.2 live days.
In making more Kate-minireport style plots for before/after cuts it seems that later data has an excess of high ID events / runtime per subrun which make it through the cuts.
Most of the sub runs which have ID event rate above 50 Hz are in run 4126 which is with SLE trigger and to take decay-e data (according to the run summary). This run is about 10 hours long. The other shorter run contributions to the events are SLE ``test'' runs which were incorrectly labeled and got into the data stream. There are also a couple of subs which have flashers.
For subs with ID event rate between 20 and 50 Hz there are SLE test runs, blasting and some subruns where it is not clear (from reading summaries) why.
About 1% of subruns are above 50Hz ID and about 2% are above 20Hz ID and account for 5.40E4 sec (.624 days) and 1.39E5 (1.613 days) livetime. This isn't much livetime and is likely a source of bad events (SLE test runs, flashers, blasting) so I'll cut out all subs with more than 20 Hz ID event rate.
There are also about a dozen subruns with less than about 5Hz ID event rate. These are mostly from run 3396 which had the LE trigger off. The other subruns were part of junked runs or unknown (again from run summaries). Since we don't care is LE trigger is off, I'll keep these subs.
Also found some subs with no OD triggers. Most of these subs are from run 4204 which is labeled as Normal trigger (SLE) and have anti trigger turned off. These should be cut. Also, we should not use subruns which get much less than the cosmic ray muon rate (~2.2 Hz)
Found some bugs in skdbntuple.c due to not understanding the (buggy and undocumented) SKDB data base. This was causeing all the v1 entries to be cut due to haveing zero runtime.
Fixing these gives a runtime before cuts (except for $4 cuts) about that which Kate got for the same runs (5.914E6 sec vs. ~6.0E6 (read from her plot)). The livetime is 5.165E6 secs for similar cuts as Kate (who gets 5.184E6 secs).
Somewhat finalized cuts are defined in runcuts.kumac. Some of the cuts Kate uses make little or no effect, such as HE trigger rate per subrun and the livetimeid/runtime per subrun so they are not cut on. The bigbelow/bigabove does remove a handfull of subruns which has more than about 10 big inner events with small outter events which are most likely flasher-rich subruns. The skiptime cut doesn't cut much but what it does cut makes sense. See this page for effects on livetimeid and livetiemod due to these cuts.
Post-SLE data can not be cut by looking at OD information due to the bug discussed below. But for an event to be considered good it must have both ID and OD data. Since it is impossible to remove subruns which have high OD deadtime the livetime will be overestimated. To estimate the overestimation, pre-SLE subruns are used. The total livetime for pre-SLE after the runcuts are applied is 2.45645E7 seconds. If, in addition, the subrun must have more than 400 seconds of OD livetime (which is essentially the livetime cut Kate used) the total pre-SLE livetime reduces to 2.4427E7 seconds. This gives an overestimation of less than 1%.
It isn't possible to go back to highe1 output tapes to get a correct livetime as there are only a few events per subrun at this level of reduction. This means the only way to get a correct livetime is to re-run all data.
Talked with CK a bit last night. He had a few suggestions for more plots. Basically we were trying to figure out what the shoulder near 1.0 was in the livetimeID/runtime plot shown here. The only thing I found in skdb_run.c is that the CALI bank is zeroed for all pedestal events, so I am pretty sure this is the source of subruns which have almost-but-not-quite 100% livetime
See this page for info about developing the livetime based subrun cuts.
Well, last night after the vid-conf I got email from Kate which had a reply from Clark to an email from her back when she was slogging through all this stuff that I am now. From it I was prompted to check the date of the skdb_run.c file which calculates the livetime and the date of the actuall highe1_toga exec which generates the skdb NDBM. Low and behold, they didn't match. Even though Clark updated the skdb_run.c code to handle the SLE trigger turnon, noone ever actually installed a new highe1_toga exec so we blithely went on running a bogus exec.
So the only recourse seems to be to re-run all the raw data through an updated highe1 or calculate the livetime by using only livetime_id and attempt to get rid of any subruns which have excessive bad OD. And of course it is not possible to re-run all data and graduate this millenium.
See this page for plots comparing different livetimes before and after SLE turnon.
It seems the reason why Kate did not have a shift in livetime when SLE turned on is that her Perl scripts use the ID livetime after SLE and the livetime before SLE.
I still don't understand, however, why the skdb routines get the livetime wrong. Clearly something is causing the OD livetime to be greatly reduced. The OD livetime is acumulated for events which have an OD and HE trigger, so any missed OD livetime chunks have a trigger but not a bank.
I'll try to strip out some raw data which has this OD trigger but no bank.
More of Kate-mini-report-style plots but seperated by pre and post SLE trigger implimentation. These plots show some possible problems with the way livetime was calcualted.
Kate's mini-report pre-subrun-cut plots for full data set using my ntuple.
Kate's mini-report pre-subrun-cut plots for sub sample of data which Kate used in her mini-report, but using my ntuple. This page also has details of how the ntuple any plots were made
I have been trying to understand the process which Kate when through and documented at
I have run her rundb.pl and runsum.pl scripts which are part of the freshly commited src/programs/rundb/ directory. Since then she committed a forgotten readlogbook.pl script. This and logbook.pl are used to deal with some hand cut subruns which are not well documented (are they at all?). They are largely based on the badrun list done by CSDH of which I know little. Not many subruns (~7%) were cut by the logbook which weren't cut elsewhere by more analytical and reproducible cuts
I have also written a run database (NDBM file) to PAW ntuple converter. This is in src/highe/skdbntuple.[ch]. It currently has trouble with Linux's db, but seems to work just fine on the alphas.
I will try to compare this ntuple to Kate's work to make sure I haven't screwed anything up. I'll do it using the same days of data. From her goodsubrun2.save file she uses runs 1743 to 2651. To run this I do
HIGHE=/home/highe skdbntuple -1 -o skdbnt-kate -r 1743 -R 2651.
Some of the cuts don't translate well to ntuples (eg, don't know how many subruns are in a particular run very easily).