Subject: Summary of muon dst production
From: Erik Katsavounidis (Erik.Katsavounidis@lngs.infn.it)
Date: Fri Sep 15 2000 - 07:38:34 EDT
Hi all,
This is a rather technical note on the final (hopefully)
production of MACRO's so-called muon astronomy DSTs. This
kind of DSTs have been traditionally used in the muon neutrino
analyses as well as in the muon astronomy searches.
The content of this e-mail is of interest mostly to people
who have performed or are currently performing analyses
using the muon astronomy DSTs.
I am copying it to our mailing lists for archiving purposes
in case any time in the future other people pose relevant
questions or use that data for analyses.
I can highlight a few things that have been derived through
this process and I believe they might be of general interest:
o MACRO data from March 1994 to June 2000 have been reprocessed
starting with the full MACRO data stream (~4TB)
o There were 168 *normal* (i.e., non calibration) MACRO RUNs of
total duration ~15 days (0.7% of livetime) that were unfortunately
not found in MACRO's official tapes. In addition, at least 56
calibration RUNs never made to the tapes (~1.2% of calibrations'
livetime). To zeroth order, these data are lost (see section
with details - at any rate my guess is that not all of the lost
data were analysable).
o The re-process of MACRO data has also revealed ~12 days of data-
taking that for various reasons NEVER made to the muon dsts before.
This reflects livetime gain. A good fraction of them should be fully
analysable. An effective increase of livetime of all muon analyses
performed so far by 0.5% is thus expected.
o I propose that this new version of muon DSTs superseeds the
old one. Gran Sasso disks currently holding "old" muon dsts
will be cleaned up and the new set will be permanently stored
there. People doing analyses and group leaders please, I need
your OK by September 30 in going ahead with this swap. Old data
will be archived on tape. Managing both old and new sets on
disk has been quite a pain over the last few months.
o The "MACRO's Muon DST web page" will be from now on at
http://wsgs02.lngs.infn.it:8000/macro/mudsts/
We hope to keep all relevant info there for any present and
future related questions, needs to use the muon dsts for
analyses etc. If you are a muon dst user, please make sure you
check out this page for updates.
Let me know if you have any comments of questions.
Thanks,
--Erik
PS: A more detailed description of the DST production follows.
================================================================================
The bulk of MACRO re-processing starting from the full stream contained
in MACRO's official data tapes was performed from February to June 2000.
There were 1-2 months prior to that that Ioannis and myself spent for code
development and testing. Once tested, more than 50% of the automated job
launching and diagnosing was performed by the scintillator technicians
at Gran Sasso. The production of the muon astronomy dst through this stream
features the following:
o MACRO (full) RUNs in ZEBRA format were first downloaded to disk, converted
back to RAW and a no-WFD, no-PHRASE DST (in RAW format) was created. This
DST is referred to from now on as a "GC-DST" (suited for ERP-GC analyses).
The MUON dsts were produced using the GC-DST as input.
o There had been several incremental upgrades and fixes that the current
muon dst producing software went through over the last 6 year. This was
actually one of the motivations in re-processing all MACRO data with the
most up-to-date code. We went beyond that in examining and eliminating
all fortran errors that had existed in the code AND HAD BEEN RESPONSIBLE
FOR SILENT CRASHES OF MUON DST PRODUCTION THROUGHOUT THE YEARS.
The code we've used in producing the muon dsts is
AXPGS0::DISK$SCRAUSA2:[MACROUSA0.ROBOT.MAKEDST]
o The muon DSTs contain at the very beginning the data of MACRO's standard
online logbook (LGB) entries. When for whatever reason a RUN had no LGB
entry at the time of the muon dst production THE MUON DST WAS NOT PRODUCED.
We have now produced all the missing LGB entries. Unfortunately, we have done
this using the "GC-DST" and NOT the full MACRO RUN. The only implication of
this is that the PHRASE rate is missing from the LGB. This was a fair
tradeoff among having the LGB timely done for the muon dst needs and going
through an extremely tedious path of downloading the full MACRO RUN,
converting it to raw etc etc.
o All detector geometry used is the one most up-to-date and includes the
correction for the 8th vertical scintillator counter (introduced by Ioannis
~1997). "Dummy" scintillator calibration constants were used, or at least
this is how they should be treated.
o All log files of the batch jobs that produced the dsts have been kept on
disk. Whenever in doubt, make sure you check the log file or ask. LOG files
can be found at
AXPGS0::DISK$SCRAUSA7:[MACRODATA.LOG...]MA_MONITOR_*_RUN%%%%%%.LOG
o Detailed detector monitor and efficiency files have been produced for all
MACRO RUNs. These can be found in http://wsgs02.lngs.infn.it:8000/macro4/lgb/
o The new muon astronomy DST that were produced through this process were
confronted to the old (real time) production. The basic criterion has been
the actual size of the file that reflects counting of muons and
thus livetime. This confrontation yielded the following observations:
- there are 6 RUNs for which the "new" production is of significantly lesser
size respect to the "old" one: (size is in blocks=512bytes)
+-------------+-------------+-------------+
| RUN | NEWSIZE | OLDSIZE |
+-------------+-------------+-------------+
| 12407 | 9296 | 13703 |
| 12408 | 419 | 1045 |
| 12409 | 174 | 3198 |
| 12410 | 1890 | 8413 |
| 13342 | 19059 | 20256 |
| 13992 | 654 | 10062 |
+-------------+-------------+-------------+
I have manually investigated the entire chain of (new) production for the
above 6 runs. I have found NO ERROR in the NEW production for the first
5 RUNs. Notice that the log file of the old dst production is not available.
This makes me conclude that the OLD production for the first 5 RUNs
has SOMETHING WRONG.
The NEW production for RUN 13992 is left incomplete due to unrecoverable
TAPE ERROR in reading the full file from tape to disk. If you hold a copy
of DD106 that contains RUN 13992 and you can successfully download it to
disk, please let me know-- this will fully recover the file!
- There are 102 RUNs that lasted each more than 0.5 hrs for which the NEW
production yielded a file signicantly (>300 blocks) larger than the OLD
one. For these 102 RUNs the NEW production yielded an average size of
7221 blocks versus an average 2235 for the OLD one. Given that a muon event
results in roughly 2 blocks, this implies that for these 102 RUNs, the
NEW production yielded 2.5x10^5 more muons or ~12 days of data taking.
There are several reasons that cut the size of the OLD dst short. These can
be split into three categories: (a) fortran errors (b) lack of LGB
(c) unknown (at least for someone investigating the case now) reason.
Obviously, none of the above three applies in the NEW production.
The list of these 102 RUNs where significant livetime/acceptance was
gained can be found in
http://wsgs02.lngs.infn.it:8000/macro/mudsts/gain_runs.txt
o The new muon DST production features identification of calibration RUNs
through detailed examination of the scintillator information. Although muon
DSTs are produced for calibration RUNs too, they are not made available
to the wider MACRO people in order to avoid confusion (and save disk space!).
Calibration runs are listed in
http://wsgs02.lngs.infn.it:8000/macro/mudsts/all_calibs.txt
If you have any reason to believe that any of runs there included is a good
run, I'd like to know.
o A number of RUNs resulted to a muon dst of zero size. Each RUN was carefully
examined in order to establish if this was legitimate. These are mostly
PHRASE-only RUNs or RUNs of extremely short duration that collected no
muon trigger. The list of these runs is available in
http://wsgs02.lngs.infn.it:8000/macro/mudsts/empty.txt
o random bits and pieces from my logbook:
- RUNs 19179 and 11849 have erroneous LGB entries. This is due to a leap
year problem (both taken on Feb 29 of a leap year). I haven't had time
to investigate further the problem. Exersice caution when using the LGB
info for these two runs.
- RUN 10880 had an LGB entry that was actually the one for RUN 10881.
Correct LGB data was produced again for RUN 10880.
o as it was already highlighted at the very beginning, a 0.6% of MACRO's
livetime was lost during the copy-to-tape phase. There are very few RUNs
that are partially recoverable (see below) while I'm not in a position to
know what PHRASE might have stored for these completely missing RUNs through
their independent run processing machine.
- there is one RUN (#10052) that is missing from the MACRO TAPEs (and
thus from the "new" production) which interestingly enough has the "old"
muon dst available. Too bad this is a junk RUN that lasted few seconds
and had the ERP broken.
- I have been keeping personal copies of MACRO data since sometime in 1997.
This makes available two full MACRO RUNs that were otherwise lost:
RUN 16720 taken in 20-NOV-1998 and lasted ~5 hours and
RUN 91068 taken in 2-JUL-1998 and lasted ~8 minutes.
The muon dsts for these two runs have been produced and merged with
the rest of the new set.
o Please take the time to run any sort of tests confronting old and new
DST production. A good fraction of the new muon dsts are available on
disk at Gran Sasso in any of the following areas:
NFS_MACRO1:[MUDST.NORMAL]
DISK$SCRAUSA16:[MACRODATA.MUDST]
DISK$SCRAUSA11:[MACRODATA.MUDST]
DISK$SCRAUSA9:[MACRODATA.MUDST]
Caltech, Frascati and Lecce should have in their hands DLT tapes with
the entire data set within few days.
This archive was generated by hypermail 2a24 : Fri Sep 15 2000 - 07:38:38 EDT