Special thanks to Bob for showing my transparencies. I'm really sorry
I missed the meeting (that is, sorry because it's impolite to snow
everyone with email instead AND sorry because I wanted to talk to you
all). He asked a couple of questions which I would be happy to answer,
and I will write a bit more to cover what I would have liked to discuss
in more detail in the meeting.
The easy one first: what about the likelihood function. I totally
agree with Francesco that it's too complicated. Most of the trouble,
however, comes from the last term, which involves the number of runs.
In fact this term is not too terribly physically relevant, and we could
drop it. I _would _argue, however, that the N+ term should be kept.
(that is, the second term, which decides how improbable it is to have n+
out of N resutls on the high side of hte prediction). This would make
things much simpler, and as I said there is no overwhelming physics
reason to keep the last term (the "run" term). I would also be very
happy to look at any other possibilities. I should say, however, that
Kolmogorov _only_ looks at the greatest deviation from the theory, and I
think it hides too much physics that should be relevant to oscillations.
Next, about the one- versus two-sided tests. This is a hard question,
But I have softened on the two-sided test, given some discussion with
other people thinking about statistics. I think the standard thinking
is that when one sees a non-discovery, one uses one-sided tests, and
when one gets a result, one uses two-sided tests to isolate the allowed
region. Given our data, I still think we need both.
In fact we already use a one-sided test on the unbinned result... that
is, we see something like 1.5 sigma too few events, and give a
probability of seven percent to see as few or fewer. At least I don't
_think_ we say that there is a 14% probability to deviate by as much
from the model (If I'm wrong, then I should know).
In the no-oscillation region, which predicts a large number of events,
we see too few, and the one-sided test again makes sense. That is, the
relevant probability is "what is the chance to see as few or fewer."
Seeing too many events can't be used as evidence again no mixing, at
least in the nu-mu/nu-tau vacuum oscillation model (it can, however,
indicate a systematic problem). In fact we already use a one-sided test
on the unbinned result... that is, we see something like 1.5 sigma too
few events, and give a probability of seven percent to see as few or
fewer. At least I don't _think_ we say that there is a 14% probability
to deviate by as much from the model (If I'm wrong, then I should know).
In the heavy mixing region, which predicts about half as many events as
the no-oscillation hypothesis, I think the same argument holds true.
Full mixing predicts the smallest possible number of events, and seeing
too _few_ shouldn't be used as evidence _against_ it. It could still
indicate some other problem (i.e. it's the wrong theory or there's some
acceptance problem).
BUT, in the "signal" region, deviation in either direction are
interesting because they indicate that the paramters in question are
either too high or too low. Here a two-sided test is probably good.
So my answer is that one test does not do everything. I use the
one-sided tests now because our 8.8% "best-fit" point doesn't convince
me we have a real signal. In an ideal world (i.e. if the 8.8% moved to
35%), we would be able to make a two-sided plot. The quotes for the
edges, however (i.e. no mixing and full mixing) should probably still be
one-sided.
Finally, right now we necessarily need the two-sided test to check the
"unusualness" of the angular distributino. Unfortunatley, no matter the
details it is going to assign a low probability to every point in
parameter space. The problem is that the fluctuation (as I believe, but
you can write systematic problem if you like) dominates the probability
no matter what paramters are chosen. The one-sided test, then, becomes
the _less_ restrictive one.
What to do? We could make a two-sided contour plot, but divide
everything by 17.8% (that is, twice 8.8% in order to "normalize" the
best-fit point to 50% probability). This was an idea raised by Doug,
which "normalizes out" the effects of the bump. Or we could keep the
one-sided exclusion points and draw in the two-sided best-fit region,
but call out a caveat that the agreement never gets above ten percent.
This is obviously an unsatisfying approach for many reasons.
So let me make a new calculation, in the "standard" way, but without
the last term in the likelihood. Then I wil make the "standard"
one-sided contours, and also a two-sided "normalized" contour.
I am very happy to get suggestions about all this. I am particualrly
interested to hear if anyone else wants to know about Kolmogorov... I
gave some numbers on this before, but called them preliminary and could
refine the resutls a bit. But only if someone else is interested.
Also, if anyone wants to point me to any other particular function, I
would be glad to put into the code.
Nat