Upload
ilene-morgan
View
230
Download
1
Embed Size (px)
Citation preview
NASSP Masters 5003F - Computational Astronomy - 2010
Lecture 7 – chi squared and all that
• Testing for goodness-of-fit continued.
• Uncertainties in the fitted parameters.
• Confidence intervals.
• The Null Hypothesis.
Hypothesis testing continued.• Procedure:
1. “Suppose the model is a perfect fit.”
2. Calculate survival function for χ2 of pure noise of N-M degrees of freedom..
3. Draw vertical at point of measured χ2.
4. Y value where this vertical intercepts the SF is the probability that a perfect model would have this χ2 value by random fluctuation.
NASSP Masters 5003F - Computational Astronomy - 2009
Su
rviv
al f
un
ctio
n
Questions answered so far:• In fitting a model, we want:
1. The best fit values of the parameters;
2. Then we want to know if these values are good enough! Ie if the model is a good fit to the data.
3. If the model passes, we want uncertainties in the best-fit parameters.
• Number 1 is accomplished. √• Number 2 is accomplished. √
NASSP Masters 5003F - Computational Astronomy - 2009
Uncertainties in the best-fit parameters• Usually what one gets is a covariance matrix
(mentioned in lecture 4):
• This is a symmetric matrix: σij2=σji
2 for all i,j.• For U=χ2, E=2(Hbestfit)-1, where Hbestfit is the Hessian,
evaluated at the best-fit values of the θi.• For U=-L, E=F-1, where F is the “Fisher Information
Matrix”:• These definitions are equivalent!
– For Gaussian data, identical.NASSP Masters 5003F - Computational Astronomy - 2009
22
221
212
21
E
ji
ji
LF
bestfit2
,ˆ
The Hessian or curvature matrix• The contours are
ellipses in the limit as the minimum is approached.– Ellipsoidal hypercontours
in the general case that M>2.
• Semiaxes aligned with the eigenvectors of H.
• Small semiaxis: large curvature;
small uncertainty in that direction.
NASSP Masters 5003F - Computational Astronomy - 2009
Contours of U:
Arrows show the eigenvectors.
1-parameter example1) Gaussian data, U=χ2.
– For this simple model, we can find the best fit θ without numerical minimization:
– Setting this to zero gives:
NASSP Masters 5003F - Computational Astronomy - 2009
N
i i
iyU1
2
2
N
i i
iyU
12
2
.1
ˆ
12
12
N
i i
N
i i
iy
Sidebar – optimum weighted average• A weighted average is:
• Since the yi are random variables, so is μ^.
• Therefore it will have a PDF and an uncertainty σμ.
• The smallest uncertainty is given for
– Exactly what we have from the χ2 fit.NASSP Masters 5003F - Computational Astronomy - 2009
i i
i ii
w
yw
21 iiw
Back to the1-parameter example.– Again, because this model is so simple, we
can calculate σθ by direct propagation of uncertainties.
• θ^ is a function of N uncorrelated random variables yi, so
• It is fairly easy to show that:
NASSP Masters 5003F - Computational Astronomy - 2009
N
ii
iy1
2
2
2ˆ
N
i i12
2
11ˆ
What does the standard approach give?• Hessian is a 1-element matrix:
• Hence
• QED.
NASSP Masters 5003F - Computational Astronomy - 2009
2
bestfit
2
2
1,1
12
i
UH
21,1
112
i
H
1-parameter example continued2) Poisson data, U=-L. (No point in using -L for gaussian
data, it’s then mathematically the same as chi squared.)
– Again it is simple to calculate the position of the minimum directly:
– Setting this to zero gives
Ie, the average of the ys.NASSP Masters 5003F - Computational Astronomy - 2009
N
iii yyU
1
!lnln
iyN
U
N
iiyN 1
1
Uncertainties in the Poisson/L case.– With our present simple model it is very easy
by propagation of uncertainties to show that
– Following the formal procedure for comparison:
– Inverting this gives the same result.
NASSP Masters 5003F - Computational Astronomy - 2009
N
ˆˆ 2
ˆˆˆ
2bestfit
2
2
1,1
NyLF i
1-parameter example continued3) Poisson data, U=“chi squared”.
– There are two flavours of “chi squared” for Poisson data!
– Note that the following is simply incorrect:
NASSP Masters 5003F - Computational Astronomy - 2009
N
i
iyU1
2
Pearson
N
i i
ii
y
yyU
1
2
Mighell 1
1,min
N
i i
i
y
yU
1
2
Don’t use Pearson’s for fitting.• It is not hard to prove it is biased.
– Eg, keeping our simple model,
– Setting this to zero gives
– In his paper, Mighell calculates the limiting value of θ^Pearson as N->∞ and shows it is not θ.
NASSP Masters 5003F - Computational Astronomy - 2009
2
2
iyN
U
N
ii
N
ii y
Ny
N 11
2Pearson
11
The Mighell formula is unbiased.– For this statistic,
– Setting this to zero gives
– Some not-too-hairy algebra shows that the limiting value of θ^Mighell as N->∞ is equal to θ.
NASSP Masters 5003F - Computational Astronomy - 2009
N
i i
iiN
i i y
yy
y
U
11 1
1,min
1
12
N
i i
N
i i
ii
y
y
yy
1
1Mighell
11
1
1,min
Goodness-of-fit:1. The Gaussian/χ2 case has been covered already.2. The Poisson/L case is a problem, because no
general PDF for L is known for this noise distribution.
– If we insist on using this, have to estimate SF via a Monte Carlo. Messy, time-consuming.
3. For the Poisson/”chi squared” case, where we have 2 competing formulae, we should do:
– Use Mighell to fit;– Use Mighell for uncertainties;– But use Pearson (with the best-fit values of θi) for
goodness-of-fit hypothesis testing.• Because it has the same PDF (thus also SF) as χ2.
NASSP Masters 5003F - Computational Astronomy - 2009
Confidence intervals• There is a hidden assumption behind
frequentist model fitting: namely that it is meaningful to talk about p(θi^).
NASSP Masters 5003F - Computational Astronomy - 2009
ip
i
• We already have some hints about its shape… and a Monte Carlo seems to offer a way to map it as accurately as we want.
Confidence intervals
NASSP Masters 5003F - Computational Astronomy - 2009i
ip
bestfit,i
2
Bayesians think this is nonsense.• Such a MC is like pretending that θ^ is the
‘true’ value, and then generating lots of hypothetical experimental data.
• But all we really know is the single set of data which we measure in the real experiment.– Plus possibly some ‘prior knowledge’.
• We don’t want p(θ^), we want p(θ).
• But we’ll continue with the frequentist way for the time being.
NASSP Masters 5003F - Computational Astronomy - 2009
Confidence intervals• We also assume that p(θi) is approximately Gaussian (which may be entirely unwarranted!!)
– We interpret this to mean that there is a 68% chance that the interval
contains the truth value θ.
NASSP Masters 5003F - Computational Astronomy - 2009
68.02
exp2
12
2
x
dx
ˆˆ,ˆˆ
Confidence intervals• Note that this is not the only
interval which contains 68% of the probability. We can move the interval up and down the θ axis as we please. The –σ to +σ version is just a convention.
• FYI
erf() is called the error function.
NASSP Masters 5003F - Computational Astronomy - 2009
2
erf2
1
2exp
2
1
02
2 axdx
a
Confidence intervals• For more than 1 parameter
the q% confidence interval is the (hyper)contour within which the probability of the truth value occuring =q.
• Again, by convention, symmetrical contours are used.
NASSP Masters 5003F - Computational Astronomy - 2009
When m=s+b (which is not always appropriate)• It is of interest to ask (probably before we
attempt to fit the parameters of s!):– Is there any signal present at all?
• In frequentist statistics this is again done via hypothesis testing. The hypothesis now is called the null hypothesis (‘null’ from Latin for ‘nothing’):– “Suppose there is no signal at all.”– and test what follows from this.
NASSP Masters 5003F - Computational Astronomy - 2009
Testing the Null Hypothesis - details1) Gaussian data, U=χ2:
– Construct the survival function (SF).• Degrees of freedom?
– Depends whether we fit the background or not.– Suppose we have Mb and Ms.– If background fitted, υ=N-Mb.– If not (in this case need to know the background from other
information), υ=N.
– From the set of measurements yi, calculate
– From the SF read off that value of probability which corresponds to Umeas.
• That is the probability that background alone would generate >=Umeas. NASSP Masters 5003F - Computational Astronomy - 2009
N
i
ii-byU1
2i
2
Meas Note ONLY include background!
Testing the Null Hypothesis – details cont.2) Poisson data, U=“χ2”:
– The PDF, therefore the SF, are not known for the Mighell statistic.
– However the PDF and SF for the Pearson statistic are identical to χ2.
Use Pearson statistic for Poisson hypothesis testing.
3) Poisson data, U=-L:– PDF and SF not known.– But one can compare two models via the Cash
statistic. (Cash W, Ap J 228, 939 (1979).
NASSP Masters 5003F - Computational Astronomy - 2009
The Cash statistic
• This is only valid providing the null model can be obtained by some combination of signal parameters.– This implies that one of the signal parameters
will be an amplitude (ie, a scalar multiplying the whole signal function).
– It also ensures that
NASSP Masters 5003F - Computational Astronomy - 2009
nullbestfit2 LLC
nullbestfit LL 0Chence
The Cash statistic• Cash showed that the PDF of C was the
same shape as that of χ2, but with υ=Mfitted.
• Note that this is rather different from the usual p(χ2
), for which υ is approx. equal to the number of data values N.
NASSP Masters 5003F - Computational Astronomy - 2009
Incomplete gamma functions - advice• Recall the survival function for χ2 is
– The incomplete gamma function can be calculated via scipy.special.gammainc.
• It is very small values of P that we are interested in however – ie where Г(υ/2,U/2)/ Г(υ /2) becomes close to 1.
• In this regime it is better to use the complementary (means, 1 minus) incomplete gamma function:– scipy.special.gammaincc <– note 2 cs.
– But NOTE the definition carefully.NASSP Masters 5003F - Computational Astronomy - 2009
2
2,21,
UUP
NASSP Masters 5003F - Computational Astronomy - 2009
General problems with fitting:
• When some of the θs are ‘near degenerate’.– Solution: avoid this.
• When several different models fit equally well (or poorly).– Solution: F-test (sometimes). Supposedly
restricted to the case in which 2 models differ by an additive component.
NASSP Masters 5003F - Computational Astronomy - 2009
Degenerate θs
Data
Model: two closegaussians – 2 par-ameters: the ampof each gaussian.
Valley in U is long and narrow. Manycombinations of θ1 and θ2 give about asgood fit; parameters strongly correlated.
Overview of the grand plan
NASSP Masters 5003F - Computational Astronomy - 2009
FrequentistFrequentist Bayesian
T B D…
Bayesian
T B D…
χ2
υ=N-Mχ2
υ=N-M
Null H
Poisson -LPoisson χ2Gaussian(χ2 -L)
GOF
Uncert
Fit
No formula(MC)
No formula(MC)
UPearson
υ=N-M
UPearson
υ=N-M
χ2
υ=Nχ2
υ=NCashυ=MCashυ=M
UPearson
υ=N
UPearson
υ=N
Minimize χ2Minimize χ2 Minimize -LMinimize -LMinimize UMighellMinimize UMighell
E=2H-1E=2H-1 E=-F-1E=-F-1
E=2H-1E=2H-1
Flowchart to disentangle the uses of χ2:
NASSP Masters 5003F - Computational Astronomy - 2009
Minimize χ2 to get best-fit θ.
Test the hypothesis that it is.Test the Null Hypothesis:
Compare to theoretical χ2 surviv-al function (num deg free = N).
P<Pcut?
Yes – there is a
signal
No – no signal.
P<Pcut?
No – model is
good.
Yes – model is
bad.
Compare to theoretical χ2 surviv-al function (num deg free = N-M).
Is there any signal at all?
Is the model an accurate des-
cription?
Decide on a cutoff probability Pcut.
Calculate χ2 for the best fit θ.
Decide on a cutoff probability Pcut.
Calculate χ2 for θ= bkg values.