Lecture 29 Notes: Models vs. Data (II)

Summary of Confidence Limits on Estimated Model Parameters After computing the best-fit values from χ 2 minimization, the model is quantitatively tested for consistency with the data. If the model is consistent with the data, next step is to compute the confidence limits on the model parameters. The following section summarizes the methods which ignores the prior information in finding confidence limits as discussed in the class. There are two reasonable approaches to find the confidence limits on the parameters estimated by minimization of χ 2 : Approach #1: This is given in the book Numerical Recipes in C section 15.6. Refer to page 697 of the book, there is a prescription to calculate the confidence interval in more than one parameter using the confidence ellipses (or ellipsoids in higher 2 contours in v-dimensions, where v are dimensions) which can be plotted using the χ approx the number of fitted parameters whose joint confidence interval is to be determined. The expression for quadratic approximation for χ 2 is given by: T 2 2 χ approx = χ best + ( θ - θ best ) H ( θ - θ best ) where N data H mn =2 ∑ i=1 1 ∂Yi ∂Yi σ i2 ∂θ m ∂θ n Note: Above expression for hessian is given on page 682 in Numerical Recipe in C with an explanation for dropping the second order derivatives. Let us consider the case in which we have 2 parameters of interests, so v = 2 and the 2 χ approx contours are ellipse in 2-D plane of two parameters θ1 and θ2. Steps to be followed to find the confidence limits: • • Choose a desired confidence limit, let us say p = 0.90. Determine ∆χ 2 value using the confidence limit. Since ∆χ 2 follows a chi-square distribution with v degrees of freedom, it can be easily calculated using chi2inv() function in MATLAB or looking up in table given on page 697 of Numerical 1 • receipes in C. If we look up in table, for v = 2 and confidence limit of 0.90, ∆χ 2 = 4.61. Using chi2inv(): chi2inv(p,v)= chi2inv(.90,2) = 4.61. Now for the desired confidence limit, acceptable region is the region inside the 2 2 contour corresponding to χ 2 =χ best +∆χ 2 = χ best + 4.61 . Now, confidence limits on the parameters can be taken as the constant chi-square boundaries. So the parameters θ1, θ2 are adjusted to get the plot of χ 2 contours. The corresponding limits on the parameters can be taken from the plot by projecting the 2 boundary for ( χ best + 4.61 ) on to the two axes θ1, θ2. One obtains slightly different uncertainty ranges if one only considers one parameter at a time instead of pairwise, see Fig. 15.6.4 in Numerical Recipes. In cases where the two parameters are strongly correlated yielding a long thin ellipse, it is better to define the model in terms of linear combinations of the parameters θnew = VTθ, the uncertainties in θnew will be less correlated. If you want to report A practical advantage of this approach is that the uncertainties in the parameters do not 2 is large, agreeing with our intuitive feeling that if the model is shrink drastically if χ best suspect, the parameter values obtained should not be well determined. The philosophical interpretation for this case might be that there is some discrepancy between the models and data that is not due to the parameter values. It is the sensitivity of χ 2 with respect to the parameters which suggests these uncertainty bounds. However, the philosophical 2 close to 0 is pretty clear. In this case, the interpretation is interpretation for the case χ best that 90% of the probability density of P ( Y data θ1 ,θ2 ) lies within the ellipse, allowing the parameters to take any value. Approach #2: In this approach, we start by assuming the data and model are true. The approach is based on the consistency test of the model with the data. We accept the parameter values if they keep χ 2 small enough to assume that the model with the parameters is consistent with the data. • • Choose the confidence limit where we would say model and data become inconsistent, say p = 0.99, i.e. we reject any parameter set θ which gives a χ 2 (θ ) so large that if θ and the model were true it is very unlikely that we would have observed the experimental data actually observed. Determine the χ 2 value corresponding to the chosen confidence limit. The χ 2 value satisfying this condition can be calculated using chi2inv() function in MATLAB. As χ 2 has ν degrees of freedom, we have: 2 χ 2required = chi2inv(P, ν) = chi2inv(0.99, ν) where ν = Ndistinctdata - Nadjustedparameters. Ndistinctdata is the number of Yn(xi)’s, it does not included repeated measurements of the same observable at same knob settings. Often ν is a large number. 2 2 < χ required is acceptable and used to determine the limits on the • The region χ approx 2 two parameters. The two parameters are varied and the contour plot of χ approx is constructed for different values of the parameters. Again the confidence limits on the parameters are taken by the constant chi-square ( χ 2required ) boundaries by projecting it on to the axes of the corresponding parameter for which the limits are desired. If the best fit is not very good, this second approach gives us tighter confidence intervals on the parameter values as compared with Approach 1. It would be wrong to interpret this as a case that parameters have been determined to high precision. The more likely correct philosophical interpretation for this is that the model looks fishy (when fit is not good) and the fishy model can match the data only if the parameters are in this narrow range. 3