A Teaching Note for Model Selection and Validation

The model selection problem is always crucial for any decision making in statistical research and management. Among the choice of many competing models, how to decide the best is even more crucial for researchers. This small article is prepared as a teaching note for deciding an appropriate model for a real life data set. We briefly describe some of the existing methods of model selection. The best model from the two competing models is decided based on the comparison of limited expected value function (LEVF) or loss elimination ratio (LER). A data set is analyzed through MINITAB software.


INTRODUCTION
I t is often believed that only one reasonable model may be constructed for a given decision making problem in market research.In empirical studies, one may be able to construct alternative models consistent with the hypothesis of the objectives (Wilson (1979), Bass (1969) and Carmone and Green (1981)).Many criteria may be used to compare quantitative marketing models.These criteria include such things as underlying assumptions, data requirements, and theoretical implications (Larreche and Montgomery (1977), Little (1979), Narasimhan and Sen (1983) and Rust (1981)).
Many methods for comparing structural forms of quantitative models have been proposed in the last three decades.These methods may be classified in terms of methodological emphasis as supermodel methods (Atkinson (1969), Johnson and Kotz (1977)), cross-validation methods (Mosteller and Tukey (1968), Stone (1974)), likelihood methods (Cox (1962), Akaike (1974)), or Bayesian methods (Smith (1973), Blattberg and Sen (1975), Rust (1981)).The model validation is a procedure of model comparison, estimating the competing models and then compiling error statistics on the other (Mosteller and Tukey (1968), Stone (1974)).The interrelatedness of the above categories of methods is exemplified by the fact that using Bayesian arguments Smith and Spiegelhalter (1980) obtained criteria closely related to those of Akaike (1974) and Schwarz information criterions.
This article examines the case in which two or more probability models are to be compared and decide how to choose the best among the competing Mathematical Journal of Interdisciplinary Sciences, Volume 1, Number 2, March 2013 models.In the next section, we present the problem.In section three some useful results and model inferences are discussed.

The pRObleM
Consider the following data set (n = 40): 85, 90, 92, 5, 10, 17, 54, 55, 58, 55, 58, 32, 33, 32, 82, 68, 34, 36, 92, 102, 103, 106, 146, 124, 142, 195, 65, 66, 68, 54, 55, 58, 143, 151, 158, 195, 114, 114, 116 and 57 (see Bain and Engelhardt, 1991).Assume that this is not a time series data, then as a preliminary investigation, one can prepare a frequency distribution or a histogram or even a sample ogive to get an assessment about the shape of the data.The ogive and histogram generally provide some indication as to the type of distribution that will model the data.For the above data set we prepared a histogram and ogive (omitted here) and concluded to be a positive skewed distribution (the MINITAB sample statistics for the above data are mean = 83, median = 68, mode = 55, sd = 47.58 and skeweness = 0.614).Instead of drawing conclusions from descriptive statistics, one can directly check the normality of the data and proceed.On our further investigation, we found that the prospective models for the above data set are Weibull and Gamma distributions (see Table 2 for p-values).In the next section, we present some useful results to decide best alternative model through a data set.

SOMe USeFUl ReSUlTS
Suppose X be the decision variable under study and f(x) the corresponding probability density function, then the limited expected value (LEV) function or the expected loss eliminated is defined as where d), that is the mean of a random variable censored at d.The other quantity of interest is the loss elimination ratio (LER) which is the ratio of the expected loss eliminated to the expected value of X. that is LER = E[X;d] /E(X), provided E(X) exists (Klugman et.al. 2008).Note that E[X;d] always exists.A quantity which needs to be compared with E[X;d] is the empirical limited expected value (ELEV) function for a sample as In life testing experiments d may be sometimes called the truncation time or the censoring time.In this respect another important characteristics of life time models is the mean residual life (MRL) at age Then E[X;d] and e(d) are related through the equality A plot of e(d) can also give some indication as to the type of distribution that will model the data.A very important problem associated with model selection is fitting of probable model to the data.In order to fit a model to a data, we need to estimate the parameters.This can be done using various methods like percentile matching, method of moments, minimum distance method, minimum chi-square method and maximum likelihood method etc.The first two are crude method and may be easy to implement but produce inferior estimates.The other three methods are more formal procedures with well defined statistical properties.Although they produce reliable estimators, they can be complex sometimes.Many often we need to employ some numerical procedures to get the estimators.For the particular problem discussed above, we use maximum likelihood method to estimate the parameters.In Table 1 we provide the MINITAB output of the model parameters estimates and Table 2 the corresponding Goodness of fit summary for the above data set.
In Table 2, we present the summary of Goodness of fit tests and their conclusion.The Table clearly shows a preference for Weibull distribution followed by Gamma distribution.It is possible to judge the best based on the p-value concept also.A p-value is a measure of how much evidence we have against the null hypotheses.It also measures the consistency by calculating the probability of observing the results from the sample of data assuming the null hypothesis is true.In the above situation, the p-value concept may not be sufficient to arrive at a final conclusion as the same models have same p-value.Therefore, to decide between the two we now use a semi parametric approach based on the agreement between the empirical and fitted LEV function as given in equations ( 1) and (2) above.These functions are crucial to the determination of the effects of coverage modifications.For the example discussed above, except Weibull and Gamma we rejected the other models because of their poor performance on likelihood measures.As Weibull and Gamma are the two competitors (see Figure 1), we computed the LEV functions for both distributions to make the final decision.In Table 3, we present those empirical estimates.From the table it is seen that for Weibull distribution the empirical expectation is closer to the LEV and hence is a better model for the data.Also the LER of Weibull is much less than that of the Gamma distribution for every d as the estimated E(X) of Weibull and Gamma is respectively 352.87 and 82.71.This again supports Weibull being the best model for the data set.
The table also includes the empirical distribution function which may be used for manually checking goodness-of-fit test.The above method works even if there are more than two competing models.The use of MRL function can also give a similar result.Since MRL and LEV functions are related it is sufficient to do only LEV function calculations.The other methods like minimum distance estimation, Bayes factor approach etc can also be used for identifying the best out of many alternatives.Generally these methods are very tedious.
For accepting any model as providing a reasonable description of the decision process, we should verify that E[X;d] and E d n ( ) are essentially in agreement for all values of d.It is because as d Mathematical Journal of InterdisciplinarySciences, Volume 1, Number 2, March 2013

Table 1 . Ml estimates of Distribution parameters
Mathematical Journal of InterdisciplinarySciences, Volume 1, Number 2, March 2013

Table 2 . Goodness of Fit Test
Mathematical Journal of InterdisciplinarySciences, Volume 1, Number 2, March 2013