Prediction-driven approach to model selection using feature selection and nonrandom hold-out validation

• In this paper, we address the forecast problem and propose a prediction drivenapproach to model selection. Furthermore, we investigate to what extent dierentprediction questions lead to dierent \best" models. Most of the studies in theeld, take an inference-driven approach to select the best model and project theresults to the future population. In contrast, we take a prediction-driven approachfor both selection criteria and validation sample. We use feature (variable) selectionand nonrandom hold-out validation to select the model with the highest predictiveperformance in an out-of-sample prediction manner. The application of interest iscar type choice modeling using the Swedish car eet data. We introduce two dier-ent types of model selection criteria: maximum likelihood which is the conventionalmethod of model selection, and root mean squared error of the prediction quantityof interest. We compare the best models obtained by dierent criterion functions.The results show that the \best" model for the purpose of prediction depends con-siderably on the prediction question of interest. Moreover, when the objective isto predict a sub-section of a population such as the total share of ethanol cars,maximizing log-likelihood is not the most accurate model selection criterion.

Nyckelord

model selection

nonrandom hold-out validation

feature selection

out of sample prediction

car type choice models

Transportvetenskap

Transport Science

Publikations- och innehållstyp

vet (ämneskategori)

ovr (ämneskategori)