Keep it Simple Stupid – On the Effect of Lower-Order Terms in BIC-Like Criteria

Abstract—We study BIC-like model selection criteria. In particular, we approximate the lower-order terms, which typically include the constant log ∫ √ det I(θ) dθ, where I(θ) is the Fisher information at parameter value θ. We observe that the constant can sometimes be a huge negative number that dominates the other terms in the criterion for moderate sample sizes. At least in the case of Markov sources, including the lower-order terms in the criteria dramatically degrades model selection accuracy. A take-home lesson is to keep it simple. I