This post summarises some R modelling tips I picked up at AMPC2011.

I got some tips from a tutorial on parameter estimation put on by Scott Brownfrom the Newcastle Cognition Lab. The R code used in the tutorial is available directly here or from the conference website.

The main tips I took from the tutorial were:

- Consider using BIC as a model comparison criterion.
- When modelling reaction times, consider modelling data as a mixture model of two processes. One process is the main process of experimental interest and another is a secondary process that otherwise contributes noise. The secondary process is used to capture what would otherwise be outliers that flow, particularly, from very slow reaction times observed when participants get distracted. Probability assigned to the two processes can be specified a priori based on knowledge of the experimental phenomena. In the specific example that Scott showed, the outlier process was given a probability of 0.03 and this was treated as a uniform distribution between 0 and the trial time-out time.
- Consider transformations model parameters for the purpose of estimation and then converting the transformed parameters back to their original scale. This can facilitate estimation and also assist in enforcing psychologically meaningful constraints on parameter values (e.g., ensuring that asymptotic reaction time is greater than zero).
- The combination of the R function
`optim`

and a custom created objective function, such as a minus log-likelihood function provides a powerful tool for parameter estimation of custom models.

- Scott Brown's tutorial includes an example of this.
- Ajay Shah has an example
of writing a likelihood function and then getting a maximum likelihood
estimate using
`optim`

. - Benjamin Bolker has great material available on the web from his book Ecological Models and Data in R. PDFs, Rnw, and R code for early versions of the chapters are provided on the website. Chapter 6 (likelihood and all that) , 7 (the gory details of model fitting), and 8 (worked likelihood estimation examples).
- Brian Ripley has a set of slides on simulation and optimisation in R.
In particular it provides a useful discussion of the various optimisation
algorithms available using
`optim`

.

- I also got the impression that it will soon be time to dive into WinBUGS.

I definitely support using Bayesian methods (ideally through some BUGS variant) to fit mixture models to reaction times, especially if you're currently hardcoding the mixture parameters. optim() is great, but the output can be impenetrable in my experience if your likelihood function doesn't have maximal/minimal value.

ReplyDeleteJohn, thanks for the tips.

ReplyDeleteThere is also nlminb, which does the same thing as optim but using yet another quasi-Newton algorithm. It's used by the lmer package for fitting mixed models, among other things.

ReplyDeleteOne nice feature that nlminb has is that if you supply a gradient function, it will check that the returned value at the solution is the same (to within a tolerance) as that computed via numerical differencing. I've found this very useful to make sure I didn't get the gradient wrong, which can happen easily with complicated objective functions. On the other hand, nlminb won't compute the Hessian matrix for you, which optim does.

HI Hong, I'll check it out. Thanks.

ReplyDeleteHi Jeromy,

ReplyDeleteIts Lee from Newcastle cognition lab. Finally found time to check out your blog. Hope all is well and I understand you have submitted, well done.

As for BUGS: I have done a lot of work with forgetting curves in BUGS and continue to use it for a range of other applications including signal detection and LBA. I think your RT work is a natural candidate for BUGS and your model selection issues lend themselves to posterior based model selection techniques such as BICM AICM and or DIC. These selection techniques allow for functional form complexity to be analyzed under different assumptions. My JMP paper 55 (2011) 25–35 outlines these issues.

I think we have a great opportunity to collaborate on some projects, I have being playing with the models you outlined in the OMPC talk (which was impressive by the way) and have them implemented at a hierarchical level (both with explicit covariance parameters [mvnorn hyper prior] and independence assumed)for the simplifying case of binomial data. We have a wealth of experience with RT fitting in Newcastle and would love the chance to work with you on these projects. Let me know if there is anything I can do to hasten your understanding of BUGS (it is buggy [pardon the pun] and I have figured out lots of tricks that will save you some time).

cheers

Lee

Hi Lee,

ReplyDeleteThanks for the comment. I'll send you an email to discuss. Thanks for the warm welcome.

Such collaborations sound very promising.