brms vs custom Stan code

[wlandau]

The implementation options we have considered are:

1. brms: use brms and follow the vignettes by Novartis/Bjorn.
2. rstantools : write a small number of Stan models by hand and pre-compile them in the package using rstantools.

Our consensus is to keep going with (1) until we hit a wall, then go to (2) if needed.

Here are some advantages/disadvantages of brms. Please let me know if there are more I missed.

brms advantages

• Already a lot developed by Bjorn et al.
• Stan code already written.
• emmeans integration to get the reference level right.
• Easier to accommodate a wide variety of fixed effect parameterizations.

brms disadvantages

• Need to recompile the model almost every run.
• Hard to explain, the details of the parameterization are a black box.
• Hard to specify informative priors because of the different fixed effects parameterizations.
• May require an untrustworthy hacky mix of custom Stan code injected into a brms Stan model file.

[wlandau]

From my perspective, the biggest factors are emmeans post-processing, pre-compiled models, and conditional means priors (#40).

(Related: I am working on a package to pre-compile CmdStan models in other packages, which would free us from the feature lag and other issues in rstan/rstantools.)

[wlandau]

Competing priorities risk splitting us into 2 different directions, and it would be great to avoid that outcome. On reflection, I think the only true deal-breaker for brms would be CMPs (#40), and if we require the parameterization to already match the prior, we can work around that limitation.

[wlandau]

Today, we decided as a group that it would be worthwhile to create a separate non-brms MMRM package to handle conditional means priors (#40) and to allow us to use a pre-compiled Stan model. We also plan to release brms.mmrm as is, with disclaimers that it is still in the development phase and we have not validated it.

[wlandau]

From #40 (comment), we could actually implement CMPs fairly straightforwardly as long as we commit to the identity link (or transform the m components individually).

[wlandau]

By the way, the instantiate package is now open-source: https://wlandau.github.io/instantiate/

[wlandau]

To summarize: we have seen several developments in favor of brms:

1. With our identity link function, @andrew-bean noted that we can put independent priors on linear combinations of regression parameters quite easily: Conditional means priors #40 (comment)
2. As mentioned in Methods vignette #36 (comment), I figured out how brms parameterizes both the residual standard deviations and the regression coefficients.
3. mixtures as priors for brms Novartis/RBesT#12, which is tightly integrated with brms, could seriously help us with MAP priors.

I know I promised to start a custom Stan implementation for next meeting, but given these new developments, I am having second thoughts. I feel much better about the future of brms.mmrm, and I think it would be worth focusing on it until it is feature-complete (all the current GitHub issues done), rigorously validated, and ready to deploy to production in our respective companies. After that, we may still want a pre-compiled Stan model, but it will be less of a risk because brms.mmrm will serve as a reference implementation in all respects.

Is everyone in the group okay with this direction? Okay to finish all the features and validation of brms.mmrm before potentially moving on to custom Stan code? cc @kkmann, @chstock, @yonicd, @andrew-bean, @kirangauthier

[wlandau]

We have decided to use brms.