Roberts DR, Bahn V, Ciuti S, et al (2017) Cross-validation strategies for data with temporal, spatial, hierarchical, or phylogenetic structure. Ecography 40:913–929. doi: 10.1111/ecog.02881
I'm sorry but you should download the package source from CRAN (https://cran.r-project.org/src/contrib/gbm_2.1.4.tar.gz), and copy and edit the function gbm(). Maybe you should add a further parameter to the function which defines the CV folds. Then load the edited function with source().
Or if only repeatability is your aim, set.seed() before gbm() might (I'm not sure) eliminate stochasticity.
Let me self-reply my question (I can do that thanks to Jane Elith and Peter Buston). Custom folds are important since they enable you to build mixed BRT models. One way to do it is to use BRT functions published as a supplement to
Elith J, Leathwick JR, Hastie T (2008) A working guide to boosted regression trees. Journal of Animal Ecology 77:802–813. doi: 10.1111/j.1365-2656.2008.01390.x
Instead of fold.vector = NULL use additional column in your data frame and make reference to that column in fold.vector. That column should contain the custom fold number (for instance, if you have 10 folds, an integer from 1 to 10) for each of the rows/cases.