In my opinion the best way is to use a FEM approach. Otherwise you can use the classicalapproach proposed by the standard (see pdf) with the difference that you should divide the teeth into many axial sections and consider each of them as a spur gear and calculate the stiffness as the mean value.

Franco

For all I know, there is no  formula to calculate mashing stiffness directly, and there are so many formulas with uncertain parameters which you won't know what it is. So I think the most efficient way is to do an experiment.