I agree with Andrew, as long as it is equal between your treatments it is less important. Given you are studying drought tolerance smaller pots will allow for more rapid drying and may cause root binding potentially effecting the applicability of your findings to field conditions.
I agree with Andrew-Pegman and Ryan William Orr. It is important to select the desired sized pot for tree crops as they are having long root systems. Based on the crop species you can select the pot size.
I guess it matters to some extent- smaller sized pots can constrain growth in some plants. A study by Poorter et al have argued that on average biomass increased by ~43% by doubling the pot size and based on various lines of evidence they suggest "..appropriate pot size is one in which the plant biomass does not exceed 1 g L–1".
But, like others have mentioned as long as they are equal between treatments it should help minimize error, which also includes but not limited to uniform volumetric soil mixing among treatments, and planting saplings at similar inital pot depths, etc.
Here's the link to the paper by Poorter et al, "https://www.publish.csiro.au/fp/FP12049"