IVI can definitely be used as a measurement of the ecological importance of the fern species.
Regarding modification of the equation for calculating of IVI, it is not advisable as the result will totally change. Moreover, there should be a proper logic behind deriving a new equation from the existing one. If you can clear your objective for modifying the equation, it would be easier to help you regarding the second part of your question.
I consider there are enough scientific references to support the application of IVI in ecological assessment, so it can be used with confidence. I am sharing an example (link) of our study that supports the above argument: http://www.sciencedirect.com/science/article/pii/S0304380005000062
GBH is specific mostly for tree and shrub species. In case of herbs or ferns many of the species are hardly tall upto the breast height. Hence, the term 'GBH' I personally feel very much ambiguous and irrelevant as far as ferns are concerned.
The IVI formula i have used on tree and shrubs data involves the inclusion of basal area. How possible will it be to have the basal area of ferns. If they are tree ferns perhaps that will be easier.
You may to investigate the use of Relative density and Relative diversity. These two parameters are used in finding Family Importance value (FIV).
For relative dominance, foliar or crown cover would be almost equivalent to basal area, and is therefore not as big an issue as you might think. Relative density could be problematic, but not greatly so, as an individual is sometimes represented by a single frond, and other times a rosette. Relative frequency is dependent on plot size, and converges with relative cover as the plot approaches zero.
All these indices are imperfect attempts to quickly assess quantity as either how much a plant dominates a resource like sunlight (cover), biomass, or reproductive potential. Crown cover is often a standard currency to compare vegetation studies irrespective of growth habit, but I understand that, perhaps, it isn't always as reproducible among observers as density and frequency would be. The best option would be to collect preliminary data using all three indices and understand their limitations, then go with whatever is practical.
is is due to the fact that the Relative dominance (sum of basal area for a species /total sum of basal area for all the species) for ferns and grasslike plant may tend to 0 that we rather canceld it in the equation, as Guadalupe et al., 2005 did?
As mentioned in many books of ecology, Herbaceous plants are measured using the 1 meter squared quadrant sampling methods. In mexico when I was studying bachelor, we did some field sampling using the coverage of the plant instead of Basal area.
Here I provide you a pdf similar to what I did.
I also don't recommend you to change the formula because most articles and books have a similar formula shown on the pdf file that I sent you and the one that you wrote.
If in your case, the herbaceous community is very distinct from others, then you will need to specify and justify the change of formula (equation).
Regarding the calculation of Relative dominance of herbaceous species , basal area is definitely less but not zero. You can just measure them with caliper. As the number of herbs is generally high in quadrats , when you will multiply the measured basal area with the number of individuals , the figure won't be negligible. In this way you can calculate the IVI without modifying the original formula. We followed the same procedure during a research work in home-gardens in India.
Whether herbs or trees the formula for calculation of dominance/IVI is same. For formula modification you may refer any standard literature or research.