It is possible for the rate of reaction to decrease with increasing substrate concentration. This could be due to substrate inhibition, in which a high concentration of substrate inhibits the reaction, although various experimental problems could also be involved (contamination of the substrate with an inhibitor of the reaction, or interference of the substrate with the measurement, for example). If substrate inhibition is occurring, you should still be able to find a lower range of substrate concentrations at which the rate increases with increasing substrate concentration, if your experimental system permits making rate measurements at lower substrate concentrations than the ones you've been using so far.

If substrate inhibition is occurring in your system, calculating the Km of the substrate requires a modification to the Michaelis-Menten equation to take substrate inhibition into account.

If you are measuring the fraction of substrate converted to product, rather than the concentration of product formed, you have to multiply the fraction by the substrate concentration to calculate the rate. Failure to do this will appear incorrectly as severe substrate inhibition.