My system is not equipped with a potentiostat, so I was wondering if it will be okay to use a multimeter to measure the resistance. I noticed that at no current, the resistance increases so I don't get a stable value for resistance either.
Your multimeter uses its own current to measure resistance. That's why multimeters have a cell or two in them. The meter is measuring the current flowing through the load, to determine the resistance of that load. In effect, the meter is behaving like an ammeter, a galvanometer, and what it reads depends on knowledge of the voltage source it uses.
So, introduce any other current in that circuit, any current generated from a source independent of (and unknown to) the multimeter, and what can you expect that meter to read? Could be anything. Could be higher or lower than what it should be, depending on the polarity of this other source.
If you apply a current with one system, then you can measure the voltage drop with the multimeter. That won't directly give resistance in most cells, but if you then measure the voltage drop with no current applied (or a different current applied) you can use the measured voltage differences and the known current(s) to calculate resistance. I should note that this technique really only works for currents and cells that are primarily resistive, and not capacitive (e.g. fuel cells/batteries, or actual resistors). It also won't work if the response is non linear (for an electrochemical system, try and keep the ddV under 20 mV...and yes, I mean ddV, you want the 0 current and applied current measurements within ~20 mV of each other, but each individual measurement could be multiple Volts).