3 phase, 3 wire(winding star/delta) as neutral is not connected the unbalanced current would not find path and the shifting on neutral would increased the voltage of lightly loaded phase, and would decreased the voltage of heavily loaded phase. In 3 phase, 4 wire as unbalanced current are getting path through neutral, no shifting, and voltage of all three phase would remained balanced. Would be more clear by drawing the voltage and current diagrams, for connected/isolated neutral.........
3 phase, 3 wire(winding star/delta) as neutral is not connected the unbalanced current would not find path and the shifting on neutral would increased the voltage of lightly loaded phase, and would decreased the voltage of heavily loaded phase. In 3 phase, 4 wire as unbalanced current are getting path through neutral, no shifting, and voltage of all three phase would remained balanced. Would be more clear by drawing the voltage and current diagrams, for connected/isolated neutral.........
Are the motor windings Wye or Delta connected? 3 phase loads should ALWAYS be delta wired to avoid high neutral currents. A schematic or wiring diagram of the motor would be helpful so we can better understand your issue.
An induction motor is generally connected in delta while running, due to several advantages. If so, there is no neutral leading to the motor. Then the imbalance in currents could be because of the single phasing faults in the motor windings or connection. If it is the overload relay functioning as a single phasing preventer that is getting tripped, then the fault could be in the motor unless there is a three phase supply problem (loss of a phase etc.). If the earth leakage breaker or the RCCB is tripping the circuit, then the unbalanced current is going elsewhere or is leaking. Otherwise the motor windings or whatever is connected to the source is taking the current ant returning it. I hope this was the query raised.
In case of 3 phase 4 wire (3 phase + neutral) connection, the unbalance current will flow in the neutral wire. In 3 phase star connected without neural, the unbalance current will lead to increased neutral voltage as compared to the ground voltage. In 3 phase delta connected, the unbalance current will flow in the delta winding of the motor and lead to more heating of the motor winding.
@Anirbid Sircar: What about a true unbalanced load, that is, a problem "after" the motor? :-) In this case, I find the answer of Bhupendra Desai most helpful (as he said, with a diagramm it would even be more helpful).
Second: Not to oversee the aspect of an unbalanced supply as mentioned by Sudha Balagopalan - a problem "before" the motor; result also is a shift of the neutral, though...
There are two types of imbalances. Negative sequence and zero sequence. The negative sequence does not need a neutral connection. So you can have negative sequence imbalances in a systems without neutral and even in delta connection.
Zero sequence current can be also produced by parasitic capacitive currents is the machine is fed by a power converter operating at high switching frequency. Differential earth leaking protections are likely to trip in this case.
If there is not a fault in the winding and a converter is not used, check the power supply. If the power supply is unbalanced this will produce negative sequence currents in the machine and pulsating torques.
It could simply be that the inductance of the motor is so large that there is a significant phase delay in the current and this is enough to trip your circuit breaker (?) at startup for example . If it is a circuit breaker that is giving you troubles you can try changing to a slower acting one.
Measure the unbalanced current of all three phases, with scale draw three vectors line 120degree apart, from same point representing three vector current, taking length scale to current, as all three phase voltage would not changed. First find resultant vector of two, and using resultant with the third find the vector, is the current flowing through neutral, of 3phase four wire. When neutral is isolated(no neutral current) for 3phase 3wire case, the voltage of heavily load phase would reduced accordingly, hence measure the unbalanced voltage reference to star point, draw unbalanced voltage diagram, would gave you the star point to neutral(zero volt) unbalanced(voltage), responsible for unbalanced phase voltage, same could apply to delta, difference wise changing phase to line.