You have almost answered the question. Kalman claims that its linear filter is optimal if the filter fully fits the linear model and all of the noise sources are white Gaussian with known statistics. If the model does not correspond to real process (A is wrong), but the Kalman filter fits the model, then you may expect for some well-known effects such as divergence, instability and bias. If you do not know exactly the noise statistics or noise is non-Gaussian, then the output may be more noisy or more biased.

First the model is always wrong. You can never model the reality perfectly, and this why a statistic filter is proposed.

Now, in theory there are two conditions that need to be hold in order to guaranty the optimal state determination, first the state and the measurement should be gaussian and second the luenberger observer solves the minimization of the euclidean norm for the estimation error. The last condition involves the controllability for A in the Luenberger Observer and translates as the observability of the system.

In conclusion, in practical implementations the only condition that needs to be hold is the system controllability, and the model uncertainty will be always there.