Did you try matlab commands obsv(A,C), ctrb(A,B).

As far I remember has another methods can be used to calculated the observability of LTI system. I recommend you to see this article :

Title: Controllability and Observability of Linear Dynamic Systems Revisited

Abstract:

A modification of well-known controllability and observability tests using Lyapunov equations and Gramians yields a reliable tool for controllability and observability testing for linear continuous and discrete time systems, as well as an alternative procedure to determine controllable, observable and minimal system realizations.

May it help you.

Best Regards.

Controllability and observability gramians can be computed

Wc = gram(sys, 'c') and Wo = gram(sys, 'o') in Matlab. The gramians are computed by solving the Lyapunov equations:

A*Wc + Wc*A' + BB' = 0 and A'*Wo + Wo*A + C'C = 0

If you have system matrices A, B, and C, Wc and Wo can be obtained as

sysc=ss(A,B,[ ],[ ]); syso=ss(A,C,[ ],[ ]);

Wc=gram(sysc,'c');Wo=gram(syso,'c');

nc=rank(Wc); no=rank(Wo);

One possible solution to your question may be as follows:

Simply convert the state space model to its equivalent transfer function using the command ss2tf and then decompose both numerator and denominator of the obtained transfer function using the command roots.Finally, the system is observable if no pole zero cancellation occurs in the transfer function model.