Is there any way to find the potential of a nonlinear dissipative system? For example the Chua oscillator is a double well system, but I don't know how to find the potential from the system equation.
It is not always possible even in the Hamiltonian case. Generally, You should rewrite your system as a second-order ODE, say a*d^2y/dt^2+b*dy/dt + c(y)=0, where y is some of function of system variables (in the simplest case - one of them). Then the potential is given by integral \int c(y)dy. I'm not sure that this procedure will be succesful with the Chua oscillator.
For example in Lagrangian mechnics/mechanics in general there exist a wider concept of "a generalized potential" and "dissipation function".
If we find a function V(q,q',t) s.t in Lagrangian formalism the generalized forces are
Q_i=(d/dt)(\partial V / \partial q_i')-\partial V/\partial q_i
and define the Lagrangian L(q,q',t)=T(q,q',t)-V(q,q',t) then Euler-Lagrange eq.s
for this function are the correct eqs. of motion when we substitute it to E-L_eqs of motion: http://en.wikipedia.org/wiki/Euler%E2%80%93Lagrange_equation
A classical example would be to find a Rayleigh dissipation function
for example for a simplest example a damped 1-Dim spring system
L:s eqs of motion are
x''+(c/m)x'+(k/m)x=0
and a dissipation function should be D=-(c/m)*(d/dx)[(1/2)*(x')^2]
Well its a quick Friday night answer but hope it answers at least to something...
In fact chua oscillator is totally unknown to me so don't have clue is it even a mechanical system...But if you want clear answers you have to ask clear questions :-).
If you want send me the equation(s) governing the dynamics of Chua oscillator.
You can send me the set of eqs. or if you have time try to reduce it to one DE.
I could dedicate today 1/2 hour to look for a generalized potential or give a statement that such can not be found.