my professor told me that DCT and DWT is not linear Transform because Cosine and Wavelet is not linear function between i found all papers said that it is linear , how i can answer him ?
Tell your professor that he's full of horsesh*t. DCT is a defined as a linear transform because it has the form
X = L . x
where L is the fourier (cosine) transform matrix, x is the signal vector and X is the transformed vector. The components of X give the coefficients of x = X0 + X1 coswt + X2 cos2wt + X3 cos3wt + ......
Cosine is a non-linear function, but keep in mind what you are doing when you carry out the cosine transform: you are representing a function with a _linear_combination_ of cosines. The transform satisfies all the requirements for linearity, i.e.
X1 + X2 = L(x1 + x2) = L(x1) + L(x2)
a . X = a.L(x) = L(a.x)
0 = L(x) if and only if x = 0
The coefficients of the original signal are multiplied by cosine functions then summed. Thus dct(ax+by)=a dct(x)+b dct(y).