A suggestion : If you can model the system as a network a possible solution differentwould be to use circuit simulation , for example Pspice .It contains quite a few functions including FT (data exportable to Excel.Scilab etc) for post processing .It is also nice to visualize the different variables in time and make alterations. Another possibility is to use Scilab(or maybe SAGE) directly to obtain transfer polynomials and plots.
If you have a discrete-time system (such as a digital filter), transfer function of that system can be analytically found by calculating the Z-transfrom of the impulse response of the system. This can also be plotted in Matlab using the command "freqz". This command returns the frequency response based on the coefficients of the digital filter. I would suggest you check out the "freqz" command.