Thanks for providing useful information

You can model your system using matlab files or you can build it in form of simulink blocks. The matlab m-files are based on the analytical mathematical model of the system. 

Any communication system is built from the transmitter a receiver and a transmission channel. You model the three components of the system.

Assume that the receiver produces the transmit symbols S(t), the channel has an impulse response h(t), the receiver signal r(t) can be expressed by 

r(t)= h(t)*  S(t) + n(t) , where * is the convolution process and n the noise power.

The role of the receiver is to find  a best estimate of S in spite of distortion by the channel and contamination by noise. Normally equalizers and matched filters are used for optimum detection.

These operation are performed using analytical formulas in the m-file formulation.

In case of Simullink you build all these components using their model in the library of elements and components  or even custom components.

wish you success

Important here to point out that the channel is modeled as an FIR filters.

The book digital communication systems by Glover and Grant helps you much to realize your project.