writing your own code for mesh generation is not something I would recommend. it is very complex and will be much less general than existing (commercial) mesh generators such as those from Abaqus, Ansys, Hyper Mesh, etc. Alternatively you could also try to employ Gmsh, Tetgen, Netgen which are free alternatives. This you could then couple with a spectral element solver you could program in MATLAB. Considering the spectral element method I would recommend two books which are very helpful
Pozrikidis, C.: Introduction to Finite and Spectral Element Methods using MATLAB, Chapman and Hall/CRC, 2014 , 830
Karniadakis, G. E. & Sherwin, S. J.: Spectral/ hp Element Methods for Computational Fluid Dynamics, Oxford Science Publications, 2005
These monographs contain all details regarding SEM.
For general comments about programming FEM in MATLAB have a look at
Ferreira, A. J. M., Gladwell, G. M. L. (Ed.) : MATLAB Codes for Finite Element Analysis, Springer Verlag, 2009
writing your own code for mesh generation is not something I would recommend. it is very complex and will be much less general than existing (commercial) mesh generators such as those from Abaqus, Ansys, Hyper Mesh, etc. Alternatively you could also try to employ Gmsh, Tetgen, Netgen which are free alternatives. This you could then couple with a spectral element solver you could program in MATLAB. Considering the spectral element method I would recommend two books which are very helpful
Pozrikidis, C.: Introduction to Finite and Spectral Element Methods using MATLAB, Chapman and Hall/CRC, 2014 , 830
Karniadakis, G. E. & Sherwin, S. J.: Spectral/ hp Element Methods for Computational Fluid Dynamics, Oxford Science Publications, 2005
These monographs contain all details regarding SEM.
For general comments about programming FEM in MATLAB have a look at
Ferreira, A. J. M., Gladwell, G. M. L. (Ed.) : MATLAB Codes for Finite Element Analysis, Springer Verlag, 2009