Neural Networks and Learning Machines by Simon O. Haykin

The textbook of Bishop is one of the best one...in my opinion

http://www.amazon.com/Pattern-Recognition-Learning-Information-Statistics/dp/0387310738

Hope it is useful

I also share Mariano's opinion. On the other hand if you want to cover more general learning topics(which touch NNs), I would also suggest:

1) Foundations of Machine Learning by Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar

2) Understanding Machine Learning: From Theory to Algorithms by Shai Shalev-Shwartz , Shai Ben-David

If this still not satisfy your level of mathematical depth(looking for proofs of universal approximation, etc.), I would suggest that you go for specific papers instead. If you need additional help in finding a specific subject, send me an email and I can point you to several specific papers.

This is an example of computational modeling:

Weakly Connected Neural Networks, Hoppensteadt, Frank C., Izhikevich, Eugene M.

http://www.springer.com/gp/book/9780387949482

For artificial neural networks you can have a look at:

"Introduction To The Theory Of Neural Computation"  A. Hertz, Anders S. Krogh and Richard G. Palmer

and for more biological realistic neural networks you can check the following survey: 

"Neural networks as spatio-temporal pattern-forming systems" by Bard Ermentrout

Please read the following books:

1. M. Anthony and P. Bartlett., Neural Network Learning: Theoretical Foundations, Cambridge University Press, 1999.

2. B. Ripley, Pattern Recognition and Neural Networks,. Cambridge University Press, 1996.

3. R. Rojas: Neural Networks, Springer-Verlag, Berlin, 1996

4. Martin Anthony, Discrete Mathematics of Neural Networks: Selected Topics,  SIAM, 2001.

5. J Heaton, Introduction to the Math of Neural Networks,  Heaton Research Inc., 2012.

Here are some more materials :

http://lia.univ-avignon.fr/chercheurs/torres/livres/book-neuro-intro.pdf

http://math.bu.edu/people/mkon/nnpap3.pdf

Part II of "Soft Computing and Intelligent Systems Design" by Karray and de Silva is really accessible with a good number of worked examples if you're planning on implementing a network. It doesn't contain the proofs for any properties or algorithms, but mentions them with the citations making a lit search easier. It's also nicer since it covers more than just multi-layer perceptions (conventional ANN) but other commonly used architectures both feedforward and recurrent.

The best one has not been written yet. There are many topics which must be formalised to get a good mathematical approach.

Neural Networks and Learning Machines by Simon O. Haykin

The problem with requesting advanced literature on the mathematics of neural networks that is heavy on theory and has little or no place for application is that you are no longer really looking for a textbook. Rather, the kind of book-length literature most likely to contain the cutting-edge algorithms, methods, theory, etc., are e.g., conference proceedings like Advances in Neural Networks. That said, there are some sources you might be interested:

Aizenberg, I. (2011). Complex-Valued Neural Networks with Multi-Valued Neurons (Studies in Computational Intelligence Vol. 353). Springer.

Ivancevic, V. G., & Ivancevic, T. T. (2010). Quantum neural computation (Intelligent Systems, Control, and Automation: Science and Engineering Vol. 40). Springer.

Montavon, G., Orr, G. B., & Müller, K-R (Eds.). Neural Networks: Tricks of the Trade (2nd Ed.). Springer.

Siddique, N. (2014). Intelligent Control: A Hybrid Approach Based on Fuzzy Logic, Neural Networks and Genetic Algorithms (Studies in Computational Intelligence Vol. 517). Springer.

Suresh, S., Sundararajan, N., & Savitha, R. (2013). Supervised Learning with Complex-valued Neural Networks (Studies in Computational Intelligence Vol. 421). Springer.

Rao, A. R., & Cecchi, G. A. (Eds.). (2011). The Relevance of the Time Domain to Neural Network Models (Springer Series in Cognitive and Neural Systems Vol. 3). Springer.

Hey, i suggest:

1) "Neural Networks, A comprehensive Foundation" by Saymon Haykin

2) "Understanding Machine Learning, From Theory to Algorithms," Shai Shalev-Shwartz and Shai Ben-David

BEST BOOK is Neural Networks by HAGAN. You will find many references of this book in MATLAB R2013a also.

Dear Behrad,

You may find these useful:

Mathematics of Neural Networks: Models, Algorithms and Applications (Operations Research/Computer Science Interfaces Series) Paperback – 2 Nov 2012

by Stephen W. Ellacott (Editor), John C. Mason (Editor), Iain J. Anderson (Editor)

The Essence of Neural Networks (Essence of Computing) Paperback – 22 Sep 1998

by Dr Robert Callan (Author)

Good luck!

Luiza

Best book for learning fundamental of NN is "Neural Network Design" by Martin T. Hagan et al.

Best manual for learning how to use the NN is manual of NN toolbox in the MATLAB 

Neural Networks using MATLAB 6.0 by S. N. Sivanandam, S. N Deepa is good book.

This is a good one

http://www.cs.cmu.edu/~tom/mlbook.html

neural networking by zurada

Neural Networks using MATLAB 6.0 by S. N. Sivanandam, S. N Deepa is good book. Exercise fully equiped with step by step calculation and Matlab programmig