This is an important question. In addition to the very helpful answers already given, there is a bit more to add.
The old adage that one learns Mathematics by doing Mathematics applies also to learning the Philosophy of Mathematics, i.e, learn about the vast and rich lanscape of the Philosophy of Mathematics by doing Mathematics.
Bertrand Russell's approach to this subject is a bit misleading, namely, a seamless development of mathematics from a few clearly stated axioms and rules of inference in pure logic: https://www.math.uwaterloo.ca/~snburris/htdocs/WWW/PDF/principia.pdf
Reflections on Mathematics, its structures and nature, are best approached in little chunks. For a novice, a very good place to start an approach to the threshold of the Philosophy of Mathematics is in the visualization of mathematical structures.
The best way to learn is when we play a game. For this question, i believe that chess game can be the best starting place to learn about the philosophy of mathematics.
Additionally, Prof. George Ifrab (French Mathematician) said that the mathematical symbols are discoveries of man using the components and characteristics of the human body.
In addition, the history of the mathematical sciences has shown that the arrival at any level made it possible to solve questions asked but also it poses new questions to solve. As a result, ancient mathematics begins with games and everyday life where there is no mathematical logic, laws and equations.
First, get yourself familiar with basic concepts in (pure/applied) mathematics and philosophy (in particular ontology and theory of knowledge). Philosophical studies and understanding cannot be based on ignorance; not all weird and exciting ideas one gets qualify as philosophy.
When thinking about concepts like infinities, one should first know what kind of role they play in mathematics, including topics that go well beyond the first year Calculus course. Things like the distinction between analytic and synthetic knowledge are important in understanding the motivation of creating theories about foundations of mathematics in the first place.
After that, you can start reading formal theories like logic. I would start from theory of language (mathematics can be understood asa kind of language, and this is the point of view taken by the most philosophers), for example, Frege and perhaps Mill. Then get familiar with Hilbert, Wittgenstein, Carnap, Russell, Kripke, etc. If you don't know the early works, it's impossible to understand what the later scholars are aiming at. Things like axiomatic set theory are certainly interesting from a methodological point of view, but they don't lead into general understanding about the nature of the topic.
Most importantly, you should first understand what mathematics is all about, and why there is a need for philosophy of mathematics. As I see it, the "what" question is usually relatively simple, but the "why" is the deep one. There is no single book or reference that will teach you that; becoming a deep thinker is a life-long project.
To understand the Philosophy of Mathematics, you should start by observing, scrutinizing and understanding the nature and in particular those natural processes, which you find makes you curious and inquisitive.
I don't know what is the point of doing philosophy (of mathematics) at all, if you are not doing it seriously. Philosophy is about understanding things. Doing things doesn't always or even often require complete understanding. But false understanding is more a problem than (acknowledged) lack of understanding.
So why do we have philosophy of mathematics? Naive books about that say that the set theory it is the basis of mathematics. This is completely wrong; mathematics was done for millennia before anybody came up with set theory. Yes, even infinity was considered already by the Ancient Greeks (who didn't have even negative numbers), and Newton wrote about it well before the set theory. Where you really need set theory is if you wish to understand concepts like cardinalities, which are mainly required in some very branches of topology, functional analysis and (perhaps) probability theory. Such things never arise in typical engineering applications.
But why do we really have things like set theory as the "foundation of mathematics? Not because it's something that naturally arises as a result of mathematical investigation. You don't suddenly find set theory underlying behind number theory, or complex analysis, or operator theory, no matter how hard and deep you study them.
Instead, the historical reason is the following. Until about 19th century, the people were doing mathematics as it was basically a branch of natural science. There was no clear distinction between physics and mathematics, i.e., direct observations from the nature (synthetic knowledge) and logical deductions (analytic knowledge) were both used in deriving mathematical results. But then the study of formal mathematics went forward, and the people were able to come with weird constructions that led into paradoxes and counterintuitive results. This turned into a crisis in mathematics: it was no longer clear what is true and what is not. It was necessary to define what is mathematics (the precise nature of mathematical objects) as something that is clear and can be studied itself (cf. Model theory, the branch of logic, that has nothing to do with mathematical modelling).
Several attempts to this direction were made. Some philosophers like Carnap tried take physics as the fundamental science, thus defining mathematics essentially as a part of physics. Some, most notably Hilbert, tried to base mathematics as a formal game ("played according to certain simple rules, with meaningless marks on paper", as he famously stated). Ultimately, both lines of reasoning turned out to be somewhat unsatisfactory (but led into useful discoveries, like computers and information theory).
Then, the research went into study of formal structures (i.e. the axiomatic systems). The people started with the earliest known axiomatic system – planar geometry – which again led into interesting discoveries; the hyperbolic geometry (which proves the independence of Euclid's axiom of parallel lines in the sense of model theory) and also to the so-called Erlangen program of Felix Klein (which has been hugely important in contemporary geometric analysis), but soon turned out to be too simple to be truly interesting in this context.
A far more useful axiomatic theory to study was the (first order) arithmetic, that could be formulated by means of the Peano Axioms. But then it turned out that this and, indeed, any (finite) system of axioms for the first order arithmetics was, by the famous result Gödel, incomplete (in the sense that there are statements that can be expressed but cannot be proven in the theory).
With these discovery, founding the "boundaries" of the mathematics turned out to be the an interesting object of study in itself. For this reason, it was important to develop a theory that would contain (i.e. allow formulating or expressing) as much as possible of the mathematics but would be simple enough to be studied as a formal structure. As I see it, this is the background of axiomatic set theory (ZFC) in mathematics. It's a concept developed for maximum usefulness in so-called metamathematics, i.e., formal/logical study of mathematical theories and formal systems (like the work of Gödel and Cohen). It's certainly useful in studying (formal aspects of) concepts like infinities, but it's definitely not something you "need to discuss infinity" (if you are only interested on sets like natural and real numbers ).
[Indeed, going into some "practical set theory" without understanding what is the background is much like trying to understand why we are writing this stuff in English without making any reference to the British Empire. Certainly, one can say that English language is the foundation of human thought in the sense that every idea expressible in some language can probably be expressed in English. But what comes to the "philosophy of human thought", it's a pretty meaningless statement.]
I don't particularly like the patronising tone of your comments, but I'll spend my time to write one more.
You make many nonsense comments about various aspects of the history of mathematics, but for example the idea that "ancient geometry" was sufficient until Cantor is so wrong that it's not even funny.
Actually, the methodological revolution in mathematics came with the Geometry of Descartes, who gave up the way "the ancients" made their geometry in favour of one that makes use of very efficient algebraic notations. Notably, the above allowed one to study things like trigonometric functions as functions (before, they were understood as geometric ratios), but for a long time nobody really understood what the concept of function (and related concepts like continuity) means in the abstract sense. There were many attempts to make these ideas precise (certain types of functions like the linear progression were studied as formulas already in the Middle Ages; I believe this was almost the only significant scientific discovery made during that era), for example, the attempt to define functions as power series, but it soon become clear that this idea is insufficient (I believe this is background for the name Function Theory that is still used more or less as a synonym for Complex Analysis).
Clearly, mathematics is not tied to any particular formalism. Many different styles of reasoning have been used in the history of mathematics. Some of them have particular advantages. For example, constructivist school of mathematics never took of as the foundation of mathematics, but it has still proved to be very useful in computer science.
Surely, study of axiomatic set theory can be useful and lead into interesting results, and any serious (theoretic) mathematician should know at least something about it (those doing e.g. topology, measure theory or functional analysis should definitely know a lot), but let's face it: arguments directly appealing to the axioms of set theory are almost never used in mathematical analysis. Mathematics was practiced for millennia before its formal foundations were settled.
In particular, an engineer learns or somebody who's knowledge about mathematics is mostly on the calculus book level learns almost nothing useful from books about axiomatic set theory. In my opinion, it's far more useful to understand why such theory is needed than to know details of the construction of real numbers, etc. And you don't really need to understand much about the set theory to understand the Cantor's diagonal argument, for example. It's something that can be explained to a high school student by using a computer program as an intuition.
Just the fact that you don't know or understand something doesn't make it wrong. It merely shows limits of your knowledge and understanding.
I don't know why you (from the very beginning) have had such a personal and hostile tone in your comments. When I gave my original answer to the person who posted the question, it was an bona fide attempt to be tell what, in my personal opinion, is useful to know in the philosophy of mathematics. I don't mind other people having different opinions. But, frankly, not much in your writings suggests that you are an expert in the topics where you claim authority. In particular, if you don't like my answers, please don't read them.
Actually, I started my academic life in mathematical logic. I studied courses like Mathematical Logic, Model Theory, Axiomatic Set Theory and Non-standard Analysis. I wrote my Master's Thesis about Descriptive Set Theory (a very formal theory which aims to establish hierarchies on subsets of natural numbers based on complexity of the formulas that is needed to describe them). I attended numerous seminars and talks about topics in mathematical logic, set theory, model theory, and so on. I also took Theoretical Philosophy as my minor, and nearly completed master level studies in this topic as well. I studied various types of logics, including intuitionist/constructivist ones, modal logics, but also history of mathematics, theories of formal languages (the people you hate like Mill, Frege, Russel, Wittgenstein, Carnap, Kripke, etc.) etc. I did even teach logic for one semester. I loved this stuff and wanted to know everything about it.
But, anyway, then got a position in another research group, and the professor I would have liked to have as my thesis advisor went to another university, so I gave up this and begun to do a Ph.D. thesis in another topic. A posteriori I can say that, it was a good career move. There are few academic positions available in logic and foundations of mathematics.
Since then, I have made about 50 papers in pure and applied mathematics (mainly complex analysis, elliptic PDEs and related topics), some of them the very top journals. I can say that I have very rarely needed anything from my background in formal foundations of mathematics. Proofs where principles like Axiom of Choice are occasionally applied in certain aspects of analysis, mainly in functional analysis of infinite dimensional Banach (etc.) spaces and measure theory. I have occasionally used the generalised induction principle (which is a neat trick that I learned in a course about Mathematical Logic), but not so often (and the principle itself is not strictly tied to mathematical logic; it's just heavily applied there). In one of my papers we used ultrafilters (an idea that is used in construction of non-standard reals), but that part was mostly written by my co-author, so I didn't really have to touch it. Of course, I occasionally refer to the set theoretic definition of a function (the condition of functionality) when explaining the students why the roots are considered as functions in the real analysis but not in complex analysis. But, overall, I would say that this stuff is approximately as useful to me as my studies in algebraic topology. Surely, it's sometimes useful (like for understanding why the people use the unintuitive homotopy definition for simply connected domains, even in the higher dimensional spaces where it doesn't even coincide with the natural definition), but one could live without it.
Don't get me wrong. I don't claim that Mathematical Logic is useless, nor regret spending my time in studying it. But it's not something you need to deeply understand in order to do or use mathematics. It's a deep, difficult, and very specialised topic that takes years to master and is mainly useful for the people who do research in this field. There are better ways to spend your time. (In)completeness of some particular axiomatic system is not a fundamental question for any engineering application I am aware of.
One should understand that the philosophy of mathematics (I suppose the original poster was asking about) is not the same as its logical foundations. The logical foundations are, of course, important but "mathematics" is not only a formal structure that you write onto paper to be studied by the future generations. It's also a living field of human activity, where the practitioners of this topic of scholarly activity work on various problems. As I see it (feel free to disagree) that understanding mathematics in this way, as a "great story" starting with early human civilisations and going somewhere in the distant future is a point of view to the philosophy of mathematics that is, based on my personal experience very useful in both studying and teaching mathematics. It allows one to meaningfully understand things like what is interesting mathematical research, what (besides technical novelty and sophistication) makes a good mathematical paper, and what are the natural problems one should study in the context of some particular theory. Why is sometimes a paper with few "impressive calculations" still acceptable to a top journal, and sometimes one that has them is rejected even from low ranking ones. More generally, what do the mathematicians do and why.
It should be noted that mere formalism doesn't give sufficient answers to such questions. If that was the case, the history of mathematics would start with the empty set and then evolve towards ever more complex formal structures, which is obviously not how it happened. Also, formal theories are, in a sense, tautologies. But then, do they tell us anything about the nature? Or is mathematics, as phrased by Hilbert, just "a game played according to certain simple rules with meaningless marks on paper?"
In particular, for an engineer, or other person who is using mathematics but not really working on research problems in it, it's very useful to understand why mathematical theories are written as they are. For example, why do we need deltas and epsilon; isn't it sufficient to think continuous functions as ones that can be graphed without lifting the chalk from the blackboard?Such naive intuition was certainly seen sufficient until relatively recently. From a pedagogical point of view, an important first step in learning formalism is in demystifying it. It certainly doesn't help, if your teacher starts by explaining that mathematical concepts come from some "Platonic Heaven", which only greatest minds can reach through careful study and meditation :)
As a side note, Plato was certainly an influential thinker, whose works important to know for any academic/scholarly type, but today they are mostly useful for understanding the history of western philosophy, and the general world view of ancient Greeks. I don't see his ontological views at all useful from a point of view of discussing foundations of modern mathematics. I'm sure that Hellenistic thinkers (who had deep philosophical problems in accepting things like zero and irrational numbers) would view things like the Peano curve, non-measurable set, fractals, etc., not to mention non-standard models of real analysis as mockery mathematics (and base this view mainly on Plato's idea realism).
I found the answers highly entertaining and informative! But, personally I believe that you do not have to understand modern mathematics to think about philosophical questions, up to some extent at least.
For example, people use real numbers to model things that are finite (but with veeery large numbers), and of course treating them as continuum makes the formulas much easier. And this leads to the question that is everything finite, in the end.
And the age old question that is the world actually expressable as formulas etc is very interesting. Even though this is more like "philosophy of physics", but in my opinion it is inseparable from "philosophy of mathematics"...
According to my point of view if you are a mathematician try to collect and review some books about number theory philosophy it may be too useful.However it seems complex and upset.BTW,I have not found good book in google search to mention here.I hope you find some.