i think physicists are largely silent on this question, as it is more philosophical in nature. a majority of physicists find it sufficient that arithmetic (and by extension all of math) /works/ and don't ponder very hard /why/ it works. if polled on their opinion, however, i think you'd find roughly an even split between physicists who think math is natural and physicists who think math is a convenient shortcut to the answer.
Interesting question and I think physicists are not ignoring it, just the answer is not known and for practical purposes often not needed. I don't know why the question is restricted to arithmetic, however.
I think that the usefulness of arithmetic is not that surprising. You can count physical objects, bodies, ... Even simple algebra is, in my opinion, useful only in some "trivial" sense. For example, experimentally you find out that hot rod is longer than the cold rod. The simplest guess is that the length is directly proportional to the temperature and you can write down corresponding law. Then, however, you find out that the dependence is not exactly linear and you add another, quadratic term. If it is not sufficient, you can add more and more terms of higher powers, the coefficients can be determined experimentally.
My point is, that whatever the dependence of the length on temperature is, you can approximate it sufficiently well by a polynomial. In this sense, there is nothing fundamental about the mathematical law which describes the length of the rod as a function of temperature. This is what I call (perhaps naively) trivial usefulness.
But what is really fundamental is when abstract mathematics turns out to describe some basic physical law. For example, the description of gravity in terms of (pseudo)Riemannian geometry is highly non-trivial, because it is not obvious. The apparatus of differential geometry gives you very deep insight into the nature of gravitation and allows you to make predictions which would be impossible in the Newtonian framework. Another similar example is the notion of connection on fiber bundles in gauge field theories. Fiber bundle is very abstract, physically non-intuitive, and yet it is a correct tool to describe quantum fields.
So, I think that some fundamental physical laws are genuinely mathematical. The notion of curvature of the spacetime cannot be inferred from observations, the notion of connection cannot be measured in usual manner. I believe that this is because the Nature really IS mathematical.
I like following "nowhere leading" argument by Roger Penrose. Mathematics exists independently of anything else, it lives in the Platonic world of ideas. However, some parts of mathematics describe correct physical laws and thus physical laws are only a part of mathematics. The physics describes entire reality and the humans are part of this reality, so our minds constitute only a part of physical world, which is only a part of mathematics. On the other hand, our minds think about many things and sometimes they think about mathematics. So, mathematics is only a part of our minds, which means that it is only a part of a physical world, which means that it is only a part of mathematics :) Which world is bigger, then?
As the number of anwers sudgest, it is not a question that can be settle and there is a large number of philosophical positions on this question.
Arithmetic is something humans have invented as part of their language for accounting and commercial transaction, for the creation of a callendar and so planning their activities. We invented tables and the word to describe tables. Does table are part of the physical world? No there are part of the human world. Why would numbers and arithmetic different? Why would we consider one of our invention for communication as part of the natural world? Because it is absolutely necessary for us to describe the natural world. Yes it is but the description belong to our culture and should not be assimilated into the natural world. But the fact that it is absolutely essential to use such concept for the description of the natural world by us or any extra-terrestrial intelligence does provide a point for that to be intrinsic to nature. It is certainly intrinsic to any system of description of nature in the universe but it is totally foreigh to Nature in itself. But Nature in itself cannot be describe. Exactly.
martin, one could make the exact opposite argument using the same examples. the fact that spacetime curvature cannot be inferred from observation can also suggest that the mathematical description of gravity is just that, a mathematical description, but a description of a thing is not the same as being that thing. it seems highly unlikely to me that nature busies itself solving differential equations all the time to make things happen. things just happen, and our diffeqs are good descriptions of the events, but they're not "natural". it's sort of like the difference between written music notes and a physical description of the waveforms that are produced by instruments being two descriptions of music, but neither are actually music.
I differ with Raskin and Scholtz on spacetime curvature. It can and was inferred from observations. With difficulty, Einstein inferred it from the equivalence principle (together with SR). This first time through the argument was convoluted and he re-cast equivalence as General Covariance. Now days we could easily infer curvature from the Shapiro delay, light bending and the extra precession of Mercury. The double bending of light and double (loosely speaking) delay in a Shapiro measurement are difficult to explain without curvature, although there are multiple interpretations of curvature. For a simplified modern inference of curvature from equivalence along with an explanation and visualization of the two orthodox interpretations of curvature, see the paper linked below.
On the question of arithmetic in nature ... something interesting I've always thought about. But I can only answer for myself. I think particles and fields can do only very simple things. The simplest kind of arithmetical functions. Complexity only arises through the interaction of multiple simple parts (and sometimes *only* in the mind of humans).
An interesting thing that many objects, particles and fields have to do in relativity theory is find what amounts to the hypotenuse of a triangle. For example, the Lorentz gamma factor is basically a solution to v2 + y2 = c2. So whatever y is (time, mass, etc.) y/c = (1-v2/c2)1/2. I do not think nature implements some kind of computer that numerically calculates this. It must be much simpler than that.
I completely accept your point of view. I especially acknowledge your "music example". I don't want to argue with you, because these are simply two different attitudes and I perfectly understand what you mean. My viewpoint is different, however. Following Fermat, I could write that I have proof that you are wrong, but this message is too short to contain my proof :)