Nature or the grandest universe is a open system, and therefore, the fundamental properties/states (mass, energy level, charge, spin, forces, etc.) about it are limitless. Hence, we have a need for infinite dimension Hilbert space to model quantum mechanics…

It is because it operates with continuous wave functions, David.

"I have a feeling that behind this question there is possible confusion regarding the meaning of Hilbert space in quantum physics, so allow me to begin my answer with an important clarification: Quantum physics is formulated in the 3+1 dimensional space in which we live. The Hilbert space that this question is about is not directly related to the number of spacetime dimensions.

Rather, the Hilbert space is about all the possible states of a quantum system. When you actually measure a property of a particle, you get a specific value. This will be one of the basis vectors of this Hilbert space. But in between measurements, the particle’s state is a combination of all possible measurement values. This is formally similar to how we describe an arbitrary vector in a vector space as a weighted sum of basis vectors.

So how many basis vectors are there in this Hilbert space? There is one for each possible outcome for the measurement. If the measurement yields only a small handful of possible outcomes (e.g., you are measuring the spin of the electron, which is either +1/2 or -1/2, with no other values possible) there are only two basis vectors. The abstract space that represents all the possible spin states of the electron will be two-dimensional.

But another measurement may have an infinite number of possible outcomes. Say, you measure the position or the momentum of a free electron. It can be anything. In between measurements, the electron is in a combination of all (infinite) possible position states or all (infinite) possible momentum states. The number of basis vectors (possible outcomes of the measurement) is infinite, hence the Hilbert space that describes the state of the electron is infinite-dimensional in this case.

Again, in all these examples, the electron “lives” in 3+1 dimensional spacetime. That does not change. It’s the state of the electron, with respect to some measurement, that is represented by the mathematical abstraction of a Hilbert space. The number of dimensions of that Hilbert space corresponds to the number of theoretically possible outcomes of that measurement." -- Viktor T. Toth, IT pro, part-time physicist, https://www.quora.com/profile/Viktor-T-Toth-1

https://www.quora.com/What-are-the-reasons-that-quantum-mechanics-is-formulated-in-an-infinite-dimension-Hilbert-space-rather-than-a-finite-dimension-one

Every infinite dimensional separable Hilbert space owns a unique companion non-separable Hilbert space that embeds its separable companion. Hilbert spaces can only cope with number systems that are division rings. This reduces the choice to the real numbers, the complex numbers and the quaternions. Quaternions are ideally suited to store dynamic geometric data as a combination of a proper time-stamp and a three-dimensional location. So an infinite dimensional separable quaternionic Hilbert space can easily archive the dynamic geometrical data of all discrete objects that exist in the universe.

BehaviorOfBasicFields; http://dx.doi.org/10.13140/RG.2.2.15517.20960

TheStructureOfPhysicalReality; http://dx.doi.org/10.13140/RG.2.2.10664.26885

Let the question be the answer to the question.

What are the reasons that classical mechanics in Hamilton-Jacobi formulation is formulated in an infinite-dimension functional space rather than a finite-dimension one?

This is a six-month-late post, but I think it is worth noting that the Hilbert space formulation of quantum mechanics, which was done by Dirac, Jordan, and von Neumann, had to be based on the two, existing at that time, descriptions (or representations) of quantum mechanics, namely the Schrödinger’s wave mechanics and the Heisenberg’s matrix mechanics, which, as Schrödinger proved, are physically equivalent (isomorphic). Thus, the new formulation had to also be equivalent (isomorphic) to wave and matrix mechanics. In wave mechanics, the state of a quantum system is described by a wave function that is square integrable and belongs to L2(R), which is an infinite dimensional and separable Hilbert space. Therefore, the equivalence (isomorphism) of wave mechanics to the Hilbert space formulation of quantum mechanics means that the abstract state space must also be infinite dimensional and separable.

Because commutator of any two operators in finite dimensional space cannot be equal to 1, which is a must if you want to have momentum and coordinate operators.

The paper "Representing Basic Physical Fields by Quaternionic Fields"; https://vixra.org/abs/2001.0298 explains why physical reality applies infinite dimensional separable Hilbert spaces in combination with non-separable Hilbert spaces. Only that combination explains the interaction between point-like objects and continuums.