As f(0) = f(0)^2, then f(0) = 0 or f(0) = 1.

If f(0) = 0, then f(x) = 0 forall x in (-1, 1).

If f(0) = 1, then by choice x=x and y = -x, we obtain

f(x)f(-x) = 1/(1 - x^2).

It is easy to verify that f(x) = 1/(1 - x), f(x) = 1/(1 + x), f(x) = 1/sqrt(1 - x^2) satisfy the equation.

Very grateful for your observations, due to which you have found two new interesting and non-trivial solutions of the problem. But still, the problem remains open ...

At the present time I have found the general solution of the problem under additional assumption, that the function f is differentiable in zero. The solution is in attached pdf-file. So, I am interested, whether this solution is represented in scientific literature. Because, it seems to me, that the solution of this problem must be known.

Very good. I am sending other remarks too.

1. If there is some $x_0$ such that $f(x_0)=0$ then clearly $f$ is identically 0.

2. So, suppose that $f(x)$ is different from $0$ for every $x$. Taking $x=y$ in the

functional equation shows that we must have $f(x)>0$ for every $x$.

3. Consider than $g(x)=log(f(tanh x))-log(cosh x)$. Then we check easily that

$g$ satisfies the Cauchy functional equation :

$g(x+y)=g(x)+g(y)$.

4. Conversely, it is easy to check that if $g$ is any real function satisfying

$g(x+y)=g(x)+g(y)$ then the function $f$ defined by

$f(x)=\exp(g(\argtanh (x) ) ) / \sqrt{1-x^2}$

is a solution to the considered functional equation.

5. Conclusion :

-- With an additional assumption on $f$, like

(a) being measurable.

(b) being bounded on a non trivial sub interval.

(c) being continuous at some point.

...

we can conclude that $g(x)=ax$ for some real $a$, and this gives the family of solutions of Yaroslav Grushka presented above.

--But, with no additional assumptions on $f$ there are many other solutions

that do not enter in the above family. because this is the case of the solutions

of the Cauchy functional equation.

@Omran, I have one simple question: How (exactly please) did you consider that $g(x)=log(f(tanh x))-log(cosh x)$?

The substitution x=tanh(u) is immediate because of the term (x+y)/(1+xy)

in the functional equation, but then we get (1+tanh(u)tanh(v)) on the right

side of the equation. It happens that

$(1+\tanh(u)\tanh(v))=\cosh(u+v)/(\cosh(u) \cosh( v))$

so it is a natural thing to divide both sides by $\cosh(u+v)$ and then you see the pattern.

Finally, the logarithm transforms the multiplication into addition and one gets

the Cauchy functional equation.

Dear Omran Kouba!

Your solution of the problem is more laconic and beautiful than what was proposed by me in a previous comment. In addition in your solution the condition about differentiability of function in zero can be weakened.

Thank you for your solution of the problem!

@Omran, very good work!