It is indeed an infinite dimensional vector space for all nonzero a & b
One example how to construct an infinite dimensional linearly independent set when a and b are positive.
Let p be a prime number. We construct a function f_p on [0, b]
At 0, b/p, 2b/p, ...b the function is 0. In between two consecutive zeros the function is a tent on the positive side. For example it goes linearly from (0,0) to
(b/2p, 1) and then linearly from (b/2p, 1) to (b/p, 0) and goes the same way on all other intervals.
Then on [b, 2b] we define it as tents on the negative side such that f(x) + a f(x + b) = 0 for all x in [0, b]. Notice that there is an unique way to do it as long as f is defined on [0, b]. We continue in the same way with positive tents on [2b, 3b] and so on and then we go on the [-b, 0] and so on.
All these functions are linearly independent because each of them has a zero (b/p) that neither one of the others have.
for any x in R exist n in Z, suth that y=x-nb in [0,b).
Let \phi(y) any function (y\in [0,b) and a not equal 0 (if a=0 only solution f(x)=0). Then f(x)=(-1/a)^n\phi(y) is solution.
f(x) is continiuosly only if \fhi(y) continiously and lim_{y\to b}\phi(y)=-\phi(0)/a.
f(x) periodic function only if (-1/a)^m=1, then period is m*b. If f real function, then m=1 (a=-1) or m=2 (a=1). If f(x) complex function, m in N (any natural).
Say b > 0. Let C_0[0, b] be the space of all continuous functions f from [0, b] to R such that f(0) = f(b).
The equation f (x) + a f (x + b) = 0 allows as shown before an unique extension to R. Call this extension F_f.
The function
H : C_0 [0, b] to C_ab (R) given by H (f) = F_f is an invertible linear operator.
Here C_ab (R) is the set of all functions from g R to R satisfying g (x) + a g (x + b) = 0.
Therefore C_ab (R) is an infinite dimensional subspace (I should say linear manifold though) of C (R).
I don't know what topology to consider on C (R) but the supremum norm does not work and definitely does not work on C_ab (R) because if |a| not = 1 except for the function 0 all functions in this space are unbounded.
However, we can define a norm on C_ab (R) by || F_f || = || f ||, so by transporting the norm on C_0 [0, b] where the sup norm is OK.
With this norm C_ab (R) is a Banach space.
One last thing. C_0 [0, b] is in fact a Banach algebra. C_ab (R) is not, unless a = - 1.
That is because F_f F_g not equal F_fg so only the linear structure transfers but not the multiplicative one.
With pervious answer of my colleagues, extremely way can exist to make the solution of main problem. One may use Fourier series with unknown coefficients that it satisfies in f(x)+af(x+b)=0. We must look infinite unknown coefficients with infinite number equations, indeed multiplicity of solutions are shown by one.
If a0 and f satisfies the proposed functional equation then h(x)= ax f(bx) is continuous and 2-periodic (h(x+2)=-h(x+1)=h(x)), in particular, it is bounded. Now, if a is not equal to +1 and f is d-periodic with f(x0) different from zero, then (h((x0+k d)/b))kεZ cannot be bounded which is absurd. So, either f is identically zero, or a=1.
Now when a=1, let g(x)=(-1)[x] f(bx) where [x] is the integer part of x then clearly g is 1- periodic continuous on R\Z and satisfies g(0)=g(0+)—g(0–)=—g(1–).
Conversely, if g:[0,1]—>R is continuous and satisfies g(0)+g(1)=0 then f(x)=(-1)[x/b]g({x/b}) where {x} is the fractional part of x is a solution of the proposed functional equation satisfying the required conditions.
A two-variable analogue of the proposed problem can be given as follows. Given real numbers, a ≠ 0, b, c ≠ 0 and d, find all continuous on R^2 real-valued functions f satisfying the equations
f(x, y) = - af(x + b, y) = - cf(x, y +d) for all x, y ϵ R.