Let's label the total number of balls in the urn with N. For a single draw, the probability of drawing a ball of a given colour C, where C = A or C = B, is (current number of balls of colour C in the urn) / N.

The probability of getting a particular sequence of colours by drawing balls from the urn is the product of probabilities for every single draw in the sequence.

The probability of getting a particular sequence of colours, when treated as a function of N, is the likelihood function for N (see https://en.wikipedia.org/wiki/Likelihood_function). When N is treated as a random variable, the probability distribution of N is proportional to the product of the likelihood function of N and the prior distribution of N. In the example of the prior distribution of N being constant (that is, if all the values of N are equally likely until we start the experiment), the probability distribution of N is proportional to the likelihood.

I'm not sure how far one can progress analytically using formulas, but I'm enclosing a Python script that computes the likelihood given a sequence of colours and a list of N values. While I did test it a bit and the results seem to make sense, I give no guarantees.

It's a pity that this inspiring question has not been answered sooner and by someone better qualified than me.

Thank you.

After I posted the question, I searched some more in the literature and I found that the problem is an inverse coupon collector's problem. This appears to be only a curiosity to mathematicians, since there is only a limited literature that I could find..

My practical application is to estimate the number of animals or certain species that live in an area, based on photographs taken by automatic cameras. The number of animals is the number of coupons (balls) and each photo is a coupon purchase or a draw from the urn. The first time an individual is photographed its coupon is collected / its ball is changed to another colour, and subsequent photos of the same animal are the same coupon or are put back in the urn. The raw data is the accumulation curve of newly photographed individuals - the aim is to estimate the number of animals without having to photograph all of them.

So you are doing a mark and recapture experiment.

They say that all models are wrong but some are useful. What is wrong with this model is that not all balls are equally likely to be drawn. Perhaps some individuals live a lifestyle that causes them to be photographed more often than other individuals (lowering the estimated number of animals), perhaps they don't want to be photographed and avoid the cameras after being photographed for the first time (increasing the estimate), perhaps it's both. However, if these effects are small enough, or if there is no better method of counting the animals, the model may be worth using.

You actually have more information than in the urn example: for a given individual, you not only know whether you have seen it before, but you know how many times you have seen it. As if you didn't return the balls of colour B but replaced them with C, and those with D, etc. Maybe the additional information can be used to improve the estimated number? Maybe it is possible to use it for checking the assumption that every individual is equally likely to be sampled?