Dear all,

my situation is the following:

I have a matrix consists of purchase probabilities for different products per user. Retrospectively I have another matrix consists of real-purchases. Now my task is to verify my purchase probabilities on the new, additional information given.

My output should be something like 'the probabilities of product 1 are overrated or underrated'.

I have tried different scores like brier-score or hit-rate BUT - and here is my problem - I have like 100.000 users and 100-500 purchases per product. So, just 1% of my real-purchase data covers the other probabilitiy matrix. Therefore my scores a really small and implicit "good estimations" but that is probably wrong.

Any idea which evaluation scores are useful in this case? Any recommendations on similar scientific approaches?