Scenario:
We have an application with several thousand of user options. Each user selects a set of options from time to time (low frequency). So each user might have an history of previous sets of selected options. Also each time a user is going to make a new selection of options, we might pre-select some options in a deterministic fashion (due to external information sources). Some options are related to others. There exist a manual classification of options, with relationships between them, which also can be used to tag user characteristics. A user Profile would consist in a group of characteristics derived from external information, and additional options selected which belong to a class.
Ranges:
Options: Thousands.
Options in User Sets: Max tens.
Users: 1Million.
Previous Sets: [0,30]
Problem to solve:
We want to help the user to build the right set of options each time it uses the application.
So when a user starts a new selection process, we:
1. Might have deterministic selected options.
2. Might have previous selected sets, which have a certain probability of being re-selected (heuristics can be retrieved for this).
3. We plan to make a short questionnaire to determine if the user has several characteristics (tags) with a high probability of being true for that user in that moment (from data analysis of all users).
4. We should be able to recommend options to the user according to the new tags, and other options that might define the user (from previous data analysis).
We do have big amounts of data of previous set selections, so that can be used to test the accuracy of the methods used.
Question
What techniques would you recommend for this scenario? Approaches, Languages, Algorithms, Databases, Frameworks, Papers that talk about similar situations, etc.
.