For ChatGPT，if you can collect all the possible pre-train data, then you can just remove the bad-feedback data from predictions for reward model.
if you can not collect all the possible pre-train data, then you need to correct the bad-feedback data from predictions for reward model.
But in both way, you need humans to label.
Not only time-consuming. The distinction good/bad is largely an "intelligence-recognized" categorization, and not a "pattern-recognized" categorization. This is a problem with many AI-tools which in reality are just pattern-recognition tools. The statement that a pattern-recognition tool caught something fundamental and essential of the subject is false: it will only have caught a difference - on average - from the rest. Just like the difference between 0 and 1 is very well recognized by computers. However, do they capture anything about "what that is"? No! What is essential in feedback? It is not the order of words (patterns). It is "what that means". Feedback is good if it is correct and helps you improve. It is bad if it is either incorrect, or if it does not help you. This can go as far as telling someone to stop a project because it is useless: if it really is useless, then that is excellent feedback. Pattern-recognition would be more successful on foul language and profanity-filtering. And even then: there may be good feedback in there - just worded impolitely.
- Badges
- Science topic
- Similar topics
- Correction