Learning from Multiple Observers with Unknown Expertise

Internet has emerged as a powerful technology for collecting labeled data from a large number of users around the world at very low cost. Consequently, each instance is often associated with a handful of labels, precluding any assessment of an individual user's quality. We present a probabilistic model for regression when there are multiple yet some unreliable observers providing continuous responses. Our approach simultaneously learns the regression function and the expertise of each observer that allow us to predict the ground truth and observers' responses on the new data. Experimental results on both synthetic and real-world data sets indicate that the proposed method has clear advantages over ``taking the average'' baseline and some state-of-art models.