Active Incentive Learning

Abstract

With the undeniable success of supervised learning in modern business operations including healthcare, revenue management or customer relation management, many crowdsourcing platforms or expert labelers are in high demand to create supervised datasets for training machine learning models. Many research have focused on which instances to query for labels while overlooking the payment-quality tradeoff that inherently resides in label purchasing problems. In this paper, we investigate the problem of how to choose the optimal payment to incentivize crowdworkers or expert decision-makers with varying labeling performance which can be uniform, class-conditioned or instance-conditioned to produce the best model performance (\eg unfairness, false omission rate and accuracy) for a given budget. Empirical results suggest that our algorithm has a robust performance across different incentive - performance tradeoff and different performance metrics.