High-Effort Crowds: Limited Liability via Tournaments

2021

Peer prediction mechanisms incentivize agents to truthfully report their signals even in the absence of verification by comparing agents' reports with those of their peers. In the detail-free multi-task setting, agents respond to multiple independent and identically distributed tasks, and the mechanism does not know the prior distribution of agents' signals. The goal is to provide an $\epsilon$-strongly truthful mechanism where truth-telling rewards agents "strictly" more than any other strategy profile (with $\epsilon$ additive error), and to do so while requiring as few tasks as possible. We design a family of mechanisms with a scoring function that maps a pair of reports to a score. The mechanism is strongly truthful if the scoring function is "prior ideal," and $\epsilon$-strongly truthful as long as the scoring function is sufficiently close to the ideal one. This reduces the above mechanism design problem to a learning problem -- specifically learni...

ACM Transactions on Economics and Computation

Peer prediction mechanisms incentivize self-interested agents to truthfully report their signals even in the absence of verification, by comparing agents’ reports with their peers. We propose two new mechanisms, Source and Target Differential Peer Prediction, and prove very strong guarantees for a very general setting. Our Differential Peer Prediction mechanisms are strongly truthful : Truth-telling is a strict Bayesian Nash equilibrium. Also, truth-telling pays strictly higher than any other equilibria, excluding permutation equilibria, which pays the same amount as truth-telling. The guarantees hold for asymmetric priors among agents which the mechanisms need not know ( detail-free ) in the single question setting . Moreover, they only require three agents , each of which submits a single item report : two report their signals (answers), and the other reports her forecast (prediction of one of the another agent’s reports). Our proof technique is straightforward, conceptually motiv...

ArXiv, 2015

Incentivizing effort and eliciting truthful responses from agents in the absence of verifiability is a major challenge faced while crowdsourcing many types of evaluation tasks like labeling images, grading assignments in online courses, etc. In this paper, we propose new reward mechanisms for such settings that, unlike most previously studied mechanisms, impose minimal assumptions on the structure and knowledge of the underlying generating model, can account for heterogeneity in the agents' abilities, require no extraneous elicitation from them, and furthermore allow their beliefs to be (almost) arbitrary. Moreover, these mechanisms have the simple and intuitive structure of output agreement mechanisms, which, despite not incentivizing truthful behavior, have nevertheless been quite popular in practice. We achieve this by leveraging a typical characteristic of many of these settings, which is the existence of a large number of similar tasks.

Proceedings of the AAAI Conference on Artificial Intelligence

We study learning statistical properties from strategic agents with private information. In this problem, agents must be incentivized to truthfully reveal their information even when it cannot be directly verified. Moreover, the information reported by the agents must be aggregated into a statistical estimate. We study two fundamental statistical properties: estimating the mean of an unknown Gaussian, and linear regression with Gaussian error. The information of each agent is one point in a Euclidean space.Our main results are two mechanisms for each of these problems which optimally aggregate the information of agents in the truth-telling equilibrium:• A minimal (non-revelation) mechanism for large populations — agents only need to report one value, but that value need not be their point.• A mechanism for small populations that is non-minimal — agents need to answer more than one question.These mechanisms are “informed truthful” mechanisms where reporting unaltered data (truth-tell...

2021

Suppose a decision maker wants to predict weather tomorrow by eliciting and aggregating information from crowd. How can the decision maker incentivize the crowds to report their information truthfully? Many truthful peer prediction mechanisms have been proposed for homogeneous agents, whose types are drawn from the same distribution. However, in many situations, the population is a hybrid crowd of different types of agents with different forms of information, and the decision maker has neither the identity of any individual nor the proportion of each types of agents in the crowd. Ignoring the heterogeneity among the agent may lead to inefficient of biased information, which would in turn lead to suboptimal decisions. In this paper, we propose the first framework for information elicitation from hybrid crowds, and two mechanisms to motivate agents to report their information truthfully. The first mechanism combines two mechanisms via linear transformations and the second is based on ...

Proceedings of the 2018 ACM Conference on Economics and Computation

A central question 1 of crowdsourcing is how to elicit expertise from agents. This is even more difficult when answers cannot be directly verified. A key challenge is that sophisticated agents may strategically withhold effort or information when they believe their payoff will be based upon comparison with other agents whose reports will likely omit this information due to lack of effort or expertise. Our work defines a natural model for this setting based on the assumption that more sophisticated agents know the beliefs of less sophisticated agents. We then provide a mechanism design framework for this setting. From this framework, we design several novel mechanisms, for both the single and multiple tasks settings, that (1) encourage agents to invest effort and provide their information honestly; (2) output a correct "hierarchy" of the information when agents are rational.

Proceedings of the Web Conference 2021, 2021

We initiate the study of information elicitation mechanisms for a crowd containing both self-interested agents, who respond to incentives, and adversarial agents, who may collude to disrupt the system. Our mechanisms work in the peer prediction setting where ground truth need not be accessible to the mechanism or even exist. We provide a meta-mechanism that reduces the design of peer prediction mechanisms to a related robust learning problem. The resulting mechanisms are ϵ-informed truthful, which means truthtelling is the highest paid ϵ-Bayesian Nash equilibrium (up to ϵerror) and pays strictly more than uninformative equilibria. The value of ϵ depends on the properties of robust learning algorithm, and typically limits to 0 as the number of tasks and agents increase. We show how to use our meta-mechanism to design mechanisms with provable guarantees in two important crowdsourcing settings even when some agents are self-interested and others are adversarial. CCS CONCEPTS • Theory of computation → Algorithmic mechanism design; Unsupervised learning and clustering; • Information systems → Incentive schemes; • Mathematics of computing → Probabilistic inference problems.

Procedia Computer Science, 2015

Participatory sensing system (PSS) is an emerging field of research where the fundamental idea is to employ the autonomous agents (in this paper humans) carrying smart devices, as a virtual sensor for collecting and transmitting information. The most grinding work is to motivate the human agents carrying smart devices in the data collecting process. For motivating the human agents, auction theory has played a central role with a framework to incentivize the agents in the PSS. In this line, works so far have mainly considered that there are several human agents to sell the data, and one data collecting centre to buy the data. In this paper, we propose a truthful mechanism in an online double auction environment that addresses the situation where multiple sellers and multiple buyers are present in the PSS. To the best of our knowledge, this is the first truthful mechanism in double auction setting.

Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, 2020

We derive conditions under which a peer-consistency mechanism can be used to elicit truthful data from non-trusted rational agents when an aggregate statistic of the collected data affects the amount of their incentives to lie. Furthermore, we discuss the relative saving that can be achieved by the mechanism, compared to the rational outcome, if no such mechanism was implemented. Our work is motivated by distributed platforms, where decentralized data oracles collect information about real-world events, based on the aggregate information provided by often self-interested participants. We compare our theoretical observations with numerical simulations on two public real datasets.

Proceedings of the 2018 ACM Conference on Economics and Computation

We build a natural connection between the learning problem, co-training, and forecast elicitation without verification (related to peer-prediction) and address them simultaneously using the same information theoretic approach. 1 In co-training/multiview learning [5] the goal is to aggregate two views of data into a prediction for a latent label. We show how to optimally combine two views of data by reducing the problem to an optimization problem. Our work gives a unified and rigorous approach to the general setting. In forecast elicitation without verification we seek to design a mechanism that elicits high quality forecasts from agents in the setting where the mechanism does not have access to the ground truth. By assuming the agents' information is independent conditioning on the outcome, we propose mechanisms where truth-telling is a strict equilibrium for both the single-task and multi-task settings. Our multi-task mechanism additionally has the property that the truth-telling equilibrium pays better than any other strategy profile and strictly better than any other "non-permutation" strategy profile.