COTA: How Specifying Confidence Creates Stronger Collective Decisions

When several people disagree, taking a simple average is tempting. But what if some of them are guessing while others know the topic well?

Imagine asking a group of people whether a statement is true. Perhaps the statement is about a scientific claim, the reliability of a data source, the emotion expressed in a news headline, or whether an extracted fact should enter a knowledge base.

You will probably get different answers. That is normal. What is more interesting is that people may also have different levels of confidence in those answers. One person may say, “I am almost certain.” Another may say, “Honestly, this is just a guess.”

Most systems that aggregate opinions do a reasonable job with the first part: they combine the answers. But they often ignore the second part: how confident people are in each individual answer.

Our new work, COTA: Confidence-aware Opinion and Trust Assessment, starts from a simple idea: if people explicitly specifies their confidence alongside their opinions, we should not throw that information away.

The central intuition: an answer given with low confidence should not necessarily count as much as an answer given with high confidence.

The problem with “one person, one vote”

Aggregating opinions is everywhere. Crowds label images for machine-learning datasets. Multiple websites report slightly different facts. Experts assess risks. Users rate content, products, or claims. In all these cases, we need some way to turn a collection of possibly conflicting views into a single useful result.

The most familiar approach is majority voting: whichever answer receives the most votes wins. This works surprisingly well in some settings, but it treats every vote as equally informative. A carefully considered answer from someone familiar with the topic receives the same weight as a random click.

The issue becomes even clearer when answers are not simply “yes” or “no.” Suppose we ask people how likely a statement is to be true. One participant might answer 90%, another 55%, and another 10%. Reducing all this nuance to a binary label loses useful information before the aggregation has even begun.

COTA is designed for this more realistic setting. It can work with continuous opinions—numbers between 0 and 1, such as probabilities—and with a confidence score attached to each opinion.

Confidence is not just a personality trait

A key detail is that confidence should be collected for each answer, not just once per person. Someone may be very knowledgeable about astronomy but far less certain when judging the emotional tone of a headline. Treating that person as globally “reliable” or “unreliable” misses an important part of the picture.

COTA therefore considers both pieces of information together: what each person says about each statement, and how confident they are in that specific judgement.

The aim is to identify a consolidated position that is most compatible with the views expressed by the population—while giving greater importance to opinions that people state with greater confidence. Put simply, COTA searches for the interpretation of the available answers that creates the most coherent and trustworthy collective picture.

This is useful because “truth” does not always mean the same thing. In some cases, there is an external fact to discover: is a scientific claim correct, did an event happen, or is a data record accurate? If the population is sufficiently informed and broadly reliable, a strong collective consensus can provide a good approximation of that factual truth.

But other questions do not have a single independently verifiable answer. Consider questions related to religion, ethics, political values, or social priorities. Asking whether a belief is simply “true” or “false” may not be the right framing. What can be meaningful, however, is understanding how a particular community perceives that belief: where it converges, where it is divided, and how strongly people hold their views.

COTA is therefore not only a truth-discovery tool. It can also be used to model community perception: a confidence-aware representation of what a group collectively believes about a set of topics.

Naturally, collective agreement should not automatically be mistaken for objective correctness. A population can be misinformed, unrepresentative, or systematically biased. If the goal is to infer factual truth and the consensus disagrees with reliable external evidence, this may reveal a limitation of the population or of the data-collection process (not a new fact about the world).

A small thought experiment

Consider three people assessing a few statements about the Sun. For each statement, they provide two things: an opinion about how likely it is to be true, and a confidence level for that opinion.

For example, one person may be completely confident that “the Sun is a star” is true. Another may believe the opposite, while still reporting high confidence. A third person may simply choose not to answer. This is already richer than a standard poll: COTA can use not only the disagreement itself, but also the certainty that accompanies each contribution.

A basic average would combine all answers in much the same way. COTA has more context to work with. Across the full set of statements, it evaluates which patterns of opinion and confidence fit together best. If a person tends to be reliable when confident, their confident answers become more informative. If someone repeatedly gives highly confident answers that are inconsistent with the broader body of evidence, that confidence becomes less persuasive.

The result is not merely a hard yes-or-no verdict. It can be a nuanced collective estimate: for example, that a statement is considered likely true, uncertain, or unlikely according to the available assessments.

How COTA builds a consensus

COTA starts with an initial estimate for every statement. It then compares this provisional estimate with the opinions supplied by the workers. Contributors whose answers align well with the evolving consensus — especially when they are confident on their opinions — contribute more to the population’s overall trustworthiness.

The algorithm repeatedly refines the estimates until it finds a stable configuration: a set of consolidated opinions that best explains the confidence-weighted responses of the group. No domain-specific knowledge is required. COTA works solely from the opinions and confidence values supplied by the contributors.

Crucially, this does not mean blindly trusting whoever sounds most certain. Confidence is useful only when it is at least somewhat connected to reliability. In fact, the more confident a worker is in an opinion that COTA identifies as incorrect, the less weight COTA assigns to that worker’s opinions overall. Thus, COTA considers confidence as one signal within the collective pattern of agreement and disagreement.

When confidence helps, and when it does not

Confidence can be a valuable signal, but it is not a guarantee of correctness. People are not always well calibrated: some may be accurate but modest, while others may be confidently wrong. In adversarial settings, someone could even deliberately provide misleading answers with high confidence.

We therefore evaluated COTA under several kinds of worker populations. In particular, we compared realistic workers, whose confidence tends to reflect the reliability of their answers, with malicious workers, who systematically provide misleading opinions. The central question was whether COTA could still find a useful consensus when a population contains a mix of both.

 

The results show a clear pattern. When confidence roughly tracks accuracy, COTA can use it to improve the final consolidated opinion. This is especially helpful in heterogeneous groups, where some people are more reliable than others. Instead of treating all contributions identically, the algorithm can make better use of the information already present in the workers’ responses.

We also evaluated the approach using real worker opinions from a dataset in which people rated the intensity of emotions in news headlines. That dataset does not include workers’ own confidence reports, so we generated confidence values related to how close each judgement was to expert assessments. This makes the experiment partly real-world and partly synthetic, and that limitation is important to keep in mind.

Even in this setting, COTA achieved the best overall results among the evaluated methods. Taken together, the experiments suggest two complementary lessons: preserving numerical opinions is valuable, and using confidence at the level of individual tasks can improve the final consensus — when that confidence contains useful information.

 

Why this matters beyond crowdsourcing

COTA is relevant whenever we need to consolidate uncertain knowledge but do not have rich information about the items being assessed. Some modern machine-learning approaches rely on features such as text embeddings, images, or other representations of the underlying data. Those approaches can be powerful, but such features may be unavailable, expensive to obtain, domain-specific, or inappropriate to use.

COTA takes a different route. It relies only on what contributors provide: their opinions and their confidence. This makes it potentially useful in data integration, collaborative knowledge bases, expert elicitation, quality assessment, and other settings where the contributors’ judgements are the main available evidence.

It may also become increasingly relevant in human–AI collaboration. If humans and AI systems both contribute assessments, a future question is how to represent and calibrate their uncertainty in a way that supports better collective decisions — not simply louder or more numerous ones.