{"generator":"Jekyll","link":[{"@attributes":{"href":"https:\/\/whiting.me\/feed.xml","rel":"self","type":"application\/atom+xml"}},{"@attributes":{"href":"https:\/\/whiting.me\/","rel":"alternate","type":"text\/html","hreflang":"en"}}],"updated":"2026-06-03T14:01:03+00:00","id":"https:\/\/whiting.me\/feed.xml","title":"Mark E. Whiting","subtitle":"Mark is Chief Scientist at Pareto and research fellow at University of Pennsylvania","entry":[{"title":"Common sense, machines, and what they don\u2019t know","link":{"@attributes":{"href":"https:\/\/whiting.me\/common-sense-llms\/","rel":"alternate","type":"text\/html","title":"Common sense, machines, and what they don\u2019t know"}},"published":"2025-10-15T00:00:00+00:00","updated":"2025-10-15T00:00:00+00:00","id":"https:\/\/whiting.me\/common-sense-llms","content":"

The framework we introduced<\/a> for quantifying common sense was built around people \u2014 individuals rating claims, and the structure of agreement across a population. The obvious next question is what happens when the raters aren\u2019t human.<\/p>\n\n

With Tuan Dung (Josh) Nguyen<\/a> and Duncan Watts<\/a>, we applied the same methodology at scale to large language models, evaluating commonsense knowledge in humans and in LLMs on the same set of claims PNAS Nexus<\/a>. The comparison is useful both ways: it tells us what current models know in the same terms we used for people, and it exposes where benchmarks designed for humans break down when pointed at a machine.<\/p>\n\n

Running alongside this, a broader community effort \u2014 organized by Vinay Chaudhri<\/a> with many others \u2014 has been articulating what a new knowledge resource for AI might look like, one that goes beyond existing knowledge graphs and taps into the kind of structured, commonsense, and expert knowledge that modern AI systems still struggle to use reliably AI Magazine<\/a>.<\/p>\n\n

Taken together, these projects push at the same question from two sides: how do we measure what machines know, and how do we build the resources that would let them know more?<\/p>","author":{"name":{}},"category":[{"@attributes":{"term":"commonsense"}},{"@attributes":{"term":"LLMs"}},{"@attributes":{"term":"AI"}},{"@attributes":{"term":"knowledge"}},{"@attributes":{"term":"Penn"}}],"summary":"The framework we introduced for quantifying common sense was built around people \u2014 individuals rating claims, and the structure of agreement across a population. The obvious next question is what happens when the raters aren\u2019t human."},{"title":"AI hasn\u2019t fixed teamwork","link":{"@attributes":{"href":"https:\/\/whiting.me\/ai-and-teamwork\/","rel":"alternate","type":"text\/html","title":"AI hasn\u2019t fixed teamwork"}},"published":"2025-09-15T00:00:00+00:00","updated":"2025-09-15T00:00:00+00:00","id":"https:\/\/whiting.me\/ai-and-teamwork","content":"

Every wave of productivity technology arrives with a promise that it will finally make teamwork better. Generative AI is the current one, and perhaps the last one.<\/p>\n\n

With Qing Xiao<\/a>, Xinlan Emily Hu<\/a>, Arvind Karunakaran<\/a>, Hong Shen<\/a>, and Hancheng Cao<\/a>, we followed a project-based software development organization longitudinally from 2023 to 2025 \u2014 through the period in which generative AI tools went from novelty to default \u2014 to see what actually changed in how teams collaborated. The preprint is on arXiv<\/a>.<\/p>\n\n

The short version of what we found is in the title: AI hasn\u2019t fixed teamwork. It has, however, shifted the collaborative culture in measurable and sometimes surprising ways \u2014 in who does what, in what counts as a contribution, and in the texture of day-to-day coordination. That\u2019s a distinct finding from either \u201cAI is transformative\u201d or \u201cAI changes nothing,\u201d and it lines up with a broader pattern in our team research<\/a>: the effects of a new tool depend on the task, the team, and the culture they arrive into.<\/p>","author":{"name":{}},"category":[{"@attributes":{"term":"AI"}},{"@attributes":{"term":"teams"}},{"@attributes":{"term":"collaboration"}},{"@attributes":{"term":"longitudinal"}}],"summary":"Every wave of productivity technology arrives with a promise that it will finally make teamwork better. Generative AI is the current one, and perhaps the last one."},{"title":"A task space for team research","link":{"@attributes":{"href":"https:\/\/whiting.me\/task-space\/","rel":"alternate","type":"text\/html","title":"A task space for team research"}},"published":"2025-09-01T00:00:00+00:00","updated":"2025-09-01T00:00:00+00:00","id":"https:\/\/whiting.me\/task-space","content":"

Teams research is fragmented. Every discipline that studies groups \u2014 organizational behavior, social psychology, HCI, economics, operations \u2014 has its own favorite tasks: brainstorming, jury deliberation, prisoner\u2019s dilemma, hidden profile, creative writing prompts, estimation games. Results on these tasks rarely talk to each other, and it\u2019s genuinely hard to tell whether that\u2019s because the findings conflict or because the tasks do. This problem extends beyond teh task to how tasks are operationalized and how experiments are parameterized and measured.<\/p>\n\n

With Xinlan Emily Hu<\/a>, Linnea Gandhi<\/a>, Duncan Watts<\/a>, and Abdullah Almaatouq<\/a>, we introduce the Task Space<\/strong>: a framework that organizes the tasks teams do along dimensions that matter for how teams actually perform them, so findings from one task can be meaningfully compared with findings from another Management Science<\/a>.<\/p>\n\n

This sits on top of a stack of earlier work that kept pointing at the same gap. In Did It Have To End This Way?<\/a> we showed that the same teams can produce very different outcomes depending on the task in front of them CSCW\u201919<\/a>. In Parallel Worlds<\/a> we re-convened the same teams without them knowing to see how much of their trajectory was locked in CSCW\u201920<\/a>. And in Online Juries<\/a> we found that for some decision tasks, teams are remarkably consistent \u2014 while for others, they\u2019re essentially random CHI\u201921<\/a>.<\/p>\n\n

The Task Space takes the next step: rather than studying individual tasks and hoping the findings generalize, it maps the space those tasks live in so that generalization can be tested directly.<\/p>","author":{"name":{}},"category":[{"@attributes":{"term":"teams"}},{"@attributes":{"term":"tasks"}},{"@attributes":{"term":"integrative"}},{"@attributes":{"term":"framework"}},{"@attributes":{"term":"Penn"}},{"@attributes":{"term":"MIT"}}],"summary":"Teams research is fragmented. Every discipline that studies groups \u2014 organizational behavior, social psychology, HCI, economics, operations \u2014 has its own favorite tasks: brainstorming, jury deliberation, prisoner\u2019s dilemma, hidden profile, creative writing prompts, estimation games. Results on these tasks rarely talk to each other, and it\u2019s genuinely hard to tell whether that\u2019s because the findings conflict or because the tasks do. This problem extends beyond teh task to how tasks are operationalized and how experiments are parameterized and measured."},{"title":"Quantifying common sense","link":{"@attributes":{"href":"https:\/\/whiting.me\/Quantifying-common-sense\/","rel":"alternate","type":"text\/html","title":"Quantifying common sense"}},"published":"2024-01-16T00:00:00+00:00","updated":"2024-01-16T00:00:00+00:00","id":"https:\/\/whiting.me\/Quantifying%20common%20sense","content":"

Common sense is one of those ideas that everyone appeals to and no one agrees on. It\u2019s supposedly universal, but it\u2019s also regularly invoked to complain that other people don\u2019t have any. That tension \u2014 universal in principle, contested in practice \u2014 is actually measurable, if you\u2019re careful about what you count.<\/p>\n\n

With Duncan Watts<\/a> and collaborators, I developed a method to quantify common sense empirically, at two levels: for an individual (how aligned is this person\u2019s take on a claim with everyone else\u2019s?) and for a collective (how much does a group actually agree?). Running the method over a large set of human-rated claims, we found that what we think of as \u201ccommonsense\u201d varies a lot depending on the kind of claim. The clearest agreement shows up on plainly-worded factual claims about the physical world; agreement drops off sharply for claims that are social, normative, or ambiguously worded. Interestingly, who the raters are matters much less than what kind of claim it is. And at the collective level, the universal common sense people often assume exists mostly doesn\u2019t PNAS<\/a>.<\/p>\n\n

The paper\u2019s full significance statement spells this out:<\/p>\n\n

\n

Common sense, while often portrayed as universal, is paradoxically also often claimed not to exist. Here, we resolve this puzzling situation by introducing a formal methodology to empirically quantify common sense both at individual and collective levels. We then demonstrate the method with a dataset involving human raters evaluating claims. We show that common sense varies considerably across types of claims but aligns most closely with plainly worded, factual claims about physical reality; in contrast, does not vary much across different types of people. We also find limited presence of collective common sense, undermining universalist claims and supporting skeptics. Finally, we argue that quantifying common sense is useful both for applications in social science and AI.<\/p>\n<\/blockquote>\n\n

The method turns out to be useful beyond humans \u2014 we\u2019ve since used it to evaluate common sense in large language models<\/a>, where having a single framework that applies to both humans and machines is genuinely handy.<\/p>\n\n

The work has received attention at a few sources, which Altmetric summarize nicely:<\/p>\n\n

<\/div>","author":{"name":{}},"category":[{"@attributes":{"term":"commonsense"}},{"@attributes":{"term":"experiments"}},{"@attributes":{"term":"meta-scientific"}},{"@attributes":{"term":"methods"}},{"@attributes":{"term":"Penn"}}],"summary":"Common sense is one of those ideas that everyone appeals to and no one agrees on. It\u2019s supposedly universal, but it\u2019s also regularly invoked to complain that other people don\u2019t have any. That tension \u2014 universal in principle, contested in practice \u2014 is actually measurable, if you\u2019re careful about what you count."},{"title":"How well can social scientists predict society?","link":{"@attributes":{"href":"https:\/\/whiting.me\/forecasting\/","rel":"alternate","type":"text\/html","title":"How well can social scientists predict society?"}},"published":"2023-04-15T00:00:00+00:00","updated":"2023-04-15T00:00:00+00:00","id":"https:\/\/whiting.me\/forecasting","content":"

One way to ask whether a theory is any good is to ask what it predicts. During COVID, a large group of social scientists ran a forecasting tournament to find out \u2014 teams competed to predict near-term changes in societal outcomes like mood, polarization, political ideology, discrimination, and life satisfaction, using whatever theoretical apparatus they wanted.<\/p>\n\n

I contributed to the effort as part of the Forecasting Collaborative<\/a>, led by Igor Grossmann<\/a> and colleagues. The headline result is sobering: across nearly every outcome, expert forecasts were not reliably better than simple statistical benchmarks, and often worse Nat Hum Behav<\/a>.<\/p>\n\n

That finding fits uncomfortably well with the argument we make in Beyond Playing 20 Questions with Nature<\/a> and in our work on common sense<\/a>: if a field can\u2019t generate reliable predictions about the phenomena it studies, it\u2019s worth asking hard questions about how the field is cumulating knowledge in the first place. Forecasting tournaments are one of the cleaner tests we have \u2014 they are hard to game, easy to score, and directly tied to the kind of claims theories are supposed to support.<\/p>","author":{"name":{}},"category":[{"@attributes":{"term":"forecasting"}},{"@attributes":{"term":"societal"}},{"@attributes":{"term":"change"}},{"@attributes":{"term":"meta-science"}},{"@attributes":{"term":"Penn"}}],"summary":"One way to ask whether a theory is any good is to ask what it predicts. During COVID, a large group of social scientists ran a forecasting tournament to find out \u2014 teams competed to predict near-term changes in societal outcomes like mood, polarization, political ideology, discrimination, and life satisfaction, using whatever theoretical apparatus they wanted."},{"title":"Integrative experiments","link":{"@attributes":{"href":"https:\/\/whiting.me\/Integrative-experiments\/","rel":"alternate","type":"text\/html","title":"Integrative experiments"}},"published":"2022-12-21T00:00:00+00:00","updated":"2022-12-21T00:00:00+00:00","id":"https:\/\/whiting.me\/Integrative%20experiments","content":"

The dominant way we run experiments in social and behavioral science \u2014 one experiment at a time, each treated as a test of a theory assumed to generalize \u2014 has a serious problem. The integration across experiments that is supposed to happen in the published record largely doesn\u2019t, and the recent push for more reliable single findings doesn\u2019t fix it. You can do every individual experiment perfectly well and still not end up with a cumulative theory.<\/p>\n\n

With Abdullah Almaatouq<\/a>, Tom Griffiths<\/a>, Jordan Suchow<\/a>, James Evans<\/a>, and Duncan Watts<\/a>, we argue that the fix has to happen at the level of experimental design. In integrative experiments<\/strong>, researchers explicitly map the space of possible experiments associated with a research question, then iteratively sample from that space. Instead of trying to defend a single experimental condition as the one that captures the phenomenon, you treat the design space itself as the object of study BBS<\/a>.<\/p>\n\n

The paper drew a large set of commentaries from across the field, which we responded to in a follow-up BBS<\/a>. The discussion \u2014 about what theories are for, how generalization should work, and whether the field needs a different unit of analysis \u2014 is, for me, the most interesting part of the project.<\/p>\n\n

Most of the other threads in my recent work connect here. Empirica<\/a> is the infrastructure that makes integrative experiments actually runnable; the common sense framework<\/a> and the task space<\/a> are both attempts to make the design space in a particular domain tractable; and the forecasting results<\/a> are one way of showing that the status quo is leaving a lot on the table.<\/p>","author":{"name":{}},"category":[{"@attributes":{"term":"integrative"}},{"@attributes":{"term":"experiments"}},{"@attributes":{"term":"meta-scientific"}},{"@attributes":{"term":"methods"}},{"@attributes":{"term":"Penn"}},{"@attributes":{"term":"MIT"}}],"summary":"The dominant way we run experiments in social and behavioral science \u2014 one experiment at a time, each treated as a test of a theory assumed to generalize \u2014 has a serious problem. The integration across experiments that is supposed to happen in the published record largely doesn\u2019t, and the recent push for more reliable single findings doesn\u2019t fix it. You can do every individual experiment perfectly well and still not end up with a cumulative theory."},{"title":"Ambiguity online","link":{"@attributes":{"href":"https:\/\/whiting.me\/online-ambiguity\/","rel":"alternate","type":"text\/html","title":"Ambiguity online"}},"published":"2022-05-01T00:00:00+00:00","updated":"2022-05-01T00:00:00+00:00","id":"https:\/\/whiting.me\/online-ambiguity","content":"

Face-to-face, a huge amount of what we communicate is nonverbal \u2014 a pause, a glance, a shrug. Online, most of that channel disappears, and the little that remains gets compressed into things like read receipts, likes, profile changes, and emoji. These signals are easy to produce and easy to misread.<\/p>\n\n

With So Yeon Park<\/a> and Michael Shanks<\/a> at Stanford, as part of the HPI-Stanford Design Thinking Research Program, we studied these \u201cnonverbal online actions\u201d \u2014 how people use them, how others interpret them, and where the gaps between sender and receiver sit DTR\u201921<\/a>. A follow-up looked specifically at what happens when those actions create confusion, and what people do to repair it DTR\u201922<\/a>.<\/p>\n\n

A sharper version of the same problem is synthetic media. With Dilrukshi Gamage<\/a>, Piyush Ghasiya<\/a>, Vamshi Bonagiri<\/a>, and Kazutoshi Sasahara<\/a>, we analyzed Reddit conversations about deepfakes to see how people actually talk about them \u2014 the concerns they raise, the distinctions they draw, and the societal implications that surface in their own words CHI\u201922<\/a>.<\/p>\n\n

Across these projects, the pattern that keeps coming up is that online communication is less a thinner version of in-person communication than it is a different medium with its own ambiguities \u2014 and the interesting research question is usually how people navigate those ambiguities, not whether they exist.<\/p>","author":{"name":{}},"category":[{"@attributes":{"term":"online"}},{"@attributes":{"term":"communication"}},{"@attributes":{"term":"nonverbal"}},{"@attributes":{"term":"deepfakes"}},{"@attributes":{"term":"Stanford"}}],"summary":"Face-to-face, a huge amount of what we communicate is nonverbal \u2014 a pause, a glance, a shrug. Online, most of that channel disappears, and the little that remains gets compressed into things like read receipts, likes, profile changes, and emoji. These signals are easy to produce and easy to misread."},{"title":"COVID dashboards","link":{"@attributes":{"href":"https:\/\/whiting.me\/COVID-dashboards\/","rel":"alternate","type":"text\/html","title":"COVID dashboards"}},"published":"2021-08-15T00:00:00+00:00","updated":"2021-08-15T00:00:00+00:00","id":"https:\/\/whiting.me\/COVID-dashboards","content":"

Together with Duncan Watts<\/a> and others on the team at the CSSLab<\/a> at Penn, and in conjunction with the city of Philadelphia, we built a collection of interactive data dashboards that visually summarize human mobility patterns over time and space for a number of cities, starting with Philadelphia, along with highlighting potentially relevant demographic correlates. The dashboards are available at covid.seas.upenn.edu<\/a>.<\/p>\n\n

The dashboards are built in Observable<\/a>. The data are a proprietary combination of cell phone GPS data, demographic data derived from the American Community Survey, and COVID-19 caseload data from the New York Times.<\/p>\n\n

A related piece of the same broader effort was the non-pharmaceutical interventions (NPI) dataset<\/strong>, where a large team of student annotators tracked how local COVID-19 policy \u2014 mask mandates, business closures, gathering limits, and so on \u2014 changed in near-real-time across many jurisdictions. Policies shifted week to week, and there was no single authoritative source for what was actually in force where, which made modeling the pandemic\u2019s trajectory much harder than it needed to be. With Benjamin Hurt<\/a>, Madhav Marathe<\/a>, Michael Bernstein<\/a>, and many others, we released the resulting annotated dataset and the annotation workflow behind it Scientific Data<\/a>.<\/p>","author":{"name":{}},"category":[{"@attributes":{"term":"COVID-19"}},{"@attributes":{"term":"dashboard"}},{"@attributes":{"term":"observable"}},{"@attributes":{"term":"data"}},{"@attributes":{"term":"visualization"}},{"@attributes":{"term":"Penn"}}],"summary":"Together with Duncan Watts and others on the team at the CSSLab at Penn, and in conjunction with the city of Philadelphia, we built a collection of interactive data dashboards that visually summarize human mobility patterns over time and space for a number of cities, starting with Philadelphia, along with highlighting potentially relevant demographic correlates. The dashboards are available at covid.seas.upenn.edu."},{"title":"Online learning at scale","link":{"@attributes":{"href":"https:\/\/whiting.me\/online-learning\/","rel":"alternate","type":"text\/html","title":"Online learning at scale"}},"published":"2021-06-15T00:00:00+00:00","updated":"2021-06-15T00:00:00+00:00","id":"https:\/\/whiting.me\/online-learning","content":"

Massive open online courses promised education at a scale that in-person teaching can\u2019t reach, but the instructor\u2013student ratio that works in a seminar doesn\u2019t survive the jump to tens of thousands of learners. So much of what makes a class work \u2014 feedback, discussion, a sense that anyone is paying attention \u2014 has to be reconstructed out of what the learners themselves do for each other.<\/p>\n\n

Over several years with Dilrukshi Gamage<\/a>, Indika Perera<\/a>, Shantha Fernando<\/a>, Thomas Staubitz<\/a> and others, we studied how to make peer-driven learning actually work. We found that aligning incentives to the quality of feedback, rather than simply its length, produced feedback learners judged more useful L@S\u201917<\/a>. We showed that seeding peer assessment groups with trained \u201cintroduced peers\u201d improved the quality of subsequent discussion TALE\u201918<\/a>. We surveyed the broader state of the field in a systematic literature review of peer assessment in MOOCs Distance Education<\/a>. And we showed that treating learners as communities of practice \u2014 rather than isolated users moving through a course \u2014 measurably improves outcomes Asian CHI\u201921<\/a>, which was recognized with a best paper award.<\/p>\n\n

The through-line across these projects is that scale doesn\u2019t remove the social structure of learning; it just changes who has to supply it.<\/p>","author":{"name":{}},"category":[{"@attributes":{"term":"MOOCs"}},{"@attributes":{"term":"peer"}},{"@attributes":{"term":"assessment"}},{"@attributes":{"term":"online"}},{"@attributes":{"term":"learning"}}],"summary":"Massive open online courses promised education at a scale that in-person teaching can\u2019t reach, but the instructor\u2013student ratio that works in a seminar doesn\u2019t survive the jump to tens of thousands of learners. So much of what makes a class work \u2014 feedback, discussion, a sense that anyone is paying attention \u2014 has to be reconstructed out of what the learners themselves do for each other."},{"title":"Running experiments at scale","link":{"@attributes":{"href":"https:\/\/whiting.me\/empirica\/","rel":"alternate","type":"text\/html","title":"Running experiments at scale"}},"published":"2021-03-15T00:00:00+00:00","updated":"2021-03-15T00:00:00+00:00","id":"https:\/\/whiting.me\/empirica","content":"

Most of the empirical work I do with collaborators needs infrastructure that doesn\u2019t really exist off the shelf. Running a single behavioral experiment with a handful of people is well supported; running thousands of them, each with groups of participants interacting in real time, is not.<\/p>\n\n

Lead by Abdullah Almaatouq<\/a>, and with Joshua Becker<\/a>, James Houghton<\/a>, Nicolas Paton<\/a>, and Duncan Watts<\/a>, we built and extended Empirica<\/a>, an open-source virtual lab designed for exactly this kind of high-throughput, macro-level experimentation. Empirica handles the hard parts \u2014 synchronization, treatment randomization, real-time interaction, and iterative design \u2014 so researchers can focus on the experiment itself BRM\u201921<\/a>.<\/p>\n\n

Empirica was one piece of a broader conversation, across many labs, about what it would take to actually scale up behavioral and social science. In a working paper with a large group of collaborators we laid out a vision for shared infrastructure, shared protocols, and shared participant access as the missing middle of the field OSF\u201921<\/a>.<\/p>\n\n

This infrastructure turned out to be essential to the work on integrative experiments<\/a> that came next \u2014 you can\u2019t run a design space full of experiments if each one costs you a month of engineering.<\/p>","author":{"name":{}},"category":[{"@attributes":{"term":"Empirica"}},{"@attributes":{"term":"infrastructure"}},{"@attributes":{"term":"virtual"}},{"@attributes":{"term":"lab"}},{"@attributes":{"term":"MIT"}},{"@attributes":{"term":"Penn"}}],"summary":"Most of the empirical work I do with collaborators needs infrastructure that doesn\u2019t really exist off the shelf. Running a single behavioral experiment with a handful of people is well supported; running thousands of them, each with groups of participants interacting in real time, is not."}]}