Extracting Human Face Similarity Judgments: Pairs or Triplets?

Journal of Visualized Experiments, 2022

Similarity judgments are commonly used to study mental representations and their neural correlates. This approach has been used to characterize perceptual spaces in many domains: colors, objects, images, words, and sounds. Ideally, one might want to compare estimates of perceived similarity between all pairs of stimuli, but this is often impractical. For example, if one asks a subject to compare the similarity of two items with the similarity of two other items, the number of comparisons grows with the fourth power of the stimulus set size. An alternative strategy is to ask a subject to rate similarities of isolated pairs, e.g., on a Likert scale. This is much more efficient (the number of ratings grows quadratically with set size rather than quartically), but these ratings tend to be unstable and have limited resolution, and the approach also assumes that there are no context effects. Here, a novel ranking paradigm for efficient collection of similarity judgments is presented, along with an analysis pipeline (software provided) that tests whether Euclidean distance models account for the data. Typical trials consist of 8 stimuli around a central reference stimulus: the subject ranks stimuli in order of their similarity to the reference. By judicious selection of combinations of stimuli used in each trial, the approach has internal controls for consistency and context effects. The approach was validated for stimuli drawn from Euclidean spaces of up to 5 dimensions. The approach is illustrated with an experiment measuring similarities among 37 words. Each trial yields the results of 28 pairwise comparisons of the form, "Was A more similar to the reference than B was to the reference?" While directly comparing all pairs of pairs of stimuli would have required 221445 trials, this design enables reconstruction of the perceptual space from 5994 such comparisons obtained from 222 trials.

2017

The card sorting problem involves the similarity judgments of pairs of photos, taken from a set of photos, by a group of participants. Given the lack of an objective standard for judging similarity, different participants may be using different strategies in judging the similarity of photos. It could be very useful to identify and study these strategies. In this paper, we present a framework for three-way analysis of judgments of similarity. Based on judgments by the set of participants, we divide all pairs of photos into three classes: a set of similar pairs that are judged by at least 60% of participants as similar; a set of dissimilar pairs that are judged by at least 60% of participants as dissimilar; and a set of undecidable pairs that have conflicting judgments. A more refined three-way classification method is also suggested based on a quantitative description of the quality of similarity judgments. The classification in terms of three classes provides an effective method to ...

Measuring perceived similarity is an important issue in visual perception of faces, since a measure of the perceived similarity between faces may be used to investigate fundamental tasks like face categorization and recognition. Despite its fundamental role, measuring perceived similarity between faces is not trivial from both a theoretical and methodological point of view. In this paper we present theoretical arguments that undermine the method currently most used to measure perceived similarity between faces in visual perception, and we propose an alternative method. We finally compare the two methods and find some empirical evidence that the proposed method can provide a more reliable evaluation of the perceived similarity between faces.

2012

In two experiments, participants were presented with a triad of morphed White and Hispanic faces paired with pseudoword labels. The meanings of these labels were manipulated to represent categorical information about the face. Labels were said to represent either the person's belief, the food s/he ate, the disease s/he had, or the person's last name. The results indicated that categorical information affects our judgments of faces. Information categories such as belief, food, and diseases were particularly strong in modifying the participants' similarity judgment of faces, whereas information characterized with last names of faces were least powerful. Previous research focuses on race face perception being affected primarily by racial indicators or racial information. Our results provide that how we perceptually analyze faces is not confined to obvious racial cues, but by non-racial semantic information as well, suggesting that category-relevant information by itself provides a strong basis for inductive generalization.

Over the last decade researchers have devised algorithms that can provide similarity measures between pairs of face images. These have been somewhat successful in estimating the similarities between face images under controlled conditions. However, those similarity measures do not parallel subjective similarity, as perceived by humans. In some applications it is important to have a similarity metric that closely parallels that of humans. This paper describes a method for discovering the high-level features that are used by humans to judge facial similarity through the use of “lexical basis functions ” gleaned from a lexicon of the English language. This method estimates the similarity of each pair of images in a set of face images by two independent methods – by the subjective evaluation of human observers, and by the use of “lexical basis functions ” to represent the multidimensional content of each image with a feature vector. The similarity measure computed with these feature vectors is shown to correlate with the subjective judgment of human observers, and thus provides both a more objective method for evaluating and expressing image content, and a possible path to automating the process of similarity measurement in the future.

Uncertainty Management with Fuzzy and Rough Sets

A fundamental task in the sorting of facial photographs is the modelling of pairwise facial similarity. A three-way analysis of facial similarity described in this work uses data obtained from card sorting of a set of facial photographs, done by a group of participants. Participants were asked to sort the photographs into an unrestricted number of piles, using their own judgements of similarity to place similar photos into the same pile. Photos placed into different piles are considered to be dissimilar. In particular, each participant compared the photo to be sorted with the last photo placed on top of each pile. The decision faced by each participant is to add the photo to an existing pile or to create a new pile. Given the lack of an objective standard for judging similarity, different participants may be using different strategies in judging the similarity of photos. It could be very useful to identify and study these strategies. An overall evaluation of similarity can be obtained by synthesizing judgments from the set of participants. A two-way analysis classifies a pair of photos as either similar or dissimilar. This may be too restrictive. Motivated by the three regions in rough set theory, in this work we present a framework for three-way analysis of judgments of facial similarity. Based on judgments by the set of participants, we divide all pairs of photos into three classes: a set of similar pairs that are judged primarily as similar; a set of dissimilar pairs that are judged primarily as dissimilar; and a set of undecidable pairs that have conflicting judgments. A more refined threeway classification method is also suggested based on a quantitative description of

Current theories of cognition are cast in terms of information processing mechanisms that use mental representations. For example, consider the mechanisms of face identification that use mental representations to identify familiar faces under various conditions of pose, illumination and ageing, or to draw resemblance between family members. Providing an explanation of these information processing mechanisms thus relies on showing how the actual information contents of these representations are used. Yet, these representational contents are rarely characterized, which in turn hinders knowledge of mechanisms. Here, we address this pervasive gap by characterizing the detailed contents of mental representations of familiar faces using a new methodological approach. We used a unique generative model of face identity information combined with perceptual judgments and reverse correlation to model the 3D representational contents of 4 familiar faces in 14 participants. We then demonstrated ...

PloS one, 2016

According to the norm-based version of the multidimensional face space model (nMDFS, Valentine, 1991), any given face and its corresponding anti-face (which deviates from the norm in exactly opposite direction as the original face) should be equidistant to a hypothetical prototype face (norm), such that by definition face and anti-face should bear the same level of perceived typicality. However, it has been argued that familiarity affects perceived typicality and that representations of familiar faces are qualitatively different (e.g., more robust and image-independent) from those for unfamiliar faces. Here we investigated the role of face familiarity for rated typicality, using two frequently used operationalisations of typicality (deviation-based: DEV), and distinctiveness (face in the crowd: FITC) for faces of celebrities and their corresponding anti-faces. We further assessed attractiveness, likeability and trustworthiness ratings of the stimuli, which are potentially related to...

arXiv (Cornell University), 2020

Advances in object recognition flourished in part because of the availability of high-quality datasets and associated benchmarks. However, these benchmarks-such as ILSVRC-are relatively task-specific, focusing predominately on predicting class labels. We introduce a publiclyavailable dataset that embodies the task-general capabilities of human perception and reasoning. The Human Similarity Judgments extension to ImageNet (ImageNet-HSJ) is composed of human similarity judgments that supplement the ILSVRC validation set. The new dataset supports a range of task and performance metrics, including the evaluation of unsupervised learning algorithms. We demonstrate two methods of assessment: using the similarity judgments directly and using a psychological embedding trained on the similarity judgments. This embedding space contains an order of magnitude more points (i.e., images) than previous efforts based on human judgments. Scaling to the full 50,000 image set was made possible through a selective sampling process that used variational Bayesian inference and model ensembles to sample aspects of the embedding space that were most uncertain. This methodological innovation not only enables scaling, but should also improve the quality of solutions by focusing sampling where it is needed. To demonstrate the utility of ImageNet-HSJ, we used the similarity ratings and the embedding space to evaluate how well several popular models conform to human similarity judgments. One finding is that more complex models that perform better on task-specific benchmarks do not better conform to human semantic judgments. In addition to the human similarity judgments, pre-trained psychological embeddings and code for inferring variational embeddings are made publicly available. Collectively, ImageNet-HSJ assets support the appraisal of internal representations and the development of more human-like models.

Journal of Memory and Language, 1997

The structural-alignment approach to similarity posits a principled distinction between object attributes and relations between objects. We examined whether this assumption holds for nonarbitrary combinations of interrelated objects. Subjects judged similarity between simple statements in which the nouns (denoting attributes) and verbs (denoting relations) were semantically interdependent. We found that semantic dependencies affected similarity judgments both by inducing inferences about the abstract combined meaning of the statements and by changing the process by which subjects arrived at their judgments. When the paired statements had matching verbs (e.g.