See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/374888271

Assessing 36 Years of the Forced Choice Design in Extra Sensory Perception

Research: A Meta-Analysis, 1987 to 2022: A Meta-Analysis, 1987 to 2022

Article in Journal of Scientific Exploration · October 2023

DOI: 10.31275/20232967

CITATIONS READS

6 437

2 authors:

Lance Storm, Patrizio Tressoldi

University of Adelaide University of Padova - Italy
75 PUBLICATIONS 1,173 CITATIONS 294 PUBLICATIONS 4,395 CITATIONS

SEE PROFILE SEE PROFILE

All content following this page was uploaded by Patrizio Tressoldi on 05 November 2023.

The user has requested enhancement of the downloaded file.

 Journal of
Anomalistics
Scientific and

Exploration
Frontier
Science

Assessing 36 Years of the Forced

SPECIAL
SUBSEC TION Choice Design in Extra Sensory
Perception Research: A Meta-
Analysis, 1987 to 2022
HIGHLIGHTS
Lance Storm
Statistical analysis shows that forced-choice experiments successfully demonstrate
[email protected]
potential extrasensory perception, yet only to a small degree.

Patrizio Tressoldi
ABSTRACT
[email protected] This meta-analysis is an update of Storm, Tressoldi, and Di Risio (2012); a meta-analysis
on forced-choice ESP studies (1987 to 2010), which use targets such as card symbols,
pictures, and letters. We formed two datasets: New Studies #1 (studies that included
actual hit rates) and New Studies #2 (Reaction Time [‘RT’] Studies; which are studies
SUBMITTED April 12, 2023
that measured only reaction time, not hits, as indicators of psi responses). New Studies
ACCEPTED June 29, 2023
PUBLISHED October 31, 2023 #1: For the period 2011 to 2022, a homogeneous dataset of 38 studies yielded a mean
effect size (ES) of 0.02 (Stouffer Z = 5.55, p = 1.43 × 10–8). New Studies #2 (‘RT’ Studies):
For the same period, a homogeneous dataset of 23 studies yielded a weaker mean ES
of 0.01 (Stouffer Z = 5.50, p = 1.90 × 10–8). The two databases were combined. In this
dataset, telepathy, clairvoyance, and precognition studies were not significantly differ-
ent from each other. Nor were target types. We updated the forced-choice database by
combining our revised original database with the new studies to form a homogeneous
database (N = 141): mean effect size (ES) of 0.02 (Stouffer Z = 8.52, p < 10-16). Effects
did not vary between investigators or laboratories, and we found a near-significant in-
cline in ES values over a 36-year period (i.e., no evidence of a decline). These results
confirm that the forced-choice design adequately tests extra-sensory perception (ESP).
We compare the overall results with those obtained in other domains, focusing on ‘se-
https://doi.org/10.31275/20232967 lected’ participants (meditators, psychics, psi-test experienced) and ‘unselected’ (i.e.,
untrained, naïve) participants.
PLATINUM OPEN ACCESS
KEYWORDS
Creative Commons License 4.0.
CC-BY-NC. Attribution required.
Consciousness, ESP, extra-sensory perception, forced-choice, meta-analysis, psi.
No commercial use.

INTRODUCTION cle, and cross) named after perceptual psychologist Karl

Zener (see Figure 1). The problem with the 52-card deck
Biologist J. B. Rhine was one of the first empirical is that extra-sensory perception (ESP) effects can be de-
investigators of alleged paranormal phenomena. He is termined statistically in a number of ways (i.e., by suit,
most famous for his card-guessing studies (Rhine et al., by face value, by color), giving skeptics the opportunity
1940/1966), first using the standard 52-card deck of play- to accuse experimenters of “hypothesis saving” (Irwin &
ing cards, and then switching to so-called Zener cards Watt, 2007, p. 52).
(consisting of five symbols: star, wavy lines, square, cir- Rhine’s various kinds of card-guessing experiments

journalofscientificexploration.org JOURNAL OF SCIENTIFIC EXPLORATION • VOL. 37, NO 3 – FALL 2023 517

FORCED CHOICE ESP META ANALYSIS 1987-2022 Lance Storm & Patrizio Tressoldi

Figure 1. The Zener card symbols: star, wavy lines, square, circle, and cross. The deck of 25 cards has five of each sym-
bol.
came to be known generally as the ‘forced-choice’ de- thoughts, feelings or activity of another conscious being”
sign, which soon produced impressive findings (Rhine, (Thalbourne, 2003, p. 125), but in the case of the forced-
1934). In the typical forced-choice ESP design, the target choice design, the agent (sender) is sensorially shielded
to be ‘guessed’ (i.e., identified without sensory clues) is from the percipient (receiver), and the agent’s thoughts
“one of a limited range of possibilities which are known are confined to ESP targets. Clairvoyance is defined as
to [the participant] in advance” (Thalbourne, 2003, p. 44). “paranormal acquisition of information concerning an
The advantage of card-guessing is that outcomes are un- object [or ESP target suitable for a forced-choice design]
ambiguous, so independent judging is not required, but or contemporary physical event” (Thalbourne, 2003, p.
the long runs of multiple trials were tedious and boring 18). Precognition is “a form of extra-sensory perception
for participants, which saw declines in hit rates. Rhine’s in which the target is some future event that cannot be
forced-choice design also came under criticism for its deduced from normally known data in the present” (Thal-
methodological flaws, such as sensory leakage due, for bourne, 2003, p. 90). The ‘future event’ may include an
example, to subtle tell-tale folds on cards or poor-quality ESP target suitable for a forced-choice study. The follow-
printing that leaked information through to the reverse ing section is a review of ESP forced-choice studies with a
sides of the cards. Although these problems were rec- focus on modalities.
tified by Rhine, card testing ultimately fell out of vogue
over the decades with the introduction of computers and Forced-Choice Telepathy, Clairvoyance, and Pre-
the adoption of more interesting, ecologically valid tar- cognition
gets. These target types still include card symbols, but Tart (1983) found 85 forced-choice studies but dis-
also used nowadays are pictures, alphabet letters, words, carded studies if they did not reach independent signif-
shapes, and so forth. Studies in recent decades now in- icance nominally set at a critical α = .05. He found that
clude fractal images, SMS messages, and Chinese char- real-time ESP (i.e., clairvoyance/telepathy) outperformed
acters. precognition. Steinkamp et al. (1998) argued that Tart’s
As the number of forced-choice studies accumulated selection criteria introduced a bias which may have put
from research conducted around the world and the tech- the precognition studies at a disadvantage.
nique of meta-analysis came to the fore, it became pos- Honorton and Ferrari (1989) only looked at precogni-
sible to evaluate large datasets of studies. A number of tion forced-choice experiments from 1935 to 1987. Partic-
forced-choice meta-analytic studies are extant in the lit- ipants had to “predict the identity of target stimuli select-
erature, and these have generally confirmed a consistent ed randomly over intervals ranging from several hundred
so-called ‘psi’ (i.e., paranormal) effect independent of ex-
milliseconds to 1 year following the subjects’ responses”
perimenters/laboratories worldwide. The forced-choice
(p. 281). Out of 309 studies, 92 (30%) showed significant
psi effect, however, has varied in strength and tends to
hit-rates at the 5% level. The authors noted that expe-
be weak (see Honorton & Ferrari, 1989; Steinkamp et al.,
rienced participants performed better than naïve par-
1998; Storm, Tressoldi, & Risio, 2012; Tart, 1983).
ticipants. They also found that precognition, although
A major aim of the present meta-analysis is to eval-
weak, produced “very robust” and highly significant re-
uate the performance of forced-choice studies conduct-
sults across a time span of more than 50 years, with the
ed since our last meta-analysis, and thus, we planned
quality of studies remaining stable, or even improving, in
to cover the period 2011 to 2022. This evaluation may
that time. Honorton and Ferrari’s meta-analysis revealed
provide statistical evidence that there is an anomalous
that the largest effect sizes were found in experiments
sensory modality that can manifest as either ‘telepathy’,
using (a) experienced (i.e., selected) participants and (b)
‘clairvoyance’, or ‘precognition’. Telepathy refers to the
trial-by-trial feedback. These are important factors for fu-
“paranormal acquisition of information concerning the
ture psi researchers to consider in their designs, and we

518 JOURNAL OF SCIENTIFIC EXPLORATION • VOL. 37, NO 3 – FALL 2023 journalofscientificexploration.org

Lance Storm & Patrizio Tressoldi FORCED CHOICE ESP META ANALYSIS 1987-2022

will assess the influence of both in our post hoc analyses. tween -0.006 and 0.14, with 17 (74%) of them positive.
The meta-analysis by Steinkamp et al. (1998), for the In conclusion, the general findings are that (i) forced-
period 1935 to 1997, was a comparison of ‘matched’ (pro- choice effects are very weak (i.e., small) but consistent
cedurally similar) clairvoyance and precognition studies in and suggestive of psi; (ii) extroversion is a significant cor-
order to test for a phenomenological difference between relate of psi (found twice); (iii) selected participants per-
the two modalities. They found 22 comparable study- form better than unselected participants (found twice),
pairs, but tests showed no difference. They concluded and (iv) there is some evidence that no psi modality (te-
that the burden of proof rested with those “who argue lepathy, clairvoyance, or precognition) is superior to any
for a difference between effect sizes under real-time and other. However, with the noted exclusion of the study by
future ESP” (p. 209). Storm et al. (2012), little attention has been given in these
For the very comprehensive expansive period 1880 to meta-analyses to the influence on psi of target types, the
1989, Steinkamp’s (2005) review of forced-choice stud- experimenter effect, the number of choices in the target
ies showed that there were few variables that correlate set (i.e., k-choices), and the decline effect. These factors
with psi success, partly because the variability in study are considered next. However, our main aim in the pres-
designs made it difficult to discern clear patterns due to ent study is to see if we can replicate the results of our
conflicting outcomes. Nevertheless, she found “low neu- original six hypotheses (see Storm et al., 2012, p. 248).
roticism, extraversion, and good social adjustment may Tests on other variables are relegated to the section, Post
be positively related to forced-choice ESP scoring” (p. Hoc Analyses.
158). She also found some support for the notion that ‘se-
lected’ (i.e., meditators, psychics, previously psi tested) Target Types
participants performed better than unselected partici- As mentioned above, target types are now quite
pants in precognition tests, whereas those who did not varied. Most trials are conducted on-screen using com-
believe in psi scored lower than those who did (see also puter monitors rather than with decks of cards or other
the meta-analysis on paranormal belief by Storm & Tress- physical objects. Apart from conventional images such
oldi, 2017). as photographs of faces, Zener cards, pictures, drawings,
Storm et al. (2012) reported the following in their letters, and numbers, ecologically valid targets are also
meta-analysis covering the period 1987 to 2010: For a ho- being used, such as racing-horse images as targets on
mogeneous dataset of 72 studies, there was a very weak simulated racetracks (e.g., Roe, Davey & Stevens, 2003),
but significant mean effect size (ES), calculated from fractal images (Luke, Delanoy, & Sherwood, 2008), and
the formula z/√n (where z is a standardized score with a
Chinese characters (Vernon, Hitchman, & Roe, 2021).
mean of 0.00 and SD of 1.00; and n is the number of stud-
Previously, we argued that there is no consensus
ies). ES did not correlate with study quality, and there was
on whether some target types are “uninteresting and/
no evidence of selective reporting. Clairvoyance and pre-
or meaningless (e.g., Zener cards, numbers, letters) com-
cognition studies were not significantly different, as was
pared to others that may be emotionally stimulating and/
found previously by Steinkamp et al. (1998). Also, ES did
or meaningful (e.g., divinatory readings, real pictures,
not vary between investigators. Storm et al. also found
video clips)” (Storm et al., 2012, p. 246). In that paper,
that target type did not make a difference to effect size,
it was shown that target type did not make a difference
but they did find suggestive evidence that the number of
to effect size, although there were significantly stronger
choices per trial was inversely related to the p-value. Ev-
mean effects for word/letter targets and for objects in the
idence of a linear incline in ES values was also found over
telepathy condition compared to the other two modali-
the period 1987 to 2010.
ties (clairvoyance and precognition). In the present study,
The most recent meta-analytic treatment of forced-
we once again test whether target type has an influence
choice studies was by Zdrenka and Wilson (2017). They
on effect size.
meta-analyzed 55 studies dating from 1945 to 2016 but
evaluated precognition studies only. They found that psi Experimenter Effect
performance correlated significantly with six individual
difference variables: “luck belief (the belief that luck is Experimenter-psi (or E-psi) becomes a problem when
primarily controllable), perceptual defensiveness, open- parapsychologists want to know that participants are ex-
ness to experience, belief in psi, extraversion, and time clusively responsible for psi; not the experimenter. How-
belief as dynamic” (p. 9). They did not present an overall ever, E-psi has not been found across a broad range of
effect size value for the 55 studies. However, of the 23 investigators in various meta-analyses (Bem & Honorton,
measures in their Table 2, effect-size values (r) ranged be- 1994; Honorton et al., 1990; Storm, Tressoldi, & Di Risio,

journalofscientificexploration.org JOURNAL OF SCIENTIFIC EXPLORATION • VOL. 37, NO 3– FALL 2023 519

FORCED CHOICE ESP META ANALYSIS 1987-2022 Lance Storm & Patrizio Tressoldi

2010). Indeed, pertinent to the present study on forced- meta-analytic review of the new forced-choice studies
choice studies only, Storm et al. (2012) found “no single (dating from 2011 to 2022) to determine if a comprehen-
group that produced effects significantly different from sive, up-to-date database is still significant, as was the
any other group” (p. 259; see also Honorton & Ferrari, case with the Storm et al. meta-analysis. Further to that,
1989). The same effect will be tested in the present study. we aim to see if we can replicate the results of our original
six hypotheses (see Storm et al., 2012, p. 248), in addition
High k-Choice Designs to testing the difference between the ‘old’ (1987-2010)
and ‘new’ (2011-2022) datasets. Tests on other variables
Parapsychologists are also interested in whether or
(feedback, participant type) are relegated to the section,
not declines in effect size might be due to the number of
Post Hoc Analyses. The following hypotheses were thus
choices k (i.e., the number of target choices) in a target
proposed:
set. Timm (2000) argued that effect size measures have
limited use if they do not adequately account for k. He
1. Forced-choice studies produce statistical evidence of a
argues that “the significance of ESP experiments must in-
communications anomaly known as ESP;
crease not only with n but also with decreasing hit prob-
2. The mean ES values for telepathy, clairvoyance, and
ability p (or with increasing number of target alternatives
precognition are different;
k = 1/p)” (p. 253). The values of k have been quite variable
3. ES values vary in strength according to target types;
over the decades in the forced-choice domain, ranging
4. ES values vary between experimenters/laboratories;
from 2 to as many as 26. Empirical support for Timm’s
5. Number of choices (k) per trial is positively related to z;
claim is minimal for forced-choice, but worthy of note.
6. ES values increased over the period of analysis (i.e.,
Specifically, Storm et al. (2012) found a significant pos-
2011–2022);
itive correlation between k values and z scores. That is,
7. The original database (1987 to 2010), and the new data-
z scores tended to increase as k increased. If this finding
base (2011 to 2022), are not different and can be com-
indicates a valid effect, researchers may wish to consider
bined to form a larger dataset.
using high k-choice designs. In the present study, we will
again test whether the number of target choices per trial Should Hypothesis 7 be supported, we will conduct
is related to z. additional (post hoc) analyses on a single combined da-
tabase to further test the validity of the forced-choice
Decline Effects
paradigm.
It is thought that significant declines in effects over
the long-term (i.e., decades) indicate some kind of arti- METHOD
fact due to, say, improvements in study quality over the
years or deliberate changes in experimental design (from Meta-Analysis Reporting Guideline
simple and fun to complex and tedious). These changes We followed the APA meta-analysis reporting stan-
are made because theoretically oriented experimenters dards guideline (MARS, Appelbaum et al., 2018).
want to understand the psi process rather than merely
prove the presence of a so-called communication anom- Study Retrieval
aly. While lengthy (chronological) declines have been
noted on occasion in the parapsychological literature, • The period of interest was January 2011 to December
generally, these declines are spurious, as shown in a large 2022. Full-text studies were retrieved from the follow-
collection of meta-analyses reviewed by Storm (in press). ing sources:
For forced-choice studies in particular, researchers have • The meta-analyses by Bem, Tressoldi, Rabeyron, and
not found declines (Bierman, 2001; Honorton & Ferrari, Duggan (2016); Google Scholar, PubMed, Scopus da-
1989). And for the period 1987 to 2010, Storm et al. (2012) tabases by using the keywords ‘forced-choice AND ex-
noted that ES values in the forced-choice domain actually tra-sensory perception’;
increased significantly in their database of 72 studies. • Specialized scientific (peer-reviewed) journals, includ-
ing the Australian Journal of Parapsychology, Conscious-
General Aims of the Present Study ness, and Cognition, Explore: The Journal of Science and
Healing, Heliyon, Journal of Anomalous Experience and
As of 2022, ten years have passed since forced-choice
Cognition, Journal of Consciousness Studies, Journal of
studies were last meta-analysed (see Storm et al., 2012).
Parapsychology, Journal of Scientific Exploration, Journal
The general aim of the present study was to conduct a
of Personality and Social Psychology, Journal of the Soci-

520 JOURNAL OF SCIENTIFIC EXPLORATION • VOL. 37, NO 3 – FALL 2023 journalofscientificexploration.org

Lance Storm & Patrizio Tressoldi FORCED CHOICE ESP META ANALYSIS 1987-2022

ety for Psychical Research, and NeuroQuantology; (i.e., mean chance expectation; or MCE). When available,
• Proceedings of the Annual Convention of the Parapsy- we collected the corresponding standard normal deviate
chological Association and the Society for Psychical Re- z value and effect size ES (where ES = z/√n). These values
search. were double-checked, and in some cases (e.g., Hitchman,
Sherwood, & Roe, 2015), we found discrepancies
The meta-analysis by Storm et al. (2012) provided between published z scores and our calculations using
the original dataset of forced-choice publications dating the Vassarstats Exact Binomial calculator (http://www.
from January 1987 to December 2010 (see the Appendix in vassarstats.net/binomialX.html), which requires only hits,
Storm et al., 2012, p. 271). That dataset contains 91 stud- trial counts, and MCE.
ies, which were reported in 65 papers conducted by 96
investigators. Variables Considered

Inclusion Criteria Each study included in the database was classified

with the following variables: Authors, Year of Publication,
To be included in the present update, studies had to: Selected or Unselected Participants, Task Type (Telepathy,
Clairvoyance, or Precognition); Target Type (pictures,
1. Be published between January 2011 and December letters, symbols, objects, etc.); Sample Size; Trials; Hits;
2022; Hits Percentage; Number of Choices (k); Type of Feedback
2. Have adopted a forced-choice procedure for the ESP (none = 0; trial-by-trial = 1; end of trials = 2), Peer Reviewed
identification of targets (therefore excluding studies (‘Yes’ or ‘No’).
that expressly tested free-response and psychokine-
sis);1 Meta-Analysis Design
3. Have used human participants only (not animals); We applied a random-effects model by using the
4. Have been carried out with groups of participants and metafor package v. 3.8 (Viechtbauer, 2010), adopting
not single cases; the restricted maximum likelihood (REML) to estimate
5. Have incorporated randomization procedures for the heterogeneity variance (Langan et al., 2019) and the
selection of targets which could not be manipulated by Hartung method to control effect-size non-normality
the experimenter or participant; (Rubio-Aparicio et al., 2018) and corresponding confidence
6. Have been peer-reviewed;2 intervals estimation. We identified effect size outliers by
7. Have sufficient information (e.g., number of trials using SPSS Stem-and-Leaf and Box-and-Whiskers Plots.
and outcomes) for the authors to calculate the direct The whole database and the code used for all
hit-rates and apply appropriate statistical tests, and the statistical analyses are available for open access
calculate effect size (ES) as z/√n (where n = number on:(https://figshare.com/articles/dataset/Forced-
of trials). Studies with only reaction times were also Choice_1987_-_2022_meta-analysis/22574218/5),for
included separately as these did not have hits data but independent reproducibility, controls, and analyses.
did have t scores by which ES values could be calculated.
RESULTS
Procedure
After all inclusion and exclusion criteria had been met,
We present a PRISMA flowchart in Appendix A (Page 27 articles with hits data were included for analysis. There
et al., 2021). This figure details the counts of papers were 52 individual experiments/treatments/conditions
across four stages: identification, screening, eligibility, (‘studies’ as distinct from ‘articles’) that had sufficient data
and inclusion. For each study, we checked the following for the calculation of specific z scores and corresponding
factors: (a) the criteria adopted for selecting participants; effect sizes (ES). (For a list of these 52, see Appendix B.)
(b) authors’ names; (c) year of publication; (d) whether There were 17 articles with no hit data (these studies
participants were selected or unselected; (e) type of measured Reaction Time; ‘RT’). In these articles were 25
ESP task (telepathy, clairvoyance, or precognition); (f) individual experiments with t scores allowing for the
number of participants; (g) number of trials; (h) number of calculation of corresponding ESs. (For a list of these 25,
alternatives k per trial in the tasks; and (i) total number of see Appendix C.) The References section lists all articles
hits (we preferred the direct hits measure as it provides a meta-analyzed (marked by asterisks). The total number of
more “conservative” result—see Honorton, 1985, p. 54). articles is 43 (not to count Bem, 2011, twice), that included
With these data, we derived the proportion of hits and a total of 77 experiments/treatments/conditions (‘studies’)
compared these to the proportions expected by chance with sufficient information to calculate ESs.

journalofscientificexploration.org JOURNAL OF SCIENTIFIC EXPLORATION • VOL. 37, NO 3– FALL 2023 521

FORCED CHOICE ESP META ANALYSIS 1987-2022 Lance Storm & Patrizio Tressoldi

Two articles in our original database (Storm et al., 5.55, p = 1.43 × 10–8 (one-tailed). A single-sample t-test
2012) were expanded to include various conditions/ revealed that ES values significantly deviated from mean
treatments (Luke, Roe, & Davison, 2008; Pitman & Owen, chance expectation (MCE), where the test statistic is zero
2004); Thalbourne and Storm (2002-2005) remains in the (i.e., MCE = 0.00), t(37) = 2.94, p = .006 (two-tailed). Eleven
2012 meta-analysis even though it was later published as studies (29% of 38 studies) were independently significant
Thalbourne and Storm (2014) and is therefore not counted at α < .05 level.
in the update. We considered it necessary to assess homogeneity in
In the original database (Storm et al., 2012) were a different context. Higgins’ I 2 (Higgins, Thompson, Deeks,
two articles now excluded due to their datasets being & Altman, 2003) indicates the proportion of effect-size
incomplete (Vernon, Sandford, & Moyo, 2019; Zilberman, variance explained by heterogeneity across effect sizes.�
1995). A further two articles by Tressoldi et al. (2009, 2010) I 2 lies between 0% and 100%. A value of 0% indicates no
were removed because it was decided that they did not observed heterogeneity, and larger values show increasing
use typical behavioral forced-choice procedures (i.e..., they heterogeneity. We found significant heterogeneity in the
used sound targets). dataset, Q(df = 37) = 436.3, p < .001, and I 2 = 93.04% (very
Also, regarding the original database (Storm et al., high heterogeneity). It is important to regard heterogeneity
2012), six articles previously overlooked from the initial not as a measure of the quality of the studies, but as a
period were added in. We will correct the original statistics measure of between-studies differences. We point out
and re-do relevant analyses—these articles are: Batthyány, that experimental designs of forced-choice studies vary
Kranz, and Erber (2009); Ertel (2010); Hadlaczky (2005); to a far greater extent than, say, free-response (especially
Savva, Child, and Smith (2004); Savva, Roe, and Smith Ganzfeld designs) in terms of type of task, targets used,
(2005); Watt and Nagtegaal (2000). number of trials, and so on.
New Studies #2 (‘RT’ Studies). For a heterogeneous
Descriptive Statistics dataset of 25 studies, mean ES = 0.006 (SD = 0.020), and
mean t = 0.54 (SD = 1.21). The skew of the ES distribution
We compiled two databases: New Studies #1, the set
was normal. However, two outliers were identified:
of articles with hits data; and (2) New Studies #2, the set of
Rabeyon (2014), and Wittmann et al. (2021, expt. 2). (See
articles with no hits data (‘RT’ studies). In New Studies #1,
Appendix C: #13 & #23.) The homogeneous dataset of 23
there are a total of 162,989 trials and 71,678 hits. In New
studies has a mean t = 0.74 (SD = 1.01); mean ES = 0.01
Studies #2, there are a total of 207,019 trials (no hits).
(SD = 0.01). Ninety-five percent CIs are as follows: t scores,
H1: Z statistics and effect sizes (ES). It was hypothesized [0.31, 1.18]; ES values, [0.004, 0.02]. Note that neither
that the new databases would yield statistical evidence of of these 95% CIs includes values of MCE (i.e., zero). For
a communications anomaly known as ESP. comparative purposes, we calculated a Stouffer Z statistic
New Studies #1. For a heterogeneous dataset of 52 since z approximates t when samples have 30+ trials,
studies, mean ES = -0.004 (SD = 0.20), and mean z = -0.84 which is the case in this homogeneous dataset. For the
(SD = 8.03). These negative values are largely attributable 23 studies, Stouffer Z = 5.50, p = 1.90 × 10–8 (one-tailed).
A single-sample t-test revealed that ES values significantly
to extreme scoring in two studies: Escolà-Gascón (2022),
deviated from chance, t(22) = 3.34, p = .003 (two-tailed).
and Escolà-Gascón et al. (2022). It is noted that the
Four studies (17% of 23 studies) were independently
skew of the ES distribution was not normal. Outliers
significant at α < .05 level. Again, there was significant
were identified from SPSS Stem-and-Leaf and Box-and-
heterogeneity, Q(22) = 104.07; p < .001; I2 = 81.02%.
Whiskers Plots as significantly deviant (“extreme”) cases.
New Studies #1 and New Studies #2 (‘RT’ Studies)
Seven of nine extremely low-scoring values were found
combined. The two databases (New Studies #1 & New
in the two Escolà-Gascón studies just mentioned (see
Studies #2), totaling 61 studies, were not significantly
Appendix B: #45, #47, #48, #49, #50, #51, & #52), and a
different on ES values, t(50.43) = 1.28, p = .207 (two-tailed).
further seven outliers removed (see Appendix B: #9, #12,
The skew was normal, but two outliers were removed: Luke
#13, #14, #16, #24, & #32). Normality (homogeneity) was
et al. (2012), and Sheldrake (2015). (See Appendix B: #10, &
achieved after the removal of these 14 studies. A dataset
#33). The homogeneous dataset of 59 studies has a mean ES
of 38 studies is now homogeneous with mean z = 0.90 (SD
= 0.016 (SD = 0.03), and mean z = 0.84 (SD = 1.47). Stouffer
= 1.74); mean ES = 0.02 (SD = 0.04). Ninety-five percent
Z = 6.42, p = 6.81 × 10–11 (one-tailed). Table 1 lists statistics
confidence intervals (CIs) are as follows: z scores, [0.33,
for the combined database of new studies only (N = 59), as
1.47]; ES values, [0.006, 0.03]. Note that neither of these
well as subgroups of experimental conditions after outliers
95% CIs includes values of MCE (i.e., zero). Stouffer Z =
and other data exclusions. Subsequent analyses in this

522 JOURNAL OF SCIENTIFIC EXPLORATION • VOL. 37, NO 3 – FALL 2023 journalofscientificexploration.org

Lance Storm & Patrizio Tressoldi FORCED CHOICE ESP META ANALYSIS 1987-2022

Table 1. Combined Database of New Studies (2011 to 2022): Effect Sizes, and 95% Confidence Intervals, p-value and I2

Datasets and Subsets n ES (SD) 95% CI p I2

Combined New Studies (N) 59 0.02 (0.03) 0.01, 0.02 3.5 × 10-6 92.6
Telepathy 4 0.025 (0.03) -0.03, 0.08 2.3 × 10 -1
97.5
Clairvoyance 14 0.007 (0.04) -0.01, 0.03 4.7 × 10-1 88.2
Precognition 41 0.017 (0.03) 0.01, 0.03 2.3 × 10 -6
90.9
Selected participants 9 0.02 (0.04) -0.01, 0.05 2.0 × 10-1 83.3
Unselected participants 50 0.02 (0.03) 0.01, 0.02 4.5 × 10 -6
92.3
With feedback 47 0.02 (0.03) 0.01, 0.03 1.1 × 10-8 91.1
No feedback 12 0.002 (0.04) -0.02, 0.03 8.6 × 10-1 87.2

paper are conducted on this larger database of 59 studies. 11 mutually exclusive experimenter groups with at least
Telepathy seems an outstanding performer as a modality, two studies in each. Some of these groupings were used in
but with only six studies in the subset, it is difficult to Storm et al. (2012): ‘Argibay’, ‘Bem’, ‘Bierman’, ‘Luke’, ‘Roe’,
gauge its pertinence (see H2 below). ‘Schlitz’, ‘Schönwetter’, ‘Simmonds-Moore’, ‘Sheldrake’,
H2: Effect size differences for telepathy, clairvoyance, ‘Storm’, and ‘Watt’. Table 2 lists the mean ES values for
and precognition. It was hypothesized that the mean ES each group. The same combined dataset was used as
values for telepathy, clairvoyance, and precognition are was tested in H2 & H3 (for the full-database tests, see
different. Table 1 lists the ES values for the three modalities. Post Hoc Analyses). Fourteen single studies (24%) did not
Four studies (7% of 59 studies) tested telepathy; 14 qualify, so these were excluded, but it would be misleading
studies (24%) tested clairvoyance; 41 studies (69%) tested to categorize these as ‘Other’ since the variability of
precognition. Telepathy produced the strongest effect. laboratory/author is too great. Mean ES values varied from
A Univariate ANOVA test was conducted, entering the -0.021 (‘Schönwetter’) to 0.049 (‘Luke’). Using a Kruskal-
variable Psi Modality, as well as the variable Target Type Wallis ANOVA, a marginally significant difference was found
(see H3 below). There was a just significant ES difference between experimenter groups, χ2(10) = 18.13, p = .053 (two-
between the three modalities, F(2, 51) = 3.16, p =.051 (two- sided). However, no two groups were significantly extreme
tailed). The greatest difference was between telepathy and
clairvoyance (0.018), but a Tukey’s HSD test showed no Table 2. Experimenter/Laboratories: Effect Sizes, SD, and
significant difference. 95% Confidence Intervals
H3: Target types. Target types may affect participants’
performances. As was done by Storm et al. (2012), data Grp. Expt./Lab. n ES (SD) 95% CI
was divided into five types of targets: (1) Pictures/
drawings/faces, (2) Symbols/fractals/I Ching hexagrams, #1 ‘Bem’ 8 0.040 (0.02) 0.02, 0.06
(3) Numbers, (4) Letters/words/messages, and (5) Objects #2 ‘Schlitz’ 3 0.004 (0.01) -0.02, 0.02
(i.e., targets that occupy 3-D physical space). The new
#6 ‘Argibay’ 4 0.042 (0.04) -0.02, 0.10
set of telepathy studies used only ‘Pictures’; clairvoyance
studies used all types except numbers; and precognition #9 ‘Simmonds-Moore’ 2 -0.018 (0.02) -0.17, 0.14
studies did not use numbers or objects. The same ANOVA
#10 ‘Schönwetter’ 2 -0.021 (0.03) -0.33, 0.29
from above (see H2) showed no significant ES difference
between target types, F(3, 51) = 0.61, p =.612 (two-tailed). #12 ‘Bierman’ 5 0.018 (0.02) -0.005, 0.04
Nevertheless, the preferred targets were ‘Symbols’ and #14 ‘Luke’ 3 0.049 (0.03) -0.04, 0.14
‘Letters’ with ES about equal (ES = 0.04).
#16 ‘Roe’ 10 0.014 (0.03) -0.004, 0.03
H4. ES differences between experimenters/
laboratories. In order to ascertain whether our database #17 ‘Sheldrake’ 3 0.040 (0.02) -.0006, 0.08
was the result of extremely positive ES values for a limited #19 ‘Storm’ 2 0.010 (0.02) -0.20, 0.22
pool of laboratories/experimenters, we conducted a
Kruskal-Wallis ANOVA on the pooled data after dividing #21 ‘Watt’ 2 0.007 (0.02) -0.14, 0.16
them into laboratory/experimenter groups. We formed n = number of studies

journalofscientificexploration.org JOURNAL OF SCIENTIFIC EXPLORATION • VOL. 37, NO 3– FALL 2023 523

FORCED CHOICE ESP META ANALYSIS 1987-2022 Lance Storm & Patrizio Tressoldi

in scoring to reach significance. The effects cannot be said Once again, 19 outliers were removed to render the
to be due to a few outstanding investigators. database homogeneous (N = 83).3 The database has mean
H5. The advantage in using a high k-choices design. We z score = 0.68 (SD = 1.70); 95%CI [0.31, 1.05]; ES = 0.02 (SD
proposed that the number of choices k per trial is positively = 0.04); 95%CI [0.01, 0.02]; Stouffer Z = 6.18 (p = 3.21 × 10-
related to z (or t). In the combined dataset (N = 59), there are 10
). Note that these 95% CIs do not include values of MCE
only three values for k choices: k = 2 (n = 47; mean z = 0.95); (i.e., zero). A single-sample t-test revealed that ES values
k = 4 (n = 9; mean z = 0.23); and k = 5 (n = 3; mean z = 0.87). significantly deviated from chance, t(82) = 4.26, p < .001
Visual inspection shows no clear trend. A Pearson’s r test (two-tailed). Of 83 studies, 16 (19%) are independently
on these values (grouped) was not significant, r(1) = -0.29, significant (α = .05).
p = .409 (one-tailed). Correlating z scores for individual The revised original database (1987 to 2010; N = 83)
studies with their k values, was also not significant, r(57) and the database of new studies (2011 to 2022; N = 59)
= -0.13, p = .158 (one-tailed). Given these analyses, z (or t) were not significantly different on ES values, t(140) = 0.08,
scores tend not to increase with k. p = .940 (two-tailed). The two databases were combined.
H6. Change in ES values over the period of analysis One outlier was removed.3 Table 3 lists statistics for the
(2011–2022). We propose that ES values have increased combined homogeneous database of studies from 1987
over this period based on a similar finding by Storm et al. to 2022 (N = 141), as well as subgroups of experimental
(2012). However, we find the correlation between year of conditions. Statistics not given in Table 3, include mean z
study and ES is negative and significant for the combined score = 0.72 (SD = 1.58); 95%CI [0.45, 1.00]; Note that most
database (N = 59), rs(57) = -0.34, p = .004 (one-tailed). Our of the 95% CIs do not include values of MCE (i.e., zero).
hypothesis failed as this statistic indicates a decline in ES Stouffer Z = 8.52 (p < 10-16). A single-sample t-test revealed
values over the 12-year period. that ES values deviated significantly from chance, t(140)
H7. The original database (1987 to 2010), and the = 5.78, p < .001 (two-tailed). Of 141 studies, 29 (21%) are
new database (2011 to 2022), are not different and can independently significant (α = .05).
be combined to form a larger dataset. The old database
in Storm et al. (2012) identified 19 outlier studies. The Post Hoc Analyses
database was reduced from 91 studies to 72 studies. Storm We re-assessed hypotheses H2 (modality) and H3
et al. reported the following: mean z score = 0.57 (SD = (target type), this time applying our tests to the largest
1.58); ES = 0.01 (SD = 0.03); Stouffer Z = 4.86, p = 5.90 × 10-7 database of forced-choice studies assembled to date (N
(p. 253). As explained above, we deleted some old studies = 141). As telepathy was not represented in two types of
that did not fit our criteria, but found new ones, warranting targets (Symbols and Numbers), and clairvoyance was not
a re-assessment of the forced-choice database of studies represented in one type of target (Numbers), relevant data
dating 1987 to 2010. were removed for this analysis only, reducing the database
There were 68 articles for this period, with 102 to N = 103. Again, we conducted a Univariate ANOVA. The
experiments/treatments/conditions (i.e., studies) with following results were obtained:
associated ES values. The heterogeneous database has a
mean z score = 1.34 (SD = 3.33); 95%CI [0.69, 1.99]; ES = (i) Psi modality, F(2, 94) = 0.30, p = .739 (two-tailed)
0.04 (SD = 0.09); 95%CI [0.03, 0.06].
Table 3. Combined Database of Studies (1987 to 2022): Effect Sizes, and 95% Confidence Intervals, p-value and I2

Datasets and Subsets n ES (SD) 95% CI p I2

Combined Studies (N) 141 0.02 (0.03) 0.009, 0.02 5.9 × 10-9 97.1
Telepathy 12 0.03 (0.04) 0.004, 0.05 2.0 × 10-1 95.6
Clairvoyance 58 0.01 (0.03) 0.001, 0.02 1.0 × 10 -1
96.4
Precognition 71 0.02 (0.03) 0.01, 0.02 1.1 × 10-6 96.7
Selected participants 15 0.03 (0.04) 0.005, 0.05 2.0 × 10 -2
91.3
Unselected participants 126 0.01 (0.03) 0.007, 0.02 2.2 × 10-7 97.0
With feedback 88 0.02 (0.03) 0.01, 0.02 1.0 × 10 -9
97.3
No feedback 53 0.01 (0.03) -0.001, 0.02 1.8 × 10-2 94.4

524 JOURNAL OF SCIENTIFIC EXPLORATION • VOL. 37, NO 3 – FALL 2023 journalofscientificexploration.org

Lance Storm & Patrizio Tressoldi FORCED CHOICE ESP META ANALYSIS 1987-2022

(ii) Target type, F(2, 94) = 3.79, p = .026 (two-tailed) to correlate with higher k values.
For the final database (N = 141), Table 3 shows
However, there were no significant differences differences between (i) the three modalities; (ii) selected
between any two specific target types. There was a and unselected participants; and (iii) feedback/no-feedback
significant modality/target-type interaction effect, F(4, conditions. A Univariate ANOVA was conducted to test
94) = 3.18, p = .017 (two-tailed). Figure 2 illustrates the these differences. There were significant ES differences
interactions between modality and target type. Letters between participant type and feedback condition:
were the most successful of the three target types, but
only for telepathy. (i) Psi modality, F(2, 131) = 0.18, p = .834 (two-tailed)
Experimenter/laboratory differences (H4) were (ii) Selected vs. unselected participants, F(1, 131) = 4.39, p
assessed separately since so much data was lost in this = .038 (two-tailed)
analysis due to the large number of single studies; 41
(29%) were excluded for this analysis. Also, we now (iii) Feedback vs. no-feedback condition, F(1, 131) = 5.43, p
had 19 mutually exclusive experimenter groups as a = .021 (two-tailed)
number of them from the old period (1987-2010) did not
conduct studies in the new period (2011 to 2022). All 19 Selected participants (mean ES = .03) were superior
groups are: ‘Argibay’, ‘Bem’, ‘Bierman’, ‘Dalkvist’, ‘Don’, in ESP performance compared to unselected participants
‘Ertel’, ‘Haraldsson’, ‘Luke’, ‘Palmer’, ‘Rao’, ‘Roe’, ‘Schlitz’, (mean ES = .01). Studies that gave feedback to participants
‘Schönwetter’, ‘Simmonds-Moore’, ‘Sheldrake’, ‘Storm’, (mean ES = .02) produced superior ESP performances than
‘Vaughan’, ‘Watt’, and ‘Wiseman’. There was a significant studies that did not give feedback (mean ES = .01). There
difference between groups of experimenters, χ2(18) = were no significant interaction effects.
34.58, p = .011 (two-sided). However, when mean ES values These findings prompted a comparison of selected
by group were checked against each other, there were no participants who received feedback with selected
significant differences. participants who did not receive feedback. The former
Finally, H5 concerning the relationship between z were very few in number (n = 5) compared to the latter (n
scores and k-choices was re-tested on the large database = 136), but we can assume equal variance (p = .205), and
(N = 141). Previously, Storm et al. (2012) reported a the difference was significant, t(139) = 1.87, p = .032 (one-
significant trend (r = 0.79). On this occasion, the effect was tailed). ‘Selected with feedback’ (mean ES = 0.04; SD =
moderate in strength but not significant, r(5) = 0.48, p = 0.04) outperformed ‘selected with no feedback’ (mean ES
.139 (one-tailed). Correlating z scores for individual studies = 0.02; SD = 0.03).
with their k values produced a significant outcome, r(139) Decline vs. incline effect. In this research field, some
= 0.20, p = .008 (one-tailed). The strongest z scores tended authors suggest episodic (within study) declines in effects

Figure 2. The target-type advantage: ESP differences on psi modality are not the same across levels of target type.

journalofscientificexploration.org JOURNAL OF SCIENTIFIC EXPLORATION • VOL. 37, NO 3– FALL 2023 525

FORCED CHOICE ESP META ANALYSIS 1987-2022 Lance Storm & Patrizio Tressoldi

Figure 3. Scatter-plot of effect sizes (ES) for forced-choice studies showing a slight incline over a 36-year period.

are inevitable and appear as slow but constant decreases experiments obtaining a z or t value equal or higher than
in the strength of effects due to repetition of similar tasks 1.65, corresponding to a one-tailed p-value of .05. For
(Bierman, 2001, Kennedy, 2003). However, as Storm the heterogeneous database (i.e., before we removed
(in press) has pointed out, there is little evidence for outliers), we counted 52 (29%) out of 179 effect sizes that
chronological (between-study) declines. We tested the are independently significant. As reported above, even 29
assumption of a chronological decline in two ways, where (21%) of 141 studies in the homogeneous database is a small
the hypothesis is that effect size (ES) covaries with year fraction of the total. This result supports the hypothesis
of publication and with a non-parametrical correlation that our database is not likely to have been contaminated
between year of study and ES. The meta-regression by publication bias, as we should expect a much larger
coefficient is zero, with a p-value = .09. The Spearman’s percentage of successful (significant) studies as a result of
rho correlation was positive and approached significance, authors withholding (not publishing) unsuccessful studies.
r(139) = 0.11, p = .098 (one-tailed), suggesting an incline. Participant comparisons across experimental designs.
The linear trend line formula is ES = [0.0004 × YEAR] - 0.79; We compared selected and unselected participants across
linear R2 = .011 (see Figure 3). Our hypothesis (H6) of an a range of experimental designs. Table 3 and our Post
incline was partially supported. Hoc tests show that selected participants outperform
Publication bias. As a means of avoiding publication unselected participants. Figures 4 and 5 show comparisons
bias, it has been the policy of the Journal of Parapsychology with forced-choice and various other meta-analyses—
since 1975 to publish all papers passing peer review namely, free-response remote viewing (Tressoldi &
whether the reported results are significant or not. Other Katz, 2023), free-response in a Ganzfeld environment
parapsychology journals have adopted the same policy. (Tressoldi & Storm, 2023), and presentiment design
Grimes, Bauch, and Ioannidis (2017) note the “top-tier (Duggan & Tressoldi, 2018). Figure 4 compares unselected
journals possess a limited number of publication slots and participants; Figure 5 compares selected participants.
are thus overwhelmingly weighted towards publishing Comparing Figure 4 with Figure 5, we see marked selection
only novel or significant results” (p. 2). Parapsychological differences in other designs.
journals are free of that pressure due to the limited number In reference to the two figures, we note that selected
of researchers in the field. We note that the majority of participants in ‘Forced-Choice’ are nearing unselected
studies in this meta-analysis (91%) were either published in participants’ in ‘Free-Response Ganzfeld’ in terms of mean
journals specializing in parapsychology or journals known ES. However, forced-choice continues to deliver smaller
to be favorable to parapsychological research. Clearly, the effects compared to the other three designs.
other 9% of journals expressed no bias at that time.
One empirical method to test if authors disseminated DISCUSSION
only experiments with positive statistical results is to The above two-stage forced-choice meta-analysis on
count how many of them reached the statistical threshold (a) two newly-formed databases, 2011 to 2022, and (b)
of p ≤ .05. In our database, we counted the number of

526 JOURNAL OF SCIENTIFIC EXPLORATION • VOL. 37, NO 3 – FALL 2023 journalofscientificexploration.org

Lance Storm & Patrizio Tressoldi FORCED CHOICE ESP META ANALYSIS 1987-2022

Figure 4. Unselected participants: Effect size with corresponding 95%CI, observed in the meta-analyses related to forced-
choice, free-response remote viewing, free-response in a Ganzfeld environment, and presentiment designs.

the enlarged complete database 1987 to 2022 (N = 141), was more successful than New Studies #2 (‘RT’ Studies),
indicate that the forced-choice domain generally produces with 29% of studies independently significant in the
significant psi effects above MCE. Up until the present former compared to 17% in the latter. Generally, however,
study, we had considered forced-choice effects to be very our results are very similar to those observed by Honorton
weak, and our position has not changed. The new findings and Ferrari (1989), Steinkamp et al. (1998), and Storm et al.
are nevertheless encouraging, even across modalities, (2012). The following is a break-down of our findings across
target types, and participant types (i.e., selected vs. a series of hypotheses.
unselected). We formed two databases—a dataset of New We proposed six hypotheses and tested H2 to H6 using
Studies #1 (studies reporting hit rates), and a dataset of the combined dataset (H1 was tested separately for both
New Studies #2 (‘RT’ Studies; i.e., measures not recording smaller datasets #1 and #2). Interestingly, Storm et al.
hits)—and although mean ES values were different (0.02 (2012) reported that effects by modality were “very weak
vs. 0.01, respectively), the difference was not statistically for precognition, clairvoyance, and even telepathy, which
significant, so we combined the two. Also, New Studies #1 was the strongest effect of the three” (p. 259)—this finding

Figure 5. Selected participants: Effect size with corresponding 95%CI, observed in the meta-analyses related to forced-
choice, free-response in a Ganzfeld environment, and free-response remote viewing designs (no data for presentiment).

journalofscientificexploration.org JOURNAL OF SCIENTIFIC EXPLORATION • VOL. 37, NO 3– FALL 2023 527

FORCED CHOICE ESP META ANALYSIS 1987-2022 Lance Storm & Patrizio Tressoldi

was repeated with telepathy still the strongest effect, words, or messages (see Figure 2).
though not significantly (H2). Second, testing selected participants, compared to
We found target type made no difference to effects in unselected participants, showed a significant mean ES
our test on the new studies (N = 59), but the test on the difference in ESP performance. This finding shores up the
large database (N = 141) was significant (H3). However, general understanding that some gifted participants tend
there were no apparent differences between specific target to obtain high scores in ESP tasks in forced-choice designs
types in the post hoc analysis. As we had already pointed (for examples, see Honorton, 1987; Kanthamani & Kelly,
out (see Storm et al., 2012, p. 260), possible reasons for 1974; Steinkamp, 2005).
these failures are to do mainly with participants’ attitudes Third, in testing studies with a feedback condition, we
and reactions to targets—specifically, targets may be found that giving participants feedback (either after trials
uninteresting and/or meaningless (i.e., Zener cards, or after runs) also gives an advantage. Given this finding,
numbers, letters, etc.) whereas experimenters should aim and the previous one concerning selection, it followed
to offer emotionally stimulating and/or meaningful targets that we should test the ‘selected with feedback’ condition
(i.e., divinatory readings, real pictures, video clips, etc.). against the ‘selected with no feedback’ condition. We
Storm et al. also argued that cognitive noise may be higher found a significant difference between the two, with
in forced-choice studies compared to Ganzfeld and other selected participants who received feedback producing a
designs where relaxation is offered. It is also thought that stronger mean ES than selected participants who did not
ecologically valid tasks are preferred by participants, such receive feedback.
as e-mail or phone call predictions, and these are always Fourth, we found no evidence of publication bias;
telepathy tests (see Sheldrake et al., 2015). Preferred otherwise, we would expect a much higher rate of
targets included various symbols (such as fractals) and text independently significant studies than a mere 21% (29 out
(i.e., words, letters, and messages). of 141 studies).
Following Storm et al. (2012), we divided our database Our fifth and final point is that the forced-choice
of new studies into mutually exclusive investigator/ domain, with ES = 0.02, has confirmed previously reported
experimenter groups. Again, we found groups tended not low ES values (see Honorton & Ferrari, 1989; Steinkamp
to outperform each other (H4). No significant differences et al., 1998; Storm et al., 2012). The effect is enduring and
were found in the larger database. consistent across time and other variables, albeit weak
Timm (2000) claimed that number of choices per trial (which is to say ‘small’), and it might be argued that there
(k) may be positively related to z score. However, when we is room for improvement. We note that the effect is weak
tested the new set of studies, no proposed advantage based because it is estimated by considering the number of trials
on k was found (see H5). A re-test on the larger database and not the number of participants, and number of trials in
was not significant when z scores were grouped by k to forced-choice protocols are much larger than those used in
form mean scores for each k-group, but correlating z scores typical free-response protocols. For all studies from 1987 to
for individual studies with k also produced a significant 2022 (N = 179) before outliers were removed, we estimate
effect. The strongest z scores tended to correlate with the typical participant performs 14 trials on average,
higher k values. whereas most participants in Ganzfeld experiments
In testing H6, we found a significant time-dependent seldom do more than one trial each. Nevertheless, the
decline in ES values in the new studies (N = 59), which covers number of forced-choice studies that are independently
a 12-year period, but for the longer period of analysis (1987 significant (21%) is better than that for the free-response
to 2022), there was a marginally significant incline (see domain, such as non-ganzfeld noise-reduction and
Figure 3). We noted earlier that the correlation between standard free-response (19% and 15%, respectively; see
year of study and ES indicated an incline, “meaning that Storm & Tressoldi, 2020, pp. 205-206), and not so far from
ES values increased over the 24-year period” (Storm et Ganzfeld (26%). Also, for experimenters to get some kind
al., 2012, p. 257). That effect was significant, and after 36 of additional advantage in forced-choice studies, they
years, there is now good evidence the incline has been are advised to test selected participants and certainly to
maintained. offer feedback. They may then find that their participants
Further tests on the largest ever database of forced- (selected with feedback; mean ES = .04) perform possibly
choice studies produced some additional findings that even better on average than participants in Bem-type
warrant mentioning: precognition studies (mean ES = .03)—see Cardeña (2018,
First, we did not find a performance difference between p. 667).
modalities, but an interaction effect showed that telepathy Given these findings, skeptics, critics, and even psi
was a ‘show-case’ modality, provided targets were letters, advocates may need to reconsider their current positions

528 JOURNAL OF SCIENTIFIC EXPLORATION • VOL. 37, NO 3 – FALL 2023 journalofscientificexploration.org

 Lance Storm & Patrizio Tressoldi FORCED CHOICE ESP META ANALYSIS 1987-2022

on the efficacy of forced-choice designs insofar as their future events [version 2; referees: 2 approved].
views may be negative. Yet, we too, suggested that the F1000Rsearch, https://doi.org/10.12688/f1000re-
forced-choice “weak effect might prove difficult to deploy search.7177.2
efficaciously” (Storm et al., 2012, p. 261). We have seen that Bierman, D. J. (2001). On the nature of anomalous phe-
there are steps that can be taken to improve that outlook. nomena: Another reality between the world of sub-
jective consciousness and the objective world of
ENDNOTES physics? In P. van Loocke (Ed.), The physical nature of
consciousness (pp. 269–292). Benjamins. https://doi.
1.
For a review of the meta-analyses of these and other
org/10.1075/aicr.29.12bie
experimental domains, see Cardeña (2018).
* Bierman, D. J. (2011). Anomalous switching of the bi-sta-
2.
Although Delorme et al. (2016) was initially peer-re-
ble percept of a Necker Cube: A preliminary study.
viewed, it was later retracted by the hosting journal and
Journal of Scientific Exploration, 25(4), 721–733.
is, therefore, officially unpublished. Hence, we excluded
* Bierman, D., & Bijl, A. (2014). Anomalous ‘retrocausal’
it from this meta-analysis on the grounds that it must
effects on performance in a Go/NoGo Task. Journal of
again go through peer review prior to publication.
Scientific Exploration, 28(3), 437-452.
3.
Higgins’ I2 = 100% × (Q – df)/Q, where Q is Cochran’s he-
* Billows, H., & Storm, L. (2015). Believe it or not: A con-
terogeneity statistic, and df is degrees of freedom. He-
firmatory study on predictors of paranormal belief,
terogeneity benchmark values for I2 are 25% (low), 50%
(moderate), and 75% (high). For details about Cochran’s and a psi test. Australian Journal of Parapsychology, 15,
Q statistic, see Lipsey and Wilson (2001). 7-35.
* Boer, De R., & Bierman, D. (2006). The roots of paranor-
REFERENCES mal belief: divergent associations or real paranormal
experiences? Proceedings of the 49th Parapsychologi-
(References marked with an asterisk indicate studies cal Association Annual Convention, 283–298.
included in the meta-analysis.) Cardeña, E. (2018). The experimental evidence for para-
psychological phenomena: A review. American Psy-
Appelbaum, M., Cooper, H., Kline, R. B., Mayo-Wilson, chologist, 73(5), 663–677. https://doi.org/10.1037/
E., Nezu, A. M., & Rao, S. M. (2018). Journal article amp0000236
reporting standards for quantitative research in psy- Curtis, J. T. & Wilson, J. P. (1997). Sensation Seeking and
chology: The APA Publications and Communications ESP test performance: a preliminary investigation,
Board task force report. American Psychologist, 73(1), Journal of the Society of Psychical Research, 62, 1–21.
3–25. https://doi.org/10.1037/amp0000191 * Dalkvist, J. (2013). Performance in group telepathy ex-
* Arora, S., Schmidt, M., Boylan, J., & Pantazatos, S. P. periments as a function of target picture characteris-
(2022). Attempt to replicate Bem’s precognitive tics. Journal of Parapsychology, 77(1), 79–106.
avoidance task and detect relationships with trait Delorme, A., Pierce, A., Michel, L., & Radin, D. (2016, May
anxiety. Journal of Consciousness Studies, 29(5-6), 17). Prediction of mortality based on facial charac-
8–20. doi.org/10.53765/20512201.29.5.008 teristics. Frontiers in Human Neuroscience, 10. doi.
Batthyány, A., Kranza, G. S., & Erber, A. (2009). Moderat- org/10.3389/fnhum.2016.00173
ing factors in precognitive habituation: The roles of Duggan, M., & Tressoldi, P. (2018). Predictive physiolog-
situational vigilance, emotional reactivity, and affect ical anticipatory activity preceding seemingly un-
regulation. Journal of Society for Psychical Research, 73, predictable stimuli: An update of Mossbridge et al.’s
65–82. meta-analysis. F1000Research, 7, 407. https://doi.
* Bem, D. J. (2011). Feeling the future: Experimental evi- org/10.12688/f1000research.14330.2
dence for anomalous retroactive influences on cog- Ertel, S. (2010). Psi in a skeptic’s lab: A successful replica-
nition and affect. Journal of Personality and Social Psy- tion of Ertel’s Ball Selection Test. Journal of Scientific
chology, 100, 407–425. doi:10.1037/a0021524 Exploration, 24(4), 581–598.
Bem, D. J., & Honorton, C. (1994). Does psi exist? Repli- * Ertel, S. (2013). Psi effect or sensory leakage: Scrutiniz-
cable evidence for an anomalous process of informa- ing the Ball Selection Test. Journal of Scientific Explo-
tion transfer. Psychological Bulletin, 115, 4–18. https:// ration, 27(3), 387–391.
doi.org/10.1037/0033-2909.115.1.4 * Escolà-Gascón, Á. (2022). Forced-choice experiment on
Bem, D., Tressoldi, P., Rabeyron, T., & Duggan, M. (2016). anomalous information reception and correlations
Feeling the future: A meta-analysis of 90 experi- with states of consciousness using the Multivariable
ments on the anomalous anticipation of random Multiaxial Suggestibility Inventory-2 (MMSI-2). Ex-

journalofscientificexploration.org JOURNAL OF SCIENTIFIC EXPLORATION • VOL. 37, NO 3– FALL 2023 529

FORCED CHOICE ESP META ANALYSIS 1987-2022 Lance Storm & Patrizio Tressoldi

plore: The Journal of Science and Healing, 18, 170–178. ments, 1935-1987. Journal of Parapsychology, 53, 281–
https://doi.org/10.1016/j.explore.2020.11.009 308.
* Escolà-Gascón, Á., Wright, A. C., & Houran, J. (2022). * Houran, J., & Lange, R. (2012). I Ching outcomes from
‘Feeling’ or ‘sensing’ the future? Testing for anom- experimental manipulations of transliminality para-
alous cognitions in clinical versus healthy popula- normal belief. Australian Journal of Parapsychology, 12,
tions. Heliyon, e11303. https://doi.org/10.1016/j.heli- 39–58.
yon.2022.e11303 * Houran, J., & Lange, R. (2014). Applying the theory of
Grimes, D. R., Bauch, C. T., & Ioannidis, J. P. A. (2017). reasoned action to a computerized test of intuition,
Modelling science trustworthiness under publish or part II: Decision-making in a hidden test of psi. Journal
perish pressure. Royal Society Open Science, 5, 171511. of the Society for Psychical Research, 77(4), 236–251.
http://dx.doi.org/10.1098/rsos.171511 Irwin, H. J., & Watt, C. A. (2007). An introduction to para-
Hadlaczky, G. (2005). Precognitive habituation. Ac- psychology (5th ed.). McFarland.
cessed on http://www.diva-portal.org/smash/get/ * Jolij, J., & Bierman, D. (2019). Two attempted retro-prim-
diva2:189200/FULLTEXT01.pdf ing replications show theory-relevant anomalous
Higgins, J. P. T., Thompson, S. G., Deeks, J. J., & Altman, connectivity. Journal of Scientific Exploration, 33(1),
D. G. (2003). Measuring inconsistency in meta-analy- 43–60. https://doi.org/10.31275/2019.1262
ses. British Medical Journal, 327, 557–560. https://doi. Kanthamani, H., & Kelly, E. F. (1974). Awareness of suc-
org/10.1136/bmj.327.7414.557 cess in an exceptional subject. Journal of Parapsychol-
* Hitchman, G. A. M., Roe, C. A., & Sherwood, S. J. (2012). ogy, 38(4), 355–382.
A re-examination of non-intentional precognition * Kekecs, Z., Palfi, B., Szaszi, B., Szecsi, P., Zrubka, M.,
with openness to experience, creativity, psi beliefs Kovacs, M., … Aczel, B. (2019, August 1). Raising the
and luck beliefs as predictors of success. Journal of value of research studies in psychological science
Parapsychology, 76(1), 109–145. by increasing the credibility of research reports:
* Hitchman, G. A. M., Roe, C. A., & Sherwood, S. J. (2015). The transparent psi project. PsyArxiv https://doi.
The relationship between lability and performance at org/10.31234/osf.io/uwk7y For the final published
intentional and nonintentional versions of an implicit versions see also https://royalsocietypublishing.org/
PMIR-type psi task. Journal of Parapsychology, 79(1), doi/pdf/10.1098/rsos.191375
65–86. Langan, D., Higgins, J. P. T., Jackson, D., Bowden, J., Veron-
* Hitchman, G. A. M., Sherwood, S. J. & Roe, C. A. (2015). iki, A. A., Kontopantelis, E., … Simmonds, M. (2019).
The relationship between latent inhibition and per- A comparison of heterogeneity variance estimators
formance at a non-intentional precognition task. in simulated random‐effects meta‐analyses. Re-
Explore: The Journal of Science and Healing, 11(2), 118– search Synthesis Methods, 10(1), 83–98. https://doi.
126. DOI: 10.1016/j.explore.2014.12.004 org/10.1002/jrsm.1316
* Hitchman, G. A. M., Pfeuffer, C. U., Roe, C. A., & Sher- * Lange, R. & Houran, J. (2013). Towards a replicable for-
wood, S. J. (2016). The effects of experimenter-par- mula for significant I Ching outcomes. Australian Jour-
ticipant interaction qualities in a goal-oriented nal of Parapsychology,13, 9–25.
nonintentional precognition task. Journal of Parapsy- Lipsey, M. W., & Wilson, D. (2001). Practical meta-analysis.
chology, 80(1), 45–69. Sage.
Honorton, C. (1985). Meta-analysis of psi ganzfeld re- * Luke, D., & Morin, S. (2014). Exploration of the validity
search: A response to Hyman. Journal of Parapsychol- and utility of a reward contingency in a non-inten-
ogy, 49, 51–91. tional forced-choice precognition task. Journal of the
Honorton, C. (1987). Precognition and real-time ESP per- Society for Psychical Research, 78(917), 207–218.
formance in a computer task with an exceptional Luke, D. P., Delanoy, D., & Sherwood, S. J. (2008). Psi may
subject. Journal of Parapsychology, 51(4), 291–320. look like luck: Perceived luckiness and beliefs about
Honorton, C., Berger, R. E., Varvoglis, M. P., Quant, M., luck in relation to precognition. Journal of Society for
Derr, P., Schechter, E. I., & Ferrari, D. C. (1990). Psi Psychical Research, 72(4),193–207.
communication in the Ganzfeld: Experiments with an * Luke, D., Zychowicz, K., Richterova, O., Tjurina, I., &
automated testing system and a comparison with a Polonnikova, J. (2012). A sideways look at the neuro-
meta-analysis of earlier studies. Journal of Parapsy- biology of psi: Precognition and circadian rhythms.
chology, 54, 99–139. NeuroQuantology, 10(3), 580–590. https://doi.
Honorton, C., & Ferrari, D. C. (1989). “Future telling”: A org/10.14704/nq.2012.10.3.614
meta-analysis of forced-choice precognition experi- * Luke, D., & Zychowicz, K. (2014). Comparison of out-

530 JOURNAL OF SCIENTIFIC EXPLORATION • VOL. 37, NO 3 – FALL 2023 journalofscientificexploration.org

Lance Storm & Patrizio Tressoldi FORCED CHOICE ESP META ANALYSIS 1987-2022

comes with nonintentional and intentional precogni- tive habituation effect: An adaptation using spider
tion tasks. Journal of Parapsychology, 78(2), 223–234. stimuli. Paper presented at the 47th Annual Convention
* Muhmenthaler, M. C., Dubravac, M., & Meier, B. (2022). of the Parapsychological Association, Vienna, Austria.
The future failed: No evidence for precognition in * Savva, L., & French, C. C. (2002). Is there time-reversed
a large scale replication attempt of Bem (2011). interference in Stroop- based tasks. The Parapsy-
Psychology of Consciousness: Theory, Research, and chological Association 45th Annual Convention, Pro-
Practice. Advance online publication. https://doi. ceedings of the Presented Papers, 194-205.
org/10.1037/cns0000342 Savva, L., Roe, C., & Smith, M. D. (2005). Further testing
Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., of the precognitive habituation effect using spider
Hoffmann, T. C., Mulrow, C. D., … Moher, D. (2021). stimuli. Proceedings of Presented Papers: The Para-
The PRISMA 2020 statement: an updated guide- psychological Association 48th Annual Convention (pp.
line for reporting systematic reviews. BMJ, 372, n71. 163–170), Parapsychological Association.
https://doi.org/10.1136/bmj.n71 * Schlitz, M., Bem, D., Marcusson-Clavertz, D., Cardeña,
* Parra, A., & Argibay, J. C. (2013a). Anomalous remote E., Lyke, J. Grover, R., Blackmore, S., Tressoldi, P.,
diagnosis: Mental and motor psi impressions under Roney-Dougal, S., Bierman, D., Jolij, J., Lobach, E.,
iconic representation of the person-target. Journal of Hartelius, G., & Delorme, A. (2021). Two replica-
Parapsychology, 77(1),123–130. tion studies of a time-reversed (psi) priming task
* Parra, A., & Argibay, J.C. (2013b). Psi and death of the and the role of expectancy in reaction times. Jour-
person-target: An experiment with highly emotional nal of Scientific Exploration, 35(1), 69–94. https://doi.
iconic representations. NeuroQuantology, 11(4), 537– org/10.31275/20211903
543. https://doi.org/10.14704/nq.2013.11.4.704 * Schlitz, M. & Delorme, A. (2021). Examining implicit
* Rabeyron, T. (2014). Retro-priming, priming, and dou- beliefs in a replication attempt of a time-reversed
ble testing: psi and replication in a test-retest design. priming task [version 2; peer review: 2 approved].
Frontiers in Human Neuroscience, 8, 1–7. doi:10.3389/ F1000Research, 10(5) https://doi.org/10.12688/
fnhum.2014.00154 f1000research.27169.2
* Rabeyron, T., Roe, C., Mousseau, M. C., & Deledalle, A. * Schönwetter, T., Ambach, W., & Vaitl, D. (2011a). Does
(2018). Anomalous experiences, mental health, and autonomic nervous system activity correlate with
creativity: Is psi the missing link? Journal of Conscious- events conventionally considered as unperceivable?
ness Studies, 25(3-4), 207–232. Using a guessing task with physiological measure-
* Rabeyron, T. & Watt, C. (2010). Paranormal experiences, ment. Journal of Parapsychology, 75(2), 327–348.
mental health and mental boundaries, and psi. Per- * Schönwetter, T., Ambach, W., & Vaitl, D. (2011b). Does
sonality and Individual Differences, 48, 487–492. doi: a modified guilty knowledge test reveal anomalous
10.1016/j.paid.2009.11.029 interactions within pairs of participants? Journal of
Rhine, J. B. (1934). Extra-sensory perception. Bos- Parapsychology, 75(1), 93–118.
ton Society for Psychical Research. https://doi. * Sheldrake, R., & Beeharee, A. (2016). Is joint attention
org/10.1037/13314-000 detectable at a distance? Three automated, inter-
Rhine, J. B., Pratt, J. G., Stuart, C. E., Smith, B. M., & Green- net-based tests. Explore: The Journal of Science and
wood, J. A. (1940/1966). Extra-sensory perception after Healing, 12(1), 34–41. https://doi.org/10.1016/j.ex-
sixty years. Bruce Humphries. plore.2015.10.006
* Roe, C. A., Grierson, S., & Lomas, A. (2012). Feeling the * Sheldrake, R., Smart, P., & Avraamides, L. (2015). Au-
future: Two independent replication attempts. In Ab- tomated tests for telephone telepathy using mobile
stracts of Presented Papers Parapsychological Associa- phones. Explore: The Journal of Science and Healing, 11,
tion 55th Annual Convention (pp. 52–53). Parapsycho- 310–319.
logical Association. * Simmonds-Moore, C. (2014). Exploring the perceptu-
Rubio-Aparicio, M., López-López, J. A., Sánchez-Meca, al biases associated with believing and disbelieving
J., Marín-Martínez, F., Viechtbauer, W., & Van den in paranormal phenomena. Consciousness and Cog-
Noortgate, W. (2018). Estimation of an overall stan- nition, 28, 30–46. https://doi.org/10.1016/j.con-
dardized mean difference in random-effects me- cog.2014.06.004
ta-analysis if the distribution of random effects de- Steinkamp, F. (2005). Forced-choice ESP experiments:
parts from normal. Research Synthesis Methods, 9(3), Their past and their future. In M. A. Thalbourne & L.
489–503. https://doi.org/10.1002/jrsm.1312 Storm (Eds.), Parapsychology in the 21st century: Es-
Savva, L., Child, R. & Smith, M. D. (2004). The precogni- says on the future of psychical research (pp. 124–163).

journalofscientificexploration.org JOURNAL OF SCIENTIFIC EXPLORATION • VOL. 37, NO 3– FALL 2023 531

FORCED CHOICE ESP META ANALYSIS 1987-2022 Lance Storm & Patrizio Tressoldi

McFarland. Report: Anomalous perception in a Ganzfeld condi-

Steinkamp, F., Milton, J., & Morris, R. L. (1998). A me- tion - A meta-analysis of more than 40 years investi-
ta-analysis of forcedchoice experiments comparing gation [version 1; peer review: awaiting peer review].
clairvoyance and precognition. Journal of Parapsychol- F1000Research, 10, 234. https://doi.org/10.12688/
ogy, 62, 193–218. f1000research.51746.2
* Storm, L., Ertel, S. & Rock, A. (2013). The sheep-goat ef- * Varvoglis, M., Bancel, P. A., Bailly, J.-P., Boban, J., &
fect as a matter of compliance vs. noncompliance: The Ahmed, D. s. (2019). The Selfield: Optimizing pre-
effect of reactance in a forced-choice ball selection cognition research. Journal of Parapsychology, 83(1),
test. Journal of Scientific Exploration, 27(3), 93–413. 13–24. https://doi.org/10.30891/jopar.2019.01.02
Storm, L. (in press). The dark spirit of the trickster arche- * Vernon, D. (2015). Exploring precognition using a repe-
type in parapsychology. Journal of Scientific Explora- tition priming paradigm. Journal of the Society for Psy-
tion. chical Research, 79, 65–79.
Storm, L., & Tressoldi, P. E. (2017). Gathering in more Vernon, D., Sandford, T., & Moyo, E. (2019). A test of telep-
sheep and goats: A meta-analysis of forced-choice athy using immersive virtual reality. Paper presented
sheep-goat studies, 1994-2015. Journal of the Society at the 62nd Annual Convention of the Parapsycholog-
for Psychical Research, 81(2), 79-107. ical Association, Paris, 2019.
Storm, L., Tressoldi, P. E., & Di Risio, L. (2010). Meta-anal- Viechtbauer, W. (2010). Conducting Meta-Analyses in
yses of freeresponse studies 1992–2008: Assessing R with the metafor package. Journal of Statistical
the noise reduction model in parapsychology. Psycho- Software, 36(3), 148. https://doi.org/10.18637/jss.
logical Bulletin, 136, 491–494. doi:10.1037/ a0019840 v036.i03
Storm, L., Tressoldi, P. E., & Di Risio, L. (2012). Meta-anal- * Wagenmakers, E. J., Wetzels, R., Borsboom, D., et al.
ysis of ESP studies, 1987-2010: Assessing the success (2012). An agenda for purely confirmatory research.
of the forced- choice design in parapsychology. Jour- Perspective in Psychological Science, 7(6), 632–638.
nal of Parapsychology, 76(2), 243–273. https://doi.org/10.1177/1745691612463078
Tart, C. T. (1983). Information acquisition rates in forced- Watt, C. & Nagtegaal, M. (2000). Luck in action? belief in
choice ESP experiments: Precognition does not work good luck, Psi-mediated instrumental response, and
as well as present-time ESP. Journal of the American games of chance. Journal of Parapsychology, 64(1),
Society for Psychical Research, 77, 293–310. 33–52.
Thalbourne, M. A., & Storm, L. (2014). A further study of * Wehrman, J. J. (2019). No evidence for retrocausation
psychopraxia using the I Ching. Australian Journal of in two classic cuing paradigms. SAGE Open, 9(2),
Parapsychology, 14, 115–142. 2158244019855852
Tressoldi, P., & Katz, D. (2023). Remote Viewing: a 1974- * Wittmann, M., Scheck, F., Feldmann, J., Glaesmann, A.,
2022 systematic review and meta-analysis. PsyArXiv, Mossbridge, J., & Bem, D. (2021). The German ver-
https://doi.org/10.31234/osf.io/r9yw5 sion of a retroactive priming task shows mixed ef-
Tressoldi, P. E., Martinelli, M., Zaccaria, E., & Massacessi, fects. Psychology of Consciousness: Theory, Research,
S. (2009). Implicit intuition: How heart rate can con- and Practice. Advance online publication. https://doi.
tribute to prediction of future events. Journal of the org/10.1037/cns0000296
Society for Psychical Research, 73, 1–16. Zdrenka, M., & Wilson, M. S. (2017). Individual difference
Tressoldi, P. E., Martinelli, M., Scartezzini, L., & Massa- correlates of psi performance in forced-choice pre-
cessi, S. (2010). Further evidence of the possibility of cognition experiments: A meta-analysis (1945-2016).
exploiting anticipatory physiological signals to assist Journal of Parapsychology, 81(1), 9–32.
implicit intuition of random events. Journal of Scientif- Zilberman, M. (1995). On the training of precognitive abil-
ic Exploration, 24, 411–424. ity. Journal of the Society for Psychical Research, 60,
Tressoldi, P. E., & Storm, L. (2023). Stage 2 Registered 289–292.

532 JOURNAL OF SCIENTIFIC EXPLORATION • VOL. 37, NO 3 – FALL 2023 journalofscientificexploration.org

Lance Storm & Patrizio Tressoldi FORCED CHOICE ESP META ANALYSIS 1987-2022

APPENDIX A: Prisma Flowchart

journalofscientificexploration.org JOURNAL OF SCIENTIFIC EXPLORATION • VOL. 37, NO 3– FALL 2023 533

FORCED CHOICE ESP META ANALYSIS 1987-2022 Lance Storm & Patrizio Tressoldi

APPENDIX B: List of New Studies with Hits Data in the Meta-Analysis and Their Results

Author(s) Year Trials Hits Z ES (z/√n) Task

1 Bem (expt. 1) 2011 3600 1912 3.72 0.062 PR
2 Bem (expt. 2) 2011 5400 2790 2.44 0.033 PR
3 Bem (expt. 3) 2011 1800 963 2.95 0.070 PR
4 Bem (expt. 5) 2011 2400 1274 2.90 0.059 PR
5 Bem (expt. 6) 2011 3600 1865 2.15 0.036 PR
6 Schönwetter et al. 2011a 657 151 -1.15 -0.045 CL
7 Schönwetter et al. 2011b 350 71 0.07 0.004 CL
8 Hitchman et al. (nonintentional) 2012 750 201 1.10 0.040 PR
9 Houran & Lange 2012 60 27 3.43 0.443 PR
10 Luke et al. 2012 200 43 -1.06 -0.075 PR
11 Dalkvist 2013 19560 9585 -2.78 -0.020 TE
12 Ertel 2013 7740 2293 21.16 0.241 CL
13 Lange & Houran 2013 60 34 5.52 0.713 PR
14 Parra & Argibay (believers; mental) 2013a 896 520 4.78 0.160 CL
15 Parra & Argibay (believers; motor) 2013a 896 482 2.24 0.075 CL
16 Parra & Argibay (live/nolive) 2013b 856 511 5.64 0.193 CL
17 Parra & Argibay (suicide/nosuicide) 2013b 856 458 2.02 0.069 CL
18 Parra & Argibay (motor live/nolive) 2013b 856 424 -0.24 -0.008 CL
19 Parra & Argibay (motor suicide/nosuicide) 2013b 856 443 0.99 0.034 CL
20 Storm et al. 2013 12016 2531 2.90 0.026 CL
21 Houran & Lange 2014 744 187 0.04 0.001 CL
22 Luke & Morin (contingent) 2014 210 57 0.64 0.044 PR
23 Luke & Morin (no-contingent) 2014 200 58 1.22 0.086 PR
24 Luke & Zychowicz (intentional; intuition) 2014 200 40 -1.55 -0.110 PR
25 Luke & Zychowicz (nonintentional; preference) 2014 200 52 0.24 0.017 PR
26 Simmonds-Moore 2014 95 22 -0.30 -0.031 CL
27 Simmonds-Moore 2014 95 24 -0.06 -0.006 CL
28 Billows & Storm 2015 3725 736 -0.35 -0.006 CL
29 Hitchman et al. (intentional) 2015 500 245 -0.40 -0.018 PR
30 Hitchman et al. (nonintentional) 2015 500 247 -0.22 -0.010 PR
31 Hitchman et al. (nonintentional) 2015 735 390 1.62 0.060 PR
32 Sheldrake et al. (three callers) 2015 1728 718 7.22 0.174 TE
33 Sheldrake et al. (two callers) 2015 660 370 3.08 0.120 TE
34 Hitchman et al. (nonintentional) 2016 624 160 0.32 0.013 PR
35 Sheldrake & Beeharee (study 1) 2016 11160 5901 6.07 0.057 TE
36 Sheldrake & Beeharee (study 2) 2016 2720 1395 1.32 0.025 TE
37 Sheldrake & Beeharee (study 3) 2016 8860 4594 3.47 0.037 TE
38 Kekecs et al. 2019 37836 18876 -0.43 -0.002 PR
39 Varvoglis et al. (all no feed) 2019 1172 570 -0.91 -0.027 PR
40 Varvoglis et al. (all with feed) 2019 1828 932 0.82 0.019 PR
41 Varvoglis et al. (selected no feed) 2019 358 179 0.00 0.000 PR
42 Varvoglis et al. (selected with feed) 2019 602 321 1.59 0.065 PR
43 Arora et al. 2022 5148 2566 -0.21 -0.003 PR
44 Escolà-Gascón (mediums; positive) 2022 900 432 -1.17 -0.039 CL
45 Escolà-Gascón (mediums; neutral) 2022 900 287 -10.83 -0.361 CL
46 Escolà-Gascón (mediums; haunted) 2022 900 443 -0.43 -0.014 CL
47 Escolà-Gascón (non-believers; positive) 2022 900 238 -14.10 -0.470 CL
48 Escolà-Gascón (non-believers; neutral) 2022 900 252 -13.17 -0.439 CL
49 Escolà-Gascón (non-believers; haunted) 2022 900 248 -13.43 -0.448 CL
50 Escolà-Gascón et al. (study 1) 2022 7110 2133 -33.72 -0.400 PR
51 Escolà-Gascón et al. (study 2; healthy) 2022 3630 1117 -23.15 -0.384 PR
52 Escolà-Gascón et al. (study 2; psychiatric) 2022 3540 1310 -15.45 -0.260 PR

534 JOURNAL OF SCIENTIFIC EXPLORATION • VOL. 37, NO 3 – FALL 2023 journalofscientificexploration.org

 Lance Storm & Patrizio Tressoldi FORCED CHOICE ESP META ANALYSIS 1987-2022

APPENDIX C: List of New Studies with No Hits Data (RT Studies) in the Meta-Analysis and Their
Results

Author(s) Year Trials t ES (z/√n) Task

1 Savva & French (expt. 1) 2002 1371 0.83 0.022 PR
2 Savva & French (expt. 2) 2002 1800 1.19 0.028 PR
3 Savva & French (expt. 3) 2002 4320 0.00 0.000 PR
4 Boer & Bierman 2006 9792 2.98 0.030 PR
5 Rabeyron & Watt 2010 4960 1.32 0.019 PR
6 Bem (expt. 4) 2011 3168 2.03 0.036 PR
7 Bem (expt. 7) 2011 9600 1.31 0.013 PR
8 Bierman 2011 5408 1.41 0.019 PR
9 Roe et al. (study 1) 2012 1504 0.69 0.018 PR
10 Roe et al. (study 2) 2012 1344 -0.64 -0.017 PR
11 Wagenmakers et al. 2012 1500 -0.22 -0.006 PR
12 Bierman & Bijl 2014 4288 2.59 0.040 PR
13 Rabeyron 2014 896 -1.35 -0.045 PR
14 Vernon 2015 9792 1.55 0.016 PR
15 Rabeyron et al. 2018 2944 -0.25 -0.005 PR
16 Jolij & Bierman (expt. 1) 2019 7104 0.21 0.002 PR
17 Jolij & Bierman (expt. 2) 2019 1984 -0.15 -0.003 PR
18 Wehrman 2019 4080 0.50 0.008 CL
19 Schlitz & Delorme 2021 4449 -0.29 -0.004 PR
20 Schlitz et al. (expt. 1) 2021 19720 0.76 0.005 PR
21 Schlitz et al. (expt. 2) 2021 22560 1.68 0.011 PR
22 Wittmann et al. (expt. 1) 2021 3640 0.53 0.009 PR
23 Wittmann et al. (expt. 2) 2021 3720 -2.34 -0.038 PR
24 Muhmenthaler et al. (expt. 1) 2022 26171 -0.45 -0.003 PR
25 Muhmenthaler et al. (expt. 2) 2022 50904 -0.46 -0.002 PR

journalofscientificexploration.org JOURNAL OF SCIENTIFIC EXPLORATION • VOL. 37, NO 3– FALL 2023 535

View publication stats