This article was downloaded by: [Lulea University of Technology]

On: 19 September 2013, At: 21:43

Publisher: Routledge
Informa Ltd Registered in England and Wales Registered Number: 1072954
Registered office: Mortimer House, 37-41 Mortimer Street, London W1T 3JH,
UK

West European Politics

Publication details, including instructions for authors
and subscription information:
http://www.tandfonline.com/loi/fwep20

Measuring Populism: Comparing

Two Methods of Content Analysis
Matthijs Rooduijn & Teun Pauwels
Published online: 01 Nov 2011.

To cite this article: Matthijs Rooduijn & Teun Pauwels (2011) Measuring Populism:
Comparing Two Methods of Content Analysis, West European Politics, 34:6, 1272-1283,
DOI: 10.1080/01402382.2011.616665

To link to this article: http://dx.doi.org/10.1080/01402382.2011.616665

PLEASE SCROLL DOWN FOR ARTICLE

Taylor & Francis makes every effort to ensure the accuracy of all the
information (the “Content”) contained in the publications on our platform.
However, Taylor & Francis, our agents, and our licensors make no
representations or warranties whatsoever as to the accuracy, completeness, or
suitability for any purpose of the Content. Any opinions and views expressed
in this publication are the opinions and views of the authors, and are not the
views of or endorsed by Taylor & Francis. The accuracy of the Content should
not be relied upon and should be independently verified with primary sources
of information. Taylor and Francis shall not be liable for any losses, actions,
claims, proceedings, demands, costs, expenses, damages, and other liabilities
whatsoever or howsoever caused arising directly or indirectly in connection
with, in relation to or arising out of the use of the Content.

This article may be used for research, teaching, and private study purposes.
Any substantial or systematic reproduction, redistribution, reselling, loan, sub-
licensing, systematic supply, or distribution in any form to anyone is expressly
 forbidden. Terms & Conditions of access and use can be found at http://
www.tandfonline.com/page/terms-and-conditions
Downloaded by [Lulea University of Technology] at 21:43 19 September 2013
 West European Politics,
Vol. 34, No. 6, 1272–1283, November 2011

RESEARCH NOTE

Measuring Populism: Comparing

Two Methods of Content Analysis
Downloaded by [Lulea University of Technology] at 21:43 19 September 2013

MATTHIJS ROODUIJN and TEUN PAUWELS

The measurement of populism – particularly over time and space – has received only
scarce attention. In this research note two different ways to measure populism are
compared: a classical content analysis and a computer-based content analysis. An
analysis of political parties in the United Kingdom, the Netherlands, Germany and Italy
demonstrates that both methods can be used to measure populism across countries and
over time. Recommendations are presented on how to combine these methods in future
comparative research on populism.

The term populism has been applied to a wide range of movements,

politicians and parties. This wide application reflects one of the most
problematic characteristics of the concept: its contextual sensitivity. Because
of this contextual sensitivity, populism is in turn plagued by a lack of
conceptual clarity (Canovan 1981; Taggart 2000; Barr 2009; Laclau 2005).
Consequently, scholars have not yet developed systematic methods to
empirically measure populism across cases and over time. There have been
some empirical investigations into populism, yet most of these are single
case studies (e.g. Mény and Surel 2002; Albertazzi and McDonnell 2008). A
more systematic and comparative perspective is still lacking. This is
problematic, because the most relevant scientific questions are empirical
and comparative in nature: are we increasingly living in a populist zeitgeist
(Mudde 2004)? What other ideologies are combined with the thin-centred
ideology of populism? Under what circumstances are mainstream parties
becoming populist? In this research note, we do not aim to answer such
empirical questions. Yet we do focus on a fundamental prerequisite for
answering these questions: the methodological issue of how populism could
be measured.
We compare two methods to measure populism: (1) a classical content
analysis where coders systematically analyse texts by means of a codebook;
and (2) a computerised content analysis in which an a priori designed

Correspondence Address: [email protected]

ISSN 0140-2382 Print/1743-9655 Online ª 2011 Taylor & Francis

http://dx.doi.org/10.1080/01402382.2011.616665
 Measuring Populism 1273

dictionary serves as a gauge of the degree of populism. The study

demonstrates that both approaches can be used to measure populism
across cases and over time. We conclude the research note by showing how
these two methods to measure populism can fruitfully be combined in one
integrated man–machine approach.
We are not the first to measure populism by means of the method of
content analysis. Jagers and Walgrave (2007) were among the first to do
that. But although their analysis formed a breakthrough in measuring
populism, their study included only one country and issues of reliability and
Downloaded by [Lulea University of Technology] at 21:43 19 September 2013

validity were not dealt with. Another content analysis of populism has been
executed by Hawkins (2009). He analysed speeches by means of ‘holistic
grading’, in which the unit of measurement is the entire text. The main
problems with Hawkins’ study are the rough – and therefore possibly
invalid – measurement due to the holistic grading method and the at times
low reliability (Kappa ¼ 0.44). More recently, Pauwels (2011) also measured
populism by means of a computer-based content analysis. Yet he studied the
Belgian case only. Building on these previous studies, we focus extensively
on issues of validity (i.e. evaluating whether we measure what we think we
are measuring) and reliability (i.e. the consistency of the measurement),
while maintaining a comparative perspective.

Populism as a Thin Ideology Consisting of Two Components

In order to measure a concept systematically, we first have to agree on a
clear definition. We build on the definition of populism provided by
Mudde (2004: 543): ‘[populism is] an ideology that considers society to be
ultimately separated into two homogeneous and antagonistic groups, ‘‘the
pure people’’ versus ‘‘the corrupt elite’’, and which argues that politics
should be an expression of the volonte´ ge´ne´rale (general will) of the
people’. Although other scholars have defined populism as a style (Jagers
and Walgrave 2007), a discourse (Hawkins 2009) or an organisational
form (see Taggart 1995), we focus on ideology for its substantive approach
and its denotative clarity (Canovan 2002; Albertazzi and McDonnell 2008;
Abts and Rummens 2007; Stanley 2008). Populism is not a ‘full ideology’,
such as liberalism or socialism, but a ‘thin ideology’ (Freeden 1998), which
only focuses on a confined range of concepts (e.g. nationalism, feminism,
ecology).
According to Mudde’s definition, the thin ideology of populism consists
of two components: people-centrism and anti-elitism. The point of
departure of every populist is the fundamental importance of the centrality
of ‘the people’ (Ionescu and Gellner 1969: 4; Mény and Surel 2002). Yet ‘the
people’ can mean many different things in many different circumstances
(Canovan 1981; Mudde 2004). It can refer, for instance, to peasants, the
working class, the electorate, the nation or no fixed group at all (see
Canovan 1981; Taggart 2000). Populists are anti-elitist because elites stand
 1274 M. Rooduijn and T. Pauwels

in the way of the centrality of the people. Elites are portrayed as corrupt and
are contrasted with the general will of the people (Mudde 2004).

Research Strategy
The unit of analysis in our content analyses is the election manifesto.1 This
unit was chosen for two reasons. The first, substantive, reason is that an
election manifesto can be seen as the document that gives the clearest
overview of what a party stands for at a certain point in time. The second,
Downloaded by [Lulea University of Technology] at 21:43 19 September 2013

more practical, reason is that election manifestos are appropriate documents

for a cross-national study, because they are reasonably comparable across
countries and over time.
We focus on election manifestos of parties in Western Europe because ‘the
main area of sustained populist growth and success over the last fifteen years
in established democracies has been Western Europe’ (Albertazzi and
McDonnell 2008: 1). More specifically, we selected the Netherlands, the
United Kingdom, Germany and Italy because these countries accommodate
a wide range of different kinds of allegedly populist parties: the Partij voor de
Vrijheid (PVV) (Vossen 2010), the Lijst Pim Fortuyn (LPF) (Mudde 2007)
and the Socialistische Partij (SP) (March 2007) in the Netherlands; the British
National Party (BNP) and the United Kingdom Independence Party (UKIP)
(Fella 2008) in the UK; the Partei des Demokratischen Sozialismus (PDS,
later Die Linke) (March 2007) in Germany; and Forza Italia (FI) and the
Lega Nord (LN) in Italy (Tarchi 2008). They range from the left (e.g. PDS/
Die Linke) to the right (e.g. the PVV), and from the electorally very
successful (e.g. Forza Italia) to relatively unsuccessful (e.g. the BNP). Our
focus is not only on the ‘usual suspects’, but also on the mainstream parties in
each country – i.e. liberal, conservative and social democratic parties.2 The
reason for doing so is to explore whether our measurement enables us to
distinguish populist parties from mainstream parties. Moreover, some
mainstream parties such as the Labour party under Tony Blair have also
been labelled as populist.
We have not only compared parties across space, but also over time.
Three election years were selected in the United Kingdom, Italy and the
Netherlands, and – for practical reasons – two election years in Germany.3
For an overview see Appendix A.

The Classical Content Analysis

In our classical content analysis of populism, the manifestos were analysed
by extensively trained coders by means of a codebook. In this codebook,
people-centrism was operationalised by the following question: ‘Do the
authors of the manifesto refer to the people?’ Coders were instructed to look
at every possible reference to the people. It did not matter whether this
reference concerned, for instance, ‘citizens’, ‘our country’, ‘society’ or ‘we’
 Measuring Populism 1275

(as in ‘we the people’). Coders were also instructed to interpret the broader
context in deciding whether to code people-centrism or not. To help the
coders, we provided an extensive list of words and combinations of words
that could refer to the people.4
Anti-elitism was measured by means of the question: ‘Do the authors of
the manifesto criticise elites?’ Critique had to concern political elites in
general. Critique on a specific party or a particular politician is not general
enough, and was therefore not coded. Because anti-elitism can be expressed
in many different ways, coders were again instructed to interpret the context
Downloaded by [Lulea University of Technology] at 21:43 19 September 2013

while coding.5
The unit of measurement is the paragraph because paragraphs are
objectively traceable distinctions between arguments. We have selected
those paragraphs as populist in which both people-centrism and anti-elitism
were present. Eventually the percentage of populist paragraphs in every
election manifesto was computed.

The Computer-Based Content Analysis

Since a classical content analysis is a very time-consuming and therefore
expensive enterprise, we have also developed a much easier applicable
measurement of populism, drawing on a computer-based content analysis.
This measurement relies on the dictionary approach in which a computer
counts the proportion of words that we consider to be indicators of
populism.6 This means that words instead of paragraphs are the unit of
measurement. Although sceptics might argue that the same word can have
different meanings depending on the context, it is mostly possible to code
words unambiguously (Laver and Garry 2000).7
After a first explorative analysis, it turned out that a measurement of
people-centrism by means of individual words only is nearly impossible. In
many instances, ‘the people’ is referred to by the words ‘we’ and ‘our’ (e.g.
‘we [the people] need to raise our voice’). Yet not every mention of the words
‘our’ or ‘we’ is a reference to the people. Often, these words refer to the
political party instead of the people (e.g. ‘we [the party] propose our plans in
the next chapter’). We therefore decided to focus only on words that refer to
anti-elitism. Although the computer-based measurement is therefore likely
to be less valid, we believe that anti-elitism is a pretty good indicator of
populism because the classical content analysis taught us that criticism
towards elites is mostly motivated by the argument that elites betray the
ordinary people. The argument cannot be reversed, however, because many
political parties will centralise the people without being negative towards
elites.8 Whether anti-elitism alone is indeed a good indicator of populism
can only be concluded from the comparison of our measurements.
Our selection of words for the dictionary was based on both empirical and
theoretical reasoning. For inspiration we used empirical examples (election
manifestos of allegedly populist parties we did not actually analyse) to make
 1276 M. Rooduijn and T. Pauwels

a list of words that such parties have used to express their negativity towards
elites. Yet theoretical reasons were decisive for our final decision whether to
include these words in our dictionary or not: only those words have been
selected that were explicitly used to position the bad elites against the good
people. The development of a dictionary is not an easy task, however. Not
every word that could refer to anti-elitism does always refer to it, while at
the same time many instances of anti-elitism can easily be missed because it
is impossible to formulate every word that could refer to anti-elitism
beforehand. Theoretically deduced words that never seemed to appear in the
Downloaded by [Lulea University of Technology] at 21:43 19 September 2013

manifestos of any party were excluded. We attempted to translate the

dictionary for the four languages as accurately as possible. However, besides
the translated ‘core words’ we also added some ‘context-specific words’. The
context-specific words are words which are too context-specific to be
translated from one language to another.9
Although we are aware of the possible pitfalls when translating the
dictionary, we argue that the theoretical argument of anti-elitism is generally
similar across cases and over time. Whatever the specific context, populists in
every country and every time period do essentially the same thing: they
position the good people against the bad elites. Because they make this same
argument, we assume that they also use similar words. We used the open
software programme Yoshikoder to measure the percentage of our dictionary
words. For a complete overview of our dictionary, see Appendix B.

Results
Validity
We focus on three types of validation: content validation, face validation
and concurrent validation. A measurement is content valid if the syste-
matised concept is adequately captured by its indicator or indicators
(Adcock and Collier 2001). We have argued that populism consists of the
combination of people-centrism and anti-elitism. In the classical analysis we
have measured it accordingly, since we verified for every measurement unit
whether there was a reference to ‘the people’ combined with a critique on
elites in general. Yet the computer-based analysis is less content valid. After
all, here the systematised concept of populism is not adequately captured by
its indicators. Instead of measuring both people-centrism and anti-elitism,
the computerised content analysis only focused on anti-elitism.
A measurement has face validity when it appears to be measuring the
concept that it intends to measure (Weber 1990). In our case, the
measurement of populism has face validity if the allegedly populist parties
turn out to be populist indeed. The results of both content analyses are
presented in Figure 1. Allegedly populist parties are represented by black
circles. Many mainstream parties are not represented by white, but by
transparent circles.
 Measuring Populism 1277

F IG U R E 1
TH E C LA SSI CA L AN D C O M PU TE RI S E D C O N T EN T AN A L Y SE S C O M P A R ED
Downloaded by [Lulea University of Technology] at 21:43 19 September 2013

Most allegedly populist parties turn out to be rather populist indeed. It

could therefore be argued that the measurements are sufficiently face valid.
In the Netherlands only the SP in 2002 and 2006 is less populist than would
be expected from the literature. Because the party was strongly populist in
1994, however, it seems as if the party has moderated its populist profile
over the years. According to both measurement methods, PDS/Die Linke in
Germany is only slightly populist. In Italy the Lega Nord and Forza Italia
score moderately high.
If we look at the mainstream parties, most parties turn out to be only
slightly populist or not populist at all. Yet there are also some mainstream
parties that turn out to be rather populist. In the United Kingdom the
Conservatives in 2001 and the Liberal Democrats in 2005 score moderately
high on our populism scales. An Italian mainstream party that turned out to
be rather populist is the social democratic DS, which turns out to be very
populist in the beginning of the 1990s. This might be due to some specific
features of the Italian political situation. Further research is needed to
determine the reasons for the high degree of populism of the DS.
Concurrent validity entails that a measure is valid if the results of a
measurement of the systematised concept in one study are empirically
 1278 M. Rooduijn and T. Pauwels

related to the results of a different measurement of the same systematised

concept in another study (Adcock and Collier 2001). We can test whether
our measurements of populism are valid by comparing their results. We can
conclude that the results are pretty concurrent. There is a strong correlation
between the results of the two measurements: Pearson’s r ¼ 0.80 (significant
at p 5 0.01). There are also some important differences between the two
measurements. For the Dutch LPF and SP (in 1994) the results differ
strongly from each other: according to the classical content analysis they are
much more populist than according to the computerised method.10 Because
Downloaded by [Lulea University of Technology] at 21:43 19 September 2013

many experts describe the LPF and the SP (in the 1990s) as populist parties
(see Mudde 2007; Lucardie 2010; Van der Brug 2003 for the LPF; and
March 2007; Voerman 2009; Vossen 2010 for the SP), it is likely that the
classical content analysis generated a better estimation of their true degree
of populism than the computerised analysis.
The validity of the computerised method thus turns out to be lower than
that of the classical approach. The computerised measurement is still
sufficiently valid, however. The biggest problem with this method is that the
indicator of anti-elitism alone does not cover the whole concept, which
consists of both people-centrism and anti-elitism. Yet empirically the effect
of this theoretical shortcoming is limited. Face validation and concurrent
validation show that anti-elitism alone is a pretty good indicator of
populism. This is very good news for comparative scholars. Despite its
contextual sensitivity, populism turns out to be measurable across countries
and over time. Not only by means of a fine-grained classical content
analysis, but even with a more easily applicable computerised method.

Reliability
In a classical content analysis inter-coder reliability is the extent to which
different coders code the same text in the same way (Krippendorff 2004).
In order to prevent low inter-coder reliability, we have extensively trained
our 11 coders (four from the Netherlands, three from the United
Kingdom, two from Germany and two from Italy). Every coder attended
three training sessions in which the codebook was explained and in which
coding examples were discussed. In between the training sessions the
coders had to complete take-home exercises. After the training sessions, we
assessed the inter-coder reliability. Coders had to complete two reliability
tests. First, all coders had to analyse a sample of paragraphs from British
election manifestos (all coders speak English), so we could calculate
whether the cross-national inter-coder reliability was sufficient. We
calculated the inter-coder reliability using Krippendorff’s alpha. The
results for cross-national reliability are a ¼ 0.73 for people-centrism and
a ¼ 0.70 for anti-elitism. Second, all coders had to analyse another sample
of paragraphs from election manifestos of parties from their own
countries, so we were able to assess the national inter-coder reliability
 Measuring Populism 1279

coefficients. The Krippendorff’s alphas range from 0.66 to 0.92. The

statistics in general are satisfactory.11
One of the advantages of the computer-based content analysis is its
accuracy. Since a computer produces the exact same results no matter how
many times one runs the analysis, Laver and Garry (2000: 625) claim that
‘[c]omputer coding is 100 percent reliable’. This is, however, a rather one-
sided way to look at reliability. It must be kept in mind that different
researchers of populism would probably end up with different dictionaries
to measure the concept, which in turn would impact upon the results. We
Downloaded by [Lulea University of Technology] at 21:43 19 September 2013

have performed a split-half test to shed some light on this issue. First, the
words of the dictionaries for each country were randomly divided into two
groups. For each country this resulted in two ‘half’ dictionaries. In a second
step we explored the Pearson correlation coefficients between the results
computed by the two ‘half’ dictionaries in each country. The results are: 0.54
for the United Kingdom, 0.42 for the Netherlands, 0.23 for Germany and
0.53 for Italy. To control for analysing ‘half’ dictionaries only, we adjusted
the split-half correlations by means of the Spearman-Brown prophecy
formula which resulted in reliability scores of 0.70 for the United Kingdom,
0.59 for the Netherlands, 0.37 for Germany and 0.69 for Italy.12 Except for
Germany, which might be explained by the lack of populist parties and
hence variation in the scores, these statistics seem sufficient. So even when
the split-half test provides a different picture than that suggested by Laver
and Garry (2000) we would nonetheless argue that the computerised
approach is reliable enough to be employed in empirical research. Whether
it is more or less reliable than the classical content analysis is difficult to
assess because of the different approaches in reliability testing.

Conclusion
The measurement of populism, particularly over time and space, has not
received much attention yet. In this research note we have paid extensive
attention to the measurement of populism over time and across countries.
By means of both a classical content analysis as well as a computerised
method, we investigated the degree of populism among political parties in
the United Kingdom, the Netherlands, Germany and Italy. The usual
suspects were the most populist according to our measurements, although
we also demonstrated that even mainstream parties might sometimes adhere
to populist appeals. Most importantly, the two methods generated similar
results. Although the classical approach turns out to be more valid, and
possibly also more reliable, both approaches can be used to measure
populism across cases and over time.
It is also possible to combine both methods. In particular for medium-n
comparative studies on populism we propose a combination of the classical
and computer-based method: an integrated man–machine approach that
filters out the disadvantages and emphasises the strong points of both
 1280 M. Rooduijn and T. Pauwels

methods. This method can be applied to every kind of text (spoken or

written): from speeches to election manifestos and from newspaper articles to
parliamentary debates. The following procedure would probably yield valid
and reasonably reliable results. First, every word that could refer to the people
or to negativity towards elites is included in a dictionary. Second, a computer
scans the relevant texts for all the selected words. Third, human coders
interpret the results of this computer analysis by means of a codebook.
Focusing on the selected words, they decide for every paragraph whether
there is a reference to the people and whether the author of the text is negative
Downloaded by [Lulea University of Technology] at 21:43 19 September 2013

towards elites. On the one hand, this approach will be much faster and
therefore less expensive than a classical content analysis. On the other hand, it
will produce more valid results than a computerised analysis because the first
selection of the computer will be double-checked by human coders who are
able to interpret the contextual meanings of the selected words.

Acknowledgements
Earlier versions of this research note have been presented at various
locations, including the ECPR Joint Sessions in Münster and the
Comparative Politics PhD Club at the University of Amsterdam. We want
to thank all participants for their valuable feedback. In particular we thank
Wouter van der Brug, Kris Deschouwer, Kirk Hawkins, Liesbet Hooghe,
Sarah de Lange and Bojan Todosijevic. Finally we want to thank the
reviewers of West European Politics for their useful comments.

Notes
1. The unit of analysis should not be confused with the unit of measurement. The classical and
the computerised content analysis approach have the same unit of analysis (election
manifestos), but different units of measurements (paragraphs and words respectively). This is
due to the different points of departure of the two methods. More on this in the next sections.
2. For Italy we simply included the most important leftist and rightist parties considering the
absence of liberal parties (see Appendix A). We also included the 1992 manifesto of the social
democrats (DS) because the number of available Italian election manifestos was rather low.
3. We only included the 2002 and 2005 party manifestos for Germany because manifestos
prior to these dates were often not available in a legible digital format, which is needed for
the computerised analysis.
4. These words are: people, citizen(s), community, society, public, population, nation(al), all
of us, each of us, everyone, our, we, voter(s), electorate, referenda, direct democracy, public
opinion, country. And words such as: United Kingdom, Britons, Netherlands, Dutch,
Italians, Gemany, etc. (depending of course on the country under analysis.)
5. For the codebook the first author can be contacted.
6. While there are different approaches available in computerised textual analyses – such as
Wordscores or Wordfish – we draw on a dictionary approach (Laver and Garry 2000). A
drawback of Wordscores is that it requires scores to be computed by other methods such as
expert surveys (Laver et al. 2003). Wordfish works well for extracting single left–right
dimensions (Slapin and Proksch 2008), while it is less suited to explore a specific ideological
aspect such as populism.
 Measuring Populism 1281

7. The word ‘taxes’, for instance, might be associated with cutting taxes but it can equally be
used to indicate that a party wants to increase taxes. In practice, however, this latter
meaning will hardly be found in party manifestos, and the word taxes is hence a good
indicator for the category ‘reduce state involvement in the economy’, identifying socio-
economic rightist parties.
8. Indeed, the classical content analysis empirically confirmed this: there is only a weak
correlation between people-centrism and anti-elitism (r ¼ 0.04, not significant at p 5 0.05),
whereas – the other way around – almost every anti-elitist paragraph also contains a
reference to the people.
9. For instance, populists in the Netherlands sometimes talk about ‘regenten’ to express anti-
elitism. This word refers to the Dutch political rulers in the sixteenth, seventeenth and
Downloaded by [Lulea University of Technology] at 21:43 19 September 2013

eighteenth centuries. Although the ‘regenten’ did not form a hereditary class, they did form
a closed group that reserved government offices for themselves. This specific word is not
used by populists in countries other than the Netherlands.
10. This becomes even more apparent when we regress the results of the two methods on each
other and look at the standardised residuals: the LPF and the SP (in 1994) are more than
three standard deviations removed from the mean residual of 0.
11. The sample of paragraphs in the reliability tests contained about 5 per cent of the total
amount of paragraphs. The results for people-centrism are: a ¼ 0.78 (NL), a ¼ 0.73 (UK),
a ¼ 0.74 (GE) and a ¼ 0.81 (IT). The results for anti-elitism are: a ¼ 0.84 (NL), a ¼ 0.66
(UK), a ¼ 0.81 (GE) and 0.81 (IT).
nr
12. Reliability ¼ 1þðn1Þr .

References
Abts, Koen, and Stefan Rummens (2007). ‘Populism versus Democracy’, Political Studies, 55:2,
405–24
Adcock, Robert, and David Collier (2001). ‘Measurement Validity: A Shared Standard for
Qualitative and Quantitative Research’, American Political Science Review, 95:3, 529–46.
Albertazzi, Daniele, and Duncan McDonnell, eds. (2008). Twenty-First Century Populism. The
Spectre of Western European Democracy. London: Palgrave.
Barr, Robert R. (2009). ‘Populists, Outsiders and Anti-establishment Politics’, Party Politics,
15:1, 29–48.
Canovan, Margaret (1981). Populism. London: Junction Books.
Canovan, Margaret (2002). ‘Taking Politics to the People: Populism as the Ideology of
Democracy’, in Yves Mény and Yves Surel (eds.), Democracies and the Populist Challenge.
Basingstoke: Palgrave, 25–44.
Fella, Stefano (2008). ‘Britain: Imperial Legacies, Institutional Constraints and New Political
Opportunities’, in Daniele Albertazzi and Duncan McDonnell (eds.), Twenty-First Century
Populism. The Spectre of Western European Democracy. London: Palgrave, 181–97.
Freeden, Michael (1998). ‘Is Nationalism a Distinct Ideology?’, Political Studies, 46:4, 748–65.
Hawkins, Kirk A. (2009). ‘Is Chávez Populist? Measuring Populist Discourse in Comparative
Perspective’, Comparative Political Studies, 42:8, 1040–67.
Ionescu, Ghita, and Ernest Gellner (1969). ‘Introduction’, in Ghita Ionescu and Ernest Gellner
(eds.), Populism: Its Meanings and National Characteristics. London: Weidenfeld and
Nicolson, 1–5.
Jagers, Jan, and Stefaan Walgrave (2007). ‘Populism as Political Communication Style: An
Empirical Study of Political Parties’ Discourse in Belgium’, European Journal of Political
Research, 46:3, 319–45.
Krippendorff, Klaus (2004). Content Analysis: An Introduction to Its Methodology. London:
Sage Publications.
Laclau, Ernesto (2005). On Populist Reason. London: Verso.
 1282 M. Rooduijn and T. Pauwels

Laver, Michael, and John Garry (2000). ‘Estimating Policy Positions from Political Texts’,
American Journal of Political Science, 44:3, 619–34.
Laver, Michael, Kenneth Benoit, and John Garry (2003). ‘Extracting Policy Positions from
Political Texts Using Words as Data’, American Political Science Review, 97:2, 311–31.
Lucardie, Paul (2010). ‘Tussen establishment en extremisme: populistische partijen in
Nederland en Vlaanderen’, Res Publica, 52:2, 149–72.
March, Luke (2007). ‘From Vanguard of the Proletariat to Vox Populi: Left-populism as a
‘‘Shadow’’ of Contemporary Socialism’, SAIS Review, XXVII:1, 63–77.
Mény, Yves, and Yves Surel (2002). Democracies and the Populist Challenge. Basingstoke:
Palgrave.
Mudde, Cas (2004). ‘The Populist Zeitgeist’, Government and Opposition, 39:4, 541–63.
Downloaded by [Lulea University of Technology] at 21:43 19 September 2013

Mudde, Cas (2007). Populist Radical Right Parties in Europe. Cambridge: Cambridge
University Press.
Pauwels, Teun (2011). ‘Measuring Populism: A Quantitative Text Analysis of Party Literature
in Belgium’, Journal of Elections, Public Opinion and Parties, 21:1, 97–119.
Slapin, Jonathan, and Sven-Oliver Proksch (2008). ‘A Scaling Model for Estimating Time-
Series Party Positions from Texts’, American Journal of Political Science, 52:3, 705–22.
Stanley, Ben (2008). ‘The Thin Ideology of Populism’, Journal of Political Ideologies, 13:1, 95–
110.
Taggart, Paul (1995). ‘New Populist Parties in Western Europe’, West European Politics, 18:1,
34–51.
Taggart, Paul (2000). Populism. Buckingham: Open University Press.
Tarchi, Marco (2008). ‘Italy: A Country of Many Populisms’, in Daniele Albertazzi and
Duncan McDonnell (eds.), Twenty-First Century Populism. The Spectre of Western European
Democracy. London: Palgrave, 84–99.
Van der Brug, Wouter (2003). ‘How the LPF Fuelled Discontent: Empirical Tests of
Explanations of LPF Support’, Acta Politica, 38:1, 89–106.
Voerman, Gerrit (2009). ‘Van Mao to marketing: Over het populisme van de SP’, Socialisme en
Democratie, 9, 26–31.
Vossen, Koen (2010). ‘Different Flavours of Populism in the Netherlands: Four Case Studies.
Paper presented at the ECPR workshop Disassembling Populism (and Putting it Back
Together Again), Muenster, March.
Weber, Robert Philip (1990). Basic Content Analysis, 2nd ed. London: Sage.

A P P EN D IX A
AN AL YSE D E LE C TI ON M AN I FE STO S

1992 1994 2001 2002 2005 2006 2008

Mainstream Parties
CDA (NL) X X X
PvdA (NL) X X X
VVD (NL) X X
D66 (NL) X X X
Conservatives (UK) X X X
Labour (UK) X X X
Liberal Democrats (UK) X X
CDU/CSU (GE) X X
SPD (GE) X X
FDP (GE) X X
DS (IT) X X
Ulivo (IT) X
PD (IT) X

(continued )
 Measuring Populism 1283

APPENDIX A
(Continued )
1992 1994 2001 2002 2005 2006 2008
Doubtful Cases*
Casa delle Libertà X
Partito della Libertà X
Allegedly Populist Parties
SP (NL) X X X
CD (NL) X
LPF (NL) X
Downloaded by [Lulea University of Technology] at 21:43 19 September 2013

PVV (NL) X
BNP (UK) X
UKIP (UK) X
PDS/Die Linke (GE) X X
Lega Nord (IT) X
Forza Italia (IT) X
*The Casa delle Libertà (CdL) was an alliance of right-wing parties, including the allegedly
populist parties Lega Nord and Forza Italia. We did not include this alliance in our category of
allegedly populist parties because the alliance also included many parties which have not been
associated with populism. The Partito della Libertà (PdL) is more than a mere alliance of
parties; it is an official political party. However, only Forza Italia merged into this party while
the Lega Nord did not. Therefore we put CdL and PdL in the category ‘doubtful cases’.

A PP E ND I X B
D IC T I O N A R Y O F T H E C O M P U T ER - B A S ED C O N T E N T A N A L Y SI S

NL UK GE IT
Core elit* elit* elit* elit*
consensus* consensus* konsens* consens*
ondemocratisch* undemocratic* undemokratisch* antidemocratic*
ondemokratisch*
referend* referend* referend* referend*
corrupt* corrupt* korrupt* corrot*
propagand* propagand* propagand* propagand*
politici* politici* politiker* politici*
*bedrog* *deceit* täusch* ingann*
*bedrieg* *deceiv* betrüg*
betrug*
*verraa* *betray* *verrat* tradi*
*verrad*
schaam* shame* scham* vergogn*
schäm*
schand* scandal* skandal* scandal*
waarheid* truth* wahrheit* verità
oneerlijk* dishonest* unfair* disonest*
unehrlich*
Context establishm* establishm* establishm* partitocrazia
heersend* ruling* *herrsch*
capitul*
kapitul*
kaste*
leugen* lüge* menzogn*
lieg* mentir*