Proceedings of the AAAI Conference on Artificial Intelligence
False information could be dangerous if the claim is not debunked timely. Fact-checking organisat... more False information could be dangerous if the claim is not debunked timely. Fact-checking organisations get a high volume of claims on different topics with immense velocity. The efficiency of the fact-checkers decreases due to 3V problems volume, velocity and variety. Especially during crises or elections, fact-checkers cannot handle user requests to verify the claim. Until now, no real-time curable centralised corpus of fact-checked articles is available. Also, the same claim is fact-checked by multiple fact-checking organisations with or without judgement. To fill this gap, we introduce FakeKG: A Knowledge Graph-Based approach for improving Automated Fact-checking. FakeKG is a centralised knowledge graph containing fact-checked articles from different sources that can be queried using the SPARQL endpoint. The proposed FakeKG can prescreen claim requests and filter them if the claim is already fact-checked and provide a judgement to the claim. It will also categorise the claim's...
We describe the fifth edition of the CheckThat! lab, part of the 2022 Conference and Labs of the ... more We describe the fifth edition of the CheckThat! lab, part of the 2022 Conference and Labs of the Evaluation Forum (CLEF). The lab evaluates technology supporting tasks related to factuality in multiple languages: Arabic, Bulgarian, Dutch, English, German, Spanish, and Turkish. Task 1 asks to identify relevant claims in tweets in terms of check-worthiness, verifiability, harmfullness, and attention-worthiness. Task 2 asks to detect previously fact-checked claims that could be relevant to fact-check a new claim. It targets both tweets and political debates/speeches. Task 3 asks to predict the veracity of the main claim in a news article. CheckThat! was the most popular lab at CLEF-2022 in terms of team registrations: 137 teams. More than one-third (37%) of them actually participated: 18, 7, and 26 teams submitted 210, 37, and 126 official runs for tasks 1, 2, and 3, respectively.
Metadata on the times at which text and image messages were submitted to a tipline and public gro... more Metadata on the times at which text and image messages were submitted to a tipline and public groups along with similarity/clustering data grouping messages. Please see the README file and the published paper for further details. Please cite the following publication if you use this data: Kazemi, A., Garimella, K., Shahi, G. K., Gaffney, D., & Hale, S. A. (2022). Research note: Tiplines to uncover misinformation on encrypted platforms: A case study of the 2019 Indian general election on WhatsApp. Harvard Kennedy School (HKS) Misinformation Review. https://doi.org/10.37016/mr-2020-91
The fifth edition of the CheckThat! Lab is held as part of the 2022 Conference and Labs of the Ev... more The fifth edition of the CheckThat! Lab is held as part of the 2022 Conference and Labs of the Evaluation Forum (CLEF). The lab evaluates technology supporting various factuality tasks in seven languages: Arabic, Bulgarian, Dutch, English, German, Spanish, and Turkish. Task 1 focuses on disinformation related to the ongoing COVID-19
Today, social media platforms have become almost a normal place in our lives, and we need to secu... more Today, social media platforms have become almost a normal place in our lives, and we need to secure and give users a clear picture of the apps we are creating. Furthermore, hate speech and fake news are increasingly circulating more and more on platforms despite surveillance efforts made by platform hosts. This work is designed to spot hate speech and fake news on social media platforms using information extracted from user-generated content. Relevant information provided to users will help address the research question, and a modified version of Twitter will be developed to show the importance of this system functioning on a particular data collection and later, the findings of this system will be discussed.
Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems
During the outbreak of the COVID-19 pandemic, many people shared their symptoms across Online Soc... more During the outbreak of the COVID-19 pandemic, many people shared their symptoms across Online Social Networks (OSNs) like Twitter, hoping for others' advice or moral support. Prior studies have shown that those who disclose health-related information across OSNs often tend to regret it and delete their publications afterwards. Hence, deleted posts containing sensitive data can be seen as manifestations of online regrets. In this work, we present an analysis of deleted content on Twitter during the outbreak of the COVID-19 pandemic. For this, we collected more than 3.67 million tweets describing COVID-19 symptoms (e.g., fever, cough, and fatigue) posted between January and April 2020. We observed that around 24% of the tweets containing personal pronouns were deleted either by their authors or by the platform after one year. As a practical application of the resulting dataset, we explored its suitability for the automatic classifcation of regrettable content on Twitter. CCS CONCEPTS • Security and privacy → Social aspects of security and privacy; Usability in security and privacy; • Human-centered computing → HCI design and evaluation methods.
Introduction With over a year of the COVID-19 pandemic -likely the defining global health crisis ... more Introduction With over a year of the COVID-19 pandemic -likely the defining global health crisis of our time , misinformation about symptoms, vaccines and infection rates is rife. In this study, we investigated the authors, content and propagation of this infodemic. Using data from over 92 professional fact-checking organizations united as the International Fact-Checking Network (IFCN) from January to July 2020, we analyzed 1 500 false and partially false tweets spread misinformation. Misinformed citizens may take decisions that will delay the mitigation of the crisis or might even have a direct adverse effect . The results of our exploratory work shed light on the spread and some characteristics of misinformation and typically used emojis and hashtags. Moreover, we have been able to describe the first set of recommendations that specifically address the role of authorities. In the following section, we describe the data we used, which methods we applied and which results in we found. Based on these, we conclude and propose future work. Data For our study, we gathered data from two different sources. The first data set consists of false and partially false tweets from fact-checking websites. The second is a random sample of tweets related to COVID-19 from the same period. Dataset-I We used an automated approach to retrieve tweets with misinformation. First, we collected the list of fact-checked news articles related to the COVID-19 from Snopes and Poynter from 04-01-2020 to 18-07-2020. We collected 7 623 fact-checked articles using the approach mentioned by Shahi . Dataset-II To understand how the misinformation around COVID19 is distinct from the other tweets on this topic, we created a background corpus of 163 096 English tweets spanning the same period as our corpus of misinformation. Methods We follow a two-way approach. We first analyze the details of user account involved in misinformation and the spread of misinformation (false or partially false data). Then, we analyze the content. With both, we investigate the propagation of misinformation on social media. To better understand who is spreading misinformation on Twitter, we investigated the Twitter accounts behind the tweets. First, we analyze the role of bots in spreading misinformation by using a bot detection API to classify accounts of authors automatically. Similarly, we analyze whether accounts are brands using an available classifier. Third, we investigate some characteristics of the accounts that reflect their popularity (e.g. follower count, favourites count, account age etc.). To analyze what misinformation around the topic of COVID-19 is circulating on Twitter, we investigate the content of tweets. Due to the relatively small number of partially false claims, we combined the data for these analyses. First, we analyze the most common hashtags and emojis. Second, we investigate the most distinctive terms in our data to better understand how COVID-19 misinformation differs from other COVID19 related content on Twitter.
<strong>Task Definition:</strong> Given the text of a news article, determine whether... more <strong>Task Definition:</strong> Given the text of a news article, determine whether the main claim made in the article is true, partially true, false, or other (e.g., claims in dispute) and also detect the topical domain of the article. This task will run in <strong>English.</strong> <strong>Subtask 3A:</strong> <strong>Multi-class fake news detection of news articles (English):</strong> Sub-task A would be the detection of fake news designed as a four-class classification problem. The training data will be released in batches and will be roughly about 1,000 articles with the respective label. Given the text of a news article, determine whether the main claim made in the article is true, partially true, false, or other. Our definitions for the categories are as follows: False - The main claim made in an article is untrue. Partially False - The main claim of an article is a mixture of true and false information. The article contains partially true and partially false information but cannot be considered as 100% true. It includes all articles in categories like partially false, partially true, mostly true, miscaptioned, misleading etc., as defined by different fact-checking services. True - This rating indicates that the primary elements of the main claim are demonstrably true. Other- An article that cannot be categorised as true, false, or partially false due to lack of evidence about its claims. This category includes articles in dispute and unproven articles. <strong>Subtask 3B: </strong>Fact-checkers require background expertise to identify the truthfulness of an article. The categorisation will help to automate the sampling process from a stream of data. Given the text of a news article, determine the topical domain of the article (English). This is a classification problem. The task is to categorise fake news articles into five or more different topical categories like health, election, conspiracy theory etc. This task will be offered for a subset of the data of Subtask 3A. <strong>Input Data</strong> The d [...]
The work presents a methodology to assess the problems behind static gene expression data modelli... more The work presents a methodology to assess the problems behind static gene expression data modelling and analysis with machine learning techniques. As a case study, transcriptomic data collected during a longitudinal study on the effects of diet on the expression of oxidative phosphorylation genes was used. Data were collected from 60 abdominally overweight men and women after an observation period of eight weeks, whilst they were following three different diets. Real-valued static gene expression data were encoded into spike trains using Gaussian receptive fields for multinomial classification using an evolving spiking neural network (eSNN) model. Results demonstrated that the proposed methodology can be used for predictive modelling of static gene expression data and future works are proposed regarding the application of eSNNs for personalised modelling.
There is currently no easy way to discover potentially problematic content on WhatsApp and other ... more There is currently no easy way to discover potentially problematic content on WhatsApp and other end-to-end encrypted platforms at scale. In this paper, we analyze the usefulness of a crowd-sourced tipline through which users can submit content (“tips”) that they want fact-checked. We compared the tips sent to a WhatsApp tipline run during the 2019 Indian general election with the messages circulating in large, public groups on WhatsApp and other social media platforms during the same period. We found that tiplines are a very useful lens into WhatsApp conversations: a significant fraction of messages and images sent to the tipline match with the content being shared on public WhatsApp groups and other social media. Our analysis also shows that tiplines cover the most popular content well, and a majority of such content is often shared to the tipline before appearing in large, public WhatsApp groups. Overall, our findings suggest tiplines can be an effective source for discovering po...
In the last decade, Social Media platforms such as Twitter have gained importance in the various ... more In the last decade, Social Media platforms such as Twitter have gained importance in the various marketing strategies of companies. This work aims to examine the presence of influential content on a textual level, by investigating characteristics of tweets in the context of social impact theory, and its dimension immediacy. To this end, we analysed influential Twitter communication data during Black Friday 2018 with methods from social media analytics such as sentiment analysis and degree centrality. Results show significant differences in communication style between companies and influencers. Companies published longer textual content and created more tweets with a positive sentiment and more first-person pronouns than influencers. These findings shall serve as a basis for a future experimental study to examine the impact of text presence on consumer cognition and the willingness to purchase.
Ontology is an important artifact of Semantic Web applications. Today, there are an enormous numb... more Ontology is an important artifact of Semantic Web applications. Today, there are an enormous number of ontologies available on the Web. Even so, finding and identifying the right ontology is not easy. This is because the majority of ontologies are either not described or described with a general-purpose metadata vocabulary like Dublin Core. On the other hand, ontology construction, irrespective of its types (e.g., general ontology, domain ontology, application ontology), is an expensive affair both in terms of human resources and other infrastructural resources. Hence, the ideal situation would be to reuse the existing ontologies to reduce the development effort and cost, and also to improve the quality of the original ontology. In the current work we present an ontology metadata vocabulary called Metadata for Ontology Description and publication (MOD). To design the vocabulary, we also propose a set of generic guiding principles and a well-established methodology which take into ac...
We describe the fourth edition of the CheckThat! Lab, part of the 2021 Conference and Labs of the... more We describe the fourth edition of the CheckThat! Lab, part of the 2021 Conference and Labs of the Evaluation Forum (CLEF). The lab evaluates technology supporting three tasks related to factuality, and it covers Arabic, Bulgarian, English, Spanish, and Turkish. Here, we present task 3, which focuses on multi-class fake news detection and topical domain detection of news articles. Overall, there were 88 submissions by 27 teams for Task 3A, and 49 submissions by 20 teams for task 3B (two team from Task 3A and seven teams from Task 3B are excluding from the ranking due to wrong submission file). The best performing system for task 3A achieved a macro F1-score of 0.84 and was ahead of the rest by a rather large margin. The performance of the systems for task 3B was overall higher than for task 3A with the top performing system achieving a macro F1-score of 0.88. In this paper, we describe the process of data collection and the task setup, including the evaluation measures used, and we g...
Twitter is becoming an increasingly important platform for disseminating information during crisi... more Twitter is becoming an increasingly important platform for disseminating information during crisis situations, such as the COVID-19 pandemic. Effective crisis communication on Twitter can shape the public perception of the crisis, influence adherence to preventative measures, and thus affect public health. Influential accounts are particularly important as they reach large audiences quickly. This study identifies influential German-language accounts from almost 3 million German tweets collected between January and May 2020 by constructing a retweet network and calculating PageRank centrality values. We capture the volatility of crisis communication by structuring the analysis into seven stages based on key events during the pandemic and profile influential accounts into roles. Our analysis shows that news and journalist accounts were influential throughout all phases, while government accounts were particularly important shortly before and after the lockdown was instantiated. We dis...
WhatsApp is a popular chat application used by over 2 billion users worldwide. However, due to en... more WhatsApp is a popular chat application used by over 2 billion users worldwide. However, due to end-to-end encryption, there is currently no easy way to fact-check content on WhatsApp at scale. In this paper, we analyze the usefulness of a crowd-sourced system on WhatsApp through which users can submit ‘tips’ containing messages they want factchecked. We compare the tips sent to a WhatsApp tipline run during the 2019 Indian national elections with the messages circulating in large, public groups on WhatsApp and other social media platforms during the same period. We find that tiplines are a very useful lens into WhatsApp conversations: a significant fraction of messages and images sent to the tipline match with the content being shared on public WhatsApp groups and other social media. Our analysis also shows that tiplines cover the most popular content well, and a majority of such content is often shared to the tipline before appearing in large, public WhatsApp groups. Overall, the a...
Proceedings of the AAAI Conference on Artificial Intelligence
False information could be dangerous if the claim is not debunked timely. Fact-checking organisat... more False information could be dangerous if the claim is not debunked timely. Fact-checking organisations get a high volume of claims on different topics with immense velocity. The efficiency of the fact-checkers decreases due to 3V problems volume, velocity and variety. Especially during crises or elections, fact-checkers cannot handle user requests to verify the claim. Until now, no real-time curable centralised corpus of fact-checked articles is available. Also, the same claim is fact-checked by multiple fact-checking organisations with or without judgement. To fill this gap, we introduce FakeKG: A Knowledge Graph-Based approach for improving Automated Fact-checking. FakeKG is a centralised knowledge graph containing fact-checked articles from different sources that can be queried using the SPARQL endpoint. The proposed FakeKG can prescreen claim requests and filter them if the claim is already fact-checked and provide a judgement to the claim. It will also categorise the claim's...
We describe the fifth edition of the CheckThat! lab, part of the 2022 Conference and Labs of the ... more We describe the fifth edition of the CheckThat! lab, part of the 2022 Conference and Labs of the Evaluation Forum (CLEF). The lab evaluates technology supporting tasks related to factuality in multiple languages: Arabic, Bulgarian, Dutch, English, German, Spanish, and Turkish. Task 1 asks to identify relevant claims in tweets in terms of check-worthiness, verifiability, harmfullness, and attention-worthiness. Task 2 asks to detect previously fact-checked claims that could be relevant to fact-check a new claim. It targets both tweets and political debates/speeches. Task 3 asks to predict the veracity of the main claim in a news article. CheckThat! was the most popular lab at CLEF-2022 in terms of team registrations: 137 teams. More than one-third (37%) of them actually participated: 18, 7, and 26 teams submitted 210, 37, and 126 official runs for tasks 1, 2, and 3, respectively.
Metadata on the times at which text and image messages were submitted to a tipline and public gro... more Metadata on the times at which text and image messages were submitted to a tipline and public groups along with similarity/clustering data grouping messages. Please see the README file and the published paper for further details. Please cite the following publication if you use this data: Kazemi, A., Garimella, K., Shahi, G. K., Gaffney, D., & Hale, S. A. (2022). Research note: Tiplines to uncover misinformation on encrypted platforms: A case study of the 2019 Indian general election on WhatsApp. Harvard Kennedy School (HKS) Misinformation Review. https://doi.org/10.37016/mr-2020-91
The fifth edition of the CheckThat! Lab is held as part of the 2022 Conference and Labs of the Ev... more The fifth edition of the CheckThat! Lab is held as part of the 2022 Conference and Labs of the Evaluation Forum (CLEF). The lab evaluates technology supporting various factuality tasks in seven languages: Arabic, Bulgarian, Dutch, English, German, Spanish, and Turkish. Task 1 focuses on disinformation related to the ongoing COVID-19
Today, social media platforms have become almost a normal place in our lives, and we need to secu... more Today, social media platforms have become almost a normal place in our lives, and we need to secure and give users a clear picture of the apps we are creating. Furthermore, hate speech and fake news are increasingly circulating more and more on platforms despite surveillance efforts made by platform hosts. This work is designed to spot hate speech and fake news on social media platforms using information extracted from user-generated content. Relevant information provided to users will help address the research question, and a modified version of Twitter will be developed to show the importance of this system functioning on a particular data collection and later, the findings of this system will be discussed.
Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems
During the outbreak of the COVID-19 pandemic, many people shared their symptoms across Online Soc... more During the outbreak of the COVID-19 pandemic, many people shared their symptoms across Online Social Networks (OSNs) like Twitter, hoping for others' advice or moral support. Prior studies have shown that those who disclose health-related information across OSNs often tend to regret it and delete their publications afterwards. Hence, deleted posts containing sensitive data can be seen as manifestations of online regrets. In this work, we present an analysis of deleted content on Twitter during the outbreak of the COVID-19 pandemic. For this, we collected more than 3.67 million tweets describing COVID-19 symptoms (e.g., fever, cough, and fatigue) posted between January and April 2020. We observed that around 24% of the tweets containing personal pronouns were deleted either by their authors or by the platform after one year. As a practical application of the resulting dataset, we explored its suitability for the automatic classifcation of regrettable content on Twitter. CCS CONCEPTS • Security and privacy → Social aspects of security and privacy; Usability in security and privacy; • Human-centered computing → HCI design and evaluation methods.
Introduction With over a year of the COVID-19 pandemic -likely the defining global health crisis ... more Introduction With over a year of the COVID-19 pandemic -likely the defining global health crisis of our time , misinformation about symptoms, vaccines and infection rates is rife. In this study, we investigated the authors, content and propagation of this infodemic. Using data from over 92 professional fact-checking organizations united as the International Fact-Checking Network (IFCN) from January to July 2020, we analyzed 1 500 false and partially false tweets spread misinformation. Misinformed citizens may take decisions that will delay the mitigation of the crisis or might even have a direct adverse effect . The results of our exploratory work shed light on the spread and some characteristics of misinformation and typically used emojis and hashtags. Moreover, we have been able to describe the first set of recommendations that specifically address the role of authorities. In the following section, we describe the data we used, which methods we applied and which results in we found. Based on these, we conclude and propose future work. Data For our study, we gathered data from two different sources. The first data set consists of false and partially false tweets from fact-checking websites. The second is a random sample of tweets related to COVID-19 from the same period. Dataset-I We used an automated approach to retrieve tweets with misinformation. First, we collected the list of fact-checked news articles related to the COVID-19 from Snopes and Poynter from 04-01-2020 to 18-07-2020. We collected 7 623 fact-checked articles using the approach mentioned by Shahi . Dataset-II To understand how the misinformation around COVID19 is distinct from the other tweets on this topic, we created a background corpus of 163 096 English tweets spanning the same period as our corpus of misinformation. Methods We follow a two-way approach. We first analyze the details of user account involved in misinformation and the spread of misinformation (false or partially false data). Then, we analyze the content. With both, we investigate the propagation of misinformation on social media. To better understand who is spreading misinformation on Twitter, we investigated the Twitter accounts behind the tweets. First, we analyze the role of bots in spreading misinformation by using a bot detection API to classify accounts of authors automatically. Similarly, we analyze whether accounts are brands using an available classifier. Third, we investigate some characteristics of the accounts that reflect their popularity (e.g. follower count, favourites count, account age etc.). To analyze what misinformation around the topic of COVID-19 is circulating on Twitter, we investigate the content of tweets. Due to the relatively small number of partially false claims, we combined the data for these analyses. First, we analyze the most common hashtags and emojis. Second, we investigate the most distinctive terms in our data to better understand how COVID-19 misinformation differs from other COVID19 related content on Twitter.
<strong>Task Definition:</strong> Given the text of a news article, determine whether... more <strong>Task Definition:</strong> Given the text of a news article, determine whether the main claim made in the article is true, partially true, false, or other (e.g., claims in dispute) and also detect the topical domain of the article. This task will run in <strong>English.</strong> <strong>Subtask 3A:</strong> <strong>Multi-class fake news detection of news articles (English):</strong> Sub-task A would be the detection of fake news designed as a four-class classification problem. The training data will be released in batches and will be roughly about 1,000 articles with the respective label. Given the text of a news article, determine whether the main claim made in the article is true, partially true, false, or other. Our definitions for the categories are as follows: False - The main claim made in an article is untrue. Partially False - The main claim of an article is a mixture of true and false information. The article contains partially true and partially false information but cannot be considered as 100% true. It includes all articles in categories like partially false, partially true, mostly true, miscaptioned, misleading etc., as defined by different fact-checking services. True - This rating indicates that the primary elements of the main claim are demonstrably true. Other- An article that cannot be categorised as true, false, or partially false due to lack of evidence about its claims. This category includes articles in dispute and unproven articles. <strong>Subtask 3B: </strong>Fact-checkers require background expertise to identify the truthfulness of an article. The categorisation will help to automate the sampling process from a stream of data. Given the text of a news article, determine the topical domain of the article (English). This is a classification problem. The task is to categorise fake news articles into five or more different topical categories like health, election, conspiracy theory etc. This task will be offered for a subset of the data of Subtask 3A. <strong>Input Data</strong> The d [...]
The work presents a methodology to assess the problems behind static gene expression data modelli... more The work presents a methodology to assess the problems behind static gene expression data modelling and analysis with machine learning techniques. As a case study, transcriptomic data collected during a longitudinal study on the effects of diet on the expression of oxidative phosphorylation genes was used. Data were collected from 60 abdominally overweight men and women after an observation period of eight weeks, whilst they were following three different diets. Real-valued static gene expression data were encoded into spike trains using Gaussian receptive fields for multinomial classification using an evolving spiking neural network (eSNN) model. Results demonstrated that the proposed methodology can be used for predictive modelling of static gene expression data and future works are proposed regarding the application of eSNNs for personalised modelling.
There is currently no easy way to discover potentially problematic content on WhatsApp and other ... more There is currently no easy way to discover potentially problematic content on WhatsApp and other end-to-end encrypted platforms at scale. In this paper, we analyze the usefulness of a crowd-sourced tipline through which users can submit content (“tips”) that they want fact-checked. We compared the tips sent to a WhatsApp tipline run during the 2019 Indian general election with the messages circulating in large, public groups on WhatsApp and other social media platforms during the same period. We found that tiplines are a very useful lens into WhatsApp conversations: a significant fraction of messages and images sent to the tipline match with the content being shared on public WhatsApp groups and other social media. Our analysis also shows that tiplines cover the most popular content well, and a majority of such content is often shared to the tipline before appearing in large, public WhatsApp groups. Overall, our findings suggest tiplines can be an effective source for discovering po...
In the last decade, Social Media platforms such as Twitter have gained importance in the various ... more In the last decade, Social Media platforms such as Twitter have gained importance in the various marketing strategies of companies. This work aims to examine the presence of influential content on a textual level, by investigating characteristics of tweets in the context of social impact theory, and its dimension immediacy. To this end, we analysed influential Twitter communication data during Black Friday 2018 with methods from social media analytics such as sentiment analysis and degree centrality. Results show significant differences in communication style between companies and influencers. Companies published longer textual content and created more tweets with a positive sentiment and more first-person pronouns than influencers. These findings shall serve as a basis for a future experimental study to examine the impact of text presence on consumer cognition and the willingness to purchase.
Ontology is an important artifact of Semantic Web applications. Today, there are an enormous numb... more Ontology is an important artifact of Semantic Web applications. Today, there are an enormous number of ontologies available on the Web. Even so, finding and identifying the right ontology is not easy. This is because the majority of ontologies are either not described or described with a general-purpose metadata vocabulary like Dublin Core. On the other hand, ontology construction, irrespective of its types (e.g., general ontology, domain ontology, application ontology), is an expensive affair both in terms of human resources and other infrastructural resources. Hence, the ideal situation would be to reuse the existing ontologies to reduce the development effort and cost, and also to improve the quality of the original ontology. In the current work we present an ontology metadata vocabulary called Metadata for Ontology Description and publication (MOD). To design the vocabulary, we also propose a set of generic guiding principles and a well-established methodology which take into ac...
We describe the fourth edition of the CheckThat! Lab, part of the 2021 Conference and Labs of the... more We describe the fourth edition of the CheckThat! Lab, part of the 2021 Conference and Labs of the Evaluation Forum (CLEF). The lab evaluates technology supporting three tasks related to factuality, and it covers Arabic, Bulgarian, English, Spanish, and Turkish. Here, we present task 3, which focuses on multi-class fake news detection and topical domain detection of news articles. Overall, there were 88 submissions by 27 teams for Task 3A, and 49 submissions by 20 teams for task 3B (two team from Task 3A and seven teams from Task 3B are excluding from the ranking due to wrong submission file). The best performing system for task 3A achieved a macro F1-score of 0.84 and was ahead of the rest by a rather large margin. The performance of the systems for task 3B was overall higher than for task 3A with the top performing system achieving a macro F1-score of 0.88. In this paper, we describe the process of data collection and the task setup, including the evaluation measures used, and we g...
Twitter is becoming an increasingly important platform for disseminating information during crisi... more Twitter is becoming an increasingly important platform for disseminating information during crisis situations, such as the COVID-19 pandemic. Effective crisis communication on Twitter can shape the public perception of the crisis, influence adherence to preventative measures, and thus affect public health. Influential accounts are particularly important as they reach large audiences quickly. This study identifies influential German-language accounts from almost 3 million German tweets collected between January and May 2020 by constructing a retweet network and calculating PageRank centrality values. We capture the volatility of crisis communication by structuring the analysis into seven stages based on key events during the pandemic and profile influential accounts into roles. Our analysis shows that news and journalist accounts were influential throughout all phases, while government accounts were particularly important shortly before and after the lockdown was instantiated. We dis...
WhatsApp is a popular chat application used by over 2 billion users worldwide. However, due to en... more WhatsApp is a popular chat application used by over 2 billion users worldwide. However, due to end-to-end encryption, there is currently no easy way to fact-check content on WhatsApp at scale. In this paper, we analyze the usefulness of a crowd-sourced system on WhatsApp through which users can submit ‘tips’ containing messages they want factchecked. We compare the tips sent to a WhatsApp tipline run during the 2019 Indian national elections with the messages circulating in large, public groups on WhatsApp and other social media platforms during the same period. We find that tiplines are a very useful lens into WhatsApp conversations: a significant fraction of messages and images sent to the tipline match with the content being shared on public WhatsApp groups and other social media. Our analysis also shows that tiplines cover the most popular content well, and a majority of such content is often shared to the tipline before appearing in large, public WhatsApp groups. Overall, the a...
Uploads
Papers by Gautam Shahi