Objectives: Xylazine is an α 2-agonist increasingly prevalent in the illicit drug supply. Our obj... more Objectives: Xylazine is an α 2-agonist increasingly prevalent in the illicit drug supply. Our objectives were to curate information about xylazine through social media from people who use drugs (PWUDs). Specifically, we sought to answer the following: (1) What are the demographics of Reddit subscribers reporting exposure to xylazine? (2) Is xylazine a desired additive? And (3) what adverse effects of xylazine are PWUDs experiencing? Methods: Natural language processing (NLP) was used to identify mentions of "xylazine" from posts by Reddit subscribers who also posted on drug-related subreddits. Posts were qualitatively evaluated for xylazine-related themes. A survey was developed to gather additional information about the Reddit subscribers. This survey was posted on subreddits that were identified by NLP to contain xylazine-related discussions from March 2022 to October 2022. Results: Seventy-six posts were extracted via NLP from 765,616 posts by 16,131 Reddit subscribers (January 2018 to August 2021). People on Reddit described xylazine as an unwanted adulterant in their opioid supply. Sixty-one participants completed the survey. Of those who disclosed their location, 25 of 50 participants (50%) reported locations in the Northeastern United States. The most common route of xylazine use was intranasal use (57%). Thirty-one of 59 (53%) reported experiencing xylazine withdrawal. Frequent adverse events reported were prolonged sedation (81%) and increased skin wounds (43%). Conclusions: Among respondents on these Reddit forums, xylazine seems to be an unwanted adulterant. People who use drugs may be experiencing adverse effects such as prolonged sedation and xylazine withdrawal. This seemed to be more common in the Northeast.
Introduction: Reddit hosts a large active community of members dedicated to the discussion of can... more Introduction: Reddit hosts a large active community of members dedicated to the discussion of cannabinoid hyperemesis syndrome. We sought to describe common themes discussed and the most frequently mentioned triggers and therapies for cannabinoid hyperemesis syndrome exacerbations in the Reddit online community. Methods: Data collected from six subreddits were filtered using natural language processing to curate posts referencing cannabinoid hyperemesis syndrome. Based on a manual review of posts, common themes were identified. A machine learning model was trained using the manually categorized data to automatically classify the themes for the rest of the posts so that their distributions could be quantified. Results: From August 2018 to November 2022, 2683 unique posts were collected. Thematic analysis resulted in five overall themes: cannabinoid hyperemesis syndrome-related science; symptom timing; cannabinoid hyperemesis syndrome treatment and prevention; cannabinoid hyperemesis syndrome diagnosis and education; and health impacts. Additionally, 447 trigger and 664 therapy-related posts were identified. The most commonly mentioned triggers for cannabinoid hyperemesis syndrome episodes included: food and drink (n ¼ 62), cannabinoids (n ¼ 45), mental health (e.g., stress, anxiety) (n ¼ 27), and alcohol (n ¼ 22). Most commonly mentioned cannabinoid hyperemesis syndrome therapies included: hot water/bathing (n ¼ 62), hydration (n ¼ 60), antiemetics (n ¼ 42), food and drink (n ¼ 38), gastrointestinal medications (n ¼ 38), behavioral therapies (e.g., meditation, yoga) (n ¼ 35), and capsaicin (n ¼ 29). Discussion: Reddit posts for cannabinoid hyperemesis syndrome provide a valuable source of community discussion and individual reports of people experiencing cannabinoid hyperemesis syndrome. Mental health and alcohol were frequently reported triggers within the posts but are not often identified in the literature. While many of the therapies mentioned are well documented, behavioral responses such as meditation and yoga have not been explored by the scientific literature. Conclusions: Knowledge shared via online social media platforms contains detailed information on self-reported cannabinoid hyperemesis syndrome disease and management experiences, which could serve as valuable data for the development of treatment strategies. Further longitudinal studies in patients with cannabinoid hyperemesis syndrome are needed to corroborate these findings.
Pretrained language models (PLMs) on domainspecific data have been proven to be effective for in-... more Pretrained language models (PLMs) on domainspecific data have been proven to be effective for in-domain natural language processing (NLP) tasks. Our work aimed to develop a language model which can be effective for the NLP tasks with the data from diverse social media platforms. We pretrained a language model on Twitter and Reddit posts in English consisting of 929M sequence blocks for 112K steps. We benchmarked our model and 3 transformer-based models-BERT, BERTweet, and RoBERTa on 40 social media text classification tasks. The results showed that although our model did not perform the best on all of the tasks, it outperformed the baseline model-BERT on most of the tasks, which illustrates the effectiveness of our model. Also, our work provides some insights of how to improve the efficiency of training PLMs.
Background: Few-shot learning (FSL) is a class of machine learning methods that require small num... more Background: Few-shot learning (FSL) is a class of machine learning methods that require small numbers of labeled instances for training. With many medical topics having limited annotated text-based data in practical settings, FSL-based natural language processing (NLP) holds substantial promise. We aimed to conduct a review to explore the current state of FSL methods for medical NLP. Methods: We searched for articles published between January 2016 and October 2022 using PubMed/Medline, Embase, ACL Anthology, and IEEE Xplore Digital Library. We also searched the preprint servers (e.g., arXiv, medRxiv, and bioRxiv) via Google Scholar to identify the latest relevant methods. We included all articles that involved FSL and any form of medical text. We abstracted articles based on the data source, target task, training set size, primary method(s)/approach(es), and evaluation metric(s). Results: Fifty-one articles met our inclusion criteria-all published after 2018, and most since 2020 (42/51; 82%). Concept extraction/named entity recognition was the most frequently addressed task (21/51; 41%), followed by text classification (16/51; 31%). Thirty-two (61%) articles reconstructed existing datasets to fit few-shot scenarios, and MIMIC-III was the most frequently used dataset (10/51; 20%). 77% of the articles attempted to incorporate prior knowledge to augment the small datasets available for training. Common methods included FSL with attention mechanisms (20/51; 39%), prototypical networks (11/51; 22%), metalearning (7/51; 14%), and prompt-based learning methods, the latter being particularly popular since 2021. Benchmarking experiments demonstrated relative underperformance of FSL methods on biomedical NLP tasks. Conclusion: Despite the potential for FSL in biomedical NLP, progress has been limited. This may be attributed to the rarity of specialized data, lack of standardized evaluation criteria, and the underperformance of FSL methods on biomedical topics. The creation of publicly-available specialized datasets for biomedical FSL may aid method development by facilitating comparative analyses.
Background: Due to the high burden of chronic pain, and the detrimental public health consequence... more Background: Due to the high burden of chronic pain, and the detrimental public health consequences of its treatment with opioids, there is a high-priority need to identify effective alternative therapies. Social media is a potentially valuable resource for knowledge about self-reported therapies by chronic pain sufferers. Methods: We attempted to (a) verify the presence of large-scale chronic pain-related chatter on Twitter, (b) develop natural language processing and machine learning methods for automatically detecting self-disclosures, (c) collect longitudinal data posted by them, and (d) semiautomatically analyze the types of chronic pain-related information reported by them. We collected data using chronic painrelated hashtags and keywords and manually annotated 4,998 posts to indicate if they were self-reports of chronic pain experiences. We trained and evaluated several state-of-the-art supervised text classification models and deployed the best-performing classifier. We collected all publicly available posts from detected cohort members and conducted manual and natural language processing-driven descriptive analyses. Results: Interannotator agreement for the binary annotation was 0.82 (Cohen's kappa). The RoBERTa model performed best (F 1 score: 0.84; 95% confidence interval: 0.80 to 0.89), and we used this model to classify all collected unlabeled posts. We discovered 22,795 self-reported chronic pain sufferers and collected over 3 million of their past posts. Further analyses revealed information about, but not limited to, alternative treatments, patient sentiments about treatments, side effects, and self-management strategies. Conclusion: Our social media based approach will result in an automatically growing large cohort over time, and the data can be leveraged to identify effective opioid-alternative therapies for diverse chronic pain types.
Nonmedical substance use surveillance via social media has the potential to provide low-cost and ... more Nonmedical substance use surveillance via social media has the potential to provide low-cost and more timely insights than traditional approaches. However, current social media-based approaches lack the capability to provide fine-grained, subpopulation-level statistics. We attempted to fill the gap by developing natural language processing methods to estimate the demographic distribution (gender, age, and race) of a large cohort (N = 288,562) of people who reported nonmedical prescription medication use on Twitter. Automatically derived distributions for opioids, stimulants, and tranquilizers were largely consistent, often with very strong correlations, with statistics reported in traditional sources such as the National Survey on Drug Use and Health. Our work represents an important stride in establishing social media as a complementary resource for substance use surveillance.
Intimate partner violence (IPV) is a preventable public health problem that affects millions of p... more Intimate partner violence (IPV) is a preventable public health problem that affects millions of people worldwide. Approximately one in four women are estimated to be or have been victims of severe violence at some point in their lives, irrespective of age, ethnicity, and economic status. Victims often report IPV experiences on social media, and automatic detection of such reports via machine learning may enable improved surveillance and targeted distribution of support and/or interventions for those in need. However, no artificial intelligence systems for automatic detection currently exists, and we attempted to address this research gap. We collected posts from Twitter using a list of IPV-related keywords, manually reviewed subsets of retrieved posts, and prepared annotation guidelines to categorize tweets into IPV-report or non-IPV-report. We annotated 6,348 tweets in total, with the inter-annotator agreement (IAA) of 0.86 (Cohen's kappa) among 1,834 double-annotated tweets. The class distribution in the annotated dataset was highly imbalanced, with only 668 posts (~11%) labeled as IPVreport. We then developed an effective natural language processing model to identify IPV-reporting tweets automatically. The developed model achieved classification F 1-scores of 0.76 for the IPV-report class and 0.97 for the non-IPV-report class. We conducted post-classification analyses to determine the causes of system errors and to ensure that the system did not exhibit biases in its decision making, particularly with respect to race and gender. Our automatic model can be an essential component for a proactive social media-based intervention and support framework, while also aiding population-level surveillance and large-scale cohort studies.
Americans bear a high chronic stress burden, particularly during the COVID-19 pandemic. Although ... more Americans bear a high chronic stress burden, particularly during the COVID-19 pandemic. Although social media have many strengths to complement the weaknesses of conventional stress measures, including surveys, they have been rarely utilized to detect individuals self-reporting chronic stress. Thus, this study aimed to develop and evaluate an automatic system on Twitter to identify users who have self-reported chronic stress experiences. Using the Twitter public streaming application programming interface, we collected tweets containing certain stress-related keywords (eg, "chronic," "constant," "stress") and then filtered the data using pre-defined text patterns. We manually annotated tweets with (without) self-report of chronic stress as positive (negative). We trained multiple classifiers and tested them via accuracy and F 1 score. We annotated 4195 tweets (1560 positives, 2635 negatives), achieving an inter-annotator agreement of 0.83 (Cohen's kappa). The classifier based on Bidirectional Encoder Representation from Transformers performed the best (accuracy of 83.6% [81.0-86.1]), outperforming the second best-performing classifier (support vector machines: 76.4% [73.5-79.3]). The past tweets from the authors of positive tweets contained useful information, including sources and health impacts of chronic stress. Our study demonstrates that users' self-reported chronic stress experiences can be automatically identified on Twitter, which has a high potential for surveillance and large-scale intervention.
Few-shot learning (FSL) is a category of machine learning models that are designed with the inten... more Few-shot learning (FSL) is a category of machine learning models that are designed with the intent of solving problems that have small amounts of labeled data available for training. FSL research progress in natural language processing (NLP), particularly within the medical domain, has been notably slow, primarily due to greater difficulties posed by domain-specific characteristics and data sparsity problems. We explored the use of novel methods for text representation and encoding combined with distance-based measures for improving FSL entity detection. In this paper, we propose a data augmentation method to incorporate semantic information from medical texts into the learning process and combine it with a nearest-neighbor classification strategy for predicting entities. Experiments performed on five biomedical text datasets demonstrate that our proposed approach often outperforms other approaches.
Introduction: Decades of research have established the association between adverse childhood expe... more Introduction: Decades of research have established the association between adverse childhood experiences (ACEs) and adult onset of chronic diseases, influenced by health behaviors and social determinants of health (SDoH). Machine Learning (ML) is a powerful tool for computing these complex associations and accurately predicting chronic health conditions. Methods: Using the 2021 Behavioral Risk Factor Surveillance Survey, we developed several ML models-random forest, logistic regression, support vector machine, Naïve Bayes, and K-Nearest Neighbor-over data from a sample of 52,268 respondents. We predicted 13 chronic health conditions based on ACE history, health behaviors, SDoH, and demographics. We further assessed each variable's importance in outcome prediction for model interpretability. We evaluated model performance via the Area Under the Curve (AUC) score. Results: With the inclusion of data on ACEs, our models outperformed or demonstrated similar accuracies to existing models in the literature that used SDoH to predict health outcomes. The most accurate models predicted diabetes, pulmonary diseases, and heart attacks. The random forest model was the most effective for diabetes (AUC = 0.784) and heart attacks (AUC = 0.732), and the logistic regression model most accurately predicted pulmonary diseases (AUC = 0.753). The strongest predictors across models were age, ever monitored blood sugar or blood pressure, count of the monitoring behaviors for blood sugar or blood pressure, BMI, time of last cholesterol check, employment status, income, count of vaccines received, health insurance status, and total ACEs. A cumulative measure of ACEs was a stronger predictor than individual ACEs. Discussion: Our models can provide an interpretable, trauma-informed framework to identify and intervene with at-risk individuals early to prevent chronic health conditions and address their inequalities in the U.S.
While social media has evolved into a useful resource for studying medication-related information... more While social media has evolved into a useful resource for studying medication-related information, observational studies of medications have continued to rely on other sources of data. Towards advancing the use of social media data for medication-related observational studies, we analyze an annotated corpus of 27,941 tweets designed for training machine learning algorithms to automatically detect users' medication intake. In particular, we assess how a baseline classifier trained on the general corpus-that is, on various types of medication-performs for specific types. For most types, the classifier performs significantly better than it does overall; however, for nervous system medications, it performs significantly worse. These results suggest that, while the general corpus may have utility for observational studies focusing on most types of medication, studying nervous system medications may benefit from training a classifier exclusively for this type. We will explore this data-level approach in future work.
Journal of the American Medical Informatics Association, Apr 11, 2019
We read with great interest the article by Cocos et al. 1 In it, the authors use one of the datas... more We read with great interest the article by Cocos et al. 1 In it, the authors use one of the datasets made public by our lab in parallel with a publication in Journal of the American Medical Informatics Association, 2 referred to by them as the Twitter ADR Dataset (v1.0) (henceforth the ADRMine Dataset). Cocos et al use state-of-the-art recurrent neural network (RNN) models for extracting adverse drug reaction (ADR) mentions in Twitter posts. We commend the authors for their clear description of the workings of neural models, and on their experiments on the use of fixed versus trainable embeddings, which can be very valuable to the natural language processing (NLP) research community. We believe that using deep learning models offer greater opportunities for mining ADR posts on social media. However, there are key choices made by the authors that require clarification to avoid a misunderstanding on the impact of their findings. In a nutshell, because the authors did not use the ADRMine Dataset in its entirety, discarding upfront all tweets with no human annotations (ie, those that do not contain any ADRs), the resulting train and test sets are biased toward the positive class. Thus, the performance measures reported for the task in Cocos et al are not comparable to those reported in Nikfarjam et al, 2 contrary to what the manuscript reports. After discarding tweets with no human annotation from the ADRMine Dataset, the authors downloaded available tweets from Twitter, and added a small set (203 tweets) to form the dataset used for their experiments. While downloading from Twitter results in an almost unavoidable reduction in the dataset size-as not all tweets are available as time goes by-it would not generally affect the class balance. The elimination of the tweets with no human annotations from the ADRMine Dataset, however, is a choice that is not discussed by Cocos et al, even though it severely impacts the
Social media has evolved into a crucial resource for obtaining large volumes of real-time informa... more Social media has evolved into a crucial resource for obtaining large volumes of real-time information. The promise of social media has been realized by the public health domain, and recent research has addressed some important challenges in that domain by utilizing social media data. Tasks such as monitoring flu trends, viral disease outbreaks, medication abuse, and adverse drug reactions are some examples of studies where data from social media have been exploited. The focus of this workshop is to explore solutions to three important natural language processing challenges for domain-specific social media text: (i) text classification, (ii) information extraction, and (iii) concept normalization. To explore different approaches to solving these problems on social media data, we designed a shared task which was open to participants globally. We designed three tasks using our in-house annotated Twitter data on adverse drug reactions. Task 1 involved automatic classification of adverse drug reaction assertive user posts; Task 2 focused on extracting specific adverse drug reaction mentions from user posts; and Task 3, which was slightly ill-defined due to the complex nature of the problem, involved normalizing user mentions of adverse drug reactions to standardized concept IDs. A total of 11 teams participated, and a total of 24 (18 for Task 1, and 6 for Task 2) system runs were submitted. Following the evaluation of the systems, and an assessment of their innovation/novelty, we accepted 7 descriptive manuscripts for publication-5 for Task 1 and 2 for Task 2. We provide descriptions of the tasks, data, and participating systems in this paper.
Introduction Adverse drug reactions (ADRs) are associated with significant health-related and fin... more Introduction Adverse drug reactions (ADRs) are associated with significant health-related and financial burden, and multiple sources are currently utilized to actively discover them. Social media has been proposed as a potential resource for monitoring ADRs, but drug-specific analytical studies comparing social media with other sources are scarce. Objectives Our objective was to develop methods to compare ADRs mentioned in social media with those in traditional sources: the US FDA Adverse Event Reporting System (FAERS), drug information databases (DIDs), and systematic reviews. Methods A total of 10,188 tweets mentioning adalimumab collected between June 2014 and August 2016 were included. ADRs in the corpus were extracted semi-automatically and manually mapped to standardized concepts in the Unified Medical Language System. ADRs were grouped into 16 biologic categories for comparisons. Frequencies, relative frequencies, disproportionality analyses, and rank ordering were used as metrics. Results There was moderate agreement between ADRs in social media and traditional sources. "Local and injection site reactions" was the top ADR in Twitter, DIDs, and systematic reviews by frequency, ranked frequency, and index ranking. The next highest ADR in Twitter-fatigue-ranked fifth and seventh in FAERS and DIDs. Conclusion Social media posts often express mild and symptomatic ADRs, but rates are measured differently in scientific sources. ADRs in FAERS are reported as absolute numbers, in DIDs as percentages, and in systematic reviews as percentages, risk ratios, or other metrics, which makes comparisons challenging; however, overlap is substantial. Social media analysis facilitates open-ended investigation of patient perspectives and may reveal concepts (e.g. anxiety) not available in traditional sources.
Free text data from social media is now widely used in natural language processing research, and ... more Free text data from social media is now widely used in natural language processing research, and one of the most common machine learning tasks performed on this data is classification. Generally speaking, performances of supervised classification algorithms on social media datasets are lower than those on texts from other sources, but recentlyproposed transformer-based models have considerably improved upon legacy state-of-theart systems. Currently, there is no study that compares the performances of different variants of transformer-based models on a wide range of social media text classification datasets. In this paper, we benchmark the performances of transformer-based pre-trained models on 25 social media text classification datasets, 6 of which are health-related. We compare three pre-trained language models, RoBERTa-base, BERTweet and Clinical-BioBERT in terms of classification accuracy. Our experiments show that RoBERTa-base and BERTweet perform comparably on most datasets, and considerably better than Clinical-BioBERT, even on health-related datasets.
Social networks, such as Twitter, have become important sources for active monitoring of user-rep... more Social networks, such as Twitter, have become important sources for active monitoring of user-reported adverse drug reactions (ADRs). Automatic extraction of ADR information can be crucial for healthcare providers, drug manufacturers, and consumers. However, because of the non-standard nature of social media language, automatically extracted ADR mentions need to be mapped to standard forms before they can be used by operational pharmacovigilance systems. We propose a modular natural language processing pipeline for mapping (normalizing) colloquial mentions of ADRs to their corresponding standardized identifiers. We seek to accomplish this task and enable customization of the pipeline so that distinct unlabeled free text resources can be incorporated to use the system for other normalization tasks. Our approach, which we call Hybrid Semantic Analysis (HSA), sequentially employs rule-based and semantic matching algorithms for mapping user-generated mentions to concept IDs in the Unified Medical Language System vocabulary. The semantic matching component of HSA is adaptive in nature and uses a regression model to combine various measures of semantic relatedness and resources to optimize normalization performance on the selected data source. On a publicly available corpus, our normalization method achieves 0.502 recall and 0.823 precision (F-measure: 0.624). Our proposed method outperforms a baseline based on latent semantic analysis and another that uses MetaMap.
We present our supervised sentiment classification system which competed in SemEval-2016 Task 4: ... more We present our supervised sentiment classification system which competed in SemEval-2016 Task 4: Sentiment Analysis in Twitter. Our system employs a Support Vector Machine (SVM) classifier trained using a number of features including n-grams, synset expansions, various sentiment scores, word clusters, and term centroids. Using weighted SVMs, to address the issue of class imbalance, our system obtains positive class F-scores of 0.694 and 0.650, and negative class F-scores of 0.391 and 0.493 over the training and test sets, respectively.
Journal of the American Medical Informatics Association, Sep 27, 2019
Objective: Twitter posts are now recognized as an important source of patient-generated data, pro... more Objective: Twitter posts are now recognized as an important source of patient-generated data, providing unique insights into population health. A fundamental step toward incorporating Twitter data in pharmacoepidemiologic research is to automatically recognize medication mentions in tweets. Given that lexical searches for medication names suffer from low recall due to misspellings or ambiguity with common words, we propose a more advanced method to recognize them. Materials and Methods: We present Kusuri, an Ensemble Learning classifier able to identify tweets mentioning drug products and dietary supplements. Kusuri (, "medication" in Japanese) is composed of 2 modules: first, 4 different classifiers (lexicon based, spelling variant based, pattern based, and a weakly trained neural network) are applied in parallel to discover tweets potentially containing medication names; second, an ensemble of deep neural networks encoding morphological, semantic, and long-range dependencies of important words in the tweets makes the final decision. Results: On a class-balanced (50-50) corpus of 15 005 tweets, Kusuri demonstrated performances close to human annotators with an F 1 score of 93.7%, the best score achieved thus far on this corpus. On a corpus made of all tweets posted by 112 Twitter users (98 959 tweets, with only 0.26% mentioning medications), Kusuri obtained an F 1 score of 78.8%. To the best of our knowledge, Kusuri is the first system to achieve this score on such an extremely imbalanced dataset. Conclusions: The system identifies tweets mentioning drug names with performance high enough to ensure its usefulness, and is ready to be integrated in pharmacovigilance, toxicovigilance, or more generally, public health pipelines that depend on medication name mentions.
Evidence based medicine (EBM) urges the medical doctor to incorporate the latest available clinic... more Evidence based medicine (EBM) urges the medical doctor to incorporate the latest available clinical evidence at point of care. A major stumbling block in the practice of EBM is the difficulty to keep up to date with the clinical advances. In this paper we describe a corpus designed for the development and testing of text processing tools for EBM, in particular for tasks related to the extraction and summarisation of answers and corresponding evidence related to a clinical query. The corpus is based on material from the Clinical Inquiries section of The Journal of Family Practice. It was gathered and annotated by a combination of automated information extraction, crowdsourcing tasks, and manual annotation. It has been used for the original summarisation task for which it was designed, as well as for other related tasks such as the appraisal of clinical evidence and the clustering of the results. The corpus is available at SourceForge (http://sourceforge.net/projects/ ebmsumcorpus/).
Objectives: Xylazine is an α 2-agonist increasingly prevalent in the illicit drug supply. Our obj... more Objectives: Xylazine is an α 2-agonist increasingly prevalent in the illicit drug supply. Our objectives were to curate information about xylazine through social media from people who use drugs (PWUDs). Specifically, we sought to answer the following: (1) What are the demographics of Reddit subscribers reporting exposure to xylazine? (2) Is xylazine a desired additive? And (3) what adverse effects of xylazine are PWUDs experiencing? Methods: Natural language processing (NLP) was used to identify mentions of "xylazine" from posts by Reddit subscribers who also posted on drug-related subreddits. Posts were qualitatively evaluated for xylazine-related themes. A survey was developed to gather additional information about the Reddit subscribers. This survey was posted on subreddits that were identified by NLP to contain xylazine-related discussions from March 2022 to October 2022. Results: Seventy-six posts were extracted via NLP from 765,616 posts by 16,131 Reddit subscribers (January 2018 to August 2021). People on Reddit described xylazine as an unwanted adulterant in their opioid supply. Sixty-one participants completed the survey. Of those who disclosed their location, 25 of 50 participants (50%) reported locations in the Northeastern United States. The most common route of xylazine use was intranasal use (57%). Thirty-one of 59 (53%) reported experiencing xylazine withdrawal. Frequent adverse events reported were prolonged sedation (81%) and increased skin wounds (43%). Conclusions: Among respondents on these Reddit forums, xylazine seems to be an unwanted adulterant. People who use drugs may be experiencing adverse effects such as prolonged sedation and xylazine withdrawal. This seemed to be more common in the Northeast.
Introduction: Reddit hosts a large active community of members dedicated to the discussion of can... more Introduction: Reddit hosts a large active community of members dedicated to the discussion of cannabinoid hyperemesis syndrome. We sought to describe common themes discussed and the most frequently mentioned triggers and therapies for cannabinoid hyperemesis syndrome exacerbations in the Reddit online community. Methods: Data collected from six subreddits were filtered using natural language processing to curate posts referencing cannabinoid hyperemesis syndrome. Based on a manual review of posts, common themes were identified. A machine learning model was trained using the manually categorized data to automatically classify the themes for the rest of the posts so that their distributions could be quantified. Results: From August 2018 to November 2022, 2683 unique posts were collected. Thematic analysis resulted in five overall themes: cannabinoid hyperemesis syndrome-related science; symptom timing; cannabinoid hyperemesis syndrome treatment and prevention; cannabinoid hyperemesis syndrome diagnosis and education; and health impacts. Additionally, 447 trigger and 664 therapy-related posts were identified. The most commonly mentioned triggers for cannabinoid hyperemesis syndrome episodes included: food and drink (n ¼ 62), cannabinoids (n ¼ 45), mental health (e.g., stress, anxiety) (n ¼ 27), and alcohol (n ¼ 22). Most commonly mentioned cannabinoid hyperemesis syndrome therapies included: hot water/bathing (n ¼ 62), hydration (n ¼ 60), antiemetics (n ¼ 42), food and drink (n ¼ 38), gastrointestinal medications (n ¼ 38), behavioral therapies (e.g., meditation, yoga) (n ¼ 35), and capsaicin (n ¼ 29). Discussion: Reddit posts for cannabinoid hyperemesis syndrome provide a valuable source of community discussion and individual reports of people experiencing cannabinoid hyperemesis syndrome. Mental health and alcohol were frequently reported triggers within the posts but are not often identified in the literature. While many of the therapies mentioned are well documented, behavioral responses such as meditation and yoga have not been explored by the scientific literature. Conclusions: Knowledge shared via online social media platforms contains detailed information on self-reported cannabinoid hyperemesis syndrome disease and management experiences, which could serve as valuable data for the development of treatment strategies. Further longitudinal studies in patients with cannabinoid hyperemesis syndrome are needed to corroborate these findings.
Pretrained language models (PLMs) on domainspecific data have been proven to be effective for in-... more Pretrained language models (PLMs) on domainspecific data have been proven to be effective for in-domain natural language processing (NLP) tasks. Our work aimed to develop a language model which can be effective for the NLP tasks with the data from diverse social media platforms. We pretrained a language model on Twitter and Reddit posts in English consisting of 929M sequence blocks for 112K steps. We benchmarked our model and 3 transformer-based models-BERT, BERTweet, and RoBERTa on 40 social media text classification tasks. The results showed that although our model did not perform the best on all of the tasks, it outperformed the baseline model-BERT on most of the tasks, which illustrates the effectiveness of our model. Also, our work provides some insights of how to improve the efficiency of training PLMs.
Background: Few-shot learning (FSL) is a class of machine learning methods that require small num... more Background: Few-shot learning (FSL) is a class of machine learning methods that require small numbers of labeled instances for training. With many medical topics having limited annotated text-based data in practical settings, FSL-based natural language processing (NLP) holds substantial promise. We aimed to conduct a review to explore the current state of FSL methods for medical NLP. Methods: We searched for articles published between January 2016 and October 2022 using PubMed/Medline, Embase, ACL Anthology, and IEEE Xplore Digital Library. We also searched the preprint servers (e.g., arXiv, medRxiv, and bioRxiv) via Google Scholar to identify the latest relevant methods. We included all articles that involved FSL and any form of medical text. We abstracted articles based on the data source, target task, training set size, primary method(s)/approach(es), and evaluation metric(s). Results: Fifty-one articles met our inclusion criteria-all published after 2018, and most since 2020 (42/51; 82%). Concept extraction/named entity recognition was the most frequently addressed task (21/51; 41%), followed by text classification (16/51; 31%). Thirty-two (61%) articles reconstructed existing datasets to fit few-shot scenarios, and MIMIC-III was the most frequently used dataset (10/51; 20%). 77% of the articles attempted to incorporate prior knowledge to augment the small datasets available for training. Common methods included FSL with attention mechanisms (20/51; 39%), prototypical networks (11/51; 22%), metalearning (7/51; 14%), and prompt-based learning methods, the latter being particularly popular since 2021. Benchmarking experiments demonstrated relative underperformance of FSL methods on biomedical NLP tasks. Conclusion: Despite the potential for FSL in biomedical NLP, progress has been limited. This may be attributed to the rarity of specialized data, lack of standardized evaluation criteria, and the underperformance of FSL methods on biomedical topics. The creation of publicly-available specialized datasets for biomedical FSL may aid method development by facilitating comparative analyses.
Background: Due to the high burden of chronic pain, and the detrimental public health consequence... more Background: Due to the high burden of chronic pain, and the detrimental public health consequences of its treatment with opioids, there is a high-priority need to identify effective alternative therapies. Social media is a potentially valuable resource for knowledge about self-reported therapies by chronic pain sufferers. Methods: We attempted to (a) verify the presence of large-scale chronic pain-related chatter on Twitter, (b) develop natural language processing and machine learning methods for automatically detecting self-disclosures, (c) collect longitudinal data posted by them, and (d) semiautomatically analyze the types of chronic pain-related information reported by them. We collected data using chronic painrelated hashtags and keywords and manually annotated 4,998 posts to indicate if they were self-reports of chronic pain experiences. We trained and evaluated several state-of-the-art supervised text classification models and deployed the best-performing classifier. We collected all publicly available posts from detected cohort members and conducted manual and natural language processing-driven descriptive analyses. Results: Interannotator agreement for the binary annotation was 0.82 (Cohen's kappa). The RoBERTa model performed best (F 1 score: 0.84; 95% confidence interval: 0.80 to 0.89), and we used this model to classify all collected unlabeled posts. We discovered 22,795 self-reported chronic pain sufferers and collected over 3 million of their past posts. Further analyses revealed information about, but not limited to, alternative treatments, patient sentiments about treatments, side effects, and self-management strategies. Conclusion: Our social media based approach will result in an automatically growing large cohort over time, and the data can be leveraged to identify effective opioid-alternative therapies for diverse chronic pain types.
Nonmedical substance use surveillance via social media has the potential to provide low-cost and ... more Nonmedical substance use surveillance via social media has the potential to provide low-cost and more timely insights than traditional approaches. However, current social media-based approaches lack the capability to provide fine-grained, subpopulation-level statistics. We attempted to fill the gap by developing natural language processing methods to estimate the demographic distribution (gender, age, and race) of a large cohort (N = 288,562) of people who reported nonmedical prescription medication use on Twitter. Automatically derived distributions for opioids, stimulants, and tranquilizers were largely consistent, often with very strong correlations, with statistics reported in traditional sources such as the National Survey on Drug Use and Health. Our work represents an important stride in establishing social media as a complementary resource for substance use surveillance.
Intimate partner violence (IPV) is a preventable public health problem that affects millions of p... more Intimate partner violence (IPV) is a preventable public health problem that affects millions of people worldwide. Approximately one in four women are estimated to be or have been victims of severe violence at some point in their lives, irrespective of age, ethnicity, and economic status. Victims often report IPV experiences on social media, and automatic detection of such reports via machine learning may enable improved surveillance and targeted distribution of support and/or interventions for those in need. However, no artificial intelligence systems for automatic detection currently exists, and we attempted to address this research gap. We collected posts from Twitter using a list of IPV-related keywords, manually reviewed subsets of retrieved posts, and prepared annotation guidelines to categorize tweets into IPV-report or non-IPV-report. We annotated 6,348 tweets in total, with the inter-annotator agreement (IAA) of 0.86 (Cohen's kappa) among 1,834 double-annotated tweets. The class distribution in the annotated dataset was highly imbalanced, with only 668 posts (~11%) labeled as IPVreport. We then developed an effective natural language processing model to identify IPV-reporting tweets automatically. The developed model achieved classification F 1-scores of 0.76 for the IPV-report class and 0.97 for the non-IPV-report class. We conducted post-classification analyses to determine the causes of system errors and to ensure that the system did not exhibit biases in its decision making, particularly with respect to race and gender. Our automatic model can be an essential component for a proactive social media-based intervention and support framework, while also aiding population-level surveillance and large-scale cohort studies.
Americans bear a high chronic stress burden, particularly during the COVID-19 pandemic. Although ... more Americans bear a high chronic stress burden, particularly during the COVID-19 pandemic. Although social media have many strengths to complement the weaknesses of conventional stress measures, including surveys, they have been rarely utilized to detect individuals self-reporting chronic stress. Thus, this study aimed to develop and evaluate an automatic system on Twitter to identify users who have self-reported chronic stress experiences. Using the Twitter public streaming application programming interface, we collected tweets containing certain stress-related keywords (eg, "chronic," "constant," "stress") and then filtered the data using pre-defined text patterns. We manually annotated tweets with (without) self-report of chronic stress as positive (negative). We trained multiple classifiers and tested them via accuracy and F 1 score. We annotated 4195 tweets (1560 positives, 2635 negatives), achieving an inter-annotator agreement of 0.83 (Cohen's kappa). The classifier based on Bidirectional Encoder Representation from Transformers performed the best (accuracy of 83.6% [81.0-86.1]), outperforming the second best-performing classifier (support vector machines: 76.4% [73.5-79.3]). The past tweets from the authors of positive tweets contained useful information, including sources and health impacts of chronic stress. Our study demonstrates that users' self-reported chronic stress experiences can be automatically identified on Twitter, which has a high potential for surveillance and large-scale intervention.
Few-shot learning (FSL) is a category of machine learning models that are designed with the inten... more Few-shot learning (FSL) is a category of machine learning models that are designed with the intent of solving problems that have small amounts of labeled data available for training. FSL research progress in natural language processing (NLP), particularly within the medical domain, has been notably slow, primarily due to greater difficulties posed by domain-specific characteristics and data sparsity problems. We explored the use of novel methods for text representation and encoding combined with distance-based measures for improving FSL entity detection. In this paper, we propose a data augmentation method to incorporate semantic information from medical texts into the learning process and combine it with a nearest-neighbor classification strategy for predicting entities. Experiments performed on five biomedical text datasets demonstrate that our proposed approach often outperforms other approaches.
Introduction: Decades of research have established the association between adverse childhood expe... more Introduction: Decades of research have established the association between adverse childhood experiences (ACEs) and adult onset of chronic diseases, influenced by health behaviors and social determinants of health (SDoH). Machine Learning (ML) is a powerful tool for computing these complex associations and accurately predicting chronic health conditions. Methods: Using the 2021 Behavioral Risk Factor Surveillance Survey, we developed several ML models-random forest, logistic regression, support vector machine, Naïve Bayes, and K-Nearest Neighbor-over data from a sample of 52,268 respondents. We predicted 13 chronic health conditions based on ACE history, health behaviors, SDoH, and demographics. We further assessed each variable's importance in outcome prediction for model interpretability. We evaluated model performance via the Area Under the Curve (AUC) score. Results: With the inclusion of data on ACEs, our models outperformed or demonstrated similar accuracies to existing models in the literature that used SDoH to predict health outcomes. The most accurate models predicted diabetes, pulmonary diseases, and heart attacks. The random forest model was the most effective for diabetes (AUC = 0.784) and heart attacks (AUC = 0.732), and the logistic regression model most accurately predicted pulmonary diseases (AUC = 0.753). The strongest predictors across models were age, ever monitored blood sugar or blood pressure, count of the monitoring behaviors for blood sugar or blood pressure, BMI, time of last cholesterol check, employment status, income, count of vaccines received, health insurance status, and total ACEs. A cumulative measure of ACEs was a stronger predictor than individual ACEs. Discussion: Our models can provide an interpretable, trauma-informed framework to identify and intervene with at-risk individuals early to prevent chronic health conditions and address their inequalities in the U.S.
While social media has evolved into a useful resource for studying medication-related information... more While social media has evolved into a useful resource for studying medication-related information, observational studies of medications have continued to rely on other sources of data. Towards advancing the use of social media data for medication-related observational studies, we analyze an annotated corpus of 27,941 tweets designed for training machine learning algorithms to automatically detect users' medication intake. In particular, we assess how a baseline classifier trained on the general corpus-that is, on various types of medication-performs for specific types. For most types, the classifier performs significantly better than it does overall; however, for nervous system medications, it performs significantly worse. These results suggest that, while the general corpus may have utility for observational studies focusing on most types of medication, studying nervous system medications may benefit from training a classifier exclusively for this type. We will explore this data-level approach in future work.
Journal of the American Medical Informatics Association, Apr 11, 2019
We read with great interest the article by Cocos et al. 1 In it, the authors use one of the datas... more We read with great interest the article by Cocos et al. 1 In it, the authors use one of the datasets made public by our lab in parallel with a publication in Journal of the American Medical Informatics Association, 2 referred to by them as the Twitter ADR Dataset (v1.0) (henceforth the ADRMine Dataset). Cocos et al use state-of-the-art recurrent neural network (RNN) models for extracting adverse drug reaction (ADR) mentions in Twitter posts. We commend the authors for their clear description of the workings of neural models, and on their experiments on the use of fixed versus trainable embeddings, which can be very valuable to the natural language processing (NLP) research community. We believe that using deep learning models offer greater opportunities for mining ADR posts on social media. However, there are key choices made by the authors that require clarification to avoid a misunderstanding on the impact of their findings. In a nutshell, because the authors did not use the ADRMine Dataset in its entirety, discarding upfront all tweets with no human annotations (ie, those that do not contain any ADRs), the resulting train and test sets are biased toward the positive class. Thus, the performance measures reported for the task in Cocos et al are not comparable to those reported in Nikfarjam et al, 2 contrary to what the manuscript reports. After discarding tweets with no human annotation from the ADRMine Dataset, the authors downloaded available tweets from Twitter, and added a small set (203 tweets) to form the dataset used for their experiments. While downloading from Twitter results in an almost unavoidable reduction in the dataset size-as not all tweets are available as time goes by-it would not generally affect the class balance. The elimination of the tweets with no human annotations from the ADRMine Dataset, however, is a choice that is not discussed by Cocos et al, even though it severely impacts the
Social media has evolved into a crucial resource for obtaining large volumes of real-time informa... more Social media has evolved into a crucial resource for obtaining large volumes of real-time information. The promise of social media has been realized by the public health domain, and recent research has addressed some important challenges in that domain by utilizing social media data. Tasks such as monitoring flu trends, viral disease outbreaks, medication abuse, and adverse drug reactions are some examples of studies where data from social media have been exploited. The focus of this workshop is to explore solutions to three important natural language processing challenges for domain-specific social media text: (i) text classification, (ii) information extraction, and (iii) concept normalization. To explore different approaches to solving these problems on social media data, we designed a shared task which was open to participants globally. We designed three tasks using our in-house annotated Twitter data on adverse drug reactions. Task 1 involved automatic classification of adverse drug reaction assertive user posts; Task 2 focused on extracting specific adverse drug reaction mentions from user posts; and Task 3, which was slightly ill-defined due to the complex nature of the problem, involved normalizing user mentions of adverse drug reactions to standardized concept IDs. A total of 11 teams participated, and a total of 24 (18 for Task 1, and 6 for Task 2) system runs were submitted. Following the evaluation of the systems, and an assessment of their innovation/novelty, we accepted 7 descriptive manuscripts for publication-5 for Task 1 and 2 for Task 2. We provide descriptions of the tasks, data, and participating systems in this paper.
Introduction Adverse drug reactions (ADRs) are associated with significant health-related and fin... more Introduction Adverse drug reactions (ADRs) are associated with significant health-related and financial burden, and multiple sources are currently utilized to actively discover them. Social media has been proposed as a potential resource for monitoring ADRs, but drug-specific analytical studies comparing social media with other sources are scarce. Objectives Our objective was to develop methods to compare ADRs mentioned in social media with those in traditional sources: the US FDA Adverse Event Reporting System (FAERS), drug information databases (DIDs), and systematic reviews. Methods A total of 10,188 tweets mentioning adalimumab collected between June 2014 and August 2016 were included. ADRs in the corpus were extracted semi-automatically and manually mapped to standardized concepts in the Unified Medical Language System. ADRs were grouped into 16 biologic categories for comparisons. Frequencies, relative frequencies, disproportionality analyses, and rank ordering were used as metrics. Results There was moderate agreement between ADRs in social media and traditional sources. "Local and injection site reactions" was the top ADR in Twitter, DIDs, and systematic reviews by frequency, ranked frequency, and index ranking. The next highest ADR in Twitter-fatigue-ranked fifth and seventh in FAERS and DIDs. Conclusion Social media posts often express mild and symptomatic ADRs, but rates are measured differently in scientific sources. ADRs in FAERS are reported as absolute numbers, in DIDs as percentages, and in systematic reviews as percentages, risk ratios, or other metrics, which makes comparisons challenging; however, overlap is substantial. Social media analysis facilitates open-ended investigation of patient perspectives and may reveal concepts (e.g. anxiety) not available in traditional sources.
Free text data from social media is now widely used in natural language processing research, and ... more Free text data from social media is now widely used in natural language processing research, and one of the most common machine learning tasks performed on this data is classification. Generally speaking, performances of supervised classification algorithms on social media datasets are lower than those on texts from other sources, but recentlyproposed transformer-based models have considerably improved upon legacy state-of-theart systems. Currently, there is no study that compares the performances of different variants of transformer-based models on a wide range of social media text classification datasets. In this paper, we benchmark the performances of transformer-based pre-trained models on 25 social media text classification datasets, 6 of which are health-related. We compare three pre-trained language models, RoBERTa-base, BERTweet and Clinical-BioBERT in terms of classification accuracy. Our experiments show that RoBERTa-base and BERTweet perform comparably on most datasets, and considerably better than Clinical-BioBERT, even on health-related datasets.
Social networks, such as Twitter, have become important sources for active monitoring of user-rep... more Social networks, such as Twitter, have become important sources for active monitoring of user-reported adverse drug reactions (ADRs). Automatic extraction of ADR information can be crucial for healthcare providers, drug manufacturers, and consumers. However, because of the non-standard nature of social media language, automatically extracted ADR mentions need to be mapped to standard forms before they can be used by operational pharmacovigilance systems. We propose a modular natural language processing pipeline for mapping (normalizing) colloquial mentions of ADRs to their corresponding standardized identifiers. We seek to accomplish this task and enable customization of the pipeline so that distinct unlabeled free text resources can be incorporated to use the system for other normalization tasks. Our approach, which we call Hybrid Semantic Analysis (HSA), sequentially employs rule-based and semantic matching algorithms for mapping user-generated mentions to concept IDs in the Unified Medical Language System vocabulary. The semantic matching component of HSA is adaptive in nature and uses a regression model to combine various measures of semantic relatedness and resources to optimize normalization performance on the selected data source. On a publicly available corpus, our normalization method achieves 0.502 recall and 0.823 precision (F-measure: 0.624). Our proposed method outperforms a baseline based on latent semantic analysis and another that uses MetaMap.
We present our supervised sentiment classification system which competed in SemEval-2016 Task 4: ... more We present our supervised sentiment classification system which competed in SemEval-2016 Task 4: Sentiment Analysis in Twitter. Our system employs a Support Vector Machine (SVM) classifier trained using a number of features including n-grams, synset expansions, various sentiment scores, word clusters, and term centroids. Using weighted SVMs, to address the issue of class imbalance, our system obtains positive class F-scores of 0.694 and 0.650, and negative class F-scores of 0.391 and 0.493 over the training and test sets, respectively.
Journal of the American Medical Informatics Association, Sep 27, 2019
Objective: Twitter posts are now recognized as an important source of patient-generated data, pro... more Objective: Twitter posts are now recognized as an important source of patient-generated data, providing unique insights into population health. A fundamental step toward incorporating Twitter data in pharmacoepidemiologic research is to automatically recognize medication mentions in tweets. Given that lexical searches for medication names suffer from low recall due to misspellings or ambiguity with common words, we propose a more advanced method to recognize them. Materials and Methods: We present Kusuri, an Ensemble Learning classifier able to identify tweets mentioning drug products and dietary supplements. Kusuri (, "medication" in Japanese) is composed of 2 modules: first, 4 different classifiers (lexicon based, spelling variant based, pattern based, and a weakly trained neural network) are applied in parallel to discover tweets potentially containing medication names; second, an ensemble of deep neural networks encoding morphological, semantic, and long-range dependencies of important words in the tweets makes the final decision. Results: On a class-balanced (50-50) corpus of 15 005 tweets, Kusuri demonstrated performances close to human annotators with an F 1 score of 93.7%, the best score achieved thus far on this corpus. On a corpus made of all tweets posted by 112 Twitter users (98 959 tweets, with only 0.26% mentioning medications), Kusuri obtained an F 1 score of 78.8%. To the best of our knowledge, Kusuri is the first system to achieve this score on such an extremely imbalanced dataset. Conclusions: The system identifies tweets mentioning drug names with performance high enough to ensure its usefulness, and is ready to be integrated in pharmacovigilance, toxicovigilance, or more generally, public health pipelines that depend on medication name mentions.
Evidence based medicine (EBM) urges the medical doctor to incorporate the latest available clinic... more Evidence based medicine (EBM) urges the medical doctor to incorporate the latest available clinical evidence at point of care. A major stumbling block in the practice of EBM is the difficulty to keep up to date with the clinical advances. In this paper we describe a corpus designed for the development and testing of text processing tools for EBM, in particular for tasks related to the extraction and summarisation of answers and corresponding evidence related to a clinical query. The corpus is based on material from the Clinical Inquiries section of The Journal of Family Practice. It was gathered and annotated by a combination of automated information extraction, crowdsourcing tasks, and manual annotation. It has been used for the original summarisation task for which it was designed, as well as for other related tasks such as the appraisal of clinical evidence and the clustering of the results. The corpus is available at SourceForge (http://sourceforge.net/projects/ ebmsumcorpus/).
Uploads
Papers by Abeed Sarker