Papers by Victor Makarenkov
One of the challenges in the NLP field is training large classification models, a task that is bo... more One of the challenges in the NLP field is training large classification models, a task that is both difficult and tedious. It is even harder when GPU hardware is unavailable. The increased availability of pre-trained and off-the-shelf word embeddings, models, and modules aim at easing the process of training large models and achieving a competitive performance. We explore the use of off-the-shelf BERT models and share the results of our experiments and compare their results to those of LSTM networks and more simple baselines. We show that the complexity and computational cost of BERT is not a guarantee for enhanced predic-tive performance in the classification tasks at hand.
Scientific writing is difficult. It is even harder for those for whom English is a second languag... more Scientific writing is difficult. It is even harder for those for whom English is a second language (ESL learners). Scholars around the world spend a significant amount of time and resources proofreading their work before submitting it for review or publication.
In this paper we present a novel machine learning based application for proper word choice task. Proper word choice is a generalization the lexical substitution (LS) and grammatical error correction (GEC) tasks. We demonstrate and evaluate the usefulness of applying bidirectional Long Short Term Memory (LSTM) tagger, for this task. While state-of-the-art grammatical error correction uses error-specific classifiers and machine translation methods, we demonstrate an unsupervised method that is based solely on a high quality text corpus and does not require manually annotated data. We use a bidirectional Recurrent Neural Network (RNN) with LSTM for learning the proper word choice based on a word's sentential context. We demonstrate and evaluate our application on both a domain-specific (scientific), writing task and a general-purpose writing task. We show that our domain-specific and general-purpose models outperform state-of-the-art general context learning. As an additional contribution of this research, we also share our code, pre-trained models, and a new ESL learner test set with the research community.
In the process of online storytelling, individual users create and consume highly diverse content... more In the process of online storytelling, individual users create and consume highly diverse content that contains a great deal of implicit beliefs and not plainly expressed narrative. It is hard to manually detect these implicit beliefs, intentions and moral foundations of the writers. We study and investigate two different tasks, each of which reflect the difficulty of detecting an implicit user's knowledge, intent or belief that may be based on writer's moral foundation: 1) political perspective detection in news articles 2) identification of informational vs. conversational questions in community question answering (CQA) archives and. In both tasks we first describe new interesting annotated datasets and make the datasets publicly available. Second, we compare various classification algorithms, and show the differences in their performance on both tasks. Third, in political perspective detection task we utilize a narrative representation language of local press to identify perspective differences between presumably neutral American and British press.
Identifying Informational vs. Conversational Questions on Community Question Answering Archives, 2018
Questions on community question answering websites usually reflect one of two intents: learning i... more Questions on community question answering websites usually reflect one of two intents: learning information or starting a conversation. In this paper, we revisit this fundamental classification task of informational versus conversational questions, which was originally introduced and studied in 2009. We use a substantially larger dataset of archived questions from Yahoo Answers, which includes the question's title, description, answers, and votes. We replicate the original experiments over this dataset, point out the common and different from the original results, and present a broad set of characteristics that distinguish the two question types. We also develop new classifiers that make use of additional data types, advanced machine learning, and a large dataset of unlabeled data, which achieve enhanced performance.
Finite satisfiability of class diagrams
Proceedings of the 6th International Workshop on Model-Driven Engineering, Verification and Validation - MoDeVVa '09, 2009
Models lie at the heart of the emerging Model Driven Development (MDD) approach, in which softwar... more Models lie at the heart of the emerging Model Driven Development (MDD) approach, in which software is developed by repeated transformations of models. Since models are intended as executable specifications, there is a need to provide correctness management on the model level. The underlying hypothesis of this research is that model level tools should be strengthened, to support model elements
Metric Driven Approach for Automatic Creation of Model Benchmarks
Model metrics measure various aspects of model complexity. They are used for evaluating the quali... more Model metrics measure various aspects of model complexity. They are used for evaluating the quality of models and anticipate management needs. Metrics can be useful also for directing model creation. The latter usage is particularly useful for automatic benchmark creation, since manual creation is exhausting and in exible. Such benchmarks are needed for experimental evaluation of model processing algorithms, and for nding their strengths and weaknesses
Finite satisfiability of class diagrams: practical occurrence and scalability of the FiniteSat algorithm
Abstract. Models lie at the heart of the emerging Model Driven De-velopment (MDD) approach, in wh... more Abstract. Models lie at the heart of the emerging Model Driven De-velopment (MDD) approach, in which software is developed by repeated transformations of models. Since models are intended as executable speci-fications, there is a need to provide correctness ...
The query-performance prediction task aims at estimating
the retrieval effectiveness of queries w... more The query-performance prediction task aims at estimating
the retrieval effectiveness of queries without obtaining relevance
feedback from users. Most of the recently proposed
predictors were empirically evaluated with various datasets
to demonstrate their merits. We propose a framework for
theoretical categorization and estimation of the value of query
performance predictors (QPP) without empirical evaluation.
We demonstrate the application of the proposed framework
on four representative selected predictors and show how it
emphasizes their strengths and weaknesses. The main contribution
of this work is the theoretical grounded categorization
of representative QPP.
Finite satisfiability of class diagrams
Proceedings of the 6th International Workshop on Model-Driven Engineering, Verification and Validation - MoDeVVa '09, 2009
Abstract. Models lie at the heart of the emerging Model Driven De-velopment (MDD) approach, in wh... more Abstract. Models lie at the heart of the emerging Model Driven De-velopment (MDD) approach, in which software is developed by repeated transformations of models. Since models are intended as executable speci-fications, there is a need to provide correctness ...
This thesis presents a metric based automatic benchmark creation method. The thesis provides patt... more This thesis presents a metric based automatic benchmark creation method. The thesis provides patterns of model based metrics, and an implemented method for translating these patterns to Alloy and automatically create benchmark models. This research was motivated by a study of nite satisability of class diagrams, extending the FiniteSat algorithm, to support the qualier constraint. This extension of FiniteSat algorithm, was also developed within this thesis. Further, during the research of practical occurrence and relevance of correctness problems within class diagrams, a problem of manual creation of class diagrams for experiments was met.
Model metrics measure various aspects of model complexity. They are used for
evaluating the qual... more Model metrics measure various aspects of model complexity. They are used for
evaluating the quality of models and anticipate management needs. Metrics can be useful
also for directing model creation. The latter usage is particularly useful for automatic
benchmark creation, since manual creation is exhausting and in exible. Such benchmarks
are needed for experimental evaluation of model processing algorithms, and for nding their
strengths and weaknesses
Models lie at the heart of the emerging model-driven engineering approach. In order to guarantee ... more Models lie at the heart of the emerging model-driven engineering approach. In order to guarantee precise, consistent, and correct models, there is a need for efficient powerful methods for verifying model correctness. Class diagram is the central language within UML. Its correctness problems involve issues of contradiction, namely the consistency problem, and issues of finite instantiation, namely the finite satisfiability problem.
This article analyzes the problem of finite satisfiability of class diagrams with class hierarchy constraints and generalization-set constraints. The article introduces the FiniteSat algorithm for efficient detection of finite satisfiability in such class diagrams, and analyzes its limitations in terms of complex hierarchy structures. FiniteSat is strengthened in two directions. First, an algorithm for identification of the cause for a finite satisfiability problem is introduced. Second, a method for propagation of generalization-set constraints in a class diagram is introduced. The propagation method serves as a preprocessing step that improves FiniteSat performance, and helps developers in clarifying intended constraints. These algorithms are implemented in the FiniteSatUSE tool [BGU Modeling Group 2011b], as part of our ongoing effort for constructing a model-level integrated development environment [BGU Modeling Group 2010a]
Models lie at the heart of the emerging Model-driven Engineering approach. In
order to guarantee... more Models lie at the heart of the emerging Model-driven Engineering approach. In
order to guarantee precise, consistent and correct models, there is an urgent need for
eËcient methods for verifying model correctness. Class diagrams are the most important
UML model. Finite satisfiability of class diagrams characterizes the ability to finitely
instantiate the classes of a class diagram, without violating any constraint. This paper
extends our previous work on eËcient recognition of finite satisfiability problems in UML
Thesis Chapters by Victor Makarenkov
PhD Thesis, 2019
Given the enormous amount of free text currently being produced that
can be analyzed for differen... more Given the enormous amount of free text currently being produced that
can be analyzed for different purposes, natural language processing (NLP)
and text analysis have received a significant amount of attention in recent
years. Success and advancements in the field can be partially attributed to
the improved algorithms and capabilities of the deep learning techniques
applied to NLP tasks. In particular, recursive neural networks (RNN), which
capture word order, have enhanced performance in a wide range of NLP
tasks. In this work, we focus specifically on the capabilities of a popular type
of RNN architecture, the long short-term memory (LSTM) network, for two
text analysis tasks.
First, we investigate the implicit dimension identification task, focusing
on two use cases: 1) political perspective identification in online news articles,
and 2) informational vs. conversational question identification in community
question answering (CQA) archives.
Second, we investigate the writing support task, and more specifically
the task of proper word choice. We evaluate this task on a dataset of edited
manuscripts originally written by English as a second language (ESL) learners.
We show that an RNN LSTM network can serve as a very effective mechanism
for the underlying statistical classification method. We utilize LSTM
networks to effectively solve both tasks and present solutions that are both
elegant and high performing. We show how unlabeled data can be utilized to
improve performance when labeled data, which is typically labor intensive
and expensive to obtain, is unavailable.
Our experiments demonstrate the enhanced performance of LSTM-based
models in all three tasks; in addition, we study the implications of using
specific parameters and classification signals in each task and their impact
on performance. We create interesting and novel datasets, which we share
with the research community.
Drafts by Victor Makarenkov
Fields such as the philosophy of language, continental philosophy, and literary studies have long... more Fields such as the philosophy of language, continental philosophy, and literary studies have long established that human language is, at its essence, ambiguous and that this quality, although challenging to communication , enriches language and points to the complexity of human thought. On the other hand, in the NLP field there have been ongoing efforts aimed at disambiguation for various downstream tasks. This work brings together computational text analysis and literary analysis to demonstrate the extent to which ambiguity in certain texts plays a key role in shaping meaning and thus requires analysis rather than elimination. We re-visit the discussion, well known in the humanities , about the role ambiguity plays in Henry James' 19th century novella, "The Turn of the Screw." We model each of the novella's two competing interpretations as a topic and computationally demonstrate that the duality between them exists consistently throughout the work and shapes, rather than obscures, its meaning. We also demonstrate that cosine similarity and word mover's distance are sensitive enough to detect ambiguity in its most subtle literary form, despite doubts to the contrary raised by literary scholars. Our analysis is built on topic word lists and word embeddings from various sources. We first claim, and then empirically show, the interdependence between computational analysis and close reading performed by a human expert.
Uploads
Papers by Victor Makarenkov
In this paper we present a novel machine learning based application for proper word choice task. Proper word choice is a generalization the lexical substitution (LS) and grammatical error correction (GEC) tasks. We demonstrate and evaluate the usefulness of applying bidirectional Long Short Term Memory (LSTM) tagger, for this task. While state-of-the-art grammatical error correction uses error-specific classifiers and machine translation methods, we demonstrate an unsupervised method that is based solely on a high quality text corpus and does not require manually annotated data. We use a bidirectional Recurrent Neural Network (RNN) with LSTM for learning the proper word choice based on a word's sentential context. We demonstrate and evaluate our application on both a domain-specific (scientific), writing task and a general-purpose writing task. We show that our domain-specific and general-purpose models outperform state-of-the-art general context learning. As an additional contribution of this research, we also share our code, pre-trained models, and a new ESL learner test set with the research community.
the retrieval effectiveness of queries without obtaining relevance
feedback from users. Most of the recently proposed
predictors were empirically evaluated with various datasets
to demonstrate their merits. We propose a framework for
theoretical categorization and estimation of the value of query
performance predictors (QPP) without empirical evaluation.
We demonstrate the application of the proposed framework
on four representative selected predictors and show how it
emphasizes their strengths and weaknesses. The main contribution
of this work is the theoretical grounded categorization
of representative QPP.
evaluating the quality of models and anticipate management needs. Metrics can be useful
also for directing model creation. The latter usage is particularly useful for automatic
benchmark creation, since manual creation is exhausting and in exible. Such benchmarks
are needed for experimental evaluation of model processing algorithms, and for nding their
strengths and weaknesses
This article analyzes the problem of finite satisfiability of class diagrams with class hierarchy constraints and generalization-set constraints. The article introduces the FiniteSat algorithm for efficient detection of finite satisfiability in such class diagrams, and analyzes its limitations in terms of complex hierarchy structures. FiniteSat is strengthened in two directions. First, an algorithm for identification of the cause for a finite satisfiability problem is introduced. Second, a method for propagation of generalization-set constraints in a class diagram is introduced. The propagation method serves as a preprocessing step that improves FiniteSat performance, and helps developers in clarifying intended constraints. These algorithms are implemented in the FiniteSatUSE tool [BGU Modeling Group 2011b], as part of our ongoing effort for constructing a model-level integrated development environment [BGU Modeling Group 2010a]
order to guarantee precise, consistent and correct models, there is an urgent need for
eËcient methods for verifying model correctness. Class diagrams are the most important
UML model. Finite satisfiability of class diagrams characterizes the ability to finitely
instantiate the classes of a class diagram, without violating any constraint. This paper
extends our previous work on eËcient recognition of finite satisfiability problems in UML
Thesis Chapters by Victor Makarenkov
can be analyzed for different purposes, natural language processing (NLP)
and text analysis have received a significant amount of attention in recent
years. Success and advancements in the field can be partially attributed to
the improved algorithms and capabilities of the deep learning techniques
applied to NLP tasks. In particular, recursive neural networks (RNN), which
capture word order, have enhanced performance in a wide range of NLP
tasks. In this work, we focus specifically on the capabilities of a popular type
of RNN architecture, the long short-term memory (LSTM) network, for two
text analysis tasks.
First, we investigate the implicit dimension identification task, focusing
on two use cases: 1) political perspective identification in online news articles,
and 2) informational vs. conversational question identification in community
question answering (CQA) archives.
Second, we investigate the writing support task, and more specifically
the task of proper word choice. We evaluate this task on a dataset of edited
manuscripts originally written by English as a second language (ESL) learners.
We show that an RNN LSTM network can serve as a very effective mechanism
for the underlying statistical classification method. We utilize LSTM
networks to effectively solve both tasks and present solutions that are both
elegant and high performing. We show how unlabeled data can be utilized to
improve performance when labeled data, which is typically labor intensive
and expensive to obtain, is unavailable.
Our experiments demonstrate the enhanced performance of LSTM-based
models in all three tasks; in addition, we study the implications of using
specific parameters and classification signals in each task and their impact
on performance. We create interesting and novel datasets, which we share
with the research community.
Drafts by Victor Makarenkov
In this paper we present a novel machine learning based application for proper word choice task. Proper word choice is a generalization the lexical substitution (LS) and grammatical error correction (GEC) tasks. We demonstrate and evaluate the usefulness of applying bidirectional Long Short Term Memory (LSTM) tagger, for this task. While state-of-the-art grammatical error correction uses error-specific classifiers and machine translation methods, we demonstrate an unsupervised method that is based solely on a high quality text corpus and does not require manually annotated data. We use a bidirectional Recurrent Neural Network (RNN) with LSTM for learning the proper word choice based on a word's sentential context. We demonstrate and evaluate our application on both a domain-specific (scientific), writing task and a general-purpose writing task. We show that our domain-specific and general-purpose models outperform state-of-the-art general context learning. As an additional contribution of this research, we also share our code, pre-trained models, and a new ESL learner test set with the research community.
the retrieval effectiveness of queries without obtaining relevance
feedback from users. Most of the recently proposed
predictors were empirically evaluated with various datasets
to demonstrate their merits. We propose a framework for
theoretical categorization and estimation of the value of query
performance predictors (QPP) without empirical evaluation.
We demonstrate the application of the proposed framework
on four representative selected predictors and show how it
emphasizes their strengths and weaknesses. The main contribution
of this work is the theoretical grounded categorization
of representative QPP.
evaluating the quality of models and anticipate management needs. Metrics can be useful
also for directing model creation. The latter usage is particularly useful for automatic
benchmark creation, since manual creation is exhausting and in exible. Such benchmarks
are needed for experimental evaluation of model processing algorithms, and for nding their
strengths and weaknesses
This article analyzes the problem of finite satisfiability of class diagrams with class hierarchy constraints and generalization-set constraints. The article introduces the FiniteSat algorithm for efficient detection of finite satisfiability in such class diagrams, and analyzes its limitations in terms of complex hierarchy structures. FiniteSat is strengthened in two directions. First, an algorithm for identification of the cause for a finite satisfiability problem is introduced. Second, a method for propagation of generalization-set constraints in a class diagram is introduced. The propagation method serves as a preprocessing step that improves FiniteSat performance, and helps developers in clarifying intended constraints. These algorithms are implemented in the FiniteSatUSE tool [BGU Modeling Group 2011b], as part of our ongoing effort for constructing a model-level integrated development environment [BGU Modeling Group 2010a]
order to guarantee precise, consistent and correct models, there is an urgent need for
eËcient methods for verifying model correctness. Class diagrams are the most important
UML model. Finite satisfiability of class diagrams characterizes the ability to finitely
instantiate the classes of a class diagram, without violating any constraint. This paper
extends our previous work on eËcient recognition of finite satisfiability problems in UML
can be analyzed for different purposes, natural language processing (NLP)
and text analysis have received a significant amount of attention in recent
years. Success and advancements in the field can be partially attributed to
the improved algorithms and capabilities of the deep learning techniques
applied to NLP tasks. In particular, recursive neural networks (RNN), which
capture word order, have enhanced performance in a wide range of NLP
tasks. In this work, we focus specifically on the capabilities of a popular type
of RNN architecture, the long short-term memory (LSTM) network, for two
text analysis tasks.
First, we investigate the implicit dimension identification task, focusing
on two use cases: 1) political perspective identification in online news articles,
and 2) informational vs. conversational question identification in community
question answering (CQA) archives.
Second, we investigate the writing support task, and more specifically
the task of proper word choice. We evaluate this task on a dataset of edited
manuscripts originally written by English as a second language (ESL) learners.
We show that an RNN LSTM network can serve as a very effective mechanism
for the underlying statistical classification method. We utilize LSTM
networks to effectively solve both tasks and present solutions that are both
elegant and high performing. We show how unlabeled data can be utilized to
improve performance when labeled data, which is typically labor intensive
and expensive to obtain, is unavailable.
Our experiments demonstrate the enhanced performance of LSTM-based
models in all three tasks; in addition, we study the implications of using
specific parameters and classification signals in each task and their impact
on performance. We create interesting and novel datasets, which we share
with the research community.