1

Recommender Systems for Good (RS4Good): Survey of Use

Cases and a Call to Action for Research that Matters
Opinion Paper

DIETMAR JANNACH, University of Klagenfurt, Austria

arXiv:2411.16645v1 [[Link]] 25 Nov 2024

ALAN SAID, University of Gothenburg, Sweden

MARKO TKALČIČ, University of Primorska, Slovenia
MARKUS ZANKER, Free University of Bozen-Bolzano and University of Klagenfurt, Italy and Austria
In the area of recommender systems, the vast majority of research efforts is spent on developing increasingly
sophisticated recommendation models, also using increasingly more computational resources. Unfortunately,
most of these research efforts target a very small set of application domains, mostly e-commerce and media
recommendation. Furthermore, many of these models are never evaluated with users, let alone put into practice.
The scientific, economic and societal value of much of these efforts by scholars therefore remains largely
unclear. To achieve a stronger positive impact resulting from these efforts, we posit that we as a research
community should more often address use cases where recommender systems contribute to societal good
(RS4Good). In this opinion piece, we first discuss a number of examples where the use of recommender systems
for problems of societal concern has been successfully explored in the literature. We then proceed by outlining
a paradigmatic shift that is needed to conduct successful RS4Good research, where the key ingredients are
interdisciplinary collaborations and longitudinal evaluation approaches with humans in the loop.

CCS Concepts: • Information systems → Recommender systems.

Additional Key Words and Phrases: Recommender Systems, Evaluation, Applications

1 INTRODUCTION
Whenever you read a paper on recommender systems published in the past twenty-five years,
chances are good that it has several or all of the following characteristics: (i) the paper proposes
a new recommendation algorithm, (ii) the algorithm is evaluated in the movie or e-commerce
domain, and (iii), the evaluation is focusing on prediction accuracy and it is based on an offline
experiment without involving humans in the loop [49]. As a result, we nowadays have hundreds—if
not thousands—of algorithmic proposals and machine learning (ML) models at our hands that were
designed for, or at least evaluated, on some movie or e-commerce dataset. Yet these achievements
are seemingly not enough. Despite or due to all this progress, research addressing similar problems
flourishes and, nowadays, involves increasingly complex deep learning architectures and, most
recently, large language models with an immense carbon footprint [86, 98].
In terms of application domain, there is of course no doubt that recommendations can create
substantial business value in e-commerce environments, see [45]. There are also various papers from
large players in the media streaming market reporting on the benefits of providing personalized
movie or video recommendations, including Netflix and YouTube [14, 36, 89]. However, due to the
strong focus on offline experimentation in scholarly research, we unfortunately do not possess
a lot of evidence on the practical effectiveness of the countless algorithmic proposals mentioned
above. Will all these algorithmic contributions with their mostly small increases in offline prediction
accuracy lead to more user engagement, higher retention rates, higher sales and more satisfied users?
For many algorithmic proposals, the answer is probably ’no’. Netflix actually never implemented

Authors’ addresses: Dietmar Jannach, [Link]@[Link], University of Klagenfurt, Austria; Alan Said, alansaid@
[Link], University of Gothenburg, Sweden; Marko Tkalčič, [Link]@[Link], University of Primorska, Slovenia;
Markus Zanker, [Link]@[Link], Free University of Bozen-Bolzano and University of Klagenfurt, Italy and Austria.
1:2 Jannach et al.

the winning solution of the million-dollar Netflix Prize competition in production1 , and in a later
paper [36] made a rather disappointed statement about the often limited correlation between
outcomes of offline experiments and the success of a deployed model.
All that puts us as researchers in a situation where we have developed countless recommendation
algorithms over the years, of which however only a very tiny fraction has been tested ‘in the wild’.
Even worse, there is both evidence from the academic literature and from industry reports that gains
in offline accuracy do not reliably translate to increased consumer or business value in practice.
Nonetheless, despite all these known limitations we observe a constant and maybe even increasing
inflow of new algorithmic papers that are evaluated in offline experiments on datasets from a
limited set of domains.
The question arises why we as a recommender systems research community continue to put so
much effort into such a small set of application domains and delimit ourselves to only a few research
methodologies. Actually, the literature contains a multitude of examples where recommender
systems are applied to a variety of application domains beyond movies and e-commerce. Moreover,
the value that is created in several of these settings goes beyond the creation of economic value
for the platform provider as well as beyond individual hedonic or utilitarian needs of consumers.
We may, for instance, think of recommender systems that guide users to a healthier life style,
recommender systems that stimulate energy-saving behavior, or systems that help users achieve
self-actualization goals.
Given that these application areas are largely underexplored in the literature, we call the research
community to re-focus our combined efforts to areas where recommender systems can contribute to
the achievement of societal goals. Following ideas of recent initiatives like ‘AI4Good’2 or ‘IR4Good’3 ,
we envision the development of an ‘RS4Good’ movement in the recommender systems community.
As a result, we hope that the future research efforts in our community will be better aligned with
the pressing challenges that we face as a society. However, as we will lay out in more depth in
the following sections, to achieve these goals it will not be sufficient to only use different datasets,
but a paradigmatic shift is required, instead. It involves the following components: (i) a focus on
problems of societal relevance, (ii) a multi-disciplinary research approach involving humans in the
loop, and, (iii) the consideration of the longitudinal effects of recommender systems.
The rest of this paper is organized as follows. Next, in Section 2, we will explain why RS4Good
ambitions must go beyond recent initiatives to only avoiding harm, and we will outline a number
of RS4Good application areas. Section 3 then describes the implications of a shift to RS4Good use
cases in terms of the applied research methodology.

2 RS4GOOD: TOPIC AREAS

In this section, we position the RS4Good initiative with respect to recent developments in the area
of ‘responsible recommendation’ and then review a number of potential application areas for future
RS4Good developments.

2.1 Defining RS4Good: From Avoiding Harm to Doing Good

Recommender systems are commonly designed to create positive effects and different types of
value for the involved stakeholders. Typical examples of such values include the avoidance of
information overload for end users or increased sales for businesses. During the past few years,
it however became more and more evident that recommender systems may also lead to harm

1 [Link]
2 [Link]
3 [Link]
Recommender Systems for Good (RS4Good): Survey of Use Cases and a Call to Action for Research that Matters 1:3

and certain undesired negative effects. In 2011, the discussion of possible ‘filter bubbles’ that may
emerge from personalized recommendation led to notable public attention [65]. Since then, we
observed increased awareness of potential problems of recommender systems, leading to various
corresponding research works and initiatives which may be collectively placed under the umbrella
terms ‘responsible recommendation’ or ‘trustworthy recommendation’.
The main aspects in this area are the awareness of recommendation biases and the consideration
of goals such as fairness, privacy protection, interpretability and explainability, various forms of
trustworthiness, robustness, auditability, or compliance with laws and regulations. Beyond the issues
of ‘filter bubbles’ and ‘echo chambers’, central topics in the news and media domain also include
the avoidance of the spread of misinformation, polarization, and radicalization. Corresponding
surveys on these topics can be found in [10, 18, 19, 24, 26, 83, 99, 100, 105].
The central underlying motivation of many of these approaches, as mentioned, is rather to avoid
harm than to proactively target at creating a positive value for society. Admittedly, these aspects
may often appear related and overlapping. In our view, however, these approaches set apart the
underlying goals of the recommender system.
Let us consider the example of a content recommender system on social media. Such recommender
systems are often designed to maximize user engagement, with the ultimate goal of increasing
user retention, ad revenue, and company growth. To maximize the engagement, a recommendation
algorithm might now learn that promoting controversial or even hateful content leads to a high
number of interactions, e.g., in terms of user comments. While this indeed may have positive
short-term effects on the mentioned business metrics, it may have negative effects as well, e.g., on
the public reputation of the platform. As a consequence, the platform might implement certain
measures in the recommendation algorithm to delimit the promotion of potentially harmful or
problematic content to a certain extent.
Such avoidance or mitigation strategies are different from our notion of RS4Good that we are
advocating in this paper. Rather than to avoid problems that may for example be caused by the
underlying business goals or by the unintended reinforcement of biases in the underlying data,
social benefit is part of the ‘genes’ and main purpose of an RS4Good recommender system. Certainly,
this does not contradict the assumption that recommender systems are designed to create economic
value for providers.
Let us think of an alternative social media platform that tries to proactively balance the coverage
of content from different political viewpoints, thereby implementing a corporate mission to reduce
radicalization and polarization in society. However, such a societal mission does not necessarily
mean that economic goals may not be pursued in parallel. The provision of balanced and diversified
content may in fact be a key factor of positioning the platform on the market, similar to independent
publishers or traditional newspapers in media environments.

2.2 Selected Use Cases of RS4Good Research

In their recent work, Felfernig et al. [29] discussed the potential role of recommender systems to
support the United Nation’s 17 Sustainability Development Goals (SDGs). These SDGs comprise
both areas that are well aligned with typical goals of recommender systems (e.g., ‘economic growth’)
as well as areas which, as of today, are barely related to existing recommender systems research
(e.g., ‘life below water’). In our following discussion of the RS4Good research landscape, we will
therefore not follow the UN’s SDG organization, but rather structure our work along application
domains where the use of recommendation systems has been already explored at least to a certain
extent. We emphasize that our discussion and categorization is not meant to be exhaustive. Instead,
it shall serve as an illustration and motivation for the many possible directions that our community
should explore in more depth in the future.
1:4 Jannach et al.

We note that our discussion of existing works is mainly organized in terms of application domains.
We however recall that whether a recommender systems is considered to be ‘for good’ does not
depend on the application domain. A recommendation system in the e-commerce domain, for
example, can be guided purely by economic objectives. It may however also have elements of
an RS4Good system, e.g., when the recommender is designed to prioritize sustainable product
alternatives. As mentioned in the previous section, it depends on the underlying goals that are
implemented in the recommender system. In the following, we review a number of RS4Good
systems in different application contexts. Figure 1 illustrates the main topic areas.4

Fig. 1. Selected RS4Good Use Cases

2.2.1 Health. The health domain is a prime example where recommender systems can be applied
for societal good in various ways. A survey of recommender systems in healthcare can be found
in [95]. In this survey, the authors identify the following main application areas.
• Food recommendation: Use cases involve the recommendation of proper diets, better nutrition,
or healthier food alternatives.
• Drug recommendation: Drug recommendation systems are designed to help patients and
health-care professionals to identify medications for certain patient conditions or to predict
drug side-effects.
• Health status prediction: Recommendation techniques like collaborative filtering can be
leveraged to make better predictions of a patient’s risk factors given their current health
status.
• Physical activity recommendation: Approaches in this area were for example developed to
suggest exercises or workouts to patients, e.g., to achieve their calorie burn goals or to
perform their activities to improve sleep [96].
• Healthcare professional recommendation: Recommender systems have also been developed
to help patients find doctors with the best expertise for their health-related issues.
A number of works in the literature cover additional topics that go beyond this classification,
e.g., the recommendation of messages within a smoking cessation app [44]. Such works fall into
the more general category of behavior change support systems [63]. Some of the works in the
literature are also targeted at certain groups of people or patients, for instance, elderly [28, 40, 41]
or people with autism [59], dementia [90], or social anxiety [1]. Today, many of the more recent
health-related recommendation approaches are implemented as (smartphone) apps. This leads to
additional research questions beyond those related to the core recommendation task. The credibility
of a health-related app is for example, important for the adoption of the recommendation, as
4We deliberately leave out the general goal of economic growth
in this list while acknowledging that the growth of individual
businesses and the economy may generally contribute to positive societal developments.
Recommender Systems for Good (RS4Good): Survey of Use Cases and a Call to Action for Research that Matters 1:5

analyzed in [64]. Also there are often ethical aspects to consider, in particular in the context of
mental health apps [74, 97]. Of course, it is often highly important to limit the risks of inadequate
recommendations in such a critical domain. This can for example be achieved by providing adequate
explanations alongside recommendations, and by leaving the final decision of the chosen health
intervention to the medical expert [17].
Generally, health recommender systems is an active research area, as also evidenced by a series
of health-related workshops held in conjunction with the ACM RecSys conference5 . The literature
on the topic is however quite scattered across various publication outlets, see also the survey on
recommenders for digital health in [11]. Various works in this area are published in computer
science outlets, others in medical journals, indicating the potential for more interdisciplinary
research.
2.2.2 Educational Recommender Systems and e-Learning. Another quite active and established ap-
plication domain for RS4Good is the use of recommendation technology to support human/student
learning processes, e.g., in the form of course recommendations or learning paths suggestions in
e-learning environments. Works in this area fall into the broader category of ‘AI in education’,
which is of UNESCO’s focus topics6 . Also, e-learning recommendations represent one form of
Technology-enhanced Learning (TEL) [23].
Several types of items can be recommended in e-learning recommender systems, including
courses and course paths, college majors, papers, web pages, lessons and various other learning
resources. Various types of traditional recommendation algorithms—both content-based, collab-
orative filtering and hybrid ones—have been explored in the literature [51]. In some approaches,
the recommendation of learning activities (and materials) is based on predicting the student’s
performance on a task that she or he has not solved yet. Given these predictions, the set of recom-
mendations is then determined based on the e-learning system’s specific goals. For example, the
system could recommend materials that the student assumedly will like and most probably will be
able to master. Following the exact opposite strategy, the system may however also recommend
materials where it assumes gaps in the student’s knowledge.
An in-depth discussion of the challenges of building e-learning recommender systems can be
found in [51]. A recommender system not only has to understand the learning goals and the prior
knowledge of the student, it has also to take into account personal characteristics, such the student’s
learning strategy. Moreover, recommendations must be made at the right point in time and within an
appropriate interactive tool environment. Generally, technology-enhanced learning processes and
educational recommender systems are a multi-faceted topic, and research in this direction usually
requires an interdisciplinary approach [3]. Furthermore, the evaluation of e-learning recommender
systems remains difficult. As mentioned above, depending on the system’s goal and strategy, one
or the other type of learning content may be appropriate and a good recommendation for a given
student. Furthermore, the true success of the system, i.e., if the student’s knowledge or learning
process, was improved, can usually only be evaluated from a longitudinal perspective.
2.2.3 Individual Development, Wellbeing and Safety. Besides healthcare and education, a number
of works address alternative ways of how recommender systems can contribute to individual
development and wellbeing of users. Some of these works are related to personal health aspects,
e.g., works by Elsweiler et al. [27], who evaluate how recommender systems can nudge users to
choose healthier recipes such as dishes with more favorable nutritional characteristics on food-
focused online sites. A related study on the role of nutrition labels on food choices can be found

5 [Link]
6 [Link]
1:6 Jannach et al.

in [87]. A more general approach to support personal wellbeing with recommender systems was
proposed by Gryard et al. in [39]. Their work is motivated by the increased availability of sensors on
Internet-of-Things wearable devices that can measure physiological signals. In their system, these
sensor signals are fed into a rule-based reasoning engine that recommends actions to (re-)achieve
‘happiness’, e.g., do physical activities, increase the intake of certain natural products, or apply
procedures from alternative medicine. A related IoT-based approach for improved mental health
was later proposed by the authors in [38]. Unfortunately, no evaluation with real users has been
performed for these approaches yet.
Recreational sport is often considered to be an important contributor to wellbeing. The use of
recommendation technology and machine learning techniques for amateur marathon runners was
for example proposed in Smyth et al. [85]. The primary recommendation tasks in their approach
range from the suggestion of personalized training plans to pacing strategies and the prediction
of realistic finish-times. Secondary use cases for runners include the selection of running events,
running routes, training partners, gear and equipment as well as advice regarding nutrition and
recovery. In [85], a number of these uses cases for recommendation technology are discussed in
more depth. The evaluation of the proposed approaches was however largely limited to offline
experiments so far.
Going beyond the realm of sports and health, Knijnenburg et al. [52] propose to develop a new
class of recommender systems for self-actualization. Instead of just recommending the “best” items
to users, such systems are designed to “support users in developing, exploring, and understanding
their own unique tastes and preferences.” As a consequence, future recommender systems for self-
actualization should help users break out of filter bubbles and rather support instead of replace
them in their decision-making tasks. A possible experiment to evaluate such a recommender system
was later outlined in [101]. The results from a user study in the news recommendation domain
are discussed in [93]. In this study, it was found that the reading intention of the participants
are influenced by their stated self-actualization goals, which could either be Broaden Horizons or
Discover the Unexplored.
As a final example in the context of individual development, wellbeing and safety we consider
recommender technology contributing to personal safety. Specifically, a number of approaches have
been proposed to recommend privacy settings to users in the online sphere. An early work [35] on
recommending privacy settings on Facebook was based on categorizing users into different groups
according to their assumed privacy values. A more elaborate technical approach to recommend
image sharing settings was later proposed in [103]. A particularity of this work is that the content
itself, i.e., the images, are analyzed for content sensitiveness. Later on, the work in [78] aims at
considering the user’s disclosure preferences while protecting their privacy. A specific model for
the domain of fitness devices can be found in [79].

2.2.4 Societal and Political Development. A number of RS4Good use cases can be identified that
target at the societal or political development of a society as a whole. In the context of public opinion
formation, for instance, recommender systems are nowadays widely employed on news sites and
social media channels. While news recommender systems can be valuable both for consumers and
providers, they are often associated with potential problems such as the creation of filter bubbles
or leading to polarization. RS4Good initiatives, as indicated above, should however go beyond
avoiding potential harm via responsible media recommendation [26]. Instead, they may, for instance,
aim at supporting a societal editorial mission or a given service mandate as many public European
broadcasters do. In [12], a discussion of challenges of building a recommender system for the British
Broadcasting Corporation (BBC) can be found. Grün and Neufeld in [37] highlight the importance
of transparency, fairness and exploration goals for the public German media service ZDF. Related
Recommender Systems for Good (RS4Good): Survey of Use Cases and a Call to Action for Research that Matters 1:7

discussions can be found in [84], where authors advocate for participatory design practices and
open data to address some of the challenges of digital public services.
Also going beyond the goal of merely avoiding harm, Lu et al. [56] conduct an online study in
the news domain to assess how personalized recommendations influence the reading behavior of
users in the context of editorial values, like diversity and coverage. Furthermore, an intervention
study reveals that a re-ranking algorithm is effective to steer readers to a more dynamic reading
behavior without a loss of accuracy. In a related work, Diakopoulos et al. [20] more generally
discuss the importance of considering professional ethics when designing responsible AI-based
solutions for journalism. Furthermore, Bauer et al. review the consideration of journalistic values7
in news recommender systems [4].
Also in the realm of social media, personalization and recommender systems are often assumed to
be the cause of echo chambers, filter bubbles and polarization [22, 91]. Donkers et al. [22] investigate
the phenomenon of ‘dual’ echo chambers with an agent-based modeling and simulation approach.
Their study reveals that counteracting the two types of echo chambers requires different proactive
diversification and depolarization strategy. In a similar vein, Stray [91] studies how to design
recommender systems for depolarization. Diversification of the presented content is considered as
one possible strategy that can work against polarization in some context, but it may even make
things worse. According to [91], recommenders should generally be designed to deprioritize content
that is found to be polarizing as well as polarization measures should be incorporated into systems
and continuously monitored.
While the news and social media domains are key use cases for recommender systems, there
are various other scenarios where recommender systems can support societal development. In the
educational sector, for example, Wilson et al. [102] propose an online recommender system that is
particularly designed to help families of low socioeconomic status to select an appropriate school
that matches parents’ preferences and the students’ needs and abilities. In the same educational
realm, Milton et al. [62] develop a book recommender system for kids that aims at increasing their
reading interest and literacy.
A number of further research works aim at fostering societal development and political en-
gagement through recommender systems in terms of supporting e-participation [8, 80] or e-
governance [13] processes. Furthermore, the authors of [21] propose a recommender system to
increase social cohesion by enabling each member of the community to feel that they are part of the
cultural heritage of society. Cultural content recommendations are also in the focus of the work
by Ferraro et al. [32], who propose a measure that reflects to what extent recommendation will
familiarize a given user population with certain content categories.
Finally, we mention two examples of works that may lead to a positive and socially desirable
economic development in quite different ways. First, Pourashraf and Mobasher [69] target the
news recommendation domain and argue that readers’ local news interests are different from their
preferences for global news. Thus, by differentiating between local and global news preferences
their localized recommendation model increased prediction accuracy with the ultimate goal to
revitalize and strengthen local media companies. The second work from Liu et al. [55] target the
domain of micro-lending, an approach to provide citizens in impoverished countries with access to
capital. In their work, the authors propose a fairness-aware model that aims to ensure that different
demographic groups have a fair chance of being recommended for a micro-loan.
2.2.5 Environment Protection and Sustainability. A diverse set of approaches to build recommender
systems ‘for good’ can be found in this category. An early work in this area represents Knijnenburg
et al. [53], who develop a system to recommend energy-saving measures to users. They specifically
7A recent, more general discussion of how to implement human values in recommender systems can be found in [92].
1:8 Jannach et al.

explore the impact of different preference elicitation methods on user satisfaction, which in turn
influences behavior outcomes. The recommendation of energy-saving measures is also in the
focus of the work by Starke et al. [88]. In their work, they analyze the effectiveness of different
psychology-informed ways of presenting recommendations to users in a persuasive way.
The reduction of emissions is a central target in other works as well, e.g., in the area of travel
and tourism. Bothos et al. [7], for instance, design a persuasive travel recommender system that
nudges users to adopt greener transportation habits. A number of other research works aims at
implementing smart mobility through recommender systems in the context of smart cities, e.g.,
by promoting the use of public transport and non-motorized mobility options, the reduction of
traffic congestions, or addressing parking management problems, see [70]. A number of additional
recommender systems use cases for smart cities are discussed in [71], where authors argue that
future systems have a higher likelihood of intent-awareness [48] due to novel technological enablers
such as Sensors and Internet-of-Things.
Other recent works in recommender systems focus on sustainability in different dimensions.
Connected to the previously discussed applications in travel and tourism, Piliponyte et al. [68]
propose a simulation-based approach to study the effects of personalized advertisement campaigns
on tourist choices. A main goal of their work is to support decision-makers in running such
campaigns with the goal of avoiding overtourism, which may often lead to natural damage. Avoiding
overcrowding at certain places is also the focus of the work of Patro et al. [67]. The goal of their multi-
objective recommendation approach is to ensure a better and fairer distribution of businesses, such
as restaurants, cafes, or malls, appearing in recommendations, thereby supporting the sustainability
of local businesses and the safety of their visitors.
Finally, a larger body of research works targets at promoting environmental protection and
sustainability through recommender systems in the agricultural domain [66]. Various smart agricul-
ture use cases can be identified in the literature, including crop recommendation based on various
variables related to weather and soil condition, fertilizer recommendation, crop protection and
pesticide recommendation or irrigation management. While the literature in this area is rich, we
note that most of the proposed recommendation approaches are non-personalized. Thus, they are
mainly applications of machine learning models for prediction purposes [54] or optimization-based
solutions for improved decision support.

3 RS4GOOD: SHIFTING OUR RESEARCH FOCUS

Our brief overview in Section 2 demonstrates that already a multitude of RS4Good use cases have
been explored in literature. However, the amount of works that focus on RS4Good-related topics
is negligible compared to the extensive body of literature devoted to proposing new algorithms
exclusively evaluated offline on the few datasets from the media and e-commerce domains. We recall
here that there is nothing wrong with research in these areas, and that such research may also fall
into the scope of RS4Good. E-commerce recommendations can for example be designed to promote
more sustainable products, and a music recommender system’s design may target at increasing the
exposure of artists from minority groups. Thus, we iterate that whether a recommender systems
serves a ‘for good’ purpose or not rather depends on the design goals of the system than on the
application area. However, not many works in the literature address such ‘for good’ purposes in
this way.
The question thus arises why we as a research community do not devote our efforts on research
that matters and fosters societal development, instead of continuing to propose increasingly complex
machine learning models that we evaluate in terms of their capability to accurately predict held-out
data points that derive primarily from the entertainment domain. And even worse, we do so while
knowing that various field tests and user studies tell us that increases in offline accuracy may in
Recommender Systems for Good (RS4Good): Survey of Use Cases and a Call to Action for Research that Matters 1:9

many cases not translate into better value for consumers or businesses [5, 15, 25, 33, 46, 58, 60].
Given all this evidence for this offline-online gap and many voices from industry, it may actually
turn out that much of our research on improving accuracy metrics in offline experiments may be
almost entirely worthless in practice.
One part of the answer to the question above may be as simply as this: that it is much easier to
do algorithm-oriented research based on offline experiments than to do RS4Good research. Typical
RS4Good research can have certain ingredients that may be considered challenging. Usually, one
needs to understand the idiosyncrasies of the particular use case, commonly leading to a need
for interdisciplinary research. Moreover, the evaluation of RS4Good recommender systems can
be demanding, as typically a human-centric research approach is required. Moreover, the aspired
positive societal impacts can many times only be assessed in a longitudinal perspective. Research
based on offline experiments does not face these challenges. Researchers can, for instance, try out
various variants of a newly developed algorithm in a largely automated way without the need
to involve humans in experiments. However, the predominant use of offline experimentation in
combination problematic offline evaluation practices may have led to a certain level of stagnation
in our field.

3.1 Offline Evaluation Practices and the Need for a Crisis

Having said that conducting research based on offline evaluations may be often easier than RS4Good
research does not necessarily imply that it is actually easy to get an algorithm paper published in
a top-level outlet. It requires inspiration for a novel technical idea8 , a working implementation,
sometimes a non-trivial mathematical exposition and analysis, as well as in-depth experimental
evaluations, which, most recently have become increasingly computationally expensive.
Nevertheless offline evaluation practices might be perceived less effort than RS4Good research
involving humans in the loop. In offline evaluation, the evaluation procedures and metrics are to
some extent standardized, whereas in user studies every single aspect of the proposed experimental
design may need to be defended against reviewers.9 Furthermore, in offline experiments, there
are many degrees of freedom regarding the experimental configuration that is used as a basis to
demonstrate an advance with respect to the “state-of-the-art”. Researchers have some freedom
when selecting baselines, evaluation datasets, preprocessing steps, data-splitting techniques, as
well as evaluation metrics and even cut-off values for the metrics [47]. Given this freedom, and
knowing about the possible existence of confirmation bias—where the expectation is that the
proposed method works well—researchers might unknowingly focus on very specific experimental
configurations supporting this bias.
As such, it is not too surprising that a number of studies in recent years have revealed that
the progress that we make in terms of algorithms that lead to better offline accuracy results may
actually be quite limited. Quite worrying, these studies show that often decade-old methods or
conceptually simple techniques based on nearest-neighbor search can outperform even the latest
neural models [30, 31, 57, 61, 72, 73, 82]. One main reason for this phantom progress [31] lies in
the apparently common practice of benchmarking a newly proposed and meticulously fine-tuned
model against baselines that were not particularly tuned for the given dataset(s). Obviously, nothing
8 Interms of technical proposals, we usually do not observe entirely novel fundamental architectures. More commonly,
existing network topologies (e.g., RNN, CNN, GAN, Transformer etc.) and learning approaches (e.g., contrastive learning,
multi-task learning) are applied in innovative ways to recommendation problems.
9 The heavy bias towards algorithm research is commonly reflected in the composition of the reviewer pools of conferences

and journals, and a substantial fraction of the reviewers may not be experts in assessing studies with humans-in-the-loop.
They might, for example, dismiss papers on user studies because of an assumedly too low number of participants, being
accustomed to working with datasets with thousands of users.
1:10 Jannach et al.

meaningful can be concluded from such an experimental setup [81]. Other reasons may include the
use of incorrect or incomplete re-implementations of baseline models, tuning of models on test
data, the use of datasets that are not suited for the task, using the same hyperparameters for all
compared models ‘for fair comparison’, or reporting the best accuracy values for different metrics
across different epochs, see also [42, 43, 94].
Furthermore, the general level of reproducibility of reported research results turns out to be quite
limited as well. While sharing code and data has become more common over the past few years, the
majority of published research works do not publicly provide the necessary resources to reproduce
the numbers that are reported in the papers [9, 31, 48]. This is by no means a novel insight, as
similar observations on the reproducibility of information retrieval and recommender systems
have been made over the past 15 years [2, 77]. As the awareness of this problem increases, more
and more conferences and journals nowadays provide reproducibility guidelines and encourage
authors to share resources. It is nonetheless today still uncommon that papers come with a complete
reproducibility package that includes all necessary materials such as the pre-processed datasets, the
code of the proposed model and the baselines, information about hyperparameter tuning, detailed
documentation and so forth.
Ensuring reproducibility unfortunately however only solves parts of the problems we face
with offline experimentation. Let us assume that for a given paper the research methodology is
correct and that all resources are shared for reproducibility. We may still need to ask what we can
conclude from an offline evaluation that reports a few percentages of accuracy improvements over
previously existing models on three academic datasets. Given the above-mentioned observations of
an offline-online gap, probably not too much. Industry reports from companies like Netflix and
Coveo mention the limited predictive power of offline tests, sometimes indicating that practitioners
often are “shooting in the dark” [36, 75, 76] in only a slightly informed way. Clearly, there are
ways in which offline experiments can be helpful. Such experiments might be used to rule out
models that perform very poorly offline. Furthermore, one can exploit offline tests to analyze certain
characteristics of generated recommendations, e.g., in terms of diversity or popularity bias, and
then decide which model might suit the given application use case best.
Overall, despite the apparent limitations of offline experiments, still most of the efforts of the
community are directed at finding models that lead to small accuracy increases on a limited
collection of datasets. Unfortunately, there are no signs of a crisis in the sense of a ‘turning point’
yet [16], where we as a community reconsider what kind of problems we address and how we
make contributions that matter with the goal to impact the real-world in a positive manner.

3.2 Challenges and Barriers of RS4Good Research

3.2.1 Methodological Challenges. When reconsidering the RS4Good use cases discussed in Sec-
tion 2, we can observe that not many of them can be addressed with traditional offline evaluations.
As briefly indicated above, many use cases necessitate a research approach that is driven by the
following requirements.
(1) Interdisciplinary: Many of the discussed RS4Good use cases require a deep understanding of
the particularities of a given domain. Research in the healthcare domain may be considered a
prime example here, which should not rely on a layman’s understanding of real-world issues
of a computer scientist. As a result, research in various RS4Good domains requires either
interdisciplinary collaboration with experts from other domains or at least an in-depth
understanding of the idiosyncrasies the domain.
(2) Human-in-the-Loop and Impact-Oriented Evaluation: The true value of RS4Good research
for different stakeholders in many cases cannot be assessed through abstract computational
Recommender Systems for Good (RS4Good): Survey of Use Cases and a Call to Action for Research that Matters 1:11

measures like precision and recall. We may consider a recommendation-based health inter-
vention or the evaluation of a novel recommendation approach in the e-learning domain
as examples. In many of these cases, studies require humans in the loop in the evaluation
process and measures (e.g., degree of health improvement or learning success) that measure
the impact of the interventions or recommendations on users.
(3) Longitudinal Perspective: Considering again the examples of health interventions or e-
learning applications, the effects of recommendations may only become visible over a longer
evaluation horizon, e.g., after several weeks of treatments or at the end of an academic year.
Likewise, if we think of using recommendations as nudges [50] towards a desired behavior,
e.g., in terms of healthy eating patterns, we must ensure that such behavior changes are
sustainable.
Certainly, not all RS4Good use cases may necessarily face all of these methodological challenges
in parallel. There may always be certain types of research without involving humans, for instance,
works that are based on simulation approaches like the study in the tourism domain by Piliponyte
et al. [68] mentioned above.10 In general, however, research in RS4Good domains might often be
much slower and effortful in terms of the required research design and the execution of experiments
than traditional offline evaluations. Moreover, research on RS4Good use cases, due to the need
to consider domain specifics, may always face the criticism of focusing on a too narrow problem
setting, and the generalizability of the findings may be questioned. As a result RS4Good research
may suffer from a basic dilemma of recommender systems research. On the one hand, researchers
in computer science generally strive to develop generalizable solutions, e.g., algorithms, that work
for a variety of use cases. On the other hand, there is a concrete danger that we overgeneralize and
make conclusions regarding the effectiveness of a new algorithm in practice based on a limited set
of experiments and using a specific set of datasets and abstract, domain-agnostic computational
measures.
3.2.2 Research Culture Barriers. The discussed methodological challenges in most cases will
probably imply a slower pace of research and publication. Studies involving human subjects often
take significant amounts of time, starting with the time needed to recruit suitable participants.
Furthermore, working in interdisciplinary teams, getting approval by ethics boards, understanding
domain specifics and collaborating with different institutions (e.g., hospitals and doctors) can be
time-consuming and effortful as well. Finally, longitudinal studies by definition require a longer
observation horizon. Such a slower pace of publication however seems largely incompatible with
the fast-paced publication culture in computer science.
For the area of machine learning, Turing-award laureate Yoshua Bengio therefore suggests to
entirely re-think the publication process, which nowadays is almost focusing on conferences [6].
This publication process is highly competitive, fast-paced, and driven by deadlines. As a result,
Bengio speculates that for many papers that are submitted to conferences and which are eventually
published, there might not have been enough time to thoroughly check them for errors. Furthermore,
such papers often appear incremental and may lack depth. Ultimately, he advocates for a ‘slow
science’11 approach, and envisions a model where research works are rather submitted to fast-
turnaround journals, allowing for multiple iterations for improvements, and conference program
committees then select already accepted works for presentation and discussion at conferences. This
way, the role of conferences can be change into a place for connecting with other people and for
highlighting and discussing the “best and most important ideas” within the scientific community.
10 Conversely, research works that are based on user studies or longitudinal developments, e.g., simulations in the e-commerce

domain [34, 104], are not necessarily “for good”.

11 [Link]
1:12 Jannach et al.

Following the discussions above, most RS4Good research will require and benefit from a ‘slow
science’ approach. We are aware that the publication process will not rapidly change in the near
future. However, we believe that we as community should not shy away from addressing important
and from a societal viewpoint relevant research questions just “because they are difficult.”

4 SUMMARY AND NEXT STEPS

With this opinionated essay, we would like to encourage scholars in the field of recommender
systems to more often address problems of societal relevance that matter in practice. Our brief
survey of examples of existing RS4Good research demonstrates that there are plenty of oppor-
tunities and open challenges in various areas from healthcare to education and individual and
societal development. We however find that successful RS4Good research requires a fundamental
methodological shift in how we do research, both in terms of our methodology and our publication
models.
To foster a research culture that values societal impact, conference organizers could, for instance,
consider introducing awards specifically for contributions to societal good. While traditional
recognitions, such as Best Paper or Test of Time awards, celebrate technological rigor and lasting
relevance in the community, awards for societal impact could highlight research that achieves
meaningful and positive change. Such recognitions could play an important role in inspiring the
community to pursue impactful RS4Good problems, aligning the achievements of the community
with societal needs.
We are aware that such changes may take their time. In our view, raising awareness in the
community of the issues outlined in this essay represents an important first step. Furthermore, the
organization of topical workshops, seminars, conference tracks and journal special issues may help
to increase the attention and refocus future efforts in recommender systems research. Generally,
we are confident that the community will soon consider personalized ranking algorithms to be
a commodity—also due to the increasing capabilities of Large Language Models—and consider
the ranking problem to be solved to the extent that a sufficient number of domain-independent
well-performing algorithmic solutions are already available by now. Such a development would
then allow us as a community to focus on RS4Good research that has the goal to contribute to a
positive development of society.

REFERENCES
[1] Mawulolo K. Ameko, Miranda L. Beltzer, Lihua Cai, Mehdi Boukhechba, Bethany A. Teachman, and Laura E.
Barnes. 2020. Offline Contextual Multi-armed Bandits for Mobile Health Interventions: A Case Study on Emotion
Regulation. In Proceedings of the 14th ACM Conference on Recommender Systems (RecSys ’20). 249–258. https:
//[Link]/10.1145/3383313.3412244
[2] Timothy G. Armstrong, Alistair Moffat, William Webber, and Justin Zobel. 2009. Improvements That Don’t Add
up: Ad-hoc Retrieval Results since 1998. In International Conference on Information and Knowledge Management,
Proceedings. ACM Press, New York, New York, USA, 601–610. [Link]
[3] Nicolas Balacheff, Sten Ludvigsen, Ton Jong, Ard Lazonder, and Sally Barnes (Eds.). 2009. Technology-Enhanced
Learning - Principles and Products. Springer.
[4] Christine Bauer, Chandni Bagchi, Olusanmi A. Hundogan, and Karin van Es. 2024. Where Are the Values? A
Systematic Literature Review on News Recommender Systems. ACM Trans. Recomm. Syst. 2, 3 (2024). https:
//[Link]/10.1145/3654805
[5] Jöran Beel and Stefan Langer. 2015. A Comparison of Offline Evaluations, Online Evaluations, and User Studies in
the Context of Research-Paper Recommender Systems. In Research and Advanced Technology for Digital Libraries:
19th International Conference on Theory and Practice of Digital Libraries, TPDL 2015. 153–168.
[6] Yoshua Bengio. 2020. Time to rethink the publication process in machine learning. [Link]
02/26/time-to-rethink-the-publication-process-in-machine-learning/
[7] Efthimios Bothos, Dimitris Apostolou, and Gregoris Mentzas. 2013. Choice architecture for environmentally sustain-
able urban mobility. In CHI ’13 Extended Abstracts on Human Factors in Computing Systems (CHI EA ’13). 1503–1508.
Recommender Systems for Good (RS4Good): Survey of Use Cases and a Call to Action for Research that Matters 1:13

[Link]
[8] Iván Cantador, Alejandro Bellogín, María E. Cortés-Cediel, and Olga Gil. 2017. Personalized recommendations in
e-participation: offline experiments for the ’Decide Madrid’ platform. In Proceedings of the International Workshop on
Recommender Systems for Citizens. [Link]
[9] Emanuele Cavenaghi, Gabriele Sottocornola, Fabio Stella, and Markus Zanker. 2023. A Systematic Study on Re-
producibility of Reinforcement Learning in Recommendation Systems. ACM Trans. Recomm. Syst. 1, 3 (2023).
[Link]
[10] Jiawei Chen, Hande Dong, Xiang Wang, Fuli Feng, Meng Wang, and Xiangnan He. 2023. Bias and Debias in
Recommender System: A Survey and Future Directions. ACM Trans. Inf. Syst. 41, 3 (2023). [Link]
3564284
[11] Kei Long Cheung, Dilara Durusu, Xincheng Sui, and Hein de Vries. 2019. How recommender systems could support
and enhance computer-tailored digital health programs: A scoping review. Digital Health 5 (2019).
[12] Benjamin Richard Clark, Kristine Grivcova, Polina Proutskova, and Duncan Martin Walker. 2023. Personalised
Recommendations for the BBC iPlayer: Initial approach and current challenges. In Proceedings of the 17th ACM
Conference on Recommender Systems (RecSys ’23). 551–553.
[13] María E. Cortés-Cediel, Iván Cantador, and Olga Gil. 2017. Recommender systems for e-governance in smart cities:
state of the art and research opportunities. In Proceedings of the International Workshop on Recommender Systems for
Citizens. [Link]
[14] Paul Covington, Jay Adams, and Emre Sargin. 2016. Deep Neural Networks for YouTube Recommendations. In
Proceedings RecSys ’16. 191–198.
[15] Paolo Cremonesi, Franca Garzotto, and Roberto Turrin. 2012. Investigating the Persuasion Potential of Recommender
Systems from a Quality Perspective: An Empirical Study. Transactions on Interactive Intelligent Systems 2, 2 (2012),
11:1–11:41.
[16] Paolo Cremonesi and Dietmar Jannach. 2021. Progress in recommender systems research: Crisis? What crisis? AI
Magazine 42, 3 (2021), 43–54.
[17] Andre Paulino de Lima, Laurentino Augusto Dantas, Marcelo Garcia Manzato, Maria Pimentel, Brunela Orlandi, and
Paula Castro. 2021. An Interpretable Recommendation Model for Gerontological Care. In Proceedings of the 15th
ACM Conference on Recommender Systems (RecSys ’21). 620–626.
[18] Yashar Deldjoo, Dietmar Jannach, Alejandro Bellogin, Alessandro Difonzo, and Dario Zanzonelli. 2023. Fairness in
Recommender Systems: Research Landscape and Future Directions. User Modeling and User-Adapted Interaction 34, 1
(2023), 59–108.
[19] Yashar Deldjoo, Tommaso Di Noia, and Felice Antonio Merra. 2021. A Survey on Adversarial Recommender
Systems: From Attack/Defense Strategies to Generative Adversarial Networks. ACM Comput. Surv. 54, 2 (2021).
[Link]
[20] Nicholas Diakopoulos, Christoph Trattner, Dietmar Jannach, Irene Costera Meijer, and Enrico Motta. 2024. Leveraging
Professional Ethics for Responsible AI. Commun. ACM 67, 2 (2024), 19–21.
[21] Belen Diaz-Agudo, Alessio Bosca, Andrea Bolioli, Guilermo Jimenez Jimenez Diaz, Tsvi Kuflik, and Alan J. Wecker.
2021. Towards Personalized Social Recommendations for Cultural Heritage Activities: Methods and technology to
enable cohesive and inclusive recommendations. In Adjunct Proceedings of the 29th ACM Conference on User Modeling,
Adaptation and Personalization (UMAP ’21). 199–202. [Link]
[22] Tim Donkers and Jürgen Ziegler. 2021. The Dual Echo Chamber: Modeling Social Media Polarization for Interventional
Recommending. In Proceedings of the 15th ACM Conference on Recommender Systems (RecSys ’21). 12–22. https:
//[Link]/10.1145/3460231.3474261
[23] Erik Duval, Mike Sharples, and Rosamund Sutherland. 2017. Technology Enhanced Learning. Springer.
[24] Michael D. Ekstrand, Anubrata Das, Robin Burke, and Fernando Diaz. 2022. Fairness in Information Access Systems.
Found. Trends Inf. Retr. 16, 1-2 (2022), 1–177.
[25] Michael D. Ekstrand, F. Maxwell Harper, Martijn C. Willemsen, and Joseph A. Konstan. 2014. User Perception
of Differences in Recommender Algorithms. In Proceedings of the 8th ACM Conference on Recommender Systems.
161–168.
[26] Mehdi Elahi, Dietmar Jannach, Lars Skjærven, Erik Knudsen, Helle Sjøvaag, Kristian Tolonen, Øyvind Holmstad, Igor
Pipkin, Eivind Throndsen, Agnes Stenbom, Eivind Fiskerud, Adrian Oesch, Loek Vredenberg, and Christoph Trattner.
2021. Towards Responsible Media Recommendation. AI and Ethics 2, 1 (2021), 103–114.
[27] David Elsweiler, Christoph Trattner, and Morgan Harvey. 2017. Exploiting Food Choice Biases for Healthier Recipe
Recommendation. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in
Information Retrieval (SIGIR ’17). 575–584.
[28] Vanesa Espín, María V. Hurtado, and Manuel Noguera. 2016. Nutrition for Elder Care: a nutritional semantic
recommender system for the elderly. Expert Systems 33, 2 (2016), 201–210.
1:14 Jannach et al.

[29] Alexander Felfernig, Manfred Wundara, Thi Ngoc Trang Tran, Seda Polat-Erdeniz, Sebastian Lubos, Merfat El Mansi,
Damian Garber, and Viet Man Le. 2023. Recommender systems for sustainability: overview and research issues.
Frontiers in Big Data 6 (2023).
[30] Maurizio Ferrari Dacrema, Simone Boglio, Paolo Cremonesi, and Dietmar Jannach. 2021. A Troubling Analysis of
Reproducibility and Progress in Recommender Systems Research. ACM Transactions on Information Systems 39, 2
(2021).
[31] Maurizio Ferrari Dacrema, Paolo Cremonesi, and Dietmar Jannach. 2019. Are We Really Making Much Progress? A
Worrying Analysis of Recent Neural Recommendation Approaches. In Proceedings of the 2019 ACM Conference on
Recommender Systems (RecSys 2019). Copenhagen.
[32] Andres Ferraro, Gustavo Ferreira, Fernando Diaz, and Georgina Born. 2022. Measuring Commonality in Recommen-
dation of Cultural Content: Recommender Systems to Enhance Cultural Citizenship. In Proceedings of the 16th ACM
Conference on Recommender Systems (RecSys ’22). 567–572. [Link]
[33] Florent Garcin, Boi Faltings, Olivier Donatsch, Ayar Alazzawi, Christophe Bruttin, and Amr Huber. 2014. Offline
and online evaluation of news recommender systems at [Link]. In Proceedings of the 8th ACM Conference on
Recommender Systems. 169–176.
[34] Nada Ghanem, Stephan Leitner, and Dietmar Jannach. 2022. Balancing Consumer and Business Value of Recommender
Systems: A Simulation-based Analysis. E-Commerce Research and Applications 55 (2022), 101195.
[35] Kambiz Ghazinour, Stan Matwin, and Marina Sokolova. 2013. Monitoring and recommending privacy settings in
social networks. In Proceedings of the Joint EDBT/ICDT 2013 Workshops (EDBT ’13). 164–168.
[36] Carlos A. Gomez-Uribe and Neil Hunt. 2015. The Netflix Recommender System: Algorithms, Business Value, and
Innovation. Transactions on Management Information Systems 6, 4 (2015), 13:1–13:19.
[37] Andreas Grün and Xenija Neufeld. 2023. Transparently Serving the Public: Enhancing Public Service Media Values
through Exploration. In Proceedings of the 17th ACM Conference on Recommender Systems (RecSys ’23). 1045–1048.
[38] Amelie Gyrard and Karima Boudaoud. 2022. Interdisciplinary IoT and Emotion Knowledge Graph-Based Recommen-
dation System to Boost Mental Health. Applied Sciences 12, 19 (2022).
[39] Amelie Gyrard and Amit Sheth. 2020. IAMHAPPY: Towards an IoT knowledge-based cross-domain well-being
recommendation system for everyday happiness. Smart Health 15 (2020), 100083.
[40] Stephan Hammer, Andreas Seiderer, Elisabeth André, Thomas Rist, Sofia Kastrinaki, Charline Hondrou, Amaryllis
Raouzaiou, Kostas Karpouzis, and Stefanos Kollias. 2015. Design of a lifestyle recommender system for the elderly:
requirement gatherings in Germany and Greece. In Proceedings of the 8th ACM International Conference on PErvasive
Technologies Related to Assistive Environments (PETRA ’15). [Link]
[41] Madita Herpich, Thomas Rist, Andreas Seiderer, and Elisabeth André. 2017. Towards a Gamified Recommender
System for the Elderly. In Proceedings of the 2017 International Conference on Digital Health (DH ’17). 211–215.
[Link]
[42] Balázs Hidasi and Ádám Tibor Czapp. 2023. The Effect of Third Party Implementations on Reproducibility. In
Proceedings of the 17th ACM Conference on Recommender Systems (RecSys ’23). 272–282. [Link]
3604915.3609487
[43] Balázs Hidasi and Ádám Tibor Czapp. 2023. Widespread Flaws in Offline Evaluation of Recommender Systems. In
Proceedings of the 17th ACM Conference on Recommender Systems, RecSys 2023. 848–855. [Link]
3604915.3608839
[44] Santiago Hors-Fraile, Francisco J. Núñez-Benjumea, Laura Carrasco Hernández, Francisco Ortega Ruiz, and Luis
Fernández-Luque. 2016. Design of two combined health recommender systems for tailoring messages in a smoking
cessation app. In Workshop on Engendering Health with RecSys co-located with 10th ACM Conference on Recommender
Systems (ACM RecSys 2016).
[45] Dietmar Jannach and Michael Jugovac. 2019. Measuring the business value of recommender systems. ACM Transactions
on Management Information Systems (TMIS) 10, 4 (2019), 1–23.
[46] Dietmar Jannach and Lukas Lerche. 2017. Offline Performance vs. Subjective Quality Experience: A Case Study in
Video Game Recommendation. In ACM Symposium on Applied Computing (ACM SAC 2017).
[47] Dietmar Jannach, Gabriel De Souza P. Moreira, and Even Oldridge. 2020. Why Are Deep Learning Models Not
Consistently Winning Recommender Systems Competitions Yet?. In ACM RecSys Challenge Workshop. Online.
[48] Dietmar Jannach and Markus Zanker. 2024. A survey on intent-aware recommender systems. ACM Transactions on
Recommender Systems (2024).
[49] Dietmar Jannach, Markus Zanker, Mouzhi Ge, and Marian Gröning. 2012. Recommender Systems in Computer
Science and Information Systems - A Landscape of Research. In 13th International Conference on Electronic Commerce
and Web Technologies (EC-Web 2012). Vienna, Austria, 76–87. [Link]
[50] Mathias Jesse and Dietmar Jannach. 2021. Digital Nudging with Recommender Systems: Survey and Future Directions.
Computers and Human Behavior Reports 3 (2021).
Recommender Systems for Good (RS4Good): Survey of Use Cases and a Call to Action for Research that Matters 1:15

[51] Aleksandra Klašnja-Milićević, Mirjana Ivanović, and Alexandros Nanopoulos. 2015. Recommender systems in
e-learning environments: a survey of the state-of-the-art and possible extensions. Artif. Intell. Rev. 44, 4 (2015),
571–604. [Link]
[52] Bart P. Knijnenburg, Saadhika Sivakumar, and Daricia Wilkinson. 2016. Recommender Systems for Self-Actualization.
In Proceedings of the 10th ACM Conference on Recommender Systems (RecSys ’16). 11–14. [Link]
2959100.2959189
[53] Bart Piet Knijnenburg, Martijn C. Willemsen, and Ron Broeders. 2014. Smart sustainability through system satisfaction:
Tailored preference elicitation for energy-saving recommenders. In 20th Americas Conference on Information Systems,
AMCIS 2014.
[54] Konstantinos G. Liakos, Patrizia Busato, Dimitrios Moshou, Simon Pearson, and Dionysis Bochtis. 2018. Machine
Learning in Agriculture: A Review. Sensors 18, 8 (2018). [Link]
[55] Weiwen Liu, Jun Guo, Nasim Sonboli, Robin Burke, and Shengyu Zhang. 2019. Personalized fairness-aware re-
ranking for microlending. In Proceedings of the 13th ACM Conference on Recommender Systems (RecSys ’19). 467–471.
[Link]
[56] Feng Lu, Anca Dumitrache, and David Graus. 2020. Beyond Optimizing for Clicks: Incorporating Editorial Values in
News Recommendation. In Proceedings of the 28th ACM Conference on User Modeling, Adaptation and Personalization
(UMAP ’20). 145–153. [Link]
[57] Malte Ludewig, Sara Latifi, Noemi Mauro, and Dietmar Jannach. 2021. Empirical Analysis of Session-Based Recom-
mendation Algorithms. User Modeling and User-Adapted Interaction 31, 1 (2021), 149–181.
[58] Andrii Maksai, Florent Garcin, and Boi Faltings. 2015. Predicting Online Performance of News Recommender Systems
Through Richer Evaluation Metrics. In Proceedings of the 9th ACM Conference on Recommender Systems. 179–186.
[59] Noemi Mauro, Liliana Ardissono, Stefano Cocomazzi, and Federica Cena. 2022. Using consumer feedback from
location-based services in PoI recommender systems for people with autism. Expert Systems with Applications 199
(2022), 116972.
[60] Sean M. McNee, Istvan Albert, Dan Cosley, Prateep Gopalkrishnan, Shyong K. Lam, Al Mamunur Rashid, Joseph A.
Konstan, and John Riedl. 2002. On the Recommending of Citations for Research Papers. In Proceedings of the 2002
ACM Conference on Computer Supported Cooperative Work (CSCW ’02). 116–125.
[61] Aleksandr Milogradskii, Oleg Lashinin, Alexander P, Marina Ananyeva, and Sergey Kolesnikov. 2024. Revisiting BPR:
A Replicability Study of a Common Recommender System Baseline. In Proceedings of the 18th ACM Conference on
Recommender Systems (RecSys ’24). 267–277. [Link]
[62] Ashlee Milton, Michael Green, Adam Keener, Joshua Ames, Michael D Ekstrand, and Maria Soledad Pera. 2019.
StoryTime: eliciting preferences from children for book recommendations. In Proceedings of the 13th ACM Conference
on Recommender Systems (RecSys ’19). 544–545. [Link]
[63] Harri Oinas-Kukkonen. 2010. Behavior Change Support Systems: A Research Model and Agenda. In Persuasive
Technology. 4–14.
[64] Kiemute Oyibo, Ifeoma Adaji, Rita Orji, and Julita Vassileva. 2018. What Drives the Perceived Credibility of Mobile
Websites: Classical or Expressive Aesthetics?. In Human-Computer Interaction. Interaction in Context. 576–594.
[65] Eli Pariser. 2011. The Filter Bubble: What the Internet Is Hiding from You. The Penguin Group.
[66] Krupa Patel and Hiren B. Patel. 2023. Machine Learning and Recommendation System in Agriculture: A Survey and
Possible Extensions. In Innovations in Computer Science and Engineering, H. S. Saini, Rishi Sayal, A. Govardhan, and
Rajkumar Buyya (Eds.). 665–674.
[67] Gourab K Patro, Abhijnan Chakraborty, Ashmi Banerjee, and Niloy Ganguly. 2020. Towards Safety and Sustain-
ability: Designing Local Recommendations for Post-pandemic World. In Proceedings of the 14th ACM Conference on
Recommender Systems (RecSys ’20). 358–367. [Link]
[68] Greta Piliponyte, David Massimo, and Francesco Ricci. 2023. The Impact of Personalised Advertisement Campaigns on
Tourist Choices in South Tyrol: A Sustainable Tourism Perspective. In Adjunct Proceedings of the 31st ACM Conference
on User Modeling, Adaptation and Personalization, UMAP 2023,. 100–103. [Link]
[69] Payam Pourashraf and Bamshad Mobasher. 2022. Using Recommender Systems to Help Revitalize Local News. In
Adjunct Proceedings of the 30th ACM Conference on User Modeling, Adaptation and Personalization (UMAP ’22 Adjunct).
80–84. [Link]
[70] Lara Quijano-Sánchez, Iván Cantador, María E. Cortés-Cediel, and Olga Gil. 2020. Recommender systems for smart
cities. Information Systems 92 (2020), 101545.
[71] Wajid Rafique, Abdelhakim Senhaji Hafid, and Junaid Qadir. 2023. Developing smart city services using intent-aware
recommendation systems: A survey. Transactions on Emerging Telecommunications Technologies 34, 4 (April 2023).
[Link]
[72] Steffen Rendle, Walid Krichene, Li Zhang, and John Anderson. 2020. Neural Collaborative Filtering vs. Matrix
Factorization Revisited. In Fourteenth ACM Conference on Recommender Systems (RecSys ’20). 240–248.
1:16 Jannach et al.

[73] Steffen Rendle, Walid Krichene, Li Zhang, and Yehuda Koren. 2022. Revisiting the Performance of iALS on Item
Recommendation Benchmarks. In RecSys ’22: Sixteenth ACM Conference on Recommender Systems. 427–435. https:
//[Link]/10.1145/3523227.3548486
[74] Darius A. Rohani, Andrea Quemada Lopategui, Nanna Tuxen, Maria Faurholt-Jepsen, Lars V. Kessing, and Jakob E.
Bardram. 2020. MUBS: A Personalized Recommender System for Behavioral Activation in Mental Health. In
Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (CHI ’20). 1–13.
[75] David Rohde. 2024. Position Paper: Why the Shooting in the Dark Method Dominates Recommender Systems
Practice; A Call to Abandon Anti-Utopian Thinking. arXiv:2402.02152 [[Link]] [Link]
[76] David Rohde. 2024. Why the Shooting in the Dark Method Dominates Recommender Systems Practice. In Proceedings
of the 18th ACM Conference on Recommender Systems (RecSys ’24). 847–849. [Link]
[77] A. Said and A. Bellogín. 2014. Comparative Recommender System Evaluation: Benchmarking Recommendation
Frameworks. In RecSys 2014 - Proceedings of the 8th ACM Conference on Recommender Systems. [Link]
1145/2645710.2645746
[78] Rim Ben Salem, Esma Aïmeur, and Hicham Hage. 2021. The Privacy versus Disclosure Appetite Dilemma: Mitigation
by Recommendation. In Proceedings of the Second Workshop on Online Misinformation- and Harm-Aware Recommender
Systems co-located with 15th ACM Conference on Recommender Systems (RecSys 2021) (CEUR Workshop Proceedings,
Vol. 3012), Antonela Tommasel, Daniela Godoy, and Arkaitz Zubiaga (Eds.). 14–32.
[79] Odnan Ref Sanchez, Ilaria Torre, Yangyang He, and Bart P. Knijnenburg. 2020. A recommendation approach for
user privacy preferences in the fitness domain. User Model. User Adapt. Interact. 30, 3 (2020), 513–565. https:
//[Link]/10.1007/S11257-019-09246-3
[80] Andrés Segura-Tinoco and Iván Cantador. 2021. On the Extraction and Use of Arguments in Recommender Systems:
A Case Study in the E-participation Domain (Long paper). In Joint Workshop Proceedings of the 3rd Edition of
Knowledge-aware and Conversational Recommender Systems (KaRS) and the 5th Edition of Recommendation in Complex
Environments (ComplexRec) (CEUR Workshop Proceedings, Vol. 2960). [Link].
[81] Faisal Shehzad and Dietmar Jannach. 2023. Everyone’s a Winner! On Hyperparameter Tuning of Recommendation
Models. In 17th ACM Conference on Recommender Systems.
[82] Faisal Shehzad and Dietmar Jannach. 2024. Performance Comparison of Session-based Recommendation Algorithms
based on GNNs. In 46th European Conference on Information Retrieval (ECIR).
[83] Mingdan Si and Qingshan Li. 2020. Shilling attacks against collaborative recommender systems: a review. Artif Intell
Rev 53 (2020), 291–319. [Link]
[84] Annelien Smets, Nils Walravens, and Pieter Ballon. 2020. Designing Recommender Systems for the Common
Good. In Adjunct Publication of the 28th ACM Conference on User Modeling, Adaptation and Personalization. 276–278.
[Link]
[85] Barry Smyth, Aonghus Lawlor, Jakim Berndsen, and Ciara Feely. 2022. Recommendations for marathon runners:
on the application of recommender systems and machine learning to support recreational marathon runners. User
Model. User Adapt. Interact. 32, 5 (2022), 787–838.
[86] Giuseppe Spillo, Allegra De Filippo, Cataldo Must, Michela Milano, and Giovanni Semeraro. 2024. Towards Green
Recommender Systems: Investigating the Impact of Data Reduction on Carbon Footprint and Algorithm Performances.
In Proceedings ACM RecSys ’24.
[87] Alain Starke, Ayoub El Majjodi, and Christoph Trattner. 2022. Boosting Health? Examining the Role of Nutrition
Labels and Preference Elicitation Methods in Food Recommendation. In Proceedings of the 9th Joint Workshop on
Interfaces and Human Decision Making for Recommender Systems co-located with 16th ACM Conference on Recommender
Systems (RecSys 2022), Vol. 3222. 67–84.
[88] Alain D. Starke, Martijn C. Willemsen, and Chris Snijders. 2021. Using Explanations as Energy-Saving Frames: A
User-Centric Recommender Study. In Adjunct Proceedings of the 29th ACM Conference on User Modeling, Adaptation
and Personalization (UMAP ’21). 229–237. [Link]
[89] Harald Steck, Linas Baltrunas, Ehtsham Elahi, Dawen Liang, Yves Raimond, and Justin Basilico. 2021. Deep Learning for
Recommender Systems: A Netflix Case Study. AI Mag. 42, 3 (2021), 7–18. [Link]
[90] Lars Steinert, Fynn Linus Kölling, Felix Putze, Dennis Küster, and Tanja Schultz. 2022. Evaluation of an Engagement-
Aware Recommender System for People with Dementia. In Proceedings of the 30th ACM Conference on User Modeling,
Adaptation and Personalization (UMAP ’22). 89–98.
[91] Jonathan Stray. 2022. Designing recommender systems to depolarize. First Monday 27, 5 (May 2022). https:
//[Link]/10.5210/fm.v27i5.12604
[92] Jonathan Stray, Alon Halevy, Parisa Assar, Dylan Hadfield-Menell, Craig Boutilier, Amar Ashar, Chloe Bakalar,
Lex Beattie, Michael Ekstrand, Claire Leibowicz, Connie Moon Sehat, Sara Johansen, Lianne Kerlin, David Vickrey,
Spandana Singh, Sanne Vrijenhoek, Amy Zhang, McKane Andrus, Natali Helberger, Polina Proutskova, Tanushree
Mitra, and Nina Vasan. 2024. Building Human Values into Recommender Systems: An Interdisciplinary Synthesis.
Recommender Systems for Good (RS4Good): Survey of Use Cases and a Call to Action for Research that Matters 1:17

ACM Trans. Recomm. Syst. 2, 3 (2024). [Link]

[93] Emily Sullivan, Dimitrios Bountouridis, Jaron Harambam, Shabnam Najafian, Felicia Loecherbach, Mykola
Makhortykh, Domokos Kelen, Daricia Wilkinson, David Graus, and Nava Tintarev. 2019. Reading News with
a Purpose: Explaining User Profiles for Self-Actualization. In Adjunct Publication of the 27th Conference on User
Modeling, Adaptation and Personalization. 241–245.
[94] Zhu Sun, Di Yu, Hui Fang, Jie Yang, Xinghua Qu, Jie Zhang, and Cong Geng. 2020. Are we evaluating rigorously?
benchmarking recommendation for reproducible evaluation and fair comparison. In Fourteenth ACM Conference on
Recommender Systems. 23–32.
[95] Thi Ngoc Trang Tran, Alexander Felfernig, Christoph Trattner, and Andreas Holzinger. 2021. Recommender systems
in the healthcare domain: state-of-the-art and research issues. Journal of Intelligent Information Systems 57, 1 (2021),
171–201. [Link]
[96] Dhruv Deepak Upadhyay, Vaibhav Pandey, Nitish Nag, and Ramesh Jain. 2020. Personalized User Modelling for Sleep
Insight. In Proceedings of the 1st International Workshop on Human-Centric Multimedia Analysis. 13–20.
[97] Lee Valentine, Simon D’Alfonso, and Reeva Lederman. 2023. Recommender systems for mental health apps: advantages
and ethical challenges. AI & SOCIETY 38, 4 (2023), 1627–1638.
[98] Tobias Vente, Lukas Wegmeth, Alan Said, and Joeran Beel. 2024. From Clicks to Carbon: The Environmental Toll of
Recommender Systems. In Proceedings of the 18th ACM Conference on Recommender Systems (Reproducibility Track).
[99] Shoujin Wang, Xiuzhen Zhang, Yan Wang, and Francesco Ricci. 2024. Trustworthy Recommender Systems. ACM
Trans. Intell. Syst. Technol. 15, 4 (2024). [Link]
[100] Yifan Wang, Weizhi Ma, Min Zhang, Yiqun Liu, and Shaoping Ma. 2023. A Survey on the Fairness of Recommender
Systems. ACM Trans. Inf. Syst. 41, 3 (2023). [Link]
[101] Daricia Wilkinson, Saadhika Sivakumar, Pratitee Sinha, and Bart P. Knijnenburg. 2017. Testing a Recommender
System for Self-Actualization. In Proceedings of the Second Workshop on Engineering Computer-Human Interaction in
Recommender Systems co-located with the 9th ACM SIGCHI Symposium on Engineering Interactive Computing Systems.
[102] David C. Wilson, Suzanne Leland, Kenneth Godwin, Andrew Baxter, Ashley Levy, Jamie Smart, Nadia Najjar, and
Jayakrishnan Andaparambil. 2009. SmartChoice: An Online Recommender System to Support Low-Income Families
in Public School Choice. AI Magazine 30, 2 (2009), 46–58. [Link]
[103] Jun Yu, Zhenzhong Kuang, Baopeng Zhang, Wei Zhang, Dan Lin, and Jianping Fan. 2018. Leveraging Content
Sensitiveness and User Trustworthiness to Recommend Fine-Grained Privacy Settings for Social Image Sharing. IEEE
Transactions on Information Forensics and Security 13, 5 (2018), 1317–1332.
[104] Jingjing Zhang, Gediminas Adomavicius, Alok Gupta, and Wolfgang Ketter. 2020. Consumption and Performance:
Understanding Longitudinal Dynamics of Recommender Systems via an Agent-Based Simulation Framework. Inf.
Syst. Res. 31, 1 (2020), 76–101. [Link]
[105] Yongfeng Zhang and Xu Chen. 2020. Explainable Recommendation: A Survey and New Perspectives. Foundations
and Trends® in Information Retrieval 14, 1 (2020), 1–101. [Link]