Thematic Semester Digital Humanities and Artificial intelligence
Final conference
Dates: dec.10-dec. 12, 2024
Place: IAP (amphi)-PSL Observatoire de Paris (Salle Cassini)
org. CNRS AISSAI
Free and open to public but registration mandatory until Dec. 6, 2024 here
Rationale
Computational Approaches including artificial intelligence have for some years now enriched the methods of digital humanities, for example for the analysis of the composition of pictorial works, the attribution of authorship for literary, scientific and journalistic texts, the automatic recognition of handwritten texts, the analysis of complex networks of knowledge circulation, etc. The aim of the AI/HN thematic semester is to create the conditions for pragmatic and prospective reflection on the practices and objects that emerge at the interface between Artificial Intelligence and Digital Humanities. In this crossing, classical and fundamental issues in humanities are renewed while shedding light on the new questions of artificial intelligence.
Thus, the question of the integration of expert knowledge, which in AI has dimensions concerning data sets, algorithmic architectures and mathematics, renews in the humanities the questions of the forms of erudition, the modalities of access to sources and the constitution of corpora. The humanities are thus led to a certain form of explicitation, via AIs, of operations that often remain otherwise in the domain of the talent or intuition of individual scholars. This effort of transparency and explicitation builds in return paths and methods for the integration of expert knowledge in AI.
Similarly, the question of explicability also affects the data, algorithmic and mathematical dimensions of AI in a global way while encountering fundamental questions in digital humanities. For example, the relations between micro and macro levels of analysis in the humanities, or between emic and etic categories are profoundly transformed by AI algorithms, which weave new links at the heart of these essential polarities in the humanities. In return, the necessarily critical approach of the humanities, where a result only makes sense if the approach that produces it can be put in relation to other results and other methods, contributes to the concrete construction of explicability and transparency for these algorithms.
New types of interdisciplinary research collectives are being built around these emerging research themes in France, Europe and internationally. For the humanities, this corresponds to a long-term trend in which data analysis and acquisition methods are increasingly based on natural and computational sciences. For artificial intelligence, this corresponds to the need to confront real and complex data sets in order to meet, for instance, the challenges of explicability and the integration of expert knowledge. The purpose of the IA/HN thematic semester is also to accompany and contribute to structuring the constitution of these new research collectives at national and international level.
Participants
Mélanie Walsh (U. of Washington, Seattle), Anna Preus (U. of Washington, Seattle), Julien Schuh (U. Paris Nanterre), Adam Faci (Huma-Num), Olga Seminck (PSL-ENS), Ludovic Montcla (INSA), Valérie Gouet (IGN), Remi Petitpierre (EPFL) , Nathalie Abadie (IGN), Mathieu Aubry (ENPC), Victor Gysembergh (Sorbonne Université), Giles Bergel (Oxford), David Rabouin (Université Paris Cité), Livio de Luca (MAP), Federico Nurra (INHA), Nicolò Dell’Unto (LU, Sweden), Paola Derudas (LU, Sweden),Jean-Christophe Carius (INHA), Thomas Sagory (MAN), Anaïs Guillem (MAP), Kévin Réby (MAP), Daniel Wilson (Cambridge), Katherine McDonough (Cambridge)
Program
Tuesday, December 10, 2024
Amphi IAP
9h-9h30 welcome/registration
9h30-9h45 Introduction (COPIL)
9h45-10h30 Mathieu Aubry (ENPC), How to bridge historical and computer vision research?
10h30-11h15 Remi Petitpierre (EPFL), A computational and spatial exploration of historical cartography
11h15-11h45 coffee break
11h45-12h30 Nathalie Abadie (IGN), From images of old trade directories to a geohistorical knowledge graph – lessons learned from the SoDUCo project
Salle Cassini-Paris Observatory
12h30-14h00 lunch
Amphi IAP
14h00-14h45 Katherine McDonough (Lancaster University/The Alan Turing Institute) & Daniel Wilson (Cambridge) MapReader:Using computer vision to rethink how we look at maps as historical sources
14h45-15h30 Victor Gysembergh (Sorbonne Université) Lessons from the application of Machine learning to process multispectral images of the “New Apulcius” palimpsest
15h30-16h coffee break
16h00-16h45 Giles Bergel (Oxford), Between seeing and reading: machine learning and Dante’s Commedia (1470-1630) in print
16h45-17h30 David Rabouin (SPHere, Université Paris Cité, Philiumm, ERC adg 101020985) Leibniz’s Mathematical Manuscripts: successes, obstacles and prospects in automatic recognition
Wednesday, December 11, 2024
Amphi IAP
9h-9h30 welcome/registration
9h30-9h45 Introduction (COPIL)
9h45-10h30 Mélanie Walsh & Anna Preus (U. Of Washington, Seattle) What do LLMs “know” about poetic form?
10h30-11h15 Olga Semick (Lattice, CNRS-ENS-PSL) Remember to forget: Do AI-models remember the literature they were exposed to during training?
11h15-11h45 coffee break
11h45-12h30 Caroline Bassett (University of Cambridge) Silicon Beached?
Salle Cassini-Paris Observatory
12h30-14h00 lunch
Amphi IAP
14h00-14h45 Ludovic Moncla (INSA) Evaluation of Transformer Models (from BERT to GPT) for Geographic Information Recognition
14h45-15h30 Julien Schuh (Paris-Nanterre) & Adam Faci (Huma-Num) AI and model readers: automating the production of artificial interpretations
15h30-16h coffee break
16h00-17h00 Round table: What do LLMs really bring to literature studies?
-
- Teaching with LLMs (M. Walsh, The ‘LLMs for Humanists’ Project; C. Bassett on the humanities environment at Cambridge)
- General Discussion: Do we really learn something new with computational literary studies? How to ensure reproducibility?
19h00-21h Coktail, Salle Cassini
Thursday, December 12, 2024
Amphi IAP
9h30-9h45 welcome/registration
9h45-10h30 Livio de Luca (MAP), Notre-Dame de Paris: A Cathedral of Digital Data and Multidisciplinary Knowledge in Heritage Science
10h30-11h15 Federico Nurra (INHA), Nicolò Dell’Unto (LU, Sweden), Paola Derudas (LU, Sweden), From AIR to (AI)R: the use of LLM for interpreting archaeological excavation data
11h15-11h45 Coffee break
11h45-12h30 Jean-Christophe Carius (INHA), Art historical sources, machine intelligence and the research process (TBC)
Salle Cassini-Paris Observatory
12h30-14h00 lunch
Amphi IAP
14h-14h45 Thomas Sagory (MAN), AI at the Musée d’Archéologie National, case studies
14h45-15h30 Anaïs Guillem, Kévin Réby (MAP CNRS/MC), Large Language Models for Thesaurus Curation using Neuro-symbolic Artificial Intelligence
15h30-16h coffee break
16h00-16h45 Valérie Gouet (IGN), Organizing and spatializing geographic iconographic content
16h45-17h15 Final remarks (COPIL)
Abstracts
Mélanie Walsh & Anna Preus (Univ. of Whasington, Seattle) What do LLMs “know” about poetic form?
Large language models (LLMs) can now generate and recognize text in a wide range of styles and genres, including highly specialized, creative genres like poetry. But what do LLMs really know about poetry? What can they know about poetry? We develop a task to evaluate how well LLMs recognize a specific aspect of poetry—poetic form—for more than 20 forms and formal elements in the English language. Poetic form captures many different poetic features, including rhyme scheme, meter, and word or line repetition. We use this task to reflect on LLMs’ current poetic capabilities, as well as the challenges and pitfalls of creating Natural Language Processing (NLP) benchmarks for poetry and for other creative tasks. In particular, we use this task to audit and reflect on poems that are included in popular pretraining datasets and memorized by popular models. Our findings have implications for digital humanities and cultural analytics scholars, poetry experts, cultural heritage professionals, and NLP researchers.
Caroline Bassett (U. of Cambridge) Silicon Beached?
The explosion of synthetic writing produces a human writing explosion. At least we can say that attempts to assess, address and understand the stakes of recent developments in AI and natural language processing have produced a plethora of work (and words) exploring the new status of writing, often focussing on machine creativity, stochastic parroting, and simulation. Digital Humanities (DH) with the brief to investigate computational transformations of cultural forms and practices and to explore ways in which these may be researched has engaged with these developments in multiple ways – and has also been challenged by them. This is partly because DH hasn’t entirely escaped the sway of that conventional distinction between what is ‘digital’ and what is ‘humanities’ which rests on a binary whereby computation (on one side) faces the word (on the other). That this can’t get us far is made the more evident by the automation of the word itself. My question then is what may be productively produced if instead of focussing on ‘the humanities’ (versus ‘the computational’) attention is paid to the question of literature and automation. Working in this vein I return to earlier debates within English literature, around automatic writing, intentionality, meaning, and ‘theory’ or critique, which have recently been revived and ask what they might have to say about new forms of writing and their literary – or post-literary – potential.
Olga Seminck (Lattice, CNRS) Remember to forget: Do AI-models remember the literature they were exposed to during training?
Recent large language models (LLMs) like ChatGPT use vast text collections for training. While some models are open-source, their training data is often unknown, which raises questions about the models’ knowledge of literature. This is particularly relevant for digital humanities (DH) research, where these models are frequently employed. “Data archaeology” involves uncovering this training data by querying the models. In this presentation, I will discuss experiments inspired by Kent Chang et al.’s 2023 article, “Speak, Memory: An Archaeology of Books Known to ChatGPT/GPT-4.” They investigated the models’ knowledge using a “name cloze task.” This method tests whether the model can fill in missing character names in sentences. Their findings showed that the models recalled many books, a number of which are under copyright. We replicated these experiments with free LLMs in English and French and found similar but less pronounced memorization. Additionally, we performed supplementary experiments. For example, we examined the Olmo model (which has accessible training data) to understand memorization processes. Our experiments led us to identify several problems with the name cloze methodology, which is sensitive to errors. We conclude that the degree of memorization of literature is relatively low in LLMs. In a pilot experiment, we propose an alternative method that has the potential to yield more robust results.
Julien Schuh (Paris-Nanterre) & Adam Faci (Huma-Num) AI and model readers: automating the production of artificial interpretations
The various forms of remote reading (frequency analysis, bag of words, etc.) suffer from the same pitfalls: on the one hand, they form reductive interpretative models that are often implicit; on the other, they overwhelm local and contextual meanings by giving equal weight to forms (words, syntactic turns of phrase) in heterogeneous corpora that were not designed to be read monolithically. The use of LLMs can help overcome these limitations by producing local and situated interpretations in an automated way. This talk will present experiments with large language models to produce artificial interpretations, instructing them to read in roles or according to the principles of various large-scale interpretive communities. Rather than seeking to impose a dominant interpretation of corpora, the aim is to enrich the field of potential interpretations.
Ludovic Moncla (INSA) Evaluation of Transformer Models (from BERT to GPT) for Geographic Information Recognition
In this presentation, we will focus on the automatic recognition of named entities, nested named entities, nominal entities, and geographic information (spatial relationships and geographic coordinates). We will present a comparative study of different methods, with a particular focus on Transformer-based models, from BERT to GPT. The objective of this task is to extract and structure geographic information as a preliminary step for analysis tasks. In addition to evaluating automatic annotation methods, we will present the annotated dataset GeoEDdA, consisting of encyclopedic articles, which was used for training and evaluation phases. We will also present a case study on toponym resolution and mapping based on the generated data.
Valérie Gouet (IGN) Organizing and spatializing geographic iconographic content
In every country, there exists many collections of iconographic contents that depict the territory at different time periods and different scales, such as aerial views belonging to mapping agency surveys or terrestrial photographs taken by a photograph for illustrating a place or an event. They are usually hosted by GLAMs (Galleries, Libraries, Archives, Museums) or mapping agencies, making them scattered in silo and documented or indexed with various standards depending on the hosting institution. Yet they represent a rich heritage touching many sectors of society, e.g. environmental cartography, urban planning, historians and geographers modeling the evolution of the territory, sustainable tourism, sociologists investigating public spaces, media for investigation and engagement, etc. With the acceleration of open data policies aimed at promoting the circulation and valorization of public data, it becomes easier to find out about this content. To exploit and valorize them, the main challenges remain in organizing them optimally, across an institution and even between institutions, in order to access them, discover them and visualize them in ways that are relevant to the various users. In this talk, we will revisit the paradigms and solutions that exist with the objective of structuring and exploring this growing and rich digital(ized) source of documentation of our territories, from their spatialization, their indexing with metadata and content analysis, their automatic linking up to their exploration and visualization. We will present the last advances in research from computer science, computer vision, artificial intelligence and digital humanities, and will illustrate these concepts in several domains with different observation scales: territory understanding from historians and sociologists point of views, collection interlinking for archives and documentation of heritage objects for restoration sites.
Remi Petitpierre (EPFL) A computational and spatial exploration of historical cartography
In this research, we propose a computational analysis of the evolution of cartography, based on computer vision. Our corpus comprises over 100,000 digitized maps published in Europe, the USA and Indonesia between the 16th and 20th centuries. Our approach is based on the segmentation of digitized maps into five semantic classes (boundaries, buildings, road network, water, non-built), as well as on the fragmentation of maps into elementary visual units, which we project into a representation space. Our analysis focuses on the evolution, and the spatial and temporal distribution of elementary cartographic units. We also investigate the dynamics of propagation between the main historical mapmaking hubs.
Nathalie Abadie (IGN), From images of old trade directories to a geohistorical knowledge graph – lessons learned from the SoDUCo project
Mathieu Aubry (ENPC) How to bridge historical and computer vision research?
Interdisciplinary scientific collaborations are often particularly rewarding, but also challenging. I will discuss two concrete attempts to try and bridge Computer Vision and Historical research. First, I will discuss the design and development of an open-source modular web platform specifically designed to enable historians to leverage Computer Vision tools without requiring any Computer Vision expertise, AIKON https://aikon-platform.github.io/. I will show results for analyzing historical scientific illustration, and outline future development. Second, I will present a project in digital paleography where Historians directly leveraged the code produced by Computer Vision research for historical analysis. I will detail advantages and drawbacks of both approaches, emphasizing in particular the high engineering effort required to build and maintain a web platform, as well as the level of technicity required to leverage research code.
Katherine McDonough (Lancaster University/The Alan Turing Institute) MapReader:Using computer vision to rethink how we look at maps as historical sources
How can humanities and social science researchers learn to work with large collections of maps as complex historical objects? For decades, scholars have been rehabilitating maps as underutilized sources that are in fact central to our understanding of the past. With improvements in computer vision and machine learning, we now can experiment with computational approaches to understanding the content of maps. But working with maps as data raises questions about how to carefully examine cartographic documents as biased, partial representations of the landscape, not as faithful snapshots of everything on the ground. The computational analysis of maps should thus build bridges between two literatures: 1) emerging processual map history (e.g. the history of map production and use) and 2) the social and environmental histories of the places maps depict. MapReader was developed with these opportunities and concerns in mind, and now incorporates two fundamental maps-as-data tasks: patch classification and text spotting. This talk reviews current uses of MapReader as well as the implications of historians working with maps at scale.
Victor Gysenbergh (Sorbonne Université) Lessons from the application of Machine learning to process multispectral images of the “New Apulcius” palimpsest
The « New Apuleius » palimpsest is a text that was discovered in 2022 in manuscript 38 (XL) of the Capitular Library in Verona. This talk will discuss the ongoing development of machine learning models to process multispectral images of the manuscript. It will present what has been done from an AI/computer science perspective, assess the results in terms of textual recovery from a scholarly point of view, and point out promising avenues for future work.
Peter Giles (Oxford), Between seeing and reading: machine learning and Dante’s Commedia (1470-1630) in print
This paper will introduce the research questions, methodologies, and initial findings of the Envisioning Dante 1472-1630 project, a collaboration between the John Rylands Research Institute at the University of Manchester and the Visual AI project at the University of Oxford. The project makes extensive use of machine learning to explore the emergence of the early printed page, through the example of one of the iconic works of European art and literature – Dante’s Commedia. The project has digitized 50,000 pages of Dante, treating each page as discrete graphic entities: all elements on the page surface (the printed letterform, the line of text, the paragraph, the heading, illustrations, paratexts and incidental features) as neutral shapes, without semantic content. The paper will outline the project’s technical methods, which include image segmentation; object detection; visual search; image comparison and image classification, all accomplished with open source software developing mostly by the Visual Geometry Group (VGG) in Oxford. It will show how generic machine learning image models can be retrained for bibliographical study; how the resulting model performs on unclassified images; and how the resulting segmentations can inform quantitative and qualitative analyses of the page layout. It will also demonstrate the use of image comparison software to reveal both differences and concordances across editions, and show how near-identical page elements (such as illustrations or ornaments derived from common printing surface) can be made searchable. Last, the paper will discuss how the discovery of thresholds within the text relate to thresholds between perceiving and reading the poem.
David Rabouin (Université Paris Cité) Leibniz’s Mathematical Manuscripts: successes, obstacles and prospects in automatic recognition
The Nachlass of Gottfried Wilhelm Leibniz includes approximately 17,000 folios dedicated solely to mathematics, documenting his pioneering explorations in areas such as differential calculus, analysis situs (now known as topology), determinants, binary arithmetic, and more. Despite the significance of this corpus, only half of it has been edited, and only a fraction of these editions have been conducted in a scientifically rigorous manner. Given the vast scale of this undertaking, Handwritten Text Recognition (HTR) emerges as a promising solution for transcribing and editing this enormous collection of manuscripts. This task has been undertaken within the framework of the PHILIUMM project (ERC advanced grant 101020985). However, numerous obstacles arise when dealing with early modern manuscripts, which are mainly drafts. These challenges are compounded when the manuscripts contain symbolic formulae, especially when these symbols are no longer in use. In this presentation, I will discuss the results obtained so far using e-scriptorium, as well as the collaborative work with Harold Mouchère (Polytech Nantes) in the recognition of equations within these manuscripts.
Livio de Luca (MAP) Notre-Dame de Paris: A Cathedral of Digital Data and Multidisciplinary Knowledge in Heritage Science
Research on cultural heritage involves the intersection of material objects and multidisciplinary studies, serving as a platform for generating collective knowledge. In the digital age, this intersection provides an ideal framework for the collective analysis and interpretation of facts, objects, and phenomena, facilitating the creation of new scientific and cultural resources—our heritage of tomorrow. How can we document the diverse perspectives focused on the same object of study? How do we analyze their dynamic interactions, overlaps, and fusions to generate new knowledge? Our research introduces a novel field—multidimensional, multidisciplinary digital data—as a foundational element for studying the mechanisms of knowledge production in heritage science. Utilizing an innovative approach in computational modeling and digitization, we leverage the Notre-Dame de Paris scientific project, which includes contributions from archaeology, anthropology, architecture, history, chemistry, physics, and computer science, to construct and analyse a comprehensive data corpus on scientific practices built around a common denominator. Our goal is to shift the focus from merely digitizing physical objects, to unveiling and analyzing the interplay between the complex characteristics of the material objects and the objects of knowledge developed by researchers through their practices.
Federico Nurra (INHA), Nicolò Dell’Unto (LU, Sweden), Paola Derudas (LU, Sweden), From AIR to (AI)R: the use of LLM for interpreting archaeological excavation data
Since 2021, the DarkLab at Lund University (LU) and the Digital Research Service at the French National Institute of Art History (INHA) have collaborated to develop state-of-the-art digital systems and tools for the management and publication of archaeological data, including information from fieldwork and artefact collections. Drawing on the combined expertise of both institutions, this partnership has led to the successful creation and launch of AIR (Archaeological Interactive Report). Following the international workshop on ‘Advanced 3D Archaeological Documentation and Linked Open Data’, held in Lund, Sweden, 17-19 April 2024, we began testing large language models (LLMs) for processing, transforming and interpreting archaeological data, with very promising results. The source data, accessible via the AIR API, is structured in JSON-LD and formalized according to the most widely used ontologies in the field, such as CIDOC CRM and CRM-Archaeo. We have tested two prominent LLMs, GPT-4 by OpenAI and the Mistral Large model by Mistral AI, to analyze this data. In this talk, we will present the results of this experiment: we will focus on data structure, standardized models, and the nuanced challenges of integrating semantics and ontologies into archaeological descriptions and narratives. The presentation will illustrate our approach to improving the interpretation of archaeological data using Large Language Models.
Jean-Christophe Carius (INHA), Art historical sources, machine intelligence and the research process (TBC)
Since 2020, the INHA (The French National Institute for Art History) has been developing the PENSE platform, PENSE is an acronym for “Plateforme d’édition numérique de sources enrichies”, or 'Digital Editions Platform of Enriched Sources' – if translated literally. The objective of this platform is to highlight the heritage collections held by the library department and to enhance the use of digital technologies applied to history within the studies and research department. The project is based on three main areas: source enrichment, which involves the collection of structured datasets that are added to the digitized source; digital editions, which refers to the digital writing and publication of scholarly work; and the creation of a methodological platform, which serves as an experimental space for exchange and knowledge sharing. The results of some experiments conducted on this platform will be presented, along with feedback on the integration of artificial intelligence into the process of digital edition of enriched sources. The overarching objective of this approach is to identify techniques and practices of machine intelligence that are likely to be additional scientific tools for researchers working from digitized sources. One of our fundamental principles is to successfully integrate machine intelligence into scholarly research processes, while ensuring that researchers retain oversight and management of the system. Additionally, we will examine the potential for integrating, on a limited scale, neural and symbolic artificial intelligence processing. Neural artificial intelligence encompasses techniques such as computer vision and large language models, while symbolic artificial intelligence encompasses technologies such as the semantic web, XML-TEI, or RDF. Finally, we will emphasize the importance of design considerations, including concept maps, process and workflow maps, and interface design, to help researchers understand the principles of the new AI tools and to encourage the emergence of new scientific applications via a collaborative approach
Thomas Sagory (MAN), AI at the Musée d’Archéologie National, case studies
The musée d’Archéologie nationale (MAN) is undergoing extensive renovations. A digital strategy accompanies a vast project involving the collections and the museographic reflection on defining a new visitor pathway. Several projects are exploring the possibilities offered by artificial intelligence for the conservation, documentation, and enhancement of the collections. Several case studies can be presented: the reconstruction of archaeological fragments, the digitization of complex objects, the 3D digitization of complex objects, and contributions to projects aimed at combating the trafficking of cultural goods.
Anaïs Guilhem, Kévin Réby (MAP), Large Language Models for Thesaurus Curation using Neuro-symbolic Artificial Intelligence
This article proposes an experiment to use Large Language Models (LLMs) for the classification and curation of thesaurus in Simple Knowledge Organization System (SKOS) formalization in the field of Cultural and Built Heritage. Large Language Models are a type of deep learning model that are trained on a vast amount of text data. These autoregressive models predict the next word in a sequence of words, given the preceding words as context. While LLMs can generate text that is grammatical and contextually appropriate, they do not understand the text they generate in the way that humans do. This research uses LLMs for the curation of built cultural heritage and heritage sciences terminology. It shows the potential of an artificial intelligence methodology with real messy data, while also preventing model bias and hallucinations. The manual curation of specialized terminology and thesauri is costly because it necessitates domain expert knowledge and takes time. But a shared, well-documented thesaurus is a crucial resource for the indexing of documentation and database curation. The experiment is based on real world data: a problem of controlled vocabulary curation occurred during the data migration at LRMH. But the manual task of curation is too long to be undertaken with the current means. The paper presents a neuro-symbolic artificial intelligence methodology to curate and structurate domain-specific thesaurus. The experiment is based on a neuro-symbolic artificial intelligence premise: the neural aspect uses the ability of LLMs for classifying and the symbolic in the integration of the results in a knowledge graph (KG). The LLMs are able to discriminate and categorize highly specialized vocabulary with proper prompt instructions and context clarification. One of the drawbacks of LLMs is the lack of explainability of the generated results. This is why a neuro-symbolic approach is adopted in this experiment. The need for explainability is crucial in the work with LLMs and prompt engineering in two aspects. Firstly, the objective is to create prompts that come close enough for the LLM to mimic the reasoning steps for the thesaurus curation. Secondly, we must provide to the human expert users the means for verification and validation. This experiment demonstrates the potential of LLMs for the semantic thesaurus curation. The proposed workflow is designed to circumvent bias and provide means for verification. The result is the transformation of around 13.000 terms of controlled vocabulary into a structured SKOS thesaurus usable for indexing specialized built heritage documentation.