The reconstruction of a high resolution image given a low resolution observation is an ill-posed ... more The reconstruction of a high resolution image given a low resolution observation is an ill-posed inverse problem in imaging. Deep learning methods rely on training data to learn an end-to-end mapping from a low-resolution input to a highresolution output. Unlike existing deep multimodal models that do not incorporate domain knowledge about the problem, we propose a multimodal deep learning design that incorporates sparse priors and allows the effective integration of information from another image modality into the network architecture. Our solution relies on a novel deep unfolding operator, performing steps similar to an iterative algorithm for convolutional sparse coding with side information; therefore, the proposed neural network is interpretable by design. The deep unfolding architecture is used as a core component of a multimodal framework for guided image super-resolution. An alternative multimodal design is investigated by employing residual learning to improve the training efficiency. The presented multimodal approach is applied to super-resolution of near-infrared and multi-spectral images as well as depth upsampling using RGB images as side information. Experimental results show that our model outperforms state-ofthe-art methods.
Tijana Ruzic UGent, Bruno Cornelis, Ljiljana Platisa UGent, Aleksandra Pizurica UGent, Ann Dooms,... more Tijana Ruzic UGent, Bruno Cornelis, Ljiljana Platisa UGent, Aleksandra Pizurica UGent, Ann Dooms, Maximiliaan Martens UGent, Marc De Mey UGent and Ingrid Daubechies (2010) Vision and material : interaction between art and science in Jan van Eyck's time, Abstracts. ... Ruzic, Tijana, Bruno Cornelis, Ljiljana Platisa, Aleksandra Pizurica, Ann Dooms, Maximiliaan Martens, Marc De Mey, and Ingrid Daubechies. 2010. “Craquelure Inpainting in Art Work.” In Vision and Material : Interaction Between Art and Science in Jan Van Eyck's Time, ...
ABSTRACT The art of Jan van Eyck is renowned for its capacity to reveal with extreme accuracy the... more ABSTRACT The art of Jan van Eyck is renowned for its capacity to reveal with extreme accuracy the materiality of object surfaces. The interaction of light with the surface of materials results in reflected light that carries to the viewer the information on the quality of the surface material. Apparently Jan van Eyck understands that seeing a visual object has more to do with tracing perceptually what kind of light is falling on the object and where it comes from than with any conceptual identification of the object. Tracing the light back to its source is in vision studies known as inverse optics. In a visual display, the light source is particularly revealed by the highlights which are mirroring that source. In his pictures Jan van Eyck uses these highlights as anchor points to indicate where in the visual scene the highest lightness is to be found. With further understanding of the physics involved, he carefully observes the specific fall off of the intensity of reflected light around highlights to characterize particular materials. His special sensibility for highlights might have originated in his involvement with the production of miniatures in illuminated books where, depending on the locations of light source and viewer and orientation of the illuminated page, gilded segments can drastically change in brightness. We will compare color and luminance distribution properties around highlights in some Eyckian pictures with comparable pictures of contemporaries (Campin, Van der Weyden) with special attention to metals and gold brocade.
Digital image processing is proving to be of great help in the analysis and documentation of our ... more Digital image processing is proving to be of great help in the analysis and documentation of our vast cultural heritage. In this paper, we present a new method for the virtual restoration of digitized paintings with special attention for the Ghent Altarpiece (1432), a large polyptych panel painting of which very few digital reproductions exist. We achieve our objective by detecting and digitally removing cracks. The detection of cracks is particularly difficult because of the varying content features in different parts of the polyptych. Three new detection methods are proposed and combined in order to detect cracks of different sizes as well as varying brightness. Semi-supervised clustering based post-processing is used to remove objects falsely labelled as cracks. For the subsequent inpainting stage, a patch-based technique is applied to handle the noisy nature of the images and to increase the performance for crack removal. We demonstrate the usefulness of our method by means of a case study where the goal is to improve readability of the depiction of text in a book, present in one of the panels, in order to assist paleographers in its deciphering.
abstract In this paper, we present a new method for virtual restoration of digitized paintings, w... more abstract In this paper, we present a new method for virtual restoration of digitized paintings, with the special focus on the Ghent Altarpiece (1432), one of Belgium's greatest masterpieces. The goal of the work is to remove cracks from the digitized painting thereby approximating how the painting looked like before ageing for nearly 600 years and aiding art historical and palaeographical analysis. For crack detection, we employ a multiscale morphological approach, which can cope with greatly varying thickness of the cracks as ...
Abstract Objective characterization of jewels in paintings, especially pearls, has been a long la... more Abstract Objective characterization of jewels in paintings, especially pearls, has been a long lasting challenge for art historians. The way an artist painted pearls reflects his ability to observing nature and his knowledge of contemporary optical theory. Moreover, the painterly execution may also be considered as an individual characteristic useful in distinguishing hands. In this work, we propose a set of image analysis techniques to analyze and measure spatial characteristics of the digital images of pearls, all relying on the so called ...
The work described here fits in the context of a larger project on the objective and relevant cha... more The work described here fits in the context of a larger project on the objective and relevant characterization of paintings and painting canvas through the analysis of multimodal digital images. We captured, amongst others, X-ray images of different canvas types, characterized by a variety of textures and weave patterns (fine and rougher texture; single thread and multiple threads per weave), including raw canvas as well as canvas processed with different primers. In this paper, we study how to characterize the canvas by extracting ...
IEEE transactions on image processing : a publication of the IEEE Signal Processing Society, Jan 25, 2016
We address the removal of canvas artifacts from high-resolution digital photographs and X-ray ima... more We address the removal of canvas artifacts from high-resolution digital photographs and X-ray images of paintings on canvas. Both imaging modalities are common investigative tools in art history and art conservation. Canvas artifacts manifest themselves very differently according to the acquisition modality; they can hamper the visual reading of the painting by art experts, for instance, in preparing a restoration campaign. Digital removal of canvas is desirable for restorers when the painting on canvas they are preparing to restore has acquired over the years a much more salient texture. We propose a new algorithm that combines a cartoon-texture decomposition method with adaptive multiscale thresholding in the frequency domain to isolate and suppress the canvas components. To illustrate the strength of the proposed method, we provide various examples, for acquisitions in both imaging modalities, for paintings with different types of canvas and from different periods. The proposed a...
IEEE transactions on image processing : a publication of the IEEE Signal Processing Society, Jan 31, 2016
In support of art investigation, we propose a new source separation method that unmixes a single ... more In support of art investigation, we propose a new source separation method that unmixes a single X-ray scan acquired from double-sided paintings. In this problem, the X-ray signals to be separated have similar morphological characteristics, which brings previous source separation methods to their limits. Our solution is to use photographs taken from the front-and back-side of the panel to drive the separation process. The crux of our approach relies on the coupling of the two imaging modalities (photographs and X-rays) using a novel coupled dictionary learning framework able to capture both common and disparate features across the modalities using parsimonious representations; the common component captures features shared by the multi-modal images, whereas the innovation component captures modality-specific information. As such, our model enables the formulation of appropriately regularized convex optimization procedures that lead to the accurate separation of the X-rays. Our dictio...
Applications of Digital Image Processing XXXI, 2008
In the past decade the use of digital data has increased significantly. The advantages of digital... more In the past decade the use of digital data has increased significantly. The advantages of digital data are, amongst others, easy editing, fast, cheap and cross-platform distribution and compact storage. The most crucial disadvantages are the unauthorized copying and copyright issues, by which authors and license holders can suffer considerable financial losses. Many inexpensive methods are readily available for editing digital data and, unlike analog information, the reproduction in the digital case is simple and robust. Hence, there is great interest in developing technology that helps to protect the integrity of a digital work and the copyrights of its owners. Watermarking, which is the embedding of a signal (known as the watermark) into the original digital data, is one method that has been proposed for the protection of digital media elements such as audio, video and images. In this article, we examine watermarking schemes for still images, based on selective quantization of the coefficients of a wavelet transformed image, i.e. sparse quantization-index modulation (QIM) watermarking. Different grouping schemes for the wavelet coefficients are evaluated and experimentally verified for robustness against several attacks. Wavelet tree-based grouping schemes yield a slightly improved performance over block-based grouping schemes. Additionally, the impact of the deployment of error correction codes on the most promising configurations is examined. The utilization of BCH-codes (Bose, Ray-Chaudhuri, Hocquenghem) results in an improved robustness as long as the capacity of the error codes is not exceeded (cliff-effect).
2014 IEEE International Conference on Image Processing (ICIP), 2014
ABSTRACT We introduce an algorithm that removes the deleterious effect of cradling on X-ray image... more ABSTRACT We introduce an algorithm that removes the deleterious effect of cradling on X-ray images of paintings on wooden panels. The algorithm consists of a three stage procedure. Firstly, the cradled regions are located automatically. The second step consists of separating the X-ray image into a textural and image component. In the last step the algorithm learns to distinguish between the texture caused by the wooden cradle and the texture belonging to the original painted wooden panel. The results obtained with our method are compared with those obtained manually by best current practice.
2013 18th International Conference on Digital Signal Processing (DSP), 2013
1 The preservation of our cultural heritage is of paramount importance. Thanks to recent developm... more 1 The preservation of our cultural heritage is of paramount importance. Thanks to recent developments in digital acquisition techniques, powerful image analysis algorithms are developed which can be useful non-invasive tools to assist in the restoration and preservation of art. In this paper we propose a semi-supervised crack detection method that can be used for high-dimensional acquisitions of paintings coming from different modalities. Our dataset consists of a recently acquired collection of images of the Ghent Altarpiece (1432), one of Northern Europe's most important art masterpieces. Our goal is to build a classifier that is able to discern crack pixels from the background consisting of non-crack pixels, making optimal use of the information that is provided by each modality. To accomplish this we employ a recently developed non-parametric Bayesian classifier, that uses tensor factorizations to characterize any conditional probability. A prior is placed on the parameters of the factorization such that every possible interaction between predictors is allowed while still identifying a sparse subset among these predictors. The proposed Bayesian classifier, which we will refer to as conditional Bayesian tensor factorization or CBTF, is assessed by visually comparing classification results with the Random Forest (RF) algorithm.
Altarpiece. Virtual restoration enables removal of the signs of aging, such as cracks. Today it i... more Altarpiece. Virtual restoration enables removal of the signs of aging, such as cracks. Today it is possible for the viewer to appreciate the finest details of the masterpiece by zooming into its high-resolution scans. Especially in these enlarged details, the appearance of inevitable signs of aging can become prominent. Virtual restoration enhances not only visual experience, which is important from the esthetical and psychological points of view, but in some cases it also facilitates deciphering the content (like text fragments) of the painting. For example, virtual restoration improves the legibility of the text in the Annunciation to Mary panel. Equally challenging is the development of new tools for painter style authentication. The second part of this Chapter reviews some of the metrics related to objective characterization of painted pearls, which are so beautiful and abundant in the works of Van Eyck. The potentials of these metrics to distinguish between painter styles are discussed as well as some side applications, such as bringing a painted object closer to the style of another painter.
The reconstruction of a high resolution image given a low resolution observation is an ill-posed ... more The reconstruction of a high resolution image given a low resolution observation is an ill-posed inverse problem in imaging. Deep learning methods rely on training data to learn an end-to-end mapping from a low-resolution input to a highresolution output. Unlike existing deep multimodal models that do not incorporate domain knowledge about the problem, we propose a multimodal deep learning design that incorporates sparse priors and allows the effective integration of information from another image modality into the network architecture. Our solution relies on a novel deep unfolding operator, performing steps similar to an iterative algorithm for convolutional sparse coding with side information; therefore, the proposed neural network is interpretable by design. The deep unfolding architecture is used as a core component of a multimodal framework for guided image super-resolution. An alternative multimodal design is investigated by employing residual learning to improve the training efficiency. The presented multimodal approach is applied to super-resolution of near-infrared and multi-spectral images as well as depth upsampling using RGB images as side information. Experimental results show that our model outperforms state-ofthe-art methods.
Tijana Ruzic UGent, Bruno Cornelis, Ljiljana Platisa UGent, Aleksandra Pizurica UGent, Ann Dooms,... more Tijana Ruzic UGent, Bruno Cornelis, Ljiljana Platisa UGent, Aleksandra Pizurica UGent, Ann Dooms, Maximiliaan Martens UGent, Marc De Mey UGent and Ingrid Daubechies (2010) Vision and material : interaction between art and science in Jan van Eyck's time, Abstracts. ... Ruzic, Tijana, Bruno Cornelis, Ljiljana Platisa, Aleksandra Pizurica, Ann Dooms, Maximiliaan Martens, Marc De Mey, and Ingrid Daubechies. 2010. “Craquelure Inpainting in Art Work.” In Vision and Material : Interaction Between Art and Science in Jan Van Eyck's Time, ...
ABSTRACT The art of Jan van Eyck is renowned for its capacity to reveal with extreme accuracy the... more ABSTRACT The art of Jan van Eyck is renowned for its capacity to reveal with extreme accuracy the materiality of object surfaces. The interaction of light with the surface of materials results in reflected light that carries to the viewer the information on the quality of the surface material. Apparently Jan van Eyck understands that seeing a visual object has more to do with tracing perceptually what kind of light is falling on the object and where it comes from than with any conceptual identification of the object. Tracing the light back to its source is in vision studies known as inverse optics. In a visual display, the light source is particularly revealed by the highlights which are mirroring that source. In his pictures Jan van Eyck uses these highlights as anchor points to indicate where in the visual scene the highest lightness is to be found. With further understanding of the physics involved, he carefully observes the specific fall off of the intensity of reflected light around highlights to characterize particular materials. His special sensibility for highlights might have originated in his involvement with the production of miniatures in illuminated books where, depending on the locations of light source and viewer and orientation of the illuminated page, gilded segments can drastically change in brightness. We will compare color and luminance distribution properties around highlights in some Eyckian pictures with comparable pictures of contemporaries (Campin, Van der Weyden) with special attention to metals and gold brocade.
Digital image processing is proving to be of great help in the analysis and documentation of our ... more Digital image processing is proving to be of great help in the analysis and documentation of our vast cultural heritage. In this paper, we present a new method for the virtual restoration of digitized paintings with special attention for the Ghent Altarpiece (1432), a large polyptych panel painting of which very few digital reproductions exist. We achieve our objective by detecting and digitally removing cracks. The detection of cracks is particularly difficult because of the varying content features in different parts of the polyptych. Three new detection methods are proposed and combined in order to detect cracks of different sizes as well as varying brightness. Semi-supervised clustering based post-processing is used to remove objects falsely labelled as cracks. For the subsequent inpainting stage, a patch-based technique is applied to handle the noisy nature of the images and to increase the performance for crack removal. We demonstrate the usefulness of our method by means of a case study where the goal is to improve readability of the depiction of text in a book, present in one of the panels, in order to assist paleographers in its deciphering.
abstract In this paper, we present a new method for virtual restoration of digitized paintings, w... more abstract In this paper, we present a new method for virtual restoration of digitized paintings, with the special focus on the Ghent Altarpiece (1432), one of Belgium's greatest masterpieces. The goal of the work is to remove cracks from the digitized painting thereby approximating how the painting looked like before ageing for nearly 600 years and aiding art historical and palaeographical analysis. For crack detection, we employ a multiscale morphological approach, which can cope with greatly varying thickness of the cracks as ...
Abstract Objective characterization of jewels in paintings, especially pearls, has been a long la... more Abstract Objective characterization of jewels in paintings, especially pearls, has been a long lasting challenge for art historians. The way an artist painted pearls reflects his ability to observing nature and his knowledge of contemporary optical theory. Moreover, the painterly execution may also be considered as an individual characteristic useful in distinguishing hands. In this work, we propose a set of image analysis techniques to analyze and measure spatial characteristics of the digital images of pearls, all relying on the so called ...
The work described here fits in the context of a larger project on the objective and relevant cha... more The work described here fits in the context of a larger project on the objective and relevant characterization of paintings and painting canvas through the analysis of multimodal digital images. We captured, amongst others, X-ray images of different canvas types, characterized by a variety of textures and weave patterns (fine and rougher texture; single thread and multiple threads per weave), including raw canvas as well as canvas processed with different primers. In this paper, we study how to characterize the canvas by extracting ...
IEEE transactions on image processing : a publication of the IEEE Signal Processing Society, Jan 25, 2016
We address the removal of canvas artifacts from high-resolution digital photographs and X-ray ima... more We address the removal of canvas artifacts from high-resolution digital photographs and X-ray images of paintings on canvas. Both imaging modalities are common investigative tools in art history and art conservation. Canvas artifacts manifest themselves very differently according to the acquisition modality; they can hamper the visual reading of the painting by art experts, for instance, in preparing a restoration campaign. Digital removal of canvas is desirable for restorers when the painting on canvas they are preparing to restore has acquired over the years a much more salient texture. We propose a new algorithm that combines a cartoon-texture decomposition method with adaptive multiscale thresholding in the frequency domain to isolate and suppress the canvas components. To illustrate the strength of the proposed method, we provide various examples, for acquisitions in both imaging modalities, for paintings with different types of canvas and from different periods. The proposed a...
IEEE transactions on image processing : a publication of the IEEE Signal Processing Society, Jan 31, 2016
In support of art investigation, we propose a new source separation method that unmixes a single ... more In support of art investigation, we propose a new source separation method that unmixes a single X-ray scan acquired from double-sided paintings. In this problem, the X-ray signals to be separated have similar morphological characteristics, which brings previous source separation methods to their limits. Our solution is to use photographs taken from the front-and back-side of the panel to drive the separation process. The crux of our approach relies on the coupling of the two imaging modalities (photographs and X-rays) using a novel coupled dictionary learning framework able to capture both common and disparate features across the modalities using parsimonious representations; the common component captures features shared by the multi-modal images, whereas the innovation component captures modality-specific information. As such, our model enables the formulation of appropriately regularized convex optimization procedures that lead to the accurate separation of the X-rays. Our dictio...
Applications of Digital Image Processing XXXI, 2008
In the past decade the use of digital data has increased significantly. The advantages of digital... more In the past decade the use of digital data has increased significantly. The advantages of digital data are, amongst others, easy editing, fast, cheap and cross-platform distribution and compact storage. The most crucial disadvantages are the unauthorized copying and copyright issues, by which authors and license holders can suffer considerable financial losses. Many inexpensive methods are readily available for editing digital data and, unlike analog information, the reproduction in the digital case is simple and robust. Hence, there is great interest in developing technology that helps to protect the integrity of a digital work and the copyrights of its owners. Watermarking, which is the embedding of a signal (known as the watermark) into the original digital data, is one method that has been proposed for the protection of digital media elements such as audio, video and images. In this article, we examine watermarking schemes for still images, based on selective quantization of the coefficients of a wavelet transformed image, i.e. sparse quantization-index modulation (QIM) watermarking. Different grouping schemes for the wavelet coefficients are evaluated and experimentally verified for robustness against several attacks. Wavelet tree-based grouping schemes yield a slightly improved performance over block-based grouping schemes. Additionally, the impact of the deployment of error correction codes on the most promising configurations is examined. The utilization of BCH-codes (Bose, Ray-Chaudhuri, Hocquenghem) results in an improved robustness as long as the capacity of the error codes is not exceeded (cliff-effect).
2014 IEEE International Conference on Image Processing (ICIP), 2014
ABSTRACT We introduce an algorithm that removes the deleterious effect of cradling on X-ray image... more ABSTRACT We introduce an algorithm that removes the deleterious effect of cradling on X-ray images of paintings on wooden panels. The algorithm consists of a three stage procedure. Firstly, the cradled regions are located automatically. The second step consists of separating the X-ray image into a textural and image component. In the last step the algorithm learns to distinguish between the texture caused by the wooden cradle and the texture belonging to the original painted wooden panel. The results obtained with our method are compared with those obtained manually by best current practice.
2013 18th International Conference on Digital Signal Processing (DSP), 2013
1 The preservation of our cultural heritage is of paramount importance. Thanks to recent developm... more 1 The preservation of our cultural heritage is of paramount importance. Thanks to recent developments in digital acquisition techniques, powerful image analysis algorithms are developed which can be useful non-invasive tools to assist in the restoration and preservation of art. In this paper we propose a semi-supervised crack detection method that can be used for high-dimensional acquisitions of paintings coming from different modalities. Our dataset consists of a recently acquired collection of images of the Ghent Altarpiece (1432), one of Northern Europe's most important art masterpieces. Our goal is to build a classifier that is able to discern crack pixels from the background consisting of non-crack pixels, making optimal use of the information that is provided by each modality. To accomplish this we employ a recently developed non-parametric Bayesian classifier, that uses tensor factorizations to characterize any conditional probability. A prior is placed on the parameters of the factorization such that every possible interaction between predictors is allowed while still identifying a sparse subset among these predictors. The proposed Bayesian classifier, which we will refer to as conditional Bayesian tensor factorization or CBTF, is assessed by visually comparing classification results with the Random Forest (RF) algorithm.
Altarpiece. Virtual restoration enables removal of the signs of aging, such as cracks. Today it i... more Altarpiece. Virtual restoration enables removal of the signs of aging, such as cracks. Today it is possible for the viewer to appreciate the finest details of the masterpiece by zooming into its high-resolution scans. Especially in these enlarged details, the appearance of inevitable signs of aging can become prominent. Virtual restoration enhances not only visual experience, which is important from the esthetical and psychological points of view, but in some cases it also facilitates deciphering the content (like text fragments) of the painting. For example, virtual restoration improves the legibility of the text in the Annunciation to Mary panel. Equally challenging is the development of new tools for painter style authentication. The second part of this Chapter reviews some of the metrics related to objective characterization of painted pearls, which are so beautiful and abundant in the works of Van Eyck. The potentials of these metrics to distinguish between painter styles are discussed as well as some side applications, such as bringing a painted object closer to the style of another painter.
In this paper, we present a new method for virtual restoration of
digitized paintings, with the ... more In this paper, we present a new method for virtual restoration of
digitized paintings, with the special focus on the Ghent Altarpiece
(1432), one of Belgium's greatest masterpieces. The goal of the work
is to remove cracks from the digitized painting thereby approximating
how the painting looked like before ageing for nearly 600 years and
aiding art historical and palaeographical analysis. For crack detection,
we employ a multiscale morphological approach, which can cope with
greatly varying thickness of the cracks as well as with their varying
intensities (from dark to the light ones). Due to the content of
the painting (with extremely many fine details) and complex type
of cracks (including inconsistent whitish clouds around them), the
available inpainting methods do not provide satisfactory results
on many parts of the painting. We show that patch-based methods outperform
pixel-based ones, but leaving still much room for improvements in
this application. We propose a new method for candidate patch selection,
which can be combined with different patch-based inpainting methods
to improve their performance in crack removal. The results demonstrate
improved performance, with less artefacts and better preserved fine
details.
Uploads
Papers by Bruno Cornelis