Deep Adversarial Belief Networks
Ashkan Panahi2019, arXiv: Learning
Sign up for access to the world's latest research
checkGet notified about relevant papers
checkSave papers to use in your research
checkJoin the discussion with peers
checkTrack your impact
Abstract
We present a novel adversarial framework for training deep belief networks (DBNs), which includes replacing the generator network in the methodology of generative adversarial networks (GANs) with a DBN and developing a highly parallelizable numerical algorithm for training the resulting architecture in a stochastic manner. Unlike the existing techniques, this framework can be applied to the most general form of DBNs with no requirement for back propagation. As such, it lays a new foundation for developing DBNs on a par with GANs with various regularization units, such as pooling and normalization. Foregoing back-propagation, our framework also exhibits superior scalability as compared to other DBN and GAN learning techniques. We present a number of numerical experiments in computer vision as well as neurosciences to illustrate the main advantages of our approach.
Related papers
Generative Adversarial Networks:Introduction and Outlook
Recently, generative adversarial networks (GANs) have become a research focus of artificial intelligence. Inspired by two-player zero-sum game, GANs comprise a generator and a discriminator, both trained under the adversarial learning idea. The goal of GANs is to estimate the potential distribution of real data samples and generate new samples from that distribution. Since their initiation, GANs have been widely studied due to their enormous prospect for applications, including image and vision computing, speech and language processing, etc. In this review paper, we summarize the state of the art of GANs and look into the future. Firstly, we survey GANs' proposal background, theoretic and implementation models, and application fields. Then, we discuss GANs' advantages and disadvantages, and their development trends. In particular, we investigate the relation between GANs and parallel intelligence, with the conclusion that GANs have a great potential in parallel systems research in terms of virtual-real interaction and integration. Clearly, GANs can provide substantial algorithmic support for parallel intelligence.
A fast learning algorithm for deep belief nets
We show how to use "complementary priors" to eliminate the explaining away effects that make inference difficult in densely-connected belief nets that have many hidden layers. Using complementary priors, we derive a fast, greedy algorithm that can learn deep, directed belief networks one layer at a time, provided the top two layers form an undirected associative memory. The fast, greedy algorithm is used to initialize a slower learning procedure that fine-tunes the weights using a contrastive version of the wake-sleep algorithm. After fine-tuning, a network with three hidden layers forms a very good generative model of the joint distribution of handwritten digit images and their labels. This generative model gives better digit classification than the best discriminative learning algorithms. The low-dimensional manifolds on which the digits lie are modelled by long ravines in the free-energy landscape of the top-level associative memory and it is easy to explore these ravines by using the directed connections to display what the associative memory has in mind.
Generative Adversarial Neural Networks and Deep Learning: Successful Cases and Advanced Approaches
International Journal of Computing, 2021
Cross-domain artificial intelligence (AI) frameworks are the keys to amplify progress in science. Cutting edge deep learning methods offer novel opportunities for retrieving, optimizing, and improving different data types. AI techniques provide new ways for enhancing and polishing existing models that are used in applied sciences. New breakthroughs in generative adversarial neural networks (GANNs/GANs) and deep learning allow to drastically increase the quality of diverse graphic samples obtained with research equipment. All these innovative approaches can be compounded into a unified academic and technological pipeline that can radically elevate and accelerate scientific research and development. The authors analyze a number of successful cases of GAN and deep learning applications in applied scientific fields (including observational astronomy, health care, materials science, deep fakes, bioinformatics, and typography) and discuss advanced approaches for increasing GAN and DL effi...
Generative adversarial networks: a survey on applications and challenges
International Journal of Multimedia Information Retrieval, 2020
Deep neural networks have attained great success in handling high dimensional data, especially images. However, generating naturalistic images containing ginormous subjects for different tasks like image classification, segmentation, object detection, reconstruction, etc., is continued to be a difficult task. Generative modelling has the potential to learn any kind of data distribution in an unsupervised manner. Variational autoencoder (VAE), autoregressive models, and generative adversarial network (GAN) are the popular generative modelling approaches that generate data distributions. Among these, GANs have gained much attention from the research community in recent years in terms of generating quality images and data augmentation. In this context, we collected research articles that employed GANs for solving various tasks from popular databases and summarized them based on their application. The main objective of this article is to present the nuts and bolts of GANs, state-of-the-art related work and its applications, evaluation metrics, challenges involved in training GANs, and benchmark datasets that would benefit naive and enthusiastic researchers who are interested in working on GANs.
HGAN: Hybrid generative adversarial network
Journal of Intelligent & Fuzzy Systems, 2021
In this paper, we present a simple approach to train Generative Adversarial Networks (GANs) in order to avoid a mode collapse issue. Implicit models such as GANs tend to generate better samples compared to explicit models that are trained on tractable data likelihood. However, GANs overlook the explicit data density characteristics which leads to undesirable quantitative evaluations and mode collapse. To bridge this gap, we propose a hybrid generative adversarial network (HGAN) for which we can enforce data density estimation via an autoregressive model and support both adversarial and likelihood framework in a joint training manner which diversify the estimated density in order to cover different modes. We propose to use an adversarial network to transfer knowledge from an autoregressive model (teacher) to the generator (student) of a GAN model. A novel deep architecture within the GAN formulation is developed to adversarially distill the autoregressive model information in additio...
Contributions to deep learning methodologies
2018
In recent years the Deep Neural Networks (DNN) has been using widely in a big range of machine learning and data-mining purposes. This pattern recognition approach can handle highly nonlinear problems. In this work, three main contributions to DNN are presented. 1A method called Semi Parallel Deep Neural Networks (SPDNN) is introduced wherein several deep architectures are mixed and merged using graph contraction technique to take advantage of all the parent networks. 2The importance of data is investigated in several attempts and an augmentation technique know as Smart Augmentation is presented. 3To extract more information from a database, multiple works on Generative Adversarial Networks (GAN) are given wherein the joint distribution of data and its ground truth is approximated and in other projects conditional generators for classification and regression problems are trained and tested.
Recent Advances of Generative Adversarial Networks in Computer Vision
IEEE Access, 2018
The appearance of generative adversarial networks (GAN) provides a new approach and framework for computer vision. Compared with traditional machine learning algorithms, GAN works via adversarial training concept and is more powerful in both feature learning and representation. GAN also exhibits some problems, such as non-convergence, model collapse, and uncontrollability due to high degree of freedom. How to improve the theory of GAN and apply it to computer-vision-related tasks have now attracted much research efforts. In this paper, recently proposed GAN models and their applications in computer vision are systematically reviewed. In particular, we firstly survey the history and development of generative algorithms, the mechanism of GAN, its fundamental network structures, and theoretical analysis of the original GAN. Classical GAN algorithms are then compared comprehensively in terms of the mechanism, visual results of generated samples, and Frechet Inception Distance. These networks are further evaluated from network construction, performance, and applicability aspects by extensive experiments conducted over public datasets. After that, several typical applications of GAN in computer vision, including high-quality samples generation, style transfer, and image translation, are examined. Finally, some existing problems of GAN are summarized and discussed and potential future research topics are forecasted. INDEX TERMS Deep learning, generative adversarial networks (GAN), computer vision (CV), image generation, style transfer, image inpainting.
Unlocking insights gans (1)
IEEE, 2024
Generative adversarial networks (GANs) are a cutting-edge approach to generative modeling in deep learning. GAN’s was proposed in 2014 by Ian Goodfellow. Since then, there has been significant growth in adversarial networks. New breakthroughs and innovative approaches in generative adversarial networks can radically elevate and increase the quality of synthetically generated images by extracting patterns from the original datasets. Among the major advancements of GAN, image synthesis is the most prominent and extensively studied application. The concept of adversarial training, where two neural networks compete against each other, has introduced a novel paradigm for learning complex patterns in the images. The paper emphasizes the vital role of GANs in strengthening and fine-tuning datasets, honing further research to create GANs capable of producing high-quality synthetic samples with constrained practice of data.
Adversarial Concurrent Training: Optimizing Robustness and Accuracy Trade-off of Deep Neural Networks
arXiv (Cornell University), 2020
Adversarial training has been proven to be an effective technique for improving the adversarial robustness of models. However, there seems to be an inherent trade-off between optimizing the model for accuracy and robustness. To this end, we propose Adversarial Concurrent Training (ACT), which employs adversarial training in a collaborative learning framework whereby we train a robust model in conjunction with a natural model in a minimax game. ACT encourages the two models to align their feature space by using the task-specific decision boundaries and explore the input space more broadly. Furthermore, the natural model acts as a regularizer, enforcing priors on features that the robust model should learn. Our analyses on the behavior of the models show that ACT leads to a robust model with lower model complexity, higher information compression in the learned representations, and high posterior entropy solutions indicative of convergence to a flatter minima. We demonstrate the effectiveness of the proposed approach across different datasets and network architectures. On ImageNet, ACT achieves 68.20% standard accuracy and 44.29% robustness accuracy under a 100-iteration untargeted attack, improving upon the standard adversarial training method's 65.70% standard accuracy and 42.36% robustness.
Generative Adversarial Nets
We propose a new framework for estimating generative models via an adversar-ial process, in which we simultaneously train two models: a generative model G that captures the data distribution, and a discriminative model D that estimates the probability that a sample came from the training data rather than G. The training procedure for G is to maximize the probability of D making a mistake. This framework corresponds to a minimax two-player game. In the space of arbitrary functions G and D, a unique solution exists, with G recovering the training data distribution and D equal to 1 2 everywhere. In the case where G and D are defined by multilayer perceptrons, the entire system can be trained with backpropagation. There is no need for any Markov chains or unrolled approximate inference networks during either training or generation of samples. Experiments demonstrate the potential of the framework through qualitative and quantitative evaluation of the generated samples.
Related papers
Investigating GAN and VAE to train DCNN
International Journal of Machine Learning and Computing
The Convolutional Neural Network (CNN) is a class of deep artificial neural network and has recently gained special attention after demonstrating breakthrough accuracies in various classification tasks. CNNs have shown remarkable performance in machine vision tasks such as image classification, natural language processing and speech recognition. There is evidence that the depth of a CNN plays an important role in performance of CNNs. However, we investigated the feasibility of improving the performance of shallow networks via fusion of the features computed by a homogenous and heterogeneous set of pre-trained networks. We also explored a recently developed framework called the Generative Adversarial Network (GAN), in which we simultaneously train two models, a Generator and a Discriminator. The Generator attempts to produce data that mirrors the probability distribution of the "true" dataset. The Discriminator is trained to distinguish between the true dataset and the counterfeit data produced by the Generator. Our work involves the application of a GAN for generation and fine tuning of synthetic data to be used to train a deep CNN. Specifically, we investigate the use of a synthetic data generator along with a GAN to create an unlimited quantity of labeled training data, without the need for hand-labeling images. We apply this technique to the detection and localization of various vehicles. We attempt to distinguish between military trucks and other types of vehicles. A successful outcome could lead to improvements in addressing security threats rapidly, and cost-effectively. We also investigate an alternative method for generating synthetic data, the Variational Auto-Encoder (VAE). Variational auto-encoders are trained to encode then decode input vectors and can also be useful for generating new training data. VAEs are capable of dimensionality reduction and synthesizing data. Finally, we evaluate our multiplicative fusion method compared to the fusion methods that we investigated previously.
Gradient Adversarial Training of Neural Networks
ArXiv, 2018
We propose gradient adversarial training, an auxiliary deep learning framework applicable to different machine learning problems. In gradient adversarial training, we leverage a prior belief that in many contexts, simultaneous gradient updates should be statistically indistinguishable from each other. We enforce this consistency using an auxiliary network that classifies the origin of the gradient tensor, and the main network serves as an adversary to the auxiliary network in addition to performing standard task-based training. We demonstrate gradient adversarial training for three different scenarios: (1) as a defense to adversarial examples we classify gradient tensors and tune them to be agnostic to the class of their corresponding example, (2) for knowledge distillation, we do binary classification of gradient tensors derived from the student or teacher network and tune the student gradient tensor to mimic the teacher's gradient tensor; and (3) for multi-task learning we cla...
Generative Adversarial Networks
We propose a new framework for estimating generative models via an adversarial process, in which we simultaneously train two models: a generative model G that captures the data distribution, and a discriminative model D that estimates the probability that a sample came from the training data rather than G. The training procedure for G is to maximize the probability of D making a mistake. This framework corresponds to a minimax two-player game. In the space of arbitrary functions G and D, a unique solution exists, with G recovering the training data distribution and D equal to 1/2 everywhere. In the case where G and D are defined by multilayer perceptrons, the entire system can be trained with backpropagation. There is no need for any Markov chains or unrolled approximate inference networks during either training or generation of samples. Experiments demonstrate the potential of the framework through qualitative and quantitative evaluation of the generated samples.
Performance Evaluation of Generative Adversarial Networks for Computer Vision Applications
Ingénierie des systèmes d information, 2020
Generative Adversarial Networks (GAN) generates model approaches using Convolution Neural Networks (CNN) to find out learning regularities and to discover the hidden patterns held in given input data. GAN is a generative model that is trained using two models such as generator and Discriminator both competing against each other to learn the probability distribution function, networks such as CNN, RNN, ANN etc. These traditional neural networks are easily fooled in misclassifying things by adding small amount of noise to original data, whereas GAN's are more stable and easier to train due to the amalgamation of Feed Forward Neural Network and CNN. In general, GAN's are simple Neural networks be trained in adversarial way to generate the data mimicking same distribution, Generator learns new possible sample, and the Discriminator learns how to differentiate generated samples from valid facts. Generated samples are similar in the nature but different from real distribution data. The generated samples make use of computer vision techniques such as visualization designs, realistic image generation, image classifications etc. In the proposed work, to realize the probability distribution Restricted-Boltzmann machines and Deep Belief networks are used. The performance of the GAN Networks is evaluated on various standard datasets to realize the complex tasks such as image prediction, handwritten digit's generation, clothing classification, image segmentation tasks etc. From the experimental results, it is clearly evident that the performance of GAN outperforms other state of the art classifiers on all the benchmark datasets.
From Beginning to BEGANing: Role of Adversarial Learning in Reshaping Generative Models
Electronics
Deep generative models, such as deep Boltzmann machines, focused on models that provided parametric specification of probability distribution functions. Such models are trained by maximizing intractable likelihood functions, and therefore require numerous approximations to the likelihood gradient. This underlying difficulty led to the development of generative machines such as generative stochastic networks, which do not represent the likelihood functions explicitly, like the earlier models, but are trained with exact backpropagation rather than the numerous approximations. These models use piecewise linear units that are having well behaved gradients. Generative machines were further extended with the introduction of an associative adversarial network leading to the generative adversarial nets (GANs) model by Goodfellow in 2014. The estimations in GANs process two multilayer perceptrons, called the generative model and the discriminative model. These are learned jointly by alternat...
Deep belief networks are compact universal approximators
Neural Computation, 2010
Deep Belief Networks (DBN) are generative models with many layers of hidden causal variables, recently introduced by Hinton et al. (2006), along with a greedy layer-wise unsupervised learning algorithm. Building on Le Roux and Bengio (2008) and Sutskever and Hinton , we show that deep but narrow generative networks do not require more parameters than shallow ones to achieve universal approximation. Exploiting the proof technique, we prove that deep but narrow feed-forward neural networks with sigmoidal units can represent any Boolean expression.
Parallelization of Deep Networks
dice.ucl.ac.be
Learning multiple levels of feature detectors in Deep Belief Networks is a promising approach both for neuro-cognitive modeling and for practical applications, but it comes at the cost of high computational requirements. Here we propose a method for the parallelization of unsupervised generative learning in deep networks based on distributing training data among multiple computational nodes in a cluster. We show that this approach significantly reduces the training time with very limited cost on performance. We also show that a layerwise convergence stopping criterion yields faster training.
A Layer-Wise Information Reinforcement Approach to Improve Learning in Deep Belief Networks
2020
With the advent of deep learning, the number of works proposing new methods or improving existent ones has grown exponentially in the last years. In this scenario, “very deep” models were emerging, once they were expected to extract more intrinsic and abstract features while supporting a better performance. However, such models suffer from the gradient vanishing problem, i.e., backpropagation values become too close to zero in their shallower layers, ultimately causing learning to stagnate. Such an issue was overcome in the context of convolution neural networks by creating “shortcut connections” between layers, in a so-called deep residual learning framework. Nonetheless, a very popular deep learning technique called Deep Belief Network still suffers from gradient vanishing when dealing with discriminative tasks. Therefore, this paper proposes the Residual Deep Belief Network, which considers the information reinforcement layer-by-layer to improve the feature extraction and knowled...
VARIOUS FRAMEWORKS 2 . 1 Deep 2 learning : Deep belief networks ( DBN )
2016
Object detection and recognition are important problems in computer vision and pattern recognition domain. Human beings are able to detect and classify objects effortlessly but replication of this ability on computer based systems has proved to be a non-trivial task. In particular, despite significant research efforts focused on meta-heuristic object detection and recognition, robust and reliable object recognition systems in real time remain elusive. Here we present a survey of one particular approach that has proved very promising for invariant feature recognition and which is a key initial stage of multi-stage network architecture methods for the high level task of object recognition.
GANEx: A complete pipeline of training, inference and benchmarking GAN experiments
2019
Deep learning (DL) is one of the standard methods in the field of multimedia research to perform data classification, detection, segmentation and generation. Within DL, generative adversarial networks (GANs) represents a new and highly popular branch of methods. GANs have the capability to generate, from random noise or conditional input, new data realizations within the dataset population. While generation is popular and highly useful in itself, GANs can also be useful to improve supervised DL. GAN-based approaches can, for example, perform segmentation or create synthetic data for training other DL models. The latter one is especially interesting in domains where not much training data exists such as medical multimedia. In this respect, performing a series of experiments involving GANs can be very time consuming due to the lack of tools that support the whole pipeline such as structured training, testing and tracking of different architectures and configurations. Moreover, the suc...
Loading Preview
Sorry, preview is currently unavailable. You can download the paper by clicking the button above.