Jinyan Guan

Improving the accuracy of water and bottom properties derived from remote sensing reflectance via artificial neural network

Proceedings of SPIE, Sep 13, 2007

Artificial neural network has been proven a useful technique for deriving water and bottom proper... more Artificial neural network has been proven a useful technique for deriving water and bottom properties from remote sensing upwelling radiance. Conventionally, a neural network is trained to minimize the overall mean square error of desired products. The approach does not explicitly take into account the change of spectral shapes of upwelling radiance. In this study, we have created four groups

Computational modeling of social face perception in humans: Leveraging the active appearance model

Face processing plays a central role in human social life. Humans readily infer social traits (e.... more Face processing plays a central role in human social life. Humans readily infer social traits (e.g. attractiveness and trustworthiness) from a stranger's face. Previous attempts to characterize the facial (physiognomic) features underlying social processing have lacked either systematicity or interpretability. Here, we utilize a statistical framework to tackle this problem, by learning a vector space to represent faces, and a linear mapping from this face space into human social trait judgments. Specifically, we obtain a face space by training the Active Appearance Model on large datasets of face images. Based on human evaluations of numerous social traits on these images, we then use regression to find linear combinations of facial features (what we call Linear Trait Axis, or LTA) that best predict human social judgments. Our model achieves state-of-the-art performance in overall predictive accuracy -comparable to the best convolutional neural network and better than human prediction of other human ratings. To interpret the LTAs, we regress them against a large repertoire of geometric features. To understand the relationship between the facial features that underlie different social, emotional, and demographic traits, we present a novel "dual space analysis" that characterizes the geometric relationship among LTA vectors. It shows that facial features important for social trait perception are largely distinct from those underlying demographic and emotion perception, contrary to previous suggestions that social trait perception is driven by over-generalization of relatively primitive demographic and emotion perception processes. In addition, we present a novel correlation decomposition analysis that quantifies how correlations in trait judgments (e.g. between attractiveness and babyfacedness) independently arise from (1) shared facial features among traits, and (2) correlation in the distribution of facial features in the human population.

Download

Multiple-Gaze Geometry: Inferring Novel 3D Locations from Gazes Observed in Monocular Video

Lecture Notes in Computer Science, 2018

We develop using person gaze direction for scene understanding. In particular, we use intersectin... more We develop using person gaze direction for scene understanding. In particular, we use intersecting gazes to learn 3D locations that people tend to look at, which is analogous to having multiple camera views. The 3D locations that we discover need not be visible to the camera. Conversely, knowing 3D locations of scene elements that draw visual attention, such as other people in the scene, can help infer gaze direction. We provide a Bayesian generative model for the temporal scene that captures the joint probability of camera parameters, locations of people, their gaze, what they are looking at, and locations of visual attention. Both the number of people in the scene and the number of extra objects that draw attention are unknown and need to be inferred. To execute this joint inference we use a probabilistic data association approach that enables principled comparison of model hypotheses. We use MCMC for inference over the discrete correspondence variables, and approximate the marginalization over continuous parameters using the Metropolis-Laplace approximation, using Hamiltonian (Hybrid) Monte Carlo for maximization. As existing data sets do not provide the 3D locations of what people are looking at, we contribute a small data set that does. On this data set, we infer what people are looking at with 59% precision compared with 13% for a baseline approach, and where those objects are within about 0.58m.

Download

Sampling bedrooms

We propose a top down approach for understanding indoor scenes such as bedrooms and living rooms.... more We propose a top down approach for understanding indoor scenes such as bedrooms and living rooms. These environments typically have the Manhattan world property that many surfaces are parallel to three principle ones. Further, the 3D geometry of the room and objects within it can largely be approximated by non overlapping simple structures such as single blocks (e.g. the room boundary), thin blocks (e.g. picture frames), and objects that are well modeled by single blocks (e.g. simple beds). We separately model the 3D geometry, the imaging process (camera parameters), and edge likelihood, to provide a generative statistical model for image data. We fit this model using data driven MCMC sampling. We combine reversible jump Metropolis Hastings samples for discrete changes in the model such as the number of blocks, and stochastic dynamics to estimate continuous parameter values in a particular parameter space that includes block positions, block sizes, and camera parameters. We tested our approach on two datasets using room box pixel orientation. Despite using only bounding box geometry and, in particular, not training on appearance, our method achieves results approaching those of others. We also introduce a new evaluation method for this domain based on ground truth camera parameters, which we found to be more sensitive to the task of understanding scene geometry.

Download

Bayesian 3D Tracking from Monocular Video

We develop a Bayesian modeling approach for tracking people in 3D from monocular video with unkno... more We develop a Bayesian modeling approach for tracking people in 3D from monocular video with unknown cameras. Modeling in 3D provides natural explanations for occlusions and smoothness discontinuities that result from projection, and allows priors on velocity and smoothness to be grounded in physical quantities: meters and seconds vs. pixels and frames. We pose the problem in the context of data association, in which observations are assigned to tracks. A correct application of Bayesian inference to multitarget tracking must address the fact that the model's dimension changes as tracks are added or removed, and thus, posterior densities of different hypotheses are not comparable. We address this by marginalizing out the trajectory parameters so the resulting posterior over data associations has constant dimension. This is made tractable by using (a) Gaussian process priors for smooth trajectories and (b) approximately Gaussian likelihood functions. Our approach provides a principled method for incorporating multiple sources of evidence; we present results using both optical flow and object detector outputs. Results are comparable to recent work on 3D tracking and, unlike others, our method requires no pre-calibrated cameras.

Download

SPM-based and ICA-based Approaches for Functional Magnetic Resonance Imaging (fMRI) Similarity Measures

I have been worked on two similarity measure techniques for fMRI images. One is SPM-based, which ... more I have been worked on two similarity measure techniques for fMRI images. One is SPM-based, which measure the similarity of Statistical Parametric Maps of the fMRI images, the other one is ICA-based, which measure the similarity of the voxels based on the Independent Component Analysis. In the SPMbased tool, I have implemented two image grouping techniques, graph-theory clustering and Replicator Dynamics, in order to group similar fMRI images. In ICA-based tool, I used bipartite graph matching to measure the similarity between fMRI images based on the similarities of the components.

Download

Bayesian Generative Modeling of Complex Dynamical Systems

This dissertation presents a Bayesian generative modeling approach for complex dynamical systems ... more This dissertation presents a Bayesian generative modeling approach for complex dynamical systems for emotion-interaction patterns within multivariate data collected in social psychology studies. While dynamical models have been used by social psychologists to study complex psychological and behavior patterns in recent years, most of these studies have been limited by using regression methods to fit the model parameters from noisy observations. These regression methods mostly rely on the estimates of the derivatives from the noisy observation, thus easily result in overfitting and fail to predict future outcomes. A Bayesian generative model solves the problem by integrating the prior knowledge of where the data comes from with the observed data through posterior distributions. It allows the development of theoretical ideas and mathematical models to be independent of the inference concerns. Besides, Bayesian generative statistical modeling allows evaluation of the model based on its predictive power instead of the model residual error reduction in regression methods to prevent overfitting in social psychology data analysis. In the proposed Bayesian generative modeling approach, this dissertation uses the State Space Model (SSM) to model the dynamics of emotion interactions. Specifically, it tests the approach in a class of psychological models aimed at explaining the emotional dynamics of interacting couples in committed relationships. The latent states of the SSM are composed of continuous real numbers that represent the level of the true emotional states of both partners. One can obtain the latent states at all subsequent time points by evolving a differential equation (typically a coupled linear oscillator (CLO)) forward in time with some known initial state at the starting time. The multivariate observed states include self-reported emotional experiences and physiological measurements of both partners during the interactions. To test whether well-being factors, such as body weight, can help to predict emotion-interaction patterns, we construct functions that determine the prior distributions of the CLO parameters of individual couples based on existing emotion theories. Besides, we allow a single latent state to generate multivariate observations and learn the group-shared coefficients that specify the relationship between the latent states and the multivariate observations. Furthermore, we model the nonlinearity of the emotional interaction by allowing smooth changes (drift) in the model parameters. By restricting the stochasticity to the parameter level, the proposed approach models the dynamics in longer periods of social interactions assuming that the interaction dynamics slowly and smoothly vary over time. The proposed approach achieves this by applying Gaussian Process (GP) priors with smooth covariance functions to the CLO parameters. Also, we propose to model the emotion regulation patterns as clusters of the dynamical parameters. To infer the parameters of the proposed Bayesian generative model from noisy experimental data, we develop a Gibbs sampler to learn the parameters of the patterns using a set of training couples. To evaluate the fitted model, we develop a multi-level cross-validation procedure for learning the group-shared parameters and distributions from training data and testing the learned models on held-out testing data. During testing, we use the learned shared model parameters to fit the individual CLO parameters to the first 80% of the time points of the testing data by Monte Carlo sampling and then predict the states of the last 20% of the time points. By evaluating models with cross-validation, one can estimate whether complex models are overfitted to noisy observations and fail to generalize to unseen data. I test our approach on both synthetic data that was generated by the generative model and real data that was collected in multiple social psychology experiments. The proposed approach has the potential to model other complex behavior since the generative model is not restricted to the forms of the underlying dynamics

Action Recognition in the Frequency Domain

arXiv (Cornell University), Sep 2, 2014

In this paper, we describe a simple strategy for mitigating variability in temporal data series b... more In this paper, we describe a simple strategy for mitigating variability in temporal data series by shifting focus onto long-term, frequency domain features that are less susceptible to variability. We apply this method to the human action recognition task and demonstrate how working in the frequency domain can yield good recognition features for commonly used optical flow and articulated pose features, which are highly sensitive to small differences in motion, viewpoint, dynamic backgrounds, occlusion and other sources of variability. We show how these frequency-based features can be used in combination with a simple forest classifier to achieve good and robust results on the popular KTH Actions dataset. * This work was partially supported by the DARPA Minds Eye program. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright annotation thereon. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of DARPA or the U.S. Government.

Download

Computational Temporal Interpersonal Emotion Systems

Reliable Offline Model-based Optimization for Industrial Process Control

arXiv (Cornell University), May 15, 2022

This study focuses on two important problems related to applying offline model-based optimization... more This study focuses on two important problems related to applying offline model-based optimization to realworld industrial control problems. The first problem is how to create a reliable probabilistic model that accurately captures the dynamics present in noisy industrial data. The second problem is how to reliably optimize control parameters without actively collecting feedback from industrial systems. Specifically, we introduce a novel cGAN ensemble-based uncertainty-aware surrogate model for reliable offline model-based optimization in industrial control problems. The effectiveness of the proposed method is demonstrated through extensive experiments conducted on two representative cases, namely a discrete control case and a continuous control case. The results of these experiments show that our method outperforms several competitive baselines in the field of offline model-based optimization for industrial control.

Download

Computational Temporal Interpersonal Emotion Systems

Bayesian Generative Modeling of Complex Dynamical Systems

This dissertation presents a Bayesian generative modeling approach for complex dynamical systems ... more This dissertation presents a Bayesian generative modeling approach for complex dynamical systems for emotion-interaction patterns within multivariate data collected in social psychology studies. While dynamical models have been used by social psychologists to study complex psychological and behavior patterns in recent years, most of these studies have been limited by using regression methods to fit the model parameters from noisy observations. These regression methods mostly rely on the estimates of the derivatives from the noisy observation, thus easily result in overfitting and fail to predict future outcomes. A Bayesian generative model solves the problem by integrating the prior knowledge of where the data comes from with the observed data through posterior distributions. It allows the development of theoretical ideas and mathematical models to be independent of the inference concerns. Besides, Bayesian generative statistical modeling allows evaluation of the model based on its predictive power instead of the model residual error reduction in regression methods to prevent overfitting in social psychology data analysis. In the proposed Bayesian generative modeling approach, this dissertation uses the State Space Model (SSM) to model the dynamics of emotion interactions. Specifically, it tests the approach in a class of psychological models aimed at explaining the emotional dynamics of interacting couples in committed relationships. The latent states of the SSM are composed of continuous real numbers that represent the level of the true emotional states of both partners. One can obtain the latent states at all subsequent time points by evolving a differential equation (typically a coupled linear oscillator (CLO)) forward in time with some known initial state at the starting time. The multivariate observed states include self-reported emotional experiences and physiological measurements of both partners during the interactions. To test whether well-being factors, such as body weight, can help to predict emotion-interaction patterns, we construct functions that determine the prior distributions of the CLO parameters of individual couples based on existing emotion theories. Besides, we allow a single latent state to generate multivariate observations and learn the group-shared coefficients that specify the relationship between the latent states and the multivariate observations. Furthermore, we model the nonlinearity of the emotional interaction by allowing smooth changes (drift) in the model parameters. By restricting the stochasticity to the parameter level, the proposed approach models the dynamics in longer periods of social interactions assuming that the interaction dynamics slowly and smoothly vary over time. The proposed approach achieves this by applying Gaussian Process (GP) priors with smooth covariance functions to the CLO parameters. Also, we propose to model the emotion regulation patterns as clusters of the dynamical parameters. To infer the parameters of the proposed Bayesian generative model from noisy experimental data, we develop a Gibbs sampler to learn the parameters of the patterns using a set of training couples. To evaluate the fitted model, we develop a multi-level cross-validation procedure for learning the group-shared parameters and distributions from training data and testing the learned models on held-out testing data. During testing, we use the learned shared model parameters to fit the individual CLO parameters to the first 80% of the time points of the testing data by Monte Carlo sampling and then predict the states of the last 20% of the time points. By evaluating models with cross-validation, one can estimate whether complex models are overfitted to noisy observations and fail to generalize to unseen data. I test our approach on both synthetic data that was generated by the generative model and real data that was collected in multiple social psychology experiments. The proposed approach has the potential to model other complex behavior since the generative model is not restricted to the forms of the underlying dynamics

A Model-Based Investigating of the Biological Origin of Human Social Perception of Faces

DMP Summer Research Internship Final Report

I have been worked on two similarity measure techniques for fMRI images. One is SPM-based, which ... more I have been worked on two similarity measure techniques for fMRI images. One is SPM-based, which measure the similarity of Statistical Parametric Maps of the fMRI images, the other one is ICA-based, which measure the similarity of the voxels based on the Independent Component Analysis. In the SPMbased tool, I have implemented two image grouping techniques, graph-theory clustering and Replicator Dynamics, in order to group similar fMRI images. In ICA-based tool, I used bipartite graph matching to measure the similarity between fMRI images based on the similarities of the components.

Download

Multiple-Gaze Geometry: Inferring Novel 3D Locations from Gazes Observed in Monocular Video

Computer Vision – ECCV 2018, 2018

We develop using person gaze direction for scene understanding. In particular, we use intersectin... more We develop using person gaze direction for scene understanding. In particular, we use intersecting gazes to learn 3D locations that people tend to look at, which is analogous to having multiple camera views. The 3D locations that we discover need not be visible to the camera. Conversely, knowing 3D locations of scene elements that draw visual attention, such as other people in the scene, can help infer gaze direction. We provide a Bayesian generative model for the temporal scene that captures the joint probability of camera parameters, locations of people, their gaze, what they are looking at, and locations of visual attention. Both the number of people in the scene and the number of extra objects that draw attention are unknown and need to be inferred. To execute this joint inference we use a probabilistic data association approach that enables principled comparison of model hypotheses. We use MCMC for inference over the discrete correspondence variables, and approximate the marginalization over continuous parameters using the Metropolis-Laplace approximation, using Hamiltonian (Hybrid) Monte Carlo for maximization. As existing data sets do not provide the 3D locations of what people are looking at, we contribute a small data set that does. On this data set, we infer what people are looking at with 59% precision compared with 13% for a baseline approach, and where those objects are within about 0.58m.

Download

Computational modeling of social face perception in humans: Leveraging the active appearance model

Face processing plays a central role in human social life. Humans readily infer social traits (e.... more Face processing plays a central role in human social life. Humans readily infer social traits (e.g. attractiveness and trustworthiness) from a stranger’s face. Previous attempts to characterize the facial (physiognomic) features underlying social processing have lacked either systematicity or interpretability. Here, we utilize a statistical framework to tackle this problem, by learning a vector space to represent faces, and a linear mapping from this face space into human social trait judgments. Specifically, we obtain a face space by training the Active Appearance Model on large datasets of face images. Based on human evaluations of numerous social traits on these images, we then use regression to find linear combinations of facial features (what we call Linear Trait Axis, or LTA) that best predict human social judgments. Our model achieves state-of-the-art performance in overall predictive accuracy – comparable to the best convolutional neural network and better than human predict...

Download

An independent component analysis based tool for exploring functional connections in the brain

SPIE Proceedings, 2009

Download

Bayesian 3D Tracking from Monocular Video

2013 IEEE International Conference on Computer Vision, 2013

We develop a Bayesian modeling approach for tracking people in 3D from monocular video with unkno... more We develop a Bayesian modeling approach for tracking people in 3D from monocular video with unknown cameras. Modeling in 3D provides natural explanations for occlusions and smoothness discontinuities that result from projection, and allows priors on velocity and smoothness to be grounded in physical quantities: meters and seconds vs. pixels and frames. We pose the problem in the context of data association, in which observations are assigned to tracks. A correct application of Bayesian inference to multitarget tracking must address the fact that the model's dimension changes as tracks are added or removed, and thus, posterior densities of different hypotheses are not comparable. We address this by marginalizing out the trajectory parameters so the resulting posterior over data associations has constant dimension. This is made tractable by using (a) Gaussian process priors for smooth trajectories and (b) approximately Gaussian likelihood functions. Our approach provides a principled method for incorporating multiple sources of evidence; we present results using both optical flow and object detector outputs. Results are comparable to recent work on 3D tracking and, unlike others, our method requires no pre-calibrated cameras.

Download

SPM-based and ICA-based Approaches for Functional Magnetic Resonance Imaging (fMRI) Similarity Measures

I have been worked on two similarity measure techniques for fMRI images. One is SPM-based, which ... more I have been worked on two similarity measure techniques for fMRI images. One is SPM-based, which measure the similarity of Statistical Parametric Maps of the fMRI images, the other one is ICA-based, which measure the similarity of the voxels based on the Independent Component Analysis. In the SPMbased tool, I have implemented two image grouping techniques, graph-theory clustering and Replicator Dynamics, in order to group similar fMRI images. In ICA-based tool, I used bipartite graph matching to measure the similarity between fMRI images based on the similarities of the components.

Download

Action Recognition in the Frequency Domain

In this paper, we describe a simple strategy for mitigating variability in temporal data series b... more In this paper, we describe a simple strategy for mitigating variability in temporal data series by shifting focus onto long-term, frequency domain features that are less susceptible to variability. We apply this method to the human action recognition task and demonstrate how working in the frequency domain can yield good recognition features for commonly used optical flow and articulated pose features, which are highly sensitive to small differences in motion, viewpoint, dynamic backgrounds, occlusion and other sources of variability. We show how these frequency-based features can be used in combination with a simple forest classifier to achieve good and robust results on the popular KTH Actions dataset. * This work was partially supported by the DARPA Minds Eye program. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright annotation thereon. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of DARPA or the U.S. Government.

Download

Related Authors

Uploads

Papers by Jinyan Guan

Log In