We consider a retailer selling a single product with limited on-hand inventory over a finite sell... more We consider a retailer selling a single product with limited on-hand inventory over a finite selling season. Customer demand arrives according to a Poisson process, the rate of which is influenced by a single action taken by the retailer (such as price adjustment, sales commission, advertisement intensity, etc.). The relationship between the action and the demand rate is not known in advance. However, the retailer is able to learn the optimal action "on the fly" as she maximizes her total expected revenue based on the observed demand reactions. Using the pricing problem as an example, we propose a dynamic "learning-while-doing" algorithm that only involves function value estimation to achieve a near-optimal performance. Our algorithm employs a series of shrinking price intervals and iteratively tests prices within that interval using a set of carefully chosen parameters. We prove that the convergence rate of our algorithm is among the fastest of all possible algo...
We propose a number of techniques for obtaining a global ranking from data that may be incomplete... more We propose a number of techniques for obtaining a global ranking from data that may be incomplete and imbalanced -- characteristics almost universal to modern datasets coming from e-commerce and internet applications. We are primarily interested in score or rating-based cardinal data. From raw ranking data, we construct pairwise rankings, represented as edge flows on an appropriate graph. Our statistical ranking method uses the graph Helmholtzian, the graph theoretic analogue of the Helmholtz operator or vector Laplacian, in much the same way the graph Laplacian is an analogue of the Laplace operator or scalar Laplacian. We study the graph Helmholtzian using combinatorial Hodge theory: we show that every edge flow representing pairwise ranking can be resolved into two orthogonal components, a gradient flow that represents the L2-optimal global ranking and a divergence-free flow (cyclic) that measures the validity of the global ranking obtained -- if this is large, then the data does...
In this paper, we propose ℓ_p-norm regularized models to seek near-optimal sparse portfolios. The... more In this paper, we propose ℓ_p-norm regularized models to seek near-optimal sparse portfolios. These sparse solutions reduce the complexity of portfolio implementation and management. Theoretical results are established to guarantee the sparsity of the second-order KKT points of the ℓ_p-norm regularized models. More interestingly, we present a theory that relates sparsity of the KKT points with Projected correlation and Projected Sharpe ratio. We also design an interior point algorithm to obtain an approximate second-order KKT solution of the ℓ_p-norm models in polynomial time with a fixed error tolerance, and then test our ℓ_p-norm modes on S&P 500 (2008-2012) data and international market data. The computational results illustrate that the ℓ_p-norm regularized models can generate portfolios of any desired sparsity with portfolio variance and portfolio return comparable to those of the unregularized Markowitz model with cardinality constraint. Our analysis of a combined model lead u...
The Alternating Direction Method of Multipliers (ADMM) has now days gained tremendous attentions ... more The Alternating Direction Method of Multipliers (ADMM) has now days gained tremendous attentions for solving large-scale machine learning and signal processing problems due to the relative simplicity. However, the two-block structure of the classical ADMM still limits the size of the real problems being solved. When one forces a more-than-two-block structure by variable-splitting, the convergence speed slows down greatly as observed in practice. Recently, a randomly assembled cyclic multi-block ADMM (RAC-MBADMM) was developed by the authors for solving general convex and nonconvex quadratic optimization problems where the number of blocks can go greater than two so that each sub-problem has a smaller size and can be solved much more efficiently. In this paper, we apply this method to solving few selected machine learning problems related to convex quadratic optimization, such as Linear Regression, LASSO, Elastic-Net, and SVM. We prove that the algorithm would converge in expectation...
OATAO is an open access repository that collects the work of some Toulouse researchers and makes ... more OATAO is an open access repository that collects the work of some Toulouse researchers and makes it freely available over the web where possible.
The contribution of this paper is twofolds. First, it is shown that while robust in terms of the ... more The contribution of this paper is twofolds. First, it is shown that while robust in terms of the average output error, the least squares estimate is sensitive to outliers with respect to the maximum output error. In fact the worst-case output error of the least squares can go unbounded. Then, a constrained logarithmic least squares for system identi cation is proposed. Analytic center algorithms are presented to solve this constrained logarithmic least squares problem.
We show that the direct extension of alternating direction method of multipliers (ADMM) with thre... more We show that the direct extension of alternating direction method of multipliers (ADMM) with three blocks is not necessarily convergent even for solving a square system of linear equations, although its convergence proof was established 40 years ago with one or two-block. However, we prove that, in each iteration if one randomly and independently permutes the updating order of variable blocks followed by the regular multiplier update, then ADMM will converge in expectation when solving any square system of linear equations with any number of blocks. We also discuss its extension to solve general convex optimization problems, in particular, linear and quadratic programs.
We focus on a permutation betting market under parimutuel call auction model where traders bet on... more We focus on a permutation betting market under parimutuel call auction model where traders bet on the final ranking of n candidates. We present a Proportional Betting mechanism for this market. Our mechanism allows the traders to bet on any subset of the n x n 'candidate-rank' pairs, and rewards them proportionally to the number of pairs that appear in the final outcome. We show that market organizer's decision problem for this mechanism can be formulated as a convex program of polynomial size. More importantly, the formulation yields a set of n x n unique marginal prices that are sufficient to price the bets in this mechanism, and are computable in polynomial-time. The marginal prices reflect the traders' beliefs about the marginal distributions over outcomes. We also propose techniques to compute the joint distribution over n! permutations from these marginal distributions. We show that using a maximum entropy criterion, we can obtain a concise parametric form (wit...
Let (G,P) be a bar framework of n vertices in general position in R^d, d <= n-1, where G is a ... more Let (G,P) be a bar framework of n vertices in general position in R^d, d <= n-1, where G is a (d+1)-lateration graph. In this paper, we present a constructive proof that (G,P) admits a positive semi-definite stress matrix with rank n-d-1. We also prove a similar result for a sensor network where the graph consists of m(>= d+1) anchors.
A Semidefinite Programming (SDP) relaxation is an effective computational method to solve a Senso... more A Semidefinite Programming (SDP) relaxation is an effective computational method to solve a Sensor Network Localization problem, which attempts to determine the locations of a group of sensors given the distances between some of them [11]. In this paper, we analyze and determine new sufficient conditions and formulations that guarantee that the SDP relaxation is exact, i.e., gives the correct solution. These conditions can be useful for designing sensor networks and managing connectivities in practice. Our main contribution is twofold: We present the first non-asymptotic bound on the connectivity or radio range requirement of the sensors in order to ensure the network is uniquely localizable. Determining this range is a key component in the design of sensor networks, and we provide a result that leads to a correct localization of each sensor, for any number of sensors. Second, we introduce a new class of graphs that can always be correctly localized by an SDP relaxation. Specificall...
We establish that mass conserving single terminal-linkage networks of chemical reactions admit po... more We establish that mass conserving single terminal-linkage networks of chemical reactions admit positive steady states regardless of network deficiency and the choice of reaction rate constants. This result holds for closed systems without material exchange across the boundary, as well as for open systems with material exchange at rates that satisfy a simple sufficient and necessary condition. Our proof uses a fixed point of a novel convex optimization formulation to find the steady state behavior of chemical reaction networks that satisfy the law of mass-action kinetics. A fixed point iteration can be used to compute these steady states, and we show that it converges for weakly reversible homogeneous systems. We report the results of our algorithm on numerical experiments.
Let P be a polytope satisfying that each row of the defining matrix has at most one positive entr... more Let P be a polytope satisfying that each row of the defining matrix has at most one positive entry. Determining whether there is an integer point in P is known to be an NP-complete problem. By introducing an integer labeling rule on an augmented set and applying a triangulation of the Euclidean space, we develop in this paper a variable dimension method for computing an integer point in P. The method starts from an arbitrary integer point and follows a finite simplicial path that either leads to an integer point in P or proves no such point exists.
In this paper we provide faster algorithms for approximately solving discounted Markov Decision P... more In this paper we provide faster algorithms for approximately solving discounted Markov Decision Processes in multiple parameter regimes. Given a discounted Markov Decision Process (DMDP) with |S| states, |A| actions, discount factor γ ∈ (0, 1), and rewards in the range [-M, M ], we show how to compute an ǫ-optimal policy, with probability 1 -δ in time 1 This contribution reflects the first nearly linear time, nearly linearly convergent algorithm for solving DMDP's for intermediate values of γ. We also show how to obtain improved sublinear time algorithms and provide an algorithm which computes an ǫ-optimal policy with probability 1 -δ in time provided we can sample from the transition function in O(1) time. Interestingly, we obtain our results by a careful modification of approximate value iteration. We show how to combine classic approximate value iteration analysis with new techniques in variance reduction. Our fastest algorithms leverage further insights to ensure that our algorithms make monotonic progress towards the optimal value. This paper is one of few instances in using sampling to obtain a linearly convergent linear programming algorithm and we hope that the analysis may be useful more broadly.
We present computational experience with an interior-point algorithm for large-scale quadratic pr... more We present computational experience with an interior-point algorithm for large-scale quadratic programming problems with box constraints. The algorithm requires a total of O (√ nL) number of iterations, where L is the size of the input data of the problem, and O (n 3) arithmetic operations per iteration. The algorithm has been implemented using vectorization and tested on an IBM 3090-600S computer with vector facilities. The computational results suggest that the efficiency of the algorithm depends on an appropriate choice of ...
We survey recent developments of using semidefinite programming for solving the sensor network an... more We survey recent developments of using semidefinite programming for solving the sensor network and graph localization problem. The semidefinite programming (SDP) relaxation based method was initially proposed by [15; 16], theoretically analyzed in [44], and further improved by a gradient-based local search method from [32]. An optimization problem is set up so as to minimize the error in sensor positions to fit distance measures. Observable criteria are developed to certify the quality of the point estimation of sensors or to detect erroneous sensors. The performance of this technique is highly satisfactory compared to other techniques. Very few anchor nodes are required to accurately estimate the position of all the unknown nodes in a network. Also the estimation errors are minimal even when the anchor nodes are not suitably placed within the network or the distance measurements are noisy.
[1], [2] have shown that dynamic spectrum management (DSM) using the market competitive equilibri... more [1], [2] have shown that dynamic spectrum management (DSM) using the market competitive equilibrium (CE), which sets a price for transmission power on each channel, leads to better system performance in terms of the total data transmission rate (by reducing cross talk), than using the Nash equilibrium (NE). But how to achieve such a CE is an open problem. We show that the CE is the solution of a linear complementarity problem (LCP) and can be computed efficiently. We propose a decentralized tâtonnement process for adjusting the prices to achieve a CE. We show that under reasonable conditions, any tâtonnement process converges to the CE. The conditions are that users of a channel experience the same noise levels and that the cross-talk effects between users are low-rank and weak.
Diagonal preconditioning has been a staple technique in optimization and machine learning. It oft... more Diagonal preconditioning has been a staple technique in optimization and machine learning. It often reduces the condition number of the design or Hessian matrix it is applied to, thereby speeding up convergence. However, rigorous analyses of how well various diagonal preconditioning procedures improve the condition number of the preconditioned matrix and how that translates into improvements in optimization are rare. In this paper, we first provide an analysis of a popular diagonal preconditioning technique based on column standard deviation and its effect on the condition number using random matrix theory. Then we identify a class of design matrices whose condition numbers can be reduced significantly by this procedure. We then study the problem of optimal diagonal preconditioning to improve the condition number of any full-rank matrix and provide a bisection algorithm and a potential reduction algorithm with O(log(1/ϵ)) iteration complexity, where each iteration consists of an SDP...
This paper concerns the worst-case complexity of cyclic coordinate descent (C-CD) for minimizing ... more This paper concerns the worst-case complexity of cyclic coordinate descent (C-CD) for minimizing a convex quadratic function, which is equivalent to Gauss-Seidel method and can be transformed to Kaczmarz method and projection onto convex sets (POCS). We observe that the known provable complexity of C-CD can be O(n^2) times slower than randomized coordinate descent (R-CD), but no example was rigorously proven to exhibit such a large gap. In this paper we show that the gap indeed exists. We prove that there exists an example for which C-CD takes at least O(n^4 κ_CD1/ϵ) operations, where κ_CD is related to Demmel's condition number and it determines the convergence rate of R-CD. It implies that in the worst case C-CD can indeed be O(n^2) times slower than R-CD, which has complexity O( n^2 κ_CD1/ϵ). Note that for this example, the gap exists for any fixed update order, not just a particular order. Based on the example, we establish several almost tight complexity bounds of C-CD for ...
A little story in the development of semidefinite programming (SDP), a major subclass of conic li... more A little story in the development of semidefinite programming (SDP), a major subclass of conic linear programming. One day in 1990, I visited the Computer Science Department of the University of Minnesota and met a young graduate student, Farid Alizadeh. He, working then on combinatorial optimization, introduced me "semidefinite optimization" or linear programming over the positive definite matrix cone. We had a very extensive discussion that afternoon and concluded that interior-point linear programming algorithms could be applicable to solving SDPs. I suggested Farid to look at the linear programming (LP) interior-point algorithms and to develop an SDP (primal) potential reduction algorithm. He worked hard for several months, and one afternoon showed up in my office in Iowa City, about 300 miles from Minneapolis. He had everything worked out, including potential function, algorithm, complexity bound, and even a "dictionary" list between LP and SDP. But he was stuck on one problem that was on how to keep the symmetry of the scaled directional matrix. We went to a bar nearby on Clinton Street in Iowa City (I paid for him since I was a third-year professor then and eager to demonstrate that I could take care of my students). After chatting for a while, I suggested that he should use scaling X −1/2 ∆X −1/2 to compute symmetric directional matrix ∆, instead of X −1 ∆ which he was using earlier, where X is the current symmetric positive definite matrix. This way, X + α∆ would remain symmetric with a step-size scalar. He returned to Minneapolis and moved to Berkeley shortly after, and few weeks later sent me an e-mail message telling me that everything had worked out beautifully. At the same time, Nesterov and Nemirovskii developed a more general and powerful theory in extending interior-point algorithms for solving convex programs, where SDP was a special case. Boyd and his group presented a wide range of SDP applications and formulations, many of which were incredibly novel and elegant.
We consider a retailer selling a single product with limited on-hand inventory over a finite sell... more We consider a retailer selling a single product with limited on-hand inventory over a finite selling season. Customer demand arrives according to a Poisson process, the rate of which is influenced by a single action taken by the retailer (such as price adjustment, sales commission, advertisement intensity, etc.). The relationship between the action and the demand rate is not known in advance. However, the retailer is able to learn the optimal action "on the fly" as she maximizes her total expected revenue based on the observed demand reactions. Using the pricing problem as an example, we propose a dynamic "learning-while-doing" algorithm that only involves function value estimation to achieve a near-optimal performance. Our algorithm employs a series of shrinking price intervals and iteratively tests prices within that interval using a set of carefully chosen parameters. We prove that the convergence rate of our algorithm is among the fastest of all possible algo...
We propose a number of techniques for obtaining a global ranking from data that may be incomplete... more We propose a number of techniques for obtaining a global ranking from data that may be incomplete and imbalanced -- characteristics almost universal to modern datasets coming from e-commerce and internet applications. We are primarily interested in score or rating-based cardinal data. From raw ranking data, we construct pairwise rankings, represented as edge flows on an appropriate graph. Our statistical ranking method uses the graph Helmholtzian, the graph theoretic analogue of the Helmholtz operator or vector Laplacian, in much the same way the graph Laplacian is an analogue of the Laplace operator or scalar Laplacian. We study the graph Helmholtzian using combinatorial Hodge theory: we show that every edge flow representing pairwise ranking can be resolved into two orthogonal components, a gradient flow that represents the L2-optimal global ranking and a divergence-free flow (cyclic) that measures the validity of the global ranking obtained -- if this is large, then the data does...
In this paper, we propose ℓ_p-norm regularized models to seek near-optimal sparse portfolios. The... more In this paper, we propose ℓ_p-norm regularized models to seek near-optimal sparse portfolios. These sparse solutions reduce the complexity of portfolio implementation and management. Theoretical results are established to guarantee the sparsity of the second-order KKT points of the ℓ_p-norm regularized models. More interestingly, we present a theory that relates sparsity of the KKT points with Projected correlation and Projected Sharpe ratio. We also design an interior point algorithm to obtain an approximate second-order KKT solution of the ℓ_p-norm models in polynomial time with a fixed error tolerance, and then test our ℓ_p-norm modes on S&P 500 (2008-2012) data and international market data. The computational results illustrate that the ℓ_p-norm regularized models can generate portfolios of any desired sparsity with portfolio variance and portfolio return comparable to those of the unregularized Markowitz model with cardinality constraint. Our analysis of a combined model lead u...
The Alternating Direction Method of Multipliers (ADMM) has now days gained tremendous attentions ... more The Alternating Direction Method of Multipliers (ADMM) has now days gained tremendous attentions for solving large-scale machine learning and signal processing problems due to the relative simplicity. However, the two-block structure of the classical ADMM still limits the size of the real problems being solved. When one forces a more-than-two-block structure by variable-splitting, the convergence speed slows down greatly as observed in practice. Recently, a randomly assembled cyclic multi-block ADMM (RAC-MBADMM) was developed by the authors for solving general convex and nonconvex quadratic optimization problems where the number of blocks can go greater than two so that each sub-problem has a smaller size and can be solved much more efficiently. In this paper, we apply this method to solving few selected machine learning problems related to convex quadratic optimization, such as Linear Regression, LASSO, Elastic-Net, and SVM. We prove that the algorithm would converge in expectation...
OATAO is an open access repository that collects the work of some Toulouse researchers and makes ... more OATAO is an open access repository that collects the work of some Toulouse researchers and makes it freely available over the web where possible.
The contribution of this paper is twofolds. First, it is shown that while robust in terms of the ... more The contribution of this paper is twofolds. First, it is shown that while robust in terms of the average output error, the least squares estimate is sensitive to outliers with respect to the maximum output error. In fact the worst-case output error of the least squares can go unbounded. Then, a constrained logarithmic least squares for system identi cation is proposed. Analytic center algorithms are presented to solve this constrained logarithmic least squares problem.
We show that the direct extension of alternating direction method of multipliers (ADMM) with thre... more We show that the direct extension of alternating direction method of multipliers (ADMM) with three blocks is not necessarily convergent even for solving a square system of linear equations, although its convergence proof was established 40 years ago with one or two-block. However, we prove that, in each iteration if one randomly and independently permutes the updating order of variable blocks followed by the regular multiplier update, then ADMM will converge in expectation when solving any square system of linear equations with any number of blocks. We also discuss its extension to solve general convex optimization problems, in particular, linear and quadratic programs.
We focus on a permutation betting market under parimutuel call auction model where traders bet on... more We focus on a permutation betting market under parimutuel call auction model where traders bet on the final ranking of n candidates. We present a Proportional Betting mechanism for this market. Our mechanism allows the traders to bet on any subset of the n x n 'candidate-rank' pairs, and rewards them proportionally to the number of pairs that appear in the final outcome. We show that market organizer's decision problem for this mechanism can be formulated as a convex program of polynomial size. More importantly, the formulation yields a set of n x n unique marginal prices that are sufficient to price the bets in this mechanism, and are computable in polynomial-time. The marginal prices reflect the traders' beliefs about the marginal distributions over outcomes. We also propose techniques to compute the joint distribution over n! permutations from these marginal distributions. We show that using a maximum entropy criterion, we can obtain a concise parametric form (wit...
Let (G,P) be a bar framework of n vertices in general position in R^d, d <= n-1, where G is a ... more Let (G,P) be a bar framework of n vertices in general position in R^d, d <= n-1, where G is a (d+1)-lateration graph. In this paper, we present a constructive proof that (G,P) admits a positive semi-definite stress matrix with rank n-d-1. We also prove a similar result for a sensor network where the graph consists of m(>= d+1) anchors.
A Semidefinite Programming (SDP) relaxation is an effective computational method to solve a Senso... more A Semidefinite Programming (SDP) relaxation is an effective computational method to solve a Sensor Network Localization problem, which attempts to determine the locations of a group of sensors given the distances between some of them [11]. In this paper, we analyze and determine new sufficient conditions and formulations that guarantee that the SDP relaxation is exact, i.e., gives the correct solution. These conditions can be useful for designing sensor networks and managing connectivities in practice. Our main contribution is twofold: We present the first non-asymptotic bound on the connectivity or radio range requirement of the sensors in order to ensure the network is uniquely localizable. Determining this range is a key component in the design of sensor networks, and we provide a result that leads to a correct localization of each sensor, for any number of sensors. Second, we introduce a new class of graphs that can always be correctly localized by an SDP relaxation. Specificall...
We establish that mass conserving single terminal-linkage networks of chemical reactions admit po... more We establish that mass conserving single terminal-linkage networks of chemical reactions admit positive steady states regardless of network deficiency and the choice of reaction rate constants. This result holds for closed systems without material exchange across the boundary, as well as for open systems with material exchange at rates that satisfy a simple sufficient and necessary condition. Our proof uses a fixed point of a novel convex optimization formulation to find the steady state behavior of chemical reaction networks that satisfy the law of mass-action kinetics. A fixed point iteration can be used to compute these steady states, and we show that it converges for weakly reversible homogeneous systems. We report the results of our algorithm on numerical experiments.
Let P be a polytope satisfying that each row of the defining matrix has at most one positive entr... more Let P be a polytope satisfying that each row of the defining matrix has at most one positive entry. Determining whether there is an integer point in P is known to be an NP-complete problem. By introducing an integer labeling rule on an augmented set and applying a triangulation of the Euclidean space, we develop in this paper a variable dimension method for computing an integer point in P. The method starts from an arbitrary integer point and follows a finite simplicial path that either leads to an integer point in P or proves no such point exists.
In this paper we provide faster algorithms for approximately solving discounted Markov Decision P... more In this paper we provide faster algorithms for approximately solving discounted Markov Decision Processes in multiple parameter regimes. Given a discounted Markov Decision Process (DMDP) with |S| states, |A| actions, discount factor γ ∈ (0, 1), and rewards in the range [-M, M ], we show how to compute an ǫ-optimal policy, with probability 1 -δ in time 1 This contribution reflects the first nearly linear time, nearly linearly convergent algorithm for solving DMDP's for intermediate values of γ. We also show how to obtain improved sublinear time algorithms and provide an algorithm which computes an ǫ-optimal policy with probability 1 -δ in time provided we can sample from the transition function in O(1) time. Interestingly, we obtain our results by a careful modification of approximate value iteration. We show how to combine classic approximate value iteration analysis with new techniques in variance reduction. Our fastest algorithms leverage further insights to ensure that our algorithms make monotonic progress towards the optimal value. This paper is one of few instances in using sampling to obtain a linearly convergent linear programming algorithm and we hope that the analysis may be useful more broadly.
We present computational experience with an interior-point algorithm for large-scale quadratic pr... more We present computational experience with an interior-point algorithm for large-scale quadratic programming problems with box constraints. The algorithm requires a total of O (√ nL) number of iterations, where L is the size of the input data of the problem, and O (n 3) arithmetic operations per iteration. The algorithm has been implemented using vectorization and tested on an IBM 3090-600S computer with vector facilities. The computational results suggest that the efficiency of the algorithm depends on an appropriate choice of ...
We survey recent developments of using semidefinite programming for solving the sensor network an... more We survey recent developments of using semidefinite programming for solving the sensor network and graph localization problem. The semidefinite programming (SDP) relaxation based method was initially proposed by [15; 16], theoretically analyzed in [44], and further improved by a gradient-based local search method from [32]. An optimization problem is set up so as to minimize the error in sensor positions to fit distance measures. Observable criteria are developed to certify the quality of the point estimation of sensors or to detect erroneous sensors. The performance of this technique is highly satisfactory compared to other techniques. Very few anchor nodes are required to accurately estimate the position of all the unknown nodes in a network. Also the estimation errors are minimal even when the anchor nodes are not suitably placed within the network or the distance measurements are noisy.
[1], [2] have shown that dynamic spectrum management (DSM) using the market competitive equilibri... more [1], [2] have shown that dynamic spectrum management (DSM) using the market competitive equilibrium (CE), which sets a price for transmission power on each channel, leads to better system performance in terms of the total data transmission rate (by reducing cross talk), than using the Nash equilibrium (NE). But how to achieve such a CE is an open problem. We show that the CE is the solution of a linear complementarity problem (LCP) and can be computed efficiently. We propose a decentralized tâtonnement process for adjusting the prices to achieve a CE. We show that under reasonable conditions, any tâtonnement process converges to the CE. The conditions are that users of a channel experience the same noise levels and that the cross-talk effects between users are low-rank and weak.
Diagonal preconditioning has been a staple technique in optimization and machine learning. It oft... more Diagonal preconditioning has been a staple technique in optimization and machine learning. It often reduces the condition number of the design or Hessian matrix it is applied to, thereby speeding up convergence. However, rigorous analyses of how well various diagonal preconditioning procedures improve the condition number of the preconditioned matrix and how that translates into improvements in optimization are rare. In this paper, we first provide an analysis of a popular diagonal preconditioning technique based on column standard deviation and its effect on the condition number using random matrix theory. Then we identify a class of design matrices whose condition numbers can be reduced significantly by this procedure. We then study the problem of optimal diagonal preconditioning to improve the condition number of any full-rank matrix and provide a bisection algorithm and a potential reduction algorithm with O(log(1/ϵ)) iteration complexity, where each iteration consists of an SDP...
This paper concerns the worst-case complexity of cyclic coordinate descent (C-CD) for minimizing ... more This paper concerns the worst-case complexity of cyclic coordinate descent (C-CD) for minimizing a convex quadratic function, which is equivalent to Gauss-Seidel method and can be transformed to Kaczmarz method and projection onto convex sets (POCS). We observe that the known provable complexity of C-CD can be O(n^2) times slower than randomized coordinate descent (R-CD), but no example was rigorously proven to exhibit such a large gap. In this paper we show that the gap indeed exists. We prove that there exists an example for which C-CD takes at least O(n^4 κ_CD1/ϵ) operations, where κ_CD is related to Demmel's condition number and it determines the convergence rate of R-CD. It implies that in the worst case C-CD can indeed be O(n^2) times slower than R-CD, which has complexity O( n^2 κ_CD1/ϵ). Note that for this example, the gap exists for any fixed update order, not just a particular order. Based on the example, we establish several almost tight complexity bounds of C-CD for ...
A little story in the development of semidefinite programming (SDP), a major subclass of conic li... more A little story in the development of semidefinite programming (SDP), a major subclass of conic linear programming. One day in 1990, I visited the Computer Science Department of the University of Minnesota and met a young graduate student, Farid Alizadeh. He, working then on combinatorial optimization, introduced me "semidefinite optimization" or linear programming over the positive definite matrix cone. We had a very extensive discussion that afternoon and concluded that interior-point linear programming algorithms could be applicable to solving SDPs. I suggested Farid to look at the linear programming (LP) interior-point algorithms and to develop an SDP (primal) potential reduction algorithm. He worked hard for several months, and one afternoon showed up in my office in Iowa City, about 300 miles from Minneapolis. He had everything worked out, including potential function, algorithm, complexity bound, and even a "dictionary" list between LP and SDP. But he was stuck on one problem that was on how to keep the symmetry of the scaled directional matrix. We went to a bar nearby on Clinton Street in Iowa City (I paid for him since I was a third-year professor then and eager to demonstrate that I could take care of my students). After chatting for a while, I suggested that he should use scaling X −1/2 ∆X −1/2 to compute symmetric directional matrix ∆, instead of X −1 ∆ which he was using earlier, where X is the current symmetric positive definite matrix. This way, X + α∆ would remain symmetric with a step-size scalar. He returned to Minneapolis and moved to Berkeley shortly after, and few weeks later sent me an e-mail message telling me that everything had worked out beautifully. At the same time, Nesterov and Nemirovskii developed a more general and powerful theory in extending interior-point algorithms for solving convex programs, where SDP was a special case. Boyd and his group presented a wide range of SDP applications and formulations, many of which were incredibly novel and elegant.
Uploads
Papers by Yinyu Ye