Papers by Eugene Feinberg
arXiv (Cornell University), Mar 6, 2012
For an upper semi-continuous set-valued mapping from one topological space to another and for a l... more For an upper semi-continuous set-valued mapping from one topological space to another and for a lower semi-continuous function defined on the product of these spaces, Berge's theorem states lower semi-continuity of the minimum of this function taken over the image sets. It assumes that the image sets are compact. For Hausdorff topological spaces, this paper extends Berge's theorem to set-valued mappings with possible noncompact image sets and studies relevant properties of minima. (Hu and Papageorgiou [5, Proposition 3.3, p. 83]). If u : X × Y → R is a lower semi-continuous function and Φ : X → 2 Y \ {∅} is a compact-valued upper semi-continuous set-valued mapping, then the function v : X → R is lower semi-continuous. Luque-Vásquez and Hernández-Lerma [7] provide an example of a continuous Φ with possible noncompact sets Φ(x) and of a lower semi-continuous function u(x, y) being inf-compact in y, when v(x) is not lower semi-continuous. In this paper, we extend Berge's theorem to possibly noncompact sets Φ(x), x ∈ X. Let Gr Z (Φ) = {(x, y) ∈ Z × Y : y ∈ Φ(x)}, where Z ⊆ X. For a topological space U, we denote by K(U) the family of all nonempty compact subsets of U. Definition 1.1. A function u : X × Y → R is called K-inf-compact on Gr X (Φ), if for every K ∈ K(X) this function is inf-compact on Gr K (Φ).
arXiv (Cornell University), Jan 9, 2014
This paper describes sufficient conditions for the existence of optimal policies for Partially Ob... more This paper describes sufficient conditions for the existence of optimal policies for Partially Observable Markov Decision Processes (POMDPs) with Borel state, observation, and action sets and with the expected total costs. Action sets may not be compact and one-step cost functions may be unbounded. The introduced conditions are also sufficient for the validity of optimality equations, semi-continuity of value functions, and convergence of value iterations to optimal values. Since POMDPs can be reduced to Completely Observable Markov Decision Processes (COMDPs), whose states are posterior state distributions, this paper focuses on the validity of the above mentioned optimality properties for COMDPs. The central question is whether transition probabilities for a COMDP are weakly continuous. We introduce sufficient conditions for this and show that the transition probabilities for a COMDP are weakly continuous, if transition probabilities of the underlying Markov Decision Process are weakly continuous and observation probabilities for the POMDP are continuous in the total variation. Moreover, the continuity in the total variation of the observation probabilities cannot be weakened to setwise continuity. The results are illustrated with counterexamples and examples.
arXiv (Cornell University), Nov 17, 2017
This note describes sufficient conditions under which total-cost and average-cost Markov decision... more This note describes sufficient conditions under which total-cost and average-cost Markov decision processes (MDPs) with general state and action spaces, and with weakly continuous transition probabilities, can be reduced to discounted MDPs. For undiscounted problems, these reductions imply the validity of optimality equations and the existence of stationary optimal policies. The reductions also provide methods for computing optimal policies. The results are applied to a capacitated inventory control problem with fixed costs and lost sales.
arXiv (Cornell University), May 18, 2017
This paper studies a periodic-review single-commodity setup-cost inventory model with backorders ... more This paper studies a periodic-review single-commodity setup-cost inventory model with backorders and holding/backlog costs satisfying quasiconvexity assumptions. We show that the Markov decision process for this inventory model satisfies the assumptions that lead to the validity of optimality equations for discounted and average-cost problems and to the existence of optimal (s, S) policies. In particular, we prove the equicontinuity of the family of discounted value functions and the convergence of optimal discounted lower thresholds to the optimal average-cost one for some sequences of discount factors converging to 1. If an arbitrary nonnegative amount of inventory can be ordered, we establish stronger convergence properties: (i) the optimal discounted lower thresholds s α converge to optimal average-cost lower threshold s; and (ii) the discounted relative value functions converge to average-cost relative value function. These convergence results previously were known only for subsequences of discount factors even for problems with convex holding/backlog costs. The results of this paper also hold for problems with fixed lead times.
arXiv (Cornell University), Jul 5, 2021
This paper studies transition probabilities from a Borel subset of a Polish space to a product of... more This paper studies transition probabilities from a Borel subset of a Polish space to a product of two Borel subsets of Polish spaces. For such transition probabilities it introduces and studies the property of semi-uniform Feller continuity. This paper provides several equivalent definitions of semiuniform Feller continuity and establishes its preservation under integration. The motivation for this study came from the theory of Markov decision processes with incomplete information, and this paper provides fundamental results useful for this theory.
arXiv (Cornell University), Aug 7, 2022
For expectation functions on metric spaces, we provide sufficient conditions for epi-convergence ... more For expectation functions on metric spaces, we provide sufficient conditions for epi-convergence under varying probability measures and integrands, and examine applications in the area of sieve estimators, mollifier smoothing, PDE-constrained optimization, and stochastic optimization with expectation constraints. As a stepping stone to epi-convergence of independent interest, we develop parametric Fatou's lemmas under mild integrability assumptions. In the setting of Suslin metric spaces, the assumptions are expressed in terms of Pasch-Hausdorff envelopes. For general metric spaces, the assumptions shift to semicontinuity of integrands also on the sample space, which then is assumed to be a metric space.
arXiv (Cornell University), Mar 27, 2019
This paper investigates natural conditions for the existence of optimal policies for a Markov dec... more This paper investigates natural conditions for the existence of optimal policies for a Markov decision process with incomplete information (MDPII) and with expected total costs. The MDPII is the classic model of a controlled stochastic process with incomplete state observations which is more general than Partially Observable Markov Decision Processes (POMDPs). For MDPIIs we introduce the notion of a semi-uniform Feller transition probability, which is stronger than the notion of a weakly continuous transition probability. We show that an MDPII has a semi-uniform Feller transition probability if and only if the corresponding belief MDP also has a semi-uniform Feller transition probability. This fact has several corollaries. In particular, it provides new and implies all known sufficient conditions for the existence of optimal policies for POMDPs with expected total costs.
arXiv (Cornell University), Dec 19, 2013
This note provides a simple example demonstrating that, if exact computations are allowed, the nu... more This note provides a simple example demonstrating that, if exact computations are allowed, the number of iterations required for the value iteration algorithm to find an optimal policy for discounted dynamic programming problems may grow arbitrarily quickly with the size of the problem. In particular, the number of iterations can be exponential in the number of actions. Thus, unlike policy iterations, the value iteration algorithm is not strongly polynomial for discounted dynamic programming.
arXiv (Cornell University), Sep 27, 2016
As is well known, average-cost optimality inequalities imply the existence of stationary optimal ... more As is well known, average-cost optimality inequalities imply the existence of stationary optimal policies for Markov Decision Processes with average costs per unit time, and these inequalities hold under broad natural conditions. This paper provides sufficient conditions for the validity of the average-cost optimality equation for an infinite state problem with weakly continuous transition probabilities and with possibly unbounded one-step costs and noncompact action sets. These conditions also imply the convergence of sequences of discounted relative value functions to average-cost relative value functions and the continuity of averagecost relative value functions. As shown in the paper, the classic periodic-review inventory control problem satisfies these conditions. Therefore, the optimality inequality holds in the form of an equality with a continuous average-cost relative value function for this problem. In addition, the K-convexity of discounted relative value functions and their convergence to average-cost relative value functions, when the discount factor increases to 1, imply the K-convexity of average-cost relative value functions. This implies that average-cost optimal (s, S) policies for the inventory control problem can be derived from the average-cost optimality equation.
arXiv (Cornell University), Sep 10, 2021
This paper describes the structure of solutions to Kolmogorov's equations for nonhomogeneous jump... more This paper describes the structure of solutions to Kolmogorov's equations for nonhomogeneous jump Markov processes and applications of these results to control of jump stochastic systems. These equations were studied by , who clarified in 1945 in the errata to that paper that some of its results covered only nonexplosive Markov processes. In this work, which is largely of a survey nature, the case of explosive processes is also considered. This paper is based on the invited talk presented by the authors at the conference "Chebyshev-200", and it describes the results of their joined studies with Manasa Mandava .
Теория вероятностей и ее применения, 2019
This note describes Fatou's lemma for a sequence of measures converging weakly to a finite measur... more This note describes Fatou's lemma for a sequence of measures converging weakly to a finite measure and for a sequence of functions whose negative parts are uniformly integrable with respect to these measures. The note also provides new formulations of uniform Fatou's lemma, uniform Lebesgue convergence theorem, the Dunford-Pettis theorem, and the fundamental theorem for Young measures based on the equivalence of uniform integrability and the apparently weaker property of asymptotic uniform integrability for sequences of functions and finite measures.
Naval Research Logistics, Aug 11, 2017
This paper studies convergence properties of optimal values and actions for discounted and averag... more This paper studies convergence properties of optimal values and actions for discounted and averagecost Markov Decision Processes (MDPs) with weakly continuous transition probabilities and applies these properties to the stochastic periodic-review inventory control problem with backorders, positive setup costs, and convex holding/backordering costs. The following results are established for MDPs with possibly noncompact action sets and unbounded cost functions: (i) convergence of value iterations to optimal values for discounted problems with possibly non-zero terminal costs, (ii) convergence of optimal finite-horizon actions to optimal infinite-horizon actions for total discounted costs, as the time horizon tends to infinity, and (iii) convergence of optimal discount-cost actions to optimal average-cost actions for infinite-horizon problems, as the discount factor tends to 1. Being applied to the setup-cost inventory control problem, the general results on MDPs imply the optimality of (s, S) policies and convergence properties of optimal thresholds. In particular this paper analyzes the setup-cost inventory control problem without two assumptions often used in the literature: (a) the demand is either discrete or continuous or (b) the backordering cost is higher than the cost of backordered inventory if the amount of backordered inventory is large. Beyer et al. [4] and the references therein. Early studies of MDPs dealt with finite-state problems and infinite-state problems with bounded costs. The case of average costs per unit time is more difficult than the case of total discounted costs. Sennott developed the theory for the average-cost criterion for countable-state problems with unbounded costs. Schäl developed the theory for uncountable state problems with discounted and average-cost criteria when action sets are compact. In particular, Schäl identified two groups of assumptions on transition probabilities: weak continuity and setwise continuity. As explained in Feinberg and Lewis [17, Section 4], models with weakly continuous transition probabilities are more natural for inventory control than models with setwise continuous transition probabilities. Hernández-Lerma and Lasserre [21] developed the theory for problems with setwise continuous transition probabilities, unbounded costs, and possibly noncompact action sets. Luque-Vasques and Hernández-Lerma [25] identified an additional technical difficulty in dealing with problems with weakly continuous transition probabilities even for finite-horizon problems by demonstrating that Berge's theorem, that ensures semi-continuity of the value function, does not hold for problems with noncompact action sets. Feinberg and Lewis [17] investigated total discounted costs for inf-compact cost functions and obtained sufficient optimality conditions for average costs. Compared to Schäl [30] these results required an additional local boundedness assumption that holds for inventory control problems, but its verification is not easy. Feinberg et al. introduced a natural class of K-inf-compact cost functions, extended Berge's theorem to noncompact action sets, and developed the theory of MDPs with weakly continuous transition probabilities, unbounded costs, and with the criteria of total discounted costs and longterm average costs. In particular, the results from do not require the validity of the local boundedness assumption. This simplifies their applications to inventory control problems. Such applications are considered in Section 6 below. The tutorial by Feinberg [11] describes in detail the applicability of recent results on MDPs to inventory control. Section 2 of this paper describes an MDP with an infinite state space, weakly continuous transition probabilities, possibly unbounded one-step costs, and possibly noncompact action sets. Sections 3 and 4 provide the results for discounted and average cost criteria. In particular, new results are provided on the following topics: (i) convergence of value iterations for discounted problems with possibly non-zero terminal values (Corollary 3.5), (ii) convergence of optimal finite-horizon actions to optimal infinite-horizon actions for total discounted costs, as the time horizon tends to infinity (Theorem 3.6), and (iii) convergence of optimal discount-cost actions to optimal average-cost actions for infinite-horizon problems, as the discount factor tends to 1 (Theorems 4.3 and 4.5). Studying the convergence of value iterations and optimal actions for discounted costs with non-zero terminal values in this paper is motivated by inventory control. As was understood by Veinott and Wagner [36], without additional assumptions (s, S) policies may not be optimal for problems with discounted costs, but they are optimal for large values of discount factors. Even for large discount factors, (s, S) policies may not be optimal for finite-horizon problems with discounted cost criteria and zero terminal costs. However, (s, S) policies are optimal for such problems with the appropriately chosen nonzero terminal costs, and this observation is useful for proving the optimality of (s, S) policies for infinite-horizon problems. Section 5 relates MDPs to problems whose dynamics are defined by stochastic equations, as this takes place for inventory control. Section 6 studies the inventory control problem with backorders, setup costs,
Naval Research Logistics, Apr 30, 2023
This paper provides sufficient conditions for the existence of solutions for two-person zero-sum ... more This paper provides sufficient conditions for the existence of solutions for two-person zero-sum games with inf/sup-compact payoff functions and with possibly noncompact decision sets for both players. Payoff functions may be unbounded, and we do not assume any convexity/concavity-type conditions. For such games expected payoff may not exist for some pairs of strategies. The results of this paper imply several classic facts. The paper also provides sufficient conditions for the existence of a value and solutions for each player. The results of this paper are illustrated with the number guessing game.
arXiv (Cornell University), Jun 1, 2018
We study discrete-time discounted constrained Markov decision processes (CMDPs) with Borel state ... more We study discrete-time discounted constrained Markov decision processes (CMDPs) with Borel state and action spaces. These CMDPs satisfy either weak (W) continuity conditions, that is, the transition probability is weakly continuous and the reward function is upper semicontinuous in state-action pairs, or setwise (S) continuity conditions, that is, the transition probability is setwise continuous and the reward function is upper semicontinuous in actions. Our main goal is to study models with unbounded reward functions, which are often encountered in applications, e.g., in consumption/investment problems. We provide some general assumptions under which the optimization problems in CMDPs are solvable in the class of randomized stationary policies and in the class of chattering policies introduced in this paper. If the initial distribution and transition probabilities are atomless, then using a general "purification result" of Feinberg and Piunovskiy we show the existence of a deterministic (stationary) optimal policy. Our main results are illustrated by examples.
arXiv (Cornell University), Mar 8, 2016
As is well-known, transition probabilities of jump Markov processes satisfy Kolmogorov's backward... more As is well-known, transition probabilities of jump Markov processes satisfy Kolmogorov's backward and forward equations. In the seminal 1940 paper, William Feller investigated solutions of Kolmogorov's equations for jump Markov processes. Recently the authors solved the problem studied by Feller and showed that the minimal solution of Kolmogorov's backward and forward equations is the transition probability of the corresponding jump Markov process if the transition rate at each state is bounded. This paper presents more general results. For Kolmogorov's backward equation, the sufficient condition for the described property of the minimal solution is that the transition rate at each state is locally integrable, and for Kolmogorov's forward equation the corresponding sufficient condition is that the transition rate at each state is locally bounded.
arXiv (Cornell University), Feb 12, 2011
Consider a measurable space with a finite vector measure. This measure defines a mapping of the σ... more Consider a measurable space with a finite vector measure. This measure defines a mapping of the σ-field into a Euclidean space. According to Lyapunov's convexity theorem, the range of this mapping is compact and, if the measure is atomless, this range is convex. Similar ranges are also defined for measurable subsets of the space. We show that the union of the ranges of all subsets having the same given vector measure is also compact and, if the measure is atomless, it is convex. We further provide a geometrically constructed convex compactum in the Euclidean space that contains this union. The equality of these two sets, that holds for two-dimensional measures, can be violated in higher dimensions.
arXiv (Cornell University), Jun 2, 2010
Consider a measurable space with an atomless finite vector measure. This measure defines a mappin... more Consider a measurable space with an atomless finite vector measure. This measure defines a mapping of the σ-field into an Euclidean space. According to the Lyapunov convexity theorem, the range of this mapping is a convex compactum. Similar ranges are also defined for measurable subsets of the space. Two subsets with the same vector measure may have different ranges. We investigate the question whether, among all the subsets having the same given vector measure, there always exists a set with the maximal range of the vector measure. The answer to this question is positive for twodimensional vector measures and negative for higher dimensions. We use the existence of maximal ranges to strengthen the Dvoretzky-Wald-Wolfowitz purification theorem for the case of two measures.
This paper presents a mixed additive-multiplicative model for load forecasting that can be flexib... more This paper presents a mixed additive-multiplicative model for load forecasting that can be flexibly adapted to accommodate various forecasting needs in a Smart Grid setting. The flexibility of the model allows forecasting the load at different levels: system level, transform substation level, and feeder level. It also enables us to conduct short-term, medium and long-term load forecasting. The model decomposes load into two additive parts. One is independent of weather but dependent on the day of the week (d) and hour of the day (h), denoted as \(L_0(d,h)\). The other is the product of a weather-independent normal load, \(L_1(d,h)\), and weather-dependent factor, f(w). Weather information (w) includes the ambient temperature, relative humidity and their lagged versions. This method has been evaluated on real data for system level, transformer level and feeder level in the Northeastern part of the USA. Unlike many other forecasting methods, this method does not suffer from the accumu...
This paper describes a sensor placement algorithm for real-time parallel power flow computations ... more This paper describes a sensor placement algorithm for real-time parallel power flow computations for transmission networks. In particular, Phasor Measurement Units (PMUs) can be such sensors. Graph partitioning is used to decompose the system into several subsystems and to locate sensors in an efficient way. Power flow calculations are then run in parallel for each area. Test results on the IEEE 118and 300-bus systems show that the proposed approach is suitable for real-time applications. Keywords–Parallel computing; power system; power flow calculation; PMU placement.
Uploads
Papers by Eugene Feinberg