Solving simple stochastic games with few coin toss positions

Journal of the ACM, 2013

Ye [2011] showed recently that the simplex method with Dantzig’s pivoting rule, as well as Howard’s policy iteration algorithm, solve discounted Markov decision processes (MDPs), with a constant discount factor, in strongly polynomial time. More precisely, Ye showed that both algorithms terminate after at most O ( mn 1− γ log n 1− γ ) iterations, where n is the number of states, m is the total number of actions in the MDP, and 0 < γ < 1 is the discount factor. We improve Ye’s analysis in two respects. First, we improve the bound given by Ye and show that Howard’s policy iteration algorithm actually terminates after at most O ( m 1− γ log n 1− γ ) iterations. Second, and more importantly, we show that the same bound applies to the number of iterations performed by the strategy iteration (or strategy improvement ) algorithm, a generalization of Howard’s policy iteration algorithm used for solving 2-player turn-based stochastic games with discounted zero-sum rewards. This provide...

Electronic Communications in Probability, 2004

Consider a two-person zero-sum game played on a random n × n-matrix where the entries are iid normal random variables. Let Z be the number of rows in the support of the optimal strategy for player I given the realization of the matrix. (The optimal strategy is a.s. unique and Z a.s. coincides with the number of columns of the support of the optimal strategy for player II.) Faris an Maier [4] make simulations that suggest that as n gets large Z has a distribution close to binomial with parameters n and 1/2 and prove that P (Z = n) ≤ 2 −(k−1). In this paper a few more theoretically rigorous steps are taken towards the limiting distribution of Z: It is shown that there exists a < 1/2 (indeed a < 0.4) such that P (1 2 − a)n < Z < (1 2 + a)n → 1 as n → ∞. It is also shown that EZ = (1 2 + o(1))n. We also prove that the value of the game with probability 1 − o(1) is at most Cn −1/2 for some C < ∞ independent of n. The proof suggests that an upper bound is in fact given by f (n)n −1 , where f (n) is any sequence such that f (n) → ∞, and it is pointed out that if this is true, then the variance of Z is o(n 2) so that any a > 0 will do in the bound on Z above.

Electronic Proceedings in Theoretical Computer Science, 2011

Games on graphs provide a natural model for reactive non-terminating systems. In such games, the interaction of two players on an arena results in an infinite path that describes a run of the system. Different settings are used to model various open systems in computer science, as for instance turnbased or concurrent moves, and deterministic or stochastic transitions. In this paper, we are interested in turn-based games, and specifically in deterministic parity games and stochastic reachability games (also known as simple stochastic games). We present a simple, direct and efficient reduction from deterministic parity games to simple stochastic games: it yields an arena whose size is linear up to a logarithmic factor in size of the original arena.

2011

Two standard algorithms for approximately solving two-player zero-sum concurrent reachability games are value iteration and strategy iteration. We prove upper and lower bounds of 2 m Q (N) 2^{m^{\ Theta (N)}} on the worst case number of iterations needed for both of these algorithms to provide non-trivial approximations to the value of a game with N non-terminal positions and m actions for each player in each position.

Lecture Notes in Computer Science, 2005

The theory of graph games with ω-regular winning conditions is the foundation for modeling and synthesizing reactive processes. In the case of stochastic reactive processes, the corresponding stochastic graph games have three players, two of them (System and Environment) behaving adversarially, and the third (Uncertainty) behaving probabilistically. We consider two problems for stochastic graph games: the qualitative problem asks for the set of states from which a player can win with probability 1 (almost-sure winning); the quantitative problem asks for the maximal probability of winning (optimal winning) from each state. We show that for Rabin winning conditions, both problems are in NP. As these problems were known to be NP-hard, it follows that they are NPcomplete for Rabin conditions, and dually, coNP-complete for Streett conditions. The proof proceeds by showing that pure memoryless strategies suffice for qualitatively and quantitatively winning stochastic graph games with Rabin conditions. This insight is of interest in its own right, as it implies that controllers for Rabin objectives have simple implementations. We also prove that for every ω-regular condition, optimal winning strategies are no more complex than almost-sure winning strategies.

2012

We study stochastic two-player games where the goal of one player is to achieve precisely a given expected value of the objective function, while the goal of the opponent is the opposite. Potential applications for such games include controller synthesis problems where the optimisation objective is to maximise or minimise a given payoff function while respecting a strict upper or lower bound, respectively. We consider a number of objective functions including reachability, ω-regular, discounted reward, and total reward.

Computing Research Repository - CORR, 2008

We consider some well known families of two-player, zero-sum, turn-based, perfect information games that can be viewed as specical cases of Shapley&#x27;s stochastic games. We show that the following tasks are polynomial time equivalent: Solving simple stochastic games, solving stochastic mean-payo games with rewards and probabilities given in unary, and solving stochastic mean-payo games with rewards and probabilities given in binary.

2020

This paper investigates the use of model-free reinforcement learning to compute the optimal value in two-player stochastic games with parity objectives. In this setting, two decision makers, player Min and player Max, compete on a finite game arena – a stochastic game graph with unknown but fixed probability distributions – to minimize and maximize, respectively, the probability of satisfying a parity objective. We give a reduction from stochastic parity games to a family of stochastic reachability games with a parameter ε, such that the value of a stochastic parity game equals the limit of the values of the corresponding simple stochastic games as the parameter ε tends to 0. Since this reduction does not require the knowledge of the probabilistic transition structure of the underlying game arena, model-free reinforcement learning algorithms, such as minimax Q-learning, can be used to approximate the value and mutual best-response strategies for both players in the underlying stocha...

2006

We consider two-player infinite games played on graphs. The games are concurrent, in that at each state the players choose their moves simultaneously and independently, and stochastic, in that the moves determine a probability distribution for the successor state. The value of a game is the maximal probability with which a player can guarantee the satisfaction of her objective. We show that the values of concurrent games with ω-regular objectives expressed as parity conditions can be computed in NP ∩ coNP. This result substantially improves the best known previous bound of 3EXPTIME. It also shows that the full class of concurrent parity games is no harder than the special cases of turnbased deterministic parity games (Emerson-Jutla) and of turn-based stochastic reachability games (Condon), for both of which NP ∩ coNP is the best known bound. While the previous, more restricted NP ∩ coNP results for graph games relied on the existence of particularly simple (pure memoryless) optimal strategies, in concurrent games with parity objectives optimal strategies may not exist, and ε-optimal strategies (which achieve the value of the game within a parameter ε > 0) require in general both randomization and infinite memory. Hence our proof must rely on a more detailed analysis of strategies and, in addition to the main result, yields two results that are interesting on their own.

2006

We consider stochastic turn-based games where the winning objectives are given by formulae of the branching-time logic PCTL. These games are generally not determined and winning strategies may require memory and/or randomization. Our main results concern history-dependent strategies. In particular, we show that the problem whether there exists a history-dependent winning strategy in 1 1 2-player games is highly undecidable, even for objectives formulated in the L(F =5/8 , F =1 , F >0 , G =1) fragment of PCTL. On the other hand, we show that the problem becomes decidable (and in fact EXPTIME-complete) for the L(F =1 , F >0 , G =1) fragment of PCTL, where winning strategies require only finite memory. This result is tight in the sense that winning strategies for L(F =1 , F >0 , G =1 , G >0) objectives may already require infinite memory.