Categories

# Repeated Games with Unknown Payoff Distributions

Repeated games with unknown payoff distributions are the same as so-called multi-armed bandit problems that have to be solved by a single decision maker. Every state of the world is correlated with a certain payoff matrix of different stage of the game.

In this context, the partial folk theorem says for the any of the functions that link every state of the world to the payoff vector it is reasonable and rational in this state. There has to always be a perfect balance where players are experimenting with learning the certain state and achieving a payoff that is as close to the specified as possible. Partial Folk Theorem holds this for those games, in which all the information about the specified state of the world is public, even when the possibility of imperfectness of information exists.

In this essay, I would like to discuss the concept of repeated games, their implications in the partial folk theorem and the issue of finding equilibrium or balance in those games.

Random variables are given vector of players’ actions and payoffs. The state of nature determines their joint distributions; it is chosen at the very start of the game and is not seen by the players. The players are able to get the information about the certain state of the world only with the help of coordinated experiments, they can observe payoff results, try various actions and this way update their opinions about the certain state.

Realized payoffs vector is thought to be publicly accessible in terms of information, so all the data about this state is public. I would like to talk about the folk theorem for a type of such games; there is always equilibrium between the processes of players learning about the state of nature an achieving the closest to feasible payoff, which is individual for the certain discussed state. One of the result uses is the introductions of the type of payoff uncertainty into an important setting; it cannot influence the forecasts of game models that assume payoffs based on common knowledge.

As Rothschild described in his work, repeated games with unkown payoff distributions are similar to the multi armed bandit problems that are being solved by one individual. In those problems, a long-lived agent has to choose the actions from the predefined set, or “arms”. There is a certain probability distribution over payoffs for every arm; the agent does not it. The agent has prior opinions about payoff distributions and they induce expectations that are not objective for each arm, not necessarily, the arm with the highest subjective expectations is the best one to choose. It can make sense for an agent to sacrifice expected payoff to get more data about the realized payoff distributions that will enable him making valid long-term decisions. By experimenting, he can reduce uncertainty but this will make him incur additional costs. If an agent has high enough discount factor, he can start experiments until he gains enough information about the certain world state with rather high degree of certainty.

The situation gets more complicate with the involvement of other agents. An arm now corresponds to a vector of action choices. Every player has to make his own certain action, so the experiments can be effective and coordinated. Some of the strategy factors can also interfere with gaining information about the true world state. For instance if one player has very low equilibrium payoff in one of the possible states of the world, it can be in his/her own interest to refuse executing certain actions. He/she may choose ignorance to the risk of finding out that the state of low payoff was realized. The situation gets even more challenging when players can witness the actions performed by the other players, and only are exposed the public outcome that signals the vector of action choices with uncertainty. The profile for player actions “a” stochastically evaluates as physical outcome “y”, and its distribution does not depend on nature state w. The distribution of payoffs, in turn, depends only on y and w. Thus, all information about the state is contained in the realized payoffs and the physical outcome y, which are publicly observed. The case where y = a corresponds to perfect monitoring.

An example of a repeated games with unknown payoff distributions where monitoring is perfect is the following partnership game: Two players must decide repeatedly whether to invest in a joint project or not (whether to “work” or to “shirk”). In each period, a player who works incurs a cost c; shirking is costless. If both players work, they each receive a random payoff x. If either player shirks, they get nothing. It is efficient for the players to invest only if the mean value of the random payoff x is greater than c, but the only way to find out the mean value is to collect a sample of realizations of x. The players must experiment by playing (work, work).

A second example, which may satisfy the informational assumption for imperfect monitoring, is the evaluation of an experimental drug. Each patient i’s outcome Ui from trying the drug depends on the dosage Di that he receives, his weight Wi (which is determined by his unobservable behavior), the (unknown) effectiveness w of the drug, and a mean zero individual-specific random factor ei, which is independently and identically distributed across patients. The outcome Ui can be written as U (Di, Wi, w) + ei. The researcher chooses the dosage, and the patient chooses his behavior (whether or not to smoke, drink, eat sensibly, exercise, etc.). Can the researcher, after observing the dosage, weight, and outcome of each patient, determine the effectiveness of the drug?

The task of the researcher is to identify the function U and the distribution of ei. That identification requires sufficient variation in weight across patients. Furthermore, the researcher may be legally restricted to testing the drug on individuals who are otherwise healthy (which depends on unobserved behavior), so she may want to specify behavior for each patient in the study in order to generate a broad enough sample. However, weight is a stochastic function of behavior, and thus not a perfect signal. Patients assigned to a strict diet and exercise program might eat ice cream instead and blame the weight gain on an unusually slow metabolism. As long as weight does not depend on the effectiveness of the drug, w, and weight is sufficiently informative about unobserved behavior, though, the informational conditions for the folk theorem are satisfied. (The formal definition of “sufficiently informative” will be given later. Weight is insufficiently informative if, for example, eating ice cream and taking up smoking to counter the resulting weight gain induces the same distribution over weight as eating vegetables.)

A third example of a repeated games with unknown payoff distributions is the case of duopolies introducing a new product to a market. Every month each firm sets its level of output; the market price that month is a function of its own output, its competitor’s output, the state of demand, and a random monthly fluctuation. The firms do not know in advance how the product will be received. That is, they do not know the demand function, although they have beliefs about its possible values. In choosing their output levels, the firms must balance three, possibly conflicting incentives: short-term profit maximization, strategic considerations, and the long-run benefits of acquiring information about the state of demand. In this example, whether or not the informational assumptions about monitoring are satisfied depends on whether the firms’ quantities are observable. If, for example, each firm observes its own output and the market price, but not its competitor’s output, then the firms are receiving private information about the state of demand. Since the market price depends directly on the output of both firms, (there is no publicly observed physical outcome that determines the price), a firm’s private information about its output is relevant for making inferences about the sate of demand, and so the informational assumptions fail. On the other hand, if government regulations require accurate reporting of output, then actions are observable, and so the result of this paper does apply.

Each state of the world corresponds to a payoff matrix whose entries are the expected values of the realized payoff distributions. Each such matrix has its own set of feasible payoffs and its own set of individually rational payoffs (i.e., where all players get at least their minmax payoff). In this context, a folk theorem implies that for any function that maps each state of the world to a payoff vector that is feasible and strictly individually rational in that state, there is a perfect public equilibrium in which players experiment to learn the realized state and achieve a payoff close to that state’s specified payo ff. I prove that such a folk theorem holds for games where all information about the state of the world is public, even in some cases, as described before, where actions are only imperfectly observed.

The proof uses the theory of repeated games with imperfect monitoring to construct an equilibrium where players experiment in the early rounds of the game to learn the payoffs, in fact always choosing the action profile that results in the fastest possible learning, and then play an equilibrium of the revealed game. One difficulty is that when payoffs are unknown it may be hard to punish a player for deviating from his equilibrium strategy. The problem is that when the state of the world is uncertain, the set of feasible payoffs is not necessarily the convex combination of the feasible sets in the different states. Instead, it is a (possibly strict) subset of that convex combination. A simple, one-player example illustrates what can go wrong: A player has two actions, L and R, and there are two states of the world, A and B. In state A, action L yields payoff 1 and action R yields payoff 0; in state B the payoffs are reversed. In each state, the set of feasible payoffs is the closed interval [0,1]. However, if the player’s beliefs put equal weight on each state, then the feasible set is the singleton point {0.5}. Because the player does not know the state, he does not know which action to play to get a given payoff; that problem generalizes to multiple agents and multiple states, where players may not know how to punish a deviator. However, it turns out that it is sufficient either to punish a deviating player with a low payoff immediately or to learn more about the payoffs and then punish him. It further turns out that a player cannot simultaneously guarantee himself a high payoff and block learning, so effective punishment strategies are possible. In fact, there exists a profile of actions for the other players such that any response either results in learning or yields the punished a payoff no higher than his lowest minmax payoff across all possible states of the world.

The main tool used in the proof is the notion of the self- generating set of payoffs developed in Abreu, Pearce, and Stacchetti (1986, 1990) and Fudenberg, Levine, and Maskin (1994). Every payoff in a self-generating set results from a perfect public equilibrium where the continuation strategies after each possible history are themselves perfect public equilibria with payoffs in the set. For repeated games with unknown payoff distributions, there may be a different set of equilibrium payoffs for each different set of beliefs about the state of the world, so the appropriate concept is self- generating sets of payoff/belief pairs. I construct such a set C, and show that for each payoff/belief pair (v, b) in C. The payoff v is achievable as the outcome of a perfect public equilibrium, starting from initial beliefs b.  Payoff/belief pairs in the interior of C are achieved using continuation payoffs that are increasing in expectation with the expected quared distance between current beliefs and next period’s updated beliefs, so that players (when they are patient enough to put a high weight on future versus current payoffs) prefer actions that yield the most information about the state. In the equilibrium to be constructed, players choose the maximal learning actions until they reach near-certainty about the state, at the boundaries of C. The set C is constructed so that when beliefs assign a probability close to one to a state, then the associated payoffs are close to that state has specified payoff.

The assumption that payoffs are publicly observable is a strong one and deserves comment. Clearly, there are many situations where it does not hold. However, there is a range of economic settings where the assumption is reasonable. Besides the examples listed above, public corporate profit reports, election results, and balance-of-trade and exchange rate statistics all are roughly consistent with the assumption of observable payoffs. In any case, the perfect observability of payoffs is not crucial for the current result; what is necessary is that all information about the state is public. If, for example, there were a second public signal z whose distribution depended only on the state w and the public outcome y, and payoffs were a function of y and z only (and not w). Then no player would have private information about the state of nature, even if payoffs are not publicly observed.2 For simplicity, however, I will assume throughout that players observe all realized payoffs.

The solution concept that will be used here, and that is used by Abreu, Pearce, and Stacchetti (1986, 1990) and Fudenberg, Levine, and Maskin (1994) (from now on, FLM), is the perfect public equilibrium (PPE). In a PPE, players use strategies that depend only on public histories; starting from any period and any public history, the strategies form a Nash equilibrium. Players cannot gain from defecting to a strategy based on private information about their own previous actions, because all the available information about the state of the world and about the actions to be chosen by the other players is contained in the public history of physical outcomes and payoffs.

Consider the case where the state of the world w is common knowledge. Given a discount factor d, a payoff vector v is generated by a set W Í RN of continuationpayoff vectors if there exists an action profile a Î DA and a function w : Y ® W such that, for all i,

There are several strands of previous work on learning payoffs in games. Aoyagi (1998) and Bolton and Harris (1999) consider extensions of the single-decision-maker multi-armed bandit problem that Rothschild (1974) first applies in an economic setting. The situation of sequential choice, where one agent after another must base a choice among a set of actions on a private signal and on the observed actions of previous agents, is studied by Banerjee (1992), Bikhchandani, Hirshleifer, and Welch (1992), and Smith and Sorensen (2000). There is also an extensive literature on repeated games with incomplete information, where strategic interaction has an important role, as it does not in either the multi- armed bandit or sequential choice. (See Aumann and Hart (1992).) In work, more closely related to this paper, Kalai and Lehrer (1995) consider repeated games where players are uncertain both about payoffs and about the strategies of their opponents. They show that play converges to a subjective equilibrium, in which players’ beliefs, though they may not converge to full information, are consistent with observed outcomes. In the special case where players know their own payoffs but not those of their opponents, Kalai and Lehrer (1993) demonstrate that players will eventually learn to play Nash equilibrium of the full information game. Gossner and Vielle (1998) examine a model similar to the one presented here, with the exception that payoffs are non-stochastic given the state of the world, so that playing an action profile once suffices to learn its payoff with certainty. They derive a folk theorem with full learning for the case of perfectly observed actions and no discounting.

This essay presents a folk theorem for the repeated games with unknown payoff distributions when all information about the state of the world is public. In fact, though, the above solutions are both more and less than a folk theorem. It is more in the sense that it guarantees equilibrium that not only produce desired payoffs but also result in learning. Even in the case of games where every action has positive expected learning, that learning result is payoff matrix but not others, or for learning to occur very gradually. The solutions are less than a folk theorem in the sense that the payoffs are achieved only approximately (although with arbitrary precision.) For some games, there may even be equilibrium without learning that Pareto dominate all learning equilibria. The game in Figure 5 illustrates the point. The value of x is either 100 or 1000, with equal probabilities. In either case, playing (A, A) in every period is an equilibrium that yields payoff (2, 2). In contrast, any path of play where players learn necessarily involves playing (B, B) at least once, which results in a strictly lower payoff.