Coordinating the Unknown Lipschitz Constant in Multiplayer Bandits

The multi-armed bandit (MAB) problem is a central model in sequential decision-making: an agent repeatedly chooses among actions with unknown rewards, balancing exploration and exploitation to maximize cumulative reward. Many deployments are inherently distributed, with several decision-makers acting concurrently and observing only part of the system feedback. In wireless systems, multiple users opportunistically access channels whose availabilities are unknown and time-varying, often without explicit coordination, as in cognitive radio and dynamic spectrum access [12, 2]. Similar abstractions arise in distributed sensing and radar networks, where nodes select waveforms or frequency bands while attempting to avoid mutual interference [9]. These settings motivate cooperative multi-player bandits in which agents coordinate implicitly under information constraints, and the relevant information structure is dictated by what each agent can observe: actions, rewards, or both.

To model continuously many actions we work with Lipschitz bandits, where the mean reward varies smoothly over a metric space so that nearby actions have similar rewards. Discretization-based methods exploit this structure, but the resolution they should use depends on the Lipschitz constant $L$. We therefore study the setting in which $L$ is not given in advance, extending the multiplayer Lipschitz bandits of [7]. Three forms of information asymmetry, described in Section III-A and referred to as Problems A, B, and C, are treated in turn: unobserved actions with common rewards, observed actions with independent rewards, and unobserved actions with independent rewards. Because players cannot communicate once learning begins, an unknown $L$ creates a difficulty absent in the single-agent problem: each player estimates $L$ from its own observations, and if these estimates disagree the players build different grids and can no longer be regarded as playing a common discrete bandit. Coordination, rather than estimation accuracy, is the binding constraint.

Contributions. We give a meta-algorithm, mECAB, that estimates an upper confidence bound on $L$ by uniform exploration of a coarse grid, fixes a discretization, and runs an existing cooperative multiplayer MAB subroutine; we prove regret bounds for all three information structures, each resting on a different agreement mechanism, namely shared rewards in Problem A, signalling through observed actions in Problem B, and a dithered quantization of the estimate in Problem C, the last of which makes the agreement probability independent of the instance in a way no deterministic rounding rule can; and we report simulations comparing Lipschitz-adaptive with non-adaptive discretization.

Consider a $d$-dimensional compact set of arms $X=[0,1]^{d}$, $d\geq 1$. Each arm $\bm{x}$ carries a reward distribution $\nu_{\bm{x}}$ supported on $[0,1]$, with mean-payoff function $f:[0,1]^{d}\to[0,1]$. At each round $t\geq 1$ the player selects $\bm{x}_{t}$ and receives $Y_{t}\sim\nu_{\bm{x}_{t}}$, drawn independently across rounds.

Assumption. $f$ is twice differentiable with Hessian uniformly bounded by $N$: for all $\bm{x},\bm{y}$, $|\bm{y}^{\top}H_{f}(\bm{x})\,\bm{y}|\leq N\|\bm{y}\|_{\infty}^{2}$. The map $\|\gradient f\|_{1}$ is continuous and attains its maximum $L$ on $[0,1]^{d}$, so $f$ is $L$-Lipschitz with respect to $\|\cdot\|_{\infty}$. Write $\mathcal{F}_{L,N}$ for the class of mean-payoff functions satisfying both conditions with parameters $L$ and $N$; all suprema below are over this class, and neither parameter is known to the players.

With $f^{\star}=\max_{\bm{x}}f(\bm{x})$, the expected regret at horizon $T$ is $R_{T}=\mathbb{E}[Tf^{\star}-\sum_{t=1}^{T}f(\bm{x}_{t})]$, the expectation being over the draws of $Y_{t}$ and any internal randomization. The goal is to minimize $R_{T}$ without knowing $L$.

Here the environment produces a single common reward observed by all players, while each player does not observe the others’ actions. Unobserved actions prevent coordination during play, and the unknown smoothness must be estimated without access to the exploration of the others. What rescues the situation is that the reward is shared, which makes the exploration statistics shared as well once the schedule is fixed in advance.

Concretely, the players agree on an ordering of the $m^{Md}$ joint bins and each samples uniformly inside the scheduled bin. Only the bin index affects the statistic collected, not the particular arm chosen inside it, so all players obtain the same empirical mean $\widehat{\mu}_{\underline{k}}=\frac{1}{E}\sum_{j=1}^{E}Z_{\underline{k},j}$ for every joint bin $\underline{k}$, and each is free to choose any arm within its own bin. Consequently every player forms the same estimate

hence the same $\widetilde{L}$ and $\widetilde{m}$, and the ordering of the coarse bins induces a consistent ordering of the finer $\widetilde{m}$-level grid that everyone can follow.

On the event of Corollary 3 the subroutine term is $O(T^{(Md+1)/(Md+2)}(L{+}1)^{Md/(Md+2)}\sqrt{\log T})$, matching the leading term up to $\sqrt{\log T}$; the same holds in Theorems 5 and 7.

In the single-agent problem with unknown $L$, discretization yields the familiar scaling $T^{(d+1)/(d+2)}$, and Problem A behaves the same way on the joint space with $Md$ in place of $d$. The action asymmetry therefore costs nothing beyond this dimensional effect, precisely because the common reward and the pre-agreed schedule force identical bin means, an identical $\widehat{L}$, and an identical grid. The dependence on the size of the discretized joint set is also unavoidable, since with $K$ actions each the induced finite problem has $K^{M}$ joint arms, to which the standard finite-armed lower bound applies.

Problem B reverses the structure: actions are observable but reward observations are not shared. The free synchronization of Problem A breaks, since the bin means $\widehat{\mu}^{\,i}_{k}$ may differ across players and would lead to different grids if nothing further were done.

The remedy is to exploit action observability to share reward information implicitly. Since every action is observed by everyone, an action can carry a signal encoding the sender’s statistics, and in a continuum this needs no departure from the action space: after collecting $E-1$ samples from a bin, a player devotes its final action in that bin to encoding its empirical mean, which the others decode and fold into their own estimate. Nothing analogous exists in Problem A, where actions are hidden, nor in finite-action models, where there is no room to encode a real number without distorting the learning problem. At a cost of one sample, each player thus gains $(M-1)(E-1)$ further samples for every bin, and forms

The effective sample size entering the concentration of $\widehat{L}$ is multiplied by $M$, which sharpens the estimate and improves the grid. Problem A had perfect alignment of the bin means but no way to convey anything beyond the common scalar reward; Problem B lacks common rewards but recovers most of the benefit of centralized averaging by broadcasting estimates through actions.

Compared with Theorem 4 the sample size inside the square root improves from $E$ to $ME$, which is exactly the pooling gain from the other players’ samples.

Problem C is the hardest of the three: rewards are not shared, so the mechanism of Problem A is unavailable, and actions are not observed, so the signalling of Problem B is unavailable too. If each player simply used (2) on its own data, the estimates would differ and the induced grids $\widetilde{m}$ would differ with them, destroying the common discrete problem the subroutine needs.

We restore agreement by quantizing the estimate, so that small discrepancies between players do not change the value they act on. Let $X^{i}=m\max_{k,s}|\widehat{\mu}_{k}^{i}-\widehat{\mu}_{k+s}^{i}|$ be the raw estimate of player $i$, the maximum running over $k\in[m-2]^{Md}$ and $s\in\{-1,1\}^{Md}$ as above. A deterministic rounding will not do: if $\overline{L}_{m}$ sits near a rounding boundary, two players whose estimates straddle it round differently however many samples they collect, and the failure probability approaches $1/2$ regardless of $E$. The boundaries are fixed while $\overline{L}_{m}$ is a property of the instance, so no deterministic rule avoids this. Instead the players agree in advance on a dither $U\sim\mathrm{Unif}[0,1)$, shared randomness requiring no communication, and set

Randomizing the offset makes the distance from $\overline{L}_{m}$ to the nearest boundary uniform rather than instance-dependent, so the probability of disagreement can be bounded with no reference to where $\overline{L}_{m}$ lies.

When the players do agree they share the same $\widetilde{m}$, and Problem C reduces to running the same discretized strategy as before; when they do not, they may follow different grids and we pay for that event in the regret.

Problem C therefore isolates what each kind of observability buys. Common rewards make $\widehat{L}$ shared automatically and no agreement term is needed; observable actions permit pooling and sharpen the concentration, improving $E$ to $ME$; when neither is available, agreement must be built from concentration together with dithered quantization. Its price is the final term, which is instance-independent and, for the stated exploration budget, of the same order as the leading one, so Problem C matches Problems A and B up to constants once $E$ is large enough.

We simulate a cooperative bandit with $M=2$ players and action dimension $d=1$ each, so $Md=2$, over $T=10^{5}$ rounds and $10$ independent trials for each configuration. A maximizer $a^{\star}$ is drawn uniformly from $[0,1]^{Md}$ once in each trial and the mean reward is $f(a)=-L\|a-a^{\star}\|_{\infty}$, which is $L$-Lipschitz with respect to $\ell_{\infty}$ and satisfies $f(a^{\star})=0$; rewards are Gaussian with unit variance. We report cumulative pseudo-regret $\sum_{t}(f(a^{\star})-f(a_{t}))$ averaged over trials, with $\pm 1$ standard deviation shading.

Both algorithms discretize the joint space and run UCB on the resulting grid, differing only in how the resolution is set. Est-$L$ explores a coarse grid of $m$ bins in each coordinate with $E$ uniform samples in each bin, estimates $\widehat{L}_{b}=m\max_{(b,b^{\prime})\in\mathcal{N}}|\widehat{\mu}(b)-\widehat{\mu}(b^{\prime})|$ over neighboring bin pairs $\mathcal{N}$, pads it as in (1), and sets $\widetilde{m}$ from the result; this phase is not aimed at collecting reward, so regret grows roughly linearly while it runs. No-$L$ skips exploration and takes $\widetilde{m}=\lceil T^{1/(Md+2)}\rceil$, avoiding the up-front cost but risking a resolution mismatched to the smoothness of $f$.

The three information structures are modelled at the level of the feedback reaching each player rather than through the signalling and quantization mechanisms themselves: Problem A supplies a common reward, Problem B pools the $M$ independent rewards of a round, the idealized effect of encoding empirical means in actions, and Problem C uses a single reward stream without pooling. This isolates the effect of feedback quality on the accuracy of $\widetilde{L}$; simulating the signalling and dithering steps directly is left to an extended version.

Fig. 1 compares the two rules for a small ($L=1$) and a large ($L=1000$) Lipschitz constant, everything else held fixed. In all three cases the Est-$L$ curves grow linearly during coarse exploration and then bend into a visibly sublinear phase once UCB begins on the refined grid, which is the tradeoff the method makes. When $L$ is small the two rules produce comparable resolutions and finish at similar levels; when $L$ is large, fixing the resolution without reference to $L$ gives a grid too coarse for the variation of $f$, and Est-$L$ overtakes it despite the initial linear segment. The information structure modulates the gain: pooling in Problem B makes $\widetilde{L}$ more accurate and flattens the later slope relative to Problems A and C, while Problem C, with the weakest feedback, is the most variable.

We extended cooperative multiplayer bandits to Lipschitz action spaces with an unknown Lipschitz constant, where the players’ estimates of $L$ must agree for a common discretization to exist. Common rewards and observable actions each deliver that agreement for free, and when neither is present a dithered quantization delivers it at no cost in the leading order of the regret. Natural next steps are adversarial rewards and structural assumptions beyond Lipschitz continuity.