Thanks for the report. As named and documented, these are weights and not probabilities. pandas will normalize the sum to 1 so that they are used a probabilities. E.g. in your example, the corresponding probabilities are:

100 / (100 + 1 + 1), 1 / (100 + 1 + 1), 1 / (100 + 1 + 1)

Marking as a closing candidate for now, let me know if you think there is still behavior that needs adjusted.

Hi @dougj892 ,

I’ve reviewed this issue and confirmed the bug:

When replace=False in DataFrame.sample(), the current implementation doesn’t validate whether the normalized weights (n * w_i / sum(w)) exceed 1, which violates PPS sampling constraints. The example with weights [100, 1, 1] and n=2 silently produces biased samples instead of raising an error.

Proposed Fix
I’ll address this by:

1. Adding Validation:

Compute max_normalized_weight = (n * weights.max()) / weights.sum() when replace=False.

Raise a ValueError if max_normalized_weight > 1, with a clear message like:

"PPS sampling without replacement requires (nw_i/sum(w)) ≤ 1 for all weights. Found max normalized weight = {max_weight:.2f}. Use replace=True, adjust weights, or reduce n."*

2)Edge Cases:

Skip validation for replace=True or weights=None.

Handle zero-sum weights gracefully.

3)Documentation:

Update the sample() docstring to clarify PPS constraints.

4)Tests:

Add test cases for invalid/valid weights (e.g., [100, 1, 1] with n=2 vs. n=1).

Verify edge cases (zero weights, empty DataFrames).

Next Steps
I’ll submit a PR shortly with these changes. Let me know if you’d prefer any adjustments to the error messaging or implementation approach.

Thanks for the review!

Hi @Tanush-Jain - Thanks a lot for the rapid response! That makes a lot of sense to me. For reference, the error message you get when you do this in Stata is "X cases has w_i*n/sum(w)>1" so I think that error message makes sense.

I am negative on restricting the weights here. Allowing weights to be greater than 1 is quite common, e.g. random.choices.

@rhshadrach @dougj892 Thanks for clarifying! I now understand that the current weights behavior intentionally supports broader use cases beyond PPS sampling.

Since this isn’t actually a bug (just a documentation gap for PPS use cases),
Probably the maintainer can (as a suggestion):

1. Propose a docs PR to:

   • Add a Note in sample()'s docstring warning that strict PPS sampling requires (n * weights / weights.sum()).max() <= 1 when replace=False.
   • Include an example like:

     # Manual PPS validation  
     if not replace and weights is not None:  
         assert (n * weights / weights.sum()).max() <= 1, "PPS sampling violation"  

2. also let the maintainers decide whether to:

   • Merge the docs update.
   • Close this issue (since it’s working as designed).

Happy to implement whichever path you prefer!

@Tanush-Jain and @rhshadrach

To clarify a bit, I agree that weights greater than 1 are totally fine. The issue with the example given is that the probabilities are not in fact proportional to the weights – because they can’t be. @rhshadrach You mentioned that the probabilities for the three units are 100/102, 1/102, and 1/102. In the example code, the probabilities for the last two units are actually ~50%. To see this note that if we sample 2 units without replacement, we are almost always going to select the first unit which means that we have a 50/50 chance of selecting either of the last two units.

This may seem like a small issue but it prevents folks like me from relying on pandas for sampling use cases. For example, I often use PPS wor sampling to select primary stage units (typically villages) for household surveys. If PPS wor sampling is not possible because the variance in weights is too large, I need to the function to return an error so that I can fix things (typically by splitting up larger PSUs or sampling some with certainty). Without an error, I will end up miscalculating the final probablity of selection of the households in my survey.

And, again, thanks so much for taking a look at this both of you!

@dougj892 - you get the same behavior using

df.sample(n=2, weights=[100/102, 1/102, 1/102], replace=False)

no?

Edit: Ah, I see, you're multiplying by n in your condition in the OP.

I'm positive on this change, but would need to go through the deprecation process of PDEP-17.