Thanks for the report! This should raise consistently. Further investigations and PRs to fix are welcome!

Edit for visibility: As @MarcoGorelli points out below, this should even raise on index construction!

The source of the issue appears to be here:

We create an unpopulated hash table, and then fail on the map_locations line. However, this only happens when self.is_mapping_populated is False, which uses:

Still, I do not see how this could then return correct results. It is certainly inefficient.

tuples = [(k,) for k in range(20000)] + [(0,)]
idx = pd.Index(tuples)
%timeit idx.drop_duplicates()
# 445 μs ± 3.66 μs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)

lists = [[k] for k in range(20000)] + [[0]]
idx = pd.Index(lists)
try:
    idx.drop_duplicates()
except TypeError:
    pass
%timeit idx.drop_duplicates()
# 3.03 s ± 18.7 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

My guess is that the hash table is somehow degenerating into saying everything is a collision, and therefore doing an O(n) lookup (where n is the size of the hash table).

Took a look. Think I see what's happening.

The root cause is that values are put in hashtables in two different ways - one where hash(val) is called (raises) and one where it isn't.

First path. Raises due to calling hash().

Second path. Does not raise.

The reason there are two code paths is:

1. In the first call to drop_duplicates(), computation of self.is_unique initializes idx._engine.mapping which needs to call PyObjectHashTable.map_locations (the snippet linked for the first path)
   

   pandas/pandas/core/indexes/base.py

   Lines 2847 to 2850 in 19ea997

   	if self.is_unique:
   	return self._view()
   	return super().drop_duplicates(keep=keep)

2. The second time around, self.is_unique is False. We then call super().drop_duplicates, which if you trace through goes down the second code path. Here, we actually rebuild an entirely new and identical (from what I can tell) hash table to idx._engine.mapping in the duplicated function of hashtable_func_helper.pxi. The natural question of course is why this is done.

As for what to do, I'm going on the premise given by @rhshadrach that we should raise twice. Intuitively, there is no need to have strong support for nonhashable values.

I'm going to suggest a few options, though I'm not yet familiar with the code so not sure which is best (if any).

1. The minimum code option is adding hash(values[i]) in the second path. This achieves the desired behavior.
2. In drop_duplicates, if computation of self.is_unique failed the first time, should it be False on the second call? I would think no. In this case, the second call will go down exactly the same code path as the first.
3. Reuse the already computed hash table when calling drop_duplicates. Not a direct fix, but would remove the second code path.

Thoughts on how to proceed?

Should this not raise even earlier? As in, should columns=['a', ['b', 'c'], ['b', 'c']] even be allowed in the DataFrame constructor? (I think not?)

that's a separate issue

Agreed @MarcoGorelli - marked as off-topic. @kiranzo - please open a new issue if you'd like to discuss this.

take

Should this not raise even earlier? As in, should columns=['a', ['b', 'c'], ['b', 'c']] even be allowed in the DataFrame constructor? (I think not?)

I was taking a look at this issue. Tracked down the index checks and constructor, noticed some things regarding lists as columns/index items:

• If all items are lists - like columns=[ ['a', 'b'], ['b', 'c'], ['b', 'c'] ], it will try to create a MultiIndex. Correct.
• If any item is a list, but NOT all - like the initial example, columns=[ 'a', ['b', 'c'], ['b', 'c'] ], there's no check for this condition, and the creation will follow as if all items, including the lists, are valid column names.

A simple solution would be to use a check for this condition in the index contructor:

...
    if any(isinstance(el, list) for el in data) and not all(isinstance(el, list) for el in data):
        raise ValueError("Index names must all be hashable, or all lists to make MultiIndex")
...

If it seems a valid solution I will make a PR for it. Thoughts?