I want to test if two or more values have membership on a list, but I’m getting an unexpected result:
>>> 'a','b' in ['b', 'a', 'foo', 'bar']
('a', True)
So, Can Python test the membership of multiple values at once in a list? What does that result mean?
See also: How to find list intersection?. Checking whether any of the specified values is in the list, is equivalent to checking if the intersection is non-empty. Checking whether all the values are in the list, is equivalent to checking if they are a subset.
Advertisement
Answer
This does what you want, and will work in nearly all cases:
>>> all(x in ['b', 'a', 'foo', 'bar'] for x in ['a', 'b']) True
The expression 'a','b' in ['b', 'a', 'foo', 'bar'] doesn’t work as expected because Python interprets it as a tuple:
>>> 'a', 'b'
('a', 'b')
>>> 'a', 5 + 2
('a', 7)
>>> 'a', 'x' in 'xerxes'
('a', True)
Other Options
There are other ways to execute this test, but they won’t work for as many different kinds of inputs. As Kabie points out, you can solve this problem using sets…
>>> set(['a', 'b']).issubset(set(['a', 'b', 'foo', 'bar']))
True
>>> {'a', 'b'} <= {'a', 'b', 'foo', 'bar'}
True
…sometimes:
>>> {'a', ['b']} <= {'a', ['b'], 'foo', 'bar'}
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: unhashable type: 'list'
Sets can only be created with hashable elements. But the generator expression all(x in container for x in items) can handle almost any container type. The only requirement is that container be re-iterable (i.e. not a generator). items can be any iterable at all.
>>> container = [['b'], 'a', 'foo', 'bar']
>>> items = (i for i in ('a', ['b']))
>>> all(x in [['b'], 'a', 'foo', 'bar'] for x in items)
True
Speed Tests
In many cases, the subset test will be faster than all, but the difference isn’t shocking — except when the question is irrelevant because sets aren’t an option. Converting lists to sets just for the purpose of a test like this won’t always be worth the trouble. And converting generators to sets can sometimes be incredibly wasteful, slowing programs down by many orders of magnitude.
Here are a few benchmarks for illustration. The biggest difference comes when both container and items are relatively small. In that case, the subset approach is about an order of magnitude faster:
>>> smallset = set(range(10)) >>> smallsubset = set(range(5)) >>> %timeit smallset >= smallsubset 110 ns ± 0.702 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each) >>> %timeit all(x in smallset for x in smallsubset) 951 ns ± 11.5 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)
This looks like a big difference. But as long as container is a set, all is still perfectly usable at vastly larger scales:
>>> bigset = set(range(100000)) >>> bigsubset = set(range(50000)) >>> %timeit bigset >= bigsubset 1.14 ms ± 13.9 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each) >>> %timeit all(x in bigset for x in bigsubset) 5.96 ms ± 37 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
Using subset testing is still faster, but only by about 5x at this scale. The speed boost is due to Python’s fast c-backed implementation of set, but the fundamental algorithm is the same in both cases.
If your items are already stored in a list for other reasons, then you’ll have to convert them to a set before using the subset test approach. Then the speedup drops to about 2.5x:
>>> %timeit bigset >= set(bigsubseq) 2.1 ms ± 49.2 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
And if your container is a sequence, and needs to be converted first, then the speedup is even smaller:
>>> %timeit set(bigseq) >= set(bigsubseq) 4.36 ms ± 31.4 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
The only time we get disastrously slow results is when we leave container as a sequence:
>>> %timeit all(x in bigseq for x in bigsubseq) 184 ms ± 994 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
And of course, we’ll only do that if we must. If all the items in bigseq are hashable, then we’ll do this instead:
>>> %timeit bigset = set(bigseq); all(x in bigset for x in bigsubseq) 7.24 ms ± 78 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
That’s just 1.66x faster than the alternative (set(bigseq) >= set(bigsubseq), timed above at 4.36).
So subset testing is generally faster, but not by an incredible margin. On the other hand, let’s look at when all is faster. What if items is ten-million values long, and is likely to have values that aren’t in container?
>>> %timeit hugeiter = (x * 10 for bss in [bigsubseq] * 2000 for x in bss); set(bigset) >= set(hugeiter) 13.1 s ± 167 ms per loop (mean ± std. dev. of 7 runs, 1 loop each) >>> %timeit hugeiter = (x * 10 for bss in [bigsubseq] * 2000 for x in bss); all(x in bigset for x in hugeiter) 2.33 ms ± 65.2 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
Converting the generator into a set turns out to be incredibly wasteful in this case. The set constructor has to consume the entire generator. But the short-circuiting behavior of all ensures that only a small portion of the generator needs to be consumed, so it’s faster than a subset test by four orders of magnitude.
This is an extreme example, admittedly. But as it shows, you can’t assume that one approach or the other will be faster in all cases.
The Upshot
Most of the time, converting container to a set is worth it, at least if all its elements are hashable. That’s because in for sets is O(1), while in for sequences is O(n).
On the other hand, using subset testing is probably only worth it sometimes. Definitely do it if your test items are already stored in a set. Otherwise, all is only a little slower, and doesn’t require any additional storage. It can also be used with large generators of items, and sometimes provides a massive speedup in that case.