I have a list of dicts, and I’d like to remove the dicts with identical key and value pairs.
For this list: [{'a': 123}, {'b': 123}, {'a': 123}]
I’d like to return this: [{'a': 123}, {'b': 123}]
Another example:
For this list: [{'a': 123, 'b': 1234}, {'a': 3222, 'b': 1234}, {'a': 123, 'b': 1234}]
I’d like to return this: [{'a': 123, 'b': 1234}, {'a': 3222, 'b': 1234}]
Advertisement
Answer
Try this:
[dict(t) for t in {tuple(d.items()) for d in l}]
The strategy is to convert the list of dictionaries to a list of tuples where the tuples contain the items of the dictionary. Since the tuples can be hashed, you can remove duplicates using set
(using a set comprehension here, older python alternative would be set(tuple(d.items()) for d in l)
) and, after that, re-create the dictionaries from tuples with dict
.
where:
l
is the original listd
is one of the dictionaries in the listt
is one of the tuples created from a dictionary
Edit: If you want to preserve ordering, the one-liner above won’t work since set
won’t do that. However, with a few lines of code, you can also do that:
l = [{'a': 123, 'b': 1234}, {'a': 3222, 'b': 1234}, {'a': 123, 'b': 1234}] seen = set() new_l = [] for d in l: t = tuple(d.items()) if t not in seen: seen.add(t) new_l.append(d) print new_l
Example output:
[{'a': 123, 'b': 1234}, {'a': 3222, 'b': 1234}]
Note: As pointed out by @alexis it might happen that two dictionaries with the same keys and values, don’t result in the same tuple. That could happen if they go through a different adding/removing keys history. If that’s the case for your problem, then consider sorting d.items()
as he suggests.