Skip to content
Advertisement

Remove duplicate JSON objects from list in python

I have a list of dict where a particular value is repeated multiple times, and I would like to remove the duplicate values.

My list:

te = [
      {
        "Name": "Bala",
        "phone": "None"
      },
      {
        "Name": "Bala",
        "phone": "None"
      },
      {
        "Name": "Bala",
        "phone": "None"
      },
      {
        "Name": "Bala",
        "phone": "None"
      }
    ]

function to remove duplicate values:

def removeduplicate(it):
    seen = set()
    for x in it:
        if x not in seen:
            yield x
            seen.add(x)

When I call this function I get generator object.

<generator object removeduplicate at 0x0170B6E8>

When I try to iterate over the generator I get TypeError: unhashable type: 'dict'

Is there a way to remove the duplicate values or to iterate over the generator

Advertisement

Answer

You can easily remove duplicate keys by dictionary comprehension, since dictionary does not allow duplicate keys, as below-

te = [
      {
        "Name": "Bala",
        "phone": "None"
      },
      {
        "Name": "Bala",
        "phone": "None"
      },
      {
        "Name": "Bala",
        "phone": "None"
      },
      {
        "Name": "Bala",
        "phone": "None"
      },
      {
          "Name": "Bala1",
          "phone": "None"
      }      
    ]

unique = { each['Name'] : each for each in te }.values()

print unique

Output-

[{'phone': 'None', 'Name': 'Bala1'}, {'phone': 'None', 'Name': 'Bala'}]
User contributions licensed under: CC BY-SA
1 People found this is helpful
Advertisement