Convert nested dict to dataframe, syntax error?

Problem

I am converting multiple nested dicts to dataframes. I have a slightly different dict that I haven’t been able to convert to a dataframe using my attempted solution. I am providing a shortened copy of my dict with dummy values as the reprex.

Reprex dict:

{'metrics': [{'metric': 'DatasetCorrelationsMetric',
   'result': {'current': {'stats': {'pearson': {'target_prediction_correlation': None,
       'abs_max_features_correlation': 0.1},
      'cramer_v': {'target_prediction_correlation': None,
       'abs_max_features_correlation': None}}},
    'reference': {'stats': {'pearson': {'target_prediction_correlation': None,
       'abs_max_features_correlation': 0.7},
      'cramer_v': {'target_prediction_correlation': None,
       'abs_max_features_correlation': None}}}}}]}

My attempted solution

Code is based on similar dict wrangling problems that I had, but I am not sure how to apply it for this specific dict.

data = {}
for result in reprex_dict['metrics']:
    data[result['result']] = {
        **{f"ref_{key}": val for key, val in result['result']['reference'].items()},
        **{f"cur_{key}": val for key, val in result['result']['current'].items()}
    }

Expected dataframe format:

cur_pearson_target_prediction_correlation	cur_pearson_abs_max_features_correlation	cur_cramer_v_target_prediction_correlation
None	0.1	None

Error message

I am currently getting this error too.

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In [403], line 7
      5 data = {}
      6 for result in corr_matrix_dict['metrics']:
----> 7     data[result['result']] = {
      8         **{f"ref_{key}": val for key, val in result['result']['reference']['stats'].items()},
      9         **{f"cur_{key}": val for key, val in result['result']['current']['stats'].items()}
     10     }

TypeError: unhashable type: 'dict'

Answer

As mentioned in the comments, the TypeError is due result['result'] (which is a dictionary) not being usable as a key. If you used some thing [like result['metric']] then the error would no longer be raised, but I think that the resulting structure is not the outcome you want.

For flattening nested data, I often take a recursive approach. Below is a simplified version of my flattenObj function:

def flattenDict(orig:dict, kList=[], kSep='_', rename={}):
    if not isinstance(orig, dict): return [(kList, orig)]
    tList, dCt = [], len([v for v in orig.values() if isinstance(v,dict)])
    for k, v in orig.items():
        kli = kList + ([] if isinstance(v,dict) and dCt==1 else [str(k)])
        tList += flattenDict(v, kli, None)
    if not isinstance(kSep, str): return tList
    return {kSep.join([rename.get(k,k) for k in kl]):v for kl,v in tList}

---

# import pandas as pd
nrMap = {'current':'cur','reference':'ref'}
rows = [flattenDict(result, rename=nrMap) for result in reprex_dict['metrics']]
rowsDf = pd.DataFrame(rows)

rows

[{'metric': 'DatasetCorrelationsMetric',
  'cur_pearson_target_prediction_correlation': None,
  'cur_pearson_abs_max_features_correlation': 0.1,
  'cur_cramer_v_target_prediction_correlation': None,
  'cur_cramer_v_abs_max_features_correlation': None,
  'ref_pearson_target_prediction_correlation': None,
  'ref_pearson_abs_max_features_correlation': 0.7,
  'ref_cramer_v_target_prediction_correlation': None,
  'ref_cramer_v_abs_max_features_correlation': None}]

---

rowsDf.T [Transposed to fit better]

---

If you don’t want the metric column, you can either drop it or omit it by defining rows as

rows = [flattenDict(result['result'], rename=nrMap) for result in reprex_dict['metrics']]