Skip to content

python SQLite3 how to getting records that match a list of values in a column then place that into pandas df

I am not experienced with SQL or SQLite3.

I have a list of ids from another table. I want to use the list as a key in my query and get all records based on the list. I want the SQL query to feed directly into a DataFrame.

import pandas as pd
import sqlite3

cnx = sqlite3.connect('c:/path/to/data.sqlite')

# the below values are ones found in "s_id"
id_list = ['C20','C23','C25','C28', ... ,'C83']
# change list to sql string. 
id_sql = ", ".join(str(x) for x in id_list)
df = pd.read_sql_query(f"SELECT * FROM table WHERE s_id in ({id_sql})", cnx)

I am getting a DatabaseError: Execution failed on sql ‘SELECT * FROM … : no such column: C20. When I saw this error I thought the code just needs a simple switch. So I tried this

df = pd.read_sql_query(f"SELECT * FROM table WHERE ({id_sql}) in s_id", cnx)

it did not work.

So how can I get this to work?

The table is like.

id s_id date assigned_to date_complete notes
0 C10 1/6/2020 Jack 1/8/2020 None
1 C20 1/10/2020 Jane 1/12/2020 Call back
2 C23 1/11/2020 Henry 1/12/2020 finished
n C83 rows of more data
n+1 D85 9/10/2021 Jeni 9/12/2021 Call back



Currently, you are missing the single quotes around your literal values and consequently the SQLite engine assumes you are attempting to query columns. However, avoid concatenation of values altogether but bind them to parameters which pandas pandas.read_sql supports with the params argument:

# the below values are ones found in "s_id"
id_list = ['C20','C23','C25','C28', ... ,'C83']

# build equal length string of ? place holders
prm_list = ", ".join("?" for _ in id_list)

# build prepared SQL statement
sql = f"SELECT * FROM table WHERE s_id IN ({prm_list})"

# run query, passing parameters and values separately
df = pd.read_sql(sql, con=cnx, params=id_list)
User contributions licensed under: CC BY-SA
2 People found this is helpful