Skip to content

Pandas data frame only showing one row when it should be showing multiple

littlebigship

My code is

from bs4 import BeautifulSoup
import pandas as pd

soup = BeautifulSoup(wd.page_source)
items = soup.find("div", {"class": "items-grid-view"})

rows_processed=[]
for item in items.findAll("div", {"class": "item-cell"}):
    itemTitle = item.find("a", {"class": "item-title"})
    itemPromo = item.find("p", {"class": "item-promo"})
    row = []
    
    row.append(itemTitle.text)
    if (itemPromo and itemPromo.text == "OUT OF STOCK"):
        row.append("Sold Out")
    else: 
        row.append("Available")

        
rows_processed.append(row)

df = pd.DataFrame.from_records(rows_processed, columns=["Item Title ", "Status"])

display(df)

When I run the cell it only presents me with the last item in the table when there are supposed to be 12 items shown. It’s not crucial that I use pandas but I’d like to learn so I’m hoping there’s a fix.

Advertisement

Answer

You had the rows_processed.append(row) outside the for loop, so you were actually just appending the last row. Also pd.DataFrame.from_records could be just pd.DataFrame.

from bs4 import BeautifulSoup
import pandas as pd

soup = BeautifulSoup(wd.page_source)
items = soup.find("div", {"class": "items-grid-view"})

rows_processed=[]
for item in items.findAll("div", {"class": "item-cell"}):
    itemTitle = item.find("a", {"class": "item-title"})
    itemPromo = item.find("p", {"class": "item-promo"})
    row = []
    
    row.append(itemTitle.text)
    if itemPromo and itemPromo.text == "OUT OF STOCK":
        row.append("Sold Out")
    else: 
        row.append("Available")

    rows_processed.append(row)
        

df = pd.DataFrame(rows_processed, columns=["Item Title ", "Status"])

print(df)