I’m new to Python and programming in general and I am having trouble with a website parsing project.
This is the code I managed to write:
import requests from bs4 import BeautifulSoup import pandas as pd pd.set_option('display.max_rows', None) pd.set_option('display.max_columns', None) pd.set_option('display.width', None) pd.set_option('display.max_colwidth', -1) import json #necessary lists url_list = [ "https://warframe.market/items/melee_riven_mod_(veiled)", "https://warframe.market/items/zaw_riven_mod_(veiled)" ] item_list = [] items_name = [] combined_data = [] iteration = 0 #looping for every url found in url_list for url in url_list: #requesting data r = requests.get(url) soup = BeautifulSoup(r.content, "html.parser") #splitting the last part of the url which has the name of the item that I want to insert in the dataframe name = url.split("/")[4] items_name.append(name) #Finding in the parsed HTML code where the JSON file starts ( it start from <script> n°2) results = soup.find_all('script')[2].text.strip() data = json.loads(results) combined_data.append(data) #combining all the data into one list #filtering only the users who sell the items and are either "ingame" or "online" for payload in combined_data[iteration]["payload"]["orders"]: if payload["order_type"] == "sell" and (payload["user"]["status"] == "online" or payload["user"]["status"] == "ingame"): p = payload item_list.append(p) #adding the items names to the item list ???? PROBLEM ????? item_list = [dict(item, **{'name':items_name[iteration]}) for item in item_list] #trying to change the list from where the data gets taken from and the items name ????? PROBLEM ???? iteration += 1 #creating a dataframe with all the values df = pd.DataFrame(item_list).sort_values(by=["platinum"])
What I’m trying to do and can’t find a solution to, is to add to item_list the name of the item which the url refers to.
e.g.
index | platinum | quantity | … | items name (problematic column) |
---|---|---|---|---|
1 | 10 | 1 | … | melee_riven_mod_(veiled) |
2 | 11 | 1 | … | melee_riven_mod_(veiled) |
3 | 12 | 2 | … | zaw_riven_mod_(veiled) |
4 | … | … | … | zaw_riven_mod_(veiled) |
But items name column has the same name for all the rows like this:
index | platinum | quantity | … | items name (problematic column) |
---|---|---|---|---|
1 | 10 | 1 | … | melee_riven_mod_(veiled) |
2 | 11 | 1 | … | melee_riven_mod_(veiled) |
3 | 12 | 2 | … | melee_riven_mod_(veiled) |
4 | … | … | … | melee_riven_mod_(veiled) |
So I wanted to ask what am I doing wrong in the for loop? It iterates 2 times which is the amount of urls in the url_list
but it doesn’t change the name of the item.
What am I not seeing?
Advertisement
Answer
Change
if payload["order_type"] == "sell" and (payload["user"]["status"] == "online" or payload["user"]["status"] == "ingame"): p = payload item_list.append(p) #adding the items names to the item list ???? PROBLEM ????? item_list = [dict(item, **{'name':items_name[iteration]}) for item in item_list]
To this:
if payload["order_type"] == "sell" and (payload["user"]["status"] == "online" or payload["user"]["status"] == "ingame"): payload['name'] = items_name[iteration] item_list.append(payload)
Note, that instead of having a separate variable iteration
and incrementing it, you can loop over url_list
using enumerate
, which provides both the item and its index at each iteration:
for iteration, url in enumerate(url_list): ....