Skip to content
Advertisement

printing same name, price,and link in BeautifulSoup python

How to Get all Product detail it prints the same things but I want others products to detail also here is the link from where I want to fetch the data of all product:https://www.nike.com/gb/w/womens-lifestyle-shoes-13jrmz5e1x6zy7ok

import requests
from bs4 import BeautifulSoup
import pandas as pd
import numpy as np
from selenium import webdriver


url= "https://www.nike.com/gb/w/womens-lifestyle-shoes-13jrmz5e1x6zy7ok"
driver = webdriver.Chrome('D:/chromedriver')
driver.get(url)
pageSource = driver.page_source
for n in range(10):
    driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
    soup = BeautifulSoup(pageSource, 'lxml')
    content= soup.findAll('div',class_='product-grid')
    content 
    for item in content:
        title= item.find('div',class_ = 'product-card__title').text
        link = item.find('a', {'class': 'product-card__link-overlay'})['href']
        price=item.find('div',class_ ='product-price css-11s12ax is--current-price').text
    print(title,price,link)

Advertisement

Answer

What happens?

  1. Their is a wrong indent with your print
  2. Their is only one element with class of product-grid

How to fix?

  1. Check the indent of your print, it should be in the loop to print the items.

  2. Change your selection to:

    content= soup.find_all('div',class_='product-card')
    
  3. Bonus: Instead findAll() better use the new syntax find_all()

Example

import requests
from bs4 import BeautifulSoup
import pandas as pd
import numpy as np
from selenium import webdriver


url= "https://www.nike.com/gb/w/womens-lifestyle-shoes-13jrmz5e1x6zy7ok"
driver = webdriver.Chrome(executable_path=r'C:Program FilesChromeDriverchromedriver.exe')
driver.get(url)
pageSource = driver.page_source
for n in range(10):
    driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
    soup = BeautifulSoup(pageSource, 'lxml')
    content= soup.find_all('div',class_='product-card')
    content 
    for item in content:
        title= item.find('div',class_ = 'product-card__title').text
        link = item.find('a', {'class': 'product-card__link-overlay'})['href']
        price=item.find('div',class_ ='product-price css-11s12ax is--current-price').text
        print(title,price,link)
User contributions licensed under: CC BY-SA
10 People found this is helpful
Advertisement