I researched but can’t find any answers to my question: I want get the main content, ignoring the commented content, how should I do? my scrapy spider looks like: But this codes give me only some nt. plz help, thank you. Answer When /text() in XPath or ::text in CSS fails to produce the desired result, …
Tag: xpath
Parsing a XML child element back as a string
I’m trying to parse a complex XML and xpath isn’t behaving like I thought it would. Here’s my sample xml: Here’s my python code: I get the output: when I expected: What am I doing wrong? Answer This XPath will get text and elements as expected Printing found nodes as OP requested Resul…
How to filter Selenium results setting a query limit?
I managed to get the data I wanted with selenium, but now I only need the first 17 data that it gives me, I need to make a kind of filter with this data, because I’m going to use conditions on top of them to use in another code. My result is below As you can see he gave me
AttributeError: ‘WebElement’ object has no attribute ‘select_by_value’ selecting dropdown menu using Selenium
I am trying to find menu prices for certain fast food restaurants from this website by different state. There is a dropdown menu where different states are the options. After I select a state (for example, California), I want to web scrape the different prices of their Ice Cream. However, I keep getting the s…
Selenium / Use pagination on site?
i want to trigger the pagination on this site: https://www.kicker.de/bundesliga/topspieler/2008-09 I found the element with this XPATH in the chrome-inspector: Now i want to click this element to go one page further – but i get an error. This is my code: But i get this error: How can i go to the next pa…
Selenium Select the exact date from Datepciker
This is continuation of this question asked earlier Earlier question I am able to select the date but the problem arises in below scenarios when there is some dates like I tried using XPath not contains but I am getting more than 1 element identified. below is my code I want to select only the dates which are…
How can I click the button and move to next page using selenium?
I am not able to click the button using selenium and move to the next page. I have tried the following commands: I have added a screenshot of the html. What am I doing wrong? Answer To click on the element with text as Production you can use either of the following Locator Strategies: Using css_selector: Usin…
Scrapy Python can‘t extract links with more stable xpath
I‘m Building a scraper for this website. I‘m using Python and scrapy Shell to extract the data that I want: xpath would be: //a[@class=“sb-card sb-card-company site-1×1 with-hover]/@href“ Using response.xpath(‘//a[@class=“sb-card sb-card-company site-1×1 with-hover]/@href“‘ returns [] I tried using …
Xpath: How to check if a tag comes before text or after text?
Assume I have the following two example pieces of HTML: <p>This is some text: <b>ABCD12345</b></p> <p><b>Name:</b> John Doe</p> I’m able to separate the <b> and non-<b> parts, but I (also) want to know how to determine whether the <b> par…
Web scraping: help needed last post and find link
First, sorry for my poor English. Actually, I have a script which scrapes a website to find comments in webpage, in python. Its for scrape all messages in page, but I will want scrape just last post. How to do this please? Too, I will want to find web links probably posted in last message, but a full link. It…