I researched but can’t find any answers to my question: I want get the main content, ignoring the commented content, how should I do? my scrapy spider looks like: But this codes give me only some nt. plz help, thank you. Answer When /text() in XPath or ::text in CSS fails to produce the desired result, I use another library.
Tag: xpath
Parsing a XML child element back as a string
I’m trying to parse a complex XML and xpath isn’t behaving like I thought it would. Here’s my sample xml: Here’s my python code: I get the output: when I expected: What am I doing wrong? Answer This XPath will get text and elements as expected Printing found nodes as OP requested Result with_tail argument prevents tail text to be
How to filter Selenium results setting a query limit?
I managed to get the data I wanted with selenium, but now I only need the first 17 data that it gives me, I need to make a kind of filter with this data, because I’m going to use conditions on top of them to use in another code. My result is below As you can see he gave me
AttributeError: ‘WebElement’ object has no attribute ‘select_by_value’ selecting dropdown menu using Selenium
I am trying to find menu prices for certain fast food restaurants from this website by different state. There is a dropdown menu where different states are the options. After I select a state (for example, California), I want to web scrape the different prices of their Ice Cream. However, I keep getting the same error message preventing me from
Selenium / Use pagination on site?
i want to trigger the pagination on this site: https://www.kicker.de/bundesliga/topspieler/2008-09 I found the element with this XPATH in the chrome-inspector: Now i want to click this element to go one page further – but i get an error. This is my code: But i get this error: How can i go to the next page using selenium? Answer The go
Selenium Select the exact date from Datepciker
This is continuation of this question asked earlier Earlier question I am able to select the date but the problem arises in below scenarios when there is some dates like I tried using XPath not contains but I am getting more than 1 element identified. below is my code I want to select only the dates which are below how
How can I click the button and move to next page using selenium?
I am not able to click the button using selenium and move to the next page. I have tried the following commands: I have added a screenshot of the html. What am I doing wrong? Answer To click on the element with text as Production you can use either of the following Locator Strategies: Using css_selector: Using xpath: Ideally to
Scrapy Python can‘t extract links with more stable xpath
I‘m Building a scraper for this website. I‘m using Python and scrapy Shell to extract the data that I want: xpath would be: //a[@class=“sb-card sb-card-company site-1×1 with-hover]/@href“ Using response.xpath(‘//a[@class=“sb-card sb-card-company site-1×1 with-hover]/@href“‘ returns [] I tried using contains(@class,“sb-card-company“) with the same result. Using other containers in the same way, changed nothing. Using a different page also had no effect. Using
Xpath: How to check if a tag comes before text or after text?
Assume I have the following two example pieces of HTML: <p>This is some text: <b>ABCD12345</b></p> <p><b>Name:</b> John Doe</p> I’m able to separate the <b> and non-<b> parts, but I (also) want to know how to determine whether the <b> part is at the start or at the end of the text (in other words; whether it has text before or
Web scraping: help needed last post and find link
First, sorry for my poor English. Actually, I have a script which scrapes a website to find comments in webpage, in python. Its for scrape all messages in page, but I will want scrape just last post. How to do this please? Too, I will want to find web links probably posted in last message, but a full link. Its