JavaScript
x
22
22
1
import scrapy2
from scrapy.http import Request3
4
5
6
class TestSpider(scrapy.Spider):7
name = 'test'8
start_urls = ['https://www.ifep.ro/justice/lawyers/lawyerspanel.aspx']9
custom_settings = {10
'CONCURRENT_REQUESTS_PER_DOMAIN': 1,11
'DOWNLOAD_DELAY': 1,12
'USER_AGENT': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36'13
}14
15
16
17
def parse(self, response):18
books = response.xpath("//div[@class='list-group']//@href").extract()19
for book in books:20
url = response.urljoin(book)21
print(url)22
I want to remove these unnecessary url from the link the website is https://www.ifep.ro/justice/lawyers/lawyerspanel.aspx
JavaScript
1
9
1
http://www.unbr.ro2
http://www.inppa.ro3
http://www.uniuneanotarilor.ro/4
Prima pagină5
http://www.executori.ro/6
http://www.csm1909.ro7
http://www.inm-lex.ro8
http://www.just.ro9
Advertisement
Answer
You can apply endswith method along with continue keyword to remove the desired urls
JavaScript
1
22
22
1
import scrapy2
from scrapy.http import Request3
4
class TestSpider(scrapy.Spider):5
name = 'test'6
start_urls = ['https://www.ifep.ro/justice/lawyers/lawyerspanel.aspx']7
custom_settings = {8
'CONCURRENT_REQUESTS_PER_DOMAIN': 1,9
'DOWNLOAD_DELAY': 1,10
'USER_AGENT': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36'11
}12
13
14
15
def parse(self, response):16
books = response.xpath("//div[@class='list-group']//@href").extract()17
for book in books:18
url = response.urljoin(book)19
if url.endswith('.ro') or url.endswith('.ro/'):20
continue21
print(url)22
Output:
JavaScript
1
16
16
1
https://www.ifep.ro/justice/lawyers/LawyerFile.aspx?RecordId=1091&Signature=3782702
https://www.ifep.ro/justice/lawyers/LawyerFile.aspx?RecordId=159077&Signature=3782703
https://www.ifep.ro/justice/lawyers/LawyerFile.aspx?RecordId=159076&Signature=3782704
https://www.ifep.ro/justice/lawyers/LawyerFile.aspx?RecordId=159075&Signature=3782705
https://www.ifep.ro/justice/lawyers/LawyerFile.aspx?RecordId=159021&Signature=3782706
https://www.ifep.ro/justice/lawyers/LawyerFile.aspx?RecordId=159020&Signature=3782707
https://www.ifep.ro/justice/lawyers/LawyerFile.aspx?RecordId=159019&Signature=3782708
https://www.ifep.ro/justice/lawyers/LawyerFile.aspx?RecordId=159018&Signature=3782709
https://www.ifep.ro/justice/lawyers/LawyerFile.aspx?RecordId=21846&Signature=37827010
https://www.ifep.ro/justice/lawyers/LawyerFile.aspx?RecordId=165927&Signature=37827011
https://www.ifep.ro/justice/lawyers/LawyerFile.aspx?RecordId=83465&Signature=37827012
https://www.ifep.ro/justice/lawyers/LawyerFile.aspx?RecordId=47724&Signature=37827013
https://www.ifep.ro/justice/lawyers/LawyerFile.aspx?RecordId=32097&Signature=37827014
https://www.ifep.ro/justice/lawyers/LawyerFile.aspx?RecordId=29573&Signature=37827015
https://www.ifep.ro/justice/lawyers/LawyerFile.aspx?RecordId=19880&Signature=37827016