I am currently trying to login brickseek’s website using FormRequest method but I am unable to login successfully. I keep on getting 404 error when using the scrapy crawl command. It seems to me that scrapy is redirecting the page incorrectly. I also noticed that my login and password are inputted in the redirected webpage which is weird.
The first page should be https://brickseek.com/login with the login information added to that page. Then once logged in the second page should be https://brickseek.com/account
It keeps redirecting me to https://brickseek.com/products/?search=&log=12345678&pwd=12345678
Here is the code:
import scrapy from scrapy.http import FormRequest from scrapy.utils.response import open_in_browser from ..items import QuotetutorialItem class QuoteSpider(scrapy.Spider): """Enter Login information located in Inspect Element, find Network, then search for login.""" name = 'login' start_urls = ['https://www.brickseek.com/login/'] def parse(self, response): """Enter Login information located in Inspect Element, find Network, then search for login.""" return FormRequest.from_response(response, formdata={'log': '12345678', 'pwd': '12345678'}, callback=self.after_login) def after_login(self, response): """Enter what you want scrapy to web scrape.""" open_in_browser(response) items = QuotetutorialItem() div_title = response.css('div.banner_content') title = div_title.css('span.banner__title-text::text').extract() items['title']: title yield items
When I enter command scrapy crawl login, I get this:
(venv) C:UsersTreborPycharmProjectsNewScraperquotetutorialquotetutorial>scrapy crawl login 2020-02-07 00:19:27 [scrapy.utils.log] INFO: Scrapy 1.8.0 started (bot: quotetutorial) 2020-02-07 00:19:27 [scrapy.utils.log] INFO: Versions: lxml 4.5.0.0, libxml2 2.9.5, cssselect 1.1.0, parsel 1.5.2, w3lib 1.21.0, Twisted 19.10.0, Python 3.8.0 (tags/v3.8.0:fa919 fd, Oct 14 2019, 19:21:23) [MSC v.1916 32 bit (Intel)], pyOpenSSL 19.1.0 (OpenSSL 1.1.1d 10 Sep 2019), cryptography 2.8, Platform Windows-10-10.0.18362-SP0 2020-02-07 00:19:28 [scrapy.crawler] INFO: Overridden settings: {'BOT_NAME': 'quotetutorial', 'NEWSPIDER_MODULE': 'quotetutorial.spiders', 'SPIDER_MODULES': ['quotetutorial.spid ers'], 'USER_AGENT': 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.131 Safari/537.36'} 2020-02-07 00:19:28 [scrapy.extensions.telnet] INFO: Telnet Password: e8e844c7cbefd148 2020-02-07 00:19:28 [scrapy.middleware] INFO: Enabled extensions: ['scrapy.extensions.corestats.CoreStats', 'scrapy.extensions.telnet.TelnetConsole', 'scrapy.extensions.logstats.LogStats'] 2020-02-07 00:19:28 [scrapy.middleware] INFO: Enabled downloader middlewares: ['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware', 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware', 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware', 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware', 'scrapy.downloadermiddlewares.retry.RetryMiddleware', 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware', 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware', 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware', 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware', 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware', 'scrapy.downloadermiddlewares.stats.DownloaderStats'] 2020-02-07 00:19:28 [scrapy.middleware] INFO: Enabled spider middlewares: ['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware', 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware', 'scrapy.spidermiddlewares.referer.RefererMiddleware', 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware', 'scrapy.spidermiddlewares.depth.DepthMiddleware'] 2020-02-07 00:19:28 [scrapy.middleware] INFO: Enabled item pipelines: [] 2020-02-07 00:19:28 [scrapy.core.engine] INFO: Spider opened 2020-02-07 00:19:28 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 2020-02-07 00:19:28 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023 2020-02-07 00:19:29 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.brickseek.com/login/> (referer: None) 2020-02-07 00:19:29 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (301) to <GET https://brickseek.com/products/?search=&user_login=12345678&user_pass=12345678> from <GET https://www.brickseek.com/products/?search=&user_login=12345678&user_pass=12345678> 2020-02-07 00:19:31 [scrapy.core.engine] DEBUG: Crawled (404) <GET https://brickseek.com/products/?search=&user_login=12345678&user_pass=12345678> (referer: https://www.bric kseek.com/login/) 2020-02-07 00:19:31 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <404 https://brickseek.com/products/?search=&user_login=12345678&user_pass=12345678>: HTTP s tatus code is not handled or not allowed 2020-02-07 00:19:31 [scrapy.core.engine] INFO: Closing spider (finished) 2020-02-07 00:19:31 [scrapy.statscollectors] INFO: Dumping Scrapy stats: {'downloader/request_bytes': 1329, 'downloader/request_count': 3, 'downloader/request_method_count/GET': 3, 'downloader/response_bytes': 41132, 'downloader/response_count': 3, 'downloader/response_status_count/200': 1, 'downloader/response_status_count/301': 1, 'downloader/response_status_count/404': 1, 'elapsed_time_seconds': 2.460197, 'finish_reason': 'finished', 'finish_time': datetime.datetime(2020, 2, 7, 6, 19, 31, 193946), 'httperror/response_ignored_count': 1, 'httperror/response_ignored_status_count/404': 1, 'log_count/DEBUG': 3, 'log_count/INFO': 11, 'request_depth_max': 1, 'response_received_count': 2, 'scheduler/dequeued': 3, 'scheduler/dequeued/memory': 3, 'scheduler/enqueued': 3, 'scheduler/enqueued/memory': 3, 'start_time': datetime.datetime(2020, 2, 7, 6, 19, 28, 733749)} 2020-02-07 00:19:31 [scrapy.core.engine] INFO: Spider closed (finished)
cURL code:
import requests cookies = { '__cfduid': 'da5ab5ff9b55ed8d9432e500501554604963', '__stripe_mid': '3689131-200-d1-9906-c3022a5de64', 'wordpress_test_cookie': 'WP+Cookie+check', 'ac_enable_tracking': '1', '_ga': 'GA1.2.114763001.15586575', '_hjid': '72b2d2b9-feef-ee-827a-2af3fe09ed', '__zlcmid': 'tZiQfA1Ce4mixx', '_form_1_': 'Sun Aug 04 2019 05:45:27 GMT-0500 (Central Daylight Time)', '_hjIncludedInSample': '1', 'WRMCembrGyQs': '1L.OZR%40s7hny', 'CpwgInmYKPQHM': 'mqJCzckWg7YOTl', 'de-EnfISbN': '%5Dj_2Wqrp8a', 'pfICHvSKW_lZr': 'ykbU6BaGwi0cS', '_gid': 'GA1.2.83233215.150787155', 'bs_zip': '*****', 'ABTasty': 'uid%3D190840545159480%26fst%3D15491552621%26pst%3D1579541792%26cs%3D1581042402480%26ns%3D8%26pvt%3D8%26pvs%3D1%26th%3D', 'cf_clearance': '7bc4c20678aa9d74cbd8517348a72f0451dcf8-1581043791-0-250', '_gat_UA-55274071-1': '1', 'wordpress_logged_in_03f9e3712464d83801b8eecd7edae': '12345678%7C151232373%7C1WXUNJmFaDGPkxwbJiPCmftQZEWYuXsRUe0qFM8kT%7C041b932ee01c4e7f5864d918a678084f4f543de2a532c64322a72ca5d39c40', } headers = { 'authority': 'brickseek.zendesk.com', 'cache-control': 'max-age=0', 'origin': 'https://brickseek.com', 'upgrade-insecure-requests': '1', 'content-type': 'application/json; charset=UTF-8', 'user-agent': 'Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Mobile Safari/537.36', 'sec-fetch-user': '?1', 'accept': '*/*', 'sec-fetch-site': 'cross-site', 'sec-fetch-mode': 'cors', 'referer': 'https://brickseek.com/', 'accept-encoding': 'gzip, deflate, br', 'accept-language': 'en-US,en;q=0.9', 'Referer': 'https://brickseek.com/', 'User-Agent': 'Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Mobile Safari/537.36', 'cookie': '__cfduid=db073c10d776e79a06f2c8849e128a601554604964; TS011baee6=0130aff232f157d75c3e3c4e190122d294ad62f857be4c386f8fa205f622df7166e3284eeec932768fd9cf24cefd766096438d32; TS01a8b35a=01da6997013403b00a91c12774d32da66071ff49f68f15b5910144b1419a5eebec6ac1031cd95090f3216192aed2407a71e9a0cca', 'if-none-match': 'W/"abd82b40b936e162ba2fd12ebca21"', 'if-modified-since': 'Thu, 03 Oct 2019 19:25:17 GMT', 'Origin': 'https://brickseek.com', 'authorization': 'd62c460eb96a27a6dc5664ae30a091', 'Upgrade-Insecure-Requests': '1', 'Pragma': 'no-cache', 'Accept-Encoding': 'identity;q=1, *;q=0', 'Accept-Language': 'en-US,en;q=0.9', 'Sec-WebSocket-Key': 'AtleA0fubdJ/pCYYR+0lQ==', 'Upgrade': 'websocket', 'Sec-WebSocket-Extensions': 'permessage-deflate; client_max_window_bits', 'Cache-Control': 'no-cache', 'Connection': 'Upgrade', 'Sec-WebSocket-Version': '13', 'Range': 'bytes=0-', } data = [ ('log', '12345678'), ('pwd', '12345678'), ('wp-submit', 'Log In'), ('redirect_to', 'https://brickseek.com/'), ('mepr_process_login_form', 'true'), ('mepr_is_login_page', 'true'), ('_LkNiouJZhOFzHR', 'H4I_a9ZkfdCDPoj'), ('_LkNiouJZhOFzHR', 'H4I_a9ZkfdCDPoj'), ('YKJmh-NQXRuD', 'O[HDXTUysdu9'), ('YKJmh-NQXRuD', 'O[HDXTUysdu9'), ('EmWpdPiAGnc', 'ybBQluhGrT5'), ('EmWpdPiAGnc', 'ybBQluhGrT5'), ('DsdaYLSwTt', 'bWTpV]1ym3'), ('DsdaYLSwTt', 'bWTpV]1ym3'), ] response = requests.post('https://brickseek.com/login/', headers=headers, cookies=cookies, data=data)
Advertisement
Answer
# -*- coding: utf-8 -*- import scrapy from scrapy.utils.response import open_in_browser data={'log': 'YOUR_USER', 'pwd': 'YOUR_PASSWORD', 'wp-submit': 'Log In', 'redirect_to': 'https://brickseek.com/', 'mepr_process_login_form': 'true', 'mepr_is_login_page': 'true', '_LkNiouJZhOFzHR': 'H4I_a9ZkfdCDPoj', 'YKJmh-NQXRuD': 'O[HDXTUysdu9', 'EmWpdPiAGnc': 'ybBQluhGrT5', 'DsdaYLSwTt': 'bWTpV]1ym3'} headers = { 'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:72.0) Gecko/20100101 Firefox/72.0', 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8', 'Accept-Language': 'en-US,en;q=0.5', 'Content-Type': 'application/x-www-form-urlencoded', 'Origin': 'https://brickseek.com', 'Connection': 'keep-alive', 'Referer': 'https://brickseek.com/login/', 'Upgrade-Insecure-Requests': '1', 'TE': 'Trailers', } class BrickseekSpider(scrapy.Spider): name = 'brickseek' allowed_domains = ['brickseek.com'] def start_requests(self): yield scrapy.http.FormRequest(url='https://brickseek.com/login/',headers=headers,formdata=data, callback=self.parse) def parse(self,response): open_in_browser(response) #your code
Full code to do it