import requests from bs4 import BeautifulSoup result = requests.get('https://www.indeed.com/?vjk=5bc59746be36d8d0') source = result.content soup = BeautifulSoup(source, "lxml") job_titles = soup.find_all("a", {"class": "jcs-JobTitle"}) print(job_titles)
The problem here that printing job_titles returns an empty list instead of the job titles in the web site
please help me fix this problem and any help would be appreciated
Advertisement
Answer
When I first went to the URL you’re requesting, I was shown a search page with no jobs listed. It was only after I submitted a search that the page was populated with results. When I returned to the original URL again, the page was still populated (possibly with cached results). The blank page is probably what you’re getting back when you get the page from requests
.
Try using the full URL with parameters that the browser forwards you to after a search. For example, the URL https://www.indeed.com/jobs?q=data%20engineer&l=Raleigh%2C%20NC&vjk=b971ec43674ab50e
gives me back 15 job title links.