I am practicing web scraping using the requests and BeautifulSoup modules on the following website:
https://www.imdb.com/title/tt0080684/
My code thus far properly outputs the json in question. I’d like help in extracting from the json only the name and description into a response dictionary.
Code
# Send HTTP requests
import requests
import json
from bs4 import BeautifulSoup
class WebScraper:
def send_http_request():
# Obtain the URL via user input
url = input('Input the URL:n')
# Get the webpage
r = requests.get(url)
soup = BeautifulSoup(r.content, 'html.parser')
# Check response object's status code
if r:
p = json.loads("".join(soup.find('script', {'type':'application/ld+json'}).contents))
print(p)
else:
print('nInvalid movie page!')
WebScraper.send_http_request()
Desired Output
{"title": "Star Wars: Episode V - The Empire Strikes Back", "description": "After the Rebels are brutally overpowered by the Empire on the ice planet Hoth, Luke Skywalker begins Jedi training with Yoda, while his friends are pursued by Darth Vader and a bounty hunter named Boba Fett all over the galaxy."}
Advertisement
Answer
You can parse the dictonary and then print a new JSON object using the dumps method:
# Send HTTP requests
import requests
import json
from bs4 import BeautifulSoup
class WebScraper:
def send_http_request():
# Obtain the URL via user input
url = input('Input the URL:n')
# Get the webpage
r = requests.get(url)
soup = BeautifulSoup(r.content, 'html.parser')
# Check response object's status code
if r:
p = json.loads("".join(soup.find('script', {'type':'application/ld+json'}).contents))
output = json.dumps({"title": p["name"], "description": p["description"]})
print(output)
else:
print('nInvalid movie page!')
WebScraper.send_http_request()
Output:
{"title": "Star Wars: Episode V - The Empire Strikes Back", "description": "Star Wars: Episode V - The Empire Strikes Back is a movie starring Mark Hamill, Harrison Ford, and Carrie Fisher. After the Rebels are brutally overpowered by the Empire on the ice planet Hoth, Luke Skywalker begins Jedi training..."}