I’m working on a web-scraping task and I can already collect the data in a very rudimentary way.
Basically, I need a function to collect a list of songs and artists from the Allmusic.com and then add the data in df. In this example, I use this link: https://www.allmusic.com/mood/tender-xa0000001119/songs
So far, I managed to accomplish most of the objective, however, I had to perform two different functions (def get_song() and def get_performer()).
I would like, if possible, an alternative to join these two functions.
The codes used are below:
JavaScript
x
37
37
1
import requests2
from bs4 import BeautifulSoup3
4
headers = {'User-Agent': 'Mozilla/5.0 (X11; Linux i586; rv:31.0) Gecko/20100101 Firefox/31.0'}5
link = "https://www.allmusic.com/mood/tender-xa0000001119/songs"6
7
8
# Function to collect songs (title)9
songs = []10
11
def get_song():12
url = link13
source_code = requests.get(url, headers=headers)14
plain_text = source_code.text15
soup = BeautifulSoup(plain_text)16
for td in soup.findAll('td', {'class': 'title'}):17
for a in td.findAll('a')[0]:18
song = a.string19
songs.append(song)20
21
# Function to collect performers22
performers = []23
24
def get_performer():25
url = link26
source_code = requests.get(url, headers=headers)27
plain_text = source_code.text28
soup = BeautifulSoup(plain_text)29
for td in soup.findAll('td', {'class': 'performer'}):30
for a in td.findAll('a'):31
performer = a.string32
performers.append(performer)33
34
get_song(), get_performer() # Here, I call the two functions, but the goal, if possible, is to use one function.35
36
df = pd.DataFrame(list(zip(songs,performers)), columns=['song', 'performer']) # df creation37
Advertisement
Answer
To get titles/performer you can use next example:
JavaScript
1
21
21
1
import requests2
import pandas as pd3
from bs4 import BeautifulSoup4
5
url = "https://www.allmusic.com/mood/tender-xa0000001119/songs"6
headers = {7
"User-Agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:91.0) Gecko/20100101 Firefox/91.0"8
}9
10
soup = BeautifulSoup(requests.get(url, headers=headers).content, "html.parser")11
12
all_data = []13
for td in soup.select("td.title"):14
title = td.get_text(strip=True)15
performer = td.find_next("td").get_text(strip=True)16
all_data.append((title, performer))17
18
df = pd.DataFrame(all_data, columns=["title", "performer"])19
print(df)20
df.to_csv("data.csv", index=False)21
Prints:
JavaScript
1
40
40
1
title performer2
0 Knock You Down Keri Hilson3
1 Down Among the Wine and Spirits Elvis Costello4
2 I Felt The Chill Elvis Costello5
3 She Handed Me A Mirror Elvis Costello6
4 I Dreamed Of My Old Lover Elvis Costello7
5 She Was No Good Elvis Costello8
6 The Crooked Line Elvis Costello9
7 Changing Partners Elvis Costello10
8 Small Town Southern Man Alan Jackson11
9 Find Your Love Drake12
10 Today Was a Fairytale Taylor Swift13
11 Need You Now Lady A14
12 American Honey Lady A15
13 Peace Dream Ringo Starr16
14 If I Died Today Tim McGraw17
15 Still Tim McGraw18
16 I Need Love Ledisi19
17 Uhh Ahh Boyz II Men20
18 Shattered Heart Brandy21
19 Right Here (Departed) Brandy22
20 Warm It Up (With Love) Brandy23
21 If I Were a Boy Beyoncé24
22 Why Does She Stay Ne-Yo25
23 Daddy Needs a Drink Drive-By Truckers26
24 Think About You Ringo Starr27
25 Liverpool 8 Ringo Starr28
26 Nefertiti Herbie Hancock29
27 River Herbie Hancock/Corinne Bailey Rae30
28 Both Sides Now Herbie Hancock31
29 Court and Spark Herbie Hancock/Norah Jones32
30 I Taught Myself How to Grow Old Ryan Adams33
31 Ghetto Kelly Rowland/Snoop Dogg34
32 Little Girl Enrique Iglesias35
33 The Magdalene Laundries Emmylou Harris36
34 Because of You Ne-Yo37
35 We Belong Together Mariah Carey38
36 Thank You for Loving Me Bon Jovi39
37 He's Younger Than You Are Sonny Rollins40
and saves data.csv (screenshot from LibreOffice):
