I am trying to run this code to scrape reviews from the google play store – but I keep getting the following error:
JavaScript
x
16
16
1
DevTools listening on ws://127.0.0.1:53044/devtools/browser/9de3e58b-6384-4809-bf01-31d47a57879f2
Traceback (most recent call last):3
File "c:/Users/Emil/Documents/Guatrain_Reviews/guatrain_reviews.py", line 20, in <module>4
Ptitle = driver.find_element_by_class_name('id-app-title').text.replace(' ','')5
File "C:UsersEmilMiniconda3envsdata_analysislibsite-packagesseleniumwebdriverremotewebdriver.py", line 564, in find_element_by_class_name6
return self.find_element(by=By.CLASS_NAME, value=name)7
File "C:UsersEmilMiniconda3envsdata_analysislibsite-packagesseleniumwebdriverremotewebdriver.py", line 978, in find_element8
'value': value})['value']9
File "C:UsersEmilMiniconda3envsdata_analysislibsite-packagesseleniumwebdriverremotewebdriver.py", line 321, in execute10
self.error_handler.check_response(response)11
File "C:UsersEmilMiniconda3envsdata_analysislibsite-packagesseleniumwebdriverremoteerrorhandler.py", line 242, in check_response12
raise exception_class(message, screen, stacktrace)13
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"class name","selector":"id-app-title"}14
(Session info: chrome=71.0.3578.98)15
(Driver info: chromedriver=2.46.628402 (536cd7adbad73a3783fdc2cab92ab2ba7ec361e1),platform=Windows NT 10.0.17134 x86_64)16
I suspect it has something to do with the id-app-title in
JavaScript
1
2
1
Ptitle = driver.find_element_by_class_name('id-app-title').text.replace(' ','')2
Could someone point out where I would find that Id for the app I am interested in OR help me identify where I am going wrong.
Thanks
EDIT
The final result I want needs to look something like this:
where for which ever app url I insert – it will extract the rating and reviews:
Thanks
Advertisement
Answer
That code is from 2016, so I’m assuming they changed the structure which is why there is no ‘id-app-title’ or anything from the original code. That’s just my assumption.
There’s a lot of work that still needs to be done with this code (like changing out the time.sleep for implicit waits by selenium, and quite frankly just to make it more robust, as I only was looking at this particular app review.EDIT SEE BELOW) It’s really complex html with tons of nested div and span tags with no specific meaning associated with the attributes/ class, etc. So I had trouble pulling out each user review element.
But essentially, I was able to open the page with the browser, have it continue to scroll down until it can click “Show More”, and just continue an x amount of times.
Once it does that, it iterates the span tags. Now I figured out every 10 span tags is relating to a single user. However if the app owner responds to a review, it offsets then by 2 so had to account for that.
I’m fairly newer to programming, so I apologize for messy code and inefficiency. I’m sure an expert would be able to provide a better solution, however, this can hopefully get you started or playing around:
JavaScript
1
67
67
1
#load webdriver function from selenium2
from selenium import webdriver3
from time import sleep4
import bs45
import pandas as pd6
import requests7
from selenium.webdriver.common.keys import Keys8
import time9
10
# Change this number to get more or less reviews11
# Current set of x=100 yielded 11,312 reviews12
x = 10013
14
link = "https://play.google.com/store/apps/details?id=uk.co.o2.android.myo2&hl=en_GB"15
16
driver = webdriver.Chrome('C:/chromedriver_win32/chromedriver.exe')17
driver.get(link + '&showAllReviews=true')18
19
num_clicks = 020
num_scrolls = 021
while num_clicks <= x and num_scrolls <= x*5:22
try:23
show_more = driver.find_element_by_xpath('//*[@id="fcxH9b"]/div[4]/c-wiz/div/div[2]/div/div[1]/div/div/div[1]/div[2]/div[2]/div/content/span')24
show_more.click()25
num_clicks += 126
27
except:28
html = driver.find_element_by_tag_name('html')29
html.send_keys(Keys.END)30
num_scrolls +=131
time.sleep(2)32
33
soup = bs4.BeautifulSoup(driver.page_source, 'html.parser')34
h2 = soup.find_all('h2')35
36
results_df = pd.DataFrame()37
for ele in h2:38
if ele.text == 'Reviews':39
c_wiz = ele.parent.parent.find_all('c-wiz')40
for sibling in c_wiz[0].next_siblings:41
try:42
#print (sibling)43
comment_shift = 044
spans = sibling.find_all('span')45
for user_block in range(0,len(spans)):46
i = user_block *1047
name = spans[i+0+comment_shift].text48
try:49
rating = spans[i+1+comment_shift].div.next_element['aria-label']50
rating = str(''.join(filter(str.isdigit, rating)))51
except:52
comment_shift += 253
continue54
date = spans[i+2+comment_shift].text55
review = spans[i+8+comment_shift].text56
print ('Name: %snRating: %snDate: %snReview: %sn' %(name, rating, date, review))57
temp_df = pd.DataFrame([[date, rating, name, review]], columns = ['Date','Rating','User','Review'])58
59
results_df = results_df.append(temp_df)60
except:61
continue62
63
results_df = results_df.reset_index(drop=True)64
results_df.to_csv('C:/reviews.csv', index=False)65
66
driver.close()67
Output:
JavaScript
1
66
66
1
print (results_df)2
Date Review3
0 31 January 2019 Was broken for pay as you go customers. Has no4
1 2 February 2019 o2 just won't be happy until their customer se...5
2 1 February 2019 Excellent quality piece of kit6
3 6 February 2019 Gud 😁7
4 23 December 2018 Can't get into the app using correct log in de...8
5 16 December 2018 The update is rubbish. I can't use MyO2 anymor...9
6 6 December 2018 Stop logging me out with every update, they ad10
7 25 December 2018 cant use this app anymore. shame i use to use 11
8 16 December 2018 Started receiving texts from 02 immediately af12
9 10 January 2019 havent been with the network long nor have i u13
10 22 December 2018 update has killed this app. why do I have to p14
11 9 January 2019 This app is now unusable for pay as you go cus15
12 26 January 2019 Wouldn't it be nice to find an app that the de...16
13 19 December 2018 wont let me log in now since the latest update17
14 13 January 2019 it was ok for a while wen u needed to put in y18
15 6 January 2019 from last update I can't login anymore. not ev...19
16 24 January 2019 I'm having 2 change review again coz I can't g20
17 5 January 2019 Changed my rating for this down from five to o21
18 22 December 2018 no longer works for me. shame as it was useful22
19 31 January 2019 total waste of time since update. not able to 23
20 23 January 2019 Despite what the description states the curren24
21 24 December 2018 When it finally lets you log in it then says t25
22 17 January 2019 Update breaks it, can't log in, log in on webs...26
23 5 January 2019 02 what have you done to app cant log in chang27
24 30 November 2018 Simple easy to use and all info available of m28
25 30 November 2018 No longer works for pay and go customers so co29
26 8 December 2018 Will not log me in after downloading the lates30
27 15 January 2019 Unable to log on to the app since the update. 31
28 1 January 2019 Very easy to use. Keeps me up to date.32
29 1 December 2018 Good app maybe it should be as colourful as th33
34
11282 12 February 2017 Just re installed this a on my new device. Ha35
11283 18 December 2016 Since updating this app on my Samsung S3 mini 36
11284 19 January 2017 Lately the app gives intermittent server error37
11285 7 December 2016 New update38
11286 12 December 2016 O2 needs to put right fast39
11287 12 February 2017 Although unlimited minutes/texts I would still40
11288 30 December 2016 Never works41
11289 13 August 2017 I have a Samsung galaxy 7 and the o2 app is no42
11290 6 December 2016 Doesn't work anymore43
11291 4 December 2016 Since the last update this app does not work f44
11292 3 December 2016 O245
11293 5 December 2016 Good app (when it opens)46
11294 11 January 2017 Stopped working and when it does work47
11295 1 December 2016 Nothing but a blue screen. Not happy.48
11296 2 December 2016 Worst app ever49
11297 18 January 2017 It's easier than trying to keep track of my ac...50
11298 16 February 2017 The new update only shows blue screen before t51
11299 15 January 2017 Mr Dimitrov52
11300 8 February 2017 Code 4 error frequently53
11301 4 January 2017 Won't work at all54
11302 27 January 2017 O2 GURU , EXCELLENT, ESQISET , PHANOMAL, SE55
11303 15 February 2017 Works well enough.56
11304 1 December 2016 Great app keeps you up to.date57
11305 28 December 2016 My 0258
11306 16 December 2016 This is a "APPY APP""59
11307 22 November 2016 Doesn't work for business account. Only shows ...60
11308 25 November 2016 Doesn't work anymore61
11309 11 November 2016 The ap won't open its just a blue screen I've 62
11310 24 November 2016 Doesn't work63
11311 12 November 2016 My 0264
65
[11312 rows x 4 columns]66
Edit:
I tried with a couple different links:
JavaScript
1
3
1
link = "https://play.google.com/store/apps/details?id=com.outfit7.mytalkingtom2"2
link = "https://play.google.com/store/apps/details?id=com.ingeniooz.hercule"3
and it appeared to work:
Output:
JavaScript
1
66
66
1
print (results_df)2
Date Review3
0 February 5, 2019 after update it is not workin before it was ev4
1 February 4, 2019 no word to describe simply 😍5
2 February 6, 2019 I loved this game6
3 February 6, 2019 it is very funny game and very nice game also7
4 February 6, 2019 😎8
5 February 6, 2019 relaxing effect9
6 February 6, 2019 this is a cool game10
7 February 6, 2019 Good game11
8 February 6, 2019 Beast12
9 February 1, 2019 Love this game, it is so much better then the 13
10 February 1, 2019 The recent updates are epic. The blender and d14
11 February 1, 2019 i like this funny game because tom is jumping 15
12 February 2, 2019 tom 2 is a great game16
13 February 3, 2019 Very very nice game17
14 February 3, 2019 I like it very much18
15 February 5, 2019 Nice and superb game.19
16 February 2, 2019 Tom is a cutipie20
17 February 2, 2019 it is so cute21
18 February 2, 2019 tr ty022
19 February 2, 2019 so good23
20 February 2, 2019 nice game24
21 February 1, 2019 Nice game25
22 February 3, 2019 i love this game26
23 February 6, 2019 l love this game as it is fun and enjoyable to27
24 February 2, 2019 love it28
25 February 5, 2019 it is so awesome 👍😍😊29
26 February 2, 2019 Amazing30
27 February 3, 2019 nice31
28 February 6, 2019 good32
29 January 30, 2019 Anish Biswa 3 to be a bit. I'm not a good idea...33
34
1770 February 2, 2019 fun35
1771 February 5, 2019 ect,36
1772 February 6, 2019 tom. is so cute37
1773 February 6, 2019 nice38
1774 January 5, 2019 urguuhtr39
1775 January 14, 2019 Very interesting game 👌😀😀40
1776 January 10, 2019 It s very very very nice41
1777 January 21, 2019 supab game😘😘😘😘42
1778 January 16, 2019 it's too funny 🐹🐹🐹🐰🐰🐰43
1779 January 20, 2019 wow Best game44
1780 January 27, 2019 It's damn good45
1781 January 28, 2019 this a good and supper game. very nice game. ,46
1782 February 4, 2019 i love this game very very very much47
1783 January 5, 2019 super48
1784 January 12, 2019 It's fun Lol49
1785 January 16, 2019 it ,s so good50
1786 January 23, 2019 fun game for kids.loved it51
1787 January 27, 2019 It's so nice52
1788 February 1, 2019 Nice The Baby games i like 😀😀😀😀53
1789 January 29, 2019 it's funny and it's fun to play54
1790 January 10, 2019 best game so cute55
1791 January 10, 2019 So Cute!56
1792 January 24, 2019 i lv this game very nice game ..57
1793 January 25, 2019 Its superb I love this game 😘58
1794 January 27, 2019 It is best game ever played😀😀😀😁😁😁59
1795 January 19, 2019 I love it!60
1796 January 20, 2019 good game!61
1797 January 16, 2019 i love this game 😍.62
1798 January 25, 2019 It is a good game for kids..63
1799 January 31, 2019 my talking tom is fun😊😊😊64
65
[1800 rows x 4 columns]66
And
JavaScript
1
19
19
1
print (results_df)2
Date Review3
0 December 2, 2018 It's a very well-thought-out an all rounded ap...4
1 January 1, 2019 L'application est superbe et hyper complète! B...5
2 December 6, 2017 Great workout diary with statistics. Easy to u6
3 June 13, 2017 I love this app! I've tried so many others, bu...7
4 March 28, 2017 Works great at what it does. You can add exerc8
5 March 21, 2017 Great9
6 December 8, 2016 Has all I need to build & adjust my workouts10
7 October 23, 2016 Goodish11
8 September 23, 2016 Great app12
9 July 18, 2016 Excellent13
10 March 9, 2016 great app.14
11 July 10, 2015 Amazing and easy to use15
12 June 5, 2015 I dreamt of this app, Hercule made it. Best ap16
13 March 18, 2015 Really good, but17
18
[14 rows x 4 columns]19