I am trying to extract this immigration data from Canada immigration website
I tried extracting the whole table, individual td tags, all returns empty list.
from bs4 import BeautifulSoup
from requests_html import HTMLSession
import requests
s = HTMLSession()
data=[]
url = "https://www.canada.ca/en/immigration-refugees-citizenship/corporate/mandate/policies-operational-instructions-agreements/ministerial-instructions/express-entry-rounds.html"
def getdata(url):
r = s.get(url)
soup = BeautifulSoup(r.text, 'html.parser')
return soup
def gettabledata(soup):
table = soup.find_all('td',{'class':'sorting_1'})
I also tried finding tr tags in a table;
table = soup.find('table', class_='table')
print("n TABLE n")
print(table)
table_body = table.find('tbody')
print("n TABLEBODY n")
print(table_body)
rows = table_body.find_all('tr')
print("n ROWS n")
print(rows)
for row in rows:
cols = row.find_all('td')
cols = [ele.text.strip() for ele in cols]
data.append([ele for ele in cols if ele])
print(data)
Also tried , but doesnt work:
divs = soup.find_all('div', class_='dataTables_wrapper')
print("divs n", divs)
for div in divs:
table = div.find('tbody',class_='wb-data-json-inited')
print("n TABLE n")
print(table)
table_body = table.find('tbody')
print("n TABLEBODY n")
print(table_body)
rows = table.find_all('tr')
print("n ROWS n")
print(rows)
for row in rows:
cols = row.find_all('td')
cols = [ele.text.strip() for ele in cols]
data.append([ele for ele in cols if ele])
print(data)
What am I missing, and how can I extract the table data?
Another doubt I have is how I can access the Program name and values just enclosed in td tags with no attributes. Each line in the table is a tr tag with a bunch of td tags in them :
<tr>
<td><a href="/content/canadasite/en/immigration-refugees-citizenship/corporate/mandate/policies-operational-instructions-agreements/ministerial-instructions/express-entry-rounds/invitations-219.html">219</a></td>
<td class="nowrap">March 30, 2022</td>
<td>Provincial Nominee Program</td>
<td>919</td>
<td>785</td>
<td>30-03-2022</td>
<td><p>Provincial Nominee Program</p></td>
</tr>
Advertisement
Answer
That table is being loaded dynamically in page. You can inspect the Network Tab in Dev Tools, and see which API is being accessed to pull the data from. This is one way of obtaining the data in that table:
import requests
import pandas as pd
r = requests.get('https://www.canada.ca/content/dam/ircc/documents/json/ee_rounds_123_en.json')
df = pd.DataFrame(r.json()['rounds'])
print(df)
This will return a dataframe:
drawNumber drawNumberURL drawDate drawDateFull drawName drawSize drawCRS mitext DrawText1 drawText2 drawDateTime drawCutOff drawDistributionAsOn dd1 dd2 dd3 dd4 dd5 dd6 dd7 dd8 dd9 dd10 dd11 dd12 dd13 dd14 dd15 dd16 dd17 dd18 0 229 <a href='/en/immigration-refugees-citizenship/corporate/mandate/policies-operational-instructions-agreements/ministerial-instructions/express-entry-rounds/invitations.html?q=229'>229</a> 2022-08-17 August 17, 2022 No Program Specified 2,250 525 <a href='/en/immigration-refugees-citizenship/corporate/mandate/policies-operational-instructions-agreements/ministerial-instructions/express-entry-rounds/invitations.html?q=229'>Invitations to apply for permanent residence under the Express Entry system #229</a> <a href='/en/immigration-refugees-citizenship/corporate/mandate/policies-operational-instructions-agreements/ministerial-instructions/express-entry-rounds/invitations.html?q=229'>See full text of Ministerial Instruction</a> Federal Skilled Worker, Canadian Experience Class, Federal Skilled Trades and Provincial Nominee Program August 17, 2022 at 13:43:47 UTC December 28, 2021 at 11:03:15 UTC August 15, 2022 538 8,221 62,753 5,435 9,129 18,831 16,465 12,893 58,113 12,200 12,721 9,801 11,138 12,253 68,440 35,745 5,137 238,947 1 228 <a href='/en/immigration-refugees-citizenship/corporate/mandate/policies-operational-instructions-agreements/ministerial-instructions/express-entry-rounds/invitations.html?q=228'>228</a> 2022-08-03 August 3, 2022 No Program Specified 2,000 533 <a href='/en/immigration-refugees-citizenship/corporate/mandate/policies-operational-instructions-agreements/ministerial-instructions/express-entry-rounds/invitations.html?q=228'>Invitations to apply for permanent residence under the Express Entry system #228</a> <a href='/en/immigration-refugees-citizenship/corporate/mandate/policies-operational-instructions-agreements/ministerial-instructions/express-entry-rounds/invitations.html?q=228'>See full text of Ministerial Instruction</a> Federal Skilled Worker, Canadian Experience Class, Federal Skilled Trades and Provincial Nominee Program August 03, 2022 at 15:16:24 UTC January 06, 2022 at 14:29:50 UTC August 2, 2022 640 8,975 62,330 5,343 9,044 18,747 16,413 12,783 57,987 12,101 12,705 9,747 11,117 12,317 68,325 35,522 5,145 238,924 2 227 <a href='/en/immigration-refugees-citizenship/corporate/mandate/policies-operational-instructions-agreements/ministerial-instructions/express-entry-rounds/invitations.html?q=227'>227</a> 2022-07-20 July 20, 2022 No Program Specified 1,750 542 <a href='/en/immigration-refugees-citizenship/corporate/mandate/policies-operational-instructions-agreements/ministerial-instructions/express-entry-rounds/invitations.html?q=227'>Invitations to apply for permanent residence under the Express Entry system #227</a> <a href='/en/immigration-refugees-citizenship/corporate/mandate/policies-operational-instructions-agreements/ministerial-instructions/express-entry-rounds/invitations.html?q=227'>See full text of Ministerial Instruction</a> Federal Skilled Worker, Canadian Experience Class, Federal Skilled Trades and Provincial Nominee Program July 20, 2022 at 16:32:49 UTC December 30, 2021 at 15:29:35 UTC July 18, 2022 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 3 226 <a href='/en/immigration-refugees-citizenship/corporate/mandate/policies-operational-instructions-agreements/ministerial-instructions/express-entry-rounds/invitations.html?q=226'>226</a> 2022-07-06 July 6, 2022 No Program Specified 1,500 557 <a href='/en/immigration-refugees-citizenship/corporate/mandate/policies-operational-instructions-agreements/ministerial-instructions/express-entry-rounds/invitations.html?q=226'>Invitations to apply for permanent residence under the Express Entry system #226</a> <a href='/en/immigration-refugees-citizenship/corporate/mandate/policies-operational-instructions-agreements/ministerial-instructions/express-entry-rounds/invitations.html?q=226'>See full text of Ministerial Instruction</a> Federal Skilled Worker, Canadian Experience Class, Federal Skilled Trades and Provincial Nominee Program July 6, 2022 at 14:34:34 UTC November 13, 2021 at 02:20:46 UTC July 11, 2022 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 4 225 <a href='/en/immigration-refugees-citizenship/corporate/mandate/policies-operational-instructions-agreements/ministerial-instructions/express-entry-rounds/invitations.html?q=225'>225</a> 2022-06-22 June 22, 2022 Provincial Nominee Program 636 752 <a href='/en/immigration-refugees-citizenship/corporate/mandate/policies-operational-instructions-agreements/ministerial-instructions/express-entry-rounds/invitations.html?q=225'>Invitations to apply for permanent residence under the Express Entry system #225</a> <a href='/en/immigration-refugees-citizenship/corporate/mandate/policies-operational-instructions-agreements/ministerial-instructions/express-entry-rounds/invitations.html?q=225'>See full text of Ministerial Instruction</a> Provincial Nominee Program June 22, 2022 at 14:13:57 UTC April 19, 2022 at 13:45:45 UTC June 20, 2022 664 8,017 55,917 4,246 7,845 16,969 15,123 11,734 53,094 10,951 11,621 8,800 10,325 11,397 64,478 33,585 4,919 220,674 ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... 225 5 <a href='/en/immigration-refugees-citizenship/corporate/mandate/policies-operational-instructions-agreements/ministerial-instructions/express-entry-rounds/invitations-5.html'>5</a> 2015-03-20 March 20, 2015 No Program Specified 1,620 481 <a href='/en/immigration-refugees-citizenship/corporate/mandate/policies-operational-instructions-agreements/ministerial-instructions/express-entry-rounds/invitations-5.html'>Invitations to apply for permanent residence under the Express Entry system #5</a> <a href='/en/immigration-refugees-citizenship/corporate/mandate/policies-operational-instructions-agreements/ministerial-instructions/express-entry-rounds/invitations-5.html'>See full text of Ministerial Instruction</a> Federal Skilled Worker, Canadian Experience Class, Federal Skilled Trades and Provincial Nominee Program March 20, 2015 at 23:59:53 UTC March 20, 2015 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 226 4 <a href='/en/immigration-refugees-citizenship/corporate/mandate/policies-operational-instructions-agreements/ministerial-instructions/express-entry-rounds/invitations-4.html'>4</a> 2015-02-27 February 27, 2015 No Program Specified 1,187 735 <a href='/en/immigration-refugees-citizenship/corporate/mandate/policies-operational-instructions-agreements/ministerial-instructions/express-entry-rounds/invitations-4.html'>Invitations to apply for permanent residence under the Express Entry system #4</a> <a href='/en/immigration-refugees-citizenship/corporate/mandate/policies-operational-instructions-agreements/ministerial-instructions/express-entry-rounds/invitations-4.html'>See full text of Ministerial Instruction</a> Federal Skilled Worker, Canadian Experience Class, Federal Skilled Trades and Provincial Nominee Program February 27, 2015 at 23:59:51 UTC February 27, 2015 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 227 3 <a href='/en/immigration-refugees-citizenship/corporate/mandate/policies-operational-instructions-agreements/ministerial-instructions/express-entry-rounds/invitations-3.html'>3</a> 2015-02-20 February 20, 2015 Canadian Experience Class 849 808 <a href='/en/immigration-refugees-citizenship/corporate/mandate/policies-operational-instructions-agreements/ministerial-instructions/express-entry-rounds/invitations-3.html'>Invitations to apply for permanent residence under the Express Entry system #3</a> <a href='/en/immigration-refugees-citizenship/corporate/mandate/policies-operational-instructions-agreements/ministerial-instructions/express-entry-rounds/invitations-3.html'>See full text of Ministerial Instruction</a> Canadian Experience Class February 20, 2015 at 11:59:47 UTC February 20, 2015 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 228 2 <a href='/en/immigration-refugees-citizenship/corporate/mandate/policies-operational-instructions-agreements/ministerial-instructions/express-entry-rounds/invitations-2.html'>2</a> 2015-02-07 February 7, 2015 No Program Specified 779 818 <a href='/en/immigration-refugees-citizenship/corporate/mandate/policies-operational-instructions-agreements/ministerial-instructions/express-entry-rounds/invitations-2.html'>Invitations to apply for permanent residence under the Express Entry system #2</a> <a href='/en/immigration-refugees-citizenship/corporate/mandate/policies-operational-instructions-agreements/ministerial-instructions/express-entry-rounds/invitations-2.html'>See full text of Ministerial Instruction</a> Federal Skilled Worker, Canadian Experience Class, Federal Skilled Trades and Provincial Nominee Program February 7, 2015 at 11:59:59 UTC February 7, 2015 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 229 1 <a href='/en/immigration-refugees-citizenship/corporate/mandate/policies-operational-instructions-agreements/ministerial-instructions/express-entry-rounds/invitations-1.html'>1</a> 2015-01-31 January 31, 2015 No Program Specified 779 886 <a href='/en/immigration-refugees-citizenship/corporate/mandate/policies-operational-instructions-agreements/ministerial-instructions/express-entry-rounds/invitations-1.html'>Invitations to apply for permanent residence under the Express Entry system #1</a> <a href='/en/immigration-refugees-citizenship/corporate/mandate/policies-operational-instructions-agreements/ministerial-instructions/express-entry-rounds/invitations-1.html'>See full text of Ministerial Instruction</a> Federal Skilled Worker, Canadian Experience Class, Federal Skilled Trades and Provincial Nominee Program January 31, 2015 at 11:59:48 UTC January 31, 2015 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 230 rows × 31 columns