Webscraping a Javascript Element, preferably without Selenium

Currently, I have used Selenium to extract text from a table on a website. Following is the code:

from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager

# Using Chrome to access web
browser = webdriver.Chrome(ChromeDriverManager().install())

# Open the website
element = browser.find_element_by_id('searchButton').click()

table_text = browser.find_element_by_class_name('table').text

while len(table_text) < 80:
    table_text = browser.find_element_by_class_name('table').text



However, I am trying to find a way to do the same with Requests/Beautiful soup or any other library where I can schedule this as a task in windows and store the result in a table at every x interval. Obviously, since I want all this to happen in the background and then trigger a notification etc.

What I Want is- Open this website, click on the search button (or trigger the corresponding javascript), and then export the table as a Dataframe or whatever.

Can you please guide me here?

thanks in advance!!


If you go to Network Tab you will get the API . You can use this post request to get all the value.Using max result field you can limit the results as well.

enter image description here


import requests
import  pandas as pd
"searchString" : "",
"searchQualificationsAndDesigns": True,
"searchDeclarationOnly": True,
"bqaApprovalStatusId" : -1,
"bqaLockStatusId" : -1,
"layers" : [],
"listingDateEarliest" : "",
"listingDateLatest" : "",
"maxResults": 5000,
"memberId": "",
"productTypeId" : 0,
"searchDeclarationOnly" : True,
"searchEndProductList" : False,
"searchMyCompany" : False,
"searchPRDProductList" : True,
"searchQualificationsAndDesigns" : True,
"searchString" : "",
"specName": 0,
"userId" : 0

headers = {'User-Agent':
       'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36'}


You can import to csv file.


enter image description here