How to get data past the “Show More” button that DOESN’T change the URL?

I am trying to scrape article titles and links from Vogue with a site search keyword. I can’t get the top 100 results because the “Show More” button obscures them. I’ve gotten around this before by using the changing URL, but Vogue’s URL does not change to include the page number, result number, etc.

import requests
from bs4 import BeautifulSoup as bs

url = ''
r = requests.get(url)
soup = bs(r.content, 'html')

links = soup.find_all('a', {'class':"summary-item-tracking__hed-link summary-item__hed-link"})
titles = soup.find_all('h2', {'class':"summary-item__hed"})

res = []
for i in range(len(titles)):
    entry = {'Title': titles[i].text.strip(), 'Link': ''+links[i]['href'].strip()}

Any tips on how to scrape the data past the “Show More” button?


You have to examine the Network from developer tools. Then you have to determine how to website requests the data. You can see the request and the response in the screenshot.


The website is using page parameter as you see.


Each page has 8 titles. So you have to use the loop to get 100 titles.


import cloudscraper,json,html
for i in range(1,14):
    url = f'{i}&sort=score%20desc&format=json'
    scraper = cloudscraper.create_scraper(browser={'browser': 'firefox','platform': 'windows','mobile': False},delay=10)
    byte_data = scraper.get(url).content
    json_data = json.loads(byte_data)
    for j in range(0,8):
        title_url = '' + (html.unescape(json_data['search']['items'][j]['url']))
        t = html.unescape(json_data['search']['items'][j]['source']['hed'])
        print(counter," - " + t + ' - ' + title_url)
        if (counter == 100):
        counter = counter + 1