Cannot extract span element with BeautifulSoup

See below. I am using BeautifulSoup to try and extract this value. What I’ve tried:

pg = requests.get(websitelink)
soup = BeautifulSoup(pg.content, 'html.parser'
value = soup.find('span',{'class':'wall-header__item_count'}).text

I’ve tried find, and find all and it returns a Nonetype. For whatever reason the wall-header item count is unable to be found with these methods, even though it appears in the HTML. How can I get this value? Thanks!

enter image description here

Answer

I’m assuming you want to get the number of total items. The number is stored within the HTML page inside the <script>. beautifulsoup doesn’t see it, but you can use re/json modules to extract it:

import re
import json
import requests

url = "https://www.nike.com/w"
html_doc = requests.get(url).text

data = re.search(r"window.INITIAL_REDUX_STATE=({.*})", html_doc).group(1)
data = json.loads(data)

# uncomment this to print all data;
# print(json.dumps(data, indent=4))

print("Total items:", data["Wall"]["pageData"]["totalResources"])

Prints (in case in my country):

Total items: 5600