Video description scrapping with Beautifulsoup

I’m trying to scrape a link in the video description on youtube, but the list always return empty.

I’ve tried to change the tag from where I’m scraping, but there is no change in either the output nor the error message.

Here’s the code I’m using:

from bs4 import BeautifulSoup
import requests

source = requests.get('https://www.youtube.com/watch?v=gqUqGaXipe8').text

soup = BeautifulSoup(source, 'lxml')

link = [i['href'] for i in soup.findAll('a', class_='yt-simple-endpoint style-scope yt-formatted-string', href=True)]

print(link)

What is wrong, and how can I solve it?

Answer

In your case, requests doesn’t return the whole HTML structure of the page. If Youtube is filling in the data using JavaScript we must run it through a real browser to get the source of the page, such as Chrome Headless using the Selenium library. Here is the general solution:

from bs4 import BeautifulSoup
from selenium import webdriver
import time

options = webdriver.ChromeOptions()
options.add_argument('--headless')
driver = webdriver.Chrome(options = options)
url = "https://www.youtube.com/watch?v=Oh1nqnZAKxw"
driver.get(url)
time.sleep(2)
soup = BeautifulSoup(driver.page_source, 'html.parser')
driver.quit()


link = [i['href'] for i in soup.select('div#meta div#description [href]')]
print(link)