Scraping with Selenium. Error: “element is not attached to the page document”

I’m scraping the link “https://www.kayak.it/flights/MIL-BCN/2021-09-01/2021-09-02/?sort=bestflight_a” with Selenium in Python.

When I run the code I receive this error and I can’t understand why (the code worked a couple weeks ago!): “Message: stale element reference: element is not attached to the page document (Session info: headless chrome=92.0.4515.159)”

Following my code:

wd = webdriver.Chrome('chromedriver',chrome_options=chrome_options)
link = 'https://www.kayak.it/flights/MIL-BCN/2021-09-01/2021-09-02/?sort=bestflight_a'
wd.get(link) 

import pprint 
detail_flights = [] 
for flight in list_flights:
    flights = ""

    if(len(flight.find_elements_by_css_selector(".mainInfo")) > 0):
       flights = flight.find_elements_by_css_selector(".mainInfo")[0].text 

    detail_flights.append({'flights': flights
                           })

len(detail_flights)
pprint.pprint(detail_flights[0:4]) 

Answer

I think they have introduce dynamic accept cookies, for few run we have button[title='Accetta'] and for we run we got this button[id='onetrust-accept-btn-handler'], there are two try blocks to handle this, also I have made some modification to get rid off stale element. Baiscally I am defining the list again in for loop to get rid off of stale element reference.

Sample code to handle and scrape :

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

wd = webdriver.Chrome('chromedriver',chrome_options = chrome_options)
wd.maximize_window()
wd.implicitly_wait(50)
#driver.get("https://account.battle.net/creation/flow/creation-full")
wait = WebDriverWait(wd, 20)

link = 'https://www.kayak.it/flights/MIL-BCN/2021-09-01/2021-09-02/?sort=bestflight_a'
wd.get(link)
try:
    wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button[title='Accetta']"))).click()
except:
    pass

try:
    wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button[id='onetrust-accept-btn-handler']"))).click()
except:
    pass

detail_flights = []
j = 0
lngth = len(wd.find_elements_by_css_selector(".mainInfo"))
for i in range(lngth):
    try:
        if len(wd.find_elements_by_css_selector(".mainInfo")) > 0:
            elements = wd.find_elements_by_css_selector(".mainInfo")
            wd.execute_script("arguments[0].scrollIntoView(true);", elements[j])
            print(elements[j].get_attribute('innerText'))
            j = j + 1
            detail_flights.append(elements[j].get_attribute('innerText'))
        else:
            print('Nothing more to scrape')
    except:
        pass

Output :

BGY Bergamo Orio al Serio
‐
BCN Barcellona-El Prat
diretto
1h 35m
6:20 – 8:00
BCN Barcellona-El Prat
‐
BGY Bergamo Orio al Serio
diretto
1h 40m
22:20 – 23:55
BGY Bergamo Orio al Serio
‐
BCN Barcellona-El Prat
diretto
1h 35m
6:20 – 8:00
BCN Barcellona-El Prat
‐
BGY Bergamo Orio al Serio
diretto
1h 40m
16:35 – 18:05
LIN Aeroporto Milano Linate
‐
BCN Barcellona-El Prat
diretto
1h 30m
6:20 – 8:00
BCN Barcellona-El Prat
‐
BGY Bergamo Orio al Serio
diretto
1h 40m
12:35 – 14:05
MXP Aeroporto Malpensa
‐
BCN Barcellona-El Prat
diretto
1h 30m
6:20 – 8:00
BCN Barcellona-El Prat
‐
BGY Bergamo Orio al Serio
diretto
1h 40m
22:20 – 23:55
BGY Bergamo Orio al Serio
‐
BCN Barcellona-El Prat
diretto
1h 35m
11:35 – 13:05
BCN Barcellona-El Prat
‐
MXP Aeroporto Malpensa
diretto
1h 30m
22:20 – 23:55
BGY Bergamo Orio al Serio
‐
BCN Barcellona-El Prat
diretto
1h 35m
12:05 – 13:45
BCN Barcellona-El Prat
‐
BGY Bergamo Orio al Serio
diretto
1h 40m
16:35 – 18:05
LIN Aeroporto Milano Linate
‐
BCN Barcellona-El Prat
diretto
1h 30m
11:35 – 13:05
BCN Barcellona-El Prat
‐
MXP Aeroporto Malpensa
diretto
1h 30m
16:35 – 18:05
LIN Aeroporto Milano Linate
‐
BCN Barcellona-El Prat
diretto
1h 30m
12:05 – 13:45
BCN Barcellona-El Prat
‐
BGY Bergamo Orio al Serio
diretto
1h 40m
14:15 – 15:50
BGY Bergamo Orio al Serio
‐
BCN Barcellona-El Prat
diretto
1h 35m
6:20 – 8:00
BCN Barcellona-El Prat
‐
BGY Bergamo Orio al Serio
diretto
1h 40m
22:20 – 23:55
BGY Bergamo Orio al Serio
‐
BCN Barcellona-El Prat
diretto
1h 35m
20:15 – 21:55
BCN Barcellona-El Prat
‐
BGY Bergamo Orio al Serio
diretto
1h 40m
14:45 – 16:25
MXP Aeroporto Malpensa
‐
BCN Barcellona-El Prat
diretto
1h 40m
6:20 – 8:00
BCN Barcellona-El Prat
‐
BGY Bergamo Orio al Serio
diretto
1h 40m
10:05 – 11:45
MXP Aeroporto Malpensa
‐
BCN Barcellona-El Prat
diretto
1h 40m
6:20 – 8:00
BCN Barcellona-El Prat
‐
BGY Bergamo Orio al Serio
diretto
1h 40m
13:00 – 14:40
MXP Aeroporto Malpensa
‐
BCN Barcellona-El Prat
diretto
1h 40m
6:20 – 8:00
BCN Barcellona-El Prat
‐
BGY Bergamo Orio al Serio
diretto
1h 40m
16:35 – 18:05
LIN Aeroporto Milano Linate
‐
BCN Barcellona-El Prat
diretto
1h 30m
20:15 – 21:55
BCN Barcellona-El Prat
‐
BGY Bergamo Orio al Serio
diretto
1h 40m
22:20 – 23:55
BGY Bergamo Orio al Serio
‐
BCN Barcellona-El Prat
diretto
1h 35m
19:05 – 20:45
BCN Barcellona-El Prat
‐
LIN Aeroporto Milano Linate
diretto
1h 40m
16:35 – 18:05
LIN Aeroporto Milano Linate
‐
BCN Barcellona-El Prat
diretto
1h 30m
19:05 – 20:45
BCN Barcellona-El Prat
‐
LIN Aeroporto Milano Linate
diretto
1h 40m

Process finished with exit code 0