Python Web Scraping Error without Any Warning Code Answer

Hello Developer, Hope you guys are doing great. Today at Tutorial Guruji Official website, we are sharing the answer of Python Web Scraping Error without Any Warning without wasting too much if your time.

The question is published on by Tutorial Guruji team.

I am trying to scrape some text from a webpage and saving them in a text file using following code (I am opening links from a text file called links.txt):

import requests
import csv
import random
import string
import re

from bs4 import BeautifulSoup

#Create random string of specific length
def randStr(chars = string.ascii_uppercase + string.digits, N=10):
    return ''.join(random.choice(chars) for _ in range(N))
    
with open("links.txt", "r") as a_file:
  for line in a_file:
    stripped_line = line.strip()
    endpoint = stripped_line
    response = requests.get(endpoint)
    data = response.text
    soup = BeautifulSoup(data, "html.parser")
    for pictags in soup.find_all('col-md-2'):
        lastfilename = randStr()
        file = open(lastfilename + ".txt", "w")
        file.write(pictags.txt)
        file.close()
        print(stripped_line)

the webpage has following attribute:

<div class="col-md-2">

Webpage: http://www.mykeyworder.com/keywords?tags=dog&exclude=&language=en

The problem is after running the code noting is happening and I am not receiving any error.

Answer

To get all keyword text from the page into a file, you can do:

import requests
from bs4 import BeautifulSoup

url = "http://www.mykeyworder.com/keywords?tags=dog&exclude=&language=en"

soup = BeautifulSoup(requests.get(url).content, "html.parser")

with open("data.txt", "w") as f_out:
    for inp in soup.select('input[type="checkbox"]'):
        print(inp["value"], file=f_out)

This creates data.txt with content:

dog
animal
canine
pet
cute
puppy
happy
young
adorable

...and so on.
We are here to answer your question about Python Web Scraping Error without Any Warning - If you find the proper solution, please don't forgot to share this with your team members.

Related Posts

Tutorial Guruji