I am trying to scrape some text from a webpage and saving them in a text file using following code (I am opening links from a text file called links.txt):
import requests import csv import random import string import re from bs4 import BeautifulSoup #Create random string of specific length def randStr(chars = string.ascii_uppercase + string.digits, N=10): return ''.join(random.choice(chars) for _ in range(N)) with open("links.txt", "r") as a_file: for line in a_file: stripped_line = line.strip() endpoint = stripped_line response = requests.get(endpoint) data = response.text soup = BeautifulSoup(data, "html.parser") for pictags in soup.find_all('col-md-2'): lastfilename = randStr() file = open(lastfilename + ".txt", "w") file.write(pictags.txt) file.close() print(stripped_line)
the webpage has following attribute:
The problem is after running the code noting is happening and I am not receiving any error.
To get all keyword text from the page into a file, you can do:
import requests from bs4 import BeautifulSoup url = "http://www.mykeyworder.com/keywords?tags=dog&exclude=&language=en" soup = BeautifulSoup(requests.get(url).content, "html.parser") with open("data.txt", "w") as f_out: for inp in soup.select('input[type="checkbox"]'): print(inp["value"], file=f_out)
data.txt with content:
dog animal canine pet cute puppy happy young adorable ...and so on.