Webscraping to add product name and corresponding prices to pandas dataframe

I am practicing webscraping and wanted to extract product names and prices to a pandas data frame.

Here is my code”

for web in website:
  r=re.get(web)
  soup = BeautifulSoup(r.content, 'html.parser')
  for i in soup.find_all("div", {"class":"row productspm"}): 
    for name in soup.find_all("h4"):
      if name.text not in productname:
        productname.append(name.text)
    for price in soup.find_all("p",{'class':"price text-right"}):
      prices.append(price.text)


print(len(productname))

When I am extracting the data. I am not getting any errors but the dataframe contains all the wrong information.

First, instead of extracting 43 products it is extracting 61 product names. Second, the prices of the products do not match the price displayed on the websites. When a product is on sale they are using a different html code and this is creating problem in scraping.

Here is the HTML code on the website for the product that is not on sale:

<div class="product-layout product-grid col-lg-3 col-md-4 col-sm-6 col-xs-12">
          <div class="product-thumb transition">
      <div class="image"><a href="---"><img src="--" alt="BREATHING BAG 3L N-LATEX PARKER" title="BREATHING BAG 3L N-LATEX PARKER" class="img-responsive center-block"></a>
          <!-- Webiarch Images Start -->
                                   
          <!-- End -->
                              <div class="topbutton">
        <button type="button" data-toggle="tooltip" title="" onclick="wishlist.add('250');" data-original-title="Add to Wish List"><svg width="20px" height="20px"><use xlink:href="#wishlist"></use></svg><span class="hidden-xs"></span></button>
        <button type="button" data-toggle="tooltip" title="" onclick="compare.add('250');" class="wishcom" data-original-title="Compare this Product"><svg width="20px" height="20px"><use xlink:href="#pcom"></use></svg><span class="hidden-xs"></span></button>
         <div class="bquickv" title="" data-toggle="tooltip" data-original-title="quickview"><div class="webi-ownstyle webi-quickview"><a href="#"><svg width="20px" height="20px"><use xlink:href="#pquick"></use></svg></a></div></div>
      </div>
      </div>
      <div class="caption">
        <h4><a href="---">BREATHING BAG 3L N-LATEX PARKER</a></h4>
        <p class="list-des">BREATHING
  BAG 3L N-LATEX PARKER..</p>
                  <div class="rating pull-left">          <span class="fa fa-stack"><i class="fa fa-star-o fa-stack-2x"></i></span>
                    <span class="fa fa-stack"><i class="fa fa-star-o fa-stack-2x"></i></span>
                    <span class="fa fa-stack"><i class="fa fa-star-o fa-stack-2x"></i></span>
                    <span class="fa fa-stack"><i class="fa fa-star-o fa-stack-2x"></i></span>
                    <span class="fa fa-stack"><i class="fa fa-star-o fa-stack-2x"></i></span>
          </div>
                

                
                    <p class="price text-right"> SAR 150</p>
                          <div class="clearfix"></div>
      <div class="button-group">
        <button type="button" onclick="cart.add('250');" class="acart">
          <span>Add to Cart</span>
        </button>
      </div>
      </div>
      
    </div>
        </div>

Here is for the products with sales on it.

<div class="product-layout product-grid col-lg-3 col-md-4 col-sm-6 col-xs-12">
          <div class="product-thumb transition">---" title="Everbrite In-Office Tooth Whitening Kit (3 Patients)" class="img-responsive center-block"></a>
          <!-- Webiarch Images Start -->
                                   
          <!-- End -->
                                 <span class="salep">sale</span>
                      <div class="topbutton">
        <button type="button" data-toggle="tooltip" title="" onclick="wishlist.add('189');" data-original-title="Add to Wish List"><svg width="20px" height="20px"><use xlink:href="#wishlist"></use></svg><span class="hidden-xs"></span></button>
        <button type="button" data-toggle="tooltip" title="" onclick="compare.add('189');" class="wishcom" data-original-title="Compare this Product"><svg width="20px" height="20px"><use xlink:href="#pcom"></use></svg><span class="hidden-xs"></span></button>
         <div class="bquickv" title="" data-toggle="tooltip" data-original-title="quickview"><div class="webi-ownstyle webi-quickview"><a href="#"><svg width="20px" height="20px"><use xlink:href="#pquick"></use></svg></a></div></div>
      </div>
      </div>
      <div class="caption">
        <h4><a href="---">Everbrite In-Office Tooth Whitening Kit (3 Patients)</a></h4>
        <p class="list-des">Everbrite In-Office Tooth Whitening Kit (3 Patients)
Used for Dentamerica Whitening System. One hour..</p>
                  <div class="rating pull-left">          <span class="fa fa-stack"><i class="fa fa-star-o fa-stack-2x"></i></span>
                    <span class="fa fa-stack"><i class="fa fa-star-o fa-stack-2x"></i></span>
                    <span class="fa fa-stack"><i class="fa fa-star-o fa-stack-2x"></i></span>
                    <span class="fa fa-stack"><i class="fa fa-star-o fa-stack-2x"></i></span>
                    <span class="fa fa-stack"><i class="fa fa-star-o fa-stack-2x"></i></span>
          </div>
                

                
                   <p class="pricedis price text-right"><span class="price-new"> SAR 275</span> <span class="price-old"> SAR 345</span></p>
                          <div class="clearfix"></div>
      <div class="button-group">
        <button type="button" onclick="cart.add('189');" class="acart">
          <span>Add to Cart</span>
        </button>
      </div>
      </div>
      
    </div>
        </div>

Can someone please let me know where I am making a mistake and how I can correct it. Thanks a lot

Here is my list of prices that I am getting:

prices
[' SAR 110', ' SAR 41', ' SAR 1,760', ' SAR 150', ' SAR 3,103', ' SAR 5,770', ' SAR 540', ' SAR 4,900', ' SAR 2,650', ' SAR 603', ' SAR 58', ' SAR 15', ' SAR 15', ' SAR 3,200', ' SAR 35', ' SAR 890', ' SAR 75', ' SAR 10,500', ' SAR 1,560', ' SAR 2,421', ' SAR 4,904', ' SAR 223', ' SAR 5,072', ' SAR 1,600', ' SAR 9,700', ' SAR 354', ' SAR 25,600', ' SAR 1,800', ' SAR 84', ' SAR 256', ' SAR 120', ' SAR 349', ' SAR 2,100', ' SAR 21,500', ' SAR 15', ' SAR 3,450']

Answer

It is very hard to answer and give a recommendation based on your input, so it would be really cool to improve your question.

What happens?

Problem of difference between name and price is the way you loop your response and append things to the lists. They are independent from each other.

How to fix that?

You should grab all the information in one step, like this:

data = []

for item in soup.select('div.row.productspm > div'):
    data.append({
        'name':item.h4.get_text(),
        'price': item.select_one('p.price').get_text('^^', strip=True).split('^^')[0]
    })

Cause it is not clear I grab only the regular price and the new price like this:

'price': item.select_one('p.price').get_text('^^', strip=True).split('^^')[0]

Example

import requests
from bs4 import BeautifulSoup
import pandas as pd


page = requests.get("https://alrazimed.me/index.php?route=product/category&path=178_115")
soup = BeautifulSoup(page.content, "html.parser")

data = []

for item in soup.select('div.row.productspm > div'):
    data.append({
        'name':item.h4.get_text(),
        'price': item.select_one('p.price').get_text('^^', strip=True).split('^^')[0]
    })

pd.DataFrame(data)

Output

    name                                                price
0   C-BRIGHT Teeth whitening accelerators               SAR 3,103
1   Everbrite At-Home Tooth Whitening Kit               SAR 120
2   Everbrite In-Office Tooth Whitening Kit (3 Pat...   SAR 275
3   Everbrite In-Office Tooth Whitening Kit (Single)    SAR 135
4   FLOCARE – 0.4% Stannous Fluoride                    SAR 35
5   LITEX 686 LED CURING AND WHITENING SYSTEM           SAR 10,500