Getting an element via LXML by XPATH

I am coding a Discord bot in a library for python, discord.py.

I don’t need help with that but with scraping some info from the site.

    @commands.command(aliases=["rubyuserinfo"])
    async def rubyinfo(self, ctx, input):
        HEADERS = {
            'User-Agent' : 'Magic Browser'
        }

        url = f'https://rubyrealms.com/user/{input}/'

        async with aiohttp.request("GET", url, headers=HEADERS) as response:
            if response.status == 200:
                print("Site is working!")
                content = await response.text()
                soup = BeautifulSoup(content, "html.parser")
                page = requests.get(url)
                tree = html.fromstring(page.content)
                stuff = tree.xpath('/html/body/div[4]/div/div[3]/div[3]/div/div[2]/div[1]/div[2]/div/p')
                print(stuff)
            else:
                print(f"The request was invalidnStatus code: {response.status}")

The website I am looking for is “https://rubyrealms.com/user/{input}/” where input is given while running h!rubyinfo USERNAME changing the link to https://rubyrealms.com/user/username/.

On the website what I want to get is their BIO which has an XPATH of

"//*[@id="content-wrap"]/div[3]/div[3]/div/div[2]/div[1]/div[2]/div/p"

where the element is:

<p class="margin-none font-color">
Hey! My name is KOMKO190, you maybe know me from the forums or discord. I am a programmer, I know a bit of JavaScript, small portion of C++, Python and html/css. Mostly python. My user ID is 7364. ||| 5th owner of Space Helmet :) </p>

Any help on how I would scrape that? The only response my bot gives is “[]”

Answer

Change your XPath expression for a relative one :

from lxml import html
import requests
page = requests.get('https://www.rubyrealms.com/user/KOMKO190/')
tree = html.fromstring(page.content)
stuff = tree.xpath('normalize-space(//h3[.="Bio"]/following-sibling::p/text())')
print (stuff)

Output :

Hey! My name is KOMKO190, you maybe know me from the forums or discord. I am a programmer, I know a bit of JavaScript, small portion of C++, Python and html/css. Mostly python. My user ID is 7364. ||| 5th owner of Space Helmet :)

Leave a Reply

Your email address will not be published. Required fields are marked *