I would like to use asyncio to get webpage html.
I run the following code in jupyter notebook:
import aiofiles import aiohttp from aiohttp import ClientSession async def get_info(url, session): resp = await session.request(method="GET", url=url) resp.raise_for_status() html = await resp.text(encoding='GB18030') with open('test_asyncio.html', 'w', encoding='utf-8-sig') as f: f.write(html) return html async def main(urls): async with ClientSession() as session: tasks = [get_info(url, session) for url in urls] return await asyncio.gather(*tasks) if __name__ == "__main__": url = ['http://huanyuntianxiazh.fang.com/house/1010123799/housedetail.htm', 'http://zhaoshangyonghefu010.fang.com/house/1010126863/housedetail.htm'] result = asyncio.run(main(url))
However, it returns
RuntimeError: asyncio.run() cannot be called from a running event loop
What is the problem?
How to solve it?
asyncio.run() documentation says:
This function cannot be called when another asyncio event loop is running in the same thread.
The problem in your case is that jupyter (IPython) is already running an event loop (for IPython ≥ 7.0):
You can now use async/await at the top level in the IPython terminal and in the notebook, it should — in most of the cases — “just work”. Update IPython to version 7+, IPykernel to version 5+, and you’re off to the races.
That’s the reason why you don’t need to start the event loop yourself in jupyter and you can directly call
await main(url) even outside asynchronous functions.
async def main(): print(1) await main()
In plain Python (≥3.7)
import asyncio async def main(): print(1) asyncio.run(main())
In your code that would give:
url = ['url1', 'url2'] result = await main(url) for text in result: pass # text contains your html (text) response