Websites with infinite scroll typically load additional content via JavaScript as you scroll, so a simple requests.get() usually won't retrieve everything. The most reliable approach is to automate a real browser using a tool like Playwright or Selenium.
Before scraping any website, make sure you're complying with its terms of service and robots.txt, and avoid sending requests at a rate that could disrupt the site.
Option 1: Playwright (recommended)
Install Playwright:
pip install playwright
playwright install
Example:
from playwright.sync_api import sync_playwright
import time
URL = "https://example.com"
with sync_playwright() as p:
browser = p.chromium.launch(headless=True)
page = browser.new_page()
page.goto(URL)
# Scroll until page stops growing
previous_height = 0
while True:
current_height = page.evaluate("document.body.scrollHeight")
if current_height == previous_height:
break
page.evaluate("window.scrollTo(0, document.body.scrollHeight)")
time.sleep(2)
previous_height = current_height
# Extract text
articles = page.locator("article")
for article in articles.all():
print(article.inner_text())
browser.close()
Option 2: Selenium
pip install selenium
Example:
from selenium import webdriver
from selenium.webdriver.common.by import By
import time
driver = webdriver.Chrome()
driver.get("https://example.com")
last_height = driver.execute_script("return document.body.scrollHeight")
while True:
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
time.sleep(2)
new_height = driver.execute_script("return document.body.scrollHeight")
if new_height == last_height:
break
last_height = new_height
elements = driver.find_elements(By.CSS_SELECTOR, "article")
for element in elements:
print(element.text)
driver.quit()
Better approach: Call the site's API directly
Many infinite-scroll sites don't actually load HTML—they request JSON behind the scenes.
For example:
Browser
↓
HTML page
↓
Scroll
↓
GET /api/posts?page=2
↓
JSON response
You can often discover these endpoints by:
- Opening Developer Tools (F12).
- Going to the Network tab.
- Filtering by Fetch/XHR.
- Scrolling the page.
- Looking for requests returning JSON.
Once you identify the endpoint, you can use requests:
import requests
url = "https://example.com/api/posts?page=1"
response = requests.get(url)
data = response.json()
for item in data["results"]:
print(item["title"])
This is usually much faster and more reliable than browser automation.
Handling "Load More" buttons
If the page uses a button instead of automatic scrolling:
while True:
try:
button = page.locator("button:has-text('Load More')")
button.click()
page.wait_for_timeout(1500)
except:
break
Avoid duplicate content
As new content loads, you'll often encounter duplicates. Track unique identifiers or URLs:
seen = set()
for article in articles:
text = article.inner_text()
if text not in seen:
seen.add(text)
print(text)
Which tool should you use?
| Situation | Best choice |
|---|
| Static HTML page | requests + BeautifulSoup |
| JavaScript/infinite scroll | Playwright |
| Infinite scroll backed by an API | requests (to the API) |
| Complex login or user interaction | Playwright |
If you share the URL of the website you're trying to scrape (or describe how it loads content), I can help determine whether browser automation is necessary or whether there's a simpler API-based approach.
