member badge
Illia Zub
Currently, the way to paginate searches is to get the
serpapi_pagination.current
and increase the
offset
or
start
parameters in the loop. Like with regular HTTP requests to
serpapi.com/search
without an API wrapper.
import os
from serpapi import GoogleSearch
params = {
"engine": "google",
"q": "coffee",
"tbm": "nws",
"api_key": os.getenv("API_KEY"),
}
search = GoogleSearch(params)
results = search.get_dict()
print(f"Current page: {results['serpapi_pagination']['current']}")
for news_result in results["news_results"]:
print(f"Title: {news_result['title']}\nLink: {news_result['link']}\n")
while 'next' in results['serpapi_pagination']:
search.params_dict[
"start"] = results['serpapi_pagination']['current'] * 10
results = search.get_dict()
print(f"Current page: {results['serpapi_pagination']['current']}")
for news_result in results["news_results"]:
print(
f"Title: {news_result['title']}\nLink: {news_result['link']}\n"
)
A more convenient way for an official API wrapper would be to provide some function like
search.paginate(callback: Callable)
which will properly calculate offset for the specific search engine and loop through pages until the end.
import os
from serpapi import GoogleSearch
def print_results(results):
print(f"Current page: {results['serpapi_pagination']['current']}")
for news_result in results["news_results"]:
print(f"Title: {news_result['title']}\nLink: {news_result['link']}\n")
params = {
"engine": "google",
"q": "coffee",
"tbm": "nws",
"api_key": os.getenv("API_KEY"),
}
search = GoogleSearch(params)
search.paginate(print_results)
What do you think?
----
Related GitHub thread: https://github.com/serpapi/google-search-results-python/issues/19
April 14, 2021