devLap

Extract GitHub Repository Data Without Hitting Rate Limits

python dev.to

GitHub's REST API is powerful but has aggressive rate limits: 60 requests per hour without a token, 5,000 with one. If you're doing any serious data extraction — searching repos, pulling contributor lists, exporting stargazers — you'll hit those limits fast.

I built a scraper that handles this properly. Here's what I learned and how you can extract GitHub data at scale without getting blocked.

The Rate Limit Problem

GitHub's API returns a 429 Too Many Requests once you exceed your quota. The response headers tell you exactly when your limit resets:

X-RateLimit-Limit: 60
X-RateLimit-Remaining: 0
X-RateLimit-Reset: 1719324000
Enter fullscreen mode Exit fullscreen mode

Without a personal access token, 60 requests per hour is nothing. A single search query + paginating through results + fetching user profiles can burn through that in minutes.

With a token (free to generate at github.com/settings/tokens), you get 5,000/hr — much more workable, but still requires careful request management.

How the Scraper Handles It

The GitHub Scraper I built handles rate limits with:

  1. Retry with backoff — When hitting 429, it reads the X-RateLimit-Reset header and waits until the limit resets
  2. Request budgeting — Tracks remaining requests and slows down before hitting zero
  3. Pagination via Link headers — GitHub uses Link: <url>; rel="next" headers, not page numbers 
# Simplified version of the retry logic
async def request_with_retry(url, headers):
    resp = await client.get(url, headers=headers)
    if resp.status_code == 429:
        reset_time = int(resp.headers.get("X-RateLimit-Reset", 0))
        wait = max(reset_time - time.time(), 1)
        await asyncio.sleep(wait)
        return await client.get(url, headers=headers)
    return resp
Enter fullscreen mode Exit fullscreen mode

What You Can Extract

The scraper supports 6 modes:

Search repos — Find repos by keyword, language, and star count:

{"scrapeType":"search_repos","searchQuery":"web scraping language:python stars:>100","maxItems":50}
Enter fullscreen mode Exit fullscreen mode

Scrape profiles — Get user details including email, company, location:

{"scrapeType":"profiles","usernames":["torvalds","gvanrossum"],"enrichProfiles":true}
Enter fullscreen mode Exit fullscreen mode

Export contributors — Who's building a project:

{"scrapeType":"contributors","repos":["scrapy/scrapy","microsoft/playwright"]}
Enter fullscreen mode Exit fullscreen mode

Stargazer export — Everyone who starred a repo (great for developer lead gen):

{"scrapeType":"stargazers","repos":["fastapi/fastapi"]}
Enter fullscreen mode Exit fullscreen mode

Example: Finding Top Python Scraping Libraries

Here's a real search result:

{"full_name":"nicoleahmed/Scrapling","stars":66129,"forks":2840,"language":"Python","topics":["scraping","web-scraping","python"],"license":"BSD-3-Clause","description":"Undetected, lightweight, and adaptive web scraping...","url":"https://github.com/nicoleahmed/Scrapling"}
Enter fullscreen mode Exit fullscreen mode

You get stars, forks, language, topics, license, and description — all the metadata you need for research or comparison.

Profile Enrichment

The enrichProfiles option fetches full user details for every username. This is especially useful for contributor and stargazer exports:

{"username":"torvalds","name":"Linus Torvalds","email":"torvalds@linux-foundation.org","company":"Linux Foundation","location":"Portland, OR","followers":228000,"public_repos":16,"bio":""}
Enter fullscreen mode Exit fullscreen mode

Note: email and company are only available if the user has made them public on their profile.

Token or No Token?

The scraper works both ways:

  • Without token: 60 req/hr, good for small extractions (< 50 items)
  • With token: 5,000 req/hr, needed for bulk exports. The token input field is marked as secret so it's never logged or exposed

Generate a token at github.com/settings/tokens — you only need the public_repo scope for reading public data.

Try It

Run it on Apify: github-scraper

The default input searches for "web scraping language:python stars:>100" and returns the top results — you can see the output format immediately.

Or call it via API:

curl -X POST "https://api.apify.com/v2/acts/ambitious_door~github-scraper/runs" \
  -H "Authorization: Bearer YOUR_APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"scrapeType": "search_repos", "searchQuery": "machine learning stars:>1000", "maxItems": 20}'
Enter fullscreen mode Exit fullscreen mode

Built with Python and httpx. Source on GitHub.

Source: dev.to

arrow_back Back to Tutorials