Skip to content

Scraper jobs for Devfolio and Unstop or other platform #677

Open
Open
Scraper jobs for Devfolio and Unstop or other platform#677
Labels
backend

Description

@ShantKhatri
ShantKhatri
opened on Jul 12, 2026

Summary

Pull hackathon listings automatically via a cron job.

Contexts

Needs to respect rate limits from both platforms, called out explicitly in the spec.

Tasks

  • Build scraper for Devfolio listings
  • Build scraper for Unstop listings
  • API fetch from the developer.events API
  • Add rate limiting / backoff to avoid getting blocked
  • Schedule as a recurring cron job

Acceptance Criteria

  • Both scrapers run on schedule without hitting rate limits
  • Scraped entries land correctly in the hackathon table
  • Tests added
  • docs updated

Area

backend

Difficulty

Advanced

Activity

  1. ShantKhatri
    added
    backend
    on Jul 12, 2026
  2. github-project-automation
    added this to DevCardon Jul 12, 2026
  3. github-project-automation
    moved this to Todo in DevCardon Jul 12, 2026

  4. github-actions commented on Jul 12, 2026

    @github-actions
    github-actionsbot
    on Jul 12, 2026 – with GitHub Actions
    Contributor

    Hi @ShantKhatri,

    Thanks for opening this issue.

Please reply with one of the following areas so the issue can be routed to the appropriate team member:

- /backend
- /web
- /mobile
- /devops

Once an area is selected, the corresponding label will be added automatically.
  5. ShantKhatri
    added a parent issue on Jul 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    backend

    Type

    No type

    Projects

    • DevCard
      Status
      Todo

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions