How to Scrape Reddit Posts and Comments with Python (2026)

This tutorial shows how to scrape Reddit posts and comments with Python. Reddit is where people say what they actually think about a product, a job, a framework or a brand, usually in long threads with the replies nested five levels deep. That makes it one of the most useful sources for market research, support mining and brand monitoring, and one of the more annoying ones to collect programmatically.

This guide shows how to get Reddit data out as clean JSON with our Reddit scraper API: a subreddit's feed, a keyword search across all of Reddit, the complete comment tree under a post, and a user's profile and history. Every example is a plain HTTPS GET, so it works from any language. If you don't write code, skip to the end for the no-code route.

What Reddit data can you get from the API?

The Reddit section of the API has 14 endpoints, all under https://api.insightsocial.app/v1/reddit/. These are the ones this guide uses:

A range means the endpoint is metered: we hold the top of the range when the call starts and charge what the call actually read. The full list, including /subreddits/search, /search/media, /post/transcript and /omni-search, is on the Reddit platform page, and every price is on Endpoint pricing.

How do I get set up?

You need one thing: an API key. Sign up, open the API section of the portal, and create a key. Keys start with isk_live_ and go in the x-api-key header. There is no Reddit app to register and no OAuth flow to run.

export INSIGHTSOCIAL_API_KEY="isk_live_..."

Sanity-check it with the free balance endpoint:

curl "https://api.insightsocial.app/v1/credits" \
  -H "x-api-key: $INSIGHTSOCIAL_API_KEY"

The free plan gives you 500 credits a month, and your first 10 calls priced at 100 credits or less cost nothing at all. That covers the subreddit, post and comment examples below. Search is the exception: a metered call reserves its ceiling before it runs, and /search with max_pages=2 holds 600 credits, more than the free plan's 500. Run it with Pro or a credit pack, or drop max_pages for a single page. You're only charged what the call actually reads. The quickstart covers the same steps in more detail.

Every response uses the same envelope: success, schema_version ("2"), data, unavailable, credits_used, credits_remaining, and on list endpoints a pagination object. Rows live in data.items, and each item wraps either a post or a comment. Ids are strings and times are ISO 8601 in UTC.

How do I scrape a subreddit's posts?

Start with the community itself. /subreddit/details is a cheap way to confirm the name is right (it's case-sensitive here) and see how big the community is:

curl "https://api.insightsocial.app/v1/reddit/subreddit/details?subreddit=dataengineering" \
  -H "x-api-key: $INSIGHTSOCIAL_API_KEY"

Then pull the feed. /subreddit takes sort (best, hot, new, top, rising) and, with sort=top, a timeframe of day, week, month, year or all. This Python script walks three pages of the month's top posts in r/dataengineering and writes them to a CSV:

import csv
import os

import requests

KEY = os.environ["INSIGHTSOCIAL_API_KEY"]
URL = "https://api.insightsocial.app/v1/reddit/subreddit"

params = {"subreddit": "dataengineering", "sort": "top", "timeframe": "month"}
rows, spent = [], 0

for page in range(3):
    body = requests.get(URL, params=params, headers={"x-api-key": KEY}, timeout=60).json()
    if not body["success"]:
        print(body["error"]["type"], body["error"]["message"])
        break
    spent += body["credits_used"]
    for item in body["data"]["items"]:
        post = item["post"]
        ext = post["ext"] or {}
        eng = post["engagement"]
        rows.append({
            "title": ext.get("title"),
            "author": post["author"]["username"],
            "score": eng.get("likes"),
            "comments": eng.get("comments"),
            "flair": ext.get("flair"),
            "url": post["url"],
            "published_at": post["published_at"],
        })
    pagination = body.get("pagination") or {}
    if not pagination.get("has_more"):
        break
    params["cursor"] = pagination["next_cursor"]

with open("dataengineering_top_month.csv", "w", newline="") as f:
    writer = csv.DictWriter(f, fieldnames=rows[0].keys() if rows else ["title"])
    writer.writeheader()
    writer.writerows(rows)

print(f"{len(rows)} posts, {spent} credits")

Two details worth knowing. First, the pagination rule is the same on every list endpoint: send pagination.next_cursor back as cursor, and stop when has_more is false. Don't stop on an empty page or a row count. Cursors start with v2c. and are opaque, so pass them back untouched, with the same parameters, within 24 hours. Second, read unavailable on each page. It lists fields Reddit normally shows that this page could not fill on any row. On subreddit feeds it usually holds items[].post.author.id: Reddit post lists carry the author's username but not their account id, so key authors on author.username. Reddit has no public view count either, so engagement.views is always null. The pagination guide covers both.

How do I search Reddit by keyword?

/search looks across all of Reddit and returns post bodies at ext.selftext at no extra cost. It takes sort (relevance, new, top, comment_count; this is Reddit's own search ranking) and timeframe, plus two filters that save you post-processing: max_age_days keeps only recent rows, and max_pages (1 to 5) walks several pages in a single request.

Here it is in Node.js, looking for a week of discussion about a product category:

const KEY = process.env.INSIGHTSOCIAL_API_KEY;
const url = new URL("https://api.insightsocial.app/v1/reddit/search");
url.searchParams.set("query", "self-hosted analytics");
url.searchParams.set("sort", "new");
url.searchParams.set("max_age_days", "7");
url.searchParams.set("max_pages", "2");

const res = await fetch(url, { headers: { "x-api-key": KEY } });
const body = await res.json();

if (!body.success) {
  console.error(body.error.type, body.error.message);
  process.exit(1);
}

for (const { post } of body.data.items) {
  const sub = post.ext?.subreddit;
  const title = post.ext?.title ?? post.content.text?.slice(0, 80);
  console.log(`r/${sub}  ${post.engagement.likes ?? 0} pts  ${title}`);
  console.log(`  ${post.url}`);
}
console.log(`credits used: ${body.credits_used}`);

To search inside one community, you have two options. /subreddit/search does it directly, but that source sends no post body unless you add include_body, which costs extra. The cheaper habit is to call /search with query=subreddit:dataengineering your terms, which returns bodies for free.

When the phrase you care about lives in replies rather than titles, use /search/comments instead. It's a flat 10 credits and every hit carries its parent post inline (ext.post_title, ext.subreddit, ext.post_url), so you don't need a second call to know what thread a comment came from.

How do I scrape the comments on a Reddit post?

Give /post/comments a post URL and it returns the whole discussion as a tree for 50 credits: each comment has text, author, engagement.likes (the score), engagement.replies, parent_id, ext.depth and its own replies[] array. A top-level comment has parent_id: null; a reply carries its parent comment's id. A comment whose score is below zero has engagement.likes: null and the score in ext.score.

curl "https://api.insightsocial.app/v1/reddit/post/comments?url=https://www.reddit.com/r/Python/comments/POST_ID/slug/" \
  -H "x-api-key: $INSIGHTSOCIAL_API_KEY"

Most analysis wants a flat table, so flatten the tree yourself. This keeps the depth and the parent so you can rebuild the thread later:

def flatten(comments, parent=None, out=None):
    out = [] if out is None else out
    for c in comments:
        ext = c["ext"] or {}
        out.append({
            "id": c["id"],
            "parent": c["parent_id"] or parent,
            "depth": ext.get("depth"),
            "author": c["author"]["username"],
            "score": c["engagement"]["likes"] if c["engagement"]["likes"] is not None else ext.get("score"),
            "text": c["text"],
        })
        flatten(c["replies"] or [], c["id"], out)
    return out

top_level = [item["comment"] for item in body["data"]["items"]]
table = flatten(top_level)

On big threads, watch data.truncated. If it's true, Reddit cut the thread short, so keep paging with pagination.next_cursor while has_more is true. Removed comments come back with text: null rather than the literal [deleted], which keeps them from polluting sentiment counts.

The API returns comments as written and doesn't score sentiment. If you need it, run your own classifier or model over text; language tells you which model to send each comment to.

How do I get a Reddit user's profile and history?

Three endpoints cover a user. /profile is the account itself:

curl "https://api.insightsocial.app/v1/reddit/profile?handle=spez" \
  -H "x-api-key: $INSIGHTSOCIAL_API_KEY"

The karma split is at data.author.ext.post_karma, comment_karma and awardee_karma. Note that author.followers is null on purpose: Reddit doesn't publish a public follower count, so karma is the reach signal you get.

/profile/posts returns their submissions across every subreddit (with sort and timeframe), at 10 credits a page. /profile/comments returns their comment history; its limit (1 to 100, default 25) controls how deep one call reads, and before=YYYY-MM-DD walks further back. It's metered from 20 up to 2,000 credits, so keep limit modest until you know you need the full history.

One gotcha: an account with no posts and an account that doesn't exist both return an empty list from /profile/posts. Call /profile first if you need to tell them apart.

What does it cost to scrape Reddit?

Pricing is per endpoint in credits, and the rules are designed so you don't pay for nothing:

• Failed and empty calls cost 0. Errors, 404s and searches with no results are never charged.
• Repeats are charged. Re-running the same call is billed again, so store the results you need.
• Shared-cache hits cost 2 credits. If someone recently fetched the same public data, you pay 2 instead of the full price, and cached: true tells you so.
• Metered calls settle below the ceiling. You're charged what the call read, never more than the top of the range.

A rough budget: 10 subreddits, three pages of top posts each, is 30 calls at 10 credits apiece, 300 credits. Add /post/comments on the 20 busiest threads at 50 each and you're at 1,300 credits for a solid week of research. The free plan covers exploration; Pro is $9.99/month for 10,000 credits. Details on Credits and Pricing.

Before a large job, add dry_run=1 to any endpoint, such as /search or /post/comments. It returns the price range in data.dry_run (credits_min, credits_max) and costs nothing.

How do I handle rate limits and errors?

Each key allows 60 requests per minute and 10 in flight at once. Over either limit you get a 429 with a Retry-After header, and nothing is charged. Branch on error.type, retry only the transient ones (RATE_LIMITED, CONCURRENCY_LIMIT, SERVICE_UNAVAILABLE, UPSTREAM_ERROR, INTERNAL_ERROR), and when paginating, retry the same cursor rather than skipping ahead. Adding an Idempotency-Key header makes a retried call free if the first attempt actually succeeded. The errors reference lists every type.

What about Reddit's official API?

Reddit runs its own Data API, which requires registering an app, authenticating with OAuth, and accepting Reddit's developer terms. If you're building something that posts, votes or moderates, or you need Reddit's blessing for a commercial product, that's the route to evaluate, and you should read Reddit's current terms and rate limits directly on their developer pages, since they have changed over the years. If you only need to read public posts, comments and profiles as JSON, a data API like this one skips the app registration and gives you the same response shape you'd get for eight other platforms.

Can I do this without writing code?

Yes. The InsightSocial Chrome extension scrapes Reddit posts, search results and subreddits from inside your own browser and exports them to CSV or Excel. It draws on the same credit balance as the API, so you can mix the two.

And if you're wiring Reddit data into an AI agent, point it at the free, keyless catalogue at GET /v1/endpoints?platform=reddit so it plans calls from real parameters and prices. The AI agents guide has a paste-in prompt.

FAQ

Do I need a Reddit account or developer app to use this?

No. You need an InsightSocial API key in the x-api-key header. There's no Reddit app, client ID or OAuth token involved.

Can I get every comment on a large thread?

Yes, with paging. /post/comments returns the tree and flags data.truncated when it's incomplete; keep following pagination.next_cursor while has_more is true.

How far back can I search?

/search and /subreddit accept timeframe values up to all, and /profile/comments walks back with before=YYYY-MM-DD. How much history exists depends on what Reddit itself still serves.

Is there a free tier?

Yes. The free plan includes 500 credits per month, and your first 10 calls priced at 100 credits or less are free. Failed and empty calls never cost anything.

Does the same code work for other platforms?

Mostly. The envelope, pagination rule, error types and the post and comment schemas are shared across all 25 platforms, so switching from /v1/reddit/search to another platform's search is usually a path change. See the platform docs and the API overview.

Why did a call cost less than the listed price?

Check charge_reason in the response: shared_cache costs 2 credits, replay costs 0, and on metered endpoints miss charges only what the call read.