Twitter/X Scraping API: A Developer Guide for 2026

If you need public X data in a script, a pipeline or an agent, you have three realistic options: pay for the official X API, maintain your own headless-browser scraper, or call a data API that has already done the scraping work. This guide covers the third route with the InsightSocial API: how to read a profile, walk an account's timeline, search tweets, pull the replies under a post, and work out what the whole job will cost before you run it.

Not a developer? You don't need any of this. The InsightSocial Chrome extension exports profiles and tweets straight from the page you're looking at, and our step-by-step guide to scraping Twitter profiles without code walks through it.

Why not just use the official X API?

You can, and for some jobs you should (posting, DMs, reading your own account's data). For reading other people's public data it has two drawbacks. First, it's billed per resource: as of September 2026, X's own pricing page lists $0.005 per post read and $0.010 per user read, paid from prepaid credits, with no free tier. Second, it's X-only. If your project also touches TikTok, Instagram or Reddit, you end up with a separate integration, auth scheme and data model for every platform.

A scraping API gives you the public side of X, returned in the same response shape as the other eight platforms. We compare the options side by side in the best Twitter/X data APIs in 2026.

Why not build your own scraper?

For a weekend project it's a fine way to learn. In production, X's logged-out surface shifts often, the internal GraphQL query IDs rotate, and the rate limits on guest traffic are strict. You'd spend more time maintaining the scraper than using the data it collects. Expect to budget for proxies, blocked sessions, and regular parser rewrites.

How do you get set up?

Sign in, open the API section of your dashboard, and copy the key that's created for you. Keys start with isk_live_ and are shown once, so put yours in an environment variable:

export INSIGHTSOCIAL_API_KEY="isk_live_..."

Every request is a plain GET with one header, x-api-key. There's no OAuth dance and no app review. The quickstart covers the response envelope in detail; the short version is that the payload sits under data, and every response tells you what it cost (credits_used) and what you have left (credits_remaining).

Your first 10 calls priced at 200 credits or less are free, and the free plan includes 500 credits a month. That covers every example in this post except the three-page search: a metered call reserves its ceiling before it runs, and max_pages=3 holds 540 credits. Use one page on the free plan, or top up for the walk.

How do you scrape a Twitter/X profile?

GET /v1/twitter/profile takes a handle (no @) and returns followers, following, tweet count, bio, avatar and banner URLs, and verification status. It costs 20 credits, and re-running the same lookup within 24 hours is free.

curl "https://api.insightsocial.app/v1/twitter/profile?handle=natgeo" \
  -H "x-api-key: $INSIGHTSOCIAL_API_KEY"

import os
import requests

BASE = "https://api.insightsocial.app/v1"
HEADERS = {"x-api-key": os.environ["INSIGHTSOCIAL_API_KEY"]}

body = requests.get(f"{BASE}/twitter/profile",
                    params={"handle": "natgeo"}, headers=HEADERS).json()

if body["success"]:
    author = body["data"]["author"]
    print(author["username"], author["followers"], author["posts_count"])
    print("cost:", body["credits_used"], "left:", body["credits_remaining"])
else:
    print(body["error"]["type"], body["error"]["message"])

If you want metrics rather than raw fields, GET /v1/twitter/profile/full (100 credits) returns the profile, recent tweets and computed figures in one call: engagement rate by views and by followers, posting cadence, top post and format mix.

How do you scrape tweets from a user's timeline?

GET /v1/twitter/user/tweets returns an account's recent tweets, newest first, about 20 per page, with full text, likes, retweets, replies, bookmarks, views, media and timestamp. It's metered at 20–100 credits per page: we hold the ceiling while the call runs and charge what it actually used.

Here is the Node version, collecting the last three pages from National Geographic's timeline:

const KEY = process.env.INSIGHTSOCIAL_API_KEY;
const url = new URL("https://api.insightsocial.app/v1/twitter/user/tweets");
url.searchParams.set("handle", "natgeo");

const tweets = [];
let spent = 0;

for (let page = 1; page <= 3; page++) {
  const body = await (await fetch(url, { headers: { "x-api-key": KEY } })).json();
  if (!body.success) {
    console.error(body.error.type, body.error.message);
    break;
  }
  spent += body.credits_used;
  for (const item of body.data.items) {
    tweets.push({
      url: item.post.url,
      text: item.post.content.text,
      likes: item.post.engagement.likes,
      views: item.post.engagement.views,
      published_at: item.post.published_at,
    });
  }
  if (!body.pagination?.has_more) break;
  url.searchParams.set("cursor", body.pagination.next_cursor);
}

console.log(`${tweets.length} tweets, ${spent} credits`);

Every tweet uses the same unified post object as posts from the other platforms, so post.engagement.likes means the same thing on X as it does on TikTok. X-specific extras live under post.ext, for example ext.retweeted_post for the original behind a retweet and ext.quote_count.

Polling a timeline without paying twice

If you poll an account every day, don't re-walk pages you already have. user/tweets accepts since (a date or ISO timestamp) and stop_at_id (the id or URL of the newest tweet you stored). The walk ends at that boundary, pagination.stopped_at tells you which one ended it, and you only pay for the pages you actually fetched. Pinned tweets are out of date order and never end the walk, so don't store one as your "newest".

How do you search tweets by keyword?

GET /v1/twitter/search/tweets takes a query and an optional sort (latest, the default, or top). There are no separate date parameters: X's own operators go inside the query string, so since:2026-09-01, until:, from:handle, min_faves:20, filter:images and quoted phrases all work.

curl -G "https://api.insightsocial.app/v1/twitter/search/tweets" \
  --data-urlencode 'query="heat pump" since:2026-09-01 min_faves:50' \
  --data-urlencode "sort=top" \
  -H "x-api-key: $INSIGHTSOCIAL_API_KEY"

One thing to know about sort=top: it ranks by engagement, not relevance, so an unquoted multi-word query can return popular posts that match none of your words. Quote the phrase or add an operator.

Search is metered at 20–180 credits per page. If you want several pages in a single request, add max_pages (1 to 5). The walk still counts as one request against your rate limit, each page is billed as one call, and the most it can cost is the listed maximum times max_pages. data.walk.stopped tells you why the walk ended.

Search also has some filters we apply on our side, after each page arrives: min_views, max_age_days and sort_rows=views. See the Twitter/X platform docs for the full parameter list.

How do you scrape replies to a tweet?

GET /v1/twitter/tweet/replies takes the tweet's full url and returns the replies (not the tweet itself), each with text, author, likes, reply count and timestamp. It's metered at 20–100 credits per page and pages with the same cursor loop.

This Python example chains two endpoints: take National Geographic's most recent tweet from the timeline, then read the conversation under it.

import os
import requests

BASE = "https://api.insightsocial.app/v1"
HEADERS = {"x-api-key": os.environ["INSIGHTSOCIAL_API_KEY"]}


def get(path, **params):
    body = requests.get(f"{BASE}/{path}", params=params,
                        headers=HEADERS, timeout=120).json()
    if not body["success"]:
        raise RuntimeError(f'{body["error"]["type"]}: {body["request_id"]}')
    return body


def paginate(path, max_pages=5, **params):
    cursor = None
    for _ in range(max_pages):
        query = dict(params, **({"cursor": cursor} if cursor else {}))
        body = get(path, **query)
        yield from body["data"]["items"]
        page = body.get("pagination") or {}
        if not page.get("has_more"):
            return
        cursor = page["next_cursor"]  # pass it back unchanged


timeline = get("twitter/user/tweets", handle="natgeo")
latest = timeline["data"]["items"][0]["post"]["url"]

for item in paginate("twitter/tweet/replies", url=latest, max_pages=3):
    c = item["comment"]
    print(c["author"]["username"], c["engagement"]["likes"], (c["text"] or "")[:80])

A tweet with no replies returns an empty list, and an empty result costs nothing. Replies use the unified comment object, so the same parser handles YouTube or Reddit comments too.

Reply lists also come back with model labels attached under computed.labels, such as the tone of each reply, at no extra credits. Name them, for example label=sentiment,question,complaint, and every row is judged before the response is sent, still at no extra cost. A few opt-in labels do add credits; the labels docs list which.

How does pagination work?

The rule is the same on every list endpoint. Send pagination.next_cursor back as the cursor parameter, byte for byte, and stop when pagination.has_more is false. Don't stop on an empty page or a count, and don't decode the cursor (it's an opaque string that starts is2.). If a page fails with a 429 or 503, retry the same cursor rather than skipping ahead. The full loop is in the pagination docs.

Other Twitter/X lists work the same way: user/followers, user/following, tweet/retweeters, user/media and search/users are 20 credits per page.

What are the rate limits?

Each key allows 60 requests per sliding minute and 10 requests in flight at once. Go over either and you get a 429 with a Retry-After header, and the rejected request costs nothing. There's no daily request quota: past those two ceilings, your credit balance is what limits volume. For a bulk job, that means up to 10 workers per key and roughly one request a second on average.

What does a real Twitter/X job cost?

Take a common brief: size up a brand's presence on X before a campaign. One profile, its recent timeline, a week of mentions, and the replies under its five latest tweets. Here's the budget from our published prices, with the floor and ceiling for the metered endpoints:

The actual bill lands somewhere between those two numbers, depending on how much each metered call reads. That's why the call reserves the ceiling and settles the actual, and why credits_used is often well below the top of the range. The free plan's 500 monthly credits can't run it as written, because each metered call must be able to cover its ceiling when it starts, and the search alone holds 540. At the ceiling, it's a small slice of Pro's 10,000 credits a month ($9.99, or $7.99 billed yearly).

Three things push the real cost down:

• Repeats are free. The exact same call from your account inside its window (24 hours for profiles, 6 hours for timelines and replies, 1 hour for search) costs 0 credits.
• Shared-cache hits cost 5 credits. If someone else recently fetched the same public data, you pay 5 instead of the full price. The response says cached: true.
• Failures and empty results cost nothing. A handle that doesn't exist or a tweet with no replies doesn't touch your balance.

To see prices without spending anything, GET /v1/endpoints is public, free and needs no key. The full table is at endpoint pricing, and the credits docs explain metering and owned windows.

Can an AI agent use this?

Yes. Every endpoint is a plain GET with one header, and the response always carries the same envelope, which suits tool-calling agents. The AI agents guide lists what to hand Claude, Cursor or another agent so it writes the integration correctly. There's also GET /v1/twitter/ai-search (100 credits), which answers a plain-English question about X and returns the posts it cited.

FAQ

Is it legal to scrape Twitter/X?

Collecting publicly visible data is generally treated differently from accessing content behind a login, but the rules depend on your jurisdiction and what you do with the data. Our API returns only data anyone can see without logging in. If you process personal data, you are responsible for complying with laws such as GDPR. When in doubt, ask a lawyer.

Can I scrape tweets without an X account or login?

Yes. You don't need an X account, a developer app or cookies. The only credential is your InsightSocial API key.

Can I get a user's full tweet history?

user/tweets pages back through the timeline about 20 tweets at a time, as far as X shows the account's public timeline. For older material on a specific topic, search/tweets with from:handle and since:/until: operators is often the better route.

Which programming language works best?

Use what your stack already runs. Anything that can send an HTTP GET and parse JSON will do, from a Python script to a Node service to a cron job calling cURL. There's no SDK to install.

Does this also cover other platforms?

Yes. The same key and response format cover Instagram, TikTok, Facebook, LinkedIn, Threads, YouTube, Reddit and Pinterest. See the social media API overview.

What happens when I run out of credits?

A call that your balance can't cover is refused with a 402 INSUFFICIENT_CREDITS before it runs, and nothing is charged. Upgrade or buy a one-time pack that never expires on the pricing page.