Data availability

A successful response tells you what one endpoint returned for one request. It is not proof that nothing else exists on the source platform.

The excerpt below is trimmed from a real GET /v1/twitter/user/tweets response. It contains each state your parser needs to handle:

{
  "success": true,
  "platform": "twitter",
  "endpoint": "/v1/twitter/user/tweets",
  "schema_version": "2",
  "data": {
    "items": [
      {
        "post": {
          "id": "2106502020849127865",
          "kind": null,
          "content": { "text": "RT @Astro_Jessica: Room with a view…", "media_urls": null },
          "author": { "id": null, "username": "NASA", "display_name": "NASA" },
          "engagement": { "views": 40169, "likes": 0, "comments": 0, "shares": 572, "saves": 0 },
          "published_at": "2026-10-03T21:49:53.000Z",
          "language": "en",
          "ext": { "author_followers": 92378243, "quote_count": 0 }
        }
      }
    ],
    "total": null,
    "truncated": null
  },
  "pagination": {
    "next_cursor": "v2c.M-GEP84JXNZPqhQ0NlEOubhbuFGG…",
    "has_more": true,
    "page_size": 19
  },
  "unavailable": ["items[].post.author.id"],
  "credits_used": 10,
  "credits_remaining": 9490,
  "request_id": "req_161a29ec13ca",
  "cached": false,
  "idempotent_replay": false,
  "charge_reason": "miss",
  "free_call": false
}

In this row, post.id and post.content.text hold values. post.kind is null because the platform did not say what format the post is. post.author.id is null on every row of the page, so the envelope lists it in unavailable. ext.author_followers is a platform-specific extra: present here, absent on other platforms.

Public data only

InsightSocial returns data that anyone can see on the platform without logging in. Private accounts and content behind a login are out of scope.

What is public changes over time. A post can be edited or deleted, an account can go private, and a platform can stop showing a field. A successful call describes what the chosen endpoint could see at that moment, in that cache state.

Four states to tell apart

unavailable never turns a success into an error. For the envelope itself see Response schema.

unavailable

unavailable is a list of paths into data, such as "post.author.id" for a single entity or "items[].post.author.id" for a page. It is [] when nothing is missing. A path is listed only when all three hold:

1. The field is one we judge. These are the fields every entity has once a platform exposes them: ids, author identity, counts, URLs and times. A null bio, external_url, content.text or duration_seconds usually means the profile has no link or the post is a photo, so those are never listed.
2. The platform exposes it. Fields a platform never shows are in the table below, once, instead of in every response.
3. It is null on every item of the page. One filled row shows the endpoint can fill it; a null on a single row is the platform hiding it on that item (a hidden like count, for example).

An empty page lists nothing: an empty search is not a missing field.

Fields judged per entity. kind is not judged on posts: it is null whenever the source does not say what format a post is, which on some endpoints is every row.

Beta endpoints, which return the platform's own structure, always send unavailable: [].

Fields a platform never shows

These fields are null on every response from that platform. They are never listed in unavailable, because nothing can fill them.

The table is checked against live traffic, and it changes when a platform changes what it shows publicly.

One page is not the whole set

A list page only proves what is on that page. A row the platform sends without an id is left out of items and does not count toward page_size, but the page is still charged at its price: the platform returned it and we paid for it. Only when every row on a page lacks an id is the call refused instead, with 503 UPSTREAM_INVALID, and nothing is charged. Follow the top-level pagination object if the response has one.

Pass pagination.next_cursor back exactly as you received it (it starts with v2c.) in the cursor parameter, with the same parameters as the first page, and stop when pagination.has_more is false. Do not stop based on data.total, which is the platform's own total and not a page count. The full loop is in Pagination.

Reaching the last page means you have every page the endpoint offered. It does not guarantee full history: endpoint depth, deletions and access restrictions can still leave gaps. When the platform itself cuts a list short, data.truncated is true.

Order is not stable

A platform can return records in a different order from one request to the next. Do not use a row's position as an identifier, and do not assume page one always spans the same time window.

When an endpoint offers a sort parameter, use it. Store the IDs and timestamps from each row, and deduplicate across pages and runs when your workflow needs it.

Region, language and date filters

Filters and depth differ per endpoint. A region, language or date parameter on one search endpoint says nothing about any other endpoint on the same platform.

Check the endpoint's parameters before you design a query. If a filter is not listed, assume it is not applied. When you filter on your side, do it only on fields that are present and fit the decision.

For example, read the TikTok profile videos reference before you build on /v1/tiktok/profile/videos. It lists the parameters that endpoint accepts and the fields it returns.

Freshness and cache state

The body field cached tells you whether the response came from a recent fetch rather than a new one. A cached response costs 2 credits. A cache hit is never guaranteed.

When your decision needs a new fetch, send Cache-Control: no-cache or add fresh=1. The call is then charged at the endpoint's normal price. Details are in Caching.

A new fetch is fresher, not more complete. It still reflects only what the platform shows at that moment.

An empty result is never charged. See Credits.

Provenance and extension fields

Store platform, endpoint and request_id alongside the IDs and URLs you keep. They let you trace any stored record back to the call that produced it.

Fields under ext.* carry platform-specific details and the IDs you need to chain one endpoint into another. Their keys are snake_case. Treat every ext field as optional, and check the unified schema for which platforms fill it. Values that every platform can mean the same way live in the core instead: the author's id is author.id, the content language is language, and a bio link is external_url.

Endpoints marked beta on their reference page (metrics and some lists) return the source's own structure under our naming rules: snake_case keys, string ids, and ISO times where the source gives a timestamp. Some carry the source's own figures and date text. Expect their shape to differ by platform, and their fields to change. profile/full is beta as well, but its data.profile is a typed profile and its data.posts[].post are typed posts; only its other keys follow these beta rules.

New optional fields can appear in responses. Ignore fields you do not recognize instead of rejecting the response, and validate only the fields you actually use.

Pick the endpoint that fits the question

The endpoint you choose sets what you can see. A profile lookup, a post list, a keyword search and a comment list answer different questions, with different filters and depth.

Start from the question, then use Choosing endpoints to find the narrowest endpoint that returns the object, coverage and controls you need.

Checklist

Before relying on an endpoint, confirm that:

• It returns the object and fields your question needs.
• The records you expect are public.
• Its region, language, date, ordering and depth controls are documented for that endpoint.
• Your loop sends pagination.next_cursor back unchanged and stops on pagination.has_more.
• You log unavailable on every response and know which fields the platform never shows.
• Your parser treats a value, null and an absent field as three different cases.
• Your freshness policy reads cached and budgets for fresh fetches at the normal price.
• Stored records keep their IDs plus request_id, platform and endpoint.
• Unknown fields are ignored, and the fields you use are validated.