devLap

Why TikTok Transcript Scraper Input Fields Constrain Behavior and Avoid Cost

python dev.to

The TikTok Transcript Scraper is a useful tool for extracting spoken content from TikTok videos. It fetches subtitles, auto-generated speech recognition (ASR) captions, and machine-translated (MT) versions. While the Actor's README covers its capabilities, a deeper dive into its documented input schema, output shape, and underlying platform behaviors reveals crucial failure modes and limitations that data engineers must anticipate and handle defensively.

How do you detect missing transcripts in the output?

You can detect missing transcripts by checking for a success: false field in the output row. When a video has no captions, is private, deleted, or a slideshow post, the Actor still emits a single row containing video metadata. Transcript-related fields will be missing or empty, and the success field will explicitly be false, along with a descriptive message.

The tiktok-transcript-scraper is designed to provide comprehensive data, but it also clearly signals when it cannot fulfill the primary request for transcripts. This is a crucial design choice that avoids silent data loss and allows for robust error handling. Instead of simply omitting the video from the output, which would require external validation to identify missing data, the Actor provides a sentinel record. This success: false flag indicates a specific failure mode directly within the dataset.

Consider the following Python snippet for processing results and identifying videos without available transcripts:

import apify_client

# Initialize the ApifyClient with your API token
client = apify_client.ApifyClient("YOUR_APIFY_TOKEN")

# Example Actor run ID
run_id = "YOUR_ACTOR_RUN_ID"

# Get a dataset client for the run's default dataset
dataset_client = client.dataset(client.run(run_id).get()['defaultDatasetId'])

# Iterate over the items in the dataset
for item in dataset_client.iterate_items():
    if not item.get('success', True): # Default to True if success field is missing for safety
        print(f"Video {item.get('postUrl', item.get('postId', 'unknown'))} had no captions:")
        print(f"  Reason: {item.get('message', 'No specific reason provided')}")
        print(f"  Metadata available: Username='{item.get('username')}', Caption='{item.get('caption')}'")
    else:
        print(f"Video {item['postUrl']} successfully scraped in {item['languageCode']}.")
        print(f"  Transcript: {item['transcript'][:100]}...") # Print first 100 chars
Enter fullscreen mode Exit fullscreen mode

This code explicitly checks for the success: false indicator. When encountered, it can log the reason and potentially trigger downstream processes to handle the absence of transcript data, perhaps by flagging the video for manual review or by attempting to use the useWhisperFallback option in a separate run if it wasn't already enabled.

When does useWhisperFallback significantly increase processing time?

Enabling the useWhisperFallback option significantly increases processing time when the Actor needs to generate transcripts for videos that lack native subtitles. This is because the Actor must download the entire video, then process its audio using a separate speech-to-text model (Whisper), which is computationally intensive and adds considerable latency compared to simply parsing existing WebVTT subtitle files.

The useWhisperFallback input parameter, defaulting to false, is a powerful feature for maximizing data coverage, but it introduces a trade-off. The Actor's primary method for extracting transcripts is to parse existing subtitle tracks. This is a quick operation, as it involves fetching relatively small WebVTT files. When useWhisperFallback is true and a video has no native subtitles, the Actor shifts to a much more resource-intensive process. It must download the entire video file to extract the audio, then submit that audio to a speech-to-text service. This conversion process is analogous to running a separate, heavy-duty AI model, dramatically increasing both the run duration and potentially the cost, especially for longer videos.

For scenarios where speed and cost efficiency are paramount, it's critical to evaluate whether the benefit of generating transcripts for every video outweighs the increased processing time and resource consumption. If your dataset frequently includes videos without native captions, enabling this fallback will make your runs take longer.

Here’s an example input for enabling the Whisper fallback:

{"postUrls":["https://www.tiktok.com/@somecreator/video/7123456789012345678"],"useWhisperFallback":true}
Enter fullscreen mode Exit fullscreen mode

This input tells the Actor to attempt Whisper ASR for the specified video if no native captions are found. Developers should monitor run times and resource usage carefully when deploying this option at scale, especially for large batches of videos where a significant portion might lack pre-existing subtitles.

What happens if I exceed the Apify synchronous run timeout?

If you exceed the Apify synchronous run timeout of 300 seconds, the API call to the tiktok-transcript-scraper will return an HTTP 408 Request Timeout error. For runs expected to last longer than five minutes, you must switch from the synchronous run endpoint to asynchronously POSTing to /v2/acts/<actor>/runs and then either polling the run status or using webhooks for completion notifications.

This platform-level constraint is critical for any data pipeline integrating with Apify. The synchronous endpoint is convenient for quick, interactive tasks, but it is not designed for long-running operations. The tiktok-transcript-scraper, especially when useWhisperFallback is enabled or when processing many videos, can easily exceed this 300-second limit. A 408 response means your client application will not receive the run's final status or output dataset ID directly, requiring you to implement an asynchronous pattern.

An asynchronous approach involves initiating the run and then either periodically querying its status or configuring a webhook to be notified upon completion.

Initiating an asynchronous run using curl:

curl -X POST \
  https://api.apify.com/v2/acts/crawlerbros~tiktok-transcript-scraper/runs?token=YOUR_APIFY_TOKEN \
  -H 'Content-Type: application/json' \
  -d '{
    "postUrls": ["https://www.tiktok.com/@natgeo/video/7637581966396656909"],
    "languages": ["en"]
  }'
Enter fullscreen mode Exit fullscreen mode

This POST request will immediately return a Run object containing the id of the newly created run. You then use this id to check the status or set up a webhook. Polling the run status might look like this:

import time
from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")
actor_id = "crawlerbros/tiktok-transcript-scraper"

run_input = {
    "postUrls": ["https://www.tiktok.com/@natgeo/video/7637581966396656909", "https://www.tiktok.com/@nasa/video/1234567890123456789"] # Example: multiple videos
}

# Start the Actor asynchronously
run = client.actor(actor_id).call(run_input=run_input)
run_id = run['id']
print(f"Actor run started with ID: {run_id}")

# Poll for run completion
while True:
    run_info = client.run(run_id).get()
    status = run_info['status']
    print(f"Run {run_id} status: {status}")
    if status in ['SUCCEEDED', 'FAILED', 'ABORTED']:
        break
    time.sleep(30) # Wait 30 seconds before polling again

if status == 'SUCCEEDED':
    print(f"Run {run_id} succeeded. Default dataset ID: {run_info['defaultDatasetId']}")
    # Process results as shown in the previous section
else:
    print(f"Run {run_id} finished with status: {status}")
Enter fullscreen mode Exit fullscreen mode

This ensures your application does not block indefinitely and can gracefully handle longer processing times by decoupling the run initiation from its completion.

How do input fields constrain the Actor's behavior and prevent certain operations?

The tiktok-transcript-scraper input schema defines postUrls and postIds as mutually exclusive arrays; providing both may lead to undefined behavior or early termination. The languages filter uses prefix-matching for BCP-47 codes, so malformed entries will silently fail to match available subtitles without raising an explicit error.

The input schema is more than just a type definition; it's a contract specifying how the Actor expects to be invoked.
Specifically:

  • postUrls vs. postIds: The README examples clearly show these as mutually exclusive inputs. While the schema defines them as separate array fields, a robust client implementation should ensure that only one is populated per run. If both are provided, the Actor likely processes postUrls and ignores postIds, or vice-versa, or simply fails to start. A defensive client application would enforce this exclusivity prior to API invocation.
  • languages filter: This field accepts an array of strings for language codes. The crucial detail is "prefix-based matching." This means ["eng"] will match eng-US and eng-GB, but ["eng-US"] will only match eng-US. If you provide a non-standard or malformed language code (e.g., ["english"] or ["fr_fr"]), it will simply not match any languageCode values generated by TikTok, effectively filtering out all results for that malformed code without explicitly raising an error. The output would simply contain no rows for the specified non-matching languages.

Here's an example demonstrating correct and incorrect input structures:

{"postUrls":["https://www.tiktok.com/@natgeo/video/7637581966396656909"],"languages":["eng-US","spa"]}
Enter fullscreen mode Exit fullscreen mode

This is valid.

{"postIds":["7637581966396656909","7652858746137234702"],"languages":["en","fr"]}
Enter fullscreen mode Exit fullscreen mode

This is also valid.

{"postUrls":["https://www.tiktok.com/@natgeo/video/7637581966396656909"],"postIds":["7637581966396656909"],"languages":["en"]}
Enter fullscreen mode Exit fullscreen mode

This input, supplying both postUrls and postIds, is problematic and should be avoided. The Actor's behavior is not guaranteed to be consistent, and it might prioritize one field over the other or fail entirely.

Defensive coding would involve input validation on the client side:

def validate_tiktok_input(input_data):
    if "postUrls" in input_data and "postIds" in input_data:
        raise ValueError("Cannot provide both 'postUrls' and 'postIds'. Choose one.")
    if "languages" in input_data:
        for lang_code in input_data["languages"]:
            # Basic BCP-47 pattern check; not exhaustive but catches common mistakes
            if not isinstance(lang_code, str) or not (len(lang_code) > 1 and all(c.isalnum() or c == '-' for c in lang_code)):
                print(f"Warning: Language code '{lang_code}' might be malformed.")
    return True

# Example usage:
try:
    validate_tiktok_input({
      "postUrls": ["https://www.tiktok.com/@natgeo/video/7637581966396656909"],
      "postIds": ["7637581966396656909"], # This will raise an error
      "languages": ["en-US", "english"]
    })
except ValueError as e:
    print(f"Input validation error: {e}")

try:
    validate_tiktok_input({
      "postUrls": ["https://www.tiktok.com/@natgeo/video/7637581966396656909"],
      "languages": ["en-US", "fr-FR", "invalid-lang"]
    })
except ValueError as e:
    print(f"Input validation error: {e}")
Enter fullscreen mode Exit fullscreen mode

This proactive validation helps prevent wasted runs due to incorrectly structured inputs or unexpected filtering behavior.

What are the output shape's constraints and how do they imply data limitations?

The output shape for tiktok-transcript-scraper is defined as a dataset where each item represents a single video-language combination. This implies that for a video with multiple language tracks, multiple output rows will be generated, one for each language. Crucially, the presence of fields like subtitleUrlExpiresAt signifies that direct access to the raw WebVTT subtitle files is transient, requiring immediate download if persistent storage of the raw files is needed.

The detailed output schema informs not just what data is available, but also its ephemeral nature and structure.

  • One row per video-language combination: This is a key design choice. If a single TikTok video has, for instance, an original ASR English transcript, an MT Spanish translation, and an MT French translation, the Actor will produce three separate output records. Each record will share common video metadata (postId, username, caption, duration) but differ in languageCode, languageId, source, segments, and transcript. This structure simplifies post-processing as each row is self-contained for a specific language version.
  • subtitleUrlExpiresAt: The presence of this field (an ISO 8601 timestamp) for subtitleUrl is a strong indicator of a time-sensitive resource. It means the subtitleUrl itself is not a permanent link and will become invalid after the specified time. Any downstream system that requires long-term archival of the raw .vtt files must fetch them promptly upon receiving the Actor's output, ideally before subtitleUrlExpiresAt. Relying on the subtitleUrl indefinitely will lead to broken links and inaccessible data.

An example output record snippet for a single language:

{"postId":"7637581966396656909","postUrl":"https://www.tiktok.com/@natgeo/video/7637581966396656909","username":"natgeo","displayName":"Nat Geo","caption":"The first official image from the James Webb Space Telescope...","createdAt":"2022-07-12T14:00:00.000Z","duration":60,"languageCode":"eng-US","languageId":"6924843187212048","subtitleFormat":"webvtt","source":"ASR","version":"1.0.0","variant":"default","subtitleUrl":"https://p16-sign.tiktokcdn-us.com/obj/tos-useast5...","subtitleUrlExpiresAt":"2026-09-16T18:30:00.000Z","isAutoGenerated":true,"isOriginalCaption":true,"segments":[{"start":0.0,"end":2.5,"text":"The first official image..."},{"start":2.6,"end":5.0,"text":"from the James Webb..."}],"segmentCount":100,"transcript":"The first official image from the James Webb Space Telescope...","transcriptDuration":58.0,"scrapedAt":"2026-09-16T16:00:00.000Z","success":true}
Enter fullscreen mode Exit fullscreen mode

If you need to persist the .vtt file, your data pipeline logic should look something like this:

import requests
from datetime import datetime

# Assuming 'item' is a scraped result from the dataset
item = {
  "postId": "7637581966396656909",
  "languageCode": "eng-US",
  "subtitleUrl": "https://p16-sign.tiktokcdn-us.com/obj/tos-useast5...", # Placeholder URL
  "subtitleUrlExpiresAt": "2026-09-16T18:30:00.000Z" # Example expiry
}

if item.get("subtitleUrl") and item.get("subtitleUrlExpiresAt"):
    expiry_time = datetime.fromisoformat(item["subtitleUrlExpiresAt"].replace('Z', '+00:00'))
    current_time = datetime.now(expiry_time.tzinfo) # Ensure timezone awareness

    # Download if still valid and needed
    if current_time < expiry_time:
        try:
            response = requests.get(item["subtitleUrl"], timeout=10)
            response.raise_for_status() # Raise an exception for HTTP errors
            filename = f"{item['postId']}_{item['languageCode']}.vtt"
            with open(filename, "w", encoding="utf-8") as f:
                f.write(response.text)
            print(f"Downloaded {filename}")
        except requests.exceptions.RequestException as e:
            print(f"Failed to download subtitle from {item['subtitleUrl']}: {e}")
    else:
        print(f"Subtitle URL for {item['postId']}_{item['languageCode']} has expired.")
Enter fullscreen mode Exit fullscreen mode

This code explicitly checks for expiry and attempts to download the VTT file, robustly handling potential network errors or expired links.

What limitations and caveats should I be aware of?

The tiktok-transcript-scraper has several important limitations. It cannot scrape private, deleted, or regionally restricted videos, which will result in success: false output rows. Slideshow (photo) posts typically lack audio and therefore subtitles, also yielding failure rows. Furthermore, while the subtitleUrl is provided, it expires within a few hours, necessitating prompt download if the raw .vtt files need to be archived long-term.

Understanding these boundaries is crucial for setting realistic expectations and designing robust data pipelines.

  • Unsupported Video Types: The Actor works exclusively with public, accessible TikTok videos. Any video that is private, deleted, or subject to regional restrictions will not yield transcripts. Instead, you'll receive a success: false row with relevant metadata and an explanatory message. Similarly, TikTok slideshow posts, which are image-based and lack an audio track, will also result in success: false entries because they have no content to transcribe.
  • Transient subtitleUrl: As discussed, the direct URL to the WebVTT subtitle file (subtitleUrl) is temporary. The subtitleUrlExpiresAt field explicitly marks its expiry. This is not a platform limitation but a design choice by TikTok for content delivery. Users must integrate immediate download logic if they require permanent storage of these .vtt files.
  • Proxy Session Limits: When using Apify's proxy services, particularly residential proxies, sessions typically persist for around 30 minutes. Datacenter proxies last longer, about 26 hours. While the tiktok-transcript-scraper handles proxy rotation internally, if you are building complex multi-step scraping workflows with shared proxy sessions or custom proxy configurations, this session expiry becomes a significant factor. For tiktok-transcript-scraper itself, this is largely transparent as it manages its own requests, but it's a general platform consideration.
  • Request Queue Consumption: An Apify RequestQueue can only be processed by one Actor or task run at a time. While multiple runs can add URLs to a single queue, you cannot fan-out processing by having several tiktok-transcript-scraper runs simultaneously consume from the same RequestQueue. If you need parallel processing for a large batch of videos, you must either distribute the URLs across multiple distinct input lists for separate runs or rely on the Actor's internal parallelism for a single, larger run.

These limitations mean that you must account for potential data gaps, plan for immediate data archival if necessary, and design your pipeline for either sequential queue processing or multiple independent input sets for concurrent runs.

What are the output schema fields and what do they mean?

The output schema for each result record provides detailed information about the extracted transcript and the video itself. Key fields include postId and postUrl for identification, username and displayName for author details, and caption and createdAt for video metadata. Transcript-specific fields are languageCode, source (indicating ASR, MT, or creator-uploaded), segments (an array of timestamped text cues), and transcript (the full plain-text version).

Understanding these fields is crucial for effectively utilizing the scraped data:

  • postId and postUrl: Unique identifiers for the TikTok video.
  • username and displayName: Information about the video's creator.
  • caption: The original text description provided with the video.
  • createdAt: The publish timestamp of the video (ISO 8601).
  • duration: The length of the video in seconds.
  • languageCode: The BCP-47 code (e.g., eng-US, fre-FR) identifying the language of the transcript in that specific output row.
  • languageId: TikTok's internal identifier for the language.
  • subtitleFormat: Always webvtt for this Actor.
  • source: The origin of the subtitle, either ASR (auto speech recognition), MT (machine translation), or other creator-uploaded captions. This helps differentiate between original spoken content and translated versions.
  • version and variant: Internal TikTok identifiers for subtitle versions.
  • subtitleUrl: A direct (but temporary) URL to the raw WebVTT subtitle file.
  • subtitleUrlExpiresAt: An ISO 8601 timestamp indicating when the subtitleUrl will become invalid.
  • isAutoGenerated: A boolean indicating if TikTok automatically generated the captions.
  • isOriginalCaption: A boolean indicating if the caption is in the video's original language.
  • segments: An array of objects, each containing start, end timestamps (in seconds), and the text of a specific caption segment. This is useful for precise timing.
  • segmentCount: The total number of segments in the segments array.
  • transcript: The full, joined plain-text transcription for the specific language in that row. Ideal for content repurposing.
  • transcriptDuration: The total duration (in seconds) covered by the captions in this transcript.
  • scrapedAt: An ISO 8601 timestamp of when the data was collected by the Actor.
  • success: A boolean flag, false if the video had no captions or could not be scraped.

These fields provide a comprehensive view of the video's transcript information and are essential for filtering, analysis, and integration into other applications.

What is the cost model for tiktok-transcript-scraper?

The tiktok-transcript-scraper uses a PAY_PER_EVENT pricing model. Every charge is for a specific event, not based on compute time or a subscription plan. The main events charged are "$0.005 per event" for each "result" (single item in the default dataset, with volume-tier prices of $0.005 for FREE, $0.00433 for BRONZE, $0.00367 for SILVER, and $0.003 for GOLD, PLATINUM, and DIAMOND tiers), and "$0.05 per GB of memory allocated to the run" for "Actor Start" (minimum one event).

The cost of running tiktok-transcript-scraper scales directly with the number of output records it generates and the memory allocated to the run.

  • Results: Each individual item stored in the default dataset incurs a charge of $0.005. Since the Actor outputs one row per video-language combination, a single video with five available language transcripts will produce five result items. The volume tiers can reduce this per-item cost significantly for higher usage volumes.
  • Actor Start: This event is charged once per run at $0.05 per GB of memory allocated. The minimum charge for Actor Start is $0.05 (for 1GB or less). This cost is fixed per run start regardless of how many videos are processed or how long the run takes (up to the platform's synchronous run timeout, if applicable).

It is crucial to note that enabling useWhisperFallback does not introduce a separate cost event, but it will indirectly increase the likelihood of generating more result events (by transcribing videos that would otherwise produce success: false rows) and potentially require more memory, thus impacting the "Actor Start" cost if you manually increase memory allocation for intensive ASR tasks. The cost does not scale with the duration of the container run, but purely on these two defined event types.

Checked against the Actor's input schema and Apify docs on 2026-09-16.

The Actor's README is the source of truth for its inputs, outputs and limits. Need a hand wiring this into your stack? Email info@crawlerbros.com

Source: dev.to

arrow_back Back to Tutorials