Indexing Large PDFs Without Timing Out: Background Processing with WP-Cron

The first version of our indexing code did exactly what you'd expect: a PDF gets uploaded, a hook fires, we parse the whole file, we store the text. Simple, and it worked fine — right up until someone uploaded a 300-page scanned course catalog and the request just… never finished.

What we started with

add_action( 'add_attachment', 'webequipe_index_pdf_on_upload' );

function webequipe_index_pdf_on_upload( $attachment_id ) {
    $file = get_attached_file( $attachment_id );

    if ( pathinfo( $file, PATHINFO_EXTENSION ) !== 'pdf' ) {
        return;
    }

    $parser = new \Smalot\PdfParser\Parser();
    $pdf    = $parser->parseFile( $file );
    $text   = $pdf->getText(); // fine for a 5-page PDF, not for a 300-page one

    webequipe_store_pdf_text( $attachment_id, $text );
}

For a handful of pages this runs in under a second. For a large, text-heavy file it can easily blow past PHP's max_execution_time — 30 seconds by default, and often less on shared hosting. When that happens, the request just gets killed mid-parse. Worst part: this was firing synchronously on the upload request itself, so the admin uploading the file was the one sitting there watching it hang.

Bumping max_execution_time isn't a real fix. A lot of hosts don't let you override it at all, and even where you can, you're still tying up one PHP worker for the entire parse — which gets worse, not better, the moment two people upload large PDFs at the same time.

The actual fix: stop trying to do it all at once

Instead of parsing the whole file in one request, we split the work into small chunks and let WP-Cron pick them up one at a time in the background.

Step one — on upload, just queue it. Don't parse anything yet.

add_action( 'add_attachment', 'webequipe_queue_pdf_for_indexing' );

function webequipe_queue_pdf_for_indexing( $attachment_id ) {
    $file = get_attached_file( $attachment_id );

    if ( pathinfo( $file, PATHINFO_EXTENSION ) !== 'pdf' ) {
        return;
    }

    webequipe_set_status( $attachment_id, 'scheduled' );

    if ( ! wp_next_scheduled( 'webequipe_process_pdf_chunk', array( $attachment_id ) ) ) {
        wp_schedule_single_event( time() + 10, 'webequipe_process_pdf_chunk', array( $attachment_id ) );
    }
}

The upload request now finishes instantly. The actual parsing hasn't started — it's just been scheduled.

Step two — the cron job processes a fixed number of pages, then reschedules itself if there's more to do.

define( 'WEBEQUIPE_PAGES_PER_RUN', 20 );

add_action( 'webequipe_process_pdf_chunk', 'webequipe_process_pdf_chunk_handler' );

function webequipe_process_pdf_chunk_handler( $attachment_id ) {
    $file   = get_attached_file( $attachment_id );
    $offset = (int) get_post_meta( $attachment_id, '_webequipe_page_offset', true );

    webequipe_set_status( $attachment_id, 'processing' );

    $parser = new \Smalot\PdfParser\Parser();
    $pdf    = $parser->parseFile( $file );
    $pages  = $pdf->getPages();
    $total  = count( $pages );

    $end = min( $offset + WEBEQUIPE_PAGES_PER_RUN, $total );

    for ( $i = $offset; $i < $end; $i++ ) {
        webequipe_store_page_text( $attachment_id, $i + 1, $pages[ $i ]->getText() );
    }

    update_post_meta( $attachment_id, '_webequipe_page_offset', $end );

    if ( $end < $total ) {
        wp_schedule_single_event( time() + 5, 'webequipe_process_pdf_chunk', array( $attachment_id ) );
    } else {
        webequipe_set_status( $attachment_id, 'indexed' );
        delete_post_meta( $attachment_id, '_webequipe_page_offset' );
    }
}

Twenty pages at a time, every few seconds, until the whole document is indexed. No single request ever does enough work to risk a timeout, no matter how large the file is.

One honest caveat on the snippet above: re-parsing the entire PDF on every chunk just to reach the next offset is wasteful for very large files — in practice you'd want to cache the parsed object or extract page references once rather than re-running parseFile() on every cron tick. I simplified that part here to keep the example readable; it's the kind of optimization that matters more once you're indexing hundreds of large files a day than it does for a single one.

Tracking progress so the admin isn't guessing

None of this is useful if nobody can tell what's actually happening to their file. We track status per attachment in a dedicated table rather than relying on post meta alone:

CREATE TABLE {$wpdb->prefix}webequipe_pdf_search_files (
    id BIGINT UNSIGNED AUTO_INCREMENT PRIMARY KEY,
    attachment_id BIGINT UNSIGNED NOT NULL,
    status VARCHAR(20) NOT NULL DEFAULT 'not_indexed',
    total_pages INT UNSIGNED DEFAULT 0,
    indexed_pages INT UNSIGNED DEFAULT 0,
    updated_at DATETIME NOT NULL,
    KEY attachment_id (attachment_id)
);

That status column is what powers the Manage PDFs screen — every file sits in one of: Not Indexed, Scheduled, Processing, Indexed, Stalled, Error, or Excluded. "Stalled" specifically exists to catch the case where a scheduled cron event never actually ran — which brings up the one WP-Cron quirk worth knowing about.

WP-Cron isn't a real cron

WordPress's cron system doesn't run on a timer in the background the way a system cron job does — it only fires when someone visits the site. On a low-traffic site, that means a scheduled indexing job can sit waiting for a while if nobody happens to load a page.

For anything beyond small personal sites, we point people toward disabling WP-Cron's default trigger and hitting wp-cron.php on a real system cron instead:

// wp-config.php
define( 'DISABLE_WP_CRON', true );

bash

# real system cron, every 5 minutes
*/5 * * * * curl -s https://example.com/wp-cron.php > /dev/null 2>&1

It's a small config change, but it's the difference between "indexing happens promptly" and "indexing happens whenever someone next visits the site."

What this actually bought us

Visitors never notice any of this — search queries run against whatever's already been indexed, and indexing itself never touches a front-end page load. On the admin side, even a huge scanned archive just ticks along in the background instead of hanging a browser tab or silently failing halfway through.

The bigger lesson generalizes past PDFs, honestly: if a task's runtime scales with user-supplied input — file size, record count, whatever — assuming one request can always finish it is asking for a timeout eventually. Chunking with an explicit, inspectable progress state turned out to be a lot more robust than just hoping the PHP process would have enough time.