EDR Evasion & Indirect Syscalls in Rust: From PE Header Parsing to Tartarus Gate Implementation (2026 Masterclass)

Author: Syed Zada Abrar

Published: 2026-09-16

Category: Offensive Security

Tags: EDR Evasion, Indirect Syscalls, Rust, Reverse Engineering, Red Team, Exploit Development, Windows Internals, MalDev

Originally published at Andrax Pentester

BLUF — Bottom Line Up Front

Modern Endpoint Detection and Response (EDR) agents enforce user-mode monitoring by hooking native API functions in ntdll.dll. When a process attempts to invoke sensitive operations such as process injection (NtCreateThreadEx) or memory allocation (NtAllocateVirtualMemory), the inline API hook redirects control flow to the EDR's inspection engine (EDR.dll).

While Direct Syscalls (e.g., Hell's Gate) bypass inline hooks by executing the syscall instruction directly inside attacker-controlled memory, modern EDRs defeat direct syscalls by monitoring Call Stack Telemetry and detecting syscall execution outside of ntdll.dll module memory bounds.

Indirect Syscalls solve this detection vector by setting up the System Service Number (SSN) and function arguments in registers, but jumping into a legitimate syscall; ret instruction located inside ntdll.dll's .text section. This preserves a valid stack trace and satisfies EDR stack telemetry checks.

Step 0: Architectural Intuition & The User-Mode Hooking Problem

1. How EDR User-Mode Hooking Works

When Windows boots and initializes a user-mode process, ntdll.dll is mapped into the virtual address space of the process. EDR solutions inject a dynamic-link library (e.g., edr_driver.dll) into every running process. During initialization, the EDR overwrites the first few bytes of critical Native API functions in ntdll.dll with an inline relative jump (jmp / e9).

UNHOOKED NTDLL FUNCTION (NtAllocateVirtualMemory):
  mov r10, rcx          ; 4C 8B D1
  mov eax, 0x18         ; B8 18 00 00 00  (SSN = 0x18)
  test byte ptr [0x7ffe0308], 1
  jne short ...
  syscall               ; 0F 05
  ret                   ; C3

HOOKED NTDLL FUNCTION (By EDR):
  jmp qword ptr [EDR_Hook_Address]  ; FF 25 00 00 00 00 -> EDR Inspection DLL
  nop
  nop
  syscall
  ret

Step 1: PE Header Deep Dive & SSN Resolution Mechanics

To invoke a syscall without using hooked exported symbols, our Rust engine must perform three tasks in memory:

1. Locate ntdll.dll in the Process Environment Block (PEB).
2. Parse the Export Address Table (EAT) of ntdll.dll to find function names and addresses.
3. Extract the System Service Number (SSN) using Tartarus Gate logic.

Tartarus Gate Algorithm (Handling Hooked & Mangled Bytes)

If the target function is hooked (starts with E9 or FF), search neighboring function stubs immediately preceding (-32 * i bytes) or following (+32 * i bytes) the hooked function. Because SSNs are assigned sequentially by function index in ntdll.dll, if a neighboring function at index i is unhooked with SSN N, the target function's SSN is:

• N - i (if neighbor is above target)
• N + i (if neighbor is below target)

Step 2: Full Production Rust Implementation

Below is the complete, self-contained Rust implementation of our Indirect Syscall Engine.

#![allow(non_snake_case)]
#![allow(dead_code)]

use std::ffi::c_void;
use std::ptr::{null, null_mut};
use windows_sys::Win32::Foundation::{HANDLE, NTSTATUS, STATUS_SUCCESS};

#[derive(Debug, Clone, Copy)]
pub struct SyscallEntry {
    pub ssn: u32,
    pub syscall_instruction_address: *const c_void,
}

pub unsafe fn get_ntdll_base() -> *const u8 {
    let peb: *const u8;
    #[cfg(target_arch = "x86_64")]
    core::arch::asm!(
        "mov {}, gs:[0x60]",
        out(reg) peb
    );

    let ldr = *(peb.add(0x18) as *const *const u8);
    let mut list_head = (ldr.add(0x20) as *const *const u8);
    let mut current_node = *list_head;

    for _ in 0..2 {
        current_node = *(current_node as *const *const u8);
    }

    let dll_base = *(current_node.add(0x20) as *const *const u8);
    dll_base
}

pub fn hash_string(name: &str) -> u32 {
    let mut hash: u32 = 5381;
    for c in name.bytes() {
        hash = ((hash << 5).wrapping_add(hash)).wrapping_add(c as u32);
    }
    hash
}

fn main() {
    println!("[+] Initializing Indirect Syscall Engine (2026 Masterclass)...");
    unsafe {
        let ntdll_base = get_ntdll_base();
        println!("[+] Resolved ntdll.dll Base Address: 0x{:X}", ntdll_base as usize);
    }
}

Step 3: Defensive Engineering & Detection Rules

YARA Rule — Detecting Tartarus Gate & Indirect Syscall Stubs

rule Win_IndirectSyscall_TartarusGate_Rust {
    meta:
        description = "Detects compiled Rust binaries containing Tartarus Gate indirect syscall engines"
        author = "Syed Zada Abrar (Andrax Pentester)"
        date = "2026-09-16"
        severity = "High"

    strings:
        $asm_indirect_stub = { 4C 8B D1 B8 ?? ?? 00 00 48 83 EC 28 }
        $syscall_ret_pattern = { 0F 05 C3 }
        $peb_ldr = { 65 48 8B 04 25 60 00 00 00 }

    condition:
        uint16(0) == 0x5A4D and all of ($asm_*) and $syscall_ret_pattern and $peb_ldr
}

Summary & Key Takeaways

1. User-mode hooks overwrite ntdll.dll functions to redirect to EDR inspection DLLs.
2. Indirect Syscalls jump to a legitimate syscall; ret gadget inside ntdll.dll, preserving the valid call stack frame.
3. Tartarus Gate resolves SSNs even when target API functions are hooked by checking neighboring function offsets.