Kernel Driver Gates and Handshakes
Let's talk today about gates and handshakes, a common security mechanism implemented in many Windows kernel drivers that expose sensitive interfaces to user mode through device objects and I/O control requests (IOCTLs).
Once a driver creates a device object and accepts requests from user mode, the driver becomes responsible for enforcing the trust boundary around whatever privileged operations it exposes.
Microsoft provides several mechanisms for this boundary: device-object security descriptors, service control permissions, handle access masks, IOCTL access requirements, caller token inspection, privilege checks, framework defaults in KMDF and UMDF, and code-integrity policy for driver loading.
These mechanisms are the intended foundation, but are they enough? A driver can restrict which principals may open its device object, require specific access rights for sensitive IOCTLs, and perform additional authorization checks against the requestor’s token or process context.
Vendor drivers often add a second layer above these mechanisms. This security layer is usually a private protocol between the driver and its intended user-mode client. It may appear as a magic constant in an IOCTL packet, an authorization IOCTL, a special open contract (buffer-size, arrangement, etc), an encrypted process identifier, a caller-image hash, a resource sign, or a registry-controlled allowlist. These mechanisms are referred to here as gates and handshakes.
What is a gate? A gate is a condition that must be satisfied before the driver accepts a request or enables a privileged operation.
What is a handshake? A handshake is a gate that requires multiple steps, state transfer, or session establishment.
The motivation behind this sort of implementations comes from the vendor who wants to a narrower policy than “any user may open this device.” Many hardware drivers are intended to serve a client application: a firmware updater, tuning utility, diagnostic tool, RGB controller, overclocking suite, anti-cheat component, or platform service. The vendor’s intended policy then becomes narrower: Only the vendor client should be able to use this kernel interface.
If the device object is exposed broadly, or if the vendor does not deploy a broker service with a restrictive service SID, the driver must implement its own notion of client recognition. Gates and handshakes are a common result.
Security from the Kernel perspective:
A Windows device object has by default a security descriptor. This descriptor determines which users, groups, services, or integrity levels can open a handle to the device. If correctly configured, this is the earliest and strongest point of enforcement, because unauthorized callers never obtain a usable handle.
The I/O manager also enforces access requirements encoded in IOCTL definitions. IOCTLs created with CTL_CODE include access bits such as FILE_READ_DATA, FILE_WRITE_DATA, or FILE_ANY_ACCESS. If a handle lacks the required access, the request can be rejected before the driver processes the IOCTL.
Drivers may also inspect requestor context. They can examine whether the request originated from user mode, query the calling process, inspect the caller token, check privileges, or enforce a service-only model. In KMDF, per-file-object context and queue configuration make it easier to associate authorization state with a handle.
What about driver signing policy and code integrity checks?
Driver signing and code integrity are separate controls. They cover whether a driver may load. They do not automatically make the driver’s device interface safe. A signed driver can still expose an unsafe IOCTL surface.
This distinction is central: code integrity controls admission of kernel code; device security and driver logic control use of the loaded driver.
If Microsoft does provide security mechanisms why do vendors have their own gates?
Gates usually exist because the vendor does not want the driver to be a general-purpose interface. The driver may expose operations such as MSR read/write, physical memory mapping, MMIO access, port I/O, PCI configuration access, EC access, SMBus transactions, ACPI method calls, firmware flashing, or kernel memory inspection. These operations are highly privileged and often difficult to constrain safely.
A vendor may therefore add a condition to be met before servicing requests. These conditionals are typically meant to authenticate an intended client relationship.
Common purposes conditional gates include:
- Restrict use only to the vendor client application
- Prevent accidental calls from unrelated tools
- Protect unstable or undocumented IOCTL structures
- Support licensing or product segmentation
- Reduce support exposure from third-party clients
- Separate diagnostic functionality from other behavior
- Bind privileged operations to a service or updater process
- Internal checks between user-mode and kernel-mode components
Types of common gates used in Kernel modules:
Let's take a deeper look into the implementations of these gates, here we will cover the most common ones, but the sky is the limit with these conditions and one as a kernel driver developer could be as creative as the code would allow and the implementation would need.
Magic-Value type of Gates:
The simplest form is a fixed value in the request packet. The driver expects a field such as:
typedef struct _REQUEST {
uint32_t Magic;
uint32_t Command;
uint64_t Address;
uint32_t Size;
} REQUEST;
The dispatch routine rejects packets whose Magic field does not match the expected constant. This design serves several purposes. It identifies the packet as belonging to the vendor’s private protocol. It allows multiple subcommands to be multiplexed through one IOCTL. It reduces accidental interpretation of malformed input. It also provides a minimal compatibility check between the user-mode client and the driver. Its security value is limited of course and the constant must be present in the vendor client or recoverable from the driver. Once known, it can be reproduced by any caller. A magic value is therefore not authentication.
The following image corresponds to nvflash.sys. The driver implements a dispatcher-level gate using a hardcoded magic value, 0x72626d41. During IOCTL handling, the first qword of the caller-supplied input buffer is read and compared against this constant. If the value does not match, the request is rejected before reaching the command-specific logic.
When the magic value is present, the dispatcher continues normal IOCTL processing and routes execution to the corresponding handler for the requested operation. In this design, the magic value acts as a protocol gate: it does not authenticate the caller, but it ensures that only requests using the driver’s expected private packet format are accepted by the dispatcher.
Magic gates are common in drivers that expose a single large control IOCTL and implement an internal command dispatcher. They are also common in drivers ported from older utility code where the packet format evolved independently of Windows access control.
Authorization IOCTLs
Authorization IOCTLs are a bit different, but sort of the same. The user-mode client first obtains a handle, then sends a specific gate to that IOCTL. The driver might keep authorization state in per-handle context and require that state for sensitive operations. The driver-side structure may resemble:
typedef struct _FILE_CONTEXT {
BOOLEAN Authorized;
HANDLE ProcessId;
} FILE_CONTEXT;
The create path allocates the context and a authorization IOCTL validates a packet and marks the context authorized.This design is a bit better than a harcoded variable because authorization is tied to a file object. A handle that completed the handshake can be distinguished from another handle opened by a different process. Some drivers implement this by requiring an encrypted process identifier. The user-mode client places its PID into a block, encrypts it with a hardcoded key, and sends the result to the driver. The driver verifies the packet and authorizes the handle.
And the following example shows an undisclosed kernel driver using a magic key, per IOCTL as a conditional check. Before an IOCTL is routed to its specific handler, the dispatcher calls a shared validation routine. This routine, the magic_gate,verifies that the input buffer is large enough to hold the requested IOCTL and then checks the first DWORD of the buffer, that should contain the key, against the hardcoded value. Requests that do not carry this value are rejected before reaching the internal handler. The mechanism does not establish a persistent authorized session, it only verifies that each request correspond to the driver’s condition.
This binds the request to a process instance, but it does not create a durable secret if the key is embedded in distributed binaries. Its effectiveness depends on the threat model. It can prevent casual use and accidental access, but it is not a strong defense against reverse engineering.
Open-Time Handshakes
In an open-time handshake, the driver enforces the private contract during IRP_MJ_CREATE. Instead of accepting a normal open and authorizing later, the driver expects additional data or accompanying kernel objects at handle creation time.
Implementation techniques include extended attributes, specific object names, shared sections, event pairs, custom create parameters, or a naming convention embedded in the open request. The driver parses this material in its create dispatch routine and either rejects the open or stores an authorized context on the file object.
This pattern is often used when the driver was built for a tightly coupled companion application. It can support version negotiation, licensing checks, shared memory setup, or request-channel initialization. Because the authorization state can be associated with the resulting file object, later IOCTL handling can depend on the fact that the handle was created through the expected path.
The security risk is that IRP_MJ_CREATE becomes a parser for user-controlled protocol data. If the handshake parser is complex, it becomes part of the attack surface. A gate that accepts extended attributes, object names, offsets, lengths, or shared memory descriptors must validate them with the same rigor as any IOCTL input.
Caller-Image Gates
Some drivers attempt to recognize the intended user-mode client by its executable image. The driver or companion logic may check the process image path, a file hash, an embedded resource, a signature property, an image timestamp, or a vendor-specific resource stamp.
As an example, the ASUS-family implements a caller-image gate through an ASUSCERT resource embedded in the user-mode executable (this changes in the new driver ASUS drivers, but it's still quite similar). This mechanism attempts to distinguish the vendor-recognized caller image from arbitrary clients before the driver interface is used.
The ASUS driver uses a caller-image gate based on an embedded PE resource. The user-mode component prepares the caller image by adding an ASUSCERT resource, while the kernel driver validates that resource during its open/initialization path. In the driver, the PE resource directory is parsed manually: each named resource entry is converted into a UNICODE_STRING and passed for comparison. That gate checks the name against the hardcoded string ASUSCERT. If the comparison fails, the entry is ignored and validation continues. If the comparison succeeds, the driver resolves the resource data and treats the caller image as having the expected ASUS certificate marker.
And the following image shows the condition itself, the gate.
The intended policy is:
The driver should accept requests only from this executable or from an executable prepared by the vendor. This approach is common in vendor utility drivers that were not built around a broker service. It is an attempt to bind privileged kernel access to a known user-mode binary rather than to every process with administrative access.
Image gates vary widely in strength. A signature validation against a trusted publisher is stronger than a path string comparison. A hash check is precise but brittle across updates. A resource stamp can be copied or regenerated if the algorithm and key are recovered. A PID allowlist can be vulnerable to process lifetime and reuse issues if not implemented carefully.
The core weakness is that caller-image identity is difficult to validate correctly from a kernel driver. Path-based checks are especially fragile. Robust designs must account for renames, hard links, reparse points, section object identity, signature validation, update behavior, and time-of-check/time-of-use issues.
Registry And Policy Gates
A driver may use registry configuration to identify allowed clients. For example, a service key may contain an AllowedProcessName value. During initialization or request processing, the driver reads this value and compares it against the caller process.
This model separates policy storage from request handling. An installer or service writes policy; the driver enforces it. It is useful when a vendor service manages driver access dynamically. Its strength depends on who can modify the policy. If local administrators can rewrite the registry value, the gate should not be considered a boundary against administrators. It may still be adequate for separating the vendor client from ordinary user processes or for reducing accidental exposure.
The following example corresponds to a gate by ksID (Microsoft Defender) KslD enforces this caller policy in the create path and when a process opens the device, the driver retrieves the current process identifier, opens a handle to that process, and queries ProcessImageFileName through ZwQueryInformationProcess. The resulting image path is compared against the configured AllowedProcessName value with RtlCompareUnicodeString, using a case-insensitive comparison. The registry for this key can be found here: HKLM\SYSTEM\CurrentControlSet\Services\KslD
If the strings do not match, the create operation fails with STATUS_ACCESS_DENIED. If they match, the driver marks the device context as accepted and records the caller PID. This makes the gate handle-creation based rather than IOCTL based: unauthorized callers are rejected before obtaining an accepted session with the device.
Capability Gates
Some driver interfaces perform setup before exposing a usable primitive. A driver handle may exist, but read, write, mapping, or translation functionality remains disabled until initialization succeeds. Examples include remapper setup, kernel-base discovery, CR3 discovery, mapped-window enumeration, or a probe read.
These gates are not primarily authentication. They are readiness checks. They prevent the user-mode client from treating a partially initialized backend as operational. In security reviews, they should be separated from authorization gates, because their purpose is correctness rather than access control.
Conclusion:
Gates and handshakes in kernel drivers are best understood as vendor-defined trust mechanisms layered on top of Windows I/O security.
Microsoft provides the foundational controls: object security, access masks, token-based authorization, service permissions, driver-signing policy, and framework support. Vendor gates are used when a driver’s intended policy is narrower or more application-specific than those controls as deployed.
Their technical meaning depends on implementation. A magic constant is packet identification. A post-open authorization IOCTL is session establishment. An open-time handshake is handle creation policy. A caller-image check is application binding. A registry allowlist is configurable policy. A readiness probe is capability validation.
For security analysis, the central question is not whether a gate exists, but what it proves, where the trust anchor resides, whether authorization is tied to the correct object, and whether the privileged IOCTL surface remains safe after the gate is passed.
