Parsing Direct3D shader bytecode

末蓝、 2022-12-20 01:56 257阅读 0赞

# [http://timjones.io/blog/archive/2015/09/02/parsing-direct3d-shader-bytecode][http_timjones.io_blog_archive_2015_09_02_parsing-direct3d-shader-bytecode] #

# Parsing Direct3D shader bytecode #

02 September 2015

## TL/DR ##

In this post, I will discuss the Direct3D shader bytecode format, and how to parse it without using the Direct3D API. This is basically a write-up of [SlimShader][], a .NET library that can read in Direct3D 10 / 11 HLSL bytecode and give you back the shader instructions and metadata as nicely structured objects.

*  [HLSL and Shader Model 4 / 5 assembly][HLSL and Shader Model 4 _ 5 assembly]
 *  [Parsing shader bytecode][]
 *  [RDEF chunk][]
 *  [ISGN chunk][]
 *  [OSGN chunk][]
 *  [SHDR chunk][]
 *  [STAT chunk][]
 *  [Other chunk types][]
 *  [Conclusion][]

## Warning: niche content ahead ##

This topic is not for everyone. More accurately, very few people will ever need to know any of this. The details of Direct3D’s shader bytecode format are really only useful to Microsoft and their hardware partners.

So why am I interested in understanding how HLSL shader bytecode works? The simple answer is: [because it’s there][because it_s there].

The longer answer is: I got started on all of this while writing a software rasteriser - [Rasterizr][] \- and I wanted to use real HLSL shaders in the software rasterisation pipeline. I didn’t want to take a dependency on Direct3D - so instead, I figured out how to disassemblecompiled HLSL shaders back to assembly instructions, and then I wrote a [virtual machine that can execute these assembly instructions][] entirely on the CPU. I implemented both an interpreter and a [JIT compiler][] (the JIT compiler actually has pretty good performance, although obviously it’s still orders of magnitude slower than a GPU). Admittedly, that was a huge amount of effort for what is mostly a pointless (albeit fun) project. Here’s a screenshot of the debugging GUI I built for Rasterizr.

![50768fa45426168c55b51aaa5770247b.png][]

I intentionally modelled Rasterizr’s API (and debugging tools) on Direct3D - I wanted Rasterizr to serve as an educational tool, to help me (and perhaps others) understand what’s going on under the bonnet (or “hood”, if you speak [Simplified English][]) in Direct3D.

Something vaguely useful did come out of my work on SlimShader - [HlslUnit][], a unit testing library for HLSL code.

If you’re still reading, I’ll assume you’re okay with the niche-ness of this topic. Let’s get started.

## HLSL and Shader Model 4 / 5 assembly ##

The following code is a simple, but complete, HLSL vertex shader. We’ll be following this shader through compilation to binary bytecode. Then we’ll figure out how to parse the resulting binary, and extract the assembly instructions and metadata back out.

*(A brief aside: we’ll stop our journey at bytecode; GPUs often have their own [intermediate][]and assembly languages, but that differs from GPU to GPU and is lower-level than we’ll get into here.)*

struct VertexShaderInput
    {
        float3 pos : POSITION;
        float2 tex : TEXCOORD;
    };
    
    struct PixelShaderInput
    {
        float4 pos : SV_POSITION;
        float2 tex : TEXCOORD;
    };
    
    float4x4 WorldViewProjection;
    
    PixelShaderInput VS(VertexShaderInput input)
    {
        PixelShaderInput output;
    
        output.pos = mul(float4(input.pos, 1), WorldViewProjection);
        output.tex = input.tex;
    
        return output;
    }

Now let’s compile this shader using `fxc.exe`, the HLSL shader compiler:

fxc.exe /Fo TestShader.o /T vs_4_0 /E VS /nologo TestShader.hlsl

The `/T` parameter sets the [target profile][] to `vs_4_0`, and the `/E` parameter sets the entry point function to `VS`.

Running that will compile the shader into `TestShader.o`, a binary file. This is the file that we’re going to be delving into.

## Parsing shader bytecode ##

`TestShader.o` contains the compiled shader, and it’s what you’d use in your game when calling [CreateVertexShader][] or [CreatePixelShader][].

Direct3D has a [D3DDisassemble][] function which takes this compiled shader (as a byte array), and disassembles it back to the assembly instructions. Almost all of the time, that’s what you should use, if that’s what you want to do.

But that’s not what we want to do; we want to understand how this file is structured, and parse it ourselves. Let’s delve into this compiled shader file. Since this is a binary file, we’ll need a tool that helps us view binary files. I use [Binary Viewer][].

This is what Binary Viewer looks like after I open `TestShader.o`:

[![1305e3b1c944916da634d7003b4bb123.png][]][1305e3b1c944916da634d7003b4bb123.png 1]

I’ve made the assumption that if you’ve got this far, you’re familiar with binary files - if not, have a look at [Wikipedia’s entry on binary files][Wikipedia_s entry on binary files].

Binary Viewer shows the raw bytes on the left, and the ASCII representation of those byteson the right. Not all byte values are valid ASCII, which is why most of the values on the right use the `◦` placeholder. But the ASCII values that *are* present are interesting.

At the very beginning of the file, the first four bytes are `0x44 0x58 0x42 0x43`. Those are the ASCII codes for the string “DXBC”. It turns out that every compiled HLSL shader starts with these four bytes. Looking at the rest of the file, we can see a number of bits of text that look relevant - we’ll get to those soon.

If you compile enough different shaders, and compare / contrast the resulting binaries, you see some patterns start to emerge. That’s the key to understanding how binary files like these are structured (unless of course someone helpfully provides you with the spec, but where would the fun be in that?).

It turns out that compiled HLSL shaders are composed of a header, followed by a number ofchunks. This is hinted at by the ASCII strings at the start of each chunk - in this case “RDEF”, “ISGN”, “OSGN”, “SHDR”, and “STAT”. You can see the chunks more clearly if I overlay each one with a different colour:

![7aa269907497edcd21836173e5392046.png][]

The uncoloured section at the beginning of the file is the header. The header contains information we’ll need to successfully parse the file, such as the location of the start of each chunk.

Here’s the structure of the header, including the specific values for our example shader. In the following tables, “byte offset” always means from the start of the file.

<table> 
 <thead> 
  <tr> 
   <th>Byte offset</th> 
   <th>Value</th> 
   <th>Description</th> 
  </tr> 
 </thead> 
 <tbody> 
  <tr> 
   <td>0-3</td> 
   <td>“DXBC”</td> 
   <td>Always the string “DXBC”</td> 
  </tr> 
  <tr> 
   <td>4-19</td> 
   <td>1331666272<br> 888713072<br> 3052922206<br> 3470416626</td> 
   <td>Checksum; the HLSL compiler uses a private algorithm to compute this</td> 
  </tr> 
  <tr> 
   <td>20-23</td> 
   <td>1</td> 
   <td>Always the number “1” - I don’t know why this exists, perhaps as a sanity check</td> 
  </tr> 
  <tr> 
   <td>24-27</td> 
   <td>848</td> 
   <td>Total size, in bytes, of the compiled shader, including the header</td> 
  </tr> 
  <tr> 
   <td>28-31</td> 
   <td>5</td> 
   <td>Chunk count</td> 
  </tr> 
 </tbody> 
</table>

Next up is the chunk index. For each chunk, you get the offset (in bytes) of the start of that chunk. Our example shader has 5 chunks, so we have 5 entries in the chunk index. To aid in understanding, I’ve added the chunk type as well, but this isn’t part of the chunk index; the chunk type is declared at the start of each chunk.

<table> 
 <thead> 
  <tr> 
   <th>Byte offset</th> 
   <th>Byte address of start of chunk</th> 
   <th>Chunk type</th> 
  </tr> 
 </thead> 
 <tbody> 
  <tr> 
   <td>32-35</td> 
   <td>52</td> 
   <td>RDEF</td> 
  </tr> 
  <tr> 
   <td>36-39</td> 
   <td>268</td> 
   <td>ISGN</td> 
  </tr> 
  <tr> 
   <td>40-43</td> 
   <td>352</td> 
   <td>OSGN</td> 
  </tr> 
  <tr> 
   <td>44-47</td> 
   <td>440</td> 
   <td>SHDR</td> 
  </tr> 
  <tr> 
   <td>48-51</td> 
   <td>724</td> 
   <td>STAT</td> 
  </tr> 
 </tbody> 
</table>

I keep using the word “chunk”, but what exactly does that mean? Chunks are used to break the data in the compiled shader into logical parts. Direct3D (probably) doesn’t always need to load every chunk; I’d guess, for example, that it doesn’t load the STAT chunk unless you use shader reflection APIs. Here are all the chunk types that I’ve seen:

<table> 
 <thead> 
  <tr> 
   <th>Chunk type</th> 
   <th>Description</th> 
  </tr> 
 </thead> 
 <tbody> 
  <tr> 
   <td>ICFE</td> 
   <td>Interface. Describes any&nbsp;<code>interface</code>s, and implementing&nbsp;<code>class</code>es, present in the source HLSL.</td> 
  </tr> 
  <tr> 
   <td>ISGN</td> 
   <td>Input signature</td> 
  </tr> 
  <tr> 
   <td>OSG5</td> 
   <td>Output signature (SM5)</td> 
  </tr> 
  <tr> 
   <td>OSGN</td> 
   <td>Output signature</td> 
  </tr> 
  <tr> 
   <td>PCSG</td> 
   <td>Patch constant signature</td> 
  </tr> 
  <tr> 
   <td>RDEF</td> 
   <td>Resource definition. Describes constant buffers and resource bindings.</td> 
  </tr> 
  <tr> 
   <td>SDGB</td> 
   <td>Shader debugging info (old-style)</td> 
  </tr> 
  <tr> 
   <td>SFI0</td> 
   <td>Not really sure… it stores a value that indicates whether double-precision floating point operations are enabled, but I don’t know why that needs to be in its own chunk</td> 
  </tr> 
  <tr> 
   <td>SHDR</td> 
   <td>Shader (SM4). The shader itself.</td> 
  </tr> 
  <tr> 
   <td>SHEX</td> 
   <td>Shader (SM5)</td> 
  </tr> 
  <tr> 
   <td>SPDB</td> 
   <td>Shader debugging info (new-style)</td> 
  </tr> 
  <tr> 
   <td>STAT</td> 
   <td>Statistics. Useful statistics about the shader, such as instruction count, declaration count, etc.</td> 
  </tr> 
 </tbody> 
</table>

[SlimShader][] exposes all of the data in all of these chunks (except the debugging chunks) through a nice friendly object model. Direct3D exposes some of this data through the [`ID3D11ShaderReflection` interface][ID3D11ShaderReflection_ interface].

I figured out a lot of this stuff the hard way: I compiled multiple shaders, and looked for similarities and differences in the resulting binaries. I also got help from:

*  The [Nuclex Framework][], in particular the [HlslShaderReflector class][], was very helpful when figuring out the RDEF, ISGN and OSGN chunks.
 *  The [Wine][] project, in particular Wine’s [shader reflection code][], had some good tips for decoding the STAT chunk.
 *  For the SHDR chunk, I mostly just used `D3D11TokenizedProgramFormat.hpp`, a header file that comes with the Windows Driver Kit (WDK).

Now that we’ve got an overview of how the file is structured, let’s turn our attention to parsing the individual chunks. I’m going to describe in detail the 5 chunks in our example compiled shader. If you want more details on the other chunk types, the [SlimShader source code][] is a good place to start.

## RDEF chunk ##

![17f9638cfdc0e5c6dee03157fecfb73a.png][]

*  [ResourceDefinitionChunk.cs][]

The first chunk in our example compiled shader is of type RDEF, or “resource definition”. Resource definition chunks are split into two main parts:

1.  Constant buffer descriptions. In our example, we have a single constant buffer named `$Globals`, which contains a single variable named `WorldViewProjection`, which is a 4x4 matrix.
2.  Resource bindings. In our example, we only have a single constant buffer, but this is where you would find bindings for textures, samplers, structured buffers, etc.

Looking at the chunk index table above, we see that the byte address of the start of the first chunk is 52. Using Binary Viewer, we start looking at the byte values starting at index 52. (If you were doing this from scratch, you’d have to painstakingly compare / contrast shader binaries until you could guess at what each value means.) Note that most of these values are stored in the binary file as 32-bit `uint`s, so we mostly, but not always, need to read 4 bytes at a time.

<table> 
 <thead> 
  <tr> 
   <th>Byte offset</th> 
   <th>Value</th> 
   <th>Description</th> 
  </tr> 
 </thead> 
 <tbody> 
  <tr> 
   <td>52-55</td> 
   <td>“RDEF”</td> 
   <td>Indicates that this is a RDEF (resource definition) chunk</td> 
  </tr> 
  <tr> 
   <td>56-59</td> 
   <td>208</td> 
   <td>Length of the chunk, in bytes</td> 
  </tr> 
  <tr> 
   <td>60-63</td> 
   <td>1</td> 
   <td>Constant buffer count</td> 
  </tr> 
  <tr> 
   <td>64-67</td> 
   <td>72</td> 
   <td>Byte offset (from start of chunk data, after chunk type and chunk length) to first constant buffer description</td> 
  </tr> 
  <tr> 
   <td>68-71</td> 
   <td>1</td> 
   <td>Resource binding count</td> 
  </tr> 
  <tr> 
   <td>72-75</td> 
   <td>28</td> 
   <td>Byte offset (from start of chunk data) to first resource binding description</td> 
  </tr> 
  <tr> 
   <td>76</td> 
   <td>0</td> 
   <td>Minor version number</td> 
  </tr> 
  <tr> 
   <td>77</td> 
   <td>4</td> 
   <td>Major version number</td> 
  </tr> 
  <tr> 
   <td>78-79</td> 
   <td>0xFFFE</td> 
   <td>Program type. “0xFFFE” means vertex shader.</td> 
  </tr> 
  <tr> 
   <td>80-83</td> 
   <td>256</td> 
   <td>Flags. “256” means “NoPreshader”.</td> 
  </tr> 
  <tr> 
   <td>84-87</td> 
   <td>156</td> 
   <td>Byte offset (from start of chunk data) to “creator” string</td> 
  </tr> 
 </tbody> 
</table>

*In this blog post, I’m only describing Shader Model 4.0 shaders. Shader Model 5.0 shaders contain some additional data; SlimShader supports both SM 4.0 and SM 5.0.*

A number of values in the data that follows are byte offsets from the “start of chunk data”. The start of the chunk data is the byte immediately following the chunk length. In this RDEF chunk, that is the byte at offset `60` from the beginning of the file.

Now we’ve parsed the “header” portion of the RDEF chunk. It’s time to parse the “body”, which is composed of constant buffer and resource binding descriptions. We know from the header that the first constant buffer description can be found at byte address `60 + 72 = 135`(`60` is the address of the start of the chunk data, and `72` is the offset to the first constant buffer). So let’s see what’s in the constant buffer description:

<table> 
 <thead> 
  <tr> 
   <th>Byte offset</th> 
   <th>Value</th> 
   <th>Description</th> 
  </tr> 
 </thead> 
 <tbody> 
  <tr> 
   <td>132-135</td> 
   <td>60</td> 
   <td>Byte offset (from start of chunk data) to constant buffer name</td> 
  </tr> 
  <tr> 
   <td>136-139</td> 
   <td>1</td> 
   <td>Variable count</td> 
  </tr> 
  <tr> 
   <td>140-143</td> 
   <td>96</td> 
   <td>Byte offset (from start of chunk data) to first variable description</td> 
  </tr> 
  <tr> 
   <td>144-147</td> 
   <td>64</td> 
   <td>Size of the constant buffer, in bytes</td> 
  </tr> 
  <tr> 
   <td>148-151</td> 
   <td>0</td> 
   <td>Flags, based on&nbsp;<a href="https://msdn.microsoft.com/en-us/library/windows/desktop/ff728729%28v=vs.85%29.aspx" rel="nofollow">D3D_SHADER_CBUFFER_FLAGS</a></td> 
  </tr> 
  <tr> 
   <td>152-155</td> 
   <td>0</td> 
   <td>Constant buffer type, based on&nbsp;<a href="https://msdn.microsoft.com/en-us/library/windows/desktop/ff476097%28v=vs.85%29.aspx" rel="nofollow">D3D11_CBUFFER_TYPE</a></td> 
  </tr> 
 </tbody> 
</table>

According to the table above, the constant buffer name will be found at `60 + 60 = 120`. Strings are null-terminated, so we keep reading until we reach a `0`:

<table> 
 <thead> 
  <tr> 
   <th>Byte offset</th> 
   <th>Value</th> 
   <th>Description</th> 
  </tr> 
 </thead> 
 <tbody> 
  <tr> 
   <td>120</td> 
   <td>“$Globals”</td> 
   <td>Constant buffer name</td> 
  </tr> 
 </tbody> 
</table>

Now let’s parse the variables for this constant buffer. In this case, there is only one constant buffer, which contains one variable. We know from the table above that the variable description starts at `60 + 96 = 156`.

<table> 
 <thead> 
  <tr> 
   <th>Byte offset</th> 
   <th>Value</th> 
   <th>Description</th> 
  </tr> 
 </thead> 
 <tbody> 
  <tr> 
   <td>156-159</td> 
   <td>120</td> 
   <td>Byte offset (from start of chunk data) to variable name</td> 
  </tr> 
  <tr> 
   <td>160-163</td> 
   <td>0</td> 
   <td>Offset from start of constant buffer (in bytes)</td> 
  </tr> 
  <tr> 
   <td>164-167</td> 
   <td>64</td> 
   <td>Variable size (in bytes)</td> 
  </tr> 
  <tr> 
   <td>168-171</td> 
   <td>2</td> 
   <td>Variable flags, based on&nbsp;<a href="https://msdn.microsoft.com/en-us/library/Bb172442%28v=VS.85%29.aspx" rel="nofollow">D3D10_SHADER_VARIABLE_FLAGS</a>.&nbsp;<code>2</code>&nbsp;means this variable is used in the shader.</td> 
  </tr> 
  <tr> 
   <td>172-175</td> 
   <td>140</td> 
   <td>Byte offset (from start of chunk data) to variable type</td> 
  </tr> 
  <tr> 
   <td>176-179</td> 
   <td>0</td> 
   <td>Byte offset (from start of chunk data) to default value. “0” means there isn’t a default value.</td> 
  </tr> 
 </tbody> 
</table>

Let’s parse the variable name, at address `60 + 120 = 180`:

<table> 
 <thead> 
  <tr> 
   <th>Byte offset</th> 
   <th>Value</th> 
   <th>Description</th> 
  </tr> 
 </thead> 
 <tbody> 
  <tr> 
   <td>180</td> 
   <td>“WorldViewProjection”</td> 
   <td>Variable name</td> 
  </tr> 
 </tbody> 
</table>

And then the variable type, at address `60 + 140 = 200`:

<table> 
 <thead> 
  <tr> 
   <th>Byte offset</th> 
   <th>Value</th> 
   <th>Description</th> 
  </tr> 
 </thead> 
 <tbody> 
  <tr> 
   <td>200-201</td> 
   <td>3</td> 
   <td>Variable class, based on&nbsp;<a href="https://msdn.microsoft.com/en-us/library/bb172440%28v=vs.85%29.aspx" rel="nofollow">D3D10_SHADER_VARIABLE_CLASS</a>. “3” means the variable is a column-major matrix.</td> 
  </tr> 
  <tr> 
   <td>202-203</td> 
   <td>3</td> 
   <td>Variable type, based on&nbsp;<a href="https://msdn.microsoft.com/en-us/library/bb172443%28v=vs.85%29.aspx" rel="nofollow">D3D10_SHADER_VARIABLE_TYPE</a>. “3” means the variable is a floating-point number.</td> 
  </tr> 
  <tr> 
   <td>204-205</td> 
   <td>4</td> 
   <td>Number of rows in matrix-type variables</td> 
  </tr> 
  <tr> 
   <td>206-207</td> 
   <td>4</td> 
   <td>Number of columns in matrix-type variables</td> 
  </tr> 
  <tr> 
   <td>208-209</td> 
   <td>0</td> 
   <td>Number of elements in array-type variables</td> 
  </tr> 
  <tr> 
   <td>210-211</td> 
   <td>0</td> 
   <td>Number of members in structure-type variables</td> 
  </tr> 
  <tr> 
   <td>212-213</td> 
   <td>0</td> 
   <td>Byte offset (from start of chunk data) to first member</td> 
  </tr> 
 </tbody> 
</table>

Whew! As you can see, there are a *lot* of different structures contained in HLSL shader binaries. And this is just the RDEF chunk type! We’re nearly there though. Next, let’s parsethe resource binding description. Going back to the first table in this section, we see that the first resource binding description can be found at address `60 + 28 = 88`.

<table> 
 <thead> 
  <tr> 
   <th>Byte offset</th> 
   <th>Value</th> 
   <th>Description</th> 
  </tr> 
 </thead> 
 <tbody> 
  <tr> 
   <td>88-91</td> 
   <td>60</td> 
   <td>Byte offset (from start of chunk data) to resource binding name</td> 
  </tr> 
  <tr> 
   <td>92-95</td> 
   <td>0</td> 
   <td>Shader input type. “0” means&nbsp;<code>cbuffer</code>.</td> 
  </tr> 
  <tr> 
   <td>96-99</td> 
   <td>0</td> 
   <td>Resource return type. “0” means not applicable.</td> 
  </tr> 
  <tr> 
   <td>100-103</td> 
   <td>0</td> 
   <td>Resource view dimension. “0” means not applicable.</td> 
  </tr> 
  <tr> 
   <td>104-107</td> 
   <td>0</td> 
   <td>Number of samples</td> 
  </tr> 
  <tr> 
   <td>108-111</td> 
   <td>0</td> 
   <td>Bind point</td> 
  </tr> 
  <tr> 
   <td>112-115</td> 
   <td>1</td> 
   <td>Bind count</td> 
  </tr> 
  <tr> 
   <td>116-119</td> 
   <td>0</td> 
   <td>Shader input flags</td> 
  </tr> 
 </tbody> 
</table>

One interesting thing to note is that the data is very efficiently packed in the binary file. There are very few, if any, wasted bytes. You would expect nothing less, but still, it’s nice to see this attention to detail firsthand.

We have one last thing to parse from the RDEF chunk - the “creator”. We can find this at address `60 + 156 = 216`:

<table> 
 <thead> 
  <tr> 
   <th>Byte offset</th> 
   <th>Value</th> 
   <th>Description</th> 
  </tr> 
 </thead> 
 <tbody> 
  <tr> 
   <td>216</td> 
   <td>“Microsoft (R) HLSL Shader Compiler 6.3.9600.16384”</td> 
   <td>Creator</td> 
  </tr> 
 </tbody> 
</table>

After all of that work, we know enough to reconstruct this line from our original HLSL code:

float4x4 WorldViewProjection;

But we also know a bit more than that; we know that:

*  the HLSL compiler automatically put this variable into a `cbuffer` called `$Globals`
 *  it is a column-major matrix
 *  it is referenced in the shader

## ISGN chunk ##

![f703e508994e5097a3e9911b9c3833ab.png][]

*  [InputSignatureChunk.cs][]

ISGN chunks define the input signature for a shader. For a vertex shader like this one, the input signature describes the shape of the data coming from the application.

ISGN chunks are much simpler than RDEF chunks. Here’s how the ISGN header is structured:

<table> 
 <thead> 
  <tr> 
   <th>Byte offset</th> 
   <th>Value</th> 
   <th>Description</th> 
  </tr> 
 </thead> 
 <tbody> 
  <tr> 
   <td>268-271</td> 
   <td>“ISGN”</td> 
   <td>Indicates that this is a RDEF (resource definition) chunk</td> 
  </tr> 
  <tr> 
   <td>272-275</td> 
   <td>76</td> 
   <td>Length of the chunk, in bytes</td> 
  </tr> 
  <tr> 
   <td>276-279</td> 
   <td>2</td> 
   <td>Element count</td> 
  </tr> 
  <tr> 
   <td>280-283</td> 
   <td>8</td> 
   <td>I don’t know what this is used for…</td> 
  </tr> 
 </tbody> 
</table>

The element data immediately follows the ISGN header. We have 2 elements, with these values:

<table> 
 <thead> 
  <tr> 
   <th>Byte offset</th> 
   <th>Value</th> 
   <th>Description</th> 
  </tr> 
 </thead> 
 <tbody> 
  <tr> 
   <td>284-287</td> 
   <td>56</td> 
   <td>Byte offset (from start of chunk data) to element name</td> 
  </tr> 
  <tr> 
   <td>288-291</td> 
   <td>0</td> 
   <td>Semantic index</td> 
  </tr> 
  <tr> 
   <td>292-295</td> 
   <td>0</td> 
   <td>System value type</td> 
  </tr> 
  <tr> 
   <td>296-299</td> 
   <td>3</td> 
   <td>Component type. “3” means floating-point</td> 
  </tr> 
  <tr> 
   <td>300-303</td> 
   <td>0</td> 
   <td>Register</td> 
  </tr> 
  <tr> 
   <td>304</td> 
   <td>7</td> 
   <td>Mask. “7” means XYZ, which means this is a 3-component vector.</td> 
  </tr> 
  <tr> 
   <td>305</td> 
   <td>7</td> 
   <td>Read-write mask. This is the same as the mask, but it is also possible for not all components to actually be used by the shader.</td> 
  </tr> 
 </tbody> 
</table>

<table> 
 <thead> 
  <tr> 
   <th>Byte offset</th> 
   <th>Value</th> 
   <th>Description</th> 
  </tr> 
 </thead> 
 <tbody> 
  <tr> 
   <td>332</td> 
   <td>“POSITION”</td> 
   <td>Element name</td> 
  </tr> 
 </tbody> 
</table>

<table> 
 <thead> 
  <tr> 
   <th>Byte offset</th> 
   <th>Value</th> 
   <th>Description</th> 
  </tr> 
 </thead> 
 <tbody> 
  <tr> 
   <td>308-311</td> 
   <td>65</td> 
   <td>Byte offset (from start of chunk data) to element name</td> 
  </tr> 
  <tr> 
   <td>312-315</td> 
   <td>0</td> 
   <td>Semantic index</td> 
  </tr> 
  <tr> 
   <td>316-319</td> 
   <td>0</td> 
   <td>System value type</td> 
  </tr> 
  <tr> 
   <td>320-323</td> 
   <td>3</td> 
   <td>Component type. “3” means floating-point</td> 
  </tr> 
  <tr> 
   <td>324-327</td> 
   <td>1</td> 
   <td>Register</td> 
  </tr> 
  <tr> 
   <td>328</td> 
   <td>3</td> 
   <td>Mask. “3” means XY, which means this is a 2-component vector.</td> 
  </tr> 
  <tr> 
   <td>329</td> 
   <td>3</td> 
   <td>Read-write mask</td> 
  </tr> 
 </tbody> 
</table>

<table> 
 <thead> 
  <tr> 
   <th>Byte offset</th> 
   <th>Value</th> 
   <th>Description</th> 
  </tr> 
 </thead> 
 <tbody> 
  <tr> 
   <td>341</td> 
   <td>“TEXCOORD”</td> 
   <td>Element name</td> 
  </tr> 
 </tbody> 
</table>

And that’s it! The input signature chunk tells us the structure of data coming into the shader:

*  A `float3` parameter whose semantic is `POSITION`, assigned to register `0`
 *  A `float2` parameter whose semantic is `TEXCOORD`, assigned to register `1`

Compare this to the original HLSL:

struct VertexShaderInput
    {
        float3 pos : POSITION;
        float2 tex : TEXCOORD;
    };

We’ve lost some information - such as the name of the `struct`, and the names of the `struct`fields. Those names aren’t needed by the GPU. Instead, the HLSL compiler has assigned register numbers to these two values.

## OSGN chunk ##

![061c4492b5a45e3eb2b6d1984dc1a061.png][]

*  [OutputSignatureChunk.cs][]

OSGN chunks define the output signature for a shader. For a vertex shader like this, the output signature describes the shape of the data being output from the vertex shader to the pixel shader (or geometry shader, or hull shader).

Happily, OSGN chunks are structured in exactly the same way as ISGN chunks. So I’ll skip to the end and show you the parsed information. The output signature for this example shader is:

*  A `float4` parameter whose semantic is `SV_POSITION`, assigned to register `0`
 *  A `float2` parameter whose semantic is `TEXCOORD`, assigned to register `1`

Again, compare that to the original HLSL:

struct PixelShaderInput
    {
        float4 pos : SV_POSITION;
        float2 tex : TEXCOORD;
    };

We’ve kept the important information, and the HLSL compiler has assigned registers to each value in the output.

## SHDR chunk ##

![2760405cacef139d1702f8ef9c990c9e.png][]

*  [ShaderProgramChunk.cs][]

The SHDR chunk is, as you might guess, where the actual shader can be found. The other chunks are basically metadata; the SHDR chunk is where you find the declarations and instructions that make up the shader itself.

It is, by far, the most complicated chunk to parse. But fortunately, it’s also the (only) chunk type that is publicly documented. If you install the [Windows Driver Kit (WDK)][Windows Driver Kit _WDK], you get a file that describes the binary format in some detail, usually located here:

C:\Program Files (x86)\Windows Kits\10\Include\10.0.10240.0\um\d3d11TokenizedProgramFormat.hpp

Because the documentation for the binary format of the SHDR chunk is readily available, I won’t go into too much detail.

Briefly, the SHDR chunk, in common with the other chunk types, is composed of a header and a body. The header is structured as follows:

<table> 
 <thead> 
  <tr> 
   <th>Byte offset</th> 
   <th>Value</th> 
   <th>Description</th> 
  </tr> 
 </thead> 
 <tbody> 
  <tr> 
   <td>440-443</td> 
   <td>“SHDR”</td> 
   <td>Indicates that this is a SHDR (shader) chunk</td> 
  </tr> 
  <tr> 
   <td>444-447</td> 
   <td>276</td> 
   <td>Length of the chunk, in bytes</td> 
  </tr> 
  <tr> 
   <td>448 (low 4 bits)</td> 
   <td>0</td> 
   <td>Minor version number</td> 
  </tr> 
  <tr> 
   <td>448 (high 4 bits)</td> 
   <td>4</td> 
   <td>Major version number</td> 
  </tr> 
  <tr> 
   <td>450-451</td> 
   <td>1</td> 
   <td>Program type. “1” means vertex shader.</td> 
  </tr> 
  <tr> 
   <td>452-455</td> 
   <td>69</td> 
   <td>Number of DWORDs in the chunk</td> 
  </tr> 
 </tbody> 
</table>

After the header come a sequence of “opcodes”. Opcodes are the binary representation of shader assembly instructions. [You can see the full list of Shader Model 5 assembly instructions here][]. Opcodes represent both declarations (i.e. texture declarations, constant buffer declarations, etc.) and instructions (i.e. add, multiply, etc.).

Each opcode has its own binary structure, but there are some commonalities. Here’s the section of `d3d11TokenizedProgramFormat.hpp` that tells us how to read the first token of each opcode.

// ----------------------------------------------------------------------------
    // Opcode Format (OpcodeToken0)
    //
    // [10:00] D3D10_SB_OPCODE_TYPE
    // if( [10:00] == D3D10_SB_OPCODE_CUSTOMDATA )
    // {
    //    Token starts a custom-data block.  See "Custom-Data Block Format".
    // }
    // else // standard opcode token
    // {
    //    [23:11] Opcode-Specific Controls
    //    [30:24] Instruction length in DWORDs including the opcode token.
    //    [31]    0 normally. 1 if extended operand definition, meaning next DWORD
    //            contains extended opcode token.
    // }
    //
    // ----------------------------------------------------------------------------

So every opcode stores the opcode type in the first 10 bits. Once you’ve read the type, you use that to know how to parse the rest of the opcode.

As an example, let’s look at the first opcode from our example shader.

<table> 
 <thead> 
  <tr> 
   <th>Byte offset</th> 
   <th>Value</th> 
   <th>Description</th> 
  </tr> 
 </thead> 
 <tbody> 
  <tr> 
   <td>456-459 (bits 0 to 10)</td> 
   <td>89</td> 
   <td>Opcode type</td> 
  </tr> 
  <tr> 
   <td>456-459 (bits 24 to 30)</td> 
   <td>4</td> 
   <td>Opcode length</td> 
  </tr> 
  <tr> 
   <td>456-459 (bit 31)</td> 
   <td>0</td> 
   <td>1 if opcode is “extended”, otherwise 0</td> 
  </tr> 
 </tbody> 
</table>

Looking at `D3D10_SB_OPCODE_TYPE` in `d3d11TokenizedProgramFormat.hpp`, we see that opcode type 89 equates to `D3D10_SB_OPCODE_DCL_CONSTANT_BUFFER`. So then we look further down the file to find how constant buffer declaration opcodes are structured:

// ----------------------------------------------------------------------------
    // Constant Buffer Declaration
    //
    // OpcodeToken0:
    //
    // [10:00] D3D10_SB_OPCODE_DCL_CONSTANT_BUFFER
    // [11]    D3D10_SB_CONSTANT_BUFFER_ACCESS_PATTERN
    // [23:12] Ignored, 0
    // [30:24] Instruction length in DWORDs including the opcode token.
    // [31]    0 normally. 1 if extended operand definition, meaning next DWORD
    //         contains extended operand description.  This dcl is currently not
    //         extended.
    //
    // OpcodeToken0 is followed by 1 operand:
    // (1) Operand, starting with OperandToken0, defining which CB slot (cb#[size])
    //     is being declared. (operand type: D3D10_SB_OPERAND_TYPE_CONSTANT_BUFFER)
    //     The indexing dimension for the register must be 
    //     D3D10_SB_OPERAND_INDEX_DIMENSION_2D, where the first index specifies
    //     which cb#[] is being declared, and the second (array) index specifies the size 
    //     of the buffer, as a count of 32-bit*4 elements.  (As opposed to when the 
    //     cb#[] is used in shader instructions, and the array index represents which 
    //     location in the constant buffer is being referenced.)
    //     If the size is specified as 0, the CB size is not known (any size CB
    //     can be bound to the slot).
    //
    // The order of constant buffer declarations in a shader indicates their
    // relative priority from highest to lowest (hint to driver).
    // 
    // ----------------------------------------------------------------------------

So for a constant buffer declaration, after the initial opcode token, we can expect to find a single operand, which defines which constant buffer slot is being declared. Operands are actually fairly complicated to parse. On the bright side, there’s a standard operand structure shared across all opcodes. But because it’s too hard (and I’m lazy), I’ll just point you at the [Operand][] source code in [SlimShader][].

If we were to parse the operand, we’d be able to extract this assembly instruction:

dcl_constantbuffer cb0[4], immediateIndexed

We can then continue to parse all the remaining opcodes in the shader chunk. When writing SlimShader, I went through all the opcodes documented in `d3d11TokenizedProgramFormat.hpp`, and wrote the parsing logic for each one. For our example shader, we’d end up with these assembly instructions:

dcl_constantbuffer cb0[4], immediateIndexed
    dcl_input v0.xyz
    dcl_input v1.xy
    dcl_output_siv o0.xyzw, position
    dcl_output o1.xy
    dcl_temps 1
    mov r0.xyz, v0.xyzx
    mov r0.w, l(1.000000)
    dp4 o0.x, r0.xyzw, cb0[0].xyzw
    dp4 o0.y, r0.xyzw, cb0[1].xyzw
    dp4 o0.z, r0.xyzw, cb0[2].xyzw
    dp4 o0.w, r0.xyzw, cb0[3].xyzw
    mov o1.xy, v1.xyxx
    ret

There are 6 declarations, and 8 instructions. When you use a shader in Direct3D, these assembly instructions are parsed from the bytecode (much like we’re doing here), and sent to the GPU. Somewhere along the way (presumably in the GPU driver), the Direct3D assembly instructions will be compiled (again) into the GPU’s own assembly language, which is lower-level and hardware-specific.

In older versions of Direct3D, you could author shaders using assembly instructions, but that hasn’t been possible for a while. These days you need to use HLSL (which is fine by me, I have enough trouble reading assembly code, let alone writing it).

## STAT chunk ##

![6a6ea7fc4bfb75048b40c0538240111c.png][]

*  [StatisticsChunk.cs][]

The STAT chunk is the least interesting chunk type (“interesting” might be a poor choice of word in a topic as dry as this one, but work with me). I don’t imagine it’s used in the Direct3D runtime, unless you use the `ID3D11ShaderReflection` interface.

Here’s the data in the STAT chunk in our example shader. I’ve guessed at what some of these values mean, and some I just don’t know, despite comparing and contrasting multiple shader binaries.

<table> 
 <thead> 
  <tr> 
   <th>Byte offset</th> 
   <th>Value</th> 
   <th>Description</th> 
  </tr> 
 </thead> 
 <tbody> 
  <tr> 
   <td>724-727</td> 
   <td>“STAT”</td> 
   <td>Indicates that this is a STAT (statistics) chunk</td> 
  </tr> 
  <tr> 
   <td>728-731</td> 
   <td>116</td> 
   <td>Length of the chunk, in bytes</td> 
  </tr> 
  <tr> 
   <td>732-735</td> 
   <td>8</td> 
   <td>Instruction count</td> 
  </tr> 
  <tr> 
   <td>736-739</td> 
   <td>1</td> 
   <td>Temp register count</td> 
  </tr> 
  <tr> 
   <td>740-743</td> 
   <td>0</td> 
   <td>Define count</td> 
  </tr> 
  <tr> 
   <td>744-747</td> 
   <td>4</td> 
   <td>Declaration count</td> 
  </tr> 
  <tr> 
   <td>748-751</td> 
   <td>4</td> 
   <td>Float instruction count</td> 
  </tr> 
  <tr> 
   <td>752-755</td> 
   <td>0</td> 
   <td>Int instruction count</td> 
  </tr> 
  <tr> 
   <td>756-759</td> 
   <td>0</td> 
   <td>Uint instruction count</td> 
  </tr> 
  <tr> 
   <td>760-763</td> 
   <td>1</td> 
   <td>Static flow control count</td> 
  </tr> 
  <tr> 
   <td>764-767</td> 
   <td>0</td> 
   <td>Dynamic flow control count</td> 
  </tr> 
  <tr> 
   <td>768-771</td> 
   <td>0</td> 
   <td>Macro instruction count - maybe? I guessed this</td> 
  </tr> 
  <tr> 
   <td>772-775</td> 
   <td>0</td> 
   <td>Temp array count</td> 
  </tr> 
  <tr> 
   <td>776-779</td> 
   <td>0</td> 
   <td>Array instruction count</td> 
  </tr> 
  <tr> 
   <td>780-783</td> 
   <td>0</td> 
   <td>Cut instruction count</td> 
  </tr> 
  <tr> 
   <td>784-787</td> 
   <td>0</td> 
   <td>Emit instruction count</td> 
  </tr> 
  <tr> 
   <td>788-791</td> 
   <td>0</td> 
   <td>Texture normal instructions</td> 
  </tr> 
  <tr> 
   <td>792-795</td> 
   <td>0</td> 
   <td>Texture load instructions</td> 
  </tr> 
  <tr> 
   <td>796-799</td> 
   <td>0</td> 
   <td>Texture comparison instructions</td> 
  </tr> 
  <tr> 
   <td>800-803</td> 
   <td>0</td> 
   <td>Texture bias instructions</td> 
  </tr> 
  <tr> 
   <td>804-807</td> 
   <td>0</td> 
   <td>Texture gradient instructions</td> 
  </tr> 
  <tr> 
   <td>808-811</td> 
   <td>3</td> 
   <td>Mov instruction count</td> 
  </tr> 
  <tr> 
   <td>812-815</td> 
   <td>0</td> 
   <td>Movc instruction count</td> 
  </tr> 
  <tr> 
   <td>816-819</td> 
   <td>0</td> 
   <td>Conversion instruction count</td> 
  </tr> 
  <tr> 
   <td>820-823</td> 
   <td>0</td> 
   <td>? - No idea…</td> 
  </tr> 
  <tr> 
   <td>824-827</td> 
   <td>0</td> 
   <td>Input primitive for geometry shaders</td> 
  </tr> 
  <tr> 
   <td>828-831</td> 
   <td>0</td> 
   <td>Primitive topology for geometry shaders</td> 
  </tr> 
  <tr> 
   <td>832-835</td> 
   <td>0</td> 
   <td>Max output vertex count for geometry shaders</td> 
  </tr> 
  <tr> 
   <td>836-839</td> 
   <td>0</td> 
   <td>? - No idea…</td> 
  </tr> 
  <tr> 
   <td>840-843</td> 
   <td>0</td> 
   <td>? - No idea…</td> 
  </tr> 
  <tr> 
   <td>844-847</td> 
   <td>0</td> 
   <td>1 for sample frequency shader, otherwise 0</td> 
  </tr> 
 </tbody> 
</table>

This data is exposed in Direct3D through a combination of [D3D11\_SHADER\_DESC][D3D11_SHADER_DESC] and [ID3D11ShaderReflection methods][ID3D11ShaderReflection_ interface].

## Other chunk types ##

There are several other chunk types not found in our example shader. The other chunk types supported by SlimShader are:

*  IFCE - [InterfacesChunk.cs][]
 *  SFI0 - [Sfi0Chunk.cs][]
 *  PCSG - [PatchConstantSignatureChunk.cs][]

## Conclusion ##

That was pretty heavy going! But I didn’t think it was worth writing this blog post without going fairly deep into the details. If you found it useful, or have questions, or can fill in any of the gaps in my knowledge, please leave a comment!

I hope to write a follow-up post describing how I used this bytecode parser to build a virtual machine that can execute Direct3D bytecode entirely on the CPU, in managed code. Stay tuned!

[http_timjones.io_blog_archive_2015_09_02_parsing-direct3d-shader-bytecode]: http://timjones.io/blog/archive/2015/09/02/parsing-direct3d-shader-bytecode
[SlimShader]: https://github.com/tgjones/slimshader
[HLSL and Shader Model 4 _ 5 assembly]: http://timjones.io/blog/archive/2015/09/02/parsing-direct3d-shader-bytecode#hlsl-sm-assembly
[Parsing shader bytecode]: http://timjones.io/blog/archive/2015/09/02/parsing-direct3d-shader-bytecode#parsing-shader-bytecode
[RDEF chunk]: http://timjones.io/blog/archive/2015/09/02/parsing-direct3d-shader-bytecode#rdef-chunk
[ISGN chunk]: http://timjones.io/blog/archive/2015/09/02/parsing-direct3d-shader-bytecode#isgn-chunk
[OSGN chunk]: http://timjones.io/blog/archive/2015/09/02/parsing-direct3d-shader-bytecode#osgn-chunk
[SHDR chunk]: http://timjones.io/blog/archive/2015/09/02/parsing-direct3d-shader-bytecode#shdr-chunk
[STAT chunk]: http://timjones.io/blog/archive/2015/09/02/parsing-direct3d-shader-bytecode#stat-chunk
[Other chunk types]: http://timjones.io/blog/archive/2015/09/02/parsing-direct3d-shader-bytecode#other-chunk-types
[Conclusion]: http://timjones.io/blog/archive/2015/09/02/parsing-direct3d-shader-bytecode#conclusion
[because it_s there]: https://en.wikiquote.org/wiki/George_Mallory
[Rasterizr]: https://github.com/tgjones/rasterizr
[virtual machine that can execute these assembly instructions]: https://github.com/tgjones/slimshader/tree/master/src/SlimShader.VirtualMachine
[JIT compiler]: https://github.com/tgjones/slimshader/tree/master/src/SlimShader.VirtualMachine.Jitter
[50768fa45426168c55b51aaa5770247b.png]: /images/20221120/90f81a82db424798abfbbf1ab136a9f1.png
[Simplified English]: https://i.imgur.com/kn488mY.jpg
[HlslUnit]: http://timjones.tw/blog/archive/2014/01/07/introducing-hlslunit-unit-tests-for-your-hlsl-shader-code
[intermediate]: http://amd-dev.wpengine.netdna-cdn.com/wordpress/media/2012/10/AMD_Intermediate_Language_%28IL%29_Specification_v2.pdf
[target profile]: https://msdn.microsoft.com/en-us/library/windows/desktop/bb509626%28v=vs.85%29.aspx
[CreateVertexShader]: https://msdn.microsoft.com/en-us/library/windows/desktop/ff476524%28v=vs.85%29.aspx
[CreatePixelShader]: https://msdn.microsoft.com/en-us/library/windows/desktop/ff476513%28v=vs.85%29.aspx
[D3DDisassemble]: https://msdn.microsoft.com/en-us/library/windows/desktop/dd607326%28v=vs.85%29.aspx
[Binary Viewer]: http://www.proxoft.com/downloads.aspx
[1305e3b1c944916da634d7003b4bb123.png]: https://img-blog.csdnimg.cn/img_convert/1305e3b1c944916da634d7003b4bb123.png
[1305e3b1c944916da634d7003b4bb123.png 1]: http://timjones.io/assets/posts/d3d-bytecode-binary-viewer.png
[Wikipedia_s entry on binary files]: https://en.wikipedia.org/wiki/Binary_file
[7aa269907497edcd21836173e5392046.png]: https://img-blog.csdnimg.cn/img_convert/7aa269907497edcd21836173e5392046.png
[ID3D11ShaderReflection_ interface]: https://msdn.microsoft.com/en-us/library/windows/desktop/ff476590%28v=vs.85%29.aspx
[Nuclex Framework]: https://devel.nuclex.org/framework
[HlslShaderReflector class]: https://devel.nuclex.org/framework/browser/graphics/Nuclex.Graphics.Native/trunk/Source/Introspection/HlslShaderReflector.cpp
[Wine]: https://github.com/mirrors/wine
[shader reflection code]: http://source.winehq.org/source/dlls/d3dcompiler_43/reflection.c
[SlimShader source code]: https://github.com/tgjones/slimshader/tree/master/src/SlimShader/Chunks
[17f9638cfdc0e5c6dee03157fecfb73a.png]: https://img-blog.csdnimg.cn/img_convert/17f9638cfdc0e5c6dee03157fecfb73a.png
[ResourceDefinitionChunk.cs]: https://github.com/tgjones/slimshader/blob/master/src/SlimShader/Chunks/Rdef/ResourceDefinitionChunk.cs
[f703e508994e5097a3e9911b9c3833ab.png]: https://img-blog.csdnimg.cn/img_convert/f703e508994e5097a3e9911b9c3833ab.png
[InputSignatureChunk.cs]: https://github.com/tgjones/slimshader/blob/master/src/SlimShader/Chunks/Xsgn/InputSignatureChunk.cs
[061c4492b5a45e3eb2b6d1984dc1a061.png]: https://img-blog.csdnimg.cn/img_convert/061c4492b5a45e3eb2b6d1984dc1a061.png
[OutputSignatureChunk.cs]: https://github.com/tgjones/slimshader/blob/master/src/SlimShader/Chunks/Xsgn/OutputSignatureChunk.cs
[2760405cacef139d1702f8ef9c990c9e.png]: https://img-blog.csdnimg.cn/img_convert/2760405cacef139d1702f8ef9c990c9e.png
[ShaderProgramChunk.cs]: https://github.com/tgjones/slimshader/blob/master/src/SlimShader/Chunks/Shex/ShaderProgramChunk.cs
[Windows Driver Kit _WDK]: https://msdn.microsoft.com/en-us/windows/hardware/dn913721.aspx
[You can see the full list of Shader Model 5 assembly instructions here]: https://msdn.microsoft.com/en-us/library/windows/desktop/hh447232%28v=vs.85%29.aspx
[Operand]: https://github.com/tgjones/slimshader/blob/master/src/SlimShader/Chunks/Shex/Tokens/Operand.cs
[6a6ea7fc4bfb75048b40c0538240111c.png]: https://img-blog.csdnimg.cn/img_convert/6a6ea7fc4bfb75048b40c0538240111c.png
[StatisticsChunk.cs]: https://github.com/tgjones/slimshader/blob/master/src/SlimShader/Chunks/Stat/StatisticsChunk.cs
[D3D11_SHADER_DESC]: https://msdn.microsoft.com/en-us/library/windows/desktop/ff476209%28v=vs.85%29.aspx
[InterfacesChunk.cs]: https://github.com/tgjones/slimshader/blob/master/src/SlimShader/Chunks/Ifce/InterfacesChunk.cs
[Sfi0Chunk.cs]: https://github.com/tgjones/slimshader/blob/master/src/SlimShader/Chunks/Sfi0/Sfi0Chunk.cs
[PatchConstantSignatureChunk.cs]: https://github.com/tgjones/slimshader/blob/master/src/SlimShader/Chunks/Xsgn/PatchConstantSignatureChunk.cs