Call of Duty 4: d3dbsp

The .d3dbsp structure, Call of Duty 4's variant on the well-known BSP format, is rather difficult to decipher. Since it's a binary file you can't simply read it. A hexadecimal editor is the best tool in this case. I am still deciphering it myself, I will post anything I happen to get known of regarding the file format.

Be sure to read here and here before you even try to decipher it. Knowing what the technique is that BSP formatted maps use, could be handy as well. Also see Call of Duty 2: d3dbsp for more information.

Every number ending with an 'h' indicates its a number using the hexadecimal count system. For other numbers one can assume the decimal count system is used.

Header & Lump index

The file starts with 3 DWORDs: the header 'IBSP' indicating this is a BSP file originally designed by ID software. After these 4 initial bytes you find the BSP version number. CoD4 uses the version number 22 (16h), which is different from CoD2. The next 4 byte group is the amount of lumps in the current BSP. The number of lumps varies from 23 to 37. Without compiled light (test maps) you get 23 lumps or 24 with lights. Stock maps have usually 37.

After the first 12 header bytes is an array filled with DWORD pairs, indicating the lumps' IDs and lengths. No absolute offsets are stored in this so called lump index in contrast to previous CoD BSP versions (each lump is connected). One can determine offsets by taking the previous lump's offset + length (padded to a multiple of 4). The first lump's offset is (3 + (number of lumps * 2)) * 4. Nonetheless, the first data byte has the offset 0. Don't count the header and the lump index!

 Note: there can be a difference of number of lumps per map. If a map has a leak
(no solid skybox or not everything is IN the skybox) the bsp only has 24 lumps (or even 23 if light is not
compiled). The bsp description yet only supports the 37-lumps version (a valid map).

Variance

Not all lumps are padded to a multiple of 4 in size. There are two exceptions:

Lump[0x07] - unknown, no padding
Lump[0x27] - Entities, no padding but +1 for c-string termination?

Example

Offset calculation of the first lump:

00 00 00 00  |  B0 C7 00 00
     ↓       |       ↓       - Little Endian transfrom
00 00 00 00  |  00 00 C7 B0
     ↓       |       ↓       - hex to dec
     0       |     51120
  Lump ID    |   Lump length


Start offset = Header length + Lump index length
             = (3 + (39 * 2)) * 4
             = 324

End offset   = Start offset  + Lump length
             = 51120 + 324
             = 51444

Start offset | End offset
    324      |   51444
     ↓       |     ↓         - dec to hex
    144      |   C8 F4

Lump[0x00] - Materials

former texture filename + flags

Texture information is 72 bytes long per texture. The first 64 bytes are used for the texture name. Then we have an DWORD for flags, and another DWORD for content flags.

Lump[0x01] - Lightmaps

3,145,728 bytes (3,072 kb) per entry.

Lump[0x02] - Light Grid Points

4 bytes per entry.

Lump[0x03] - Light Grid Colors

168 bytes per entry.

Lump[0x04] - Planes

16 bytes per entry.

Lump[0x05] - Brush Sides

8 bytes per entry.

Lump[0x06] - unknown

Equal to count of brush sides?

Lump[0x07] - unknown

Lump[0x08] - Brushes

4 bytes per entry.

Lump[0x09] - Layered Tri Soups

24 bytes per entry.

Lump[0x0A] - Layered Verts

ID 0Ah = 10, former Vertex Data
68 bytes per entry.

Lump[0x0B] - Layered Indexes

ID 0Bh = 11
2 bytes per entry.

[Lump[0x13] - Portal Verts

ID 13h = 19
12 bytes per entry.

Lump[0x18] - Layered AABB Trees

ID 18h = 24
12 bytes per entry.

Lump[0x19] - Cells

ID 19h = 25, former Bounding Box
112 bytes per entry.

Lump[0x1A] - Portals

ID 1Ah = 26
16 bytes per entry.

Lump[0x1B] - Nodes

ID 1Bh = 27
36 bytes per entry.

Lump[0x1C] - Leafs

ID 1Ch = 28
24 bytes per entry.

Lump[0x1D] - Leaf Brushes

ID 1Dh = 29
4 bytes per entry.

Lump[0x1F] - Collision Verts

ID 1Bh = 31, former assumed as nodes
12 bytes per entry.

Lump[0x20] - Collision Tris

ID 20h = 32 6 bytes per entry.

Lump[0x21] - Collision Edge Walk

ID 21h = 33
Variable in length.

Lump[0x22] - Collision Borders

ID 22h = 34
28 bytes per entry.

Lump[0x23] - Collision Parts

ID 23h = 35
12 bytes per entry.

Lump[0x24] - Collision AABBs

ID 24h = 36
32 bytes per entry.

Lump[0x25] - Models

ID 25h = 37
48 bytes per entry.

Lump[0x27] - Entities

ID 27h = 39, also known as EntData
Varying bytes per entry depending on size of entity. Basically a char[] terminated by a binary zero (=string). The entities lump is readable and stored almost the same way described in the .MAP file structure.

Lump[0x28] - Paths

ID 28h = 40
Variable in length.

Lump[0x29] - Reflection Probes

ID 29h = 41
131,140 bytes (128 kB) per entry.

Lump[0x2A] - Layered Data

ID 2Ah = 42
Variable in length.

Lump[0x2B] - Primary Lights

ID 2Bh = 43
128 bytes per entry.

Lump[0x2C] - Light Grid Header

ID 2Ch = 44
50 bytes per entry? (variable in length?)

Lump[0x2D] - Light Grid Rows

ID 2Dh = 45
16 bytes per entry? (variable in length?)

Lump[0x2F] - Simple Tri Soups

ID 2Fh = 47
24 bytes per entry.

Lump[0x30] - Simple Verts

ID 30h = 48, former Vertex Data
68 bytes per entry. - Redundancy??

Lump[0x31] - Simple Indexes

ID 31h = 49
2 bytes per entry.

Lump[0x33] - Simple AABB Trees

ID 33h = 51
12 bytes per entry.

Lump[0x34] - Light Regions

ID 34h = 52
1 byte per entry.

Lump[0x35] - Light Region Hulls

ID 35h = 53
76 bytes per entry.

Lump[0x36] - Light Region Axes

ID 36h = 54
20 bytes per entry.

Unidentified lumps

Predicted lump IDs in brackets:

[0x1E] Leaf Surfaces
[0x??] Visibility
[0x??] Cull Groups
[0x??] Cull Group Indexes

Note: If not listed, either missing or ID not in use.

BSP data in FastFile

Eventhough IW said the fastfiles were encrypted, in fact they are zlib-compressed memory dumps with a 12 byte header. Due to the process of building a fastfile, the linker loads all files (referenced in the csv) into memory. Assumedly, this happens in the same fashion like the game engine loads the data for later use into a large data structure (might be a database, have a look at the console messages). It seems that there is a pre-processing or a different way of storing the bsp data at runtime. Whether or not active changes are made, the data differs in any case. At this point it looks difficult to get all desired data, but the fastfile construction is still under investigation.

(Almost) unchanged lumps:   2, 3, ? (9th), 31, 32, 33, 34, 36, 44, 45, 52
Partially changed:          0, 1, 19, 35, 37, 39, 53, 54
Not or hardly recognizable: 4, 5, ? (8th), 8, 9, 10, 11, 24, 25, 26, 27, 28, 29, 42, 43, 47, 48, 49, 51

Made by Daevius, CoDEmanX & Silicone_Milk