Call of Duty 4: d3dbsp - Revision history

CoDEmanX: Undo revision 15869 by Randallphillips (talk)

2012-01-04T13:57:37Z

Undo revision 15869 by Randallphillips (talk)

← Older revision		Revision as of 13:57, 4 January 2012
Line 281:		Line 281:
	[[Category:Technical Reference]]		[[Category:Technical Reference]]
	[[Category:File formats]]		[[Category:File formats]]

	~~[http://cvresumewriters.com/ resume writers]~~

	~~[http://essaywritingservices.org/prices.php write my essay]~~

Randallphillips at 11:40, 24 June 2011

2011-06-24T11:40:59Z

New page

The .d3dbsp structure, Call of Duty 4's variant on the well-known BSP format, is rather difficult to decipher. Since it's a binary file you can't simply read it. A hexadecimal editor is the best tool in this case. I am still deciphering it myself, I will post anything I happen to get known of regarding the file format.

Be sure to read [http://www.geocities.com/cofrdrbob/bspformat.html here] and [http://en.wikipedia.org/wiki/BSP_(file_extension) here] before you even try to decipher it. Knowing what the technique is that BSP formatted maps use, could be handy as well. Also see [[Call of Duty 2: d3dbsp]] for more information.

Every number ending with an 'h' indicates its a number using the hexadecimal count system. For other numbers one can assume the decimal count system is used.

=Header & Lump index=
The file starts with 3 DWORDs: the header 'IBSP' indicating this is a BSP file originally designed by ID software. After these 4 initial bytes you find the BSP version number. CoD4 uses the version number 22 (16h), which is different from CoD2. The next 4 byte group is the amount of lumps in the current BSP. The number of lumps varies from 23 to 37. Without compiled light (test maps) you get 23 lumps or 24 with lights. Stock maps have usually 37.

After the first 12 header bytes is an array filled with DWORD pairs, indicating the lumps' '''IDs and lengths'''. No absolute offsets are stored in this so called lump index in contrast to previous CoD BSP versions (each lump is connected). One can determine offsets by taking the previous lump's offset + length (padded to a multiple of 4). The first lump's offset is <code>(3 + (number of lumps * 2)) * 4</code>. Nonetheless, the first data byte has the offset 0. Don't count the header and the lump index!

[[Image:Lightbulb.png]]'' Note: there can be a difference of number of lumps per map. If a map has a leak
(no solid skybox or not everything is IN the skybox) the bsp only has 24 lumps (or even 23 if light is not
compiled). The bsp description yet only supports the 37-lumps version (a valid map).''

==Variance==
Not all lumps are padded to a multiple of 4 in size. There are two exceptions:

*[[#Lump.5B0x07.5D_-_unknown|Lump[0x07] - unknown]], no padding
*[[#Lump.5B0x27.5D_-_Entities|Lump[0x27] - Entities]], no padding but +1 for c-string termination

==Example==
[[Image:D3DBSP_v22.png]]

Offset calculation of the first lump:
<pre>
00 00 00 00 | B0 C7 00 00
↓ | ↓ - Little Endian transfrom
00 00 00 00 | 00 00 C7 B0
↓ | ↓ - hex to dec
0 | 51120
Lump ID | Lump length

Start offset = Header length + Lump index length
= (3 + (39 * 2)) * 4
= 324

End offset = Start offset + Lump length
= 51120 + 324
= 51444

Start offset | End offset
324 | 51444
↓ | ↓ - dec to hex
144 | C8 F4
</pre>

=Lumps=

==Lump[0x00] - Materials==
''former texture filename + flags''

Texture information is 72 bytes long per texture. The first 64 bytes are used for the texture name. Then we have an DWORD for flags, and another DWORD for content flags.

==Lump[0x01] - Lightmaps==
3,145,728 bytes (3,072 kb) per entry.

==Lump[0x02] - Light Grid Points==
4 bytes per entry.

==Lump[0x03] - Light Grid Colors==
168 bytes per entry.

==Lump[0x04] - Planes==
16 bytes per entry.

==Lump[0x05] - Brush Sides==
8 bytes per entry.

==Lump[0x06] - unknown==
Equal to count of brush sides?

==Lump[0x07] - unknown==
 

==Lump[0x08] - Brushes==
4 bytes per entry.

==Lump[0x09] - Layered Tri Soups==
24 bytes per entry.

==Lump[0x0A] - Layered Verts==
''ID 0Ah = 10'', former Vertex Data 
68 bytes per entry.

==Lump[0x0B] - Layered Indexes==
''ID 0Bh = 11'' 
2 bytes per entry.

==[Lump[0x13] - Portal Verts==
''ID 13h = 19'' 
12 bytes per entry.

==Lump[0x18] - Layered AABB Trees==
''ID 18h = 24'' 
12 bytes per entry.

==Lump[0x19] - Cells==
''ID 19h = 25, former Bounding Box'' 
112 bytes per entry.

==Lump[0x1A] - Portals==
''ID 1Ah = 26'' 
16 bytes per entry.

==Lump[0x1B] - Nodes==
''ID 1Bh = 27'' 
36 bytes per entry.

==Lump[0x1C] - Leafs==
''ID 1Ch = 28'' 
24 bytes per entry.

==Lump[0x1D] - Leaf Brushes==
''ID 1Dh = 29'' 
4 bytes per entry.

==Lump[0x1F] - Collision Verts==
''ID 1Bh = 31, former assumed as nodes'' 
12 bytes per entry.

==Lump[0x20] - Collision Tris==
''ID 20h = 32''
6 bytes per entry.

==Lump[0x21] - Collision Edge Walk==
''ID 21h = 33'' 
Variable in length.

==Lump[0x22] - Collision Borders==
''ID 22h = 34'' 
28 bytes per entry.

==Lump[0x23] - Collision Parts==
''ID 23h = 35'' 
12 bytes per entry.

==Lump[0x24] - Collision AABBs==
''ID 24h = 36'' 
32 bytes per entry.

==Lump[0x25] - Models==
''ID 25h = 37'' 
48 bytes per entry.

==Lump[0x27] - Entities==
''ID 27h = 39, also known as EntData'' 
Varying bytes per entry depending on size of entity. Basically a char[] terminated by a binary zero (=string).
The entities lump is readable and stored almost the same way described in the [[Call_of_Duty_4:_.MAP_file_structure#Entity|.MAP file structure]].

==Lump[0x28] - Paths ==
''ID 28h = 40'' 
Variable in length.

==Lump[0x29] - Reflection Probes==
''ID 29h = 41'' 
131,140 bytes (128 kB) per entry.

==Lump[0x2A] - Layered Data==
''ID 2Ah = 42'' 
Variable in length.

==Lump[0x2B] - Primary Lights==
''ID 2Bh = 43'' 
128 bytes per entry.

==Lump[0x2C] - Light Grid Header==
''ID 2Ch = 44'' 
50 bytes per entry? (variable in length?)

==Lump[0x2D] - Light Grid Rows==
''ID 2Dh = 45'' 
16 bytes per entry? (variable in length?)

==Lump[0x2F] - Simple Tri Soups==
''ID 2Fh = 47'' 
24 bytes per entry.

==Lump[0x30] - Simple Verts==
''ID 30h = 48, former Vertex Data'' 
68 bytes per entry. - Redundancy??

==Lump[0x31] - Simple Indexes==
''ID 31h = 49'' 
2 bytes per entry.

==Lump[0x33] - Simple AABB Trees==
''ID 33h = 51'' 
12 bytes per entry.

==Lump[0x34] - Light Regions==
''ID 34h = 52'' 
1 byte per entry.

==Lump[0x35] - Light Region Hulls==
''ID 35h = 53'' 
76 bytes per entry.

==Lump[0x36] - Light Region Axes==
''ID 36h = 54'' 
20 bytes per entry.

==Unidentified lumps==
Predicted lump IDs in brackets:

*[0x1E] Leaf Surfaces
*[0x??] Visibility
*[0x??] Cull Groups
*[0x??] Cull Group Indexes

Note: If not listed, either missing or ID not in use.

=BSP data in FastFile=
Even though IW said the fastfiles were encrypted, in fact they are zlib-compressed memory dumps with a 12 byte header. Due to the process of building a fastfile, the linker loads all files (referenced in the csv) into memory. Assumedly, this happens in the same fashion like the game engine loads the data for later use into a large data structure (might be a database, have a look at the console messages). It seems that there is a pre-processing or a different way of storing the bsp data at runtime. Whether or not active changes are made, the data differs in any case. At this point it looks difficult to get all desired data, but the fastfile construction is still under investigation.

<pre>
(Almost) unchanged lumps: 2, 3, ? 7?, 31, 32, 33, 34, 36, 44, 45, 52
Partially changed: 0, 1, 19, 35, 37, 39, 53, 54
Not or hardly recognizable: 4, 5, 6?, 8, 9, 10, 11, 24, 25, 26, 27, 28, 29, 42, 43, 47, 48, 49, 51
</pre>

[[Image:Under Construction Small.png]]

@Daevius: I couldn't find Lump 41 (5,114,460 bytes) and wasn't able to clearly identify the 9th and 10th lump (in order of the index). Lengths look good, but offsets seem to be screwed. 605 bytes of entity data (29th lump) are at the end of the 28th, in the entity lump are 285 bytes data of the following lump in return.

--[[User:CoDEmanX|CoDEmanX]] 03:04, 28 March 2009 (UTC)

''Made by Daevius, CoDEmanX & Silicone_Milk''

[[Category:Call of Duty 4]]
[[Category:Technical Reference]]
[[Category:File formats]]

[http://cvresumewriters.com/ resume writers]

[http://essaywritingservices.org/prices.php write my essay]

CoDEmanX at 13:20, 29 July 2009

2009-07-29T13:20:35Z

← Older revision		Revision as of 13:20, 29 July 2009
Line 177:		Line 177:
	''ID 27h = 39, also known as EntData''<br>		''ID 27h = 39, also known as EntData''<br>
	Varying bytes per entry depending on size of entity. Basically a char[] terminated by a binary zero (=string).		Varying bytes per entry depending on size of entity. Basically a char[] terminated by a binary zero (=string).
	The entities lump is readable and stored almost the same way described in the [[Call_of_Duty_4:~~_MAP_file_structure~~#Entity\|.MAP file structure]].		The entities lump is readable and stored almost the same way described in the [[Call_of_Duty_4:_.MAP_file_structure#Entity\|.MAP file structure]].

CoDEmanX at 13:10, 29 July 2009

2009-07-29T13:10:43Z

← Older revision		Revision as of 13:10, 29 July 2009
Line 277:		Line 277:

	''Made by Daevius, CoDEmanX & Silicone_Milk''		''Made by Daevius, CoDEmanX & Silicone_Milk''

	[[Category:Call of Duty 4]]		[[Category:Call of Duty 4]]
	[[Category:Technical Reference]]		[[Category:Technical Reference]]
			[[Category:File formats]]

Zeroy: /* BSP data in FastFile */

2009-04-08T15:17:56Z

BSP data in FastFile

← Older revision		Revision as of 15:17, 8 April 2009
Line 259:		Line 259:

	=BSP data in FastFile=		=BSP data in FastFile=
	~~Eventhough~~ IW said the fastfiles were encrypted, in fact they are zlib-compressed memory dumps with a 12 byte header. Due to the process of building a fastfile, the linker loads all files (referenced in the csv) into memory. Assumedly, this happens in the same fashion like the game engine loads the data for later use into a large data structure (might be a database, have a look at the console messages). It seems that there is a pre-processing or a different way of storing the bsp data at runtime. Whether or not active changes are made, the data differs in any case. At this point it looks difficult to get all desired data, but the fastfile construction is still under investigation.		Even though IW said the fastfiles were encrypted, in fact they are zlib-compressed memory dumps with a 12 byte header. Due to the process of building a fastfile, the linker loads all files (referenced in the csv) into memory. Assumedly, this happens in the same fashion like the game engine loads the data for later use into a large data structure (might be a database, have a look at the console messages). It seems that there is a pre-processing or a different way of storing the bsp data at runtime. Whether or not active changes are made, the data differs in any case. At this point it looks difficult to get all desired data, but the fastfile construction is still under investigation.

	<pre>		<pre>

CoDEmanX: +1 for string termination (verified)

2009-04-01T13:09:45Z

+1 for string termination (verified)

← Older revision		Revision as of 13:09, 1 April 2009
Line 19:		Line 19:

	*[[#Lump.5B0x07.5D_-_unknown\|Lump[0x07] - unknown]], no padding		*[[#Lump.5B0x07.5D_-_unknown\|Lump[0x07] - unknown]], no padding
	*[[#Lump.5B0x27.5D_-_Entities\|Lump[0x27] - Entities]], no padding but +1 for c-string termination?		*[[#Lump.5B0x27.5D_-_Entities\|Lump[0x27] - Entities]], no padding but +1 for c-string termination

CoDEmanX at 14:04, 31 March 2009

2009-03-31T14:04:44Z

← Older revision		Revision as of 14:04, 31 March 2009
Line 262:		Line 262:

	<pre>		<pre>
	(Almost) unchanged lumps: 2, 3, ? ~~(9th)~~, 31, 32, 33, 34, 36, 44, 45, 52		(Almost) unchanged lumps: 2, 3, ? 7?, 31, 32, 33, 34, 36, 44, 45, 52
	Partially changed: 0, 1, 19, 35, 37, 39, 53, 54		Partially changed: 0, 1, 19, 35, 37, 39, 53, 54
	Not or hardly recognizable: 4, 5, ? ~~(8th)~~, 8, 9, 10, 11, 24, 25, 26, 27, 28, 29, 42, 43, 47, 48, 49, 51		Not or hardly recognizable: 4, 5, 6?, 8, 9, 10, 11, 24, 25, 26, 27, 28, 29, 42, 43, 47, 48, 49, 51
	</pre>		</pre>

Line 270:		Line 270:
	[[Image:Under Construction Small.png]]		[[Image:Under Construction Small.png]]

	@Daevius: I couldn't find Lump 41 (5,114,460 bytes) and wasn't able to identify the ~~8th~~ and ~~9th~~ lump (in order of the index). Lengths look good, but offsets seem to be screwed. 605 bytes of entity data (29th lump) are at the end of the 28th, in the entity lump are 285 bytes data of the following lump in return.		@Daevius: I couldn't find Lump 41 (5,114,460 bytes) and wasn't able to clearly identify the 9th and 10th lump (in order of the index). Lengths look good, but offsets seem to be screwed. 605 bytes of entity data (29th lump) are at the end of the 28th, in the entity lump are 285 bytes data of the following lump in return.

	--[[User:CoDEmanX\|CoDEmanX]] 03:04, 28 March 2009 (UTC)		--[[User:CoDEmanX\|CoDEmanX]] 03:04, 28 March 2009 (UTC)

CoDEmanX at 03:04, 28 March 2009

2009-03-28T03:04:27Z

← Older revision		Revision as of 03:04, 28 March 2009
Line 6:		Line 6:


	==Header & Lump index==		=Header & Lump index=
	The file starts with 3 DWORDs: the header 'IBSP' indicating this is a BSP file originally designed by ID software. After these 4 initial bytes you find the BSP version number. CoD4 uses the version number 22 (16h), which is different from CoD2. The next 4 byte group is the amount of lumps in the current BSP. The number of lumps varies from 23 to 37. Without compiled light (test maps) you get 23 lumps or 24 with lights. Stock maps have usually 37.		The file starts with 3 DWORDs: the header 'IBSP' indicating this is a BSP file originally designed by ID software. After these 4 initial bytes you find the BSP version number. CoD4 uses the version number 22 (16h), which is different from CoD2. The next 4 byte group is the amount of lumps in the current BSP. The number of lumps varies from 23 to 37. Without compiled light (test maps) you get 23 lumps or 24 with lights. Stock maps have usually 37.

Line 15:		Line 15:
	compiled). The bsp description yet only supports the 37-lumps version (a valid map).''		compiled). The bsp description yet only supports the 37-lumps version (a valid map).''

	===Variance===		==Variance==
	Not all lumps are padded to a multiple of 4 in size. There are two exceptions:		Not all lumps are padded to a multiple of 4 in size. There are two exceptions:

Line 22:		Line 22:


	===Example===		==Example==
	[[Image:D3DBSP_v22.png]]		[[Image:D3DBSP_v22.png]]

Line 50:		Line 50:
	</pre>		</pre>


			=Lumps=

	==Lump[0x00] - Materials==		==Lump[0x00] - Materials==
Line 256:		Line 258:


	== BSP data in FastFile ==		=BSP data in FastFile=
	Eventhough IW said the fastfiles were encrypted, in fact they are zlib-compressed memory dumps with a 12 byte header. Due to the process of building a fastfile, the linker loads all files (referenced in the csv) into memory. Assumedly, this happens in the same fashion like the game engine loads the data for later use into a large data structure (might be a database, have a look at the console messages). It seems that there is a pre-processing or a different way of storing the bsp data at runtime. Whether or not active changes are made, the data differs in any case. At this point it looks difficult to get all desired data, but the fastfile construction is still under investigation.		Eventhough IW said the fastfiles were encrypted, in fact they are zlib-compressed memory dumps with a 12 byte header. Due to the process of building a fastfile, the linker loads all files (referenced in the csv) into memory. Assumedly, this happens in the same fashion like the game engine loads the data for later use into a large data structure (might be a database, have a look at the console messages). It seems that there is a pre-processing or a different way of storing the bsp data at runtime. Whether or not active changes are made, the data differs in any case. At this point it looks difficult to get all desired data, but the fastfile construction is still under investigation.

Line 264:		Line 266:
	Not or hardly recognizable: 4, 5, ? (8th), 8, 9, 10, 11, 24, 25, 26, 27, 28, 29, 42, 43, 47, 48, 49, 51		Not or hardly recognizable: 4, 5, ? (8th), 8, 9, 10, 11, 24, 25, 26, 27, 28, 29, 42, 43, 47, 48, 49, 51
	</pre>		</pre>


			[[Image:Under Construction Small.png]]

			@Daevius: I couldn't find Lump 41 (5,114,460 bytes) and wasn't able to identify the 8th and 9th lump (in order of the index). Lengths look good, but offsets seem to be screwed. 605 bytes of entity data (29th lump) are at the end of the 28th, in the entity lump are 285 bytes data of the following lump in return.

			--[[User:CoDEmanX\|CoDEmanX]] 03:04, 28 March 2009 (UTC)

CoDEmanX at 02:52, 28 March 2009

2009-03-28T02:52:12Z

← Older revision		Revision as of 02:52, 28 March 2009
Line 260:		Line 260:

	<pre>		<pre>
	(Almost) unchanged lumps: ~~44, 45,~~ 2, 3, 9 ?, 31, 32, 33, 34, 36, 52		(Almost) unchanged lumps: 2, 3, ? (9th), 31, 32, 33, 34, 36, 44, 45, 52
	Partially changed: 0, 1, 19, 35, 37, 39, 53, 54		Partially changed: 0, 1, 19, 35, 37, 39, 53, 54
	Not or hardly recognizable: 4, 5, 8 ?, 8, 9, 10~~, 42~~, 11, 24, 25, 26, 27, 28, 29, 43, 47, 48, 49, 51		Not or hardly recognizable: 4, 5, ? (8th), 8, 9, 10, 11, 24, 25, 26, 27, 28, 29, 42, 43, 47, 48, 49, 51
	</pre>		</pre>

CoDEmanX at 02:50, 28 March 2009

2009-03-28T02:50:46Z

← Older revision		Revision as of 02:50, 28 March 2009
Line 260:		Line 260:

	<pre>		<pre>
	(Almost) unchanged lumps: 2, 3, ~~4, 5,~~ 9, 22, 23, 24, 25, ~~27, 31~~,		(Almost) unchanged lumps: 44, 45, 2, 3, 9 ?, 31, 32, 33, 34, 36, 52
	Partially changed: 1, 15, 26, 28, 29, 32, 33		Partially changed: 0, 1, 19, 35, 37, 39, 53, 54
	Not or hardly recognizable: 6, 7, 8, 10, 11, 12, 13, 14, 16, 17, 18, 19, 20, 21, 30, 34, 35, 36, 37		Not or hardly recognizable: 4, 5, 8 ?, 8, 9, 10, 42, 11, 24, 25, 26, 27, 28, 29, 43, 47, 48, 49, 51
	</pre>		</pre>