Call of Duty 2: d3dbsp

From COD Modding & Mapping Wiki
Revision as of 00:19, 11 March 2012 by CoDEmanX (talk | contribs) (-spam)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigation Jump to search
Deciphering the Call of Duty 2 D3DBSP format

The .d3dbsp structure, Call of Duty 2's variant on the well-known BSP format, is rather difficult to decipher. Since it's a binary file you can't simply read it. A hexadecimal editor is the best tool in this case. I am still deciphering it myself, I will post anything I happen to get known of regarding the file format.


Be sure to read here and here before you even try to decipher it. Knowing what the technique is that BSP formatted maps use, could be handy as well. The Call of Duty 2 engine is based on its ancestor, so is the BSP: Call of Duty 1: d3dbsp

Every number ending with an 'h' indicates its a number using the hexadecimal count system. For other numbers one can assume the decimal count system is used.

Numbers are stored in little-endian. The endianness describes the order in which a sequence of bytes are stored in computer memory. Little-endian is an order in which the "little end" (least significant value in the sequence) is stored first.


An useful however not documented feature of the cod2map.exe is the -info parameter. It analyzes the given BSP and prints the name, length and entry count of all lumps to console. One can use it for deductions regarding the lump and entry sizes and also to identify the kind of content.


Header

The file starts with 2 DWORDS: the header 'IBSP' indicating this is a BSP file originally designed by ID software and the version number 4 (4h).

Then there is an array, filled with DWORD pairs, indicating the lump's offset and length respectively. The array ends with 2 empty (zero) DWORDs. Each lump that has size 0 in the list is simply not filled / not used.

 Note: there can be a difference of number of lumps per map. If a map has a leak
(no solid skybox or not everything is IN the skybox) the bsp only has 24 lumps (or even 23 if light is not
compiled). The bsp description yet only supports the 37-lumps version (a valid map).


Lumps

A BSP is organized in several sections, called lumps. Each lump stores data with different functionality. It's a data unit containing sub-sections, or better: entries. Every entry is a binary representations of something, a plane custom shot glasses for instance. Such a plane is stored as a normal vector (3 floats) and a distance (1 float). The length of the datatype "float" is 4 byte (DWORD), so the size of one plane entry is 4 DWORDs (4*4 = 16 bytes).


Lump[0] - Materials

former Textures

Texture information is 72 bytes long per texture. The first 64 bytes are used for the texture name. Then we have an DWORD for flags, and another DWORD for content flags.

struct Material
{
    char name[64];
    unsigned char flags[2][4];
};


Example:

d_beachpebble                                                   0x00b00000 0x00000004
stalingrad_ground_trench                                        0x00e00000 0x00000001
duhoc_dirt_ground1                                              0x00600000 0x00000001


Flags:

Surface type:
-------------
         -          - <error> (asset manager will fail)
0x00000000 0x00000001 <none>
0x01600000 0x00000001 asphalt
0x00100000 0x00000001 bark
0x00200000 0x00000001 brick
0x00300000 0x00000001 carpet
0x00400000 0x00000001 cloth
0x00500000 0x00000001 concrete
0x00600000 0x00000001 dirt
0x00700000 0x00000001 flesh
0x00800000 0x00000002 foliage
0x00900000 0x00000010 glass
0x00a00000 0x00000001 grass
0x00b00000 0x00000001 gravel
0x00c00000 0x00000001 ice
0x00d00000 0x00000001 metal
0x00e00000 0x00000001 mud
0x00f00000 0x00000001 paper
0x01000000 0x00000001 plaster
0x01100000 0x00000001 rock
0x01200000 0x00000001 sand
0x01300000 0x00000001 snow
0x01400000 0x00000020 water
0x01500000 0x00000001 wood

Surface properties:
-------------------
0x00000000 0x00000080 missileClip
0x00000000 0x00002000 bulletClip
0x00000000 0x00010000 playerClip
0x00000000 0x00000200 vehicleClip
0x00000000 0x00020000 aiClip
0x00000000 0x00000400 itemClip
0x00000000 0x00000041 canShootClip
0x00000000 0x00001000 aiSightClip
0x00000001 0x00000001 noFallDamage
0x00002000 0x00000001 noSteps
0x00000010 0x00000001 noImpact
0x00000020 0x00000001 noMarks
0x00000000 0x80000000 noDrop
0x00000002 0x00000001 slick
0x00000008 0x00000001 ladder
0x02000000 0x01000001 mantleOn
0x04000000 0x01000001 mantleOver
0x00000000 0x00000001 noLightmap
0x00020000 0x00000001 noDynamicLight
0x00040000 0x00000001 noCastShadow
0x00000080 0x00000001 noDraw
0x00000000 0x00000001 noFog
0x00000000 0x00000001 drawToggle
0x00000004 0x00000800 sky
0x00000000 0x00000001 radialNormals
0x00000000 0x00000004 nonColliding
0x00004000 0x00000000 nonSolid
0x00000000 0x20000001 transparent \__ (two entries
0x00000000 0x28000001 transparent /    added to lump)
0x00000000 0x08000001 detail
0x00000000 0x10000001 structural
0x80000000 0x00000000 portal
0x00000000 0x00000001 occluder
         -          - origin (no lump entry)


Combinations (examples):

0x00000000 0x00002080 missleClip & bulletClip
0x00000000 0xb0000000 noDrop & transparent & structural
0x00000000 0x90000000 noDrop & structural
0x00000004 0x20000800 transparent & sky \__ (two entries
0x00000004 0x28000800 transparent & sky /    added to lump)
0x01400088 0x00000020 water & noFog & noDraw & ladder

These flags can be set in AssMan, nonetheless some Radiant functions influence them as well, e.g.

"Make Weapon Clip" on a wood-textured brush

0x01500000 0x00000001 \__ usual wood flags
0x01500000 0x00002080 /   first flag like wood, second flag missleClip & bulletClip

Note: the brush lost its collision (less planes in lump 4 etc.)


Lump[1] - Lightmaps

former World Lightning

Contains most of the pre-processed lightning. Not present if compiled without lightning, otherwise a multiple of 4.096 KB.

Without: World geometry is rendered overbright


Lump[2] - Light Grid Hash

former Light Volume

Precalculated (sun)light in the 3d space of the playable area, also called light grid. Each entry is 8 bytes long.

Without: a lot less light influence and a little rainbow colors on entities (e.g. player)


Lump[3] - Light Grid Values

former Model Lightning

Base (sun)light for model entities (ground lighting for xmodels). 24 bytes per entry.

Without: models show up very dark


Lump[4] - Planes

Each plane is defined by 4 DWORDs (16 bytes), 3 for the normal vector and 1 for the offset of the origin. Planes are faces with infinite length, we suspect they have to do with the rendering system.

struct Plane
{
    float normal[3];
    float distance;
};


Example:

 0 |  0        0         1         128
 1 |  0        0         1         320
 2 |  0        1         0        -384
 3 |  1        0         0         384
 4 |  0        1         0        -128
 5 |  1        0         0         128
 6 |  0        0.707107  0.707107  90.5097


Lump[5] - Brushsides

Without: CMod_LoadBrushes: bad side count

struct Brushside
{
    union Column1 // first 6 entries indicated by an entry in lump 6 are distances, rest is plane ID's
    {
        unsigned int plane;
        float distance;
    } column1;

    unsigned int texture;
};


Lump 5 and 6 are used for the collision map, however the second column holds texture IDs. The ID is a reference to lump 0 and can be used to look up the texture flags. These tell the game wether the brush is used for collision detection only, acts as ladder, player clip etc.

Each entry has a length of 8 bytes. Two columns, first either distance or plane id:

  • float  distance - plane parallel to x, y or z axis
  • ushort plane_id - slant plane reference to lump 4 (#)


The first six entries per brush (lump 6) are distances defining a bounding box, followed by optional plane IDs for more complex, but still convex brushes. Imagine a wooden cube (bounding box), where you cut off pieces with an axe. Note that the axe's blade is infinite, you can't do partial cuts.

The normal vector's direction represents the plane's facing. The decision, which part will be kept of the volume (brush), is made here. The part the vector points is dropped. If you negate the normal vector, it points to the remaining volume.


Example:

     128       20
     384       18
    -384       17
    -128       19
     128       15
     320       16
     13 #      21
     128       12
     384       14
     128       13
     384       14
     128       11
     320       14
     10 #      14
     12 #      14

Order of the bounding box sides:

  • 1 - left
  • 2 - right
  • 3 - front
  • 4 - back
  • 5 - bottom
  • 6 - top


Bounding Box:

The minimum or smallest bounding or enclosing box
that indicates the maximum extents of an object.
In our case, each two sides are parallel to
one of the axis (x/y/z) to shape an aligned cube.
(technical term: AABB - axis-aligned bounding box)


Second column is a texture id, texture lump:

0 | dawnville2_wartorn_brick07                                      0x00200000 0x00000001
1 | dawnville2_wood_boards01                                        0x01500000 0x00000001
2 | dawnville2_wood_boards01                                        0x01500000 0x00002080

Lump[6] - Brushes

Without: no collision with brush geometry. Footage (Xfire)


struct Brush
{
    unsigned short sides;
    unsigned short material_id;
};


Brushside index, two ushorts per entry (2 byte unsigned integer, 4 bytes each row).

  • First column:

Amount of brushsides, start id can be calculated (sum_of_previous_amounts - 1)
For end id, add the entry value to the start id (sum_of_previous_amounts - 1 + amount_of_brushsides)

   7   15
   8   11
  • Second column:

Might be texture id, perhaps one of the applied textures as default.
If more than one texture is mapped to the brush, it may get overwritten (brushsides lump texture id).

New theory: the texture flags (properties) are actually referenced to set up the brush characteristic and the engine behaves accordingly.


Lump[7] - TriangleSoups

Without: LoadMap: funny lump size in (*.d3dbsp)

It's like the missing link between the triangles (lump 9) and vertices (lump 8). It refers to both of them defining the texture per triangle, giving the offset in the vertex lump to which the numbers in lump 9 point to (ID's in lump 9 point to vertex as in: offset + ID, offset is given in this lump). Also offset in triangles lump is given.

16 bytes per entry:

struct TriangleSoup
{
    unsigned short material_id;

    unsigned short draw_order;

    unsigned int vertex_offset;
    unsigned short vertex_length;

    unsigned short triangle_length;
    unsigned int triangle_offset;
};


Example:

       0        0        0   24   84        0
       1        0       24    8   24       84
       2        0       32   14   18      108
       3        0       46   40   66      126
       4        0       86   16   24      192
       5        0      102   78  144      216


Lump[8] - Vertices

or DrawVerts

Every vertex entry is 17 DWORDs long (68 bytes). First 3 DWORDs define the position (x y and z). Next 3 DWORDs define the normal vector of the vertex (x y and z) (WHY?!), then a DWORD which defines the colour (BGR) and the opacity (A). Then 2 DWORDs which have to do with texture shifting, and 2 more DWORDs for something yet unknown. At the end we have 6 DWORDs having to do with texture alignment (rotation?).

struct DrawVertex
{
    float position[3];
    float normal[3];
    unsigned char rgba[4];

    float uv[2];
    float st[2];

    float unknown[6];
};


Example:

     256      256      -64  0.235702 0.235702 0.942809 0xff80a0c0  1.000000  1.000000  0.023438  0.001953  0.970143  0.000000 -0.242536 -0.057166  0.971825 -0.228665 
       0      256        0  0.119573 0.119573 0.985599 0xff80a0c0  0.000000  1.000000  0.007813  0.001953  0.992719  0.000117 -0.120451 -0.014518  0.992825 -0.118689 
       0        0        0  0        0        1        0xff80a0c0  0.000000  0.000000  0.007813  0.017578  1.000000  0.000000  0.000000  0.000000  1.000000  0.000000 
     256        0        0  0.119573 0.119573 0.985599 0xff80a0c0  1.000000  0.000000  0.023438  0.017578  0.992719  0.000117 -0.120451 -0.014518  0.992825 -0.118689 
    -100        0        0  0        0        1        0xffffffff  0.804688  0.000000  0.001709  0.017578  1.000000  0.000000  0.000000  0.000000 -1.000000  0.000000 
       0        0        0  0        0        1        0xffffffff  1.000000  0.000000  0.007813  0.017578  1.000000  0.000000  0.000000  0.000000 -1.000000  0.000000 


Lump[9] - Triangles

or DrawIndexes

A face is made up of 3 vertexes, giving a triangle. One entry is actually 2 bytes long (ushort), but 3 sequently stored make up one triangle. So 3 entries can be treated as 1.

struct DrawIndex
{
    unsigned short vertex[3];
};


Example:

   8    9    6 
  12   13   10 
  22   23   20 
  20   21   22 
   2   19    0 
   0    5    2 
  18    3   16 
  16   17   18 
  10   14   12 
   6   15    8 
  11    7    4 
   4    1   11 
   6    7    4 
   4    5    6 
   2    3    0 
   0    1    2 


Lump[10] - Cull Groups

32 bytes per entry.


Lump[11] - Cull Group Indexes

4 bytes per entry.


Lump[17] - Portal Verts

12 bytes per entry.

struct PortalVertex
{
    float unknown[3];
};

Lump[18] - Occluder

20 bytes per entry. Fine portalling within a cell, Valve Engine example


Lump[19] - Occluder Planes

4 bytes per entry.


Lump[20] - Occluder Edges

4 bytes per entry.


Lump[21] - Occluder Indexes

2 bytes per entry.


Lump[22] - AABB Trees

Without: game crashes with application error

12 bytes per entry. Web links: [1] [2]

struct AABBTree
{
    unsigned int unknown[3];
};


Example:

       0     1278        8 
       0      300        6 
     300       22        2 
     322      292        5 
     614       17        2 


Lump[23] - Cells

Assumed to be VIS portals, contains a cell in Skybox dimension (first entry). 52 bytes per entry (example needs update!).

struct Cell
{
    float unknown[3];
};


Example:

     120     -392      120 
     392     -120      328 
       0        0        0 
       0        0        0 


Lump[24] - Portals

16 bytes per entry.

struct Portal
{
    unsigned int unknown[4];
};


Lump[25] - Nodes

Without: Map has no Nodes

36 bytes per entry.

struct Node
{
    int unknown[9];
};


Example:

       1       -2        1      -16      -16       -8        8        8       16 
       5        2       -8      -16      -16       -8        8        8        8 
       0        3       -7       -8      -16       -8        8        8        8 
       3       -3        4       -8      -16        0        8        8        8 
       4       -4        5       -8      -16        0        0        8        8 
       2       -5       -6       -8      -16        0        0        0        8 


Lump[26] - Leafs

Without: Map has no Leafs

36 bytes per entry.

struct Leaf
{
    int unknown[9];
};


Example:

       0        0        0        0        0        0        0        0        0 
      -1       -1        0        0        0        1       -1        0        0 
      -1       -1        0        0        1        1       -1        0        0 
      -1       -1        0        0        2        1       -1        0        0 
       0        0        0        0        3        0        0        0        0 
      -1       -1        0        0        3        1       -1        0        0 
      -1       -1        0        0        4        1       -1        0        0 

The 5th column is possibly an id of lump 27.


Lump[27] - Leaf Brushes

Without: Stuck in loadscreen, game closes after a couple seconds without error.

This lump contains a list of brush IDs (lump 6), there might be an index lump for it. 4 bytes per entry.

struct LeafBrush
{
    int unknown;
};


Example:

      53
      52
      51
      50
      14
       1


Lump[29] - Collision Verts

Without: no visible changes

16 bytes per entry.

struct CollisionVertex
{
    float unknown[4];
};


Lump[30] - Collision Edges

former Player Terrain Collision

Without: terrain retains collision, but player can't move smoothly on the ground. It's easy to get stuck or float around. Still okay where terrain is very even. Footage (Xfire). 56 bytes per entry.

struct CollisionEdge
{
    unsigned int unknown;

    float position[3];
    float normal[3][3];
    float distance;
};


Example:

       0       64      128        8        0        0        1  -0.707107  -0.707107        0  0.707107  -0.707107        0  90.5097 
       0      128       64        8        0        1        0          0          0       -1        -1          0        0       64 
       0       64       64        8        1        0        0          0          0       -1         0          1        0       64 
       0       64      128        8        0        1        0          0          0        1         1          0        0       64 
       0      128      128        8        1        0        0          0          0        1         0         -1        0       64 
       0      128      128        8        0        0        1  -0.707107  -0.707107        0  0.707107  -0.707107        0  90.5097 


Lump[31] - Collision Tris

Without: game crashes with application error

72 bytes per entry.

struct CollisionTriangles
{
    float normal[3];
    float distance;
    float unknown[8];
    int id[6];
};


Lump[32] - Collision Borders

Without: no visible changes

28 bytes per entry.


Lump[33] - Collision Parts

Without: game crashes with application error

12 bytes per entry.


Lump[34] - Collision AABBs

Without: game crashes with application error

32 bytes per entry.


Lump[35] - Models

Without: Map with no models

Possibly similar to "rigid groups of world geometry": Quake3 - Model Lump

Seems to contain script_brushmodels, maybe more. 48 bytes per entry.

struct Models
{
    float position[6];
    int id[6];
};


Example:

  -18048   -11904     -896    -2944     6144     5160        0     5748        0        0        0     8260 
    -112     -112      -48      112      112       48     5748        0     3072        0     8260        1 
  -11072    -7504       16   -11014    -7382      144     5748        0     3072        0     8261        1 
    -112     -112      -48      112      112       48     5748        0     3072        0     8262        1 
   -9208    -6264        0    -9150    -6142      128     5748        0     3072        0     8263        1 
     -40      -40        0       40       40       56     5748        0     3072        0     8264        1 
  -17920   -11904     -512    -3072     6144     -384     5748        0     3072        0     8265        1 
  -17849   -11977     -512    -3001     6071     -384     5748        0     3072        0     8266        1 


Current investigation status:


How a script_brushmodel appears in the BSP (lump 37):

{  // ----- worldspawn block start
 {
   // brush definition inside of the worldspawn entity
   // all geometry elements which belongs to a group of lump 35 are stored sequently
 }
}  // ----- worldspawn block end

{
 "classname" "script_brushmodel"
 "model" "*1"
 "targetname" "auto1"
}


In the source .map it looks different:

// entity 1
{
"targetname" "auto1"
"classname" "script_brushmodel"

 // brush 0
 {
   // ...
 }

 // brush 1
 {
   // ...
 }

 // brush 2
 {
   // ...
 }

}

It requires a transformation to get script_brushmodels back. The missing link between the geometry and the entity is lump 35. model *n is an id and refers to an entry of this lump. It references 3 other lumps in turn.


Example:
 [1]     [2]      [3]      [4]      [5]      [6]       [7]      [8]      [9]      [10]    [11]      [12]

  0      256      192      112      320      256        0        1        0        0        0        1 
 16        0        0      152      128      112        1        4        0        0        1        3 
208        0      192      320      128      312        5        2        0        0        4        2 

 1 - left   (x)
 2 - front  (y)
 3 - bottom (z)
 4 - right  (x)
 5 - back   (y)
 6 - top    (z)
 7 - texture group (start)
 8 - texture group (amount)
 9 - mesh (start)
10 - mesh (amount)
11 - brush (start)
12 - brush (amount)

 1-6 : bounding box
 7+8 : texture group id reference (lump 7)
 9+10: patch id reference
11+12: brush id reference (lump 6)

Note: No planes in lump 4 for meshes / patches. Additional verts and tris if surface is visible. Mesh related lumps: 29, 30, 31, 33 and 34. One or more of them may contain collision data.


Lump[36] - Visibility

former VIS

Present if VIS (portalling) has been calculated during the compile process. Apparently a single block of data.

Without: /r_singlecell 1 still shows the portals, forced VIS?


Lump[37] - Entities

or EntData

The entities lump is readable and stored (almost) the same way described in the .MAP file structure.

struct Entity
{
    char data[];
};

Or, after getting rid of redundant data, one might interpret the entity lump like this:

struct Entity
{
    std::map<std::string, std::string> pairs;
};

In other words, the Entities lump consists of a a number of Entity structs. Each entity struct holds a list of string pairs.


Couple entity types in this lump got a "model" key/value pair with a star and a number (*n), causing Radiant to crash and the compiler to ignore the whole block (see lump 35). You also have to remove certain other things til the decompiler supports a reconstruction method for the necessary data:

  • model *n
  • fx
  • script_exploder
  • trigger
  • gndLt (optional, ignored by Radiant)

Furthermore, you need to recreate the skybox at the moment, as there will likely be leaks in the ground caused from caulk/clip/... textured geometry. It's not visible in game as you know and therefore not stored in the usual lumps for vertices, triangles and so on. But it has collision, and the collision data is either stored in the brush or terrain collmap lumps. Note that many models are clipped using invisible textures as well. If no collmap file is present for a model, create one or it will lack collision after compile. A decompiler would have to reconstruct the collision map to avoid this.


Lump[38] - Paths

SinglePlayer only, connected waypoints and/or paths? Variable in length.


Unidentified lumps

Not seen on any map so far, presumably not in use. Feasible lump number predictions in brackets:

  • [12] Shadow Verts
  • [13] Shadow Indices
  • [14] Shadow Clusters
  • [15] Shadow AABB Trees
  • [16] Shadow Sources
  • [28] Leaf Surfaces
  • [39] Lights


Web Links


Made by CoDEmanX and Daevius