Call of Duty 4: FastFile Format

*** Article and investigation in early progress ***

Preface

Be aware, FF files are little endian, so swap those bytes 'round!

The global file layout starts with a header. After which you get a list a model tag names, after which you get an index list. This list gives information about the rest of the file, which is a concatenation of files, most of which are reinterpreted. Each file is separated by the separation key: 0xFFFFFFFF. But be aware, each file itself can also contain such a sequence which is not meant to be the separation key! So it is necessary to understand each data type in the FF file.

Header

• Byte 0-3: decompressed fastfile size minus 44 (0x2C)
• Byte 4-7: about the total size of referenced data, e.g. the required memory for IWI textures if material files are in the ff
• Byte 8-11: unknown, might be a flag
• Byte 12-23: unknown
• Byte 24-27: somehow related to ff size?
• Byte 28-43: unknown
• Byte 44-47: equal to number of entries in "1st index" and amount of (model tag/joint/notetrack) strings (times 4 for index length)
• Byte 48-51: separator? (FF FF FF FF)
• Byte 52-55: number of records* ("2nd index", times 8 for index length)
• Byte 56-63: separator? (FF FF FF FF FF FF FF FF)
• Byte 64-x: string list (xmodelsurfs?) --> not always, depends on the content (required offsets are in the FF header)*

* under re-investigation

Example: smallest possible fastfile (mod.ff)

29 00 00 00  decompressed fastfile size minus 44 
00 00 00 00  
1C 00 00 00  unknown

00 00 00 00 00 00 00 00 00 00 00 00

0D 00 00 00  data size in memory (?)

00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

01 00 00 00  one index entry

FF FF FF FF 1F 00 00 00  a rawfile
FF FF FF FF FF FF FF FF  index end

00 00 00 00  rawfile length (empty, no errors occured during compilation)
FF FF FF FF  separator
6D 6F 64 00  fastfile name as rawfile name
         00  rawfile string (empty)

Example: mp_backlot_load.ff

Contains:

• 2x shaders
• 3x materials
• (1x fastfile name)

Materials:

1. $victorybackdrop - mile_high_victory_screen.iwi (699,092 bytes)
2. $defeatbackdrop - (none)
3. $levelbriefing - loadscreen_mp_backlot.iwi (524,316 bytes)

-----------  [start of file and header]

71 03 00 00  Decompressed file size - 44 (hex: 371 + 2C = 39D; dec: 881 + 44 = 925)

B8 AA 12 00  Size of both IWI's the materials in FF point to - 56 (2x28, 28 bytes = IWI header)
             0x12AAB8          = 1,223,352
             699,092 + 524,316 = 1,223,408
             difference:                56

A4 00 00 00  Might be a flag

00 00 00 00  Unknown
00 00 00 00  Unknown
00 00 00 00  Unknown
E5 00 00 00  Size of fastfile in memory (that is at least without header and separators, and maybe more)
00 00 00 00  Unknown
00 00 00 00  Unknown
00 00 00 00  Unknown
00 00 00 00  Unknown

-----------  [end of header] (44 bytes), [start of index]

00 00 00 00  Unknown, is sometimes FF FF FF FF

06 00 00 00  Number of records in index (6)

FF FF FF FF  05 00 00 00  1st: shader
FF FF FF FF  05 00 00 00  2nd: shader
FF FF FF FF  04 00 00 00  3rd: material
FF FF FF FF  04 00 00 00  4th: material
FF FF FF FF  04 00 00 00  5th: material
FF FF FF FF  1F 00 00 00  6th: rawfile

-----------  [end of index]

Index

4 byte identifier (fastfile entry type), 4 byte offset (FF FF FF FF = -1, meaning right after the previous block?)

Separator (FF FF FF FF) after last 4-byte couple + 00 00 00 00 ?

Types

• 00 00 00 01 - physpreset
• 00 00 00 02 - xanim
• 00 00 00 03 - xmodel
• 00 00 00 04 - material
• 00 00 00 05 - shader / techset
• 00 00 00 06 - image?
• 00 00 00 07 - soundalias (+audio data)
• 00 00 00 0B - map_ents (d3dbsp)
• 00 00 00 0C - related to col_map_mp (d3dbsp)
• 00 00 00 0E - related to game_map_mp (d3dbsp)
• 00 00 00 10 - related to col_map_mp / gfx_map (d3dbsp)
• 00 00 00 13 - font
• 00 00 00 14 - menu
• 00 00 00 16 - localized string asset
• 00 00 00 17 - physpreset??
• 00 00 00 19 - fx
• 00 00 00 1F - raw file
• 00 00 00 20 - stringtable (comma separated list)

Content

Since there is a FF FF FF FF separator in front of every block and none at the very end (after last 1F-block), we can assume that each block has a start separation and no end separation. Any FF FF FF FFs you find after data blocks are actually the start separators of the next one block. This means the index end separator is rather FF FF FF FF than 8x FF!

00 00 00 04 (material)

-----------  [block start]

FF FF FF FF  start separation

00 2B 01 01  First 4 bytes seem to be 4 (unsigned) char's.
             1st byte is so far always 0x00 (perhaps it's a short together with the second byte).
             2nd byte so far either 0x2B or 0x00, I think this is a length.
             3rd byte so far 0x01, when 0x00 (all 4) the content is only a number of 0x00's, then the filename (0x00 termination)
             4th byte either 0x00 or 0x01. Can be a length or a bool.
             Then 0x10 bytes of more header.
FF FF FF FF  Separator
00           When fourth byte is 01, there's a 0x00 following?             
             Then there's garbage (0xFF) and some shorts it looks of length in the second byte.
             Then more information...
02 00 0E E0  Often ends with this

-----------  [block end]

00 00 00 05 (shader)

A shader consists of 3 levels. One, the top level, consists of a header (with a name inside). Then the next level is a pack of shaders, it has a header, shaders and at the end a name. Then the bottom level is the shader itself, it has a header (containing a name of the source file and the length of the data) and the raw shader data.

-----------  [block start]

FF FF FF FF  Start separation

FF FF FF FF  ID/reference? First is always 0xFFFFFFFF, later on you can also see 0x??????40
00 00 00 00  0x18 bytes of 0x00
FF FF FF FF  Or 0x00000000 (if no content further on)
00 00 00 00  0x74 bytes which are often 0x00
?? ?? .. 00  Then techset file name (0x00 termination) if fist dword is 0xFFFFFFFF?

-----------  [start of shader pack]

             1 dword of ID/reference?
             1 dword of some character-length options / flags?
             3 dwords of ID/references?
             1 dword of some character-length sizes
FF FF FF FF  Separator
             0x64 bytes of some short length options / flags? For the first 05 entry? Or if the references above are 0xFFFFFFFF?

-----------  [start of shader]

             2 dwords of references
FF FF FF FF  Separator
?? ?? 00 00  2 shorts, first short multiplied by 4 gives the length of the shader, second is a flag? When set no string follows?
?? ?? .. 00  String if previous second short is 00 00, 0x00 terminated (name of shader source file)
             Raw shader file (except the first 4 bytes, which, in the shader file, denotes the length of the file)
FF FF 00 00  which ends with this (always?)

-----------  [end of shader]

             Can either have more shaders (go back to [start of shader]) or shaders have ended

             Variable length of 'stuff'
41 10 AB A0  which ends with this (always?)
?? ?? .. 00  String, 0x00 terminated (technique file name)

-----------  [end of shader pack]

             Can either have more shader packs (go back to [start of shader pack]) or shader packs have ended

-----------  [block end]

00 00 00 1F (rawfile)

A rawfile. Often a rawfile named right after the fastfile's name is implicitly added. It contains the errors thrown when it was compiling / linking.

-----------  [block start]

FF FF FF FF  Start separation

64 00 00 00  Length of the file's content (example: decimal 100; does not count the string termination!)

FF FF FF FF  Separator

6D 6F 64 00  File name or if it's the last 1F-block FastFile name (example: mod.)
?? ?? .. 00  Plain text content of length above (string, terminated by 0x00)

-----------  [block end OR even end of file]

00 00 00 20 (stringtable)

A compiled stringtable (comma separated list, *.csv). Even values are stored as strings.

-----------  [block start]

FF FF FF FF  start separation

05 00 00 00  (int) columns
02 00 00 00  (int) rows

FF FF FF FF  separator
?? ?? .. 00  filename (path/*.csv) + string termination (0x00)

-----------  [table cell description]

?? ?? ?? ??  columns * rows integers (5 * 2 * 4 = 40 bytes in total, 5 * 2 = 10 cells)
             FF FF FF FF - if the string wasn't used before and follows after the cell description
             ?? ?? ?? 4? - if an identical string was already used (in another csv). It is some kind of memory offset (pointer). Base address calculation not figured out yet (0x40000072 ?)

-----------  [cell content / strings]

?? ?? .. 00  string + termination (0x00)
             Number of strings = amount of FF FF FF FFs in table cell description

-----------  [block end]

--CoDEmanX 01:45, 8 December 2009 (UTC)

--Daevius 14:24, 5 November 2009 (UTC)