Call of Duty 4: FastFile Format

From COD Modding & Mapping Wiki
Revision as of 15:04, 24 June 2011 by Randallphillips (talk | contribs)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigation Jump to search

*** Article and investigation in early progress ***

Preface

Be aware, FF files are little endian, so swap those bytes 'round!

The data block types (also called "index identifier") are written in big endian in this article, e.g. 00 00 00 1F is actually 1F 00 00 00 in ff, although ints and floats in the examples are kept little endian!


The global file layout starts with a header. After which you get a string list (model tag names, animation notetracks, maybe more; not always present). Followed by an index list. This list gives information about the rest of the file, which is a concatenation of compiled files (reinterpreted and linked data generated from source files like d3dbsp and stringtables, hereinafter referred to as data blocks).

  • Header
  • String list index
  • String list
  • Data block index
  • Data blocks

Each file is separated by the separation key: 0xFFFFFFFF. But be aware, each file itself can also contain such a sequence which is not meant to be the separation key! So it is necessary to understand each data type in the FF file.

Edit: The latest research led to the assumption, that it's no real separation key but an address pointer (which is 0xFFFFFFFF if the data follows or 0x??????4? if the content is located elsewhere).

It seems to be 0xFFFFFFFF replaces pointers used during compilation, leading to the following data block structure in case of some data blocks (at least rawfile):

  • Block header (containing one or more 0xFFFFFFFF pointers, size needs to be known to properly read files)
  • Block data (one for each 0xFFFFFFFF data, can be anything from a string to raw binary data, possibly a 00 marks the end for strings?)


Header

(Note: not up to date and not generalized)

  • Byte 0-3: decompressed fastfile size minus 44 (0x2C)
  • Byte 4-7: about the total size of referenced data, e.g. the required memory for IWI textures if material files are in the ff
  • Byte 8-11: unknown, might be a flag
  • Byte 12-23: unknown
  • Byte 24-27: somehow related to ff size?
  • Byte 28-43: unknown
  • Byte 44-47: equal to number of entries in "1st index" and amount of (model tag/joint/notetrack) strings (times 4 for index length)
  • Byte 48-51: separator? (FF FF FF FF)
  • Byte 52-55: number of records* ("2nd index", times 8 for index length)
  • Byte 56-63: separator? (FF FF FF FF FF FF FF FF)
  • Byte 64-x: string list (xmodelsurfs?) --> not always, depends on the content (required offsets are in the FF header)*

* under re-investigation


Example: smallest possible fastfile (mod.ff)

29 00 00 00  decompressed fastfile size minus 44 
00 00 00 00  
1C 00 00 00  unknown

00 00 00 00 00 00 00 00 00 00 00 00

0D 00 00 00  data size in memory (?)

00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

01 00 00 00  one index entry

FF FF FF FF 1F 00 00 00  a rawfile

FF FF FF FF  index end

FF FF FF FF  start separator
00 00 00 00  rawfile length (empty, no errors occured during compilation)
FF FF FF FF  separator
6D 6F 64 00  fastfile name as rawfile name
         00  rawfile string (empty)


Example: mp_backlot_load.ff

Contains:

  • 2x shaders
  • 3x materials
  • (1x fastfile name)


Materials:

  1. $victorybackdrop - mile_high_victory_screen.iwi (699,092 bytes)
  2. $defeatbackdrop - (none)
  3. $levelbriefing - loadscreen_mp_backlot.iwi (524,316 bytes)
-----------  [start of file and header]

71 03 00 00  Decompressed file size - 44 (hex: 371 + 2C = 39D; dec: 881 + 44 = 925)

B8 AA 12 00  Size of both IWI's the materials in FF point to - 56 (2x28, 28 bytes = IWI header)
             0x12AAB8          = 1,223,352
             699,092 + 524,316 = 1,223,408
             difference:                56

A4 00 00 00  Might be a flag

00 00 00 00  Unknown
00 00 00 00  Unknown
00 00 00 00  Unknown
E5 00 00 00  Size of fastfile in memory (that is at least without header and separators, and maybe more)
00 00 00 00  Unknown
00 00 00 00  Unknown
00 00 00 00  Unknown
00 00 00 00  Unknown

-----------  [end of header] (44 bytes), [start of index]

00 00 00 00  Unknown, is sometimes FF FF FF FF

06 00 00 00  Number of records in index (6)

FF FF FF FF  05 00 00 00  1st: shader
FF FF FF FF  05 00 00 00  2nd: shader
FF FF FF FF  04 00 00 00  3rd: material
FF FF FF FF  04 00 00 00  4th: material
FF FF FF FF  04 00 00 00  5th: material
FF FF FF FF  1F 00 00 00  6th: rawfile

FF FF FF FF  end separator (the following FF FF FF FF is the start seperator of the next data block!)

-----------  [end of index]


Index

Starts with an int telling the number of dword couples. Such a couple is composed of 2 ints:

4 byte offset (FF FF FF FF = -1, meaning right after the previous block?)
4 byte identifier (data block type)

Separator (FF FF FF FF) after last couple

Types

  • xmodelpieces = 0,
  • physpreset = 1,
  • xanim = 2,
  • xmodel = 3,
  • material = 4,
  • pixelshader = 5,
  • techset = 6,
  • image = 7,
  • sndcurve = 8,
  • loaded_sound = 9,
  • col_map_sp = 0x0a,
  • col_map_mp = 0x0b,
  • com_map = 0x0c,
  • game_map_sp = 0x0d,
  • game_map_mp = 0x0e,
  • map_ents = 0x0f,
  • gfx_map = 0x10,
  • lightdef = 0x11,
  • ui_map = 0x12,
  • font = 0x13,
  • menufile = 0x14,
  • menu = 0x15,
  • localize = 0x16,
  • weapon = 0x17,
  • snddriverglobals = 0x18,
  • impactfx = 0x19,
  • aitype = 0x1a,
  • mptype = 0x1b,
  • character = 0x1c,
  • xmodelalias = 0x1d,
  • rawfile = 0x1f,
  • stringtable = 0x20

Content

Since there is a FF FF FF FF separator (respectively pointer) in front of every block and none at the very end (after last 1F-block), we can assume that each block has a start separation and no end separation. Thus, any FF FF FF FFs you find after data blocks are actually the start separators of the next block. This means the index end separator is rather FFFFFFFF than FFFFFFFFFFFFFFFF!


00 00 00 04 (material)

-----------  [block start]

00 2B 01 01  First 4 bytes seem to be 4 (unsigned) char's.
             1st byte is so far always 0x00 (perhaps it's a short together with the second byte).
             2nd byte so far either 0x2B or 0x00, I think this is a length.
             3rd byte so far 0x01, when 0x00 (all 4) the content is only a number of 0x00's, then the filename (0x00 termination)
             4th byte either 0x00 or 0x01. Can be a length or a bool.
             Then 0x10 bytes of more header.
FF FF FF FF  Separator
00           When fourth byte is 01, there's a 0x00 following?             
             Then there's garbage (0xFF) and some shorts it looks of length in the second byte.
             Then more information...
02 00 0E E0  Often ends with this

-----------  [block end]

00 00 00 05 (shader)

A shader consists of 3 levels. One, the top level, consists of a header (with a name inside). Then the next level is a pack of shaders, it has a header, shaders and at the end a name. Then the bottom level is the shader itself, it has a header (containing a name of the source file and the length of the data) and the raw shader data. 

-----------  [block start]

             36 pointers (0x90 bytes) (bytes 0x19 up to 0x22 say if we get any content at all?)
?? ?? .. 00  Then techset file name (0x00 termination) if pointer is 0xFFFFFFFF?

-----------  [start of shader pack]

             pointer
             1 dword of some character-length options / flags?
             3 pointers
             1 dword of some character-length sizes
FF FF FF FF  Separator
             0x64 bytes of some short length options / flags? If the second pointer above is 0xFFFFFFFF

-----------  [start of shader]

             pointer, if not 0xFFFFFFFF, no filename is further on
             pointer, often 00 00 00 00
FF FF FF FF  Separator
?? ?? 00 00  2 shorts, first short multiplied by 4 gives the length of the shader, second is a flag?
?? ?? .. 00  String if first pointer is 0xFFFFFFFF, 0x00 terminated (name of shader source file)
             Raw shader file (except the first 4 bytes, which, in the shader file, denotes the length of the file)
FF FF 00 00  which ends with this (always?)

-----------  [end of shader]

             Can either have more shaders (go back to [start of shader]) or shaders have ended

             Variable length of 'stuff' (technique commands?)
41 10 AB A0  which ends with this (always?)
?? ?? .. 00  String, 0x00 terminated (technique file name)

-----------  [end of shader pack]

             Can either have more shader packs (go back to [start of shader pack]) or shader packs have ended

-----------  [block end]

00 00 00 06 (image)

Stores some information about the specified IWI in fastfile. Does not include the actual image data. Length is 52 bytes + IWI name (+1 for string termination). 

-----------  [block start]

03 00 00 00
FE FF FF FF  separator?
00 00 00 00
00 00 00 00

70 55 01 00  IWI size minus 28 (0x1C)
70 55 01 00  same

00 01 00 01
01 00 03 00
FF FF FF FF  separator?

73 70 .. 00  IWI name without file extension (string + termination, variable size)

00 00 00 01
00 01 01 00

44 58 54 35 00 00 00 00  fixed size char[] for type? (here: DXT5....)

-----------  [block end]

00 00 00 13 (font)

See also Call of Duty: Font System. Implicitly adds shaders. 

-----------  [block start] 216 byte + 2 strings
18 00 00 00  int font_size
BF 00 00 00  int number_of_records
FE FF FF FF  ?
FE FF FF FF  ?
FF FF FF FF  separator
66 6F .. 00  string font_name (e.g. fonts/bigFont.)
FF FF FF FF  separator
00 2B 01 01  ?
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
FF FF FF FF 00 FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF
FF FF 01 00 01 00 03 00
0D 00 00 40  pointer?
FF FF FF FF
00 00 00 00
FF FF FF FF
66 6F .. 00  string font_material (e.g. fonts/gamefonts_pc.)
41 10 AB A0 63 70 EB 00  ?

-----------  [image block] 112 bytes + 1 string

FF FF FF FF  start separator / pointer
03 00 00 00
FE FF FF FF  separator?
00 00 01 00  ?
00 00 00 00
D0 AA 0A 00  IWI size minus 28 (0x1C)
D0 AA 0A 00  same
00 02 00 04
01 00 03 00  ?
FF FF FF FF  separator
67 61 .. 00  IWI name
00 00 00 02
00 04 01 00  ?
44 58 54 35
00 00 00 00  compression (here: DXT5....)

-----------  [unknown block] 168 bytes

65 91 28 19
02 00 0E E0  ?
FF FF FF FF  separator?
00 04 01 01  ?
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
FF FF FF FF 00 FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF
01 00 01 00 03 00  ?
0D 00 00 40  pointer? (same as above)
FF FF FF FF  separator?
00 00 00 00

-----------  [font block] 48 bytes + 1 string + number_of_records * 24 bytes

FF FF FF FF  separator/pointer
66 6F .. 00  font effect name? (e.g. fonts/gamefonts_pc_glow.)
41 10 AB A0
63 70 EB 00
95 06 00 40
25 91 28 19
02 00 0E E0  ?

?? ?? ?? ??  font file entries (24 bytes each)

-----------  [block end]

00 00 00 16 (localized string asset)

A localized string asset consists of an alias name and a translated string. During compilation, it takes the given alias name (REFERENCE) from *.str file and prepands the *.str file's name in upper-case plus an underscore. Example:


example.str 

VERSION             "1"
CONFIG              "C:\trees\cod3\cod3\bin\StringEd.cfg"
FILENOTES           ""
//--------------------------------

REFERENCE           MARINES_NAME
LANG_ENGLISH        "US Airforce"

//--------------------------------
ENDMARKER


mod.ff 

ÿÿÿÿ
ÿÿÿÿ
EXAMPLE_MARINES_NAME.
US Airforce.


Length is 8 bytes + string sizes (string count = 0-2, depends on the first two ints). 

-----------  [block start]

FF FF FF FF  either pointer or FFFFFFFF if the string wasn't used before
FF FF FF FF  same. If two pointers are used here, then no strings will follow.
55 53 .. 00  alias name for the localized
53 54 .. 00  actual localized string

-----------  [block end]

00 00 00 1F (rawfile)

A rawfile. Often a rawfile named right after the fastfile's name is implicitly added. It contains the errors thrown when it was compiling / linking. 

-----------  [block start]

64 00 00 00  Length of the file's content (example: decimal 100; does not count the string termination!)

FF FF FF FF  Separator

6D 6F 64 00  File name or if it's the last 1F-block FastFile name (example: mod.)
?? ?? .. 00  Plain text content of length above (string, terminated by 0x00)

-----------  [block end OR even end of file]

00 00 00 20 (stringtable)

A compiled stringtable (comma separated list, *.csv). Even values are stored as strings. 

-----------  [block start]

05 00 00 00  (int) columns
02 00 00 00  (int) rows

FF FF FF FF  separator
?? ?? .. 00  filename (path/*.csv) + string termination (0x00)

-----------  [table cell description]

?? ?? ?? ??  columns * rows integers (5 * 2 * 4 = 40 bytes in total, 5 * 2 = 10 cells)
             FF FF FF FF - if the string wasn't used before and follows after the cell description
             ?? ?? ?? 4? - if an identical string was already used (in another csv). It is some kind of memory offset (pointer). Base address calculation not figured out yet (0x40000072 ?)

-----------  [cell content / strings]

?? ?? .. 00  string + termination (0x00)
             Number of strings = amount of FF FF FF FFs in table cell description

-----------  [block end]

--NTAuthority 10:54, 3 June 2010 (UTC) (some tiny information)

--CoDEmanX 01:45, 8 December 2009 (UTC)

--Daevius 23:00, 8 December 2009 (UTC)




resume writing services

Retrieved from "https://wiki.zeroy.com/index.php?title=Call_of_Duty_4:_FastFile_Format&oldid=15874"