Old Source Engine VPK mounting system

Table of contents

Introduction

Before the Steampipe update, game content was kept inside GCF archives. Those would get mounted to the Source Engine's virtual file system and the game would access them from the archive itself, without extracting. (Well, for the most part, because GCF archives have a flag that'd make Steam extract necessary files before the game is first ran.) However, after the Steampipe update, the GCF format was scrapped and completely removed from Steam. Alongside GCF, the legacy Steam API (steam.dll) was also scrapped in favor of steam_api.dll and steamclient.dll. To not break old games, Valve decided to make the steam2wrapper.

About the Steam2 Wrapper

The Steam2 wrapper is a steam.dll replacement that acts kind-of as a proxy between the new post-Steampipe Steam and the game. It's the thing that handles the VPK mounting and makes the VGUI friends dialog still work. Note that the version of Steam2 wrapper inside the main Steam directory isn't the one that the old Source games use, instead they come with an old build of it dated Wed Feb 26 2014. The Steam2 wrapper uses an internal Valve library called vpklib to open VPK files and provide it to the game's virtual file system, the library is also used in newer Source Engine builds for the filesystem module and vpk.exe.

Into the VPK mounting system

The VPK mounting process happens entirely in steam.dll, well, at least the "first" part of it because filesystem_steam tries to redo it afterwards as well. The process goes like this:

CFileSystem_Steam::MountSteamContent tries to load steam.dll -> steam.SteamStartup -> sub_1000AA70 (vpk_entry) -> inside vpk_entry app dependencies are enumerated -> sub_1000A960 (mount_vpk)

After it loads steam.dll, it tries to do the process itself again which goes like this:

CFileSystem_Steam::MountSteamContent-> steam.MountFilesytem -> mount_vpk

This behaviour can be also observed from the Windows' debug log through software like DbgView. Enabling debug output from the Steam2 wrapper is later explained further in this article.

00000027    3.56093121  [20780] Mounting d:\steam\steamapps\common\source sdk base\vpks\depot_215_dir.vpk   
00000028    3.56212473  [20780] Mounting d:\steam\steamapps\common\source sdk base\vpks\depot_208_dir.vpk   
00000029    3.56450486  [20780] Mounting d:\steam\steamapps\common\source sdk base\vpks\depot_207_dir.vpk   
00000030    3.56573367  [20780] Mounting d:\steam\steamapps\common\source sdk base\vpks\depot_206_dir.vpk   
00000031    3.56595469  [20780] Mounting d:\steam\steamapps\common\source sdk base\vpks\depot_212_dir.vpk   
00000032    3.56704235  [20780] Mounting d:\steam\steamapps\common\source sdk base\vpks\depot_213_dir.vpk   
00000033    3.56837034  [20780] Mounting d:\steam\steamapps\common\source sdk base\vpks\depot_308_dir.vpk   
00000034    3.56847954  [20780] Mounting d:\steam\steamapps\common\source sdk base\vpks\depot_309_dir.vpk   
00000035    3.56900430  [20780] Mounting d:\steam\steamapps\common\source sdk base\vpks\depot_381_dir.vpk   
00000036    3.57062411  [20780] Mounting d:\steam\steamapps\common\source sdk base\vpks\depot_421_dir.vpk   
00000037    3.57099271  [20780] Mounting d:\steam\steamapps\common\source sdk base\vpks\depot_422_dir.vpk   
00000038    3.57142329  [20780] Mounting d:\steam\steamapps\common\source sdk base\vpks\depot_423_dir.vpk   
00000039    3.57181120  [20780] SteamMountAppFilesystem

…

00000579    7.42786551  [9408] SteamStartup 
00000580    7.42814875  [9408] SteamEnumerateApp appid 320, dependencies 3  
00000581    7.42829370  [9408] SteamEnumerateAppDependency AppID 320, depot 321 
00000582    7.42840338  [9408] SteamIsAppSubscribed appid 321: yes  
00000583    7.42862082  [9408] SteamMountFilesystem uDepotId 321, path ""   
00000584    7.42874336  [9408] SteamEnumerateAppDependency AppID 320, depot 320 
00000585    7.42884779  [9408] SteamIsAppSubscribed appid 320: yes  
00000586    7.42898655  [9408] SteamMountFilesystem uDepotId 320, path ""   
00000587    7.42906332  [9408] SteamEnumerateAppDependency AppID 320, depot 232371  
00000588    7.42917204  [9408] SteamIsAppSubscribed appid 232371: yes   
00000589    7.42936468  [9408] SteamMountFilesystem uDepotId 232371, path ""    
...
00000626    7.43344784  [9408] SteamEnumerateAppDependency AppID 215, depot 422 
00000627    7.43352175  [9408] SteamIsAppSubscribed appid 422: yes  
00000628    7.43381691  [9408] Mounting d:\steam\steamapps\common\source sdk base\vpks\depot_422_dir.vpk    
00000629    7.43383741  [9408] SteamMountFilesystem uDepotId 422, path ""   
00000630    7.43391275  [9408] SteamEnumerateAppDependency AppID 215, depot 423 
00000631    7.43398380  [9408] SteamIsAppSubscribed appid 423: yes  
00000632    7.43417740  [9408] Mounting d:\steam\steamapps\common\source sdk base\vpks\depot_423_dir.vpk    
00000633    7.43419743  [9408] SteamMountFilesystem uDepotId 423, path ""   
00000634    7.43427420  [9408] SteamEnumerateAppDependency AppID 215, depot 1004    
00000635    7.43434620  [9408] SteamIsAppSubscribed appid 1004: yes 
00000636    7.43447208  [9408] SteamMountFilesystem uDepotId 1004, path ""

Upon further investigation of the Steam2 wrapper, I have found out that it tries to read a file called "steam_vpks.vdf", which to my knowledge has not yet been documented anywhere online. However, I was able to reverse engineer the format and get it to work. The format looks as follows:

steam_vpks.vdf example

"does_this_even_matter"
{
 "spewfileio" "true"
 "mount"
 {
  "vpk" "vpks/depot_206"
  "vpk" "vpks/depot_207"
  "vpk" "vpks/depot_208"
  "vpk" "vpks/depot_212"
  "vpk" "vpks/depot_215"
  // other vpks can be mounted here as well
 }
}

… and here is a snippet of pseudo code generated by IDA. This is a part of the vpk_entry function:

 if ( (unsigned __int8)sub_10011A50(v3, (char)"vpk", (int)hSteamVPKs_idk, 0) )
  {
    v5 = (_DWORD *)sub_10009BA0(&Str);
    SubKey = GetSubKey(v5, "mount", 0);
    v7 = (_DWORD *)sub_10009BA0(&Str);
    String = (wchar_t *)GetString(v7, "spewfileio");
    bDebug = sub_10010010(String, 0, 0);
    if ( SubKey )
    {
      for ( i = (_DWORD *)sub_1000FE10(SubKey); i; i = (_DWORD *)sub_1000FE30(i) )
      {
        v10 = GetString(i, 0);
        v11 = (char *)sub_10010FD0(v10, (int)&unk_100397BA);
        mount_vpk(v11);
      }
      v12 = 1;
    }

The steam_vpks.vdf file disables the automatic dependency enumeration and mounting by taking a different code path, so you have to include the base VPKs in the steam_vpks.vdf file. The bDebug variable is also set by the -debugsteam2wrapper command line parameter, however as you can see, it's also overriden in vpk_entry if you have the steam_vpks.vdf file and it's valid. The "debug mode" also logs every find, stat, open and close call, even the ones which end up reading directly from the filesystem, so keep in mind it's probably a good idea to leave it disabled while you don't need it.

So, how do I mount VPK files?

I've found 3 ways to do it:

SteamMountFilesystem

SteamMountFilesystem is the way filesystem_steam.dll handles it. There isn't much to say about it as just calling the function should work, but I wouldn't call it the best way as it looks for the depot in a very specific path, that is:

<game_path>/vpks/depot_<depot_id>_dir.vpk

Calling mount_vpk directly

This one is also a rather simple one. Simply call the function with the path and the VPK file should get mounted. The path should be relative to the directory hl2.exe is in, for example in case of Source SDK Base 2006, the path would be relative to steamapps\common\Source SDK Base.

The steam_vpks.vdf file

Start by copying the example steam_vpks.vdf I provided earlier and replace the example VPKs with the ones that your base game needs to work (such as depots 206, 207 and 208). Then at the end you can add the VPK you want to mount, you should also probably make spewfileio false to not get your debug log spammed.

Out of the three ways I listed, I only thoroughly tested the last two. If you can't see your content, consider making a fake game directory. As in, structure your VPK file like the following:

and add it to your gameinfo.txt as:

            Game                fake_game_dir

A sidenote about the VPK files coming with the game and the vpk(lib) version used

I deem the VPK files and the vpklib version built into the steam.dll a very specific one. The VPK files coming with all Valve games and licensed games that utilize this mounting system are special, specifically it's pretty much a valid VPK2 file except for the checksums at the end. The tree and chunk checksums have been replaced by something that looks like corruption, which might've been caused by the checksums being uninitialized memory aligned next to each other. Here's how it looks like in the depot_215_dir.vpk file from Source SDK Base 2006. This pattern can also be seen in the depot_307_dir.vpk file from Source SDK Base 2007.

    {
        "PersonaName"       "martin

Here is another pattern, this one is found in depot_309_dir.vpk and depot_308_dir.vpk from Source SDK Base 2007.

    {
        "0"     "0100000065a3e27eb

As to what the actual cause of the corruption is, I can only speculate. The VPK files might've been made by some internal vpk.exe build, which licensees would also have, because this can be observed in both Valve and licensed games as I have stated before.

Small demo

Offsets/binary information

Hashes of binaries examined

    File: FileSystem_Steam.dll
  CRC-32: 554bdd04
   SHA-1: b4df74a7d66128c7698c45e21ef1bcd802c2e553
 SHA-256: 60389fc40694cf54646f03f84471f2d1994af760de76a77498b7c0924b40ed26
 SHA-512: 907ebb08faebc9a10941bdb72ac90be6877906ecc2c13208ee9bc7ceb682ab4d7723d9418817326a35478b1db9c5dff64f1025db4b2fc9ad61905a8e30b34cfc

    File: steam.dll
  CRC-32: cc3c72e4
   SHA-1: 7ea23a50f764e3dac18ec2609607469d92088f74
 SHA-256: 18ccb20a1c8ab916a2f93aa846d8c9730fd80c0d279cdee0ac7987d8a14a7f71
 SHA-512: 676bdc922cffd1228f98a7375339b50d3ddf87cb858031cd1c0a8e871c9b013687b83f78942281c3236ee2b555a1f065c3ce0ff57bca9a6771f321f018256182

Offsets

Binary Function Name Function Address/RVA
steam.dll mount_vpk 0xA960 from base
steam.dll vpk_entry 0xAA70 from base
filesystem_steam.dll CFileSystem_Steam::MountSteamContent 0x16C70 from base
steam.dll SteamMountFilesystem n/a (GetProcAddress)

Example code

Example code used while making and testing this can be found here. You will need Minhook and Source SDK in order to compile it. If you do not know anything about C++, it's probably not a great idea to do so.

Special thanks

… and everyone else that I've forgotten.

Contact

You can contact me via email here. Feedback is greatly appreciated.