Make sure the VPK header is always initialized to 0 instead of actual values, as otherwise it would cause the reader to succeed on truncated files if at least the first byte matches that of the VPK magic. This also lead to the simplification of the tree header writer; instead of writing the tree size apart from the header, write the whole thing at once.
* Make sure the workspace path actually exists before attempting to pack it.
* Make sure the VPK directory tree file was parsed correctly before unpacking store.
* Log debug output for each pack operation to a file.
* Fix bug in s_DirFileRegex regex pattern, which would include a trailing '_' in the context part of the directory tree file name.
* Fix bug in 'GetLevelName()' and 'GetDirNameParts()' causing it to parse the path as well, prune the path before running the regex.
* Renamed 'GetLevelName()' to 'PackedStore_GetDirLevelName()'.
* Renamed 'GetDirNameParts()' to 'PackedStore_GetDirNameParts()'.
* Write a front-end enable file when building client VPK's.
We do this as if we happen to not have any manifest whatsoever, but already opened the block file, and the block file already existed and was still valid, it would be emptied out. Only open it if we are actually going to write into it!
Make the utility methods statics within the implementation so it could be used by other structs/classes of the VPK builder as well, these utility functions weren't useful outside the class. This also allowed for the removal of the singleton which had no purpose whatsoever.
The KeyValues class belongs here. Also reimplemented most loading methods for KeyValues, and adjusted the VPK building code to account for it. Pointers to the engine's implementation of KeyValues have been moved to a separate header ('keyvalues_iface.h'), as this allows external tools code to utilize the standalone KeyValues class implementation. Playlist utilities are completely separated from the KeyValues header; these have nothing to do with KeyValues other than manipulating a global KeyValues object for the playlists, and thus have been named as such and moved to rtech/playlists.
- Only process referenced pack files, previously, if an entry in the VPK file was build into a pack of index 128, the code would generate a list of pack files from 0 to 128.
- Added documentation to VPK structure.
This commit fixes the following bugs:
- Use of STL map on CUtlString, CUtlString does not have a lesser operator, thus it always fails to check whether or not the string is already present in the map. The tree structure cannot contain duplicates per tree scope.
- Use of backward slashes in vpk file tree structure, path separators should always be a forwards slash. The code was originally designed to be forwards slash only, but some paths were adjusted to take backwards slash, as the KeyValues class treats the '/' char as a delimiter for subfields. This has therefore lead into the whole code changing to it. The code has now been changed to the point where only the KeyValues utilities take the backward slash paths.
- Trailing slash character on file path, that ended up getting written as-is in the path tree.
The VPK tree builder is now part of a separate nested class in 'VPKDir_t'.
Light code cleanup: use references instead of pointers as much as possible, additional use of const qualifiers, and usage of provided iteration macros for CUtlVector.
Should be made static to avoid unwanted bloat in compiled output, as each translation unit will create a new initializer/destructor. -10KiB compiled code for GameSDK.
This implementation was done to calculate buffer CRC on the go, as that was what the VPK file system uses for the file crc's. But it later turned out to be unnecessary, and was never reverted.
These methods where only used for and by the VPKDir_t structure.
Also renamed variable and method names to reflect new changes regarding the use of KeyValues by creating a structure with all values defined during file lookup rather than doing this during the pack process.
The CPackedStore class now heavily relies on the engine's FileSystem and KeyValues API. Since its part of the engine, it would make more sense to utilize the engine's features instead. It also allows for easier modifications.
* The logic behind the build process hasn't changed.
* Don't call copy constructor on entry blocks when building the directory file.
* Don't call copy constructor for string when we don't want to sanitize the directory file name.
The sanitizer can retrieve the directory file name from the block file name if this is passed instead. This was broken in the previous system. Sanitization can be opted out if the 3rd parameter of the 'fs_vpk_unpack' command is omitted.
* Use reference to array index.
* Only obtain array size once during loop.
* Use static buffer of 1024*1024 for decompressing.
* Removed extraneous goto.
* Fixed bug where shared data would be mapped wrong due to the last optimization.
* This increased packing performance by 40/60 seconds when rebuilding all vpk's.
More detailed operation statistics.
Discard VPK directory files who's header doesn't match engine requirements.
Separated manifest properly between language and context (you can now package for each language and context from a single workspace).
Cast to proper types.
Set the archive offset only instead of the entire block. The entry could still have its own flags. The only thing needed for saving data is sharing the same offset and size.
Fixed bug where the compressed buffer size equals the source buffer size getting packed into the VPK chunk. In all occurrences this resulted in corrupted data upon export.
The reason for data being corrupt upon export is because the engine checks the equality of the compressed and decompressed size fields in the directory file, if they are equal, the engine doesn't attempt to decompress the block. So we end up with a still compressed block on the disk (technically not corrupt in the context of the compression lib, but useless as-is on the disk).
If a compressed file doesn't get lower in size we are better out storing it rather than compressing it.
Added a new condition in lzham::lzham_lib_compress_memory which checks source and destination buffer size equality.
Initial proper implementation pending cleanup.
The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not.
When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them.
I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag).
User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them.
fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only>
locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk).
The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
VPK's could now be repacked with compression and multiple files. The output is identical to the input as far as currently tested (mp_rr_canyonlands_staging).
The 'fs_pack_vpk' allows compressing specific files into a VPK (compressed). This implementation has better compression ratio's then currently available VPK tools with compression (at the cost of speed).
The implementation is early, a directory iterator with a files for flags is planned soon to fully rebuild VPK's on-demand. A dedicated standalone program is also planned.
The output dir and archive formats should match 1:1 with the original game
