R5Reloaded/r5sdk
1
0
Fork 0
mirror of https://github.com/Mauler125/r5sdk.git synced 2025-02-09 19:15:03 +01:00
Code Issues Packages Projects Releases Wiki Activity
r5sdk/r5dev/vpklib/packedstore.cpp

986 lines
37 KiB
C++
Raw Normal View History

CPackedStore: light optimizations * Don't call copy constructor on entry blocks when building the directory file. * Don't call copy constructor for string when we don't want to sanitize the directory file name.
2022-10-12 23:39:26 +02:00
//=============================================================================//
//
// Purpose: Valve Pak utility class.
//
//=============================================================================//
// packedstore.cpp
//
Improve consistency throughout packedstore class No modifications to the existing logic have been performed. Some parameter names have been renamed to improve code readability and maintainability.
2022-11-16 00:54:51 +01:00
// Note: VPK's are created in pairs of a directory file and pack file(s).
// - <locale><target>_<level>.bsp.pak000_dir.vpk --> directory file.
// - <target>_<level>.bsp.pak000_<patch>.vpk --> pack file.
//
// - Assets larger than 1MiB are fragmented into chunks of 1MiB or smaller (ENTRY_MAX_LEN).
// - A VPK directory file could be patched up to 512 times before full rebuild is required.
CPackedStore: light optimizations * Don't call copy constructor on entry blocks when building the directory file. * Don't call copy constructor for string when we don't want to sanitize the directory file name.
2022-10-12 23:39:26 +02:00
//
/////////////////////////////////////////////////////////////////////////////////
Update packedstore.cpp
2022-06-02 01:59:03 +02:00
#include "core/stdafx.h"
#include "tier1/cvar.h"
#include "mathlib/adler32.h"
#include "mathlib/crc32.h"
Implement data sharing algorithm in VPK packing This reduces size of the resulting VPK (checks entry block in hash map). mp_common compresses to 32mb instead of 37.3mb (original is 38.5mb).
2022-06-04 13:55:24 +02:00
#include "mathlib/sha1.h"
CPackedStore refactor The CPackedStore class now heavily relies on the engine's FileSystem and KeyValues API. Since its part of the engine, it would make more sense to utilize the engine's features instead. It also allows for easier modifications. * The logic behind the build process hasn't changed.
2022-11-23 12:18:33 +01:00
#include "filesystem/filesystem.h"
#include "vpc/keyvalues.h"
Update packedstore.cpp
2022-06-02 01:59:03 +02:00
#include "vpklib/packedstore.h"
Use unicode character set instead and light cleanup/bug fixes
2022-03-04 12:22:17 +01:00
Use static buffer for file chunking and compressing during VPK packing Avoid using new every time. Chunk can be at most 1024 * 1024.
2022-10-02 12:17:03 +02:00
//-----------------------------------------------------------------------------
Add convar for adjusting LZHAM compressor max helper threads
2022-11-17 20:37:12 +01:00
// Static buffers for fragmenting/compressing the source files and decompressing
Use static buffer for file chunking and compressing during VPK packing Avoid using new every time. Chunk can be at most 1024 * 1024.
2022-10-02 12:17:03 +02:00
//-----------------------------------------------------------------------------
More VPK compressing/decompressing optimizations * Use reference to array index. * Only obtain array size once during loop. * Use static buffer of 1024*1024 for decompressing. * Removed extraneous goto. * Fixed bug where shared data would be mapped wrong due to the last optimization. * This increased packing performance by 40/60 seconds when rebuilding all vpk's.
2022-10-02 19:24:00 +02:00
static uint8_t s_EntryBuf[ENTRY_MAX_LEN];
static uint8_t s_DecompBuf[ENTRY_MAX_LEN];
Use static buffer for file chunking and compressing during VPK packing Avoid using new every time. Chunk can be at most 1024 * 1024.
2022-10-02 12:17:03 +02:00
Update packedstore.cpp
2022-06-02 01:59:03 +02:00
//-----------------------------------------------------------------------------
// Purpose: initialize parameters for compression algorithm
//-----------------------------------------------------------------------------
void CPackedStore::InitLzCompParams(void)
{
/*| PARAMETERS ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||*/
Additional cleanup Cast to proper types. Set the archive offset only instead of the entire block. The entry could still have its own flags. The only thing needed for saving data is sharing the same offset and size.
2022-06-05 12:28:49 +02:00
m_lzCompParams.m_dict_size_log2 = VPK_DICT_SIZE;
Set VPK compression level through ConVar
2022-09-11 00:16:31 +02:00
m_lzCompParams.m_level = GetCompressionLevel();
m_lzCompParams.m_compress_flags = lzham_compress_flags::LZHAM_COMP_FLAG_DETERMINISTIC_PARSING;
Add convar for adjusting LZHAM compressor max helper threads
2022-11-17 20:37:12 +01:00
m_lzCompParams.m_max_helper_threads = fs_packedstore_max_helper_threads->GetInt();
Update packedstore.cpp
2022-06-02 01:59:03 +02:00
}
//-----------------------------------------------------------------------------
// Purpose: initialize parameters for decompression algorithm
//-----------------------------------------------------------------------------
void CPackedStore::InitLzDecompParams(void)
{
/*| PARAMETERS ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||*/
Additional cleanup Cast to proper types. Set the archive offset only instead of the entire block. The entry could still have its own flags. The only thing needed for saving data is sharing the same offset and size.
2022-06-05 12:28:49 +02:00
m_lzDecompParams.m_dict_size_log2 = VPK_DICT_SIZE;
Remove experimental CRC implementation in LZHAM This implementation was done to calculate buffer CRC on the go, as that was what the VPK file system uses for the file crc's. But it later turned out to be unnecessary, and was never reverted.
2022-12-04 01:33:10 +01:00
m_lzDecompParams.m_decompress_flags = lzham_decompress_flags::LZHAM_DECOMP_FLAG_OUTPUT_UNBUFFERED;
Update packedstore.cpp
2022-06-02 01:59:03 +02:00
m_lzDecompParams.m_struct_size = sizeof(lzham_decompress_params);
}
Set VPK compression level through ConVar
2022-09-11 00:16:31 +02:00
//-----------------------------------------------------------------------------
// Purpose: gets the LZHAM compression level
// output : lzham_compress_level
//-----------------------------------------------------------------------------
lzham_compress_level CPackedStore::GetCompressionLevel(void) const
{
const char* pszLevel = fs_packedstore_compression_level->GetString();
if(strcmp(pszLevel, "fastest") == NULL)
return lzham_compress_level::LZHAM_COMP_LEVEL_FASTEST;
else if (strcmp(pszLevel, "faster") == NULL)
return lzham_compress_level::LZHAM_COMP_LEVEL_FASTER;
else if (strcmp(pszLevel, "default") == NULL)
return lzham_compress_level::LZHAM_COMP_LEVEL_DEFAULT;
else if (strcmp(pszLevel, "better") == NULL)
return lzham_compress_level::LZHAM_COMP_LEVEL_BETTER;
else if (strcmp(pszLevel, "uber") == NULL)
return lzham_compress_level::LZHAM_COMP_LEVEL_UBER;
else
return lzham_compress_level::LZHAM_COMP_LEVEL_DEFAULT;
}
Update packedstore.cpp
2022-06-02 01:59:03 +02:00
//-----------------------------------------------------------------------------
// Purpose: obtains and returns the entry block to the vector
Move some CPackedStore methods to VPKDir_t These methods where only used for and by the VPKDir_t structure. Also renamed variable and method names to reflect new changes regarding the use of KeyValues by creating a structure with all values defined during file lookup rather than doing this during the pack process.
2022-11-23 17:09:48 +01:00
// Input : hDirectoryFile -
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
// output : vector<VPKEntryBlock_t>
Update packedstore.cpp
2022-06-02 01:59:03 +02:00
//-----------------------------------------------------------------------------
Move some CPackedStore methods to VPKDir_t These methods where only used for and by the VPKDir_t structure. Also renamed variable and method names to reflect new changes regarding the use of KeyValues by creating a structure with all values defined during file lookup rather than doing this during the pack process.
2022-11-23 17:09:48 +01:00
vector<VPKEntryBlock_t> CPackedStore::GetEntryBlocks(FileHandle_t hDirectoryFile) const
Update packedstore.cpp
2022-06-02 01:59:03 +02:00
{
string svName, svPath, svExtension;
Move some CPackedStore methods to VPKDir_t These methods where only used for and by the VPKDir_t structure. Also renamed variable and method names to reflect new changes regarding the use of KeyValues by creating a structure with all values defined during file lookup rather than doing this during the pack process.
2022-11-23 17:09:48 +01:00
vector<VPKEntryBlock_t> vEntryBlocks;
while (!(svExtension = FileSystem()->ReadString(hDirectoryFile)).empty())
Update packedstore.cpp
2022-06-02 01:59:03 +02:00
{
Move some CPackedStore methods to VPKDir_t These methods where only used for and by the VPKDir_t structure. Also renamed variable and method names to reflect new changes regarding the use of KeyValues by creating a structure with all values defined during file lookup rather than doing this during the pack process.
2022-11-23 17:09:48 +01:00
while (!(svPath = FileSystem()->ReadString(hDirectoryFile)).empty())
Update packedstore.cpp
2022-06-02 01:59:03 +02:00
{
Move some CPackedStore methods to VPKDir_t These methods where only used for and by the VPKDir_t structure. Also renamed variable and method names to reflect new changes regarding the use of KeyValues by creating a structure with all values defined during file lookup rather than doing this during the pack process.
2022-11-23 17:09:48 +01:00
while (!(svName = FileSystem()->ReadString(hDirectoryFile)).empty())
Update packedstore.cpp
2022-06-02 01:59:03 +02:00
{
Set VPK compression level through ConVar
2022-09-11 00:16:31 +02:00
const string svFilePath = FormatEntryPath(svPath, svName, svExtension);
Move some CPackedStore methods to VPKDir_t These methods where only used for and by the VPKDir_t structure. Also renamed variable and method names to reflect new changes regarding the use of KeyValues by creating a structure with all values defined during file lookup rather than doing this during the pack process.
2022-11-23 17:09:48 +01:00
vEntryBlocks.push_back(VPKEntryBlock_t(hDirectoryFile, svFilePath));
Update packedstore.cpp
2022-06-02 01:59:03 +02:00
}
}
}
Move some CPackedStore methods to VPKDir_t These methods where only used for and by the VPKDir_t structure. Also renamed variable and method names to reflect new changes regarding the use of KeyValues by creating a structure with all values defined during file lookup rather than doing this during the pack process.
2022-11-23 17:09:48 +01:00
return vEntryBlocks;
Update packedstore.cpp
2022-06-02 01:59:03 +02:00
}
//-----------------------------------------------------------------------------
// Purpose: scans the input directory and returns the paths to the vector
Move some CPackedStore methods to VPKDir_t These methods where only used for and by the VPKDir_t structure. Also renamed variable and method names to reflect new changes regarding the use of KeyValues by creating a structure with all values defined during file lookup rather than doing this during the pack process.
2022-11-23 17:09:48 +01:00
// Input : &svWorkspace -
// Output : vpk keyvalues vector of all existing entry paths
Update packedstore.cpp
2022-06-02 01:59:03 +02:00
//-----------------------------------------------------------------------------
Move some CPackedStore methods to VPKDir_t These methods where only used for and by the VPKDir_t structure. Also renamed variable and method names to reflect new changes regarding the use of KeyValues by creating a structure with all values defined during file lookup rather than doing this during the pack process.
2022-11-23 17:09:48 +01:00
vector<VPKKeyValues_t> CPackedStore::GetEntryValues(const string& svWorkspace) const
Update packedstore.cpp
2022-06-02 01:59:03 +02:00
{
Move some CPackedStore methods to VPKDir_t These methods where only used for and by the VPKDir_t structure. Also renamed variable and method names to reflect new changes regarding the use of KeyValues by creating a structure with all values defined during file lookup rather than doing this during the pack process.
2022-11-23 17:09:48 +01:00
vector<VPKKeyValues_t> vEntryValues;
vector<string> vIgnoredList = GetIgnoreList(svWorkspace);
Add support for .vpkignore file Folders can be globally excluded using the '.vpkignore' file (placed in the workspace root). To exclude specific files, use the manifest file.
2022-06-06 23:08:53 +02:00
Move some CPackedStore methods to VPKDir_t These methods where only used for and by the VPKDir_t structure. Also renamed variable and method names to reflect new changes regarding the use of KeyValues by creating a structure with all values defined during file lookup rather than doing this during the pack process.
2022-11-23 17:09:48 +01:00
fs::recursive_directory_iterator dir(svWorkspace), end;
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
while (dir != end)
{
Move some CPackedStore methods to VPKDir_t These methods where only used for and by the VPKDir_t structure. Also renamed variable and method names to reflect new changes regarding the use of KeyValues by creating a structure with all values defined during file lookup rather than doing this during the pack process.
2022-11-23 17:09:48 +01:00
const vector<string>::iterator it = std::find(vIgnoredList.begin(), vIgnoredList.end(),
Add file extension support to .vpkignore
2022-06-07 13:44:31 +02:00
GetExtension(dir->path().filename().u8string(), true, true));
Move some CPackedStore methods to VPKDir_t These methods where only used for and by the VPKDir_t structure. Also renamed variable and method names to reflect new changes regarding the use of KeyValues by creating a structure with all values defined during file lookup rather than doing this during the pack process.
2022-11-23 17:09:48 +01:00
if (it != vIgnoredList.end())
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
{
Add file extension support to .vpkignore
2022-06-07 13:44:31 +02:00
dir.disable_recursion_pending(); // Skip all ignored folders and extensions.
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
}
CPackedStore: do not pack empty files Empty files have no place in a VPK.
2022-09-10 00:07:42 +02:00
else if (dir->file_size() > 0) // Empty files are not supported.
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
{
Move some CPackedStore methods to VPKDir_t These methods where only used for and by the VPKDir_t structure. Also renamed variable and method names to reflect new changes regarding the use of KeyValues by creating a structure with all values defined during file lookup rather than doing this during the pack process.
2022-11-23 17:09:48 +01:00
const string svEntryPath = dir->path().u8string();
if (!GetExtension(svEntryPath).empty())
CPackedStore: do not pack empty files Empty files have no place in a VPK.
2022-09-10 00:07:42 +02:00
{
Move some CPackedStore methods to VPKDir_t These methods where only used for and by the VPKDir_t structure. Also renamed variable and method names to reflect new changes regarding the use of KeyValues by creating a structure with all values defined during file lookup rather than doing this during the pack process.
2022-11-23 17:09:48 +01:00
vEntryValues.push_back(VPKKeyValues_t(ConvertToUnixPath(svEntryPath)));
CPackedStore: do not pack empty files Empty files have no place in a VPK.
2022-09-10 00:07:42 +02:00
}
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
}
dir++;
}
Move some CPackedStore methods to VPKDir_t These methods where only used for and by the VPKDir_t structure. Also renamed variable and method names to reflect new changes regarding the use of KeyValues by creating a structure with all values defined during file lookup rather than doing this during the pack process.
2022-11-23 17:09:48 +01:00
return vEntryValues;
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
}
//-----------------------------------------------------------------------------
Move some CPackedStore methods to VPKDir_t These methods where only used for and by the VPKDir_t structure. Also renamed variable and method names to reflect new changes regarding the use of KeyValues by creating a structure with all values defined during file lookup rather than doing this during the pack process.
2022-11-23 17:09:48 +01:00
// Purpose: scans the input directory and returns the values to the vector if path exists in manifest
// Input : &svWorkspace -
CPackedStore refactor The CPackedStore class now heavily relies on the engine's FileSystem and KeyValues API. Since its part of the engine, it would make more sense to utilize the engine's features instead. It also allows for easier modifications. * The logic behind the build process hasn't changed.
2022-11-23 12:18:33 +01:00
// *pManifestKV -
Move some CPackedStore methods to VPKDir_t These methods where only used for and by the VPKDir_t structure. Also renamed variable and method names to reflect new changes regarding the use of KeyValues by creating a structure with all values defined during file lookup rather than doing this during the pack process.
2022-11-23 17:09:48 +01:00
// Output : vpk keyvalues vector of all existing and included entry paths
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
//-----------------------------------------------------------------------------
Move some CPackedStore methods to VPKDir_t These methods where only used for and by the VPKDir_t structure. Also renamed variable and method names to reflect new changes regarding the use of KeyValues by creating a structure with all values defined during file lookup rather than doing this during the pack process.
2022-11-23 17:09:48 +01:00
vector<VPKKeyValues_t> CPackedStore::GetEntryValues(const string& svWorkspace, KeyValues* pManifestKV) const
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
{
Move some CPackedStore methods to VPKDir_t These methods where only used for and by the VPKDir_t structure. Also renamed variable and method names to reflect new changes regarding the use of KeyValues by creating a structure with all values defined during file lookup rather than doing this during the pack process.
2022-11-23 17:09:48 +01:00
vector<VPKKeyValues_t> vEntryValues;
CPackedStore refactor The CPackedStore class now heavily relies on the engine's FileSystem and KeyValues API. Since its part of the engine, it would make more sense to utilize the engine's features instead. It also allows for easier modifications. * The logic behind the build process hasn't changed.
2022-11-23 12:18:33 +01:00
if (!pManifestKV)
{
Light optimizations for string utils * Changed 'CreateDirectories' to only copy sanitized path if pointer to string is passed. * Changed 'IsValidBase64' to only tokenize base64 value if pointer to string is passed.
2023-02-04 15:59:45 +01:00
Warning(eDLL_T::FS, "Invalid VPK build manifest KV; unable to parse entry list\n");
Move some CPackedStore methods to VPKDir_t These methods where only used for and by the VPKDir_t structure. Also renamed variable and method names to reflect new changes regarding the use of KeyValues by creating a structure with all values defined during file lookup rather than doing this during the pack process.
2022-11-23 17:09:48 +01:00
return vEntryValues;
CPackedStore refactor The CPackedStore class now heavily relies on the engine's FileSystem and KeyValues API. Since its part of the engine, it would make more sense to utilize the engine's features instead. It also allows for easier modifications. * The logic behind the build process hasn't changed.
2022-11-23 12:18:33 +01:00
}
Add support for .vpkignore file Folders can be globally excluded using the '.vpkignore' file (placed in the workspace root). To exclude specific files, use the manifest file.
2022-06-06 23:08:53 +02:00
Move some CPackedStore methods to VPKDir_t These methods where only used for and by the VPKDir_t structure. Also renamed variable and method names to reflect new changes regarding the use of KeyValues by creating a structure with all values defined during file lookup rather than doing this during the pack process.
2022-11-23 17:09:48 +01:00
vector<string> vIgnoredList = GetIgnoreList(svWorkspace);
fs::recursive_directory_iterator dir(svWorkspace), end;
CPackedStore refactor The CPackedStore class now heavily relies on the engine's FileSystem and KeyValues API. Since its part of the engine, it would make more sense to utilize the engine's features instead. It also allows for easier modifications. * The logic behind the build process hasn't changed.
2022-11-23 12:18:33 +01:00
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
while (dir != end)
Update packedstore.cpp
2022-06-02 01:59:03 +02:00
{
Move some CPackedStore methods to VPKDir_t These methods where only used for and by the VPKDir_t structure. Also renamed variable and method names to reflect new changes regarding the use of KeyValues by creating a structure with all values defined during file lookup rather than doing this during the pack process.
2022-11-23 17:09:48 +01:00
const vector<string>::iterator it = std::find(vIgnoredList.begin(), vIgnoredList.end(),
Add file extension support to .vpkignore
2022-06-07 13:44:31 +02:00
GetExtension(dir->path().filename().u8string(), true, true));
Move some CPackedStore methods to VPKDir_t These methods where only used for and by the VPKDir_t structure. Also renamed variable and method names to reflect new changes regarding the use of KeyValues by creating a structure with all values defined during file lookup rather than doing this during the pack process.
2022-11-23 17:09:48 +01:00
if (it != vIgnoredList.end())
Update packedstore.cpp
2022-06-02 01:59:03 +02:00
{
Add file extension support to .vpkignore
2022-06-07 13:44:31 +02:00
dir.disable_recursion_pending(); // Skip all ignored folders and extensions.
Log truncated files if they are listed in the VPK build manifest
2023-01-26 03:00:53 +01:00
continue;
Update packedstore.cpp
2022-06-02 01:59:03 +02:00
}
Log truncated files if they are listed in the VPK build manifest
2023-01-26 03:00:53 +01:00
const string svFullPath = ConvertToWinPath(dir->path().u8string());
if (!GetExtension(svFullPath).empty())
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
{
Log truncated files if they are listed in the VPK build manifest
2023-01-26 03:00:53 +01:00
// Remove workspace path by offsetting it by its size.
const char* pszEntry = (svFullPath.c_str() + svWorkspace.length());
KeyValues* pEntryKV = pManifestKV->FindKey(pszEntry);
CPackedStore refactor The CPackedStore class now heavily relies on the engine's FileSystem and KeyValues API. Since its part of the engine, it would make more sense to utilize the engine's features instead. It also allows for easier modifications. * The logic behind the build process hasn't changed.
2022-11-23 12:18:33 +01:00
Log truncated files if they are listed in the VPK build manifest
2023-01-26 03:00:53 +01:00
if (pEntryKV)
{
if (!dir->file_size()) // Empty files are not supported.
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
{
Log truncated files if they are listed in the VPK build manifest
2023-01-26 03:00:53 +01:00
Warning(eDLL_T::FS, "File '%s' listed in build manifest appears truncated\n", dir->path().relative_path().c_str());
continue;
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
}
Log truncated files if they are listed in the VPK build manifest
2023-01-26 03:00:53 +01:00
vEntryValues.push_back(VPKKeyValues_t(
ConvertToUnixPath(svFullPath),
pEntryKV->GetInt("preloadSize", NULL),
pEntryKV->GetInt("loadFlags", static_cast<uint32_t>(EPackedLoadFlags::LOAD_VISIBLE) | static_cast<uint32_t>(EPackedLoadFlags::LOAD_CACHE)),
pEntryKV->GetInt("textureFlags", static_cast<uint16_t>(EPackedTextureFlags::TEXTURE_DEFAULT)),
pEntryKV->GetBool("useCompression", true),
pEntryKV->GetBool("useDataSharing", true))
);
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
}
}
dir++;
Update packedstore.cpp
2022-06-02 01:59:03 +02:00
}
Move some CPackedStore methods to VPKDir_t These methods where only used for and by the VPKDir_t structure. Also renamed variable and method names to reflect new changes regarding the use of KeyValues by creating a structure with all values defined during file lookup rather than doing this during the pack process.
2022-11-23 17:09:48 +01:00
return vEntryValues;
Update packedstore.cpp
2022-06-02 01:59:03 +02:00
}
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
//-----------------------------------------------------------------------------
CPackedStore: fix directory file name sanitizer The sanitizer can retrieve the directory file name from the block file name if this is passed instead. This was broken in the previous system. Sanitization can be opted out if the 3rd parameter of the 'fs_vpk_unpack' command is omitted.
2022-10-11 15:37:34 +02:00
// Purpose: gets the parts of the directory file name
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
// Input : &svDirectoryName -
Improve consistency throughout packedstore class No modifications to the existing logic have been performed. Some parameter names have been renamed to improve code readability and maintainability.
2022-11-16 00:54:51 +01:00
// nCaptureGroup - (1 = locale + target, 2 = level)
CPackedStore: fix directory file name sanitizer The sanitizer can retrieve the directory file name from the block file name if this is passed instead. This was broken in the previous system. Sanitization can be opted out if the 3rd parameter of the 'fs_vpk_unpack' command is omitted.
2022-10-11 15:37:34 +02:00
// Output : part of directory file name as string
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
//-----------------------------------------------------------------------------
VPK system improvements More detailed operation statistics. Discard VPK directory files who's header doesn't match engine requirements. Separated manifest properly between language and context (you can now package for each language and context from a single workspace).
2022-06-05 17:28:39 +02:00
string CPackedStore::GetNameParts(const string& svDirectoryName, int nCaptureGroup) const
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
{
std::smatch smRegexMatches;
CPackedStore: fix directory file name sanitizer The sanitizer can retrieve the directory file name from the block file name if this is passed instead. This was broken in the previous system. Sanitization can be opted out if the 3rd parameter of the 'fs_vpk_unpack' command is omitted.
2022-10-11 15:37:34 +02:00
std::regex_search(svDirectoryName, smRegexMatches, DIR_REGEX);
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
VPK system improvements More detailed operation statistics. Discard VPK directory files who's header doesn't match engine requirements. Separated manifest properly between language and context (you can now package for each language and context from a single workspace).
2022-06-05 17:28:39 +02:00
return smRegexMatches[nCaptureGroup].str();
}
//-----------------------------------------------------------------------------
Improve consistency throughout packedstore class No modifications to the existing logic have been performed. Some parameter names have been renamed to improve code readability and maintainability.
2022-11-16 00:54:51 +01:00
// Purpose: gets the level name from the directory file name
VPK system improvements More detailed operation statistics. Discard VPK directory files who's header doesn't match engine requirements. Separated manifest properly between language and context (you can now package for each language and context from a single workspace).
2022-06-05 17:28:39 +02:00
// Input : &svDirectoryName -
Improve consistency throughout packedstore class No modifications to the existing logic have been performed. Some parameter names have been renamed to improve code readability and maintainability.
2022-11-16 00:54:51 +01:00
// Output : level name as string (e.g. "mp_rr_box")
VPK system improvements More detailed operation statistics. Discard VPK directory files who's header doesn't match engine requirements. Separated manifest properly between language and context (you can now package for each language and context from a single workspace).
2022-06-05 17:28:39 +02:00
//-----------------------------------------------------------------------------
Improve consistency throughout packedstore class No modifications to the existing logic have been performed. Some parameter names have been renamed to improve code readability and maintainability.
2022-11-16 00:54:51 +01:00
string CPackedStore::GetLevelName(const string& svDirectoryName) const
VPK system improvements More detailed operation statistics. Discard VPK directory files who's header doesn't match engine requirements. Separated manifest properly between language and context (you can now package for each language and context from a single workspace).
2022-06-05 17:28:39 +02:00
{
std::smatch smRegexMatches;
CPackedStore: fix directory file name sanitizer The sanitizer can retrieve the directory file name from the block file name if this is passed instead. This was broken in the previous system. Sanitization can be opted out if the 3rd parameter of the 'fs_vpk_unpack' command is omitted.
2022-10-11 15:37:34 +02:00
std::regex_search(svDirectoryName, smRegexMatches, DIR_REGEX);
VPK system improvements More detailed operation statistics. Discard VPK directory files who's header doesn't match engine requirements. Separated manifest properly between language and context (you can now package for each language and context from a single workspace).
2022-06-05 17:28:39 +02:00
return smRegexMatches[1].str() + smRegexMatches[2].str();
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
}
//-----------------------------------------------------------------------------
CPackedStore refactor The CPackedStore class now heavily relies on the engine's FileSystem and KeyValues API. Since its part of the engine, it would make more sense to utilize the engine's features instead. It also allows for easier modifications. * The logic behind the build process hasn't changed.
2022-11-23 12:18:33 +01:00
// Purpose: gets the manifest file associated with the VPK name (must be freed after wards)
Improve consistency throughout packedstore class No modifications to the existing logic have been performed. Some parameter names have been renamed to improve code readability and maintainability.
2022-11-16 00:54:51 +01:00
// Input : &svWorkspace -
Add support for .vpkignore file Folders can be globally excluded using the '.vpkignore' file (placed in the workspace root). To exclude specific files, use the manifest file.
2022-06-06 23:08:53 +02:00
// &svManifestName -
Move some CPackedStore methods to VPKDir_t These methods where only used for and by the VPKDir_t structure. Also renamed variable and method names to reflect new changes regarding the use of KeyValues by creating a structure with all values defined during file lookup rather than doing this during the pack process.
2022-11-23 17:09:48 +01:00
// Output : KeyValues (build manifest pointer)
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
//-----------------------------------------------------------------------------
CPackedStore refactor The CPackedStore class now heavily relies on the engine's FileSystem and KeyValues API. Since its part of the engine, it would make more sense to utilize the engine's features instead. It also allows for easier modifications. * The logic behind the build process hasn't changed.
2022-11-23 12:18:33 +01:00
KeyValues* CPackedStore::GetManifest(const string& svWorkspace, const string& svManifestName) const
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
{
CPackedStore refactor The CPackedStore class now heavily relies on the engine's FileSystem and KeyValues API. Since its part of the engine, it would make more sense to utilize the engine's features instead. It also allows for easier modifications. * The logic behind the build process hasn't changed.
2022-11-23 12:18:33 +01:00
string svPathOut = fmt::format("{:s}{:s}{:s}.{:s}", svWorkspace, "manifest/", svManifestName, "txt");
KeyValues* pManifestKV = FileSystem()->LoadKeyValues(IFileSystem::TYPE_COMMON, svPathOut.c_str(), "GAME");
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
CPackedStore refactor The CPackedStore class now heavily relies on the engine's FileSystem and KeyValues API. Since its part of the engine, it would make more sense to utilize the engine's features instead. It also allows for easier modifications. * The logic behind the build process hasn't changed.
2022-11-23 12:18:33 +01:00
if (!pManifestKV)
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
{
CPackedStore refactor The CPackedStore class now heavily relies on the engine's FileSystem and KeyValues API. Since its part of the engine, it would make more sense to utilize the engine's features instead. It also allows for easier modifications. * The logic behind the build process hasn't changed.
2022-11-23 12:18:33 +01:00
Warning(eDLL_T::FS, "Failed to parse VPK build manifest: '%s'\n", svPathOut.c_str());
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
}
CPackedStore refactor The CPackedStore class now heavily relies on the engine's FileSystem and KeyValues API. Since its part of the engine, it would make more sense to utilize the engine's features instead. It also allows for easier modifications. * The logic behind the build process hasn't changed.
2022-11-23 12:18:33 +01:00
return pManifestKV;
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
}
Add support for .vpkignore file Folders can be globally excluded using the '.vpkignore' file (placed in the workspace root). To exclude specific files, use the manifest file.
2022-06-06 23:08:53 +02:00
//-----------------------------------------------------------------------------
// Purpose: gets the contents from the global ignore list (.vpkignore)
Improve consistency throughout packedstore class No modifications to the existing logic have been performed. Some parameter names have been renamed to improve code readability and maintainability.
2022-11-16 00:54:51 +01:00
// Input : &svWorkspace -
// Output : a string vector of ignored directories/files and extensions
Add support for .vpkignore file Folders can be globally excluded using the '.vpkignore' file (placed in the workspace root). To exclude specific files, use the manifest file.
2022-06-06 23:08:53 +02:00
//-----------------------------------------------------------------------------
Improve consistency throughout packedstore class No modifications to the existing logic have been performed. Some parameter names have been renamed to improve code readability and maintainability.
2022-11-16 00:54:51 +01:00
vector<string> CPackedStore::GetIgnoreList(const string& svWorkspace) const
Add support for .vpkignore file Folders can be globally excluded using the '.vpkignore' file (placed in the workspace root). To exclude specific files, use the manifest file.
2022-06-06 23:08:53 +02:00
{
CPackedStore refactor The CPackedStore class now heavily relies on the engine's FileSystem and KeyValues API. Since its part of the engine, it would make more sense to utilize the engine's features instead. It also allows for easier modifications. * The logic behind the build process hasn't changed.
2022-11-23 12:18:33 +01:00
string svIgnore = svWorkspace + ".vpkignore";
Open ignore file in text mode
2022-11-24 11:48:23 +01:00
FileHandle_t hIgnoreFile = FileSystem()->Open(svIgnore.c_str(), "rt", "GAME");
CPackedStore refactor The CPackedStore class now heavily relies on the engine's FileSystem and KeyValues API. Since its part of the engine, it would make more sense to utilize the engine's features instead. It also allows for easier modifications. * The logic behind the build process hasn't changed.
2022-11-23 12:18:33 +01:00
if (!hIgnoreFile)
{
Warning(eDLL_T::FS, "No ignore file provided; continuing build without...\n");
return vector<string>();
}
Add support for .vpkignore file Folders can be globally excluded using the '.vpkignore' file (placed in the workspace root). To exclude specific files, use the manifest file.
2022-06-06 23:08:53 +02:00
vector<string> vIgnore;
CPackedStore refactor The CPackedStore class now heavily relies on the engine's FileSystem and KeyValues API. Since its part of the engine, it would make more sense to utilize the engine's features instead. It also allows for easier modifications. * The logic behind the build process hasn't changed.
2022-11-23 12:18:33 +01:00
char szIgnore[MAX_PATH];
while (FileSystem()->ReadLine(szIgnore, sizeof(szIgnore) - 1, hIgnoreFile))
Add support for .vpkignore file Folders can be globally excluded using the '.vpkignore' file (placed in the workspace root). To exclude specific files, use the manifest file.
2022-06-06 23:08:53 +02:00
{
CPackedStore refactor The CPackedStore class now heavily relies on the engine's FileSystem and KeyValues API. Since its part of the engine, it would make more sense to utilize the engine's features instead. It also allows for easier modifications. * The logic behind the build process hasn't changed.
2022-11-23 12:18:33 +01:00
if (!strstr(szIgnore, "//"))
Add support for .vpkignore file Folders can be globally excluded using the '.vpkignore' file (placed in the workspace root). To exclude specific files, use the manifest file.
2022-06-06 23:08:53 +02:00
{
CPackedStore refactor The CPackedStore class now heavily relies on the engine's FileSystem and KeyValues API. Since its part of the engine, it would make more sense to utilize the engine's features instead. It also allows for easier modifications. * The logic behind the build process hasn't changed.
2022-11-23 12:18:33 +01:00
if (char* pEOL = strchr(szIgnore, '\n'))
Add support for .vpkignore file Folders can be globally excluded using the '.vpkignore' file (placed in the workspace root). To exclude specific files, use the manifest file.
2022-06-06 23:08:53 +02:00
{
CPackedStore refactor The CPackedStore class now heavily relies on the engine's FileSystem and KeyValues API. Since its part of the engine, it would make more sense to utilize the engine's features instead. It also allows for easier modifications. * The logic behind the build process hasn't changed.
2022-11-23 12:18:33 +01:00
// Null newline character.
*pEOL = '\0';
if (pEOL - szIgnore > 0)
Add support for .vpkignore file Folders can be globally excluded using the '.vpkignore' file (placed in the workspace root). To exclude specific files, use the manifest file.
2022-06-06 23:08:53 +02:00
{
CPackedStore refactor The CPackedStore class now heavily relies on the engine's FileSystem and KeyValues API. Since its part of the engine, it would make more sense to utilize the engine's features instead. It also allows for easier modifications. * The logic behind the build process hasn't changed.
2022-11-23 12:18:33 +01:00
// Null carriage return.
if (*(pEOL - 1) == '\r')
{
*(pEOL - 1) = '\0';
}
Add support for .vpkignore file Folders can be globally excluded using the '.vpkignore' file (placed in the workspace root). To exclude specific files, use the manifest file.
2022-06-06 23:08:53 +02:00
}
}
CPackedStore refactor The CPackedStore class now heavily relies on the engine's FileSystem and KeyValues API. Since its part of the engine, it would make more sense to utilize the engine's features instead. It also allows for easier modifications. * The logic behind the build process hasn't changed.
2022-11-23 12:18:33 +01:00
vIgnore.push_back(szIgnore);
Add support for .vpkignore file Folders can be globally excluded using the '.vpkignore' file (placed in the workspace root). To exclude specific files, use the manifest file.
2022-06-06 23:08:53 +02:00
}
}
CPackedStore refactor The CPackedStore class now heavily relies on the engine's FileSystem and KeyValues API. Since its part of the engine, it would make more sense to utilize the engine's features instead. It also allows for easier modifications. * The logic behind the build process hasn't changed.
2022-11-23 12:18:33 +01:00
FileSystem()->Close(hIgnoreFile);
Add support for .vpkignore file Folders can be globally excluded using the '.vpkignore' file (placed in the workspace root). To exclude specific files, use the manifest file.
2022-06-06 23:08:53 +02:00
return vIgnore;
}
Update packedstore.cpp
2022-06-02 01:59:03 +02:00
//-----------------------------------------------------------------------------
Update terminology entirely for packedstore.h/.cpp VPK tooling also works on the R1 game (Titanfall 1)
2022-06-06 14:54:22 +02:00
// Purpose: formats the file entry path
CPackedStore refactor The CPackedStore class now heavily relies on the engine's FileSystem and KeyValues API. Since its part of the engine, it would make more sense to utilize the engine's features instead. It also allows for easier modifications. * The logic behind the build process hasn't changed.
2022-11-23 12:18:33 +01:00
// Input : &svPath -
Add support for .vpkignore file Folders can be globally excluded using the '.vpkignore' file (placed in the workspace root). To exclude specific files, use the manifest file.
2022-06-06 23:08:53 +02:00
// &svName -
// &svExtension -
CPackedStore: fix directory file name sanitizer The sanitizer can retrieve the directory file name from the block file name if this is passed instead. This was broken in the previous system. Sanitization can be opted out if the 3rd parameter of the 'fs_vpk_unpack' command is omitted.
2022-10-11 15:37:34 +02:00
// Output : formatted entry path
Update packedstore.cpp
2022-06-02 01:59:03 +02:00
//-----------------------------------------------------------------------------
CPackedStore refactor The CPackedStore class now heavily relies on the engine's FileSystem and KeyValues API. Since its part of the engine, it would make more sense to utilize the engine's features instead. It also allows for easier modifications. * The logic behind the build process hasn't changed.
2022-11-23 12:18:33 +01:00
string CPackedStore::FormatEntryPath(const string& svPath, const string& svName, const string& svExtension) const
Update packedstore.cpp
2022-06-02 01:59:03 +02:00
{
CPackedStore refactor The CPackedStore class now heavily relies on the engine's FileSystem and KeyValues API. Since its part of the engine, it would make more sense to utilize the engine's features instead. It also allows for easier modifications. * The logic behind the build process hasn't changed.
2022-11-23 12:18:33 +01:00
return fmt::format("{:s}{:s}{:s}.{:s}", svPath, svPath.empty() ? "" : "/", svName, svExtension);
Update packedstore.cpp
2022-06-02 01:59:03 +02:00
}
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
//-----------------------------------------------------------------------------
// Purpose: builds the VPK manifest file
// Input : &vBlock -
Move some CPackedStore methods to VPKDir_t These methods where only used for and by the VPKDir_t structure. Also renamed variable and method names to reflect new changes regarding the use of KeyValues by creating a structure with all values defined during file lookup rather than doing this during the pack process.
2022-11-23 17:09:48 +01:00
// &svWorkspace -
Add support for .vpkignore file Folders can be globally excluded using the '.vpkignore' file (placed in the workspace root). To exclude specific files, use the manifest file.
2022-06-06 23:08:53 +02:00
// &svManifestName -
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
//-----------------------------------------------------------------------------
CPackedStore refactor The CPackedStore class now heavily relies on the engine's FileSystem and KeyValues API. Since its part of the engine, it would make more sense to utilize the engine's features instead. It also allows for easier modifications. * The logic behind the build process hasn't changed.
2022-11-23 12:18:33 +01:00
void CPackedStore::BuildManifest(const vector<VPKEntryBlock_t>& vBlock, const string& svWorkspace, const string& svManifestName) const
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
{
CPackedStore refactor The CPackedStore class now heavily relies on the engine's FileSystem and KeyValues API. Since its part of the engine, it would make more sense to utilize the engine's features instead. It also allows for easier modifications. * The logic behind the build process hasn't changed.
2022-11-23 12:18:33 +01:00
KeyValues kv("BuildManifest");
KeyValues* pManifestKV = kv.FindKey("BuildManifest", true);
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
Additional VPK lib cleanup and performance improvements
2022-10-09 12:08:54 +02:00
for (const VPKEntryBlock_t& vEntry : vBlock)
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
{
Improve consistency throughout packedstore class No modifications to the existing logic have been performed. Some parameter names have been renamed to improve code readability and maintainability.
2022-11-16 00:54:51 +01:00
const VPKChunkDescriptor_t& vDescriptor = vEntry.m_vFragments[0];
CPackedStore refactor The CPackedStore class now heavily relies on the engine's FileSystem and KeyValues API. Since its part of the engine, it would make more sense to utilize the engine's features instead. It also allows for easier modifications. * The logic behind the build process hasn't changed.
2022-11-23 12:18:33 +01:00
KeyValues* pEntryKV = pManifestKV->FindKey(ConvertToWinPath(vEntry.m_svEntryPath).c_str(), true);
pEntryKV->SetInt("preloadSize", vEntry.m_iPreloadSize);
pEntryKV->SetInt("loadFlags", vDescriptor.m_nLoadFlags);
pEntryKV->SetInt("textureFlags", vDescriptor.m_nTextureFlags);
pEntryKV->SetBool("useCompression", vDescriptor.m_nCompressedSize != vDescriptor.m_nUncompressedSize);
pEntryKV->SetBool("useDataSharing", true);
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
}
CPackedStore refactor The CPackedStore class now heavily relies on the engine's FileSystem and KeyValues API. Since its part of the engine, it would make more sense to utilize the engine's features instead. It also allows for easier modifications. * The logic behind the build process hasn't changed.
2022-11-23 12:18:33 +01:00
string svPathOut = fmt::format("{:s}{:s}{:s}.{:s}", svWorkspace, "manifest/", svManifestName, "txt");
CUtlBuffer uBuf(0i64, 0, CUtlBuffer::TEXT_BUFFER);
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
CPackedStore refactor The CPackedStore class now heavily relies on the engine's FileSystem and KeyValues API. Since its part of the engine, it would make more sense to utilize the engine's features instead. It also allows for easier modifications. * The logic behind the build process hasn't changed.
2022-11-23 12:18:33 +01:00
kv.RecursiveSaveToFile(uBuf, 0);
Light optimizations for string utils * Changed 'CreateDirectories' to only copy sanitized path if pointer to string is passed. * Changed 'IsValidBase64' to only tokenize base64 value if pointer to string is passed.
2023-02-04 15:59:45 +01:00
FileSystem()->CreateDirHierarchy(fmt::format("{:s}{:s}", svWorkspace, "manifest/").c_str(), "GAME");
CPackedStore refactor The CPackedStore class now heavily relies on the engine's FileSystem and KeyValues API. Since its part of the engine, it would make more sense to utilize the engine's features instead. It also allows for easier modifications. * The logic behind the build process hasn't changed.
2022-11-23 12:18:33 +01:00
FileSystem()->WriteFile(svPathOut.c_str(), "GAME", uBuf);
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
}
Update packedstore.cpp
2022-06-02 01:59:03 +02:00
//-----------------------------------------------------------------------------
// Purpose: validates extraction result with precomputed CRC32 hash
Move some CPackedStore methods to VPKDir_t These methods where only used for and by the VPKDir_t structure. Also renamed variable and method names to reflect new changes regarding the use of KeyValues by creating a structure with all values defined during file lookup rather than doing this during the pack process.
2022-11-23 17:09:48 +01:00
// Input : &svAssetPath -
Remove experimental CRC implementation in LZHAM This implementation was done to calculate buffer CRC on the go, as that was what the VPK file system uses for the file crc's. But it later turned out to be unnecessary, and was never reverted.
2022-12-04 01:33:10 +01:00
// : nFileCRC -
Update packedstore.cpp
2022-06-02 01:59:03 +02:00
//-----------------------------------------------------------------------------
Remove experimental CRC implementation in LZHAM This implementation was done to calculate buffer CRC on the go, as that was what the VPK file system uses for the file crc's. But it later turned out to be unnecessary, and was never reverted.
2022-12-04 01:33:10 +01:00
void CPackedStore::ValidateCRC32PostDecomp(const string& svAssetPath, const uint32_t nFileCRC)
Update packedstore.cpp
2022-06-02 01:59:03 +02:00
{
CPackedStore refactor The CPackedStore class now heavily relies on the engine's FileSystem and KeyValues API. Since its part of the engine, it would make more sense to utilize the engine's features instead. It also allows for easier modifications. * The logic behind the build process hasn't changed.
2022-11-23 12:18:33 +01:00
FileHandle_t hAsset = FileSystem()->Open(svAssetPath.c_str(), "rb", "GAME");
if (!hAsset)
{
Error(eDLL_T::FS, NO_ERROR, "%s - Unable to open '%s' (insufficient rights?)\n", __FUNCTION__, svAssetPath.c_str());
return;
}
uint32_t nLen = FileSystem()->Size(hAsset);
uint8_t* pBuf = MemAllocSingleton()->Alloc<uint8_t>(nLen);
FileSystem()->Read(pBuf, nLen, hAsset);
FileSystem()->Close(hAsset);
Remove experimental CRC implementation in LZHAM This implementation was done to calculate buffer CRC on the go, as that was what the VPK file system uses for the file crc's. But it later turned out to be unnecessary, and was never reverted.
2022-12-04 01:33:10 +01:00
uint32_t nCrc32 = crc32::update(NULL, pBuf, nLen);
CPackedStore refactor The CPackedStore class now heavily relies on the engine's FileSystem and KeyValues API. Since its part of the engine, it would make more sense to utilize the engine's features instead. It also allows for easier modifications. * The logic behind the build process hasn't changed.
2022-11-23 12:18:33 +01:00
MemAllocSingleton()->Free(pBuf);
Update packedstore.cpp
2022-06-02 01:59:03 +02:00
Remove experimental CRC implementation in LZHAM This implementation was done to calculate buffer CRC on the go, as that was what the VPK file system uses for the file crc's. But it later turned out to be unnecessary, and was never reverted.
2022-12-04 01:33:10 +01:00
if (nCrc32 != nFileCRC)
Update packedstore.cpp
2022-06-02 01:59:03 +02:00
{
Remove experimental CRC implementation in LZHAM This implementation was done to calculate buffer CRC on the go, as that was what the VPK file system uses for the file crc's. But it later turned out to be unnecessary, and was never reverted.
2022-12-04 01:33:10 +01:00
Warning(eDLL_T::FS, "Computed checksum '0x%lX' doesn't match expected checksum '0x%lX'. File may be corrupt!\n", nCrc32, nFileCRC);
nCrc32 = NULL;
Update packedstore.cpp
2022-06-02 01:59:03 +02:00
}
}
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
//-----------------------------------------------------------------------------
Improve consistency throughout packedstore class No modifications to the existing logic have been performed. Some parameter names have been renamed to improve code readability and maintainability.
2022-11-16 00:54:51 +01:00
// Purpose: packs all files from workspace path into VPK file
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
// Input : &vPair -
Improve consistency throughout packedstore class No modifications to the existing logic have been performed. Some parameter names have been renamed to improve code readability and maintainability.
2022-11-16 00:54:51 +01:00
// &svWorkspace -
// &svBuildPath -
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
// bManifestOnly -
//-----------------------------------------------------------------------------
Improve consistency throughout packedstore class No modifications to the existing logic have been performed. Some parameter names have been renamed to improve code readability and maintainability.
2022-11-16 00:54:51 +01:00
void CPackedStore::PackWorkspace(const VPKPair_t& vPair, const string& svWorkspace, const string& svBuildPath, bool bManifestOnly)
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
{
Move some CPackedStore methods to VPKDir_t These methods where only used for and by the VPKDir_t structure. Also renamed variable and method names to reflect new changes regarding the use of KeyValues by creating a structure with all values defined during file lookup rather than doing this during the pack process.
2022-11-23 17:09:48 +01:00
const string svPackFilePath = string(svBuildPath + vPair.m_svPackName);
CPackedStore refactor The CPackedStore class now heavily relies on the engine's FileSystem and KeyValues API. Since its part of the engine, it would make more sense to utilize the engine's features instead. It also allows for easier modifications. * The logic behind the build process hasn't changed.
2022-11-23 12:18:33 +01:00
FileHandle_t hPackFile = FileSystem()->Open(svPackFilePath.c_str(), "wb", "GAME");
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
CPackedStore refactor The CPackedStore class now heavily relies on the engine's FileSystem and KeyValues API. Since its part of the engine, it would make more sense to utilize the engine's features instead. It also allows for easier modifications. * The logic behind the build process hasn't changed.
2022-11-23 12:18:33 +01:00
if (!hPackFile)
{
Error(eDLL_T::FS, NO_ERROR, "%s - Unable to write to '%s' (read-only?)\n", __FUNCTION__, svPackFilePath.c_str());
return;
}
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
Move some CPackedStore methods to VPKDir_t These methods where only used for and by the VPKDir_t structure. Also renamed variable and method names to reflect new changes regarding the use of KeyValues by creating a structure with all values defined during file lookup rather than doing this during the pack process.
2022-11-23 17:09:48 +01:00
vector<VPKKeyValues_t> vEntryValues;
CPackedStore refactor The CPackedStore class now heavily relies on the engine's FileSystem and KeyValues API. Since its part of the engine, it would make more sense to utilize the engine's features instead. It also allows for easier modifications. * The logic behind the build process hasn't changed.
2022-11-23 12:18:33 +01:00
vector<VPKEntryBlock_t> vEntryBlocks;
KeyValues* pManifestKV = nullptr;
Add support for .vpkignore file Folders can be globally excluded using the '.vpkignore' file (placed in the workspace root). To exclude specific files, use the manifest file.
2022-06-06 23:08:53 +02:00
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
if (bManifestOnly)
{
CPackedStore refactor The CPackedStore class now heavily relies on the engine's FileSystem and KeyValues API. Since its part of the engine, it would make more sense to utilize the engine's features instead. It also allows for easier modifications. * The logic behind the build process hasn't changed.
2022-11-23 12:18:33 +01:00
pManifestKV = GetManifest(svWorkspace, GetLevelName(vPair.m_svDirectoryName));
Move some CPackedStore methods to VPKDir_t These methods where only used for and by the VPKDir_t structure. Also renamed variable and method names to reflect new changes regarding the use of KeyValues by creating a structure with all values defined during file lookup rather than doing this during the pack process.
2022-11-23 17:09:48 +01:00
vEntryValues = GetEntryValues(svWorkspace, pManifestKV);
CPackedStore refactor The CPackedStore class now heavily relies on the engine's FileSystem and KeyValues API. Since its part of the engine, it would make more sense to utilize the engine's features instead. It also allows for easier modifications. * The logic behind the build process hasn't changed.
2022-11-23 12:18:33 +01:00
if (pManifestKV)
{
pManifestKV->DeleteThis();
}
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
}
Additional cleanup Cast to proper types. Set the archive offset only instead of the entire block. The entry could still have its own flags. The only thing needed for saving data is sharing the same offset and size.
2022-06-05 12:28:49 +02:00
else // Pack all files in workspace.
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
{
Move some CPackedStore methods to VPKDir_t These methods where only used for and by the VPKDir_t structure. Also renamed variable and method names to reflect new changes regarding the use of KeyValues by creating a structure with all values defined during file lookup rather than doing this during the pack process.
2022-11-23 17:09:48 +01:00
vEntryValues = GetEntryValues(svWorkspace);
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
}
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
Add convar for adjusting LZHAM compressor max helper threads
2022-11-17 20:37:12 +01:00
uint64_t nSharedTotal = NULL;
uint32_t nSharedCount = NULL;
VPK system improvements More detailed operation statistics. Discard VPK directory files who's header doesn't match engine requirements. Separated manifest properly between language and context (you can now package for each language and context from a single workspace).
2022-06-05 17:28:39 +02:00
Move some CPackedStore methods to VPKDir_t These methods where only used for and by the VPKDir_t structure. Also renamed variable and method names to reflect new changes regarding the use of KeyValues by creating a structure with all values defined during file lookup rather than doing this during the pack process.
2022-11-23 17:09:48 +01:00
for (size_t i = 0, ps = vEntryValues.size(); i < ps; i++)
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
{
Move some CPackedStore methods to VPKDir_t These methods where only used for and by the VPKDir_t structure. Also renamed variable and method names to reflect new changes regarding the use of KeyValues by creating a structure with all values defined during file lookup rather than doing this during the pack process.
2022-11-23 17:09:48 +01:00
const VPKKeyValues_t& vEntryValue = vEntryValues[i];
FileHandle_t hAsset = FileSystem()->Open(vEntryValue.m_svEntryPath.c_str(), "rb", "GAME");
CPackedStore refactor The CPackedStore class now heavily relies on the engine's FileSystem and KeyValues API. Since its part of the engine, it would make more sense to utilize the engine's features instead. It also allows for easier modifications. * The logic behind the build process hasn't changed.
2022-11-23 12:18:33 +01:00
if (!hAsset)
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
{
Move some CPackedStore methods to VPKDir_t These methods where only used for and by the VPKDir_t structure. Also renamed variable and method names to reflect new changes regarding the use of KeyValues by creating a structure with all values defined during file lookup rather than doing this during the pack process.
2022-11-23 17:09:48 +01:00
Error(eDLL_T::FS, NO_ERROR, "%s - Unable to open '%s' (insufficient rights?)\n", __FUNCTION__, vEntryValue.m_svEntryPath.c_str());
CPackedStore refactor The CPackedStore class now heavily relies on the engine's FileSystem and KeyValues API. Since its part of the engine, it would make more sense to utilize the engine's features instead. It also allows for easier modifications. * The logic behind the build process hasn't changed.
2022-11-23 12:18:33 +01:00
continue;
}
More VPK compressing/decompressing optimizations * Use reference to array index. * Only obtain array size once during loop. * Use static buffer of 1024*1024 for decompressing. * Removed extraneous goto. * Fixed bug where shared data would be mapped wrong due to the last optimization. * This increased packing performance by 40/60 seconds when rebuilding all vpk's.
2022-10-02 19:24:00 +02:00
Move some CPackedStore methods to VPKDir_t These methods where only used for and by the VPKDir_t structure. Also renamed variable and method names to reflect new changes regarding the use of KeyValues by creating a structure with all values defined during file lookup rather than doing this during the pack process.
2022-11-23 17:09:48 +01:00
const char* szDestPath = (vEntryValue.m_svEntryPath.c_str() + svWorkspace.length());
CPackedStore refactor The CPackedStore class now heavily relies on the engine's FileSystem and KeyValues API. Since its part of the engine, it would make more sense to utilize the engine's features instead. It also allows for easier modifications. * The logic behind the build process hasn't changed.
2022-11-23 12:18:33 +01:00
uint32_t nLen = FileSystem()->Size(hAsset);
uint8_t* pBuf = MemAllocSingleton()->Alloc<uint8_t>(nLen);
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
CPackedStore refactor The CPackedStore class now heavily relies on the engine's FileSystem and KeyValues API. Since its part of the engine, it would make more sense to utilize the engine's features instead. It also allows for easier modifications. * The logic behind the build process hasn't changed.
2022-11-23 12:18:33 +01:00
FileSystem()->Read(pBuf, nLen, hAsset);
FileSystem()->Seek(hAsset, 0, FileSystemSeek_t::FILESYSTEM_SEEK_HEAD);
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
CPackedStore refactor The CPackedStore class now heavily relies on the engine's FileSystem and KeyValues API. Since its part of the engine, it would make more sense to utilize the engine's features instead. It also allows for easier modifications. * The logic behind the build process hasn't changed.
2022-11-23 12:18:33 +01:00
DevMsg(eDLL_T::FS, "Packing entry '%zu' ('%s')\n", i, szDestPath);
Move some CPackedStore methods to VPKDir_t These methods where only used for and by the VPKDir_t structure. Also renamed variable and method names to reflect new changes regarding the use of KeyValues by creating a structure with all values defined during file lookup rather than doing this during the pack process.
2022-11-23 17:09:48 +01:00
vEntryBlocks.push_back(VPKEntryBlock_t(pBuf, nLen, FileSystem()->Tell(hPackFile), vEntryValue.m_iPreloadSize, 0, vEntryValue.m_nLoadFlags, vEntryValue.m_nTextureFlags, szDestPath));
CPackedStore::PackAll optimizations * Hash chunk before compression to determine if it already exists or not (don't run compression if it does). * Use the result of vector::insert for data sharing (insert already checks if it exists or not, and returns the iterator). * This optimization saves around 20 seconds total when rebuilding all VPK's.
2022-10-02 02:16:51 +02:00
Move some CPackedStore methods to VPKDir_t These methods where only used for and by the VPKDir_t structure. Also renamed variable and method names to reflect new changes regarding the use of KeyValues by creating a structure with all values defined during file lookup rather than doing this during the pack process.
2022-11-23 17:09:48 +01:00
VPKEntryBlock_t& vEntryBlock = vEntryBlocks[i];
for (size_t j = 0, es = vEntryBlock.m_vFragments.size(); j < es; j++)
CPackedStore refactor The CPackedStore class now heavily relies on the engine's FileSystem and KeyValues API. Since its part of the engine, it would make more sense to utilize the engine's features instead. It also allows for easier modifications. * The logic behind the build process hasn't changed.
2022-11-23 12:18:33 +01:00
{
Move some CPackedStore methods to VPKDir_t These methods where only used for and by the VPKDir_t structure. Also renamed variable and method names to reflect new changes regarding the use of KeyValues by creating a structure with all values defined during file lookup rather than doing this during the pack process.
2022-11-23 17:09:48 +01:00
VPKChunkDescriptor_t& vDescriptor = vEntryBlock.m_vFragments[j];
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
CPackedStore refactor The CPackedStore class now heavily relies on the engine's FileSystem and KeyValues API. Since its part of the engine, it would make more sense to utilize the engine's features instead. It also allows for easier modifications. * The logic behind the build process hasn't changed.
2022-11-23 12:18:33 +01:00
FileSystem()->Read(s_EntryBuf, vDescriptor.m_nCompressedSize, hAsset);
vDescriptor.m_nPackFileOffset = FileSystem()->Tell(hPackFile);
More VPK compressing/decompressing optimizations * Use reference to array index. * Only obtain array size once during loop. * Use static buffer of 1024*1024 for decompressing. * Removed extraneous goto. * Fixed bug where shared data would be mapped wrong due to the last optimization. * This increased packing performance by 40/60 seconds when rebuilding all vpk's.
2022-10-02 19:24:00 +02:00
Move some CPackedStore methods to VPKDir_t These methods where only used for and by the VPKDir_t structure. Also renamed variable and method names to reflect new changes regarding the use of KeyValues by creating a structure with all values defined during file lookup rather than doing this during the pack process.
2022-11-23 17:09:48 +01:00
if (vEntryValue.m_bUseDataSharing)
CPackedStore refactor The CPackedStore class now heavily relies on the engine's FileSystem and KeyValues API. Since its part of the engine, it would make more sense to utilize the engine's features instead. It also allows for easier modifications. * The logic behind the build process hasn't changed.
2022-11-23 12:18:33 +01:00
{
string svEntryHash = sha1(string(reinterpret_cast<char*>(s_EntryBuf), vDescriptor.m_nUncompressedSize));
auto p = m_mChunkHashMap.insert({ svEntryHash, vDescriptor });
Implement data sharing algorithm in VPK packing This reduces size of the resulting VPK (checks entry block in hash map). mp_common compresses to 32mb instead of 37.3mb (original is 38.5mb).
2022-06-04 13:55:24 +02:00
CPackedStore refactor The CPackedStore class now heavily relies on the engine's FileSystem and KeyValues API. Since its part of the engine, it would make more sense to utilize the engine's features instead. It also allows for easier modifications. * The logic behind the build process hasn't changed.
2022-11-23 12:18:33 +01:00
if (!p.second) // Map to existing chunk to avoid having copies of the same data.
Implement data sharing algorithm in VPK packing This reduces size of the resulting VPK (checks entry block in hash map). mp_common compresses to 32mb instead of 37.3mb (original is 38.5mb).
2022-06-04 13:55:24 +02:00
{
CPackedStore refactor The CPackedStore class now heavily relies on the engine's FileSystem and KeyValues API. Since its part of the engine, it would make more sense to utilize the engine's features instead. It also allows for easier modifications. * The logic behind the build process hasn't changed.
2022-11-23 12:18:33 +01:00
DevMsg(eDLL_T::FS, "Mapping chunk '%zu' ('%s') to existing chunk at '0x%llx'\n", j, svEntryHash.c_str(), p.first->second.m_nPackFileOffset);
Use static buffer for file chunking and compressing during VPK packing Avoid using new every time. Chunk can be at most 1024 * 1024.
2022-10-02 12:17:03 +02:00
CPackedStore refactor The CPackedStore class now heavily relies on the engine's FileSystem and KeyValues API. Since its part of the engine, it would make more sense to utilize the engine's features instead. It also allows for easier modifications. * The logic behind the build process hasn't changed.
2022-11-23 12:18:33 +01:00
vDescriptor = p.first->second;
nSharedTotal += vDescriptor.m_nCompressedSize;
nSharedCount++;
Use static buffer for file chunking and compressing during VPK packing Avoid using new every time. Chunk can be at most 1024 * 1024.
2022-10-02 12:17:03 +02:00
CPackedStore refactor The CPackedStore class now heavily relies on the engine's FileSystem and KeyValues API. Since its part of the engine, it would make more sense to utilize the engine's features instead. It also allows for easier modifications. * The logic behind the build process hasn't changed.
2022-11-23 12:18:33 +01:00
continue;
More VPK compressing/decompressing optimizations * Use reference to array index. * Only obtain array size once during loop. * Use static buffer of 1024*1024 for decompressing. * Removed extraneous goto. * Fixed bug where shared data would be mapped wrong due to the last optimization. * This increased packing performance by 40/60 seconds when rebuilding all vpk's.
2022-10-02 19:24:00 +02:00
}
CPackedStore refactor The CPackedStore class now heavily relies on the engine's FileSystem and KeyValues API. Since its part of the engine, it would make more sense to utilize the engine's features instead. It also allows for easier modifications. * The logic behind the build process hasn't changed.
2022-11-23 12:18:33 +01:00
}
Move some CPackedStore methods to VPKDir_t These methods where only used for and by the VPKDir_t structure. Also renamed variable and method names to reflect new changes regarding the use of KeyValues by creating a structure with all values defined during file lookup rather than doing this during the pack process.
2022-11-23 17:09:48 +01:00
if (vEntryValue.m_bUseCompression)
CPackedStore refactor The CPackedStore class now heavily relies on the engine's FileSystem and KeyValues API. Since its part of the engine, it would make more sense to utilize the engine's features instead. It also allows for easier modifications. * The logic behind the build process hasn't changed.
2022-11-23 12:18:33 +01:00
{
Light CPackedStore cleanup
2022-12-04 14:17:14 +01:00
lzham_compress_status_t lzCompStatus = lzham_compress_memory(&m_lzCompParams, s_EntryBuf, &vDescriptor.m_nCompressedSize, s_EntryBuf,
Remove experimental CRC implementation in LZHAM This implementation was done to calculate buffer CRC on the go, as that was what the VPK file system uses for the file crc's. But it later turned out to be unnecessary, and was never reverted.
2022-12-04 01:33:10 +01:00
vDescriptor.m_nUncompressedSize, nullptr);
CPackedStore refactor The CPackedStore class now heavily relies on the engine's FileSystem and KeyValues API. Since its part of the engine, it would make more sense to utilize the engine's features instead. It also allows for easier modifications. * The logic behind the build process hasn't changed.
2022-11-23 12:18:33 +01:00
Light CPackedStore cleanup
2022-12-04 14:17:14 +01:00
if (lzCompStatus != lzham_compress_status_t::LZHAM_COMP_STATUS_SUCCESS)
More VPK compressing/decompressing optimizations * Use reference to array index. * Only obtain array size once during loop. * Use static buffer of 1024*1024 for decompressing. * Removed extraneous goto. * Fixed bug where shared data would be mapped wrong due to the last optimization. * This increased packing performance by 40/60 seconds when rebuilding all vpk's.
2022-10-02 19:24:00 +02:00
{
CPackedStore refactor The CPackedStore class now heavily relies on the engine's FileSystem and KeyValues API. Since its part of the engine, it would make more sense to utilize the engine's features instead. It also allows for easier modifications. * The logic behind the build process hasn't changed.
2022-11-23 12:18:33 +01:00
Warning(eDLL_T::FS, "Status '%d' for chunk '%zu' within entry '%zu' in block '%hu' (chunk packed without compression)\n",
Light CPackedStore cleanup
2022-12-04 14:17:14 +01:00
lzCompStatus, j, i, vEntryBlocks[i].m_iPackFileIndex);
CPackedStore refactor The CPackedStore class now heavily relies on the engine's FileSystem and KeyValues API. Since its part of the engine, it would make more sense to utilize the engine's features instead. It also allows for easier modifications. * The logic behind the build process hasn't changed.
2022-11-23 12:18:33 +01:00
More VPK compressing/decompressing optimizations * Use reference to array index. * Only obtain array size once during loop. * Use static buffer of 1024*1024 for decompressing. * Removed extraneous goto. * Fixed bug where shared data would be mapped wrong due to the last optimization. * This increased packing performance by 40/60 seconds when rebuilding all vpk's.
2022-10-02 19:24:00 +02:00
vDescriptor.m_nCompressedSize = vDescriptor.m_nUncompressedSize;
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
}
}
CPackedStore refactor The CPackedStore class now heavily relies on the engine's FileSystem and KeyValues API. Since its part of the engine, it would make more sense to utilize the engine's features instead. It also allows for easier modifications. * The logic behind the build process hasn't changed.
2022-11-23 12:18:33 +01:00
else // Write data uncompressed.
{
vDescriptor.m_nCompressedSize = vDescriptor.m_nUncompressedSize;
}
vDescriptor.m_bIsCompressed = vDescriptor.m_nCompressedSize != vDescriptor.m_nUncompressedSize;
FileSystem()->Write(s_EntryBuf, vDescriptor.m_nCompressedSize, hPackFile);
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
}
CPackedStore refactor The CPackedStore class now heavily relies on the engine's FileSystem and KeyValues API. Since its part of the engine, it would make more sense to utilize the engine's features instead. It also allows for easier modifications. * The logic behind the build process hasn't changed.
2022-11-23 12:18:33 +01:00
MemAllocSingleton()->Free(pBuf);
FileSystem()->Close(hAsset);
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
}
CPackedStore refactor The CPackedStore class now heavily relies on the engine's FileSystem and KeyValues API. Since its part of the engine, it would make more sense to utilize the engine's features instead. It also allows for easier modifications. * The logic behind the build process hasn't changed.
2022-11-23 12:18:33 +01:00
DevMsg(eDLL_T::FS, "*** Build block totaling '%zu' bytes with '%zu' shared bytes among '%lu' chunks\n", FileSystem()->Tell(hPackFile), nSharedTotal, nSharedCount);
FileSystem()->Close(hPackFile);
Use static buffer for file chunking and compressing during VPK packing Avoid using new every time. Chunk can be at most 1024 * 1024.
2022-10-02 12:17:03 +02:00
Update terminology entirely for packedstore.h/.cpp VPK tooling also works on the R1 game (Titanfall 1)
2022-06-06 14:54:22 +02:00
m_mChunkHashMap.clear();
More VPK compressing/decompressing optimizations * Use reference to array index. * Only obtain array size once during loop. * Use static buffer of 1024*1024 for decompressing. * Removed extraneous goto. * Fixed bug where shared data would be mapped wrong due to the last optimization. * This increased packing performance by 40/60 seconds when rebuilding all vpk's.
2022-10-02 19:24:00 +02:00
memset(s_EntryBuf, '\0', sizeof(s_EntryBuf));
VPK system improvements More detailed operation statistics. Discard VPK directory files who's header doesn't match engine requirements. Separated manifest properly between language and context (you can now package for each language and context from a single workspace).
2022-06-05 17:28:39 +02:00
Move some CPackedStore methods to VPKDir_t These methods where only used for and by the VPKDir_t structure. Also renamed variable and method names to reflect new changes regarding the use of KeyValues by creating a structure with all values defined during file lookup rather than doing this during the pack process.
2022-11-23 17:09:48 +01:00
VPKDir_t vDirectory;
vDirectory.BuildDirectoryFile(svBuildPath + vPair.m_svDirectoryName, vEntryBlocks);
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
}
//-----------------------------------------------------------------------------
Improve consistency throughout packedstore class No modifications to the existing logic have been performed. Some parameter names have been renamed to improve code readability and maintainability.
2022-11-16 00:54:51 +01:00
// Purpose: rebuilds manifest and extracts all files from specified VPK file
Move some CPackedStore methods to VPKDir_t These methods where only used for and by the VPKDir_t structure. Also renamed variable and method names to reflect new changes regarding the use of KeyValues by creating a structure with all values defined during file lookup rather than doing this during the pack process.
2022-11-23 17:09:48 +01:00
// Input : &vDirectory -
Improve consistency throughout packedstore class No modifications to the existing logic have been performed. Some parameter names have been renamed to improve code readability and maintainability.
2022-11-16 00:54:51 +01:00
// &svWorkspace -
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
//-----------------------------------------------------------------------------
Move some CPackedStore methods to VPKDir_t These methods where only used for and by the VPKDir_t structure. Also renamed variable and method names to reflect new changes regarding the use of KeyValues by creating a structure with all values defined during file lookup rather than doing this during the pack process.
2022-11-23 17:09:48 +01:00
void CPackedStore::UnpackWorkspace(const VPKDir_t& vDirectory, const string& svWorkspace)
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
{
Move some CPackedStore methods to VPKDir_t These methods where only used for and by the VPKDir_t structure. Also renamed variable and method names to reflect new changes regarding the use of KeyValues by creating a structure with all values defined during file lookup rather than doing this during the pack process.
2022-11-23 17:09:48 +01:00
if (vDirectory.m_vHeader.m_nHeaderMarker != VPK_HEADER_MARKER ||
vDirectory.m_vHeader.m_nMajorVersion != VPK_MAJOR_VERSION ||
vDirectory.m_vHeader.m_nMinorVersion != VPK_MINOR_VERSION)
VPK system improvements More detailed operation statistics. Discard VPK directory files who's header doesn't match engine requirements. Separated manifest properly between language and context (you can now package for each language and context from a single workspace).
2022-06-05 17:28:39 +02:00
{
Light cleanup Use NO_ERROR instead of NULL for error code parameter passed to Error(..).
2022-09-14 01:14:51 +02:00
Error(eDLL_T::FS, NO_ERROR, "Unsupported VPK directory file (invalid header criteria)\n");
VPK system improvements More detailed operation statistics. Discard VPK directory files who's header doesn't match engine requirements. Separated manifest properly between language and context (you can now package for each language and context from a single workspace).
2022-06-05 17:28:39 +02:00
return;
}
CPackedStore refactor The CPackedStore class now heavily relies on the engine's FileSystem and KeyValues API. Since its part of the engine, it would make more sense to utilize the engine's features instead. It also allows for easier modifications. * The logic behind the build process hasn't changed.
2022-11-23 12:18:33 +01:00
Move some CPackedStore methods to VPKDir_t These methods where only used for and by the VPKDir_t structure. Also renamed variable and method names to reflect new changes regarding the use of KeyValues by creating a structure with all values defined during file lookup rather than doing this during the pack process.
2022-11-23 17:09:48 +01:00
BuildManifest(vDirectory.m_vEntryBlocks, svWorkspace, GetLevelName(vDirectory.m_svDirectoryPath));
const string svPath = RemoveFileName(vDirectory.m_svDirectoryPath) + '/';
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
Move some CPackedStore methods to VPKDir_t These methods where only used for and by the VPKDir_t structure. Also renamed variable and method names to reflect new changes regarding the use of KeyValues by creating a structure with all values defined during file lookup rather than doing this during the pack process.
2022-11-23 17:09:48 +01:00
for (size_t i = 0, fs = vDirectory.m_vPackFile.size(); i < fs; i++)
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
{
Move some CPackedStore methods to VPKDir_t These methods where only used for and by the VPKDir_t structure. Also renamed variable and method names to reflect new changes regarding the use of KeyValues by creating a structure with all values defined during file lookup rather than doing this during the pack process.
2022-11-23 17:09:48 +01:00
const string svPackFile = svPath + vDirectory.m_vPackFile[i];
CPackedStore refactor The CPackedStore class now heavily relies on the engine's FileSystem and KeyValues API. Since its part of the engine, it would make more sense to utilize the engine's features instead. It also allows for easier modifications. * The logic behind the build process hasn't changed.
2022-11-23 12:18:33 +01:00
// Read from each pack file.
FileHandle_t hPackFile = FileSystem()->Open(svPackFile.c_str(), "rb", "GAME");
if (!hPackFile)
{
Error(eDLL_T::FS, NO_ERROR, "%s - Unable to open '%s' (insufficient rights?)\n", __FUNCTION__, svPackFile.c_str());
continue;
}
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
Move some CPackedStore methods to VPKDir_t These methods where only used for and by the VPKDir_t structure. Also renamed variable and method names to reflect new changes regarding the use of KeyValues by creating a structure with all values defined during file lookup rather than doing this during the pack process.
2022-11-23 17:09:48 +01:00
for (size_t j = 0, es = vDirectory.m_vEntryBlocks.size(); j < es; j++)
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
{
Move some CPackedStore methods to VPKDir_t These methods where only used for and by the VPKDir_t structure. Also renamed variable and method names to reflect new changes regarding the use of KeyValues by creating a structure with all values defined during file lookup rather than doing this during the pack process.
2022-11-23 17:09:48 +01:00
const VPKEntryBlock_t& vEntryBlock = vDirectory.m_vEntryBlocks[j];
if (vEntryBlock.m_iPackFileIndex != static_cast<uint16_t>(i))
Additional VPK system improvements Added DevMsg logs to show the elapsed time of the operation. Fixed post decompress CRC/ADLER error checking. Slight cleanup.
2022-06-04 03:12:46 +02:00
{
More VPK compressing/decompressing optimizations * Use reference to array index. * Only obtain array size once during loop. * Use static buffer of 1024*1024 for decompressing. * Removed extraneous goto. * Fixed bug where shared data would be mapped wrong due to the last optimization. * This increased packing performance by 40/60 seconds when rebuilding all vpk's.
2022-10-02 19:24:00 +02:00
continue;
Additional VPK system improvements Added DevMsg logs to show the elapsed time of the operation. Fixed post decompress CRC/ADLER error checking. Slight cleanup.
2022-06-04 03:12:46 +02:00
}
Cast and format to proper types
2022-06-04 03:19:05 +02:00
else // Chunk belongs to this block.
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
{
Light optimizations for string utils * Changed 'CreateDirectories' to only copy sanitized path if pointer to string is passed. * Changed 'IsValidBase64' to only tokenize base64 value if pointer to string is passed.
2023-02-04 15:59:45 +01:00
string svFilePath;
CreateDirectories(svWorkspace + vEntryBlock.m_svEntryPath, &svFilePath);
CPackedStore refactor The CPackedStore class now heavily relies on the engine's FileSystem and KeyValues API. Since its part of the engine, it would make more sense to utilize the engine's features instead. It also allows for easier modifications. * The logic behind the build process hasn't changed.
2022-11-23 12:18:33 +01:00
FileHandle_t hAsset = FileSystem()->Open(svFilePath.c_str(), "wb", "GAME");
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
CPackedStore refactor The CPackedStore class now heavily relies on the engine's FileSystem and KeyValues API. Since its part of the engine, it would make more sense to utilize the engine's features instead. It also allows for easier modifications. * The logic behind the build process hasn't changed.
2022-11-23 12:18:33 +01:00
if (!hAsset)
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
{
CPackedStore refactor The CPackedStore class now heavily relies on the engine's FileSystem and KeyValues API. Since its part of the engine, it would make more sense to utilize the engine's features instead. It also allows for easier modifications. * The logic behind the build process hasn't changed.
2022-11-23 12:18:33 +01:00
Error(eDLL_T::FS, NO_ERROR, "%s - Unable to write to '%s' (read-only?)\n", __FUNCTION__, svFilePath.c_str());
Additional VPK system improvements Added DevMsg logs to show the elapsed time of the operation. Fixed post decompress CRC/ADLER error checking. Slight cleanup.
2022-06-04 03:12:46 +02:00
continue;
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
}
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
Move some CPackedStore methods to VPKDir_t These methods where only used for and by the VPKDir_t structure. Also renamed variable and method names to reflect new changes regarding the use of KeyValues by creating a structure with all values defined during file lookup rather than doing this during the pack process.
2022-11-23 17:09:48 +01:00
DevMsg(eDLL_T::FS, "Unpacking entry '%zu' from block '%zu' ('%s')\n", j, i, vEntryBlock.m_svEntryPath.c_str());
for (size_t k = 0, cs = vEntryBlock.m_vFragments.size(); k < cs; k++)
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
{
Move some CPackedStore methods to VPKDir_t These methods where only used for and by the VPKDir_t structure. Also renamed variable and method names to reflect new changes regarding the use of KeyValues by creating a structure with all values defined during file lookup rather than doing this during the pack process.
2022-11-23 17:09:48 +01:00
const VPKChunkDescriptor_t& vChunk = vEntryBlock.m_vFragments[k];
Update terminology entirely for packedstore.h/.cpp VPK tooling also works on the R1 game (Titanfall 1)
2022-06-06 14:54:22 +02:00
m_nChunkCount++;
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
CPackedStore refactor The CPackedStore class now heavily relies on the engine's FileSystem and KeyValues API. Since its part of the engine, it would make more sense to utilize the engine's features instead. It also allows for easier modifications. * The logic behind the build process hasn't changed.
2022-11-23 12:18:33 +01:00
FileSystem()->Seek(hPackFile, vChunk.m_nPackFileOffset, FileSystemSeek_t::FILESYSTEM_SEEK_HEAD);
FileSystem()->Read(s_EntryBuf, vChunk.m_nCompressedSize, hPackFile);
Additional VPK system improvements Added DevMsg logs to show the elapsed time of the operation. Fixed post decompress CRC/ADLER error checking. Slight cleanup.
2022-06-04 03:12:46 +02:00
Update terminology entirely for packedstore.h/.cpp VPK tooling also works on the R1 game (Titanfall 1)
2022-06-06 14:54:22 +02:00
if (vChunk.m_bIsCompressed)
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
{
More VPK compressing/decompressing optimizations * Use reference to array index. * Only obtain array size once during loop. * Use static buffer of 1024*1024 for decompressing. * Removed extraneous goto. * Fixed bug where shared data would be mapped wrong due to the last optimization. * This increased packing performance by 40/60 seconds when rebuilding all vpk's.
2022-10-02 19:24:00 +02:00
size_t nDstLen = sizeof(s_DecompBuf);
assert(vChunk.m_nCompressedSize <= nDstLen);
if (vChunk.m_nCompressedSize > nDstLen)
break; // Corrupt or invalid chunk descriptor.
Light CPackedStore cleanup
2022-12-04 14:17:14 +01:00
lzham_decompress_status_t lzDecompStatus = lzham_decompress_memory(&m_lzDecompParams, s_DecompBuf,
Remove experimental CRC implementation in LZHAM This implementation was done to calculate buffer CRC on the go, as that was what the VPK file system uses for the file crc's. But it later turned out to be unnecessary, and was never reverted.
2022-12-04 01:33:10 +01:00
&nDstLen, s_EntryBuf, vChunk.m_nCompressedSize, nullptr);
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
Light CPackedStore cleanup
2022-12-04 14:17:14 +01:00
if (lzDecompStatus != lzham_decompress_status_t::LZHAM_DECOMP_STATUS_SUCCESS)
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
{
Light cleanup Use NO_ERROR instead of NULL for error code parameter passed to Error(..).
2022-09-14 01:14:51 +02:00
Error(eDLL_T::FS, NO_ERROR, "Status '%d' for chunk '%zu' within entry '%zu' in block '%hu' (chunk not decompressed)\n",
Light CPackedStore cleanup
2022-12-04 14:17:14 +01:00
lzDecompStatus, m_nChunkCount, i, vEntryBlock.m_iPackFileIndex);
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
}
Additional VPK system improvements Added DevMsg logs to show the elapsed time of the operation. Fixed post decompress CRC/ADLER error checking. Slight cleanup.
2022-06-04 03:12:46 +02:00
else // If successfully decompressed, write to file.
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
{
CPackedStore refactor The CPackedStore class now heavily relies on the engine's FileSystem and KeyValues API. Since its part of the engine, it would make more sense to utilize the engine's features instead. It also allows for easier modifications. * The logic behind the build process hasn't changed.
2022-11-23 12:18:33 +01:00
FileSystem()->Write(s_DecompBuf, nDstLen, hAsset);
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
}
}
More VPK compressing/decompressing optimizations * Use reference to array index. * Only obtain array size once during loop. * Use static buffer of 1024*1024 for decompressing. * Removed extraneous goto. * Fixed bug where shared data would be mapped wrong due to the last optimization. * This increased packing performance by 40/60 seconds when rebuilding all vpk's.
2022-10-02 19:24:00 +02:00
else // If not compressed, write source data into output file.
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
{
CPackedStore refactor The CPackedStore class now heavily relies on the engine's FileSystem and KeyValues API. Since its part of the engine, it would make more sense to utilize the engine's features instead. It also allows for easier modifications. * The logic behind the build process hasn't changed.
2022-11-23 12:18:33 +01:00
FileSystem()->Write(s_EntryBuf, vChunk.m_nUncompressedSize, hAsset);
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
}
}
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
CPackedStore refactor The CPackedStore class now heavily relies on the engine's FileSystem and KeyValues API. Since its part of the engine, it would make more sense to utilize the engine's features instead. It also allows for easier modifications. * The logic behind the build process hasn't changed.
2022-11-23 12:18:33 +01:00
FileSystem()->Close(hAsset);
Move some CPackedStore methods to VPKDir_t These methods where only used for and by the VPKDir_t structure. Also renamed variable and method names to reflect new changes regarding the use of KeyValues by creating a structure with all values defined during file lookup rather than doing this during the pack process.
2022-11-23 17:09:48 +01:00
if (m_nChunkCount == vEntryBlock.m_vFragments.size()) // Only validate after last entry in block had been written.
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
{
Add convar for adjusting LZHAM compressor max helper threads
2022-11-17 20:37:12 +01:00
m_nChunkCount = NULL;
Remove experimental CRC implementation in LZHAM This implementation was done to calculate buffer CRC on the go, as that was what the VPK file system uses for the file crc's. But it later turned out to be unnecessary, and was never reverted.
2022-12-04 01:33:10 +01:00
ValidateCRC32PostDecomp(svFilePath, vEntryBlock.m_nFileCRC);
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
}
More VPK compressing/decompressing optimizations * Use reference to array index. * Only obtain array size once during loop. * Use static buffer of 1024*1024 for decompressing. * Removed extraneous goto. * Fixed bug where shared data would be mapped wrong due to the last optimization. * This increased packing performance by 40/60 seconds when rebuilding all vpk's.
2022-10-02 19:24:00 +02:00
}
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
}
CPackedStore refactor The CPackedStore class now heavily relies on the engine's FileSystem and KeyValues API. Since its part of the engine, it would make more sense to utilize the engine's features instead. It also allows for easier modifications. * The logic behind the build process hasn't changed.
2022-11-23 12:18:33 +01:00
FileSystem()->Close(hPackFile);
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
}
More VPK compressing/decompressing optimizations * Use reference to array index. * Only obtain array size once during loop. * Use static buffer of 1024*1024 for decompressing. * Removed extraneous goto. * Fixed bug where shared data would be mapped wrong due to the last optimization. * This increased packing performance by 40/60 seconds when rebuilding all vpk's.
2022-10-02 19:24:00 +02:00
memset(s_EntryBuf, '\0', sizeof(s_EntryBuf));
memset(s_DecompBuf, '\0', sizeof(s_DecompBuf));
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
}
CPackedStore refactor The CPackedStore class now heavily relies on the engine's FileSystem and KeyValues API. Since its part of the engine, it would make more sense to utilize the engine's features instead. It also allows for easier modifications. * The logic behind the build process hasn't changed.
2022-11-23 12:18:33 +01:00
//-----------------------------------------------------------------------------
// Purpose: 'VPKKeyValues_t' memory constructor
// Input : &svEntryPath -
// iPreloadSize -
// nLoadFlags -
// nTextureFlags -
// bUseCompression -
// bUseDataSharing -
//-----------------------------------------------------------------------------
VPKKeyValues_t::VPKKeyValues_t(const string& svEntryPath, uint16_t iPreloadSize, uint32_t nLoadFlags, uint16_t nTextureFlags, bool bUseCompression, bool bUseDataSharing)
{
m_svEntryPath = svEntryPath;
m_iPreloadSize = iPreloadSize;
m_nLoadFlags = nLoadFlags;
m_nTextureFlags = nTextureFlags;
m_bUseCompression = bUseCompression;
m_bUseDataSharing = bUseDataSharing;
}
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
//-----------------------------------------------------------------------------
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
// Purpose: 'VPKEntryBlock_t' file constructor
Move some CPackedStore methods to VPKDir_t These methods where only used for and by the VPKDir_t structure. Also renamed variable and method names to reflect new changes regarding the use of KeyValues by creating a structure with all values defined during file lookup rather than doing this during the pack process.
2022-11-23 17:09:48 +01:00
// Input : hDirectoryFile -
Avoid string copy in 'VPKEntryBlock_t::VPKEntryBlock_t' Avoid a copy when formatting string to that of the VPK directory tree structure.
2022-11-23 14:49:40 +01:00
// &svEntryPath -
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
//-----------------------------------------------------------------------------
Move some CPackedStore methods to VPKDir_t These methods where only used for and by the VPKDir_t structure. Also renamed variable and method names to reflect new changes regarding the use of KeyValues by creating a structure with all values defined during file lookup rather than doing this during the pack process.
2022-11-23 17:09:48 +01:00
VPKEntryBlock_t::VPKEntryBlock_t(FileHandle_t hDirectoryFile, const string& svEntryPath)
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
{
Improve consistency throughout packedstore class No modifications to the existing logic have been performed. Some parameter names have been renamed to improve code readability and maintainability.
2022-11-16 00:54:51 +01:00
m_svEntryPath = svEntryPath; // Set the entry path.
Avoid string copy in 'VPKEntryBlock_t::VPKEntryBlock_t' Avoid a copy when formatting string to that of the VPK directory tree structure.
2022-11-23 14:49:40 +01:00
StringReplace(m_svEntryPath, "\\", "/"); // Flip windows-style backslash to forward slash.
Light optimizations for string utils * Changed 'CreateDirectories' to only copy sanitized path if pointer to string is passed. * Changed 'IsValidBase64' to only tokenize base64 value if pointer to string is passed.
2023-02-04 15:59:45 +01:00
StringReplace(m_svEntryPath, " /", ""); // Remove space character representing VPK root.
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
Move some CPackedStore methods to VPKDir_t These methods where only used for and by the VPKDir_t structure. Also renamed variable and method names to reflect new changes regarding the use of KeyValues by creating a structure with all values defined during file lookup rather than doing this during the pack process.
2022-11-23 17:09:48 +01:00
FileSystem()->Read(&m_nFileCRC, sizeof(uint32_t), hDirectoryFile); //
FileSystem()->Read(&m_iPreloadSize, sizeof(uint16_t), hDirectoryFile); //
FileSystem()->Read(&m_iPackFileIndex, sizeof(uint16_t), hDirectoryFile); //
CPackedStore refactor The CPackedStore class now heavily relies on the engine's FileSystem and KeyValues API. Since its part of the engine, it would make more sense to utilize the engine's features instead. It also allows for easier modifications. * The logic behind the build process hasn't changed.
2022-11-23 12:18:33 +01:00
uint16_t nMarker = 0;
Improve consistency throughout packedstore class No modifications to the existing logic have been performed. Some parameter names have been renamed to improve code readability and maintainability.
2022-11-16 00:54:51 +01:00
do // Loop through all chunks in the entry and add to list.
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
{
Move some CPackedStore methods to VPKDir_t These methods where only used for and by the VPKDir_t structure. Also renamed variable and method names to reflect new changes regarding the use of KeyValues by creating a structure with all values defined during file lookup rather than doing this during the pack process.
2022-11-23 17:09:48 +01:00
VPKChunkDescriptor_t entry(hDirectoryFile);
Improve consistency throughout packedstore class No modifications to the existing logic have been performed. Some parameter names have been renamed to improve code readability and maintainability.
2022-11-16 00:54:51 +01:00
m_vFragments.push_back(entry);
CPackedStore refactor The CPackedStore class now heavily relies on the engine's FileSystem and KeyValues API. Since its part of the engine, it would make more sense to utilize the engine's features instead. It also allows for easier modifications. * The logic behind the build process hasn't changed.
2022-11-23 12:18:33 +01:00
Move some CPackedStore methods to VPKDir_t These methods where only used for and by the VPKDir_t structure. Also renamed variable and method names to reflect new changes regarding the use of KeyValues by creating a structure with all values defined during file lookup rather than doing this during the pack process.
2022-11-23 17:09:48 +01:00
FileSystem()->Read(&nMarker, sizeof(nMarker), hDirectoryFile);
CPackedStore refactor The CPackedStore class now heavily relies on the engine's FileSystem and KeyValues API. Since its part of the engine, it would make more sense to utilize the engine's features instead. It also allows for easier modifications. * The logic behind the build process hasn't changed.
2022-11-23 12:18:33 +01:00
} while (nMarker != static_cast<uint16_t>(PACKFILEINDEX_END));
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
}
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
//-----------------------------------------------------------------------------
// Purpose: 'VPKEntryBlock_t' memory constructor
CPackedStore refactor The CPackedStore class now heavily relies on the engine's FileSystem and KeyValues API. Since its part of the engine, it would make more sense to utilize the engine's features instead. It also allows for easier modifications. * The logic behind the build process hasn't changed.
2022-11-23 12:18:33 +01:00
// Input : *pData -
Move some CPackedStore methods to VPKDir_t These methods where only used for and by the VPKDir_t structure. Also renamed variable and method names to reflect new changes regarding the use of KeyValues by creating a structure with all values defined during file lookup rather than doing this during the pack process.
2022-11-23 17:09:48 +01:00
// nLen
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
// nOffset -
Improve consistency throughout packedstore class No modifications to the existing logic have been performed. Some parameter names have been renamed to improve code readability and maintainability.
2022-11-16 00:54:51 +01:00
// iPreloadSize -
// iPackFileIndex -
Add support for .vpkignore file Folders can be globally excluded using the '.vpkignore' file (placed in the workspace root). To exclude specific files, use the manifest file.
2022-06-06 23:08:53 +02:00
// nLoadFlags -
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
// nTextureFlags -
Improve consistency throughout packedstore class No modifications to the existing logic have been performed. Some parameter names have been renamed to improve code readability and maintainability.
2022-11-16 00:54:51 +01:00
// &svEntryPath -
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
//-----------------------------------------------------------------------------
CPackedStore refactor The CPackedStore class now heavily relies on the engine's FileSystem and KeyValues API. Since its part of the engine, it would make more sense to utilize the engine's features instead. It also allows for easier modifications. * The logic behind the build process hasn't changed.
2022-11-23 12:18:33 +01:00
VPKEntryBlock_t::VPKEntryBlock_t(const uint8_t* pData, size_t nLen, int64_t nOffset, uint16_t iPreloadSize,
Improve consistency throughout packedstore class No modifications to the existing logic have been performed. Some parameter names have been renamed to improve code readability and maintainability.
2022-11-16 00:54:51 +01:00
uint16_t iPackFileIndex, uint32_t nLoadFlags, uint16_t nTextureFlags, const string& svEntryPath)
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
{
CPackedStore refactor The CPackedStore class now heavily relies on the engine's FileSystem and KeyValues API. Since its part of the engine, it would make more sense to utilize the engine's features instead. It also allows for easier modifications. * The logic behind the build process hasn't changed.
2022-11-23 12:18:33 +01:00
m_nFileCRC = crc32::update(NULL, pData, nLen);
Improve consistency throughout packedstore class No modifications to the existing logic have been performed. Some parameter names have been renamed to improve code readability and maintainability.
2022-11-16 00:54:51 +01:00
m_iPreloadSize = iPreloadSize;
m_iPackFileIndex = iPackFileIndex;
Update terminology entirely for packedstore.h/.cpp VPK tooling also works on the R1 game (Titanfall 1)
2022-06-06 14:54:22 +02:00
m_svEntryPath = svEntryPath;
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
CPackedStore refactor The CPackedStore class now heavily relies on the engine's FileSystem and KeyValues API. Since its part of the engine, it would make more sense to utilize the engine's features instead. It also allows for easier modifications. * The logic behind the build process hasn't changed.
2022-11-23 12:18:33 +01:00
size_t nFragmentCount = (nLen + ENTRY_MAX_LEN - 1) / ENTRY_MAX_LEN;
size_t nFileSize = nLen;
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
int64_t nCurrentOffset = nOffset;
VPK system improvements More detailed operation statistics. Discard VPK directory files who's header doesn't match engine requirements. Separated manifest properly between language and context (you can now package for each language and context from a single workspace).
2022-06-05 17:28:39 +02:00
Improve consistency throughout packedstore class No modifications to the existing logic have been performed. Some parameter names have been renamed to improve code readability and maintainability.
2022-11-16 00:54:51 +01:00
for (size_t i = 0; i < nFragmentCount; i++) // Fragment data into 1 MiB chunks.
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
{
Improve consistency throughout packedstore class No modifications to the existing logic have been performed. Some parameter names have been renamed to improve code readability and maintainability.
2022-11-16 00:54:51 +01:00
size_t nSize = std::min<uint64_t>(ENTRY_MAX_LEN, nFileSize);
nFileSize -= nSize;
m_vFragments.push_back(VPKChunkDescriptor_t(nLoadFlags, nTextureFlags, nCurrentOffset, nSize, nSize));
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
nCurrentOffset += nSize;
}
}
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
//-----------------------------------------------------------------------------
Update terminology entirely for packedstore.h/.cpp VPK tooling also works on the R1 game (Titanfall 1)
2022-06-06 14:54:22 +02:00
// Purpose: 'VPKChunkDescriptor_t' file constructor
Move some CPackedStore methods to VPKDir_t These methods where only used for and by the VPKDir_t structure. Also renamed variable and method names to reflect new changes regarding the use of KeyValues by creating a structure with all values defined during file lookup rather than doing this during the pack process.
2022-11-23 17:09:48 +01:00
// Input : hDirectoryFile -
VPK system improvements More detailed operation statistics. Discard VPK directory files who's header doesn't match engine requirements. Separated manifest properly between language and context (you can now package for each language and context from a single workspace).
2022-06-05 17:28:39 +02:00
//-----------------------------------------------------------------------------
Move some CPackedStore methods to VPKDir_t These methods where only used for and by the VPKDir_t structure. Also renamed variable and method names to reflect new changes regarding the use of KeyValues by creating a structure with all values defined during file lookup rather than doing this during the pack process.
2022-11-23 17:09:48 +01:00
VPKChunkDescriptor_t::VPKChunkDescriptor_t(FileHandle_t hDirectoryFile)
VPK system improvements More detailed operation statistics. Discard VPK directory files who's header doesn't match engine requirements. Separated manifest properly between language and context (you can now package for each language and context from a single workspace).
2022-06-05 17:28:39 +02:00
{
Move some CPackedStore methods to VPKDir_t These methods where only used for and by the VPKDir_t structure. Also renamed variable and method names to reflect new changes regarding the use of KeyValues by creating a structure with all values defined during file lookup rather than doing this during the pack process.
2022-11-23 17:09:48 +01:00
FileSystem()->Read(&m_nLoadFlags, sizeof(uint32_t), hDirectoryFile); //
FileSystem()->Read(&m_nTextureFlags, sizeof(uint16_t), hDirectoryFile); //
FileSystem()->Read(&m_nPackFileOffset, sizeof(uint64_t), hDirectoryFile); //
FileSystem()->Read(&m_nCompressedSize, sizeof(uint64_t), hDirectoryFile); //
FileSystem()->Read(&m_nUncompressedSize, sizeof(uint64_t), hDirectoryFile); //
Improve consistency throughout packedstore class No modifications to the existing logic have been performed. Some parameter names have been renamed to improve code readability and maintainability.
2022-11-16 00:54:51 +01:00
m_bIsCompressed = (m_nCompressedSize != m_nUncompressedSize);
VPK system improvements More detailed operation statistics. Discard VPK directory files who's header doesn't match engine requirements. Separated manifest properly between language and context (you can now package for each language and context from a single workspace).
2022-06-05 17:28:39 +02:00
}
//-----------------------------------------------------------------------------
Update terminology entirely for packedstore.h/.cpp VPK tooling also works on the R1 game (Titanfall 1)
2022-06-06 14:54:22 +02:00
// Purpose: 'VPKChunkDescriptor_t' memory constructor
Add support for .vpkignore file Folders can be globally excluded using the '.vpkignore' file (placed in the workspace root). To exclude specific files, use the manifest file.
2022-06-06 23:08:53 +02:00
// Input : nLoadFlags -
// nTextureFlags -
// nArchiveOffset -
// nCompressedSize -
// nUncompressedSize -
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
//-----------------------------------------------------------------------------
Additional VPK lib cleanup and performance improvements
2022-10-09 12:08:54 +02:00
VPKChunkDescriptor_t::VPKChunkDescriptor_t(uint32_t nLoadFlags, uint16_t nTextureFlags,
Move some CPackedStore methods to VPKDir_t These methods where only used for and by the VPKDir_t structure. Also renamed variable and method names to reflect new changes regarding the use of KeyValues by creating a structure with all values defined during file lookup rather than doing this during the pack process.
2022-11-23 17:09:48 +01:00
uint64_t nPackFileOffset, uint64_t nCompressedSize, uint64_t nUncompressedSize)
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
{
Update terminology entirely for packedstore.h/.cpp VPK tooling also works on the R1 game (Titanfall 1)
2022-06-06 14:54:22 +02:00
m_nLoadFlags = nLoadFlags;
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
m_nTextureFlags = nTextureFlags;
Move some CPackedStore methods to VPKDir_t These methods where only used for and by the VPKDir_t structure. Also renamed variable and method names to reflect new changes regarding the use of KeyValues by creating a structure with all values defined during file lookup rather than doing this during the pack process.
2022-11-23 17:09:48 +01:00
m_nPackFileOffset = nPackFileOffset;
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
m_nCompressedSize = nCompressedSize;
m_nUncompressedSize = nUncompressedSize;
Light CPackedStore cleanup
2022-12-04 14:17:14 +01:00
m_bIsCompressed = (m_nCompressedSize != m_nUncompressedSize);
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
}
Move some CPackedStore methods to VPKDir_t These methods where only used for and by the VPKDir_t structure. Also renamed variable and method names to reflect new changes regarding the use of KeyValues by creating a structure with all values defined during file lookup rather than doing this during the pack process.
2022-11-23 17:09:48 +01:00
//-----------------------------------------------------------------------------
// Purpose: builds a valid file name for the VPK
// Input : svLanguage -
// svTarget -
// &svLevel -
// nPatch -
// Output : a vpk file pair (block and directory file names)
//-----------------------------------------------------------------------------
VPKPair_t::VPKPair_t(string svLanguage, string svTarget, const string& svLevel, int nPatch)
{
if (std::find(DIR_LOCALE.begin(), DIR_LOCALE.end(), svLanguage) == DIR_LOCALE.end())
{
svLanguage = DIR_LOCALE[0];
}
if (std::find(DIR_TARGET.begin(), DIR_TARGET.end(), svTarget) == DIR_TARGET.end())
{
svTarget = DIR_TARGET[0];
}
m_svPackName = fmt::format("{:s}_{:s}.bsp.pak000_{:03d}{:s}", svTarget, svLevel, nPatch, ".vpk");
m_svDirectoryName = fmt::format("{:s}{:s}_{:s}.bsp.pak000_{:s}", svLanguage, svTarget, svLevel, "dir.vpk");
}
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
//-----------------------------------------------------------------------------
// Purpose: 'VPKDir_t' file constructor
Improve consistency throughout packedstore class No modifications to the existing logic have been performed. Some parameter names have been renamed to improve code readability and maintainability.
2022-11-16 00:54:51 +01:00
// Input : &svPath -
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
//-----------------------------------------------------------------------------
Move some CPackedStore methods to VPKDir_t These methods where only used for and by the VPKDir_t structure. Also renamed variable and method names to reflect new changes regarding the use of KeyValues by creating a structure with all values defined during file lookup rather than doing this during the pack process.
2022-11-23 17:09:48 +01:00
VPKDir_t::VPKDir_t(const string& svDirectoryPath)
{
Init(svDirectoryPath);
}
//-----------------------------------------------------------------------------
// Purpose: 'VPKDir_t' file constructor with sanitation
// Input : svDirectoryName -
// bSanitizeName - retrieve the directory file name from block name
// Output : VPKDir_t
//-----------------------------------------------------------------------------
VPKDir_t::VPKDir_t(const string& svDirectoryPath, bool bSanitizeName)
{
if (!bSanitizeName)
{
Init(svDirectoryPath);
return;
}
std::smatch smRegexMatches;
std::regex_search(svDirectoryPath, smRegexMatches, BLOCK_REGEX);
if (smRegexMatches.empty())
{
Init(svDirectoryPath);
return;
}
string svSanitizedName = svDirectoryPath;
StringReplace(svSanitizedName, smRegexMatches[0], "pak000_dir");
bool bHasLocale = false;
for (const string& svLocale : DIR_LOCALE)
{
if (svSanitizedName.find(svLocale) != string::npos)
{
bHasLocale = true;
break;
}
}
if (!bHasLocale) // Only sanitize if no locale was provided.
{
string svPackDirPrefix;
svPackDirPrefix.append(DIR_LOCALE[0]);
for (const string& svTarget : DIR_TARGET)
{
if (svSanitizedName.find(svTarget) != string::npos)
{
svPackDirPrefix.append(svTarget);
StringReplace(svSanitizedName, svTarget, svPackDirPrefix);
break;
}
}
}
Init(svSanitizedName);
}
//-----------------------------------------------------------------------------
// Purpose: 'VPKDir_t' file constructor
// Input : &svDirectoryPath -
//-----------------------------------------------------------------------------
void VPKDir_t::Init(const string& svDirectoryPath)
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
{
CPackedStore refactor The CPackedStore class now heavily relies on the engine's FileSystem and KeyValues API. Since its part of the engine, it would make more sense to utilize the engine's features instead. It also allows for easier modifications. * The logic behind the build process hasn't changed.
2022-11-23 12:18:33 +01:00
// Create stream to read from each pack file.
Move some CPackedStore methods to VPKDir_t These methods where only used for and by the VPKDir_t structure. Also renamed variable and method names to reflect new changes regarding the use of KeyValues by creating a structure with all values defined during file lookup rather than doing this during the pack process.
2022-11-23 17:09:48 +01:00
FileHandle_t hDirectory = FileSystem()->Open(svDirectoryPath.c_str(), "rb", "GAME");
CPackedStore refactor The CPackedStore class now heavily relies on the engine's FileSystem and KeyValues API. Since its part of the engine, it would make more sense to utilize the engine's features instead. It also allows for easier modifications. * The logic behind the build process hasn't changed.
2022-11-23 12:18:33 +01:00
if (!hDirectory)
{
Move some CPackedStore methods to VPKDir_t These methods where only used for and by the VPKDir_t structure. Also renamed variable and method names to reflect new changes regarding the use of KeyValues by creating a structure with all values defined during file lookup rather than doing this during the pack process.
2022-11-23 17:09:48 +01:00
Error(eDLL_T::FS, NO_ERROR, "%s - Unable to open '%s' (insufficient rights?)\n", __FUNCTION__, svDirectoryPath.c_str());
CPackedStore refactor The CPackedStore class now heavily relies on the engine's FileSystem and KeyValues API. Since its part of the engine, it would make more sense to utilize the engine's features instead. It also allows for easier modifications. * The logic behind the build process hasn't changed.
2022-11-23 12:18:33 +01:00
return;
}
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
CPackedStore refactor The CPackedStore class now heavily relies on the engine's FileSystem and KeyValues API. Since its part of the engine, it would make more sense to utilize the engine's features instead. It also allows for easier modifications. * The logic behind the build process hasn't changed.
2022-11-23 12:18:33 +01:00
FileSystem()->Read(&m_vHeader.m_nHeaderMarker, sizeof(uint32_t), hDirectory);
FileSystem()->Read(&m_vHeader.m_nMajorVersion, sizeof(uint16_t), hDirectory); //
FileSystem()->Read(&m_vHeader.m_nMinorVersion, sizeof(uint16_t), hDirectory); //
FileSystem()->Read(&m_vHeader.m_nDirectorySize, sizeof(uint32_t), hDirectory); //
FileSystem()->Read(&m_vHeader.m_nSignatureSize, sizeof(uint32_t), hDirectory); //
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
CPackedStore refactor The CPackedStore class now heavily relies on the engine's FileSystem and KeyValues API. Since its part of the engine, it would make more sense to utilize the engine's features instead. It also allows for easier modifications. * The logic behind the build process hasn't changed.
2022-11-23 12:18:33 +01:00
m_vEntryBlocks = g_pPackedStore->GetEntryBlocks(hDirectory);
Move some CPackedStore methods to VPKDir_t These methods where only used for and by the VPKDir_t structure. Also renamed variable and method names to reflect new changes regarding the use of KeyValues by creating a structure with all values defined during file lookup rather than doing this during the pack process.
2022-11-23 17:09:48 +01:00
m_svDirectoryPath = svDirectoryPath; // Set path to vpk directory file.
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
CPackedStore refactor The CPackedStore class now heavily relies on the engine's FileSystem and KeyValues API. Since its part of the engine, it would make more sense to utilize the engine's features instead. It also allows for easier modifications. * The logic behind the build process hasn't changed.
2022-11-23 12:18:33 +01:00
m_nPackFileCount = 0;
Improve consistency throughout packedstore class No modifications to the existing logic have been performed. Some parameter names have been renamed to improve code readability and maintainability.
2022-11-16 00:54:51 +01:00
for (VPKEntryBlock_t vEntry : m_vEntryBlocks)
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
{
Improve consistency throughout packedstore class No modifications to the existing logic have been performed. Some parameter names have been renamed to improve code readability and maintainability.
2022-11-16 00:54:51 +01:00
if (vEntry.m_iPackFileIndex > m_nPackFileCount)
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
{
Improve consistency throughout packedstore class No modifications to the existing logic have been performed. Some parameter names have been renamed to improve code readability and maintainability.
2022-11-16 00:54:51 +01:00
m_nPackFileCount = vEntry.m_iPackFileIndex;
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
}
}
Improve consistency throughout packedstore class No modifications to the existing logic have been performed. Some parameter names have been renamed to improve code readability and maintainability.
2022-11-16 00:54:51 +01:00
for (uint16_t i = 0; i < m_nPackFileCount + 1; i++)
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
{
Move some CPackedStore methods to VPKDir_t These methods where only used for and by the VPKDir_t structure. Also renamed variable and method names to reflect new changes regarding the use of KeyValues by creating a structure with all values defined during file lookup rather than doing this during the pack process.
2022-11-23 17:09:48 +01:00
string svPackPath = GetPackFile(svDirectoryPath, i);
m_vPackFile.push_back(svPackPath);
}
}
//-----------------------------------------------------------------------------
// Purpose: formats pack file path for specific directory file
// Input : &svDirectoryPath -
// iPackFileIndex -
// output : string
//-----------------------------------------------------------------------------
string VPKDir_t::GetPackFile(const string& svDirectoryPath, uint16_t iPackFileIndex) const
{
string svPackChunkName = StripLocalePrefix(svDirectoryPath);
string svPackChunkIndex = fmt::format("{:s}{:03d}", "pak000_", iPackFileIndex);
StringReplace(svPackChunkName, "pak000_dir", svPackChunkIndex);
return svPackChunkName;
}
//-----------------------------------------------------------------------------
// Purpose: strips locale prefix from file path
// Input : &svDirectoryPath -
// Output : directory filename without locale prefix
//-----------------------------------------------------------------------------
string VPKDir_t::StripLocalePrefix(const string& svDirectoryPath) const
{
string svFileName = GetFileName(svDirectoryPath);
for (const string& svLocale : DIR_LOCALE)
{
if (svFileName.find(svLocale) != string::npos)
{
StringReplace(svFileName, svLocale, "");
break;
}
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
}
Move some CPackedStore methods to VPKDir_t These methods where only used for and by the VPKDir_t structure. Also renamed variable and method names to reflect new changes regarding the use of KeyValues by creating a structure with all values defined during file lookup rather than doing this during the pack process.
2022-11-23 17:09:48 +01:00
return svFileName;
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
}
//-----------------------------------------------------------------------------
Split 'VPKDir_t::Build' into multiple methods Each process of writing out a VPK directory is now within its own method.
2022-11-23 12:27:57 +01:00
// Purpose: writes the vpk directory header
Move some CPackedStore methods to VPKDir_t These methods where only used for and by the VPKDir_t structure. Also renamed variable and method names to reflect new changes regarding the use of KeyValues by creating a structure with all values defined during file lookup rather than doing this during the pack process.
2022-11-23 17:09:48 +01:00
// Input : hDirectoryFile -
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
//-----------------------------------------------------------------------------
Move some CPackedStore methods to VPKDir_t These methods where only used for and by the VPKDir_t structure. Also renamed variable and method names to reflect new changes regarding the use of KeyValues by creating a structure with all values defined during file lookup rather than doing this during the pack process.
2022-11-23 17:09:48 +01:00
void VPKDir_t::WriteHeader(FileHandle_t hDirectoryFile) const
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
{
Move some CPackedStore methods to VPKDir_t These methods where only used for and by the VPKDir_t structure. Also renamed variable and method names to reflect new changes regarding the use of KeyValues by creating a structure with all values defined during file lookup rather than doing this during the pack process.
2022-11-23 17:09:48 +01:00
FileSystem()->Write(&m_vHeader.m_nHeaderMarker, sizeof(uint32_t), hDirectoryFile);
FileSystem()->Write(&m_vHeader.m_nMajorVersion, sizeof(uint16_t), hDirectoryFile);
FileSystem()->Write(&m_vHeader.m_nMinorVersion, sizeof(uint16_t), hDirectoryFile);
FileSystem()->Write(&m_vHeader.m_nDirectorySize, sizeof(uint32_t), hDirectoryFile);
FileSystem()->Write(&m_vHeader.m_nSignatureSize, sizeof(uint32_t), hDirectoryFile);
Split 'VPKDir_t::Build' into multiple methods Each process of writing out a VPK directory is now within its own method.
2022-11-23 12:27:57 +01:00
}
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
Split 'VPKDir_t::Build' into multiple methods Each process of writing out a VPK directory is now within its own method.
2022-11-23 12:27:57 +01:00
//-----------------------------------------------------------------------------
// Purpose: writes the directory tree size
Move some CPackedStore methods to VPKDir_t These methods where only used for and by the VPKDir_t structure. Also renamed variable and method names to reflect new changes regarding the use of KeyValues by creating a structure with all values defined during file lookup rather than doing this during the pack process.
2022-11-23 17:09:48 +01:00
// Input : hDirectoryFile -
Split 'VPKDir_t::Build' into multiple methods Each process of writing out a VPK directory is now within its own method.
2022-11-23 12:27:57 +01:00
//-----------------------------------------------------------------------------
Move some CPackedStore methods to VPKDir_t These methods where only used for and by the VPKDir_t structure. Also renamed variable and method names to reflect new changes regarding the use of KeyValues by creating a structure with all values defined during file lookup rather than doing this during the pack process.
2022-11-23 17:09:48 +01:00
void VPKDir_t::WriteTreeSize(FileHandle_t hDirectoryFile) const
Split 'VPKDir_t::Build' into multiple methods Each process of writing out a VPK directory is now within its own method.
2022-11-23 12:27:57 +01:00
{
Move some CPackedStore methods to VPKDir_t These methods where only used for and by the VPKDir_t structure. Also renamed variable and method names to reflect new changes regarding the use of KeyValues by creating a structure with all values defined during file lookup rather than doing this during the pack process.
2022-11-23 17:09:48 +01:00
FileSystem()->Seek(hDirectoryFile, offsetof(VPKDir_t, m_vHeader.m_nDirectorySize), FileSystemSeek_t::FILESYSTEM_SEEK_HEAD);
FileSystem()->Write(&m_vHeader.m_nDirectorySize, sizeof(uint32_t), hDirectoryFile);
FileSystem()->Write(&PACKFILEINDEX_SEP, sizeof(uint32_t), hDirectoryFile);
Split 'VPKDir_t::Build' into multiple methods Each process of writing out a VPK directory is now within its own method.
2022-11-23 12:27:57 +01:00
}
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
Split 'VPKDir_t::Build' into multiple methods Each process of writing out a VPK directory is now within its own method.
2022-11-23 12:27:57 +01:00
//-----------------------------------------------------------------------------
// Purpose: writes the vpk chunk descriptors
Move some CPackedStore methods to VPKDir_t These methods where only used for and by the VPKDir_t structure. Also renamed variable and method names to reflect new changes regarding the use of KeyValues by creating a structure with all values defined during file lookup rather than doing this during the pack process.
2022-11-23 17:09:48 +01:00
// Input : hDirectoryFile -
Split 'VPKDir_t::Build' into multiple methods Each process of writing out a VPK directory is now within its own method.
2022-11-23 12:27:57 +01:00
// &vMap -
// Output : number of descriptors written
//-----------------------------------------------------------------------------
Move some CPackedStore methods to VPKDir_t These methods where only used for and by the VPKDir_t structure. Also renamed variable and method names to reflect new changes regarding the use of KeyValues by creating a structure with all values defined during file lookup rather than doing this during the pack process.
2022-11-23 17:09:48 +01:00
uint64_t VPKDir_t::WriteDescriptor(FileHandle_t hDirectoryFile, std::map<string, std::map<string, std::list<VPKEntryBlock_t>>>& vMap) const
Split 'VPKDir_t::Build' into multiple methods Each process of writing out a VPK directory is now within its own method.
2022-11-23 12:27:57 +01:00
{
uint64_t nDescriptors = NULL;
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
for (auto& iKeyValue : vMap)
{
Move some CPackedStore methods to VPKDir_t These methods where only used for and by the VPKDir_t structure. Also renamed variable and method names to reflect new changes regarding the use of KeyValues by creating a structure with all values defined during file lookup rather than doing this during the pack process.
2022-11-23 17:09:48 +01:00
FileSystem()->Write(iKeyValue.first.c_str(), (iKeyValue.first.length() + 1), hDirectoryFile);
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
for (auto& jKeyValue : iKeyValue.second)
{
Move some CPackedStore methods to VPKDir_t These methods where only used for and by the VPKDir_t structure. Also renamed variable and method names to reflect new changes regarding the use of KeyValues by creating a structure with all values defined during file lookup rather than doing this during the pack process.
2022-11-23 17:09:48 +01:00
FileSystem()->Write(jKeyValue.first.c_str(), (jKeyValue.first.length() + 1), hDirectoryFile);
VPK system improvements More detailed operation statistics. Discard VPK directory files who's header doesn't match engine requirements. Separated manifest properly between language and context (you can now package for each language and context from a single workspace).
2022-06-05 17:28:39 +02:00
for (auto& vEntry : jKeyValue.second)
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
{
CPackedStore refactor The CPackedStore class now heavily relies on the engine's FileSystem and KeyValues API. Since its part of the engine, it would make more sense to utilize the engine's features instead. It also allows for easier modifications. * The logic behind the build process hasn't changed.
2022-11-23 12:18:33 +01:00
string pszEntryPath = GetFileName(vEntry.m_svEntryPath, true);
Move some CPackedStore methods to VPKDir_t These methods where only used for and by the VPKDir_t structure. Also renamed variable and method names to reflect new changes regarding the use of KeyValues by creating a structure with all values defined during file lookup rather than doing this during the pack process.
2022-11-23 17:09:48 +01:00
FileSystem()->Write(pszEntryPath.c_str(), (pszEntryPath.length() + 1), hDirectoryFile);
CPackedStore refactor The CPackedStore class now heavily relies on the engine's FileSystem and KeyValues API. Since its part of the engine, it would make more sense to utilize the engine's features instead. It also allows for easier modifications. * The logic behind the build process hasn't changed.
2022-11-23 12:18:33 +01:00
Move some CPackedStore methods to VPKDir_t These methods where only used for and by the VPKDir_t structure. Also renamed variable and method names to reflect new changes regarding the use of KeyValues by creating a structure with all values defined during file lookup rather than doing this during the pack process.
2022-11-23 17:09:48 +01:00
FileSystem()->Write(&vEntry.m_nFileCRC, sizeof(uint32_t), hDirectoryFile);
FileSystem()->Write(&vEntry.m_iPreloadSize, sizeof(uint16_t), hDirectoryFile);
FileSystem()->Write(&vEntry.m_iPackFileIndex, sizeof(uint16_t), hDirectoryFile);
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
Improve consistency throughout packedstore class No modifications to the existing logic have been performed. Some parameter names have been renamed to improve code readability and maintainability.
2022-11-16 00:54:51 +01:00
for (size_t i = 0, nc = vEntry.m_vFragments.size(); i < nc; i++)
Additional VPK lib cleanup and performance improvements
2022-10-09 12:08:54 +02:00
{
/*Write chunk descriptor*/
Improve consistency throughout packedstore class No modifications to the existing logic have been performed. Some parameter names have been renamed to improve code readability and maintainability.
2022-11-16 00:54:51 +01:00
const VPKChunkDescriptor_t* pDescriptor = &vEntry.m_vFragments[i];
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
Move some CPackedStore methods to VPKDir_t These methods where only used for and by the VPKDir_t structure. Also renamed variable and method names to reflect new changes regarding the use of KeyValues by creating a structure with all values defined during file lookup rather than doing this during the pack process.
2022-11-23 17:09:48 +01:00
FileSystem()->Write(&pDescriptor->m_nLoadFlags, sizeof(uint32_t), hDirectoryFile);
FileSystem()->Write(&pDescriptor->m_nTextureFlags, sizeof(uint16_t), hDirectoryFile);
FileSystem()->Write(&pDescriptor->m_nPackFileOffset, sizeof(uint64_t), hDirectoryFile);
FileSystem()->Write(&pDescriptor->m_nCompressedSize, sizeof(uint64_t), hDirectoryFile);
FileSystem()->Write(&pDescriptor->m_nUncompressedSize, sizeof(uint64_t), hDirectoryFile);
Additional VPK lib cleanup and performance improvements
2022-10-09 12:08:54 +02:00
if (i != (nc - 1))
{
Move some CPackedStore methods to VPKDir_t These methods where only used for and by the VPKDir_t structure. Also renamed variable and method names to reflect new changes regarding the use of KeyValues by creating a structure with all values defined during file lookup rather than doing this during the pack process.
2022-11-23 17:09:48 +01:00
FileSystem()->Write(&PACKFILEINDEX_SEP, sizeof(uint16_t), hDirectoryFile);
Additional VPK lib cleanup and performance improvements
2022-10-09 12:08:54 +02:00
}
Improve consistency throughout packedstore class No modifications to the existing logic have been performed. Some parameter names have been renamed to improve code readability and maintainability.
2022-11-16 00:54:51 +01:00
else // Mark end of entry.
Additional VPK lib cleanup and performance improvements
2022-10-09 12:08:54 +02:00
{
Move some CPackedStore methods to VPKDir_t These methods where only used for and by the VPKDir_t structure. Also renamed variable and method names to reflect new changes regarding the use of KeyValues by creating a structure with all values defined during file lookup rather than doing this during the pack process.
2022-11-23 17:09:48 +01:00
FileSystem()->Write(&PACKFILEINDEX_END, sizeof(uint16_t), hDirectoryFile);
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
}
Additional VPK lib cleanup and performance improvements
2022-10-09 12:08:54 +02:00
nDescriptors++;
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
}
}
Move some CPackedStore methods to VPKDir_t These methods where only used for and by the VPKDir_t structure. Also renamed variable and method names to reflect new changes regarding the use of KeyValues by creating a structure with all values defined during file lookup rather than doing this during the pack process.
2022-11-23 17:09:48 +01:00
FileSystem()->Write(&PACKFILEINDEX_SEP, sizeof(uint8_t), hDirectoryFile);
Split 'VPKDir_t::Build' into multiple methods Each process of writing out a VPK directory is now within its own method.
2022-11-23 12:27:57 +01:00
}
Move some CPackedStore methods to VPKDir_t These methods where only used for and by the VPKDir_t structure. Also renamed variable and method names to reflect new changes regarding the use of KeyValues by creating a structure with all values defined during file lookup rather than doing this during the pack process.
2022-11-23 17:09:48 +01:00
FileSystem()->Write(&PACKFILEINDEX_SEP, sizeof(uint8_t), hDirectoryFile);
Split 'VPKDir_t::Build' into multiple methods Each process of writing out a VPK directory is now within its own method.
2022-11-23 12:27:57 +01:00
}
Move some CPackedStore methods to VPKDir_t These methods where only used for and by the VPKDir_t structure. Also renamed variable and method names to reflect new changes regarding the use of KeyValues by creating a structure with all values defined during file lookup rather than doing this during the pack process.
2022-11-23 17:09:48 +01:00
FileSystem()->Write(&PACKFILEINDEX_SEP, sizeof(uint8_t), hDirectoryFile);
Split 'VPKDir_t::Build' into multiple methods Each process of writing out a VPK directory is now within its own method.
2022-11-23 12:27:57 +01:00
return nDescriptors;
}
//-----------------------------------------------------------------------------
// Purpose: builds the vpk directory tree
// Input : &vEntryBlocks -
// &vMap -
//-----------------------------------------------------------------------------
void VPKDir_t::BuildDirectoryTree(const vector<VPKEntryBlock_t>& vEntryBlocks, std::map<string, std::map<string, std::list<VPKEntryBlock_t>>>& vMap) const
{
for (const VPKEntryBlock_t& vBlock : vEntryBlocks)
{
string svExtension = GetExtension(vBlock.m_svEntryPath);
string svFilePath = RemoveFileName(vBlock.m_svEntryPath);
if (svFilePath.empty())
{
svFilePath = ' '; // Has to be padded with a space character if empty [root].
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
}
Split 'VPKDir_t::Build' into multiple methods Each process of writing out a VPK directory is now within its own method.
2022-11-23 12:27:57 +01:00
if (!vMap.count(svExtension))
{
vMap.insert({ svExtension, std::map<string, std::list<VPKEntryBlock_t>>() });
}
if (!vMap[svExtension].count(svFilePath))
{
vMap[svExtension].insert({ svFilePath, std::list<VPKEntryBlock_t>() });
}
vMap[svExtension][svFilePath].push_back(vBlock);
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
}
Split 'VPKDir_t::Build' into multiple methods Each process of writing out a VPK directory is now within its own method.
2022-11-23 12:27:57 +01:00
}
CPackedStore refactor The CPackedStore class now heavily relies on the engine's FileSystem and KeyValues API. Since its part of the engine, it would make more sense to utilize the engine's features instead. It also allows for easier modifications. * The logic behind the build process hasn't changed.
2022-11-23 12:18:33 +01:00
Split 'VPKDir_t::Build' into multiple methods Each process of writing out a VPK directory is now within its own method.
2022-11-23 12:27:57 +01:00
//-----------------------------------------------------------------------------
// Purpose: builds the vpk directory file
Move some CPackedStore methods to VPKDir_t These methods where only used for and by the VPKDir_t structure. Also renamed variable and method names to reflect new changes regarding the use of KeyValues by creating a structure with all values defined during file lookup rather than doing this during the pack process.
2022-11-23 17:09:48 +01:00
// Input : &svDirectoryPath -
Split 'VPKDir_t::Build' into multiple methods Each process of writing out a VPK directory is now within its own method.
2022-11-23 12:27:57 +01:00
// &vEntryBlocks -
//-----------------------------------------------------------------------------
Move some CPackedStore methods to VPKDir_t These methods where only used for and by the VPKDir_t structure. Also renamed variable and method names to reflect new changes regarding the use of KeyValues by creating a structure with all values defined during file lookup rather than doing this during the pack process.
2022-11-23 17:09:48 +01:00
void VPKDir_t::BuildDirectoryFile(const string& svDirectoryPath, const vector<VPKEntryBlock_t>& vEntryBlocks)
Split 'VPKDir_t::Build' into multiple methods Each process of writing out a VPK directory is now within its own method.
2022-11-23 12:27:57 +01:00
{
Move some CPackedStore methods to VPKDir_t These methods where only used for and by the VPKDir_t structure. Also renamed variable and method names to reflect new changes regarding the use of KeyValues by creating a structure with all values defined during file lookup rather than doing this during the pack process.
2022-11-23 17:09:48 +01:00
FileHandle_t hDirectoryFile = FileSystem()->Open(svDirectoryPath.c_str(), "wb", "GAME");
if (!hDirectoryFile)
Split 'VPKDir_t::Build' into multiple methods Each process of writing out a VPK directory is now within its own method.
2022-11-23 12:27:57 +01:00
{
Move some CPackedStore methods to VPKDir_t These methods where only used for and by the VPKDir_t structure. Also renamed variable and method names to reflect new changes regarding the use of KeyValues by creating a structure with all values defined during file lookup rather than doing this during the pack process.
2022-11-23 17:09:48 +01:00
Error(eDLL_T::FS, NO_ERROR, "%s - Unable to write to '%s' (read-only?)\n", __FUNCTION__, svDirectoryPath.c_str());
Split 'VPKDir_t::Build' into multiple methods Each process of writing out a VPK directory is now within its own method.
2022-11-23 12:27:57 +01:00
return;
}
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
Split 'VPKDir_t::Build' into multiple methods Each process of writing out a VPK directory is now within its own method.
2022-11-23 12:27:57 +01:00
auto vMap = std::map<string, std::map<string, std::list<VPKEntryBlock_t>>>();
BuildDirectoryTree(vEntryBlocks, vMap);
Move some CPackedStore methods to VPKDir_t These methods where only used for and by the VPKDir_t structure. Also renamed variable and method names to reflect new changes regarding the use of KeyValues by creating a structure with all values defined during file lookup rather than doing this during the pack process.
2022-11-23 17:09:48 +01:00
WriteHeader(hDirectoryFile);
uint64_t nDescriptors = WriteDescriptor(hDirectoryFile, vMap);
VPK system improvements More detailed operation statistics. Discard VPK directory files who's header doesn't match engine requirements. Separated manifest properly between language and context (you can now package for each language and context from a single workspace).
2022-06-05 17:28:39 +02:00
Move some CPackedStore methods to VPKDir_t These methods where only used for and by the VPKDir_t structure. Also renamed variable and method names to reflect new changes regarding the use of KeyValues by creating a structure with all values defined during file lookup rather than doing this during the pack process.
2022-11-23 17:09:48 +01:00
m_vHeader.m_nDirectorySize = static_cast<uint32_t>(FileSystem()->Tell(hDirectoryFile) - sizeof(VPKDirHeader_t));
WriteTreeSize(hDirectoryFile);
Split 'VPKDir_t::Build' into multiple methods Each process of writing out a VPK directory is now within its own method.
2022-11-23 12:27:57 +01:00
Move some CPackedStore methods to VPKDir_t These methods where only used for and by the VPKDir_t structure. Also renamed variable and method names to reflect new changes regarding the use of KeyValues by creating a structure with all values defined during file lookup rather than doing this during the pack process.
2022-11-23 17:09:48 +01:00
FileSystem()->Close(hDirectoryFile);
Split 'VPKDir_t::Build' into multiple methods Each process of writing out a VPK directory is now within its own method.
2022-11-23 12:27:57 +01:00
DevMsg(eDLL_T::FS, "*** Build directory totaling '%zu' bytes with '%zu' entries and '%zu' descriptors\n",
Use correct size type format specifier in packetstore.cpp Using %llu is also correct, but this is the standard approach.
2022-08-08 17:51:09 +02:00
size_t(sizeof(VPKDirHeader_t) + m_vHeader.m_nDirectorySize), vEntryBlocks.size(), nDescriptors);
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
}
Improve consistency throughout packedstore class No modifications to the existing logic have been performed. Some parameter names have been renamed to improve code readability and maintainability.
2022-11-16 00:54:51 +01:00
//-----------------------------------------------------------------------------
// Singleton
//-----------------------------------------------------------------------------
Update packedstore.cpp
2022-06-02 01:59:03 +02:00
CPackedStore* g_pPackedStore = new CPackedStore();
Reference in New Issue Copy Permalink