R5Reloaded/r5sdk
1
0
Fork 0
mirror of https://github.com/Mauler125/r5sdk.git synced 2025-02-09 19:15:03 +01:00
Code Issues Packages Projects Releases Wiki Activity
r5sdk/r5dev/vpklib/packedstore.h

222 lines
8.6 KiB
C
Raw Normal View History

CPackedStore: light optimizations * Don't call copy constructor on entry blocks when building the directory file. * Don't call copy constructor for string when we don't want to sanitize the directory file name.
2022-10-12 23:39:26 +02:00
#ifndef PACKEDSTORE_H
#define PACKEDSTORE_H
/*******************************************************************
* ██████╗ ██╗ ██╗ ██╗██████╗ ██╗ ██╗ ██╗ ██╗██████╗ *
* ██╔══██╗███║ ██║ ██║██╔══██╗██║ ██╔╝ ██║ ██║██╔══██╗ *
* ██████╔╝╚██║ ██║ ██║██████╔╝█████╔╝ ██║ ██║██████╔╝ *
* ██╔══██╗ ██║ ╚██╗ ██╔╝██╔═══╝ ██╔═██╗ ██║ ██║██╔══██╗ *
* ██║ ██║ ██║ ╚████╔╝ ██║ ██║ ██╗ ███████╗██║██████╔╝ *
* ╚═╝ ╚═╝ ╚═╝ ╚═══╝ ╚═╝ ╚═╝ ╚═╝ ╚══════╝╚═╝╚═════╝ *
*******************************************************************/
VpkLib: check for manifest file first before opening the pack file We do this as if we happen to not have any manifest whatsoever, but already opened the block file, and the block file already existed and was still valid, it would be emptied out. Only open it if we are actually going to write into it!
2024-01-13 23:55:01 +01:00
#include "public/ipackedstore.h"
CPackedStore refactor The CPackedStore class now heavily relies on the engine's FileSystem and KeyValues API. Since its part of the engine, it would make more sense to utilize the engine's features instead. It also allows for easier modifications. * The logic behind the build process hasn't changed.
2022-11-23 12:18:33 +01:00
#include "public/ifilesystem.h"
CPackedStore refactor * Improve naming convention, and make it more consistent. * Use engine types as much as possible. * Reduced the amount of string copy constructions.
2023-05-29 21:41:17 +02:00
#include "public/tier1/strtools.h"
#include "public/tier1/utlvector.h"
#include "public/tier1/utlstring.h"
Code base refactor + major performance and readability improvement. Read description for details. * Codebase restructured to SourceSDK codebase style and .cpp/.h assertion paths in the game executable. * Document most functions with valve style 'Purpose' blocks. * Rename variables to match the rest of the codebase and Valve's naming convention. * Dedicated DLL and the SDKLauncher now share the same codebase as the DevSDK. * Obtain globals or pointers directly instead of waiting for runtime initialized data. * Dynamically search for all functions and globals (this doesn't count for dedicated yet!). * Initialize most in-SDK variables. * Move certain prints and other utilities under ConVars to reduce verbosity and increase performance. * Print all pattern scan results through a virtual function to make it easier to add and debug new patterns in the future. * Type global var pointers appropriately if class or type is known and implemented. * Forward declare 'CClient' class to avoid having 2 'g_pClient' copies. * Add IDA's pseudo definitions for easier prototyping with decompiled assembly code. * RPAK decompress Command callback implementation. * Load decompressed RPaks from 'paks\Win32\' overriding the ones in 'paks\Win64\' (the decompress callback will automatically fix the header and write it to 'paks\Win32\'). * VPK decompress Command callback implementation. * Move CRC32 ands Adler32 to implementation files. * Server will print out more details about the connecting client. * Upgrade ImGui lib to v1.86. * Don't compile id3dx.h for dedicated. * Don't compile id3dx.cpp for dedicated * Implement DevMsg print function allowing to print information to the in-game VGUI/RUI console overlay, ImGui console overlay and the external windows console * Fixed bug where the Error function would not properly terminate the process when an error is called. This caused access violations for critical/non-recoverable errors. * Fixed bug where the game would crash if the console or server browser was enabled while the game was still starting up. * Several bug fixes for the dedicated server (warning: dedicated is still considered work-in-progress!).
2021-12-25 22:36:38 +01:00
#include "thirdparty/lzham/include/lzham.h"
Additional cleanup Cast to proper types. Set the archive offset only instead of the entire block. The entry could still have its own flags. The only thing needed for saving data is sharing the same offset and size.
2022-06-05 12:28:49 +02:00
constexpr unsigned int VPK_HEADER_MARKER = 0x55AA1234;
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
constexpr unsigned int VPK_MAJOR_VERSION = 2;
constexpr unsigned int VPK_MINOR_VERSION = 3;
Additional cleanup Cast to proper types. Set the archive offset only instead of the entire block. The entry could still have its own flags. The only thing needed for saving data is sharing the same offset and size.
2022-06-05 12:28:49 +02:00
constexpr unsigned int VPK_DICT_SIZE = 20;
constexpr int ENTRY_MAX_LEN = 1024 * 1024;
Add convar for adjusting LZHAM compressor max helper threads
2022-11-17 20:37:12 +01:00
constexpr int PACKFILEPATCH_MAX = 512;
CPackedStore refactor The CPackedStore class now heavily relies on the engine's FileSystem and KeyValues API. Since its part of the engine, it would make more sense to utilize the engine's features instead. It also allows for easier modifications. * The logic behind the build process hasn't changed.
2022-11-23 12:18:33 +01:00
constexpr int PACKFILEINDEX_SEP = 0x0;
Improve consistency throughout packedstore class No modifications to the existing logic have been performed. Some parameter names have been renamed to improve code readability and maintainability.
2022-11-16 00:54:51 +01:00
constexpr int PACKFILEINDEX_END = 0xffff;
CPackedStore refactor * Improve naming convention, and make it more consistent. * Use engine types as much as possible. * Reduced the amount of string copy constructions.
2023-05-29 21:41:17 +02:00
constexpr const char VPK_IGNORE_FILE[] = ".vpkignore";
Code base refactor + major performance and readability improvement. Read description for details. * Codebase restructured to SourceSDK codebase style and .cpp/.h assertion paths in the game executable. * Document most functions with valve style 'Purpose' blocks. * Rename variables to match the rest of the codebase and Valve's naming convention. * Dedicated DLL and the SDKLauncher now share the same codebase as the DevSDK. * Obtain globals or pointers directly instead of waiting for runtime initialized data. * Dynamically search for all functions and globals (this doesn't count for dedicated yet!). * Initialize most in-SDK variables. * Move certain prints and other utilities under ConVars to reduce verbosity and increase performance. * Print all pattern scan results through a virtual function to make it easier to add and debug new patterns in the future. * Type global var pointers appropriately if class or type is known and implemented. * Forward declare 'CClient' class to avoid having 2 'g_pClient' copies. * Add IDA's pseudo definitions for easier prototyping with decompiled assembly code. * RPAK decompress Command callback implementation. * Load decompressed RPaks from 'paks\Win32\' overriding the ones in 'paks\Win64\' (the decompress callback will automatically fix the header and write it to 'paks\Win32\'). * VPK decompress Command callback implementation. * Move CRC32 ands Adler32 to implementation files. * Server will print out more details about the connecting client. * Upgrade ImGui lib to v1.86. * Don't compile id3dx.h for dedicated. * Don't compile id3dx.cpp for dedicated * Implement DevMsg print function allowing to print information to the in-game VGUI/RUI console overlay, ImGui console overlay and the external windows console * Fixed bug where the Error function would not properly terminate the process when an error is called. This caused access violations for critical/non-recoverable errors. * Fixed bug where the game would crash if the console or server browser was enabled while the game was still starting up. * Several bug fixes for the dedicated server (warning: dedicated is still considered work-in-progress!).
2021-12-25 22:36:38 +01:00
CPackedStore refactor * Improve naming convention, and make it more consistent. * Use engine types as much as possible. * Reduced the amount of string copy constructions.
2023-05-29 21:41:17 +02:00
static const char* const DIR_TARGET[]
Use static buffer for file chunking and compressing during VPK packing Avoid using new every time. Chunk can be at most 1024 * 1024.
2022-10-02 12:17:03 +02:00
{
"server",
"client"
};
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
VPK utility code improvements - Only process referenced pack files, previously, if an entry in the VPK file was build into a pack of index 128, the code would generate a list of pack files from 0 to 128. - Added documentation to VPK structure.
2023-07-06 00:49:17 +02:00
//-----------------------------------------------------------------------------
// KeyValues structure for the VPK manifest file. This struct gets populated by
// the VPK's corresponding manifest file, which ultimately determines how each
// asset is getting packed into the VPK.
//-----------------------------------------------------------------------------
CPackedStore refactor The CPackedStore class now heavily relies on the engine's FileSystem and KeyValues API. Since its part of the engine, it would make more sense to utilize the engine's features instead. It also allows for easier modifications. * The logic behind the build process hasn't changed.
2022-11-23 12:18:33 +01:00
struct VPKKeyValues_t
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
{
Light VPK code optimizations Cache string buffer ptr.
2023-07-08 02:18:18 +02:00
static constexpr uint16_t TEXTURE_FLAGS_DEFAULT =
VPK utility code improvements - Only process referenced pack files, previously, if an entry in the VPK file was build into a pack of index 128, the code would generate a list of pack files from 0 to 128. - Added documentation to VPK structure.
2023-07-06 00:49:17 +02:00
EPackedTextureFlags::TEXTURE_DEFAULT;
Light VPK code optimizations Cache string buffer ptr.
2023-07-08 02:18:18 +02:00
static constexpr uint32_t LOAD_FLAGS_DEFAULT =
VPK utility code improvements - Only process referenced pack files, previously, if an entry in the VPK file was build into a pack of index 128, the code would generate a list of pack files from 0 to 128. - Added documentation to VPK structure.
2023-07-06 00:49:17 +02:00
EPackedLoadFlags::LOAD_VISIBLE | EPackedLoadFlags::LOAD_CACHE;
CPackedStore refactor The CPackedStore class now heavily relies on the engine's FileSystem and KeyValues API. Since its part of the engine, it would make more sense to utilize the engine's features instead. It also allows for easier modifications. * The logic behind the build process hasn't changed.
2022-11-23 12:18:33 +01:00
CPackedStore refactor * Improve naming convention, and make it more consistent. * Use engine types as much as possible. * Reduced the amount of string copy constructions.
2023-05-29 21:41:17 +02:00
CUtlString m_EntryPath;
CPackedStore refactor The CPackedStore class now heavily relies on the engine's FileSystem and KeyValues API. Since its part of the engine, it would make more sense to utilize the engine's features instead. It also allows for easier modifications. * The logic behind the build process hasn't changed.
2022-11-23 12:18:33 +01:00
uint16_t m_iPreloadSize;
uint32_t m_nLoadFlags;
uint16_t m_nTextureFlags;
bool m_bUseCompression;
CPackedStore: Rename 'useDataSharing' to 'deDuplicate' More descriptive in what it does (deduplication of existing copies).
2023-04-03 20:08:36 +02:00
bool m_bDeduplicate;
CPackedStore refactor The CPackedStore class now heavily relies on the engine's FileSystem and KeyValues API. Since its part of the engine, it would make more sense to utilize the engine's features instead. It also allows for easier modifications. * The logic behind the build process hasn't changed.
2022-11-23 12:18:33 +01:00
VPK utility code improvements - Only process referenced pack files, previously, if an entry in the VPK file was build into a pack of index 128, the code would generate a list of pack files from 0 to 128. - Added documentation to VPK structure.
2023-07-06 00:49:17 +02:00
VPKKeyValues_t(const CUtlString& svEntryPath = "",
uint16_t iPreloadSize = NULL,
uint32_t nLoadFlags = LOAD_FLAGS_DEFAULT,
uint16_t nTextureFlags = TEXTURE_FLAGS_DEFAULT,
bool bUseCompression = true, bool bDeduplicate = true);
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
};
VPK utility code improvements - Only process referenced pack files, previously, if an entry in the VPK file was build into a pack of index 128, the code would generate a list of pack files from 0 to 128. - Added documentation to VPK structure.
2023-07-06 00:49:17 +02:00
//-----------------------------------------------------------------------------
// An asset packed into a VPK is carved into 'ENTRY_MAX_LEN' chunks, the chunk
// is then optionally compressed. A chunk is NOT compressed if the compressed
// size equals the uncompressed size.
//-----------------------------------------------------------------------------
Update terminology entirely for packedstore.h/.cpp VPK tooling also works on the R1 game (Titanfall 1)
2022-06-06 14:54:22 +02:00
struct VPKChunkDescriptor_t
Code base refactor + major performance and readability improvement. Read description for details. * Codebase restructured to SourceSDK codebase style and .cpp/.h assertion paths in the game executable. * Document most functions with valve style 'Purpose' blocks. * Rename variables to match the rest of the codebase and Valve's naming convention. * Dedicated DLL and the SDKLauncher now share the same codebase as the DevSDK. * Obtain globals or pointers directly instead of waiting for runtime initialized data. * Dynamically search for all functions and globals (this doesn't count for dedicated yet!). * Initialize most in-SDK variables. * Move certain prints and other utilities under ConVars to reduce verbosity and increase performance. * Print all pattern scan results through a virtual function to make it easier to add and debug new patterns in the future. * Type global var pointers appropriately if class or type is known and implemented. * Forward declare 'CClient' class to avoid having 2 'g_pClient' copies. * Add IDA's pseudo definitions for easier prototyping with decompiled assembly code. * RPAK decompress Command callback implementation. * Load decompressed RPaks from 'paks\Win32\' overriding the ones in 'paks\Win64\' (the decompress callback will automatically fix the header and write it to 'paks\Win32\'). * VPK decompress Command callback implementation. * Move CRC32 ands Adler32 to implementation files. * Server will print out more details about the connecting client. * Upgrade ImGui lib to v1.86. * Don't compile id3dx.h for dedicated. * Don't compile id3dx.cpp for dedicated * Implement DevMsg print function allowing to print information to the in-game VGUI/RUI console overlay, ImGui console overlay and the external windows console * Fixed bug where the Error function would not properly terminate the process when an error is called. This caused access violations for critical/non-recoverable errors. * Fixed bug where the game would crash if the console or server browser was enabled while the game was still starting up. * Several bug fixes for the dedicated server (warning: dedicated is still considered work-in-progress!).
2021-12-25 22:36:38 +01:00
{
VPK utility code improvements - Only process referenced pack files, previously, if an entry in the VPK file was build into a pack of index 128, the code would generate a list of pack files from 0 to 128. - Added documentation to VPK structure.
2023-07-06 00:49:17 +02:00
uint32_t m_nLoadFlags;
// Texture flags (only used if the entry is a vtf).
uint16_t m_nTextureFlags;
// Offset in pack file.
uint64_t m_nPackFileOffset;
uint64_t m_nCompressedSize;
uint64_t m_nUncompressedSize;
Code base refactor + major performance and readability improvement. Read description for details. * Codebase restructured to SourceSDK codebase style and .cpp/.h assertion paths in the game executable. * Document most functions with valve style 'Purpose' blocks. * Rename variables to match the rest of the codebase and Valve's naming convention. * Dedicated DLL and the SDKLauncher now share the same codebase as the DevSDK. * Obtain globals or pointers directly instead of waiting for runtime initialized data. * Dynamically search for all functions and globals (this doesn't count for dedicated yet!). * Initialize most in-SDK variables. * Move certain prints and other utilities under ConVars to reduce verbosity and increase performance. * Print all pattern scan results through a virtual function to make it easier to add and debug new patterns in the future. * Type global var pointers appropriately if class or type is known and implemented. * Forward declare 'CClient' class to avoid having 2 'g_pClient' copies. * Add IDA's pseudo definitions for easier prototyping with decompiled assembly code. * RPAK decompress Command callback implementation. * Load decompressed RPaks from 'paks\Win32\' overriding the ones in 'paks\Win64\' (the decompress callback will automatically fix the header and write it to 'paks\Win32\'). * VPK decompress Command callback implementation. * Move CRC32 ands Adler32 to implementation files. * Server will print out more details about the connecting client. * Upgrade ImGui lib to v1.86. * Don't compile id3dx.h for dedicated. * Don't compile id3dx.cpp for dedicated * Implement DevMsg print function allowing to print information to the in-game VGUI/RUI console overlay, ImGui console overlay and the external windows console * Fixed bug where the Error function would not properly terminate the process when an error is called. This caused access violations for critical/non-recoverable errors. * Fixed bug where the game would crash if the console or server browser was enabled while the game was still starting up. * Several bug fixes for the dedicated server (warning: dedicated is still considered work-in-progress!).
2021-12-25 22:36:38 +01:00
CPackedStore refactor * Improve naming convention, and make it more consistent. * Use engine types as much as possible. * Reduced the amount of string copy constructions.
2023-05-29 21:41:17 +02:00
VPKChunkDescriptor_t()
: m_nLoadFlags(0)
, m_nTextureFlags(0)
, m_nPackFileOffset(0)
, m_nCompressedSize(0)
, m_nUncompressedSize(0)
{
}
Move some CPackedStore methods to VPKDir_t These methods where only used for and by the VPKDir_t structure. Also renamed variable and method names to reflect new changes regarding the use of KeyValues by creating a structure with all values defined during file lookup rather than doing this during the pack process.
2022-11-23 17:09:48 +01:00
VPKChunkDescriptor_t(FileHandle_t hDirectoryFile);
VPK utility code improvements - Only process referenced pack files, previously, if an entry in the VPK file was build into a pack of index 128, the code would generate a list of pack files from 0 to 128. - Added documentation to VPK structure.
2023-07-06 00:49:17 +02:00
VPKChunkDescriptor_t(uint32_t nLoadFlags, uint16_t nTextureFlags,
uint64_t nPackFileOffset, uint64_t nCompressedSize, uint64_t nUncompressedSize);
Code base refactor + major performance and readability improvement. Read description for details. * Codebase restructured to SourceSDK codebase style and .cpp/.h assertion paths in the game executable. * Document most functions with valve style 'Purpose' blocks. * Rename variables to match the rest of the codebase and Valve's naming convention. * Dedicated DLL and the SDKLauncher now share the same codebase as the DevSDK. * Obtain globals or pointers directly instead of waiting for runtime initialized data. * Dynamically search for all functions and globals (this doesn't count for dedicated yet!). * Initialize most in-SDK variables. * Move certain prints and other utilities under ConVars to reduce verbosity and increase performance. * Print all pattern scan results through a virtual function to make it easier to add and debug new patterns in the future. * Type global var pointers appropriately if class or type is known and implemented. * Forward declare 'CClient' class to avoid having 2 'g_pClient' copies. * Add IDA's pseudo definitions for easier prototyping with decompiled assembly code. * RPAK decompress Command callback implementation. * Load decompressed RPaks from 'paks\Win32\' overriding the ones in 'paks\Win64\' (the decompress callback will automatically fix the header and write it to 'paks\Win32\'). * VPK decompress Command callback implementation. * Move CRC32 ands Adler32 to implementation files. * Server will print out more details about the connecting client. * Upgrade ImGui lib to v1.86. * Don't compile id3dx.h for dedicated. * Don't compile id3dx.cpp for dedicated * Implement DevMsg print function allowing to print information to the in-game VGUI/RUI console overlay, ImGui console overlay and the external windows console * Fixed bug where the Error function would not properly terminate the process when an error is called. This caused access violations for critical/non-recoverable errors. * Fixed bug where the game would crash if the console or server browser was enabled while the game was still starting up. * Several bug fixes for the dedicated server (warning: dedicated is still considered work-in-progress!).
2021-12-25 22:36:38 +01:00
};
VPK utility code improvements - Only process referenced pack files, previously, if an entry in the VPK file was build into a pack of index 128, the code would generate a list of pack files from 0 to 128. - Added documentation to VPK structure.
2023-07-06 00:49:17 +02:00
//-----------------------------------------------------------------------------
// An asset packed into a VPK is represented as an entry block.
//-----------------------------------------------------------------------------
See description * Added 2 new FileSystem pointers with new features in their classes. * Register all factory instances created by the GameDLL in the SDK. * Added new command 'fs_mount_vpk' to mount a specified VPK file. * Renamed 'fs_decompress_pak' to 'fs_unpack_vpk'. * Some renaming of Factory and VPK types. * Some light optimizations/cleanup.
2022-04-09 00:59:42 +02:00
struct VPKEntryBlock_t
Code base refactor + major performance and readability improvement. Read description for details. * Codebase restructured to SourceSDK codebase style and .cpp/.h assertion paths in the game executable. * Document most functions with valve style 'Purpose' blocks. * Rename variables to match the rest of the codebase and Valve's naming convention. * Dedicated DLL and the SDKLauncher now share the same codebase as the DevSDK. * Obtain globals or pointers directly instead of waiting for runtime initialized data. * Dynamically search for all functions and globals (this doesn't count for dedicated yet!). * Initialize most in-SDK variables. * Move certain prints and other utilities under ConVars to reduce verbosity and increase performance. * Print all pattern scan results through a virtual function to make it easier to add and debug new patterns in the future. * Type global var pointers appropriately if class or type is known and implemented. * Forward declare 'CClient' class to avoid having 2 'g_pClient' copies. * Add IDA's pseudo definitions for easier prototyping with decompiled assembly code. * RPAK decompress Command callback implementation. * Load decompressed RPaks from 'paks\Win32\' overriding the ones in 'paks\Win64\' (the decompress callback will automatically fix the header and write it to 'paks\Win32\'). * VPK decompress Command callback implementation. * Move CRC32 ands Adler32 to implementation files. * Server will print out more details about the connecting client. * Upgrade ImGui lib to v1.86. * Don't compile id3dx.h for dedicated. * Don't compile id3dx.cpp for dedicated * Implement DevMsg print function allowing to print information to the in-game VGUI/RUI console overlay, ImGui console overlay and the external windows console * Fixed bug where the Error function would not properly terminate the process when an error is called. This caused access violations for critical/non-recoverable errors. * Fixed bug where the game would crash if the console or server browser was enabled while the game was still starting up. * Several bug fixes for the dedicated server (warning: dedicated is still considered work-in-progress!).
2021-12-25 22:36:38 +01:00
{
VPK utility code improvements - Only process referenced pack files, previously, if an entry in the VPK file was build into a pack of index 128, the code would generate a list of pack files from 0 to 128. - Added documentation to VPK structure.
2023-07-06 00:49:17 +02:00
// Crc32 for the uncompressed entry.
uint32_t m_nFileCRC;
uint16_t m_iPreloadSize;
// Index of the pack file that contains this entry.
uint16_t m_iPackFileIndex;
// Vector of all the chunks of a given entry
// (chunks have a size limit of 1 MiB, anything
// over this limit is fragmented into smaller chunks).
CUtlVector<VPKChunkDescriptor_t> m_Fragments;
CUtlString m_EntryPath;
Update terminology entirely for packedstore.h/.cpp VPK tooling also works on the R1 game (Titanfall 1)
2022-06-06 14:54:22 +02:00
CPackedStore refactor * Improve naming convention, and make it more consistent. * Use engine types as much as possible. * Reduced the amount of string copy constructions.
2023-05-29 21:41:17 +02:00
VPKEntryBlock_t(FileHandle_t pFile, const char* svEntryPath);
Move some CPackedStore methods to VPKDir_t These methods where only used for and by the VPKDir_t structure. Also renamed variable and method names to reflect new changes regarding the use of KeyValues by creating a structure with all values defined during file lookup rather than doing this during the pack process.
2022-11-23 17:09:48 +01:00
VPKEntryBlock_t(const uint8_t* pData, size_t nLen, int64_t nOffset, uint16_t iPreloadSize,
CPackedStore refactor * Improve naming convention, and make it more consistent. * Use engine types as much as possible. * Reduced the amount of string copy constructions.
2023-05-29 21:41:17 +02:00
uint16_t iPackFileIndex, uint32_t nLoadFlags, uint16_t nTextureFlags, const char* svEntryPath);
VPKEntryBlock_t(const VPKEntryBlock_t& other)
: m_nFileCRC(other.m_nFileCRC)
, m_iPreloadSize(other.m_iPreloadSize)
, m_iPackFileIndex(other.m_iPackFileIndex)
, m_EntryPath(other.m_EntryPath)
{
// Has to be explicitly copied!
m_Fragments = other.m_Fragments;
}
Code base refactor + major performance and readability improvement. Read description for details. * Codebase restructured to SourceSDK codebase style and .cpp/.h assertion paths in the game executable. * Document most functions with valve style 'Purpose' blocks. * Rename variables to match the rest of the codebase and Valve's naming convention. * Dedicated DLL and the SDKLauncher now share the same codebase as the DevSDK. * Obtain globals or pointers directly instead of waiting for runtime initialized data. * Dynamically search for all functions and globals (this doesn't count for dedicated yet!). * Initialize most in-SDK variables. * Move certain prints and other utilities under ConVars to reduce verbosity and increase performance. * Print all pattern scan results through a virtual function to make it easier to add and debug new patterns in the future. * Type global var pointers appropriately if class or type is known and implemented. * Forward declare 'CClient' class to avoid having 2 'g_pClient' copies. * Add IDA's pseudo definitions for easier prototyping with decompiled assembly code. * RPAK decompress Command callback implementation. * Load decompressed RPaks from 'paks\Win32\' overriding the ones in 'paks\Win64\' (the decompress callback will automatically fix the header and write it to 'paks\Win32\'). * VPK decompress Command callback implementation. * Move CRC32 ands Adler32 to implementation files. * Server will print out more details about the connecting client. * Upgrade ImGui lib to v1.86. * Don't compile id3dx.h for dedicated. * Don't compile id3dx.cpp for dedicated * Implement DevMsg print function allowing to print information to the in-game VGUI/RUI console overlay, ImGui console overlay and the external windows console * Fixed bug where the Error function would not properly terminate the process when an error is called. This caused access violations for critical/non-recoverable errors. * Fixed bug where the game would crash if the console or server browser was enabled while the game was still starting up. * Several bug fixes for the dedicated server (warning: dedicated is still considered work-in-progress!).
2021-12-25 22:36:38 +01:00
};
VPK utility code improvements - Only process referenced pack files, previously, if an entry in the VPK file was build into a pack of index 128, the code would generate a list of pack files from 0 to 128. - Added documentation to VPK structure.
2023-07-06 00:49:17 +02:00
//-----------------------------------------------------------------------------
// The VPK directory file header.
//-----------------------------------------------------------------------------
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
struct VPKDirHeader_t
Code base refactor + major performance and readability improvement. Read description for details. * Codebase restructured to SourceSDK codebase style and .cpp/.h assertion paths in the game executable. * Document most functions with valve style 'Purpose' blocks. * Rename variables to match the rest of the codebase and Valve's naming convention. * Dedicated DLL and the SDKLauncher now share the same codebase as the DevSDK. * Obtain globals or pointers directly instead of waiting for runtime initialized data. * Dynamically search for all functions and globals (this doesn't count for dedicated yet!). * Initialize most in-SDK variables. * Move certain prints and other utilities under ConVars to reduce verbosity and increase performance. * Print all pattern scan results through a virtual function to make it easier to add and debug new patterns in the future. * Type global var pointers appropriately if class or type is known and implemented. * Forward declare 'CClient' class to avoid having 2 'g_pClient' copies. * Add IDA's pseudo definitions for easier prototyping with decompiled assembly code. * RPAK decompress Command callback implementation. * Load decompressed RPaks from 'paks\Win32\' overriding the ones in 'paks\Win64\' (the decompress callback will automatically fix the header and write it to 'paks\Win32\'). * VPK decompress Command callback implementation. * Move CRC32 ands Adler32 to implementation files. * Server will print out more details about the connecting client. * Upgrade ImGui lib to v1.86. * Don't compile id3dx.h for dedicated. * Don't compile id3dx.cpp for dedicated * Implement DevMsg print function allowing to print information to the in-game VGUI/RUI console overlay, ImGui console overlay and the external windows console * Fixed bug where the Error function would not properly terminate the process when an error is called. This caused access violations for critical/non-recoverable errors. * Fixed bug where the game would crash if the console or server browser was enabled while the game was still starting up. * Several bug fixes for the dedicated server (warning: dedicated is still considered work-in-progress!).
2021-12-25 22:36:38 +01:00
{
VPK utility code improvements - Only process referenced pack files, previously, if an entry in the VPK file was build into a pack of index 128, the code would generate a list of pack files from 0 to 128. - Added documentation to VPK structure.
2023-07-06 00:49:17 +02:00
uint32_t m_nHeaderMarker; // File magic.
uint16_t m_nMajorVersion; // Vpk major version.
uint16_t m_nMinorVersion; // Vpk minor version.
uint32_t m_nDirectorySize; // Directory tree size.
uint32_t m_nSignatureSize; // Directory signature.
Move some CPackedStore methods to VPKDir_t These methods where only used for and by the VPKDir_t structure. Also renamed variable and method names to reflect new changes regarding the use of KeyValues by creating a structure with all values defined during file lookup rather than doing this during the pack process.
2022-11-23 17:09:48 +01:00
};
VPK utility code improvements - Only process referenced pack files, previously, if an entry in the VPK file was build into a pack of index 128, the code would generate a list of pack files from 0 to 128. - Added documentation to VPK structure.
2023-07-06 00:49:17 +02:00
//-----------------------------------------------------------------------------
// The VPK directory tree structure.
//-----------------------------------------------------------------------------
Initial implementation of compressed VPK rebuilding The 'fs_pack_vpk' allows compressing specific files into a VPK (compressed). This implementation has better compression ratio's then currently available VPK tools with compression (at the cost of speed). The implementation is early, a directory iterator with a files for flags is planned soon to fully rebuild VPK's on-demand. A dedicated standalone program is also planned. The output dir and archive formats should match 1:1 with the original game
2022-05-30 02:56:15 +02:00
struct VPKDir_t
{
VPK utility code improvements - Only process referenced pack files, previously, if an entry in the VPK file was build into a pack of index 128, the code would generate a list of pack files from 0 to 128. - Added documentation to VPK structure.
2023-07-06 00:49:17 +02:00
VPKDirHeader_t m_Header;
CUtlString m_DirFilePath;
Light VPK code optimizations Cache string buffer ptr.
2023-07-08 02:18:18 +02:00
CUtlVector<VPKEntryBlock_t> m_EntryBlocks;
VPK utility code improvements - Only process referenced pack files, previously, if an entry in the VPK file was build into a pack of index 128, the code would generate a list of pack files from 0 to 128. - Added documentation to VPK structure.
2023-07-06 00:49:17 +02:00
// This set only contains packfile indices used
// by the directory tree, notated as pak000_xxx.
std::set<uint16_t> m_PakFileIndices;
VpkLib: move header check to dir construction and flag error on failure Moved the construction as otherwise we would iterate over potentially bad or incompatible data.
2024-01-14 00:43:15 +01:00
bool m_bInitFailed;
Code base refactor + major performance and readability improvement. Read description for details. * Codebase restructured to SourceSDK codebase style and .cpp/.h assertion paths in the game executable. * Document most functions with valve style 'Purpose' blocks. * Rename variables to match the rest of the codebase and Valve's naming convention. * Dedicated DLL and the SDKLauncher now share the same codebase as the DevSDK. * Obtain globals or pointers directly instead of waiting for runtime initialized data. * Dynamically search for all functions and globals (this doesn't count for dedicated yet!). * Initialize most in-SDK variables. * Move certain prints and other utilities under ConVars to reduce verbosity and increase performance. * Print all pattern scan results through a virtual function to make it easier to add and debug new patterns in the future. * Type global var pointers appropriately if class or type is known and implemented. * Forward declare 'CClient' class to avoid having 2 'g_pClient' copies. * Add IDA's pseudo definitions for easier prototyping with decompiled assembly code. * RPAK decompress Command callback implementation. * Load decompressed RPaks from 'paks\Win32\' overriding the ones in 'paks\Win64\' (the decompress callback will automatically fix the header and write it to 'paks\Win32\'). * VPK decompress Command callback implementation. * Move CRC32 ands Adler32 to implementation files. * Server will print out more details about the connecting client. * Upgrade ImGui lib to v1.86. * Don't compile id3dx.h for dedicated. * Don't compile id3dx.cpp for dedicated * Implement DevMsg print function allowing to print information to the in-game VGUI/RUI console overlay, ImGui console overlay and the external windows console * Fixed bug where the Error function would not properly terminate the process when an error is called. This caused access violations for critical/non-recoverable errors. * Fixed bug where the game would crash if the console or server browser was enabled while the game was still starting up. * Several bug fixes for the dedicated server (warning: dedicated is still considered work-in-progress!).
2021-12-25 22:36:38 +01:00
Fix incorrect VPK tree during build This commit fixes the following bugs: - Use of STL map on CUtlString, CUtlString does not have a lesser operator, thus it always fails to check whether or not the string is already present in the map. The tree structure cannot contain duplicates per tree scope. - Use of backward slashes in vpk file tree structure, path separators should always be a forwards slash. The code was originally designed to be forwards slash only, but some paths were adjusted to take backwards slash, as the KeyValues class treats the '/' char as a delimiter for subfields. This has therefore lead into the whole code changing to it. The code has now been changed to the point where only the KeyValues utilities take the backward slash paths. - Trailing slash character on file path, that ended up getting written as-is in the path tree. The VPK tree builder is now part of a separate nested class in 'VPKDir_t'.
2023-07-05 23:04:07 +02:00
class CTreeBuilder
{
public:
typedef std::map<std::string, std::list<VPKEntryBlock_t>> PathContainer_t;
typedef std::map<std::string, PathContainer_t> TypeContainer_t;
void BuildTree(const CUtlVector<VPKEntryBlock_t>& entryBlocks);
Adjust types and format specifiers Make it more correct.
2023-07-15 16:25:35 +02:00
int WriteTree(FileHandle_t hDirectoryFile) const;
Fix incorrect VPK tree during build This commit fixes the following bugs: - Use of STL map on CUtlString, CUtlString does not have a lesser operator, thus it always fails to check whether or not the string is already present in the map. The tree structure cannot contain duplicates per tree scope. - Use of backward slashes in vpk file tree structure, path separators should always be a forwards slash. The code was originally designed to be forwards slash only, but some paths were adjusted to take backwards slash, as the KeyValues class treats the '/' char as a delimiter for subfields. This has therefore lead into the whole code changing to it. The code has now been changed to the point where only the KeyValues utilities take the backward slash paths. - Trailing slash character on file path, that ended up getting written as-is in the path tree. The VPK tree builder is now part of a separate nested class in 'VPKDir_t'.
2023-07-05 23:04:07 +02:00
private:
TypeContainer_t m_FileTree;
};
Move some CPackedStore methods to VPKDir_t These methods where only used for and by the VPKDir_t structure. Also renamed variable and method names to reflect new changes regarding the use of KeyValues by creating a structure with all values defined during file lookup rather than doing this during the pack process.
2022-11-23 17:09:48 +01:00
VPKDir_t()
{
Fix incorrect VPK tree during build This commit fixes the following bugs: - Use of STL map on CUtlString, CUtlString does not have a lesser operator, thus it always fails to check whether or not the string is already present in the map. The tree structure cannot contain duplicates per tree scope. - Use of backward slashes in vpk file tree structure, path separators should always be a forwards slash. The code was originally designed to be forwards slash only, but some paths were adjusted to take backwards slash, as the KeyValues class treats the '/' char as a delimiter for subfields. This has therefore lead into the whole code changing to it. The code has now been changed to the point where only the KeyValues utilities take the backward slash paths. - Trailing slash character on file path, that ended up getting written as-is in the path tree. The VPK tree builder is now part of a separate nested class in 'VPKDir_t'.
2023-07-05 23:04:07 +02:00
m_Header.m_nHeaderMarker = VPK_HEADER_MARKER; m_Header.m_nMajorVersion = VPK_MAJOR_VERSION;
CPackedStore refactor * Improve naming convention, and make it more consistent. * Use engine types as much as possible. * Reduced the amount of string copy constructions.
2023-05-29 21:41:17 +02:00
m_Header.m_nMinorVersion = VPK_MINOR_VERSION; m_Header.m_nDirectorySize = NULL, m_Header.m_nSignatureSize = NULL;
VpkLib: move header check to dir construction and flag error on failure Moved the construction as otherwise we would iterate over potentially bad or incompatible data.
2024-01-14 00:43:15 +01:00
m_bInitFailed = false;
Move some CPackedStore methods to VPKDir_t These methods where only used for and by the VPKDir_t structure. Also renamed variable and method names to reflect new changes regarding the use of KeyValues by creating a structure with all values defined during file lookup rather than doing this during the pack process.
2022-11-23 17:09:48 +01:00
};
CPackedStore refactor * Improve naming convention, and make it more consistent. * Use engine types as much as possible. * Reduced the amount of string copy constructions.
2023-05-29 21:41:17 +02:00
VPKDir_t(const CUtlString& svDirectoryFile);
VPKDir_t(const CUtlString& svDirectoryFile, bool bSanitizeName);
Initial implementation of compressed VPK rebuilding The 'fs_pack_vpk' allows compressing specific files into a VPK (compressed). This implementation has better compression ratio's then currently available VPK tools with compression (at the cost of speed). The implementation is early, a directory iterator with a files for flags is planned soon to fully rebuild VPK's on-demand. A dedicated standalone program is also planned. The output dir and archive formats should match 1:1 with the original game
2022-05-30 02:56:15 +02:00
CPackedStore refactor * Improve naming convention, and make it more consistent. * Use engine types as much as possible. * Reduced the amount of string copy constructions.
2023-05-29 21:41:17 +02:00
void Init(const CUtlString& svPath);
VpkLib: move header check to dir construction and flag error on failure Moved the construction as otherwise we would iterate over potentially bad or incompatible data.
2024-01-14 00:43:15 +01:00
inline bool Failed() const { return m_bInitFailed; }
Move some CPackedStore methods to VPKDir_t These methods where only used for and by the VPKDir_t structure. Also renamed variable and method names to reflect new changes regarding the use of KeyValues by creating a structure with all values defined during file lookup rather than doing this during the pack process.
2022-11-23 17:09:48 +01:00
CPackedStore refactor * Improve naming convention, and make it more consistent. * Use engine types as much as possible. * Reduced the amount of string copy constructions.
2023-05-29 21:41:17 +02:00
CUtlString StripLocalePrefix(const CUtlString& svDirectoryFile) const;
VPK utility code improvements - Only process referenced pack files, previously, if an entry in the VPK file was build into a pack of index 128, the code would generate a list of pack files from 0 to 128. - Added documentation to VPK structure.
2023-07-06 00:49:17 +02:00
CUtlString GetPackFileNameForIndex(uint16_t iPackFileIndex) const;
Move some CPackedStore methods to VPKDir_t These methods where only used for and by the VPKDir_t structure. Also renamed variable and method names to reflect new changes regarding the use of KeyValues by creating a structure with all values defined during file lookup rather than doing this during the pack process.
2022-11-23 17:09:48 +01:00
void WriteHeader(FileHandle_t hDirectoryFile) const;
void WriteTreeSize(FileHandle_t hDirectoryFile) const;
Split 'VPKDir_t::Build' into multiple methods Each process of writing out a VPK directory is now within its own method.
2022-11-23 12:27:57 +01:00
CPackedStore: light code cleanup Light code cleanup: use references instead of pointers as much as possible, additional use of const qualifiers, and usage of provided iteration macros for CUtlVector.
2023-07-04 22:08:44 +02:00
void BuildDirectoryFile(const CUtlString& svDirectoryFile, const CUtlVector<VPKEntryBlock_t>& entryBlocks);
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
};
VPK utility code improvements - Only process referenced pack files, previously, if an entry in the VPK file was build into a pack of index 128, the code would generate a list of pack files from 0 to 128. - Added documentation to VPK structure.
2023-07-06 00:49:17 +02:00
//-----------------------------------------------------------------------------
// Contains the VPK directory name, and the pack file name. Used for building
// the VPK file.
// !TODO[ AMOS ]: Remove this when patching is implemented!
//-----------------------------------------------------------------------------
struct VPKPair_t
{
CUtlString m_PackName;
CUtlString m_DirName;
VPKPair_t(const char* svLocale, const char* svTarget, const char* svLevel, int nPatch);
};
//-----------------------------------------------------------------------------
// VPK utility class.
//-----------------------------------------------------------------------------
VpkLib: rename CPackedStore to CPackedStoreBuilder Avoid confusion with the game's CPackedStore class.
2024-01-12 11:02:54 +01:00
class CPackedStoreBuilder
Code base refactor + major performance and readability improvement. Read description for details. * Codebase restructured to SourceSDK codebase style and .cpp/.h assertion paths in the game executable. * Document most functions with valve style 'Purpose' blocks. * Rename variables to match the rest of the codebase and Valve's naming convention. * Dedicated DLL and the SDKLauncher now share the same codebase as the DevSDK. * Obtain globals or pointers directly instead of waiting for runtime initialized data. * Dynamically search for all functions and globals (this doesn't count for dedicated yet!). * Initialize most in-SDK variables. * Move certain prints and other utilities under ConVars to reduce verbosity and increase performance. * Print all pattern scan results through a virtual function to make it easier to add and debug new patterns in the future. * Type global var pointers appropriately if class or type is known and implemented. * Forward declare 'CClient' class to avoid having 2 'g_pClient' copies. * Add IDA's pseudo definitions for easier prototyping with decompiled assembly code. * RPAK decompress Command callback implementation. * Load decompressed RPaks from 'paks\Win32\' overriding the ones in 'paks\Win64\' (the decompress callback will automatically fix the header and write it to 'paks\Win32\'). * VPK decompress Command callback implementation. * Move CRC32 ands Adler32 to implementation files. * Server will print out more details about the connecting client. * Upgrade ImGui lib to v1.86. * Don't compile id3dx.h for dedicated. * Don't compile id3dx.cpp for dedicated * Implement DevMsg print function allowing to print information to the in-game VGUI/RUI console overlay, ImGui console overlay and the external windows console * Fixed bug where the Error function would not properly terminate the process when an error is called. This caused access violations for critical/non-recoverable errors. * Fixed bug where the game would crash if the console or server browser was enabled while the game was still starting up. * Several bug fixes for the dedicated server (warning: dedicated is still considered work-in-progress!).
2021-12-25 22:36:38 +01:00
{
public:
ReVPK: add option for num helper threads and compress levels Allow the user to set the # amount of helper threads and the compress level of the VPK's.
2024-01-13 14:02:46 +01:00
void InitLzEncoder(const lzham_int32 maxHelperThreads = -1, const char* compressionLevel = "default");
void InitLzDecoder(void);
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
CPackedStore: Fix bug and improve code readability Moved deduplication code to a separate function. Unnested packing and unpacking code a bit more.
2023-04-03 22:46:10 +02:00
bool Deduplicate(const uint8_t* pEntryBuffer, VPKChunkDescriptor_t& descriptor, const size_t chunkIndex);
VokLib: rename pack/unpack methods More correct names.
2024-01-13 14:09:37 +01:00
void PackStore(const VPKPair_t& vpkPair, const char* workspaceName, const char* buildPath);
void UnpackStore(const VPKDir_t& vpkDir, const char* workspaceName = "");
Update terminology entirely for packedstore.h/.cpp VPK tooling also works on the R1 game (Titanfall 1)
2022-06-06 14:54:22 +02:00
private:
ReVPK: add option for num helper threads and compress levels Allow the user to set the # amount of helper threads and the compress level of the VPK's.
2024-01-13 14:02:46 +01:00
lzham_compress_params m_Encoder; // LZham compression parameters.
lzham_decompress_params m_Decoder; // LZham decompression parameters.
CPackedStore refactor * Improve naming convention, and make it more consistent. * Use engine types as much as possible. * Reduced the amount of string copy constructions.
2023-05-29 21:41:17 +02:00
std::unordered_map<string, const VPKChunkDescriptor_t&> m_ChunkHashMap;
Code base refactor + major performance and readability improvement. Read description for details. * Codebase restructured to SourceSDK codebase style and .cpp/.h assertion paths in the game executable. * Document most functions with valve style 'Purpose' blocks. * Rename variables to match the rest of the codebase and Valve's naming convention. * Dedicated DLL and the SDKLauncher now share the same codebase as the DevSDK. * Obtain globals or pointers directly instead of waiting for runtime initialized data. * Dynamically search for all functions and globals (this doesn't count for dedicated yet!). * Initialize most in-SDK variables. * Move certain prints and other utilities under ConVars to reduce verbosity and increase performance. * Print all pattern scan results through a virtual function to make it easier to add and debug new patterns in the future. * Type global var pointers appropriately if class or type is known and implemented. * Forward declare 'CClient' class to avoid having 2 'g_pClient' copies. * Add IDA's pseudo definitions for easier prototyping with decompiled assembly code. * RPAK decompress Command callback implementation. * Load decompressed RPaks from 'paks\Win32\' overriding the ones in 'paks\Win64\' (the decompress callback will automatically fix the header and write it to 'paks\Win32\'). * VPK decompress Command callback implementation. * Move CRC32 ands Adler32 to implementation files. * Server will print out more details about the connecting client. * Upgrade ImGui lib to v1.86. * Don't compile id3dx.h for dedicated. * Don't compile id3dx.cpp for dedicated * Implement DevMsg print function allowing to print information to the in-game VGUI/RUI console overlay, ImGui console overlay and the external windows console * Fixed bug where the Error function would not properly terminate the process when an error is called. This caused access violations for critical/non-recoverable errors. * Fixed bug where the game would crash if the console or server browser was enabled while the game was still starting up. * Several bug fixes for the dedicated server (warning: dedicated is still considered work-in-progress!).
2021-12-25 22:36:38 +01:00
};
ReVPK: bug fixes & improvements * Make sure the workspace path actually exists before attempting to pack it. * Make sure the VPK directory tree file was parsed correctly before unpacking store. * Log debug output for each pack operation to a file. * Fix bug in s_DirFileRegex regex pattern, which would include a trailing '_' in the context part of the directory tree file name. * Fix bug in 'GetLevelName()' and 'GetDirNameParts()' causing it to parse the path as well, prune the path before running the regex. * Renamed 'GetLevelName()' to 'PackedStore_GetDirLevelName()'. * Renamed 'GetDirNameParts()' to 'PackedStore_GetDirNameParts()'. * Write a front-end enable file when building client VPK's.
2024-01-14 01:39:11 +01:00
CUtlString PackedStore_GetDirLevelName(const CUtlString& dirFileName);
CUtlString PackedStore_GetDirNameParts(const CUtlString& dirFileName, const int nCaptureGroup);
Code base refactor + major performance and readability improvement. Read description for details. * Codebase restructured to SourceSDK codebase style and .cpp/.h assertion paths in the game executable. * Document most functions with valve style 'Purpose' blocks. * Rename variables to match the rest of the codebase and Valve's naming convention. * Dedicated DLL and the SDKLauncher now share the same codebase as the DevSDK. * Obtain globals or pointers directly instead of waiting for runtime initialized data. * Dynamically search for all functions and globals (this doesn't count for dedicated yet!). * Initialize most in-SDK variables. * Move certain prints and other utilities under ConVars to reduce verbosity and increase performance. * Print all pattern scan results through a virtual function to make it easier to add and debug new patterns in the future. * Type global var pointers appropriately if class or type is known and implemented. * Forward declare 'CClient' class to avoid having 2 'g_pClient' copies. * Add IDA's pseudo definitions for easier prototyping with decompiled assembly code. * RPAK decompress Command callback implementation. * Load decompressed RPaks from 'paks\Win32\' overriding the ones in 'paks\Win64\' (the decompress callback will automatically fix the header and write it to 'paks\Win32\'). * VPK decompress Command callback implementation. * Move CRC32 ands Adler32 to implementation files. * Server will print out more details about the connecting client. * Upgrade ImGui lib to v1.86. * Don't compile id3dx.h for dedicated. * Don't compile id3dx.cpp for dedicated * Implement DevMsg print function allowing to print information to the in-game VGUI/RUI console overlay, ImGui console overlay and the external windows console * Fixed bug where the Error function would not properly terminate the process when an error is called. This caused access violations for critical/non-recoverable errors. * Fixed bug where the game would crash if the console or server browser was enabled while the game was still starting up. * Several bug fixes for the dedicated server (warning: dedicated is still considered work-in-progress!).
2021-12-25 22:36:38 +01:00
///////////////////////////////////////////////////////////////////////////////
CPackedStore: light optimizations * Don't call copy constructor on entry blocks when building the directory file. * Don't call copy constructor for string when we don't want to sanitize the directory file name.
2022-10-12 23:39:26 +02:00
VPK utility code improvements - Only process referenced pack files, previously, if an entry in the VPK file was build into a pack of index 128, the code would generate a list of pack files from 0 to 128. - Added documentation to VPK structure.
2023-07-06 00:49:17 +02:00
#endif // PACKEDSTORE_H
Reference in New Issue Copy Permalink