R5Reloaded/r5sdk
1
0
Fork 0
mirror of https://github.com/Mauler125/r5sdk.git synced 2025-02-09 19:15:03 +01:00
Code Issues Packages Projects Releases Wiki Activity
r5sdk/r5dev/vpklib/packedstore.h

200 lines
8.2 KiB
C
Raw Normal View History

CPackedStore: light optimizations * Don't call copy constructor on entry blocks when building the directory file. * Don't call copy constructor for string when we don't want to sanitize the directory file name.
2022-10-12 23:39:26 +02:00
#ifndef PACKEDSTORE_H
#define PACKEDSTORE_H
/*******************************************************************
* ██████╗ ██╗ ██╗ ██╗██████╗ ██╗ ██╗ ██╗ ██╗██████╗ *
* ██╔══██╗███║ ██║ ██║██╔══██╗██║ ██╔╝ ██║ ██║██╔══██╗ *
* ██████╔╝╚██║ ██║ ██║██████╔╝█████╔╝ ██║ ██║██████╔╝ *
* ██╔══██╗ ██║ ╚██╗ ██╔╝██╔═══╝ ██╔═██╗ ██║ ██║██╔══██╗ *
* ██║ ██║ ██║ ╚████╔╝ ██║ ██║ ██╗ ███████╗██║██████╔╝ *
* ╚═╝ ╚═╝ ╚═╝ ╚═══╝ ╚═╝ ╚═╝ ╚═╝ ╚══════╝╚═╝╚═════╝ *
*******************************************************************/
CPackedStore refactor The CPackedStore class now heavily relies on the engine's FileSystem and KeyValues API. Since its part of the engine, it would make more sense to utilize the engine's features instead. It also allows for easier modifications. * The logic behind the build process hasn't changed.
2022-11-23 12:18:33 +01:00
#include "public/ifilesystem.h"
CPackedStore refactor * Improve naming convention, and make it more consistent. * Use engine types as much as possible. * Reduced the amount of string copy constructions.
2023-05-29 21:41:17 +02:00
#include "public/tier1/strtools.h"
#include "public/tier1/utlvector.h"
#include "public/tier1/utlstring.h"
Code base refactor + major performance and readability improvement. Read description for details. * Codebase restructured to SourceSDK codebase style and .cpp/.h assertion paths in the game executable. * Document most functions with valve style 'Purpose' blocks. * Rename variables to match the rest of the codebase and Valve's naming convention. * Dedicated DLL and the SDKLauncher now share the same codebase as the DevSDK. * Obtain globals or pointers directly instead of waiting for runtime initialized data. * Dynamically search for all functions and globals (this doesn't count for dedicated yet!). * Initialize most in-SDK variables. * Move certain prints and other utilities under ConVars to reduce verbosity and increase performance. * Print all pattern scan results through a virtual function to make it easier to add and debug new patterns in the future. * Type global var pointers appropriately if class or type is known and implemented. * Forward declare 'CClient' class to avoid having 2 'g_pClient' copies. * Add IDA's pseudo definitions for easier prototyping with decompiled assembly code. * RPAK decompress Command callback implementation. * Load decompressed RPaks from 'paks\Win32\' overriding the ones in 'paks\Win64\' (the decompress callback will automatically fix the header and write it to 'paks\Win32\'). * VPK decompress Command callback implementation. * Move CRC32 ands Adler32 to implementation files. * Server will print out more details about the connecting client. * Upgrade ImGui lib to v1.86. * Don't compile id3dx.h for dedicated. * Don't compile id3dx.cpp for dedicated * Implement DevMsg print function allowing to print information to the in-game VGUI/RUI console overlay, ImGui console overlay and the external windows console * Fixed bug where the Error function would not properly terminate the process when an error is called. This caused access violations for critical/non-recoverable errors. * Fixed bug where the game would crash if the console or server browser was enabled while the game was still starting up. * Several bug fixes for the dedicated server (warning: dedicated is still considered work-in-progress!).
2021-12-25 22:36:38 +01:00
#include "thirdparty/lzham/include/lzham.h"
Additional cleanup Cast to proper types. Set the archive offset only instead of the entire block. The entry could still have its own flags. The only thing needed for saving data is sharing the same offset and size.
2022-06-05 12:28:49 +02:00
constexpr unsigned int VPK_HEADER_MARKER = 0x55AA1234;
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
constexpr unsigned int VPK_MAJOR_VERSION = 2;
constexpr unsigned int VPK_MINOR_VERSION = 3;
Additional cleanup Cast to proper types. Set the archive offset only instead of the entire block. The entry could still have its own flags. The only thing needed for saving data is sharing the same offset and size.
2022-06-05 12:28:49 +02:00
constexpr unsigned int VPK_DICT_SIZE = 20;
constexpr int ENTRY_MAX_LEN = 1024 * 1024;
Add convar for adjusting LZHAM compressor max helper threads
2022-11-17 20:37:12 +01:00
constexpr int PACKFILEPATCH_MAX = 512;
CPackedStore refactor The CPackedStore class now heavily relies on the engine's FileSystem and KeyValues API. Since its part of the engine, it would make more sense to utilize the engine's features instead. It also allows for easier modifications. * The logic behind the build process hasn't changed.
2022-11-23 12:18:33 +01:00
constexpr int PACKFILEINDEX_SEP = 0x0;
Improve consistency throughout packedstore class No modifications to the existing logic have been performed. Some parameter names have been renamed to improve code readability and maintainability.
2022-11-16 00:54:51 +01:00
constexpr int PACKFILEINDEX_END = 0xffff;
CPackedStore refactor * Improve naming convention, and make it more consistent. * Use engine types as much as possible. * Reduced the amount of string copy constructions.
2023-05-29 21:41:17 +02:00
constexpr const char VPK_IGNORE_FILE[] = ".vpkignore";
Code base refactor + major performance and readability improvement. Read description for details. * Codebase restructured to SourceSDK codebase style and .cpp/.h assertion paths in the game executable. * Document most functions with valve style 'Purpose' blocks. * Rename variables to match the rest of the codebase and Valve's naming convention. * Dedicated DLL and the SDKLauncher now share the same codebase as the DevSDK. * Obtain globals or pointers directly instead of waiting for runtime initialized data. * Dynamically search for all functions and globals (this doesn't count for dedicated yet!). * Initialize most in-SDK variables. * Move certain prints and other utilities under ConVars to reduce verbosity and increase performance. * Print all pattern scan results through a virtual function to make it easier to add and debug new patterns in the future. * Type global var pointers appropriately if class or type is known and implemented. * Forward declare 'CClient' class to avoid having 2 'g_pClient' copies. * Add IDA's pseudo definitions for easier prototyping with decompiled assembly code. * RPAK decompress Command callback implementation. * Load decompressed RPaks from 'paks\Win32\' overriding the ones in 'paks\Win64\' (the decompress callback will automatically fix the header and write it to 'paks\Win32\'). * VPK decompress Command callback implementation. * Move CRC32 ands Adler32 to implementation files. * Server will print out more details about the connecting client. * Upgrade ImGui lib to v1.86. * Don't compile id3dx.h for dedicated. * Don't compile id3dx.cpp for dedicated * Implement DevMsg print function allowing to print information to the in-game VGUI/RUI console overlay, ImGui console overlay and the external windows console * Fixed bug where the Error function would not properly terminate the process when an error is called. This caused access violations for critical/non-recoverable errors. * Fixed bug where the game would crash if the console or server browser was enabled while the game was still starting up. * Several bug fixes for the dedicated server (warning: dedicated is still considered work-in-progress!).
2021-12-25 22:36:38 +01:00
CPackedStore refactor * Improve naming convention, and make it more consistent. * Use engine types as much as possible. * Reduced the amount of string copy constructions.
2023-05-29 21:41:17 +02:00
static const char* const DIR_TARGET[]
Use static buffer for file chunking and compressing during VPK packing Avoid using new every time. Chunk can be at most 1024 * 1024.
2022-10-02 12:17:03 +02:00
{
"server",
"client"
};
CPackedStore refactor * Improve naming convention, and make it more consistent. * Use engine types as much as possible. * Reduced the amount of string copy constructions.
2023-05-29 21:41:17 +02:00
static const char* const DIR_LOCALE[]
Use static buffer for file chunking and compressing during VPK packing Avoid using new every time. Chunk can be at most 1024 * 1024.
2022-10-02 12:17:03 +02:00
{
"english",
"french",
"german",
"italian",
"japanese",
"korean",
"polish",
"portuguese",
"russian",
"spanish",
"tchinese"
};
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
CPackedStore refactor The CPackedStore class now heavily relies on the engine's FileSystem and KeyValues API. Since its part of the engine, it would make more sense to utilize the engine's features instead. It also allows for easier modifications. * The logic behind the build process hasn't changed.
2022-11-23 12:18:33 +01:00
struct VPKKeyValues_t
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
{
CPackedStore refactor The CPackedStore class now heavily relies on the engine's FileSystem and KeyValues API. Since its part of the engine, it would make more sense to utilize the engine's features instead. It also allows for easier modifications. * The logic behind the build process hasn't changed.
2022-11-23 12:18:33 +01:00
static constexpr uint16_t TEXTURE_FLAGS_DEFAULT = static_cast<uint16_t>(EPackedTextureFlags::TEXTURE_DEFAULT);
static constexpr uint32_t LOAD_FLAGS_DEFAULT = static_cast<uint32_t>(EPackedLoadFlags::LOAD_VISIBLE) | static_cast<uint32_t>(EPackedLoadFlags::LOAD_CACHE);
CPackedStore refactor * Improve naming convention, and make it more consistent. * Use engine types as much as possible. * Reduced the amount of string copy constructions.
2023-05-29 21:41:17 +02:00
CUtlString m_EntryPath;
CPackedStore refactor The CPackedStore class now heavily relies on the engine's FileSystem and KeyValues API. Since its part of the engine, it would make more sense to utilize the engine's features instead. It also allows for easier modifications. * The logic behind the build process hasn't changed.
2022-11-23 12:18:33 +01:00
uint16_t m_iPreloadSize;
uint32_t m_nLoadFlags;
uint16_t m_nTextureFlags;
bool m_bUseCompression;
CPackedStore: Rename 'useDataSharing' to 'deDuplicate' More descriptive in what it does (deduplication of existing copies).
2023-04-03 20:08:36 +02:00
bool m_bDeduplicate;
CPackedStore refactor The CPackedStore class now heavily relies on the engine's FileSystem and KeyValues API. Since its part of the engine, it would make more sense to utilize the engine's features instead. It also allows for easier modifications. * The logic behind the build process hasn't changed.
2022-11-23 12:18:33 +01:00
CPackedStore refactor * Improve naming convention, and make it more consistent. * Use engine types as much as possible. * Reduced the amount of string copy constructions.
2023-05-29 21:41:17 +02:00
VPKKeyValues_t(const CUtlString& svEntryPath = "", uint16_t iPreloadSize = NULL, uint32_t nLoadFlags = LOAD_FLAGS_DEFAULT,
CPackedStore: Rename 'useDataSharing' to 'deDuplicate' More descriptive in what it does (deduplication of existing copies).
2023-04-03 20:08:36 +02:00
uint16_t nTextureFlags = TEXTURE_FLAGS_DEFAULT, bool bUseCompression = true, bool bDeduplicate = true);
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
};
Update terminology entirely for packedstore.h/.cpp VPK tooling also works on the R1 game (Titanfall 1)
2022-06-06 14:54:22 +02:00
struct VPKChunkDescriptor_t
Code base refactor + major performance and readability improvement. Read description for details. * Codebase restructured to SourceSDK codebase style and .cpp/.h assertion paths in the game executable. * Document most functions with valve style 'Purpose' blocks. * Rename variables to match the rest of the codebase and Valve's naming convention. * Dedicated DLL and the SDKLauncher now share the same codebase as the DevSDK. * Obtain globals or pointers directly instead of waiting for runtime initialized data. * Dynamically search for all functions and globals (this doesn't count for dedicated yet!). * Initialize most in-SDK variables. * Move certain prints and other utilities under ConVars to reduce verbosity and increase performance. * Print all pattern scan results through a virtual function to make it easier to add and debug new patterns in the future. * Type global var pointers appropriately if class or type is known and implemented. * Forward declare 'CClient' class to avoid having 2 'g_pClient' copies. * Add IDA's pseudo definitions for easier prototyping with decompiled assembly code. * RPAK decompress Command callback implementation. * Load decompressed RPaks from 'paks\Win32\' overriding the ones in 'paks\Win64\' (the decompress callback will automatically fix the header and write it to 'paks\Win32\'). * VPK decompress Command callback implementation. * Move CRC32 ands Adler32 to implementation files. * Server will print out more details about the connecting client. * Upgrade ImGui lib to v1.86. * Don't compile id3dx.h for dedicated. * Don't compile id3dx.cpp for dedicated * Implement DevMsg print function allowing to print information to the in-game VGUI/RUI console overlay, ImGui console overlay and the external windows console * Fixed bug where the Error function would not properly terminate the process when an error is called. This caused access violations for critical/non-recoverable errors. * Fixed bug where the game would crash if the console or server browser was enabled while the game was still starting up. * Several bug fixes for the dedicated server (warning: dedicated is still considered work-in-progress!).
2021-12-25 22:36:38 +01:00
{
CPackedStore: light optimizations * Don't call copy constructor on entry blocks when building the directory file. * Don't call copy constructor for string when we don't want to sanitize the directory file name.
2022-10-12 23:39:26 +02:00
uint32_t m_nLoadFlags; // Load flags.
uint16_t m_nTextureFlags; // Texture flags (only used if the entry is a vtf).
Improve consistency throughout packedstore class No modifications to the existing logic have been performed. Some parameter names have been renamed to improve code readability and maintainability.
2022-11-16 00:54:51 +01:00
uint64_t m_nPackFileOffset; // Offset in pack file.
CPackedStore: light optimizations * Don't call copy constructor on entry blocks when building the directory file. * Don't call copy constructor for string when we don't want to sanitize the directory file name.
2022-10-12 23:39:26 +02:00
uint64_t m_nCompressedSize; // Compressed size of chunk.
uint64_t m_nUncompressedSize; // Uncompressed size of chunk.
Code base refactor + major performance and readability improvement. Read description for details. * Codebase restructured to SourceSDK codebase style and .cpp/.h assertion paths in the game executable. * Document most functions with valve style 'Purpose' blocks. * Rename variables to match the rest of the codebase and Valve's naming convention. * Dedicated DLL and the SDKLauncher now share the same codebase as the DevSDK. * Obtain globals or pointers directly instead of waiting for runtime initialized data. * Dynamically search for all functions and globals (this doesn't count for dedicated yet!). * Initialize most in-SDK variables. * Move certain prints and other utilities under ConVars to reduce verbosity and increase performance. * Print all pattern scan results through a virtual function to make it easier to add and debug new patterns in the future. * Type global var pointers appropriately if class or type is known and implemented. * Forward declare 'CClient' class to avoid having 2 'g_pClient' copies. * Add IDA's pseudo definitions for easier prototyping with decompiled assembly code. * RPAK decompress Command callback implementation. * Load decompressed RPaks from 'paks\Win32\' overriding the ones in 'paks\Win64\' (the decompress callback will automatically fix the header and write it to 'paks\Win32\'). * VPK decompress Command callback implementation. * Move CRC32 ands Adler32 to implementation files. * Server will print out more details about the connecting client. * Upgrade ImGui lib to v1.86. * Don't compile id3dx.h for dedicated. * Don't compile id3dx.cpp for dedicated * Implement DevMsg print function allowing to print information to the in-game VGUI/RUI console overlay, ImGui console overlay and the external windows console * Fixed bug where the Error function would not properly terminate the process when an error is called. This caused access violations for critical/non-recoverable errors. * Fixed bug where the game would crash if the console or server browser was enabled while the game was still starting up. * Several bug fixes for the dedicated server (warning: dedicated is still considered work-in-progress!).
2021-12-25 22:36:38 +01:00
CPackedStore refactor * Improve naming convention, and make it more consistent. * Use engine types as much as possible. * Reduced the amount of string copy constructions.
2023-05-29 21:41:17 +02:00
VPKChunkDescriptor_t()
: m_nLoadFlags(0)
, m_nTextureFlags(0)
, m_nPackFileOffset(0)
, m_nCompressedSize(0)
, m_nUncompressedSize(0)
{
}
Move some CPackedStore methods to VPKDir_t These methods where only used for and by the VPKDir_t structure. Also renamed variable and method names to reflect new changes regarding the use of KeyValues by creating a structure with all values defined during file lookup rather than doing this during the pack process.
2022-11-23 17:09:48 +01:00
VPKChunkDescriptor_t(FileHandle_t hDirectoryFile);
VPKChunkDescriptor_t(uint32_t nLoadFlags, uint16_t nTextureFlags, uint64_t nPackFileOffset, uint64_t nCompressedSize, uint64_t nUncompressedSize);
Code base refactor + major performance and readability improvement. Read description for details. * Codebase restructured to SourceSDK codebase style and .cpp/.h assertion paths in the game executable. * Document most functions with valve style 'Purpose' blocks. * Rename variables to match the rest of the codebase and Valve's naming convention. * Dedicated DLL and the SDKLauncher now share the same codebase as the DevSDK. * Obtain globals or pointers directly instead of waiting for runtime initialized data. * Dynamically search for all functions and globals (this doesn't count for dedicated yet!). * Initialize most in-SDK variables. * Move certain prints and other utilities under ConVars to reduce verbosity and increase performance. * Print all pattern scan results through a virtual function to make it easier to add and debug new patterns in the future. * Type global var pointers appropriately if class or type is known and implemented. * Forward declare 'CClient' class to avoid having 2 'g_pClient' copies. * Add IDA's pseudo definitions for easier prototyping with decompiled assembly code. * RPAK decompress Command callback implementation. * Load decompressed RPaks from 'paks\Win32\' overriding the ones in 'paks\Win64\' (the decompress callback will automatically fix the header and write it to 'paks\Win32\'). * VPK decompress Command callback implementation. * Move CRC32 ands Adler32 to implementation files. * Server will print out more details about the connecting client. * Upgrade ImGui lib to v1.86. * Don't compile id3dx.h for dedicated. * Don't compile id3dx.cpp for dedicated * Implement DevMsg print function allowing to print information to the in-game VGUI/RUI console overlay, ImGui console overlay and the external windows console * Fixed bug where the Error function would not properly terminate the process when an error is called. This caused access violations for critical/non-recoverable errors. * Fixed bug where the game would crash if the console or server browser was enabled while the game was still starting up. * Several bug fixes for the dedicated server (warning: dedicated is still considered work-in-progress!).
2021-12-25 22:36:38 +01:00
};
See description * Added 2 new FileSystem pointers with new features in their classes. * Register all factory instances created by the GameDLL in the SDK. * Added new command 'fs_mount_vpk' to mount a specified VPK file. * Renamed 'fs_decompress_pak' to 'fs_unpack_vpk'. * Some renaming of Factory and VPK types. * Some light optimizations/cleanup.
2022-04-09 00:59:42 +02:00
struct VPKEntryBlock_t
Code base refactor + major performance and readability improvement. Read description for details. * Codebase restructured to SourceSDK codebase style and .cpp/.h assertion paths in the game executable. * Document most functions with valve style 'Purpose' blocks. * Rename variables to match the rest of the codebase and Valve's naming convention. * Dedicated DLL and the SDKLauncher now share the same codebase as the DevSDK. * Obtain globals or pointers directly instead of waiting for runtime initialized data. * Dynamically search for all functions and globals (this doesn't count for dedicated yet!). * Initialize most in-SDK variables. * Move certain prints and other utilities under ConVars to reduce verbosity and increase performance. * Print all pattern scan results through a virtual function to make it easier to add and debug new patterns in the future. * Type global var pointers appropriately if class or type is known and implemented. * Forward declare 'CClient' class to avoid having 2 'g_pClient' copies. * Add IDA's pseudo definitions for easier prototyping with decompiled assembly code. * RPAK decompress Command callback implementation. * Load decompressed RPaks from 'paks\Win32\' overriding the ones in 'paks\Win64\' (the decompress callback will automatically fix the header and write it to 'paks\Win32\'). * VPK decompress Command callback implementation. * Move CRC32 ands Adler32 to implementation files. * Server will print out more details about the connecting client. * Upgrade ImGui lib to v1.86. * Don't compile id3dx.h for dedicated. * Don't compile id3dx.cpp for dedicated * Implement DevMsg print function allowing to print information to the in-game VGUI/RUI console overlay, ImGui console overlay and the external windows console * Fixed bug where the Error function would not properly terminate the process when an error is called. This caused access violations for critical/non-recoverable errors. * Fixed bug where the game would crash if the console or server browser was enabled while the game was still starting up. * Several bug fixes for the dedicated server (warning: dedicated is still considered work-in-progress!).
2021-12-25 22:36:38 +01:00
{
CPackedStore refactor * Improve naming convention, and make it more consistent. * Use engine types as much as possible. * Reduced the amount of string copy constructions.
2023-05-29 21:41:17 +02:00
uint32_t m_nFileCRC; // Crc32 for the uncompressed entry.
uint16_t m_iPreloadSize; // Preload bytes.
uint16_t m_iPackFileIndex; // Index of the pack file that contains this entry.
CUtlVector<VPKChunkDescriptor_t> m_Fragments; // Vector of all the chunks of a given entry (chunks have a size limit of 1 MiB, anything over this limit is fragmented into smaller chunks).
CUtlString m_EntryPath; // Path to entry within vpk.
Update terminology entirely for packedstore.h/.cpp VPK tooling also works on the R1 game (Titanfall 1)
2022-06-06 14:54:22 +02:00
CPackedStore refactor * Improve naming convention, and make it more consistent. * Use engine types as much as possible. * Reduced the amount of string copy constructions.
2023-05-29 21:41:17 +02:00
VPKEntryBlock_t(FileHandle_t pFile, const char* svEntryPath);
Move some CPackedStore methods to VPKDir_t These methods where only used for and by the VPKDir_t structure. Also renamed variable and method names to reflect new changes regarding the use of KeyValues by creating a structure with all values defined during file lookup rather than doing this during the pack process.
2022-11-23 17:09:48 +01:00
VPKEntryBlock_t(const uint8_t* pData, size_t nLen, int64_t nOffset, uint16_t iPreloadSize,
CPackedStore refactor * Improve naming convention, and make it more consistent. * Use engine types as much as possible. * Reduced the amount of string copy constructions.
2023-05-29 21:41:17 +02:00
uint16_t iPackFileIndex, uint32_t nLoadFlags, uint16_t nTextureFlags, const char* svEntryPath);
VPKEntryBlock_t(const VPKEntryBlock_t& other)
: m_nFileCRC(other.m_nFileCRC)
, m_iPreloadSize(other.m_iPreloadSize)
, m_iPackFileIndex(other.m_iPackFileIndex)
, m_EntryPath(other.m_EntryPath)
{
// Has to be explicitly copied!
m_Fragments = other.m_Fragments;
}
Code base refactor + major performance and readability improvement. Read description for details. * Codebase restructured to SourceSDK codebase style and .cpp/.h assertion paths in the game executable. * Document most functions with valve style 'Purpose' blocks. * Rename variables to match the rest of the codebase and Valve's naming convention. * Dedicated DLL and the SDKLauncher now share the same codebase as the DevSDK. * Obtain globals or pointers directly instead of waiting for runtime initialized data. * Dynamically search for all functions and globals (this doesn't count for dedicated yet!). * Initialize most in-SDK variables. * Move certain prints and other utilities under ConVars to reduce verbosity and increase performance. * Print all pattern scan results through a virtual function to make it easier to add and debug new patterns in the future. * Type global var pointers appropriately if class or type is known and implemented. * Forward declare 'CClient' class to avoid having 2 'g_pClient' copies. * Add IDA's pseudo definitions for easier prototyping with decompiled assembly code. * RPAK decompress Command callback implementation. * Load decompressed RPaks from 'paks\Win32\' overriding the ones in 'paks\Win64\' (the decompress callback will automatically fix the header and write it to 'paks\Win32\'). * VPK decompress Command callback implementation. * Move CRC32 ands Adler32 to implementation files. * Server will print out more details about the connecting client. * Upgrade ImGui lib to v1.86. * Don't compile id3dx.h for dedicated. * Don't compile id3dx.cpp for dedicated * Implement DevMsg print function allowing to print information to the in-game VGUI/RUI console overlay, ImGui console overlay and the external windows console * Fixed bug where the Error function would not properly terminate the process when an error is called. This caused access violations for critical/non-recoverable errors. * Fixed bug where the game would crash if the console or server browser was enabled while the game was still starting up. * Several bug fixes for the dedicated server (warning: dedicated is still considered work-in-progress!).
2021-12-25 22:36:38 +01:00
};
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
struct VPKDirHeader_t
Code base refactor + major performance and readability improvement. Read description for details. * Codebase restructured to SourceSDK codebase style and .cpp/.h assertion paths in the game executable. * Document most functions with valve style 'Purpose' blocks. * Rename variables to match the rest of the codebase and Valve's naming convention. * Dedicated DLL and the SDKLauncher now share the same codebase as the DevSDK. * Obtain globals or pointers directly instead of waiting for runtime initialized data. * Dynamically search for all functions and globals (this doesn't count for dedicated yet!). * Initialize most in-SDK variables. * Move certain prints and other utilities under ConVars to reduce verbosity and increase performance. * Print all pattern scan results through a virtual function to make it easier to add and debug new patterns in the future. * Type global var pointers appropriately if class or type is known and implemented. * Forward declare 'CClient' class to avoid having 2 'g_pClient' copies. * Add IDA's pseudo definitions for easier prototyping with decompiled assembly code. * RPAK decompress Command callback implementation. * Load decompressed RPaks from 'paks\Win32\' overriding the ones in 'paks\Win64\' (the decompress callback will automatically fix the header and write it to 'paks\Win32\'). * VPK decompress Command callback implementation. * Move CRC32 ands Adler32 to implementation files. * Server will print out more details about the connecting client. * Upgrade ImGui lib to v1.86. * Don't compile id3dx.h for dedicated. * Don't compile id3dx.cpp for dedicated * Implement DevMsg print function allowing to print information to the in-game VGUI/RUI console overlay, ImGui console overlay and the external windows console * Fixed bug where the Error function would not properly terminate the process when an error is called. This caused access violations for critical/non-recoverable errors. * Fixed bug where the game would crash if the console or server browser was enabled while the game was still starting up. * Several bug fixes for the dedicated server (warning: dedicated is still considered work-in-progress!).
2021-12-25 22:36:38 +01:00
{
CPackedStore: light optimizations * Don't call copy constructor on entry blocks when building the directory file. * Don't call copy constructor for string when we don't want to sanitize the directory file name.
2022-10-12 23:39:26 +02:00
uint32_t m_nHeaderMarker; // File magic.
uint16_t m_nMajorVersion; // Vpk major version.
uint16_t m_nMinorVersion; // Vpk minor version.
uint32_t m_nDirectorySize; // Directory tree size.
uint32_t m_nSignatureSize; // Directory signature.
Initial implementation of compressed VPK rebuilding The 'fs_pack_vpk' allows compressing specific files into a VPK (compressed). This implementation has better compression ratio's then currently available VPK tools with compression (at the cost of speed). The implementation is early, a directory iterator with a files for flags is planned soon to fully rebuild VPK's on-demand. A dedicated standalone program is also planned. The output dir and archive formats should match 1:1 with the original game
2022-05-30 02:56:15 +02:00
};
Move some CPackedStore methods to VPKDir_t These methods where only used for and by the VPKDir_t structure. Also renamed variable and method names to reflect new changes regarding the use of KeyValues by creating a structure with all values defined during file lookup rather than doing this during the pack process.
2022-11-23 17:09:48 +01:00
struct VPKPair_t
{
CPackedStore refactor * Improve naming convention, and make it more consistent. * Use engine types as much as possible. * Reduced the amount of string copy constructions.
2023-05-29 21:41:17 +02:00
CUtlString m_PackName;
CUtlString m_DirName;
Move some CPackedStore methods to VPKDir_t These methods where only used for and by the VPKDir_t structure. Also renamed variable and method names to reflect new changes regarding the use of KeyValues by creating a structure with all values defined during file lookup rather than doing this during the pack process.
2022-11-23 17:09:48 +01:00
CPackedStore refactor * Improve naming convention, and make it more consistent. * Use engine types as much as possible. * Reduced the amount of string copy constructions.
2023-05-29 21:41:17 +02:00
VPKPair_t(const char* svLocale, const char* svTarget, const char* svLevel, int nPatch);
Move some CPackedStore methods to VPKDir_t These methods where only used for and by the VPKDir_t structure. Also renamed variable and method names to reflect new changes regarding the use of KeyValues by creating a structure with all values defined during file lookup rather than doing this during the pack process.
2022-11-23 17:09:48 +01:00
};
Initial implementation of compressed VPK rebuilding The 'fs_pack_vpk' allows compressing specific files into a VPK (compressed). This implementation has better compression ratio's then currently available VPK tools with compression (at the cost of speed). The implementation is early, a directory iterator with a files for flags is planned soon to fully rebuild VPK's on-demand. A dedicated standalone program is also planned. The output dir and archive formats should match 1:1 with the original game
2022-05-30 02:56:15 +02:00
struct VPKDir_t
{
CPackedStore refactor * Improve naming convention, and make it more consistent. * Use engine types as much as possible. * Reduced the amount of string copy constructions.
2023-05-29 21:41:17 +02:00
VPKDirHeader_t m_Header; // Dir header.
CUtlVector<VPKEntryBlock_t> m_EntryBlocks; // Vector of entry blocks.
uint16_t m_PackFileCount; // Number of pack patches (pack file count-1).
CUtlVector<CUtlString> m_PackFiles; // Vector of pack file names.
CUtlString m_DirFilePath; // Path to vpk_dir file.
Code base refactor + major performance and readability improvement. Read description for details. * Codebase restructured to SourceSDK codebase style and .cpp/.h assertion paths in the game executable. * Document most functions with valve style 'Purpose' blocks. * Rename variables to match the rest of the codebase and Valve's naming convention. * Dedicated DLL and the SDKLauncher now share the same codebase as the DevSDK. * Obtain globals or pointers directly instead of waiting for runtime initialized data. * Dynamically search for all functions and globals (this doesn't count for dedicated yet!). * Initialize most in-SDK variables. * Move certain prints and other utilities under ConVars to reduce verbosity and increase performance. * Print all pattern scan results through a virtual function to make it easier to add and debug new patterns in the future. * Type global var pointers appropriately if class or type is known and implemented. * Forward declare 'CClient' class to avoid having 2 'g_pClient' copies. * Add IDA's pseudo definitions for easier prototyping with decompiled assembly code. * RPAK decompress Command callback implementation. * Load decompressed RPaks from 'paks\Win32\' overriding the ones in 'paks\Win64\' (the decompress callback will automatically fix the header and write it to 'paks\Win32\'). * VPK decompress Command callback implementation. * Move CRC32 ands Adler32 to implementation files. * Server will print out more details about the connecting client. * Upgrade ImGui lib to v1.86. * Don't compile id3dx.h for dedicated. * Don't compile id3dx.cpp for dedicated * Implement DevMsg print function allowing to print information to the in-game VGUI/RUI console overlay, ImGui console overlay and the external windows console * Fixed bug where the Error function would not properly terminate the process when an error is called. This caused access violations for critical/non-recoverable errors. * Fixed bug where the game would crash if the console or server browser was enabled while the game was still starting up. * Several bug fixes for the dedicated server (warning: dedicated is still considered work-in-progress!).
2021-12-25 22:36:38 +01:00
Fix incorrect VPK tree during build This commit fixes the following bugs: - Use of STL map on CUtlString, CUtlString does not have a lesser operator, thus it always fails to check whether or not the string is already present in the map. The tree structure cannot contain duplicates per tree scope. - Use of backward slashes in vpk file tree structure, path separators should always be a forwards slash. The code was originally designed to be forwards slash only, but some paths were adjusted to take backwards slash, as the KeyValues class treats the '/' char as a delimiter for subfields. This has therefore lead into the whole code changing to it. The code has now been changed to the point where only the KeyValues utilities take the backward slash paths. - Trailing slash character on file path, that ended up getting written as-is in the path tree. The VPK tree builder is now part of a separate nested class in 'VPKDir_t'.
2023-07-05 23:04:07 +02:00
class CTreeBuilder
{
public:
typedef std::map<std::string, std::list<VPKEntryBlock_t>> PathContainer_t;
typedef std::map<std::string, PathContainer_t> TypeContainer_t;
void BuildTree(const CUtlVector<VPKEntryBlock_t>& entryBlocks);
uint64_t WriteTree(FileHandle_t hDirectoryFile) const;
private:
TypeContainer_t m_FileTree;
};
Move some CPackedStore methods to VPKDir_t These methods where only used for and by the VPKDir_t structure. Also renamed variable and method names to reflect new changes regarding the use of KeyValues by creating a structure with all values defined during file lookup rather than doing this during the pack process.
2022-11-23 17:09:48 +01:00
VPKDir_t()
{
Fix incorrect VPK tree during build This commit fixes the following bugs: - Use of STL map on CUtlString, CUtlString does not have a lesser operator, thus it always fails to check whether or not the string is already present in the map. The tree structure cannot contain duplicates per tree scope. - Use of backward slashes in vpk file tree structure, path separators should always be a forwards slash. The code was originally designed to be forwards slash only, but some paths were adjusted to take backwards slash, as the KeyValues class treats the '/' char as a delimiter for subfields. This has therefore lead into the whole code changing to it. The code has now been changed to the point where only the KeyValues utilities take the backward slash paths. - Trailing slash character on file path, that ended up getting written as-is in the path tree. The VPK tree builder is now part of a separate nested class in 'VPKDir_t'.
2023-07-05 23:04:07 +02:00
m_Header.m_nHeaderMarker = VPK_HEADER_MARKER; m_Header.m_nMajorVersion = VPK_MAJOR_VERSION;
CPackedStore refactor * Improve naming convention, and make it more consistent. * Use engine types as much as possible. * Reduced the amount of string copy constructions.
2023-05-29 21:41:17 +02:00
m_Header.m_nMinorVersion = VPK_MINOR_VERSION; m_Header.m_nDirectorySize = NULL, m_Header.m_nSignatureSize = NULL;
m_PackFileCount = NULL;
Move some CPackedStore methods to VPKDir_t These methods where only used for and by the VPKDir_t structure. Also renamed variable and method names to reflect new changes regarding the use of KeyValues by creating a structure with all values defined during file lookup rather than doing this during the pack process.
2022-11-23 17:09:48 +01:00
};
CPackedStore refactor * Improve naming convention, and make it more consistent. * Use engine types as much as possible. * Reduced the amount of string copy constructions.
2023-05-29 21:41:17 +02:00
VPKDir_t(const CUtlString& svDirectoryFile);
VPKDir_t(const CUtlString& svDirectoryFile, bool bSanitizeName);
Initial implementation of compressed VPK rebuilding The 'fs_pack_vpk' allows compressing specific files into a VPK (compressed). This implementation has better compression ratio's then currently available VPK tools with compression (at the cost of speed). The implementation is early, a directory iterator with a files for flags is planned soon to fully rebuild VPK's on-demand. A dedicated standalone program is also planned. The output dir and archive formats should match 1:1 with the original game
2022-05-30 02:56:15 +02:00
CPackedStore refactor * Improve naming convention, and make it more consistent. * Use engine types as much as possible. * Reduced the amount of string copy constructions.
2023-05-29 21:41:17 +02:00
void Init(const CUtlString& svPath);
Move some CPackedStore methods to VPKDir_t These methods where only used for and by the VPKDir_t structure. Also renamed variable and method names to reflect new changes regarding the use of KeyValues by creating a structure with all values defined during file lookup rather than doing this during the pack process.
2022-11-23 17:09:48 +01:00
CPackedStore refactor * Improve naming convention, and make it more consistent. * Use engine types as much as possible. * Reduced the amount of string copy constructions.
2023-05-29 21:41:17 +02:00
CUtlString StripLocalePrefix(const CUtlString& svDirectoryFile) const;
CUtlString GetPackFile(const CUtlString& svDirectoryPath, uint16_t iPackFileIndex) const;
Move some CPackedStore methods to VPKDir_t These methods where only used for and by the VPKDir_t structure. Also renamed variable and method names to reflect new changes regarding the use of KeyValues by creating a structure with all values defined during file lookup rather than doing this during the pack process.
2022-11-23 17:09:48 +01:00
void WriteHeader(FileHandle_t hDirectoryFile) const;
void WriteTreeSize(FileHandle_t hDirectoryFile) const;
Split 'VPKDir_t::Build' into multiple methods Each process of writing out a VPK directory is now within its own method.
2022-11-23 12:27:57 +01:00
CPackedStore: light code cleanup Light code cleanup: use references instead of pointers as much as possible, additional use of const qualifiers, and usage of provided iteration macros for CUtlVector.
2023-07-04 22:08:44 +02:00
void BuildDirectoryFile(const CUtlString& svDirectoryFile, const CUtlVector<VPKEntryBlock_t>& entryBlocks);
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
};
Code base refactor + major performance and readability improvement. Read description for details. * Codebase restructured to SourceSDK codebase style and .cpp/.h assertion paths in the game executable. * Document most functions with valve style 'Purpose' blocks. * Rename variables to match the rest of the codebase and Valve's naming convention. * Dedicated DLL and the SDKLauncher now share the same codebase as the DevSDK. * Obtain globals or pointers directly instead of waiting for runtime initialized data. * Dynamically search for all functions and globals (this doesn't count for dedicated yet!). * Initialize most in-SDK variables. * Move certain prints and other utilities under ConVars to reduce verbosity and increase performance. * Print all pattern scan results through a virtual function to make it easier to add and debug new patterns in the future. * Type global var pointers appropriately if class or type is known and implemented. * Forward declare 'CClient' class to avoid having 2 'g_pClient' copies. * Add IDA's pseudo definitions for easier prototyping with decompiled assembly code. * RPAK decompress Command callback implementation. * Load decompressed RPaks from 'paks\Win32\' overriding the ones in 'paks\Win64\' (the decompress callback will automatically fix the header and write it to 'paks\Win32\'). * VPK decompress Command callback implementation. * Move CRC32 ands Adler32 to implementation files. * Server will print out more details about the connecting client. * Upgrade ImGui lib to v1.86. * Don't compile id3dx.h for dedicated. * Don't compile id3dx.cpp for dedicated * Implement DevMsg print function allowing to print information to the in-game VGUI/RUI console overlay, ImGui console overlay and the external windows console * Fixed bug where the Error function would not properly terminate the process when an error is called. This caused access violations for critical/non-recoverable errors. * Fixed bug where the game would crash if the console or server browser was enabled while the game was still starting up. * Several bug fixes for the dedicated server (warning: dedicated is still considered work-in-progress!).
2021-12-25 22:36:38 +01:00
class CPackedStore
{
public:
Create RCON ConCommand
2022-02-06 15:59:46 +01:00
void InitLzCompParams(void);
void InitLzDecompParams(void);
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
Set VPK compression level through ConVar
2022-09-11 00:16:31 +02:00
lzham_compress_level GetCompressionLevel(void) const;
Update terminology entirely for packedstore.h/.cpp VPK tooling also works on the R1 game (Titanfall 1)
2022-06-06 14:54:22 +02:00
CPackedStore refactor * Improve naming convention, and make it more consistent. * Use engine types as much as possible. * Reduced the amount of string copy constructions.
2023-05-29 21:41:17 +02:00
void GetEntryBlocks(CUtlVector<VPKEntryBlock_t>& entryBlocks, FileHandle_t hDirectoryFile) const;
bool GetEntryValues(CUtlVector<VPKKeyValues_t>& entryValues, const CUtlString& workspacePath, const CUtlString& dirFileName) const;
Update terminology entirely for packedstore.h/.cpp VPK tooling also works on the R1 game (Titanfall 1)
2022-06-06 14:54:22 +02:00
CPackedStore refactor * Improve naming convention, and make it more consistent. * Use engine types as much as possible. * Reduced the amount of string copy constructions.
2023-05-29 21:41:17 +02:00
CUtlString GetNameParts(const CUtlString& dirFileName, int nCaptureGroup) const;
CUtlString GetLevelName(const CUtlString& dirFileName) const;
Add support for .vpkignore file Folders can be globally excluded using the '.vpkignore' file (placed in the workspace root). To exclude specific files, use the manifest file.
2022-06-06 23:08:53 +02:00
CPackedStore refactor * Improve naming convention, and make it more consistent. * Use engine types as much as possible. * Reduced the amount of string copy constructions.
2023-05-29 21:41:17 +02:00
KeyValues* GetManifest(const CUtlString& workspacePath, const CUtlString& manifestFile) const;
bool GetIgnoreList(CUtlVector<CUtlString>& ignoreList, const CUtlString& workspacePath) const;
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
CPackedStore refactor * Improve naming convention, and make it more consistent. * Use engine types as much as possible. * Reduced the amount of string copy constructions.
2023-05-29 21:41:17 +02:00
CUtlString FormatEntryPath(const CUtlString& filePath, const CUtlString& fileName, const CUtlString& fileExt) const;
void BuildManifest(const CUtlVector<VPKEntryBlock_t>& entryBlocks, const CUtlString& workspacePath, const CUtlString& manifestName) const;
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
CPackedStore refactor * Improve naming convention, and make it more consistent. * Use engine types as much as possible. * Reduced the amount of string copy constructions.
2023-05-29 21:41:17 +02:00
void ValidateCRC32PostDecomp(const CUtlString& assetPath, const uint32_t nFileCRC);
CPackedStore: Fix bug and improve code readability Moved deduplication code to a separate function. Unnested packing and unpacking code a bit more.
2023-04-03 22:46:10 +02:00
bool Deduplicate(const uint8_t* pEntryBuffer, VPKChunkDescriptor_t& descriptor, const size_t chunkIndex);
CPackedStore refactor * Improve naming convention, and make it more consistent. * Use engine types as much as possible. * Reduced the amount of string copy constructions.
2023-05-29 21:41:17 +02:00
bool ShouldPrune(const CUtlString& filePath, CUtlVector<CUtlString>& ignoreList) const;
void PackWorkspace(const VPKPair_t& vpkPair, const char* workspaceName, const char* buildPath);
void UnpackWorkspace(const VPKDir_t& vpkDir, const char* workspaceName = "");
Update terminology entirely for packedstore.h/.cpp VPK tooling also works on the R1 game (Titanfall 1)
2022-06-06 14:54:22 +02:00
private:
CPackedStore refactor * Improve naming convention, and make it more consistent. * Use engine types as much as possible. * Reduced the amount of string copy constructions.
2023-05-29 21:41:17 +02:00
lzham_compress_params m_lzCompParams; // LZham compression parameters.
lzham_decompress_params m_lzDecompParams; // LZham decompression parameters.
std::unordered_map<string, const VPKChunkDescriptor_t&> m_ChunkHashMap;
Code base refactor + major performance and readability improvement. Read description for details. * Codebase restructured to SourceSDK codebase style and .cpp/.h assertion paths in the game executable. * Document most functions with valve style 'Purpose' blocks. * Rename variables to match the rest of the codebase and Valve's naming convention. * Dedicated DLL and the SDKLauncher now share the same codebase as the DevSDK. * Obtain globals or pointers directly instead of waiting for runtime initialized data. * Dynamically search for all functions and globals (this doesn't count for dedicated yet!). * Initialize most in-SDK variables. * Move certain prints and other utilities under ConVars to reduce verbosity and increase performance. * Print all pattern scan results through a virtual function to make it easier to add and debug new patterns in the future. * Type global var pointers appropriately if class or type is known and implemented. * Forward declare 'CClient' class to avoid having 2 'g_pClient' copies. * Add IDA's pseudo definitions for easier prototyping with decompiled assembly code. * RPAK decompress Command callback implementation. * Load decompressed RPaks from 'paks\Win32\' overriding the ones in 'paks\Win64\' (the decompress callback will automatically fix the header and write it to 'paks\Win32\'). * VPK decompress Command callback implementation. * Move CRC32 ands Adler32 to implementation files. * Server will print out more details about the connecting client. * Upgrade ImGui lib to v1.86. * Don't compile id3dx.h for dedicated. * Don't compile id3dx.cpp for dedicated * Implement DevMsg print function allowing to print information to the in-game VGUI/RUI console overlay, ImGui console overlay and the external windows console * Fixed bug where the Error function would not properly terminate the process when an error is called. This caused access violations for critical/non-recoverable errors. * Fixed bug where the game would crash if the console or server browser was enabled while the game was still starting up. * Several bug fixes for the dedicated server (warning: dedicated is still considered work-in-progress!).
2021-12-25 22:36:38 +01:00
};
///////////////////////////////////////////////////////////////////////////////
extern CPackedStore* g_pPackedStore;
CPackedStore: light optimizations * Don't call copy constructor on entry blocks when building the directory file. * Don't call copy constructor for string when we don't want to sanitize the directory file name.
2022-10-12 23:39:26 +02:00
#endif // PACKEDSTORE_H
Reference in New Issue Copy Permalink