R5Reloaded/r5sdk
1
0
Fork 0
mirror of https://github.com/Mauler125/r5sdk.git synced 2025-02-09 19:15:03 +01:00
Code Issues Packages Projects Releases Wiki Activity
r5sdk/r5dev/vpklib/packedstore.cpp

879 lines
32 KiB
C++
Raw Normal View History

Update terminology entirely for packedstore.h/.cpp VPK tooling also works on the R1 game (Titanfall 1)
2022-06-06 14:54:22 +02:00
/*******************************************************************
* ██████╗ ██╗ ██╗ ██╗██████╗ ██╗ ██╗ ██╗ ██╗██████╗ *
* ██╔══██╗███║ ██║ ██║██╔══██╗██║ ██╔╝ ██║ ██║██╔══██╗ *
* ██████╔╝╚██║ ██║ ██║██████╔╝█████╔╝ ██║ ██║██████╔╝ *
* ██╔══██╗ ██║ ╚██╗ ██╔╝██╔═══╝ ██╔═██╗ ██║ ██║██╔══██╗ *
* ██║ ██║ ██║ ╚████╔╝ ██║ ██║ ██╗ ███████╗██║██████╔╝ *
* ╚═╝ ╚═╝ ╚═╝ ╚═══╝ ╚═╝ ╚═╝ ╚═╝ ╚══════╝╚═╝╚═════╝ *
*******************************************************************/
Update packedstore.cpp
2022-06-02 01:59:03 +02:00
#include "core/stdafx.h"
#include "tier1/cvar.h"
#include "mathlib/adler32.h"
#include "mathlib/crc32.h"
Implement data sharing algorithm in VPK packing This reduces size of the resulting VPK (checks entry block in hash map). mp_common compresses to 32mb instead of 37.3mb (original is 38.5mb).
2022-06-04 13:55:24 +02:00
#include "mathlib/sha1.h"
Update packedstore.cpp
2022-06-02 01:59:03 +02:00
#include "vpklib/packedstore.h"
Use unicode character set instead and light cleanup/bug fixes
2022-03-04 12:22:17 +01:00
Use static buffer for file chunking and compressing during VPK packing Avoid using new every time. Chunk can be at most 1024 * 1024.
2022-10-02 12:17:03 +02:00
//-----------------------------------------------------------------------------
More VPK compressing/decompressing optimizations * Use reference to array index. * Only obtain array size once during loop. * Use static buffer of 1024*1024 for decompressing. * Removed extraneous goto. * Fixed bug where shared data would be mapped wrong due to the last optimization. * This increased packing performance by 40/60 seconds when rebuilding all vpk's.
2022-10-02 19:24:00 +02:00
// Static buffers for chunking/compressing the source files and decompressing
Use static buffer for file chunking and compressing during VPK packing Avoid using new every time. Chunk can be at most 1024 * 1024.
2022-10-02 12:17:03 +02:00
//-----------------------------------------------------------------------------
More VPK compressing/decompressing optimizations * Use reference to array index. * Only obtain array size once during loop. * Use static buffer of 1024*1024 for decompressing. * Removed extraneous goto. * Fixed bug where shared data would be mapped wrong due to the last optimization. * This increased packing performance by 40/60 seconds when rebuilding all vpk's.
2022-10-02 19:24:00 +02:00
static uint8_t s_EntryBuf[ENTRY_MAX_LEN];
static uint8_t s_DecompBuf[ENTRY_MAX_LEN];
Use static buffer for file chunking and compressing during VPK packing Avoid using new every time. Chunk can be at most 1024 * 1024.
2022-10-02 12:17:03 +02:00
Update packedstore.cpp
2022-06-02 01:59:03 +02:00
//-----------------------------------------------------------------------------
// Purpose: initialize parameters for compression algorithm
//-----------------------------------------------------------------------------
void CPackedStore::InitLzCompParams(void)
{
/*| PARAMETERS ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||*/
Additional cleanup Cast to proper types. Set the archive offset only instead of the entire block. The entry could still have its own flags. The only thing needed for saving data is sharing the same offset and size.
2022-06-05 12:28:49 +02:00
m_lzCompParams.m_dict_size_log2 = VPK_DICT_SIZE;
Set VPK compression level through ConVar
2022-09-11 00:16:31 +02:00
m_lzCompParams.m_level = GetCompressionLevel();
m_lzCompParams.m_compress_flags = lzham_compress_flags::LZHAM_COMP_FLAG_DETERMINISTIC_PARSING;
Update packedstore.cpp
2022-06-02 01:59:03 +02:00
m_lzCompParams.m_max_helper_threads = -1;
}
//-----------------------------------------------------------------------------
// Purpose: initialize parameters for decompression algorithm
//-----------------------------------------------------------------------------
void CPackedStore::InitLzDecompParams(void)
{
/*| PARAMETERS ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||*/
Additional cleanup Cast to proper types. Set the archive offset only instead of the entire block. The entry could still have its own flags. The only thing needed for saving data is sharing the same offset and size.
2022-06-05 12:28:49 +02:00
m_lzDecompParams.m_dict_size_log2 = VPK_DICT_SIZE;
Update packedstore.cpp
2022-06-02 01:59:03 +02:00
m_lzDecompParams.m_decompress_flags = lzham_decompress_flags::LZHAM_DECOMP_FLAG_OUTPUT_UNBUFFERED | lzham_decompress_flags::LZHAM_DECOMP_FLAG_COMPUTE_CRC32;
m_lzDecompParams.m_struct_size = sizeof(lzham_decompress_params);
}
//-----------------------------------------------------------------------------
CPackedStore: fix directory file name sanitizer The sanitizer can retrieve the directory file name from the block file name if this is passed instead. This was broken in the previous system. Sanitization can be opted out if the 3rd parameter of the 'fs_vpk_unpack' command is omitted.
2022-10-11 15:37:34 +02:00
// Purpose: gets a directory structure for specified file.
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
// Input : svPackDirFile -
CPackedStore: fix directory file name sanitizer The sanitizer can retrieve the directory file name from the block file name if this is passed instead. This was broken in the previous system. Sanitization can be opted out if the 3rd parameter of the 'fs_vpk_unpack' command is omitted.
2022-10-11 15:37:34 +02:00
// bSanitizeName - retrieve the directory file name from block name
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
// Output : VPKDir_t
Update packedstore.cpp
2022-06-02 01:59:03 +02:00
//-----------------------------------------------------------------------------
CPackedStore: fix directory file name sanitizer The sanitizer can retrieve the directory file name from the block file name if this is passed instead. This was broken in the previous system. Sanitization can be opted out if the 3rd parameter of the 'fs_vpk_unpack' command is omitted.
2022-10-11 15:37:34 +02:00
VPKDir_t CPackedStore::GetDirectoryFile(string svPackDirFile, bool bSanitizeName) const
Update packedstore.cpp
2022-06-02 01:59:03 +02:00
{
/*| PACKDIRFILE |||||||||||||||||||||||||||||||||||||||||||||||||||||||||*/
std::smatch smRegexMatches;
CPackedStore: fix directory file name sanitizer The sanitizer can retrieve the directory file name from the block file name if this is passed instead. This was broken in the previous system. Sanitization can be opted out if the 3rd parameter of the 'fs_vpk_unpack' command is omitted.
2022-10-11 15:37:34 +02:00
if (!bSanitizeName)
return VPKDir_t(svPackDirFile);
Update packedstore.cpp
2022-06-02 01:59:03 +02:00
CPackedStore: fix directory file name sanitizer The sanitizer can retrieve the directory file name from the block file name if this is passed instead. This was broken in the previous system. Sanitization can be opted out if the 3rd parameter of the 'fs_vpk_unpack' command is omitted.
2022-10-11 15:37:34 +02:00
std::regex_search(svPackDirFile, smRegexMatches, BLOCK_REGEX);
if (smRegexMatches.empty())
return VPKDir_t(svPackDirFile);
StringReplace(svPackDirFile, smRegexMatches[0], "pak000_dir");
bool bHasLocale = false;
string svPackDirPrefix;
for (size_t i = 0, nl = DIR_LOCALE.size(); i < nl; i++)
{
const string& svLocale = DIR_LOCALE[i];
if (svPackDirFile.find(svLocale) != string::npos)
{
bHasLocale = true;
CPackedStore::GetDirectoryFile optimizations Break when we have a locale, or found the context. Removed extraneous logic/vars.
2022-10-11 16:31:00 +02:00
break;
CPackedStore: fix directory file name sanitizer The sanitizer can retrieve the directory file name from the block file name if this is passed instead. This was broken in the previous system. Sanitization can be opted out if the 3rd parameter of the 'fs_vpk_unpack' command is omitted.
2022-10-11 15:37:34 +02:00
}
}
CPackedStore::GetDirectoryFile optimizations Break when we have a locale, or found the context. Removed extraneous logic/vars.
2022-10-11 16:31:00 +02:00
if (!bHasLocale)
Update packedstore.cpp
2022-06-02 01:59:03 +02:00
{
CPackedStore: fix directory file name sanitizer The sanitizer can retrieve the directory file name from the block file name if this is passed instead. This was broken in the previous system. Sanitization can be opted out if the 3rd parameter of the 'fs_vpk_unpack' command is omitted.
2022-10-11 15:37:34 +02:00
svPackDirPrefix.append(DIR_LOCALE[0]);
Update packedstore.cpp
2022-06-02 01:59:03 +02:00
CPackedStore: fix directory file name sanitizer The sanitizer can retrieve the directory file name from the block file name if this is passed instead. This was broken in the previous system. Sanitization can be opted out if the 3rd parameter of the 'fs_vpk_unpack' command is omitted.
2022-10-11 15:37:34 +02:00
for (size_t i = 0, nc = DIR_CONTEXT.size(); i < nc; i++)
Update packedstore.cpp
2022-06-02 01:59:03 +02:00
{
CPackedStore: fix directory file name sanitizer The sanitizer can retrieve the directory file name from the block file name if this is passed instead. This was broken in the previous system. Sanitization can be opted out if the 3rd parameter of the 'fs_vpk_unpack' command is omitted.
2022-10-11 15:37:34 +02:00
const string& svContext = DIR_CONTEXT[i];
if (svPackDirFile.find(svContext) != string::npos)
Update packedstore.cpp
2022-06-02 01:59:03 +02:00
{
CPackedStore: fix directory file name sanitizer The sanitizer can retrieve the directory file name from the block file name if this is passed instead. This was broken in the previous system. Sanitization can be opted out if the 3rd parameter of the 'fs_vpk_unpack' command is omitted.
2022-10-11 15:37:34 +02:00
svPackDirPrefix.append(svContext);
CPackedStore::GetDirectoryFile optimizations Break when we have a locale, or found the context. Removed extraneous logic/vars.
2022-10-11 16:31:00 +02:00
StringReplace(svPackDirFile, svContext, svPackDirPrefix);
break;
Update packedstore.cpp
2022-06-02 01:59:03 +02:00
}
CPackedStore: fix directory file name sanitizer The sanitizer can retrieve the directory file name from the block file name if this is passed instead. This was broken in the previous system. Sanitization can be opted out if the 3rd parameter of the 'fs_vpk_unpack' command is omitted.
2022-10-11 15:37:34 +02:00
}
}
return VPKDir_t(svPackDirFile);
Update packedstore.cpp
2022-06-02 01:59:03 +02:00
}
packedstore cleanup Add const qualifiers to params. Enable compression flags.
2022-06-02 15:20:22 +02:00
//-----------------------------------------------------------------------------
Update terminology entirely for packedstore.h/.cpp VPK tooling also works on the R1 game (Titanfall 1)
2022-06-06 14:54:22 +02:00
// Purpose: formats pack file path for specific directory file
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
// Input : &svPackDirFile -
// iArchiveIndex -
// output : string
packedstore cleanup Add const qualifiers to params. Enable compression flags.
2022-06-02 15:20:22 +02:00
//-----------------------------------------------------------------------------
Update terminology entirely for packedstore.h/.cpp VPK tooling also works on the R1 game (Titanfall 1)
2022-06-06 14:54:22 +02:00
string CPackedStore::GetPackFile(const string& svPackDirFile, uint16_t iArchiveIndex) const
packedstore cleanup Add const qualifiers to params. Enable compression flags.
2022-06-02 15:20:22 +02:00
{
/*| ARCHIVES ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||*/
string svPackChunkFile = StripLocalePrefix(svPackDirFile);
ostringstream oss;
oss << std::setw(3) << std::setfill('0') << iArchiveIndex;
string svPackChunkIndex = "pak000_" + oss.str();
StringReplace(svPackChunkFile, "pak000_dir", svPackChunkIndex);
return svPackChunkFile;
}
Set VPK compression level through ConVar
2022-09-11 00:16:31 +02:00
//-----------------------------------------------------------------------------
// Purpose: gets the LZHAM compression level
// output : lzham_compress_level
//-----------------------------------------------------------------------------
lzham_compress_level CPackedStore::GetCompressionLevel(void) const
{
const char* pszLevel = fs_packedstore_compression_level->GetString();
if(strcmp(pszLevel, "fastest") == NULL)
return lzham_compress_level::LZHAM_COMP_LEVEL_FASTEST;
else if (strcmp(pszLevel, "faster") == NULL)
return lzham_compress_level::LZHAM_COMP_LEVEL_FASTER;
else if (strcmp(pszLevel, "default") == NULL)
return lzham_compress_level::LZHAM_COMP_LEVEL_DEFAULT;
else if (strcmp(pszLevel, "better") == NULL)
return lzham_compress_level::LZHAM_COMP_LEVEL_BETTER;
else if (strcmp(pszLevel, "uber") == NULL)
return lzham_compress_level::LZHAM_COMP_LEVEL_UBER;
else
return lzham_compress_level::LZHAM_COMP_LEVEL_DEFAULT;
}
Update packedstore.cpp
2022-06-02 01:59:03 +02:00
//-----------------------------------------------------------------------------
// Purpose: obtains and returns the entry block to the vector
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
// Input : *pReader -
// output : vector<VPKEntryBlock_t>
Update packedstore.cpp
2022-06-02 01:59:03 +02:00
//-----------------------------------------------------------------------------
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
vector<VPKEntryBlock_t> CPackedStore::GetEntryBlocks(CIOStream* pReader) const
Update packedstore.cpp
2022-06-02 01:59:03 +02:00
{
/*| ENTRYBLOCKS |||||||||||||||||||||||||||||||||||||||||||||||||||||||||*/
string svName, svPath, svExtension;
vector<VPKEntryBlock_t> vBlocks;
while (!(svExtension = pReader->ReadString()).empty())
{
while (!(svPath = pReader->ReadString()).empty())
{
while (!(svName = pReader->ReadString()).empty())
{
Set VPK compression level through ConVar
2022-09-11 00:16:31 +02:00
const string svFilePath = FormatEntryPath(svPath, svName, svExtension);
Update packedstore.cpp
2022-06-02 01:59:03 +02:00
vBlocks.push_back(VPKEntryBlock_t(pReader, svFilePath));
}
}
}
return vBlocks;
}
//-----------------------------------------------------------------------------
// Purpose: scans the input directory and returns the paths to the vector
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
// Input : &svPathIn -
CPackedStore: fix directory file name sanitizer The sanitizer can retrieve the directory file name from the block file name if this is passed instead. This was broken in the previous system. Sanitization can be opted out if the 3rd parameter of the 'fs_vpk_unpack' command is omitted.
2022-10-11 15:37:34 +02:00
// Output : a string vector of all included entry paths
Update packedstore.cpp
2022-06-02 01:59:03 +02:00
//-----------------------------------------------------------------------------
Update terminology entirely for packedstore.h/.cpp VPK tooling also works on the R1 game (Titanfall 1)
2022-06-06 14:54:22 +02:00
vector<string> CPackedStore::GetEntryPaths(const string& svPathIn) const
Update packedstore.cpp
2022-06-02 01:59:03 +02:00
{
vector<string> vPaths;
Add support for .vpkignore file Folders can be globally excluded using the '.vpkignore' file (placed in the workspace root). To exclude specific files, use the manifest file.
2022-06-06 23:08:53 +02:00
vector<string> vIgnore = GetIgnoreList(svPathIn);
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
fs::recursive_directory_iterator dir(svPathIn), end;
while (dir != end)
{
CPackedStore: do not pack empty files Empty files have no place in a VPK.
2022-09-10 00:07:42 +02:00
const vector<string>::iterator it = std::find(vIgnore.begin(), vIgnore.end(),
Add file extension support to .vpkignore
2022-06-07 13:44:31 +02:00
GetExtension(dir->path().filename().u8string(), true, true));
if (it != vIgnore.end())
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
{
Add file extension support to .vpkignore
2022-06-07 13:44:31 +02:00
dir.disable_recursion_pending(); // Skip all ignored folders and extensions.
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
}
CPackedStore: do not pack empty files Empty files have no place in a VPK.
2022-09-10 00:07:42 +02:00
else if (dir->file_size() > 0) // Empty files are not supported.
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
{
CPackedStore: do not pack empty files Empty files have no place in a VPK.
2022-09-10 00:07:42 +02:00
const string svPath = dir->path().u8string();
if (!GetExtension(svPath).empty())
{
vPaths.push_back(ConvertToUnixPath(svPath));
}
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
}
dir++;
}
return vPaths;
}
//-----------------------------------------------------------------------------
// Purpose: scans the input directory and returns the paths to the vector if path exists in manifest
// Input : &svPathIn -
// &jManifest -
CPackedStore: fix directory file name sanitizer The sanitizer can retrieve the directory file name from the block file name if this is passed instead. This was broken in the previous system. Sanitization can be opted out if the 3rd parameter of the 'fs_vpk_unpack' command is omitted.
2022-10-11 15:37:34 +02:00
// Output : a string vector of all included entry paths
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
//-----------------------------------------------------------------------------
Update terminology entirely for packedstore.h/.cpp VPK tooling also works on the R1 game (Titanfall 1)
2022-06-06 14:54:22 +02:00
vector<string> CPackedStore::GetEntryPaths(const string& svPathIn, const nlohmann::json& jManifest) const
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
{
vector<string> vPaths;
Add support for .vpkignore file Folders can be globally excluded using the '.vpkignore' file (placed in the workspace root). To exclude specific files, use the manifest file.
2022-06-06 23:08:53 +02:00
vector<string> vIgnore = GetIgnoreList(svPathIn);
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
fs::recursive_directory_iterator dir(svPathIn), end;
while (dir != end)
Update packedstore.cpp
2022-06-02 01:59:03 +02:00
{
CPackedStore: do not pack empty files Empty files have no place in a VPK.
2022-09-10 00:07:42 +02:00
const vector<string>::iterator it = std::find(vIgnore.begin(), vIgnore.end(),
Add file extension support to .vpkignore
2022-06-07 13:44:31 +02:00
GetExtension(dir->path().filename().u8string(), true, true));
if (it != vIgnore.end())
Update packedstore.cpp
2022-06-02 01:59:03 +02:00
{
Add file extension support to .vpkignore
2022-06-07 13:44:31 +02:00
dir.disable_recursion_pending(); // Skip all ignored folders and extensions.
Update packedstore.cpp
2022-06-02 01:59:03 +02:00
}
CPackedStore: do not pack empty files Empty files have no place in a VPK.
2022-09-10 00:07:42 +02:00
else if (dir->file_size() > 0) // Empty files are not supported.
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
{
CPackedStore: do not pack empty files Empty files have no place in a VPK.
2022-09-10 00:07:42 +02:00
const string svPath = dir->path().u8string();
if (!GetExtension(svPath).empty())
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
{
CPackedStore: do not pack empty files Empty files have no place in a VPK.
2022-09-10 00:07:42 +02:00
if (!jManifest.is_null())
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
{
CPackedStore: do not pack empty files Empty files have no place in a VPK.
2022-09-10 00:07:42 +02:00
try
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
{
CPackedStore: do not pack empty files Empty files have no place in a VPK.
2022-09-10 00:07:42 +02:00
const string svEntryPath = ConvertToUnixPath(svPath);
if (jManifest.contains(StringReplaceC(svEntryPath, svPathIn, "")))
{
vPaths.push_back(svEntryPath);
}
}
catch (const std::exception& ex)
{
Warning(eDLL_T::FS, "Exception while reading VPK control file: '%s'\n", ex.what());
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
}
}
}
}
dir++;
Update packedstore.cpp
2022-06-02 01:59:03 +02:00
}
return vPaths;
}
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
//-----------------------------------------------------------------------------
CPackedStore: fix directory file name sanitizer The sanitizer can retrieve the directory file name from the block file name if this is passed instead. This was broken in the previous system. Sanitization can be opted out if the 3rd parameter of the 'fs_vpk_unpack' command is omitted.
2022-10-11 15:37:34 +02:00
// Purpose: gets the parts of the directory file name
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
// Input : &svDirectoryName -
CPackedStore: fix directory file name sanitizer The sanitizer can retrieve the directory file name from the block file name if this is passed instead. This was broken in the previous system. Sanitization can be opted out if the 3rd parameter of the 'fs_vpk_unpack' command is omitted.
2022-10-11 15:37:34 +02:00
// nCaptureGroup - (1 = locale + context, 2 = levelname)
// Output : part of directory file name as string
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
//-----------------------------------------------------------------------------
VPK system improvements More detailed operation statistics. Discard VPK directory files who's header doesn't match engine requirements. Separated manifest properly between language and context (you can now package for each language and context from a single workspace).
2022-06-05 17:28:39 +02:00
string CPackedStore::GetNameParts(const string& svDirectoryName, int nCaptureGroup) const
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
{
std::smatch smRegexMatches;
CPackedStore: fix directory file name sanitizer The sanitizer can retrieve the directory file name from the block file name if this is passed instead. This was broken in the previous system. Sanitization can be opted out if the 3rd parameter of the 'fs_vpk_unpack' command is omitted.
2022-10-11 15:37:34 +02:00
std::regex_search(svDirectoryName, smRegexMatches, DIR_REGEX);
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
VPK system improvements More detailed operation statistics. Discard VPK directory files who's header doesn't match engine requirements. Separated manifest properly between language and context (you can now package for each language and context from a single workspace).
2022-06-05 17:28:39 +02:00
return smRegexMatches[nCaptureGroup].str();
}
//-----------------------------------------------------------------------------
// Purpose: gets the source of the directory file name
// Input : &svDirectoryName -
CPackedStore: fix directory file name sanitizer The sanitizer can retrieve the directory file name from the block file name if this is passed instead. This was broken in the previous system. Sanitization can be opted out if the 3rd parameter of the 'fs_vpk_unpack' command is omitted.
2022-10-11 15:37:34 +02:00
// Output : source name as string (e.g. "mp_rr_box")
VPK system improvements More detailed operation statistics. Discard VPK directory files who's header doesn't match engine requirements. Separated manifest properly between language and context (you can now package for each language and context from a single workspace).
2022-06-05 17:28:39 +02:00
//-----------------------------------------------------------------------------
string CPackedStore::GetSourceName(const string& svDirectoryName) const
{
std::smatch smRegexMatches;
CPackedStore: fix directory file name sanitizer The sanitizer can retrieve the directory file name from the block file name if this is passed instead. This was broken in the previous system. Sanitization can be opted out if the 3rd parameter of the 'fs_vpk_unpack' command is omitted.
2022-10-11 15:37:34 +02:00
std::regex_search(svDirectoryName, smRegexMatches, DIR_REGEX);
VPK system improvements More detailed operation statistics. Discard VPK directory files who's header doesn't match engine requirements. Separated manifest properly between language and context (you can now package for each language and context from a single workspace).
2022-06-05 17:28:39 +02:00
return smRegexMatches[1].str() + smRegexMatches[2].str();
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
}
//-----------------------------------------------------------------------------
Fix spelling errors Overall spelling improvements and cleanup..
2022-09-09 19:47:31 +02:00
// Purpose: gets the manifest file associated with the VPK name
Add support for .vpkignore file Folders can be globally excluded using the '.vpkignore' file (placed in the workspace root). To exclude specific files, use the manifest file.
2022-06-06 23:08:53 +02:00
// Input : &svWorkSpace -
// &svManifestName -
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
// Output : json
//-----------------------------------------------------------------------------
Revert declaration
2022-06-04 01:30:18 +02:00
nlohmann::json CPackedStore::GetManifest(const string& svWorkSpace, const string& svManifestName) const
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
{
ostringstream ostream;
ostream << svWorkSpace << "manifest/" << svManifestName << ".json";
fs::path fsPath = fs::current_path() /= ostream.str();
Fix return bug CPackedStore::GetManifest() did not return a value in all control paths. Use proper size types for entry block constructor.
2022-06-06 00:21:13 +02:00
nlohmann::json jsOut;
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
if (fs::exists(fsPath))
{
try
{
General cleanup Use c++ methods, remove unnecessary casts, unnecessary use of auto, etc..
2022-08-09 15:19:12 +02:00
ifstream iManifest(fsPath.u8string(), std::ios::binary);
Revert declaration
2022-06-04 01:30:18 +02:00
jsOut = nlohmann::json::parse(iManifest);
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
return jsOut;
}
catch (const std::exception& ex)
{
Use correct size type format specifier in packetstore.cpp Using %llu is also correct, but this is the standard approach.
2022-08-08 17:51:09 +02:00
Warning(eDLL_T::FS, "Exception while parsing VPK control file: '%s'\n", ex.what());
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
return jsOut;
}
}
Fix return bug CPackedStore::GetManifest() did not return a value in all control paths. Use proper size types for entry block constructor.
2022-06-06 00:21:13 +02:00
return jsOut;
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
}
Add support for .vpkignore file Folders can be globally excluded using the '.vpkignore' file (placed in the workspace root). To exclude specific files, use the manifest file.
2022-06-06 23:08:53 +02:00
//-----------------------------------------------------------------------------
// Purpose: gets the contents from the global ignore list (.vpkignore)
// Input : &svWorkSpace -
CPackedStore: fix directory file name sanitizer The sanitizer can retrieve the directory file name from the block file name if this is passed instead. This was broken in the previous system. Sanitization can be opted out if the 3rd parameter of the 'fs_vpk_unpack' command is omitted.
2022-10-11 15:37:34 +02:00
// Output : a string vector of ignored directories/files
Add support for .vpkignore file Folders can be globally excluded using the '.vpkignore' file (placed in the workspace root). To exclude specific files, use the manifest file.
2022-06-06 23:08:53 +02:00
//-----------------------------------------------------------------------------
vector<string> CPackedStore::GetIgnoreList(const string& svWorkSpace) const
{
fs::path fsIgnore = svWorkSpace + ".vpkignore";
ifstream iStream(fsIgnore);
vector<string> vIgnore;
if (iStream)
{
string svIgnore;
while (std::getline(iStream, svIgnore))
{
string::size_type nPos = svIgnore.find("//");
if (nPos == string::npos)
{
Cleanup
2022-06-07 13:57:29 +02:00
if (!svIgnore.empty() &&
std::find(vIgnore.begin(), vIgnore.end(), svIgnore) == vIgnore.end())
Add support for .vpkignore file Folders can be globally excluded using the '.vpkignore' file (placed in the workspace root). To exclude specific files, use the manifest file.
2022-06-06 23:08:53 +02:00
{
vIgnore.push_back(svIgnore);
}
}
}
}
return vIgnore;
}
Update packedstore.cpp
2022-06-02 01:59:03 +02:00
//-----------------------------------------------------------------------------
Update terminology entirely for packedstore.h/.cpp VPK tooling also works on the R1 game (Titanfall 1)
2022-06-06 14:54:22 +02:00
// Purpose: formats the file entry path
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
// Input : svPath -
Add support for .vpkignore file Folders can be globally excluded using the '.vpkignore' file (placed in the workspace root). To exclude specific files, use the manifest file.
2022-06-06 23:08:53 +02:00
// &svName -
// &svExtension -
CPackedStore: fix directory file name sanitizer The sanitizer can retrieve the directory file name from the block file name if this is passed instead. This was broken in the previous system. Sanitization can be opted out if the 3rd parameter of the 'fs_vpk_unpack' command is omitted.
2022-10-11 15:37:34 +02:00
// Output : formatted entry path
Update packedstore.cpp
2022-06-02 01:59:03 +02:00
//-----------------------------------------------------------------------------
Update terminology entirely for packedstore.h/.cpp VPK tooling also works on the R1 game (Titanfall 1)
2022-06-06 14:54:22 +02:00
string CPackedStore::FormatEntryPath(string svPath, const string& svName, const string& svExtension) const
Update packedstore.cpp
2022-06-02 01:59:03 +02:00
{
if (!svPath.empty())
{
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
svPath += '/';
Update packedstore.cpp
2022-06-02 01:59:03 +02:00
}
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
return svPath + svName + '.' + svExtension;
Update packedstore.cpp
2022-06-02 01:59:03 +02:00
}
//-----------------------------------------------------------------------------
// Purpose: strips locale prefix from file path
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
// Input : &svDirectoryFile -
CPackedStore: fix directory file name sanitizer The sanitizer can retrieve the directory file name from the block file name if this is passed instead. This was broken in the previous system. Sanitization can be opted out if the 3rd parameter of the 'fs_vpk_unpack' command is omitted.
2022-10-11 15:37:34 +02:00
// Output : directory filename without locale prefix
Update packedstore.cpp
2022-06-02 01:59:03 +02:00
//-----------------------------------------------------------------------------
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
string CPackedStore::StripLocalePrefix(const string& svDirectoryFile) const
Update packedstore.cpp
2022-06-02 01:59:03 +02:00
{
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
fs::path fsDirectoryFile(svDirectoryFile);
string svFileName = fsDirectoryFile.filename().u8string();
Update packedstore.cpp
2022-06-02 01:59:03 +02:00
Additional VPK lib cleanup and performance improvements
2022-10-09 12:08:54 +02:00
for (const string& svLocale : DIR_LOCALE)
Update packedstore.cpp
2022-06-02 01:59:03 +02:00
{
Additional VPK lib cleanup and performance improvements
2022-10-09 12:08:54 +02:00
if (svFileName.find(svLocale) != string::npos)
Update packedstore.cpp
2022-06-02 01:59:03 +02:00
{
Additional VPK lib cleanup and performance improvements
2022-10-09 12:08:54 +02:00
StringReplace(svFileName, svLocale, "");
Update packedstore.cpp
2022-06-02 01:59:03 +02:00
break;
}
}
return svFileName;
}
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
//-----------------------------------------------------------------------------
// Purpose: builds a valid file name for the VPK
// Input : svLanguage -
// svContext -
// &svPakName -
// nPatch -
CPackedStore: fix directory file name sanitizer The sanitizer can retrieve the directory file name from the block file name if this is passed instead. This was broken in the previous system. Sanitization can be opted out if the 3rd parameter of the 'fs_vpk_unpack' command is omitted.
2022-10-11 15:37:34 +02:00
// Output : a vpk file pair (block and directory file names)
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
//-----------------------------------------------------------------------------
VPKPair_t CPackedStore::BuildFileName(string svLanguage, string svContext, const string& svPakName, int nPatch) const
{
if (std::find(DIR_LOCALE.begin(), DIR_LOCALE.end(), svLanguage) == DIR_LOCALE.end())
{
svLanguage = DIR_LOCALE[0];
}
if (std::find(DIR_CONTEXT.begin(), DIR_CONTEXT.end(), svContext) == DIR_CONTEXT.end())
{
svContext = DIR_CONTEXT[0];
}
VPKPair_t vPair;
vPair.m_svBlockName = fmt::format("{:s}_{:s}.bsp.pak000_{:03d}{:s}", svContext, svPakName, nPatch, ".vpk");
vPair.m_svDirectoryName = fmt::format("{:s}{:s}_{:s}.bsp.pak000_{:s}", svLanguage, svContext, svPakName, "dir.vpk");
return vPair;
}
//-----------------------------------------------------------------------------
// Purpose: builds the VPK manifest file
// Input : &vBlock -
Add support for .vpkignore file Folders can be globally excluded using the '.vpkignore' file (placed in the workspace root). To exclude specific files, use the manifest file.
2022-06-06 23:08:53 +02:00
// &svWorkSpace -
// &svManifestName -
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
//-----------------------------------------------------------------------------
void CPackedStore::BuildManifest(const vector<VPKEntryBlock_t>& vBlock, const string& svWorkSpace, const string& svManifestName) const
{
Revert declaration
2022-06-04 01:30:18 +02:00
nlohmann::json jEntry;
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
Additional VPK lib cleanup and performance improvements
2022-10-09 12:08:54 +02:00
for (const VPKEntryBlock_t& vEntry : vBlock)
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
{
Additional VPK lib cleanup and performance improvements
2022-10-09 12:08:54 +02:00
jEntry[vEntry.m_svEntryPath] =
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
{
Additional VPK lib cleanup and performance improvements
2022-10-09 12:08:54 +02:00
{ "preloadSize", vEntry.m_iPreloadSize },
{ "loadFlags", vEntry.m_vChunks[0].m_nLoadFlags },
{ "textureFlags", vEntry.m_vChunks[0].m_nTextureFlags },
{ "useCompression", vEntry.m_vChunks[0].m_nCompressedSize != vEntry.m_vChunks[0].m_nUncompressedSize },
Implement data sharing algorithm in VPK packing This reduces size of the resulting VPK (checks entry block in hash map). mp_common compresses to 32mb instead of 37.3mb (original is 38.5mb).
2022-06-04 13:55:24 +02:00
{ "useDataSharing", true }
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
};
}
VPK system improvements More detailed operation statistics. Discard VPK directory files who's header doesn't match engine requirements. Separated manifest properly between language and context (you can now package for each language and context from a single workspace).
2022-06-05 17:28:39 +02:00
string svPathOut = svWorkSpace + "manifest/";
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
fs::create_directories(svPathOut);
ofstream oManifest(svPathOut + svManifestName + ".json");
oManifest << jEntry.dump(4);
}
Update packedstore.cpp
2022-06-02 01:59:03 +02:00
//-----------------------------------------------------------------------------
// Purpose: validates extraction result with precomputed ADLER32 hash
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
// Input : &svAssetFile -
Update packedstore.cpp
2022-06-02 01:59:03 +02:00
//-----------------------------------------------------------------------------
void CPackedStore::ValidateAdler32PostDecomp(const string& svAssetFile)
{
CIOStream reader(svAssetFile, CIOStream::Mode_t::READ);
Additional VPK system improvements Added DevMsg logs to show the elapsed time of the operation. Fixed post decompress CRC/ADLER error checking. Slight cleanup.
2022-06-04 03:12:46 +02:00
m_nAdler32 = adler32::update(NULL, reader.GetData(), reader.GetSize());
Update packedstore.cpp
2022-06-02 01:59:03 +02:00
if (m_nAdler32 != m_nAdler32_Internal)
{
Additional cleanup Cast to proper types. Set the archive offset only instead of the entire block. The entry could still have its own flags. The only thing needed for saving data is sharing the same offset and size.
2022-06-05 12:28:49 +02:00
Warning(eDLL_T::FS, "Computed checksum '0x%lX' doesn't match expected checksum '0x%lX'. File may be corrupt!\n", m_nAdler32, m_nAdler32_Internal);
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
m_nAdler32 = NULL;
m_nAdler32_Internal = NULL;
Update packedstore.cpp
2022-06-02 01:59:03 +02:00
}
}
//-----------------------------------------------------------------------------
// Purpose: validates extraction result with precomputed CRC32 hash
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
// Input : &svAssetFile -
Update packedstore.cpp
2022-06-02 01:59:03 +02:00
//-----------------------------------------------------------------------------
void CPackedStore::ValidateCRC32PostDecomp(const string& svAssetFile)
{
CIOStream reader(svAssetFile, CIOStream::Mode_t::READ);
Additional VPK system improvements Added DevMsg logs to show the elapsed time of the operation. Fixed post decompress CRC/ADLER error checking. Slight cleanup.
2022-06-04 03:12:46 +02:00
m_nCrc32 = crc32::update(NULL, reader.GetData(), reader.GetSize());
Update packedstore.cpp
2022-06-02 01:59:03 +02:00
if (m_nCrc32 != m_nCrc32_Internal)
{
Additional cleanup Cast to proper types. Set the archive offset only instead of the entire block. The entry could still have its own flags. The only thing needed for saving data is sharing the same offset and size.
2022-06-05 12:28:49 +02:00
Warning(eDLL_T::FS, "Computed checksum '0x%lX' doesn't match expected checksum '0x%lX'. File may be corrupt!\n", m_nCrc32, m_nCrc32_Internal);
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
m_nCrc32 = NULL;
m_nCrc32_Internal = NULL;
Update packedstore.cpp
2022-06-02 01:59:03 +02:00
}
}
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
//-----------------------------------------------------------------------------
// Purpose: packs all files from specified path into VPK file
// Input : &vPair -
// &svPathIn -
// &svPathOut -
// bManifestOnly -
//-----------------------------------------------------------------------------
void CPackedStore::PackAll(const VPKPair_t& vPair, const string& svPathIn, const string& svPathOut, bool bManifestOnly)
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
{
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
CIOStream writer(svPathOut + vPair.m_svBlockName, CIOStream::Mode_t::WRITE);
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
vector<string> vPaths;
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
vector<VPKEntryBlock_t> vEntryBlocks;
CPackedStore::PackAll optimizations * Hash chunk before compression to determine if it already exists or not (don't run compression if it does). * Use the result of vector::insert for data sharing (insert already checks if it exists or not, and returns the iterator). * This optimization saves around 20 seconds total when rebuilding all VPK's.
2022-10-02 02:16:51 +02:00
const nlohmann::json jManifest = GetManifest(svPathIn, GetSourceName(vPair.m_svDirectoryName));
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
Add support for .vpkignore file Folders can be globally excluded using the '.vpkignore' file (placed in the workspace root). To exclude specific files, use the manifest file.
2022-06-06 23:08:53 +02:00
GetIgnoreList(svPathIn);
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
if (bManifestOnly)
{
Update terminology entirely for packedstore.h/.cpp VPK tooling also works on the R1 game (Titanfall 1)
2022-06-06 14:54:22 +02:00
vPaths = GetEntryPaths(svPathIn, jManifest);
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
}
Additional cleanup Cast to proper types. Set the archive offset only instead of the entire block. The entry could still have its own flags. The only thing needed for saving data is sharing the same offset and size.
2022-06-05 12:28:49 +02:00
else // Pack all files in workspace.
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
{
Update terminology entirely for packedstore.h/.cpp VPK tooling also works on the R1 game (Titanfall 1)
2022-06-06 14:54:22 +02:00
vPaths = GetEntryPaths(svPathIn);
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
}
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
VPK system improvements More detailed operation statistics. Discard VPK directory files who's header doesn't match engine requirements. Separated manifest properly between language and context (you can now package for each language and context from a single workspace).
2022-06-05 17:28:39 +02:00
uint64_t nSharedTotal = 0i64;
uint32_t nSharedCount = 0i32;
More VPK compressing/decompressing optimizations * Use reference to array index. * Only obtain array size once during loop. * Use static buffer of 1024*1024 for decompressing. * Removed extraneous goto. * Fixed bug where shared data would be mapped wrong due to the last optimization. * This increased packing performance by 40/60 seconds when rebuilding all vpk's.
2022-10-02 19:24:00 +02:00
for (size_t i = 0, ps = vPaths.size(); i < ps; i++)
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
{
More VPK compressing/decompressing optimizations * Use reference to array index. * Only obtain array size once during loop. * Use static buffer of 1024*1024 for decompressing. * Removed extraneous goto. * Fixed bug where shared data would be mapped wrong due to the last optimization. * This increased packing performance by 40/60 seconds when rebuilding all vpk's.
2022-10-02 19:24:00 +02:00
const string& svPath = vPaths[i];
CIOStream reader(svPath, CIOStream::Mode_t::READ);
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
if (reader.IsReadable())
{
More VPK compressing/decompressing optimizations * Use reference to array index. * Only obtain array size once during loop. * Use static buffer of 1024*1024 for decompressing. * Removed extraneous goto. * Fixed bug where shared data would be mapped wrong due to the last optimization. * This increased packing performance by 40/60 seconds when rebuilding all vpk's.
2022-10-02 19:24:00 +02:00
const string svDestPath = StringReplaceC(svPath, svPathIn, "");
Add support for .vpkignore file Folders can be globally excluded using the '.vpkignore' file (placed in the workspace root). To exclude specific files, use the manifest file.
2022-06-06 23:08:53 +02:00
uint16_t iPreloadSize = 0i16;
Update terminology entirely for packedstore.h/.cpp VPK tooling also works on the R1 game (Titanfall 1)
2022-06-06 14:54:22 +02:00
uint32_t nLoadFlags = static_cast<uint32_t>(EPackedLoadFlags::LOAD_VISIBLE) | static_cast<uint32_t>(EPackedLoadFlags::LOAD_CACHE);
Add support for .vpkignore file Folders can be globally excluded using the '.vpkignore' file (placed in the workspace root). To exclude specific files, use the manifest file.
2022-06-06 23:08:53 +02:00
uint16_t nTextureFlags = static_cast<uint16_t>(EPackedTextureFlags::TEXTURE_DEFAULT); // !TODO: Reverse these.
Implement data sharing algorithm in VPK packing This reduces size of the resulting VPK (checks entry block in hash map). mp_common compresses to 32mb instead of 37.3mb (original is 38.5mb).
2022-06-04 13:55:24 +02:00
bool bUseCompression = true;
bool bUseDataSharing = true;
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
if (!jManifest.is_null())
{
try
{
Additional cleanup Cast to proper types. Set the archive offset only instead of the entire block. The entry could still have its own flags. The only thing needed for saving data is sharing the same offset and size.
2022-06-05 12:28:49 +02:00
nlohmann::json jEntry = jManifest[svDestPath];
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
if (!jEntry.is_null())
{
Add support for .vpkignore file Folders can be globally excluded using the '.vpkignore' file (placed in the workspace root). To exclude specific files, use the manifest file.
2022-06-06 23:08:53 +02:00
iPreloadSize = jEntry.at("preloadSize").get<uint32_t>();
Update terminology entirely for packedstore.h/.cpp VPK tooling also works on the R1 game (Titanfall 1)
2022-06-06 14:54:22 +02:00
nLoadFlags = jEntry.at("loadFlags").get<uint32_t>();
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
nTextureFlags = jEntry.at("textureFlags").get<uint16_t>();
bUseCompression = jEntry.at("useCompression").get<bool>();
Implement data sharing algorithm in VPK packing This reduces size of the resulting VPK (checks entry block in hash map). mp_common compresses to 32mb instead of 37.3mb (original is 38.5mb).
2022-06-04 13:55:24 +02:00
bUseDataSharing = jEntry.at("useDataSharing").get<bool>();
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
}
}
catch (const std::exception& ex)
{
Use correct size type format specifier in packetstore.cpp Using %llu is also correct, but this is the standard approach.
2022-08-08 17:51:09 +02:00
Warning(eDLL_T::FS, "Exception while reading VPK control file: '%s'\n", ex.what());
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
}
}
More VPK compressing/decompressing optimizations * Use reference to array index. * Only obtain array size once during loop. * Use static buffer of 1024*1024 for decompressing. * Removed extraneous goto. * Fixed bug where shared data would be mapped wrong due to the last optimization. * This increased packing performance by 40/60 seconds when rebuilding all vpk's.
2022-10-02 19:24:00 +02:00
DevMsg(eDLL_T::FS, "Packing entry '%zu' ('%s')\n", i, svDestPath.c_str());
Add support for .vpkignore file Folders can be globally excluded using the '.vpkignore' file (placed in the workspace root). To exclude specific files, use the manifest file.
2022-06-06 23:08:53 +02:00
vEntryBlocks.push_back(VPKEntryBlock_t(reader.GetVector(), writer.GetPosition(), iPreloadSize, 0, nLoadFlags, nTextureFlags, svDestPath));
More VPK compressing/decompressing optimizations * Use reference to array index. * Only obtain array size once during loop. * Use static buffer of 1024*1024 for decompressing. * Removed extraneous goto. * Fixed bug where shared data would be mapped wrong due to the last optimization. * This increased packing performance by 40/60 seconds when rebuilding all vpk's.
2022-10-02 19:24:00 +02:00
VPKEntryBlock_t& vEntry = vEntryBlocks[i];
for (size_t j = 0, es = vEntry.m_vChunks.size(); j < es; j++)
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
{
More VPK compressing/decompressing optimizations * Use reference to array index. * Only obtain array size once during loop. * Use static buffer of 1024*1024 for decompressing. * Removed extraneous goto. * Fixed bug where shared data would be mapped wrong due to the last optimization. * This increased packing performance by 40/60 seconds when rebuilding all vpk's.
2022-10-02 19:24:00 +02:00
VPKChunkDescriptor_t& vDescriptor = vEntry.m_vChunks[j];
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
More VPK compressing/decompressing optimizations * Use reference to array index. * Only obtain array size once during loop. * Use static buffer of 1024*1024 for decompressing. * Removed extraneous goto. * Fixed bug where shared data would be mapped wrong due to the last optimization. * This increased packing performance by 40/60 seconds when rebuilding all vpk's.
2022-10-02 19:24:00 +02:00
reader.Read(s_EntryBuf, vDescriptor.m_nUncompressedSize);
vDescriptor.m_nArchiveOffset = writer.GetPosition();
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
CPackedStore::PackAll optimizations * Hash chunk before compression to determine if it already exists or not (don't run compression if it does). * Use the result of vector::insert for data sharing (insert already checks if it exists or not, and returns the iterator). * This optimization saves around 20 seconds total when rebuilding all VPK's.
2022-10-02 02:16:51 +02:00
if (bUseDataSharing)
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
{
More VPK compressing/decompressing optimizations * Use reference to array index. * Only obtain array size once during loop. * Use static buffer of 1024*1024 for decompressing. * Removed extraneous goto. * Fixed bug where shared data would be mapped wrong due to the last optimization. * This increased packing performance by 40/60 seconds when rebuilding all vpk's.
2022-10-02 19:24:00 +02:00
string svEntryHash = sha1(string(reinterpret_cast<char*>(s_EntryBuf), vDescriptor.m_nUncompressedSize));
auto p = m_mChunkHashMap.insert({ svEntryHash, vDescriptor });
CPackedStore::PackAll optimizations * Hash chunk before compression to determine if it already exists or not (don't run compression if it does). * Use the result of vector::insert for data sharing (insert already checks if it exists or not, and returns the iterator). * This optimization saves around 20 seconds total when rebuilding all VPK's.
2022-10-02 02:16:51 +02:00
More VPK compressing/decompressing optimizations * Use reference to array index. * Only obtain array size once during loop. * Use static buffer of 1024*1024 for decompressing. * Removed extraneous goto. * Fixed bug where shared data would be mapped wrong due to the last optimization. * This increased packing performance by 40/60 seconds when rebuilding all vpk's.
2022-10-02 19:24:00 +02:00
if (!p.second) // Map to existing chunk to avoid having copies of the same data.
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
{
CPackedStore::PackAll optimizations * Hash chunk before compression to determine if it already exists or not (don't run compression if it does). * Use the result of vector::insert for data sharing (insert already checks if it exists or not, and returns the iterator). * This optimization saves around 20 seconds total when rebuilding all VPK's.
2022-10-02 02:16:51 +02:00
DevMsg(eDLL_T::FS, "Mapping chunk '%zu' ('%s') to existing chunk at '0x%llx'\n", j, svEntryHash.c_str(), p.first->second.m_nArchiveOffset);
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
More VPK compressing/decompressing optimizations * Use reference to array index. * Only obtain array size once during loop. * Use static buffer of 1024*1024 for decompressing. * Removed extraneous goto. * Fixed bug where shared data would be mapped wrong due to the last optimization. * This increased packing performance by 40/60 seconds when rebuilding all vpk's.
2022-10-02 19:24:00 +02:00
vDescriptor = p.first->second;
nSharedTotal += vDescriptor.m_nCompressedSize;
CPackedStore::PackAll optimizations * Hash chunk before compression to determine if it already exists or not (don't run compression if it does). * Use the result of vector::insert for data sharing (insert already checks if it exists or not, and returns the iterator). * This optimization saves around 20 seconds total when rebuilding all VPK's.
2022-10-02 02:16:51 +02:00
nSharedCount++;
More VPK compressing/decompressing optimizations * Use reference to array index. * Only obtain array size once during loop. * Use static buffer of 1024*1024 for decompressing. * Removed extraneous goto. * Fixed bug where shared data would be mapped wrong due to the last optimization. * This increased packing performance by 40/60 seconds when rebuilding all vpk's.
2022-10-02 19:24:00 +02:00
continue;
Implement data sharing algorithm in VPK packing This reduces size of the resulting VPK (checks entry block in hash map). mp_common compresses to 32mb instead of 37.3mb (original is 38.5mb).
2022-06-04 13:55:24 +02:00
}
}
More VPK compressing/decompressing optimizations * Use reference to array index. * Only obtain array size once during loop. * Use static buffer of 1024*1024 for decompressing. * Removed extraneous goto. * Fixed bug where shared data would be mapped wrong due to the last optimization. * This increased packing performance by 40/60 seconds when rebuilding all vpk's.
2022-10-02 19:24:00 +02:00
if (bUseCompression)
Implement data sharing algorithm in VPK packing This reduces size of the resulting VPK (checks entry block in hash map). mp_common compresses to 32mb instead of 37.3mb (original is 38.5mb).
2022-06-04 13:55:24 +02:00
{
More VPK compressing/decompressing optimizations * Use reference to array index. * Only obtain array size once during loop. * Use static buffer of 1024*1024 for decompressing. * Removed extraneous goto. * Fixed bug where shared data would be mapped wrong due to the last optimization. * This increased packing performance by 40/60 seconds when rebuilding all vpk's.
2022-10-02 19:24:00 +02:00
m_lzCompStatus = lzham_compress_memory(&m_lzCompParams, s_EntryBuf,
&vDescriptor.m_nCompressedSize, s_EntryBuf,
vDescriptor.m_nUncompressedSize, &m_nAdler32_Internal, &m_nCrc32_Internal);
Use static buffer for file chunking and compressing during VPK packing Avoid using new every time. Chunk can be at most 1024 * 1024.
2022-10-02 12:17:03 +02:00
More VPK compressing/decompressing optimizations * Use reference to array index. * Only obtain array size once during loop. * Use static buffer of 1024*1024 for decompressing. * Removed extraneous goto. * Fixed bug where shared data would be mapped wrong due to the last optimization. * This increased packing performance by 40/60 seconds when rebuilding all vpk's.
2022-10-02 19:24:00 +02:00
if (m_lzCompStatus != lzham_compress_status_t::LZHAM_COMP_STATUS_SUCCESS)
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
{
More VPK compressing/decompressing optimizations * Use reference to array index. * Only obtain array size once during loop. * Use static buffer of 1024*1024 for decompressing. * Removed extraneous goto. * Fixed bug where shared data would be mapped wrong due to the last optimization. * This increased packing performance by 40/60 seconds when rebuilding all vpk's.
2022-10-02 19:24:00 +02:00
Warning(eDLL_T::FS, "Status '%d' for chunk '%zu' within entry '%zu' in block '%hu' (chunk packed without compression)\n",
m_lzCompStatus, j, i, vEntryBlocks[i].m_iPackFileIndex);
Use static buffer for file chunking and compressing during VPK packing Avoid using new every time. Chunk can be at most 1024 * 1024.
2022-10-02 12:17:03 +02:00
More VPK compressing/decompressing optimizations * Use reference to array index. * Only obtain array size once during loop. * Use static buffer of 1024*1024 for decompressing. * Removed extraneous goto. * Fixed bug where shared data would be mapped wrong due to the last optimization. * This increased packing performance by 40/60 seconds when rebuilding all vpk's.
2022-10-02 19:24:00 +02:00
vDescriptor.m_nCompressedSize = vDescriptor.m_nUncompressedSize;
}
}
else // Write data uncompressed.
{
vDescriptor.m_nCompressedSize = vDescriptor.m_nUncompressedSize;
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
}
More VPK compressing/decompressing optimizations * Use reference to array index. * Only obtain array size once during loop. * Use static buffer of 1024*1024 for decompressing. * Removed extraneous goto. * Fixed bug where shared data would be mapped wrong due to the last optimization. * This increased packing performance by 40/60 seconds when rebuilding all vpk's.
2022-10-02 19:24:00 +02:00
vDescriptor.m_bIsCompressed = vDescriptor.m_nCompressedSize != vDescriptor.m_nUncompressedSize;
writer.Write(s_EntryBuf, vDescriptor.m_nCompressedSize);
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
}
}
}
Fix spelling errors Overall spelling improvements and cleanup..
2022-09-09 19:47:31 +02:00
DevMsg(eDLL_T::FS, "*** Build block totaling '%zu' bytes with '%zu' shared bytes among '%lu' chunks\n", writer.GetPosition(), nSharedTotal, nSharedCount);
Use static buffer for file chunking and compressing during VPK packing Avoid using new every time. Chunk can be at most 1024 * 1024.
2022-10-02 12:17:03 +02:00
Update terminology entirely for packedstore.h/.cpp VPK tooling also works on the R1 game (Titanfall 1)
2022-06-06 14:54:22 +02:00
m_mChunkHashMap.clear();
More VPK compressing/decompressing optimizations * Use reference to array index. * Only obtain array size once during loop. * Use static buffer of 1024*1024 for decompressing. * Removed extraneous goto. * Fixed bug where shared data would be mapped wrong due to the last optimization. * This increased packing performance by 40/60 seconds when rebuilding all vpk's.
2022-10-02 19:24:00 +02:00
memset(s_EntryBuf, '\0', sizeof(s_EntryBuf));
VPK system improvements More detailed operation statistics. Discard VPK directory files who's header doesn't match engine requirements. Separated manifest properly between language and context (you can now package for each language and context from a single workspace).
2022-06-05 17:28:39 +02:00
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
VPKDir_t vDir = VPKDir_t();
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
vDir.Build(svPathOut + vPair.m_svDirectoryName, vEntryBlocks);
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
}
//-----------------------------------------------------------------------------
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
// Purpose: extracts all files from specified VPK file
Update terminology entirely for packedstore.h/.cpp VPK tooling also works on the R1 game (Titanfall 1)
2022-06-06 14:54:22 +02:00
// Input : &vDir -
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
// &svPathOut -
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
//-----------------------------------------------------------------------------
Update terminology entirely for packedstore.h/.cpp VPK tooling also works on the R1 game (Titanfall 1)
2022-06-06 14:54:22 +02:00
void CPackedStore::UnpackAll(const VPKDir_t& vDir, const string& svPathOut)
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
{
Update terminology entirely for packedstore.h/.cpp VPK tooling also works on the R1 game (Titanfall 1)
2022-06-06 14:54:22 +02:00
if (vDir.m_vHeader.m_nHeaderMarker != VPK_HEADER_MARKER ||
vDir.m_vHeader.m_nMajorVersion != VPK_MAJOR_VERSION ||
vDir.m_vHeader.m_nMinorVersion != VPK_MINOR_VERSION)
VPK system improvements More detailed operation statistics. Discard VPK directory files who's header doesn't match engine requirements. Separated manifest properly between language and context (you can now package for each language and context from a single workspace).
2022-06-05 17:28:39 +02:00
{
Light cleanup Use NO_ERROR instead of NULL for error code parameter passed to Error(..).
2022-09-14 01:14:51 +02:00
Error(eDLL_T::FS, NO_ERROR, "Unsupported VPK directory file (invalid header criteria)\n");
VPK system improvements More detailed operation statistics. Discard VPK directory files who's header doesn't match engine requirements. Separated manifest properly between language and context (you can now package for each language and context from a single workspace).
2022-06-05 17:28:39 +02:00
return;
}
Update terminology entirely for packedstore.h/.cpp VPK tooling also works on the R1 game (Titanfall 1)
2022-06-06 14:54:22 +02:00
BuildManifest(vDir.m_vEntryBlocks, svPathOut, GetSourceName(vDir.m_svDirPath));
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
More VPK compressing/decompressing optimizations * Use reference to array index. * Only obtain array size once during loop. * Use static buffer of 1024*1024 for decompressing. * Removed extraneous goto. * Fixed bug where shared data would be mapped wrong due to the last optimization. * This increased packing performance by 40/60 seconds when rebuilding all vpk's.
2022-10-02 19:24:00 +02:00
for (size_t i = 0, fs = vDir.m_vPackFile.size(); i < fs; i++)
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
{
Update terminology entirely for packedstore.h/.cpp VPK tooling also works on the R1 game (Titanfall 1)
2022-06-06 14:54:22 +02:00
fs::path fspVpkPath(vDir.m_svDirPath);
string svPath = fspVpkPath.parent_path().u8string() + '\\' + vDir.m_vPackFile[i];
Additional VPK system improvements Added DevMsg logs to show the elapsed time of the operation. Fixed post decompress CRC/ADLER error checking. Slight cleanup.
2022-06-04 03:12:46 +02:00
CIOStream iStream(svPath, CIOStream::Mode_t::READ); // Create stream to read from each archive.
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
More VPK compressing/decompressing optimizations * Use reference to array index. * Only obtain array size once during loop. * Use static buffer of 1024*1024 for decompressing. * Removed extraneous goto. * Fixed bug where shared data would be mapped wrong due to the last optimization. * This increased packing performance by 40/60 seconds when rebuilding all vpk's.
2022-10-02 19:24:00 +02:00
for (size_t j = 0, es = vDir.m_vEntryBlocks.size(); j < es; j++)
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
{
More VPK compressing/decompressing optimizations * Use reference to array index. * Only obtain array size once during loop. * Use static buffer of 1024*1024 for decompressing. * Removed extraneous goto. * Fixed bug where shared data would be mapped wrong due to the last optimization. * This increased packing performance by 40/60 seconds when rebuilding all vpk's.
2022-10-02 19:24:00 +02:00
const VPKEntryBlock_t& vBlock = vDir.m_vEntryBlocks[j];
if (vBlock.m_iPackFileIndex != static_cast<uint16_t>(i))
Additional VPK system improvements Added DevMsg logs to show the elapsed time of the operation. Fixed post decompress CRC/ADLER error checking. Slight cleanup.
2022-06-04 03:12:46 +02:00
{
More VPK compressing/decompressing optimizations * Use reference to array index. * Only obtain array size once during loop. * Use static buffer of 1024*1024 for decompressing. * Removed extraneous goto. * Fixed bug where shared data would be mapped wrong due to the last optimization. * This increased packing performance by 40/60 seconds when rebuilding all vpk's.
2022-10-02 19:24:00 +02:00
continue;
Additional VPK system improvements Added DevMsg logs to show the elapsed time of the operation. Fixed post decompress CRC/ADLER error checking. Slight cleanup.
2022-06-04 03:12:46 +02:00
}
Cast and format to proper types
2022-06-04 03:19:05 +02:00
else // Chunk belongs to this block.
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
{
More VPK compressing/decompressing optimizations * Use reference to array index. * Only obtain array size once during loop. * Use static buffer of 1024*1024 for decompressing. * Removed extraneous goto. * Fixed bug where shared data would be mapped wrong due to the last optimization. * This increased packing performance by 40/60 seconds when rebuilding all vpk's.
2022-10-02 19:24:00 +02:00
string svFilePath = CreateDirectories(svPathOut + vBlock.m_svEntryPath);
Additional VPK system improvements Added DevMsg logs to show the elapsed time of the operation. Fixed post decompress CRC/ADLER error checking. Slight cleanup.
2022-06-04 03:12:46 +02:00
CIOStream oStream(svFilePath, CIOStream::Mode_t::WRITE);
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
Additional VPK system improvements Added DevMsg logs to show the elapsed time of the operation. Fixed post decompress CRC/ADLER error checking. Slight cleanup.
2022-06-04 03:12:46 +02:00
if (!oStream.IsWritable())
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
{
Light cleanup Use NO_ERROR instead of NULL for error code parameter passed to Error(..).
2022-09-14 01:14:51 +02:00
Error(eDLL_T::FS, NO_ERROR, "Unable to write file '%s'\n", svFilePath.c_str());
Additional VPK system improvements Added DevMsg logs to show the elapsed time of the operation. Fixed post decompress CRC/ADLER error checking. Slight cleanup.
2022-06-04 03:12:46 +02:00
continue;
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
}
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
More VPK compressing/decompressing optimizations * Use reference to array index. * Only obtain array size once during loop. * Use static buffer of 1024*1024 for decompressing. * Removed extraneous goto. * Fixed bug where shared data would be mapped wrong due to the last optimization. * This increased packing performance by 40/60 seconds when rebuilding all vpk's.
2022-10-02 19:24:00 +02:00
DevMsg(eDLL_T::FS, "Unpacking entry '%zu' from block '%zu' ('%s')\n", j, i, vBlock.m_svEntryPath.c_str());
for (size_t k = 0, cs = vBlock.m_vChunks.size(); k < cs; k++)
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
{
More VPK compressing/decompressing optimizations * Use reference to array index. * Only obtain array size once during loop. * Use static buffer of 1024*1024 for decompressing. * Removed extraneous goto. * Fixed bug where shared data would be mapped wrong due to the last optimization. * This increased packing performance by 40/60 seconds when rebuilding all vpk's.
2022-10-02 19:24:00 +02:00
const VPKChunkDescriptor_t& vChunk = vBlock.m_vChunks[k];
Update terminology entirely for packedstore.h/.cpp VPK tooling also works on the R1 game (Titanfall 1)
2022-06-06 14:54:22 +02:00
m_nChunkCount++;
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
Update terminology entirely for packedstore.h/.cpp VPK tooling also works on the R1 game (Titanfall 1)
2022-06-06 14:54:22 +02:00
iStream.SetPosition(vChunk.m_nArchiveOffset);
More VPK compressing/decompressing optimizations * Use reference to array index. * Only obtain array size once during loop. * Use static buffer of 1024*1024 for decompressing. * Removed extraneous goto. * Fixed bug where shared data would be mapped wrong due to the last optimization. * This increased packing performance by 40/60 seconds when rebuilding all vpk's.
2022-10-02 19:24:00 +02:00
iStream.Read(s_EntryBuf, vChunk.m_nCompressedSize);
Additional VPK system improvements Added DevMsg logs to show the elapsed time of the operation. Fixed post decompress CRC/ADLER error checking. Slight cleanup.
2022-06-04 03:12:46 +02:00
Update terminology entirely for packedstore.h/.cpp VPK tooling also works on the R1 game (Titanfall 1)
2022-06-06 14:54:22 +02:00
if (vChunk.m_bIsCompressed)
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
{
More VPK compressing/decompressing optimizations * Use reference to array index. * Only obtain array size once during loop. * Use static buffer of 1024*1024 for decompressing. * Removed extraneous goto. * Fixed bug where shared data would be mapped wrong due to the last optimization. * This increased packing performance by 40/60 seconds when rebuilding all vpk's.
2022-10-02 19:24:00 +02:00
size_t nDstLen = sizeof(s_DecompBuf);
assert(vChunk.m_nCompressedSize <= nDstLen);
if (vChunk.m_nCompressedSize > nDstLen)
break; // Corrupt or invalid chunk descriptor.
m_lzDecompStatus = lzham_decompress_memory(&m_lzDecompParams, s_DecompBuf,
&nDstLen, s_EntryBuf, vChunk.m_nCompressedSize, &m_nAdler32_Internal, &m_nCrc32_Internal);
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
if (m_lzDecompStatus != lzham_decompress_status_t::LZHAM_DECOMP_STATUS_SUCCESS)
{
Light cleanup Use NO_ERROR instead of NULL for error code parameter passed to Error(..).
2022-09-14 01:14:51 +02:00
Error(eDLL_T::FS, NO_ERROR, "Status '%d' for chunk '%zu' within entry '%zu' in block '%hu' (chunk not decompressed)\n",
More VPK compressing/decompressing optimizations * Use reference to array index. * Only obtain array size once during loop. * Use static buffer of 1024*1024 for decompressing. * Removed extraneous goto. * Fixed bug where shared data would be mapped wrong due to the last optimization. * This increased packing performance by 40/60 seconds when rebuilding all vpk's.
2022-10-02 19:24:00 +02:00
m_lzDecompStatus, m_nChunkCount, i, vBlock.m_iPackFileIndex);
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
}
Additional VPK system improvements Added DevMsg logs to show the elapsed time of the operation. Fixed post decompress CRC/ADLER error checking. Slight cleanup.
2022-06-04 03:12:46 +02:00
else // If successfully decompressed, write to file.
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
{
More VPK compressing/decompressing optimizations * Use reference to array index. * Only obtain array size once during loop. * Use static buffer of 1024*1024 for decompressing. * Removed extraneous goto. * Fixed bug where shared data would be mapped wrong due to the last optimization. * This increased packing performance by 40/60 seconds when rebuilding all vpk's.
2022-10-02 19:24:00 +02:00
oStream.Write(s_DecompBuf, nDstLen);
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
}
}
More VPK compressing/decompressing optimizations * Use reference to array index. * Only obtain array size once during loop. * Use static buffer of 1024*1024 for decompressing. * Removed extraneous goto. * Fixed bug where shared data would be mapped wrong due to the last optimization. * This increased packing performance by 40/60 seconds when rebuilding all vpk's.
2022-10-02 19:24:00 +02:00
else // If not compressed, write source data into output file.
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
{
More VPK compressing/decompressing optimizations * Use reference to array index. * Only obtain array size once during loop. * Use static buffer of 1024*1024 for decompressing. * Removed extraneous goto. * Fixed bug where shared data would be mapped wrong due to the last optimization. * This increased packing performance by 40/60 seconds when rebuilding all vpk's.
2022-10-02 19:24:00 +02:00
oStream.Write(s_EntryBuf, vChunk.m_nUncompressedSize);
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
}
}
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
More VPK compressing/decompressing optimizations * Use reference to array index. * Only obtain array size once during loop. * Use static buffer of 1024*1024 for decompressing. * Removed extraneous goto. * Fixed bug where shared data would be mapped wrong due to the last optimization. * This increased packing performance by 40/60 seconds when rebuilding all vpk's.
2022-10-02 19:24:00 +02:00
if (m_nChunkCount == vBlock.m_vChunks.size()) // Only validate after last entry in block had been written.
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
{
Update terminology entirely for packedstore.h/.cpp VPK tooling also works on the R1 game (Titanfall 1)
2022-06-06 14:54:22 +02:00
m_nChunkCount = 0;
More VPK compressing/decompressing optimizations * Use reference to array index. * Only obtain array size once during loop. * Use static buffer of 1024*1024 for decompressing. * Removed extraneous goto. * Fixed bug where shared data would be mapped wrong due to the last optimization. * This increased packing performance by 40/60 seconds when rebuilding all vpk's.
2022-10-02 19:24:00 +02:00
m_nCrc32_Internal = vBlock.m_nFileCRC;
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
Implement data sharing algorithm in VPK packing This reduces size of the resulting VPK (checks entry block in hash map). mp_common compresses to 32mb instead of 37.3mb (original is 38.5mb).
2022-06-04 13:55:24 +02:00
oStream.Flush();
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
ValidateCRC32PostDecomp(svFilePath);
}
More VPK compressing/decompressing optimizations * Use reference to array index. * Only obtain array size once during loop. * Use static buffer of 1024*1024 for decompressing. * Removed extraneous goto. * Fixed bug where shared data would be mapped wrong due to the last optimization. * This increased packing performance by 40/60 seconds when rebuilding all vpk's.
2022-10-02 19:24:00 +02:00
}
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
}
}
More VPK compressing/decompressing optimizations * Use reference to array index. * Only obtain array size once during loop. * Use static buffer of 1024*1024 for decompressing. * Removed extraneous goto. * Fixed bug where shared data would be mapped wrong due to the last optimization. * This increased packing performance by 40/60 seconds when rebuilding all vpk's.
2022-10-02 19:24:00 +02:00
memset(s_EntryBuf, '\0', sizeof(s_EntryBuf));
memset(s_DecompBuf, '\0', sizeof(s_DecompBuf));
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
}
//-----------------------------------------------------------------------------
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
// Purpose: 'VPKEntryBlock_t' file constructor
// Input : *pReader -
Add support for .vpkignore file Folders can be globally excluded using the '.vpkignore' file (placed in the workspace root). To exclude specific files, use the manifest file.
2022-06-06 23:08:53 +02:00
// svEntryPath -
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
//-----------------------------------------------------------------------------
Update terminology entirely for packedstore.h/.cpp VPK tooling also works on the R1 game (Titanfall 1)
2022-06-06 14:54:22 +02:00
VPKEntryBlock_t::VPKEntryBlock_t(CIOStream* pReader, string svEntryPath)
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
{
Update terminology entirely for packedstore.h/.cpp VPK tooling also works on the R1 game (Titanfall 1)
2022-06-06 14:54:22 +02:00
StringReplace(svEntryPath, "\\", "/"); // Flip windows-style backslash to forward slash.
StringReplace(svEntryPath, " /", "" ); // Remove space character representing VPK root.
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
Update terminology entirely for packedstore.h/.cpp VPK tooling also works on the R1 game (Titanfall 1)
2022-06-06 14:54:22 +02:00
this->m_svEntryPath = svEntryPath; // Set the entry path.
Add support for .vpkignore file Folders can be globally excluded using the '.vpkignore' file (placed in the workspace root). To exclude specific files, use the manifest file.
2022-06-06 23:08:53 +02:00
pReader->Read<uint32_t>(this->m_nFileCRC); //
pReader->Read<uint16_t>(this->m_iPreloadSize); //
Update terminology entirely for packedstore.h/.cpp VPK tooling also works on the R1 game (Titanfall 1)
2022-06-06 14:54:22 +02:00
pReader->Read<uint16_t>(this->m_iPackFileIndex); //
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
Update terminology entirely for packedstore.h/.cpp VPK tooling also works on the R1 game (Titanfall 1)
2022-06-06 14:54:22 +02:00
do // Loop through all chunks in the entry and push them to the vector.
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
{
Update terminology entirely for packedstore.h/.cpp VPK tooling also works on the R1 game (Titanfall 1)
2022-06-06 14:54:22 +02:00
VPKChunkDescriptor_t entry(pReader);
this->m_vChunks.push_back(entry);
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
} while (pReader->Read<uint16_t>() != UINT16_MAX);
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
}
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
//-----------------------------------------------------------------------------
// Purpose: 'VPKEntryBlock_t' memory constructor
// Input : &vData -
// nOffset -
Add support for .vpkignore file Folders can be globally excluded using the '.vpkignore' file (placed in the workspace root). To exclude specific files, use the manifest file.
2022-06-06 23:08:53 +02:00
// nPreloadSize -
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
// nArchiveIndex -
Add support for .vpkignore file Folders can be globally excluded using the '.vpkignore' file (placed in the workspace root). To exclude specific files, use the manifest file.
2022-06-06 23:08:53 +02:00
// nLoadFlags -
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
// nTextureFlags -
Add support for .vpkignore file Folders can be globally excluded using the '.vpkignore' file (placed in the workspace root). To exclude specific files, use the manifest file.
2022-06-06 23:08:53 +02:00
// &svBlockPath -
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
//-----------------------------------------------------------------------------
Additional VPK lib cleanup and performance improvements
2022-10-09 12:08:54 +02:00
VPKEntryBlock_t::VPKEntryBlock_t(const vector<uint8_t> &vData, int64_t nOffset, uint16_t nPreloadSize,
uint16_t nArchiveIndex, uint32_t nLoadFlags, uint16_t nTextureFlags, const string& svEntryPath)
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
{
Light cleanup
2022-09-09 02:28:03 +02:00
m_nFileCRC = crc32::update(NULL, vData.data(), vData.size());
Add support for .vpkignore file Folders can be globally excluded using the '.vpkignore' file (placed in the workspace root). To exclude specific files, use the manifest file.
2022-06-06 23:08:53 +02:00
m_iPreloadSize = nPreloadSize;
Update terminology entirely for packedstore.h/.cpp VPK tooling also works on the R1 game (Titanfall 1)
2022-06-06 14:54:22 +02:00
m_iPackFileIndex = nArchiveIndex;
m_svEntryPath = svEntryPath;
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
Fix return bug CPackedStore::GetManifest() did not return a value in all control paths. Use proper size types for entry block constructor.
2022-06-06 00:21:13 +02:00
size_t nEntryCount = (vData.size() + ENTRY_MAX_LEN - 1) / ENTRY_MAX_LEN;
size_t nDataSize = vData.size();
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
int64_t nCurrentOffset = nOffset;
VPK system improvements More detailed operation statistics. Discard VPK directory files who's header doesn't match engine requirements. Separated manifest properly between language and context (you can now package for each language and context from a single workspace).
2022-06-05 17:28:39 +02:00
Additional VPK lib cleanup and performance improvements
2022-10-09 12:08:54 +02:00
for (size_t i = 0; i < nEntryCount; i++) // Fragment data into 1 MiB chunks.
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
{
Fix return bug CPackedStore::GetManifest() did not return a value in all control paths. Use proper size types for entry block constructor.
2022-06-06 00:21:13 +02:00
size_t nSize = std::min<uint64_t>(ENTRY_MAX_LEN, nDataSize);
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
nDataSize -= nSize;
Add support for .vpkignore file Folders can be globally excluded using the '.vpkignore' file (placed in the workspace root). To exclude specific files, use the manifest file.
2022-06-06 23:08:53 +02:00
m_vChunks.push_back(VPKChunkDescriptor_t(nLoadFlags, nTextureFlags, nCurrentOffset, nSize, nSize));
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
nCurrentOffset += nSize;
}
}
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
//-----------------------------------------------------------------------------
Update terminology entirely for packedstore.h/.cpp VPK tooling also works on the R1 game (Titanfall 1)
2022-06-06 14:54:22 +02:00
// Purpose: 'VPKChunkDescriptor_t' file constructor
VPK system improvements More detailed operation statistics. Discard VPK directory files who's header doesn't match engine requirements. Separated manifest properly between language and context (you can now package for each language and context from a single workspace).
2022-06-05 17:28:39 +02:00
// Input : *pReader -
//-----------------------------------------------------------------------------
Update terminology entirely for packedstore.h/.cpp VPK tooling also works on the R1 game (Titanfall 1)
2022-06-06 14:54:22 +02:00
VPKChunkDescriptor_t::VPKChunkDescriptor_t(CIOStream* pReader)
VPK system improvements More detailed operation statistics. Discard VPK directory files who's header doesn't match engine requirements. Separated manifest properly between language and context (you can now package for each language and context from a single workspace).
2022-06-05 17:28:39 +02:00
{
Update terminology entirely for packedstore.h/.cpp VPK tooling also works on the R1 game (Titanfall 1)
2022-06-06 14:54:22 +02:00
pReader->Read<uint32_t>(this->m_nLoadFlags); //
VPK system improvements More detailed operation statistics. Discard VPK directory files who's header doesn't match engine requirements. Separated manifest properly between language and context (you can now package for each language and context from a single workspace).
2022-06-05 17:28:39 +02:00
pReader->Read<uint16_t>(this->m_nTextureFlags); //
pReader->Read<uint64_t>(this->m_nArchiveOffset); //
pReader->Read<uint64_t>(this->m_nCompressedSize); //
pReader->Read<uint64_t>(this->m_nUncompressedSize); //
this->m_bIsCompressed = (this->m_nCompressedSize != this->m_nUncompressedSize);
}
//-----------------------------------------------------------------------------
Update terminology entirely for packedstore.h/.cpp VPK tooling also works on the R1 game (Titanfall 1)
2022-06-06 14:54:22 +02:00
// Purpose: 'VPKChunkDescriptor_t' memory constructor
Add support for .vpkignore file Folders can be globally excluded using the '.vpkignore' file (placed in the workspace root). To exclude specific files, use the manifest file.
2022-06-06 23:08:53 +02:00
// Input : nLoadFlags -
// nTextureFlags -
// nArchiveOffset -
// nCompressedSize -
// nUncompressedSize -
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
//-----------------------------------------------------------------------------
Additional VPK lib cleanup and performance improvements
2022-10-09 12:08:54 +02:00
VPKChunkDescriptor_t::VPKChunkDescriptor_t(uint32_t nLoadFlags, uint16_t nTextureFlags,
uint64_t nArchiveOffset, uint64_t nCompressedSize, uint64_t nUncompressedSize)
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
{
Update terminology entirely for packedstore.h/.cpp VPK tooling also works on the R1 game (Titanfall 1)
2022-06-06 14:54:22 +02:00
m_nLoadFlags = nLoadFlags;
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
m_nTextureFlags = nTextureFlags;
m_nArchiveOffset = nArchiveOffset;
m_nCompressedSize = nCompressedSize;
m_nUncompressedSize = nUncompressedSize;
}
//-----------------------------------------------------------------------------
// Purpose: 'VPKDir_t' file constructor
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
// Input : &szPath -
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
//-----------------------------------------------------------------------------
VPKDir_t::VPKDir_t(const string& svPath)
{
CIOStream reader(svPath, CIOStream::Mode_t::READ);
VPK system improvements More detailed operation statistics. Discard VPK directory files who's header doesn't match engine requirements. Separated manifest properly between language and context (you can now package for each language and context from a single workspace).
2022-06-05 17:28:39 +02:00
reader.Read<uint32_t>(this->m_vHeader.m_nHeaderMarker);
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
reader.Read<uint16_t>(this->m_vHeader.m_nMajorVersion); //
reader.Read<uint16_t>(this->m_vHeader.m_nMinorVersion); //
reader.Read<uint32_t>(this->m_vHeader.m_nDirectorySize); //
reader.Read<uint32_t>(this->m_nFileDataSize); //
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
Update terminology entirely for packedstore.h/.cpp VPK tooling also works on the R1 game (Titanfall 1)
2022-06-06 14:54:22 +02:00
this->m_vEntryBlocks = g_pPackedStore->GetEntryBlocks(&reader);
this->m_svDirPath = svPath; // Set path to vpk directory file.
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
Update terminology entirely for packedstore.h/.cpp VPK tooling also works on the R1 game (Titanfall 1)
2022-06-06 14:54:22 +02:00
for (VPKEntryBlock_t vEntry : this->m_vEntryBlocks)
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
{
Update terminology entirely for packedstore.h/.cpp VPK tooling also works on the R1 game (Titanfall 1)
2022-06-06 14:54:22 +02:00
if (vEntry.m_iPackFileIndex > this->m_iPackFileCount)
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
{
Update terminology entirely for packedstore.h/.cpp VPK tooling also works on the R1 game (Titanfall 1)
2022-06-06 14:54:22 +02:00
this->m_iPackFileCount = vEntry.m_iPackFileIndex;
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
}
}
Update terminology entirely for packedstore.h/.cpp VPK tooling also works on the R1 game (Titanfall 1)
2022-06-06 14:54:22 +02:00
for (uint16_t i = 0; i < this->m_iPackFileCount + 1; i++)
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
{
Update terminology entirely for packedstore.h/.cpp VPK tooling also works on the R1 game (Titanfall 1)
2022-06-06 14:54:22 +02:00
string svArchivePath = g_pPackedStore->GetPackFile(svPath, i);
this->m_vPackFile.push_back(svArchivePath);
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
}
}
//-----------------------------------------------------------------------------
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
// Purpose: builds the vpk directory file
// Input : &svDirectoryFile -
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
// &vEntryBlocks -
//-----------------------------------------------------------------------------
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
void VPKDir_t::Build(const string& svDirectoryFile, const vector<VPKEntryBlock_t>& vEntryBlocks)
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
{
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
CIOStream writer(svDirectoryFile, CIOStream::Mode_t::WRITE);
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
auto vMap = std::map<string, std::map<string, std::list<VPKEntryBlock_t>>>();
Additional VPK lib cleanup and performance improvements
2022-10-09 12:08:54 +02:00
uint64_t nDescriptors = 0;
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
writer.Write<uint32_t>(this->m_vHeader.m_nHeaderMarker);
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
writer.Write<uint16_t>(this->m_vHeader.m_nMajorVersion);
writer.Write<uint16_t>(this->m_vHeader.m_nMinorVersion);
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
writer.Write<uint32_t>(this->m_vHeader.m_nDirectorySize);
Add support for .vpkignore file Folders can be globally excluded using the '.vpkignore' file (placed in the workspace root). To exclude specific files, use the manifest file.
2022-06-06 23:08:53 +02:00
writer.Write<uint32_t>(this->m_vHeader.m_nSignatureSize);
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
for (VPKEntryBlock_t vBlock : vEntryBlocks)
{
Update terminology entirely for packedstore.h/.cpp VPK tooling also works on the R1 game (Titanfall 1)
2022-06-06 14:54:22 +02:00
string svExtension = GetExtension(vBlock.m_svEntryPath);
string svFilePath = RemoveFileName(vBlock.m_svEntryPath);
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
if (svFilePath.empty())
{
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
svFilePath = ' '; // Has to be padded with a space character if empty [root].
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
}
if (!vMap.count(svExtension))
{
vMap.insert({ svExtension, std::map<string, std::list<VPKEntryBlock_t>>() });
}
if (!vMap[svExtension].count(svFilePath))
{
vMap[svExtension].insert({ svFilePath, std::list<VPKEntryBlock_t>() });
}
vMap[svExtension][svFilePath].push_back(vBlock);
}
for (auto& iKeyValue : vMap)
{
writer.WriteString(iKeyValue.first);
for (auto& jKeyValue : iKeyValue.second)
{
writer.WriteString(jKeyValue.first);
VPK system improvements More detailed operation statistics. Discard VPK directory files who's header doesn't match engine requirements. Separated manifest properly between language and context (you can now package for each language and context from a single workspace).
2022-06-05 17:28:39 +02:00
for (auto& vEntry : jKeyValue.second)
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
{
Additional VPK lib cleanup and performance improvements
2022-10-09 12:08:54 +02:00
/*Write entry block*/
Update terminology entirely for packedstore.h/.cpp VPK tooling also works on the R1 game (Titanfall 1)
2022-06-06 14:54:22 +02:00
writer.WriteString(GetFileName(vEntry.m_svEntryPath, true));
Additional VPK lib cleanup and performance improvements
2022-10-09 12:08:54 +02:00
writer.Write(vEntry.m_nFileCRC);
writer.Write(vEntry.m_iPreloadSize);
writer.Write(vEntry.m_iPackFileIndex);
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
Additional VPK lib cleanup and performance improvements
2022-10-09 12:08:54 +02:00
for (size_t i = 0, nc = vEntry.m_vChunks.size(); i < nc; i++)
{
/*Write chunk descriptor*/
const VPKChunkDescriptor_t* pDescriptor = &vEntry.m_vChunks[i];
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
Additional VPK lib cleanup and performance improvements
2022-10-09 12:08:54 +02:00
writer.Write(pDescriptor->m_nLoadFlags);
writer.Write(pDescriptor->m_nTextureFlags);
writer.Write(pDescriptor->m_nArchiveOffset);
writer.Write(pDescriptor->m_nCompressedSize);
writer.Write(pDescriptor->m_nUncompressedSize);
if (i != (nc - 1))
{
const ushort s = 0;
writer.Write(s);
}
else
{
const ushort s = UINT16_MAX;
writer.Write(s);
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
}
Additional VPK lib cleanup and performance improvements
2022-10-09 12:08:54 +02:00
nDescriptors++;
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
}
}
writer.Write<uint8_t>('\0');
}
writer.Write<uint8_t>('\0');
}
writer.Write<uint8_t>('\0');
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
m_vHeader.m_nDirectorySize = static_cast<uint32_t>(writer.GetPosition() - sizeof(VPKDirHeader_t));
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
Proper VPK repacking Initial proper implementation pending cleanup. The new system builds a manifest file when a VPK is unpacked. The manifest files contains data such as the entry flags and texture flags. It also contains a field determining whether the file should be compressed or not. When a user repacks a pack, the system attempts to load this manifest file and does a lookup to the object to retrieve the flags (most of these flags are unknown, but they are used by the engine and are necessary for stuff like cubemaps and texture files to work correctly. Cubemaps won't work with proper flags, and textures (decals, particle system components, etc..) will look washed out without them. I think some also determine whether a file within the VPK should be cached or not, so simply marking everything as 0x101 will probably end up in more CPU time and higher filesystem cache usage (depot/ is only 0x1, I don't think anything there is getting cached ever without the 0x100 flag). User could also repack a VPK while excluding anything that is not in the manifest file. So you could unpack all VPK's into a single directory (each VPK has its own manifest file tied to its level name), and rebuild all the VPK's with only the files that where originally in them. fs_pack_vpk command usage: <locale> <context> <level_name> <manifest_only> locale determines the pak language (default english), context determines whether is a server/client vpk, level_name determines the BSP name of the pak, manifest_only determines whether the pack system should only include files within the manifest (leaving this arg out will build all files into the vpk). The VPK workspace path is determined with ConVar 'fs_packedstore_workspace'.
2022-06-04 01:08:23 +02:00
writer.SetPosition(offsetof(VPKDir_t, m_vHeader.m_nDirectorySize));
writer.Write(this->m_vHeader.m_nDirectorySize);
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
writer.Write(0);
VPK system improvements More detailed operation statistics. Discard VPK directory files who's header doesn't match engine requirements. Separated manifest properly between language and context (you can now package for each language and context from a single workspace).
2022-06-05 17:28:39 +02:00
Fix spelling errors Overall spelling improvements and cleanup..
2022-09-09 19:47:31 +02:00
DevMsg(eDLL_T::FS, "*** Build directory totaling '%zu' bytes with '%zu' entries and '%zu' descriptors\n",
Use correct size type format specifier in packetstore.cpp Using %llu is also correct, but this is the standard approach.
2022-08-08 17:51:09 +02:00
size_t(sizeof(VPKDirHeader_t) + m_vHeader.m_nDirectorySize), vEntryBlocks.size(), nDescriptors);
Update packedstore.cpp
2022-06-02 02:00:35 +02:00
}
Update packedstore.cpp
2022-06-02 01:59:03 +02:00
///////////////////////////////////////////////////////////////////////////////
CPackedStore* g_pPackedStore = new CPackedStore();
Reference in New Issue Copy Permalink