Most of the pak load functions are now reverse engineered, documented and ready to be used from the API handle. Structure is confirmed to align with the engine.
FS_OpenAsyncFile is inline in StreamDB_Init, and in our implementation, FS_OpenAsyncFile is the only place channel id's get mapped to file slots, so it will be null for stbsp's. Commented assert out as there isn't anything we could do about this besides rebuilding StreamDB_Init which is a large task.
The engine assigns channel ID's to paks, which it then uses to determine whether to error out or warn when a certain pak file failed to load.
Paks loaded with a higher channel ID, such as the load screens or gladiator cards, won't call Error() in the engine if the loading thereof failed, where paks with a lower channel ID, such as common.rpak will.
Paks with a higher channel ID are quite verbose in the console, as the game often switches between gladiator card rpaks from scripts to update the banners in the world dynamically. User can now explicitly set the channel ID, with the default being 4 which only logs core engine and sdk paks, along with preloaded and level paks.
The async file read system uses a similar concept, this was confirmed when reversing Pak_SetupBuffersAndLoad as this calls FS_OpenAsyncFile for its patch files with the log channel that was assigned during Pak_LoadAsync. This code already had logging, but has been adjusted to follow the design of the system. Async file reads however are disabled by default as these are very verbose and will degrade performance of the game.
Change callbacks actually take a structure of 2 pointers, one being for the callback itself, and the other being 'userdata' which typically is used to sync the ConVar with VGUI slider elements. This issue was noticed after implementing the ADS scalars and attempting to hook them up to VGUI, only to find out it would crash due to this lacking detail. All change callback prototypes had to be adjusted.
Mutex is technically no longer needed as imgui now draws in main and gets buffer swapped / rendered in the render thread just like the rest of the game. But the mutex was still in because of the shared script function 'Script_GetAvailablePlaylists' that can be used on the server. Both the 'playlist_reload' and 'downloadPlaylists' commands will now join the active server job if ran to prevent crashing in the engine or sdk when ran during the game.
There was also a performance difference between 'playlist_reload' and 'downloadPlaylists' while the commands are identical. There was a bug where the 'playlist_reload' one called 'Playlists_SDKInit' twice. This has also been fixed in this commit.
During beta tests on a Linux system, we encountered an issue where the I/O could be slower and thus result in not enough bytes being streamed by the time we invoke the decoder (lenStreamed == bufSizeNeeded at this point), after 16 calls with lenStreamed == bufSizeNeeded, the engine errors as ZSTD_NO_FORWARD_PROGRESS_MAX would be reached. Added value of 'ZSTD_nextSrcSizeToDecompress()' to 'PakDecoder_s::bufferSizeNeeded' to make sure we never call the decoder without any new streamed bytes. Also increased the value of ZSTD_NO_FORWARD_PROGRESS_MAX to 1024 since this fixed the issue without applying the aforementioned patch, this was increased as a hardening measure.
Implemented CustomEvent in code, which supports:
- bool|int|float|string|vector|array|table
- nested arrays and tables, up to a depth of 64
Also improved foundation code for LiveAPI:
- added ability to log liveapi events to a file on the disk (rotates between each match or round, depending on how the abstracted functions are called in scripts)
- when the system is enabled through cvars, code will be invoked on the fly
- when the system is disabled through cvars, the system will be shutdown properly on the fly (properly handling socket closing, log file finishing, etc)
- if the socket system is enabled/disabled on the fly using cvars, related code will be called to initiate or shutdown the connections.
The generated proto.cpp/h file has been moved to the protoc project as it was causing some compiler warnings that we suppress on the thirdparty (vendored) code.
Marked convars/concommands as FCVAR_SERVER_FRAME_THREAD and removed main thread dispatching code and comments that are no longer in effect. In the RCONServer execute handler, a ThreadJoinServerJob() is placed if the commandbase is flagged FCVAR_SERVER_FRAME_THREAD since RCON dispatches the command, or sets the convar directly.
The WebSocket system is now fully split up. The global network module initialization is moved to DLL init so the websocket class could be used for anything. The LiveAPI system now also mostly runs in the server frame thread, the convar change callbacks still need to be latched to the server frame thread however.
Fully implemented ConVar class so we could statically construct all SDK convars, this avoids a level of indirection, and allows for creating ConVar's everywhere in the project.
This patch also removed the settings tab of the ImGui server browser, as it has threading issues, while it technically never caused a crash yet, it has been removed as there was no point keeping it vs the work required to make it thread save (it only managed 2 convars which are perfectly manageable through cfg's or the in-game console).
Also temporarily disabled the creation of ConVar's in the mod system due to a memory leak, we would allocate and register a convar based on details parsed out of a mod file definition, but never unregister and free it.
Properly implement the ConCommandBase and ConCommand classes so we could statically construct all ConCommand objects in the global scope of each translation unit, this way we don't need to put them in a global file and deal with preprocessor directives to compile then in/out for certain projects.
Add missing types to fully mapped out PakGlobals_s struct, which is a 13MiB+ structure! This covers the vast majority of the pakfile system which is why we had to change a bunch of stuff for this patch. This patch also comes with:
- Reversed 'JobFifoLock_s' structure
- Reversed 'PakTracker_s' structure
- Reversed 'PakAssetTracker_s' structure
Many globals have been dropped as they were covered by the large PakGlobals_s singleton.
The pak decoder logic has been changed up as well, we now use a decode mode enumerant which will make it easier to add in more decoders for the pak files in the future.
Pak files above 4GB caused the async file system to throw an error about failing to load audio banks, removed casts and promoted everything correctly. Also removed irrelevant comments in the public pakfile header.
The printf specifier does not allow limiting buffer reads. Made the FourCCToString more performant by using a fixed size stack array and creating the fourcc into that which we now use to properly print out the asset magic.
The pak_decompress callback has been fully moved to a dedicated buffered pak decoder function, also added descriptive error to the buffered pak encode function. Fixed a bug in the streamed pak decoder where the next required streamed buffer size would always be below actually required on any subsequent patch rpaks that are getting loaded. Also fixed assert in Pak_ZStreamDecode, as the inbuf position can equal total streamed size, but shall never exceed it!
* Pak_InitDecoder() now takes the output buf and mask as parameters.
* Pak_InitDecoder() checks if provided masks are a power of 2 (required).
* Pak_ComputeRingBufferFrame() now uses the bit mask instead of modulo to determine # bytes used.
* Fixed a bug where PakDecoder_t::bufferSizeNeeded could be bigger than the file stream, causing a deadlock.
The new decoder can now decode patched and custom rpaks files, all issues so far have been fixed. The code still needs some testing and bench marking to make sure
Must not include pak header size since we parse compressed frame header and determine actual frame size from there, else we get 2 equal number that somehow don't equal (Pak_InitDecoder returns frame size + pak header size)
