During beta tests on a Linux system, we encountered an issue where the I/O could be slower and thus result in not enough bytes being streamed by the time we invoke the decoder (lenStreamed == bufSizeNeeded at this point), after 16 calls with lenStreamed == bufSizeNeeded, the engine errors as ZSTD_NO_FORWARD_PROGRESS_MAX would be reached. Added value of 'ZSTD_nextSrcSizeToDecompress()' to 'PakDecoder_s::bufferSizeNeeded' to make sure we never call the decoder without any new streamed bytes. Also increased the value of ZSTD_NO_FORWARD_PROGRESS_MAX to 1024 since this fixed the issue without applying the aforementioned patch, this was increased as a hardening measure.
Add missing types to fully mapped out PakGlobals_s struct, which is a 13MiB+ structure! This covers the vast majority of the pakfile system which is why we had to change a bunch of stuff for this patch. This patch also comes with:
- Reversed 'JobFifoLock_s' structure
- Reversed 'PakTracker_s' structure
- Reversed 'PakAssetTracker_s' structure
Many globals have been dropped as they were covered by the large PakGlobals_s singleton.
The pak decoder logic has been changed up as well, we now use a decode mode enumerant which will make it easier to add in more decoders for the pak files in the future.
The pak_decompress callback has been fully moved to a dedicated buffered pak decoder function, also added descriptive error to the buffered pak encode function. Fixed a bug in the streamed pak decoder where the next required streamed buffer size would always be below actually required on any subsequent patch rpaks that are getting loaded. Also fixed assert in Pak_ZStreamDecode, as the inbuf position can equal total streamed size, but shall never exceed it!
* Pak_InitDecoder() now takes the output buf and mask as parameters.
* Pak_InitDecoder() checks if provided masks are a power of 2 (required).
* Pak_ComputeRingBufferFrame() now uses the bit mask instead of modulo to determine # bytes used.
* Fixed a bug where PakDecoder_t::bufferSizeNeeded could be bigger than the file stream, causing a deadlock.
The new decoder can now decode patched and custom rpaks files, all issues so far have been fixed. The code still needs some testing and bench marking to make sure
Allows paks compressed with ZSTD, to be stream decoded at run time. Currently seems to work pretty good, but needs to be further battle tested + cleaned up still.
* split rtech_game and rtech_utils cpp files into multiple files
* rebuilt several large pak load routines for debugging and custom implementations
* moved rson code to rtech_game
* reworked and improved engine and sdk pak precache system
* reversed more of the jobthreads system
