The same constants are used between Recast and Detour, but Recast had to declare and match the constant locally as it couldn't access the one from the Detour library. This patch avoids duplicate definitions and potential issues during refactors.
Detour_LevelInit resets the global NavMesh query, but if code is going to utilize the global Detour query for an existing query on a subsequent frame without attaching a NavMesh in the mean time, the code will crash. The previously attached NavMesh type has to be reattached after the hotswap.
Slight optimization to skip unsupported ones out before computing anything, and also improves the generation results as we try to get the best one out of the supported ones, instead of taking one from supported and unsupported and then later check if its unsupported. (if the best type is 9, and 9 isn't supported while 8 is, and 8 would've been the candidate after 9, then this patch will still give the link a chance for 8 while the previous implementation would've skipped it entirely and returned DT_NULL_TRAVERSE_TYPE, ultimately resulting in no link whatsoever.
The algorithm has been moved to the dtNavMesh class, and now features a parameter based design allowing code that utilizes the algorithm to define engine-specific tests such as the raycasting, traverse masking, etc. The code can now be hooked up to any level editor without changing a line of code inside the Recast & Detour library. All parameters have been documented thoroughly. The code now also properly returns errors if out of memory, or if invalid parameters were provided. This commit does not affect the results of the traverse link generation, the navmesh remains identical to the one generated on the previous design.
pmin and pmax pointed to these stack arrays, but these stack arrays were inside the loop while pmin and pmax, along with their usages were outside the scope of the loop causing undefined behavior when they pointed to the currently out-of-scope stack variables. Moved the arrays outside the loop so they remain valid up to the last point they are needed.
dtMeshHeader::bvNodeCount is always set to polycount*2, but when we have null bv nodes, then these will also be used during dtNavMeshQuery::queryPolygonsInTile, which can yield incorrect results when the null nodes overlaps query box. Also added an additional optimization to only store non-null BV nodes which can save a significant amount of memory during the runtime. After this bug fix and optimization, the resulting mesh still works as intended in-game.
Fully unlinked can be removed entirely, semi unlinked needs a rebuild to remove unlinked polygons (still work in progress). Added ability to automatically remove fully unlinked tiles.
This function was removed in commit 041d02cd654d94bf4b059c212cb6a63a39e5324e but it should've been kept in as it was still used for dtNavMeshQuery::findRandomPointAroundCircle. Only the quantizer should be removed.
Titanfall 2 navmeshes flag their polygons as follows. The pattern was always that if a polygon connects to a polygon on a neighboring tile, it should be flagged as EDITOR_POLYFLAGS_HAS_NEIGHBOUR, and if the polygon's surface area isn't higher than 120 (NAVMESH_SMALL_POLYGON_THRESHOLD) it should be flagged as EDITOR_POLYFLAGS_TOO_SMALL.
Let recast compute this instead of detour. We need this before the detour navmesh is getting build as we have to determine whether a polygon is too small before that, and flag it as such.
Instead of using just the polygon edges, use the detail edges as well. Ray cast errors have been fully eliminated; nothing ever clips through geometry anymore. The results are also a lot more detailed now as we can connect various sub edges together. The resulting navmesh also performs better in-game, the npc's are now a lot more fluent with jumps and climbs, they rarely clip though geometry, if they do its typically an error with the input geometry used to build the navmesh.
There was also a bug in GetBestTraverseType; the exclusive check (where at least one overlap had to occur) returned true if both overlap tests failed. This has been fixed by flipping the check and now checking if at least one overlaps when an exclusive test was specified.
