In a recent Metal regression, Metal sometimes does not trigger the
[CAMetalDrawable addPresentedHandler:] callback on the final few (1-3)
CAMetalDrawable presentations, and retains internal memory associated
with these CAMetalDrawables. This does not occur for any CAMetalDrawable
presentations prior to those final few.
Most apps typically don't care much what happens after the last few
CAMetalDrawables are presented, and typically end shortly after that.
However, for some apps, such as Vulkan CTS WSI tests, which serially create
potentially hundreds, or thousands, of CAMetalLayers and MTLDevices,these
retained device memory allocations can pile up and cause the CTS WSI tests
to stall, block, or crash.
This issue has proven very difficult to debug, or replicate in incrementally
controlled environments. It appears consistently in some scenarios, and never
in other, almost identical scenarios.
For example, the MoltenVK Cube demo consistently runs without encountering
this issue, but CTS WSI test dEQP-VK.wsi.macos.swapchain.render.basic
consistently triggers the issue. Both apps run almost identical Vulkan
command paths, and identical swapchain image presentation paths, and
result in GPU captures that have identical swapchain image presentations.
We may ultimately have to wait for Apple to fix the core issue, but this
update includes workarounds that helps in some cases. During vkQueueWaitIdle()
and vkDeviceWaitIdle(), wait a short while for any in-flight swapchain image
presentations to finish, and attempt to force completion by calling
MVKPresentableSwapchainImage::forcePresentationCompletion(), which releases
the current CAMetalDrawable, and attempts to retrieve a new one, to trigger
the callback on the current CAMetalDrawable.
In exploring possible work-arounds for this issue, this update adds significant
structural improvements in the handling of swapchains, and quite a bit of new
performance and logging functionality that is useful for debugging purposes.
- Add several additional performance trackers, available via logging,
or the mvk_private_api.h API.
- Rename MVKPerformanceTracker members, and refactor performance result
collection, to support tracking and logging memory use, or other measurements,
in addition to just durations.
- Redefine MVKQueuePerformance to add tracking separate performance metrics for
MTLCommandBuffer retrieval, encoding, and execution, plus swapchain presentation.
- Add MVKDevicePerformance as part of MVKPerformanceStatistics to track device
information, including GPU device memory allocated, and update device memory
results whenever performance content is requested.
- Add MVKConfigActivityPerformanceLoggingStyle::
MVK_CONFIG_ACTIVITY_PERFORMANCE_LOGGING_STYLE_DEVICE_LIFETIME_ACCUMULATE
to accumulate performance and memory results across multiple serial
invocations of VkDevices, during the lifetime of the app process. This
is useful for accumulating performance results across multiple CTS tests.
- Log destruction of VkDevice, VkPhysicalDevice, and VkInstance, to bookend
the corresponding logs performed upon their creation.
- Include consumed GPU memory in log when VkPhysicalDevice is destroyed.
- Add mvkGetAvailableMTLDevicesArray() to support consistency when retrieving
MTLDevices available on the system.
- Add mvkVkCommandName() to generically map command use to a command name.
- MVKDevice:
- Support MTLPhysicalDevice.recommendedMaxWorkingSetSize on iOS & tvOS.
- Include available and consumed GPU memory in log of GPU device at
VkInstance creation time.
- MVKQueue:
- Add handleMTLCommandBufferError() to handle errors for all
MTLCommandBuffer executions.
- Track time to retrieve a MTLCommandBuffer.
- If MTLCommandBuffer could not be retrieved during queue submission,
report error, signal queue submission completion, and return
VK_ERROR_OUT_OF_POOL_MEMORY.
- waitIdle() simplify to use [MTLCommandBuffer waitUntilCompleted],
plus also wait for in-flight presentations to complete, and attempt
to force them to complete if they are stuck.
- MVKPresentableSwapchainImage:
- Don't track presenting MTLCommandBuffer.
- Add limit on number of attempts to retrieve a drawable, and report
VK_ERROR_OUT_OF_POOL_MEMORY if drawable cannot be retrieved.
- Return VkResult from acquireAndSignalWhenAvailable() to notify upstream
if MTLCommandBuffer could not be created.
- Track presentation time.
- Notify MVKQueue when presentation has completed.
- Add forcePresentationCompletion(), which releases the current
CAMetalDrawable, and attempts to retrieve a new one, to trigger the
callback on the current CAMetalDrawable. Called when a swapchain is
destroyed, or by queue if waiting for presentation to complete stalls,
- If destroyed while in flight, stop tracking swapchain and
don't notify when presentation completes.
- MVKSwapchain:
- Track active swapchain in MVKSurface to check oldSwapchain
- Track MVKSurface to access layer and detect lost surface.
- Don't track layer and layer observer, since MVKSurface handles these.
- On destruction, wait until all in-flight presentable images have returned.
- Remove empty and unused releaseUndisplayedSurfaces() function.
- MVKSurface:
- Consolidate constructors into initLayer() function.
- Update logic to test for valid layer and to set up layer observer.
- MVKSemaphoreImpl:
- Add getReservationCount()
- MVKBaseObject:
- Add reportResult() and reportWarning() functions to support logging
and reporting Vulkan results that are not actual errors.
- Rename MVKCommandUse::kMVKCommandUseEndCommandBuffer to
kMVKCommandUseBeginCommandBuffer, since that's where it is used.
- Update MVK_CONFIGURATION_API_VERSION and MVK_PRIVATE_API_VERSION to 38.
- Cube Demo support running a maximum number of frames.
In the rare case where vertex attribute buffers are bound to MVKCommandEncoder,
are not used by first pipeline, but are used by a subsequent pipeline, and no
other bindings are changed, the MVKResourcesCommandEncoderState will not appear
to be dirty to the second pipeline, and the buffer will not be bound to Metal.
When reverting a binding to dirty if it is not used by a pipeline, also revert
the enclosing MVKResourcesCommandEncoderState to dirty state.
Update MoltenVK to version 1.2.6 (unrelated).
- Guard against Intel returning zero values for CPU & GPU timestamps.
- Apply lowpass filter on timestampPeriod updates, to avoid wild temporary
changes, particularly at startup before GPU has been really exercised.
- In MoltenVK Xcode projects, set iOS & tvOS deployment targets to 12.0,
to avoid warnings while building MoltenVK.
- Add DYLD_LIBRARY_PATH to runcts script, to ensure Vulkan and MoltenVK
libraries are found during CTS runs.
- Update Whats_New.md and MoltenVK_Runtime_UserGuide.md documents.
Xcode simulator always requires 256B buffer alignment, even when running
on Apple Silicon. Previously, it was assumed that Apple Silicon would use
it's native 16B buffer alignment.
The [MTLDevice sampleTimestamps:gpuTimestamp:] function turns out to be
synchronized with other queue activities, and can block GPU execution
if it is called between MTLCommandBuffer submissions. On non-Apple-Silicon
devices, it was called before and after every vkQueueSubmit() submission,
to track the correlation between GPU and CPU timestamps, and was delaying
the start of GPU work on the next submission (on Apple Silicon, both
CPU & GPU timestamps are specified in nanoseconds, and the call was bypassed).
Move timestamp correlation from vkQueueSubmit() to
vkGetPhysicalDeviceProperties(), where it is used to update
VkPhysicalDeviceLimits::timestampPeriod on non-Apple-Silicon devices.
Delegate MVKPhysicalDevice::getProperties(VkPhysicalDeviceProperties2*)
to MVKPhysicalDevice::getProperties(VkPhysicalDeviceProperties*), plus
minimize wasted effort if pNext is empty (unrelated).
Move the declaration of several MVKPhysicalDevice member structs to
potentially reduce member spacing (unrelated).
When compiling tessellation vertex shaders, MVKGraphicsPipeline
pass array of MVKMTLFunction instead of MTLFunctions to retain
MTLFunctions for duration of processing.
Needed to complete fix for #1874
We can't wait until getMTLComputeEncoder() is called to dirty the state,
because this call will be avoided by dirty checks themselves.
Those checks are comparing against leftover and now incorrect state since
the previous encoder has already ended.
It needs to be dirtied on encoder end.
- Remove visionOS from multi-platform builds because it
requires Xcode 15+ and will abort a multi-platform build.
- Define TARGET_OS_XR for older SDK's.
- A number of SDK deprecation warnings remain when building for visionOS.
These cannot be removed without significant refactoring.
- Build visionOS dependencies for Release build by default.
- Fix local variable initialization warning (unrelated).
This provides feedback that indicates:
* how long it took to compile each shader stage and the pipeline as a
whole;
* whether or not the pipeline or any shader stage were found in any
supplied pipeline cache; and
* whether or not any supplied base pipeline were used to accelerate
pipeline creation.
This is similar to the performance statistics that MoltenVK already
collects.
Since we don't use any supplied base pipeline at all, this
implementation never sets
`VK_PIPELINE_CREATION_FEEDBACK_BASE_PIPELINE_ACCELERATION_BIT`. However,
I've identified several places where we could probably use the base
pipeline to accelerate pipeline creation. One day, I should probably
implement that.
Likewise, because we don't yet support using `MTLBinaryArchive`s,
`VK_PIPELINE_CREATION_FEEDBACK_APPLICATION_PIPELINE_CACHE_HIT_BIT` is
never set on the whole pipeline, though it *is* set for individual
stages, on the assumption that any shader found in a cache is likely to
be found in Metal's own implicit cache.
In this implementation, shader stage compilation time includes any time
needed to build the `MTLComputePipelineState`s needed for vertex and
tessellation control shaders in tessellated pipelines.
This patch also changes compilation of the vertex stage
`MTLComputePipelineState`s in tessellated pipelines to be eager instead
of lazy. We really ought to have been doing this anyway, in order to
report pipeline failures at creation time instead of draw time. I'm not
happy, though, that we now pay the cost of all three pipeline states all
the time, instead of just the ones that are used.
This also gets rid of some fields of `MVKGraphicsPipeline` that were
only used during pipeline construction, which should save some memory,
particularly for apps that create lots of pipelines.
This extension allows apps to provide a hint to the presentation engine
indicating which parts of the surface need updating. To provide this
hint, we call `-[CALayer setNeedsDisplayInRect:]`, which indicates that
only the given rectangle needs updating.
I'm not sure if this will have any effect, especially if
`CAMetalLayer.presentsWithTransaction` is `NO`. Luckily for us, this is
only a hint, and it is permissible for the presentation engine to do
nothing with the hint.
The tests don't work because they apparently can't handle
`VK_SUBOPTIMAL_KHR` being returned.
In this commit, I've added support for Xcode 15, and added a case for MSL version 3.1. I added this because I noticed xcode was throwing some warnings about an unhandled switch case.
- Fix unreachable code in MVKDeferredOperation::join().
- Refactor code so deferred functions call back to MVKDeferredOperation
instance to update current status, and deferred function execution
returns result of individual thread execution.
- MVKDeferredOperation use MVKSmallVector for _functionParameters.
- MVKDeferredOperation use a single mutex lock.
- Add additional comments explaining design to developers of
future extensions that use deferred operations.
To reduce complexity and repetitive copy-pasted spaghetti code,
the design approach here was to implement triangle fan conversion on
MVKCmdDrawIndexedIndirect, as the most general of the draw commands,
and then populate and invoke a synthetic MVKCmdDrawIndexedIndirect
command from the other draw commands.
- Rename pipeline factory shader cmdDrawIndexedIndirectMultiviewConvertBuffers()
to cmdDrawIndexedIndirectConvertBuffers, and in addition to original support
for modifying indirect content to support multiview, add support for
converting triangle fan indirect content and indexes to triangle list.
- Modify MVKCmdDrawIndexedIndirect to track need to convert triangle fans
to triangle list, and invoke kernel function when needed.
- Modify MVKCmdDraw, MVKCmdDrawIndexed, and MVKCmdDrawIndirect to populate
and invoke a synthetic MVKCmdDrawIndexedIndirect command to convert triangle
fans to triangle lists.
- Add pipeline factory shader cmdDrawIndirectPopulateIndexes() to convert
non-indexed indirect content to indexed indirect content.
- MVKCmdDrawIndexedIndirect add support for zero divisor vertex buffers
potentially coming from MVKCmdDraw and MVKCmdDrawIndexed.
- Rename pipeline factory shader cmdDrawIndexedIndirectConvertBuffers()
to cmdDrawIndexedIndirectTessConvertBuffers() so it will be invoked from
MVKCommandEncodingPool::getCmdDrawIndirectTessConvertBuffersMTLComputePipelineState()
(unrelated).
This just provides support for the `SPV_KHR_non_semantic_info`
extension, which supports extended instruction sets that do not affect
the semantics of a SPIR-V shader (e.g. debug info). SPIRV-Cross already
handles these instruction sets, so no additional work is required on our
part to support this extension.
This extension has a direct Metal equivalent in the
`-[MTLDevice sampleTimestamps:gpuTimestamp:]` method. However, that
method returns CPU timestamps in the Mach absolute time domain, which is
*not* that of `CLOCK_MONOTONIC_RAW` but of `CLOCK_UPTIME_RAW`. The
function that corresponds to `CLOCK_MONOTONIC_RAW` is
`mach_continuous_time()`. Therefore, this implementation uses the
`mach_continuous_time()` function for the CPU timestamp. Perhaps we
should lobby the WG for `VK_TIME_DOMAIN_CLOCK_UPTIME_RAW_EXT`.
This turned out to be a little bit more involved than I had hoped. But,
with this, we can now use the `VK_FORMAT_A4R4G4B4_UNORM_PACK16` and
`VK_FORMAT_A4B4G4R4_UNORM_PACK16` formats from shaders, use them as blit
sources, and even clear them. Storage images and render targets of these
formats aren't supported, however. To support the latter would require
the insertion of a swizzle into the fragment shader before returning.
The former cannot be reasonably supported.
This commit changes the MVKSmallVector to a C-Style array because the parameters array will not have to resize, and it seems that a C-Style array is just as capable of doing the job.
The changes are as follows:
* Moved the code around to fit with the ordering system
* Added a function to get available cpu cores
* Renamed variables with _ in front of them
* Added mutexes and lock guards for the getters and setters of the max concurrency and result variables
* Made max concurrency dynamic by returning 0 when the operation is finished
Implemented deferred host operations in this commit. It was pretty simple with nothing Metal specific. I am a bit concerned on the return types of MVKDeferredOperation::join and about how to store the operation to be executed for later.
- When forcing the window system to use the same high-power GPU as the app,
move the call to MTLCreateSystemDefaultDevice() to MVKDevice constructor,
instead of MVKDevice::createSwapchain(), and test whether the
VK_KHR_swapchain extension is enabled to determine the need to swap GPUs.
- After calling MTLCreateSystemDefaultDevice() the GPU will already be
the same high-power GPU, so remove attempting to replace the MTLDevice.
- Remove MVKPhysicalDevice::replaceMTLDevice() as no longer used.
- Remove many unnecessary inline declarations in MVKDevice.h (unrelated).
As of macOS Big Sur and iOS/tvOS 14, the `discard_fragment()` function
in MSL is defined to have demote semantics; that is, fragment shader
output is discarded, but the fragment shader thread continues to run as
a helper invocation. This is very useful for Direct3D emulation, since
this is the semantic that HLSL `discard` has.
Signed-off-by: Chip Davis <chip@holochip.com>
- [MTLDrawable presentAtTime:] syncs to display vsync. To support
VK_PRESENT_MODE_IMMEDIATE_KHR while using VkPresentTimeGOOGLE::presentID,
only call presentAtTime: if VkPresentTimeGOOGLE::desiredPresentTime has
been explicitly set to a non-zero value.
- Clarify initially clearing MVKImagePresentInfo to all zeros.
- Only log performance stats on FPS logging if logging style is explicitly
set to MVK_CONFIG_ACTIVITY_PERFORMANCE_LOGGING_STYLE_FRAME_COUNT (unrelated).
The same compute encoder is used across dispatches and other commands,
which may override compute state, and end up breaking subsequent dispatches.
- Mark compute encoding state dirty when following commands,
which use Metal compute encoders, are issued:
- vkCmdCopyBuffer()
- vkCmdClearColorImage()
- vkCmdClearDepthStencilImage()
- vkCmdFillBuffer()
- vkCmdCopyQueryPoolResults()
- MVKCommandEncoder move marking compute state dirty from
endCurrentMetalEncoding() to getMTLComputeEncoder().
- For efficiency, don't prematurely force end of query copy compute encoder
used on renderpass end, in case compute dispatches follow.
- Update MoltenVK to 1.2.5 (unrelated).
Advertise VK_KHR_depth_stencil_resolve extension on early iOS devices,
since VK_RESOLVE_MODE_SAMPLE_ZERO_BIT is supported on all devices,
even if other resolve modes are not, and makes it consistent with
Vulkan 1.2 mandatory support for VK_RESOLVE_MODE_SAMPLE_ZERO_BIT.
