Improve behavior of swapchain image presentation stalls caused by Metal regression.

In a recent Metal regression, Metal sometimes does not trigger the
[CAMetalDrawable addPresentedHandler:] callback on the final few (1-3)
CAMetalDrawable presentations, and retains internal memory associated
with these CAMetalDrawables. This does not occur for any CAMetalDrawable
presentations prior to those final few.

Most apps typically don't care much what happens after the last few
CAMetalDrawables are presented, and typically end shortly after that.

However, for some apps, such as Vulkan CTS WSI tests, which serially create
potentially hundreds, or thousands, of CAMetalLayers and MTLDevices,these
retained device memory allocations can pile up and cause the CTS WSI tests
to stall, block, or crash.

This issue has proven very difficult to debug, or replicate in incrementally
controlled environments. It appears consistently in some scenarios, and never
in other, almost identical scenarios.

For example, the MoltenVK Cube demo consistently runs without encountering
this issue, but CTS WSI test dEQP-VK.wsi.macos.swapchain.render.basic
consistently triggers the issue. Both apps run almost identical Vulkan
command paths, and identical swapchain image presentation paths, and
result in GPU captures that have identical swapchain image presentations.

We may ultimately have to wait for Apple to fix the core issue, but this
update includes workarounds that helps in some cases. During vkQueueWaitIdle()
and vkDeviceWaitIdle(), wait a short while for any in-flight swapchain image
presentations to finish, and attempt to force completion by calling
MVKPresentableSwapchainImage::forcePresentationCompletion(), which releases
the current CAMetalDrawable, and attempts to retrieve a new one, to trigger
the callback on the current CAMetalDrawable.

In exploring possible work-arounds for this issue, this update adds significant
structural improvements in the handling of swapchains, and quite a bit of new
performance and logging functionality that is useful for debugging purposes.

- Add several additional performance trackers, available via logging,
  or the mvk_private_api.h API.
- Rename MVKPerformanceTracker members, and refactor performance result
  collection, to support tracking and logging memory use, or other measurements,
  in addition to just durations.
- Redefine MVKQueuePerformance to add tracking separate performance metrics for
  MTLCommandBuffer retrieval, encoding, and execution, plus swapchain presentation.
- Add MVKDevicePerformance as part of MVKPerformanceStatistics to track device
  information, including GPU device memory allocated, and update device memory
  results whenever performance content is requested.
- Add MVKConfigActivityPerformanceLoggingStyle::
  MVK_CONFIG_ACTIVITY_PERFORMANCE_LOGGING_STYLE_DEVICE_LIFETIME_ACCUMULATE
  to accumulate performance and memory results across multiple serial
  invocations of VkDevices, during the lifetime of the app process. This
  is useful for accumulating performance results across multiple CTS tests.
- Log destruction of VkDevice, VkPhysicalDevice, and VkInstance, to bookend
  the corresponding logs performed upon their creation.
- Include consumed GPU memory in log when VkPhysicalDevice is destroyed.
- Add mvkGetAvailableMTLDevicesArray() to support consistency when retrieving
  MTLDevices available on the system.
- Add mvkVkCommandName() to generically map command use to a command name.
- MVKDevice:
    - Support MTLPhysicalDevice.recommendedMaxWorkingSetSize on iOS & tvOS.
    - Include available and consumed GPU memory in log of GPU device at
      VkInstance creation time.
- MVKQueue:
    - Add handleMTLCommandBufferError() to handle errors for all
      MTLCommandBuffer executions.
    - Track time to retrieve a MTLCommandBuffer.
    - If MTLCommandBuffer could not be retrieved during queue submission,
      report error, signal queue submission completion, and return
      VK_ERROR_OUT_OF_POOL_MEMORY.
    - waitIdle() simplify to use [MTLCommandBuffer waitUntilCompleted],
      plus also wait for in-flight presentations to complete, and attempt
      to force them to complete if they are stuck.
- MVKPresentableSwapchainImage:
    - Don't track presenting MTLCommandBuffer.
    - Add limit on number of attempts to retrieve a drawable, and report
      VK_ERROR_OUT_OF_POOL_MEMORY if drawable cannot be retrieved.
    - Return VkResult from acquireAndSignalWhenAvailable() to notify upstream
      if MTLCommandBuffer could not be created.
    - Track presentation time.
	- Notify MVKQueue when presentation has completed.
	- Add forcePresentationCompletion(), which releases the current
	  CAMetalDrawable, and attempts to retrieve a new one, to trigger the
	  callback on the current CAMetalDrawable. Called when a swapchain is
	  destroyed, or by queue if waiting for presentation to complete stalls,
	- If destroyed while in flight, stop tracking swapchain and
	  don't notify when presentation completes.
- MVKSwapchain:
    - Track active swapchain in MVKSurface to check oldSwapchain
    - Track MVKSurface to access layer and detect lost surface.
    - Don't track layer and layer observer, since MVKSurface handles these.
    - On destruction, wait until all in-flight presentable images have returned.
    - Remove empty and unused releaseUndisplayedSurfaces() function.
- MVKSurface:
	- Consolidate constructors into initLayer() function.
    - Update logic to test for valid layer and to set up layer observer.
- MVKSemaphoreImpl:
    - Add getReservationCount()
- MVKBaseObject:
    - Add reportResult() and reportWarning() functions to support logging
      and reporting Vulkan results that are not actual errors.
- Rename MVKCommandUse::kMVKCommandUseEndCommandBuffer to
  kMVKCommandUseBeginCommandBuffer, since that's where it is used.
- Update MVK_CONFIGURATION_API_VERSION and MVK_PRIVATE_API_VERSION to 38.
- Cube Demo support running a maximum number of frames.

Demos/Cube/Cube.xcodeproj/project.pbxproj

@@ -3,7 +3,7 @@
 	archiveVersion = 1;
 	classes = {
 	};
-	objectVersion = 52;
+	objectVersion = 54;
 	objects = {
 
 /* Begin PBXBuildFile section */

Demos/Cube/iOS/DemoViewController.m

@@ -30,15 +30,9 @@
 	struct demo demo;
 }
 
--(void) dealloc {
+/** Since this is a single-view app, initialize Vulkan as view is appearing. */
-	demo_cleanup(&demo);
+-(void) viewWillAppear: (BOOL) animated {
-	[_displayLink release];
+	[super viewWillAppear: animated];
-	[super dealloc];
-}
-
-/** Since this is a single-view app, init Vulkan when the view is loaded. */
--(void) viewDidLoad {
-	[super viewDidLoad];
 
 	self.view.contentScaleFactor = UIScreen.mainScreen.nativeScale;
 
@@ -68,6 +62,13 @@
 	demo_resize(&demo);
 }
 
+-(void) viewDidDisappear: (BOOL) animated {
+	[_displayLink invalidate];
+	[_displayLink release];
+	demo_cleanup(&demo);
+	[super viewDidDisappear: animated];
+}
+
 @end
 
 

Demos/Cube/macOS/DemoViewController.m

@@ -18,6 +18,7 @@
 
 #import "DemoViewController.h"
 #import <QuartzCore/CAMetalLayer.h>
+#import <CoreVideo/CVDisplayLink.h>
 
 #include <MoltenVK/mvk_vulkan.h>
 #include "../../Vulkan-Tools/cube/cube.c"
@@ -27,27 +28,34 @@
 #pragma mark DemoViewController
 
 @implementation DemoViewController {
-	CVDisplayLinkRef	_displayLink;
+	CVDisplayLinkRef _displayLink;
 	struct demo demo;
+	uint32_t _maxFrameCount;
+	uint64_t _frameCount;
+	BOOL _stop;
+	BOOL _useDisplayLink;
 }
 
--(void) dealloc {
+/** Since this is a single-view app, initialize Vulkan as view is appearing. */
-	demo_cleanup(&demo);
+-(void) viewWillAppear {
-	CVDisplayLinkRelease(_displayLink);
+	[super viewWillAppear];
-	[super dealloc];
-}
-
-/** Since this is a single-view app, initialize Vulkan during view loading. */
--(void) viewDidLoad {
-	[super viewDidLoad];
 
 	self.view.wantsLayer = YES;		// Back the view with a layer created by the makeBackingLayer method.
 
-	// Enabling this will sync the rendering loop with the natural display link (60 fps).
+	// Enabling this will sync the rendering loop with the natural display link
-	// Disabling this will allow the rendering loop to run flat out, limited only by the rendering speed.
+	// (monitor refresh rate, typically 60 fps). Disabling this will allow the
-	bool useDisplayLink = true;
+	// rendering loop to run flat out, limited only by the rendering speed.
+	_useDisplayLink = YES;
 
-	VkPresentModeKHR vkPresentMode = useDisplayLink ? VK_PRESENT_MODE_FIFO_KHR : VK_PRESENT_MODE_IMMEDIATE_KHR;
+	// If this value is set to zero, the demo will render frames until the window is closed.
+	// If this value is not zero, it establishes a maximum number of frames that will be
+	// rendered, and once this count has been reached, the demo will stop rendering.
+	// Once rendering is finished, if _useDisplayLink is false, the demo will immediately
+	// clean up the Vulkan objects, or if _useDisplayLink is true, the demo will delay
+	// cleaning up Vulkan objects until the window is closed.
+	_maxFrameCount = 0;
+
+	VkPresentModeKHR vkPresentMode = _useDisplayLink ? VK_PRESENT_MODE_FIFO_KHR : VK_PRESENT_MODE_IMMEDIATE_KHR;
 	char vkPresentModeStr[64];
 	sprintf(vkPresentModeStr, "%d", vkPresentMode);
 
@@ -55,19 +63,33 @@
 	int argc = sizeof(argv)/sizeof(char*);
 	demo_main(&demo, self.view.layer, argc, argv);
 
-	if (useDisplayLink) {
+	_stop = NO;
+	_frameCount = 0;
+	if (_useDisplayLink) {
 		CVDisplayLinkCreateWithActiveCGDisplays(&_displayLink);
-		CVDisplayLinkSetOutputCallback(_displayLink, &DisplayLinkCallback, &demo);
+		CVDisplayLinkSetOutputCallback(_displayLink, &DisplayLinkCallback, self);
 		CVDisplayLinkStart(_displayLink);
 	} else {
 		dispatch_async(dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0), ^{
-			while(true) {
+			do {
 				demo_draw(&demo);
-			}
+				_stop = _stop || (_maxFrameCount && ++_frameCount >= _maxFrameCount);
+			} while( !_stop );
+			demo_cleanup(&demo);
 		});
 	}
 }
 
+-(void) viewDidDisappear {
+	_stop = YES;
+	if (_useDisplayLink) {
+		CVDisplayLinkRelease(_displayLink);
+		demo_cleanup(&demo);
+	}
+
+	[super viewDidDisappear];
+}
+
 
 #pragma mark Display loop callback function
 
@@ -78,7 +100,11 @@ static CVReturn DisplayLinkCallback(CVDisplayLinkRef displayLink,
 									CVOptionFlags flagsIn,
 									CVOptionFlags* flagsOut,
 									void* target) {
-	demo_draw((struct demo*)target);
+	DemoViewController* demoVC =(DemoViewController*)target;
+	if ( !demoVC->_stop ) {
+		demo_draw(&demoVC->demo);
+		demoVC->_stop = (demoVC->_maxFrameCount && ++demoVC->_frameCount >= demoVC->_maxFrameCount);
+	}
 	return kCVReturnSuccess;
 }
 

Docs/Whats_New.md

@@ -20,6 +20,9 @@ Released TBD
 
 - Fix rare case where vertex attribute buffers are not bound to Metal 
   when no other bindings change between pipelines.
+- Improve behavior of swapchain image presentation stalls caused by Metal regression.
+- Add several additional performance trackers, available via logging, or the `mvk_private_api.h` API.
+- Update `MVK_CONFIGURATION_API_VERSION` and `MVK_PRIVATE_API_VERSION` to `38`.
 
 
 

MoltenVK/MoltenVK/API/mvk_config.h

@@ -51,7 +51,7 @@ extern "C" {
 #define MVK_VERSION     MVK_MAKE_VERSION(MVK_VERSION_MAJOR, MVK_VERSION_MINOR, MVK_VERSION_PATCH)
 
 
-#define MVK_CONFIGURATION_API_VERSION   37
+#define MVK_CONFIGURATION_API_VERSION   38
 
 /** Identifies the level of logging MoltenVK should be limited to outputting. */
 typedef enum MVKConfigLogLevel {
@@ -138,10 +138,11 @@ typedef enum MVKConfigCompressionAlgorithm {
 
 /** Identifies the style of activity performance logging to use. */
 typedef enum MVKConfigActivityPerformanceLoggingStyle {
-	MVK_CONFIG_ACTIVITY_PERFORMANCE_LOGGING_STYLE_FRAME_COUNT     = 0,	/**< Repeatedly log performance after a configured number of frames. */
+	MVK_CONFIG_ACTIVITY_PERFORMANCE_LOGGING_STYLE_FRAME_COUNT                = 0,	/**< Repeatedly log performance after a configured number of frames. */
-	MVK_CONFIG_ACTIVITY_PERFORMANCE_LOGGING_STYLE_IMMEDIATE       = 1,	/**< Log immediately after each performance measurement. */
+	MVK_CONFIG_ACTIVITY_PERFORMANCE_LOGGING_STYLE_IMMEDIATE                  = 1,	/**< Log immediately after each performance measurement. */
-	MVK_CONFIG_ACTIVITY_PERFORMANCE_LOGGING_STYLE_DEVICE_LIFETIME = 2,	/**< Log at the end of the VkDevice lifetime. This is useful for one-shot apps such as testing frameworks. */
+	MVK_CONFIG_ACTIVITY_PERFORMANCE_LOGGING_STYLE_DEVICE_LIFETIME            = 2,	/**< Log at the end of the VkDevice lifetime. This is useful for one-shot apps such as testing frameworks. */
-	MVK_CONFIG_ACTIVITY_PERFORMANCE_LOGGING_STYLE_MAX_ENUM        = 0x7FFFFFFF,
+	MVK_CONFIG_ACTIVITY_PERFORMANCE_LOGGING_STYLE_DEVICE_LIFETIME_ACCUMULATE = 3,	/**< Log at the end of the VkDevice lifetime, but continue to accumulate across mulitiple VkDevices throughout the app process. This is useful for testing frameworks that create many VkDevices serially. */
+	MVK_CONFIG_ACTIVITY_PERFORMANCE_LOGGING_STYLE_MAX_ENUM                   = 0x7FFFFFFF,
 } MVKConfigActivityPerformanceLoggingStyle;
 
 /**
@@ -786,6 +787,8 @@ typedef struct {
 	/**
 	 * Controls when MoltenVK should log activity performance events.
 	 *
+	 * The performanceTracking parameter must also be enabled.
+	 *
 	 * The value of this parameter must be changed before creating a VkDevice,
 	 * for the change to take effect.
 	 *

MoltenVK/MoltenVK/API/mvk_private_api.h

@@ -44,7 +44,7 @@ typedef unsigned long MTLArgumentBuffersTier;
  */
 
 
-#define MVK_PRIVATE_API_VERSION   37
+#define MVK_PRIVATE_API_VERSION   38
 
 
 /** Identifies the type of rounding Metal uses for float to integer conversions in particular calculatons. */
@@ -153,13 +153,16 @@ typedef struct {
 	VkDeviceSize hostMemoryPageSize;				/**< The size of a page of host memory on this platform. */
 } MVKPhysicalDeviceMetalFeatures;
 
-/** MoltenVK performance of a particular type of activity. */
+/**
+ * MoltenVK performance of a particular type of activity.
+ * Durations are recorded in milliseconds. Memory sizes are recorded in kilobytes.
+ */
 typedef struct {
-    uint32_t count;             /**< The number of activities of this type. */
+    uint32_t count;       /**< The number of activities of this type. */
-	double latestDuration;      /**< The latest (most recent) duration of the activity, in milliseconds. */
+	double latest;        /**< The latest (most recent) value of the activity. */
-    double averageDuration;     /**< The average duration of the activity, in milliseconds. */
+    double average;       /**< The average value of the activity. */
-    double minimumDuration;     /**< The minimum duration of the activity, in milliseconds. */
+    double minimum;       /**< The minimum value of the activity. */
-    double maximumDuration;     /**< The maximum duration of the activity, in milliseconds. */
+    double maximum;       /**< The maximum value of the activity. */
 } MVKPerformanceTracker;
 
 /** MoltenVK performance of shader compilation activities. */
@@ -186,12 +189,20 @@ typedef struct {
 
 /** MoltenVK performance of queue activities. */
 typedef struct {
-	MVKPerformanceTracker mtlQueueAccess;               /** Create an MTLCommandQueue or access an existing cached instance. */
+	MVKPerformanceTracker retrieveMTLCommandBuffer;     /** Retrieve a MTLCommandBuffer from a MTLQueue. */
-	MVKPerformanceTracker mtlCommandBufferCompletion;   /** Completion of a MTLCommandBuffer on the GPU, from commit to completion callback. */
+	MVKPerformanceTracker commandBufferEncoding;        /** Encode a single VkCommandBuffer to a MTLCommandBuffer (excludes MTLCommandBuffer encoding from configured immediate prefilling). */
-	MVKPerformanceTracker nextCAMetalDrawable;			/** Retrieve next CAMetalDrawable from CAMetalLayer during presentation. */
+	MVKPerformanceTracker submitCommandBuffers;         /** Submit and encode all VkCommandBuffers in a vkQueueSubmit() operation to MTLCommandBuffers (including both prefilled and deferred encoding). */
-	MVKPerformanceTracker frameInterval;				/** Frame presentation interval (1000/FPS). */
+	MVKPerformanceTracker mtlCommandBufferExecution;    /** Execute a MTLCommandBuffer on the GPU, from commit to completion callback. */
+	MVKPerformanceTracker retrieveCAMetalDrawable;      /** Retrieve next CAMetalDrawable from a CAMetalLayer. */
+	MVKPerformanceTracker presentSwapchains;            /** Present the swapchains in a vkQueuePresentKHR() on the GPU, from commit to presentation callback. */
+	MVKPerformanceTracker frameInterval;                /** Frame presentation interval (1000/FPS). */
 } MVKQueuePerformance;
 
+/** MoltenVK performance of device activities. */
+typedef struct {
+	MVKPerformanceTracker gpuMemoryAllocated;		/** GPU memory allocated (in KB). */
+} MVKDevicePerformance;
+
 /**
  * MoltenVK performance. You can retrieve a copy of this structure using the vkGetPerformanceStatisticsMVK() function.
  *
@@ -209,6 +220,7 @@ typedef struct {
 	MVKShaderCompilationPerformance shaderCompilation;	/** Shader compilations activities. */
 	MVKPipelineCachePerformance pipelineCache;			/** Pipeline cache activities. */
 	MVKQueuePerformance queue;          				/** Queue activities. */
+	MVKDevicePerformance device;          				/** Device activities. */
 } MVKPerformanceStatistics;
 
 

MoltenVK/MoltenVK/Commands/MVKCommandBuffer.h

@@ -536,9 +536,6 @@ protected:
 #pragma mark -
 #pragma mark Support functions
 
-/** Returns a name, suitable for use as a MTLCommandBuffer label, based on the MVKCommandUse. */
-NSString* mvkMTLCommandBufferLabel(MVKCommandUse cmdUse);
-
 /** Returns a name, suitable for use as a MTLRenderCommandEncoder label, based on the MVKCommandUse. */
 NSString* mvkMTLRenderCommandEncoderLabel(MVKCommandUse cmdUse);
 

MoltenVK/MoltenVK/Commands/MVKCommandBuffer.mm

@@ -120,7 +120,7 @@ VkResult MVKCommandBuffer::begin(const VkCommandBufferBeginInfo* pBeginInfo) {
 
     if(_device->shouldPrefillMTLCommandBuffers() && !(_isSecondary || _supportsConcurrentExecution)) {
 		@autoreleasepool {
-			_prefilledMTLCmdBuffer = [_commandPool->getMTLCommandBuffer(0) retain];    // retained
+			_prefilledMTLCmdBuffer = [_commandPool->getMTLCommandBuffer(kMVKCommandUseBeginCommandBuffer, 0) retain];    // retained
 			auto prefillStyle = mvkConfig().prefillMetalCommandBuffers;
 			if (prefillStyle == MVK_CONFIG_PREFILL_METAL_COMMAND_BUFFERS_STYLE_IMMEDIATE_ENCODING ||
 				prefillStyle == MVK_CONFIG_PREFILL_METAL_COMMAND_BUFFERS_STYLE_IMMEDIATE_ENCODING_NO_AUTORELEASE ) {
@@ -335,11 +335,19 @@ void MVKCommandBuffer::recordBindPipeline(MVKCmdBindPipeline* mvkBindPipeline) {
 #pragma mark -
 #pragma mark MVKCommandEncoder
 
+// Activity performance tracking is put here to deliberately exclude when
+// MVKConfiguration::prefillMetalCommandBuffers is set to immediate prefilling,
+// because that would include app time between command submissions.
 void MVKCommandEncoder::encode(id<MTLCommandBuffer> mtlCmdBuff,
 							   MVKCommandEncodingContext* pEncodingContext) {
+	MVKDevice* mvkDev = getDevice();
+	uint64_t startTime = mvkDev->getPerformanceTimestamp();
+
     beginEncoding(mtlCmdBuff, pEncodingContext);
     encodeCommands(_cmdBuffer->_head);
     endEncoding();
+
+	mvkDev->addActivityPerformance(mvkDev->_performanceStatistics.queue.commandBufferEncoding, startTime);
 }
 
 void MVKCommandEncoder::beginEncoding(id<MTLCommandBuffer> mtlCmdBuff, MVKCommandEncodingContext* pEncodingContext) {
@@ -1169,19 +1177,6 @@ MVKCommandEncoder::~MVKCommandEncoder() {
 #pragma mark -
 #pragma mark Support functions
 
-NSString* mvkMTLCommandBufferLabel(MVKCommandUse cmdUse) {
-	switch (cmdUse) {
-		case kMVKCommandUseEndCommandBuffer:                return @"vkEndCommandBuffer (Prefilled) CommandBuffer";
-		case kMVKCommandUseQueueSubmit:                     return @"vkQueueSubmit CommandBuffer";
-		case kMVKCommandUseQueuePresent:                    return @"vkQueuePresentKHR CommandBuffer";
-		case kMVKCommandUseQueueWaitIdle:                   return @"vkQueueWaitIdle CommandBuffer";
-		case kMVKCommandUseDeviceWaitIdle:                  return @"vkDeviceWaitIdle CommandBuffer";
-		case kMVKCommandUseAcquireNextImage:                return @"vkAcquireNextImageKHR CommandBuffer";
-		case kMVKCommandUseInvalidateMappedMemoryRanges:    return @"vkInvalidateMappedMemoryRanges CommandBuffer";
-		default:                                            return @"Unknown Use CommandBuffer";
-	}
-}
-
 NSString* mvkMTLRenderCommandEncoderLabel(MVKCommandUse cmdUse) {
     switch (cmdUse) {
 		case kMVKCommandUseBeginRendering:                  return @"vkCmdBeginRendering RenderEncoder";

MoltenVK/MoltenVK/Commands/MVKCommandPool.h

@@ -82,7 +82,7 @@ public:
 	 * Returns a retained MTLCommandBuffer created from the indexed queue
 	 * within the queue family for which this command pool was created.
 	 */
-	id<MTLCommandBuffer> getMTLCommandBuffer(uint32_t queueIndex);
+	id<MTLCommandBuffer> getMTLCommandBuffer(MVKCommandUse cmdUse, uint32_t queueIndex);
 
 	/** Release any held but unused memory back to the system. */
 	void trim();

MoltenVK/MoltenVK/Commands/MVKCommandPool.mm

@@ -77,8 +77,8 @@ void MVKCommandPool::freeCommandBuffers(uint32_t commandBufferCount,
 	}
 }
 
-id<MTLCommandBuffer> MVKCommandPool::getMTLCommandBuffer(uint32_t queueIndex) {
+id<MTLCommandBuffer> MVKCommandPool::getMTLCommandBuffer(MVKCommandUse cmdUse, uint32_t queueIndex) {
-	return _device->getQueue(_queueFamilyIndex, queueIndex)->getMTLCommandBuffer(kMVKCommandUseEndCommandBuffer, true);
+	return _device->getQueue(_queueFamilyIndex, queueIndex)->getMTLCommandBuffer(cmdUse, true);
 }
 
 // Clear the command type pool member variables.

MoltenVK/MoltenVK/GPUObjects/MVKDevice.h

@@ -53,7 +53,6 @@ class MVKSemaphore;
 class MVKTimelineSemaphore;
 class MVKDeferredOperation;
 class MVKEvent;
-class MVKSemaphoreImpl;
 class MVKQueryPool;
 class MVKShaderModule;
 class MVKPipelineCache;
@@ -440,6 +439,11 @@ protected:
 #pragma mark -
 #pragma mark MVKDevice
 
+typedef enum {
+	MVKActivityPerformanceValueTypeDuration,
+	MVKActivityPerformanceValueTypeByteCount,
+} MVKActivityPerformanceValueType;
+
 typedef struct MVKMTLBlitEncoder {
 	id<MTLBlitCommandEncoder> mtlBlitEncoder = nil;
 	id<MTLCommandBuffer> mtlCmdBuffer = nil;
@@ -704,13 +708,17 @@ public:
 	void addActivityPerformance(MVKPerformanceTracker& activityTracker,
 								uint64_t startTime, uint64_t endTime = 0) {
 		if (_isPerformanceTracking) {
-			updateActivityPerformance(activityTracker, startTime, endTime);
+			updateActivityPerformance(activityTracker, mvkGetElapsedMilliseconds(startTime, endTime));
+		}
+	};
 
-			// Log call not locked. Very minor chance that the tracker data will be updated during log call,
+	/**
-			// resulting in an inconsistent report. Not worth taking lock perf hit for rare inline reporting.
+	 * If performance is being tracked, adds the performance for an activity
-			if (_activityPerformanceLoggingStyle == MVK_CONFIG_ACTIVITY_PERFORMANCE_LOGGING_STYLE_IMMEDIATE) {
+	 * with a kilobyte count, to the given performance statistics.
-				logActivityPerformance(activityTracker, _performanceStatistics, true);
+	 */
-			}
+	void addActivityByteCount(MVKPerformanceTracker& activityTracker, uint64_t byteCount) {
+		if (_isPerformanceTracking) {
+			updateActivityPerformance(activityTracker, double(byteCount / KIBI));
 		}
 	};
 
@@ -885,8 +893,11 @@ protected:
 	template<typename S> void enableFeatures(S* pRequested, VkBool32* pEnabledBools, const VkBool32* pRequestedBools, const VkBool32* pAvailableBools, uint32_t count);
 	void enableExtensions(const VkDeviceCreateInfo* pCreateInfo);
     const char* getActivityPerformanceDescription(MVKPerformanceTracker& activity, MVKPerformanceStatistics& perfStats);
-	void logActivityPerformance(MVKPerformanceTracker& activity, MVKPerformanceStatistics& perfStats, bool isInline = false);
+	MVKActivityPerformanceValueType getActivityPerformanceValueType(MVKPerformanceTracker& activity, MVKPerformanceStatistics& perfStats);
-	void updateActivityPerformance(MVKPerformanceTracker& activity, uint64_t startTime, uint64_t endTime);
+	void logActivityInline(MVKPerformanceTracker& activity, MVKPerformanceStatistics& perfStats);
+	void logActivityDuration(MVKPerformanceTracker& activity, MVKPerformanceStatistics& perfStats, bool isInline = false);
+	void logActivityByteCount(MVKPerformanceTracker& activity, MVKPerformanceStatistics& perfStats, bool isInline = false);
+	void updateActivityPerformance(MVKPerformanceTracker& activity, double currentValue);
 	void getDescriptorVariableDescriptorCountLayoutSupport(const VkDescriptorSetLayoutCreateInfo* pCreateInfo,
 														   VkDescriptorSetLayoutSupport* pSupport,
 														   VkDescriptorSetVariableDescriptorCountLayoutSupport* pVarDescSetCountSupport);
@@ -908,7 +919,6 @@ protected:
 	id<MTLSamplerState> _defaultMTLSamplerState = nil;
 	id<MTLBuffer> _dummyBlitMTLBuffer = nil;
     uint32_t _globalVisibilityQueryCount = 0;
-	MVKConfigActivityPerformanceLoggingStyle _activityPerformanceLoggingStyle = MVK_CONFIG_ACTIVITY_PERFORMANCE_LOGGING_STYLE_FRAME_COUNT;
 	bool _isPerformanceTracking = false;
 	bool _isCurrentlyAutoGPUCapturing = false;
 	bool _isUsingMetalArgumentBuffers = false;
@@ -1056,6 +1066,15 @@ protected:
 #pragma mark -
 #pragma mark Support functions
 
+/**
+ * Returns an autoreleased array containing the MTLDevices available on this system,
+ * sorted according to power, with higher power GPU's at the front of the array.
+ * This ensures that a lazy app that simply grabs the first GPU will get a high-power
+ * one by default. If MVKConfiguration::forceLowPowerGPU is enabled, the returned
+ * array will only include low-power devices.
+ */
+NSArray<id<MTLDevice>>* mvkGetAvailableMTLDevicesArray();
+
 /** Returns the registry ID of the specified device, or zero if the device does not have a registry ID. */
 uint64_t mvkGetRegistryID(id<MTLDevice> mtlDevice);
 

MoltenVK/MoltenVK/GPUObjects/MVKDevice.mm

@@ -3056,32 +3056,23 @@ uint64_t MVKPhysicalDevice::getVRAMSize() {
 	}
 }
 
+// If possible, retrieve from the MTLDevice, otherwise from available memory size, or a fixed conservative estimate.
 uint64_t MVKPhysicalDevice::getRecommendedMaxWorkingSetSize() {
-#if MVK_MACOS
+#if MVK_XCODE_14 || MVK_MACOS
 	if ( [_mtlDevice respondsToSelector: @selector(recommendedMaxWorkingSetSize)]) {
 		return _mtlDevice.recommendedMaxWorkingSetSize;
 	}
 #endif
-#if MVK_IOS_OR_TVOS
-	// GPU and CPU use shared memory. Estimate the current free memory in the system.
 	uint64_t freeMem = mvkGetAvailableMemorySize();
-	if (freeMem) { return freeMem; }
+	return freeMem ? freeMem : 256 * MEBI;
-#endif
-
-	return 128 * MEBI;		// Conservative minimum for macOS GPU's & iOS shared memory
 }
 
+// If possible, retrieve from the MTLDevice, otherwise use the current memory used by this process.
 uint64_t MVKPhysicalDevice::getCurrentAllocatedSize() {
 	if ( [_mtlDevice respondsToSelector: @selector(currentAllocatedSize)] ) {
 		return _mtlDevice.currentAllocatedSize;
 	}
-#if MVK_IOS_OR_TVOS
-	// We can use the current memory used by this process as a reasonable approximation.
 	return mvkGetUsedMemorySize();
-#endif
-#if MVK_MACOS
-	return 0;
-#endif
 }
 
 // When using argument buffers, Metal imposes a hard limit on the number of MTLSamplerState
@@ -3249,31 +3240,14 @@ bool MVKPhysicalDevice::needsCounterSetRetained() {
 }
 
 void MVKPhysicalDevice::logGPUInfo() {
-	string devTypeStr;
-	switch (_properties.deviceType) {
-		case VK_PHYSICAL_DEVICE_TYPE_DISCRETE_GPU:
-			devTypeStr = "Discrete";
-			break;
-		case VK_PHYSICAL_DEVICE_TYPE_INTEGRATED_GPU:
-			devTypeStr = "Integrated";
-			break;
-		case VK_PHYSICAL_DEVICE_TYPE_VIRTUAL_GPU:
-			devTypeStr = "Virtual";
-			break;
-		case VK_PHYSICAL_DEVICE_TYPE_CPU:
-			devTypeStr = "CPU Emulation";
-			break;
-		default:
-			devTypeStr = "Unknown";
-			break;
-	}
-
 	string logMsg = "GPU device:";
 	logMsg += "\n\t\tmodel: %s";
 	logMsg += "\n\t\ttype: %s";
 	logMsg += "\n\t\tvendorID: %#06x";
 	logMsg += "\n\t\tdeviceID: %#06x";
 	logMsg += "\n\t\tpipelineCacheUUID: %s";
+	logMsg += "\n\t\tGPU memory available: %llu MB";
+	logMsg += "\n\t\tGPU memory used: %llu MB";
 	logMsg += "\n\tsupports the following Metal Versions, GPU's and Feature Sets:";
 	logMsg += "\n\t\tMetal Shading Language %s";
 
@@ -3356,9 +3330,29 @@ void MVKPhysicalDevice::logGPUInfo() {
 	}
 #endif
 
+	string devTypeStr;
+	switch (_properties.deviceType) {
+		case VK_PHYSICAL_DEVICE_TYPE_DISCRETE_GPU:
+			devTypeStr = "Discrete";
+			break;
+		case VK_PHYSICAL_DEVICE_TYPE_INTEGRATED_GPU:
+			devTypeStr = "Integrated";
+			break;
+		case VK_PHYSICAL_DEVICE_TYPE_VIRTUAL_GPU:
+			devTypeStr = "Virtual";
+			break;
+		case VK_PHYSICAL_DEVICE_TYPE_CPU:
+			devTypeStr = "CPU Emulation";
+			break;
+		default:
+			devTypeStr = "Unknown";
+			break;
+	}
+
 	NSUUID* nsUUID = [[NSUUID alloc] initWithUUIDBytes: _properties.pipelineCacheUUID];		// temp retain
-	MVKLogInfo(logMsg.c_str(), _properties.deviceName, devTypeStr.c_str(),
+	MVKLogInfo(logMsg.c_str(), getName(), devTypeStr.c_str(),
 			   _properties.vendorID, _properties.deviceID, nsUUID.UUIDString.UTF8String,
+			   getRecommendedMaxWorkingSetSize() / MEBI, getCurrentAllocatedSize() / MEBI,
 			   SPIRVToMSLConversionOptions::printMSLVersion(_metalFeatures.mslVersion).c_str());
 	[nsUUID release];																		// temp release
 }
@@ -3366,7 +3360,11 @@ void MVKPhysicalDevice::logGPUInfo() {
 MVKPhysicalDevice::~MVKPhysicalDevice() {
 	mvkDestroyContainerContents(_queueFamilies);
 	[_timestampMTLCounterSet release];
+
+	uint64_t memUsed = getCurrentAllocatedSize();	// Retrieve before releasing MTLDevice
 	[_mtlDevice release];
+
+	MVKLogInfo("Destroyed VkPhysicalDevice for GPU %s with %llu MB of GPU memory still allocated.", getName(), memUsed / MEBI);
 }
 
 
@@ -4185,30 +4183,52 @@ void MVKDevice::applyMemoryBarrier(VkPipelineStageFlags srcStageMask,
 	}
 }
 
-void MVKDevice::updateActivityPerformance(MVKPerformanceTracker& activity,
+void MVKDevice::updateActivityPerformance(MVKPerformanceTracker& activity, double currentValue) {
-										  uint64_t startTime, uint64_t endTime) {
-
-	double currInterval = mvkGetElapsedMilliseconds(startTime, endTime);
 	lock_guard<mutex> lock(_perfLock);
 
-	activity.latestDuration = currInterval;
+	activity.latest = currentValue;
-	activity.minimumDuration = ((activity.minimumDuration == 0.0)
+	activity.minimum = ((activity.minimum == 0.0)
-								? currInterval :
+								? currentValue :
-								min(currInterval, activity.minimumDuration));
+								min(currentValue, activity.minimum));
-	activity.maximumDuration = max(currInterval, activity.maximumDuration);
+	activity.maximum = max(currentValue, activity.maximum);
-	double totalInterval = (activity.averageDuration * activity.count++) + currInterval;
+	double total = (activity.average * activity.count++) + currentValue;
-	activity.averageDuration = totalInterval / activity.count;
+	activity.average = total / activity.count;
+
+	if (mvkConfig().activityPerformanceLoggingStyle == MVK_CONFIG_ACTIVITY_PERFORMANCE_LOGGING_STYLE_IMMEDIATE) {
+		logActivityInline(activity, _performanceStatistics);
+	}
 }
 
-void MVKDevice::logActivityPerformance(MVKPerformanceTracker& activity, MVKPerformanceStatistics& perfStats, bool isInline) {
+void MVKDevice::logActivityInline(MVKPerformanceTracker& activity, MVKPerformanceStatistics& perfStats) {
-	MVKLogInfo("%s%s%s avg: %.3f ms, latest: %.3f ms, min: %.3f ms, max: %.3f ms, count: %d",
+	if (getActivityPerformanceValueType(activity, _performanceStatistics) == MVKActivityPerformanceValueTypeByteCount) {
-			   (isInline ? "" : "  "),
+		logActivityByteCount(activity, _performanceStatistics, true);
+	} else {
+		logActivityDuration(activity, _performanceStatistics, true);
+	}
+}
+void MVKDevice::logActivityDuration(MVKPerformanceTracker& activity, MVKPerformanceStatistics& perfStats, bool isInline) {
+	const char* fmt = (isInline
+					   ? "%s performance avg: %.3f ms, latest: %.3f ms, min: %.3f ms, max: %.3f ms, count: %d"
+					   : "  %-45s avg: %.3f ms, latest: %.3f ms, min: %.3f ms, max: %.3f ms, count: %d");
+	MVKLogInfo(fmt,
 			   getActivityPerformanceDescription(activity, perfStats),
-			   (isInline ? " performance" : ""),
+			   activity.average,
-			   activity.averageDuration,
+			   activity.latest,
-			   activity.latestDuration,
+			   activity.minimum,
-			   activity.minimumDuration,
+			   activity.maximum,
-			   activity.maximumDuration,
+			   activity.count);
+}
+
+void MVKDevice::logActivityByteCount(MVKPerformanceTracker& activity, MVKPerformanceStatistics& perfStats, bool isInline) {
+	const char* fmt = (isInline
+					   ? "%s avg: %5llu MB, latest: %5llu MB, min: %5llu MB, max: %5llu MB, count: %d"
+					   : "  %-45s avg: %5llu MB, latest: %5llu MB, min: %5llu MB, max: %5llu MB, count: %d");
+	MVKLogInfo(fmt,
+			   getActivityPerformanceDescription(activity, perfStats),
+			   uint64_t(activity.average) / KIBI,
+			   uint64_t(activity.latest) / KIBI,
+			   uint64_t(activity.minimum) / KIBI,
+			   uint64_t(activity.maximum) / KIBI,
 			   activity.count);
 }
 
@@ -4218,49 +4238,71 @@ void MVKDevice::logPerformanceSummary() {
 	MVKPerformanceStatistics perfStats;
 	getPerformanceStatistics(&perfStats);
 
-	logActivityPerformance(perfStats.queue.frameInterval, perfStats);
+#define logDuration(s)   logActivityDuration(perfStats.s, perfStats)
-	logActivityPerformance(perfStats.queue.nextCAMetalDrawable, perfStats);
+#define logByteCount(s)  logActivityByteCount(perfStats.s, perfStats)
-	logActivityPerformance(perfStats.queue.mtlCommandBufferCompletion, perfStats);
+
-	logActivityPerformance(perfStats.queue.mtlQueueAccess, perfStats);
+	logDuration(queue.frameInterval);
-	logActivityPerformance(perfStats.shaderCompilation.hashShaderCode, perfStats);
+	logDuration(queue.retrieveMTLCommandBuffer);
-	logActivityPerformance(perfStats.shaderCompilation.spirvToMSL, perfStats);
+	logDuration(queue.commandBufferEncoding);
-	logActivityPerformance(perfStats.shaderCompilation.mslCompile, perfStats);
+	logDuration(queue.submitCommandBuffers);
-	logActivityPerformance(perfStats.shaderCompilation.mslLoad, perfStats);
+	logDuration(queue.mtlCommandBufferExecution);
-	logActivityPerformance(perfStats.shaderCompilation.mslCompress, perfStats);
+	logDuration(queue.retrieveCAMetalDrawable);
-	logActivityPerformance(perfStats.shaderCompilation.mslDecompress, perfStats);
+	logDuration(queue.presentSwapchains);
-	logActivityPerformance(perfStats.shaderCompilation.shaderLibraryFromCache, perfStats);
+	logDuration(shaderCompilation.hashShaderCode);
-	logActivityPerformance(perfStats.shaderCompilation.functionRetrieval, perfStats);
+	logDuration(shaderCompilation.spirvToMSL);
-	logActivityPerformance(perfStats.shaderCompilation.functionSpecialization, perfStats);
+	logDuration(shaderCompilation.mslCompile);
-	logActivityPerformance(perfStats.shaderCompilation.pipelineCompile, perfStats);
+	logDuration(shaderCompilation.mslLoad);
-	logActivityPerformance(perfStats.pipelineCache.sizePipelineCache, perfStats);
+	logDuration(shaderCompilation.mslCompress);
-	logActivityPerformance(perfStats.pipelineCache.readPipelineCache, perfStats);
+	logDuration(shaderCompilation.mslDecompress);
-	logActivityPerformance(perfStats.pipelineCache.writePipelineCache, perfStats);
+	logDuration(shaderCompilation.shaderLibraryFromCache);
+	logDuration(shaderCompilation.functionRetrieval);
+	logDuration(shaderCompilation.functionSpecialization);
+	logDuration(shaderCompilation.pipelineCompile);
+	logDuration(pipelineCache.sizePipelineCache);
+	logDuration(pipelineCache.readPipelineCache);
+	logDuration(pipelineCache.writePipelineCache);
+	logByteCount(device.gpuMemoryAllocated);
+#undef logDuration
+#undef logByteCount
 }
 
 const char* MVKDevice::getActivityPerformanceDescription(MVKPerformanceTracker& activity, MVKPerformanceStatistics& perfStats) {
-	if (&activity == &perfStats.shaderCompilation.hashShaderCode) { return "Hash shader SPIR-V code"; }
+#define ifActivityReturnName(s, n)  if (&activity == &perfStats.s) return n
-	if (&activity == &perfStats.shaderCompilation.spirvToMSL) { return "Convert SPIR-V to MSL source code"; }
+	ifActivityReturnName(shaderCompilation.hashShaderCode,         "Hash shader SPIR-V code");
-	if (&activity == &perfStats.shaderCompilation.mslCompile) { return "Compile MSL source code into a MTLLibrary"; }
+	ifActivityReturnName(shaderCompilation.spirvToMSL,             "Convert SPIR-V to MSL source code");
-	if (&activity == &perfStats.shaderCompilation.mslLoad) { return "Load pre-compiled MSL code into a MTLLibrary"; }
+	ifActivityReturnName(shaderCompilation.mslCompile,             "Compile MSL into a MTLLibrary");
-	if (&activity == &perfStats.shaderCompilation.mslCompress) { return "Compress MSL source code after compiling a MTLLibrary"; }
+	ifActivityReturnName(shaderCompilation.mslLoad,                "Load pre-compiled MSL into a MTLLibrary");
-	if (&activity == &perfStats.shaderCompilation.mslDecompress) { return "Decompress MSL source code during pipeline cache write"; }
+	ifActivityReturnName(shaderCompilation.mslCompress,            "Compress MSL after compiling a MTLLibrary");
-	if (&activity == &perfStats.shaderCompilation.shaderLibraryFromCache) { return "Retrieve shader library from the cache"; }
+	ifActivityReturnName(shaderCompilation.mslDecompress,          "Decompress MSL for pipeline cache write");
-	if (&activity == &perfStats.shaderCompilation.functionRetrieval) { return "Retrieve a MTLFunction from a MTLLibrary"; }
+	ifActivityReturnName(shaderCompilation.shaderLibraryFromCache, "Retrieve shader library from the cache");
-	if (&activity == &perfStats.shaderCompilation.functionSpecialization) { return "Specialize a retrieved MTLFunction"; }
+	ifActivityReturnName(shaderCompilation.functionRetrieval,      "Retrieve a MTLFunction from a MTLLibrary");
-	if (&activity == &perfStats.shaderCompilation.pipelineCompile) { return "Compile MTLFunctions into a pipeline"; }
+	ifActivityReturnName(shaderCompilation.functionSpecialization, "Specialize a retrieved MTLFunction");
-	if (&activity == &perfStats.pipelineCache.sizePipelineCache) { return "Calculate cache size required to write MSL to pipeline cache"; }
+	ifActivityReturnName(shaderCompilation.pipelineCompile,        "Compile MTLFunctions into a pipeline");
-	if (&activity == &perfStats.pipelineCache.readPipelineCache) { return "Read MSL from pipeline cache"; }
+	ifActivityReturnName(pipelineCache.sizePipelineCache,          "Calculate pipeline cache size");
-	if (&activity == &perfStats.pipelineCache.writePipelineCache) { return "Write MSL to pipeline cache"; }
+	ifActivityReturnName(pipelineCache.readPipelineCache,          "Read MSL from pipeline cache");
-	if (&activity == &perfStats.queue.mtlQueueAccess) { return "Access MTLCommandQueue"; }
+	ifActivityReturnName(pipelineCache.writePipelineCache,         "Write MSL to pipeline cache");
-	if (&activity == &perfStats.queue.mtlCommandBufferCompletion) { return "Complete MTLCommandBuffer"; }
+	ifActivityReturnName(queue.retrieveMTLCommandBuffer,           "Retrieve a MTLCommandBuffer");
-	if (&activity == &perfStats.queue.nextCAMetalDrawable) { return "Retrieve a CAMetalDrawable from CAMetalLayer"; }
+	ifActivityReturnName(queue.commandBufferEncoding,              "Encode VkCommandBuffer to MTLCommandBuffer");
-	if (&activity == &perfStats.queue.frameInterval) { return "Frame interval"; }
+	ifActivityReturnName(queue.submitCommandBuffers,               "vkQueueSubmit() encoding to MTLCommandBuffers");
-	return "Unknown performance activity";
+	ifActivityReturnName(queue.mtlCommandBufferExecution,          "Execute a MTLCommandBuffer on GPU");
+	ifActivityReturnName(queue.retrieveCAMetalDrawable,            "Retrieve a CAMetalDrawable");
+	ifActivityReturnName(queue.presentSwapchains,                  "Present swapchains in on GPU");
+	ifActivityReturnName(queue.frameInterval,                      "Frame interval");
+	ifActivityReturnName(device.gpuMemoryAllocated,                "GPU memory allocated");
+	return                                                         "Unknown performance activity";
+#undef ifActivityReturnName
+}
+
+MVKActivityPerformanceValueType MVKDevice::getActivityPerformanceValueType(MVKPerformanceTracker& activity, MVKPerformanceStatistics& perfStats) {
+	if (&activity == &perfStats.device.gpuMemoryAllocated) return MVKActivityPerformanceValueTypeByteCount;
+	return MVKActivityPerformanceValueTypeDuration;
 }
 
 void MVKDevice::getPerformanceStatistics(MVKPerformanceStatistics* pPerf) {
-    lock_guard<mutex> lock(_perfLock);
+	addActivityByteCount(_performanceStatistics.device.gpuMemoryAllocated,
+						 _physicalDevice->getCurrentAllocatedSize());
 
+	lock_guard<mutex> lock(_perfLock);
     if (pPerf) { *pPerf = _performanceStatistics; }
 }
 
@@ -4597,33 +4639,15 @@ MVKDevice::MVKDevice(MVKPhysicalDevice* physicalDevice, const VkDeviceCreateInfo
 	startAutoGPUCapture(MVK_CONFIG_AUTO_GPU_CAPTURE_SCOPE_DEVICE, getMTLDevice());
 
 	MVKLogInfo("Created VkDevice to run on GPU %s with the following %d Vulkan extensions enabled:%s",
-			   _pProperties->deviceName,
+			   getName(), _enabledExtensions.getEnabledCount(), _enabledExtensions.enabledNamesString("\n\t\t", true).c_str());
-			   _enabledExtensions.getEnabledCount(),
-			   _enabledExtensions.enabledNamesString("\n\t\t", true).c_str());
 }
 
+// Perf stats that last the duration of the app process.
+static MVKPerformanceStatistics _processPerformanceStatistics = {};
+
 void MVKDevice::initPerformanceTracking() {
-
 	_isPerformanceTracking = mvkConfig().performanceTracking;
-	_activityPerformanceLoggingStyle = mvkConfig().activityPerformanceLoggingStyle;
+	_performanceStatistics = _processPerformanceStatistics;
-
-	_performanceStatistics.shaderCompilation.hashShaderCode = {};
-    _performanceStatistics.shaderCompilation.spirvToMSL = {};
-    _performanceStatistics.shaderCompilation.mslCompile = {};
-    _performanceStatistics.shaderCompilation.mslLoad = {};
-	_performanceStatistics.shaderCompilation.mslCompress = {};
-	_performanceStatistics.shaderCompilation.mslDecompress = {};
-	_performanceStatistics.shaderCompilation.shaderLibraryFromCache = {};
-    _performanceStatistics.shaderCompilation.functionRetrieval = {};
-    _performanceStatistics.shaderCompilation.functionSpecialization = {};
-    _performanceStatistics.shaderCompilation.pipelineCompile = {};
-	_performanceStatistics.pipelineCache.sizePipelineCache = {};
-	_performanceStatistics.pipelineCache.writePipelineCache = {};
-	_performanceStatistics.pipelineCache.readPipelineCache = {};
-	_performanceStatistics.queue.mtlQueueAccess = {};
-	_performanceStatistics.queue.mtlCommandBufferCompletion = {};
-	_performanceStatistics.queue.nextCAMetalDrawable = {};
-	_performanceStatistics.queue.frameInterval = {};
 }
 
 void MVKDevice::initPhysicalDevice(MVKPhysicalDevice* physicalDevice, const VkDeviceCreateInfo* pCreateInfo) {
@@ -4920,9 +4944,16 @@ void MVKDevice::reservePrivateData(const VkDeviceCreateInfo* pCreateInfo) {
 }
 
 MVKDevice::~MVKDevice() {
-	if (_activityPerformanceLoggingStyle == MVK_CONFIG_ACTIVITY_PERFORMANCE_LOGGING_STYLE_DEVICE_LIFETIME) {
+	if (_isPerformanceTracking) {
-		MVKLogInfo("Device activity performance summary:");
+		auto perfLogStyle = mvkConfig().activityPerformanceLoggingStyle;
-		logPerformanceSummary();
+		if (perfLogStyle == MVK_CONFIG_ACTIVITY_PERFORMANCE_LOGGING_STYLE_DEVICE_LIFETIME) {
+			MVKLogInfo("Device activity performance summary:");
+			logPerformanceSummary();
+		} else if (perfLogStyle == MVK_CONFIG_ACTIVITY_PERFORMANCE_LOGGING_STYLE_DEVICE_LIFETIME_ACCUMULATE) {
+			MVKLogInfo("Process activity performance summary:");
+			logPerformanceSummary();
+			_processPerformanceStatistics = _performanceStatistics;
+		}
 	}
 
 	for (auto& queues : _queuesByQueueFamilyIndex) {
@@ -4938,12 +4969,58 @@ MVKDevice::~MVKDevice() {
 	stopAutoGPUCapture(MVK_CONFIG_AUTO_GPU_CAPTURE_SCOPE_DEVICE);
 
 	mvkDestroyContainerContents(_privateDataSlots);
+
+	MVKLogInfo("Destroyed VkDevice on GPU %s with %d Vulkan extensions enabled.",
+			   getName(), _enabledExtensions.getEnabledCount());
 }
 
 
 #pragma mark -
 #pragma mark Support functions
 
+NSArray<id<MTLDevice>>* mvkGetAvailableMTLDevicesArray() {
+	NSMutableArray* mtlDevs = [NSMutableArray array];	// autoreleased
+
+#if MVK_MACOS
+	NSArray* rawMTLDevs = [MTLCopyAllDevices() autorelease];
+	bool forceLowPower = mvkConfig().forceLowPowerGPU;
+
+	// Populate the array of appropriate MTLDevices
+	for (id<MTLDevice> md in rawMTLDevs) {
+		if ( !forceLowPower || md.isLowPower ) { [mtlDevs addObject: md]; }
+	}
+
+	// Sort by power
+	[mtlDevs sortUsingComparator: ^(id<MTLDevice> md1, id<MTLDevice> md2) {
+		BOOL md1IsLP = md1.isLowPower;
+		BOOL md2IsLP = md2.isLowPower;
+
+		if (md1IsLP == md2IsLP) {
+			// If one device is headless and the other one is not, select the
+			// one that is not headless first.
+			BOOL md1IsHeadless = md1.isHeadless;
+			BOOL md2IsHeadless = md2.isHeadless;
+			if (md1IsHeadless == md2IsHeadless ) {
+				return NSOrderedSame;
+			}
+			return md2IsHeadless ? NSOrderedAscending : NSOrderedDescending;
+		}
+
+		return md2IsLP ? NSOrderedAscending : NSOrderedDescending;
+	}];
+
+	// If the survey found at least one device, return the array.
+	if (mtlDevs.count) { return mtlDevs; }
+
+#endif	// MVK_MACOS
+
+	// For other OS's, or for macOS if the survey returned empty, use the default device.
+	id<MTLDevice> md = [MTLCreateSystemDefaultDevice() autorelease];
+	if (md) { [mtlDevs addObject: md]; }
+
+	return mtlDevs;		// retained
+}
+
 uint64_t mvkGetRegistryID(id<MTLDevice> mtlDevice) {
 	return [mtlDevice respondsToSelector: @selector(registryID)] ? mtlDevice.registryID : 0;
 }

MoltenVK/MoltenVK/GPUObjects/MVKImage.h

@@ -31,6 +31,7 @@
 class MVKImage;
 class MVKImageView;
 class MVKSwapchain;
+class MVKQueue;
 class MVKCommandEncoder;
 
 
@@ -409,8 +410,8 @@ protected:
 	virtual id<CAMetalDrawable> getCAMetalDrawable() = 0;
 	void detachSwapchain();
 
+	std::mutex _detachmentLock;
 	MVKSwapchain* _swapchain;
-	std::mutex _swapchainLock;
 	uint32_t _swapchainIndex;
 };
 
@@ -429,6 +430,7 @@ typedef struct MVKSwapchainImageAvailability {
 /** Presentation info. */
 typedef struct  {
 	MVKPresentableSwapchainImage* presentableImage;
+	MVKQueue* queue;				// The queue on which the vkQueuePresentKHR() command was executed.
 	MVKFence* fence;				// VK_EXT_swapchain_maintenance1 fence signaled when resources can be destroyed
 	uint64_t desiredPresentTime;  	// VK_GOOGLE_display_timing desired presentation time in nanoseconds
 	uint32_t presentID;           	// VK_GOOGLE_display_timing presentID
@@ -454,12 +456,22 @@ public:
 	/** Presents the contained drawable to the OS. */
 	void presentCAMetalDrawable(id<MTLCommandBuffer> mtlCmdBuff, MVKImagePresentInfo presentInfo);
 
+	/** Called when the presentation begins. */
+	void beginPresentation(const MVKImagePresentInfo& presentInfo);
+
+	/** Called via callback when the presentation completes. */
+	void endPresentation(const MVKImagePresentInfo& presentInfo, uint64_t actualPresentTime = 0);
+
+	/** If this image is stuck in-flight, attempt to force it to complete. */
+	void forcePresentationCompletion();
 
 #pragma mark Construction
 
 	MVKPresentableSwapchainImage(MVKDevice* device, const VkImageCreateInfo* pCreateInfo,
 								 MVKSwapchain* swapchain, uint32_t swapchainIndex);
 
+	void destroy() override;
+
 	~MVKPresentableSwapchainImage() override;
 
 protected:
@@ -471,15 +483,14 @@ protected:
 	MVKSwapchainImageAvailability getAvailability();
 	void makeAvailable(const MVKSwapchainSignaler& signaler);
 	void makeAvailable();
-	void acquireAndSignalWhenAvailable(MVKSemaphore* semaphore, MVKFence* fence);
+	VkResult acquireAndSignalWhenAvailable(MVKSemaphore* semaphore, MVKFence* fence);
-	void renderWatermark(id<MTLCommandBuffer> mtlCmdBuff);
 
-	id<CAMetalDrawable> _mtlDrawable;
+	id<CAMetalDrawable> _mtlDrawable = nil;
-	id<MTLCommandBuffer> _presentingMTLCmdBuff;
 	MVKSwapchainImageAvailability _availability;
 	MVKSmallVector<MVKSwapchainSignaler, 1> _availabilitySignalers;
-	MVKSwapchainSignaler _preSignaler;
+	MVKSwapchainSignaler _preSignaler = {};
 	std::mutex _availabilityLock;
+	uint64_t _presentationStartTime = 0;
 };
 
 

MoltenVK/MoltenVK/GPUObjects/MVKImage.mm

@@ -19,6 +19,7 @@
 #include "MVKImage.h"
 #include "MVKQueue.h"
 #include "MVKSwapchain.h"
+#include "MVKSurface.h"
 #include "MVKCommandBuffer.h"
 #include "MVKCmdDebug.h"
 #include "MVKFoundation.h"
@@ -1192,8 +1193,9 @@ MVKSwapchainImage::MVKSwapchainImage(MVKDevice* device,
 }
 
 void MVKSwapchainImage::detachSwapchain() {
-	lock_guard<mutex> lock(_swapchainLock);
+	lock_guard<mutex> lock(_detachmentLock);
 	_swapchain = nullptr;
+	_device = nullptr;
 }
 
 void MVKSwapchainImage::destroy() {
@@ -1245,7 +1247,7 @@ static void signalAndUnmarkAsTracked(const MVKSwapchainSignaler& signaler) {
 	unmarkAsTracked(signaler);
 }
 
-void MVKPresentableSwapchainImage::acquireAndSignalWhenAvailable(MVKSemaphore* semaphore, MVKFence* fence) {
+VkResult MVKPresentableSwapchainImage::acquireAndSignalWhenAvailable(MVKSemaphore* semaphore, MVKFence* fence) {
 	lock_guard<mutex> lock(_availabilityLock);
 
 	// Upon acquisition, update acquisition ID immediately, to move it to the back of the chain,
@@ -1256,18 +1258,21 @@ void MVKPresentableSwapchainImage::acquireAndSignalWhenAvailable(MVKSemaphore* s
 	// This is not done earlier so the texture is retained for any post-processing such as screen captures, etc.
 	releaseMetalDrawable();
 
+	VkResult rslt = VK_SUCCESS;
 	auto signaler = MVKSwapchainSignaler{fence, semaphore, semaphore ? semaphore->deferSignal() : 0};
 	if (_availability.isAvailable) {
 		_availability.isAvailable = false;
 
-		// If signalling through a MTLEvent, and there's no command buffer presenting me, use an ephemeral MTLCommandBuffer.
+		// If signalling through a MTLEvent, signal through an ephemeral MTLCommandBuffer.
 		// Another option would be to use MTLSharedEvent in MVKSemaphore, but that might
 		// impose unacceptable performance costs to handle this particular case.
 		@autoreleasepool {
 			MVKSemaphore* mvkSem = signaler.semaphore;
-			id<MTLCommandBuffer> mtlCmdBuff = (mvkSem && mvkSem->isUsingCommandEncoding()
+			id<MTLCommandBuffer> mtlCmdBuff = nil;
-											   ? _device->getAnyQueue()->getMTLCommandBuffer(kMVKCommandUseAcquireNextImage)
+			if (mvkSem && mvkSem->isUsingCommandEncoding()) {
-											   : nil);
+				mtlCmdBuff = _device->getAnyQueue()->getMTLCommandBuffer(kMVKCommandUseAcquireNextImage);
+				if ( !mtlCmdBuff ) { rslt = VK_ERROR_OUT_OF_POOL_MEMORY; }
+			}
 			signal(signaler, mtlCmdBuff);
 			[mtlCmdBuff commit];
 		}
@@ -1277,17 +1282,20 @@ void MVKPresentableSwapchainImage::acquireAndSignalWhenAvailable(MVKSemaphore* s
 		_availabilitySignalers.push_back(signaler);
 	}
 	markAsTracked(signaler);
+
+	return rslt;
 }
 
 id<CAMetalDrawable> MVKPresentableSwapchainImage::getCAMetalDrawable() {
-	while ( !_mtlDrawable ) {
+	if ( !_mtlDrawable ) {
-		@autoreleasepool {      // Reclaim auto-released drawable object before end of loop
+		@autoreleasepool {
-			uint64_t startTime = _device->getPerformanceTimestamp();
+			uint32_t attemptCnt = _swapchain->getImageCount() * 2;	// Attempt a resonable number of times
-
+			for (uint32_t attemptIdx = 0; !_mtlDrawable && attemptIdx < attemptCnt; attemptIdx++) {
-			_mtlDrawable = [_swapchain->_mtlLayer.nextDrawable retain];
+				uint64_t startTime = _device->getPerformanceTimestamp();
-			if ( !_mtlDrawable ) { MVKLogError("CAMetalDrawable could not be acquired."); }
+				_mtlDrawable = [_swapchain->_surface->getCAMetalLayer().nextDrawable retain];	// retained
-
+				_device->addActivityPerformance(_device->_performanceStatistics.queue.retrieveCAMetalDrawable, startTime);
-			_device->addActivityPerformance(_device->_performanceStatistics.queue.nextCAMetalDrawable, startTime);
+			}
+			if ( !_mtlDrawable ) { reportError(VK_ERROR_OUT_OF_POOL_MEMORY, "CAMetalDrawable could not be acquired after %d attempts.", attemptCnt); }
 		}
 	}
 	return _mtlDrawable;
@@ -1299,22 +1307,20 @@ void MVKPresentableSwapchainImage::presentCAMetalDrawable(id<MTLCommandBuffer> m
 														  MVKImagePresentInfo presentInfo) {
 	lock_guard<mutex> lock(_availabilityLock);
 
-	_swapchain->willPresentSurface(getMTLTexture(0), mtlCmdBuff);
+	_swapchain->renderWatermark(getMTLTexture(0), mtlCmdBuff);
 
 	// According to Apple, it is more performant to call MTLDrawable present from within a
 	// MTLCommandBuffer scheduled-handler than it is to call MTLCommandBuffer presentDrawable:.
 	// But get current drawable now, intead of in handler, because a new drawable might be acquired by then.
 	// Attach present handler before presenting to avoid race condition.
 	id<CAMetalDrawable> mtlDrwbl = getCAMetalDrawable();
+	addPresentedHandler(mtlDrwbl, presentInfo);
 	[mtlCmdBuff addScheduledHandler: ^(id<MTLCommandBuffer> mcb) {
 		// Try to do any present mode transitions as late as possible in an attempt
 		// to avoid visual disruptions on any presents already on the queue.
 		if (presentInfo.presentMode != VK_PRESENT_MODE_MAX_ENUM_KHR) {
 			mtlDrwbl.layer.displaySyncEnabledMVK = (presentInfo.presentMode != VK_PRESENT_MODE_IMMEDIATE_KHR);
 		}
-		if (presentInfo.hasPresentTime) {
-			addPresentedHandler(mtlDrwbl, presentInfo);
-		}
 		if (presentInfo.desiredPresentTime) {
 			[mtlDrwbl presentAtTime: (double)presentInfo.desiredPresentTime * 1.0e-9];
 		} else {
@@ -1362,34 +1368,45 @@ void MVKPresentableSwapchainImage::presentCAMetalDrawable(id<MTLCommandBuffer> m
 // Pass MVKImagePresentInfo by value because it may not exist when the callback runs.
 void MVKPresentableSwapchainImage::addPresentedHandler(id<CAMetalDrawable> mtlDrawable,
 													   MVKImagePresentInfo presentInfo) {
+	beginPresentation(presentInfo);
+
 #if !MVK_OS_SIMULATOR
 	if ([mtlDrawable respondsToSelector: @selector(addPresentedHandler:)]) {
-		retain();	// Ensure this image is not destroyed while awaiting presentation
+		[mtlDrawable addPresentedHandler: ^(id<MTLDrawable> mtlDrwbl) {
-		[mtlDrawable addPresentedHandler: ^(id<MTLDrawable> drawable) {
+			endPresentation(presentInfo, mtlDrwbl.presentedTime * 1.0e9);
-			// Since we're in a callback, it's possible that the swapchain has been released by now.
-			// Lock the swapchain, and test if it is present before doing anything with it.
-			lock_guard<mutex> cblock(_swapchainLock);
-			if (_swapchain) { _swapchain->recordPresentTime(presentInfo, drawable.presentedTime * 1.0e9); }
-			release();
 		}];
-		return;
+	} else
-	}
 #endif
-
+	{
-	// If MTLDrawable.presentedTime/addPresentedHandler isn't supported,
+		// If MTLDrawable.presentedTime/addPresentedHandler isn't supported,
-	// treat it as if the present happened when requested.
+		// treat it as if the present happened when requested.
-	// Since this function may be called in a callback, it's possible that
+		endPresentation(presentInfo);
-	// the swapchain has been released by the time this function runs.
+	}
-	// Lock the swapchain, and test if it is present before doing anything with it.
-	lock_guard<mutex> lock(_swapchainLock);
-	if (_swapchain) {_swapchain->recordPresentTime(presentInfo); }
 }
 
-// Resets the MTLTexture and CAMetalDrawable underlying this image.
+// Ensure this image and the swapchain are not destroyed while awaiting presentation
+void MVKPresentableSwapchainImage::beginPresentation(const MVKImagePresentInfo& presentInfo) {
+	retain();
+	_swapchain->beginPresentation(presentInfo);
+	presentInfo.queue->beginPresentation(presentInfo);
+	_presentationStartTime = getDevice()->getPerformanceTimestamp();
+}
+
+void MVKPresentableSwapchainImage::endPresentation(const MVKImagePresentInfo& presentInfo,
+												   uint64_t actualPresentTime) {
+	{	// Scope to avoid deadlock if release() is run within detachment lock
+		// If I have become detached from the swapchain, it means the swapchain, and possibly the
+		// VkDevice, have been destroyed by the time of this callback, so do not reference them.
+		lock_guard<mutex> lock(_detachmentLock);
+		if (_device) { _device->addActivityPerformance(_device->_performanceStatistics.queue.presentSwapchains, _presentationStartTime); }
+		if (_swapchain) { _swapchain->endPresentation(presentInfo, actualPresentTime); }
+	}
+	presentInfo.queue->endPresentation(presentInfo);
+	release();
+}
+
+// Releases the CAMetalDrawable underlying this image.
 void MVKPresentableSwapchainImage::releaseMetalDrawable() {
-    for (uint8_t planeIndex = 0; planeIndex < _planes.size(); ++planeIndex) {
-        _planes[planeIndex]->releaseMTLTexture();
-    }
     [_mtlDrawable release];
 	_mtlDrawable = nil;
 }
@@ -1417,6 +1434,13 @@ void MVKPresentableSwapchainImage::makeAvailable() {
 	}
 }
 
+// Clear the existing CAMetalDrawable and retrieve and release a new transient one,
+// in an attempt to trigger the existing CAMetalDrawable to complete it's callback.
+void MVKPresentableSwapchainImage::forcePresentationCompletion() {
+	releaseMetalDrawable();
+	if (_swapchain) { @autoreleasepool { [_swapchain->_surface->getCAMetalLayer() nextDrawable]; } }
+}
+
 
 #pragma mark Construction
 
@@ -1426,11 +1450,14 @@ MVKPresentableSwapchainImage::MVKPresentableSwapchainImage(MVKDevice* device,
 														   uint32_t swapchainIndex) :
 	MVKSwapchainImage(device, pCreateInfo, swapchain, swapchainIndex) {
 
-	_mtlDrawable = nil;
-
 	_availability.acquisitionID = _swapchain->getNextAcquisitionID();
 	_availability.isAvailable = true;
-	_preSignaler = MVKSwapchainSignaler{nullptr, nullptr, 0};
+}
+
+
+void MVKPresentableSwapchainImage::destroy() {
+	forcePresentationCompletion();
+	MVKSwapchainImage::destroy();
 }
 
 // Unsignaled signalers will exist if this image is acquired more than it is presented.

MoltenVK/MoltenVK/GPUObjects/MVKInstance.h

@@ -179,7 +179,6 @@ protected:
 	void propagateDebugName() override {}
 	void initProcAddrs();
 	void initDebugCallbacks(const VkInstanceCreateInfo* pCreateInfo);
-	NSArray<id<MTLDevice>>* getAvailableMTLDevicesArray();
 	VkDebugReportFlagsEXT getVkDebugReportFlagsFromLogLevel(MVKConfigLogLevel logLevel);
 	VkDebugUtilsMessageSeverityFlagBitsEXT getVkDebugUtilsMessageSeverityFlagBitsFromLogLevel(MVKConfigLogLevel logLevel);
 	VkDebugUtilsMessageTypeFlagsEXT getVkDebugUtilsMessageTypesFlagBitsFromLogLevel(MVKConfigLogLevel logLevel);

MoltenVK/MoltenVK/GPUObjects/MVKInstance.mm

@@ -238,94 +238,37 @@ void MVKInstance::debugReportMessage(MVKVulkanAPIObject* mvkAPIObj, MVKConfigLog
 
 VkDebugReportFlagsEXT MVKInstance::getVkDebugReportFlagsFromLogLevel(MVKConfigLogLevel logLevel) {
 	switch (logLevel) {
-		case MVK_CONFIG_LOG_LEVEL_DEBUG:
+		case MVK_CONFIG_LOG_LEVEL_ERROR:    return VK_DEBUG_REPORT_ERROR_BIT_EXT;
-			return VK_DEBUG_REPORT_DEBUG_BIT_EXT;
+		case MVK_CONFIG_LOG_LEVEL_WARNING:  return VK_DEBUG_REPORT_WARNING_BIT_EXT;
-		case MVK_CONFIG_LOG_LEVEL_INFO:
+		case MVK_CONFIG_LOG_LEVEL_INFO:     return VK_DEBUG_REPORT_INFORMATION_BIT_EXT;
-			return VK_DEBUG_REPORT_INFORMATION_BIT_EXT;
+		case MVK_CONFIG_LOG_LEVEL_DEBUG:    return VK_DEBUG_REPORT_DEBUG_BIT_EXT;
-		case MVK_CONFIG_LOG_LEVEL_WARNING:
+		default:                            return VK_DEBUG_REPORT_ERROR_BIT_EXT;
-			return VK_DEBUG_REPORT_WARNING_BIT_EXT;
-		case MVK_CONFIG_LOG_LEVEL_ERROR:
-		default:
-			return VK_DEBUG_REPORT_ERROR_BIT_EXT;
 	}
 }
 
 VkDebugUtilsMessageSeverityFlagBitsEXT MVKInstance::getVkDebugUtilsMessageSeverityFlagBitsFromLogLevel(MVKConfigLogLevel logLevel) {
 	switch (logLevel) {
-		case MVK_CONFIG_LOG_LEVEL_DEBUG:
+		case MVK_CONFIG_LOG_LEVEL_ERROR:    return VK_DEBUG_UTILS_MESSAGE_SEVERITY_ERROR_BIT_EXT;
-			return VK_DEBUG_UTILS_MESSAGE_SEVERITY_VERBOSE_BIT_EXT;
+		case MVK_CONFIG_LOG_LEVEL_WARNING:  return VK_DEBUG_UTILS_MESSAGE_SEVERITY_WARNING_BIT_EXT;
-		case MVK_CONFIG_LOG_LEVEL_INFO:
+		case MVK_CONFIG_LOG_LEVEL_INFO:     return VK_DEBUG_UTILS_MESSAGE_SEVERITY_INFO_BIT_EXT;
-			return VK_DEBUG_UTILS_MESSAGE_SEVERITY_INFO_BIT_EXT;
+		case MVK_CONFIG_LOG_LEVEL_DEBUG:    return VK_DEBUG_UTILS_MESSAGE_SEVERITY_VERBOSE_BIT_EXT;
-		case MVK_CONFIG_LOG_LEVEL_WARNING:
+		default:                            return VK_DEBUG_UTILS_MESSAGE_SEVERITY_ERROR_BIT_EXT;
-			return VK_DEBUG_UTILS_MESSAGE_SEVERITY_WARNING_BIT_EXT;
-		case MVK_CONFIG_LOG_LEVEL_ERROR:
-		default:
-			return VK_DEBUG_UTILS_MESSAGE_SEVERITY_ERROR_BIT_EXT;
 	}
 }
 
 VkDebugUtilsMessageTypeFlagsEXT MVKInstance::getVkDebugUtilsMessageTypesFlagBitsFromLogLevel(MVKConfigLogLevel logLevel) {
 	switch (logLevel) {
-		case MVK_CONFIG_LOG_LEVEL_DEBUG:
+		case MVK_CONFIG_LOG_LEVEL_ERROR:    return VK_DEBUG_UTILS_MESSAGE_TYPE_VALIDATION_BIT_EXT;
-		case MVK_CONFIG_LOG_LEVEL_INFO:
+		case MVK_CONFIG_LOG_LEVEL_WARNING:  return VK_DEBUG_UTILS_MESSAGE_TYPE_VALIDATION_BIT_EXT;
-		case MVK_CONFIG_LOG_LEVEL_WARNING:
+		case MVK_CONFIG_LOG_LEVEL_DEBUG:    return VK_DEBUG_UTILS_MESSAGE_TYPE_GENERAL_BIT_EXT;
-			return VK_DEBUG_UTILS_MESSAGE_TYPE_GENERAL_BIT_EXT;
+		case MVK_CONFIG_LOG_LEVEL_INFO:     return VK_DEBUG_UTILS_MESSAGE_TYPE_GENERAL_BIT_EXT;
-		case MVK_CONFIG_LOG_LEVEL_ERROR:
+		default:                            return VK_DEBUG_UTILS_MESSAGE_TYPE_VALIDATION_BIT_EXT;
-		default:
-			return VK_DEBUG_UTILS_MESSAGE_TYPE_VALIDATION_BIT_EXT;
 	}
 }
 
 
 #pragma mark Object Creation
 
-// Returns an autoreleased array containing the MTLDevices available on this system, sorted according
-// to power, with higher power GPU's at the front of the array. This ensures that a lazy app that simply
-// grabs the first GPU will get a high-power one by default. If MVKConfiguration::forceLowPowerGPU is set,
-// the returned array will only include low-power devices.
-NSArray<id<MTLDevice>>* MVKInstance::getAvailableMTLDevicesArray() {
-	NSMutableArray* mtlDevs = [NSMutableArray array];
-
-#if MVK_MACOS
-	NSArray* rawMTLDevs = [MTLCopyAllDevices() autorelease];
-	if (rawMTLDevs) {
-		bool forceLowPower = mvkConfig().forceLowPowerGPU;
-
-		// Populate the array of appropriate MTLDevices
-		for (id<MTLDevice> md in rawMTLDevs) {
-			if ( !forceLowPower || md.isLowPower ) { [mtlDevs addObject: md]; }
-		}
-
-		// Sort by power
-		[mtlDevs sortUsingComparator: ^(id<MTLDevice> md1, id<MTLDevice> md2) {
-			BOOL md1IsLP = md1.isLowPower;
-			BOOL md2IsLP = md2.isLowPower;
-
-			if (md1IsLP == md2IsLP) {
-				// If one device is headless and the other one is not, select the
-				// one that is not headless first.
-				BOOL md1IsHeadless = md1.isHeadless;
-				BOOL md2IsHeadless = md2.isHeadless;
-				if (md1IsHeadless == md2IsHeadless ) {
-					return NSOrderedSame;
-				}
-				return md2IsHeadless ? NSOrderedAscending : NSOrderedDescending;
-			}
-
-			return md2IsLP ? NSOrderedAscending : NSOrderedDescending;
-		}];
-
-	}
-#endif	// MVK_MACOS
-
-#if MVK_IOS_OR_TVOS
-	id<MTLDevice> md = [MTLCreateSystemDefaultDevice() autorelease];
-	if (md) { [mtlDevs addObject: md]; }
-#endif	// MVK_IOS_OR_TVOS
-
-	return mtlDevs;		// retained
-}
-
 MVKInstance::MVKInstance(const VkInstanceCreateInfo* pCreateInfo) : _enabledExtensions(this) {
 
 	initDebugCallbacks(pCreateInfo);	// Do before any creation activities
@@ -347,7 +290,7 @@ MVKInstance::MVKInstance(const VkInstanceCreateInfo* pCreateInfo) : _enabledExte
 	// This effort creates a number of autoreleased instances of Metal
 	// and other Obj-C classes, so wrap it all in an autorelease pool.
 	@autoreleasepool {
-		NSArray<id<MTLDevice>>* mtlDevices = getAvailableMTLDevicesArray();
+		NSArray<id<MTLDevice>>* mtlDevices = mvkGetAvailableMTLDevicesArray();
 		_physicalDevices.reserve(mtlDevices.count);
 		for (id<MTLDevice> mtlDev in mtlDevices) {
 			_physicalDevices.push_back(new MVKPhysicalDevice(this, mtlDev));
@@ -782,5 +725,9 @@ MVKInstance::~MVKInstance() {
 
 	lock_guard<mutex> lock(_dcbLock);
 	mvkDestroyContainerContents(_debugReportCallbacks);
+
+	MVKLogInfo("Destroyed VkInstance for Vulkan version %s with %d Vulkan extensions enabled.",
+			   mvkGetVulkanVersionString(_appInfo.apiVersion).c_str(),
+			   _enabledExtensions.getEnabledCount());
 }
 

MoltenVK/MoltenVK/GPUObjects/MVKPixelFormats.mm

@@ -1482,26 +1482,21 @@ void MVKPixelFormats::addMTLVertexFormatCapabilities(id<MTLDevice> mtlDevice,
 	}
 }
 
-// If supporting a physical device, retrieve the MTLDevice from it,
+// If supporting a physical device, retrieve the MTLDevice from it, otherwise
-// otherwise create a temp copy of the system default MTLDevice.
+// retrieve the array of physical GPU devices, and use the first one.
+// Retrieving the GPUs creates a number of autoreleased instances of Metal
+// and other Obj-C classes, so wrap it all in an autorelease pool.
 void MVKPixelFormats::modifyMTLFormatCapabilities() {
 	if (_physicalDevice) {
 		modifyMTLFormatCapabilities(_physicalDevice->getMTLDevice());
 	} else {
-#if MVK_IOS_OR_TVOS
+		@autoreleasepool {
-		id<MTLDevice> mtlDevice = MTLCreateSystemDefaultDevice();	// temp retained
+			auto* mtlDevs = mvkGetAvailableMTLDevicesArray();
-#endif
+			if (mtlDevs.count) { modifyMTLFormatCapabilities(mtlDevs[0]); }
-#if MVK_MACOS
+		}
-		NSArray<id<MTLDevice>>* mtlDevices = MTLCopyAllDevices();	// temp retained
-		id<MTLDevice> mtlDevice = [mtlDevices count] > 0 ? [mtlDevices[0] retain] : MTLCreateSystemDefaultDevice();			// temp retained
-		[mtlDevices release];										// temp release
-#endif
-		modifyMTLFormatCapabilities(mtlDevice);
-		[mtlDevice release];										// release temp instance
 	}
 }
 
-
 // Mac Catalyst does not support feature sets, so we redefine them to GPU families in MVKDevice.h.
 #if MVK_MACCAT
 #define addFeatSetMTLPixFmtCaps(FEAT_SET, MTL_FMT, CAPS)  \

MoltenVK/MoltenVK/GPUObjects/MVKQueue.h

@@ -86,6 +86,9 @@ public:
 	/** Returns a pointer to the Vulkan instance. */
 	MVKInstance* getInstance() override { return _device->getInstance(); }
 
+	/** Return the name of this queue. */
+	const std::string& getName() { return _name; }
+
 #pragma mark Queue submissions
 
 	/** Submits the specified command buffers to the queue. */
@@ -97,8 +100,11 @@ public:
 	/** Block the current thread until this queue is idle. */
 	VkResult waitIdle(MVKCommandUse cmdUse);
 
-	/** Return the name of this queue. */
+	/** Mark the beginning of a swapchain image presentation. */
-	const std::string& getName() { return _name; }
+	void beginPresentation(const MVKImagePresentInfo& presentInfo);
+
+	/** Mark the end of a swapchain image presentation. */
+	void endPresentation(const MVKImagePresentInfo& presentInfo);
 
 
 #pragma mark Metal
@@ -140,25 +146,29 @@ protected:
 	void initName();
 	void initExecQueue();
 	void initMTLCommandQueue();
-	void initGPUCaptureScopes();
 	void destroyExecQueue();
 	VkResult submit(MVKQueueSubmission* qSubmit);
 	NSString* getMTLCommandBufferLabel(MVKCommandUse cmdUse);
+	void handleMTLCommandBufferError(id<MTLCommandBuffer> mtlCmdBuff);
+	void waitSwapchainPresentations(MVKCommandUse cmdUse);
 
 	MVKQueueFamily* _queueFamily;
-	uint32_t _index;
+	MVKSemaphoreImpl _presentationCompletionBlocker;
-	float _priority;
+	std::unordered_map<MVKPresentableSwapchainImage*, uint32_t> _presentedImages;
-	dispatch_queue_t _execQueue;
-	id<MTLCommandQueue> _mtlQueue;
 	std::string _name;
-	NSString* _mtlCmdBuffLabelEndCommandBuffer;
+	dispatch_queue_t _execQueue;
-	NSString* _mtlCmdBuffLabelQueueSubmit;
+	id<MTLCommandQueue> _mtlQueue = nil;
-	NSString* _mtlCmdBuffLabelQueuePresent;
+	NSString* _mtlCmdBuffLabelBeginCommandBuffer = nil;
-	NSString* _mtlCmdBuffLabelDeviceWaitIdle;
+	NSString* _mtlCmdBuffLabelQueueSubmit = nil;
-	NSString* _mtlCmdBuffLabelQueueWaitIdle;
+	NSString* _mtlCmdBuffLabelQueuePresent = nil;
-	NSString* _mtlCmdBuffLabelAcquireNextImage;
+	NSString* _mtlCmdBuffLabelDeviceWaitIdle = nil;
-	NSString* _mtlCmdBuffLabelInvalidateMappedMemoryRanges;
+	NSString* _mtlCmdBuffLabelQueueWaitIdle = nil;
-	MVKGPUCaptureScope* _submissionCaptureScope;
+	NSString* _mtlCmdBuffLabelAcquireNextImage = nil;
+	NSString* _mtlCmdBuffLabelInvalidateMappedMemoryRanges = nil;
+	MVKGPUCaptureScope* _submissionCaptureScope = nil;
+	std::mutex _presentedImagesLock;
+	float _priority;
+	uint32_t _index;
 };
 
 
@@ -178,7 +188,7 @@ public:
 	 *
 	 * Upon completion of this function, no further calls should be made to this instance.
 	 */
-	virtual void execute() = 0;
+	virtual VkResult execute() = 0;
 
 	MVKQueueSubmission(MVKQueue* queue,
 					   uint32_t waitSemaphoreCount,
@@ -190,6 +200,7 @@ protected:
 	friend class MVKQueue;
 
 	virtual void finish() = 0;
+	MVKDevice* getDevice() { return _queue->getDevice(); }
 
 	MVKQueue* _queue;
 	MVKSmallVector<std::pair<MVKSemaphore*, uint64_t>> _waitSemaphores;
@@ -206,7 +217,7 @@ protected:
 class MVKQueueCommandBufferSubmission : public MVKQueueSubmission {
 
 public:
-	void execute() override;
+	VkResult execute() override;
 
 	MVKQueueCommandBufferSubmission(MVKQueue* queue, const VkSubmitInfo* pSubmit, VkFence fence, MVKCommandUse cmdUse);
 
@@ -217,7 +228,7 @@ protected:
 
 	id<MTLCommandBuffer> getActiveMTLCommandBuffer();
 	void setActiveMTLCommandBuffer(id<MTLCommandBuffer> mtlCmdBuff);
-	void commitActiveMTLCommandBuffer(bool signalCompletion = false);
+	VkResult commitActiveMTLCommandBuffer(bool signalCompletion = false);
 	void finish() override;
 	virtual void submitCommandBuffers() {}
 
@@ -238,20 +249,10 @@ template <size_t N>
 class MVKQueueFullCommandBufferSubmission : public MVKQueueCommandBufferSubmission {
 
 public:
-	MVKQueueFullCommandBufferSubmission(MVKQueue* queue, const VkSubmitInfo* pSubmit, VkFence fence) :
+	MVKQueueFullCommandBufferSubmission(MVKQueue* queue,
-		MVKQueueCommandBufferSubmission(queue, pSubmit, fence, kMVKCommandUseQueueSubmit) {
+										const VkSubmitInfo* pSubmit,
-
+										VkFence fence,
-			// pSubmit can be null if just tracking the fence alone
+										MVKCommandUse cmdUse);
-			if (pSubmit) {
-				uint32_t cbCnt = pSubmit->commandBufferCount;
-				_cmdBuffers.reserve(cbCnt);
-				for (uint32_t i = 0; i < cbCnt; i++) {
-					MVKCommandBuffer* cb = MVKCommandBuffer::getMVKCommandBuffer(pSubmit->pCommandBuffers[i]);
-					_cmdBuffers.push_back(cb);
-					setConfigurationResult(cb->getConfigurationResult());
-				}
-			}
-		}
 
 protected:
 	void submitCommandBuffers() override;
@@ -267,7 +268,7 @@ protected:
 class MVKQueuePresentSurfaceSubmission : public MVKQueueSubmission {
 
 public:
-	void execute() override;
+	VkResult execute() override;
 
 	MVKQueuePresentSurfaceSubmission(MVKQueue* queue,
 									 const VkPresentInfoKHR* pPresentInfo);

MoltenVK/MoltenVK/GPUObjects/MVKQueue.mm

@@ -18,6 +18,7 @@
 
 #include "MVKInstance.h"
 #include "MVKQueue.h"
+#include "MVKSurface.h"
 #include "MVKSwapchain.h"
 #include "MVKSync.h"
 #include "MVKFoundation.h"
@@ -68,7 +69,7 @@ void MVKQueue::propagateDebugName() { setLabelIfNotNil(_mtlQueue, _debugName); }
 
 // Execute the queue submission under an autoreleasepool to ensure transient Metal objects are autoreleased.
 // This is critical for apps that don't use standard OS autoreleasing runloop threading.
-static inline void execute(MVKQueueSubmission* qSubmit) { @autoreleasepool { qSubmit->execute(); } }
+static inline VkResult execute(MVKQueueSubmission* qSubmit) { @autoreleasepool { return qSubmit->execute(); } }
 
 // Executes the submmission, either immediately, or by dispatching to an execution queue.
 // Submissions to the execution queue are wrapped in a dedicated autoreleasepool.
@@ -80,10 +81,12 @@ VkResult MVKQueue::submit(MVKQueueSubmission* qSubmit) {
 	if ( !qSubmit ) { return VK_SUCCESS; }     // Ignore nils
 
 	VkResult rslt = qSubmit->getConfigurationResult();     // Extract result before submission to avoid race condition with early destruction
-	if (_execQueue) {
+	if (rslt == VK_SUCCESS) {
-		dispatch_async(_execQueue, ^{ execute(qSubmit); } );
+		if (_execQueue) {
-	} else {
+			dispatch_async(_execQueue, ^{ execute(qSubmit); } );
-		execute(qSubmit);
+		} else {
+			rslt = execute(qSubmit);
+		}
 	}
 	return rslt;
 }
@@ -103,19 +106,19 @@ VkResult MVKQueue::submit(uint32_t submitCount, const VkSubmitInfo* pSubmits, Vk
 		MVKQueueCommandBufferSubmission* mvkSub;
 		uint32_t cbCnt = pVkSub->commandBufferCount;
 		if (cbCnt <= 1) {
-			mvkSub = new MVKQueueFullCommandBufferSubmission<1>(this, pVkSub, fenceOrNil);
+			mvkSub = new MVKQueueFullCommandBufferSubmission<1>(this, pVkSub, fenceOrNil, cmdUse);
 		} else if (cbCnt <= 16) {
-			mvkSub = new MVKQueueFullCommandBufferSubmission<16>(this, pVkSub, fenceOrNil);
+			mvkSub = new MVKQueueFullCommandBufferSubmission<16>(this, pVkSub, fenceOrNil, cmdUse);
 		} else if (cbCnt <= 32) {
-			mvkSub = new MVKQueueFullCommandBufferSubmission<32>(this, pVkSub, fenceOrNil);
+			mvkSub = new MVKQueueFullCommandBufferSubmission<32>(this, pVkSub, fenceOrNil, cmdUse);
 		} else if (cbCnt <= 64) {
-			mvkSub = new MVKQueueFullCommandBufferSubmission<64>(this, pVkSub, fenceOrNil);
+			mvkSub = new MVKQueueFullCommandBufferSubmission<64>(this, pVkSub, fenceOrNil, cmdUse);
 		} else if (cbCnt <= 128) {
-			mvkSub = new MVKQueueFullCommandBufferSubmission<128>(this, pVkSub, fenceOrNil);
+			mvkSub = new MVKQueueFullCommandBufferSubmission<128>(this, pVkSub, fenceOrNil, cmdUse);
 		} else if (cbCnt <= 256) {
-			mvkSub = new MVKQueueFullCommandBufferSubmission<256>(this, pVkSub, fenceOrNil);
+			mvkSub = new MVKQueueFullCommandBufferSubmission<256>(this, pVkSub, fenceOrNil, cmdUse);
 		} else {
-			mvkSub = new MVKQueueFullCommandBufferSubmission<512>(this, pVkSub, fenceOrNil);
+			mvkSub = new MVKQueueFullCommandBufferSubmission<512>(this, pVkSub, fenceOrNil, cmdUse);
 		}
 
         VkResult subRslt = submit(mvkSub);
@@ -128,29 +131,62 @@ VkResult MVKQueue::submit(const VkPresentInfoKHR* pPresentInfo) {
 	return submit(new MVKQueuePresentSurfaceSubmission(this, pPresentInfo));
 }
 
-// Create an empty submit struct and fence, submit to queue and wait on fence.
 VkResult MVKQueue::waitIdle(MVKCommandUse cmdUse) {
 
-	if (_device->getConfigurationResult() != VK_SUCCESS) { return _device->getConfigurationResult(); }
+	VkResult rslt = _device->getConfigurationResult();
+	if (rslt != VK_SUCCESS) { return rslt; }
 
-	VkFenceCreateInfo vkFenceInfo = {
+	auto* mtlCmdBuff = getMTLCommandBuffer(cmdUse);
-		.sType = VK_STRUCTURE_TYPE_FENCE_CREATE_INFO,
+	[mtlCmdBuff commit];
-		.pNext = nullptr,
+	[mtlCmdBuff waitUntilCompleted];
-		.flags = 0,
-	};
 
-	// The MVKFence is retained by the command submission, and may outlive this function while
+	waitSwapchainPresentations(cmdUse);
-	// the command submission finishes, so we can't allocate MVKFence locally on the stack.
+
-	MVKFence* mvkFence = new MVKFence(_device, &vkFenceInfo);
+	return VK_SUCCESS;
-	VkFence vkFence = (VkFence)mvkFence;
+}
-	submit(0, nullptr, vkFence, cmdUse);
+
-	VkResult rslt = mvkWaitForFences(_device, 1, &vkFence, false);
+// If there are any swapchain presentations in flight, wait a few frames for them to complete.
-	mvkFence->destroy();
+// If they don't complete within a few frames, attempt to force them to complete, and wait another
-	return rslt;
+// few frames for that to happen. If there are still swapchain presentations that haven't completed,
+// log a warning, and force them to end presentation, so the images and drawables will be released.
+void MVKQueue::waitSwapchainPresentations(MVKCommandUse cmdUse) {
+	auto waitFrames = _device->_pMetalFeatures->maxSwapchainImageCount + 2;
+	if (_presentationCompletionBlocker.wait((waitFrames/60.0) * 1e9)) { return; }
+
+	auto imgCnt = _presentationCompletionBlocker.getReservationCount();
+	MVKPresentableSwapchainImage* images[imgCnt];
+	mvkClear(images, imgCnt);
+
+	{
+		// Scope of image lock limited to creating array copy of uncompleted presentations
+		// Populate a working array of the unpresented images.
+		lock_guard<mutex> lock(_presentedImagesLock);
+		size_t imgIdx = 0;
+		for (auto imgPair : _presentedImages) { images[imgIdx++] = imgPair.first; }
+	}
+
+	// Attempt to force each image to complete presentation through the callback.
+	for (size_t imgIdx = 0; imgIdx < imgCnt && _presentationCompletionBlocker.getReservationCount(); imgIdx++) {
+		auto* img = images[imgIdx];
+		if (img) { img->forcePresentationCompletion(); }
+	}
+
+	// Wait for forced presentation completions. If we still have unfinished swapchain image
+	// presentations, log a warning, and force each image to end, so that it can be released.
+	if ( !_presentationCompletionBlocker.wait((waitFrames/60.0) * 1e9) ) {
+		reportWarning(VK_TIMEOUT, "%s timed out after %d frames while awaiting %d swapchain image presentations to complete.",
+					  mvkVkCommandName(cmdUse), waitFrames * 2, _presentationCompletionBlocker.getReservationCount());
+		for (size_t imgIdx = 0; imgIdx < imgCnt; imgIdx++) {
+			auto* img = images[imgIdx];
+			if (_presentedImages.count(img)) { img->endPresentation({.queue = this, .presentableImage = img}); }
+		}
+	}
 }
 
 id<MTLCommandBuffer> MVKQueue::getMTLCommandBuffer(MVKCommandUse cmdUse, bool retainRefs) {
 	id<MTLCommandBuffer> mtlCmdBuff = nil;
+	MVKDevice* mvkDev = getDevice();
+	uint64_t startTime = mvkDev->getPerformanceTimestamp();
 #if MVK_XCODE_12
 	if ([_mtlQueue respondsToSelector: @selector(commandBufferWithDescriptor:)]) {
 		MTLCommandBufferDescriptor* mtlCmdBuffDesc = [MTLCommandBufferDescriptor new];	// temp retain
@@ -167,53 +203,145 @@ id<MTLCommandBuffer> MVKQueue::getMTLCommandBuffer(MVKCommandUse cmdUse, bool re
 	} else {
 		mtlCmdBuff = [_mtlQueue commandBufferWithUnretainedReferences];
 	}
-	setLabelIfNotNil(mtlCmdBuff, getMTLCommandBufferLabel(cmdUse));
+	mvkDev->addActivityPerformance(mvkDev->_performanceStatistics.queue.retrieveMTLCommandBuffer, startTime);
+	NSString* mtlCmdBuffLabel = getMTLCommandBufferLabel(cmdUse);
+	setLabelIfNotNil(mtlCmdBuff, mtlCmdBuffLabel);
+	[mtlCmdBuff addCompletedHandler: ^(id<MTLCommandBuffer> mtlCB) { handleMTLCommandBufferError(mtlCB); }];
+
+	if ( !mtlCmdBuff ) { reportError(VK_ERROR_OUT_OF_POOL_MEMORY, "%s could not be acquired.", mtlCmdBuffLabel.UTF8String); }
 	return mtlCmdBuff;
 }
 
 NSString* MVKQueue::getMTLCommandBufferLabel(MVKCommandUse cmdUse) {
-#define CASE_GET_LABEL(cmdUse)  \
+#define CASE_GET_LABEL(cu)  \
-	case kMVKCommandUse ##cmdUse:  \
+	case kMVKCommandUse ##cu:  \
-		if ( !_mtlCmdBuffLabel ##cmdUse ) { _mtlCmdBuffLabel ##cmdUse = [[NSString stringWithFormat: @"%@ on Queue %d-%d", mvkMTLCommandBufferLabel(kMVKCommandUse ##cmdUse), _queueFamily->getIndex(), _index] retain]; }  \
+		if ( !_mtlCmdBuffLabel ##cu ) { _mtlCmdBuffLabel ##cu = [[NSString stringWithFormat: @"%s MTLCommandBuffer on Queue %d-%d", mvkVkCommandName(kMVKCommandUse ##cu), _queueFamily->getIndex(), _index] retain]; }  \
-		return _mtlCmdBuffLabel ##cmdUse
+		return _mtlCmdBuffLabel ##cu
 
 	switch (cmdUse) {
-		CASE_GET_LABEL(EndCommandBuffer);
+		CASE_GET_LABEL(BeginCommandBuffer);
 		CASE_GET_LABEL(QueueSubmit);
 		CASE_GET_LABEL(QueuePresent);
 		CASE_GET_LABEL(QueueWaitIdle);
 		CASE_GET_LABEL(DeviceWaitIdle);
 		CASE_GET_LABEL(AcquireNextImage);
 		CASE_GET_LABEL(InvalidateMappedMemoryRanges);
-		default: return mvkMTLCommandBufferLabel(cmdUse);
+		default:
+			MVKAssert(false, "Uncached MTLCommandBuffer label for command use %s.", mvkVkCommandName(cmdUse));
+			return [NSString stringWithFormat: @"%s MTLCommandBuffer on Queue %d-%d", mvkVkCommandName(cmdUse), _queueFamily->getIndex(), _index];
 	}
 #undef CASE_GET_LABEL
 }
 
+#if MVK_XCODE_12
+static const char* mvkStringFromMTLCommandEncoderErrorState(MTLCommandEncoderErrorState errState) {
+	switch (errState) {
+		case MTLCommandEncoderErrorStateUnknown:   return "unknown";
+		case MTLCommandEncoderErrorStateAffected:  return "affected";
+		case MTLCommandEncoderErrorStateCompleted: return "completed";
+		case MTLCommandEncoderErrorStateFaulted:   return "faulted";
+		case MTLCommandEncoderErrorStatePending:   return "pending";
+	}
+	return "unknown";
+}
+#endif
+
+void MVKQueue::handleMTLCommandBufferError(id<MTLCommandBuffer> mtlCmdBuff) {
+	if (mtlCmdBuff.status != MTLCommandBufferStatusError) { return; }
+
+	// If a command buffer error has occurred, report the error. If the error affects
+	// the physical device, always mark both the device and physical device as lost.
+	// If the error is local to this command buffer, optionally mark the device (but not the
+	// physical device) as lost, depending on the value of MVKConfiguration::resumeLostDevice.
+	VkResult vkErr = VK_ERROR_UNKNOWN;
+	bool markDeviceLoss = !mvkConfig().resumeLostDevice;
+	bool markPhysicalDeviceLoss = false;
+	switch (mtlCmdBuff.error.code) {
+		case MTLCommandBufferErrorBlacklisted:
+		case MTLCommandBufferErrorNotPermitted:	// May also be used for command buffers executed in the background without the right entitlement.
+#if MVK_MACOS && !MVK_MACCAT
+		case MTLCommandBufferErrorDeviceRemoved:
+#endif
+			vkErr = VK_ERROR_DEVICE_LOST;
+			markDeviceLoss = true;
+			markPhysicalDeviceLoss = true;
+			break;
+		case MTLCommandBufferErrorTimeout:
+			vkErr = VK_TIMEOUT;
+			break;
+#if MVK_XCODE_13
+		case MTLCommandBufferErrorStackOverflow:
+#endif
+		case MTLCommandBufferErrorPageFault:
+		case MTLCommandBufferErrorOutOfMemory:
+		default:
+			vkErr = VK_ERROR_OUT_OF_DEVICE_MEMORY;
+			break;
+	}
+	reportError(vkErr, "MTLCommandBuffer \"%s\" execution failed (code %li): %s",
+				mtlCmdBuff.label ? mtlCmdBuff.label.UTF8String : "",
+				mtlCmdBuff.error.code, mtlCmdBuff.error.localizedDescription.UTF8String);
+	if (markDeviceLoss) { getDevice()->markLost(markPhysicalDeviceLoss); }
+
+#if MVK_XCODE_12
+	if (&MTLCommandBufferEncoderInfoErrorKey != nullptr) {
+		if (NSArray<id<MTLCommandBufferEncoderInfo>>* mtlEncInfo = mtlCmdBuff.error.userInfo[MTLCommandBufferEncoderInfoErrorKey]) {
+			MVKLogInfo("Encoders for %p \"%s\":", mtlCmdBuff, mtlCmdBuff.label ? mtlCmdBuff.label.UTF8String : "");
+			for (id<MTLCommandBufferEncoderInfo> enc in mtlEncInfo) {
+				MVKLogInfo(" - %s: %s", enc.label.UTF8String, mvkStringFromMTLCommandEncoderErrorState(enc.errorState));
+				if (enc.debugSignposts.count > 0) {
+					MVKLogInfo("   Debug signposts:");
+					for (NSString* signpost in enc.debugSignposts) {
+						MVKLogInfo("    - %s", signpost.UTF8String);
+					}
+				}
+			}
+		}
+	}
+	if ([mtlCmdBuff respondsToSelector: @selector(logs)]) {
+		bool isFirstMsg = true;
+		for (id<MTLFunctionLog> log in mtlCmdBuff.logs) {
+			if (isFirstMsg) {
+				MVKLogInfo("Shader log messages:");
+				isFirstMsg = false;
+			}
+			MVKLogInfo("%s", log.description.UTF8String);
+		}
+	}
+#endif
+}
+
+// _presentedImages counts presentations per swapchain image, because the presentation of an image can
+// begin before the previous presentation of that image has indicated that it has completed via a callback.
+void MVKQueue::beginPresentation(const MVKImagePresentInfo& presentInfo) {
+	lock_guard<mutex> lock(_presentedImagesLock);
+	_presentationCompletionBlocker.reserve();
+	_presentedImages[presentInfo.presentableImage]++;
+}
+
+void MVKQueue::endPresentation(const MVKImagePresentInfo& presentInfo) {
+	lock_guard<mutex> lock(_presentedImagesLock);
+	_presentationCompletionBlocker.release();
+	if (_presentedImages[presentInfo.presentableImage]) {
+		_presentedImages[presentInfo.presentableImage]--;
+	}
+	if ( !_presentedImages[presentInfo.presentableImage] ) {
+		_presentedImages.erase(presentInfo.presentableImage);
+	}
+}
 
 #pragma mark Construction
 
 #define MVK_DISPATCH_QUEUE_QOS_CLASS		QOS_CLASS_USER_INITIATED
 
-MVKQueue::MVKQueue(MVKDevice* device, MVKQueueFamily* queueFamily, uint32_t index, float priority)
+MVKQueue::MVKQueue(MVKDevice* device, MVKQueueFamily* queueFamily, uint32_t index, float priority) : MVKDeviceTrackingMixin(device) {
-        : MVKDeviceTrackingMixin(device) {
-
 	_queueFamily = queueFamily;
 	_index = index;
 	_priority = priority;
 
-	_mtlCmdBuffLabelEndCommandBuffer = nil;
-	_mtlCmdBuffLabelQueueSubmit = nil;
-	_mtlCmdBuffLabelQueuePresent = nil;
-	_mtlCmdBuffLabelDeviceWaitIdle = nil;
-	_mtlCmdBuffLabelQueueWaitIdle = nil;
-	_mtlCmdBuffLabelAcquireNextImage = nil;
-	_mtlCmdBuffLabelInvalidateMappedMemoryRanges = nil;
-
 	initName();
 	initExecQueue();
 	initMTLCommandQueue();
-	initGPUCaptureScopes();
 }
 
 void MVKQueue::initName() {
@@ -236,23 +364,15 @@ void MVKQueue::initExecQueue() {
 	}
 }
 
-// Retrieves and initializes the Metal command queue.
+// Retrieves and initializes the Metal command queue and Xcode GPU capture scopes
 void MVKQueue::initMTLCommandQueue() {
-	uint64_t startTime = _device->getPerformanceTimestamp();
 	_mtlQueue = _queueFamily->getMTLCommandQueue(_index);	// not retained (cached in queue family)
-	_device->addActivityPerformance(_device->_performanceStatistics.queue.mtlQueueAccess, startTime);
-}
 
-// Initializes Xcode GPU capture scopes
-void MVKQueue::initGPUCaptureScopes() {
 	_submissionCaptureScope = new MVKGPUCaptureScope(this);
-
 	if (_queueFamily->getIndex() == mvkConfig().defaultGPUCaptureScopeQueueFamilyIndex &&
 		_index == mvkConfig().defaultGPUCaptureScopeQueueIndex) {
-
 		getDevice()->startAutoGPUCapture(MVK_CONFIG_AUTO_GPU_CAPTURE_SCOPE_FRAME, _mtlQueue);
 		_submissionCaptureScope->makeDefault();
-
 	}
 	_submissionCaptureScope->beginScope();	// Allow Xcode to capture the first frame if desired.
 }
@@ -261,7 +381,7 @@ MVKQueue::~MVKQueue() {
 	destroyExecQueue();
 	_submissionCaptureScope->destroy();
 
-	[_mtlCmdBuffLabelEndCommandBuffer release];
+	[_mtlCmdBuffLabelBeginCommandBuffer release];
 	[_mtlCmdBuffLabelQueueSubmit release];
 	[_mtlCmdBuffLabelQueuePresent release];
 	[_mtlCmdBuffLabelDeviceWaitIdle release];
@@ -306,7 +426,7 @@ MVKQueueSubmission::~MVKQueueSubmission() {
 #pragma mark -
 #pragma mark MVKQueueCommandBufferSubmission
 
-void MVKQueueCommandBufferSubmission::execute() {
+VkResult MVKQueueCommandBufferSubmission::execute() {
 
 	_queue->_submissionCaptureScope->beginScope();
 
@@ -321,7 +441,7 @@ void MVKQueueCommandBufferSubmission::execute() {
 
 	// Commit the last MTLCommandBuffer.
 	// Nothing after this because callback might destroy this instance before this function ends.
-	commitActiveMTLCommandBuffer(true);
+	return commitActiveMTLCommandBuffer(true);
 }
 
 // Returns the active MTLCommandBuffer, lazily retrieving it from the queue if needed.
@@ -341,24 +461,11 @@ void MVKQueueCommandBufferSubmission::setActiveMTLCommandBuffer(id<MTLCommandBuf
 	[_activeMTLCommandBuffer enqueue];
 }
 
-#if MVK_XCODE_12
-static const char* mvkStringFromErrorState(MTLCommandEncoderErrorState errState) {
-	switch (errState) {
-		case MTLCommandEncoderErrorStateUnknown: return "unknown";
-		case MTLCommandEncoderErrorStateAffected: return "affected";
-		case MTLCommandEncoderErrorStateCompleted: return "completed";
-		case MTLCommandEncoderErrorStateFaulted: return "faulted";
-		case MTLCommandEncoderErrorStatePending: return "pending";
-	}
-	return "unknown";
-}
-#endif
-
 // Commits and releases the currently active MTLCommandBuffer, optionally signalling
 // when the MTLCommandBuffer is done. The first time this is called, it will wait on
 // any semaphores. We have delayed signalling the semaphores as long as possible to
 // allow as much filling of the MTLCommandBuffer as possible before forcing a wait.
-void MVKQueueCommandBufferSubmission::commitActiveMTLCommandBuffer(bool signalCompletion) {
+VkResult MVKQueueCommandBufferSubmission::commitActiveMTLCommandBuffer(bool signalCompletion) {
 
 	// If using inline semaphore waiting, do so now.
 	// When prefilled command buffers are used, multiple commits will happen because native semaphore
@@ -386,66 +493,21 @@ void MVKQueueCommandBufferSubmission::commitActiveMTLCommandBuffer(bool signalCo
 	id<MTLCommandBuffer> mtlCmdBuff = signalCompletion ? getActiveMTLCommandBuffer() : _activeMTLCommandBuffer;
 	_activeMTLCommandBuffer = nil;
 
-	MVKDevice* mvkDev = _queue->getDevice();
+	MVKDevice* mvkDev = getDevice();
 	uint64_t startTime = mvkDev->getPerformanceTimestamp();
 	[mtlCmdBuff addCompletedHandler: ^(id<MTLCommandBuffer> mtlCB) {
-		if (mtlCB.status == MTLCommandBufferStatusError) {
+		mvkDev->addActivityPerformance(mvkDev->_performanceStatistics.queue.mtlCommandBufferExecution, startTime);
-			// If a command buffer error has occurred, report the error. If the error affects
+		if (signalCompletion) { this->finish(); }	// Must be the last thing the completetion callback does.
-			// the physical device, always mark both the device and physical device as lost.
-			// If the error is local to this command buffer, optionally mark the device (but not the
-			// physical device) as lost, depending on the value of MVKConfiguration::resumeLostDevice.
-			getVulkanAPIObject()->reportError(VK_ERROR_DEVICE_LOST, "MTLCommandBuffer \"%s\" execution failed (code %li): %s", mtlCB.label ? mtlCB.label.UTF8String : "", mtlCB.error.code, mtlCB.error.localizedDescription.UTF8String);
-			switch (mtlCB.error.code) {
-				case MTLCommandBufferErrorBlacklisted:
-				case MTLCommandBufferErrorNotPermitted:	// May also be used for command buffers executed in the background without the right entitlement.
-#if MVK_MACOS && !MVK_MACCAT
-				case MTLCommandBufferErrorDeviceRemoved:
-#endif
-					mvkDev->markLost(true);
-					break;
-				default:
-					if ( !mvkConfig().resumeLostDevice ) { mvkDev->markLost(); }
-					break;
-			}
-#if MVK_XCODE_12
-			if (mvkConfig().debugMode) {
-				if (&MTLCommandBufferEncoderInfoErrorKey != nullptr) {
-					if (NSArray<id<MTLCommandBufferEncoderInfo>>* mtlEncInfo = mtlCB.error.userInfo[MTLCommandBufferEncoderInfoErrorKey]) {
-						MVKLogInfo("Encoders for %p \"%s\":", mtlCB, mtlCB.label ? mtlCB.label.UTF8String : "");
-						for (id<MTLCommandBufferEncoderInfo> enc in mtlEncInfo) {
-							MVKLogInfo(" - %s: %s", enc.label.UTF8String, mvkStringFromErrorState(enc.errorState));
-							if (enc.debugSignposts.count > 0) {
-								MVKLogInfo("   Debug signposts:");
-								for (NSString* signpost in enc.debugSignposts) {
-									MVKLogInfo("    - %s", signpost.UTF8String);
-								}
-							}
-						}
-					}
-				}
-			}
-#endif
-		}
-#if MVK_XCODE_12
-		if (mvkConfig().debugMode && [mtlCB respondsToSelector: @selector(logs)]) {
-			bool isFirstMsg = true;
-			for (id<MTLFunctionLog> log in mtlCB.logs) {
-				if (isFirstMsg) {
-					MVKLogInfo("Shader log messages:");
-					isFirstMsg = false;
-				}
-				MVKLogInfo("%s", log.description.UTF8String);
-			}
-		}
-#endif
-
-		// Ensure finish() is the last thing the completetion callback does.
-		mvkDev->addActivityPerformance(mvkDev->_performanceStatistics.queue.mtlCommandBufferCompletion, startTime);
-		if (signalCompletion) { this->finish(); }
 	}];
 
 	[mtlCmdBuff commit];
 	[mtlCmdBuff release];		// retained
+
+	// If we need to signal completion, but an error occurred and the MTLCommandBuffer
+	// was not created, call the finish() function directly.
+	if (signalCompletion && !mtlCmdBuff) { finish(); }
+
+	return mtlCmdBuff ? VK_SUCCESS : VK_ERROR_OUT_OF_POOL_MEMORY;
 }
 
 // Be sure to retain() any API objects referenced in this function, and release() them in the
@@ -474,10 +536,11 @@ void MVKQueueCommandBufferSubmission::finish() {
 MVKQueueCommandBufferSubmission::MVKQueueCommandBufferSubmission(MVKQueue* queue,
 																 const VkSubmitInfo* pSubmit,
 																 VkFence fence,
-																 MVKCommandUse cmdUse) :
+																 MVKCommandUse cmdUse)
-	MVKQueueSubmission(queue,
+	: MVKQueueSubmission(queue,
-					   (pSubmit ? pSubmit->waitSemaphoreCount : 0),
+						 (pSubmit ? pSubmit->waitSemaphoreCount : 0),
-					   (pSubmit ? pSubmit->pWaitSemaphores : nullptr)),
+						 (pSubmit ? pSubmit->pWaitSemaphores : nullptr)),
+
 	_commandUse(cmdUse),
 	_emulatedWaitDone(false) {
 
@@ -524,7 +587,31 @@ MVKQueueCommandBufferSubmission::~MVKQueueCommandBufferSubmission() {
 
 template <size_t N>
 void MVKQueueFullCommandBufferSubmission<N>::submitCommandBuffers() {
+	MVKDevice* mvkDev = getDevice();
+	uint64_t startTime = mvkDev->getPerformanceTimestamp();
+
 	for (auto& cb : _cmdBuffers) { cb->submit(this, &_encodingContext); }
+
+	mvkDev->addActivityPerformance(mvkDev->_performanceStatistics.queue.submitCommandBuffers, startTime);
+}
+
+template <size_t N>
+MVKQueueFullCommandBufferSubmission<N>::MVKQueueFullCommandBufferSubmission(MVKQueue* queue,
+																			const VkSubmitInfo* pSubmit,
+																			VkFence fence,
+																			MVKCommandUse cmdUse)
+	: MVKQueueCommandBufferSubmission(queue, pSubmit, fence, cmdUse) {
+
+	// pSubmit can be null if just tracking the fence alone
+	if (pSubmit) {
+		uint32_t cbCnt = pSubmit->commandBufferCount;
+		_cmdBuffers.reserve(cbCnt);
+		for (uint32_t i = 0; i < cbCnt; i++) {
+			MVKCommandBuffer* cb = MVKCommandBuffer::getMVKCommandBuffer(pSubmit->pCommandBuffers[i]);
+			_cmdBuffers.push_back(cb);
+			setConfigurationResult(cb->getConfigurationResult());
+		}
+	}
 }
 
 
@@ -534,24 +621,31 @@ void MVKQueueFullCommandBufferSubmission<N>::submitCommandBuffers() {
 // If the semaphores are encodable, wait on them by encoding them on the MTLCommandBuffer before presenting.
 // If the semaphores are not encodable, wait on them inline after presenting.
 // The semaphores know what to do.
-void MVKQueuePresentSurfaceSubmission::execute() {
+VkResult MVKQueuePresentSurfaceSubmission::execute() {
 	id<MTLCommandBuffer> mtlCmdBuff = _queue->getMTLCommandBuffer(kMVKCommandUseQueuePresent);
 	[mtlCmdBuff enqueue];
-	for (auto& ws : _waitSemaphores) { ws.first->encodeWait(mtlCmdBuff, 0); }
 
 	// Add completion handler that will destroy this submission only once the MTLCommandBuffer
 	// is finished with the resources retained here, including the wait semaphores.
 	// Completion handlers are also added in presentCAMetalDrawable() to retain the swapchain images.
-	[mtlCmdBuff addCompletedHandler: ^(id<MTLCommandBuffer> mcb) {
+	[mtlCmdBuff addCompletedHandler: ^(id<MTLCommandBuffer> mtlCB) { this->finish(); }];
-		this->finish();
+
-	}];
+	for (auto& ws : _waitSemaphores) {
+		auto& sem4 = ws.first;
+		sem4->encodeWait(mtlCmdBuff, 0);	// Encoded semaphore waits
+		sem4->encodeWait(nil, 0);			// Inline semaphore waits
+	}
 
 	for (int i = 0; i < _presentInfo.size(); i++ ) {
 		_presentInfo[i].presentableImage->presentCAMetalDrawable(mtlCmdBuff, _presentInfo[i]);
 	}
 
-	for (auto& ws : _waitSemaphores) { ws.first->encodeWait(nil, 0); }
 	[mtlCmdBuff commit];
+
+	// If an error occurred and the MTLCommandBuffer was not created, call finish() directly.
+	if ( !mtlCmdBuff ) { finish(); }
+
+	return mtlCmdBuff ? VK_SUCCESS : VK_ERROR_OUT_OF_POOL_MEMORY;
 }
 
 void MVKQueuePresentSurfaceSubmission::finish() {
@@ -563,7 +657,7 @@ void MVKQueuePresentSurfaceSubmission::finish() {
 	cs->beginScope();
 	if (_queue->_queueFamily->getIndex() == mvkConfig().defaultGPUCaptureScopeQueueFamilyIndex &&
 		_queue->_index == mvkConfig().defaultGPUCaptureScopeQueueIndex) {
-		_queue->getDevice()->stopAutoGPUCapture(MVK_CONFIG_AUTO_GPU_CAPTURE_SCOPE_FRAME);
+		getDevice()->stopAutoGPUCapture(MVK_CONFIG_AUTO_GPU_CAPTURE_SCOPE_FRAME);
 	}
 
 	this->destroy();
@@ -623,6 +717,7 @@ MVKQueuePresentSurfaceSubmission::MVKQueuePresentSurfaceSubmission(MVKQueue* que
 	for (uint32_t scIdx = 0; scIdx < scCnt; scIdx++) {
 		MVKSwapchain* mvkSC = (MVKSwapchain*)pPresentInfo->pSwapchains[scIdx];
 		MVKImagePresentInfo presentInfo = {};	// Start with everything zeroed
+		presentInfo.queue = _queue;
 		presentInfo.presentableImage = mvkSC->getPresentableImage(pPresentInfo->pImageIndices[scIdx]);
 		presentInfo.presentMode = pPresentModes ? pPresentModes[scIdx] : VK_PRESENT_MODE_MAX_ENUM_KHR;
 		presentInfo.fence = pFences ? (MVKFence*)pFences[scIdx] : nullptr;

MoltenVK/MoltenVK/GPUObjects/MVKSurface.h

@@ -35,6 +35,7 @@
 #endif
 
 class MVKInstance;
+class MVKSwapchain;
 
 @class MVKBlockObserver;
 
@@ -55,11 +56,8 @@ public:
 	/** Returns a pointer to the Vulkan instance. */
 	MVKInstance* getInstance() override { return _mvkInstance; }
 
-    /** Returns the CAMetalLayer underlying this surface.  */
+    /** Returns the CAMetalLayer underlying this surface. */
-    inline CAMetalLayer* getCAMetalLayer() {
+	CAMetalLayer* getCAMetalLayer();
-        std::lock_guard<std::mutex> lock(_layerLock);
-        return _mtlCAMetalLayer;
-    }
 
 
 #pragma mark Construction
@@ -75,13 +73,16 @@ public:
 	~MVKSurface() override;
 
 protected:
+	friend class MVKSwapchain;
+
 	void propagateDebugName() override {}
-	void initLayerObserver();
+	void initLayer(CAMetalLayer* mtlLayer, const char* vkFuncName);
 	void releaseLayer();
 
-	MVKInstance* _mvkInstance;
-	CAMetalLayer* _mtlCAMetalLayer;
-	MVKBlockObserver* _layerObserver;
 	std::mutex _layerLock;
+	MVKInstance* _mvkInstance = nullptr;
+	CAMetalLayer* _mtlCAMetalLayer = nil;
+	MVKBlockObserver* _layerObserver = nil;
+	MVKSwapchain* _activeSwapchain = nullptr;
 };
 

MoltenVK/MoltenVK/GPUObjects/MVKSurface.mm

@@ -29,12 +29,15 @@
 
 #pragma mark MVKSurface
 
+CAMetalLayer* MVKSurface::getCAMetalLayer() {
+	std::lock_guard<std::mutex> lock(_layerLock);
+	return _mtlCAMetalLayer;
+}
+
 MVKSurface::MVKSurface(MVKInstance* mvkInstance,
 					   const VkMetalSurfaceCreateInfoEXT* pCreateInfo,
 					   const VkAllocationCallbacks* pAllocator) : _mvkInstance(mvkInstance) {
-
+	initLayer((CAMetalLayer*)pCreateInfo->pLayer, "vkCreateMetalSurfaceEXT");
-	_mtlCAMetalLayer = (CAMetalLayer*)[pCreateInfo->pLayer retain];
-	initLayerObserver();
 }
 
 // pCreateInfo->pView can be either a CAMetalLayer or a view (NSView/UIView).
@@ -47,36 +50,30 @@ MVKSurface::MVKSurface(MVKInstance* mvkInstance,
 
 	// If it's a view (NSView/UIView), extract the layer, otherwise assume it's already a CAMetalLayer.
 	if ([obj isKindOfClass: [PLATFORM_VIEW_CLASS class]]) {
+		obj = ((PLATFORM_VIEW_CLASS*)obj).layer;
 		if ( !NSThread.isMainThread ) {
-			MVKLogInfo("%s(): You are not calling this function from the main thread. %s should only be accessed from the main thread. When using this function outside the main thread, consider passing the CAMetalLayer itself in %s::pView, instead of the %s.",
+			MVKLogWarn("%s(): You are not calling this function from the main thread. %s should only be accessed from the main thread. When using this function outside the main thread, consider passing the CAMetalLayer itself in %s::pView, instead of the %s.",
 					   STR(vkCreate_PLATFORM_SurfaceMVK), STR(PLATFORM_VIEW_CLASS), STR(Vk_PLATFORM_SurfaceCreateInfoMVK), STR(PLATFORM_VIEW_CLASS));
 		}
-		obj = ((PLATFORM_VIEW_CLASS*)obj).layer;
 	}
 
 	// Confirm that we were provided with a CAMetalLayer
-	if ([obj isKindOfClass: [CAMetalLayer class]]) {
+	initLayer([obj isKindOfClass: CAMetalLayer.class] ? (CAMetalLayer*)obj : nil,
-		_mtlCAMetalLayer = (CAMetalLayer*)[obj retain];		// retained
+			  STR(vkCreate_PLATFORM_SurfaceMVK));
-	} else {
-		setConfigurationResult(reportError(VK_ERROR_INITIALIZATION_FAILED,
-										   "%s(): On-screen rendering requires a layer of type CAMetalLayer.",
-										   STR(vkCreate_PLATFORM_SurfaceMVK)));
-		_mtlCAMetalLayer = nil;
-	}
-
-	initLayerObserver();
 }
 
-// Sometimes, the owning view can replace its CAMetalLayer. In that case, the client needs to recreate the surface.
+void MVKSurface::initLayer(CAMetalLayer* mtlLayer, const char* vkFuncName) {
-void MVKSurface::initLayerObserver() {
 
-	_layerObserver = nil;
+	_mtlCAMetalLayer = [mtlLayer retain];	// retained
-	if ( ![_mtlCAMetalLayer.delegate isKindOfClass: [PLATFORM_VIEW_CLASS class]] ) { return; }
+	if ( !_mtlCAMetalLayer ) { setConfigurationResult(reportError(VK_ERROR_SURFACE_LOST_KHR, "%s(): On-screen rendering requires a layer of type CAMetalLayer.", vkFuncName)); }
 
-	_layerObserver = [MVKBlockObserver observerWithBlock: ^(NSString* path, id, NSDictionary*, void*) {
+	// Sometimes, the owning view can replace its CAMetalLayer.
-		if ( ![path isEqualToString: @"layer"] ) { return; }
+	// When that happens, the app needs to recreate the surface.
-		this->releaseLayer();
+	if ([_mtlCAMetalLayer.delegate isKindOfClass: [PLATFORM_VIEW_CLASS class]]) {
-	} forObject: _mtlCAMetalLayer.delegate atKeyPath: @"layer"];
+		_layerObserver = [MVKBlockObserver observerWithBlock: ^(NSString* path, id, NSDictionary*, void*) {
+			if ([path isEqualToString: @"layer"]) { this->releaseLayer(); }
+		} forObject: _mtlCAMetalLayer.delegate atKeyPath: @"layer"];
+	}
 }
 
 void MVKSurface::releaseLayer() {

MoltenVK/MoltenVK/GPUObjects/MVKSwapchain.h

@@ -28,8 +28,6 @@
 
 class MVKWatermark;
 
-@class MVKBlockObserver;
-
 
 #pragma mark -
 #pragma mark MVKSwapchain
@@ -76,19 +74,8 @@ public:
 	/** Releases swapchain images. */
 	VkResult releaseImages(const VkReleaseSwapchainImagesInfoEXT* pReleaseInfo);
 
-	/** Returns whether the parent surface is now lost and this swapchain must be recreated. */
-	bool getIsSurfaceLost() { return _surfaceLost; }
-
-	/** Returns whether this swapchain is optimally sized for the surface. */
-	bool hasOptimalSurface();
-
 	/** Returns the status of the surface. Surface loss takes precedence over sub-optimal errors. */
-	VkResult getSurfaceStatus() {
+	VkResult getSurfaceStatus();
-		if (_device->getConfigurationResult() != VK_SUCCESS) { return _device->getConfigurationResult(); }
-		if (getIsSurfaceLost()) { return VK_ERROR_SURFACE_LOST_KHR; }
-		if ( !hasOptimalSurface() ) { return VK_SUBOPTIMAL_KHR; }
-		return VK_SUCCESS;
-	}
 
 	/** Adds HDR metadata to this swapchain. */
 	void setHDRMetadataEXT(const VkHdrMetadataEXT& metadata);
@@ -118,31 +105,28 @@ protected:
 						  VkSwapchainPresentScalingCreateInfoEXT* pScalingInfo,
 						  uint32_t imgCnt);
 	void initSurfaceImages(const VkSwapchainCreateInfoKHR* pCreateInfo, uint32_t imgCnt);
-	void releaseLayer();
+	bool getIsSurfaceLost();
-	void releaseUndisplayedSurfaces();
+	bool hasOptimalSurface();
 	uint64_t getNextAcquisitionID();
-    void willPresentSurface(id<MTLTexture> mtlTexture, id<MTLCommandBuffer> mtlCmdBuff);
     void renderWatermark(id<MTLTexture> mtlTexture, id<MTLCommandBuffer> mtlCmdBuff);
     void markFrameInterval();
-	void recordPresentTime(const MVKImagePresentInfo& presentInfo, uint64_t actualPresentTime = 0);
+	void beginPresentation(const MVKImagePresentInfo& presentInfo);
+	void endPresentation(const MVKImagePresentInfo& presentInfo, uint64_t actualPresentTime = 0);
 
-	CAMetalLayer* _mtlLayer = nil;
+	MVKSurface* _surface = nullptr;
     MVKWatermark* _licenseWatermark = nullptr;
 	MVKSmallVector<MVKPresentableSwapchainImage*, kMVKMaxSwapchainImageCount> _presentableImages;
 	MVKSmallVector<VkPresentModeKHR, 2> _compatiblePresentModes;
 	static const int kMaxPresentationHistory = 60;
 	VkPastPresentationTimingGOOGLE _presentTimingHistory[kMaxPresentationHistory];
 	std::atomic<uint64_t> _currentAcquisitionID = 0;
-    MVKBlockObserver* _layerObserver = nil;
 	std::mutex _presentHistoryLock;
-	std::mutex _layerLock;
 	uint64_t _lastFrameTime = 0;
 	VkExtent2D _mtlLayerDrawableExtent = {0, 0};
 	uint32_t _currentPerfLogFrameCount = 0;
 	uint32_t _presentHistoryCount = 0;
 	uint32_t _presentHistoryIndex = 0;
 	uint32_t _presentHistoryHeadIndex = 0;
-	std::atomic<bool> _surfaceLost = false;
 	bool _isDeliberatelyScaled = false;
 };
 

MoltenVK/MoltenVK/GPUObjects/MVKSwapchain.mm

@@ -95,9 +95,8 @@ VkResult MVKSwapchain::acquireNextImage(uint64_t timeout,
 	// Return the index of the image with the shortest wait,
 	// and signal the semaphore and fence when it's available
 	*pImageIndex = minWaitImage->_swapchainIndex;
-	minWaitImage->acquireAndSignalWhenAvailable((MVKSemaphore*)semaphore, (MVKFence*)fence);
+	VkResult rslt = minWaitImage->acquireAndSignalWhenAvailable((MVKSemaphore*)semaphore, (MVKFence*)fence);
-
+	return rslt ? rslt : getSurfaceStatus();
-	return getSurfaceStatus();
 }
 
 VkResult MVKSwapchain::releaseImages(const VkReleaseSwapchainImagesInfoEXT* pReleaseInfo) {
@@ -110,10 +109,18 @@ VkResult MVKSwapchain::releaseImages(const VkReleaseSwapchainImagesInfoEXT* pRel
 
 uint64_t MVKSwapchain::getNextAcquisitionID() { return ++_currentAcquisitionID; }
 
-// Releases any surfaces that are not currently being displayed,
+bool MVKSwapchain::getIsSurfaceLost() {
-// so they can be used by a different swapchain.
+	VkResult surfRslt = _surface->getConfigurationResult();
-void MVKSwapchain::releaseUndisplayedSurfaces() {}
+	setConfigurationResult(surfRslt);
+	return surfRslt != VK_SUCCESS;
+}
 
+VkResult MVKSwapchain::getSurfaceStatus() {
+	if (_device->getConfigurationResult() != VK_SUCCESS) { return _device->getConfigurationResult(); }
+	if (getIsSurfaceLost()) { return VK_ERROR_SURFACE_LOST_KHR; }
+	if ( !hasOptimalSurface() ) { return VK_SUBOPTIMAL_KHR; }
+	return VK_SUCCESS;
+}
 
 // This swapchain is optimally sized for the surface if the app has specified deliberate
 // swapchain scaling, or the CAMetalLayer drawableSize has not changed since the swapchain
@@ -121,22 +128,16 @@ void MVKSwapchain::releaseUndisplayedSurfaces() {}
 bool MVKSwapchain::hasOptimalSurface() {
 	if (_isDeliberatelyScaled) { return true; }
 
-	VkExtent2D drawExtent = mvkVkExtent2DFromCGSize(_mtlLayer.drawableSize);
+	auto* mtlLayer = _surface->getCAMetalLayer();
+	VkExtent2D drawExtent = mvkVkExtent2DFromCGSize(mtlLayer.drawableSize);
 	return (mvkVkExtent2DsAreEqual(drawExtent, _mtlLayerDrawableExtent) &&
-			mvkVkExtent2DsAreEqual(drawExtent, mvkGetNaturalExtent(_mtlLayer)));
+			mvkVkExtent2DsAreEqual(drawExtent, mvkGetNaturalExtent(mtlLayer)));
 }
 
 
 #pragma mark Rendering
 
-// Called automatically when a swapchain image is about to be presented to the surface by the queue.
+// Renders the watermark image to the surface.
-// Activities include marking the frame interval and rendering the watermark if needed.
-void MVKSwapchain::willPresentSurface(id<MTLTexture> mtlTexture, id<MTLCommandBuffer> mtlCmdBuff) {
-    markFrameInterval();
-    renderWatermark(mtlTexture, mtlCmdBuff);
-}
-
-// If the product has not been fully licensed, renders the watermark image to the surface.
 void MVKSwapchain::renderWatermark(id<MTLTexture> mtlTexture, id<MTLCommandBuffer> mtlCmdBuff) {
     if (mvkConfig().displayWatermark) {
         if ( !_licenseWatermark ) {
@@ -159,21 +160,20 @@ void MVKSwapchain::renderWatermark(id<MTLTexture> mtlTexture, id<MTLCommandBuffe
 
 // Calculates and remembers the time interval between frames.
 void MVKSwapchain::markFrameInterval() {
-	if ( !(mvkConfig().performanceTracking || _licenseWatermark) ) { return; }
-
 	uint64_t prevFrameTime = _lastFrameTime;
-	_lastFrameTime = mvkGetTimestamp();
+	_lastFrameTime = _device->getPerformanceTimestamp();
 
 	if (prevFrameTime == 0) { return; }		// First frame starts at first presentation
 
 	_device->addActivityPerformance(_device->_performanceStatistics.queue.frameInterval, prevFrameTime, _lastFrameTime);
 
-	uint32_t perfLogCntLimit = mvkConfig().performanceLoggingFrameCount;
+	auto& mvkCfg = mvkConfig();
-	if ((perfLogCntLimit > 0) && (++_currentPerfLogFrameCount >= perfLogCntLimit)) {
+	bool shouldLogOnFrames = mvkCfg.performanceTracking && mvkCfg.activityPerformanceLoggingStyle == MVK_CONFIG_ACTIVITY_PERFORMANCE_LOGGING_STYLE_FRAME_COUNT;
+	if (shouldLogOnFrames && (mvkCfg.performanceLoggingFrameCount > 0) && (++_currentPerfLogFrameCount >= mvkCfg.performanceLoggingFrameCount)) {
 		_currentPerfLogFrameCount = 0;
 		MVKLogInfo("Performance statistics reporting every: %d frames, avg FPS: %.2f, elapsed time: %.3f seconds:",
-				   perfLogCntLimit,
+				   mvkCfg.performanceLoggingFrameCount,
-				   (1000.0 / _device->_performanceStatistics.queue.frameInterval.averageDuration),
+				   (1000.0 / _device->_performanceStatistics.queue.frameInterval.average),
 				   mvkGetElapsedMilliseconds() / 1000.0);
 		if (mvkConfig().activityPerformanceLoggingStyle == MVK_CONFIG_ACTIVITY_PERFORMANCE_LOGGING_STYLE_FRAME_COUNT) {
 			_device->logPerformanceSummary();
@@ -181,6 +181,119 @@ void MVKSwapchain::markFrameInterval() {
 	}
 }
 
+VkResult MVKSwapchain::getRefreshCycleDuration(VkRefreshCycleDurationGOOGLE *pRefreshCycleDuration) {
+	if (_device->getConfigurationResult() != VK_SUCCESS) { return _device->getConfigurationResult(); }
+
+	auto* mtlLayer = _surface->getCAMetalLayer();
+#if MVK_VISIONOS
+	// TODO: See if this can be obtained from OS instead
+	NSInteger framesPerSecond = 90;
+#elif MVK_IOS_OR_TVOS || MVK_MACCAT
+	NSInteger framesPerSecond = 60;
+	UIScreen* screen = mtlLayer.screenMVK;
+	if ([screen respondsToSelector: @selector(maximumFramesPerSecond)]) {
+		framesPerSecond = screen.maximumFramesPerSecond;
+	}
+#elif MVK_MACOS && !MVK_MACCAT
+	NSScreen* screen = mtlLayer.screenMVK;
+	CGDirectDisplayID displayId = [[[screen deviceDescription] objectForKey:@"NSScreenNumber"] unsignedIntValue];
+	CGDisplayModeRef mode = CGDisplayCopyDisplayMode(displayId);
+	double framesPerSecond = CGDisplayModeGetRefreshRate(mode);
+	CGDisplayModeRelease(mode);
+#if MVK_XCODE_13
+	if (framesPerSecond == 0 && [screen respondsToSelector: @selector(maximumFramesPerSecond)])
+		framesPerSecond = [screen maximumFramesPerSecond];
+#endif
+
+	// Builtin panels, e.g., on MacBook, report a zero refresh rate.
+	if (framesPerSecond == 0)
+		framesPerSecond = 60.0;
+#endif
+
+	pRefreshCycleDuration->refreshDuration = (uint64_t)1e9 / framesPerSecond;
+	return VK_SUCCESS;
+}
+
+VkResult MVKSwapchain::getPastPresentationTiming(uint32_t *pCount, VkPastPresentationTimingGOOGLE *pPresentationTimings) {
+	if (_device->getConfigurationResult() != VK_SUCCESS) { return _device->getConfigurationResult(); }
+
+	VkResult res = VK_SUCCESS;
+
+	std::lock_guard<std::mutex> lock(_presentHistoryLock);
+	if (pPresentationTimings == nullptr) {
+		*pCount = _presentHistoryCount;
+	} else {
+		uint32_t countRemaining = std::min(_presentHistoryCount, *pCount);
+		uint32_t outIndex = 0;
+
+		res = (*pCount >= _presentHistoryCount) ? VK_SUCCESS : VK_INCOMPLETE;
+		*pCount = countRemaining;
+
+		while (countRemaining > 0) {
+			pPresentationTimings[outIndex] = _presentTimingHistory[_presentHistoryHeadIndex];
+			countRemaining--;
+			_presentHistoryCount--;
+			_presentHistoryHeadIndex = (_presentHistoryHeadIndex + 1) % kMaxPresentationHistory;
+			outIndex++;
+		}
+	}
+
+	return res;
+}
+
+void MVKSwapchain::beginPresentation(const MVKImagePresentInfo& presentInfo) {}
+
+void MVKSwapchain::endPresentation(const MVKImagePresentInfo& presentInfo, uint64_t actualPresentTime) {
+
+	markFrameInterval();
+
+	std::lock_guard<std::mutex> lock(_presentHistoryLock);
+	if (_presentHistoryCount < kMaxPresentationHistory) {
+		_presentHistoryCount++;
+	} else {
+		_presentHistoryHeadIndex = (_presentHistoryHeadIndex + 1) % kMaxPresentationHistory;
+	}
+
+	// If actual present time is not available, use desired time instead, and if that
+	// hasn't been set, use the current time, which should be reasonably accurate (sub-ms),
+	// since we are here as part of the addPresentedHandler: callback.
+	if (actualPresentTime == 0) { actualPresentTime = presentInfo.desiredPresentTime; }
+	if (actualPresentTime == 0) { actualPresentTime = CACurrentMediaTime() * 1.0e9; }
+
+	_presentTimingHistory[_presentHistoryIndex].presentID = presentInfo.presentID;
+	_presentTimingHistory[_presentHistoryIndex].desiredPresentTime = presentInfo.desiredPresentTime;
+	_presentTimingHistory[_presentHistoryIndex].actualPresentTime = actualPresentTime;
+	// These details are not available in Metal
+	_presentTimingHistory[_presentHistoryIndex].earliestPresentTime = actualPresentTime;
+	_presentTimingHistory[_presentHistoryIndex].presentMargin = 0;
+	_presentHistoryIndex = (_presentHistoryIndex + 1) % kMaxPresentationHistory;
+}
+
+void MVKSwapchain::setLayerNeedsDisplay(const VkPresentRegionKHR* pRegion) {
+	auto* mtlLayer = _surface->getCAMetalLayer();
+	if (!pRegion || pRegion->rectangleCount == 0) {
+		[mtlLayer setNeedsDisplay];
+		return;
+	}
+
+	for (uint32_t i = 0; i < pRegion->rectangleCount; ++i) {
+		CGRect cgRect = mvkCGRectFromVkRectLayerKHR(pRegion->pRectangles[i]);
+#if MVK_MACOS
+		// VK_KHR_incremental_present specifies an upper-left origin, but macOS by default
+		// uses a lower-left origin.
+		cgRect.origin.y = mtlLayer.bounds.size.height - cgRect.origin.y;
+#endif
+		// We were given rectangles in pixels, but -[CALayer setNeedsDisplayInRect:] wants them
+		// in points, which is pixels / contentsScale.
+		CGFloat scaleFactor = mtlLayer.contentsScale;
+		cgRect.origin.x /= scaleFactor;
+		cgRect.origin.y /= scaleFactor;
+		cgRect.size.width /= scaleFactor;
+		cgRect.size.height /= scaleFactor;
+		[mtlLayer setNeedsDisplayInRect:cgRect];
+	}
+}
+
 #if MVK_MACOS
 struct CIE1931XY {
 	uint16_t x;
@@ -237,19 +350,31 @@ void MVKSwapchain::setHDRMetadataEXT(const VkHdrMetadataEXT& metadata) {
 	CAEDRMetadata* caMetadata = [CAEDRMetadata HDR10MetadataWithDisplayInfo: colorVolData
 																contentInfo: lightLevelData
 														 opticalOutputScale: 1];
-	_mtlLayer.EDRMetadata = caMetadata;
+	auto* mtlLayer = _surface->getCAMetalLayer();
+	mtlLayer.EDRMetadata = caMetadata;
+	mtlLayer.wantsExtendedDynamicRangeContent = YES;
 	[caMetadata release];
 	[colorVolData release];
 	[lightLevelData release];
-	_mtlLayer.wantsExtendedDynamicRangeContent = YES;
 #endif
 }
 
 
 #pragma mark Construction
 
-MVKSwapchain::MVKSwapchain(MVKDevice* device,
+MVKSwapchain::MVKSwapchain(MVKDevice* device, const VkSwapchainCreateInfoKHR* pCreateInfo)
-						   const VkSwapchainCreateInfoKHR* pCreateInfo) : MVKVulkanAPIDeviceObject(device) {
+	: MVKVulkanAPIDeviceObject(device),
+	_surface((MVKSurface*)pCreateInfo->surface) {
+
+	// Check if oldSwapchain is properly set
+	auto* oldSwapchain = (MVKSwapchain*)pCreateInfo->oldSwapchain;
+	if (oldSwapchain == _surface->_activeSwapchain) {
+		_surface->_activeSwapchain = this;
+	} else {
+		setConfigurationResult(reportError(VK_ERROR_NATIVE_WINDOW_IN_USE_KHR, "vkCreateSwapchainKHR(): pCreateInfo->oldSwapchain does not match the VkSwapchain that is in use by the surface"));
+		return;
+	}
+
 	memset(_presentTimingHistory, 0, sizeof(_presentTimingHistory));
 
 	// Retrieve the scaling and present mode structs if they are supplied.
@@ -280,10 +405,6 @@ MVKSwapchain::MVKSwapchain(MVKDevice* device,
 		}
 	}
 
-	// If applicable, release any surfaces (not currently being displayed) from the old swapchain.
-	MVKSwapchain* oldSwapchain = (MVKSwapchain*)pCreateInfo->oldSwapchain;
-	if (oldSwapchain) { oldSwapchain->releaseUndisplayedSurfaces(); }
-
 	uint32_t imgCnt = mvkClamp(pCreateInfo->minImageCount,
 							   _device->_pMetalFeatures->minSwapchainImageCount,
 							   _device->_pMetalFeatures->maxSwapchainImageCount);
@@ -333,85 +454,80 @@ void MVKSwapchain::initCAMetalLayer(const VkSwapchainCreateInfoKHR* pCreateInfo,
 									VkSwapchainPresentScalingCreateInfoEXT* pScalingInfo,
 									uint32_t imgCnt) {
 
-	MVKSurface* mvkSrfc = (MVKSurface*)pCreateInfo->surface;
+	if ( getIsSurfaceLost() ) { return; }
-	_mtlLayer = mvkSrfc->getCAMetalLayer();
-	if ( !_mtlLayer ) {
-		setConfigurationResult(mvkSrfc->getConfigurationResult());
-		_surfaceLost = true;
-		return;
-	}
 
+	auto* mtlLayer = _surface->getCAMetalLayer();
 	auto minMagFilter = mvkConfig().swapchainMinMagFilterUseNearest ? kCAFilterNearest : kCAFilterLinear;
-	_mtlLayer.device = getMTLDevice();
+	mtlLayer.device = getMTLDevice();
-	_mtlLayer.pixelFormat = getPixelFormats()->getMTLPixelFormat(pCreateInfo->imageFormat);
+	mtlLayer.pixelFormat = getPixelFormats()->getMTLPixelFormat(pCreateInfo->imageFormat);
-	_mtlLayer.maximumDrawableCountMVK = imgCnt;
+	mtlLayer.maximumDrawableCountMVK = imgCnt;
-	_mtlLayer.displaySyncEnabledMVK = (pCreateInfo->presentMode != VK_PRESENT_MODE_IMMEDIATE_KHR);
+	mtlLayer.displaySyncEnabledMVK = (pCreateInfo->presentMode != VK_PRESENT_MODE_IMMEDIATE_KHR);
-	_mtlLayer.minificationFilter = minMagFilter;
+	mtlLayer.minificationFilter = minMagFilter;
-	_mtlLayer.magnificationFilter = minMagFilter;
+	mtlLayer.magnificationFilter = minMagFilter;
-	_mtlLayer.contentsGravity = getCALayerContentsGravity(pScalingInfo);
+	mtlLayer.contentsGravity = getCALayerContentsGravity(pScalingInfo);
-	_mtlLayer.framebufferOnly = !mvkIsAnyFlagEnabled(pCreateInfo->imageUsage, (VK_IMAGE_USAGE_TRANSFER_SRC_BIT |
+	mtlLayer.framebufferOnly = !mvkIsAnyFlagEnabled(pCreateInfo->imageUsage, (VK_IMAGE_USAGE_TRANSFER_SRC_BIT |
 																			   VK_IMAGE_USAGE_TRANSFER_DST_BIT |
 																			   VK_IMAGE_USAGE_SAMPLED_BIT |
 																			   VK_IMAGE_USAGE_STORAGE_BIT));
 	// Remember the extent to later detect if it has changed under the covers,
 	// and set the drawable size of the CAMetalLayer from the extent.
 	_mtlLayerDrawableExtent = pCreateInfo->imageExtent;
-	_mtlLayer.drawableSize = mvkCGSizeFromVkExtent2D(_mtlLayerDrawableExtent);
+	mtlLayer.drawableSize = mvkCGSizeFromVkExtent2D(_mtlLayerDrawableExtent);
 
 	if (pCreateInfo->compositeAlpha != VK_COMPOSITE_ALPHA_INHERIT_BIT_KHR) {
-		_mtlLayer.opaque = pCreateInfo->compositeAlpha == VK_COMPOSITE_ALPHA_OPAQUE_BIT_KHR;
+		mtlLayer.opaque = pCreateInfo->compositeAlpha == VK_COMPOSITE_ALPHA_OPAQUE_BIT_KHR;
 	}
 
 	switch (pCreateInfo->imageColorSpace) {
 		case VK_COLOR_SPACE_SRGB_NONLINEAR_KHR:
-			_mtlLayer.colorspaceNameMVK = kCGColorSpaceSRGB;
+			mtlLayer.colorspaceNameMVK = kCGColorSpaceSRGB;
-			_mtlLayer.wantsExtendedDynamicRangeContentMVK = NO;
+			mtlLayer.wantsExtendedDynamicRangeContentMVK = NO;
 			break;
 		case VK_COLOR_SPACE_DISPLAY_P3_NONLINEAR_EXT:
-			_mtlLayer.colorspaceNameMVK = kCGColorSpaceDisplayP3;
+			mtlLayer.colorspaceNameMVK = kCGColorSpaceDisplayP3;
-			_mtlLayer.wantsExtendedDynamicRangeContentMVK = YES;
+			mtlLayer.wantsExtendedDynamicRangeContentMVK = YES;
 			break;
 		case VK_COLOR_SPACE_EXTENDED_SRGB_LINEAR_EXT:
-			_mtlLayer.colorspaceNameMVK = kCGColorSpaceExtendedLinearSRGB;
+			mtlLayer.colorspaceNameMVK = kCGColorSpaceExtendedLinearSRGB;
-			_mtlLayer.wantsExtendedDynamicRangeContentMVK = YES;
+			mtlLayer.wantsExtendedDynamicRangeContentMVK = YES;
 			break;
 		case VK_COLOR_SPACE_EXTENDED_SRGB_NONLINEAR_EXT:
-			_mtlLayer.colorspaceNameMVK = kCGColorSpaceExtendedSRGB;
+			mtlLayer.colorspaceNameMVK = kCGColorSpaceExtendedSRGB;
-			_mtlLayer.wantsExtendedDynamicRangeContentMVK = YES;
+			mtlLayer.wantsExtendedDynamicRangeContentMVK = YES;
 			break;
 		case VK_COLOR_SPACE_DISPLAY_P3_LINEAR_EXT:
-			_mtlLayer.colorspaceNameMVK = kCGColorSpaceExtendedLinearDisplayP3;
+			mtlLayer.colorspaceNameMVK = kCGColorSpaceExtendedLinearDisplayP3;
-			_mtlLayer.wantsExtendedDynamicRangeContentMVK = YES;
+			mtlLayer.wantsExtendedDynamicRangeContentMVK = YES;
 			break;
 		case VK_COLOR_SPACE_DCI_P3_NONLINEAR_EXT:
-			_mtlLayer.colorspaceNameMVK = kCGColorSpaceDCIP3;
+			mtlLayer.colorspaceNameMVK = kCGColorSpaceDCIP3;
-			_mtlLayer.wantsExtendedDynamicRangeContentMVK = YES;
+			mtlLayer.wantsExtendedDynamicRangeContentMVK = YES;
 			break;
 		case VK_COLOR_SPACE_BT709_NONLINEAR_EXT:
-			_mtlLayer.colorspaceNameMVK = kCGColorSpaceITUR_709;
+			mtlLayer.colorspaceNameMVK = kCGColorSpaceITUR_709;
-			_mtlLayer.wantsExtendedDynamicRangeContentMVK = NO;
+			mtlLayer.wantsExtendedDynamicRangeContentMVK = NO;
 			break;
 		case VK_COLOR_SPACE_BT2020_LINEAR_EXT:
-			_mtlLayer.colorspaceNameMVK = kCGColorSpaceExtendedLinearITUR_2020;
+			mtlLayer.colorspaceNameMVK = kCGColorSpaceExtendedLinearITUR_2020;
-			_mtlLayer.wantsExtendedDynamicRangeContentMVK = YES;
+			mtlLayer.wantsExtendedDynamicRangeContentMVK = YES;
 			break;
 #if MVK_XCODE_12
 		case VK_COLOR_SPACE_HDR10_ST2084_EXT:
-			_mtlLayer.colorspaceNameMVK = kCGColorSpaceITUR_2100_PQ;
+			mtlLayer.colorspaceNameMVK = kCGColorSpaceITUR_2100_PQ;
-			_mtlLayer.wantsExtendedDynamicRangeContentMVK = YES;
+			mtlLayer.wantsExtendedDynamicRangeContentMVK = YES;
 			break;
 		case VK_COLOR_SPACE_HDR10_HLG_EXT:
-			_mtlLayer.colorspaceNameMVK = kCGColorSpaceITUR_2100_HLG;
+			mtlLayer.colorspaceNameMVK = kCGColorSpaceITUR_2100_HLG;
-			_mtlLayer.wantsExtendedDynamicRangeContentMVK = YES;
+			mtlLayer.wantsExtendedDynamicRangeContentMVK = YES;
 			break;
 #endif
 		case VK_COLOR_SPACE_ADOBERGB_NONLINEAR_EXT:
-			_mtlLayer.colorspaceNameMVK = kCGColorSpaceAdobeRGB1998;
+			mtlLayer.colorspaceNameMVK = kCGColorSpaceAdobeRGB1998;
-			_mtlLayer.wantsExtendedDynamicRangeContentMVK = NO;
+			mtlLayer.wantsExtendedDynamicRangeContentMVK = NO;
 			break;
 		case VK_COLOR_SPACE_PASS_THROUGH_EXT:
-			_mtlLayer.colorspace = nil;
+			mtlLayer.colorspace = nil;
-			_mtlLayer.wantsExtendedDynamicRangeContentMVK = NO;
+			mtlLayer.wantsExtendedDynamicRangeContentMVK = NO;
 			break;
 		default:
 			setConfigurationResult(reportError(VK_ERROR_FORMAT_NOT_SUPPORTED, "vkCreateSwapchainKHR(): Metal does not support VkColorSpaceKHR value %d.", pCreateInfo->imageColorSpace));
@@ -421,22 +537,6 @@ void MVKSwapchain::initCAMetalLayer(const VkSwapchainCreateInfoKHR* pCreateInfo,
 	// TODO: set additional CAMetalLayer properties before extracting drawables:
 	//	- presentsWithTransaction
 	//	- drawsAsynchronously
-
-	if ( [_mtlLayer.delegate isKindOfClass: [PLATFORM_VIEW_CLASS class]] ) {
-		// Sometimes, the owning view can replace its CAMetalLayer. In that case, the client
-		// needs to recreate the swapchain, or no content will be displayed.
-		_layerObserver = [MVKBlockObserver observerWithBlock: ^(NSString* path, id, NSDictionary*, void*) {
-			if ( ![path isEqualToString: @"layer"] ) { return; }
-			this->releaseLayer();
-		} forObject: _mtlLayer.delegate atKeyPath: @"layer"];
-	}
-}
-
-void MVKSwapchain::releaseLayer() {
-	std::lock_guard<std::mutex> lock(_layerLock);
-	_surfaceLost = true;
-	[_layerObserver release];
-	_layerObserver = nil;
 }
 
 // Initializes the array of images used for the surface of this swapchain.
@@ -459,13 +559,13 @@ void MVKSwapchain::initSurfaceImages(const VkSwapchainCreateInfoKHR* pCreateInfo
 		}
 	}
 
+	auto* mtlLayer = _surface->getCAMetalLayer();
     VkExtent2D imgExtent = pCreateInfo->imageExtent;
-
     VkImageCreateInfo imgInfo = {
         .sType = VK_STRUCTURE_TYPE_IMAGE_CREATE_INFO,
         .pNext = VK_NULL_HANDLE,
         .imageType = VK_IMAGE_TYPE_2D,
-        .format = getPixelFormats()->getVkFormat(_mtlLayer.pixelFormat),
+        .format = getPixelFormats()->getVkFormat(mtlLayer.pixelFormat),
         .extent = { imgExtent.width, imgExtent.height, 1 },
         .mipLevels = 1,
         .arrayLayers = 1,
@@ -494,131 +594,21 @@ void MVKSwapchain::initSurfaceImages(const VkSwapchainCreateInfoKHR* pCreateInfo
 
 	NSString* screenName = @"Main Screen";
 #if MVK_MACOS && !MVK_MACCAT
-	if ([_mtlLayer.screenMVK respondsToSelector:@selector(localizedName)]) {
+	if ([mtlLayer.screenMVK respondsToSelector:@selector(localizedName)]) {
-		screenName = _mtlLayer.screenMVK.localizedName;
+		screenName = mtlLayer.screenMVK.localizedName;
 	}
 #endif
     MVKLogInfo("Created %d swapchain images with initial size (%d, %d) and contents scale %.1f for screen %s.",
-			   imgCnt, imgExtent.width, imgExtent.height, _mtlLayer.contentsScale, screenName.UTF8String);
+			   imgCnt, imgExtent.width, imgExtent.height, mtlLayer.contentsScale, screenName.UTF8String);
 }
 
-VkResult MVKSwapchain::getRefreshCycleDuration(VkRefreshCycleDurationGOOGLE *pRefreshCycleDuration) {
-	if (_device->getConfigurationResult() != VK_SUCCESS) { return _device->getConfigurationResult(); }
-
-#if MVK_VISIONOS
-    // TODO: See if this can be obtained from OS instead
-    NSInteger framesPerSecond = 90;
-#elif MVK_IOS_OR_TVOS || MVK_MACCAT
-	NSInteger framesPerSecond = 60;
-	UIScreen* screen = _mtlLayer.screenMVK;
-	if ([screen respondsToSelector: @selector(maximumFramesPerSecond)]) {
-		framesPerSecond = screen.maximumFramesPerSecond;
-	}
-#elif MVK_MACOS && !MVK_MACCAT
-	NSScreen* screen = _mtlLayer.screenMVK;
-	CGDirectDisplayID displayId = [[[screen deviceDescription] objectForKey:@"NSScreenNumber"] unsignedIntValue];
-	CGDisplayModeRef mode = CGDisplayCopyDisplayMode(displayId);
-	double framesPerSecond = CGDisplayModeGetRefreshRate(mode);
-	CGDisplayModeRelease(mode);
-#if MVK_XCODE_13
-	if (framesPerSecond == 0 && [screen respondsToSelector: @selector(maximumFramesPerSecond)])
-     	framesPerSecond = [screen maximumFramesPerSecond];
-#endif
-
-	// Builtin panels, e.g., on MacBook, report a zero refresh rate.
-	if (framesPerSecond == 0)
-		framesPerSecond = 60.0;
-#endif
-
-	pRefreshCycleDuration->refreshDuration = (uint64_t)1e9 / framesPerSecond;
-	return VK_SUCCESS;
-}
-
-VkResult MVKSwapchain::getPastPresentationTiming(uint32_t *pCount, VkPastPresentationTimingGOOGLE *pPresentationTimings) {
-	if (_device->getConfigurationResult() != VK_SUCCESS) { return _device->getConfigurationResult(); }
-
-	VkResult res = VK_SUCCESS;
-
-	std::lock_guard<std::mutex> lock(_presentHistoryLock);
-	if (pPresentationTimings == nullptr) {
-		*pCount = _presentHistoryCount;
-	} else {
-		uint32_t countRemaining = std::min(_presentHistoryCount, *pCount);
-		uint32_t outIndex = 0;
-
-		res = (*pCount >= _presentHistoryCount) ? VK_SUCCESS : VK_INCOMPLETE;
-		*pCount = countRemaining;
-
-		while (countRemaining > 0) {
-			pPresentationTimings[outIndex] = _presentTimingHistory[_presentHistoryHeadIndex];
-			countRemaining--;
-			_presentHistoryCount--;
-			_presentHistoryHeadIndex = (_presentHistoryHeadIndex + 1) % kMaxPresentationHistory;
-			outIndex++;
-		}
-	}
-
-	return res;
-}
-
-void MVKSwapchain::recordPresentTime(const MVKImagePresentInfo& presentInfo, uint64_t actualPresentTime) {
-	std::lock_guard<std::mutex> lock(_presentHistoryLock);
-	if (_presentHistoryCount < kMaxPresentationHistory) {
-		_presentHistoryCount++;
-	} else {
-		_presentHistoryHeadIndex = (_presentHistoryHeadIndex + 1) % kMaxPresentationHistory;
-	}
-
-	// If actual present time is not available, use desired time instead, and if that
-	// hasn't been set, use the current time, which should be reasonably accurate (sub-ms),
-	// since we are here as part of the addPresentedHandler: callback.
-	if (actualPresentTime == 0) { actualPresentTime = presentInfo.desiredPresentTime; }
-	if (actualPresentTime == 0) { actualPresentTime = CACurrentMediaTime() * 1.0e9; }
-
-	_presentTimingHistory[_presentHistoryIndex].presentID = presentInfo.presentID;
-	_presentTimingHistory[_presentHistoryIndex].desiredPresentTime = presentInfo.desiredPresentTime;
-	_presentTimingHistory[_presentHistoryIndex].actualPresentTime = actualPresentTime;
-	// These details are not available in Metal
-	_presentTimingHistory[_presentHistoryIndex].earliestPresentTime = actualPresentTime;
-	_presentTimingHistory[_presentHistoryIndex].presentMargin = 0;
-	_presentHistoryIndex = (_presentHistoryIndex + 1) % kMaxPresentationHistory;
-}
-
-void MVKSwapchain::setLayerNeedsDisplay(const VkPresentRegionKHR* pRegion) {
-	if (!pRegion || pRegion->rectangleCount == 0) {
-		[_mtlLayer setNeedsDisplay];
-		return;
-	}
-
-	for (uint32_t i = 0; i < pRegion->rectangleCount; ++i) {
-		CGRect cgRect = mvkCGRectFromVkRectLayerKHR(pRegion->pRectangles[i]);
-#if MVK_MACOS
-		// VK_KHR_incremental_present specifies an upper-left origin, but macOS by default
-		// uses a lower-left origin.
-		cgRect.origin.y = _mtlLayer.bounds.size.height - cgRect.origin.y;
-#endif
-		// We were given rectangles in pixels, but -[CALayer setNeedsDisplayInRect:] wants them
-		// in points, which is pixels / contentsScale.
-		CGFloat scaleFactor = _mtlLayer.contentsScale;
-		cgRect.origin.x /= scaleFactor;
-		cgRect.origin.y /= scaleFactor;
-		cgRect.size.width /= scaleFactor;
-		cgRect.size.height /= scaleFactor;
-		[_mtlLayer setNeedsDisplayInRect:cgRect];
-	}
-}
-
-// A retention loop exists between the swapchain and its images. The swapchain images
-// retain the swapchain because they can be in flight when the app destroys the swapchain.
-// Release the images now, when the app destroys the swapchain, so they will be destroyed when
-// no longer held by the presentation flow, and will in turn release the swapchain for destruction.
 void MVKSwapchain::destroy() {
+	if (_surface->_activeSwapchain == this) { _surface->_activeSwapchain = nullptr; }
 	for (auto& img : _presentableImages) { _device->destroyPresentableSwapchainImage(img, NULL); }
 	MVKVulkanAPIDeviceObject::destroy();
 }
 
 MVKSwapchain::~MVKSwapchain() {
     if (_licenseWatermark) { _licenseWatermark->destroy(); }
-	releaseLayer();
 }
 

MoltenVK/MoltenVK/GPUObjects/MVKSync.h

@@ -63,6 +63,9 @@ public:
 	/** Returns whether this instance is in a reserved state. */
 	bool isReserved();
 
+	/** Returns the number of outstanding reservations. */
+	uint32_t getReservationCount();
+
 	/**
 	 * Blocks processing on the current thread until any or all (depending on configuration) outstanding
      * reservations have been released, or until the specified timeout interval in nanoseconds expires.
@@ -89,20 +92,19 @@ public:
 	 * require a separate call to the release() function to cause the semaphore to stop blocking.
 	 */
     MVKSemaphoreImpl(bool waitAll = true, uint32_t reservationCount = 0)
-        : _shouldWaitAll(waitAll), _reservationCount(reservationCount) {}
+        : _reservationCount(reservationCount), _shouldWaitAll(waitAll) {}
 
-    /** Destructor. */
     ~MVKSemaphoreImpl();
 
 
 private:
 	bool operator()();
-    inline bool isClear() { return _reservationCount == 0; }    // Not thread-safe
+    bool isClear() { return _reservationCount == 0; }    // Not thread-safe
 
 	std::mutex _lock;
 	std::condition_variable _blocker;
-	bool _shouldWaitAll;
 	uint32_t _reservationCount;
+	bool _shouldWaitAll;
 };
 
 

MoltenVK/MoltenVK/GPUObjects/MVKSync.mm

@@ -50,6 +50,11 @@ bool MVKSemaphoreImpl::isReserved() {
 	return !isClear();
 }
 
+uint32_t MVKSemaphoreImpl::getReservationCount() {
+	lock_guard<mutex> lock(_lock);
+	return _reservationCount;
+}
+
 bool MVKSemaphoreImpl::wait(uint64_t timeout, bool reserveAgain) {
     unique_lock<mutex> lock(_lock);
 

MoltenVK/MoltenVK/Utility/MVKBaseObject.h

@@ -50,7 +50,7 @@ public:
 	void reportMessage(MVKConfigLogLevel logLevel, const char* format, ...) __printflike(3, 4);
 
 	/**
-	 * Report a Vulkan error message, on behalf of the object, which may be nil.
+	 * Report a message, on behalf of the object, which may be nil.
 	 * Reporting includes logging to a standard system logging stream, and if the object
 	 * is not nil and has access to the VkInstance, the message will also be forwarded
 	 * to the VkInstance for output to the Vulkan debug report messaging API.
@@ -58,14 +58,19 @@ public:
 	static void reportMessage(MVKBaseObject* mvkObj, MVKConfigLogLevel logLevel, const char* format, ...) __printflike(3, 4);
 
 	/**
-	 * Report a Vulkan error message, on behalf of the object, which may be nil.
+	 * Report a Vulkan result message. This includes logging to a standard system logging stream,
+	 * and some subclasses will also forward the message to their VkInstance for output to the
+	 * Vulkan debug report messaging API.
+	 */
+	VkResult reportResult(VkResult vkRslt, MVKConfigLogLevel logLevel, const char* format, ...) __printflike(4, 5);
+
+	/**
+	 * Report a Vulkan result message, on behalf of the object. which may be nil.
 	 * Reporting includes logging to a standard system logging stream, and if the object
 	 * is not nil and has access to the VkInstance, the message will also be forwarded
 	 * to the VkInstance for output to the Vulkan debug report messaging API.
-	 *
-	 * This is the core reporting implementation. Other similar functions delegate here.
 	 */
-	static void reportMessage(MVKBaseObject* mvkObj, MVKConfigLogLevel logLevel, const char* format, va_list args) __printflike(3, 0);
+	static VkResult reportResult(MVKBaseObject* mvkObj, VkResult vkRslt, MVKConfigLogLevel logLevel, const char* format, ...) __printflike(4, 5);
 
 	/**
 	 * Report a Vulkan error message. This includes logging to a standard system logging stream,
@@ -83,19 +88,29 @@ public:
 	static VkResult reportError(MVKBaseObject* mvkObj, VkResult vkErr, const char* format, ...) __printflike(3, 4);
 
 	/**
-	 * Report a Vulkan error message, on behalf of the object. which may be nil.
+	 * Report a Vulkan warning message. This includes logging to a standard system logging stream,
+	 * and some subclasses will also forward the message to their VkInstance for output to the
+	 * Vulkan debug report messaging API.
+	 */
+	VkResult reportWarning(VkResult vkRslt, const char* format, ...) __printflike(3, 4);
+
+	/**
+	 * Report a Vulkan warning message, on behalf of the object. which may be nil.
 	 * Reporting includes logging to a standard system logging stream, and if the object
 	 * is not nil and has access to the VkInstance, the message will also be forwarded
 	 * to the VkInstance for output to the Vulkan debug report messaging API.
-	 *
-	 * This is the core reporting implementation. Other similar functions delegate here.
 	 */
-	static VkResult reportError(MVKBaseObject* mvkObj, VkResult vkErr, const char* format, va_list args) __printflike(3, 0);
+	static VkResult reportWarning(MVKBaseObject* mvkObj, VkResult vkRslt, const char* format, ...) __printflike(3, 4);
 
 	/** Destroys this object. Default behaviour simply deletes it. Subclasses may override to delay deletion. */
 	virtual void destroy() { delete this; }
 
     virtual ~MVKBaseObject() {}
+
+protected:
+	static VkResult reportResult(MVKBaseObject* mvkObj, VkResult vkRslt, MVKConfigLogLevel logLevel, const char* format, va_list args) __printflike(4, 0);
+	static void reportMessage(MVKBaseObject* mvkObj, MVKConfigLogLevel logLevel, const char* format, va_list args) __printflike(3, 0);
+
 };
 
 

MoltenVK/MoltenVK/Utility/MVKBaseObject.mm

@@ -26,24 +26,19 @@
 using namespace std;
 
 
-static const char* getReportingLevelString(MVKConfigLogLevel logLevel) {
-	switch (logLevel) {
-		case MVK_CONFIG_LOG_LEVEL_DEBUG:
-			return "mvk-debug";
-		case MVK_CONFIG_LOG_LEVEL_INFO:
-			return "mvk-info";
-		case MVK_CONFIG_LOG_LEVEL_WARNING:
-			return "mvk-warn";
-		case MVK_CONFIG_LOG_LEVEL_ERROR:
-		default:
-			return "mvk-error";
-	}
-}
-
-
 #pragma mark -
 #pragma mark MVKBaseObject
 
+static const char* getReportingLevelString(MVKConfigLogLevel logLevel) {
+	switch (logLevel) {
+		case MVK_CONFIG_LOG_LEVEL_ERROR:    return "mvk-error";
+		case MVK_CONFIG_LOG_LEVEL_WARNING:  return "mvk-warn";
+		case MVK_CONFIG_LOG_LEVEL_INFO:     return "mvk-info";
+		case MVK_CONFIG_LOG_LEVEL_DEBUG:    return "mvk-debug";
+		default:                            return "mvk-unknown";
+	}
+}
+
 string MVKBaseObject::getClassName() { return mvk::getTypeName(this); }
 
 void MVKBaseObject::reportMessage(MVKConfigLogLevel logLevel, const char* format, ...) {
@@ -102,10 +97,43 @@ void MVKBaseObject::reportMessage(MVKBaseObject* mvkObj, MVKConfigLogLevel logLe
 	free(redoBuff);
 }
 
+VkResult MVKBaseObject::reportResult(VkResult vkErr, MVKConfigLogLevel logLevel, const char* format, ...) {
+	va_list args;
+	va_start(args, format);
+	VkResult rslt = reportResult(this, vkErr, logLevel, format, args);
+	va_end(args);
+	return rslt;
+}
+
+VkResult MVKBaseObject::reportResult(MVKBaseObject* mvkObj, VkResult vkErr, MVKConfigLogLevel logLevel, const char* format, ...) {
+	va_list args;
+	va_start(args, format);
+	VkResult rslt = reportResult(mvkObj, vkErr, logLevel, format, args);
+	va_end(args);
+	return rslt;
+}
+
+VkResult MVKBaseObject::reportResult(MVKBaseObject* mvkObj, VkResult vkRslt, MVKConfigLogLevel logLevel, const char* format, va_list args) {
+
+	// Prepend the result code to the format string
+	const char* vkRsltName = mvkVkResultName(vkRslt);
+	size_t rsltLen = strlen(vkRsltName) + strlen(format) + 4;
+	char fmtStr[rsltLen];
+	snprintf(fmtStr, rsltLen, "%s: %s", vkRsltName, format);
+
+	// Report the message
+	va_list lclArgs;
+	va_copy(lclArgs, args);
+	reportMessage(mvkObj, logLevel, fmtStr, lclArgs);
+	va_end(lclArgs);
+
+	return vkRslt;
+}
+
 VkResult MVKBaseObject::reportError(VkResult vkErr, const char* format, ...) {
 	va_list args;
 	va_start(args, format);
-	VkResult rslt = reportError(this, vkErr, format, args);
+	VkResult rslt = reportResult(this, vkErr, MVK_CONFIG_LOG_LEVEL_ERROR, format, args);
 	va_end(args);
 	return rslt;
 }
@@ -113,25 +141,23 @@ VkResult MVKBaseObject::reportError(VkResult vkErr, const char* format, ...) {
 VkResult MVKBaseObject::reportError(MVKBaseObject* mvkObj, VkResult vkErr, const char* format, ...) {
 	va_list args;
 	va_start(args, format);
-	VkResult rslt = reportError(mvkObj, vkErr, format, args);
+	VkResult rslt = reportResult(mvkObj, vkErr, MVK_CONFIG_LOG_LEVEL_ERROR, format, args);
 	va_end(args);
 	return rslt;
 }
 
-// This is the core reporting implementation. Other similar functions delegate here.
+VkResult MVKBaseObject::reportWarning(VkResult vkErr, const char* format, ...) {
-VkResult MVKBaseObject::reportError(MVKBaseObject* mvkObj, VkResult vkErr, const char* format, va_list args) {
+	va_list args;
-
+	va_start(args, format);
-	// Prepend the error code to the format string
+	VkResult rslt = reportResult(this, vkErr, MVK_CONFIG_LOG_LEVEL_WARNING, format, args);
-	const char* vkRsltName = mvkVkResultName(vkErr);
+	va_end(args);
-	size_t rsltLen = strlen(vkRsltName) + strlen(format) + 4;
+	return rslt;
-	char fmtStr[rsltLen];
+}
-	snprintf(fmtStr, rsltLen, "%s: %s", vkRsltName, format);
+
-
+VkResult MVKBaseObject::reportWarning(MVKBaseObject* mvkObj, VkResult vkErr, const char* format, ...) {
-	// Report the error
+	va_list args;
-	va_list lclArgs;
+	va_start(args, format);
-	va_copy(lclArgs, args);
+	VkResult rslt = reportResult(mvkObj, vkErr, MVK_CONFIG_LOG_LEVEL_WARNING, format, args);
-	reportMessage(mvkObj, MVK_CONFIG_LOG_LEVEL_ERROR, fmtStr, lclArgs);
+	va_end(args);
-	va_end(lclArgs);
+	return rslt;
-
-	return vkErr;
 }

MoltenVK/MoltenVK/Utility/MVKFoundation.cpp

@@ -21,6 +21,44 @@
 
 #define CASE_STRINGIFY(V)  case V: return #V
 
+const char* mvkVkCommandName(MVKCommandUse cmdUse) {
+	switch (cmdUse) {
+		case kMVKCommandUseBeginCommandBuffer:           return "vkBeginCommandBuffer (prefilled VkCommandBuffer)";
+		case kMVKCommandUseQueueSubmit:                  return "vkQueueSubmit";
+		case kMVKCommandUseAcquireNextImage:             return "vkAcquireNextImageKHR";
+		case kMVKCommandUseQueuePresent:                 return "vkQueuePresentKHR";
+		case kMVKCommandUseQueueWaitIdle:                return "vkQueueWaitIdle";
+		case kMVKCommandUseDeviceWaitIdle:               return "vkDeviceWaitIdle";
+		case kMVKCommandUseInvalidateMappedMemoryRanges: return "vkInvalidateMappedMemoryRanges";
+		case kMVKCommandUseBeginRendering:               return "vkCmdBeginRendering";
+		case kMVKCommandUseBeginRenderPass:              return "vkCmdBeginRenderPass";
+		case kMVKCommandUseNextSubpass:                  return "vkCmdNextSubpass";
+		case kMVKCommandUseRestartSubpass:               return "Metal renderpass restart on barrier";
+		case kMVKCommandUsePipelineBarrier:              return "vkCmdPipelineBarrier";
+		case kMVKCommandUseBlitImage:                    return "vkCmdBlitImage";
+		case kMVKCommandUseCopyImage:                    return "vkCmdCopyImage";
+		case kMVKCommandUseResolveImage:                 return "vkCmdResolveImage (resolve stage)";
+		case kMVKCommandUseResolveExpandImage:           return "vkCmdResolveImage (expand stage)";
+		case kMVKCommandUseResolveCopyImage:             return "vkCmdResolveImage (copy stage)";
+		case kMVKCommandUseCopyBuffer:                   return "vkCmdCopyBuffer";
+		case kMVKCommandUseCopyBufferToImage:            return "vkCmdCopyBufferToImage";
+		case kMVKCommandUseCopyImageToBuffer:            return "vkCmdCopyImageToBuffer";
+		case kMVKCommandUseFillBuffer:                   return "vkCmdFillBuffer";
+		case kMVKCommandUseUpdateBuffer:                 return "vkCmdUpdateBuffer";
+		case kMVKCommandUseClearAttachments:             return "vkCmdClearAttachments";
+		case kMVKCommandUseClearColorImage:              return "vkCmdClearColorImage";
+		case kMVKCommandUseClearDepthStencilImage:       return "vkCmdClearDepthStencilImage";
+		case kMVKCommandUseResetQueryPool:               return "vkCmdResetQueryPool";
+		case kMVKCommandUseDispatch:                     return "vkCmdDispatch";
+		case kMVKCommandUseTessellationVertexTessCtl:    return "vkCmdDraw (vertex and tess control stages)";
+		case kMVKCommandUseDrawIndirectConvertBuffers:   return "vkCmdDrawIndirect (convert indirect buffers)";
+		case kMVKCommandUseCopyQueryPoolResults:         return "vkCmdCopyQueryPoolResults";
+		case kMVKCommandUseAccumOcclusionQuery:          return "Post-render-pass occlusion query accumulation";
+		case kMVKCommandUseRecordGPUCounterSample:       return "Record GPU Counter Sample";
+		default:                                         return "Unknown Vulkan command";
+	}
+}
+
 const char* mvkVkResultName(VkResult vkResult) {
 	switch (vkResult) {
 

MoltenVK/MoltenVK/Utility/MVKFoundation.h

@@ -63,7 +63,7 @@ typedef struct {
 /** Tracks the Vulkan command currently being used. */
 typedef enum : uint8_t {
     kMVKCommandUseNone = 0,                     /**< No use defined. */
-	kMVKCommandUseEndCommandBuffer,             /**< vkEndCommandBuffer (prefilled VkCommandBuffer). */
+	kMVKCommandUseBeginCommandBuffer,           /**< vkBeginCommandBuffer (prefilled VkCommandBuffer). */
     kMVKCommandUseQueueSubmit,                  /**< vkQueueSubmit. */
 	kMVKCommandUseAcquireNextImage,             /**< vkAcquireNextImageKHR. */
     kMVKCommandUseQueuePresent,                 /**< vkQueuePresentKHR. */
@@ -104,6 +104,9 @@ enum MVKGraphicsStage {
 	kMVKGraphicsStageRasterization	/**< The rest of the pipeline. */
 };
 
+/** Returns the name of the command defined by the command use. */
+const char* mvkVkCommandName(MVKCommandUse cmdUse);
+
 /** Returns the name of the result value. */
 const char* mvkVkResultName(VkResult vkResult);
 

MoltenVK/MoltenVK/Utility/MVKLogging.h

@@ -57,9 +57,9 @@ extern "C" {
  *		MVKLogErrorIf(cond, fmt, ...)	- same as MVKLogError if boolean "cond" condition expression evaluates to YES,
  *										  otherwise logs nothing.
  *
- *		MVKLogWarning(fmt, ...)		- recommended for not immediately harmful errors
+ *		MVKLogWarn(fmt, ...)			- recommended for not immediately harmful errors
  *										- will print if MVK_LOG_LEVEL_WARNING is set on.
- *		MVKLogWarningIf(cond, fmt, ...) - same as MVKLogWarning if boolean "cond" condition expression evaluates to YES,
+ *		MVKLogWarnIf(cond, fmt, ...) 	- same as MVKLogWarn if boolean "cond" condition expression evaluates to YES,
  *										  otherwise logs nothing.
  *
  *		MVKLogInfo(fmt, ...)			- recommended for general, infrequent, information messages
@@ -67,7 +67,7 @@ extern "C" {
  *		MVKLogInfoIf(cond, fmt, ...)	- same as MVKLogInfo if boolean "cond" condition expression evaluates to YES,
  *										  otherwise logs nothing.
  *
- *		MVKLogDebug(fmt, ...)		- recommended for temporary use during debugging
+ *		MVKLogDebug(fmt, ...)			- recommended for temporary use during debugging
  *										- will print if MVK_LOG_LEVEL_DEBUG is set on.
  *		MVKLogDebugIf(cond, fmt, ...)	- same as MVKLogDebug if boolean "cond" condition expression evaluates to YES,
  *										  otherwise logs nothing.
@@ -148,11 +148,11 @@ extern "C" {
 
 // Warning logging - for not immediately harmful errors
 #if MVK_LOG_LEVEL_WARNING
-#	define MVKLogWarning(fmt, ...)			MVKLogWarningImpl(fmt, ##__VA_ARGS__)
+#	define MVKLogWarn(fmt, ...)				MVKLogWarnImpl(fmt, ##__VA_ARGS__)
-#	define MVKLogWarningIf(cond, fmt, ...)	if(cond) { MVKLogWarningImpl(fmt, ##__VA_ARGS__); }
+#	define MVKLogWarnIf(cond, fmt, ...)		if(cond) { MVKLogWarnImpl(fmt, ##__VA_ARGS__); }
 #else
-#    define MVKLogWarning(...)
+#    define MVKLogWarn(...)
-#    define MVKLogWarningIf(cond, fmt, ...)
+#    define MVKLogWarnIf(cond, fmt, ...)
 #endif
 
 // Info logging - for general, non-performance affecting information messages
@@ -182,11 +182,11 @@ extern "C" {
 #	define MVKLogTraceIf(cond, fmt, ...)
 #endif
 
-#define MVKLogErrorImpl(fmt, ...)		reportMessage(MVK_CONFIG_LOG_LEVEL_ERROR, fmt, ##__VA_ARGS__)
+#define MVKLogErrorImpl(fmt, ...)			reportMessage(MVK_CONFIG_LOG_LEVEL_ERROR, fmt, ##__VA_ARGS__)
-#define MVKLogWarningImpl(fmt, ...)		reportMessage(MVK_CONFIG_LOG_LEVEL_WARNING, fmt, ##__VA_ARGS__)
+#define MVKLogWarnImpl(fmt, ...)			reportMessage(MVK_CONFIG_LOG_LEVEL_WARNING, fmt, ##__VA_ARGS__)
-#define MVKLogInfoImpl(fmt, ...)		reportMessage(MVK_CONFIG_LOG_LEVEL_INFO, fmt, ##__VA_ARGS__)
+#define MVKLogInfoImpl(fmt, ...)			reportMessage(MVK_CONFIG_LOG_LEVEL_INFO, fmt, ##__VA_ARGS__)
-#define MVKLogDebugImpl(fmt, ...)		reportMessage(MVK_CONFIG_LOG_LEVEL_DEBUG, fmt, ##__VA_ARGS__)
+#define MVKLogDebugImpl(fmt, ...)			reportMessage(MVK_CONFIG_LOG_LEVEL_DEBUG, fmt, ##__VA_ARGS__)
-#define MVKLogTraceImpl(fmt, ...)		reportMessage(MVK_CONFIG_LOG_LEVEL_DEBUG, fmt, ##__VA_ARGS__)
+#define MVKLogTraceImpl(fmt, ...)			reportMessage(MVK_CONFIG_LOG_LEVEL_DEBUG, fmt, ##__VA_ARGS__)
 
 // Assertions
 #ifdef NS_BLOCK_ASSERTIONS

Scripts/runcts

@@ -113,7 +113,7 @@ export MVK_CONFIG_USE_METAL_ARGUMENT_BUFFERS=0          #(2 = VK_EXT_descriptor_
 export MVK_CONFIG_VK_SEMAPHORE_SUPPORT_STYLE=2          #(2 = MTLEvents always)
 export MVK_CONFIG_SHADER_COMPRESSION_ALGORITHM=0        #(2 = ZLIB, 3 = LZ4)
 export MVK_CONFIG_PERFORMANCE_TRACKING=0
-export MVK_CONFIG_ACTIVITY_PERFORMANCE_LOGGING_STYLE=2  #(2 = Device lifetime)
+export MVK_CONFIG_ACTIVITY_PERFORMANCE_LOGGING_STYLE=3  #(2 = Device lifetime, 3 = Process lifetime)
 
 # -------------- Operation --------------------