Two steps about Client Upgrade in Post-Installation part stuck in In progress

ISSUE

Two steps about Client Upgrade in Post-Installation part stuck in In progress during an upgrade from 1909 to 2006

These two steps are : 

  1. Updating Client folder on Site Server 

  2. Updating Configuration Manager Client Package 

hman.log

Loaded client upgrade settings from DB successfully. FullClientPackageID=SG100002, StagingClientPackageID=SG100017, ClientUpgradePackageID=SG100003, PilotingUpgradePackageID=SG10074E, ClientUpgradeAdvertisementID=SG120000, ClientPilotingAdvertisementID=SG120BFF
Both UpgradeAdvertisementID and PilotingAdvertisementID existed in the output of ClientDeploymentSettings, while, the advertisementID 'SG120000' couldn't be found in table ProgramOffers, that's why these two steps stuck there, and keep being retried. 
    SQL>>>select PkgID,PkgProgram,CollectionID,SourceSite,OfferName,OfferComment,Hierarchy,AvailableSched,UnavailableSched,MandatorySched,PresentTime,ExpirationTime,TimeEnableFlag,OfferFlags,DeviceFlags,RemoteClientFlags,Priority,AssignmentID,MandatoryCountdown,SourceLocaleID,LastModificationTime,OfferTypeID,ISVString,AdditionalProperties from ProgramOffers where OfferID = 'SG120000' and Action <> 3
SQL>>>>> Done.
ERROR: Failed to get the advertisement object for advertisement SG120000.
ERROR: Failed to add a new mandatory schedule to the client upgrade advertisement SG120000. Will continue on next cycle.
WARN: It fails to process the client content in the update package A200E842-5CB5-4BD3-AB77-2999B59721AE. Will retry in next cycle.

Solution

1. Go to the top-level site 
2. Navigate into <ConfigMgr>\inboxes\hman.box 
3. Create an empty CliUpg.acu file in <ConfigMgr>\inboxes\hman.box 
4. Restart SMS_EXECUTIVE service from services.msc panel 

Management point role installation fails on the secondary site with “Child process exited with non-zero code 102”

Issue

After an upgrade from 2010 to 2103, the management point role installation on a secondary site keeps failing with the following exceptions in sitecom.log –

08-31-2021 12:40:37.867 SMS_SITE_COMPONENT_MANAGER 4408 (0x1138) Installed service SMS_SERVER_BOOTSTRAP_STLCANSCCM.
08-31-2021 12:40:37.868 SMS_SITE_COMPONENT_MANAGER 4408 (0x1138) Starting service SMS_SERVER_BOOTSTRAP_STLCANSCCM with command-line arguments "SWK D:\Program Files\Microsoft Configuration Manager /install D:\Program Files\Microsoft Configuration Manager\bin\x64\rolesetup.exe SMSBGB "…
08-31-2021 12:42:17.015 SMS_SITE_COMPONENT_MANAGER 4408 (0x1138) Execution of "D:\Program Files\Microsoft Configuration Manager\bin\x64\rolesetup.exe /install /siteserver:CONTOSO.LOCAL.LAB" on server CONTOSO.LOCAL.LAB failed: Child process exited with non-zero code 102.
08-31-2021 12:42:17.016 SMS_SITE_COMPONENT_MANAGER 4408 (0x1138) bgbisapi.msi could not install correctly on server CONTOSO.LOCAL.LAB.
08-31-2021 12:42:17.016 SMS_SITE_COMPONENT_MANAGER 4408 (0x1138) Bootstrap operation failed.

102 here means “The semaphore is set and cannot be closed“. However, it is hard to figure out what that refers to.

Error Code: 0x66 (102)
Error Name: ERROR_SEM_IS_SET
Error Source: Windows
Error Message: The semaphore is set and cannot be closed.

Wikipedia explains it this way –

A semaphore, in computer science, is a protected variable (an entity storing a value) or abstract data type (an entity grouping several variables that may or may not be numerical) which constitutes the classic method for restricting access to shared resources, such as shared memory, in a multiprogramming environment (a system where several programs may be executing, or taking turns to execute, at once). Semaphores exist in many variants, though usually the term refers to a counting semaphore, since a binary semaphore is better known as a mutex. A counting semaphore is a counter for a set of available resources, rather than a locked/unlocked flag of a single resource. It was invented by Edsger Dijkstra. Semaphores are the classic solution to preventing race conditions in the dining philosophers problem, although they do not prevent resource deadlocks.

Capturing a ProcMon, I found the process where 102 was thrown and launched the process tree, where the unlodctr.exe caught my eye. This command “D:\Program Files\Microsoft Configuration Manager\bin\x64\rolesetup.exe /install /siteserver:CONTOSO.LOCAL.LAB” was actually calling “C:\Windows\system32\unlodctr.exe” “sms notification server” to unload the performance counters for sms notification server. But it failed.

The procmon content is as below –

12:42:15.2625967 8/31/2021 12:42:15 rolesetup.exe 5516 Process Exit SUCCESS Exit Status: 102, User Time: 0.0937500 seconds, Kernel Time: 0.2812500 seconds, Private Bytes: 2,666,496, Peak Private Bytes: 2,916,352, Working Set: 13,082,624, Peak Working Set: 13,430,784

SOLUTION

1. Open an elevated CMD
2. Run
   1. cd \windows\system32
   2. lodctr /R
   3. Restart the secondary site

If the above does not work, you can further try the following:

1. Uninstall the MP role from SCCM console
2. Delete CCM name instance from WMI
  a. Open WBEMTest and connect to root
  b. Click enum classes and choose "Recursive" then click okay
  c. Under Query Result go to _NAMESPACE and double click it.
  d. Hit the "Instances" button on the right hand side.
  e. Then highlight the _NAMESPACE.name="ccm" value and click delete to remove it.
3. Install MP again.

References

• https://docs.microsoft.com/en-us/windows-server/administration/windows-commands/unlodctr
• https://docs.microsoft.com/en-us/troubleshoot/windows-server/performance/rebuild-performance-counter-library-values

Feature Update 1909 failed with 0xC1900208

The customer was trying to upgrade from 1703 to 1909 via Feature Update but came across the following exception.

” 0xC1900208 ” means – The system does not pass the compat scan to install the update.

setuperr.log has the following –

06-07-2021 07:34:12.000 MOUPG CDlpTask::CheckUserInterruptEx(3060): Result = 0x800704D3
06-07-2021 07:34:12.000 MOUPG CDlpTask::CheckUserInterrupt(3112): Result = 0x800704D3
06-07-2021 07:34:15.000 MOUPG CSetupManager::ExecuteInstallMode(822): Result = 0xC1900208
06-07-2021 07:34:15.000 MOUPG CSetupManager::ExecuteDownlevelMode(392): Result = 0xC1900208
06-07-2021 07:34:34.000 MOUPG CSetupManager::Execute(284): Result = 0xC1900208
06-07-2021 07:34:34.000 MOUPG CSetupHost::Execute(420): Result = 0xC1900208

But what was blocking the upgrade?

An easy way is to launch setup.exe from the C:\$WINDOWS.~BT directory. Afer you the first screens, the setup will warn you about possible incompatible apps.

Alternatively, you can also refer to the C:\$WINDOWS.~BT\Sources\Panther\CompatData[datetime].xml, eg. CompatData_2021_06_07_07_34_10_3_006f0018.xml. There can usually be multiple Compatdata.xml files. But the most recent one should contain the information you desire. In my example, it is the VMware Player that brought the issue.

CompatData xml log shows an incompatible software that was blocking the upgrade.

Solution is –

1. Uninstall the incompatible software, in my case it is VMware Player
2. Clear SCCM Cache by opening PowerShell as Admin and run:
   $resman= New-Object -ComObject "UIResource.UIResourceMgr"
$cacheInfo=$resman.GetCacheInfo()
$cacheinfo.GetCacheElements() | foreach {$cacheInfo.DeleteCacheElement($_.CacheElementID)}
3. Remove the content of C:\WINDOWS\SoftwareDistribution\Download folder.
4. Note that you may need to stop the Windows Update service (wuauserv) before renaming or deleting the content in that directory. To stop that service, open PowerShell as Admin and run: Stop-Service wuauserv
5. Restart that service after deleting the content: Start-Service wuauserv
6. Remove the C:\$WINDOWS.~BT folder (Note that this folder is hidden)
7. Restart the computer
8. Try a Software Update Deployment Evaluation Cycle and Software Update Scan Cycle
9. Wait a couple of minute for the scan to complete and retry the deployment