PXE boot crashed

When it comes PXE boot, various issue can occur. I was reported one incident that all the clients fail to PXE boot and that the clients reboot all of a sudden halfway.

The smsts.log has the following information.

01-05-2021 17:06:25.756 TSPxe 1432 (0x598) CLibSMSMessageWinHttpTransport::Send: WinHttpOpenRequest - URL: contoso.com:80 CCM_POST /ccm_system/request
01-05-2021 17:06:25.756 TSPxe 1432 (0x598) Not in SSL.
01-05-2021 17:06:28.013 TSPxe 1432 (0x598) Request was successful.
01-05-2021 17:06:28.560 TSPxe 1432 (0x598) dwBodyLength <= m_nMaxReplySize, HRESULT=80004005 (..\libsmsmessaging.cpp,2404) 01-05-2021 17:06:28.560 TSPxe 1432 (0x598) reply message body length is too long (17733718, 16777216) 01-05-2021 17:06:28.560 TSPxe 1432 (0x598) DoRequest (sReply, true), HRESULT=80004005 (..\libsmsmessaging.cpp,3167) 01-05-2021 17:06:28.560 TSPxe 1432 (0x598) oPolicyAssignments.RequestAssignments(), HRESULT=80004005 (..\tspolicy.cpp,1318) 01-05-2021 17:06:28.560 TSPxe 1432 (0x598) Failed to request policy assignments (Code 0x80004005) 01-05-2021 17:06:28.560 TSPxe 1432 (0x598) m_pPolicyManager->init( sMP, nHttpPort, nHttpsPort, sSiteCode, bUseCRL, L"", sMediaPfx, sMediaGuid, sEnterpriseCert, sServerCerts, sSiteSigningCert, sAuthenticator), HRESULT=80004005 (tsmediawizardcontrol.cpp,1189)
01-05-2021 17:06:28.560 TSPxe 1432 (0x598) Failed to initialize policy from Management Point
01-05-2021 17:06:28.560 TSPxe 1432 (0x598) Exiting TSMediaWizardControl::GetPolicy.
01-05-2021 17:06:28.560 TSPxe 1432 (0x598) GetPolicy(), HRESULT=80004005 (tsmediawizardcontrol.cpp,2547)
01-05-2021 17:06:28.560 TSPxe 1432 (0x598) RunWizardForPXE(), HRESULT=80004005 (tsmediawizardcontrol.cpp,2900)
01-05-2021 17:06:28.560 TSPxe 1432 (0x598) oTSMediaWizardControl.Run(sMediaRoot, true, sTSLaunchMode), HRESULT=80004005 (tsmbootstrap.cpp,1139)
01-05-2021 17:06:28.560 TSPxe 1432 (0x598) Failed to run from PXE in WinPE
01-05-2021 17:06:28.560 TSPxe 1432 (0x598) Execute( eExecutionEnv, sConfigPath, sTSXMLFile, uBootCount, bReloadEnv, &uExitCode ), HRESULT=80004005 (tsmbootstrap.cpp,1291)
01-05-2021 17:06:28.560 TSPxe 1432 (0x598) Exiting with return code 0x80004005
01-05-2021 17:06:28.575 TSBootShell 724 (0x2d4) Execution complete.
01-05-2021 17:06:28.575 TSBootShell 724 (0x2d4) hMap != 0, HRESULT=80070002 (..\environmentscope.cpp,493)
01-05-2021 17:06:28.575 TSBootShell 724 (0x2d4) m_pGlobalScope->open(), HRESULT=80070002 (..\environmentlib.cpp,335)
01-05-2021 17:06:28.575 TSBootShell 724 (0x2d4) this->open(), HRESULT=80070002 (..\environmentlib.cpp,561)
01-05-2021 17:06:28.575 TSBootShell 724 (0x2d4) ::RegOpenKeyExW (HKEY_LOCAL_MACHINE, sKey.c_str(), 0, KEY_READ, &hSubKey), HRESULT=80070002 (..\utils.cpp,1107)
01-05-2021 17:06:28.575 TSBootShell 724 (0x2d4) RegOpenKeyExW is unsuccessful for Software\Microsoft\SMS\Task Sequence
01-05-2021 17:06:28.575 TSBootShell 724 (0x2d4) GetTsRegValue() is unsuccessful. 0x80070002.
01-05-2021 17:06:28.575 TSBootShell 724 (0x2d4) End program:
01-05-2021 17:06:28.575 TSBootShell 724 (0x2d4) Finalizing logging from process 660
01-05-2021 17:06:28.575 TSBootShell 724 (0x2d4) Finalizing logs to root of first available drive
01-05-2021 17:06:28.575 TSBootShell 724 (0x2d4) LOGGING: Setting log directory to "X:\SMSTSLog".

The problem lies in “reply message body length is too long“. By default, the maximum of policy size allowed for a device is 16M. This is hardcoded and cannot be adjusted. To solve the issue, just manual delete unnecessary deployments to the device or device collection. Suppose you deploy task sequence to All Unknown Computers. In that case, go to its Properties by right click and check if there are too many deployments. If yes, remove some of them that you no longer need.

Here is an explanation from Microsoft Technet –

The limit is on OVERALL policy (not just Task Sequence policy) to the device. As mentioned this is not just Task Sequence policy but all deployments to the device – applications, packages, software updates, etc. Reducing the deployments reduces the policy which fixes the issue. Again this does not just have to be Task Sequence deployments – it can be any deployment. In fact one of the biggest culprits is usually a large amount of software updates deployed to the device. Reduce the amount of updates deployed to the device and it will also fix the problem.

BTW 16MB is actually a HUGE amount of policy. More careful targeting and maintenance again usually resolve this problem.

There are multiple bugs on the issue and work has been done lately to help improve the situation, mainly by being more efficient with policy and getting rid of old policy that is no longer needed (e.g. backward compatibility with ConfigMgr 2007). The first of these efforts are now in ConfigMgr 1902 with additional improvements expected in 1906 (including hopefully fixing the bug that currently limits the policy to 16MB). If this is a serious issue and you have found that all of their deployments are valid and there is no cleanup to do, you may want to consider upgrading to 1902 so that they get some of these improvements.

References

• https://social.technet.microsoft.com/Forums/ie/en-US/c0a8c80a-d16d-48a1-82e8-2d92cac47b12/issue-osd-before-select-ts?forum=ConfigMgrCBOSD

Office 365 update failed to download with error 0x8007362e

When you use SCCM to download Office 365 updates, you may come across some annoying errors. For example, 0x8007362e, which means –

Error Code: 0x362E (13870)
Error Name: ERROR_IPSEC_IKE_INVALID_HASH
Error Source: Windows
Error Message: Hash verification failed

One of the possible reasons is that the server from which you downloaded the office update has the wrong update file. In fact, Microsoft provides a cluster of servers servicing the updates download. Theoretically, all those servers should have the same file content for a given office update. However, it may not be the case in reality. Sometimes things can go against your expectation as Microsoft also makes mistakes. If it is the file downloaded from the server caused the issue, one easy solution is to specify IP mappings in the hosts file in C:\Windows\System32\drivers\etc where SCCM console is installed, eg. –

23.36.252.68 officecdn.microsoft.com

You may wonder how should I know which IP I should use? Good question. If you capture a network trace file during the update download, you can find the actual IP address used for officedn.microsoft.com. And if you have another environment where you happen to succeed in downloading the same office update, you can capture a network trace file and look for the actual IP used during the download and then you can use that IP address for the other environment. Of course, this is not the best way. But you have no choice because there is nothing you can do about the updates servers that Microsoft provides. But Microsoft usually will fix the update file in short time.

What is interesting is that 0x8007362e can also mean other reasons, though literally it indicates a problem of hash verification. Let’s take a look at the following issue. One customer tried to download Microsoft 365 Apps Update – Semi-Annual Enterprise Channel Version 2002 for x64 based Edition (build 12527.21416).

PatchDownloader.log prints the following information –

01-06-2021 14:41:58.545 Software Updates Patch Downloader 78556 (0x132dc) Query to run: select f.FileName, ct.ContentSource from SMS_CIToContent c join SMS_CIContentFiles f on c.ContentID = f.ContentID join SMS_Content ct on c.ContentID = ct.ContentID where c.ContentDownloaded = 1 and f.FileHash = 'Sha256:67D805F422D110F3A346EA9F09747D59ED4BBB3AC1C5A9CE3AD8C9B534D892D9'
01-06-2021 14:41:59.101 Software Updates Patch Downloader 78556 (0x132dc) Downloading content for ContentID = 16801840, FileName = office\data\16.0.12527.21416\stream.x64.x-none.dat.
01-06-2021 14:41:59.102 Software Updates Patch Downloader 78556 (0x132dc) Proxy is enabled for download, using registry settings or defaults.
01-06-2021 14:41:59.110 Software Updates Patch Downloader 30668 (0x77cc) Connecting - Adding file range by calling HttpAddRequestHeaders, range string = "Range: bytes=0-"
01-06-2021 14:42:54.147 Software Updates Patch Downloader 30668 (0x77cc) Download http://officecdn.microsoft.com/pr/7ffbc6bf-bc32-4f92-8982-f9dd17fd3114/office/data/16.0.12527.21416/stream.x64.x-none.dat in progress: 10 percent complete
01-06-2021 14:43:40.883 Software Updates Patch Downloader 30668 (0x77cc) Download http://officecdn.microsoft.com/pr/7ffbc6bf-bc32-4f92-8982-f9dd17fd3114/office/data/16.0.12527.21416/stream.x64.x-none.dat in progress: 20 percent complete
01-06-2021 14:44:31.577 Software Updates Patch Downloader 30668 (0x77cc) Download http://officecdn.microsoft.com/pr/7ffbc6bf-bc32-4f92-8982-f9dd17fd3114/office/data/16.0.12527.21416/stream.x64.x-none.dat in progress: 30 percent complete
01-06-2021 14:45:14.107 Software Updates Patch Downloader 30668 (0x77cc) Download http://officecdn.microsoft.com/pr/7ffbc6bf-bc32-4f92-8982-f9dd17fd3114/office/data/16.0.12527.21416/stream.x64.x-none.dat in progress: 40 percent complete
01-06-2021 14:46:14.106 Software Updates Patch Downloader 30668 (0x77cc) Download http://officecdn.microsoft.com/pr/7ffbc6bf-bc32-4f92-8982-f9dd17fd3114/office/data/16.0.12527.21416/stream.x64.x-none.dat in progress: 50 percent complete
01-06-2021 14:47:04.850 Software Updates Patch Downloader 30668 (0x77cc) Download http://officecdn.microsoft.com/pr/7ffbc6bf-bc32-4f92-8982-f9dd17fd3114/office/data/16.0.12527.21416/stream.x64.x-none.dat in progress: 60 percent complete
01-06-2021 14:47:48.153 Software Updates Patch Downloader 30668 (0x77cc) Download http://officecdn.microsoft.com/pr/7ffbc6bf-bc32-4f92-8982-f9dd17fd3114/office/data/16.0.12527.21416/stream.x64.x-none.dat in progress: 70 percent complete
01-06-2021 14:48:47.669 Software Updates Patch Downloader 30668 (0x77cc) Download http://officecdn.microsoft.com/pr/7ffbc6bf-bc32-4f92-8982-f9dd17fd3114/office/data/16.0.12527.21416/stream.x64.x-none.dat in progress: 80 percent complete
01-06-2021 14:49:45.376 Software Updates Patch Downloader 30668 (0x77cc) Download http://officecdn.microsoft.com/pr/7ffbc6bf-bc32-4f92-8982-f9dd17fd3114/office/data/16.0.12527.21416/stream.x64.x-none.dat in progress: 90 percent complete
01-06-2021 14:50:48.126 Software Updates Patch Downloader 30668 (0x77cc) Download http://officecdn.microsoft.com/pr/7ffbc6bf-bc32-4f92-8982-f9dd17fd3114/office/data/16.0.12527.21416/stream.x64.x-none.dat to C:\Users\KWS~1\AppData\Local\Temp\1\CAB7C18.tmp.dat returns 0
01-06-2021 14:50:48.131 Software Updates Patch Downloader 30668 (0x77cc) Using machine settings for CRL checking.
01-06-2021 14:50:48.135 Software Updates Patch Downloader 30668 (0x77cc) Cert revocation check is disabled so cert revocation list will not be checked.
01-06-2021 14:50:48.137 Software Updates Patch Downloader 30668 (0x77cc) To enable cert revocation check use: UpdDwnldCfg.exe /checkrevocation
01-06-2021 14:50:48.138 Software Updates Patch Downloader 30668 (0x77cc) Verifying file hash C:\Users\KWS~1\AppData\Local\Temp\1\CAB7C18.tmp.dat
01-06-2021 14:51:02.110 Software Updates Patch Downloader 30668 (0x77cc) Hash verification of file C:\Users\KWS~1\AppData\Local\Temp\1\CAB7C18.tmp.dat failed, error 0x17
01-06-2021 14:51:02.389 Software Updates Patch Downloader 78556 (0x132dc) ERROR: DownloadUpdateContent() failed with hr=0x8007362e

I tried to use IP mapping in hosts file as mentioned in the begging of the article, but in vain. Then I noticed the highlighted query in the above log. Turing to the site database, I ran the query below –

select * from vCI_ContentFiles where Content_ID='16801840' and FileHash='Sha256:67D805F422D110F3A346EA9F09747D59ED4BBB3AC1C5A9CE3AD8C9B534D892D9'

With no surprise, it returns one result exactly the same as printed in the PatchDownloader.log. The source url column has the actual download address of the update –

http://officecdn.microsoft.com/pr/7ffbc6bf-bc32-4f92-8982-f9dd17fd3114/office/data/16.0.12527.21416/stream.x64.x-none.dat

I downloaded the file with this link and verified its true hash value with –

Get-FileHash -Algorithm sha256 "C:\Users\KWS\Downloads\stream.x64.x-none.dat"

The actual hash value is 42649E74276D9746430BC323B3AF0F32404667A9279022BC591E7223B4CE6D3A. Next step is easy. I ran the following DML statement to update the database –

Update vCI_ContentFiles set FileHash='Sha256:42649E74276D9746430BC323B3AF0F32404667A9279022BC591E7223B4CE6D3A' where Content_ID='16801840' and FileHash='Sha256:67D805F422D110F3A346EA9F09747D59ED4BBB3AC1C5A9CE3AD8C9B534D892D9'

Finally, the update got downloaded successfully.

01-06-2021 17:04:03.768 Software Updates Patch Downloader 59304 (0xe7a8) Query to run: select f.FileName, ct.ContentSource from SMS_CIToContent c join SMS_CIContentFiles f on c.ContentID = f.ContentID join SMS_Content ct on c.ContentID = ct.ContentID where c.ContentDownloaded = 1 and f.FileHash = 'Sha256:42649E74276D9746430BC323B3AF0F32404667A9279022BC591E7223B4CE6D3A'
01-06-2021 17:04:04.146 Software Updates Patch Downloader 59304 (0xe7a8) Downloading content for ContentID = 16801840, FileName = office\data\16.0.12527.21416\stream.x64.x-none.dat.
01-06-2021 17:04:04.147 Software Updates Patch Downloader 59304 (0xe7a8) Proxy is enabled for download, using registry settings or defaults.
01-06-2021 17:04:04.154 Software Updates Patch Downloader 44036 (0xac04) Connecting - Adding file range by calling HttpAddRequestHeaders, range string = "Range: bytes=0-"
01-06-2021 17:04:44.258 Software Updates Patch Downloader 44036 (0xac04) Download http://officecdn.microsoft.com/pr/7ffbc6bf-bc32-4f92-8982-f9dd17fd3114/office/data/16.0.12527.21416/stream.x64.x-none.dat in progress: 10 percent complete
01-06-2021 17:05:35.130 Software Updates Patch Downloader 44036 (0xac04) Download http://officecdn.microsoft.com/pr/7ffbc6bf-bc32-4f92-8982-f9dd17fd3114/office/data/16.0.12527.21416/stream.x64.x-none.dat in progress: 20 percent complete
01-06-2021 17:06:22.481 Software Updates Patch Downloader 44036 (0xac04) Download http://officecdn.microsoft.com/pr/7ffbc6bf-bc32-4f92-8982-f9dd17fd3114/office/data/16.0.12527.21416/stream.x64.x-none.dat in progress: 30 percent complete
01-06-2021 17:07:08.299 Software Updates Patch Downloader 44036 (0xac04) Download http://officecdn.microsoft.com/pr/7ffbc6bf-bc32-4f92-8982-f9dd17fd3114/office/data/16.0.12527.21416/stream.x64.x-none.dat in progress: 40 percent complete
01-06-2021 17:08:30.809 Software Updates Patch Downloader 44036 (0xac04) Download http://officecdn.microsoft.com/pr/7ffbc6bf-bc32-4f92-8982-f9dd17fd3114/office/data/16.0.12527.21416/stream.x64.x-none.dat in progress: 50 percent complete
01-06-2021 17:09:04.307 Software Updates Patch Downloader 44036 (0xac04) Download http://officecdn.microsoft.com/pr/7ffbc6bf-bc32-4f92-8982-f9dd17fd3114/office/data/16.0.12527.21416/stream.x64.x-none.dat in progress: 60 percent complete
01-06-2021 17:09:36.654 Software Updates Patch Downloader 44036 (0xac04) Download http://officecdn.microsoft.com/pr/7ffbc6bf-bc32-4f92-8982-f9dd17fd3114/office/data/16.0.12527.21416/stream.x64.x-none.dat in progress: 70 percent complete
01-06-2021 17:10:12.204 Software Updates Patch Downloader 44036 (0xac04) Download http://officecdn.microsoft.com/pr/7ffbc6bf-bc32-4f92-8982-f9dd17fd3114/office/data/16.0.12527.21416/stream.x64.x-none.dat in progress: 80 percent complete
01-06-2021 17:10:53.899 Software Updates Patch Downloader 44036 (0xac04) Download http://officecdn.microsoft.com/pr/7ffbc6bf-bc32-4f92-8982-f9dd17fd3114/office/data/16.0.12527.21416/stream.x64.x-none.dat in progress: 90 percent complete
01-06-2021 17:11:29.600 Software Updates Patch Downloader 44036 (0xac04) Download http://officecdn.microsoft.com/pr/7ffbc6bf-bc32-4f92-8982-f9dd17fd3114/office/data/16.0.12527.21416/stream.x64.x-none.dat to C:\Users\KWS~1\AppData\Local\Temp\1\CABBA65.tmp.dat returns 0
01-06-2021 17:11:29.606 Software Updates Patch Downloader 44036 (0xac04) Using machine settings for CRL checking.
01-06-2021 17:11:29.607 Software Updates Patch Downloader 44036 (0xac04) Cert revocation check is disabled so cert revocation list will not be checked.
01-06-2021 17:11:29.607 Software Updates Patch Downloader 44036 (0xac04) To enable cert revocation check use: UpdDwnldCfg.exe /checkrevocation
01-06-2021 17:11:29.608 Software Updates Patch Downloader 44036 (0xac04) Verifying file hash C:\Users\KWS~1\AppData\Local\Temp\1\CABBA65.tmp.dat
01-06-2021 17:11:46.191 Software Updates Patch Downloader 44036 (0xac04) File hash verified: C:\Users\KWS~1\AppData\Local\Temp\1\CABBA65.tmp.dat

Hope this article helps if you run into similar issues.

Thanks for reading.

Package validation status sending failed with reply has “no message header marker” & “Failed to send status message (80004005)”

The primary site has dozens of secondary sites. Each of the secondary sites has management point role installed and one remote distribution point. Each package sent to the distribution will be validated and if the validation is successful the validation will be sent to the management point.

However, some distribution points have a problem – package validation status fail to send to its management point. If we reassign the remote distribution point to the primary site, the issue will be gone. But once reassigned back to the secondary site, the issues re-appears. The issue can also be reproduced by running the command –

smsdpmon.exe <PACKAGE-ID>

For example –

smsdpmon.exe kwspkg001

The smsdpmon.log on the distribution point has the following exceptions –

11-27-2020 13:01:03.047 SMS_Distribution_Point_Monitoring 4208 (0x1070) Start to evaluate package share for package 'kwspkg001' version 0 …
11-27-2020 13:01:03.062 SMS_Distribution_Point_Monitoring 4208 (0x1070) Package kwspkg001 is verified successfully
11-27-2020 13:01:03.062 SMS_Distribution_Point_Monitoring 4208 (0x1070) Report state message 0x40000950 to MP
11-27-2020 13:01:03.078 SMS_Distribution_Point_Monitoring 4208 (0x1070) Report Body: kwspkg001["Display=\kwsdistpoint.contoso.com\"]MSWNET:["SMS_SITE=SKQ"]\kwsdistpoint.contoso.com\
11-27-2020 13:01:03.093 SMS_Distribution_Point_Monitoring 4208 (0x1070) SSL, using authenticator in request.
11-27-2020 13:01:03.093 SMS_Distribution_Point_Monitoring 4208 (0x1070) In SSL, but with no client cert.
11-27-2020 13:01:03.203 SMS_Distribution_Point_Monitoring 4208 (0x1070) Report status message 0x40000950 to MP
11-27-2020 13:01:03.234 SMS_Distribution_Point_Monitoring 4208 (0x1070) SSL, using authenticator in request.
11-27-2020 13:01:03.234 SMS_Distribution_Point_Monitoring 4208 (0x1070) In SSL, but with no client cert.
11-27-2020 13:01:03.312 SMS_Distribution_Point_Monitoring 4208 (0x1070) reply has no message header marker
11-27-2020 13:01:03.312 SMS_Distribution_Point_Monitoring 4208 (0x1070) Failed to send status message (80004005)
11-27-2020 13:01:03.312 SMS_Distribution_Point_Monitoring 4208 (0x1070) Error sending status message to management point(s) 'https://kwsmgmtpoint.contoso.com' (port 80) from remote DP. Verify management point(s). (code 0x80004005)
11-27-2020 13:01:03.312 SMS_Distribution_Point_Monitoring 4208 (0x1070) CSMSDPMonitoring::ReportStatusMessage failed; 0x80004005

The ClientAuth.log on the secondary site’s management point has the following information registered –

11-27-2020 13:01:03.202 ClientAuth 13592 (0x3518) Error verifying message from client 'b18b40dc-389d-4a60-8a42-f2330f9e71ba' (0x87d00238).

Searching with SMSID ‘b18b40dc-389d-4a60-8a42-f2330f9e71ba‘ in the tables ClientKeyData, ClientKeyData_GP, ClientKeyDataCertExtend, I found that relevant information is complete. ClientkeyData_GP should be replicated between the primary site and the secondary site. If all is good, the same information found in the ClientKeyData_GP table in the database of the secondary site should also exist in the site database of the primary site. While checking the information in the ClientKeyData_GP table of the database of the primary site, I was surprised to find that the expected entry was missing. That is where the problem comes from – replication about ClientKeyData_GP table between the secondary site and the primary site is not working as it should.

But database replication looks good in sccm console – I had no idea why. Initialization and replication was good. All the receive and sent time stamps are the latest. rcmctrl.log all good! Checking the vLogs in the database of the secondary site with the following query, no errors.

SELECT * FROM vLogs WHERE LogText like 'error' order by LogTime DESC;

Examining the vLogs of the database of the primary site, wow, a few exceptions popped into my eyes –

ERROR 545, Level 16, State 1, Procedure SMSDBAuditTrigger_SC_SiteDefinition_Property_INS_UPD_DEL, Line 8, Message: Explicit value must be specified for identity column in table 'SCCM_Audit' either when IDENTITY_INSERT is set to ON or when a replication user is inserting into a NOT FOR REPLICATION identity column.

ERROR 545, Level 16, State 1, Procedure SMSDBMON_CM_RoleSSLCertificates_CM_RoleSSLCertificates_ins, Line 3, Message: Explicit value must be specified for identity column in table 'TableChangeNotifications' either when IDENTITY_INSERT is set to ON or when a replication user is inserting into a NOT FOR REPLICATION identity column.

ERROR 545, Level 16, State 1, Procedure tr_ClientKeyData_Revoked, Line 17, Message: Explicit value must be specified for identity column in table 'CertRevocationNotify' either when IDENTITY_INSERT is set to ON or when a replication user is inserting into a NOT FOR REPLICATION identity column.

ERROR 545, Level 16, State 1, Procedure trCollTrack_v_EAS_Property, Line 14, Message: Explicit value must be specified for identity column in table 'CollectionNotifications' either when IDENTITY_INSERT is set to ON or when a replication user is inserting into a NOT FOR REPLICATION identity column.

ERROR: LogPoisonMessage Exception [System.Data.SqlClient.SqlException (0x80131904): Explicit value must be specified for identity column in table 'DrsPoisonMessageInfo' either when IDENTITY_INSERT is set to ON or when a replication user is inserting into a NOT FOR REPLICATION identity column. at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action1 wrapCloseInAction) at System.Data.SqlClient.SqlInternalConnectionSmi.EventSink.DispatchMessages(Boolean ignoreNonFatalMessages) at System.Data.SqlClient.SqlCommand.RunExecuteNonQuerySmi(Boolean sendToPipe) at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(TaskCompletionSource1 completion, String methodName, Boolean sendToPipe, Int32 timeout, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry) at System.Data.SqlClient.SqlCommand.ExecuteNonQuery() at Microsoft.ConfigurationManager.DataReplicationService.DatabaseOperation.ExecuteNonQuery(String commandText, List`1 commandParameters, Int32 timeOut) at Microsoft.ConfigurationManager.DataReplicationService.DrsLogging.LogPoisonMessage(String tableName, String exceptionMessage, String poisonMessageSiteCode, DataTable poisonMsgTable, Int64 lastSyncVersionToSource, Int32 replicationID, String procedureName) ClientConnectionId:00000000-0000-0000-0000-000000000000 Error Number:545,State:1,Class:16]

NOT FOR REPLICATION of the ID columns of 5 tables CertRevocationNotify，DrsPoisonMessageInfo ，TableChangeNotifications，CollectionNotifications，SCCM_Audit should be OFF instead of ON.

So, I suggested the solution –

ALTER TABLE CollectionNotifications ALTER COLUMN RecordID DROP NOT FOR REPLICATION;
ALTER TABLE CertRevocationNotify ALTER COLUMN ID DROP NOT FOR REPLICATION;
ALTER TABLE DrsPoisonMessageInfo ALTER COLUMN ID DROP NOT FOR REPLICATION;
ALTER TABLE TableChangeNotifications  ALTER COLUMN RecordID DROP NOT FOR REPLICATION;
ALTER TABLE SCCM_Audit ALTER COLUMN ID DROP NOT FOR REPLICATION;

After that the issue was gone.

References

• Replicate Identity Columns – SQL Server | Microsoft Docs

• Change Not For Replication Value for SQL Server Identity Columns (mssqltips.com)

Catalog of Third-Party Softeware Updates Sync now failed

We followed the official guide to configure third-party software updates for Configuration Manager. After having added custom catalog, we tried the menu Sync now on the head ribbon. Unfortunately, it does not work.

Logs show the following exceptions:

WSUSCtrl:
Attempting connection to local WSUS server 2720 (0x0AA0)
System.Net.WebException: The request failed with HTTP status 403: Forbidden. at Microsoft.UpdateServices.Administration.AdminProxy.CreateUpdateServer(Object[] args) at Microsoft.SystemsManagementServer.WSUS.WSUSServer.ConnectToWSUSServer(String ServerName, Boolean UseSSL, Int32 PortNumber) 2720 (0x0AA0)
Failures reported during periodic health check by the WSUS Server CONTOSO.COM. Will retry check in 1 minutes 2720 (0x0AA0)
Waiting for changes for 1 minutes 2720 (0x0AA0)
Timed Out… 2720 (0x0AA0)

SMS_ISVUPDATES_SYNCAGENT.log:
==================== Exception Detail Start ======================= 5184 (0x1440)
Exception type: WebException 5184 (0x1440)
Exception HRESULT: -2146233079 5184 (0x1440)
Exception Message: The request failed with HTTP status 403: Forbidden. 5184 (0x1440)
Exception source Microsoft.UpdateServices.Administration 5184 (0x1440)
Exception TargetSite Microsoft.UpdateServices.Administration.IUpdateServer CreateUpdateServer(System.Object[]) 5184 (0x1440)
Stack at Microsoft.UpdateServices.Administration.AdminProxy.CreateUpdateServer(Object[] args)~~ at Microsoft.ConfigurationManager.ISVUpdatesSyncAgent.WSUS.UpdateServicesWrapper.Connect() 5184 (0x1440)
===================== Exception Detail End ======================== 5184 (0x1440)

WCM.log:
Successfully connected to server: CONTOSO.com, port: 8531, useSSL: True 70592 (0x113C0)
Waiting for changes for 59 minutes 70592 (0x113C0)
Wait timed out after 59 minutes while waiting for at least one trigger event. 70592 (0x113C0)
Timed Out… 70592 (0x113C0)

In addition, SCCM SMS_WSUS_CONTROL_MANAGER kept printing repeated Error 7000 and 7003.

WSUS Control Manager failed to monitor WSUS Server “CONTOSO.COM”. Possible cause: WSUS Server version 3.0 SP2 or above is not installed or cannot be contacted. Solution: Verify that the WSUS Server version 3.0 SP2 or greater is installed. Verify that the IIS ports configured in the site are same as those configured on the WSUS IIS website.

Then we checked the health on SUP:

1. Open CMD as admin
2. Navigate to C:\Program Files\Update Services\Tools
3. Run: WSUSUtil.exe checkhealth

I found Event ID 12002, 12052, 12042, 12022, 12032, 12012 in Windows Server Update Services event log-

The description for Event ID 12052 from source Windows Server Update Services cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.

If the event originated on another computer, the display information had to be saved with the event.

The following information was included with the event:

The DSS Authentication Web Service is not working.

The only step we missed was configure wsus for ssl with wsusutil configuressel. So we tried the following steps:

1. Open CMD as admin and navigate to C:\Program Files\Update Services\Tools
2. Run (Case sensitive): wsusutil.exe configuressl <FQDN-OF-WSUS-SERVER>
3. Restart WSUS Service from services and WSUS Administration from IIS Administration control

Wonderful! The issue was gone.

References

• https://docs.microsoft.com/en-us/mem/configmgr/sum/deploy-use/third-party-software-updates#enable-third-party-updates-on-the-sup
• https://msunleashed.wordpress.com/2011/05/02/sms_wsus_control_manager-event-id-7000-7003/

Package distribution keeps failing with “CSendFileAction::AddFile failed; 0x80070570”

Two days ago, I took an issue about package failing to update on distribution point with the following Asset Message in Content Status:

Package Transfer Manager failed to update the package “CT001N”, Version 2 on distribution point MP server. Review PkgXferMgr.log for more information about this failure.

Looking at the the PkgXferMgr.log from the primary site, I found the following exceptions:

Sending file ‘\DEVDP1.contoso.com\SMS_DP$\CF2FB2829436844A999B071455159D15CA1DE04D057EB721BA05767DE003C052-CT0001N.3438.temp’ 333856 (0x51820)
Adding PFC20200407.ini file in CT0001N.3438. 333856 (0x51820)
ExecStaticMethod failed (80070570) SMS_DistributionPoint, AddFile 333856 (0x51820)
CSendFileAction::AddFile failed; 0x80070570 333856 (0x51820)
Failed to add the file PFC20200407.ini in content CT0001N.3438. Error 0x80070570 333856 (0x51820)
CSendFileAction::AddFileMetaData failed; 0x80070570 333856 (0x51820)
CSendFileAction::SendFiles failed; 0x80070570 333856 (0x51820)
CSendFileAction::SendContent failed; 0x80070570 333856 (0x51820)

0x80070570 indicates “The file or directory is corrupted and unreadable.” The quickest way to fix this is to remove the broken package from the target DP and redistribute the package.