Category: Content Management

Primary Server fails to distribute content to distribution point servers with 0x800706BA

Primary Server fails to distribute content to distribution point servers with 0x800706BA

ISSUE

Primary Server fails to distribute content to distribution point servers with 0x800706BA

BACKGROUND

The primary server was Windows Server 2012 R2 and worked well before upgraded to Windows Server 2019.

The upgraded Primary server has no latest patch installed.

LOG

distmgr.log – 

SMS_DISTRIBUTION_MANAGER 15420 (0x3c3c) CWmi::Connect() failed to connect to \contoso.lab.com\root\CIMv2. Error = 0x800706BA

Connecting CIMv2 from the primary server to the distribution point server fails with 0x80070005 meaning Access is denied.

Solution

  1. Open registry on the distribution point
  2. Path : HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Ole\AppCompat
  3. Value Name: RequireIntegrityActivationAuthenticationLevel
  4. Type: dword
  5. Value Data: 0x00000000
  6. Restart the distribution point server

You can do this in PowerShell:

# Add the registry
New-ItemProperty -Path Registry::HKLM\SOFTWARE\Microsoft\Ole\AppCompat -Name RequireIntegrityActivationAuthenticationLevel -Value 0 -PropertyType DWORD -Force

# Restart the machine
Restart-Computer

Refer to – https://support.microsoft.com/en-us/topic/kb5004442-manage-changes-for-windows-dcom-server-security-feature-bypass-cve-2021-26414-f1400b52-c141-43d2-941e-37ed901c769c

Or apply the latest patches up to at least June to the primary server and distribution point servers.

Package failed to distribute to remote distribution point

Package failed to distribute to remote distribution point

ISSUE

Package failed to distribute to remote distribution point

ANALYSIS

distmgr.log on the primary server has the following.

SMS_DISTRIBUTION_MANAGER    44660 (0xae74)    Start adding package CON008AB to server ["Display=\\contosodp.test.lab\"]MSWNET:["SMS_SITE=FWD"]\\contosodp.test.lab\...
SMS_DISTRIBUTION_MANAGER    44660 (0xae74)    ~Created DP processing thread 11880 for addition or update of package CON008AB on server ["Display=\\contosodp.test.lab\"]MSWNET:["SMS_SITE=FWD"]\\contosodp.test.lab\
SMS_DISTRIBUTION_MANAGER    11880 (0x2e68)    DP Thread: Attempting to add or update package CON008AB on DP ["Display=\\contosodp.test.lab\"]MSWNET:["SMS_SITE=FWD"]\\contosodp.test.lab\
SMS_DISTRIBUTION_MANAGER    11880 (0x2e68)    DPConnection::Connect: For contosodp.test.lab, logged-on as test\sccm_admin~
SMS_DISTRIBUTION_MANAGER    11880 (0x2e68)    The current user context will be used for connecting to ["Display=\\contosodp.test.lab\"]MSWNET:["SMS_SITE=FWD"]\\contosodp.test.lab\.~
SMS_DISTRIBUTION_MANAGER    11880 (0x2e68)    Failed to make a network connection to \\contosodp.test.lab\ADMIN$ (0x4b8).~
SMS_DISTRIBUTION_MANAGER    11880 (0x2e68)    ~Cannot establish connection to ["Display=\\contosodp.test.lab\"]MSWNET:["SMS_SITE=FWD"]\\contosodp.test.lab\. Error = 1208
SMS_DISTRIBUTION_MANAGER    11880 (0x2e68)    STATMSG: ID=2323 SEV=E LEV=M SOURCE="SMS Server" COMP="SMS_DISTRIBUTION_MANAGER" SYS=contosopri.test.lab SITE=FWD PID=2872 TID=11880 GMTDATE=Thu Mar 10 07:20:38.816 2022 ISTR0="30" ISTR1="8" ISTR2="" ISTR3="" ISTR4="" ISTR5="" ISTR6="" ISTR7="" ISTR8="" ISTR9="" NUMATTRS=2 LE=0X0 AID0=400 AVAL0="CON008AB" AID1=404 AVAL1="["Display=\\contosodp.test.lab\"]MSWNET:["SMS_SITE=FWD"]\\contosodp.test.lab\"
SMS_DISTRIBUTION_MANAGER    11880 (0x2e68)    Error occurred. Performing error cleanup prior to returning.
SMS_DISTRIBUTION_MANAGER    44660 (0xae74)    ~DP thread for package CON008AB with thread handle 0000000000006358 and thread ID 11880 ended.
SMS_DISTRIBUTION_MANAGER    44660 (0xae74)    DP thread with ID 11880 failed to process DP action

I tested the SMB port with TNC contosodp.test.lab -Port 445 and the result was good. Checked the primary server computer account and test\sccm_admin account were in the local admin group of the distribution point and that the account test\sccm_admin was not disabled or locked in AD. All looked good.

Ben, I turned to capturing a network trace to find out what actually was going on with the network communication between the primary server and the distribution point.

Here is the network trace. 

10.50.1.52	10.10.126.33	TCP	66	64478 → 445 [SYN, ECN, CWR] Seq=0 Win=8192 Len=0 MSS=1460 WS=256 SACK_PERM=1
10.10.126.33	10.50.1.52	TCP	66	445 → 64478 [SYN, ACK] Seq=0 Ack=1 Win=8192 Len=0 MSS=1296 WS=256 SACK_PERM=1
10.50.1.52	10.10.126.33	TCP	54	64478 → 445 [ACK] Seq=1 Ack=1 Win=262912 Len=0
10.50.1.52	10.10.126.33	SMB2	232	Negotiate Protocol Request
10.10.126.33	10.50.1.52	SMB2	366	Negotiate Protocol Response
	10.50.1.52	10.10.126.33	SMB2	1947	Session Setup Request
10.10.126.33	10.50.1.52	TCP	60	445 → 64478 [ACK] Seq=313 Ack=2072 Win=66048 Len=0
10.10.126.33	10.50.1.52	SMB2	262	KRB Error: KRB5KRB_AP_ERR_TKT_NYV
	10.50.1.52	10.10.126.33	SMB2	1890	Session Setup Request
10.10.126.33	10.50.1.52	TCP	60	445 → 64478 [ACK] Seq=521 Ack=3908 Win=66048 Len=0
10.10.126.33	10.50.1.52	SMB2	248	KRB Error: KRB5KRB_AP_ERR_TKT_NYV
10.50.1.52	10.10.126.33	TCP	54	64478 → 445 [RST, ACK] Seq=3908 Ack=715 Win=0 Len=0

KRB5KRB_AP_ERR_TKT_NYV caught the eye, which means ticket not yet valid.

“Check the time synchronization between the client, filer and KDC. It should be less than 5 minutes and should be in same time zone. If they are in sync, the ticket is not valid, it needs to be created. Add the proper principals to the KDC and create the proper key tab files for the client and filer and retry the Kerberos mount as per the procedure.”

After examination, we did find that the culprit was indeed the time difference between the distribution point and the KDC (generally the AD) which was more than 5 minutes!

SOLUTION

Package was distributed successfully after making the time difference between the distribution point and the KDC less than 5 minutes.

References

Packages failed to redistribute to distribution point with “FileOpen failed; 0x80070570”

This is another issue following my previous post “Package distribution keeps failing with “CSendFileAction::AddFile failed; 0x80070570”“. The primary server failed to distribute packages to its distribution point due to the exceptions in smsdpprov.log on the distribution point:

[13B0][Thu 05/14/2020 10:07:51]:FileOpen failed; 0x80070570
[13B0][Thu 05/14/2020 10:07:51]:CContentDefinition::CreatePackedSignature failed; 0x80070570
[13B0][Thu 05/14/2020 10:07:51]:Failed to create packed signatures for content ‘a4e5db1b-4b44-4c84-82a3-4586628003d2’ for package ‘CAS007C6’. Error code: 0X80070570
[350][Thu 05/14/2020 10:14:36]:CreateFileW failed for D:\SMSSIG$\CAS00850.6.tar

0X80070570 refers to “The file or directory is corrupted and unreadable”. When I used PsExec to connect to the remote distribution point as system account and created a file by 

PsExec -s \\dpserverCMD
fsutil file createnew D:\SMSSIG$\test.txt 1024

I could create successfully the test.txt file in D:\SMSSIG$. However, the file test.txt is not visible in D:\SMSSIG$ directory in the file explorer. The strange thing is that the test.txt could be opened if I put in the address bar in the file explorer D:\SMSSIG$\test.txt and pressed enter.

What is going on? I scratched my head. Then I tried to look for the ghost file from the command prompt:

dir D:\SMSSIG$ | findstr test.txt

Guess what? The file could not be found! This can explain the exceptions in the smsdpprov.log. It can be deduced that there is something wrong with the directory SMSSIG$. So, I turned to the System event log and came across the following information:

Log Name: System
Source: Ntfs
Date: 2020/5/12 15:59:47
Event ID: 55
Task Category: None
Level: Error
Keywords:
User: SYSTEM
Computer: dpserver.contoso.com
Description:
A corruption was discovered in the file system structure on volume D:.
A corruption was found in a file system index structure. The file reference number is 0x20000000000ac. The name of the file is “\SMSSIG$”. The corrupted index attribute is “:$I30:$INDEX_ALLOCATION”.

Ntfs 55 indicates that file corresponding metadata was damaged when the system accesses the file. Based on this information, I ran the following commands to fix the drive:

  1. Scan the problematic disk completely
chkntfs D: >C:\chkntfs.txt
chkdsk D: /scan >C:\chkdskscan.txt
  1. After Step 1 completes, run CHKDSK command to attempt to fix the corruption
Chkdsk D: /f /r >C:\chkdskresult.txt

Note: When the volume is in use, chkdsk will prompt the following to unmount the volume, we need to manually enter Y to process the dismount operation. The operation will not be displayed, you need to enter Y manually, the chkdsk command can continue to execute. You can refresh the file and then double click to open the file to view the command execution. You can also use CMtrace to view the file.

Chkdsk may run if this volume is dismounted first.
ALL OPENED HANDLES TO THIS VOLUME WOULD THEN BE INVALID.
Would you like to force a dismount on this volume? (Y/N)

During the process, if you see

there is an active paging file on it. Would you like to schedule

just put in Y, which means “schedule this volume to be checked the next time the system restarts”. Then restart the machine and the CHKDSK will run automatically on reboot.

After this operation, packages distribution returned back to normal. Regarding the failed packages which count as many as over seventy, I used this modified script to redistribute the packages in batch.

Redistribute Failed packages in SCCM

Kindly test your environment before using production.

The script will check specified DP for packages that has failed according to summarizer events and redist any failed packages.

When working with ConfigMgr you always end up distributing content to several DPs. This normaly goes off without a hitch but from time to time this fails. If you then have several DPs spread across a large geographical area, WAN links may be questionable. So when a package then fails most are not happy to redistribute the content to all DPs again.

Function Update-ContentForSingleDP{
param (
[Parameter(Mandatory=$true)]
$SiteCode,
[Parameter(Mandatory=$true)]
$DPFQDN
)
$Failures = Get-WmiObject -Namespace root\sms\site_$SiteCode -Class sms_packagestatusDistPointsSummarizer | Where-Object State -GT 2 | Where-Object SourceNALPath -Match $DPFQDN 
$DP = Get-WmiObject -Namespace root\sms\site_$SiteCode -Class sms_distributionpoint | Where-Object ServerNalPath -match $DPFQDN
foreach ($Failure in $Failures) { 
    $PackageID = $Failure.PackageID 
    Write-Output "Failed PackageID: $PackageID" 

    $DPPackage = $DP | Where-Object PackageID -EQ $PackageID 
    $DPPackage.RefreshNow = $true 
    $DPPackage.put() | Out-Null
}
}

Note that here I assume that objects whose State is greater than 2 are failed packages instead of the original -EQ 2 in the MSDN script because some failed packages have other values grater than 2.

References

Design a site like this with WordPress.com
Get started