Issue
Software Center is empty. There are no applications nor updates. We tested by deploying one application and one update but in vain.
Observation
CcmMessaging.log has no errors, all the messages being sent successfully to the management point associated with a secondary site.
05-24-2021 23:13:46.793 CcmMessaging 19792 (0x4d50) Sending async message '{0D37A652-BCC8-4169-8A1A-6E1574E2A070}' to outgoing queue 'mp:[http]mp_locationmanager'
05-24-2021 23:13:48.215 CcmMessaging 15764 (0x3d94) Sending outgoing message '{0D37A652-BCC8-4169-8A1A-6E1574E2A070}'. Flags 0x201, sender account empty
05-24-2021 23:13:48.516 CcmMessaging 15764 (0x3d94) Message '{0D37A652-BCC8-4169-8A1A-6E1574E2A070}' got reply '{DA626C8B-433E-42C4-8360-965EF2D60683}' to local endpoint queue 'LS_ReplyLocations'
05-24-2021 23:13:48.518 CcmMessaging 15764 (0x3d94) OutgoingMessage(Queue='mp_[http]mp_locationmanager', ID={0D37A652-BCC8-4169-8A1A-6E1574E2A070}): Delivered successfully to host 'secondarysite.contoso.com'.
Looking at LocationServices.log, we found the following messages –
05-24-2021 22:59:00.664 LocationServices 5924 (0x1724) 1 proxy MP errors in the last 10 minutes, threshold is 5. 05-24-2021 22:59:49.974 LocationServices 13528 (0x34d8) 2 proxy MP errors in the last 10 minutes, threshold is 5. 05-24-2021 23:00:01.747 LocationServices 9620 (0x2594) 3 proxy MP errors in the last 10 minutes, threshold is 5. 05-24-2021 23:00:31.063 LocationServices 13528 (0x34d8) 4 proxy MP errors in the last 10 minutes, threshold is 5. 05-24-2021 23:00:46.937 LocationServices 15764 (0x3d94) Proxy MP error threshold reached, moving to next MP.
DataTransferService.log –
05-24-2021 23:04:44.994 DataTransferService 15764 (0x3d94) BITS compatible path: http://secondarysite.contoso.com:80/SMS_MP/.sms_pol?%7BF4CA6035-61A2-4146-89C0-50DB591157A0%7D.SHA256:36A294B138C353F4487562349357BE61D23B9C631D8707BC79B712A5D836DC0F
05-24-2021 23:04:44.994 DataTransferService 15764 (0x3d94) BITSHelper: Full source path to be transferred = http://secondarysite.contoso.com:80/SMS_MP/.sms_pol?%7BF4CA6035-61A2-4146-89C0-50DB591157A0%7D.SHA256:36A294B138C353F4487562349357BE61D23B9C631D8707BC79B712A5D836DC0F
05-24-2021 23:04:44.994 DataTransferService 15764 (0x3d94) BITSHelper, remote name = http://secondarysite.contoso.com:80/SMS_MP/.sms_pol?%7BF4CA6035-61A2-4146-89C0-50DB591157A0%7D.SHA256:36A294B138C353F4487562349357BE61D23B9C631D8707BC79B712A5D836DC0F, local name = C:\WINDOWS\CCM\Staging{F4CA6035-61A2-4146-89C0-50DB591157A0}.1.00.tmp
Visiting the highlighted link in the browser on the client computer returns 500 Server Internal Error.
Bits job, retrieved by running “bitsadmin /list /allusers /verbose“, has the following information. Obviously, something goes wrong with the management point on the secondary site. More likely relevant to IIS.
UID: {A5A891C5-4EB7-4EB2-BE44-19BB40F66CCA} DISPLAY: 'CCMDTS Job'
TYPE: DOWNLOAD STATE: TRANSIENT_ERROR OWNER: NT AUTHORITY\SYSTEM
PRIORITY: HIGH FILES: 0 / 2 BYTES: 0 / UNKNOWN
CREATION TIME: 5/24/2021 11:04:44 PM MODIFICATION TIME: 5/24/2021 11:17:58 PM
COMPLETION TIME: UNKNOWN ACL FLAGS:
NOTIFY INTERFACE: REGISTERED NOTIFICATION FLAGS: 11
RETRY DELAY: 60 NO PROGRESS TIMEOUT: 28800 ERROR COUNT: 14
PROXY USAGE: NO_PROXY PROXY LIST: NULL PROXY BYPASS LIST: NULL
ERROR FILE: http://secondarysite.contoso.com:80/SMS_MP/.sms_pol?%7BF4CA6035-61A2-4146-89C0-50DB591157A0%7D.SHA256:36A294B138C353F4487562349357BE61D23B9C631D8707BC79B712A5D836DC0F -> C:\WINDOWS\CCM\Staging{F4CA6035-61A2-4146-89C0-50DB591157A0}.1.00.tmp
ERROR CODE: 0x801901f4 - HTTP status 500: An unexpected condition prevented the server from fulfilling the request.
ERROR CONTEXT: 0x00000005 - The error occurred while the remote file was being processed.
We enabled the failed request tracing for IIS on the secondary site, then reproduced the issue by visiting the highlighted URL in red. No surprise, still 500 error. At the same time, failed request tracing logs are generated. Here is what we got –
It points to the getpolicy.dll, which is a file from SCCM itself. Okay, now we know that it is not the IIS that brings the issue but the management point itself.
In MP_Framework.log, we found this information –
MPDB ERROR - CONNECTION PARAMETERS SQL Server Name : secondarysite.contoso.com\CONFIGMGRSEC SQL Database Name : CM_SS1 Integrated Auth : True MPDB ERROR - EXTENDED INFORMATION MPDB Method : ExecuteSP() MPDB Method HRESULT : 0x80004005 Error Description : Login failed for user 'lnsvr_admin'. OLEDB IID : {0C733A63-2A1C-11CE-ADE5-00AA0044773D} ProgID : Microsoft SQL Server Native Client 11.0 MPDB ERROR - INFORMATION FROM DRIVER SQL Server Name : SCCMSiteDBSQL02 Native Error no. : 18456 Error State : 1 Class (Severity) : 14 Line number in SP : 1
But what is this ” lnsvr_admin ” user? We tried to find login failure information from the secondary site database CM_SS1 but got nothing (In fact, ” SQL Server Name : SCCMSiteDBSQL02 ” pointed out already the right target sql server but we neglected that information). What is going on? It dawned on us that there could be a linked server object in this secondary site database which is using ” lnsvr_admin ” as the connection account. We were right –
Right click on the linked server object and select Test Connection. It failed.
On the linked server’s events log, we could find similar messages –
The customer told us that that account’s password has been changed by their sql server administrator but they never did any change to the linked server object. All is clear now – the account ” lnsvr_admin ” has the wrong password in the linked server object and thus failed to connect to the linked server.
Solution
After we updated the password for ” lnsvr_admin ” to the correct one, all the errors in MP_Framework.log went away and the ” proxy MP errors …” also disappeared from LocationServices.log and Software Center finally shows applications and updates.
SCCM Monde 