Clients show offline in SCCM console

Clients show offline in SCCM console

ISSUE

All the clients show offline with an x icon in SCCM console

Observation

CcmNotificationAgent.log

10-01-2021 12:01:27.772 BgbAgent 1820 (0x71c) Connecting to server with IP: 10.2.253.4 Port: 10123
10-01-2021 12:01:29.848 BgbAgent 1820 (0x71c) Failed to connect to server with IP v4 address with error 10061. Try next IP…
10-01-2021 12:01:29.848 BgbAgent 1820 (0x71c) Failed to signin bgb client with error = 80004005.
10-01-2021 12:02:29.856 BgbAgent 1820 (0x71c) Connecting to server with IP: 10.2.253.4 Port: 10123
10-01-2021 12:02:31.993 BgbAgent 1820 (0x71c) Failed to connect to server with IP v4 address with error 10061. Try next IP…
10-01-2021 12:02:31.993 BgbAgent 1820 (0x71c) Failed to signin bgb client with error = 80004005.
10-01-2021 16:59:16.575 BgbAgent 21004 (0x520c) Fallback to HTTP connection.
10-01-2021 16:59:16.851 BgbAgent 21004 (0x520c) [CCMHTTP] ERROR: URL=https://contoso.lab/bgb/handler.ashx?RequestType=LogIn, Port=443, Options=31, Code=0, Text=CCM_E_BAD_HTTP_STATUS_CODE
10-01-2021 16:59:16.851 BgbAgent 21004 (0x520c) [CCMHTTP] ERROR INFO: StatusCode=500 StatusText=Internal Server Error
10-01-2021 16:59:16.862 BgbAgent 21004 (0x520c) Raising event:
instance of CCM_CcmHttp_Status
{
ClientID = "GUID:C044DC1C-1935-4B0B-BEF9-AEB76A6C6989";
DateTime = "20211001085916.862000+000";
HostName = "contoso.lab";
HRESULT = "0x87d0027e";
ProcessID = 108;
StatusCode = 500;
ThreadID = 21004;
};
10-01-2021 16:59:16.866 BgbAgent 21004 (0x520c) Successfully queued event on HTTP/HTTPS failure for server 'contoso.lab'.
10-01-2021 16:59:16.867 BgbAgent 21004 (0x520c) Failed to post Login with error code 87d0027e.
10-01-2021 16:59:16.867 BgbAgent 21004 (0x520c) Failed to signin bgb client with error = 87d0027e.

Status code 500 indicates something goes wrong with the server itself. Checking if management point is in good status. The MP is installed correctly. Then I checked BGB status.

bgbsetup

10-04-2021 15:37:41.000 SMSBGB Setup Started….
10-04-2021 15:37:41.000 Parameters: E:\Program Files\Microsoft Configuration Manager\bin\x64\rolesetup.exe /deinstall /siteserver:SIN5VSCMPS001 SMSBGB 0
10-04-2021 15:37:41.000 Deinstalling the SMSBGB
10-04-2021 15:37:41.000 CTool::RegisterComPlusService: run command line: "C:\Windows\Microsoft.NET\Framework64\v4.0.30319\RegSvcs.exe" /u "E:\Program Files\Microsoft Configuration Manager\bin\x64\microsoft.configurationmanager.bgbserverchannel.dll"
10-04-2021 15:37:44.000 CTool::RegisterComPlusService: Failed to unregister E:\Program Files\Microsoft Configuration Manager\bin\x64\microsoft.configurationmanager.bgbserverchannel.dll with .Net Fx 4.0
10-04-2021 15:37:44.000 Failed to unregister BGB server channel DLL E:\Program Files\Microsoft Configuration Manager\bin\x64\microsoft.configurationmanager.bgbserverchannel.dll. Error = 0x80004005.
10-04-2021 15:37:44.000 CTool::RegisterComPlusService: run command line: "C:\Windows\Microsoft.NET\Framework64\v4.0.30319\RegSvcs.exe" /u "E:\Program Files\Microsoft Configuration Manager\bin\x64\BGBServer\microsoft.configurationmanager.bgbserverchannel.dll"
10-04-2021 15:37:45.000 CTool::RegisterComPlusService: Failed to unregister E:\Program Files\Microsoft Configuration Manager\bin\x64\BGBServer\microsoft.configurationmanager.bgbserverchannel.dll with .Net Fx 4.0
10-04-2021 15:37:45.000 Failed to unregister BGB server channel DLL E:\Program Files\Microsoft Configuration Manager\bin\x64\BGBServer\microsoft.configurationmanager.bgbserverchannel.dll. Error = 0x80004005.
10-04-2021 15:37:45.000 DeleteBgbServerApplication: failed to find the application
10-04-2021 15:38:03.000 Fatal MSI Error - bgbisapi.msi could not be installed.

bgbisapiMSI.log – it says there is not enough disk space while there actually sufficient disk space.

05-10-2021 10:29:46.600 MSI (s) 76 (0x4c) Doing action: CcmCheckFreeDiskSpace
05-10-2021 10:29:46.600 Action ended InstallValidate. Return value 1.
05-10-2021 10:29:46.600 MSI (s) 76 (0x4c) Note: 1: 2235 2: 3: ExtendedType 4: SELECT Action,Type,Source,Target, NULL, ExtendedType FROM CustomAction WHERE Action = 'CcmCheckFreeDiskSpace'
05-10-2021 10:29:46.600 Action start CcmCheckFreeDiskSpace.
05-10-2021 10:29:46.601 MSI (s) 76 (0x4c) Product: BGB http proxy -- There is not enough available disk space on to complete this operation. Installation requires at least 10MB free disk space.
05-10-2021 10:29:46.601
05-10-2021 10:29:46.601 There is not enough available disk space on to complete this operation. Installation requires at least 10MB free disk space.
05-10-2021 10:29:46.601 Action ended CcmCheckFreeDiskSpace. Return value 3.
05-10-2021 10:29:46.601 Action ended INSTALL. Return value 3.
05-10-2021 10:29:46.601 Property(S): UpgradeCode = {57F5D44D-5328-44DB-8DA2-EB252C1F810D}

Solution

1. Go to this registry key on the MP server: HKEY_CLASSES_ROOT\Installer\Products\

2. Find the subkey entries for the item within this key.

   A. The display name will be found under the ProductCode String Value called ProductName.

Example:  HKEY_CLASSES_ROOT\Installer\Products\86F1F994E7D7D6C4DA16688376A94C6B\

Reg String Value ProductName = ConfigMgr Management Point

  B. The actual MSI will be found in a Subkey of the ProductCode called SourceList under PackageName. For example: HKEY_CLASSES_ROOT\Installer\Products\57F5D44D-5328-44DB-8DA2-EB252C1F810D\SourceList

Reg String Value PackageName = BGBAPI.msi

3. Identify these keys, Export and backup the ProductCode subkey, eg.

HKEY_CLASSES_ROOT\Installer\Products\57F5D44D-5328-44DB-8DA2-EB252C1F810D

4. Delete the registry key  HKEY_CLASSES_ROOT\Installer\Products\57F5D44D-5328-44DB-8DA2-EB252C1F810D

6. Restart Site Component Manager service.

Management point role installation fails on the primary site with “Child process exited with non-zero code 102”

Management point role installation fails on the primary site with “Child process exited with non-zero code 102”

Issue

Management point role installation fails on the primary site with “Child process exited with non-zero code 102”

09-27-2021 11:02:01.440    SMS_SITE_COMPONENT_MANAGER    23296 (0x5b00)            Starting service SMS_SERVER_BOOTSTRAP_WXSCCM with command-line arguments "WX1 C:\Program Files\Microsoft Configuration Manager /install C:\Program Files\Microsoft Configuration Manager\bin\x64\rolesetup.exe SMSMP "...
09-27-2021 11:02:08.489    SMS_SITE_COMPONENT_MANAGER    23296 (0x5b00)              Execution of "C:\Program Files\Microsoft Configuration Manager\bin\x64\rolesetup.exe /install /siteserver:WXSCCM.HYNIX-CN.COM" on server WXSCCM.HYNIX-CN.COM failed: Child process exited with non-zero code 102.

Analysis

Reviewing the mpMSI.log, I found the following:

09-27-2021 11:02:06.791    Action start    CcmValidateCustomWebSite.
09-27-2021 11:02:06.791    [11:02:06] Found 3 web site(s).
09-27-2021 11:02:06.791    [11:02:06] WARNING: Could not find site with name 'SMSWeb'
09-27-2021 11:02:06.791    [11:02:06] @@ERR:25001
09-27-2021 11:02:06.864    MSI (s)    192 (0xc0)    Product: ConfigMgr Management Point -- Error 25001. Setup failed due to unexpected circumstances
09-27-2021 11:02:06.864    The error code is 87D00215
09-27-2021 11:02:06.864    
09-27-2021 11:02:06.864    Error 25001. Setup failed due to unexpected circumstances
09-27-2021 11:02:06.864    The error code is 87D00215
09-27-2021 11:02:06.864    CustomAction CcmValidateCustomWebSite returned actual error code 1603 (note this may not be 100% accurate if translation happened inside sandbox)

87D00215 refers to –

Error Code: 0x87D00215 (2278556181)
Error Name: CCM_E_ITEMNOTFOUND
Error Source: Configuration Manager
Error Message: Item not found

Solution

The customer does not have any custom site. So, the solution is:

  1. Go to \Administration\Overview\Site Configuration\Sites
  2. Select the target site, then Properties
  3. Go to the tab Ports
  4. Uncheck Use custom web site
  5. The roles will get reinstalled accordingly

References

Software Center shows nothing, no Applications, no updates

Issue

Software Center is empty. There are no applications nor updates. We tested by deploying one application and one update but in vain.

Observation

CcmMessaging.log has no errors, all the messages being sent successfully to the management point associated with a secondary site.

05-24-2021 23:13:46.793    CcmMessaging    19792 (0x4d50)    Sending async message '{0D37A652-BCC8-4169-8A1A-6E1574E2A070}' to outgoing queue 'mp:[http]mp_locationmanager'
05-24-2021 23:13:48.215    CcmMessaging    15764 (0x3d94)    Sending outgoing message '{0D37A652-BCC8-4169-8A1A-6E1574E2A070}'. Flags 0x201, sender account empty
05-24-2021 23:13:48.516    CcmMessaging    15764 (0x3d94)    Message '{0D37A652-BCC8-4169-8A1A-6E1574E2A070}' got reply '{DA626C8B-433E-42C4-8360-965EF2D60683}' to local endpoint queue 'LS_ReplyLocations'
05-24-2021 23:13:48.518    CcmMessaging    15764 (0x3d94)    OutgoingMessage(Queue='mp_[http]mp_locationmanager', ID={0D37A652-BCC8-4169-8A1A-6E1574E2A070}): Delivered successfully to host 'secondarysite.contoso.com'.

Looking at LocationServices.log, we found the following messages –

05-24-2021 22:59:00.664    LocationServices    5924 (0x1724)    1 proxy MP errors in the last 10 minutes, threshold is 5.
05-24-2021 22:59:49.974    LocationServices    13528 (0x34d8)    2 proxy MP errors in the last 10 minutes, threshold is 5.
05-24-2021 23:00:01.747    LocationServices    9620 (0x2594)    3 proxy MP errors in the last 10 minutes, threshold is 5.
05-24-2021 23:00:31.063    LocationServices    13528 (0x34d8)    4 proxy MP errors in the last 10 minutes, threshold is 5.
05-24-2021 23:00:46.937    LocationServices    15764 (0x3d94)    Proxy MP error threshold reached, moving to next MP.

DataTransferService.log –

05-24-2021 23:04:44.994 DataTransferService 15764 (0x3d94) BITS compatible path: http://secondarysite.contoso.com:80/SMS_MP/.sms_pol?%7BF4CA6035-61A2-4146-89C0-50DB591157A0%7D.SHA256:36A294B138C353F4487562349357BE61D23B9C631D8707BC79B712A5D836DC0F
05-24-2021 23:04:44.994 DataTransferService 15764 (0x3d94) BITSHelper: Full source path to be transferred = http://secondarysite.contoso.com:80/SMS_MP/.sms_pol?%7BF4CA6035-61A2-4146-89C0-50DB591157A0%7D.SHA256:36A294B138C353F4487562349357BE61D23B9C631D8707BC79B712A5D836DC0F
05-24-2021 23:04:44.994 DataTransferService 15764 (0x3d94) BITSHelper, remote name = http://secondarysite.contoso.com:80/SMS_MP/.sms_pol?%7BF4CA6035-61A2-4146-89C0-50DB591157A0%7D.SHA256:36A294B138C353F4487562349357BE61D23B9C631D8707BC79B712A5D836DC0F, local name = C:\WINDOWS\CCM\Staging{F4CA6035-61A2-4146-89C0-50DB591157A0}.1.00.tmp

Visiting the highlighted link in the browser on the client computer returns 500 Server Internal Error.

Bits job, retrieved by running “bitsadmin /list /allusers /verbose“, has the following information. Obviously, something goes wrong with the management point on the secondary site. More likely relevant to IIS.

UID: {A5A891C5-4EB7-4EB2-BE44-19BB40F66CCA} DISPLAY: 'CCMDTS Job'
TYPE: DOWNLOAD STATE: TRANSIENT_ERROR OWNER: NT AUTHORITY\SYSTEM
PRIORITY: HIGH FILES: 0 / 2 BYTES: 0 / UNKNOWN
CREATION TIME: 5/24/2021 11:04:44 PM MODIFICATION TIME: 5/24/2021 11:17:58 PM
COMPLETION TIME: UNKNOWN ACL FLAGS:
NOTIFY INTERFACE: REGISTERED NOTIFICATION FLAGS: 11
RETRY DELAY: 60 NO PROGRESS TIMEOUT: 28800 ERROR COUNT: 14
PROXY USAGE: NO_PROXY PROXY LIST: NULL PROXY BYPASS LIST: NULL
ERROR FILE: http://secondarysite.contoso.com:80/SMS_MP/.sms_pol?%7BF4CA6035-61A2-4146-89C0-50DB591157A0%7D.SHA256:36A294B138C353F4487562349357BE61D23B9C631D8707BC79B712A5D836DC0F -> C:\WINDOWS\CCM\Staging{F4CA6035-61A2-4146-89C0-50DB591157A0}.1.00.tmp
ERROR CODE: 0x801901f4 - HTTP status 500: An unexpected condition prevented the server from fulfilling the request.

ERROR CONTEXT: 0x00000005 - The error occurred while the remote file was being processed.

We enabled the failed request tracing for IIS on the secondary site, then reproduced the issue by visiting the highlighted URL in red. No surprise, still 500 error. At the same time, failed request tracing logs are generated. Here is what we got –

It points to the getpolicy.dll, which is a file from SCCM itself. Okay, now we know that it is not the IIS that brings the issue but the management point itself.

In MP_Framework.log, we found this information –

MPDB ERROR - CONNECTION PARAMETERS
SQL Server Name : secondarysite.contoso.com\CONFIGMGRSEC
SQL Database Name : CM_SS1
Integrated Auth : True

MPDB ERROR - EXTENDED INFORMATION
MPDB Method : ExecuteSP()
MPDB Method HRESULT : 0x80004005
Error Description : Login failed for user 'lnsvr_admin'.
OLEDB IID : {0C733A63-2A1C-11CE-ADE5-00AA0044773D}
ProgID : Microsoft SQL Server Native Client 11.0

MPDB ERROR - INFORMATION FROM DRIVER
SQL Server Name : SCCMSiteDBSQL02
Native Error no. : 18456
Error State : 1
Class (Severity) : 14
Line number in SP : 1

But what is this ” lnsvr_admin ” user? We tried to find login failure information from the secondary site database CM_SS1 but got nothing (In fact, ” SQL Server Name : SCCMSiteDBSQL02 ” pointed out already the right target sql server but we neglected that information). What is going on? It dawned on us that there could be a linked server object in this secondary site database which is using ” lnsvr_admin ” as the connection account. We were right –

Right click on the linked server object and select Test Connection. It failed.

On the linked server’s events log, we could find similar messages –

The customer told us that that account’s password has been changed by their sql server administrator but they never did any change to the linked server object. All is clear now – the account ” lnsvr_admin ” has the wrong password in the linked server object and thus failed to connect to the linked server.

Solution

After we updated the password for ” lnsvr_admin ” to the correct one, all the errors in MP_Framework.log went away and the ” proxy MP errors …” also disappeared from LocationServices.log and Software Center finally shows applications and updates.

References

Management Point fails to install with “ERROR: Cannot use SMS issued certificate for SSL role.”

Management Point fails to install with “ERROR: Cannot use SMS issued certificate for SSL role.”

One customer called me reporting their management point kept failing to get installed. The mpMSI log throws errors about SMS issued certificate.

mpMSI.log:

[7:49:13] Verification of Certificate chain returned 00000000
[7:49:13] Completed validation of Certificate [Thumbprint 136F81A0B31D713E4089E7973F5A46FD74F086EB] issued to ‘azProdSCCM1.dlgrd.wa.gov.au’
[7:49:13] ERROR: Cannot use SMS issued certificate for SSL role.
[7:49:13] ERROR: Validate IIS failed with 0x80004005.
[7:49:13] @@ERR:25055
MSI (s) (74!54) [07:49:13:599]: Product: ConfigMgr Management Point — Error 25055. Internet Information Services Default Web Site is not correctly configured for SSL
Error 25055. Internet Information Services Default Web Site is not correctly configured for SSL

CustomAction CcmValidateServerConfig returned actual error code 1603 (note this may not be 100% accurate if translation happened inside sandbox)
Action ended 7:49:13: CcmValidateServerConfig. Return value 3.
Action ended 7:49:13: INSTALL. Return value 3.

SMS_MP_CONTROL MANAGER component status keeps logging the following information:

Severity        Type        Site code        Date / Time        System        Component        Message ID        Description
Error        Detail        DSA        11/11/2020 4:27:48 PM        AZPRODSCCM1.DLGRD.WA.GOV.AU        SMS_MP_CONTROL_MANAGER        4964        Site Component Manager failed to install this component, because Secure Sockets Layer (SSL) is not configured properly on the Internet Information Server.
Possible cause: No Server Certificate is attached to the designated Web Site. 
Solution: Refer to ConfigMgr Documentation regarding how to create and attach a proper Server Certificate to the designated Web Site.    
Possible cause: The Server Certificate is invalid or expired. 
Solution: Refer to ConfigMgr Documentation regarding how to create and attach a proper Server Certificate to the designated Web Site.

Looking at its Communication Security in Site’s Properties, I found that they configured E-HTTP for site systems that use IIS. HTTPS or HTTP checked with Use Configuration Manager generated certificate for HTTP site systems ticked.

With E-HTTP, there are some prerequisites to comply with:

  • A management point configured for HTTP client connections. Set this option on the General tab of the management point role properties.
  • A distribution point configured for HTTP client connections. Set this option on the Communication tab of the distribution point role properties. Don’t enable the option to Allow clients to connect anonymously.
  • Onboard the site to Azure AD for cloud management.
  • For Scenario 3 only: A client running Windows 10 version 1803 or later, and joined to Azure AD. The client requires this configuration for Azure AD device authentication.

For details, refer to – https://docs.microsoft.com/en-us/mem/configmgr/core/plan-design/hierarchy/enhanced-http#prerequisites

Then I went to check its Management Point settings in Servers and Site System Roles only to find that HTTPS option was checked.

So, the solution is quite simple:

Check HTTP instead of HTTPS

After that, the management point was installed successfully.

References

https://docs.microsoft.com/en-us/mem/configmgr/core/plan-design/hierarchy/enhanced-http

Design a site like this with WordPress.com
Get started