ISSUE
Hardware Inventory Scan state stops updating after upgrading to 2203. It stays empty or keeps the old date without any change.
LOGS
inventoryAgent.log says the report has been successfully sent. But be vigilant. This can be a misleading message. That information only indicates that InventoryAgent iteslf sends the report to CcmMessageing component, which is actually responsible for sending the report to the management point via Bits.
InventoryAgent 29432 (0x72f8) Inventory: Temp report = C:\Windows\CCM\Inventory\Temp\77aea040-e9a5-4a57-9159-91eae40810b9.xml InventoryAgent 23620 (0x5c44) Inventory: Successfully sent report. Destination:mp:MP_HinvEndpoint, ID: {A781FEBD-D4C2-44B7-98B9-6A84F02ED431}, Timeout: 80640 minutes MsgMode: Signed, Not Encrypted
CcmMessaging.log has interesting messages. It starts sending the report but keeps retrying. Why? Because it fails to send the report.
CcmMessaging 23620 (0x5c44) Sending async message '{A781FEBD-D4C2-44B7-98B9-6A84F02ED431}' to outgoing queue 'mp:mp_hinvendpoint' CcmMessaging 28148 (0x6df4) Upload location request: <UploadRequest NetworkType="Intranet"><MessageID>{A781FEBD-D4C2-44B7-98B9-6A84F02ED431}</MessageID></UploadRequest> CcmMessaging 28148 (0x6df4) Upload location response: <UploadReply><Host>contonsomp.lab.com</Host><URL>http://contonsomp.lab.com/CCM_Incoming/{A781FEBD-D4C2-44B7-98B9-6A84F02ED431}</URL></UploadReply> CcmMessaging 28148 (0x6df4) Uploading 'C:\Windows\CCM\ServiceData\LocalPayload\{A781FEBD-D4C2-44B7-98B9-6A84F02ED431}' to 'http://contonsomp.lab.com/CCM_Incoming/{A781FEBD-D4C2-44B7-98B9-6A84F02ED431}' for message '{A781FEBD-D4C2-44B7-98B9-6A84F02ED431}' CcmMessaging 28148 (0x6df4) UpdateURLWithTransportSettings(): OLD URL - http://contonsomp.lab.com/CCM_Incoming/{A781FEBD-D4C2-44B7-98B9-6A84F02ED431} CcmMessaging 28148 (0x6df4) UpdateURLWithTransportSettings(): NEW URL - http://contonsomp.lab.com:80/CCM_Incoming/{A781FEBD-D4C2-44B7-98B9-6A84F02ED431} CcmMessaging 28148 (0x6df4) OutgoingMessage(Queue='mp_mp_hinvendpoint', ID={A781FEBD-D4C2-44B7-98B9-6A84F02ED431}): Started upload to 'http://contonsomp.lab.com/CCM_Incoming/{A781FEBD-D4C2-44B7-98B9-6A84F02ED431}'. CcmMessaging 28148 (0x6df4) OutgoingMessage(Queue='mp_mp_hinvendpoint', ID={A781FEBD-D4C2-44B7-98B9-6A84F02ED431}): In pending state, will retry in 5 seconds. CcmMessaging 15848 (0x3de8) OutgoingMessage(Queue='mp_mp_hinvendpoint', ID={A781FEBD-D4C2-44B7-98B9-6A84F02ED431}): In pending state, will retry in 30 seconds. CcmMessaging 28148 (0x6df4) OutgoingMessage(Queue='mp_mp_hinvendpoint', ID={A781FEBD-D4C2-44B7-98B9-6A84F02ED431}): In pending state, will retry in 60 seconds. CcmMessaging 28148 (0x6df4) OutgoingMessage(Queue='mp_mp_hinvendpoint', ID={A781FEBD-D4C2-44B7-98B9-6A84F02ED431}): In pending state, will retry in 300 seconds.
Bits event log:
BITS stopped transferring the CCM Message Upload {A781FEBD-D4C2-44B7-98B9-6A84F02ED431} transfer job that is associated with the http://nt-sccm-pss-p1.corp.taifungbank.com:80/CCM_Incoming/{A781FEBD-D4C2-44B7-98B9-6A84F02ED431} URL. The status code is 0x80070003.
0x80070003 means –
Error Code: 0x80070003
Error Name: ERROR_PATH_NOT_FOUND
Error Source: Windows
Error Message: The system cannot find the path specified.
I suspected the file
C:\Windows\CCM\ServiceData\LocalPayload\{A781FEBD-D4C2-44B7-98B9-6A84F02ED431}
could be somehow missing but that was not true. That files was right there in the LocalPayload folder. And I could see other old files piling up in that folder. Uninstalling McAfee from the machine did not help at all.
You can get a similar message with bitsadmin /list /allusers /verbose.
GUID: {b99b7eda-b80e-482d-b73b-765419a9fd2f} DISPLAY: 'CCM Message Upload {A781FEBD-D4C2-44B7-98B9-6A84F02ED431}' TYPE: UPLOAD STATE: ERROR OWNER: NT AUTHORITY\SYSTEM PRIORITY: NORMAL FILES: 0 / 1 BYTES: 0 / 4343 CREATION TIME: 5/26/2022 5:22:56 PM MODIFICATION TIME: 5/30/2022 5:22:56 PM COMPLETION TIME: UNKNOWN ACL FLAGS: NOTIFY INTERFACE: UNREGISTERED NOTIFICATION FLAGS: 3 RETRY DELAY: 600 NO PROGRESS TIMEOUT: 1209600 ERROR COUNT: 1119 PROXY USAGE: NO_PROXY PROXY LIST: NULL PROXY BYPASS LIST: NULL ERROR FILE: http://contonsomp.lab.com:80/CCM_Incoming/{A781FEBD-D4C2-44B7-98B9-6A84F02ED431} -> C:\Windows\CCM\ServiceData\LocalPayload\{A781FEBD-D4C2-44B7-98B9-6A84F02ED431} ERROR CODE: 0x80070003 - The system cannot find the path specified. ERROR CONTEXT: 0x00000005 - The error occurred while the remote file was being processed. DESCRIPTION: JOB FILES: 0 / 4343 WORKING http://contonsomp.lab.com:80/CCM_Incoming/{A781FEBD-D4C2-44B7-98B9-6A84F02ED431} -> C:\Windows\CCM\ServiceData\LocalPayload\{A781FEBD-D4C2-44B7-98B9-6A84F02ED431} NOTIFICATION COMMAND LINE: none owner MIC integrity level: SYSTEM owner elevated ? true This job is read-only to the current CMD window because the job's mandatory integrity level of SYSTEM is higher than the window's level of HIGH. Peercaching flags Enable download from peers :false Enable serving to peers :false CUSTOM HEADERS: NULL
MP_Hinv.log on the management does not contain any information from the target client. So does dataldr.log on the site server.
The IIS log on the management point contonsomp.lab.com has the following:
192.168.108.16 BITS_POST /CCM_Incoming/{A781FEBD-D4C2-44B7-98B9-6A84F02ED431} (bits_error:{8CF09A6A-0865-43C6-B500-BCDFB9C84085},404,0x80070003) 80 - 192.168.13.206 Microsoft+BITS/7.5 - 404 0 0 1496 11
404 and 0x80070003 are eye-catching.
Checking further, it is found that C:\Program Files\Microsoft Configuration Manager\CCM\Incoming folder, which is expected to have a subfolder named BITS-Sessions, is empty. What is weird is that mpcontrol.log shows no errors at all.
Solution
Reinstall managment point role and the issue was gone.