Problem description
The CloudCache (CC) upload job consistently reports a PHOENIX410 error in the management console. Despite this error notification, the actual data upload completes successfully, and there is no data left pending in the synchronization queue. The error is characterized by the message "Failed to end session" appearing at the conclusion of the upload task.
Cause
The root cause is a time drift between the CloudCache device system clock and the NTP (Network Time Protocol) server.
When the system clock on the CloudCache appliance is not synchronized with the backend services, the security tokens used to authenticate the session become invalid or are perceived as expired by the server. Even if the data transfer is finished, the command to formally "end the session" fails because the token is no longer trusted due to the time mismatch.
Traceback
The following entries in the Controlservice.log indicate the synchronization mismatch and resulting token failure:
Plaintext
level=debug ts=2025-08-22T01:00:00.734751006Z filename=sessionhandler.go:324 jobType=flush ... msg="Failed to end session" err="Token expired: Token expired: Token expired" level=debug ts=2025-08-22T01:05:00.796368654Z filename=sessionhandler.go:333 ... msg="Session ended" ... error="errortype.ServiceError{Err:Token expired: Token expired: Token expired, CodePrefix:\"DaintreeControl\", Code:1022, Escape:false, Retry:true, Data:map[string]interface {}(nil)}"
Resolution
To resolve this issue, you must synchronize the CloudCache device's system time with the NTP server.
Step 1: Check current synchronization status Run the following command on the CloudCache device: timedatectl status
Example output showing a desynchronized clock:
CloudCache0001:~$ timedatectl status
Local time: Thu 2025-10-09 03:35:06 UTC
Universal time: Thu 2025-10-09 03:35:06 UTC
RTC time: Thu 2025-10-09 03:35:04
Time zone: Etc/UTC (UTC, +0000)
System clock synchronized: no
NTP service: active
RTC in local TZ: no
jhansen@HAM-VS-CloudCache0001:~$
[!IMPORTANT] Note that System clock synchronized: no indicates the drift causing the PHOENIX410 error.
Step 2: Enable System Clock Synchronization Force the system to synchronize with the NTP server by running: sudo timedatectl set-ntp true
Step 3: Monitor next upload cycle Allow the next scheduled upload job to run. The job should now transition to a "Successful" status without the PHOENIX410 error.
Verification
To verify the fix, check the timedatectl status again to ensure the clock is locked:
Bash
jhansen@HAM-VS-CloudCache0001:~$ timedatectl status
Local time: Thu 2025-10-09 05:31:47 UTC
Universal time: Thu 2025-10-09 05:31:47 UTC
RTC time: Thu 2025-10-09 05:31:47
Time zone: Etc/UTC (UTC, +0000)
System clock synchronized: yes
NTP service: active
RTC in local TZ: no
jhansen@HAM-VS-CloudCache0001:~$
Confirmation is successful when System clock synchronized displays yes.