Problem Description
An Amazon S3 restore job completes with a status of "Completed with errors" (Error Code 0). While the job summary indicates successful processing for most objects, specific files are reported as "Missed." In this scenario, the Druva console UI may not immediately display the per-object failure reason, but the restore logs indicate that critical files (e.g., terraform.tfstate) failed to write to the destination bucket despite the backup of those same files having finished successfully.
Cause
The failure is typically caused by an AWS KMS key policy misconfiguration on the destination target.
When restoring data to an S3 bucket that uses default encryption with AWS KMS (SSE-KMS), the IAM role performing the restore requires specific permissions that differ from those required during a backup:
Backup: Generally requires
kms:Decryptto read the encrypted source data.Restore: Requires both
kms:GenerateDataKeyandkms:Decrypt. Thekms:GenerateDataKeypermission is mandatory for thePutObjectoperation to encrypt the new objects being written to the destination bucket.
If the CloudRanger IAM role (e.g., cloudranger-s3-airgap-backup-xxxx) lacks kms:GenerateDataKey in the KMS key policy, AWS returns an AccessDenied (HTTP 403) error and the file copy fails.
Traceback / Logs
The following patterns in the restore logs identify this specific permission gap:
Failed File Examples:
.../IAM_Staging.../terraform.tfstate (FileSize: 21298) .../IAM_UAT.../terraform.tfstate (FileSize: 21156)
Error Metadata:
Service: Amazon S3
Operation: PutObject
Status Code: 403
Error Code: AccessDenied
Details:
kms:GenerateDataKeypermission missingIAM Role:
arn:aws:iam::123456789012:role/cloudranger-s3-airgap-backup-xxxx
Resolution
Step 1: Update the KMS Key Policy
Modify the Key Policy of the KMS key used by the destination S3 bucket to grant the backup IAM role the required permissions. Add the following statement to the policy:
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::123456789012:role/cloudranger-s3-airgap-backup-xxxx"
},
"Action": [
"kms:GenerateDataKey",
"kms:Decrypt"
],
"Resource": "*"
} Step 2: Verify Environmental Constraints
Service Control Policies (SCPs): Ensure no AWS organization-level SCPs are explicitly denying KMS actions for the backup role.
Region Alignment: Confirm that the KMS key and the destination S3 bucket reside in the same AWS region.
IAM Policy: Ensure the IAM role itself also has these permissions allowed if it is not already covered by an administrator policy.
Step 3: Retry the Restore
After updating the KMS key policy, rerun the restore job. The previously missed files should now process successfully.
Reference: IAM Roles and Permissions for Amazon S3