Post

The `aws s3 sync` Flag That Deletes Your Production Data (Without Warning)

Are you using `aws s3 sync` to manage critical data, unaware of a specific flag that can lead to silent, irreversible data loss in production environments?

Posted Updated
Illustration representing The `aws s3 sync` Flag That Deletes Your Production Data (Without Warning)
Illustration representing The `aws s3 sync` Flag That Deletes Your Production Data (Without Warning)
By Kamal K.
7 min read
The `aws s3 sync` Flag That Deletes Your Production Data (Without Warning)

You set up a scheduled S3 sync to back up your production uploads, assuming your data is safe. One day, an upstream pipeline fails, creating an empty source bucket, and the automated sync runs exactly as scheduled. Because the AWS CLI operates over stateless API calls without a “dry-run by default” safety net, your entire production destination bucket is quietly wiped out.

TL;DR: Appending --delete to aws s3 sync forces a strict state match, meaning an empty source bucket will instantly trigger the deletion of all destination objects. This post demonstrates the failure mechanics of this hidden gotcha, provides an idempotent deployment alternative, and outlines how to use native S3 Replication to protect critical data.

What you’ll walk away with:

  • The exact REST API semantics that make CLI synchronization dangerous in automated pipelines.
  • A concrete methodology to audit your CI/CD pipelines for unsafe state reconciliation flags.
  • A robust, non-destructive fallback using bucket versioning and asynchronous replication.

Why Does aws s3 sync --delete Wipe Destination Buckets?

The aws s3 sync command with the --delete flag performs a strict unidirectional state reconciliation. If a file exists in the destination but not in the source, the CLI issues a DeleteObject API call to remove it. Consequently, an empty source bucket guarantees complete destination data erasure.

State reconciliation is the process of computing the difference between a current target environment and a desired source configuration, applying only the necessary changes to make them identical. The AWS CLI version 2.14 executes this process by paginating through both buckets in batches of 1,000 objects. If the source batch is empty, it assumes the desired state is an empty bucket.

You will not receive a prompt, and the command will exit with a success code. The terminal simply outputs a list of deleted objects as they vanish.

1

aws s3 sync s3://ci-build-artifacts s3://prod-release-backups --delete

1
2

delete: s3://prod-release-backups/release-1.1.tar.gz
delete: s3://prod-release-backups/config.json

If you are orchestrating large state backups for Beyond Chatbots: The Economics of Deploying Agent Fleets on AWS Trainium3, native replication guarantees your fleet’s memory isn’t accidentally purged by a misconfigured bash script. Relying on shell scripts for infrastructure replication creates severe operational blind spots.

Never use --delete on a destination bucket that serves as a historical archive, audit trail, or long-term backup.

How Do You Safely Replicate S3 Data Without Risking Deletion?

The safest method is to abandon CLI sync entirely and configure Cross-Region Replication (CRR) or Same-Region Replication (SRR) at the bucket level. S3 Replication handles asynchronous copying within the AWS backbone and explicitly ignores delete markers by default, protecting your destination data.

When you must use the CLI for ad-hoc tasks, you should remove the destructive flag. Below is a common CI/CD script failure mode, followed by the safe, non-destructive approach.

1
2
3
4
5

  # BAD: A failed upstream job results in $SRC_DIR being empty, wiping production.
- aws s3 sync $SRC_DIR s3://prod-data-bucket/ --delete

  # GOOD: Copies new and modified files without removing existing destination assets.
+ aws s3 sync $SRC_DIR s3://prod-data-bucket/

To understand why native replication is safer, look at the architecture. The CLI acts as a middleman, generating massive volumes of HTTP requests from the runner host. Native replication happens entirely within the S3 control plane.

graph TD
  A["Source Bucket (Empty)"] -->|"aws s3 sync --delete"| B["CI/CD Runner"]
  B -->|"DeleteObject API"| C["Dest Bucket (Wiped)"]
  A2["Source Bucket (Empty)"] -->|"S3 Replication Rule"| C2["Dest Bucket (Safe)"]
  C2 -.->|"Ignores delete markers"| C2

When preparing massive datasets for Customizing Intelligence: A First Look at AWS Nova Forge, the same architectural rule applies. Moving heavy object storage through a runner bottleneck is inherently fragile.

View verbose API trace showing silent deletion
1
2
3

2023-10-27 10:00:00 UTC - botocore.endpoint - DEBUG - Making request for OperationModel(name=ListObjectsV2)
2023-10-27 10:00:01 UTC - botocore.endpoint - DEBUG - Making request for OperationModel(name=DeleteObject)
2023-10-27 10:00:01 UTC - botocore.parsers - DEBUG - Response headers: {'x-amz-request-id': '...', 'x-amz-version-id': 'null'}

Notice that the CLI blindly executes DeleteObject API calls the moment ListObjectsV2 returns an empty array for the source prefix.

Always enable S3 Object Versioning on destination buckets to allow immediate rollback if accidental DeleteObject calls are issued.

What Are The Best Alternatives to AWS S3 Sync?

The best alternatives depend on whether you need one-time bulk transfers or continuous replication. S3 Batch Operations is ideal for one-time massive copies, while native S3 Replication is the standard for continuous, hands-off synchronization without the risk of script-driven deletions.

Cleaning up these legacy synchronization commands is exactly the type of modernization discussed in Refactoring with AI: The ‘Rack Drop’. You should replace brittle shell scripts with resilient, managed AWS infrastructure configurations.

Mechanism Risk of Data Loss Best For
aws s3 sync --delete Very High Temporary scratch spaces.
aws s3 sync Low Ad-hoc terminal operations.
S3 Batch Operations Low One-time billion-object copies.
S3 Replication Zero (by default) Continuous production backups.

Configure S3 Replication metrics using Amazon CloudWatch to monitor replication latency instead of relying on cron job exit codes.

How Can You Audit Your Environment For Unsafe Sync Commands?

You must scan your CI/CD pipeline definitions, cron jobs, and infrastructure-as-code scripts for any invocation of aws s3 sync that includes the --delete argument. Replacing these with native AWS replication features or appending explicit --exclude filters drastically reduces your blast radius.

Start your audit by searching your source control repositories. Look for automated scripts running on Jenkins, GitHub Actions, or GitLab CI.

  • Run grep -r "aws s3 sync.*--delete" . across your infrastructure repositories.
  • Verify S3 Object Versioning is enabled (aws s3api get-bucket-versioning --bucket <name>).
  • Migrate cross-account backups to use IAM-secured S3 Replication rules.
  • Enforce --dryrun requirements for all local developer scripts using sync logic.

If you absolutely must use the CLI in a pipeline, wrap the sync command in a pre-flight check. Ensure the source directory actually contains data before calling the AWS API.

1
2
3
4
5
6
7

# Safely verify source data exists before running a destructive sync
if [ "$(ls -A /source/data/ 2>/dev/null)" ]; then
    aws s3 sync /source/data/ s3://prod-bucket/ --delete
else
    echo "Error: Source directory is empty. Aborting sync."
    exit 1
fi

Append --dryrun during script development to log the exact DeleteObject calls AWS would make without actually executing them.

Bottom Line

Using aws s3 sync --delete in automated pipelines turns temporary upstream errors into permanent production data loss. Default to native S3 Replication for ongoing synchronization tasks, as it safely ignores delete markers by design. Audit your CI/CD runners today to remove destructive CLI flags before a failed artifact build wipes your deployment bucket.

FAQ

What exactly does the --delete flag do in AWS S3 sync?

It forces the destination bucket to exactly mirror the source bucket. If an object exists in the destination but is missing from the source, the CLI issues an API call to permanently delete the destination object.

Does S3 versioning protect against aws s3 sync --delete?

Yes. If S3 Object Versioning is enabled on the destination bucket, the CLI’s deletion simply places a delete marker over the object rather than permanently erasing the data. You can restore the object by removing the delete marker.

Can I restore data deleted by aws s3 sync if versioning was turned off?

No. If bucket versioning is suspended or disabled, DeleteObject API calls are permanent and irreversible. The data cannot be recovered unless you have an external backup.

How do I test an S3 sync command without modifying the destination?

Append the --dryrun flag to your command. The CLI will output exactly which objects would be copied, modified, or deleted without making any actual API changes to your bucket.

Does S3 Replication automatically delete files in the destination?

No. By default, S3 Replication does not replicate delete markers from the source to the destination. You have to explicitly configure delete marker replication if you want the destination to mirror source deletions.

Further Reading

🚀 Ready to get hands-on? Spin up an interactive AI or Kubernetes Sandbox at Aicademy Labs for free.

AWS
aws s3 data-loss cli-tools hidden-gotcha
This post is licensed under CC BY 4.0 by the author.