AWS Storage Blog
Category: Developer Tools
Scalable cross-cloud data migration to Amazon S3 with distributed rclone
Migrating petabytes of data across cloud providers is one of the most operationally demanding tasks an organization can take on. At this scale, simple transfer approaches break down. Teams lose track of what has been copied and what has failed. Transfers stall and require constant manual intervention to restart. In some cases, teams need to […]
Implement single-exchange tokens for short-lived Amazon S3 presigned URLs with Terraform
Organizations across industries use signed URLs to grant temporary, credential-less access to private resources such as receipts, medical or financial records, legal files, or confidential reports. However, signed URLs can be reused by anyone until they expire, creating security risks if a URL is shared or inadvertently disclosed. This risk can be mitigated by vending […]
Improve resilience of low latency applications with Amazon EBS and AWS FIS
Organizations running mission-critical applications and databases face challenges to ensure their systems can withstand unexpected storage performance degradation. Storage latency issues can emerge unpredictably in production environments and impact the business. Although unit, integration, and regression testing ensure that systems work under normal conditions, they don’t cover unexpected failures that result in expensive outages. This […]
Test and build application resilience using Amazon EBS latency injection
As businesses strive to build highly available applications, they must prevent disruptions that can lead to downtime and revenue loss. Robust monitoring systems help identify failures proactively, but chaos engineering has emerged as a systematic approach to building resilient systems by uncovering potential issues before they become outages. Chaos engineering is especially critical for storage […]
Accelerating Amazon S3 Batch Operations at scale with on-demand manifest generation
Modern enterprises routinely manage billions of objects across their cloud storage environments, needing efficient bulk operations for disaster recovery, compliance management, data transfer, and cost optimization. Performing these operations manually or through custom scripts becomes impractical at scale, often creating operational bottlenecks when time-sensitive actions are necessary. Organizations frequently need to identify and process specific […]
Detect malware threats using AWS Transfer Family
Securely sharing files over SFTP, FTP, and FTPS is a staple within many business-to-business (B2B) workflows. Across industries, companies use file transfer to transmit inventory, invoice, and compliance information. It is critical for companies to make sure that shared files do not have any malicious content that could compromise their systems. Guaranteeing the shared files […]
Encrypt and decrypt files with PGP and AWS Transfer Family
1/11/2024: Updates made due to CloudShell migration to Amazon Linux 2023 (AL2023). Protecting sensitive data is not a novel idea. Customers in industries like financial services and healthcare regularly exchange files containing sensitive data, including Personal Identifiable Information (PII) and financial records with their users. Pretty Good Privacy (PGP) encryption of these files is often […]
Using presigned URLs to identify per-requester usage of Amazon S3
Many software-as-a-service (SaaS) product offerings have a pay-as-you-go pricing model, charging customers only for the resources consumed. However, a pay-as-you-go pricing is only viable when you can accurately track each customer’s use of resources, such as compute capacity, storage, and networking bandwidth. Without this data, SaaS providers do not have visibility into resource consumption of […]
Use AWS Backup and CI/CD tools to automate centralized backup across AWS services
Automating and scaling your data protection and backup strategy helps you reduce manual overhead from time-consuming configuration, minimizes the risk for errors, provides visibility on drift detection, and enhances backup policy compliance across distributed AWS workloads or accounts. Incorporating backup in your disaster recovery (DR) and business continuity plan (BCP), along with the automation of […]
Allowing external users to securely and directly upload files to Amazon S3
Organizations are often required to store files, images, and other digital assets in a repository. In many cases, the source of these files are partners or individuals who are not connected to internal systems and requires corporate authentication in order to upload the files. Customers traditionally use servers to handle file uploads, which can use […]






