Amazon S3 storage classes&lifecycle configuration

 Each object in Amazon S3 has a storage class associated with it. 

Storage classes for frequently accessed objects

• S3 Standard

• S3 Express One Zone

S3 Express One Zone is the first S3 storage class where you can select a single Availability Zone with the option to co-locate your object storage with your compute resources, which provides the highest possible access speed. Additionally, to further increase access speed and support hundreds of thousands of requests per second, data in S3 Express One Zone storage class is stored in a new bucket type: an Amazon S3 directory bucket. Each directory bucket can support hundreds of thousands of transactions per second (TPS), irrespective of key names or access pattern.

Storage class for automatically optimizing data with changing or unknown access patterns

• S3 Intelligent-Tiering

S3 Intelligent-Tiering automatically stores objects in three access tiers:

• Frequent Access – Objects that are uploaded or transitioned to S3 Intelligent-Tiering are automatically stored in the Frequent Access tier. 

• Infrequent Access – S3 Intelligent-Tiering moves objects that have not been accessed in 30 consecutive days to the Infrequent Access tier. 

• Archive Instant Access – With S3 Intelligent-Tiering, any existing objects that have not been accessed for 90 consecutive days are automatically moved to the Archive Instant Access tier. 

In addition to these three tiers, S3 Intelligent-Tiering offers two optional archive access tiers: 

• Archive Access – S3 Intelligent-Tiering provides you with the option to activate the Archive Access tier for data that can be accessed asynchronously. After activation, the Archive Access tier automatically archives objects that have not been accessed for a minimum of 90 consecutive days. 

• Deep Archive Access – S3 Intelligent-Tiering provides you with the option to activate the Deep Archive Access tier for data that can be accessed asynchronously. After activation, the Deep Archive Access tier automatically archives objects that have not been accessed for a minimum of 180 consecutive days.

We recommend the following: 

• S3 Standard-IA – Use for your primary or only copy of data that can't be re-created. 

• S3 One Zone-IA – Use if you can re-create the data if the Availability Zone fails, and for object replicas when configuring S3 Cross-Region Replication (CRR).

Amazon S3 offers multiple S3 Glacier storage classes that are designed to provide cost-effective solutions to storing long-term data that isn't accessed often. The S3 Glacier storage classes are:

  • S3 Glacier Instant Retrieval

  • S3 Glacier Flexible Retrieval

  • S3 Glacier Deep Archive

Creating a lifecycle configuration 

An S3 Lifecycle configuration is an XML file that consists of a set of rules with predefined actions that you want Amazon S3 to perform on objects during their lifetime.

Unsupported lifecycle transitions 

Amazon S3 does not support any of the following lifecycle transitions. You can't transition from the following: 

• Any storage class to the S3 Standard storage class

存储时间指创建多少天后才可以？
STANDARD-IA：

# aws s3api get-bucket-lifecycle-configuration --bucket bucket-for-logging-1
{
    "Rules": [
        {
            "ID": "e38ff268-c38b-41f2-b569-525cbbaef132",
            "Filter": {
                "Prefix": "jgp/"
            },
            "Status": "Enabled",
            "Transitions": [
                {
                    "Days": 30,
                    "StorageClass": "STANDARD_IA"
                }
            ]
        }
    ]
}
假设您的桶未启用版本控制，且希望在创建30天后将对象名前缀为“documents/”的对象转为低频访问存储对象，创建60天后转为归档存储对象，创建1年之后永久删除。为此，您可以向桶附加以下生命周期配置。该生命周期配置定义的规则包含两个操作（Transition和Expiration），该规则应用于键前缀为“documents”（在Prefix元素中指定）的对象。
<LifecycleConfiguration> 
    <Rule> 
        <ID>sample-rule</ID> 
        <Prefix>documents/</Prefix> 
        <Status>Enabled</Status> 
        <Expiration> 
             <Days>365</Days> 
        </Expiration> 
        <Transition>  
             <Days>30</Days> 
             <StorageClass>WARM</StorageClass>
        </Transition>  
        <Transition>  
             <Days>60</Days>  
             <StorageClass>COLD</StorageClass>  
        </Transition> 
    </Rule> 
</LifecycleConfiguration>
天数指创建多少天后，不是指要在该类型保持的天数

Q： What does "Days' in Transition action must be greater than or equal to 30 for storageClass 'STANDARD_IA' " mean?

A： The message you're encountering is related to Amazon S3 lifecycle rules and storage classes. It indicates a specific requirement for transitioning objects to the STANDARD_IA storage class.

Here's a breakdown of the message:

  • Days' in Transition action: This refers to the number of days that an object must remain in its current storage class before it can be transitioned to the STANDARD_IA class.
  • must be greater than or equal to 30: This specifies the minimum number of days required. In this case, objects need to be in a different storage class for at least 30 days before being transitioned to STANDARD_IA.
  • storageClass 'STANDARD_IA': This refers to the specific Amazon S3 storage class where this rule applies. STANDARD_IA is a lower-cost storage class designed for infrequently accessed data.

In simpler terms, Amazon S3 enforces a 30-day waiting period before you can move objects to the STANDARD_IA storage class. This ensures that you don't accidentally transition objects you might still need to access quickly.

Here are some additional points to consider:

  • This rule only applies when transitioning objects to STANDARD_IA storage class. There are no minimum waiting periods for transitioning to other S3 storage classes like Glacier.
  • You can set lifecycle rules to automatically manage object transitions based on criteria like time since creation or last access.

If you're encountering this error message, it means your S3 lifecycle rule is trying to transition objects to STANDARD_IA before they meet the 30-day minimum requirement. You'll need to adjust your rule to either increase the 'Days' in Transition value to 30 or more, or transition the objects to a different storage class that doesn't have this restriction.





评论

发表评论

此博客中的热门博文

码率单位

Feature Bandwidth Traffic Throughput Definition Maximum data transfer rate Actual data being transmitted Actual data transfer rate achieved Measure Capacity (bps) Data volume (bytes or packets) Transfer rate (bps) Static vs. Dynamic Static (connection property) Dynamic (varies over time) Dynamic (affected by network conditions) Impact on Performance Determines potential data transfer speed Affects network congestion and utilization Measures actual network performance 表格一览： 名称 符号 b/s B/s 示例 比特每秒 b/s 1 1/8 字节每秒 B/s 8 1 千比特每秒 kb/s 1000 = 10^3 125 = 10^3/8 二进制千比特每秒 Kib/s 1024 = 2^10 128 = 2^10/8 千字节每秒 kB/s 8000 = 8*10^3 1000 = 10^3 二进制千字节每秒 KiB/s 8192 = 8*2^10 1024 = 2^10 k 与 Ki k 与 Ki 分别表示 kilo-（千） 与 kibi-（二进制千） 。作为前缀使用时， k 表示 1,000，Ki 表示1,024 ，因为“Ki”来源于它在计算机方面 2^10 = 1,024 的使用。不幸的是，人们常常错误地用 K 代替 Ki。此外，不知晓其中微妙差别的广大公众，常常不加区别地使用“Kbps”与“Kibps”，造成了混乱。总之，使用“Kibps”通常是正确的。 The bit rate is expressed in the unit bit per second (symbol: bit/s), often in conjunction with an SI prefix such as kilo (1...
阅读全文

Process vs. Thread

图片
  A process is an executing instance of a program with its own memory and resources, while  a thread is the smallest unit of execution within a process that can run concurrently with other threads in the same process .  Processes are isolated with separate address spaces and require system-provided inter-process communication to share data, whereas threads share the process's memory and resources, allowing for faster communication and context switching.   Process vs. Thread Process: An independent program in execution.  Has its own separate memory address space.  Carries more state information and resources (like open files and system handles).  Data sharing requires Inter-Process Communication (IPC) mechanisms.  Considered a "heavyweight" unit of execution due to resource overhead for creation and context switching.  Thread: A subset of a process , executing within the process's context.  Shares the process's address space, memory, and ...
阅读全文

日志文件系统

图片
Journaling filesystem 基本原理 更新文件系统以反映文件和目录的更改通常需要许多单独的写入操作。这使得写入之间可能发生中断（例如电源故障或系统崩溃），从而使数据结构处于无效的中间状态。[1] 例如，删除 Unix 文件系统上的文件涉及三个步骤：[7] 删除其目录条目。（Removing its directory entry.） 将索引节点释放到空闲索引节点池中。（Releasing the inode to the pool of free inodes.） 将所有磁盘块返回到空闲磁盘块池。（Returning all disk blocks to the pool of free disk blocks.） 如果在步骤1之后和步骤2之前发生崩溃，就会出现孤立的inode，从而导致存储泄漏；如果在步骤2和步骤3之间发生崩溃，则该文件先前使用的块将无法用于新文件，从而有效地减少了文件系统的存储容量。重新安排步骤也无济于事。如果步骤 3 在步骤 1 之前，则它们之间的崩溃可能允许文件的块重新用于新文件，这意味着部分删除的文件将包含另一个文件的部分内容，并且对任一文件的修改都会显示在两个文件中。另一方面，如果步骤 2 在步骤 1 之前，则它们之间的崩溃将导致文件无法访问，尽管该文件看起来存在。 检测此类不一致并从中恢复通常需要完整地检查其数据结构，例如通过 fsck （文件系统检查器）等工具。[2] 这通常必须在下次安装文件系统以进行读写访问之前完成。如果文件系统很大并且 I/O 带宽相对较小，则这可能会花费很长时间，并且如果它阻止系统的其余部分恢复联机，则会导致更长的停机时间。 为了防止这种情况，日志文件系统分配一个特殊区域——日志——在其中记录它将提前进行的更改。崩溃后，恢复只需从文件系统读取日志并重播该日志中的更改，直到文件系统再次保持一致。因此，这些更改被认为是原子的（不可整除），因为它们要么成功（最初成功或在恢复期间完全重播），要么根本不重播（被跳过，因为它们在之前尚未完全写入日志）发生崩溃）。  JFS(1990)->NTFS(1993)->XFS(1994)->Apple HFS Plus(1998)->ext3(2001)->ext4(2008) ext和ext2文件系统不是日志文件系统 日誌式檔...
阅读全文