DynamoDB Stream Records and Stream Operations

A DynamoDB stream contains information about both the previous value and the changed value of the item (as applicable). The stream also includes the change type (INSERT, REMOVE, and MODIFY) and the primary key for the item that changed.

Each stream record represents a single data modification in the DynamoDB table to which the stream belongs. Each stream record is assigned a sequence number, reflecting the order in which the record was published to the stream.

For new tables, use the CreateTable API call and specify the ViewType parameter to choose what information you want to include in the stream. For an existing table, use the UpdateTable API call and specify the ViewType parameter to choose what information to include in the stream.

The ViewType parameter takes the following values:

ViewType: {

                    { KEYS_ONLY,

                      NEW_IMAGE,

                      OLD_IMAGE,

                      NEW_AND_OLD_IMAGES}

                }

The values have the following meaning:

• KEYS_ONLY: Only the name of the key of items that changed are included in the stream.

• NEW_IMAGE: The name of the key and the item after the update (new item) are included in the stream.

• OLD_IMAGE: The name of the key and the item before the update (old item) are included in the stream.

• NEW_AND_OLD_IMAGES: The name of the key, the item before (old item) and after (new item) the update are included in the stream.

Note:

Stream records are organized into shards (groups). Each shard acts as a container for multiple stream records, and contains information required for accessing and iterating through records. Shards are created and deleted automatically, as needed.  If you disable a stream, any shards that are open will be closed. 

A shard might split in response to high levels of write activity on parent table, and applications can process records from multiple shards in parallel. Applications must always process a parent shard before it processes a child shard to ensure that stream records are processed in correct order. DynamoDB Streams Kinesis Adapter will automatically handle this.

Enabling DynamoDB Streams

DynamoDB Streams have to be enabled on a per-table basis. DynamoDB Streams for an existing table can be enabled from the console under the Manage Stream, enabling  a view type. Under Stream details, verify Stream enabled is set to ‘Yes’ after enabling.

You can start reading from the stream as soon as you start creating it, but the stream may not include all updates to the table until the status changes to ENABLED. You can use the DescribeStream API to get the current status of the stream.

Accessing the Stream

You can access a stream with API call using DynamoDB SDK or using Kinesis Client Library (KCL). KCL helps you consume and process the data from a stream and also helps you manage tasks such as load balancing across multiple readers, responding to instance failures, and checkpointing processed records.

You can use the DynamoDB Streams Adapter, which implements the Amazon Kinesis interface, to allow your application to use the Amazon KCL to access DynamoDB Streams.

Updating Stream

If you want to change the type of information stored in a stream after it has been created, you must disable the stream and create a new one using the UpdateTable API.

Each update in a DynamoDB stream will include a parameter that specifies whether the update was a deletion, insertion of a new item, or a modification to an existing item.

Deleting Table / Restart Streams

After deleting table, the stream will persist for 24 hours. After 24 hours, the stream will be deleted automatically from DynamoDB Streams.

If you turn off DynamoDB Streams, the stream will persist for 24 hours but will not be updated with any additional changes made to your table. If you turn Streams back on, this will create a new stream that contains the changes made to your table starting from the time that new stream was created.

Guarantees

1. Changes made to any individual item will appear in the correct order. Changes made to different items may appear in a different order.

2. Each stream record appears exactly once in the stream.