DynamoDB Streams Overview

DynamoDB Streams provides a time-ordered sequence of item-level changes made to data in a table in the last 24 hours.

You can access a stream with a simple API call and update other data stores or take actions based on the changes made to your table.

Developers could also use DynamoDB Streams to keep data warehousing tools, such as Redshift, in sync for real-time analytics.

Capacity Provisioning

Capacity for your stream is managed automatically in DynamoDB Streams.  

You can read updates from your stream in DynamoDB Streams at up to twice the rate of the provisioned write capacity of your DynamoDB table. For example, if you have provisioned enough capacity to update 1,000 items per second in your DynamoDB table, you could read up to 2,000 updates per second from your stream.

DynamoDB Logstash Plugin for Elasticsearch

DynamoDB also integrates with Elasticsearch using the Amazon DynamoDB Logstash Plugin, thus enabling developers to add free-text search for DynamoDB content

Logstash is an open source data pipeline that works together with Elasticsearch to help you process logs and other event data. The Logstash Plugin make is easy to integrate DynamoDB tables with Elasticsearch clusters. The Logstash Plugin is available on GitHub. It is free to download and use.

DynamoDB Streams – Important Points

1. AWS maintains separate endpoints for DynamoDB and DynamoDB Streams.

2. The AWS SDKs provide separate clients for DynamoDB and DynamoDB Streams.

3. The naming convention for DynamoDB Streams endpoints is streams.dynamodb.<region>.amazonaws.com. For example, if you use the endpoint dynamodb.us-west-2.amazonaws.com to access DynamoDB, you would use the endpoint streams.dynamodb.us-west-2.amazonaws.com to access DynamoDB Streams.

4. DynamoDB Streams operates asynchronously, so there is no performance impact on a table if you enable a stream.

5. You will receive a ResourceInUseException if you attempt to enable stream on a table that already has a stream, and you will receive a ValidationException if you attempt to disable a stream on a table which does not have a stream.

6. Every stream is uniquely identified by an Amazon Resource Name (ARN). Issue a DynamoDB DescribeTable request and look for the LatestStreamArn element in the response.