Kinesis introduction - Amazon Web Services (AWS)教程

- [Instructor] Amazon Kinesis make it easy to collect, process, and analyze real-time streaming data. Amazon Kinesis represents a family of services that are ideal for processing real-time big data. Each of the services in the family is designed to help you with a specific task. In the Kinesis family, we have Kinesis Data Streams, that is a low-latency streaming ingesting service at scale. Kinesis Video Streams that allows you to process live video data in real time. And Kinesis Firehose that allows you to load streams of data into S3 Redshift and others. In this video, we are going to focus on Amazon Kinesis Data Streams. Kinesis Data Streams takes the data generated by different producers and ingests them into shards. Then the different consumers will read the data from the shards. Consumers can be other Kinesis Data Streams, Amazon EC2 instances, AWS Lambda functions and other applications. When defining a new Kinesis Data Stream, you need to define how many shards you are going to need, and that is defined based on the amount of data that you are going to ingest per second. Data that is produced is stored in those shards. Shards are partition of data, and that data is ordered within the shard. Then the consumers will consume the data from the different shards. Data is stored by default in the shard for 24 hours and you can extend the retention up to 365 days if you need so. Remember that you will pay for that data storage. Let's see now some use cases for Kinesis Data Streams. There are many use cases in which organizations are using Kinesis Data Streams. For example, log and event data collection. Kinesis Data Streams can be used to collect logs and event data from different sources. That data can be aggregated, for example, and displayed in some dashboard. Real-time analytics. You can gain real-time insights from your data as the events are coming in to the stream and being processed. Click-through data. Your apps and web applications send data to Kinesis whenever a user performs an activity on the site. Kinesis can take hundreds of thousands of records and make them available for you as soon as they are produced. Financial data. Kinesis can be used to collect data through high frequency data and detect fraudulent activities in real time. Let's talk now about some important concepts that you will need to know when working with Kinesis Data Streams. A shard is a unique sequence of data records in a stream and provides a fixed unit of capacity. Each shard can support five read transactions per second or two megabytes per second, and they can write up to a thousand records per second up to a maximum total write of one megabyte per second. Inside a shard, you are going to have records. Records are immutable unit of data stored in a Kinesis Data Stream shard. Records are composed of a sequence number, a partition key, and a data block. Kinesis Data Stream doesn't inspect change or interpret the data block in any way. The partition key is a value that determines in which shard the record will be stored. Kinesis Data Stream uses a hash function to define in which shard a partition will be. In one shard, you can have multiple partitions. So now that you understand a bit more how shard works, it's important to highlight that shards are very important to increase the throughput of a stream. If you need to have more write or read capacity in your stream, you need to have more shards. Here is where the capacity mode comes into picture. The capacity mode of a stream determines how the capacity is managed in the stream. Currently, Kinesis Data Stream supports on-demand and provision mode. In the on-demand mode, Kinesis Data Stream will provision the shard for you in a serverless way. This means that Kinesis Data Stream will scale up and down the stream accordingly to the data that is being ingested. On the provision mode, you are going to define how many shards you are going to need and you're going to scale up and down manually the stream.