Kinesis Firehose: Kinesis Data Source To S3

Basic Configuration in this directory creates kinesis firehose stream with MSK Cluster as source and S3 bucket as destination with a basic configuration.

Usage

To run this example you need to execute:

$ terraform init
$ terraform plan
$ terraform apply

Note that this example may create resources which cost money. Run terraform destroy when you don't need these resources.

• Send Message to Kafka

Documentation

# Create Client Machine
sudo su -
sudo yum -y install java-11
wget https://archive.apache.org/dist/kafka/2.8.1/kafka_2.12-2.8.1.tgz
tar -xzf kafka_2.12-2.8.1.tgz
wget https://github.com/aws/aws-msk-iam-auth/releases/download/v1.1.1/aws-msk-iam-auth-1.1.1-all.jar
mv aws-msk-iam-auth-1.1.1-all.jar kafka_2.12-2.8.1/libs/
vi kafka_2.12-2.8.1/bin/client.properties
security.protocol=SASL_SSL
sasl.mechanism=AWS_MSK_IAM
sasl.jaas.config=software.amazon.msk.auth.iam.IAMLoginModule required;
sasl.client.callback.handler.class=software.amazon.msk.auth.iam.IAMClientCallbackHandler

# Create Topic
export BS=my-endpoint
./kafka_2.12-2.8.1/bin/kafka-topics.sh --bootstrap-server $BS --command-config kafka_2.12-2.8.1/bin/client.properties --create --topic demo-topic --partitions 6

# Produce data
./kafka_2.12-2.8.1/bin/kafka-console-producer.sh --broker-list $BS --producer.config kafka_2.12-2.8.1/bin/client.properties --topic demo-topic

# Consume Data
./kafka_2.12-2.8.1/bin/kafka-console-consumer.sh --bootstrap-server $BS --consumer.config kafka_2.12-2.8.1/bin/client.properties --topic demo-topic --from-beginning

Requirements

Name	Version
terraform	>= 0.13.1
aws	~> 5.0
random	~> 3.0

Providers

Name	Version
aws	~> 5.0
random	~> 3.0

Modules

Name	Source	Version
ec2	terraform-aws-modules/ec2-instance/aws	n/a
firehose	../../../	n/a
msk_cluster	terraform-aws-modules/msk-kafka-cluster/aws	2.3.0
security_group	terraform-aws-modules/security-group/aws	~> 5.0
vpc	terraform-aws-modules/vpc/aws	~> 5.0

Resources

Name	Type
aws_msk_cluster_policy.this	resource
aws_s3_bucket.s3	resource
random_pet.this	resource
aws_availability_zones.available	data source

Inputs

Name	Description	Type	Default	Required
name_prefix	Name prefix to use in resources	string	"msk-to-s3-basic"	no
tags	Default Tags to be added to all resources.	map(string)	{}	no

Outputs

Name	Description
kinesis_data_stream_name	The name of the Kinesis Firehose Stream
kinesis_firehose_arn	The ARN of the Kinesis Firehose Stream
kinesis_firehose_destination_id	The Destination id of the Kinesis Firehose Stream
kinesis_firehose_role_arn	The ARN of the IAM role created for Kinesis Firehose Stream
kinesis_firehose_version_id	The Version id of the Kinesis Firehose Stream
msk_arn	MSK Topic Endpoint
msk_brokers_endpoint	Brokers endpoints
s3_bucket_arn	S3 Bucket ARN
topic_name	MSK Topic Name