This post explains how you can ingest data from a MQTT broker such as VerneMQ into your data lake via IoT Core and Kinesis Data Firehose. We’ll set up a data processing pipeline from start to finish in Terraform.
MQTT is one of the core protocols that enables IoT applications. In this post we’ll first introduce MQTT and some of the core concepts. We’ll also take a look at AWS IoT Core, the MQTT broker that comes bundled in AWS.
Reliable Data Streaming on AWS We should agree that in our digital world streaming and especially data streaming becomes more and more important if it isn’t already. Besides performance/throughput and security for a data streaming system reliability
Over the last years, demands and expectations on data security have increased dramatically. The main drivers are local data privacy regulations like the EU-GDPR, which imply awareness of sensitive data and an overview of potential risks. Amazon has offere
Amazon Ion is one of the data serialization formats you can use when exporting data from DynamoDB to S3. Recently, I tried to select data from one of these exports with Athena after using a Glue Crawler to create the schema and table. It didn’t work
In this post I’ll introduce DynamoDB, a very powerful fully managed NoSQL wide-column data store in AWS. We will talk about data structures, the APIs to read and write data, indexes, as well as performance and cost considerations. In the end you wil