Menu

📰
0

Reddit - Please wait for verification

Amazon Web Services (AWS): S3, EC2, SQS, RDS, DynamoDB, IAM, CloudFormation, Route 53, VPC and more·/u/Direct-Value4452·3 days ago
#I8PlqKhL
Reading 0:00
15s threshold

​ The part I’m stuck on is how much of our warehouse ingestion should stay AWS-native versus using a separate ETL tool. Current setup is pretty normal: a few RDS Postgres/MySQL databases, some SaaS sources, S3 files from vendors, and CSV uploads that still show up more often than I’d like. Data volume is not huge, but we do need scheduled loads, retries, basic mapping, and occasional backfills. I’ve looked at Glue, DMS, Lambda scripts, Airflow, and a few managed ETL tools. Glue seems useful, but maybe more work than we need for basic SaaS ingestion. DMS makes sense for database replication, but not really for every source. Lambda scripts are fine until there are too many small edge cases. For smaller AWS-based data setups, what AWS ETL tools or approaches have actually worked well long term? Do you keep most of it AWS-native, use external connectors, or mix both depending on the source? submitted by /u/Direct-Value4452 [link] [comments]

Read More