Day 23 - Building S3 Security & Operations Monitoring using Terraform, CloudTrail, CloudWatch and SNS
For Day 23 of my AWS Terraform journey, I built a security monitoring solution for Amazon S3 using Terraform modules.
The goal of this project was to monitor S3 object-level activity, detect suspicious access patterns, and send email alerts automatically using native AWS services.
This project uses:
- Amazon S3
- AWS CloudTrail
- Amazon CloudWatch Logs
- CloudWatch Metric Filters
- CloudWatch Alarms
- Amazon SNS
- Terraform Modules
Project Objective
The monitoring solution performs the following:
- Creates a monitored S3 bucket with a random suffix
- Captures S3 object-level API activity using CloudTrail Data Events
- Sends CloudTrail logs to CloudWatch Logs
-
Uses Metric Filters to detect:
- AccessDenied or 403 errors
-
Access to restricted prefixes such as
private/*
- Triggers CloudWatch Alarms when suspicious activity is detected
- Sends email notifications through Amazon SNS
Architecture Diagram
This diagram should show:
- User accessing S3 bucket
- CloudTrail capturing S3 Data Events
- CloudWatch Logs receiving logs
- Metric Filters analyzing logs
- CloudWatch Alarms triggering
- SNS sending email alerts
Project Folder Structure
I organized the Terraform configuration using modules for better separation of responsibilities.
day-23 - S3 Security & Operations Monitoring/
├── versions.tf
├── provider.tf
├── variables.tf
├── main.tf
├── outputs.tf
├── terraform.tfvars.example
└── modules/
├── log_ingest_s3/
│ ├── main.tf
│ ├── variables.tf
│ └── outputs.tf
├── security_metrics/
│ ├── main.tf
│ ├── variables.tf
│ └── outputs.tf
├── security_alarms/
│ ├── main.tf
│ └── variables.tf
└── sns_security/
├── main.tf
├── variables.tf
└── outputs.tf
Terraform Module Design
1. log_ingest_s3 Module
This module is responsible for:
- Creating the monitored S3 bucket
- Creating the CloudTrail log bucket
- Creating the CloudWatch Log Group
- Configuring CloudTrail Data Events
- Creating IAM permissions for CloudTrail logging
This module acts as the ingestion layer for security events.
2. security_metrics Module
This module creates CloudWatch Metric Filters.
The filters monitor CloudTrail logs for:
- AccessDenied or 403 errors
-
Access to restricted prefixes like
private/*
These filters convert log patterns into CloudWatch metrics.
3. security_alarms Module
This module creates CloudWatch Alarms based on the custom metrics.
The alarms trigger when:
- Failed access attempts occur
- Restricted objects are accessed
Threshold used:
Threshold = 1
This means even a single suspicious event triggers an alarm.
4. sns_security Module
This module creates:
- SNS Topic
- Email Subscription
When an alarm enters the ALARM state, SNS sends an email notification automatically.
Terraform Configuration
The root main.tf acts as the orchestrator and connects all modules together.
Example flow:
S3 Activity
↓
CloudTrail Data Events
↓
CloudWatch Logs
↓
Metric Filters
↓
CloudWatch Alarms
↓
SNS Email Notification
Deployment Steps
Initialize Terraform
terraform init
This screenshot should show providers downloading successfully.
Validate Terraform Configuration
terraform validate
Terraform Plan
terraform plan -var="security_alert_email=your-email@example.com"
This screenshot should show all AWS resources Terraform plans to create.
Terraform Apply
terraform apply -var="security_alert_email=katta.jayant@gmail.com"
This screenshot should show successful resource creation.
SNS Email Confirmation
After deployment, AWS SNS sends a confirmation email.
The subscription must be confirmed before alerts can be received.
Testing the Monitoring System
Retrieve Bucket Name
terraform output monitored_bucket_name
Test Restricted Prefix Access
I downloaded a file from the private/ prefix to generate a monitored event.
BUCKET_NAME=$(terraform output -raw monitored_bucket_name)
aws s3 cp s3://$BUCKET_NAME/private/secret-file.txt downloaded-secret.txt
This action triggers the Restricted Prefix alarm.
Test Failed Access Request
Next, I attempted to access a non-existing object.
aws s3 cp s3://$BUCKET_NAME/ghost-file.txt .
This generates an AccessDenied or failed request event.
Verifying CloudWatch Alarms
CloudTrail Data Events usually take several minutes to appear in CloudWatch Logs.
After waiting a few minutes, both alarms entered the ALARM state.
This screenshot should show alarms in ALARM status.
Email Notification Validation
Once the alarms triggered, SNS sent an email notifications automatically.
This screenshot should show the actual alert emails received.
Key Learnings
This project helped me understand the full monitoring pipeline for S3 object-level activity.
I learned the difference between CloudTrail Management Events and Data Events. For S3 object monitoring, Data Events are required.
CloudWatch Metric Filters are extremely useful for converting log activity into actionable metrics.
SNS provides a simple but effective notification mechanism for operational monitoring.
Using Terraform modules made the project cleaner and easier to organize compared to placing everything in a single file.
Challenges Faced
One important observation was the delay in CloudTrail Data Events appearing inside CloudWatch Logs.
The alarms do not trigger instantly. Sometimes the delay can range from 5 to 15 minutes depending on log delivery timing.
Another important part was configuring IAM permissions correctly for CloudTrail integration with CloudWatch Logs.
Cleanup
After validation, I destroyed all resources to avoid unnecessary AWS charges.
terraform destroy -var="security_alert_email=katta.jayant@gmail.com"
Final Thoughts
This project was a good hands-on exercise in combining security monitoring with Infrastructure as Code.
Instead of only provisioning resources, this setup focuses on observability and operational awareness inside AWS.
It also resembles traditional database auditing concepts where failed access attempts and sensitive object access are monitored continuously.
As I continue this AWS Terraform journey, I am starting to understand how monitoring, alerting, and security operations integrate together in real cloud environments.
Comments
Post a Comment