Bucket Connections - Security Best Practices

Bucket Connections: Security Best Practices

Overview

This document outlines security best practices for connecting AWS S3 and Google Cloud Storage buckets to Pillar.

Authentication Methods Comparison

Provider	Method	Credentials
AWS S3	IAM User with Access Keys	Access Key ID + Secret Access Key
GCS	Service Account	JSON Key File

Why These Methods?

AWS S3: IAM Users

Why not IAM Roles?

• IAM Roles are designed for AWS resources (EC2, Lambda) or AssumeRole cross-account access
• For external applications accessing S3, IAM Users with Access Keys are the standard approach
• Roles require additional infrastructure (STS, AssumeRole API calls)

Best Practice:

• ✅ Create dedicated IAM user per application
• ✅ Minimal permissions (ListBucket, GetObject only)
• ✅ Restrict to specific buckets in the policy
• ❌ Never use root account credentials

GCS: Service Accounts

Why not user accounts?

• Service accounts are designed for application-to-application authentication
• User accounts are for people, service accounts are for services
• Service accounts don’t have passwords and can’t log into GCP console

Best Practice:

• ✅ Create dedicated service account per application
• ✅ Minimal permissions (Storage Object Viewer)
• ✅ Bucket-level IAM for additional restriction
• ❌ Don’t use user credentials or default service accounts

Advanced: More Secure Alternatives

AWS: Cross-Account IAM Roles (Enterprise)

When to use:

• You want to avoid long-lived credentials
• You have infrastructure to assume roles
• You need time-limited, revocable access

Setup:

1. Create IAM Role in bucket-owning account
2. Add trust policy allowing Pillar’s AWS account
3. Pillar assumes role with STS for temporary credentials

Pros:

• No long-lived access keys
• Temporary credentials (1-12 hours)
• Easy to revoke (just update trust policy)

Cons:

• More complex setup
• Requires Pillar to have AWS infrastructure
• Higher latency (STS calls)

GCS: Workload Identity (Enterprise)

When to use:

• Pillar is running on GCP (GKE, Compute Engine)
• You want to avoid service account keys
• You need the most secure option

Setup:

1. Bind Kubernetes service account to GCS service account
2. Workload automatically gets credentials
3. No JSON keys needed

Pros:

• No keys to manage or rotate
• Automatic credential rotation
• Most secure option

Cons:

• Only works if Pillar runs on GCP
• Complex initial setup
• Not available for external applications

Principle of Least Privilege

Minimal S3 Permissions

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:ListBucket",
        "s3:GetObject"
      ],
      "Resource": [
        "arn:aws:s3:::specific-bucket-name",
        "arn:aws:s3:::specific-bucket-name/*"
      ]
    }
  ]
}

Why these permissions?

• s3:ListBucket - Required to discover files
• s3:GetObject - Required to download files

Not needed (unless writing back):

• ❌ s3:PutObject
• ❌ s3:DeleteObject
• ❌ s3:* (never use wildcard)

Minimal GCS Permissions

Predefined role:

roles/storage.objectViewer

Custom role permissions:

storage.buckets.get
storage.objects.list
storage.objects.get

Not needed:

• ❌ storage.objects.create
• ❌ storage.objects.delete
• ❌ roles/storage.admin

Credential Management

Storage

Method	Security Level	Recommendation
Plain text file	❌ Low	Never
Password manager	✅ Good	For individuals
Secrets manager	✅✅ Best	For teams/production
Environment variables	⚠️ Medium	OK for development

Best practice in Pillar:

• Store encrypted in database
• Never log credentials
• Mask in UI (show last 4 chars only)
• Use HTTPS for transmission

Rotation Schedule

Credential Type	Recommended Rotation
AWS Access Keys	Every 90 days
GCS Service Account Keys	Every 90 days
IAM Roles (temporary)	Automatic (1-12 hours)
Workload Identity	Automatic

Implementing rotation:

1. Create new credentials
2. Update in Pillar
3. Test connection
4. Delete old credentials

Monitoring & Auditing

AWS CloudTrail

Enable CloudTrail to monitor S3 access:

{
  "eventName": "GetObject",
  "userIdentity": {
    "userName": "pillar-s3-sync"
  },
  "requestParameters": {
    "bucketName": "my-bucket"
  }
}

What to monitor:

• Unusual access patterns
• Access from unexpected IPs
• Failed authentication attempts
• Permission denied errors

GCS Audit Logs

Enable Cloud Audit Logs:

gcloud logging read "resource.type=gcs_bucket AND protoPayload.authenticationInfo.principalEmail=pillar-gcs-sync@*"

What to monitor:

• Service account activity
• Bucket access patterns
• Permission changes
• Failed operations

Security Checklist

For AWS S3

• Created dedicated IAM user (not root)
• Applied minimal permissions policy
• Restricted to specific bucket(s)
• Stored credentials securely
• Enabled CloudTrail logging
• Configured MFA on AWS account
• Planned key rotation schedule
• Tested revocation process

For GCS

• Created dedicated service account
• Applied minimal permissions role
• Configured bucket-level IAM
• Stored JSON key securely
• Enabled Cloud Audit Logs
• Verified organization policies
• Planned key rotation schedule
• Tested revocation process

Common Security Mistakes

❌ Using Root/Admin Credentials

Problem:

• Grants unlimited access to all resources
• Can’t easily revoke without affecting everything
• Violates principle of least privilege

Solution:

• Always create dedicated IAM users or service accounts
• Grant only permissions needed for the specific task

❌ Overly Permissive Policies

Problem:

// DON'T DO THIS
{
  "Effect": "Allow",
  "Action": "s3:*",
  "Resource": "*"
}

Solution:

• Specify exact actions needed
• Limit to specific resources (buckets)

❌ Sharing Credentials

Problem:

• Same credentials used across multiple applications
• Can’t track which app accessed what
• Revoking affects all applications

Solution:

• One credential set per application
• Use descriptive names
• Monitor usage separately

❌ Hardcoding Credentials

Problem:

// DON'T DO THIS
const accessKey = "AKIAI...";
const secretKey = "wJalr...";

Solution:

• Use environment variables
• Use secrets management systems
• Never commit to version control

Compliance Considerations

HIPAA

If handling healthcare data:

• ✅ Enable encryption at rest (S3/GCS)
• ✅ Enable encryption in transit (HTTPS)
• ✅ Enable audit logging
• ✅ Sign BAA with cloud provider
• ✅ Implement access controls

GDPR

If handling EU personal data:

• ✅ Choose appropriate bucket region
• ✅ Enable audit logs for data access
• ✅ Implement data retention policies
• ✅ Document data processing activities
• ✅ Enable right to erasure

SOC 2

For security compliance:

• ✅ Document access control procedures
• ✅ Enable comprehensive logging
• ✅ Implement change management
• ✅ Regular access reviews
• ✅ Incident response procedures

Incident Response

If credentials are compromised:

Immediate actions:

1. ✅ Revoke credentials immediately
2. ✅ Create new credentials with different name
3. ✅ Update in Pillar
4. ✅ Check audit logs for unauthorized access
5. ✅ Notify security team
6. ✅ Document the incident

AWS:

# Disable access key
aws iam update-access-key --access-key-id AKIAI... --status Inactive --user-name pillar-s3-sync

GCS:

# Delete service account key
gcloud iam service-accounts keys delete KEY_ID --iam-account=pillar-gcs-sync@project.iam.gserviceaccount.com

Resources

AWS

• IAM Best Practices
• S3 Security
• CloudTrail

GCP

• Service Account Best Practices
• GCS Access Control
• Cloud Audit Logs

Security Frameworks

• CIS Benchmarks
• NIST Cybersecurity Framework