Connecting Your AWS S3 Bucket

Connecting Your S3 Bucket to Pillar

This guide will help you securely connect your AWS S3 bucket to Pillar for automatic document synchronization.

Overview

Pillar needs read-only access to your S3 bucket to automatically sync documents into your knowledge base. This guide covers the currently supported connection method using IAM Users with Access Keys.

Security Note

AWS recommends using temporary credentials (IAM Roles with AssumeRole) instead of long-lived access keys for third-party access. Pillar plans to support this more secure method in the future. For now, we support IAM Users with Access Keys, which is acceptable for read-only operations when following best practices.

Why this is acceptable:

• ✅ Read-only permissions - Pillar only requests s3:ListBucket and s3:GetObject (no write/delete)
• ✅ Dedicated IAM user - Separate credentials just for Pillar, easily revoked
• ✅ Encrypted storage - Credentials are encrypted in Pillar’s database
• ✅ Minimal scope - Limited to specific bucket(s) only

Best practices we recommend:

• 🔄 Rotate access keys every 90 days
• 🔐 Enable CloudTrail to monitor access
• 🎯 Limit permissions to specific bucket(s)
• 🚨 Set up AWS CloudWatch alerts for unusual activity

Prerequisites

• AWS Account with access to IAM and S3
• An existing S3 bucket with documents
• Admin permissions to create IAM users and policies

---

Step 1: Create a Dedicated IAM User

Why? Never use your root AWS credentials. A dedicated IAM user with minimal permissions is more secure and can be revoked without affecting other services.

1. Sign in to the AWS IAM Console
2. Navigate to Users → Create user
3. Enter user name: pillar-s3-sync (or your preferred name)
4. Do NOT select AWS Management Console access (not needed)
5. Click Next: Permissions

Step 2: Set Permissions (Read-Only Access)

Important: Pillar only needs read-only access to your S3 bucket. The permissions below allow Pillar to list and download files, but NOT create, modify, or delete anything.

On the permissions screen, you’ll see three options:

• Add user to group
• Copy permissions from existing user
• Attach policies directly ← Select this option

Create a Read-Only Policy

1. Select Attach policies directly
2. Click Create policy (this opens a new tab)
3. Choose the JSON tab
4. Paste the following policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "PillarReadAccess",
      "Effect": "Allow",
      "Action": [
        "s3:ListBucket",
        "s3:GetObject",
        "s3:GetObjectVersion"
      ],
      "Resource": [
        "arn:aws:s3:::YOUR-BUCKET-NAME",
        "arn:aws:s3:::YOUR-BUCKET-NAME/*"
      ]
    }
  ]
}

Important: Replace YOUR-BUCKET-NAME with your actual bucket name.

5. Click Next: Tags, then click Next: Review
6. Name the policy: pillar-s3-read-policy
7. Add description (optional): “Read-only access to S3 bucket for Pillar document sync”
8. Click Create policy
9. Return to the Create user tab and click the refresh icon (↻) next to Create policy
10. Search for pillar-s3-read-policy and check the box next to it
11. Click Next: Tags (you can skip adding tags)
12. Click Next: Review then Create user

Alternative: Use Bucket Policy (Advanced)

If you prefer to manage permissions at the bucket level instead of creating an IAM policy:

1. In Step 2 above, you can skip attaching policies to the user
2. Instead, go to your S3 bucket → Permissions → Bucket Policy
3. Add this policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "PillarAccess",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::YOUR-ACCOUNT-ID:user/pillar-s3-sync"
      },
      "Action": [
        "s3:ListBucket",
        "s3:GetObject"
      ],
      "Resource": [
        "arn:aws:s3:::YOUR-BUCKET-NAME",
        "arn:aws:s3:::YOUR-BUCKET-NAME/*"
      ]
    }
  ]
}

Step 3: Create Access Keys

After creating the IAM user, you need to generate access keys for programmatic access:

1. Go to IAM Users Console
2. Click on your pillar-s3-sync user
3. Go to the Security credentials tab
4. Scroll down to Access keys section
5. Click Create access key
6. Select Third-party service (or Application running outside AWS)

About the AWS Warning

⚠️ At this point, AWS will show a warning: “Alternative recommended: As a best practice, use temporary security credentials (IAM roles) instead of creating long-term credentials like access keys.”

You can safely proceed because:

• ✅ This is for read-only access (minimal risk if credentials leak)
• ✅ You’re creating a dedicated IAM user (not root credentials)
• ✅ Permissions are scoped to specific bucket(s) (not account-wide)
• ✅ Pillar encrypts and securely stores these credentials
• ✅ You can revoke access anytime by deleting the key

AWS recommends IAM Roles for production multi-tenant scenarios, which Pillar plans to support in the future. For now, this method is appropriate for read-only bucket access.

7. Check the acknowledgment box and click Next
8. Optionally add a description tag (e.g., “Pillar document sync”)
9. Click Create access key
10. IMPORTANT: Copy both the Access Key ID and Secret Access Key immediately
    • Click Download .csv file to save them securely
    • You cannot view the secret key again after leaving this page
11. Store them securely (password manager recommended)

Step 4: Configure Pillar

1. In Pillar, navigate to Knowledge Base → Data Sources → Add Bucket Connection

2. Select Amazon S3

3. Enter the following:

   • Bucket Name: Your S3 bucket name (e.g., my-company-docs)
   • Region: Your bucket’s region (e.g., us-east-1)
   • Access Key ID: From Step 3
   • Secret Access Key: From Step 3
   • Prefix (optional): Folder path to sync (e.g., docs/ to only sync the docs folder)
   • File Types (optional): Comma-separated extensions (e.g., pdf, txt, md, docx)
   • Sync Frequency: How often to check for updates (default: 60 minutes)

4. Click Test Connection to verify

5. Click Create Connection

Security Best Practices

✅ DO

• Create a dedicated IAM user for Pillar (never use root credentials)
• Use minimal permissions (only ListBucket and GetObject)
• Limit to specific bucket(s) in the IAM policy
• Rotate access keys every 90 days
• Enable CloudTrail to audit S3 access
• Use MFA on your AWS account

❌ DON’T

• Share root AWS credentials
• Grant write permissions unless necessary
• Use wildcard (*) in Resource ARNs
• Hardcode credentials in code or version control
• Share access keys via email or unsecured channels

Permissions Explained

Required Permissions (Read-Only)

Pillar only needs READ access to sync documents from your S3 bucket. No write or delete permissions are required.

Permission	Purpose	Type
s3:ListBucket	List files in your bucket to discover new/changed documents	Read
s3:GetObject	Download document contents for indexing	Read
s3:GetObjectVersion	(Optional) Access versioned objects if versioning is enabled	Read

What Pillar CANNOT do with these permissions:

• ❌ Create, upload, or modify files (s3:PutObject)
• ❌ Delete files (s3:DeleteObject)
• ❌ Change bucket settings or policies
• ❌ Access other buckets not specified in the policy

Optional Write Permissions (Not Recommended)

If you have a specific use case where Pillar needs to write back to your bucket (e.g., for processed documents), you can add:

{
  "Action": [
    "s3:PutObject",
    "s3:DeleteObject"
  ]
}

Note: For standard document syncing, write permissions are NOT needed.

Advanced: Cross-Region Buckets

If your bucket is in a different region than your default AWS region:

1. Make sure to specify the correct region in Pillar
2. The IAM user works globally, but S3 buckets are regional
3. Ensure no region-specific SCP (Service Control Policy) restrictions

Advanced: Cross-Account Access

If you want to connect an S3 bucket from a different AWS account:

In the Bucket Account:

1. Create an IAM role (not user) in the bucket-owning account
2. Add Pillar’s AWS account ID to the trust policy
3. Attach the permissions policy to the role

In Pillar:

Contact Pillar support to configure AssumeRole access (requires additional setup).

Troubleshooting

”Access Denied” Errors

1. Check bucket name: Ensure it’s spelled correctly and exists
2. Check region: Bucket region must match what you entered
3. Check IAM permissions: Verify the policy includes both bucket and object ARNs
4. Check bucket policy: Ensure no explicit Deny statements
5. Check S3 Block Public Access: This shouldn’t affect IAM user access, but verify

Connection Test Fails

1. Verify access keys are copied correctly (no extra spaces)
2. Check if the IAM user is in the correct AWS account
3. Verify the bucket exists and you have access
4. Check AWS CloudTrail for specific error messages

Documents Not Syncing

1. Check prefix/path settings (must not start with /)
2. Verify file types filter if specified
3. Check sync frequency settings
4. Verify documents are in supported formats

Supported File Types

Pillar supports the following document formats:

• Documents: PDF, DOC, DOCX, TXT, RTF
• Spreadsheets: XLS, XLSX, CSV
• Presentations: PPT, PPTX
• Markup: MD, HTML
• Code: Most text-based formats

Data Privacy & Security

• Encryption in Transit: All data is transmitted over HTTPS/TLS
• Encryption at Rest: Documents are encrypted in Pillar’s knowledge base
• Credential Storage: Access keys are encrypted and stored securely
• No Data Retention: Pillar only indexes document content; original files remain in your S3 bucket
• Access Logs: Monitor your AWS CloudTrail for all S3 access by Pillar

Revoking Access

To disconnect or revoke Pillar’s access:

1. In Pillar: Delete the bucket connection from Data Sources
2. In AWS:
   • Go to IAM → Users → pillar-s3-sync
   • Delete the access key or delete the user entirely
   • Remove any bucket policies granting access

Getting Help

• Documentation: [Link to help center]
• Support: support@pillar.ai
• AWS IAM Docs: https://docs.aws.amazon.com/IAM/
• S3 Permissions Guide: https://docs.aws.amazon.com/AmazonS3/latest/userguide/access-policy-language-overview.html

---

Quick Reference: Minimal IAM Policy

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:ListBucket",
        "s3:GetObject"
      ],
      "Resource": [
        "arn:aws:s3:::YOUR-BUCKET-NAME",
        "arn:aws:s3:::YOUR-BUCKET-NAME/*"
      ]
    }
  ]
}

Remember to replace YOUR-BUCKET-NAME with your actual bucket name!

---

Future: IAM Roles with AssumeRole (Coming Soon)

Pillar is working on supporting AWS’s recommended approach using IAM Roles with temporary credentials. This will provide:

Benefits of IAM Roles

• 🔐 Temporary credentials that automatically expire (1-12 hours)
• 🔄 No rotation needed - credentials refresh automatically
• 🛡️ External ID protection - prevents confused deputy attacks
• 📊 Better audit trail - each access generates AssumeRole logs
• ✅ AWS recommended - follows security best practices

How it will work (when available):

Instead of creating an IAM User with access keys, you’ll:

1. Create an IAM Role in your AWS account (not an IAM User)
2. Add a trust policy allowing Pillar’s AWS account to assume the role
3. Include an External ID that Pillar provides (unique per customer)
4. Provide the Role ARN to Pillar (not credentials)

Pillar will then call AWS STS AssumeRole to get temporary credentials before each sync.

Migration path:

When IAM Role support is available:

• Existing connections using access keys will continue to work
• You’ll be able to upgrade to IAM Roles from your connection settings
• Step-by-step migration guide will be provided

Want to be notified when this is available? Contact support@pillar.ai to be added to the early access list.

For enterprise customers:

If your organization requires IAM Roles now due to security policies, please contact our enterprise support team. We can work with you to implement a custom solution.