Cloud Storage

Sync documents from an Amazon S3 or Google Cloud Storage (GCS) bucket. Pillar reads files from your bucket, extracts the content, and indexes it for the co-pilot. Your files stay in your bucket — Pillar only needs read access.

When to Use

• You have a large library of documents stored in S3 or GCS
• You want automatic syncing when files change
• Your content is managed through a pipeline that outputs to cloud storage

Supported File Types

Maximum file size: 100 MB

Setup

Amazon S3

Create an IAM User

1. Open the AWS Console and navigate to IAM → Users
2. Click Create user and enter a username (e.g., pillar-s3-reader)
3. Select Attach policies directly

Set Permissions

Create a custom policy with read-only access to your bucket. Replace YOUR-BUCKET-NAME with your actual bucket name:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::YOUR-BUCKET-NAME",
        "arn:aws:s3:::YOUR-BUCKET-NAME/*"
      ]
    }
  ]
}

This grants only two permissions:

Pillar never writes, deletes, or modifies your data.

Generate Access Keys

1. Click on the IAM user you created
2. Go to the Security credentials tab
3. Under Access keys, click Create access key
4. Select Application running outside AWS
5. Copy both the Access Key ID and Secret Access Key

Enter Credentials in Pillar

Multiple Buckets

To grant access to multiple buckets, add their ARNs to the policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::bucket-one",
        "arn:aws:s3:::bucket-one/*",
        "arn:aws:s3:::bucket-two",
        "arn:aws:s3:::bucket-two/*"
      ]
    }
  ]
}

Google Cloud Storage

Create a Service Account

1. Open the Google Cloud Console
2. Navigate to IAM & Admin → Service Accounts
3. Click + Create Service Account
4. Enter a name (e.g., pillar-bucket-reader) and click Create and Continue

Grant Permissions

1. In the role selection, search for Storage Object Viewer (roles/storage.objectViewer)
2. Select it and click Continue, then Done

Generate a JSON Key

1. Click on your service account in the list
2. Go to the Keys tab
3. Click Add Key → Create new key
4. Select JSON and click Create

A JSON file will download. It looks like this:

{
  "type": "service_account",
  "project_id": "your-project-id",
  "private_key_id": "abc123...",
  "private_key": "-----BEGIN PRIVATE KEY-----\n...\n-----END PRIVATE KEY-----\n",
  "client_email": "pillar-bucket-reader@your-project.iam.gserviceaccount.com",
  "client_id": "123456789",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://oauth2.googleapis.com/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/..."
}

Enter Credentials in Pillar

Restricting to a Specific Bucket

For tighter security, grant bucket-level access instead of project-level:

1. Go to Cloud Storage → Buckets → your bucket → Permissions
2. Click Grant Access
3. Enter the service account email
4. Select Storage Object Viewer
5. Save, then remove the project-level permission

Sync Behavior

• First sync: All matching files are downloaded and indexed
• Periodic sync: Pillar re-syncs daily to pick up changes
• Manual sync: Trigger a sync anytime from the dashboard
• Incremental updates: Only new or changed files are reprocessed (detected via content hashing)

Path Prefix

Use the optional prefix field to sync only files under a specific path:

• Prefix docs/ syncs only files in the docs/ folder
• Prefix products/manuals/ syncs only files in that subfolder
• Leave empty to sync the entire bucket

Security Best Practices

• Dedicated credentials: Create a separate IAM user or service account just for Pillar
• Read-only access: Only grant list and read permissions — never write or delete
• Rotate keys: Delete and recreate credentials every 90 days
• Never commit credentials: Keep access keys and JSON files out of source control

Troubleshooting

Access Denied / Permission Denied

• Verify the IAM user or service account has the correct read-only policy
• For S3, check that the policy includes both the bucket ARN and bucket/* for objects
• For GCS, verify the service account has Storage Object Viewer on the bucket

Invalid Credentials

• S3: Access Key ID should start with AKIA. Check for extra spaces.
• GCS: Make sure you pasted the entire JSON file contents, including the curly braces.

Bucket Not Found

• Verify the bucket name is spelled correctly (case-sensitive for S3)
• For S3, ensure the region matches where the bucket was created

Finding Your S3 Bucket Region

1. Go to S3 in the AWS Console
2. Click your bucket → Properties tab
3. Region is shown at the top