Connecting Your Google Cloud Storage Bucket

Connecting Your Google Cloud Storage Bucket to Pillar

This guide will help you securely connect your GCS bucket to Pillar for automatic document synchronization.

Overview

Pillar needs read access to your GCS bucket to automatically sync documents. We recommend creating a dedicated service account with minimal permissions.

Prerequisites

• Google Cloud Platform account
• Existing GCS bucket with documents
• Permissions to create service accounts (IAM Admin role)

---

Step 1: Create a Service Account

Why? Service accounts are designed for application-to-application authentication with minimal permissions.

1. Go to GCP IAM Console
2. Select your project
3. Click Create Service Account
4. Enter details:
   • Name: pillar-gcs-sync
   • Description: “Service account for Pillar document sync”
5. Click Create and Continue

Step 2: Grant Bucket-Level Permissions

Important: Assign permissions at the bucket level, not the project level. This ensures the service account only has access to the specific bucket you want to sync.

1. Skip the project-level permissions during service account creation

   • Click Continue → Done (leave roles empty)

2. Go to Cloud Storage Browser

3. Click on your specific bucket name

4. Click the Permissions tab at the top

5. Click Grant Access

6. Enter the following:

   • New principals: Your service account email
     (e.g., pillar-gcs-sync@your-project.iam.gserviceaccount.com)
   • Role: Search for and select Storage Object Viewer

7. Click Save

Why this works: When assigned at the bucket level, Storage Object Viewer includes all three required permissions for that specific bucket:

• storage.buckets.get - Access bucket metadata
• storage.objects.list - List files in the bucket
• storage.objects.get - Download files for indexing

Security benefit: The service account can only access this one bucket, not all buckets in your project.

Skip “Grant users access” (Optional Step)

After creating the service account, you may see a screen asking to grant users access to it. You can skip this entirely.

• Leave “Service account users role” empty
• Leave “Service account admins role” empty
• Click Done

Why? These fields let other users impersonate or manage the service account. Since Pillar authenticates with the JSON key directly, this isn’t needed.

Step 3: Create Service Account Key

1. Return to Service Accounts
2. Click on your pillar-gcs-sync service account
3. Go to Keys tab
4. Click Add Key → Create new key
5. Select JSON format
6. Click Create
7. IMPORTANT: The JSON file downloads automatically - store it securely!

The JSON key file looks like this:

{
  "type": "service_account",
  "project_id": "your-project-id",
  "private_key_id": "...",
  "private_key": "-----BEGIN PRIVATE KEY-----\n...\n-----END PRIVATE KEY-----\n",
  "client_email": "pillar-gcs-sync@your-project.iam.gserviceaccount.com",
  "client_id": "...",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://oauth2.googleapis.com/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": "..."
}

Step 4: Configure Pillar

1. In Pillar, navigate to Knowledge Base → Data Sources → Add Bucket Connection

2. Select Google Cloud Storage

3. Enter the following:

   • Bucket Name: Your GCS bucket name (e.g., my-company-docs)
   • Service Account JSON: Paste the entire JSON key file contents
   • Prefix (optional): Folder path to sync (e.g., docs/ to only sync the docs folder)
   • File Types (optional): Comma-separated extensions (e.g., pdf, txt, md, docx)
   • Sync Frequency: How often to check for updates (default: 60 minutes)

4. Click Test Connection to verify

5. Click Create Connection

Security Best Practices

✅ DO

• Create dedicated service accounts for each application
• Use minimal permissions (Storage Object Viewer only)
• Limit to specific buckets using bucket-level IAM
• Rotate service account keys annually
• Enable Cloud Audit Logs to monitor access
• Use workload identity if running on GCP (advanced)
• Store JSON keys securely (password manager, secrets manager)

❌ DON’T

• Grant broad roles like “Owner” or “Editor”
• Use user credentials instead of service accounts
• Grant write permissions unless necessary
• Commit service account keys to version control
• Share keys via email or unsecured channels
• Use the same service account for multiple applications

Permissions Explained

Required Permissions

Permission	Purpose
storage.buckets.get	Access bucket metadata
storage.objects.list	List files in the bucket
storage.objects.get	Download document contents for indexing

Optional Permissions

If you want Pillar to write back to your bucket:

storage.objects.create
storage.objects.delete

Advanced: Cross-Project Buckets

To connect a bucket from a different GCP project:

1. Create the service account in Project A
2. Grant the service account access to the bucket in Project B:
   • Go to the bucket in Project B’s Cloud Storage Browser
   • Click Permissions tab
   • Click Grant Access
   • Add the service account email as a principal with Storage Object Viewer role

Advanced: Workload Identity (for GCP-hosted Pillar)

If Pillar is running on GKE or other GCP services:

1. Create a Kubernetes service account
2. Bind it to the GCS service account
3. No JSON keys needed!

Contact Pillar support for setup assistance.

Troubleshooting

”Access Denied” or “Forbidden” Errors

1. Check bucket name: Ensure it’s spelled correctly
2. Check bucket-level permissions: Verify Storage Object Viewer is assigned to the service account on the specific bucket (not at project level)
3. Check the service account email: Ensure you copied the correct email address
4. Check organization policies: Some organizations restrict service account usage
5. Verify service account is enabled: Check it hasn’t been disabled in IAM

”Invalid Credentials” Error

1. Verify the JSON key is complete and valid JSON
2. Check if the service account key was disabled or deleted
3. Ensure no extra characters or spaces in the JSON
4. Try creating a new key

Connection Test Fails

1. Verify bucket exists and is in the same project
2. Check if the service account is enabled
3. Verify Cloud Storage API is enabled in your project
4. Check GCP Cloud Logging for specific error messages

Documents Not Syncing

1. Check prefix/path settings (must not start with /)
2. Verify file types filter if specified
3. Check sync frequency settings
4. Verify documents are in supported formats
5. Check if bucket has any object lifecycle policies

Supported File Types

Pillar supports the following document formats:

• Documents: PDF, DOC, DOCX, TXT, RTF
• Spreadsheets: XLS, XLSX, CSV
• Presentations: PPT, PPTX
• Markup: MD, HTML
• Code: Most text-based formats

Data Privacy & Security

• Encryption in Transit: All data is transmitted over HTTPS/TLS
• Encryption at Rest: Documents are encrypted in Pillar’s knowledge base
• Credential Storage: Service account keys are encrypted and stored securely
• No Data Retention: Pillar only indexes document content; original files remain in your GCS bucket
• Access Logs: Monitor GCP Cloud Audit Logs for all GCS access by Pillar
• Compliance: GCS supports various compliance standards (HIPAA, SOC 2, etc.)

Key Rotation

To rotate service account keys:

1. Create a new key for the service account
2. Update the key in Pillar
3. Test the connection
4. Delete the old key from GCP

Recommended rotation frequency: Every 90 days

Revoking Access

To disconnect or revoke Pillar’s access:

1. In Pillar: Delete the bucket connection from Data Sources
2. In GCP:
   • Go to IAM & Admin → Service Accounts
   • Click on pillar-gcs-sync
   • Delete all keys or disable/delete the service account
   • Remove bucket-level permissions if configured

Cost Considerations

GCS charges for:

• Storage: Based on data stored
• Operations: API calls (list, get operations)
• Network: Data egress outside GCP

Pillar’s sync operations are minimal, but be aware:

• Initial sync may perform many GET operations
• Regular syncs only check for changes (LIST operations)
• Consider sync frequency vs. cost trade-offs

Getting Help

• Documentation: [Link to help center]
• Support: support@pillar.ai
• GCP IAM Docs: https://cloud.google.com/iam/docs
• GCS Permissions: https://cloud.google.com/storage/docs/access-control/iam

---

Quick Reference

Required Role (at bucket level):

Storage Object Viewer (roles/storage.objectViewer)

Permissions Included:

storage.buckets.get     - Access bucket metadata
storage.objects.list    - List files in bucket
storage.objects.get     - Download files for indexing

Quick Reference: Service Account Email Format

[SA-NAME]@[PROJECT-ID].iam.gserviceaccount.com

Example:

pillar-gcs-sync@acme-corp-prod.iam.gserviceaccount.com