Amazon S3 Data Warehouse

Amazon S3 Data Warehouse Integration

The Amazon S3 Data Warehouse integration provides a native solution for storing your event, dispatch, and visitor data in a scalable, cost-effective manner. Ours Privacy automatically deposits daily data into your specified S3 bucket in parquet format, containing all events and dispatches that occurred on your account, plus visitor records that have been recently updated.

How the Integration Works

• Scheduled Deposits: Events and dispatches are automatically collected and deposited into your S3 bucket on your chosen sync frequency — daily by default, or hourly (see Sync Frequency)
• Visitor Updates: Visitor data contains records where the last seen timestamp is greater than or equal to yesterday, requiring upsert processing
• Parquet Format: Data is stored in efficient parquet format, optimized for analytics and querying
• Complete Data: All events and dispatches from your account are included in the daily deposits
• Flexible Access: Once in your S3 bucket, you can process, analyze, or move the data as needed
• Custom KMS Encryption: Buckets with customer-managed KMS keys are supported

Data Organization

The data in your S3 bucket is organized in a partitioned structure:

s3://your-bucket/
  ├── events/                       # daily (canonical, sealed day)
  │   └── YYYY/MM/DD/*.parquet
  ├── dispatches/
  │   └── YYYY/MM/DD/*.parquet
  ├── visitors/
  │   └── YYYY/MM/DD/*.parquet
  └── live/                         # hourly (opt-in, one immutable folder per hour)
      ├── events/
      │   └── YYYY/MM/DD/HH/*.parquet
      ├── dispatches/
      │   └── YYYY/MM/DD/HH/*.parquet
      └── visitors/
          └── YYYY/MM/DD/HH/*.parquet

The daily feed under events/, dispatches/, and visitors/ is always present. The live/ tree only appears when hourly sync is enabled.

This partitioning by event/year/month/day (and hour, for the live feed) makes it easy to:

• Query specific time periods efficiently
• Manage data retention policies
• Process historical data in batches

Getting Started

To set up the Amazon S3 Data Warehouse integration:

1. Contact your account manager to enable the integration and provide you with the IAM policy for your S3 Bucket
2. Provide your S3 bucket details and add the policy to your bucket provided by your account manager
3. Choose your Sync frequency in the destination settings — daily by default, or hourly (see Sync Frequency)
4. Configure any additional processing or analytics tools you plan to use with the data

Once configured, your event and dispatch data will be automatically deposited into your S3 bucket on your chosen schedule, ready for your use in analytics, reporting, or other data processing workflows.

Sync Frequency

Choose how often data is delivered to your bucket using the Sync frequency setting on the destination:

• Every 24 hours (default): Once per day, the previous full day of events, dispatches, and recently updated visitors is deposited under that day's partition (events/YYYY/MM/DD/). This is the canonical, complete record for each finished day.
• Hourly: Shortly after each hour closes, that completed hour is deposited into its own immutable folder under the live/ prefix (live/events/YYYY/MM/DD/HH/). Each hour is written once and never modified, so the feed is append-safe and never contains duplicate records.

Hourly sync is opt-in and available on request. Because each hour is written once into a new folder and never overwritten, it needs no changes to your bucket policy beyond the write access already granted — the same IAM policy that serves the daily feed works as-is, and the feed loads cleanly into append-based tools such as the BigQuery Data Transfer Service. Each hour is sealed about 15 minutes after it closes (to let all of that hour's events finish landing), so the live/ feed runs roughly an hour behind real time. Contact support@oursprivacy.com to enable hourly delivery.

Data Format

The parquet files deposited in your S3 bucket contain your complete data, including:

• Event names and properties
• Dispatch details and status
• User information
• Timestamps
• All associated metadata

Best Practices

• Ensure your S3 bucket has appropriate access policies (you will need to contact a member of the Ours Privacy team for this)
• Consider setting up lifecycle policies to manage data retention
• Use AWS Athena or similar tools to query the parquet files directly
• Take advantage of the partitioning structure for efficient querying
• If your bucket uses a customer-managed KMS key, contact our support team to configure encryption settings for your destination