Google BigQuery

Google BigQuery Data Warehouse Integration

The Google BigQuery Data Warehouse integration provides a powerful solution for storing and analyzing your event and dispatch data. We utilize Google's S3 transfer integration to automatically move your data from S3 to BigQuery, enabling efficient querying and analysis of your complete event dataset.

Read more here

How the Integration Works

• Daily S3 to BigQuery Transfers: Events and dispatches are automatically transferred from S3 to BigQuery on a daily basis
• Complete Data: All events and dispatches from your account are included in the transfers
• Efficient Querying: BigQuery's SQL-based querying engine allows for fast, in-place analysis of your data
• Scalable Storage: BigQuery's infrastructure handles large volumes of data efficiently

Data Organization

The data in BigQuery mirrors the S3 structure, organized by:

event_dispatches_rolled_up/
  └── year=YYYY/
      └── month=MM/
          └── day=DD/
events_rolled_up/
  └── year=YYYY/
      └── month=MM/
          └── day=DD/

This partitioning makes it easy to:

• Query specific time periods efficiently
• Process historical data in batches
• Manage data retention policies

Getting Started

To set up the Google BigQuery integration:

1. Contact your account manager to enable the integration
2. Follow the Ours Privacy S3 Setup Guide

Data Format

The data transferred to BigQuery contains your complete event and dispatch data, including:

• Event names and properties
• Dispatch details and status
• User information
• Timestamps
• All associated metadata

Best Practices

• Ensure your BigQuery dataset has appropriate access policies
• Consider setting up partitioning and clustering for optimal query performance
• Use BigQuery's built-in analytics capabilities to analyze your data
• Take advantage of the partitioning structure for efficient querying
• Monitor your BigQuery usage and costs