Deploy Cloud Composer DAGs Across Projects with Build
Master cross-project DAG deployment for Cloud Composer using Cloud Build automation. This tutorial covers CI/CD pipelines, permissions, and production-ready deployment strategies.
Learning how to deploy Cloud Composer DAGs across projects is essential for Professional Data Engineer exam candidates. This tutorial demonstrates building an automated deployment pipeline using Cloud Build that can distribute Directed Acyclic Graphs (DAGs) from a central repository to Cloud Composer environments running in different Google Cloud projects. By the end of this guide, you'll have a working CI/CD system that automatically deploys DAGs to multiple Cloud Composer instances whenever code changes are committed.
This deployment pattern matters because organizations commonly separate their environments into different GCP projects for development, staging, and production. You need a reliable, automated way to promote DAGs through these environments without manual file copying. The approach covered here creates a single source of truth for your orchestration code while maintaining proper isolation between projects.
Why Cross-Project DAG Deployment Matters
Many organizations structure their Google Cloud infrastructure with separate projects for different purposes. A data engineering team might maintain Cloud Composer environments in a dev project, a staging project, and multiple production projects serving different business units. Managing DAG deployments across these environments manually becomes error-prone and time-consuming.
Automating DAG deployments across projects provides several advantages. You establish consistent deployment processes, reduce human error, enable faster iteration cycles, and create audit trails for compliance requirements. For the Professional Data Engineer certification, understanding how to architect these multi-project deployments demonstrates knowledge of production-ready data orchestration patterns.
Prerequisites and Requirements
Before starting this tutorial, you'll need at least two GCP projects with billing enabled (source and target projects). Cloud Composer environments should already be created in your target projects. You'll need Owner or Editor permissions on both projects, or specific IAM roles including Cloud Composer Administrator and Cloud Build Editor. Install and configure Google Cloud SDK locally, and make sure you have Git for version control. You should have a basic understanding of Apache Airflow DAG structure. Plan for 45 to 60 minutes to complete this tutorial.
You'll work with Cloud Source Repositories or GitHub for source control, Cloud Build for automation, and Cloud Storage buckets that back your Cloud Composer environments.
Architecture Overview
The implementation uses Cloud Build as the deployment orchestrator. When you commit DAG files to a source repository, Cloud Build triggers automatically. The build process copies your DAG files to the Cloud Storage bucket associated with your target Cloud Composer environment. Cloud Composer monitors this bucket and automatically loads new or updated DAG files into the Airflow scheduler.
For cross-project deployments, Cloud Build runs in your source project but needs permission to write to Cloud Storage buckets in your target projects. This requires configuring service account permissions across project boundaries. The build configuration file defines which DAGs deploy to which environments based on branch names or directory structures.
Step 1: Prepare Your Source Repository
Create a repository structure that organizes DAGs by environment or project. This structure helps manage which DAGs deploy where.
First, create a new directory for your project and initialize Git:
mkdir composer-multi-project
cd composer-multi-project
git init
Create a directory structure separating DAGs by environment:
mkdir -p dags/dev
mkdir -p dags/staging
mkdir -p dags/prod
Create a sample DAG file in the dev directory. Save this as dags/dev/sample_etl_dag.py:
from airflow import DAG
from airflow.operators.bash import BashOperator
from datetime import datetime, timedelta
default_args = {
'owner': 'data-engineering',
'depends_on_past': False,
'start_date': datetime(2024, 1, 1),
'email_on_failure': False,
'email_on_retry': False,
'retries': 1,
'retry_delay': timedelta(minutes=5),
}
with DAG(
'sample_etl_pipeline',
default_args=default_args,
description='Sample ETL pipeline for cross-project deployment',
schedule_interval=timedelta(days=1),
catchup=False,
) as dag:
extract_task = BashOperator(
task_id='extract_data',
bash_command='echo "Extracting data from source systems"',
)
transform_task = BashOperator(
task_id='transform_data',
bash_command='echo "Transforming data"',
)
load_task = BashOperator(
task_id='load_data',
bash_command='echo "Loading data to destination"',
)
extract_task >> transform_task >> load_task
This creates a simple DAG that demonstrates the deployment process. In production, your DAGs would contain actual data processing logic using operators for BigQuery, Dataflow, or other Google Cloud services.
Step 2: Configure Cross-Project IAM Permissions
The Cloud Build service account from your source project needs permission to write to Cloud Storage buckets in your target projects. This step establishes the cross-project trust relationship.
First, identify your Cloud Build service account. In your source project, run:
export SOURCE_PROJECT_ID="your-source-project-id"
export SOURCE_PROJECT_NUMBER=$(gcloud projects describe $SOURCE_PROJECT_ID --format="value(projectNumber)")
export CLOUD_BUILD_SA="${SOURCE_PROJECT_NUMBER}@cloudbuild.gserviceaccount.com"
echo $CLOUD_BUILD_SA
Note the service account email that appears. Now grant this service account permissions in your target project. Switch to your target project and grant the necessary role:
export TARGET_PROJECT_ID="your-target-project-id"
export TARGET_COMPOSER_BUCKET="gs://us-central1-your-composer-env-bucket"
gcloud projects add-iam-policy-binding $TARGET_PROJECT_ID \
--member="serviceAccount:${CLOUD_BUILD_SA}" \
--role="roles/composer.admin"
The composer.admin role provides permissions to update the Cloud Composer environment and access its Cloud Storage bucket. For production deployments with stricter security requirements, you can create a custom role with only the specific permissions needed to write to the DAGs folder in Cloud Storage.
Verify the permission was granted:
gcloud projects get-iam-policy $TARGET_PROJECT_ID \
--flatten="bindings[].members" \
--filter="bindings.members:${CLOUD_BUILD_SA}"
You should see the Cloud Build service account listed with the composer.admin role.
Step 3: Identify Cloud Composer Bucket Paths
Cloud Composer stores DAGs in a specific folder within its Cloud Storage bucket. You need to identify the exact path for each target environment.
For each Cloud Composer environment, retrieve the bucket information:
gcloud composer environments describe your-composer-env-name \
--location us-central1 \
--project $TARGET_PROJECT_ID \
--format="get(config.dagGcsPrefix)"
This command returns the full GCS path where DAGs should be copied, such as gs://us-central1-your-composer-env-a1b2c3-bucket/dags. Record these paths for each environment where you'll deploy DAGs. You'll reference them in your Cloud Build configuration.
Step 4: Create the Cloud Build Configuration
The Cloud Build configuration file defines the deployment steps. Create a file named cloudbuild.yaml in your repository root:
steps:
# Deploy dev DAGs to dev Composer environment
- name: 'gcr.io/cloud-builders/gsutil'
args:
- '-m'
- 'rsync'
- '-r'
- '-d'
- 'dags/dev/'
- 'gs://us-central1-dev-composer-a1b2c3-bucket/dags/'
id: 'deploy-dev-dags'
# Deploy staging DAGs to staging Composer environment (different project)
- name: 'gcr.io/cloud-builders/gsutil'
args:
- '-m'
- 'rsync'
- '-r'
- '-d'
- 'dags/staging/'
- 'gs://us-central1-staging-composer-d4e5f6-bucket/dags/'
id: 'deploy-staging-dags'
# Deploy prod DAGs to production Composer environment (different project)
- name: 'gcr.io/cloud-builders/gsutil'
args:
- '-m'
- 'rsync'
- '-r'
- '-d'
- 'dags/prod/'
- 'gs://us-central1-prod-composer-g7h8i9-bucket/dags/'
id: 'deploy-prod-dags'
waitFor: ['deploy-staging-dags'] # Only deploy to prod after staging succeeds
options:
logging: CLOUD_LOGGING_ONLY
This configuration uses gsutil rsync to synchronize DAG files. The -m flag enables parallel uploads for better performance. The -r flag syncs recursively, and -d deletes files in the destination that don't exist in the source, keeping environments clean.
The waitFor directive in the production step ensures staging deployment completes successfully before deploying to production. This provides a basic safety gate in your deployment pipeline.
Step 5: Set Up Cloud Build Triggers
Configure Cloud Build to automatically run your deployment pipeline when code changes are pushed to your repository.
First, connect your repository to Cloud Build. If using Cloud Source Repositories:
gcloud source repos create composer-dags --project=$SOURCE_PROJECT_ID
git remote add google \
https://source.developers.google.com/p/$SOURCE_PROJECT_ID/r/composer-dags
Create a Cloud Build trigger that runs on commits to the main branch:
gcloud builds triggers create cloud-source-repositories \
--repo=composer-dags \
--branch-pattern="^main$" \
--build-config=cloudbuild.yaml \
--description="Deploy DAGs to all Composer environments" \
--project=$SOURCE_PROJECT_ID
For GitHub repositories, you can create a trigger through the GCP Console that connects to your GitHub account and repository. The trigger configuration would specify the same branch pattern and build configuration file.
Verify your trigger was created:
gcloud builds triggers list --project=$SOURCE_PROJECT_ID
Step 6: Test the Deployment Pipeline
Commit your DAG files and Cloud Build configuration to trigger the first deployment.
git add .
git commit -m "Initial DAG deployment configuration"
git push google main
This push triggers Cloud Build automatically. Monitor the build progress:
gcloud builds list --project=$SOURCE_PROJECT_ID --limit=5
View detailed logs for the most recent build:
gcloud builds log $(gcloud builds list --project=$SOURCE_PROJECT_ID --limit=1 --format="value(id)") \
--project=$SOURCE_PROJECT_ID
You should see output showing gsutil copying your DAG files to each Cloud Composer bucket. The build should complete with a SUCCESS status.
Step 7: Verify DAG Deployment in Cloud Composer
After Cloud Build completes, verify that your DAGs appear in each Cloud Composer environment. Cloud Composer typically picks up new DAG files within 1 to 2 minutes.
Check if the DAG file exists in the bucket:
gsutil ls gs://us-central1-dev-composer-a1b2c3-bucket/dags/
Access the Airflow web interface for your Cloud Composer environment through the GCP Console. Navigate to Cloud Composer, select your environment, and click the Airflow web server link. In the Airflow UI, you should see your sample_etl_pipeline DAG listed.
If the DAG doesn't appear, check the DAG processor logs in Airflow. Parsing errors in your DAG code prevent it from loading. The Airflow UI shows import errors prominently on the dashboard.
Real-World Application Examples
A subscription meal kit service uses this cross-project deployment pattern to manage data pipelines across regional operations. Their data engineering team maintains DAGs in a central repository. Development DAGs run in a sandbox project where engineers test new data transformations. Staging DAGs in a separate project process recent production data for validation. Production DAGs deploy to three regional projects (US, Europe, Asia) that each run Cloud Composer environments processing orders, inventory, and delivery logistics for their regions.
A telehealth platform deploys Cloud Composer DAGs across projects separated by compliance boundaries. Their HIPAA-compliant production environment runs in an isolated project with strict access controls. DAGs that process patient data deploy only to this project. Meanwhile, DAGs handling operational analytics and business intelligence deploy to a separate analytics project. The Cloud Build pipeline routes DAGs to appropriate environments based on directory structure, ensuring sensitive data workflows never accidentally deploy to non-compliant environments.
A mobile gaming studio manages separate Cloud Composer environments for each game title, with projects organized by game and environment. Their central data platform team maintains common DAG templates for player analytics, revenue reporting, and event processing. Game-specific customizations live in separate directories. When the platform team updates a shared template, Cloud Build deploys the changes to all game projects simultaneously, ensuring consistent data processing standards while allowing game teams to maintain their specialized workflows.
Environment-Specific Configuration
Production DAG deployments often require environment-specific configuration values like BigQuery dataset names, Cloud Storage paths, or API endpoints that differ between dev, staging, and production.
One approach uses Airflow variables that you set directly in each Cloud Composer environment. Your DAG code retrieves these at runtime:
from airflow.models import Variable
bigquery_dataset = Variable.get('etl_target_dataset')
gcs_staging_bucket = Variable.get('staging_bucket')
Set these variables in each environment using gcloud commands or the Airflow UI. This keeps environment-specific values out of your DAG code.
Another approach uses separate configuration files that deploy alongside your DAGs. Your Cloud Build configuration can substitute values during deployment using build variables or separate configuration files per environment stored in different directories.
Branch-Based Deployment Strategy
You can refine the deployment strategy to deploy different branches to different environments. This allows developers to test DAGs in development without affecting production.
Modify your Cloud Build trigger configuration to create separate triggers for different branches:
# Trigger for dev environment from develop branch
gcloud builds triggers create cloud-source-repositories \
--repo=composer-dags \
--branch-pattern="^develop$" \
--build-config=cloudbuild-dev.yaml \
--project=$SOURCE_PROJECT_ID
# Trigger for production environment from main branch
gcloud builds triggers create cloud-source-repositories \
--repo=composer-dags \
--branch-pattern="^main$" \
--build-config=cloudbuild-prod.yaml \
--project=$SOURCE_PROJECT_ID
Create separate build configuration files that deploy to different environments. The cloudbuild-dev.yaml file would only copy DAGs to dev environments, while cloudbuild-prod.yaml handles production deployments.
Common Issues and Troubleshooting
If your build fails with permission errors, verify the Cloud Build service account has the necessary roles in the target project. Use gcloud projects get-iam-policy to confirm the binding exists. Remember that IAM changes can take up to 60 seconds to propagate.
When DAGs don't appear in the Airflow UI after successful deployment, check that you copied files to the correct bucket path. The path must end with /dags. Verify using gsutil ls on the exact path returned by the gcloud composer environments describe command.
If DAGs appear but show import errors, check the Cloud Composer environment's installed Python packages. Your DAG might depend on packages not available in the environment. Add required packages using gcloud composer environments update with the --update-pypi-packages-from-file flag.
Build performance can degrade as you accumulate many DAG files. The gsutil -m rsync command with the -m flag enables parallel transfers which significantly speeds up large deployments. For very large DAG repositories, consider splitting deployments by directory rather than syncing everything in a single step.
Security Considerations for Production
The composer.admin role provides broad permissions. For production systems, create a custom IAM role with minimal permissions. The Cloud Build service account only needs storage.objects.create, storage.objects.delete, and storage.objects.get permissions on the specific Cloud Storage bucket used by Cloud Composer.
Create a custom role in your target project:
gcloud iam roles create composerDagDeployer \
--project=$TARGET_PROJECT_ID \
--title="Composer DAG Deployer" \
--description="Limited permissions for deploying DAGs to Composer" \
--permissions=storage.objects.create,storage.objects.delete,storage.objects.get,storage.objects.list \
--stage=GA
Then grant this custom role on the specific bucket rather than at the project level. This follows the principle of least privilege, reducing the potential impact if the service account credentials are compromised.
Monitoring and Alerting
Set up monitoring to track deployment success and failures. Cloud Build integrates with Cloud Logging and Cloud Monitoring. Create a logs-based metric that counts failed builds:
gcloud logging metrics create composer_deployment_failures \
--description="Count of failed Composer DAG deployments" \
--log-filter='resource.type="cloud_build"
severity="ERROR"
resource.labels.build_trigger_id="your-trigger-id"' \
--project=$SOURCE_PROJECT_ID
Create an alerting policy in Cloud Monitoring that notifies your team when this metric exceeds zero. This ensures deployment failures receive immediate attention.
You can also configure Cloud Build to publish messages to Pub/Sub topics on build completion. Subscribe to these topics to trigger additional workflows, send notifications to Slack or email, or update deployment tracking systems.
Integration with Other Google Cloud Services
This deployment pattern integrates naturally with other GCP services. Store your DAG repository in Cloud Source Repositories for a fully integrated Google Cloud solution. Use Secret Manager to store sensitive credentials that DAGs need, granting the Cloud Composer service account access to specific secrets rather than hardcoding credentials.
For teams using BigQuery extensively, DAGs often reference datasets, tables, and views. Consider deploying your BigQuery schema definitions using similar Cloud Build automation. This creates a complete infrastructure-as-code approach for your data platform.
Dataflow jobs triggered by your DAGs can use templates stored in Cloud Storage. Deploy these templates using the same Cloud Build pipeline that handles your DAGs, ensuring your orchestration code and processing code stay synchronized.
Testing DAGs Before Deployment
Add a testing step to your Cloud Build pipeline that validates DAG syntax before deploying. Create a build step that runs Python syntax checks:
steps:
# Test DAG syntax
- name: 'python:3.8-slim'
entrypoint: 'bash'
args:
- '-c'
- |
pip install apache-airflow==2.5.0
python -m py_compile dags/**/*.py
id: 'test-dag-syntax'
# Deploy only if tests pass
- name: 'gcr.io/cloud-builders/gsutil'
args: ['-m', 'rsync', '-r', '-d', 'dags/prod/', 'gs://prod-bucket/dags/']
waitFor: ['test-dag-syntax']
This prevents deploying DAGs with syntax errors that would fail to parse in Cloud Composer. You can expand testing to include unit tests for custom operators, validation of DAG structure, or linting checks using tools like pylint or flake8.
Cost Optimization
Cloud Build offers a generous free tier with 120 build-minutes per day. DAG deployments typically complete in under one minute, so small to medium teams often stay within free tier limits. For larger teams with frequent deployments, consider using Cloud Build's private pools with preemptible workers to reduce costs.
The gsutil rsync operation only transfers changed files, minimizing Cloud Storage operations costs. If you have hundreds of DAG files but typically change only a few per deployment, rsync efficiently transfers only the differences.
Cloud Composer costs remain unchanged by this deployment method. You're simply automating file transfers that would otherwise happen manually. The automation actually helps optimize costs by enabling better environment management and faster troubleshooting of issues.
Next Steps and Advanced Patterns
After mastering basic cross-project DAG deployment, explore these advanced patterns. Implement approval gates for production deployments by integrating Cloud Build with third-party CI/CD tools or using Cloud Build's manual approval step (currently in preview). This requires human review before deploying to production environments.
Consider implementing canary deployments where new DAG versions initially deploy to a single Cloud Composer worker in production. Monitor metrics and logs before rolling out to all workers. This requires more sophisticated orchestration but provides additional safety for critical production workflows.
Investigate using Terraform or Config Connector to manage Cloud Composer environments themselves as code. Combined with the DAG deployment pipeline covered here, this creates a fully automated data orchestration platform where both infrastructure and application code deploy through version-controlled pipelines.
Explore using Cloud Build to deploy other Airflow components like plugins, custom operators, or configuration files to your Cloud Composer environments using the same cross-project pattern.
Summary
You've built a complete CI/CD pipeline that automatically deploys Cloud Composer DAGs across multiple GCP projects. This system provides automated, reliable DAG deployments from a central source repository to distributed Cloud Composer environments. You configured cross-project IAM permissions, created Cloud Build triggers, and implemented testing and monitoring for your deployment pipeline.
The skills you practiced here are directly applicable to Professional Data Engineer exam scenarios involving multi-environment data orchestration, CI/CD automation for data pipelines, and cross-project resource management in Google Cloud. You've gained hands-on experience with Cloud Build, Cloud Composer, Cloud Storage, and IAM that translates directly to production data engineering work.
For comprehensive preparation covering this topic and all other Professional Data Engineer exam objectives, check out the Professional Data Engineer course. The course provides in-depth coverage of Cloud Composer, Cloud Build, and the complete range of GCP data services you'll encounter on the exam.