How to Switch Between gcloud Configurations

Managing multiple Google Cloud projects and environments becomes much easier when you know how to switch between gcloud configurations. This tutorial will walk you through creating, configuring, and activating different gcloud configurations, allowing you to quickly move between development, staging, and production environments without manually resetting project settings each time.

For anyone preparing for the Professional Data Engineer certification, understanding how to switch between gcloud configurations is essential. You'll frequently need to work across multiple GCP projects, whether you're building data pipelines in a development environment, testing in staging, or managing production workloads. This skill saves time and reduces errors caused by accidentally running commands in the wrong project.

What You'll Accomplish

By the end of this tutorial, you'll be able to create multiple gcloud configurations, set project-specific properties for each, and switch between them instantly. You'll set up configurations for different environments that include project IDs, default compute zones, and regions. This approach eliminates the need to remember and reset these values every time you change contexts.

Prerequisites and Requirements

Before you begin, make sure you have Google Cloud SDK installed on your local machine or access to Cloud Shell. You'll need a Google Cloud account with access to at least two projects (or permission to create them). Basic familiarity with the command line interface helps. You should have Project Editor or Owner role on the projects you'll be working with.

Estimated time to complete: 20 minutes

Why gcloud Configurations Matter

Without configurations, you'd need to manually run commands like gcloud config set project every time you switch contexts. When working as a data engineer across multiple Google Cloud projects, this becomes tedious and error-prone.

A genomics research lab might maintain separate projects for patient data processing, research analytics, and public datasets. A mobile gaming studio manages development, QA, and production environments for their player analytics pipeline. A solar energy company has distinct projects for real-time monitoring, historical analysis, and predictive maintenance workloads.

In each case, switching between projects dozens of times per day becomes inefficient. Configurations solve this problem by bundling all relevant settings together.

Step 1: Authenticate with Google Cloud

Before you can use functional gcloud commands, you must authenticate. The gcloud CLI needs permission to access your Google Cloud resources.

Run the following command:

gcloud auth login

This command opens a browser window prompting you to log in with your Google account. Once you complete the authentication flow, the CLI stores your credentials locally. If you're working in Cloud Shell, you're already authenticated and can skip this step.

Verify your authentication by checking which account is active:

gcloud auth list

You should see your email address marked as active with an asterisk.

Step 2: Set Your Initial Default Project

The gcloud CLI requires a default project before executing commands that interact with GCP resources. Set your initial project with:

gcloud config set project PROJECT_ID

Replace PROJECT_ID with your actual project ID. For example:

gcloud config set project solar-monitor-dev-2024

The command returns a confirmation message showing the updated property. This sets the project for your default configuration, which gcloud automatically creates when you first install the SDK.

Step 3: Create Your First Named Configuration

Now you'll create a dedicated configuration for a specific environment. Let's create a development configuration for our solar energy monitoring platform example.

Run this command:

gcloud config configurations create dev-environment

The command creates a new configuration named "dev-environment" and automatically activates it. You'll see a message confirming the creation and activation.

When you create a configuration, it starts empty. You need to populate it with the settings relevant to that environment.

Step 4: Configure Properties for the Development Environment

Set the project ID for this configuration:

gcloud config set project solar-monitor-dev-2024

Set the default compute zone where your development resources run:

gcloud config set compute/zone us-central1-a

Set the default compute region:

gcloud config set compute/region us-central1

These settings now apply whenever you activate the dev-environment configuration. When you launch Compute Engine instances, deploy Dataflow jobs, or provision other regional resources, they'll default to these locations unless you explicitly override them.

Step 5: Create a Production Configuration

Create a second configuration for your production environment:

gcloud config configurations create prod-environment

Configure the production project settings:

gcloud config set project solar-monitor-prod-2024
gcloud config set compute/zone us-east1-b
gcloud config set compute/region us-east1

Notice that the production environment uses a different region. Many organizations separate development and production geographically for isolation and compliance reasons.

Step 6: View All Your Configurations

List all configurations to see what you've created:

gcloud config configurations list

This command displays a table showing all configurations, their associated properties, and which one is currently active (marked with IS_ACTIVE set to True). You'll see columns for NAME, IS_ACTIVE, ACCOUNT, PROJECT, DEFAULT_ZONE, and DEFAULT_REGION.

How to Switch Between gcloud Configurations

To switch to a different configuration, use the activate command:

gcloud config configurations activate dev-environment

This immediately changes your active configuration to dev-environment. All subsequent gcloud commands will use the project, zone, and region associated with that configuration.

Switch to production:

gcloud config configurations activate prod-environment

Now your commands operate in the production context. This is the core skill for managing multiple Google Cloud environments efficiently.

Step 7: Verify Your Active Configuration

Check which configuration is currently active and view its properties:

gcloud config configurations describe prod-environment

This shows all properties set for the specified configuration. Alternatively, view just the active configuration's properties:

gcloud config list

This command displays the account, project, zone, and region for whichever configuration is currently active. Run this command before executing critical operations to confirm you're working in the correct environment.

Real-World Application Examples

Let's look at how different organizations use gcloud configurations in practice.

Example 1: Telehealth Platform Data Engineering

A telehealth platform maintains four separate GCP projects: a development project for building new data pipelines, a staging project that mirrors production for testing, a production project handling live patient appointment data, and a research project with de-identified data for analytics.

Their data engineer creates four configurations:

gcloud config configurations create telehealth-dev
gcloud config configurations create telehealth-staging
gcloud config configurations create telehealth-prod
gcloud config configurations create telehealth-research

Each configuration includes the appropriate project ID, plus zone and region settings that match where BigQuery datasets and Dataflow pipelines run. When the engineer needs to deploy a new data transformation pipeline, they activate telehealth-dev, test the code, then switch to telehealth-staging for validation before finally activating telehealth-prod for deployment.

Example 2: Freight Logistics Multi-Region Setup

A freight logistics company processes shipment tracking data in multiple regions to reduce latency. They maintain separate projects for their US and European operations, each with development and production environments.

Their configurations might include:

gcloud config configurations create us-dev
gcloud config set project freight-track-us-dev
gcloud config set compute/region us-central1

gcloud config configurations create us-prod
gcloud config set project freight-track-us-prod
gcloud config set compute/region us-central1

gcloud config configurations create eu-dev
gcloud config set project freight-track-eu-dev
gcloud config set compute/region europe-west1

gcloud config configurations create eu-prod
gcloud config set project freight-track-eu-prod
gcloud config set compute/region europe-west1

The engineer switches between configurations based on which region and environment they're deploying to, ensuring data pipeline components land in the correct Google Cloud project and region.

Example 3: Video Streaming Service Cost Management

A video streaming service uses separate projects to manage costs and isolate workloads. They have distinct projects for data ingestion, processing, storage, and analytics. Each data engineer on the team works across all these projects but focuses on specific components.

They create configurations like:

gcloud config configurations create streaming-ingest
gcloud config configurations create streaming-process
gcloud config configurations create streaming-analytics

When working on the ingestion pipeline that receives viewer event data, they activate streaming-ingest. When optimizing BigQuery queries for viewer insights, they switch to streaming-analytics. This separation helps with billing analysis and resource organization.

Additional Configuration Properties

You can set other useful properties in your configurations beyond project, zone, and region.

Set a default dataset for BigQuery operations:

gcloud config set bigquery/dataset viewer_events

Set the default format for command output:

gcloud config set core/format json

Disable prompts for confirmation:

gcloud config set core/disable_prompts true

View all available properties you can configure:

gcloud config list --all

Common Issues and Troubleshooting

Configuration Not Found Error

If you see an error that a configuration doesn't exist, verify the name with:

gcloud config configurations list

Configuration names are case-sensitive. Make sure you're using the exact name you created.

Permission Denied When Switching Projects

If you get permission errors after switching configurations, verify that your authenticated account has access to the project:

gcloud projects get-iam-policy PROJECT_ID --flatten="bindings[].members" --filter="bindings.members:user:your-email@example.com"

You need at least the Viewer role to list resources, and appropriate roles like Editor or specific service roles to create or modify resources.

Wrong Configuration Active

If commands execute in the wrong project, double-check your active configuration:

gcloud config configurations list

The IS_ACTIVE column shows which configuration is currently in use. Activate the correct one before proceeding.

Configuration Properties Not Taking Effect

Some commands allow you to override configuration defaults with explicit flags. If a command doesn't use your configured region, check whether you're passing flags like --region that override the configuration.

Best Practices for Configuration Management

Use descriptive configuration names that clearly indicate their purpose. Names like "prod-environment" or "eu-prod" are more maintainable than "config1" or "test".

Always verify your active configuration before running commands that create, modify, or delete resources. A quick gcloud config configurations list check prevents costly mistakes.

Document your configurations in your team's runbook or wiki. Include the project IDs, regions, and purposes for each configuration so team members understand the setup.

Consider setting different output formats for different configurations. For example, your development configuration might use table format for readability, while automation configurations use JSON for parsing.

Keep your production configurations secure. If your machine is shared or has multiple users, be cautious about storing credentials for production projects. Use service accounts with minimal necessary permissions for automated operations.

Integration with Other GCP Services

Your gcloud configurations affect how you interact with all Google Cloud services. When you run commands for BigQuery, Cloud Storage, Dataflow, or Pub/Sub, they inherit the project setting from your active configuration.

For example, when you create a BigQuery dataset:

bq mk --dataset viewer_analytics

The dataset is created in whichever project your active gcloud configuration specifies. The same applies to Cloud Storage buckets, Pub/Sub topics, and Dataflow jobs.

This integration extends to the Cloud Console as well. When you run gcloud app browse or gcloud compute ssh, these commands use your configuration's project context to determine which resources to access.

For Dataflow pipelines, your configured region determines where the workers run by default:

gcloud dataflow jobs run job-name --gcs-location gs://dataflow-templates/latest/Word_Count --parameters inputFile=gs://bucket/input.txt,output=gs://bucket/output

Without specifying --region, this uses the region from your active configuration.

Advanced Configuration Techniques

You can temporarily override a configuration property without switching configurations by setting environment variables. For example:

CLOUDSDK_CORE_PROJECT=temporary-project-id gcloud compute instances list

This runs a single command in a different project context without changing your active configuration.

You can also chain configuration switches in scripts:

gcloud config configurations activate dev-environment && \
gcloud dataflow jobs run dev-job --gcs-location gs://template-location && \
gcloud config configurations activate prod-environment

This pattern activates a configuration, runs a command, then switches back. However, be cautious with this approach because if the middle command fails, you might end up in an unexpected configuration state.

Monitoring and Cost Management

Different configurations often correspond to different billing accounts or cost centers. Use labels and configuration names that align with your organization's cost tracking structure.

When working with configurations tied to different projects, you can quickly check spending across them:

gcloud config configurations activate dev-environment
gcloud billing projects describe PROJECT_ID

gcloud config configurations activate prod-environment
gcloud billing projects describe PROJECT_ID

This helps you verify that each project is linked to the correct billing account.

Security Considerations

Configurations store sensitive information including project IDs and default settings. They don't store credentials directly, but they reference which account to use.

Review your stored credentials periodically:

gcloud auth list

Revoke credentials you no longer need:

gcloud auth revoke ACCOUNT_EMAIL

For production configurations, consider using service accounts instead of personal accounts. Activate a service account with:

gcloud auth activate-service-account --key-file=path/to/key.json

Then create a configuration that uses this service account for production operations.

Deleting Configurations

If you no longer need a configuration, remove it with:

gcloud config configurations delete config-name

This deletes the configuration and all its property settings. It does not affect the actual GCP project or resources, only your local configuration.

Next Steps and Enhancements

Now that you can switch between gcloud configurations, consider these enhancements.

Integrate configuration switches into your deployment scripts and CI/CD pipelines. This ensures automated processes use the correct environment context.

Look at using gcloud configurations with Terraform or other infrastructure-as-code tools. You can set environment variables that tell these tools which GCP project to target based on your active configuration.

Learn about the gcloud interactive mode, which provides auto-completion and inline documentation. It respects your active configuration while making command construction easier.

Investigate using configuration properties to customize behavior for tools like kubectl when working with Google Kubernetes Engine across multiple projects.

Wrapping Up

You've learned how to switch between gcloud configurations to manage multiple Google Cloud projects and environments efficiently. You created named configurations, set project-specific properties including project IDs, zones, and regions, and practiced activating different configurations to change your working context instantly.

This skill is fundamental for data engineers working across development, staging, and production environments. You'll use this daily when building data pipelines, deploying Dataflow jobs, managing BigQuery datasets, or orchestrating workflows across multiple GCP projects.

The ability to quickly switch contexts without manually resetting properties reduces errors and saves significant time. Whether you're managing a telehealth platform's patient data pipelines, processing logistics tracking information across regions, or optimizing video streaming analytics, configurations keep your work organized and efficient.

For preparation beyond gcloud configuration management, including hands-on practice with BigQuery, Dataflow, Pub/Sub, and other essential services, check out the Professional Data Engineer course.