NCC Demo & PoC Guide v6

(*** Draft Version - Work in Progress - mostly complete ***)

Last updated: July 08,2025

Intro

This guide will help you build NCC for testing and demonstration purposes. The goal is make it simple and quick to use gcloud commands to build a static, isolated environment.

Diagram - What you'll build

Future Enhancements

For admin use - 2025-04(Apr) Future Enhancements - NCC Demo & PoC Guide 

How to use the guide

Suggestions

• To be safe, start by building this as an isolated environment in its own project with its own VPCs and no networking to reach the other projects / VPCs.

Pre-requisites

• Permissions to:

• Create and operate a project
• Create and operate lots of networking and network security such as VPCs, subnets, firewall rules, Cloud NAT, Cloud Router and NCC hub and spokes
• Create and operate VMs for testing
• Create and operate Cloud SQL instances and BigQuery datasets

• Resources that the script creates that could exceed quotas depending on your specific quotas:

• 12 Cloud Routers (referenced below how to request additional quota)
• 6 VPCs

• Org policies:

• Some org policies are set in the script. See below in section 1
• Other org policies are not set, though eg:

• region1 and region2 need to be allowed. See below in Variables section for which regions)

Build notes

Steps:

• Create a project
• Complete the variables below
• For the gcloud script below, copy/paste/run larger or smaller sections as needed, eg the entirety of section #1 can be run all at once.

• If there's an error, the script will stop and no further commands will run, so you'll need to find where the script failed, fix the error (likely a variable value that is not accepted) and copy the rest of script to re-run
• Alternatively, you can copy and paste each individual command or a few commands in a section, especially if you want to view the output.
• Note: if you copy selectively, make sure to not copy &&\ on the last command copied since &&\ tells the shell to expect another command

Variables

Click on each variable to add values which will customize the gcloud commands below to fit your environment. Note there are some variables to be entered in later sections after the components have been created.

#1- gcloud script - Add resources

# RUN THE FIRST FEW COMMANDS INDIVIDUALLY

# Set project

gcloud config set project  &&\

# Enable APIs and org policy

# From Enabling services

gcloud services enable bigquery.googleapis.com cloudquotas.googleapis.com compute.googleapis.com dns.googleapis.com networkconnectivity.googleapis.com sqladmin.googleapis.com servicenetworking.googleapis.com &&\

#From

gcloud resource-manager org-policies disable-enforce sql.restrictAuthorizedNetworks --project= &&\

echo "constraint: constraints/compute.restrictVpnPeerIPs

listPolicy:

  allValues: ALLOW" > policy.yaml && \

gcloud resource-manager org-policies set-policy --project= policy.yaml && \

# If needed, request quota increase for Cloud Routers (need 12) in this project

# Note, this command is in Preview. From Request a quota increase adjustment when a quota preference hasn't been set yet. Can also request additional quota in the GCP console in IAM & Admin > Quotas & System Limits:

gcloud beta quotas preferences create --preferred-value=15 --quota-id=ROUTERS-per-project --service=compute.googleapis.com --project= --billing-project= --email= --justification=testing && \

# How to find quota-id for routers: gcloud beta quotas info list --service=compute.googleapis.com --project= --billing-project= | grep compute.googleapis.com/routers -A 8

# quota-id is after the last "/" in name

# START HERE IF COPYING IN BULK

# Estimated time to complete the script for this section: 19+ min

# Add NCC hub

gcloud network-connectivity hubs create ncchub --description="ncc hub created from gcloud" --export-psc --preset-topology=star &&\

# Add VPCs and subnets

gcloud compute networks create vpc1 --bgp-routing-mode=global --subnet-mode=custom &&\

gcloud compute networks subnets create subnet11 --network=vpc1 --enable-flow-logs --enable-private-ip-google-access --range=10.11.11.0/24 --region= &&\

gcloud compute networks subnets create subnet12 --network=vpc1 --enable-flow-logs --enable-private-ip-google-access --range=10.12.12.0/24 --region= &&\

gcloud compute networks create vpc2 --bgp-routing-mode=global --subnet-mode=custom &&\

gcloud compute networks subnets create subnet21 --network=vpc2 --enable-flow-logs --enable-private-ip-google-access --range=10.21.21.0/24 --region= &&\

gcloud compute networks subnets create subnet22 --network=vpc2 --enable-flow-logs --enable-private-ip-google-access --range=10.22.22.0/24 --region= &&\

gcloud compute networks create vpc3 --bgp-routing-mode=global --subnet-mode=custom &&\

gcloud compute networks subnets create subnet31 --network=vpc3 --enable-flow-logs --enable-private-ip-google-access --range=10.31.31.0/24 --region= &&\

gcloud compute networks subnets create subnet32 --network=vpc3 --enable-flow-logs --enable-private-ip-google-access --range=10.32.32.0/24 --region= &&\

gcloud compute networks create vpc4 --bgp-routing-mode=global --subnet-mode=custom &&\

gcloud compute networks subnets create subnet41 --network=vpc4 --enable-flow-logs --enable-private-ip-google-access --range=10.41.41.0/24 --region= &&\

gcloud compute networks subnets create subnet42 --network=vpc4 --enable-flow-logs --enable-private-ip-google-access --range=10.42.42.0/24 --region= &&\

gcloud compute networks create vpc-routing --bgp-routing-mode=global --subnet-mode=custom &&\

gcloud compute networks subnets create subnet-routing1 --network=vpc-routing --enable-flow-logs --enable-private-ip-google-access --range=10.1.1.0/24 --region= &&\

gcloud compute networks subnets create subnet-routing2 --network=vpc-routing --enable-flow-logs --enable-private-ip-google-access --range=10.2.2.0/24 --region= &&\

gcloud compute networks create vpcdc --bgp-routing-mode=global --subnet-mode=custom &&\

gcloud compute networks subnets create subnet110 --network=vpcdc --enable-flow-logs --enable-private-ip-google-access --range=10.110.110.0/24 --region= &&\

gcloud compute networks subnets create subnet120 --network=vpcdc --enable-flow-logs --enable-private-ip-google-access --range=10.120.120.0/24 --region=

# Add VPN tunnels between routing VPC and DC VPC

# From Create HA VPN gateways to connect VPC networks > Create the HA VPN gateways

gcloud compute vpn-gateways create havpngw-vpc-routing --network=vpc-routing --region= --stack-type=IPV4_ONLY &&\

gcloud compute vpn-gateways create havpngw-vpcdc --network=vpcdc --region= --stack-type=IPV4_ONLY &&\

# From  > Create Cloud Routers

gcloud compute routers create cldrtr-havpngw-vpc-routing --region= --network=vpc-routing --asn=65500 --advertisement-mode=CUSTOM --set-advertisement-groups=ALL_SUBNETS &&\

gcloud compute routers create cldrtr-havpngw-vpcdc --region= --network=vpcdc --asn=65501 --advertisement-mode=CUSTOM --set-advertisement-groups=ALL_SUBNETS &&\

# From  > Create VPN Tunnels

gcloud compute vpn-tunnels create tunnel0-havpngw-vpc-routing --peer-gcp-gateway=havpngw-vpcdc --region= --ike-version=2 --shared-secret=secret --router=cldrtr-havpngw-vpc-routing --vpn-gateway=havpngw-vpc-routing --interface=0 &&\

gcloud compute vpn-tunnels create tunnel1-havpngw-vpc-routing --peer-gcp-gateway=havpngw-vpcdc --region= --ike-version=2 --shared-secret=secret --router=cldrtr-havpngw-vpc-routing --vpn-gateway=havpngw-vpc-routing --interface=1 &&\

gcloud compute vpn-tunnels create tunnel0-havpngw-vpcdc --peer-gcp-gateway=havpngw-vpc-routing --region= --ike-version=2 --shared-secret=secret --router=cldrtr-havpngw-vpcdc --vpn-gateway=havpngw-vpcdc --interface=0 &&\

gcloud compute vpn-tunnels create tunnel1-havpngw-vpcdc --peer-gcp-gateway=havpngw-vpc-routing --region= --ike-version=2 --shared-secret=secret --router=cldrtr-havpngw-vpcdc --vpn-gateway=havpngw-vpcdc --interface=1 &&\

# From  > Create BGP sessions

gcloud compute routers add-interface cldrtr-havpngw-vpc-routing --interface-name=rtrint-tunnel0-havpngw-vpc-routing --ip-address=169.254.0.1 --mask-length=30 --vpn-tunnel=tunnel0-havpngw-vpc-routing --region=  &&\

gcloud compute routers add-bgp-peer cldrtr-havpngw-vpc-routing --peer-name=peer-tunnel0-havpngw-vpc-routing --interface=rtrint-tunnel0-havpngw-vpc-routing --peer-ip-address=169.254.0.2 --peer-asn=65501 --region= &&\

gcloud compute routers add-interface cldrtr-havpngw-vpc-routing --interface-name=rtrint-tunnel1-havpngw-vpc-routing --ip-address=169.254.1.1 --mask-length=30 --vpn-tunnel=tunnel1-havpngw-vpc-routing --region=  &&\

gcloud compute routers add-bgp-peer cldrtr-havpngw-vpc-routing --peer-name=peer-tunnel1-havpngw-vpc-routing --interface=rtrint-tunnel1-havpngw-vpc-routing --peer-ip-address=169.254.1.2 --peer-asn=65501 --region= &&\

gcloud compute routers add-interface cldrtr-havpngw-vpcdc --interface-name=rtrint-tunnel0-havpngw-vpcdc --ip-address=169.254.0.2 --mask-length=30 --vpn-tunnel=tunnel0-havpngw-vpcdc --region=  &&\

gcloud compute routers add-bgp-peer cldrtr-havpngw-vpcdc --peer-name=peer-tunnel0-havpngw-vpcdc --interface=rtrint-tunnel0-havpngw-vpcdc --peer-ip-address=169.254.0.1 --peer-asn=65500 --region= &&\

gcloud compute routers add-interface cldrtr-havpngw-vpcdc --interface-name=rtrint-tunnel1-havpngw-vpcdc --ip-address=169.254.1.2 --mask-length=30 --vpn-tunnel=tunnel1-havpngw-vpcdc --region=  &&\

gcloud compute routers add-bgp-peer cldrtr-havpngw-vpcdc --peer-name=peer-tunnel1-havpngw-vpcdc --interface=rtrint-tunnel1-havpngw-vpcdc --peer-ip-address=169.254.1.1 --peer-asn=65500 --region= &&\

# Add NCC Spokes

# Lesson learned: How to dynamically advertise routes outside of GCP in NCC? => "To automatically advertise VPC spoke subnet IP address ranges to hybrid spokes, use the --include-import-ranges flag with the ALL_IPV4_RANGES field during spoke creation. By default, the --include-import-ranges field is empty, which means that no hub subnets are imported to new or existing hybrid spokes until the ALL_IPV4_RANGES is specified." Source: Network Connectivity Center overview > Import of hub subnets for hybrid spokes

gcloud network-connectivity spokes linked-vpc-network create nccspk-vpc1 --hub=projects//locations/global/hubs/ncchub --description="ncc spoke vpc1 created from gcloud" --vpc-network=projects//global/networks/vpc1 --global --group=edge &&\

gcloud network-connectivity spokes linked-vpc-network create nccspk-vpc2 --hub=projects//locations/global/hubs/ncchub --description="ncc spoke vpc2 created from gcloud" --vpc-network=projects//global/networks/vpc2 --global --group=edge &&\

gcloud network-connectivity spokes linked-vpc-network create nccspk-vpc3 --hub=projects//locations/global/hubs/ncchub --description="ncc spoke vpc3 created from gcloud" --vpc-network=projects//global/networks/vpc3 --global --group=center &&\

gcloud network-connectivity spokes linked-vpc-network create nccspk-vpc4 --hub=projects//locations/global/hubs/ncchub --description="ncc spoke vpc4 created from gcloud" --vpc-network=projects//global/networks/vpc4 --global --group=center &&\

gcloud network-connectivity spokes linked-vpn-tunnels create nccspk-vpn-in-vpc-routing --hub=projects//locations/global/hubs/ncchub --description="NCC hybrid spoke VPN created from gcloud" --vpn-tunnels=projects//regions//vpnTunnels/tunnel0-havpngw-vpc-routing,projects//regions//vpnTunnels/tunnel1-havpngw-vpc-routing --region= --include-import-ranges=ALL_IPV4_RANGES --group=center &&\

# Add Cloud Router & Cloud NAT for VMs to load packages, 1 per subnet except for routing VPC

# No diagram since for this document Cloud NAT is only to download packages on VMs

# Create Cloud Routers

gcloud compute routers create cldrtr-vpc1- --project= --network=vpc1 --region= && \

gcloud compute routers create cldrtr-vpc2- --project= --network=vpc2 --region= && \

gcloud compute routers create cldrtr-vpc3- --project= --network=vpc3 --region= && \

gcloud compute routers create cldrtr-vpc4- --project= --network=vpc4 --region= && \

gcloud compute routers create cldrtr-vpcdc- --project= --network=vpcdc --region= && \

gcloud compute routers create cldrtr-vpc1- --project= --network=vpc1 --region= && \

gcloud compute routers create cldrtr-vpc2- --project= --network=vpc2 --region= && \

gcloud compute routers create cldrtr-vpc3- --project= --network=vpc3 --region= && \

gcloud compute routers create cldrtr-vpc4- --project= --network=vpc4 --region= && \

gcloud compute routers create cldrtr-vpcdc- --project= --network=vpcdc --region= && \

# Create Cloud NAT

gcloud compute routers nats create pubnatgw-cldrtr-vpc1- --router=cldrtr-vpc1- --region= --nat-all-subnet-ip-ranges --auto-allocate-nat-external-ips && \

gcloud compute routers nats create pubnatgw-cldrtr-vpc2- --router=cldrtr-vpc2- --region= --nat-all-subnet-ip-ranges --auto-allocate-nat-external-ips && \

gcloud compute routers nats create pubnatgw-cldrtr-vpc3- --router=cldrtr-vpc3- --region= --nat-all-subnet-ip-ranges --auto-allocate-nat-external-ips && \

gcloud compute routers nats create pubnatgw-cldrtr-vpc4- --router=cldrtr-vpc4- --region= --nat-all-subnet-ip-ranges --auto-allocate-nat-external-ips && \

gcloud compute routers nats create pubnatgw-cldrtr-vpcdc- --router=cldrtr-vpcdc- --region= --nat-all-subnet-ip-ranges --auto-allocate-nat-external-ips && \

gcloud compute routers nats create pubnatgw-cldrtr-vpc1- --router=cldrtr-vpc1- --region= --nat-all-subnet-ip-ranges --auto-allocate-nat-external-ips && \

gcloud compute routers nats create pubnatgw-cldrtr-vpc2- --router=cldrtr-vpc2- --region= --nat-all-subnet-ip-ranges --auto-allocate-nat-external-ips && \

gcloud compute routers nats create pubnatgw-cldrtr-vpc3- --router=cldrtr-vpc3- --region= --nat-all-subnet-ip-ranges --auto-allocate-nat-external-ips && \

gcloud compute routers nats create pubnatgw-cldrtr-vpc4- --router=cldrtr-vpc4- --region= --nat-all-subnet-ip-ranges --auto-allocate-nat-external-ips && \

gcloud compute routers nats create pubnatgw-cldrtr-vpcdc- --router=cldrtr-vpcdc- --region= --nat-all-subnet-ip-ranges --auto-allocate-nat-external-ips && \

# Add test VMs, 1 per subnet except for routing VPC

# Note about accessing BigQuery from a GCE VM, hence the need for the --scopes parameter on GCE VMs below: "Note that the level of access that a service account has is determined by a combination of access scopes and IAM roles so you must configure both access scopes and IAM roles for the service account to work properly." Source: gcloud compute instances create > --scopes

gcloud compute instances create testvm--c-vpc1 --zone=-c --machine-type=e2-micro --shielded-secure-boot --metadata=startup-script=sudo\ apt-get\ update$'\n'sudo\ apt-get\ install\ -y\ postgresql-client$'\n'sudo\ apt-get\ install\ -y\ dnsutils,enable-oslogin=true --network-interface=private-network-ip=10.11.11.2,subnet=subnet11,no-address --scopes=bigquery &&\

gcloud compute instances create testvm--c-vpc1 --zone=-c --machine-type=e2-micro --shielded-secure-boot --metadata=startup-script=sudo\ apt-get\ update$'\n'sudo\ apt-get\ install\ -y\ postgresql-client$'\n'sudo\ apt-get\ install\ -y\ dnsutils,enable-oslogin=true --network-interface=private-network-ip=10.12.12.2,subnet=subnet12,no-address --scopes=bigquery &&\

gcloud compute instances create testvm--c-vpc2 --zone=-c --machine-type=e2-micro --shielded-secure-boot --metadata=startup-script=sudo\ apt-get\ update$'\n'sudo\ apt-get\ install\ -y\ postgresql-client$'\n'sudo\ apt-get\ install\ -y\ dnsutils,enable-oslogin=true --network-interface=private-network-ip=10.21.21.2,subnet=subnet21,no-address --scopes=bigquery &&\

gcloud compute instances create testvm--c-vpc2 --zone=-c --machine-type=e2-micro --shielded-secure-boot --metadata=startup-script=sudo\ apt-get\ update$'\n'sudo\ apt-get\ install\ -y\ postgresql-client$'\n'sudo\ apt-get\ install\ -y\ dnsutils,enable-oslogin=true --network-interface=private-network-ip=10.22.22.2,subnet=subnet22,no-address --scopes=bigquery &&\

gcloud compute instances create testvm--c-vpc3 --zone=-c --machine-type=e2-micro --shielded-secure-boot --metadata=startup-script=sudo\ apt-get\ update$'\n'sudo\ apt-get\ install\ -y\ postgresql-client$'\n'sudo\ apt-get\ install\ -y\ dnsutils,enable-oslogin=true --network-interface=private-network-ip=10.31.31.2,subnet=subnet31,no-address --scopes=bigquery &&\

gcloud compute instances create testvm--c-vpc3 --zone=-c --machine-type=e2-micro --shielded-secure-boot --metadata=startup-script=sudo\ apt-get\ update$'\n'sudo\ apt-get\ install\ -y\ postgresql-client$'\n'sudo\ apt-get\ install\ -y\ dnsutils,enable-oslogin=true --network-interface=private-network-ip=10.32.32.2,subnet=subnet32,no-address --scopes=bigquery &&\

gcloud compute instances create testvm--c-vpc4 --zone=-c --machine-type=e2-micro --shielded-secure-boot --metadata=startup-script=sudo\ apt-get\ update$'\n'sudo\ apt-get\ install\ -y\ postgresql-client$'\n'sudo\ apt-get\ install\ -y\ dnsutils,enable-oslogin=true --network-interface=private-network-ip=10.41.41.2,subnet=subnet41,no-address --scopes=bigquery &&\

gcloud compute instances create testvm--c-vpc4 --zone=-c --machine-type=e2-micro --shielded-secure-boot --metadata=startup-script=sudo\ apt-get\ update$'\n'sudo\ apt-get\ install\ -y\ postgresql-client$'\n'sudo\ apt-get\ install\ -y\ dnsutils,enable-oslogin=true --network-interface=private-network-ip=10.42.42.2,subnet=subnet42,no-address --scopes=bigquery &&\

gcloud compute instances create testvm--c-vpcdc --zone=-c --machine-type=e2-micro --shielded-secure-boot --metadata=startup-script=sudo\ apt-get\ update$'\n'sudo\ apt-get\ install\ -y\ postgresql-client$'\n'sudo\ apt-get\ install\ -y\ dnsutils,enable-oslogin=true --network-interface=private-network-ip=10.110.110.2,subnet=subnet110,no-address --scopes=bigquery &&\

gcloud compute instances create testvm--c-vpcdc --zone=-c --machine-type=e2-micro --shielded-secure-boot --metadata=startup-script=sudo\ apt-get\ update$'\n'sudo\ apt-get\ install\ -y\ postgresql-client$'\n'sudo\ apt-get\ install\ -y\ dnsutils,enable-oslogin=true --network-interface=private-network-ip=10.120.120.2,subnet=subnet120,no-address --scopes=bigquery &&\

# Add network firewall policy

gcloud compute network-firewall-policies create fw-policy --description DESCRIPTION --global &&\

# Add firewall policy rule - allow IAP IP range

# Same image as above

gcloud compute network-firewall-policies rules create 100 --action allow --firewall-policy fw-policy --direction INGRESS --src-ip-ranges=35.235.240.0/20 --layer4-configs all --global-firewall-policy &&\

# Add firewall policy rule - allow VMs to reach each other

# Same image as above

gcloud compute network-firewall-policies rules create 200 --action allow --firewall-policy fw-policy --direction INGRESS --src-ip-ranges=10.0.0.0/8 --layer4-configs all --global-firewall-policy &&\

# Link firewall policy to VPCs

# Same image as above

gcloud compute network-firewall-policies associations create --firewall-policy fw-policy --network vpc1 --global-firewall-policy &&\

gcloud compute network-firewall-policies associations create --firewall-policy fw-policy --network vpc2 --global-firewall-policy &&\

gcloud compute network-firewall-policies associations create --firewall-policy fw-policy --network vpc3 --global-firewall-policy &&\

gcloud compute network-firewall-policies associations create --firewall-policy fw-policy --network vpc4 --global-firewall-policy &&\

gcloud compute network-firewall-policies associations create --firewall-policy fw-policy --network vpcdc --global-firewall-policy

# STOP COPYING HERE - SECTION 2, 3 and 4 COMMANDS ARE TO BE RUN INDIVIDUALLY OR IN SMALLER GROUPS OF COMMANDS

#2- Network Reachability

# RUN THE COMMANDS IN THIS SECTION INDIVIDUALLY

Ping script for reachability testing in VPC1 (SSH to each VM in VPC1 to run)

ping -c 5 10.11.11.2 && ping -c 5 10.12.12.2 &&\

ping -c 5 10.31.31.2 && ping -c 5 10.32.32.2 &&\

ping -c 5 10.41.41.2 && ping -c 5 10.42.42.2 &&\

ping -c 5 10.110.110.2 && ping -c 5 10.120.120.2

Note no ping to VMs in VPC2. VPCs 1 & 2 are in Edge Group so they cannot reach each other.

Ping script for reachability testing in VPC2 (SSH to each VM in VPC2 to run)

ping -c 5 10.21.21.2 && ping -c 5 10.22.22.2 &&\

ping -c 5 10.31.31.2 && ping -c 5 10.32.32.2 &&\

ping -c 5 10.41.41.2 && ping -c 5 10.42.42.2 &&\

ping -c 5 10.110.110.2 && ping -c 5 10.120.120.2

Note no ping to VMs in VPC1. VPCs 1 & 2 are in Edge Group so they cannot reach each other.

Ping script for reachability testing in other VPCs (SSH to each VM to run)

ping -c 5 10.11.11.2 && ping -c 5 10.12.12.2 &&\

ping -c 5 10.21.21.2 && ping -c 5 10.22.22.2 &&\

ping -c 5 10.31.31.2 && ping -c 5 10.32.32.2 &&\

ping -c 5 10.41.41.2 && ping -c 5 10.42.42.2 &&\

ping -c 5 10.110.110.2 && ping -c 5 10.120.120.2

#3- PSC Cloud SQL Reachability

# RUN THE COMMANDS IN THIS SECTION IN SMALLER GROUPS AND INDIVIDUALLY

# Add PSC-based Cloud SQL Instance

# From Create a Cloud SQL instance

gcloud sql instances create cldsql-postgres-psc --database-version=POSTGRES_16 --project= --region= --enable-private-service-connect --allowed-psc-projects= --availability-type=REGIONAL --no-assign-ip --tier=db-g1-small --root-password= --edition=ENTERPRISE && \

# From Get the service attachment

gcloud sql instances describe cldsql-postgres-psc --project= | grep pscServiceAttachmentLink

# Record the PSC service attachment URI from the previous step here:  

# From Create a Private Service Connect Endpoint

gcloud compute addresses create ipaddr-psce-cldsql-postgres-psc --project= --region= --subnet=subnet41 && \

gcloud compute forwarding-rules create psce-cldsql-postgres --address=ipaddr-psce-cldsql-postgres-psc --project= --region= --network=vpc4 --target-service-attachment= --allow-psc-global-access && \

# List Compute Engine service account:

gcloud iam service-accounts list | grep EMAIL:

# Record the Compute Engine service account from the previous step here:  

# Give compute engine service account permissions to view Cloud SQL (to run psql commands in next section)

gcloud projects add-iam-policy-binding  --member=serviceAccount: --role=roles/cloudsql.admin && \

# Note: You may be able to use a lower level of permissions.

# Test connection to PSC-based Cloud SQL

# Get PSC Endpoint IP address:

gcloud compute addresses describe ipaddr-psce-cldsql-postgres-psc --project= --region= | grep address:

Record the IP address from the previous step                        here:  

# Before next step, note: With current setup, you will *not* be able to reach Cloud SQL from VPC DC:

• "If you need to make propagated Private Service Connect connections available to on-premises networks through the hub's hybrid spokes, the hub's single routing VPC network must also be connected as a VPC spoke"

• The current limit is 1 VPC Spoke that also contains hybrid spokes.
• Reference VPC spokes overview >  Limitations of Dynamic Route Exchange

# SSH to any VM other than VMs in VPC DC (see note above) and run:

psql -h  -p 5432 -U postgres

# Enter password:

PSQL commands: https://hasura.io/blog/top-psql-commands-and-flags-you-need-to-know-postgresql 

• List all databases - \l

• List database tables - \dt

• From https://www.w3schools.com/postgresql/postgresql_create_table.php, create a table -

CREATE TABLE cars (

  brand VARCHAR(255),

  model VARCHAR(255),

  year INT

);

• List database tables - \dt

• Quit psql - \q

# Add NCC Spoke - Make Routing VPC also a VPC Spoke

# In order to pass PSC Endpoints for Published Services (VPC-based managed services) to on-prem, the NCC Routing VPC (has hybrid connections or router appliances) must also be a VPC Spoke. See https://cloud.google.com/network-connectivity/docs/network-connectivity-center/concepts/vpc-spokes-overview#limitations-of-route-exchange.

gcloud network-connectivity spokes linked-vpc-network create nccspk-vpc-routing --hub=projects//locations/global/hubs/ncchub --description="ncc spoke vpcrouting created from gcloud" --vpc-network=projects//global/networks/vpc-routing --global --group=center

# Reach Cloud SQL from VPC-DC (on-prem) - Routing VPC to VPC Spoke

# SSH to a VM in vpcdc and run:

psql -h  -p 5432 -U postgres

# Enter password:

#4- PSA Cloud SQL

# RUN THE COMMANDS IN THIS SECTION IN SMALLER GROUPS AND INDIVIDUALLY

# Add PSA-based Cloud SQL Instance

# From Configure private services access > Create an IP allocation

gcloud compute addresses create psa-range --global --purpose=VPC_PEERING --addresses=172.16.0.0 --prefix-length=16 --description="Private Services Access (PSA) range for Cloud SQL, created by gcloud" --network=vpc3

# From Configure private services access > Create a private connection

gcloud services vpc-peerings connect --service=servicenetworking.googleapis.com --ranges=psa-range --network=vpc3

# From Create a Cloud SQL instance

gcloud sql instances create cldsql-postgres-psa --database-version=POSTGRES_16 --project= --region= --network=vpc3 --availability-type=REGIONAL --no-assign-ip --tier=db-g1-small --root-password= --edition=ENTERPRISE

# Test 1 connection to PSA-based Cloud SQL - Limited reachability

# Get PSA-based Cloud SQL IP address:

gcloud sql instances describe cldsql-postgres-psa | grep ipAddress:

Record the IP address from the previous step                   here:  

# Before next step, note: Current setup has not yet enabled reachability to PSA-based Cloud SQL from other VPCs than just vpc3. The next step will enable reachability from other VPCs.

# SSH to a VM in vpc3 and run:

psql -h  -p 5432 -U postgres

# Enter password:

# Go to section above, # Test connection to PSC-based Cloud SQL, for specific psql commands to run once connected from the VM to the Cloud SQL instance

# Add NCC Spoke - VPC Producer Spoke for PSA-based Cloud SQL in VPC3

gcloud network-connectivity spokes linked-producer-vpc-network create nccprodspk-vpc3 --hub=projects//locations/global/hubs/ncchub --description="ncc VPC producer spoke vpc3 created from gcloud" --network=projects//global/networks/vpc3 --peering=servicenetworking-googleapis-com --global --group=center

# Estimated time to complete this command: 1+ min

# Test 2 connection to PSA-based Cloud SQL - Full reachability

# SSH to *any* VM and run:

psql -h  -p 5432 -U postgres

# Enter password:

# Go to section above, # Test connection to PSC-based Cloud SQL, for specific psql commands to run once connected from the VM to the Cloud SQL instance

#5- PSC BQ

# START HERE IF COPYING IN BULK

# Add BQ dataset

bq --location= mk --dataset \:test && \

# Give compute engine service account permissions to BigQuery (to run command gcloud sql instances describe)

gcloud projects add-iam-policy-binding  --member=serviceAccount: --role=roles/bigquery.admin && \

# Note: You may be able to use a lower level of permissions.

# Add PSC Endpoint for BQ per VPC

# From Create an endpoint

# The following group of commands are for vpc1 to privately reach bigquery

gcloud compute addresses create ipaddr-psce-bq-vpc1 --global --purpose=PRIVATE_SERVICE_CONNECT --addresses=192.168.1.1 --network=vpc1 && \

gcloud compute forwarding-rules create pscebqvpc1 --global --network=vpc1 --address=ipaddr-psce-bq-vpc1 --target-google-apis-bundle=all-apis && \

# If you'd like the rest of the VPCs to be able to privately reach bigquery, run the rest of the commands in this section

gcloud compute addresses create ipaddr-psce-bq-vpc2 --global --purpose=PRIVATE_SERVICE_CONNECT --addresses=192.168.1.2 --network=vpc2 && \

gcloud compute forwarding-rules create pscebqvpc2 --global --network=vpc2 --address=ipaddr-psce-bq-vpc2 --target-google-apis-bundle=all-apis && \

gcloud compute addresses create ipaddr-psce-bq-vpc3 --global --purpose=PRIVATE_SERVICE_CONNECT --addresses=192.168.1.3 --network=vpc3 && \

gcloud compute forwarding-rules create pscebqvpc3 --global --network=vpc3 --address=ipaddr-psce-bq-vpc3 --target-google-apis-bundle=all-apis && \

gcloud compute addresses create ipaddr-psce-bq-vpc4 --global --purpose=PRIVATE_SERVICE_CONNECT --addresses=192.168.1.4 --network=vpc4 && \

gcloud compute forwarding-rules create pscebqvpc4 --global --network=vpc4 --address=ipaddr-psce-bq-vpc4 --target-google-apis-bundle=all-apis && \

gcloud compute addresses create ipaddr-psce-bq-vpc-routing --global --purpose=PRIVATE_SERVICE_CONNECT --addresses=192.168.1.100 --network=vpc-routing && \

gcloud compute forwarding-rules create pscebqvpcrouting --global --network=vpc-routing --address=ipaddr-psce-bq-vpc-routing --target-google-apis-bundle=all-apis && \

# DNS for BQ (and other googleapis.com services)

# The following group of commands are for vpc1 to privately reach bigquery

gcloud dns managed-zones create googleapis-com-vpc1 --dns-name googleapis.com --visibility private --networks=vpc1 --description="" && \

gcloud dns record-sets create private.googleapis.com. --rrdatas 192.168.1.1 --type A --ttl 60 --zone googleapis-com-vpc1 && \

gcloud dns record-sets create bigquery.googleapis.com. --rrdatas private.googleapis.com. --type CNAME --ttl 60 --zone googleapis-com-vpc1 && \

# Note: instead of creating a CNAME record for bigquery.googleapis.com., you could instead create a CNAME for *.googleapis.com. which would resolve all googleapis.com, not just BigQuery. Since this test is just for BigQuery, we add it specifically.

# If you'd like the rest of the VPCs to be able to privately reach bigquery, run the rest of the commands in this section

gcloud dns managed-zones create googleapis-com-vpc2 --dns-name googleapis.com --visibility private --networks=vpc2 --description="" && \

gcloud dns record-sets create private.googleapis.com. --rrdatas 192.168.1.2 --type A --ttl 60 --zone googleapis-com-vpc2 && \

gcloud dns record-sets create bigquery.googleapis.com. --rrdatas private.googleapis.com. --type CNAME --ttl 60 --zone googleapis-com-vpc2 && \

gcloud dns managed-zones create googleapis-com-vpc3 --dns-name googleapis.com --visibility private --networks=vpc3 --description="" && \

gcloud dns record-sets create private.googleapis.com. --rrdatas 192.168.1.3 --type A --ttl 60 --zone googleapis-com-vpc3 && \

gcloud dns record-sets create bigquery.googleapis.com. --rrdatas private.googleapis.com. --type CNAME --ttl 60 --zone googleapis-com-vpc3 && \

gcloud dns managed-zones create googleapis-com-vpc4 --dns-name googleapis.com --visibility private --networks=vpc4 --description="" && \

gcloud dns record-sets create private.googleapis.com. --rrdatas 192.168.1.4 --type A --ttl 60 --zone googleapis-com-vpc4 && \

gcloud dns record-sets create bigquery.googleapis.com. --rrdatas private.googleapis.com. --type CNAME --ttl 60 --zone googleapis-com-vpc4 && \

gcloud dns managed-zones create googleapis-com-vpcdc --dns-name googleapis.com --visibility private --networks=vpcdc --description="" && \

gcloud dns record-sets create private.googleapis.com. --rrdatas 192.168.1.100 --type A --ttl 60 --zone googleapis-com-vpcdc && \

gcloud dns record-sets create bigquery.googleapis.com. --rrdatas private.googleapis.com. --type CNAME --ttl 60 --zone googleapis-com-vpcdc && \

# Note: DNS in vpcdc is for simulation only. DNS over a hybrid connection to your own on-prem would be done differently with DNS forwarding zones and/or DNS inbound policies.

# Add routing on vpc-routing Cloud Router to PSC Endpoints for reachability from vpcdc (simulated on-prem DC)

gcloud compute routers update cldrtr-havpngw-vpc-routing --region= --add-advertisement-ranges=192.168.1.0/24="private path to BigQuery / Google APIs"

# STOP HERE IF COPYING ALL - RUN REMAINING STEPS INDIVIDUALLY

# Test privately reaching BQ from VPCs

# Verify that bigquery.googleapis.com is directed to the PSC Endpoints (in range 192.168.1.x) by SSHing to a VM and running:

nslookup bigquery.googleapis.com

# Output should look like this (last Address should be in range 192.168.1.x):

Server:         127.0.0.53

Address:        127.0.0.53#53

Non-authoritative answer:

bigquery.googleapis.com canonical name = private.googleapis.com.

Name:   private.googleapis.com

Address: 192.168.1.1   

# Note: the bq command that follows uses the API bigquery.googleapis.com

# List BigQuery datasets by running:

bq ls

# Output should look like this:

datasetId  

 -----------

  test_1    

# Optional: Test privately reaching BQ from VPCs after deleting some PSC Endpoints

…to verify that private reachability is removed

END OF CREATING RESOURCES

#6- Delete resources

# START HERE IF COPYING IN BULK

# Delete PSC Endpoint for BQ per

# From Create an endpoint

# The following group of commands are for vpc1-1 to privately reach bigquery

gcloud compute forwarding-rules delete pscebqvpc1 pscebqvpc2 pscebqvpc3 pscebqvpc4 pscebqvpcrouting --global --quiet && \

gcloud compute addresses delete ipaddr-psce-bq-vpc1 ipaddr-psce-bq-vpc2 ipaddr-psce-bq-vpc3 ipaddr-psce-bq-vpc4 ipaddr-psce-bq-vpc-routing --global --quiet && \

# DNS for BQ (and other googleapis.com services)

# The following group of commands are for vpc1 to privately reach bigquery

gcloud dns record-sets delete private.googleapis.com. --type A --zone googleapis-com-vpc1 && \

gcloud dns record-sets delete bigquery.googleapis.com. --type CNAME --zone googleapis-com-vpc1 && \

gcloud dns managed-zones delete googleapis-com-vpc1 && \

gcloud dns record-sets delete private.googleapis.com. --type A --zone googleapis-com-vpc2 && \

gcloud dns record-sets delete bigquery.googleapis.com. --type CNAME --zone googleapis-com-vpc2 && \

gcloud dns managed-zones delete googleapis-com-vpc2 && \

gcloud dns record-sets delete private.googleapis.com. --type A --zone googleapis-com-vpc3 && \

gcloud dns record-sets delete bigquery.googleapis.com. --type CNAME --zone googleapis-com-vpc3 && \

gcloud dns managed-zones delete googleapis-com-vpc3 && \

gcloud dns record-sets delete private.googleapis.com. --type A --zone googleapis-com-vpc4 && \

gcloud dns record-sets delete bigquery.googleapis.com. --type CNAME --zone googleapis-com-vpc4 && \

gcloud dns managed-zones delete googleapis-com-vpc4 && \

gcloud dns record-sets delete private.googleapis.com. --type A --zone googleapis-com-vpcdc && \

gcloud dns record-sets delete bigquery.googleapis.com. --type CNAME --zone googleapis-com-vpcdc && \

gcloud dns managed-zones delete googleapis-com-vpcdc && \

# Delete BQ dataset

# Remove compute engine service account permissions to BigQuery (to run command gcloud sql instances describe)

gcloud projects remove-iam-policy-binding  --member=serviceAccount: --role=roles/bigquery.admin && \

# Delete dataset

bq --location= rm --dataset --force=true \:test && \

# Delete NCC Spoke and PSA-based Cloud SQL Instance

gcloud network-connectivity spokes delete nccprodspk-vpc3 --global --quiet &&\

gcloud sql instances delete cldsql-postgres-psa --quiet && \

gcloud compute addresses delete psa-range --global --quiet && \

# Delete NCC Spoke and PSC-based Cloud SQL Instance

gcloud network-connectivity spokes delete nccspk-vpc-routing --global --quiet &&\

# Give compute engine service account permissions to view Cloud SQL (to run psql commands in next section)

gcloud projects remove-iam-policy-binding  --member=serviceAccount: --role=roles/cloudsql.admin && \

# From &&\

gcloud compute forwarding-rules delete psce-cldsql-postgres --region= --quiet && \

gcloud compute addresses delete ipaddr-psce-cldsql-postgres-psc --region= --quiet && \

# From &&\

gcloud sql instances delete cldsql-postgres-psc --quiet && \

# Unlink firewall policy to VPC Spokes

gcloud compute network-firewall-policies associations delete --firewall-policy fw-policy --name network-vpc1 --global-firewall-policy &&\

gcloud compute network-firewall-policies associations delete --firewall-policy fw-policy --name network-vpc2 --global-firewall-policy &&\

gcloud compute network-firewall-policies associations delete --firewall-policy fw-policy --name network-vpc3 --global-firewall-policy &&\

gcloud compute network-firewall-policies associations delete --firewall-policy fw-policy --name network-vpc4 --global-firewall-policy &&\

gcloud compute network-firewall-policies associations delete --firewall-policy fw-policy --name network-vpcdc --global-firewall-policy &&\

# Delete network firewall policy

gcloud compute network-firewall-policies delete fw-policy --global &&\

# Delete test VMs

gcloud compute instances delete testvm--c-vpc1 testvm--c-vpc2 testvm--c-vpc3 testvm--c-vpc4 testvm--c-vpcdc --zone=-c --quiet &&\

gcloud compute instances delete testvm--c-vpc1 testvm--c-vpc2 testvm--c-vpc3 testvm--c-vpc4 testvm--c-vpcdc --zone=-c --quiet &&\

# Delete Cloud NAT & Cloud Router

gcloud compute routers nats delete pubnatgw-cldrtr-vpc1- --router=cldrtr-vpc1- --region= --quiet && \

gcloud compute routers nats delete pubnatgw-cldrtr-vpc2- --router=cldrtr-vpc2- --region=  --quiet && \

gcloud compute routers nats delete pubnatgw-cldrtr-vpc3- --router=cldrtr-vpc3- --region=  --quiet && \

gcloud compute routers nats delete pubnatgw-cldrtr-vpc4- --router=cldrtr-vpc4-  --region=  --quiet && \

gcloud compute routers nats delete pubnatgw-cldrtr-vpcdc- --router=cldrtr-vpcdc- --region= --quiet && \

gcloud compute routers nats delete pubnatgw-cldrtr-vpc1- --router=cldrtr-vpc1- --region= --quiet && \

gcloud compute routers nats delete pubnatgw-cldrtr-vpc2- --router=cldrtr-vpc2- --region= --quiet && \

gcloud compute routers nats delete pubnatgw-cldrtr-vpc3- --router=cldrtr-vpc3- --region= --quiet && \

gcloud compute routers nats delete pubnatgw-cldrtr-vpc4- --router=cldrtr-vpc4- --region= --quiet && \

gcloud compute routers nats delete pubnatgw-cldrtr-vpcdc- --router=cldrtr-vpcdc- --region= --quiet && \

gcloud compute routers delete cldrtr-vpc1- cldrtr-vpc2- cldrtr-vpc3- cldrtr-vpc4- cldrtr-vpcdc- --region= --quiet && \

gcloud compute routers delete cldrtr-vpc1- cldrtr-vpc2- cldrtr-vpc3- cldrtr-vpc4- cldrtr-vpcdc- --region= --quiet && \

# Delete Most NCC Spokes

gcloud network-connectivity spokes delete nccspk-vpc1 --global --quiet &&\

gcloud network-connectivity spokes delete nccspk-vpc2 --global --quiet &&\

gcloud network-connectivity spokes delete nccspk-vpc3 --global --quiet &&\

gcloud network-connectivity spokes delete nccspk-vpc4 --global --quiet &&\

gcloud network-connectivity spokes delete nccspk-vpn-in-vpc-routing --region= --quiet &&\

# Delete NCC hub

gcloud network-connectivity hubs delete ncchub --quiet &&\

# Delete VPN tunnels

# Must delete NCC spokes before this step

gcloud compute vpn-tunnels delete tunnel0-havpngw-vpc-routing tunnel1-havpngw-vpc-routing tunnel0-havpngw-vpcdc tunnel1-havpngw-vpcdc  --region= --quiet &&\

gcloud compute routers delete cldrtr-havpngw-vpc-routing cldrtr-havpngw-vpcdc --region= --quiet &&\

gcloud compute vpn-gateways delete havpngw-vpc-routing havpngw-vpcdc --region= --quiet &&\

# Delete VPCs and subnets

gcloud compute networks subnets delete subnet11 subnet21 subnet31 subnet41 subnet-routing1 subnet110 --region= --quiet &&\

gcloud compute networks subnets delete subnet12 subnet22 subnet32 subnet42 subnet-routing2 subnet120 --region= --quiet &&\

gcloud compute networks delete vpc1 vpc2 vpc3 vpc4 vpc-routing vpcdc --quiet

# STOP COPYING HERE

END OF DELETING RESOURCES