We have a Spark master node on a Google Cloud Dataproc cluster, which we want to connect by Hostname and NOT the Internal IP. We want to connect/ping these VMs from one another. Rationale: When we drop/create any of the VMs/Clusters, the internal IP changes, but we dont want to change bunch of connection strings everytime.
GCLOUD command line lists the Master node VM:
vn524i0@m-c02zf1nylvdt ~$ gcloud compute instances list | grep anvil
anvil-dataproc-m us-east1-a custom-16-65536 10.22.162.40 RUNNING
From another GCP VM (in same region) when I try to ping the VM using the internal IP, I am able to (so ICMP is enabled):
vn524i0@m-c02zf1nylvdt ~$ ping 10.22.162.40 -c 1
PING 10.22.162.40 (10.22.162.40): 56 data bytes
64 bytes from 10.22.162.40: icmp_seq=0 ttl=56 time=140.232 ms
--- 10.22.162.40 ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
But when I try to ping by hostname, I get Unknown Host error:
vn524i0@m-c02zf1nylvdt ~$ ping anvil-dataproc-m
ping: cannot resolve anvil-dataproc-m: Unknown host
I’ve followed the global & Zonal DNS guide and used the hostname pattern suggested by Google as per this page: https://cloud.google.com/compute/docs/internal-dns#zonal-dnsblonde-hair