Cannot ping by hostname but can ping by IP - Google Cloud

We have a Spark master node on a Google Cloud Dataproc cluster, which we want to connect by Hostname and NOT the Internal IP. We want to connect/ping these VMs from one another. Rationale: When we drop/create any of the VMs/Clusters, the internal IP changes, but we dont want to change bunch of connection strings everytime.

GCLOUD command line lists the Master node VM:

vn524i0@m-c02zf1nylvdt ~$ gcloud compute instances list | grep anvil
anvil-dataproc-m           us-east1-a  custom-16-65536               10.22.162.40                RUNNING

From another GCP VM (in same region) when I try to ping the VM using the internal IP, I am able to (so ICMP is enabled):

vn524i0@m-c02zf1nylvdt ~$ ping 10.22.162.40 -c 1
PING 10.22.162.40 (10.22.162.40): 56 data bytes
64 bytes from 10.22.162.40: icmp_seq=0 ttl=56 time=140.232 ms

--- 10.22.162.40 ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss

But when I try to ping by hostname, I get Unknown Host error:

vn524i0@m-c02zf1nylvdt ~$ ping anvil-dataproc-m
ping: cannot resolve anvil-dataproc-m: Unknown host

I’ve followed the global & Zonal DNS guide and used the hostname pattern suggested by Google as per this page: https://cloud.google.com/compute/docs/internal-dns#zonal-dnsblonde-hair