GCP IAM Authentication and Authorization 101

Stefanie Lai

Dec 20, 2020·8 min read

Just a few days ago, on 14th Dec., Google IAM experienced a 50-minute outage🔥 and resulted in the unavailability of all the user verification services that rely on it worldwide. Youtube alone lost 1.7 million💸 in advertising revenue, while the losses of other websites were utterly incalculable.

• What is Google IAM?
• Why does this matter so much?
• How much do you know about Authentication and Authorization on GCP?
• How do you align GCP IAM with Kubernetes RBAC?

What’s a User

Traditional web systems often have independent user designs, storing data containing basic user information, such as ages, names, birthdays, emails, phone numbers, etc., in the database. The most important data is user passwords, using which users log in to the website after inputting the username.

There are a wider variety of users in the cloud era, though the user definition has not changed much. Google users are no longer limited to Google only. Many websites have integrated Google’s third-party login now, and by obtaining the users’ basic information after user authorization, they enable users to log in directly.

It brings much convenience.

• It is no longer required to build an independent user management system. Relying on a giant like Google and its huge user base, you can save resources and expand the users’ scope.
• Reduce security vulnerabilities. The data leakage on the Internet and its consequences of losing user information are not rare. Meanwhile, the loss caused by the imprudent management of ordinary user passwords (59% of users always use the same set of passwords) is unpredictable.
• The authenticity and validity of user information are higher. The reliability of Google account information is generally assumed to be much higher than many websites(users tend to fill in false information).

But don’t forget the disadvantages. Having witnessed the serious consequences caused by Google IAM’s outage, you can imagine how crashing you would be if you are completely dependent on Google’s SRE to fulfill your obligations.🙏

GCP IAM?

Before talking about GCP IAM, you need to understand Kubernetes RBAC. And if not, please read Kubernetes Authentication and Authorization 101.

IAM (Identity and Access Management), by understanding it, you will know how to answer the following questions on GCP.

• Who are you?
• What can you do?

Identities

The User term on GCP is similar to that of Kubernetes. Identities on GCP can also be categorized into ordinary users and Google Service accounts.

The usual sources of ordinary users are:

• Google accounts
• Service accounts belong to your applications
• Google groups (collection of google accounts and service accounts), like group:[group-name]@iam.spotify.com
• G suite domains
• Cloud identity domains

Google Service Account(GSA), used by APP on GKE or VM running code on your behalf.

• User-managed GSA: for example, [service-account-name]@[project-id].iam.gserviceaccount.com. And, you can choose the service account name.
• Default GSA: commonly seen, it automatically creates a default GSA(default@[project-id].iam.gserviceaccount.com) when you use GCP services.
• Google-managed GSA: used internally on GCP. Normally users don’t need to deal with them.

GSA uses private/public RSA key-pairs to authenticate with Google.

IAM roles

IAM roles are encapsulations of various GCP resource use permissions. Generally, they can be divided into three categories.

• Primitive roles: Owner, Editor, Viewer. Roles defined in advance when creating the GCP project and will not be assigned to ordinary users.
• Predefined roles: Roles for different resources, which are similar to Kubernetes Roles, such as compite, gke, network, etc.,
• Custom roles: Roles that are customized and user-maintained and are for more granular access.

Role Bindings

It forms a policy combining identity with roles, and then IAM assigns the policy to different users.

The process is almost the same as the Rolebindings of Kubernetes RBAC, but GCP Rolebindings are not only cluster-wide and namespace-wide, but also have IAM hierarchy as well. From the top to the bottom is organization -> folder -> project -> resource.

Authentication

After understanding IAM’s definition, let’s try to figure out how to define users and authenticate them.

Usually, when you join the company, you will get your own Gmail mailbox (Here, take Google as an example, not against Microsoft fans) and be added to various Google groups within the company, as shown below.

(It includes internal and external groups.)

Then you can easily login into the browser. But if you are to operate GKE on the command line and use kubectl, then you are required to use gcloud on the CLI for authentication. Suppose you already have a GCP project and a new GKE cluster.

• Install Google SDK.
• Enable GCP Kubernetes API.
• Set up gcloud config. Usually, there are two ways to do it.

run gcloud init, and follow the guide prompt, config your CLI step by step.

Or log in first, gcloud auth login, then run gcloud config set to change items by the flag, such as zone, region, project-id. This is more useful when you only want to update one or two items, such as changing to another project in the same region, then you can add the flag-project=xxx.

• Add your Google credential into kubeconfig so that you can run kubectl against the GKE cluster. Run command gcloud container clusters get-credentials {{cluster-name}}

To make all this work, you need to make sure your google identity has at least container.clusters.get permission.

After all the steps, you can run kubectl in CLI finally. But how?

OpenID Token

The reason why users can directly use kubectl to complete Kubernetes APIServer authentication after passing cloud provider authentication is OpenID Connect.

Simply put, the cloud provider gives an ID Token to your local kubeconfig via OAuth2, then you can complete user authentication with this token in the same way as the APIServer bearer token. For more about the bearer token, refer to Kubernetes Authentication and Authorization 101.

The flow chart is as follows.

When you have configured the above gcloud correctly, you can see something similar to the following when you open the local ~/.kube/config file.

When executing the kubectl command, the token in the user corresponding to the cluster in kubeconfig will be automatically put into the HTTP header requested to pass APIServer verification. You can also set the desired token via token flag.

Authorization

It’s time to clear your mind about the GCP authorization mechanism. I believe you are familiar with Kubernetes RBAC.

Except for the identities mentioned above, GCP has its own Roles and Rolebindings.

We can apply IAM configurations to the clusters through YAML by using Google Config Connector. See the example below:

There is one IAMPolicyMember and one IAMPolicy in the YAML.

• IAMPolicyMember member-test gives the storage.admin role to a GSA(serviceAccount:test@project-test.iam.gserviceaccount.com).
• IAMPolicy policy-test gives two roles, compute.disk.get and compute.disk.create to the same GSA.

To be noted, the GCP role works on specific resources. Here in the example is the Google cloud project project-test. It can also be other kinds of Google cloud resources, such as BigTable, PubSubTopic, etc.

Custom Role

Sometimes in order to follow the security principle of least privilege, we need to create custom roles since the default roles are not enough.

Bespoke a Custom Role is so easy with Config Connector. For instance, to avoid grant PubSubTopicadmin role, define a custom role with reader and sender permissions, and bind it to the GSA.

External User

I have encountered a special situation where we need to assign permissions to people other than the GCP default owner. To implement it, we need to assign a reasonable GSA to an external user or group.

You can assign a GSA with the predefined role roles/iam.serviceAccountAdmin or
roles/iam.serviceAccountTokenCreator. And the app using this GSA can automatically create related GSA with external user identities and apply it to the cluster, and automatically grant these users the authority to operate on your cluster.

You can fetch all IAM roles by running a gcloud command gcloud iam roles list. I haven’t found a document online containing all the current IAM roles for all the resources on GCP.

How to align IAM with RBAC

After comprehending the fundamental concepts of RBAC and IAM, it’s time to figure out how to use them in the GKE cluster.

• Is there any conflict?
• What’s the difference?
• Do they align with each other?

These questions haunted me for a while at the beginning of my learning Kubernetes and using GCP. By answering them, I quickly absorbed the concepts and took them into practice.

Difference

The essence of either IAM or RBAC is the same, binding the permissions of the resources they manage to respective users.

They can cooperate without interference in a cluster, and manage their resources, respectively, requiring users to configure Kubernetes user (ServiceAccount) and Google IAM identity at the same time. Absolutely, it is not easy, requiring more configuration files. So Google proposed a relatively simplified solution.IAM workload identity.

The key is to make a Kubernetes service account act as a Google service account in the format of serviceAccount:project-id.svc.id.goog[k8s-namespace/ksa-name]. ksa-name: Kubernetes Service Account name.

In simple, if you bind a Kubernetes Service Account with a Google Service Account, and then the Pod attached to this Kubernetes Service Account can access the corresponding GCP resources.

Again, it could be set in UI, CLI, and done by Config Connector. I’ll show you the Config Connector way, which is also the YAML way.

The above default Kubernetes Service Account will have the authority to use any GCP resources bind to the IAMServiceAccount.

Remember, there are limitations to workload identity, and here are some important ones.

• Only one pool per GCP project.
• GKE metadata server will be enabled, and compute metadata will be affected if you’re using compute VM.
• Enabling it may cause serval seconds outage on the Pod.

IAMServiceAccountKey

There is another way to use GSA in Kubernetes resources, that is using IAMServiceAccountKey. Take a look at the following example.

• First, define GSA and IAMPolicyMember.

• Then, define IAMServiceAccountKey.

• Use the IAMServiceAccountKey in a Deployment.

This is also a workaround for using client authentication in Kubernetes resources, passing APIServer verification by injecting the key directly into the Deployment. For more details, please refer to Google documentation.

Keep in mind, abusing GSA and IAMServiceAccountKey is a terrible practice. Some best practices about using GSA and its credentials are,

• Always use IAMServiceAccountKey or token, and never use a secret key directly in the code.
• Use different GSA in different contexts .
• Always add an expiration date on the keys.
• Clean up unused keys.

In the end

When using GKE, Authentication and Authorization are hard to understand and use in the beginning especially when it’s entangled with RBAC. But things get better once you have fully understood the concepts and the difference between them.

The critical point is always the resource. Every resource has a set of roles or APIs. When you figure out how to find them quickly and correctly, then you’re good to go.

Thanks for reading!