Adopt Open ID Connect (OIDC) in Terraform for secure multi-account CI/CD to AWS

In this post we will explore how we can increase our security posture by leveraging Open ID Connect (OIDC) as authentication mechanism in CI/CD pipelines for deploying Terraform based infrastructure to Amazon Web Services (AWS).

The first design principle of the Security Pillar of the AWS Well-Architected Framework sounds as follows:
Implement a strong identity foundation: Implement the principle of least privilege and enforce separation of duties with appropriate authorization for each interaction with your AWS resources. Centralize identity management, and aim to eliminate reliance on long-term static credentials.”

The technical implementation guidance for SEC02-BP02 Use temporary credentials advises to leverage temporary security credentials instead of long-term credentials for all AWS API and CLI requests.

“Avoiding the use of long-term credentials in favor of temporary credentials should go hand in hand with a strategy of reducing the usage of IAM users in favor of federation and IAM roles.

While IAM users have been used for both human and machine identities in the past, we now recommend not using them to avoid the risks in using long-term access keys.”

For automated deployments to AWS with CI/CD solutions such as GitHub, GitLab, Terraform Cloud and so on, IAM users with static access keys where traditionally the way to go, with the hassle of rotating them at a reasonable interval.

Lately, the advent of Open ID Connect (OIDC) has made this both simpler and more secure. This technology effectively eliminates the need for rotation of long-term access keys. However, there’s been some drawbacks for Terraform users using the AWS S3 state backend, which did not support assume role with web identity. Additionally, for the Terraform AWS provider, an IAM Identity Provider had to be configured in every target AWS account.

With the release of Terraform 1.6 assume role with web identity support for the S3 backend was finally included (even though this was not clearly communicated in the release announcement, it’s stated in the changelog: assume_role_with_web_identity nested block for assuming a role with dynamic credentials such as a JSON Web Token. (#31244)).

This means that we now are able to deploy an Identity Provider configuration in a centralized CI/CD tooling account and assume IAM roles to each target workload account!

In this post I’ll take you through the steps on how you can deploy infrastructure with Terraform and GitHub Actions using Open ID Connect (OIDC) and AWS IAM AssumeRoleWithWebIdentity.

Figure 1. Diagram of authentication flow

Each job requests an OIDC token from GitHub’s OIDC provider, which responds with an automatically generated JSON web token (JWT) that is unique for each workflow job where it is generated. When the job runs, the OIDC token is presented to the cloud provider. To validate the token, the cloud provider checks if the OIDC token’s subject and other claims are a match for the conditions that were preconfigured on the cloud role’s OIDC trust definition. For more information see https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/about-security-hardening-with-openid-connect#understanding-the-oidc-token.

A) Create Identity Provider in the tooling account

Normally I provision resources with Terraform, but for the sake of simplicity in this example we’re going to use the CLI, so in your tooling CI/CD account, open AWS Cloudshell.

You can add GitHub as an IdP in your account with a single AWS CLI command. The following code will create an Open ID Connect (OIDC) Provider for GitHub.com with their OIDC thumbprint 938fd4d98bab03faadb97b34396831e3780aea1.

aws iam create-open-id-connect-provider --url "https://token.actions.githubusercontent.com" --thumbprint-list "6938fd4d98bab03faadb97b34396831e3780aea1" --client-id-list 'sts.amazonaws.com'

B) Create IAM roles in the tooling account

In CloudShell: vim trustpolicyforGitHubOIDC.json. Adapt AWS account ID on line 7 and GitHub organization and repo name on line 12 to your use-case. For more details see https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/configuring-openid-connect-in-amazon-web-services#configuring-the-role-and-trust-policy.

Copy the AWS tooling account ID and keep it available in a temporary text document, it’s going to be referenced a few times throughout this setup procedure.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Federated": "arn:aws:iam::<id-of-tooling-account>:oidc-provider/token.actions.githubusercontent.com"  # Replace
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
                "StringEquals": {
                    "token.actions.githubusercontent.com:sub": "repo:organization/repo-name:ref:refs/heads/main",      # Replace
                    "token.actions.githubusercontent.com:aud": "sts.amazonaws.com"
                }
            }
        }
    ]
}

Then create the IAM role for AssumeRoleWithWebIdentity.

aws iam create-role --role-name GitHubAction-AssumeRoleWithAction --assume-role-policy-document file://trustpolicyforGitHubOIDC.json

Create a corresponding IAM role for cicd purposes, to be assumed by the previously created role in the tooling account.

In CloudShell: vim trustpolicyforcicd.json. Adapt AWS account ID on line 8 to your tooling account ID.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "sts:AssumeRole",
            "Principal": {
                "AWS": "arn:aws:iam::<id-of-tooling-account>:role/GitHubAction-AssumeRoleWithAction"
            }
        }
    ]
}

Then create the IAM role for AssumeRole.

aws iam create-role --role-name cicd --assume-role-policy-document file://trustpolicyforcicd.json

C) Create IAM role in workload accounts(s) with corresponding trust policy

This enables the IAM role in the tooling account to assume IAM roles in workload AWS accounts for CI/CD purposes.

In CloudShell in a sample worklod AWS account: vim trustpolicyforcicd.json. Adapt account ID on line 7 to your tooling account ID.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::<id-of-tooling-account>:role/GitHubAction-AssumeRoleWithAction"
            },
            "Action": "sts:AssumeRole"
        }
    ]
}

Then create the IAM role for AssumeRole.

aws iam create-role --role-name cicd --assume-role-policy-document file://trustpolicyforcicd.json

The AWS configurations are now complete. Let’s jump to GitHub.com.

D) Set up GitHub Actions workflow

As with GitHub Action implementations, there are many roads that lead to Rome, but here is a simplified example should work as a starting point. Feel free to adopt to your use-case and change account ID on line 5 to your tooling account ID.

TF_VERSION is your desired Terraform version of choice, but do note that this functionality is only supported from version 1.6.0.

Step Get OIDC Token on line 29 issues a short-lived OIDC token, stores it at the path configured in AWS_WEB_IDENTITY_TOKEN_FILE, configures it for a profile called “cicd” and then issues a request to AWS STS to get-caller-id to verify the profile “cicd” is working.

Sample .github/workflows/tf-cicd-aws.yml

---
name: "Terraform CI/CD to AWS"
env:  
  AWS_REGION: "eu-central-1"
  AWS_ROLE_ARN: "arn:aws:iam::<id-of-tooling-account>:role/GitHubAction-AssumeRoleWithAction"  # Replace
  AWS_WEB_IDENTITY_TOKEN_FILE: "/tmp/web_identity_token_file"
  TF_VERSION: "1.6.0"

"on":
  pull_request:
  push:
    branches:    
      - main

# Permission can be added at job level or workflow level    
permissions:
      id-token: write   # This is required for requesting the JWT
      contents: read    # This is required for actions/checkout

jobs:
  Terraform:
    name: Terraform
    runs-on: ubuntu-latest
    strategy:
      matrix: { dir: ['aws-accounts/tooling', 'aws-accounts/workload-prod-01'] }               # Replace, declare folder(s) to traverse
    #env:
    #  working-directory: aws-accounts/tooling
    steps:
      - name: Get OIDC Token
        id: get_oidc_token
        run: |
          curl -s -H "Authorization: bearer $ACTIONS_ID_TOKEN_REQUEST_TOKEN" "$ACTIONS_ID_TOKEN_REQUEST_URL&audience=sts.amazonaws.com" | jq -r '.value' > /tmp/web_identity_token_file
          mkdir ~/.aws
          echo -e "[profile cicd]\nrole_arn=${AWS_ROLE_ARN}\nweb_identity_token_file=${AWS_WEB_IDENTITY_TOKEN_FILE}" >> ~/.aws/config
          aws sts get-caller-identity --profile cicd

      - name: Checkout
        uses: actions/checkout@v3

      - uses: hashicorp/setup-terraform@v2
        with:
          terraform_version: ${{ env.TF_VERSION }}

      - name: Terraform format
        id: fmt
        run: terraform fmt -check
        working-directory: ${{ matrix.dir }}
        continue-on-error: true

      - name: Terraform init
        id: init
        run: terraform init
        working-directory: ${{ matrix.dir }}

      - name: Terraform validate
        id: validate
        run: terraform validate
        working-directory: ${{ matrix.dir }}

      - name: Terraform plan
        id: plan
        run: terraform plan -no-color -input=false -refresh=true -detailed-exitcode
        working-directory: ${{ matrix.dir }}

      - name: Checkov GitHub Action
        uses: bridgecrewio/checkov-action@v12.1347.0
        with:
          directory: ${{ matrix.dir }}
          # Run scan on all checks but a specific check identifier (comma separated)
          # skip_check: # optional
          download_external_modules: false
          quiet: true
          soft_fail: true    # Replace to false to break on errors

      - name: Plan output
        id: output
        uses: actions/github-script@v3
        if: github.event_name == 'pull_request'
        env:
          PLAN: "terraform\n${{ steps.plan.outputs.stdout }}"
        with:
          github-token: ${{ secrets.GITHUB_TOKEN }}
          script: |
            const output = `#### Terraform Format and Style 🖌\`${{ steps.fmt.outcome }}\`
            ### Workspace

            \`${process.env.TF_WORKSPACE}\`

            #### Terraform Initialization ⚙️\`${{ steps.init.outcome }}\`
            #### Terraform Plan 📖\`${{ steps.plan.outcome }}\`
            <details><summary>Show Plan</summary>

            \`\`\`hcl
            ${process.env.PLAN}
            \`\`\`

            </details>

            **Pusher**: @${{ github.actor }}
            **Action**: ${{ github.event_name }}
            `;
            github.issues.createComment({
              issue_number: context.issue.number,
              owner: context.repo.owner,
              repo: context.repo.repo,
              body: output
            })
      - name: Terraform apply
        id: apply        
        run: terraform apply -input=false -auto-approve
        working-directory: ${{ matrix.dir }}

E) Arrange your Terraform code

Sample for backend.tf.

Change account ID on line 13 to your tooling account ID. Change values for lines 8-10 as well.

If you are starting from scratch with Terraform state on AWS this module can help you get going: https://github.com/cloudposse/terraform-aws-tfstate-backend .

terraform {
  backend "s3" {
    profile        = "cicd"
    session_name   = "SESSION_NAME"
    external_id    = "EXTERNAL_ID"
    encrypt        = true
    acl            = "bucket-owner-full-control"
    dynamodb_table = "tf-build-terraform-state-lock"                                                    # Replace to your configuration
    bucket         = "tf-build-terraform-state"                                                         # Replace to your configuration
    key            = "workload-prod-01/terraform.tfstate"                                               # Replace to your configuration
    region         = "eu-central-1"
    assume_role_with_web_identity = {
      role_arn           = "arn:aws:iam::<id-of-tooling-account:role/GitHubAction-AssumeRoleWithAction" # Replace to your configuration
      web_identity_token_file = "/tmp/web_identity_token_file"
    }
  }
}

Sample for provider.tf

provider "aws" {
    region = var.aws_region
    profile = var.profile_cicd
    assume_role {
        role_arn = "arn:aws:iam::${var.aws_account_id}:role/${var.profile_cicd}"
        session_name = "SESSION_NAME"
        external_id  = "EXTERNAL_ID"
    }
}

Sample for variables.tf

variable "aws_region" {
  type = string
  default = "eu-central-1"
}

variable "aws_account_id" {
  type = number
}

variable "profile_cicd" {
  type = string
  default = "cicd"
}

Sample for terraform.tfvars

aws_region = "eu-central-1"  # Replace to your region
profile_cicd = "cicd"
aws_account_id = 1234567890  # Replace to your destination AWS account ID

Sample for main.tf

module "s3_bucket" {
  source = "terraform-aws-modules/s3-bucket/aws"
  versioning = {
    enabled = true
  }
}

As always, adjust variables and desired configurations to your use-case.

Please let me know if you have any questions or feedback, you’ll find my contact details at the /about/ page.

Conclusion

In this article we saw how Open ID Connect (OIDC) can be leveraged to avoid configuring long-lived static access keys (IAM users) in CI/CD pipelines.

We configured an Identity Provider in an AWS account dedicated for tooling purposes with corresponding IAM roles in the tooling + workload accounts.

A GitHub Actions workflow was then configured and sample Terraform code was provided for demonstration.

You now no longer need to worry about rotating static secrets for this use-case (or introduce more complexity with HashiCorp Vault [disclaimer: might be valuable in certain situations]).

References


Posted

in

by