Debugging the AWS AccessDeniedException Exception

Firstly, why the article… I am currently on a learning path of implementing infrastructure as code (IAC) using Terraform in AWS. I have read the docs and want to follow the standard security advice of granting least privilege.

Following this advice the role used to apply IAC should only have the permissions it requires to do its job and no more. In theory this ticks all the boxes, but it practice when your role doesn’t have the required permission to carry out the command it doesn’t always tell you the permission required and responds with a generic AccessDeniedException exception.

What an unhelpful error message!

My initial attempts to overcome the problem whilst adhering to least privilege was to start turning on groups of permissions which I thought related to the code that had changed. This is time consuming, especially when a service has 50+ permissions. If you include 10 permissions, rerun your command and it succeeds you still don’t know which of the 10 you require… Ahhhh! Temptation at this point may lead you to start using the wildcard * and move away from least privilege.

Photo by Ryan Snaadt on Unsplash

There must be a better way.

Fortunately there is a better way and it can be achieved using AWS services including AWS CloudTrail, Amazon S3 and Amazon Athena.

How are we going to use these services?

How do I get this up and running?

Step 1 - Create an AWS CloudTrail trail

• Login to the AWS Console and navigate to AWS CloudTrail
• Hit create trail
• Enter a trail name
• Enter an AWS KMS Alias or for demo purposes uncheck SSE-KMS Encryption (not recommended outside of the demo)
• All other fields can be left as default

• Hit next
• Management events must be checked as this logs the management operations auditing

• Hit next
• Hit create trail

Step 2 - Create Athena Table

• From CloudTrail, hit event history on the left navigation menu
• Hit create Athena table on the right of the UI

• Choose your CloudTrail S3 storage location
• Create table

Step 3 - Run the command which generates the AccessDeniedException exception

• Run the command to generate the AccessDeniedException exception

Step 4 - Query the logs

• Navigate to Athena
• If first time use, hit Get Started, on the query editor page you will need to set up a query result location in Amazon S3
• Set data source as AwsDataCatalog
• Set database as default
• You should see your CloudTrail log table we created in step 2
• In the New query window paste the following comand

SELECT 
    errormessage,
    errorcode,
    eventsource, 
    eventname,
    useridentity.sessioncontext.attributes.creationdate,
    useridentity.sessioncontext.sessionissuer.arn
FROM cloudtrail_logs_aws_cloudtrail_logs_example_events
WHERE useridentity.sessioncontext.sessionissuer.arn IS NOT NULL AND 
errorcode = 'AccessDenied'
ORDER BY useridentity.sessioncontext.attributes.creationdate DESC
LIMIT 10

• Hit run query, hopefully the results provided look like the image below which provides details about the role used, action performed, the resource and the permission required

Success!

If you don’t have any results returned you may need to wait a few minutes for CloudTrail or Athena to complete it’s processing.

Summary

I spent quite a few hours trying to overcome this problem before discovering the AWS services which were able to solve it in a more efficient, less onerous way.

Hopefully this article helps you on your journey of wanting to adhere to least privilege whilst keeping your sanity during development!