Using ‘Untitled Goose’ to Investigate Account Compromise in the Microsoft Cloud

CISA Untitled Goose Tool is a Useful Addition to Microsoft 365 Security Practitioner Toolkit.

The Cybersecurity & Infrastructure Security Agency (CISA) is a U.S.-based organization whose sole purpose is to help US-based organizations stay secure. Over the past couple of months, they have released exciting tools such as SCUBA (Secure Cloud Business Applications), an application that scans your Microsoft 365 environment and checks security configurations in your tenant, such as Conditional Access and SharePoint sharing settings… CISA’s own secure score, if you would.

At the end of March 2023, CISA released a new tool called ‘Untitled Goose.’ It is a post-incident hunting tool to help security practitioners sift through security logs in Microsoft 365, Azure Active Directory, and Azure Cloud to find potential malicious activity.

Why is a Tool Like Untitled Goose Needed?

It is important to start off by mentioning that the Untitled Goose tool is not for every organization. This tool (in my opinion) is meant for organizations that don’t have mature SOC and SIEM operations. Why is that? Those organizations won’t have a SIEM environment (such as Microsoft Sentinel) where all logs are collected and stored.

The Untitled Goose tool is designed to run if a breach is suspected. It will retrieve artifacts from the tenant and help verify if an attacker has compromised a user account or device. After Untitled Goose has exported the logs, you need to analyze the logs and identify malicious indicators.

Installation

 A whole slew of prerequisites must be completed before you can run the tool:

• Python version 3 must be installed. This can be installed from the Microsoft Store.

Note: the link to the Microsoft store redirects to Python version 3.9. This is because the 3.10 version on the Microsoft Store currently installs Python 3.10.11, which is incompatible with the tool.

• Mozilla Firefox is required to authenticate from the tool.
• An administrator account with read-only account permissions. I recommend creating a brand-new account (more about this topic later).
  • The account should have notifications via the Microsoft Authenticator app enabled and number matching disabled, as these settings are not compatible with the tool.
• An app registration with application permission on the Graph API, Defender for Endpoint, and Defender 365 API’s. A full list of permissions can be found on the prerequisites page on GitHub.
• Reader access to an Azure Subscription or Management Group for your app registration.
  • To configure those permissions, it is recommended to assign the roles Reader, Storage Blob Data Reader, and Storage Queue Data Reader to the application on the root management group.

To save you some time, I created a PowerShell script to create the app registrations, assign the API permissions, and create a secret:

Script: #Install-Module Microsoft.Graph -Scope CurrentUser

Import-Module Microsoft.Graph

Connect-MgGraph -Scopes "Application.ReadWrite.All"

$context = Get-MgContext

#Creates the application
$App = New-MgApplication -DisplayName 'Untitled Goose'

#adds the secret
$secret =  Add-MgApplicationPassword -ApplicationId $App.id

#adds the permissions
// The full list of permissions being added can be found here - https://github.com/cisagov/untitledgoosetool#requirements
$params = @{
"requiredResourceAccess"= @(
        @{
            "resourceAppId"= "00000003-0000-0000-c000-000000000000"
            "resourceAccess"= @(
                @{
                    "id"= "472e4a4d-bb4a-4026-98d1-0b0d74cb74a5"
                    "type"= "Role"
                }
                @{
                    "id"= "607c7344-0eed-41e5-823a-9695ebe1b7b0"
                    "type"= "Role"
                }
                @{
                    "id"= "1260ad83-98fb-4785-abbb-d6cc1806fd41"
                    "type"= "Role"
                }
                @{
                    "id"= "246dd0d5-5bd0-4def-940b-0421030a5b68"
                    "type"= "Role"
                }
                @{
                    "id"= "e321f0bb-e7f7-481e-bb28-e3b0b32d4bd0"
                    "type"= "Role"
                }
                @{
                    "id"= "5e0edab9-c148-49d0-b423-ac253e121825"
                    "type"= "Role"
                }
                @{
                    "id"= "bf394140-e372-4bf9-a898-299cfc7564e5"
                    "type"= "Role"
                }
                @{
                    "id"= "6e472fd1-ad78-48da-a0f0-97ab2c6b769e"
                    "type"= "Role"
                }
                @{
                    "id"= "dc5007c0-2d7d-4c42-879c-2dab87571379"
                    "type"= "Role"
                }
                @{
                    "id"= "b0afded3-3588-46d8-8b3d-9842eff778da"
                    "type"= "Role"
                }
                @{
                    "id"= "230c1aed-a721-4c5d-9cb4-a90514e508ef"
                    "type"= "Role"
                }
                @{
                    "id"= "40f97065-369a-49f4-947c-6a255697ae91"
                    "type"= "Role"
                }
                @{
                    "id"= "7ab1d382-f21e-4acd-a863-ba3e13f7da61"
                    "type"= "Role"
                }
                @{
                    "id"= "38d9df27-64da-44fd-b7c5-a6fbac20248f"
                    "type"= "Role"
                }
                @{
                    "id"= "dbb9058a-0e50-45d7-ae91-66909b5d4664"
                    "type"= "Role"
                }
                @{
                    "id"= "9e640839-a198-48fb-8b9a-013fd6f6cbcd"
                    "type"= "Role"
                }
                @{
                    "id"= "c7fbd983-d9aa-4fa7-84b8-17382c103bc4"
                    "type"= "Role"
                }
                @{
                    "id"= "b86848a7-d5b1-41eb-a9b4-54a4e6306e97"
                    "type"= "Role"
                }
            )
        }
        @{
            "resourceAppId"= "fc780465-2017-40d4-a0c5-307022471b92"
            "resourceAccess"= @(
                @{
                    "id"= "41d209c7-2511-4fc9-b899-8008a3976f09"
                    "type"= "Role"
                }
                @{
                    "id"= "41269fc5-d04d-4bfd-bce7-43a51cea049a"
                    "type"= "Role"
                }
                @{
                    "id"= "37f71c98-d198-41ae-964d-2c49aab74926"
                    "type"= "Role"
                }
                @{
                    "id"= "6443965c-7dd2-4cfd-b38f-bb7772bee163"
                    "type"= "Role"
                }
                @{
                    "id"= "71fe6b80-7034-4028-9ed8-0f316df9c3ff"
                    "type"= "Role"
                }
                @{
                    "id"= "ea8291d3-4b9a-44b5-bc3a-6cea3026dc79"
                    "type"= "Role"
                }
                @{
                    "id"= "93489bf5-0fbc-4f2d-b901-33f2fe08ff05"
                    "type"= "Role"
                }
                @{
                    "id"= "a8bc2240-f96a-46a1-bad5-6a960b7327a1"
                    "type"= "Role"
                }
            )
        }
        @{
            "resourceAppId"= "8ee8fdad-f234-4243-8f3b-15c294843740"
            "resourceAccess"= @(
                @{
                    "id"= "7734e8e5-8dde-42fc-b5ae-6eafea078693"
                    "type"= "Role"
                }
            )
        }
    )
}
Update-MgApplication -ApplicationId $App.id -BodyParameter $params

#Add redirect url
$params = @{
    RedirectUris = @("https://practical365.com")
}
Update-MgApplication -ApplicationId $App.id -IsFallbackPublicClient -PublicClient $params

#Waiting a bit before permissions are loaded and consent can be given
Start-Sleep -s 30

#Grants interactive Consent
$URL = "https://login.microsoftonline.com/$($context.TenantId)/adminconsent?client_id=$($app.appid)"
Start-Process $URL

Write-Host "App details:"
Write-Host "Tenant Id: $($context.TenantId) "
Write-Host "App id: $($app.Id) "
Write-Host "Secret: $($secret.SecretText)"

At the end, it opens a browser asking you to consent to the application permissions on the app registration. The only thing left to do is assign the Azure permissions.

If you have completed all perquisites, you can now download the sources files from GitHub and execute the following command to install the tool (make sure your command prompt is open in the unzipped folder from GitHub):

python3 -m pip install .

By running this command, a subset of Python libraries will be installed, together with the main library called ‘Goosey.’

Note: If you receive a warning about missing PATH variables, make sure to fix that first. The location where Goosey is installed should be configured as a PATH variable.

Configuration

The tool requires you to add your credentials in plain text in a configuration file. That is why I recommend using a ‘throw-away’ administrator account and app registration, which you should remove after running the tool. This will ensure the tool does not become a security liability.

To set up the tool, run the following Python command python3 scripts/generate_conf.py’ and open the ‘auth.conf’ file in the goose directory. Within this configuration file, add the following values:

• Username of your read-only Exchange role
• The password of the account
• Application ID of the app registration we created
• Application secret of the app registration

The code repository also contains a configuration file named ‘.conf.’ This must be configured before the tool can run. The file contains basic information about your tenant.

• tenant: The Tenant identifier.
• us_government: Should only be set to ‘true’ if you have a GCC High tenant (a specific kind of tenant reserved for United States Government Agencies), otherwise, this can be set to ‘false.’
• exo_us_government: A boolean that should be set to ‘true’ if you have a US government tenant.
• subscriptionid: Defines what Azure subscriptions will be checked. I recommend setting this to ‘All,’ as this will take all your Azure subscriptions in scope.
• m365: Defines if you have a Microsoft 365 environment, it’s recommended to set it to yes. If this is configured to no, only Azure, MDE, and Defender for IOT will be scanned.
• date_start: Configures the start time to use when retrieving Azure AD Sign-in Logs. If you are just testing the tool, set it up to the day before yesterday to avoid retrieving too much information. As an example, this could be: 2023-04-10.

After providing the credentials and configuring the tool, you can authenticate by using the ‘goosey auth’ command. Don’t be surprised by how bad the authentication experience is here. During the authentication process, multiple MFA requests will be generated, and the account used cannot have number matching enabled. It teaches an administrator that they should just accept multiple MFA prompts without thinking and disable certain security tools to make a tool work. To me, this is a big oversight for an organization such as CISA, which aims to increase security.

In the configuration file (.conf) are boolean values that determine what kind of logs will be checked when you run the tool. By default, all of the values are set to false, and you need to enable them one by one. You might think it is best to enable them all, but this will put a lot of load on the tool, meaning you might get throttled by the Microsoft Graph API. As a start, I recommend enabling the following configuration to retrieve some important tenant information which you can use to explore the tool with.

• Applications: Audits app registrations and service principals in Azure AD.
• Directory_roles: Azure AD administrator roles.
• Vm_config: Configuration of Azure Virtual Machines.
• Risk_detections: Risk detections seen on logins by Azure AD Identity Protection.

This tool also integrates with Defender for IOT. If you have it deployed, be sure to update the D4IOT-specific files (.auth_d4iot, .config_d4iot).

Unfortunately, there is no guidance from CISA available on what configurations you should enable. I had hoped there would be a recommended or default configuration list in which to get started, but there is nothing besides a description for each property.

Letting the Goose Graze

After you have worked your way through the installation and configuration, it is time to run the tool. There are three different commands you can use:

• Graze
  • This command is meant to set up time boundaries for you to use in subsequent ‘honk’ commands. By using the graze commands, you can set up time filters for the requested data.
• Honk
  • This command dumps information from your environment into different files. By default, it will read your configuration file, but you can also retrieve all information from a specific category by providing parameters.
  • As an example: ‘goosey honk –mde’ will export all Defender for Endpoint data.
• Message Trace
  • This is the most basic command. It runs a message trace in Exchange Online using the parameters (sender, recipient…) defined in the configuration file.

After you have executed a command, the tool exports the data in JSON or CSV format. Depending on your environment, this can take a while. Be sure to keep an eye on the tool, if it takes more than 60 minutes, your session will expire as our access token is no longer valid, and you will need to reauthenticate manually.

Interpreting the data

After running the tool, a ton of files are in the ‘outputs’ folder. Now begins the real work of looking over the exported logs to search for traces of a potential compromise. This can be done by loading the logs in a database tool or SIEM. Interpreting this amount of logs requires in-depth knowledge of the Microsoft 365 Cloud as you will receive thousands of lines of log files, and it is up to you to distinguish what are default configurations and what might indicate a breach.

Close-off

After using previous CISA tools (such as SCUBA), I had high hopes for the Untitled Goose tool. While it is promising, and the amount of data it can retrieve is awesome, it lacks some finesse. The fact that they teach administrators to save passwords and application secrets in plain text is beyond me. Combined with the fact that the configuration is cumbersome, this is not user-friendly. I imagine CISA will improve the experience in the next couple of iterations, which will be beneficial.  This tool is not expected to be run by everyone. It is geared towards incident responders who arrive at a breached environment and want to extract as much information as possible to investigate. For them, a tool like this will be a godsend.