Menu

Official website

Create a Grafana Alert for expiring Microsoft Entra ID secrets

author image Giovanni Barbaro

20 Jun 2024

min read

azure grafana monitoring alert entraid en

Microsoft Entra ID is a key component of every cloud deployment. It allows, among other things, for "App Registrations" to authenticate users and other applications.
This is usually done by authenticating the App Registration itself with a client-id and a client-secret, similar to other IAM solutions.

Entra ID has a peculiarity though, it assigns an expiration date to client-secrets in order to force frequent rotation of those. This is very good from a security perspective, but caught my team and I by surprise on more than one occasion as suddenly our applications stopped working because of such secrets expiring.

Unfortunately Azure doesn’t seem to provide a built-in alert system for the expiring secrets, and the solutions we found online were not quite to our liking (this excellent guide suggests a solution making use of powershell scripts ) as we were hoping to fire the alerts from our Grafana instance since all the contact points were already configured.

After a long investigation and some open source contributions, we finally arrived to the solution I’m sharing here!

Overview

The basic idea is to allow Grafana to use Microsoft’s Graph API as a Datasource for its metrics and alerts. This means that Grafana itself will need to be authenticated as a Microsoft Entra ID Service Principal. To do that we’ll do the following:

  • Azure

    • Create an App Registration

    • Create an App Registration Secret

    • Grant permission to call Graph API

  • Grafana

    • Install "Infinity Data Source" plugin

    • Create a Graph API data source

    • Create an alert for expiring secrets

Azure

We’ll be basing our guide on the official guide offered by Microsoft

By the end of this section you’ll need to know your App Registration’s <tenant-id>, <client-id> and <client-secret>

Create an App Registration

  1. Login to Entra ID

  2. Access the Microsoft Entra ID service

  3. From the menu on the left, select the "App Registrations" blade

  4. From the top, click "New Registration"

  5. Fill the form (no redirection URL is needed)

  6. Click "Register"

New Registration

Create an App Registration Secret

On this step, we’ll create the <client-secret> needed for Grafana to authenticate itself as the AppRegistration. Worry not, it will be possible for Grafana to report on this secret as well!

  1. Open the App Registration object you just created (you should be automatically redirected there)

  2. From the overview page, write down your <client-id> (Application (client) ID) and <tenant-id> (Directory (tenant) ID)

  3. From the menu on the left, select the "Certificates & Secrets" blade

  4. From the top, click "Client Secrets"

  5. Click on "+ New client secret"

  6. Add a description and an expiration date

  7. Click "Add"

  8. IMPORTANT: Store the value that will be displayed in the "Value" column. This is a confidential value, so store it safely. This will be your <client-secret>

Registration Secret
Registration Secret Form

Grant permission to call Graph API

Now we’ll grant our newly created App Registration the permission to query the "Applications" stored in Entra ID. We’ll use this query in order to fetch the expiring secrets.

  1. Open the App Registration (you should still be there)

  2. From the menu on the left, select the "API Permissions" blade. You should see that the base "Microsoft.Graph User.Read" permission is already allowed

  3. From the top, click "Add a permission"

  4. From the menu that will pop-up you’ll have to select
    Microsoft APIs → Microsoft Graph → Delegated Permission → Application → Application.Read.All

  5. Once the right permission is selected, click on "Add Permission"

Registration Permission

Grafana

It is now the time to connect Grafana to the Graph API. In order to do this, we’ll make use of a special plugin we recently discovered: Infinity

This plugin allows to use web endpoint as a Data source, which fits our needs perfectly.

Install "Infinity Data Source" plugin

Using the official guide as a reference

  1. Login to your Grafana instance

  2. Click Administration > Plugins and data > Plugins in the side navigation menu to view all plugins.

  3. Browse for "Infinity".

  4. Click the plugin’s logo.

  5. Click Install.

Infinity Plugin

Create a Graph API data source

  1. In Grafana, click Connections > Data Sources

  2. On the top right, click "+ Add new data source"

  3. Pick Infinity as your Data source

  4. Configure the "Authentication" blade section

Property Value

Auth type

Oauth2

Grant type

Client Credentials

Auth Style

Auto

Client ID

<client-id>

Client Secret

<client-secret>

Token URL

https://login.microsoftonline.com/<tenant-id>/oauth2/v.0/token

Scopes

https://graph.microsoft.com/.default

Allowed hosts

https://graph.microsoft.com

  1. (Optional, but recommended) Configure the "Health Check" section

Property Value

Enable custom health check

Enabled

Health check URL

https://graph.microsoft.com/v1.0/applications

  1. Click on "Save & test"

Graph Api Data Source
Graph Api Health Check

Create an alert for expiring secrets

Now it’s time to piece it all together. The idea of this alert will be the following:

  1. Query the https://graph.microsoft.com/v1.0/applications endpoint

  2. Extract get all the secrets expiration dates

  3. Filter based on the number of days you need to be notified

To do so:

  1. In Grafana, click Alerting > Alert Rules

  2. Click "New alert rule" from the top right corner

  3. In the "2. Define query and alert condition" do the following

    • Remove the Expressions B and C

    • In the Query A fill as follows

Property Value Explanation

Datasource (top left corner)

The newly created Graph API datasource

Type

JSON

Parser

Backend

We need to use this parser because it’s the only one that supports alerts

Source

URL

Format

Table

Method

GET

URL

https://graph.microsoft.com/v1.0/applications

This will return all the applications in the tenant. If you want to perform some filtering consider using the $filter query parameter or add extra conditions to the "Filter" section down below

Parsing options & Result fields

Rows/Root

value.passwordCredentials

This will return all the credentials for all the app registrations. If your team sticks to one secret per App Registration, consider using only value as that could provide more information to the alert. (We won’t be providing the configuration for value)

Computed columns, Filter, Group by

Computed Columns - Expression

tomillis(totime(endDateTime))

Takes the endDateTime field which is represented as "yyyy-MM-ddThh:mm:ssZ" and converts it to Unix milliseconds

Computed Columns - As

Value

Filter

Value < ${__to} + <num-of-days> * 86400000

Compares the endDateTime to the ${__to} value (the current time) plus how many days in advance you want to be notified. The days need to be converted to millis, hence the * 86400000

  1. Configure the rest as you see fit, but make sure to set "Alert state if no data or all values are null" to "Keep Last State"

Alert Config

Conclusion

With this, Grafana will be able to send alerts whenever a secret is about to expire. Unfortunately, at this moment it’s not possible to display exactly which App Registration will be affected due to limitations of the backend parser, but simply logging into Entra ID’s App Registration > Owned Applications page should inform you of which Apps have soon-to-be-expired secrets!

I hope that this guide will help you and your team mitigate exceptions caused by secrets sneakily expiring just as it did for us.

Hopefully, in the not so far distant future, Azure will allow us to more easily get alerts from its key services. Until then, we’ll stick to this setup!

expand_less