Verify Conditional Access Policies using Report Only mode

In this post I will be covering the toping on how to verify Conditional Access Policies using Report Only Mode. It is always recommended to verify CA Policies before applying it in production. Report only mode comes as a handy tool to simulate the behaviour to check and see what could have happened if policy would have applied.

Conditional Access Policy

As we know that Conditional Access Policy provides organization the flexibility and control what users / administrator can control in terms of right access controls. We can implement several controls to secure the login, devices. Few of the implementations could implementing Multi-factor Authentication (MFA) for users, restrict access to Azure portal from external network, block access to Office 365 if device is not patched and the list goes on. There can be n number of scenarios to be implemented based upon requirement. But what happens when you create Conditional Access policy and that blocks everyone to login / access because of wrong configuration we implemented without verifying it. This is where Conditional Access Report-Only mode comes into picture.

What is Conditional Access Report-only mode

Conditional Access Policy have 3 states to apply – On, Off and Report-only. When you create Conditional Access policy, it will get apply automatically as it does not require assignments unlike any applications, configuration profiles. Hence, Microsoft came up with Report-only mode so as to evaluate the impact and risk of implementing Conditional Access policy.

Report-only mode is Conditional Access policy state, when set you can evaluate the behaviour of Conditional Access policy through Azure Active Directory > Monitoring > Sign-in logs. It will provide you the information on what would have happened whether success, failure while trying to access Cloud resources in terms of right access controls.

Setup Conditional Access report-only mode

Log on to Azure Portal and navigate to Azure AD Conditional Access. Create new Conditional Access policy or pen any existing one. I have a policy where I have restricted access to Azure portal for external location. I might block myself as well if conditions are not specified correctly, I haven’t excluded current user or I haven’t used azure break glass account. Hence, I am going to change the policy mode to Report-only under Enable policy. Click on Save.

We can see the Policy Name with State showing as Report-only and we are ready to test login attempt.

Check Sign-in logs for Report-only mode

I will try to access Azure portal from external location, and what I can see, I am able to login because Conditional Access policy is not enforced (in report-only mode). I wanted to see the behaviour whether I was allowed to login or not.

Under Azure Active Directory, navigate to Azure AD Conditional Access > Monitoring > Sign-in logs, I can see list of all sign-in attempts. Under User sign-ins (interactive) tab, I can see Status as Success for my last attempt along with following information:

• Date
• Request ID
• User
• Application
• Status
• IP address
• Location
• Conditional Access
• Authentication requirement

Click on the top row to evaluate it further which will open Activity Details: Sign-ins for this particular attempt.

Under Basic info, I can see basic information for my login attempt with success.

If I wanted to see which Conditional Access policy was applied, click on Conditional Access tab. I can see there are 2 policies, one is applied successfully which is related to MFA, while other is disabled.

We are interested for Report-only tab, click on it. It will show Policy Name for all Report-only mode CA policy. The first one “Restrict access to Azure portal external location” is showing Grant Controls as “Block” and Result as Report-only: Failure, which is an indication of Conditional Access policy would have blocked if CA policy mode was set to On.

There could be 4 results for Report-only status:

• Report-only: Success
• Report-only: Failure
• Report-only: User action required
• Report-only: Not applied

To further dig into it, click on Policy Name showing status as Failure.

This will open Conditional Access Policy details showing all Assignments, Conditions and Access controls including Location, Client app, Device and User risk.

The IP address range is out of the scope of my Named locations which I used to specify IP address range, hence any IP address out of this range will be considered as Matched and eventually Grant Controls shows Block

Configure Conditional Access insights and reporting

Conditional Access insights and reporting gives you a powerful way to understand the impact of Conditional Access policies. This is really helpful in scenarios where multiple Conditional Access policies are applied while granting the control.

Prerequisites for Conditional Access insights and reporting

We need to create Log Analytics workspace where sign-in logs will be saved. Users need to have either Contributor or Owner role assigned to Log Analytics workspace roles.

Users need to have Azure AD Premium P1 or P2 License assigned. User should be part of following roles to access insights and reporting:

• Conditional Access administrator
• Security reader
• Security administrator
• Global reader
• Global administrator

Create Log Analytics workspace for CA insights and reporting

Under Azure Portal, navigate to Log Analytics workspaces and click on Create.

Select existing Subscription and specify Resource group¸ I proceeded with creating a new group with name LogAnalyticsCAInsightsAndReporting. Provide Instance details such as Name and Region. Click Review + Create. Click on Create once again to proceed with creating the Log Analytics workspace.

Wait for some time to complete the deployment of creating Workspace. Finally we can see the message Your deployment is complete.

Integrate Azure AD sign-in logs in Azure Monitor logs

As our Log Analytics workspace is ready, we are ready to integrate sign-in logs with Log Analytics. The process is called setting up Azure AD logs with Azure Monitor. You must be global administrator or security administrator to perform this task.

Navigate to Azure Active Directory > Diagnostic settings and click on Add diagnostic setting.

This will open Diagnostic setting blade, Specify Diagnostic setting name as “Signin Attempt”. Under Logs > Categories, we have following options to select:

AuditLogs
SignInLogs
NonInteractiveUserSignInLogs
ServicePrincipalSignInLogs
ManagedIdentitySignInLogs
ProvisioningLogs
ADFSSignInLogs
RiskyUsers
UserRiskEvents

Select AuditLogs and SignInLogs, under Destination details select “Send to Log Analytics workspace”. Specify existing subscription and select Log Analytics workspace we created in previous step, click on Create.

Allow some time for integration to get completed. Once done, we can see successful creation of diagnostic setting under Azure Active Directory.

Verify Insights and Reporting status

We are done with the integration of Audit logs and Sign-in logs. Let’s navigate to Azure AD Conditional Access > Insights and reporting.

We can see the detailed information here under User sign-ins. Under Conditional Access policy, we have options to:

• Select all enabled policies
• Select individual enabled policy
• Select individual report-only policy

Select the option for CA Policy you are looking for along with Time range, users, app etc.

We can see in-depth detail of how many users have tried sign-in along with Success, Failure, User action required and Not applied status.

A colourful graphs with report will be available when you scroll down. You can see Breakdown per condition and sign-in status for Device State – Total, Device platform – Total, Client app – Total.

While scrolling down further, we can see Sign-in details with

User sign-in count – Total

Sign-in events – Total

Important Links

What is Conditional Access report-only mode? – Azure Active Directory | Microsoft Docs

Conditional Access insights and reporting workbook – Azure Active Directory | Microsoft Docs

Stream Azure Active Directory logs to Azure Monitor logs | Microsoft Docs