Get Usage Logs from Azure Rights Management

Introduction

You can configure the Azure Rights Management service to generate and give you a log of every request that the Azure Rights Management service fulfils for your tenant as soon as it happens.

This information is very useful for a variety of reasons:

• Analyzing for business insight. The Azure Rights Management service writes logs in W3C Extended Log (Weblog) format into an Azure storage account that you provide for this purpose. You can then feed the logs into a repository of your choice (database, OLAP, Map/Reduce, etc.) in order to analyze and report.
  

Examples of insight you can gain are: who is accessing your RMS-protected assets, what are they accessing, from what devices, from where, are they getting a satisfactory experience, who all has read a given document etc.

• Monitoring for abuse. The Azure Rights Management service shares logs with you in near-realtime (99.9% of logs available to you within 15 minutes of the action). This allows you to continuously monitor usage of your Azure Rights Management service's assets.
  

You know your employees best and are uniquely qualified to identify an abuse pattern. As an example, your administrators may want to be alerted if there is a spike in accesses to your assets during off hours (insider trying to steal a bunch of documents?), or if the same user accesses from two different IP addresses within 15 minutes (potential password compromise?).

• Performing Forensics. When there is an information leak, the top two questions are:
  
  • Who recently accessed the specific document that leaked?
    
  • What information did a specific suspect access recently?
    

With the Azure Rights Management service's architecture, your documents can flow any way you want (email, cloud storage such as Dropbox, USB, etc.) but recipients must get a license from the Azure Rights Management service to open and consume those documents. Therefore the Azure Rights Management service's logs are a definitive source of information for forensics, as long as your assets are protected with the Azure Rights Management service.

Objectives of this paper

This document provides information about the logging capability of the Azure Rights Management service and thus about monitoring and controlling its usage. More particularly, it provided an in-depth description of this capability, how to enable it in your environment and your related subscription of the Microsoft Rights Management cloud-hosted service along with the standard format used for the log files.

Furthermore, by following the steps outlined in this document you should be able to successfully prepare your environment to leverage the logging capability, enable it and efficiently monitor the usage of the service over the time, and consequently start using the Azure Rights Management service within your organization to create and consume protected content in compliance with your own security and IT policies in place.

Note    The logging capability is available to you whether you let Microsoft generate your key (the default) or you bring your own key (BYOK) (see whitepaper Bring Your Own Key with Azure Rights Management). But when you bring your own key, the Azure Rights Management service also logs every usage of your key in addition to front-end request logs. This allows you to monitor in near real-time how your key is being used.

Non-objectives of this paper

This document doesn't offer a full description of the Microsoft Rights Management services offerings. It rather simply focusses on key aspects in the context of this paper that aims at providing the readers an understanding on how to leverage and enable logging capability in your environment and your related subscription of the Microsoft Rights Management cloud-hosted service.

Note    For an overview of the NEW Microsoft Rights Management services offerings, see the whitepaper Microsoft Rights Management services, the online documentation, the series of whitepapers on RMS to which the current document belong as well as the posts on the RMS Team blog.

Organization of this paper

To cover the aforementioned objectives, this document is organized by themes, which are covered in the following sections:

• Opting in to receive logs.
  
• Retrieving your logs.
  
• Processing your logs.
  
• Managing the logging capability.
  
• Understanding the log format.
  

About the audience

This document is intended for IT professionals and system architects who are interested in understanding the logging capability of the Azure Rights Management service and, thus monitoring and controlling its usage.

Opting in to receive logs

Opting in for logs is completely optional. All functionality of the Azure Rights Management service will work the same way whether or not you opt in to receive logs.

The Azure Rights Management service provides you logs for no extra charge. You must provide an Azure storage account to receive logs in, and you will be billed for the storage used.

Understanding the pre-requisites

To exercise the Usage Logging feature, the pre-requisites are as follows:

The rest of this document will guide you through the entire process to benefit from such a capability.

Setting up an Azure storage

As previously outlined, the Azure Rights Management service writes logs to an Azure storage account that you provide. The rest of this section steps you through how to create such an account.

The steps below assumes you already have an Azure account. For testing purpose, you can subscribe for a free Azure 1-month trial.

We support individual accounts, but recommend organizational accounts.

We recommend you set up a dedicated storage account for the Azure Rights Management service's logs. You will need to share the storage account keys with the Azure Rights Management service, and potentially with other staff in your organization that report on your logs.

To set up an Azure storage, proceed with the following steps:

1. Open a browser and navigate to the Azure management portal at https://manage.windowsazure.com.
   
2. Sign in with your Azure account credentials
   
3. Select STORAGE
   in the left pane and click NEW at the bottom of the screen. Select STORAGE and QUICK CREATE.
   

1. Type a unique name for your storage account URL, for example "corpcontoso" for our factious company Contoso Corporation and select a location corresponding to the one of your RMS tenant, North Europe in our case. Click CREATE STORAGE ACCOUNT.
   

Wait for Azure to create your account. Once complete, you will see a status of Online as shown next.

1. Click MANAGE ACCESS KEYS at the bottom of the screen. A Manage Access Keys dialog pops up and shows your primary and secondary access keys. Copy the primary access key to the clipboard, you will need this in the next step.
   

Configuring the Azure Rights Management service to log to the Azure storage account

The configuration leverages the cmdlets of the Azure Rights Management administration module for Windows PowerShell. Most Azure Rights Management service's administrative tasks need this package.

Note        Windows PowerShell
is a task-based command-line shell and scripting language that is designed for system/service administration and automation. It uses administrative tasks called cmdlets. Each cmdlet has required and optional arguments, called parameters, that identify which objects to act on or control how the cmdlet performs its task. You can combine cmdlets in scripts to perform complex functions that give you more control and help you automate the administration of Windows, applications and online services in the Cloud. It has become a common way to manage the latest generation of Microsoft products and services.

A Windows PowerShell "module" is a package that contains Windows PowerShell commands, cmdlets, providers, functions, variables, and aliases. The Azure Rights Management Administration Module for Windows PowerShell is a separate installation package which includes cmdlets specifically designed for the Azure Rights Management service tenant-based administration.

For more information about Windows PowerShell, please see the Windows PowerShell Web site, the Windows PowerShell online help, and the Windows PowerShell Weblog
Windows PowerShell Software Development Kit (SDK) that includes a programmer's guide along with a full reference.

Installing Windows PowerShell for Azure Rights Management

This section walks you through the installation of the Microsoft Rights Management administration module.

Note    For additional information, see the Microsoft TechNet article Install Windows PowerShell for rights management.

Installing the prerequisite software

The Microsoft Rights Management administration module requires the Microsoft Online Services Sign-In Assistant (MOS SIA) 7.0, which is 7.250.4551.0 as of this writing. So we will install it manually.

Note    The Microsoft Online Services Sign-In Assistant (MOS SIA) 7.0 provides end user sign-in capabilities to Microsoft Online Services, such Office 365 and the Azure Rights Management service. In the context of this paper, the MOS SIA is used to authenticate users to these services through a set of dynamic link library files (DLLs) and a Windows service as described in the community article Description of Microsoft Online Services Sign-In Assistant (MOS SIA).

To install the Microsoft Online Sign-In Assistant (MOS SIA) 7.0, proceed with the following steps:

1. Open a browser session and navigate to the following link: Microsoft Online Services Sign-In Assistant for IT Professionals RTW and click Download.
   
2. In the Choose the download you want page, check msoidcli_64bit.msi (or msoidcli_32bit.msi depending on your Windows environment) and click Next.
   

1. Click Run. The Microsoft Online Services Sign-in Assistant Setup wizard opens.
   

1. On the license terms page, select I accept the terms in the License Agreement and Privacy Statement and click Install. A User Account Control dialog pops up.
   

1. In the User Account Control dialog, click Yes to execute the setup.
   

1. On the completion page, click Finish.
   

Installing the Microsoft Rights Management administration module

To connect Windows PowerShell to the Azure Rights Management service, proceed with the following steps:

1. Download the Microsoft Rights Management Administration module from http://go.microsoft.com/fwlink/?LinkId=257721.
   
2. In the Choose the download you want page, check WindowsAzureADRightsManagementAdministration_x64.exe (or WindowsAzureADRightsManagementAdministration_x86.exe depending on your Windows environment) and click Next.
   

1. Click Run. A Microsoft Rights Management Administration Setup wizard opens up.
   

1. On the Welcome page, select the Next option.
   

1. On the End-User
   License Agreement page, select I accept the terms in the License Agreement and click Next.
   

1. On the Ready to Install page, click Install. An User Account Control dialog pops up.
   

1. Click Yes.
   

1. On the completion page, click Finish.
   

The Microsoft Rights Management administration module for Windows PowerShell provides a set of Windows PowerShell cmdlets that provide administrative (advanced) capabilities for the Azure Rights Management service. These cmdlets will be used later in this document with for the logging capability. More especially, in the context of this document, the cmdlets relevant to the logging capability are the following ones:

• Disable-AadrmUsageLogFeature
  
• Enable-AadrmUsageLogFeature
  
• Get-AadrmUsageLog
  
• Get-AadrmUsageLogFeature
  
• Get-AadrmUsageLogLastCounterValue
  
• Get-AadrmUsageLogStorageAccount
  
• Set-AadrmUsageLogStorageAccount
  

Detailed help is available from an elevated Windows PowerShell command prompt by executing these commands with the -? option.

Telling Azure Rights Management about your storage account and enabling logging

To configure the Azure Rights Management service to log to the Azure storage account, proceed with the following steps:

1. Open an elevated Windows PowerShell command prompt.
   
2. Then import the Azure Rights Management module for Windows PowerShell and connect to the Azure Rights Management service by typing the following commands.
   

PS C:\Windows\system32> Import-Module AADRM

PS C:\Windows\system32> Connect-AadrmService -verbose

You will be prompted for your credentials.

1. Enter your Azure Rights Management service's tenant credentials (the set of credentials should have Global Administrator privilege) and wait to be authenticated. In the Windows PowerShell Credential Request window that opens up, provide the credentials for the administrator account such as:
   

Username: philber@demorms.onmicrosoft.com

Password: ****************

1. Next, run the following commands to specify the Azure Rights Management service where you want your logs. Replace the key value in the Convert-SecureString cmdlet with the one that you copied in the Azure management portal for your storage account's access key (copied to the clipboard). Replace "corpcontoso" in the example below with your storage account name.
   

PS C:\Windows\system32> $accesskey = Convert-SecureString " wUjKVV14XXUCrdpuLsIa8yQ5IgUmLSOLmlgS/CcHNZXiurEORjTItdtPf4OpCaIwGNyijjMPxvDEOG21HRKR7A==" –asplaintext -force

PS C:\Windows\System32> Set-AadrmUsageLogStorageAccount -StorageAccount corpcontoso -AccessKey $accesskey

corpcontoso was set as the storage account for the usage log feature for the Rights management service.

PS C:\Windows\system32>

1. Finally, run the following cmdlet to enable logging:
   

PS C:\Windows\system32> Enable-AadrmUsageLogFeature

The usage log feature is enabled for the Rights management service.

PS C:\Windows\system32>

From this point onwards the Azure Rights Management service will log all requests served on behalf of your tenant to your storage account. Logs before this point are not available.

Retrieving your logs

The Azure Rights Management service writes logs to your Azure storage account as a series of blobs. Each blob contains one or more log records, in W3C Extended Log (Weblog) format. The blob names are numbers, in the order they were created.

Section § Understanding the log format describes the log format in detail.

Logs may take up to a few minutes to show up in your storage account after the Azure Rights Management service serves a request for your tenant. 99.9% of logs should appear in your account within 15 minutes after the request is served.

You can retrieve these logs in two ways:

1. The simplest method is to use the Get-AadrmUsageLog cmdlet in the Microsoft Rights Management module for Windows PowerShell package.
   
2. Alternatively, if you want to customize what/how/where you download, you can use the Azure SDK or other tooling.
   

Retrieving your logs with Windows PowerShell

The Get-AadrmUsageLog cmdlet lets you download logs to your computer. This cmdlet downloads each blob as a file to the location you specify. You may analyze these files locally or import them into a database or Hadoop storage to do serious crunching.

Example 1: still from the previous elevated Windows PowerShell command prompt, run the following command to download ALL available logs from your storage account to your C:\Logs folder:

PS C:\Windows\system32> Get-AadrmUsageLog –Path "C:\Logs"

1527

PS C:\Windows\system32>

Example 2: still from the previous elevated Windows PowerShell command prompt, run the following command to download a specific range of blob:

PS C:\Windows\system32> Get-AadrmUsageLog –Path "C:\Logs" –FromCounter 1024 –ToCounter 2047

2047

PS C:\Windows\system32>

The cmdlet displays on the console the name of the last blob it downloaded. You can assign this to a variable as shown next. This allows you to run Get-AadrmUsageLog in a loop or a scheduled job, downloading only the incremental logs each time.

PS C:\Windows\system32> $LastBlobName = Get-AadrmUsageLog –Path "C:\Logs"

1527

PS C:\Windows\system32> $LastBlobName

1527

PS C:\Windows\system32>

To download blobs starting from the last downloaded one to the last available on, run the following command.

PS C:\Windows\system32> $LastBlobName = Get-AadrmUsageLog –Path "C:\Logs" –FromCounter $LastBlobName –ToCounter (Get-AadrmUsageLogLastCounterValue)

Retrieving your logs with Azure Storage SDK

In some situations you may want more flexibility than Get-AadrmUsageLog provides. E.g. you may need to delegate the downloading of logs to a person or process that cannot have your Azure Rights Management service's administrative credentials. Or you may want to poll for logs in realtime in order to monitor abuse.

In such situations, you can retrieve the logs using the Azure Storage SDK directly.

Processing your logs

The storage account you created for logs is like a mailbox. It supports reading directly from the storage account, but is not optimized for this. For best performance and cost we recommend you download the logs to destination appropriate for your application use case, such as a local folder, a database, or a Map/Reduce repository.

Here is an example of how to aggregate all your downloaded log files into a CSV format using the Microsoft Log Parser utility, which is a tool to convert between various well-known log formats. The same tool can be used to convert to SYSLOG format, or import into a database:

C:\users\joesmith\logs> logparser –i:w3c –o:csv "SELECT * INTO AllLogs.csv FROM *.log"

Statistics:

-----------

Elements processed: 36

Elements output: 36

Execution time: 0.08 seconds

C:\users\joesmith\logs>

To interpret the logs, see Section § Understanding the log format.

Managing the logging capability

Suspending and resuming logging

Should you for any reason not want logs any more, run the Disable-AadrmUsageLogFeature cmdlet from the elevated Windows PowerShell command prompt:

PS C:\Windows\System32> Disable-AadrmUsageLogFeature

The usage log feature is disbled for the Rights management service.

PS C:\Windows\System32>

The Azure Rights Management service stops writing logs to your account immediately. However it retains your storage account information. So you can resume logging again. To resume, just run the cmdlet Enable-AadrmUsageLogFeature as illustrated before.

PS C:\Windows\system32> Enable-AadrmUsageLogFeature

The usage log feature is enabled for the Rights management service.

PS C:\Windows\system32>

If you are not sure what state your logging is in, use the Get-AadrmUsageLogFeature cmdlet. In the following example, the return value of "True" means logging is currently enabled.

PS C:\Windows\system32> Get-AadrmUsageLogFeature

True

PS C:\Windows\system32>

Managing your log storage

You manage your storage account directly. You can leave your logs there for as long as you wish. Or you may delete the logs after you download them. Or you can prune selectively. The Azure Rights Management service does not read the blobs back, so what you do with your logs will not interfere with the Azure Rights Management service.

The one exception to the previous statement is the blob named "metadata" in the container "rms-metadata". Do not modify or delete this blob. The Azure Rights Management service uses this blob to keep track of the last blob number it wrote. If you accidentally delete the metadata blob, Azure RMS will start a new container for logs, with a blob number starting from 1. All future downloads using the Get-AadrmUsageLog cmdlet will download logs from the new container only. The logs in the previous container will be preserved but orphaned. You can download previous logs with the Azure storage SDK, but the Get-AadrmUsageLog cmdlet will have no knowledge of it.

You will be billed for space used for as long as you keep your logs in Azure storage.

You can also delegate the management function to a vendor by sharing your storage account name and access key.

Rolling your storage account details

If you change the vendor that processes your logs, or if you suspect that your storage account key was breached, then you must regenerate your storage account keys. In order to prevent interruption in service, Windows Azure storage gives you two keys, a primary and a secondary key. Regenerate the key that you are NOT using, e.g. if you configured the Azure Rights Management service to use the primary access key, then regenerate the secondary access key by clicking regenerate.

Once regenerated, configure the Azure Rights Management service to use the new key by running the cmdlet Set-AadrmUsageLogConfig.

PS C:\Windows\system32> $accesskey = Convert-SecureString " wUjKVV14XXUCrdpuLsIa8yQ5IgUmLSOLmlgS/CcHNZXiurEORjTItdtPf4OpCaIwGNyijjMPxvDEOG21HRKR7A==" –asplaintext -force

PS C:\Windows\System32> Set-AadrmUsageLogConfig -StorageAccount corpcontoso -AccessKey $accesskey

corpcontoso was set as the storage account for the usage log feature for the Rights management service.

PS C:\Windows\system32>

Now you can regenerate the other access key.

To switch to a different storage account, run Set-AadrmUsageLogConfig with the new storage account and key. Do this only if really necessary. Doing so will cause your new logs to go to the new storage account. The previous logs will stay in the previous storage account. The Get-AadrmUsageLog cmdlet (and likely most other applications) will know of the latest storage account and logs only.

Delegating access to your logs

You can delegate the processing of your RMS logs to a vendor by sharing your storage account name and access key. Since they will not have your Azure Rights Management service administrator credentials, they cannot use the Get-AadrmUsageLog cmdlet to download logs. Instead, they must retrieve the logs using the Windows Azure Storage SDK or other relevant tooling.

It is safe to share your storage account name and access key with your vendor as long as that storage account is dedicated to your logs. Access to that key does not grant access to any of your other storage accounts nor to your Azure Rights Management service tenant itself.

Understanding the log format

This section explains the format of the logs.

Understanding the storage account layout

The first time the Azure Rights Management service writes logs to your storage account, it creates two containers.

• Rms-metadata. This is reserved for internal use by the Azure Rights Management service. Do not modify or delete this. This is the only part of your storage account that the Azure Rights Management service reads back and expects to be untouched.
  
• Rms-logs-<guid>. This container is where the Azure Rights Management service writes the logs. Everything in this container is yours, you can keep it there or delete it.
  

Over time the Azure Rights Management service may create additional containers of the form Rms-logs-<guid>. Specifically, if the container Rms-metadata is accidentally deleted or corrupted, the Azure Rights Management service resets its contents, creates a new Rms-logs-<guid> container to write all further logs to. Previous log containers are left untouched but they are orphaned from the Azure Rights Management service's perspective.

The Azure Rights Management service writes the logs as a series of blobs. Each blob contains one or more log records, in extended W3C log format as previously outlined.

Understanding the log sequence and how to interpret it

The name of each blob is a number. Within each log container the first blob is named "000000001". Each blob is named sequentially in the order it was created. Each blob contains one or more log records. Each log record has a UTC time, which indicates when the corresponding request was served by the Azure Rights Management service.

The Azure Rights Management service's logging system is optimized to get log records to you efficiently, rather than in strict sequential order. The order of blobs, as well as the order of log records within a blob, may be out of chronological sequence. The only reason the blobs are named sequentially is to enable you to download them efficiently incrementally.

Let's illustrate this with some examples:

• The log records in blob 000000004 may overlap chronologically with the log records in blob 000000003. In the extreme case, all log records in blob 000000004 may have been generated before all log records in blob 000000003.
  
• Similarly the second log record in a blob may have been generated before the first log record, but simply got written to storage in reverse order.
  

Before you analyze the logs, we recommend that you download and import the logs into a repository where you can sort the logs based on timestamp. You can download using the Get-AadrmUsageLog cmdlet or on your own using the Azure Storage SDK.

If the logs are not chronological, you may be wondering how to find out if you have received all logs until a certain point. The Azure Rights Management service will write 99.9% of logs within 15 minutes of the request happening. So in practice if you collect all logs until 15 minutes past the point in time you are interested in, you should receive almost all logs.

One other caveat you should be aware of is that the timestamp on each log record is the local time of the server that processed the request. Because the Azure Rights Management service runs on multiple servers across multiple data centers, sometimes the logs may seem out of sequence even when sorted by timestamp. They are close though, within ±1min. This is typically not an issue for most real-world use cases for log analysis.

Understanding the blob format

Each blob is in W3C Extended log format. Here is an example of a log file.

#Software: RMS

#Version: 1.0

#Fields: date    time    row-id    request-type    user-id    result    correlation-id    content-id    c-info    c-ip

2013-09-19    13:46:44    0d07036b-c66c-4e92-b887-f59ecd61dc96    AcquireLicense    'janet@corp-contoso.com'    'Success'    ad18e935-bcf9-4b51-9d34-cf3391c451ef    {9312A0DF-DA57-4854-9160-603A1ED06CB3}    'MSIPC;version=1.0.622.36;AppName=WINWORD.EXE;AppVersion=15.0.4535.1000;AppArch=x86;OSName=Windows;OSVersion=6.2.9200;OSArch=x86'    94.245.87.113

As you can see, it starts with the following two lines.

#Software: RMS

#Version: 1.0

The first line identifies that these are the Azure Rights Management service's logs. The second line identifies that the rest of the blob follows the version 1.0 specification (as depicted as part of this document). Applications that parse these logs are advised to verify these two lines before parsing the rest.

The third line enumerates a list of field names, separated by tabs.

#Fields: date    time    row-id    request-type    user-id    result    correlation-id    content-id    c-info    c-ip

Each of the subsequent lines is a log record. The values of the fields are in the same order as the definition line above, and are separated by tabs.

The following table provides the field definitions.

Understanding the special values of user-id

The following values of user-id are special and do not map to a real user.

• 'microsoftrmsonline@<GUID>.rms.<region>.aadrm.com' – This means an Office 365 service (e.g. Exchange Online or SharePoint Online) is making the request where:
  
  • <GUID> is the GUID for your tenant.
    
  • <region> is the region where your tenant is registered:
    
    • ap. Asia region.
      
    • eu. European Union region.
      
    • na. North America region.
      
• If your company has deployed the Rights Management connector (see whitepaper Leverage the Rights Management Connector for your premises), then requests from the connector will be logged with the service principal that you have configured the connector to use.
  

Understanding the common request types

The number of request types in Azure Rights Management is long and growing. Here are some of the common request types.