這次要分享的是一個storage account log分析的solution，首先還是按照老規矩，先講下這個事的背景，以及我們需要這麼做的原因，我們都知道storage account可以作爲非常實用的保存數據的存儲，擴展性極強，便於使用，無需維護，還是按需付費，所以越來越多數據開始被存放在storage account中，但是同時，我們也需要越來越重視storage account的安全與合規，比如可以藉助global azure security center中的功能，對storage account中的文件進行病毒掃描，另外，還可以對storage account使用service endpoint或者private endpoint來把他變成一個私有的服務
這些其實都不是我們要講的重點，我們今天要講的是如何使用Log Analytics分析storage account訪問log，爲什麼需要這麼做呢？Storage access的log是存儲在storage裏的，查閱並不方便，都是一些像IIS Log這種的字段，一般我的做法是下載下來之後手動轉成CSV，然後利用excel進行分析，這種做法也很簡單，但是步驟其實比較繁瑣，而如果可以直接用Log Analytics進行分析的話相當於就是直接查詢數據庫裏數據的感覺，也不需要頻繁的轉換數據了，具體怎麼做可以看下
首先第一步我們需要開啓storage account的訪問log，有些人可能不太清楚，默認其實是不會記錄詳細的訪問log的，需要在diagnostic這裏開啓
開啓之後，嘗試訪問幾次，等待一段時間之後，在storage explorer裏就可以看到log了
而拿到的數據默認都是這種格式
2.0;2020-12-29T04:09:17.6971151Z;GetBlob;SASSuccess;200;681;3;sas;;mxyarmtemplate;blob;"https://mxyarmtemplate.blob.core.windows.net:443/template/Linuxvmdeploy.json?sv=2019-12-12&si=testpolicy&sr=c&sig=XXXXX";"/mxyarmtemplate/template/Linuxvmdeploy.json";c76b0385-401e-004d-3f98-dd4ba9000000;0;124.126.17.6:50735;2019-12-12;294;0;410;6805;0;;;""0x8D89F61D1748E94"";Sunday, 13-Dec-20 12:22:54 GMT;;"Mozilla/5.0 (Windows NT; Windows NT 10.0; zh-CN) WindowsPowerShell/5.1.18362.1171";;;;;;;;;;
2.0;2020-12-29T04:09:23.4252152Z;GetBlob;SASSuccess;200;728;4;sas;;mxyarmtemplate;blob;"https://mxyarmtemplate.blob.core.windows.net:443/template/Linuxvmdeploy.json?sv=2019-12-12&si=testpolicy&sr=c&sig=XXXXX";"/mxyarmtemplate/template/Linuxvmdeploy.json";c76b0db9-401e-004d-2a98-dd4ba9000000;0;124.126.17.6:50735;2019-12-12;294;0;410;6805;0;;;""0x8D89F61D1748E94"";Sunday, 13-Dec-20 12:22:54 GMT;;"Mozilla/5.0 (Windows NT; Windows NT 10.0; zh-CN) WindowsPowerShell/5.1.18362.1171";;;;;;;;;;這種格式是沒辦法被log analytics接受的，我們目前採用的辦法是把storage account的訪問log下載下來，然後手動上傳到log analytics的方式，log analytics可以接受的格式是JSON，所以我們需要先把這種格式準換成JSON，可以用下邊的腳本來做這件事
Function ConvertSemicolonToURLEncoding([String] $InputText)
{
$ReturnText = ""
$chars = $InputText.ToCharArray()
$StartConvert = $false
foreach($c in $chars)
{
if($c -eq '"') {
$StartConvert = ! $StartConvert
}
if($StartConvert -eq $true -and $c -eq ';')
{
$ReturnText += "%3B"
} else {
$ReturnText += $c
}
}
return $ReturnText
}
Function FormalizeJsonValue($Text)
{
$Text1 = ""
if($Text.IndexOf("`"") -eq 0) { $Text1=$Text } else {$Text1="`"" + $Text+ "`""}
if($Text1.IndexOf("%3B") -ge 0) {
$ReturnText = $Text1.Replace("%3B", ";")
} else {
$ReturnText = $Text1
}
return $ReturnText
}
Function ConvertLogLineToJson([String] $logLine)
{
$logLineEncoded = ConvertSemicolonToURLEncoding($logLine)
$elements = $logLineEncoded.split(';')
$FormattedElements = New-Object System.Collections.ArrayList
foreach($element in $elements)
{
$NewText = FormalizeJsonValue($element)
$FormattedElements.Add($NewText) > null
}
$Columns =
( "version-number",
"request-start-time",
"operation-type",
"request-status",
"http-status-code",
"end-to-end-latency-in-ms",
"server-latency-in-ms",
"authentication-type",
"requester-account-name",
"owner-account-name",
"service-type",
"request-url",
"requested-object-key",
"request-id-header",
"operation-count",
"requester-ip-address",
"request-version-header",
"request-header-size",
"request-packet-size",
"response-header-size",
"response-packet-size",
"request-content-length",
"request-md5",
"server-md5",
"etag-identifier",
"last-modified-time",
"conditions-used",
"user-agent-header",
"referrer-header",
"client-request-id"
)
# Propose json payload
$logJson = "[{";
For($i = 0;$i -lt $Columns.Length;$i++)
{
$logJson += "`"" + $Columns[$i] + "`":" + $FormattedElements[$i]
if($i -lt $Columns.Length - 1) {
$logJson += ","
}
}
$logJson += "}]";
return $logJson
}這幾個函數可以把storage account的訪問log直接轉換成JSON格式
轉換完成後，結合下邊的腳本把log post到log analytics就可以了，注意需要把customid和key之類的變量替換成自己的實際值
$TimeStampField = ""
$LogType = ""
$SharedKey = ""
$CustomerId = ""
$ResourceGroup = ""
Function Build-Signature ($customerId, $sharedKey, $date, $contentLength, $method, $contentType, $resource)
{
$xHeaders = "x-ms-date:" + $date
$stringToHash = $method + "`n" + $contentLength + "`n" + $contentType + "`n" + $xHeaders + "`n" + $resource
$bytesToHash = [Text.Encoding]::UTF8.GetBytes($stringToHash)
$keyBytes = [Convert]::FromBase64String($sharedKey)
$sha256 = New-Object System.Security.Cryptography.HMACSHA256
$sha256.Key = $keyBytes
$calculatedHash = $sha256.ComputeHash($bytesToHash)
$encodedHash = [Convert]::ToBase64String($calculatedHash)
$authorization = 'SharedKey {0}:{1}' -f $customerId,$encodedHash
return $authorization
}
#
# Create the function to create and post the request
#
Function Post-LogAnalyticsData($customerId, $sharedKey, $body, $logType)
{
<#
CustomerID Log Analytics 工作區的唯一標識符。
#>
$method = "POST"
$contentType = "application/json"
$resource = "/api/logs"
$rfc1123date = [DateTime]::UtcNow.ToString("r")
$contentLength = $body.Length
$signature = Build-Signature `
-customerId $customerId `
-sharedKey $sharedKey `
-date $rfc1123date `
-contentLength $contentLength `
-method $method `
-contentType $contentType `
-resource $resource
$uri = "https://" + $customerId + ".ods.opinsights.azure.com" + $resource + "?api-version=2016-04-01"
$headers = @{
"Authorization" = $signature;
"Log-Type" = $logType;
"x-ms-date" = $rfc1123date;
"time-generated-field" = $TimeStampField;
}
$response = Invoke-WebRequest -Uri $uri -Method $method -ContentType $contentType -Headers $headers -Body $body -UseBasicParsing
return $response.StatusCode
}
$storageAccount = Get-AzStorageAccount -ResourceGroupName $ResourceGroup -Name $StorageAccountName -ErrorAction SilentlyContinue
if($null -eq $storageAccount)
{
throw "The storage account specified does not exist in this subscription."
}
$storageContext = $storageAccount.Context
$containers = New-Object System.Collections.ArrayList
$container = Get-AzStorageContainer -Context $storageContext -Name "$ContainerName" -ErrorAction SilentlyContinue |
ForEach-Object { $containers.Add($_) } | Out-Null
Write-Output("> Container count: {0}" -f $containers.Count)
$token = $Null
$maxReturn = 5000
$successPost = 0
$failedPost = 0
# Enumerate containers
$containers | ForEach-Object {
$container = $_.CloudBlobContainer
Write-Output("> Reading container {0}" -f $container.Name)
do {
$blobs = Get-AzStorageBlob -Context $storageContext -Container $container.Name -MaxCount $maxReturn -ContinuationToken $token
if($Null -eq $blobs) {
break
}
#Set-StrictMode will cause Get-AzStorageBlob returns result in different data types when there is only one blob
if($blobs.GetType().Name -eq "AzureStorageBlob") {
$token = $Null
} else {
$token = $blobs[$blobs.Count - 1].ContinuationToken;
}
# Enumerate log blobs
foreach($blob in $blobs)
{
Write-Output("> Downloading blob: {0}" -f $blob.Name)
$filename = ".\log.txt"
Get-AzStorageBlobContent -Context $storageContext -Container $container.Name -Blob $blob.Name -Destination $filename -Force > Null
Write-Output("> Posting logs to log analytic workspace: {0}" -f $blob.Name)
$lines = Get-Content $filename
foreach($line in $lines)
{
$json = ConvertLogLineToJson($line)
$response = Post-LogAnalyticsData -customerId $customerId -sharedKey $sharedKey -body ([System.Text.Encoding]::UTF8.GetBytes($json)) -logType $logType
if($response -eq "200") {
$successPost++
} else {
$failedPost++
Write-Output "> Failed to post one log to Log Analytics workspace"
}
}
remove-item $filename -Force
}
}
While ($token -ne $Null)
Write-Output "> Log lines posted to Log Analytics workspace: success = $successPost, failure = $failedPost"
}腳本運行之後，可以看到已經在下載、轉換、然後上傳log了
LA裏已經可以查到log了，這個StorageAccountLog是在腳本里自己設置的LogType，不是自動生成的
