一，引言
在實際的項目中，Azure Data Factroy 中的 Data Flow 並不能徹底幫我們完成一系列複製邏輯計算， 比如我們需要針對數據集的每一行數據進行判斷計算，Data Flow 就顯的有些喫力。別怕，Azure Data Factory 提供了調用 Azure Function 的組件，有了代碼的加持，那麼解決更復雜的都能迎刃而解！！那麼就開始今天的表演吧
--------------------我是分割線--------------------↳
1，Azure Data Factory（一）入門簡介
2，Azure Data Factory（二）複製數據
3，Azure Data Factory（三）集成 Azure Devops 實現CI/CD
4，Azure Data Factory（四）集成 Logic App 的郵件通知提醒
5，Azure Data Factory（五）Blob Storage 密鑰管理問題
6，Azure Data Factory（六）數據集類型爲Dataverse的Link測試
7，Azure Data Factory（七）數據集驗證之用戶託管憑證
8，Azure Data Factory（八）數據集驗證之服務主體（Service Principal）
9，Azure Data Factory（九）基礎知識回顧
10，Azure Data Factory（十）Data Flow 組件詳解
11，Azure Data Factory（十一）Data Flow 的使用解析
12，Azure Data Factory（十二）傳參調用 Azure Function
二，正文
1，準備 Azure Function
打開 Azure Portal ，點擊 "Create a resource" 快速創建 Azure Function
以下就是剛創建好的 Azure Function，Operating System 選擇 "Windows"，Runtime 選擇："node js"
添加 名字爲 “Http_skip_holiday” 的 Function
Function Code：
1 const intercept = require("azure-function-log-intercept"); 2 3 module.exports = async function (context, req) { 4 context.log('JavaScript HTTP trigger function processed a request.'); 5 intercept(context); 6 let lo_date = (req.query.lo_date || (req.body && req.body.lo_date)); 7 let skipday = (req.query.skipday || (req.body && req.body.skipday)); 8 context.log("req.body:"+req.body); 9 context.log("lo_date:"+req.body.lo_date); 10 context.log("req.body:"+req.body.skipday); 11 //server Info 12 13 // Holiday Handling 14 let holidayArray = ['2023-01-01','2023-01-06','2023-01-07','2023-01-13','2023-01-14','2023-01-21','2023-01-27','2023-01-28']; 15 context.log("holidayArray.length: ", holidayArray.length); 16 17 let due_dateObj= calculate_dueDate(context,lo_date,holidayArray,skipday) 18 context.log("due_dateObj.Step: ", due_dateObj.Step); 19 context.res = { 20 status: 200, /* Defaults to 200 */ 21 body: due_dateObj 22 }; 23 } 24 25 function calculate_dueDate(context,lodate, holidayArray, num) { 26 "use strict"; 27 let DueDateObj={}; 28 let lo_date = new Date(lodate); 29 let Year = lo_date.getFullYear(); 30 let Month = lo_date.getMonth(); 31 let day = lo_date.getDate(); 32 33 let dueDate; 34 let step = num; 35 let isWorkDay = false; 36 do { 37 38 let currentDate = new Date(Year, Month, day + step); 39 40 if (currentDate.toDateString() in holidayArray || (currentDate.getDay() < 1)) { 41 step++; 42 } else { 43 isWorkDay = true; 44 } 45 } while (!isWorkDay); 46 47 48 dueDate = new Date(Year, Month, day + step); 49 DueDateObj.DueDate=dueDate.toString("yyyy-MM-dd"); 50 DueDateObj.Step=step; 51 context.log("dueDate:"+dueDate.toString("yyyy-MM-dd")); 52 return DueDateObj; 53 }
開啓 Function 後，，我們使用 Postman 進行測試
注意：1）打開 Function 的 Filesystem Logs
2）如果Function 的訪問基本不是 "" 那麼就得在調用 Function 的 Url 後面加上驗證身份的 Code
Postman 進行結果測試
2，Data Factory 中配置調用 Function
1）使用 LookUp 查詢需要更新的數據集
2）利用 Foreach 循環編輯數據集，並根據每一天數據的 "inputdate","skipday" 作爲參數調用 Azure Function
Foreach 的數據集合：
@activity('Lookup_Data').output.value
Function 的 Body 參數配置
@concat('{"lo_date":"',item().inputdate,'","skipday":',item().skipday,'}')
pipeline code
{ "name": "test_pipeline", "properties": { "activities": [ { "name": "Lookup_Data", "type": "Lookup", "dependsOn": [], "policy": { "timeout": "0.12:00:00", "retry": 0, "retryIntervalInSeconds": 30, "secureOutput": false, "secureInput": false }, "userProperties": [], "typeProperties": { "source": { "type": "DelimitedTextSource", "storeSettings": { "type": "AzureBlobStorageReadSettings", "recursive": true, "wildcardFolderPath": "AAA", "wildcardFileName": { "value": "@concat('User_*.csv')", "type": "Expression" }, "enablePartitionDiscovery": false }, "formatSettings": { "type": "DelimitedTextReadSettings" } }, "dataset": { "referenceName": "AZURE_BLOB_CSV", "type": "DatasetReference", "parameters": { "ContainerName": "test", "DirectoryPath": "AAA", "FileName": { "value": "@concat('User_*.csv')", "type": "Expression" } } }, "firstRowOnly": false } }, { "name": "ForEach UPDATE Date", "type": "ForEach", "dependsOn": [ { "activity": "Lookup_Data", "dependencyConditions": [ "Succeeded" ] } ], "userProperties": [], "typeProperties": { "items": { "value": "@activity('Lookup_Data').output.value", "type": "Expression" }, "activities": [ { "name": "Azure_Function_SkipHoliday", "type": "AzureFunctionActivity", "dependsOn": [], "policy": { "timeout": "0.12:00:00", "retry": 0, "retryIntervalInSeconds": 30, "secureOutput": false, "secureInput": false }, "userProperties": [], "typeProperties": { "functionName": "Http_skip_holiday", "method": "POST", "body": { "value": "@concat('{\"lo_date\":\"',item().inputdate,'\",\"skipday\":',item().skipday,'}')", "type": "Expression" } }, "linkedServiceName": { "referenceName": "AzureFunction_LinkService", "type": "LinkedServiceReference" } } ] } } ], "annotations": [] } }
點擊 Debug 進行調試
三，結尾
Azure Data Factory（ADF）是Azure中的託管數據集成服務，允許我們迭代地構建、編排和監視您的ETL工作流程。Azure Functions現在已與ADF集成，允許我們在數據工廠管道中運行Azure函數作爲步驟。大家多多練習！！！
作者：Allen
版權：轉載請在文章明顯位置註明作者及出處。如發現錯誤，歡迎批評指正。