TensorFlow 篇 | TensorFlow 2.x 基於 HParams 的超參數調優

{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/47/479f52106550097c602b598bb2372647.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"「"},{"type":"text","marks":[{"type":"strong"}],"text":"導語"},{"type":"text","text":"」 當我們創建了 Keras 模型並開始進行訓練時，一般都會指定一些超參數（如學習率）的值來對訓練的過程進行調控，而這些超參數的取值會對模型訓練的結果產生很大的影響，因此在機器學習工作流程中一項十分重要的步驟就是要確定模型超參數的最佳取值，亦即超參數調優。在 TensorFlow 中，我們可以使用 HParams 插件很方便地完成這一調優過程。"}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"什麼是超參數"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"機器學習和深度學習的模型中往往包含成千上萬個參數，其中有的參數可以通過模型訓練並利用反向傳播算法來進行優化，比如模型中的權重 ("},{"type":"codeinline","content":[{"type":"text","text":"weights"}]},{"type":"text","text":") 和偏差 ("},{"type":"codeinline","content":[{"type":"text","text":"bias"}]},{"type":"text","text":") 等，我們稱之爲參數 ("},{"type":"codeinline","content":[{"type":"text","text":"parameters"}]},{"type":"text","text":")。還有一些參數不能通過模型訓練來進行優化，比如學習率 ("},{"type":"codeinline","content":[{"type":"text","text":"learning rate"}]},{"type":"text","text":") 、深度神經網絡中的隱含層 ("},{"type":"codeinline","content":[{"type":"text","text":"hidden layers"}]},{"type":"text","text":") 的個數以及隱含層的神經元 ("},{"type":"codeinline","content":[{"type":"text","text":"hidden units"}]},{"type":"text","text":") 個數等，我們稱之爲超參數 ("},{"type":"codeinline","content":[{"type":"text","text":"hyper parameters"}]},{"type":"text","text":")。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"超參數是用來調節整個模型的訓練過程的，它們只是配置變量，並不直接參與到訓練的過程中，因此需要我們不斷調整與嘗試，以使得模型效果達到最優。需要注意，在一次訓練迭代的過程中，參數是不斷進行更新的，而超參數是恆定不變的。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"一般而言，我們會根據模型在訓練集和驗證集上的損失大小，以及預定義的評估指標來選擇最優超參數組合，並最終將該組超參數應用於正式訓練以及線上的 "},{"type":"codeinline","content":[{"type":"text","text":"Serving"}]},{"type":"text","text":" 服務。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"超參數調優策略"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"模型的超參數一般都有很多，而且每個超參數也會有衆多的候選值，形成的參數空間較大，如果僅依賴人工逐項測試，時間和精力成本無疑是巨大的，因此有許多自動調參算法被提出。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"目前比較常用的超參數自動調優策略包括網格搜索 ("},{"type":"codeinline","content":[{"type":"text","text":"grid search"}]},{"type":"text","text":") 和隨機搜索 ("},{"type":"codeinline","content":[{"type":"text","text":"random search"}]},{"type":"text","text":") 等。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"網格搜索"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"numberedlist","attrs":{"start":"1","normalizeStart":1},"content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":"網格搜索是指在所有候選的超參數取值中，嘗試所有的超參數組合，並選取其中使模型效果達到最優的超參數組合作爲最終解。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"numberedlist","attrs":{"start":"2","normalizeStart":"2"},"content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","text":"比如模型有 "},{"type":"codeinline","content":[{"type":"text","text":"2"}]},{"type":"text","text":" 個超參數，每個超參數有 "},{"type":"codeinline","content":[{"type":"text","text":"3"}]},{"type":"text","text":" 個候選值，那麼所有的超參數組合就有 "},{"type":"codeinline","content":[{"type":"text","text":"3*3=9"}]},{"type":"text","text":" 種，網格搜索會嘗試所有 "},{"type":"codeinline","content":[{"type":"text","text":"9"}]},{"type":"text","text":" 組超參數並從中選擇最佳組合。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"numberedlist","attrs":{"start":"3","normalizeStart":"3"},"content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":3,"align":null,"origin":null},"content":[{"type":"text","text":"網格搜索的缺點是，模型訓練評估的次數會隨超參數數量以及超參數候選值數量的增加而呈指數級增長，從而產生很高的算力和時間成本，因此它並不適用於超參數的數量及候選值數量較多的情況。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"隨機搜索"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"numberedlist","attrs":{"start":"1","normalizeStart":1},"content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":"隨機搜索是指每次選取隨機的超參數組合進行嘗試，它可以手動設置嘗試的次數，避免遍歷整個超參數空間，從而可以減少嘗試的成本。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"numberedlist","attrs":{"start":"2","normalizeStart":"2"},"content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","text":"相比網格搜索，隨機搜索能夠更高效地進行超參數調優，但是隨機搜索並不能保證一定能選取到最優的超參數取值，具有不確定性。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"HParams 超參數調優步驟"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"codeinline","content":[{"type":"text","text":"TensorFlow"}]},{"type":"text","text":" 提供了 "},{"type":"codeinline","content":[{"type":"text","text":"HParams"}]},{"type":"text","text":" 插件來輔助我們進行超參數調優， "},{"type":"codeinline","content":[{"type":"text","text":"HParams"}]},{"type":"text","text":" 插件支持網格搜索和隨機搜索兩種策略。下面以 "},{"type":"codeinline","content":[{"type":"text","text":"Keras"}]},{"type":"text","text":" 模型訓練爲例，介紹使用 "},{"type":"codeinline","content":[{"type":"text","text":"HParams"}]},{"type":"text","text":" 進行超參數調優的步驟。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"定義超參數"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"numberedlist","attrs":{"start":"1","normalizeStart":1},"content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":"使用 "},{"type":"codeinline","content":[{"type":"text","text":"HParam"}]},{"type":"text","text":" 類初始化定義所有的超參數，並指定超參數的取值域 ("},{"type":"codeinline","content":[{"type":"text","text":"Domain"}]},{"type":"text","text":")。 "},{"type":"codeinline","content":[{"type":"text","text":"Domain"}]},{"type":"text","text":" 有三種類型，一種爲 "},{"type":"codeinline","content":[{"type":"text","text":"IntInterval"}]},{"type":"text","text":" 表示連續的整數取值， "},{"type":"codeinline","content":[{"type":"text","text":"Discrete"}]},{"type":"text","text":" 表示離散取值，可以是整數，浮點數以及字符串等， "},{"type":"codeinline","content":[{"type":"text","text":"RealInterval"}]},{"type":"text","text":" 表示連續的浮點數取值。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"numberedlist","attrs":{"start":"2","normalizeStart":"2"},"content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","text":"例 "},{"type":"codeinline","content":[{"type":"text","text":"HP_DEEP_LAYERS = hp.HParam(\"deep_layers\", hp.IntInterval(1, 3))"}]},{"type":"text","text":" 表示連續的整數取值的超參數，"},{"type":"codeinline","content":[{"type":"text","text":"1"}]},{"type":"text","text":" 爲最小值， "},{"type":"codeinline","content":[{"type":"text","text":"3"}]},{"type":"text","text":" 爲最大值，取值範圍是 "},{"type":"codeinline","content":[{"type":"text","text":"[1, 3]"}]},{"type":"text","text":" 。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"numberedlist","attrs":{"start":"3","normalizeStart":"3"},"content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":3,"align":null,"origin":null},"content":[{"type":"text","text":"例 "},{"type":"codeinline","content":[{"type":"text","text":"HP_DEEP_LAYER_SIZE = hp.HParam(\"deep_layer_size\", hp.Discrete([32, 64, 128]))"}]},{"type":"text","text":" 表示離散取值的超參數，可以取參數列表 ("},{"type":"codeinline","content":[{"type":"text","text":"list"}]},{"type":"text","text":") 中的任一元素。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"numberedlist","attrs":{"start":"4","normalizeStart":"4"},"content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":4,"align":null,"origin":null},"content":[{"type":"text","text":"例 "},{"type":"codeinline","content":[{"type":"text","text":"HP_LEARNING_RATE = hp.HParam(\"learning_rate\", hp.RealInterval(0.001, 0.1))"}]},{"type":"text","text":" 表示連續的浮點數取值的超參數， "},{"type":"codeinline","content":[{"type":"text","text":"0.001"}]},{"type":"text","text":" 爲最小值， "},{"type":"codeinline","content":[{"type":"text","text":"0.1"}]},{"type":"text","text":" 爲最大值，取值範圍是 "},{"type":"codeinline","content":[{"type":"text","text":"[0.001, 0.1]"}]},{"type":"text","text":" 。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"定義評估指標"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"numberedlist","attrs":{"start":"1","normalizeStart":1},"content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":"使用 "},{"type":"codeinline","content":[{"type":"text","text":"Metric"}]},{"type":"text","text":" 類定義我們要用到的評估指標，後面會根據這些指標來選取最優的超參數組合。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"numberedlist","attrs":{"start":"2","normalizeStart":"2"},"content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","text":"例 "},{"type":"codeinline","content":[{"type":"text","text":"hp.Metric(\"epoch_auc\", group=\"validation\", display_name=\"auc (val.)\")"}]},{"type":"text","text":" 表示要用到的評估指標爲 "},{"type":"codeinline","content":[{"type":"text","text":"epoch_auc"}]},{"type":"text","text":" 。指標必須是被 "},{"type":"codeinline","content":[{"type":"text","text":"Tensorboard"}]},{"type":"text","text":" 回調函數記錄的或者自定義的標量 ("},{"type":"codeinline","content":[{"type":"text","text":"scalar"}]},{"type":"text","text":") ，一般存儲在日誌文件中，可視化展示時會被 "},{"type":"codeinline","content":[{"type":"text","text":"HPARAMS"}]},{"type":"text","text":" 面板調用並顯示。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"numberedlist","attrs":{"start":"3","normalizeStart":"3"},"content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":3,"align":null,"origin":null},"content":[{"type":"codeinline","content":[{"type":"text","text":"Metric"}]},{"type":"text","text":" 構造函數的第一個參數 "},{"type":"codeinline","content":[{"type":"text","text":"tag"}]},{"type":"text","text":" 表示指標的名稱，對於使用 "},{"type":"codeinline","content":[{"type":"text","text":"Tensorboard"}]},{"type":"text","text":" 回調函數 ("},{"type":"codeinline","content":[{"type":"text","text":"callbacks"}]},{"type":"text","text":") 記錄的指標，其名稱一般爲 "},{"type":"codeinline","content":[{"type":"text","text":"epoch_tag"}]},{"type":"text","text":" 或 "},{"type":"codeinline","content":[{"type":"text","text":"batch_tag"}]},{"type":"text","text":" ，如 "},{"type":"codeinline","content":[{"type":"text","text":"epoch_auc"}]},{"type":"text","text":" 。對於自定義的指標， "},{"type":"codeinline","content":[{"type":"text","text":"tag"}]},{"type":"text","text":" 則爲 "},{"type":"codeinline","content":[{"type":"text","text":"tf.summary.scalar(\"test_auc\", auc, step=1)"}]},{"type":"text","text":" 中設定的名稱，這裏爲 "},{"type":"codeinline","content":[{"type":"text","text":"test_auc"}]},{"type":"text","text":" 。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"numberedlist","attrs":{"start":"4","normalizeStart":"4"},"content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":4,"align":null,"origin":null},"content":[{"type":"codeinline","content":[{"type":"text","text":"Metric"}]},{"type":"text","text":" 構造函數的第二個參數 "},{"type":"codeinline","content":[{"type":"text","text":"group"}]},{"type":"text","text":" 表示指標存儲的路徑，比如訓練的指標存儲在 "},{"type":"codeinline","content":[{"type":"text","text":"train"}]},{"type":"text","text":" 目錄下，驗證的指標存儲在 "},{"type":"codeinline","content":[{"type":"text","text":"validation"}]},{"type":"text","text":" 目錄下，自定義的指標可以存儲在 "},{"type":"codeinline","content":[{"type":"text","text":"test"}]},{"type":"text","text":" 目錄下等，詳見示例程序。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"numberedlist","attrs":{"start":"5","normalizeStart":"5"},"content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":5,"align":null,"origin":null},"content":[{"type":"codeinline","content":[{"type":"text","text":"Metric"}]},{"type":"text","text":" 構造函數的第三個參數 "},{"type":"codeinline","content":[{"type":"text","text":"display_name"}]},{"type":"text","text":" 表示指標在 "},{"type":"codeinline","content":[{"type":"text","text":"HPARAMS"}]},{"type":"text","text":" 面板中顯示的名稱。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"配置 HParams"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"numberedlist","attrs":{"start":"1","normalizeStart":1},"content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":"通過 "},{"type":"codeinline","content":[{"type":"text","text":"hp.hparams_config(hparams=HPARAMS, metrics=METRICS)"}]},{"type":"text","text":" 可以按需設置將要選取的超參數以及用於評估的指標。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"numberedlist","attrs":{"start":"2","normalizeStart":"2"},"content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"codeinline","content":[{"type":"text","text":"hparams_config"}]},{"type":"text","text":" 方法有兩個參數，它們分別表示所有待選超參數 "},{"type":"codeinline","content":[{"type":"text","text":"HParam"}]},{"type":"text","text":" 的列表 ("},{"type":"codeinline","content":[{"type":"text","text":"list"}]},{"type":"text","text":") 和所有評估指標 "},{"type":"codeinline","content":[{"type":"text","text":"Metric"}]},{"type":"text","text":" 的列表 ("},{"type":"codeinline","content":[{"type":"text","text":"list"}]},{"type":"text","text":") 。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"numberedlist","attrs":{"start":"3","normalizeStart":"3"},"content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":3,"align":null,"origin":null},"content":[{"type":"text","text":"如果不進行此項全局設置， "},{"type":"codeinline","content":[{"type":"text","text":"HParams"}]},{"type":"text","text":" 默認會記錄所有在模型中"},{"type":"codeinline","content":[{"type":"text","text":"使用到的"}]},{"type":"text","text":"超參數以及模型輸出的 "},{"type":"codeinline","content":[{"type":"text","text":"所有指標值"}]},{"type":"text","text":" 並在 "},{"type":"codeinline","content":[{"type":"text","text":"HPARAMS"}]},{"type":"text","text":" 面板中顯示。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"構建超參數模型"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"numberedlist","attrs":{"start":"1","normalizeStart":1},"content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":"一般是是將超參數以字典 ("},{"type":"codeinline","content":[{"type":"text","text":"dict"}]},{"type":"text","text":") 的形式傳遞給模型構造函數以完成模型的構建。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"numberedlist","attrs":{"start":"2","normalizeStart":"2"},"content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","text":"模型構造函數的代碼如下所示："}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"python"},"content":[{"type":"text","text":"def model_fn(hparams):\n model = keras.models.Sequential()\n\n for _ in range(hparams[HP_DEEP_LAYERS]):\n model.add(\n keras.layers.Dense(\n units=hparams[HP_DEEP_LAYER_SIZE],\n activation=\"relu\",\n use_bias=True,\n ))\n model.add(keras.layers.Dense(units=1, activation=\"sigmoid\"))\n model.compile(\n optimizer=tf.keras.optimizers.Adam(\n learning_rate=hparams[HP_LEARNING_RATE]),\n loss=tf.keras.losses.BinaryCrossentropy(),\n metrics=[\"AUC\"],\n )\n\n return model"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"numberedlist","attrs":{"start":"3","normalizeStart":"3"},"content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":3,"align":null,"origin":null},"content":[{"type":"text","text":"超參數字典的 "},{"type":"codeinline","content":[{"type":"text","text":"key"}]},{"type":"text","text":" 爲上面定義的 "},{"type":"codeinline","content":[{"type":"text","text":"HParam"}]},{"type":"text","text":" 對象， "},{"type":"codeinline","content":[{"type":"text","text":"value"}]},{"type":"text","text":" 爲基本數據類型的值。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"numberedlist","attrs":{"start":"4","normalizeStart":"4"},"content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":4,"align":null,"origin":null},"content":[{"type":"text","text":"模型的構建過程與一般模型構建相同，只不過將原來固定的參數換成超參數字典中的 "},{"type":"codeinline","content":[{"type":"text","text":"value"}]},{"type":"text","text":" 值。當然，也可以預先定義好一個可配置的 "},{"type":"codeinline","content":[{"type":"text","text":"subclass"}]},{"type":"text","text":" 模型，然後將超參數傳入該模型，即可更加方便地完成模型的構建。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"模型訓練"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"numberedlist","attrs":{"start":"1","normalizeStart":1},"content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":"模型訓練時需要指定 "},{"type":"codeinline","content":[{"type":"text","text":"fit"}]},{"type":"text","text":" 方法的 "},{"type":"codeinline","content":[{"type":"text","text":"callback"}]},{"type":"text","text":" 參數，不僅需要包含 "},{"type":"codeinline","content":[{"type":"text","text":"Tensorboard"}]},{"type":"text","text":" 回調函數，還需要包括 "},{"type":"codeinline","content":[{"type":"text","text":"hp.KerasCallback"}]},{"type":"text","text":" 回調函數。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"numberedlist","attrs":{"start":"2","normalizeStart":"2"},"content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","text":"其中第一個回調函數用於記錄損失 ("},{"type":"codeinline","content":[{"type":"text","text":"loss"}]},{"type":"text","text":") 以及指標 ("},{"type":"codeinline","content":[{"type":"text","text":"metrics"}]},{"type":"text","text":") 的值，第二個回調函數用來記錄本次訓練使用的超參數組合以及計算最後的損失值以及指標值。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"numberedlist","attrs":{"start":"3","normalizeStart":"3"},"content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":3,"align":null,"origin":null},"content":[{"type":"codeinline","content":[{"type":"text","text":"hp.KerasCallback(logdir, hparams)"}]},{"type":"text","text":" 中第一個參數爲記錄 "},{"type":"codeinline","content":[{"type":"text","text":"HParams"}]},{"type":"text","text":" 日誌的目錄，第二個參數爲超參數字典，與傳遞給模型構造函數的字典相同。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"numberedlist","attrs":{"start":"4","normalizeStart":"4"},"content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":4,"align":null,"origin":null},"content":[{"type":"text","text":"如果使用某組超參數執行了多次訓練，那麼最終 "},{"type":"codeinline","content":[{"type":"text","text":"HPARAMS"}]},{"type":"text","text":" 面板的顯示結果爲多次評估結果的平均值。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"可視化調參結果"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"numberedlist","attrs":{"start":"1","normalizeStart":1},"content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":"如果進行 "},{"type":"codeinline","content":[{"type":"text","text":"2"}]},{"type":"text","text":" 次超參數選擇，其日誌根目錄 "},{"type":"codeinline","content":[{"type":"text","text":"mlp"}]},{"type":"text","text":" 的結構如下所示："}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"shell"},"content":[{"type":"text","text":"mlp\n├── 0\n│   ├── events.out.tfevents.1589257272.alexander.4918.34.v2\n│   ├── test\n│   │   └── events.out.tfevents.1589257274.alexander.4918.2418.v2\n│   ├── train\n│   │   └── events.out.tfevents.1589257272.alexander.4918.95.v2\n│   └── validation\n│   └── events.out.tfevents.1589257273.alexander.4918.1622.v2\n├── 1\n│   ├── events.out.tfevents.1589257274.alexander.4918.2575.v2\n│   ├── test\n│   │   └── events.out.tfevents.1589257275.alexander.4918.4958.v2\n│   ├── train\n│   │   └── events.out.tfevents.1589257274.alexander.4918.2636.v2\n│   └── validation\n│   └── events.out.tfevents.1589257274.alexander.4918.4162.v2\n└── events.out.tfevents.1589257272.alexander.4918.5.v2"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"numberedlist","attrs":{"start":"2","normalizeStart":"2"},"content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","text":"其中 "},{"type":"codeinline","content":[{"type":"text","text":"0"}]},{"type":"text","text":" 和 "},{"type":"codeinline","content":[{"type":"text","text":"1"}]},{"type":"text","text":" 目錄分別存儲了一組超參數訓練與驗證後的結果數據。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"numberedlist","attrs":{"start":"3","normalizeStart":"3"},"content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":3,"align":null,"origin":null},"content":[{"type":"text","text":"模型的訓練和驗證結果（包括 "},{"type":"codeinline","content":[{"type":"text","text":"loss"}]},{"type":"text","text":" 以及 "},{"type":"codeinline","content":[{"type":"text","text":"metrics"}]},{"type":"text","text":") 會以 "},{"type":"codeinline","content":[{"type":"text","text":"events.out.tfevents"}]},{"type":"text","text":" 文件的形式保存在 "},{"type":"codeinline","content":[{"type":"text","text":"Tensorboard"}]},{"type":"text","text":" 回調函數指定的目錄下，本示例中爲 "},{"type":"codeinline","content":[{"type":"text","text":"0"}]},{"type":"text","text":" 或 "},{"type":"codeinline","content":[{"type":"text","text":"1"}]},{"type":"text","text":" 目錄下的 "},{"type":"codeinline","content":[{"type":"text","text":"train"}]},{"type":"text","text":" 和 "},{"type":"codeinline","content":[{"type":"text","text":"validation"}]},{"type":"text","text":" 目錄。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"numberedlist","attrs":{"start":"4","normalizeStart":"4"},"content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":4,"align":null,"origin":null},"content":[{"type":"codeinline","content":[{"type":"text","text":"HParams"}]},{"type":"text","text":" 記錄的日誌會保存在 "},{"type":"codeinline","content":[{"type":"text","text":"0"}]},{"type":"text","text":" 或 "},{"type":"codeinline","content":[{"type":"text","text":"1"}]},{"type":"text","text":" 根目錄下的 "},{"type":"codeinline","content":[{"type":"text","text":"events.out.tfevents"}]},{"type":"text","text":" 文件中。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"numberedlist","attrs":{"start":"5","normalizeStart":"5"},"content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":5,"align":null,"origin":null},"content":[{"type":"text","text":"啓動 "},{"type":"codeinline","content":[{"type":"text","text":"Tensorboard"}]},{"type":"text","text":" 並指定其 "},{"type":"codeinline","content":[{"type":"text","text":"logdir"}]},{"type":"text","text":" 參數爲 "},{"type":"codeinline","content":[{"type":"text","text":"mlp"}]},{"type":"text","text":" ，然後選擇 "},{"type":"codeinline","content":[{"type":"text","text":"HPARAMS"}]},{"type":"text","text":" 面板即可看到可視化的調參結果。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"完整超參數調優示例"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"網格搜索示例"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"網格搜索需要遍歷所有的超參數組合，所以此時在初始化 "},{"type":"codeinline","content":[{"type":"text","text":"HParam"}]},{"type":"text","text":" 超參數對象時應該儘量使用 "},{"type":"codeinline","content":[{"type":"text","text":"Discrete"}]},{"type":"text","text":" 域類型，方便數據遍歷。當然 "},{"type":"codeinline","content":[{"type":"text","text":"IntInterval"}]},{"type":"text","text":" 和 "},{"type":"codeinline","content":[{"type":"text","text":"RealInterval"}]},{"type":"text","text":" 域類型的數據也可以通過指定步長的方式來進行遍歷。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如果超參數對象的域類型是 "},{"type":"codeinline","content":[{"type":"text","text":"IntInterval"}]},{"type":"text","text":" 和 "},{"type":"codeinline","content":[{"type":"text","text":"RealInterval"}]},{"type":"text","text":" 時，可以通過該對象的 "},{"type":"codeinline","content":[{"type":"text","text":"domain.min_value"}]},{"type":"text","text":" 和 "},{"type":"codeinline","content":[{"type":"text","text":"domain.max_value"}]},{"type":"text","text":" 屬性獲取超參數候選值的最小值和最大值。如果是 "},{"type":"codeinline","content":[{"type":"text","text":"Discrete"}]},{"type":"text","text":" 類型，可以通過 "},{"type":"codeinline","content":[{"type":"text","text":"domain.values"}]},{"type":"text","text":" 屬性獲取到該超參數所有候選值的列表 ("},{"type":"codeinline","content":[{"type":"text","text":"list"}]},{"type":"text","text":")。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"網格搜索的示例代碼如下所示(搜索步驟參見 "},{"type":"codeinline","content":[{"type":"text","text":"run_all"}]},{"type":"text","text":" 函數)："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"python"},"content":[{"type":"text","text":"import os\nimport tensorflow as tf\nfrom tensorflow import keras\nfrom tensorboard.plugins.hparams import api as hp\nfrom absl import app, flags\nimport shutil\nimport numpy as np\n\nFLAGS = flags.FLAGS\nflags.DEFINE_string(\"logdir\", \"mlp\", \"logs dir\")\n\nHP_DEEP_LAYERS = hp.HParam(\"deep_layers\", hp.IntInterval(1, 3))\nHP_DEEP_LAYER_SIZE = hp.HParam(\"deep_layer_size\", hp.Discrete([32, 64, 128]))\nHP_LEARNING_RATE = hp.HParam(\"learning_rate\", hp.RealInterval(0.001, 0.1))\n\nHPARAMS = [\n HP_DEEP_LAYERS,\n HP_DEEP_LAYER_SIZE,\n HP_LEARNING_RATE,\n]\n\nMETRICS = [\n hp.Metric(\n \"epoch_auc\",\n group=\"validation\",\n display_name=\"auc (val.)\",\n ),\n hp.Metric(\n \"epoch_loss\",\n group=\"validation\",\n display_name=\"loss (val.)\",\n ),\n hp.Metric(\n \"batch_auc\",\n group=\"train\",\n display_name=\"auc (train)\",\n ),\n hp.Metric(\n \"batch_loss\",\n group=\"train\",\n display_name=\"loss (train)\",\n ),\n hp.Metric(\n \"test_auc\",\n group=\"test\",\n display_name=\"auc (test)\",\n ),\n]\n\ndef model_fn(hparams):\n model = keras.models.Sequential()\n\n for _ in range(hparams[HP_DEEP_LAYERS]):\n model.add(\n keras.layers.Dense(\n units=hparams[HP_DEEP_LAYER_SIZE],\n activation=\"relu\",\n use_bias=True,\n ))\n model.add(keras.layers.Dense(units=1, activation=\"sigmoid\"))\n\n model.compile(\n optimizer=tf.keras.optimizers.Adam(\n learning_rate=hparams[HP_LEARNING_RATE]),\n loss=tf.keras.losses.BinaryCrossentropy(),\n metrics=[\"AUC\"],\n )\n\n return model\n\ndef run(data, hparams, base_logdir, session_id):\n model = model_fn(hparams)\n logdir = os.path.join(base_logdir, session_id)\n\n tensorboard_callback = tf.keras.callbacks.TensorBoard(\n log_dir=logdir,\n update_freq=10,\n profile_batch=0,\n )\n hparams_callback = hp.KerasCallback(logdir, hparams)\n\n ((x_train, y_train), (x_val, y_val), (x_test, y_test)) = data\n\n model.fit(\n x=x_train,\n y=y_train,\n epochs=2,\n batch_size=128,\n validation_data=(x_val, y_val),\n callbacks=[tensorboard_callback, hparams_callback],\n )\n\n test_dir = os.path.join(logdir, \"test\")\n with tf.summary.create_file_writer(test_dir).as_default():\n _, auc = model.evaluate(x_test, y_test)\n tf.summary.scalar(\"test_auc\", auc, step=1)\n\ndef prepare_data():\n x_train, y_train = (\n np.random.rand(6000, 32),\n np.random.randint(2, size=(6000, 1)),\n )\n\n x_val, y_val = (\n np.random.rand(1000, 32),\n np.random.randint(2, size=(1000, 1)),\n )\n\n x_test, y_test = (\n np.random.rand(1000, 32),\n np.random.randint(2, size=(1000, 1)),\n )\n\n return ((x_train, y_train), (x_val, y_val), (x_test, y_test))\n\ndef run_all(logdir):\n data = prepare_data()\n with tf.summary.create_file_writer(logdir).as_default():\n hp.hparams_config(hparams=HPARAMS, metrics=METRICS)\n\n session_index = 0\n for deep_layers in range(HP_DEEP_LAYERS.domain.min_value,\n HP_DEEP_LAYERS.domain.max_value):\n for deep_layer_size in HP_DEEP_LAYER_SIZE.domain.values:\n for learning_rate in np.arange(HP_LEARNING_RATE.domain.min_value,\n HP_LEARNING_RATE.domain.max_value,\n 0.01):\n hparams = {\n HP_DEEP_LAYERS: deep_layers,\n HP_DEEP_LAYER_SIZE: deep_layer_size,\n HP_LEARNING_RATE: learning_rate,\n }\n session_id = str(session_index)\n session_index += 1\n print(\"--- Running training session %d\" % (session_index))\n hparams_string = str(hparams)\n print(hparams_string)\n run(\n data=data,\n hparams=hparams,\n base_logdir=logdir,\n session_id=session_id,\n )\n\ndef main(argv):\n del argv # Unused args\n logdir = FLAGS.logdir\n shutil.rmtree(logdir, ignore_errors=True)\n print(\"Saving output to %s.\" % logdir)\n run_all(logdir=logdir)\n print(\"Done. Output saved to %s.\" % logdir)\n\nif __name__ == \"__main__\":\n app.run(main)"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"隨機搜索示例"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通過調用超參數對象的域 ("},{"type":"codeinline","content":[{"type":"text","text":"domain"}]},{"type":"text","text":") 屬性的 "},{"type":"codeinline","content":[{"type":"text","text":"sample_uniform()"}]},{"type":"text","text":" 方法可以從該超參數的候選值中隨機選取一個值，然後就可以使用隨機生成的超參數組合進行訓練了。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"codeinline","content":[{"type":"text","text":"sample_uniform"}]},{"type":"text","text":" 方法還可以接收一個帶有種子 ("},{"type":"codeinline","content":[{"type":"text","text":"seed"}]},{"type":"text","text":") 的僞隨機數生成器，如 "},{"type":"codeinline","content":[{"type":"text","text":"random.Random(seed)"}]},{"type":"text","text":"，這在分佈式訓練的超參數調優過程中十分重要，通過指定同一個僞隨機數生成器，可以保證所有 "},{"type":"codeinline","content":[{"type":"text","text":"worker"}]},{"type":"text","text":" 節點每次獲取到的超參數組合都是一致的，從而確保分佈式訓練能夠正常進行。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"隨機搜索 "},{"type":"codeinline","content":[{"type":"text","text":"run_all"}]},{"type":"text","text":" 部分的代碼如下所示，其它部分的代碼與網格搜索相同。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"python"},"content":[{"type":"text","text":"def run_all(logdir):\n data = prepare_data()\n with tf.summary.create_file_writer(logdir).as_default():\n hp.hparams_config(hparams=HPARAMS, metrics=METRICS)\n\n session_index = 0\n for _ in range(8):\n hparams = {h: h.domain.sample_uniform() for h in HPARAMS}\n hparams_string = str(hparams)\n session_id = str(session_index)\n session_index += 1\n print(\"--- Running training session %d\" % (session_index))\n print(hparams_string)\n run(\n data=data,\n hparams=hparams,\n base_logdir=logdir,\n session_id=session_id,\n )"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"HPARAMS 面板"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"啓動 "},{"type":"codeinline","content":[{"type":"text","text":"Tensorboard"}]},{"type":"text","text":" 後，即可在頁面的上方看到 "},{"type":"codeinline","content":[{"type":"text","text":"HPARAMS"}]},{"type":"text","text":" 選項，點擊該選項就能夠看到 "},{"type":"codeinline","content":[{"type":"text","text":"HPARAMS"}]},{"type":"text","text":" 面板 ("},{"type":"codeinline","content":[{"type":"text","text":"Dashboard"}]},{"type":"text","text":") 了。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"codeinline","content":[{"type":"text","text":"HPARAMS"}]},{"type":"text","text":" 面板提供了左右兩個窗格，左邊的窗格提供了篩選功能，右邊的窗格提供了可視化評估結果的功能，下面來分別對它們的功用進行說明。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"篩選窗格"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"篩選窗格提供了篩選功能，以控制右邊窗格的可視化渲染。它可以選擇用於展示的超參數以及指標，可以篩選被展示的超參數以及指標的值，還可以對可視化的結果進行排序等。如下圖所示："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/a1/a158992ef68f14802e0096196a20447d.png","alt":"選擇顯示某些超參數和指標","title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"可視化窗格"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"可視化窗格包含三個視圖 ("},{"type":"codeinline","content":[{"type":"text","text":"view"}]},{"type":"text","text":") ，分別包含不同的信息。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"表格視圖 ("},{"type":"codeinline","content":[{"type":"text","text":"Table View"}]},{"type":"text","text":") 以表格的形式列出了所有的超參數組合以及對應的各項指標的值，還可以通過點擊 "},{"type":"codeinline","content":[{"type":"text","text":"Show Metrics"}]},{"type":"text","text":" 來展示指標隨 "},{"type":"codeinline","content":[{"type":"text","text":"batch"}]},{"type":"text","text":" 或 "},{"type":"codeinline","content":[{"type":"text","text":"epoch"}]},{"type":"text","text":" 的變化趨勢圖。如下圖所示："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/39/39e3f996858e2a72309459b9d24d50f8.gif","alt":"表格視圖","title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"平行座標視圖 ("},{"type":"codeinline","content":[{"type":"text","text":"Parallel Coordinates View"}]},{"type":"text","text":") 由一系列表示超參數和指標的豎向座標軸組成，對於每一超參數的取值以及對應的指標值，都會通過一條線連接起來。點擊任意一條線都會對該組取值進行高亮顯示，可以在座標軸上用鼠標標記一個區域，這時只會顯示在該區域內的取值，這對於判斷哪組超參數更爲重要十分有幫助。如下圖所示："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/85/85f4fd5a0c924f6303021542c67987fc.gif","alt":"平行座標視圖","title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"散點圖視圖 ("},{"type":"codeinline","content":[{"type":"text","text":"Scatter Plot View"}]},{"type":"text","text":") 由一系列超參數與指標之間關聯的散點圖組成，它可以幫助發現超參數之間或超參數與指標之間的潛在關聯。如下圖所示："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/c3/c3afdf3b46bb3c0d1c5d4636db29f388.gif","alt":"散點圖視圖","title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"注意事項"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"numberedlist","attrs":{"start":"1","normalizeStart":1},"content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"codeinline","content":[{"type":"text","text":"RealInterval"}]},{"type":"text","text":" 的取值如果從 "},{"type":"codeinline","content":[{"type":"text","text":"0"}]},{"type":"text","text":" 開始，則要以浮點數 "},{"type":"codeinline","content":[{"type":"text","text":"0."}]},{"type":"text","text":" 形式表示。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"numberedlist","attrs":{"start":"2","normalizeStart":"2"},"content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","text":"每一組超參數都需要獨立的訓練過程，所以要將不同組超參數訓練的日誌文件寫到不同的目錄下。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"numberedlist","attrs":{"start":"3","normalizeStart":"3"},"content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":3,"align":null,"origin":null},"content":[{"type":"text","text":"在進行超參數調優時，模型訓練方法 "},{"type":"codeinline","content":[{"type":"text","text":"fit"}]},{"type":"text","text":" 中的 "},{"type":"codeinline","content":[{"type":"text","text":"metrics"}]},{"type":"text","text":" 參數要設置爲字符串類型或者一個全局的 "},{"type":"codeinline","content":[{"type":"text","text":"Metric"}]},{"type":"text","text":" 對象。這樣 "},{"type":"codeinline","content":[{"type":"text","text":"Tensorboard"}]},{"type":"text","text":" 記錄的 "},{"type":"codeinline","content":[{"type":"text","text":"metrics"}]},{"type":"text","text":" 名稱才能在多次訓練中保持一致如 "},{"type":"codeinline","content":[{"type":"text","text":"epoch_auc"}]},{"type":"text","text":"，而不會出現 "},{"type":"codeinline","content":[{"type":"text","text":"epoch_auc_1"}]},{"type":"text","text":"， "},{"type":"codeinline","content":[{"type":"text","text":"epoch_"},{"type":"text","marks":[{"type":"italic"}],"text":"auc_"},{"type":"text","text":"2"}]},{"type":"text","text":" 這種情況，從而使得 "},{"type":"codeinline","content":[{"type":"text","text":"HPARAMS"}]},{"type":"text","text":" 面板能夠正常獲取到 "},{"type":"codeinline","content":[{"type":"text","text":"metrics"}]},{"type":"text","text":" 的值並進行展示。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"numberedlist","attrs":{"start":"4","normalizeStart":"4"},"content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":4,"align":null,"origin":null},"content":[{"type":"text","text":"在分佈式訓練中進行超參數隨機搜索時，需要指定一個帶種子的僞隨機數生成器，使得每個 "},{"type":"codeinline","content":[{"type":"text","text":"worker"}]},{"type":"text","text":" 節點選取到的隨機值都一致，從而確保分佈式訓練能夠正常進行。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"參考資料"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"numberedlist","attrs":{"start":"1","normalizeStart":1},"content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https://www.tensorflow.org/tensorboard/hyperparametertuningwith_hparams","title":"Hyperparameter Tuning with the HParams Dashboard"},"content":[{"type":"text","text":"Hyperparameter Tuning with the HParams Dashboard"}]}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https://github.com/tensorflow/tensorboard/blob/master/tensorboard/plugins/hparams/hparams_demo.py","title":"Hparams demo"},"content":[{"type":"text","text":"Hparams demo"}]}]}]}]}]}