Keras 在fit-generator中獲取驗證數據的y_true和y_preds

在Keras網絡訓練過程中，fit-generator爲我們提供了很多便利。調用fit-generator時，每個epoch訓練結束後會使用驗證數據檢測模型性能，Keras使用model.evaluate_generator提供該功能。然而我遇到了需要提取驗證集y_pred的需求，在網上沒有找到現有的功能實現方法，於是自己對源碼進行了微調，實現了可配置提取驗證集模型預測結果的功能，記錄如下。

原理簡介

通過查看源代碼，發現Keras調用了model.evaluate_generator驗證數據，該函數最終調用的是TensorFlow（我用的後端是tf）的TF_SessionRunCallable函數，封裝得很死，功能是以數據爲輸入，輸出模型預測的結果並與真實標籤比較並計算評價函數得到結果。

過程中不保存、不返回預測結果，這部分沒有辦法修改，但可以在評價數據的同時對數據進行預測，得到結果並記錄下來，傳入到epoch_logs中，隨後在回調函數的on_epoch_end中盡情使用。

代碼修改

Keras版本 2.2.4 其他版本不保證一定使用相同的方法，但大體思路不變

model.fit_generator

找到fit_generator函數定義位置，加入控制參數get_predict：

def fit_generator(self, generator,
                      steps_per_epoch=None,
                      epochs=1,
                      verbose=1,
                      callbacks=None,
                      validation_data=None,
                      validation_steps=None,
                      class_weight=None,
                      max_queue_size=10,
                      workers=1,
                      use_multiprocessing=False,
                      shuffle=True,
                      initial_epoch=0,
                      get_predict = False):    # 加入 get_predict

return training_generator.fit_generator(
            self, generator,
            steps_per_epoch=steps_per_epoch,
            epochs=epochs,
            verbose=verbose,
            callbacks=callbacks,
            validation_data=validation_data,
            validation_steps=validation_steps,
            class_weight=class_weight,
            max_queue_size=max_queue_size,
            workers=workers,
            use_multiprocessing=use_multiprocessing,
            shuffle=shuffle,
            initial_epoch=initial_epoch,
            get_predict = get_predict) # 加入 get_predict

training_generator.fit_generator

找到training_generator.fit_generator定義位置，加入get_predict:

def fit_generator(model,
                  generator,
                  steps_per_epoch=None,
                  epochs=1,
                  verbose=1,
                  callbacks=None,
                  validation_data=None,
                  validation_steps=None,
                  class_weight=None,
                  max_queue_size=10,
                  workers=1,
                  use_multiprocessing=False,
                  shuffle=True,
                  initial_epoch=0,
                  get_predict = False): # 加入 get_predict

修改 # Epoch finished. 註釋後的模塊，可以看到Keras中fit_generator就是用model.evaluate_generator對驗證集評估的：

# Epoch finished.
if steps_done >= steps_per_epoch and do_validation:
    if val_gen:
        
        if get_predict:
            ## 如果啓動獲取預測結果功能，那麼將get_predict設置爲True
            ## 返回值會包括 gts_and_preds 
            val_outs, gts_and_preds = model.evaluate_generator(
                val_enqueuer_gen,
                validation_steps,
                workers=0,
                get_predict=get_predict)
        else:
            val_outs = model.evaluate_generator(
                val_enqueuer_gen,
                validation_steps,
                workers=0)                            
    else:
        # No need for try/except because
        # data has already been validated.
        val_outs = model.evaluate(
            val_x, val_y,
            batch_size=batch_size,
            sample_weight=val_sample_weights,
            verbose=0)
    val_outs = to_list(val_outs)
    # Same labels assumed.
    for l, o in zip(out_labels, val_outs):
        epoch_logs['val_' + l] = o
    
    ## 將返回值 gts_and_preds 保存到 log 中
    if get_predict:
        epoch_logs['val_gts_and_preds'] = gts_and_preds
        
if callback_model.stop_training:
    break

model.evaluate_generator

進入model.evaluate_generator函數，加入get_predict變量：

    def evaluate_generator(self, generator,
                           steps=None,
                           max_queue_size=10,
                           workers=1,
                           use_multiprocessing=False,
                           verbose=0,
                           get_predict=False): # 加入get_predict變量

return training_generator.evaluate_generator(
            self, generator,
            steps=steps,
            max_queue_size=max_queue_size,
            workers=workers,
            use_multiprocessing=use_multiprocessing,
            verbose=verbose,
            get_predict=get_predict) # 加入get_predict變量

training_generator.evaluate_generator

進入training_generator.evaluate_generator，添加get_predict變量，新建三個變量：

def evaluate_generator(model, generator,
                       steps=None,
                       max_queue_size=10,
                       workers=1,
                       use_multiprocessing=False,
                       verbose=0,
                       get_predict=False):  # 加入get_predict變量
    """See docstring for `Model.evaluate_generator`."""
    model._make_test_function()

    if hasattr(model, 'metrics'):
        for m in model.stateful_metric_functions:
            m.reset_states()
        stateful_metric_indices = [
            i for i, name in enumerate(model.metrics_names)
            if str(name) in model.stateful_metric_names]
    else:
        stateful_metric_indices = []

    steps_done = 0
    wait_time = 0.01
    outs_per_batch = []
    batch_sizes = []
      
    if get_predict:
        preds_dict={} # 新建保存結果的dict
        gt_per_batch = [] # 新建 y_true 的 list
        pr_per_batch = [] # 新建 y_pred 的 list

在覈心循環while steps_done < steps:中加入預測變量的內容：

while steps_done < steps:
            generator_output = next(output_generator)
            if not hasattr(generator_output, '__len__'):
                raise ValueError('Output of generator should be a tuple '
                                 '(x, y, sample_weight) '
                                 'or (x, y). Found: ' +
                                 str(generator_output))
            if len(generator_output) == 2:
                x, y = generator_output
                sample_weight = None
            elif len(generator_output) == 3:
                x, y, sample_weight = generator_output
            else:
                raise ValueError('Output of generator should be a tuple '
                                 '(x, y, sample_weight) '
                                 'or (x, y). Found: ' +
                                 str(generator_output))
            outs = model.test_on_batch(x, y, sample_weight=sample_weight)
            outs = to_list(outs)
            outs_per_batch.append(outs)
            
            ## 加入預測功能，保存preds和y_true
            if get_predict:
                preds = model.predict_on_batch(x)
                gt_per_batch.append(y.tolist())
                pr_per_batch.append(preds.tolist())

            if x is None or len(x) == 0:
                # Handle data tensors support when no input given
                # step-size = 1 for data tensors
                batch_size = 1
            elif isinstance(x, list):
                batch_size = x[0].shape[0]
            elif isinstance(x, dict):
                batch_size = list(x.values())[0].shape[0]
            else:
                batch_size = x.shape[0]
            if batch_size == 0:
                raise ValueError('Received an empty batch. '
                                 'Batches should contain '
                                 'at least one item.')
            steps_done += 1
            batch_sizes.append(batch_size)
            if verbose == 1:
                progbar.update(steps_done)
        ## 將結果保存到dict中
        if get_predict:
            preds_dict['y_true'] = gt_per_batch
            preds_dict['y_pred'] = pr_per_batch

修改返回值：

    if get_predict:
        return unpack_singleton(averages), preds_dict
    
    else:
        return unpack_singleton(averages)

至此核心的功能已經實現，但還有一個小問題。

keras.callbacks.TensorBoard._write_logs

Keras的Tensorboard會記錄logs中的內容，但是他只認識 int, float 等數值格式，我們保存在log中的複雜字典他沒辦法寫入tesnorboard，需要對_write_logs做微小的調整：

def _write_logs(self, logs, index):
        for name, value in logs.items():
            if name in ['batch', 'size']:
                continue
            summary = tf.Summary()
            summary_value = summary.value.add()
            if isinstance(value, np.ndarray):
                summary_value.simple_value = value.item()
            ## 跳過我們生成的字典
            elif isinstance(value, dict):
                pass
            else:
                summary_value.simple_value = value
            summary_value.tag = name
            self.writer.add_summary(summary, index)
        self.writer.flush()

大功告成！

測試

隨便寫個帶on_epoch_end的回調函數，將get_predict設置爲True，測試logs中是否有我們想要的數據：

 model.fit_generator(
        generator = train_data_generator,
        steps_per_epoch = 10,
        epochs = config.Epochs,
        verbose = 1,
        use_multiprocessing=False,
        validation_data=val_data_generator,
        validation_steps=10,
        callbacks = callbacks,
        get_predict= True
    )

回調函數設斷點，輸出logs：

logs['val_gts_and_preds']
{'y_pred': [[[2.5419962184969336e-05, 0.9999746084213257],
             [0.6694663763046265, 0.33053362369537354],
             [0.3561754524707794, 0.643824577331543]],
            [[5.548826155499231e-12, 1.0],
             [2.701560219975363e-08, 1.0],
             [4.0011427699937485e-06, 0.9999959468841553]],
            [[7.97858723533551e-11, 1.0],
             [2.3924835659272503e-06, 0.999997615814209],
             [3.359668880875688e-07, 0.9999996423721313]],
            [[0.06622887402772903, 0.9337711930274963],
             [4.1211248458239425e-07, 0.9999996423721313],
             [8.561290087527595e-06, 0.9999914169311523]],
            [[9.313887403550325e-07, 0.9999990463256836],
             [2.614793537247806e-08, 1.0],
             [8.66139725985704e-06, 0.9999912977218628]],
            [[7.047830763440288e-09, 1.0],
             [0.010548637248575687, 0.9894513487815857],
             [1.8744471252940542e-10, 1.0]],
            [[8.760089875714527e-11, 1.0],
             [0.0015734446933493018, 0.9984265565872192],
             [1.5642463040421717e-06, 0.9999984502792358]],
            [[0.004750440828502178, 0.9952495098114014],
             [6.984401466070267e-07, 0.9999992847442627],
             [0.00013592069444712251, 0.9998641014099121]],
            [[7.22906318140204e-11, 1.0],
             [2.402198795437016e-08, 1.0],
             [9.673745138272238e-10, 1.0]],
            [[3.1848256298872e-07, 0.9999996423721313],
             [0.0035940599627792835, 0.9964058995246887],
             [1.9458911912351162e-11, 1.0]]],
 'y_true': [[[0.0, 1.0], [0.0, 1.0], [0.0, 1.0]],
            [[0.0, 1.0], [0.0, 1.0], [0.0, 1.0]],
            [[0.0, 1.0], [0.0, 1.0], [0.0, 1.0]],
            [[0.0, 1.0], [0.0, 1.0], [0.0, 1.0]],
            [[0.0, 1.0], [0.0, 1.0], [0.0, 1.0]],
            [[0.0, 1.0], [0.0, 1.0], [0.0, 1.0]],
            [[0.0, 1.0], [0.0, 1.0], [0.0, 1.0]],
            [[0.0, 1.0], [0.0, 1.0], [0.0, 1.0]],
            [[0.0, 1.0], [0.0, 1.0], [0.0, 1.0]],
            [[0.0, 1.0], [0.0, 1.0], [0.0, 1.0]]]}

之後這些結果任君處置了；

附

將get_predict設爲 False 時則屏蔽了我們做出的所有修改，與原始Keras代碼完全相同；

目前沒有發現其他的問題，有任何不對頭可以隨時交流。

Keras 在fit-generator中獲取驗證數據的y_true和y_preds

原理簡介

代碼修改

model.fit_generator

training_generator.fit_generator

model.evaluate_generator

training_generator.evaluate_generator

keras.callbacks.TensorBoard._write_logs

測試

附

Ubuntu 下 Pytorch, Tensorflow 對應的Python、英偉達顯卡驅動、CUDA、CUDNN版本與環境信息查看方法

Linux 軟鏈接的增、刪、改、查

Keras 在fit-generator中獲取驗證數據的y_true和y_preds

keras 預訓練模型的使用方法

解決 Windows 10 家庭版無法使用NFS服務的問題

https://yachay.unat.edu.pe/blog/index.php?comment_area=format_blog&comment_component=blog&comment_co

linux以太網驅動總結