keras 0.x 升級2.x 後，multimodel多模態融合層

Keras 0.x

• Merge-在Keras的早期版本中，用於合併來自2個或更多Sequential 模型的輸入，並且在舊的Graph容器中也內部使用。 該層將模型作爲layer參數，而不是tensor。
• 如果您想要自定義合併模式，則可以傳遞一個lambda作爲mode參數。 

model1 = Sequential()
model1.add(...)

model2 = Sequential()
model2.add(...)

model3 = Sequential()
model3.add(Merge([model1, model2], mode='sum')
model3.add(...)

model.3.compile(...)
model3.fit(...)

Keras 1.x

• merge-在Keras 1.0中，引入了Functional API，merge函數在Merge對象的簡單函數包裝器，此函數執行在tensor上，而不是layer模型上。

input1 = Input((10,))
x1 = Dense(10)(input1)

input2 = Input((10,))
x2 = Dense(10)(input2)

y = merge([x1, x2], 'sum') 

Keras 2.x

• 在Keras 2.x中，代替單個Merge層和切換不同模式的一個mode的merge，我們爲每個模式用對應的分離層。可以用Sequential模型layer或者包裝的Functional API函數。

input1 = Input((10,))
x1 = Dense(10)(input1)

input2 = Input((10,))
x2 = Dense(10)(input2)

y = add([x1, x2])   # other modes: multiply, concatenate, dot

# Alternatively, using a layer object:

y = Add()([x1, x2])  # other layers: Multiply, Concatenate, Dot

如果您想要自定義merge模式，則可以編寫一個繼承keras.layers._Merge的自定義層（請參閱keras / layers / merge.py中現有合併層和函數的代碼，或者使用lambda層：

def weighted_sum(X):
    x1, x2 = X
    return 0.2 * x1 + 0.8 * x2

y = Lambda(weighted_sum, output_shape=(None, dim))   # output_shape argument not required for TF backend

Keras 0.x到Keras 2.x

用Sequential + Merge函數

model1 = Sequential()
model1.add(...)

model2 = Sequential()
model2.add(...)

x = add(model1.output, model2.output])

model3 = Sequential()
model3.add(...)

model3_output = model3(x)

model = Model([model1.input, model2.input], model3_output)  # wrap everything up in a functional 
model.compile(...)
model.fit(...)

 

Keras 0.x 多模態融合的例子

def linear_model_combined(optimizer='Adadelta'):    
    modela = Sequential()
    modela.add(Flatten(input_shape=(100, 34)))
    modela.add(Dense(1024))
    modela.add(Activation('relu'))
    modela.add(Dense(512))

    modelb = Sequential()
    modelb.add(Flatten(input_shape=(100, 34)))
    modelb.add(Dense(1024))
    modelb.add(Activation('relu'))
    modelb.add(Dense(512))

    model_combined = Sequential()

    model_combined.add(Merge([modela, modelb], mode='concat'))

    model_combined.add(Activation('relu'))
    model_combined.add(Dense(256))
    model_combined.add(Activation('relu'))

    model_combined.add(Dense(4))
    model_combined.add(Activation('softmax'))

    model_combined.compile(loss='categorical_crossentropy', optimizer=optimizer, metrics=['accuracy'])

    return model_combined

 

Keras 2.x

用Functional API的多模型融合

def linear_model_combined(optimizer='Adadelta'):    

    # declare input
    inlayer =Input(shape=(100, 34))
    flatten = Flatten()(inlayer)

    modela = Dense(1024)(flatten)
    modela = Activation('relu')(modela)
    modela = Dense(512)(modela)

    modelb = Dense(1024)(flatten)
    modelb = Activation('relu')(modelb)
    modelb = Dense(512)(modelb)

    model_concat = concatenate([modela, modelb])


    model_concat = Activation('relu')(model_concat)
    model_concat = Dense(256)(model_concat)
    model_concat = Activation('relu')(model_concat)

    model_concat = Dense(4)(model_concat)
    model_concat = Activation('softmax')(model_concat)

    model_combined = Model(inputs=inlayer,outputs=model_concat)

    model_combined.compile(loss='categorical_crossentropy', optimizer=optimizer, metrics=['accuracy'])

    return model_combined

用 Sequential，多模型融合

def linear_model_combined(optimizer='Adadelta'): 
    from keras.models import Model, Sequential
    from keras.layers.core import Dense, Flatten, Activation, Dropout
    from keras.layers import add

    modela = Sequential()
    modela.add(Flatten(input_shape=(100, 34)))
    modela.add(Dense(1024))
    modela.add(Activation('relu'))
    modela.add(Dense(512))

    modelb = Sequential()
    modelb.add(Flatten(input_shape=(100, 34)))
    modelb.add(Dense(1024))
    modelb.add(Activation('relu'))
    modelb.add(Dense(512))

    merged_output = add([modela.output, modelb.output])   

    model_combined = Sequential()
    model_combined.add(Activation('relu'))
    model_combined.add(Dense(256))
    model_combined.add(Activation('relu'))
    model_combined.add(Dense(4))
    model_combined.add(Activation('softmax'))

    final_model = Model([modela.input, modelb.input], model_combined(merged_output))

    final_model.compile(loss='categorical_crossentropy', optimizer=optimizer, metrics=['accuracy'])

    return final_model