keras 0.x 升级2.x 后，multimodel多模态融合层

Keras 0.x

• Merge-在Keras的早期版本中，用于合并来自2个或更多Sequential 模型的输入，并且在旧的Graph容器中也内部使用。 该层将模型作为layer参数，而不是tensor。
• 如果您想要自定义合并模式，则可以传递一个lambda作为mode参数。 

model1 = Sequential()
model1.add(...)

model2 = Sequential()
model2.add(...)

model3 = Sequential()
model3.add(Merge([model1, model2], mode='sum')
model3.add(...)

model.3.compile(...)
model3.fit(...)

Keras 1.x

• merge-在Keras 1.0中，引入了Functional API，merge函数在Merge对象的简单函数包装器，此函数执行在tensor上，而不是layer模型上。

input1 = Input((10,))
x1 = Dense(10)(input1)

input2 = Input((10,))
x2 = Dense(10)(input2)

y = merge([x1, x2], 'sum') 

Keras 2.x

• 在Keras 2.x中，代替单个Merge层和切换不同模式的一个mode的merge，我们为每个模式用对应的分离层。可以用Sequential模型layer或者包装的Functional API函数。

input1 = Input((10,))
x1 = Dense(10)(input1)

input2 = Input((10,))
x2 = Dense(10)(input2)

y = add([x1, x2])   # other modes: multiply, concatenate, dot

# Alternatively, using a layer object:

y = Add()([x1, x2])  # other layers: Multiply, Concatenate, Dot

如果您想要自定义merge模式，则可以编写一个继承keras.layers._Merge的自定义层（请参阅keras / layers / merge.py中现有合并层和函数的代码，或者使用lambda层：

def weighted_sum(X):
    x1, x2 = X
    return 0.2 * x1 + 0.8 * x2

y = Lambda(weighted_sum, output_shape=(None, dim))   # output_shape argument not required for TF backend

Keras 0.x到Keras 2.x

用Sequential + Merge函数

model1 = Sequential()
model1.add(...)

model2 = Sequential()
model2.add(...)

x = add(model1.output, model2.output])

model3 = Sequential()
model3.add(...)

model3_output = model3(x)

model = Model([model1.input, model2.input], model3_output)  # wrap everything up in a functional 
model.compile(...)
model.fit(...)

 

Keras 0.x 多模态融合的例子

def linear_model_combined(optimizer='Adadelta'):    
    modela = Sequential()
    modela.add(Flatten(input_shape=(100, 34)))
    modela.add(Dense(1024))
    modela.add(Activation('relu'))
    modela.add(Dense(512))

    modelb = Sequential()
    modelb.add(Flatten(input_shape=(100, 34)))
    modelb.add(Dense(1024))
    modelb.add(Activation('relu'))
    modelb.add(Dense(512))

    model_combined = Sequential()

    model_combined.add(Merge([modela, modelb], mode='concat'))

    model_combined.add(Activation('relu'))
    model_combined.add(Dense(256))
    model_combined.add(Activation('relu'))

    model_combined.add(Dense(4))
    model_combined.add(Activation('softmax'))

    model_combined.compile(loss='categorical_crossentropy', optimizer=optimizer, metrics=['accuracy'])

    return model_combined

 

Keras 2.x

用Functional API的多模型融合

def linear_model_combined(optimizer='Adadelta'):    

    # declare input
    inlayer =Input(shape=(100, 34))
    flatten = Flatten()(inlayer)

    modela = Dense(1024)(flatten)
    modela = Activation('relu')(modela)
    modela = Dense(512)(modela)

    modelb = Dense(1024)(flatten)
    modelb = Activation('relu')(modelb)
    modelb = Dense(512)(modelb)

    model_concat = concatenate([modela, modelb])


    model_concat = Activation('relu')(model_concat)
    model_concat = Dense(256)(model_concat)
    model_concat = Activation('relu')(model_concat)

    model_concat = Dense(4)(model_concat)
    model_concat = Activation('softmax')(model_concat)

    model_combined = Model(inputs=inlayer,outputs=model_concat)

    model_combined.compile(loss='categorical_crossentropy', optimizer=optimizer, metrics=['accuracy'])

    return model_combined

用 Sequential，多模型融合

def linear_model_combined(optimizer='Adadelta'): 
    from keras.models import Model, Sequential
    from keras.layers.core import Dense, Flatten, Activation, Dropout
    from keras.layers import add

    modela = Sequential()
    modela.add(Flatten(input_shape=(100, 34)))
    modela.add(Dense(1024))
    modela.add(Activation('relu'))
    modela.add(Dense(512))

    modelb = Sequential()
    modelb.add(Flatten(input_shape=(100, 34)))
    modelb.add(Dense(1024))
    modelb.add(Activation('relu'))
    modelb.add(Dense(512))

    merged_output = add([modela.output, modelb.output])   

    model_combined = Sequential()
    model_combined.add(Activation('relu'))
    model_combined.add(Dense(256))
    model_combined.add(Activation('relu'))
    model_combined.add(Dense(4))
    model_combined.add(Activation('softmax'))

    final_model = Model([modela.input, modelb.input], model_combined(merged_output))

    final_model.compile(loss='categorical_crossentropy', optimizer=optimizer, metrics=['accuracy'])

    return final_model