Adaptation module design for stacking two CNNs

I’m trying to stack two different CNNs using an adaptation module to bridge them, but I’m having a hard time determining the adaption module’s layer hyperparameters correctly.

To be more precise, I would like to train the adaptation module to bridge two convolutional layers:

1. Layer A with output shape: (29,29,256)
2. Layer B with input shape: (8,8,384)

So, after Layer A, I sequentially add the adaptation module, for which I choose:

• Conv2D layer with 384 filters with kernel size: (3,3) / Output shape: (29,29,384)
• MaxPool2D with pool size: (2,2), strides: (4,4) and padding: “same” / Output shape: (8,8,384)

Finally, I try to add layer B to the model, but I get the following error from tensorflow:

InvalidArgumentError: Dimensions must be equal, but are 384 and 288 for '{{node batch_normalization_159/FusedBatchNormV3}} = FusedBatchNormV3[T=DT_FLOAT, U=DT_FLOAT, data_format="NHWC", epsilon=0.001, exponential_avg_factor=1, is_training=false](Placeholder, batch_normalization_159/scale, batch_normalization_159/ReadVariableOp, batch_normalization_159/FusedBatchNormV3/ReadVariableOp, batch_normalization_159/FusedBatchNormV3/ReadVariableOp_1)' with input shapes: [?,8,8,384], [288], [288], [288], [288].

There’s a minimal reproducible example of it:

from keras.applications.inception_resnet_v2 import InceptionResNetV2
from keras.applications.mobilenet import MobileNet
from keras.layers import Conv2D, MaxPool2D
from keras.models import Sequential

mobile_model = MobileNet(weights='imagenet')
server_model = InceptionResNetV2(weights='imagenet')

hybrid = Sequential()

for i, layer in enumerate(mobile_model.layers):
  if i <= 36:
    layer.trainable = False
    hybrid.add(layer)

hybrid.add(Conv2D(384, kernel_size=(3,3), padding='same'))
hybrid.add(MaxPool2D(pool_size=(2,2), strides=(4,4), padding='same'))

for i, layer in enumerate(server_model.layers):
  if i >= 610:
    layer.trainable = False
    hybrid.add(layer)

Answer

Sequential models only support models where the layers are arranged like a linked list – each layer takes the output of only one layer, and each layer’s output is only fed to a single layer. Your two base models have residual blocks, which breaks the above assumption, and turns the model architecture into directed acyclic graph (DAG).

To do what you want to do, you’ll need to use the Functional API. With the Functional API, you explicitly control the intermediate activations aka KerasTensors.

For the first model, you can skip that extra work and just create a new model from a subset of the existing graph like this

sub_mobile = keras.models.Model(mobile_model.inputs, mobile_model.layers[36].output)

Wiring some of the layers of the second model is much more difficult. It’s easy to slice off the end of a keras model – it’s much more difficult to slice of the beginning because of the need for a tf.keras.Input placeholder. To do this successfully, you’ll need to write a model walking algorithm that go through the layers, tracks the output KerasTensors, then calls each layer with the new inputs to create a new output KerasTensor.

You could avoid all that work by simply finding some source code for an InceptionResNet and adding layers via Python rather than introspecting an existing model. Here’s one which may fit the bill.

https://github.com/yuyang-huang/keras-inception-resnet-v2/blob/master/inception_resnet_v2.py