Easiest way to see the output of a hidden layer in Tensorflow/Keras?

I am working on a GAN and I’m trying to diagnose how any why mode collapse occurs. I want to be able to look “under the hood” and see what the outputs of various layers in the network look like for the last minibatch. I saw you can do something like model.layers[5].output, but this produces a tensor of shape [None, 64, 64, 512], which looks like an empty tensor and not the actual output from the previous run. My only other idea is to recompile a model that’s missing all the layers after the one I’m interested in and then run a minibatch through, but this seems like an extremely inefficient way to do it. I’m wondering if there’s an easier way. I want to run some statistics on layer outputs during the training process to see where things might be going wrong.

Answer

I did this for a GAN I was training myself. The method I used extends to both the generator (G) and discriminator (D) of a GAN.

The idea is to make a model with the same input as D or G, but with outputs according to each layer in the model that you require.

For me, I found it useful to check the activations. In Keras, with some model model (which will be D or G for you and me)

activation_layers = []
activation_names = []

# obtain the layers in a given model, but skip the first 6
# as these generally are the input / non-convolutional layers
model_layers = [layer for layer in sub_model.layers][6:]
# print the names of what we are looking at.
print("MODEL LAYERS:", model_layers)

for layer in model_layers:
    # check if the layer is an activation
    if isinstance(layer, (Activation, LeakyReLU)):
        # append the output of this layer for later
        activation_layers.append(layer.output)
        # name it with a signature of its output for clarity
        activation_names.append(layer.name + str(layer.output_shape[1]))

# now create a model which outputs every activation
activation_model = Model(inputs=model.inputs, outputs=activation_layers)
# this outputs a list of layers when given an input, so for G
noise = np.random.normal(size=(1,32,32,1)) # random image shape (change for yourself)
model_activations = model.predict(noise)

Now the rest is quite model-specific. This is the basic method for checking the outputs of the layers in a given model.

Note it can be done before, during or after training. It also does not need re-compiling.

The plotting of activation maps in this case is relatively straight forward and as you mentioned, you will probably have something specific you want to do. Still, I have to link this beautiful example here.