在用ConvLSTM2D处理图像数据时,通常先将图像经过Conv2D的操作,然后把卷积结果输入到LSTM中。这个过程需要经历的过程是:(1)把shape=[n,h,w,c]的数据输入到Conv2D的卷积层中,得到卷积结果。(2)把卷积结果通过split变形成shape = [n,d,h,w,c]形式。(3)把shape=[n,d,h,w,c]形式的数据输入到LSTM中。的每个step中,这个过程很繁琐。在keras中,用TimeDistributed简化了这一过程。示例如下:
import tensorflow.compat.v1 as tf from keras.layers import ConvLSTM2D,TimeDistributed,Conv2D import numpy as np inputs_np = tf.convert_to_tensor(np.random.random((4,6,256,256,3)).astype(np.float32)) # shape = [5,6,10,10,3] # inputs = tf.concat([inputs_np[:,i,...] for i in range(6)],axis=0) # conv1 = Conv2D(filters=10,kernel_size=(3,3),strides=(1,1))(inputs) # conv1 =tf.concat([tf.expand_dims(v,axis=1) for v in tf.split(conv1,num_or_size_splits=6)],axis=1) conv1 = TimeDistributed(Conv2D(filters=10,kernel_size=(3,3),strides=(1,1)),input_shape=(6,256,256,3))(inputs_np) lstm_out= ConvLSTM2D(filters=1,kernel_size=(3,3),strides=(1,1),padding='valid',activation='tanh',return_sequences=True)(conv1) with tf.Session() as sess: sess.run(tf.global_variables_initializer()) lstm_out_ = sess.run(lstm_out) print(lstm_out_.shape)程序中,用TimeDistributed计算conv1的过程等价于注释行的计算过程。
注意:在TimeDistributed的参数中,input_shape=[time_steps,h,w,c]:
(1)不包含batch维度,
(2)第一个维度是clip_len, 或者LSTM的长度
(3)TimeDistributed 的操作对象只能是keras的一个layer对象,不能是自定义的函数
(4)TimeDistributed也是keras的一个层(layer)对象。
(5)用TimeDistributed封装了一个layer之后,模型的默认变量名发生了变化,但是没有新增变量。
如下例,当用TimeDistributed封装了一个Conv2d之后,模型的默认变量名由 “conv2d/kernel:0”变成了“time_distributed/kernel:0”。
(6)针对默认变量名的变化,可以使用“name=”的layer参数进行纠正。如下例中的“NEW1”中的操作。
import tensorflow.compat.v1 as tf from tensorflow.keras.layers import ConvLSTM2D, TimeDistributed, Conv2D import numpy as np inputs_np = tf.convert_to_tensor(np.random.random((4, 6, 256, 256, 3)).astype(np.float32)) # shape = [5,6,10,10,3] with tf.variable_scope("RAW"): inputs = tf.concat([inputs_np[:,i,...] for i in range(6)],axis=0) conv1 = Conv2D(filters=10,kernel_size=(3,3),strides=(1,1))(inputs) conv1 =tf.concat([tf.expand_dims(v,axis=1) for v in tf.split(conv1,num_or_size_splits=6)],axis=1) with tf.variable_scope("NEW"): conv2 = TimeDistributed(Conv2D(filters=10, kernel_size=(3, 3), strides=(1, 1)), input_shape=(6, 256, 256, 3))(inputs_np) with tf.variable_scope("NEW1"): conv3 = TimeDistributed(Conv2D(filters=10, kernel_size=(3, 3), strides=(1, 1)), input_shape=(6, 256, 256, 3),name="conv2d")(inputs_np) lstm_out = ConvLSTM2D(filters=1, kernel_size=(3, 3), strides=(1, 1), padding='valid', activation='tanh',return_sequences=True)(conv1) all_vars = tf.global_variables() for var in all_vars: print(var.name) """ 打印结果: RAW/conv2d/kernel:0 RAW/conv2d/bias:0 NEW/time_distributed/kernel:0 NEW/time_distributed/bias:0 NEW1/conv2d/kernel:0 NEW1/conv2d/bias:0 conv_lst_m2d/kernel:0 conv_lst_m2d/recurrent_kernel:0 conv_lst_m2d/bias:0 """
