3D CNN to infer throttle and angle for a RC autonomous car . Implementation of https://arxiv.org/pdf/1412.0767.pdf

3dcnn.py

# A basic end-to-end autonomous car uses 2d images to infer throttle and steering. However, speed and acceleration can change this. If the car is already speeding, throttle should decrease. If the car already has speed, then steering should be proportionally less
# One way to improve this is with hardware, by adding a RPM speed sensor (optical or magnetic), a GPS, an IMU or a combination. This is used as an input to the model
# If one does not have a speed sensor, another way is to update the model so it takes into account the last frames, and by using Kera's 3d convolution methods. Reference paper to read : https://arxiv.org/pdf/1412.0767.pdf
# This can slow down inference and result in lower fps

img_in3D = Input(shape=(3, 120, 160, 3), name='img_in') # First layer, input layer, takes 3 Shapes comes from camera.py resolution, RGB
x = img_in3D x = Cropping3D(cropping = ((0, 0),(60, 0),(0, 0)))(x) # Depending on your camera angle. Most cameras are pointed downards so the top X pixels can be safely cropped
x = Convolution3D(8, (3, 3, 3), strides=(1, 2, 2), activation='relu')(x) # CNN in 2d + time dimension
x = MaxPooling3D(pool_size=(1, 2, 2))(x) # MaxPooling make it less sensitive to position in image. Camera angle fixed, so may not to be needed
x = BatchNormalization()(x) x = Dropout(0.1)(x)  # Randomly drop out (turn off) 10% of the neurons (Prevent overfitting)
x = Flatten(name='flattened')(x)  # Flatten to 1D (Fully connected)
x = Dense(50, activation='relu')(x)  # Classify the data into 100 features, make all negatives 0
x = Dropout(0.2)(x) # Randomly drop out (turn off) 20% of the neurons (Prevent overfitting)

angle_out = Dense(15, activation='softmax', name='angle_out')(x) # Connect every input with every output and output 15 hidden units. Use Softmax to give percentage. 15 categories and find best one based off percentage 0.0-1.0
    # continous output of throttle
throttle_out = Dense(1, activation='relu', name='throttle_out')(x) # Reduce to 1 number, Positive number only

model = Model(inputs=[img_in3D], outputs=[angle_out, throttle_out]) 
model.compile(optimizer='adam', loss={'angle_out':'categorical_crossentropy',
'throttle_out': 'mean_absolute_error'}, loss_weights={'angle_out': 0.9, 'throttle_out': 0.01})

return model