cowplot's plot_grid function doesn't work with labels if a custom theme has been set using theme()

plot_grid_problem.R

library(ggplot2)

## set custom theme (even with no options)
theme_set(theme())

## Error: Aesthetics must be either length 1 or the same as the data (1): colour and family
a <- ggplot(cars, aes(speed, dist)) + geom_point()
cowplot::plot_grid(a, a, labels = c("A", "B"))

## Works with no labels
cowplot::plot_grid(a, a)

## Works when theme reset
theme_set(theme_classic())
cowplot::plot_grid(a, a, labels = c("A", "B"))

Because the custom theme does not set the required aesthetic for cowplot:::draw_plot_label to use.
This works:
theme_set(theme(text=element_text(colour = "black", font= "Times"))
And this works as well: cowplot::plot_grid(a, a, labels = c("A", "B"), label_fontfamily = "Times", label_colour = "black")