DRM cdev pager is not compatible with ARM pmap.
Actually, this is the only problematic part of linux KPI and/or kernel VM. And it need longer clarification: The DRM device pager is based on OBJT_DEVICE, OBJT_MDEVICE or OBJT_SG objects. All these pagers uses fake pages (vm_page_getfake(), vm_page_updatefake()) for mmaping memory used by GPU to user space. On ARM, fake pages aliases are allowed only if they are backed by fictitious pages (i.e. memory on GPU card), not by normal memory pages (CPU memory). The later breaks ARM basic pmap assumption that
paddr == VM_PAGE_TO_PHYS(PHYS_TO_VM_PAGE(paddr))
and I’m unable, nor as ARM pmap coauthor, to find any reasonable solution. I spent weeks of tests on this, but without success.
Summarization what is missing in linux KPI, including my short subjective (and unverified) comment:
https://elixir.bootlin.com/linux/latest/source/include/linux/of.h and related of_*.h
Lots of function is used, I don’t see any fatal problem here
Lots (if not all) of functions are used. Our analogy is dev/extres/clk. There are significant functional differences and don’t see way how to lkpi'ise this in general way without converting each single clock driver to be linux identical Interface. See 
Easy one. See
Our analogy is dev/extres/regulator. Rest is same as for clocks. See 
We have nothing like that. Hard to say how much is this difficult.
We don’t have iommu implementation on arm at all.
I don’t see any fatal problem here.
Platform device related functions
Hard to say how much is this difficult. Coincidence with rest of device tree can be problematic.
devm_ related functions
< Contained in clock, reset, regulators headers >
It depends on platform_device. Rest looks easy.
The biggest difference between GPU card (including i386/amd64 GPUs on CPU/mainboard) and GPU integrated in (ARM) SoCs is the fact that the first one is very “self-contained” solution.
For GPU cards, one driver (rareon/i915) controls (with BIOS/firmware help) all its resources (clocks/regulators/resets/gpios). Additionally, it consumes some resources from kernel (memory, timers, interrupts). These kernel resources are covered by high level API which ensures their stability/uniformity across all platforms.
But, as opposite, SoC based GPUs consumes resources generated by other blocks, controlled by ther own drivers outside of DRM (clocks/regulators/resets/gpios). So
GPU driver depends on exact behavior of (mainly) clock, regulator, reset and gpio drivers. Although there is a uniform API layer, given request can be satisfied by many ways resulting in different chip settings and different side effects. There is no way to emulate exact Linux behavior other than copy Linux driver 1:1. Moreover, and again mainly for clocks (or regulators), Linux developers uses dark tricks and hacks to bypass limitation of API. For example, on tegra, call to clk_enable(foo_clock) also de-assert reset for block foo, although Tegra has implemented an independent reset API. And other SoC do their own tricks :(
For Radeon GPU inserted to PCIe slot on ARM board is drm-next is right way, I'm sure. And I’m ready to help to make drm-next arm (and arm64) compatible.
But to my best knowledge, and by all these above, plan to use unmodified Linux drivers for SoC based graphic is quite unrealistic.