Skip to content

Instantly share code, notes, and snippets.

@thestinger
Last active December 23, 2016 06:47
Show Gist options
  • Save thestinger/dd64f03993623f5058b1a9811d68c4aa to your computer and use it in GitHub Desktop.
Save thestinger/dd64f03993623f5058b1a9811d68c4aa to your computer and use it in GitHub Desktop.
NOT READY FOR PUBLISHING (i.e. future blog post): notes about problems with the Android Open Source Project

The Neglected Android Open Source Project

What is the Android Open Source Project?

The Android Open Source Project is essentially the Android source code for Nexus devices and the Pixel C. However, it's somewhat different from the internal source tree used to build stock Android for those devices.

It's important to note that it's only closely tied to the internal source code for stable releases. The master branch of AOSP does not closely track the internal development. The internal work is released as part of a stable update, and then merged into the AOSP master branch afterwards. AOSP master and the internal source tree are repeatedly merged into each other to some extent during a year of development, as some work happens out in the open (lots of the low-level userspace work) while other work happens internally. For some repositories, there's substantial amounts of both. The merges are primarily from AOSP into the internal tree until new stable source code is released. These strange workflows result in all kinds of oddities like the DO NOT MERGE in the summaries of many commits that are indeed merged, since it has nothing to do with a commit not being ready for merging into that source tree. Instead, it prevents merging of AOSP commits back into the internal tree.

Kernel sources are included in the broader AOSP umbrella, but they are not checked out as part of the normal source tree used for building. Instead, they are bundled as prebuilt binaries in repositories for each device. This is the same way things work for Google's internal source tree, but there are no public development branches for the AOSP device kernels, meaning that there isn't really a way to submit a device-specific kernel patch.

Issue tracker

The Android issue tracker is the internal Google bug tracker. Issue reports are never made public, and they are heavily referenced by id in commit messages and even the source code. Commit messages often have no explanation for the changes beyond a bug id that's not usable from outside of Google. The only way to cope with this is knowing someone at Google to use them the same way that Richard Stallman fetches public web pages with wget.

A public Android issue tracker tracker does exist, but solely to receive external reports which are then transferred onto the internal bug tracker. Discussion by Google employees is usually limited to the internal bug tracker.

Proprietary blobs

Device support requires proprietary firmware, libraries and other files. Google used to make releases of these proprietary binaries for each device, allowing the Android Open Source Project to be built for Nexus devices in an official way. These have never been published for the Nexus 9, 5X and 6P. The only way to make a fully functional build is extracting a bunch of proprietary blobs from the factory images in order to rebuild the /vendor partition (for functional verified boot, among other things) and to obtain all of the blobs for /system. It has been claimed that all of the device support blobs were moved to /vendor, but that's only nearly true on the Nexus 9. On the 5X and 6P, there are over a hundred in /system. Regardless, it is necessary to rebuild /vendor in order to provide fully functional builds and full updates.

Extracting blobs from factory images has been greatly complicated by dex preoptimization. Apps built in to the system partition on the 5X and 6P are shipped with classes.dex stripped out, since they are precompiled to native code in oat files. The oat files have to be reverse engineered to dex files, which was mostly sorted out for Marshmallow but is now unavailable for the time being with Nougat.

Another example of a tricky case is the over-the-air update system for the Nexus 5X. A proprietary LG library is used to update the low-level firmware partitions other than the radio (the dozen partitions contained in bootloader.img). It's statically linked into the update-binary executable contained in the stock over-the-air updates. There is no way to obtain this library as an archive / object files. CopperheadOS simply has to bundle the stock update-binary into over-the-air updates right now, which prevents making any extensions to the OTA system.

Google is aware that using the stock vendor partition with a custom build breaks verified boot, so it was disabled for the Nexus 5X and Nexus 6P, but not the Nexus 9. A much saner alternative would be adding support for reassembling vendor.img by integrating the android-prepare-vendor generator-vendor.sh script into the AOSP build process.

Obsolete / untested projects due to Play

Google switched to Play versions of many apps, services and libraries. Some of the AOSP projects that were replaced on stock are now nearly fully abandoned and only receive extremely critical fixes, but often not for stable releases (only master). In other cases such as DeskClock, the AOSP project is still maintained and is likely still the basis for the Play app. However, these apps are mostly untested and end up with breakage that's not present in the Play builds. Play apps also receive updates via Play regularly, while AOSP apps only get security updates and occasionally some critical fixes until the next major operating system release.

Functionality specific to AOSP

Some projects are ONLY used for the open-source Android builds. This code is particularly prone to being broken, and also often doesn't receive even critical fixes for stable releases.

Device-specific branches

Most of the supported devices are simply built from the main stable branch of Android. However, new devices often end up in device-specific branches since they weren't supported by that stable branch, and Google doesn't merge in a bunch of new functionality after releases. This may not seem like it would cause AOSP-specific problems, but the level of neglect for AOSP is MUCH higher for these device-specific branches.

The Marshmallow release cycle was strange in that most devices went from marshmallow-release -> marshmallow-mr1-release -> marshmallow-mr2-release, but the Nexus 5X and 6P split off from marshmallow-mr1-release -> marshmallow-dr1.5-release -> marshmallow-dr1.6-release. The reason for this is not publicly known. It resulted in some strange cases of functionality specific to the 5X and 6P, some of which is now gone in Nougat. These dr1.5/dr1.6 branches were particularly bad, with issues like mass imports of broken translations not corresponding properly to the version of the apps in AOSP. The translations may have come from the current versions used on Play.

Chrome

Chrome on the desktop is much different than on Android in terms of the relationship to Chromium. On the desktop, they are nearly equivalent with Chromium supporting the same Google services and other features. It's even possible to use Chrome's Pepper Flash and Widevine EME plugins in Chromium. On Android, Chromium only recently gained support for building the proper standalone Chrome browser app. Unlike the desktop version where built-in implementations are used, all features reliant on Google services require having Play Services present. This applies even to features like Safe Browsing (malware / phishing protection) that are supported by other browsers like Firefox. Features in Chrome that are specific to Android tend to be developed in private, released in Chrome and then later landed upstream. This is quite different than normal Chromium development which is almost entirely in the open, with a public bug tracker, almost fully public code review and very little internal code (which extends the Chromium source tree, rather than outright replacing lots of it like AOSP).

Android's WebView widget is also built from the same source tree, and in the past this was fully open-source and worked well. However, recently Google moved to unified Chrome/WebView builds ("monochrome" builds) and that code is not yet upstream, meaning that everyone else still has to do independent builds. Worse than that, Nougat support for the WebView is not yet upstream so the only way to have an up-to-date WebView with all security fixes on Android Nougat is using the Google built packages, despite it being a supposedly open-source project.

Source releases

In theory, Android Open Source Project branches/tags are pushed shortly after new Android releases. This is generally true, but there are often delays and it's rare for there to be a monthly security update without major issues in the initial source release. There are often missing branches/tags (sometimes the commits are present regardless, if you know the revision to look for), the wrong sources end up pushed for a given reference (a different device's kernel, a different branch, etc.), the manifest referencing repositories may be screwed up in various ways (wrong tag, referencing an extra repository that is no longer part of the source tree and therefore not tagged, etc.) along with other more unique problems.

Compatibility Test Suite

Android has an extensive test suite known as the CTS. The CTS plays a big role in the Android ecosystem since devices using the "Android" branding and shipping with Google Play need to pass the full test suite. Despite being called the Android Open Source Project, AOSP does not pass all of the tests. A small amount of the required functionality is missing (for example, AOSP claims to have a network location service but does not) while most of the issues are caused by AOSP-specific bugs in components that are not used in stock, or where Play variants are used instead.

Lack of communication

All of these issues have been communicated to Google, often many times by many different people. However, there is no clear point of contact for the community and there is rarely if ever a response about these issues. Google employees often independently care about these issues and listen to the community, but they are doing so in an unofficial capacity and most of the developers that the community interacts with do not work on stable branches or release engineering.

Google hasn't provided reasoning for changes like the insanely restrictive terms of use for factory images / ota updates or the lack of binary releases for the Nexus 5X, 6P and 9. The community has to make assumptions about the reasons for these issues and which vendors are to blame when it's suspected that Google is simply bending to their will. They do not acknowledge that the problems exist and there's no work on alternative solutions to these problems. For example, Google could stop stripping out classes.dex for carrier/device-related apks by setting LOCAL_DEX_PREOPT := unstripped even if they are not going to provide binary releases with them anymore.

@jduck
Copy link

jduck commented Sep 15, 2016

@jduck
Copy link

jduck commented Sep 15, 2016

Another thing that may alienate external contributions is the lack of proper attribution for patches. Some people do this well, others on the team do a terrible job. I've seen my patches both lose attribution (instead being credited to a googler) and live on with attribution despite being profoundly changed (CTS). Case-by-case basis I guess. Perhaps also depends on the specific case handler's experience with git or other source control tools.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment