The Android Open Source Project is essentially the Android source code for Nexus devices and the Pixel C. However, it's somewhat different from the internal source tree used to build stock Android for those devices.
It's important to note that it's only closely tied to the internal source code for stable releases. The master branch of AOSP does not closely track the internal development. The internal work is released as part of a stable update, and then merged into the AOSP master branch afterwards. AOSP master and the internal source tree are repeatedly merged into each other to some extent during a year of development, as some work happens out in the open (lots of the low-level userspace work) while other work happens internally. For some repositories, there's substantial amounts of both. The merges are primarily from AOSP into the internal tree until new stable source code is released. These strange workflows result in all kinds of oddities like the DO NOT MERGE in the summaries of many commits that are indeed merged, since it has nothing to do with a commit not being ready for merging into that source tree. Instead, it prevents merging of AOSP commits back into the internal tree.
Kernel sources are included in the broader AOSP umbrella, but they are not checked out as part of the normal source tree used for building. Instead, they are bundled as prebuilt binaries in repositories for each device. This is the same way things work for Google's internal source tree, but there are no public development branches for the AOSP device kernels, meaning that there isn't really a way to submit a device-specific kernel patch.
The Android issue tracker is the internal Google bug tracker. Issue reports are never made public, and they are heavily referenced by id in commit messages and even the source code. Commit messages often have no explanation for the changes beyond a bug id that's not usable from outside of Google. The only way to cope with this is knowing someone at Google to use them the same way that Richard Stallman fetches public web pages with wget.
A public Android issue tracker tracker does exist, but solely to receive external reports which are then transferred onto the internal bug tracker. Discussion by Google employees is usually limited to the internal bug tracker.
Device support requires proprietary firmware, libraries and other files. Google used to make releases of these proprietary binaries for each device, allowing the Android Open Source Project to be built for Nexus devices in an official way. These have never been published for the Nexus 9, 5X and 6P. The only way to make a fully functional build is extracting a bunch of proprietary blobs from the factory images in order to rebuild the /vendor partition (for functional verified boot, among other things) and to obtain all of the blobs for /system. It has been claimed that all of the device support blobs were moved to /vendor, but that's only nearly true on the Nexus 9. On the 5X and 6P, there are over a hundred in /system. Regardless, it is necessary to rebuild /vendor in order to provide fully functional builds and full updates.
Extracting blobs from factory images has been greatly complicated by dex preoptimization. Apps built in to the system partition on the 5X and 6P are shipped with classes.dex stripped out, since they are precompiled to native code in oat files. The oat files have to be reverse engineered to dex files, which was mostly sorted out for Marshmallow but is now unavailable for the time being with Nougat.
Another example of a tricky case is the over-the-air update system for the Nexus 5X. A proprietary LG library is used to update the low-level firmware partitions other than the radio (the dozen partitions contained in bootloader.img). It's statically linked into the update-binary executable contained in the stock over-the-air updates. There is no way to obtain this library as an archive / object files. CopperheadOS simply has to bundle the stock update-binary into over-the-air updates right now, which prevents making any extensions to the OTA system.
Google is aware that using the stock vendor partition with a custom build breaks verified boot, so it was disabled for the Nexus 5X and Nexus 6P, but not the Nexus 9. A much saner alternative would be adding support for reassembling vendor.img by integrating the android-prepare-vendor generator-vendor.sh script into the AOSP build process.
Google switched to Play versions of many apps, services and libraries. Some of the AOSP projects that were replaced on stock are now nearly fully abandoned and only receive extremely critical fixes, but often not for stable releases (only master). In other cases such as DeskClock, the AOSP project is still maintained and is likely still the basis for the Play app. However, these apps are mostly untested and end up with breakage that's not present in the Play builds. Play apps also receive updates via Play regularly, while AOSP apps only get security updates and occasionally some critical fixes until the next major operating system release.
Some projects are ONLY used for the open-source Android builds. This code is particularly prone to being broken, and also often doesn't receive even critical fixes for stable releases.
Most of the supported devices are simply built from the main stable branch of Android. However, new devices often end up in device-specific branches since they weren't supported by that stable branch, and Google doesn't merge in a bunch of new functionality after releases. This may not seem like it would cause AOSP-specific problems, but the level of neglect for AOSP is MUCH higher for these device-specific branches.
The Marshmallow release cycle was strange in that most devices went from marshmallow-release -> marshmallow-mr1-release -> marshmallow-mr2-release, but the Nexus 5X and 6P split off from marshmallow-mr1-release -> marshmallow-dr1.5-release -> marshmallow-dr1.6-release. The reason for this is not publicly known. It resulted in some strange cases of functionality specific to the 5X and 6P, some of which is now gone in Nougat. These dr1.5/dr1.6 branches were particularly bad, with issues like mass imports of broken translations not corresponding properly to the version of the apps in AOSP. The translations may have come from the current versions used on Play.
Chrome on the desktop is much different than on Android in terms of the relationship to Chromium. On the desktop, they are nearly equivalent with Chromium supporting the same Google services and other features. It's even possible to use Chrome's Pepper Flash and Widevine EME plugins in Chromium. On Android, Chromium only recently gained support for building the proper standalone Chrome browser app. Unlike the desktop version where built-in implementations are used, all features reliant on Google services require having Play Services present. This applies even to features like Safe Browsing (malware / phishing protection) that are supported by other browsers like Firefox. Features in Chrome that are specific to Android tend to be developed in private, released in Chrome and then later landed upstream. This is quite different than normal Chromium development which is almost entirely in the open, with a public bug tracker, almost fully public code review and very little internal code (which extends the Chromium source tree, rather than outright replacing lots of it like AOSP).
Android's WebView widget is also built from the same source tree, and in the past this was fully open-source and worked well. However, recently Google moved to unified Chrome/WebView builds ("monochrome" builds) and that code is not yet upstream, meaning that everyone else still has to do independent builds. Worse than that, Nougat support for the WebView is not yet upstream so the only way to have an up-to-date WebView with all security fixes on Android Nougat is using the Google built packages, despite it being a supposedly open-source project.
In theory, Android Open Source Project branches/tags are pushed shortly after new Android releases. This is generally true, but there are often delays and it's rare for there to be a monthly security update without major issues in the initial source release. There are often missing branches/tags (sometimes the commits are present regardless, if you know the revision to look for), the wrong sources end up pushed for a given reference (a different device's kernel, a different branch, etc.), the manifest referencing repositories may be screwed up in various ways (wrong tag, referencing an extra repository that is no longer part of the source tree and therefore not tagged, etc.) along with other more unique problems.
Android has an extensive test suite known as the CTS. The CTS plays a big role in the Android ecosystem since devices using the "Android" branding and shipping with Google Play need to pass the full test suite. Despite being called the Android Open Source Project, AOSP does not pass all of the tests. A small amount of the required functionality is missing (for example, AOSP claims to have a network location service but does not) while most of the issues are caused by AOSP-specific bugs in components that are not used in stock, or where Play variants are used instead.
All of these issues have been communicated to Google, often many times by many different people. However, there is no clear point of contact for the community and there is rarely if ever a response about these issues. Google employees often independently care about these issues and listen to the community, but they are doing so in an unofficial capacity and most of the developers that the community interacts with do not work on stable branches or release engineering.
Google hasn't provided reasoning for changes like the insanely restrictive terms of use for factory images / ota updates or the lack of binary releases for the Nexus 5X, 6P and 9. The community has to make assumptions about the reasons for these issues and which vendors are to blame when it's suspected that Google is simply bending to their will. They do not acknowledge that the problems exist and there's no work on alternative solutions to these problems. For example, Google could stop stripping out classes.dex for carrier/device-related apks by setting LOCAL_DEX_PREOPT := unstripped even if they are not going to provide binary releases with them anymore.
I assume this section is meant to stop here. This seems like information that can be put into the executive summary/introduction (and probably with the sub-heading that's here).
The intro should make the point that AOSP is in poor shape. It can be supported with citations from android-building from all the people complaining that it's not possible to build a usable device based on AOSP. Can explain this post will touch on why this issue exists.
This seems to be more suited for an AOSP vs. Internal code branches. It should probably be prefixed with the re-explanation that Google uses a closed development model vs. a true open source one. I'm probably biased, but I think it was well covered in Android Hacker's Handbook in chapter 1 under "Open source, mostly" heading. This stuff seems to fit under a heading "AOSP vs. Internal" or "Android Development Model" or something similar.
Could also reference https://source.android.com/source/code-lines.html
This deserves it's own section IMHO. "Linux Kernel Handling", perhaps. Is there more to this btw?
This should make a reference back to the Android development model (or become a sub-heading of that). The main point is that both internal and external bug trackers exist but Android's developers use the internal one primarily. That should be stated prominently and then supported with the facts present. It should end with a statement about how this is not in line with open-source software initiatives and how it complicates collaboration.
I don't see a mention of the openness of bugs even in the external tracker. Many bugs are not being made public ever. I think I understand that it's a maintenance nightmare to deal with that, but I feel it's something that can be automated...
This is very well stated. Personally I tried to modify "am" once and was quickly disappointed with the realization I'd need to build an entire ROM in order to test the change (due to framework signing enforcements). Meh.
Wow. I didn't realize the problem got worse :-/
I think there are some bits of information on how this was accomplished in Andrew Boie's presentation on OTA updates from Embedded Linux Con 2015. Perhaps LG should just open-source their updater modifications in device/lge/bullhead?
This is terrible IMHO. It makes sense for a non-development build but for not for release builds. AOSP could support this by doing this only for "eng" builds. Even saner would be supporting binary blobs as before instead of requiring the "prepare-vendor" extraction step at all IMHO.
Perhaps these two sections should be merged under a heading "Drifting Towards Closed Source". Is that too bold of a claim to make? Are there other points that would support this idea? Were there any actual changes (improvements?) made to AOSP apps you had in mind between M and N?
While I cannot argue that there are branches in some repositories, they do not seem to be officially supported. See https://source.android.com/source/build-numbers.html where they use the term branch but then use tags in the column :-/
The tags seem to be what they use the most. Trying to make sense of the branches (which maybe are only used by a minority of Android developers in Google) seems like a bad idea to me still. If this is causing you pain you may consider changing your approach =) Regardless, it is still good to complain that AOSP branching doesn't make much sense...
"Main stable branch" means "master" to me, but I am sure that's not the case. I think they do try to merge device specific changes when possible (see some of the Nexus 6 Lollipop builds for example). Whenever they can figure out how to support both branches with the same code they do AFAIK.
I think the neglect level for these device specific branches can be explained by the idea that development is always moving forward. When you think about it, once a device is released, it's not development anymore it's just maintenance/support. Even porting the latest version of Android is a maintenance/support task and not really development. Real development is happening on the next-generation device that is yet to be released to the public. In those developers' eyes that's "the old device" even the day after it gets released to the public, or maybe even before.
You hit on something that I see as a big problem for AOSP in general. There is a profound lack of transparency. I'm not sure if the idea is that Google engineers cannot make mistakes and therefore shouldn't discuss them or what. That is, is it a culture problem or is it a PR problem or what the fuck? Transparency builds trust. Without it, we are left wondering -- and that is exhausting.
I don't think it's fair to say "only recently gained support for building the proper standalone Chrome browser app". I managed to build Chrome for Android myself around Android 4.4. That said, the ability to do so may change on a day-by-day basis.
It might also be worth mentioning that even doing that required including SVN-commit-specific proprietary blobs. It also required following arcane steps that I dug up after reading several out-of-date pieces of documentation. I originally wanted to build based on a git/gclient check out but eventually needed to use SVN to make it work. It was such a pain in the ass that I still have no deleted that directory from my machine =)
This is a new development that I think you said was in the process of being fixed. It might help to say that and link to the relevant places to show the evidence. I genuinely think the Chromium/Chrome guys actually care about OSS and do put in effort to make things buildable. They may not have pulled it off in time for the Android N release, but hopefully they will soon. After all, they do need to build things themselves, right? Whatever was needed to do that exists somewhere... I'm not sure if red tape, technical issues, or what stand in the way...
Some of this should be merged above. There's information that is key to what AOSP is and how branching/tagging etc is done in AOSP. It helps explain the Android development model too.
That said, some of this seems to be related to releases specifically. I think everyone who pays attention to Android releases knows it has been a very rocky history. From "There won't be AOSP releases for 3.x" to flawless releases. The in-betweens are many. I don't think the problem is limited to AOSP either, but then again this post is about AOSP so maybe we don't care to complain about other Nexus release issues (omitting bootloaders, etc).
This is very damning. It's not even clear if some of the tests can pass at all given the recent hardening that's taken place in N. Nexus devices do not appear to pass some of the tests as they are written either. Unfortunately the specific example I had in mind turned out to not be true (dmesg_restrict) so maybe this is not correct...
Some of these failures could also be caused by proprietary blob issues too.
Yes. AOSP needs a community manager BADLY. Possibly even a team of them. Without that, you can expect AOSP to continue to languish and fail to engage the greater community at the level that it could. Perhaps Google does not want to involve the community though... Who knows.
Only speculating, but there may be management issues inside Android that create a culture or climate where caring about open source is viewed negatively. That certainly seems to be the case for security previously but we've seen a big change there in the last 12-18 months.
I think some of these things are obvious. They have licensed proprietary code from outside parties (qcom, lge, etc) to make their devices happen. This means restrictive terms for use of any form containing these bits. As for the lack of binary updates, my guess would be either laziness or a systemic de-prioritization of open source / AOSP support.
Sometimes I wonder if there is a requirement to subscribe to a philosophy that Android should be shiny and happy all the time. Nothing negative, always cake and candy. This is not how the real world works. People make mistakes -- all the time. To err is human. People should not be afraid to admit their mistakes as long as they dedicate themselves to learning from them.
Anyway... There are some things that are not present that should be called out as well.