gistfile1.txt

Zorinaq
mrb's blog
"I Contribute to the Windows Kernel. We Are Slower Than Other Operating Systems. Here Is Why."
I was explaining on Hacker News why Windows fell behind Linux in terms of operating system kernel performance and innovation. And out of nowhere an anonymous Microsoft developer who contributes to the Windows NT kernel wrote a fantastic and honest response acknowledging this problem and explaining its cause. His post has been deleted! Why the censorship? I am reposting it here. This is too insightful to be lost. [Edit: The anonymous poster himself deleted his post as he thought it was too cruel and did not help make his point, which is about the social dynamics of spontaneous contribution. However he let me know he does not mind the repost at the condition I redact the SHA1 hash info, which I did.] [Edit: A second statement, apologetic, has been made by the anonymous person. See update at the bottom.]

"""

I'm a developer in Windows and contribute to the NT kernel. (Proof: the SHA1 hash of revision #102 of [Edit: filename redacted] is [Edit: hash redacted].) I'm posting through Tor for obvious reasons.

Windows is indeed slower than other operating systems in many scenarios, and the gap is worsening. The cause of the problem is social. There's almost none of the improvement for its own sake, for the sake of glory, that you see in the Linux world.

Granted, occasionally one sees naive people try to make things better. These people almost always fail. We can and do improve performance for specific scenarios that people with the ability to allocate resources believe impact business goals, but this work is Sisyphean. There's no formal or informal program of systemic performance improvement. We started caring about security because pre-SP3 Windows XP was an existential threat to the business. Our low performance is not an existential threat to the business.

See, component owners are generally openly hostile to outside patches: if you're a dev, accepting an outside patch makes your lead angry (due to the need to maintain this patch and to justify in in shiproom the unplanned design change), makes test angry (because test is on the hook for making sure the change doesn't break anything, and you just made work for them), and PM is angry (due to the schedule implications of code churn). There's just no incentive to accept changes from outside your own team. You can always find a reason to say "no", and you have very little incentive to say "yes".

There's also little incentive to create changes in the first place. On linux-kernel, if you improve the performance of directory traversal by a consistent 5%, you're praised and thanked. Here, if you do that and you're not on the object manager team, then even if you do get your code past the Ob owners and into the tree, your own management doesn't care. Yes, making a massive improvement will get you noticed by senior people and could be a boon for your career, but the improvement has to be very large to attract that kind of attention. Incremental improvements just annoy people and are, at best, neutral for your career. If you're unlucky and you tell your lead about how you improved performance of some other component on the system, he'll just ask you whether you can accelerate your bug glide.

Is it any wonder that people stop trying to do unplanned work after a little while?

Another reason for the quality gap is that that we've been having trouble keeping talented people. Google and other large Seattle-area companies keep poaching our best, most experienced developers, and we hire youths straight from college to replace them. You find SDEs and SDE IIs maintaining hugely import systems. These developers mean well and are usually adequately intelligent, but they don't understand why certain decisions were made, don't have a thorough understanding of the intricate details of how their systems work, and most importantly, don't want to change anything that already works.

These junior developers also have a tendency to make improvements to the system by implementing brand-new features instead of improving old ones. Look at recent Microsoft releases: we don't fix old features, but accrete new ones. New features help much more at review time than improvements to old ones.

(That's literally the explanation for PowerShell. Many of us wanted to improve cmd.exe, but couldn't.)

More examples:

We can't touch named pipes. Let's add %INTERNAL_NOTIFICATION_SYSTEM%! And let's make it inconsistent with virtually every other named NT primitive.
We can't expose %INTERNAL_NOTIFICATION_SYSTEM% to the rest of the world because we don't want to fill out paperwork and we're not losing sales because we only have 1990s-era Win32 APIs available publicly.
We can't touch DCOM. So we create another %C#_REMOTING_FLAVOR_OF_THE_WEEK%!
XNA. Need I say more?
Why would anyone need an archive format that supports files larger than 2GB?
Let's support symbolic links, but make sure that nobody can use them so we don't get blamed for security vulnerabilities (Great! Now we get to look sage and responsible!)
We can't touch Source Depot, so let's hack together SDX!
We can't touch SDX, so let's pretend for four releases that we're moving to TFS while not actually changing anything!
Oh god, the NTFS code is a purple opium-fueled Victorian horror novel that uses global recursive locks and SEH for flow control. Let's write ReFs instead. (And hey, let's start by copying and pasting the NTFS source code and removing half the features! Then let's add checksums, because checksums are cool, right, and now with checksums we're just as good as ZFS? Right? And who needs quotas anyway?)
We just can't be fucked to implement C11 support, and variadic templates were just too hard to implement in a year. (But ohmygosh we turned "^" into a reference-counted pointer operator. Oh, and what's a reference cycle?)
Look: Microsoft still has some old-fashioned hardcore talented developers who can code circles around brogrammers down in the valley. These people have a keen appreciation of the complexities of operating system development and an eye for good, clean design. The NT kernel is still much better than Linux in some ways --- you guys be trippin' with your overcommit-by-default MM nonsense --- but our good people keep retiring or moving to other large technology companies, and there are few new people achieving the level of technical virtuosity needed to replace the people who leave. We fill headcount with nine-to-five-with-kids types, desperate-to-please H1Bs, and Google rejects. We occasionally get good people anyway, as if by mistake, but not enough. Is it any wonder we're falling behind? The rot has already set in.

"""

Edit: This anonymous poster contacted me anonymously to make a second statement, worried by the attention his words are getting:

"""

All this has gotten out of control. I was much too harsh, and I didn't intend this as some kind of massive exposé. This is just grumbling. I didn't appreciate the appetite people outside Microsoft have for Kremlinology. I should have thought through my post much more thoroughly. I want to apologize for presenting a misleading impression of what it's like on the inside.

First, I want to clarify that much of what I wrote is tongue-in-cheek and over the top --- NTFS does use SEH internally, but the filesystem is very solid and well tested. The people who maintain it are some of the most talented and experienced I know. (Granted, I think they maintain ugly code, but ugly code can back good, reliable components, and ugliness is inherently subjective.) The same goes for our other core components. Yes, there are some components that I feel could benefit from more experienced maintenance, but we're not talking about letting monkeys run the place. (Besides: you guys have systemd, which if I'm going to treat it the same way I treated NTFS, is an all-devouring octopus monster about crawl out of the sea and eat Tokyo and spit it out as a giant binary logfile.)

In particular, I don't have special insider numbers on poaching, and what I wrote is a subjective assessment written from a very limited point of view --- I watched some very dear friends leave and I haven't been impressed with new hires, but I am *not* HR. I don't have global facts and figures. I may very well be wrong on overall personnel flow rates, and I shouldn't have made the comment I did: I stated it with far more authority than my information merits.

Windows and Microsoft still have plenty of technical talent. We do not ship code that someone doesn't maintain and understand, even if it takes a little while for new people to ramp up sometimes. While I have read and write access to the Windows source and commit to it once in a while, so do tens and tens of thousands of other people all over the world. I am nobody special. I am not Deep Throat. I'm not even Steve Yegge. I'm not the Windows equivalent of Ingo Molnar. While I personally think the default restrictions placed on symlinks limited their usefulness, there *was* a reasoned engineering analysis --- it wasn't one guy with an ulterior motive trying to avoid a bad review score. In fact, that practically never happens, at least consciously. We almost never make decisions individually, and while I maintain that social dynamics discourage risk-taking and spontaneous individual collaboration, I want to stress that we are not insane and we are not dysfunctional. The social forces I mentioned act as a drag on innovation, and I think we should do something about the aspects of our culture that I highlighted, but we're far from crippled. The negative effects are more like those incurred by mounting an unnecessary spoiler on a car than tearing out the engine block. What's indisputable fact is that our engineering division regularly runs and releases dependable, useful software that runs all over the world. No matter what you think of the Windows 8 UI, the system underneath is rock-solid, as was Windows 7, and I'm proud of having been a small part of this entire process.

I also want to apologize for what I said about devdiv. Look: I might disagree with the priorities of our compiler team, and I might be mystified by why certain C++ features took longer to implement for us than for the competition, but seriously good people work on the compiler. Of course they know what reference cycles are. We're one of the only organizations on earth that's built an impressive optimizing compiler from scratch, for crap's sake.

Last, I'm here because I've met good people and feel like I'm part of something special. I wouldn't be here if I thought Windows was an engineering nightmare. Everyone has problems, but people outside the company seem to infuse ours with special significance. I don't get that. In any case, I feel like my first post does wrong by people who are very dedicated and who work quite hard. They don't deserve the broad and ugly brush I used to paint them.

P.S. I have no problem with family people, and want to retract the offhand comment I made about them. I work with many awesome colleagues who happen to have children at home. What I really meant to say is that I don't like people who see what we do as more of a job than a passion, and it feels like we have a lot of these people these days. Maybe everyone does, though, or maybe I'm just completely wrong.

"""

mrb Friday 10 May 2013 at 9:14 pm | ¶ | Default
Used tags: performance, windows
fourteen comments

 
Well that is sad. But hey, every software piece out there turns into a monster eventually.

Monk, - 11-05-’13 00:03
 
welcome to IT. You can’t do shit to improve anything without some asshat taking offense. We might as well wear coveralls with our names on them and just say yes boss.

Hukklebuk, - 11-05-’13 02:43
 
What’s wrong with “9-5 with kids types,” it doesn’t make you a worse programmer to want a healthy work life balance. If you’re able to pull that off it even suggests a level of skill that your company will allow you out of the “you have to do overtime” culture.

I found that comment very offensive but other than that the article was a very interesting read.

James Pike, - 11-05-’13 03:10
 
Seems to me the obvious problem is project managers over valuing enterprise customer gripes rather than general everyday problems that any fool can see from 1000 yards. I always thought the inner workings of MS would be a modern day self organising meritocracy where experienced geeks intimate with Windows could recognise and improve key areas… Developer led. But instead it seems more like internal company politics and hierarchy are encumbering progress. This is even obvious from a consumer point of view, changing tack on major UI components of your flagship OS exhibits this.

As a .Net developer I am gravely concerned about the future of MS, should I be using C#, or JS with RT? Why is the in-app ad-ecosystem so bad and unprofitable? Why did the UK get the Surface RT pro so late and what’s with the un-consumer-friendly naming convention? COM/DCOM/RT/WTF?

Most guys I know are sticking with C# and branching into web, but then most of those guys think the ASP.Net Event Model is a good thing and try to make web apps with desktop like behaviour.

Me, I’m branching out into app-development for IOS and Android, and returning to my long-term passion JavaScript. I plan to switch to Linux and run windows and osx in a VM. And to be honest I feel a little inadequate because I’ve hedged my bets on windows for so long that I’m a little worried about the Linux learning curve.

And even when I’m looking for contracts, most people are asking for frontend or php or JavaScript.

I know MS are trying to attract developers to their app-store but frigging wake up MS. Developers are abandoning you because you’ve abandoned developers.

codingjohnson, - 11-05-’13 03:34
 
what he means with “9-5 with kids” is dudes who don’t take their shit seriously enough to go the extra mile without acting like they are in agony because “i have kids”

they are generally just trying to do the bare minimum and roll out because the only thing they care about is their “real life at home” and work is just a way to sustain that.

real hackers can have kids but still be passionate. those people don’t get called “9-5 with kids” they still get called a hacker. yes they can have kids and work/life balance but work/life balance means balance. it doesnt mean putting your work into a protective little risk-free box and going home every day right when the school bell rings.

if you are this butt hurt simply by the use of the phrase, maybe you are one of those people.

http://www.someecards.com/usercards/view..

alex, - 11-05-’13 04:01
 
Who gives a fuck what you found offensive, James?

BridgeLiver, - 11-05-’13 04:05
 
alex and BridgeLiver: watch your language. I want my blog to remain civil.

mrb, - 11-05-’13 04:12
 
15 years of MS only technology experience has ended up for me 2 years ago when I read Android developers guide. The ideas behind it seemed powerful as nothing else. JS Ecma5 standard astonished my imagination of the ideas and possibilities behind it. 
Seeing Microsoft out of these innovations I have it up. New ideas rule the world and me.

Vasyl, - 11-05-’13 04:28
 
alex &
BridgeLiver if your passion for the code of some ungrateful soul sucking entity takes more than one unnecessary minute of the scarce time with your kids your priorities are screwed.

joeg, - 11-05-’13 05:04
 
Alex, everyone is entitled to decide for them whats their “real life”. Solely for this reason, slavery is Abolished from this planet. If you want people to be motivated, blame game and guilt therapy is pretty bad tool. I guess you must be one of those who stayed back in MS because you could not make it to Google. How does THAT feel for sake of balance ?

BridgeLiver, your attitude will soon take you where NO ONE will give a fuck about you.

9-5-with-kids, - 11-05-’13 05:16
 
It sounds like the problems they have taken on are of the ‘everyone else is handicapping themselves this way, so we can get away with it too’ variety. When a company loses an experienced expert with irreplacable domain knowledge, replace them with a new college grad. The organization takes an enormous hit in every technical respect, but you save a lot of money. And since all of your competitors are already doing it, you will have aggressive business types winning all the arguments when it comes time for budgeting.

Society has already decided that every productivity increase brought about by technology and computers will benefit the business exclusively, with none of their benefits being shared with the employee. If productivity doubles, profit doubles, workload doubles, and compensation remains stagnant. It’s easy to actually reduce the % of your budget going to talent every year while still growing profit, and this is exactly how you do it.

It will work precisely until the workers figure out that thanks to the Internet the only thing standing between them working 1/10th as much and making 2x the income is some software to connect people who can provide work with people who need work done, cutting employers out of the loop.

codetaku, - 11-05-’13 05:27
 
How in the hell could a man enjoy being awakened at 6:30 a.m. by an alarm clock, leap out of bed, dress, force-feed, shit, piss, brush teeth and hair, and fight traffic to get to a place where essentially you made lots of money for somebody else and were asked to be grateful for the opportunity to do so?

- Charles Bukowski

9-5-with-kids, - 11-05-’13 05:29
 
I worked in Blvd 42 in the early-mid 90’s and I can’t imagine having 9-5 kids working on campus. I hadn’t heard that phrase before but I run into a lot of young programmers who were never taught some of the basics.

My direct report was a fantastic mentor. He had been at MSFT forever & was responsible for the first CD driver. Besides our code, he also focused on our build environment. Everyone had the same version of Visual Studios and during a product cycle he wouldn’t allow any updates or changes. He understood, as I have always taught, just changing the text of a string could affect your compiled code in ways you’d never imagine. These are the kind of habits you learn from education & experience.

I also occasionally participated in new employee interviews. The focus was always how someone thought more than what kind of education they had.

Thanks for reporting this comment and thanks to the gentleman who posted. It really is interesting to hear and picture how things work behind the scenes.

BillP

Bill Pytlovany, (URL) - 11-05-’13 07:02
 
“What’s wrong with “9-5 with kids types,”

I found that comment very offensive but other than that the article was a very interesting read.
James Pike, – 11-05-’13 03:10
“

I read it more that these kids coming out of college have self-entitlement issues with no backing to warrant it.

Hellscreamgold, - 11-05-’13 07:44
2013-05-11 12:50 UTC: I see ~50 HTTP requests per second, ~5 Mbit/s outbound traffic, and PivotX/PHP unable to handle the load at 100% CPU... Comments are disabled for this post due to the need to make the page static. I knew it! I predicted that PivotX's performance was going to be an issue, in my very first blog post!
« ASIC Development Cost… | Home |
 
About Me

Marc Bevand, aka mrb. I am passionate about high-performance computing, GPGPU, computer and network security, free and open source software, assembly optimization (AMD64, x86, ATI CAL IL), low-level programming, security vulnerability research and exploitation, reversing, Linux, *BSD, Solaris kernels internals, operating systems design, new hardware advances... I breathe all of this every day. Check out my personal page to learn more about my passions and what I do.

Tag Cloud

amd, amusing, android, attack, bitcoin, bruteforcing, bug, cpu, cryptography, finance, google, gpu, hack, hardware, intel, largescale, network, nvidia, performance, perl, pivotx, qemu, sata, security, ssd, storage, sysadmin, virtualization, web, zfs(all)
Latest Comments

Hellscreamgold op "I Contribute to …: “What’s wrong with “9-5 with kids types,” I found that comment very offensive but other than that…
Bill Pytlovany op "I Contribute to …: I worked in Blvd 42 in the early-mid 90’s and I can’t imagine having 9-5 kids working on campus. I …
9-5-with-kids op "I Contribute to …: How in the hell could a man enjoy being awakened at 6:30 a.m. by an alarm clock, leap out of bed, dr…
codetaku op "I Contribute to …: It sounds like the problems they have taken on are of the ‘everyone else is handicapping themselves …
9-5-with-kids op "I Contribute to …: Alex, everyone is entitled to decide for them whats their “real life”. Solely for this reason, slave…
joeg op "I Contribute to …: alex & BridgeLiver if your passion for the code of some ungrateful soul sucking entity takes more th…
Pages

About Me
Links
Archives

May 2010
Jun 2010
Jul 2010
Aug 2010
Sep 2010
Oct 2010
Nov 2010
Dec 2010
Jan 2011
Feb 2011
Mar 2011
Feb 2013
Apr 2013
May 2013
Categories

Default
Linkdump
Meta

   
Links

None
Top | Zorinaq - Powered by PivotX - 2.3: pre-alpha