Podcast transcript

podcast.txt

*Bubble sound effect fading out^
*Splash sound effect*

Agent 1: Where is it? Where did you hide it ?

Ivan: Huh?

Agent 1: The hard disk?

Ivan: You must look inside your heart and to find the motherland…*incoherent mumbling: UglyfyJS2, top the download charts of NPM it vill; trust me* -* gets splashed*

Agent 1: Dammit Ivan. That’s right we know who you are, spy, where you live. Maybe you should have thought twice before bringing your family to this country. Now tell me, where did you hide that data?

Agent 2: ahh just let him go, sport.

Agent 1: but- 

Agent 2: it’s 1973, act a little more civilised will ya. Besides, boys in the lab just told me this was just a pet project, some Ugly-my ‘’’J’’’’S’’’’ thing, won’t really go anywhere.  

Agent 1:  No, isn’t this about JS?  The joint services?  This smells like a dirty commie to me.

Agent 2: No, it is this thing - sounds like a coffee, some “Java Script”We probably spent more money on trying to find this guy than what he stole will ever be worth. Just kill him, his family and hit the bar.

Agent 1: Fine, but this is on you

Ivan: Now we can sleep freely, right frien- *Gun shot*

*Millennial music*

*hard cut to someone digging disk out of ground. Implied with by shoveling sounds*
Oh boy
Enterpriser 1: A hard drive? pretty old, an... IDE one. 

*harder cut to garage. Enterpriser one typing furiously. Footsteps fading in*

Enterpriser 2: Okay I got here as fast as I could, what do we have here?

Enterpriser 1: Since the dawn of time mankind has always strived to make things smaller, and now we can: by getting rid of js. But not with a hyphen, with a dot. And with a no, but with a node.

E2: Sounds redundant.

Enterpriser 1: Yeah well I guess getting filthy  rich is pretty redundant

Enterpriser 2: All I see is a floppy disk welded onto  your server block. Javascript.lp.. PROPERTY OF THE NATIONAL SECRET SCIENCE AGENCY? Where, how did you get this? What even is this?.

Enterpriser !: I’ll tell you what it is

*Typing ends with last audible click* 

Enterpriser ! : It’s web scale

*Intro theme* 

Hi, I’m Wisesa
I’m Sam
    And welcome to tickets of our time, where we look at the bugs, dramas and personalities that define our software.

Back to the plot.  Our real story begins on June the 6th; 2017.  Eric Tucker, working hard, when he got a call from his boss:

*Ring ring*

MA: CHECK PROD

E1: Sorry, how can I help?

MA: CAN YOU CHECK PROD???

E1: Sorry, what is the problem?

MA: WHY DO I EVEN PAY YOU TO WORK HERE IF YOU DON”T EVEN KNOW HOW TO CHECK PROD?  ARE YOU CAPABLE OF LISTENING TO INSTRUCTIONS?

E1: Yes sir, I am checking it now.
*keyboard clicks*
E1: Oh, I see the problem now.  Sorry I’ll be right at it.

MA: YOU BETTER BE AT IT… I’LL BE ROUND TO CHECK BEFORE YOU KNOW IT!

*time passes*

MA: So; what have you got for me?

E1:  Well turns out the deployment failed.  Something went wrong in the build process - with the zipping. 

MA: ARE YOU QUESTIONING MY BUILD PROCESS?  QUESTIONING YOUR MANAGER IS A SERIOUS OFFENCE!

E1: no no no sir.  Just a bug in a library we are using.  I’ll report it and get it fixed.

MA: YOU BETTER REPORT IT… BEFORE I REPORT YOU FOR YOUR INCOMPETENCE.

Eric sat down at his computer, and began typing the pug report.  He had found a bug in UglifyJS - an amazing tool to obfuscate and compress javascript files.  He wrote the report:

----typing-noise----
E1:  npm install of 2.8.28 is downloading files with timestamp of Dec 31, 1969

The package is not usable. I cannot zip up the build folder because zip cannot handle files with a creation date before 1980.

I had to switch to 2.8.27. Can you not just republish 2.8.28 to fix these timestamps?
----typing-noise---

But there was something our Eric Tucker did not notice.  UglifyJS maintainer Alex Lam was unsure why zip files should be involved at all; this was npm, or node package manager, package after all, not WinRAR.

Hacker News was quick to pick up on this; giving it the title “How not to behave on GitHub issues”.  The GitHub issue a smogersboard of demanding users and confused maintainers.  As one commenter said:

    Which side is behaving inappropriately? Hard to tell.. 

To the reply from HN user mdlap:

Inappropriate: Users being impolite and overly demanding.
Inappropriate: Maintainers (and supporters) insisting that a problem in their publishing process (whether or not they caused it) isn't their problem because the code works.

But what really happened here?

Why did this bug exists and who was at fault?  Was it maintainer Alex Lam,... or the user Eric Tucker?

So firstly, ZIP file really do fail with timestamps older than 1980.  In the unix world; where most nodejs stuff happens; timestamps revolve around the unix epoch of 1970.  Usually time is stored as a 64 bit integer, the number of seconds since the epoch.  But for zips, time is stored in the DOS format - which is really different.

It stores the time as 2 parts - 16bits for time and 16 bits for day.  Now things get a bit tricky here.  Every day 86,400 seconds.  But in 16 bits, you can only store from 0 to 65k. To get around this, the time is stored as 2 second increments - rather than the normal 1 second from the UNIX world.  The day component is then stored as the number of days after 1980 - which is after the UNIX epoch of 1970.

So ZIPS are special snowflakes when it comes to time.  If your system is dealing with ZIPs, and isn’t running DOS - then it is a good plan to check out these edge cases!

2ndly, it turned out the bug title was correct; there was a problem with the relase.  The tarball did infact have a file with a bogus creation date.  If you have version 2.8.28 of Uglify JS, might be lucky to have a README with the file creation date at the unix epoch.

Turns out, this was a bug produced when publishing NPM packages from Windows.  So the rage in our ticket today was misdirected - it wasn’t even a bug in UglifyJS!

The bug was in the “npm pack” command, which creates a tarball given an NPM package. On seemingly any OS, the file timestamps would be packed into the tarball as 0.  Nikolai Vavilov found the cause.  To pack a file into the tarball, it file fstat to get the information on the file.  Then it would copy the returned object from fstat into it’s own object - which was used to write the tarball.  But it never copied the “mtime”, or modification time, property.  When the mtime was not updated, it defaulted to 0 - the unix epoch.

This wasn’t a simple and obvious bug.  The copy code was looping over all the properties, but the mtime property was hidden.  Layers of abstraction and indirection made this a tricky bug to find and fix.  The bug wasn’t even in NPM - it was actually a subpackage called fstream.  And the bug wasn’t caused by fstream - because it was caused by an update the NODE JS core packages.



So that concluded our ticket for today.  A ticket that travelled down the line, from UglifyJS, to NPM, to fstream and finally to a small change in NodeJS.

This time - our ending is less clear cut.  Eric Tucker, the bug reported really did find a legitimate bug.  But Alex Lam was right to hold off from fixing it - this wasn’t a bug in Uglyfy’s code.

If you liked this story make sure to subscribe in your favourite podcast app, or share www.sam.today/podcast with your friends.

Until next time, this was Tickets of our Time - available from www.sam.today/podcast


---------

Show notes:

Issues with ZIP: https://stackoverflow.com/questions/3725662/what-is-the-earliest-timestamp-value-that-is-supported-in-zip-file-format
Bug in NPM: https://github.com/npm/npm/issues/16734

Ticket this time: https://github.com/mishoo/UglifyJS2/issues/2054