Thanks for the really well thought out response!
For what it's worth, I also don't think what I've shown will be what programming looks like in 10 years, but I think something like it could obviate the need for a lot of the programming we do now. That's probably a different discussion though.
If Lotus Notes was so cool, why aren’t we using it today? Also, if programming in Excel is so awesome, why am I not developing banking software using it?
A combination of accidents of history, failures to adapt, and poor underlying foundations. It's hard to exactly explain the fate of any "failed" project (fwiw, IBM Notes is still a $1B business..), but Lotus took several missteps despite having truly amazing and obviously better technological solutions in many cases. Lotus Improv could have easily killed Excel and the entire spreadsheet market, but didn't due to absolutely horrible positioning (they continued to sell 1-2-3 at the same time, and never adequately explained why you'd want Improv over it). Notes continued to develop but never fully found its way into the next decade. It also suffers under its own weight.
Excel is arguably one of the single most successful pieces of software ever developed. Hundreds of millions of people use it every single day and people do truly incredible things with it, but it is a very specific tool - a tool that limits you to thinking on a 2d grid of numbers. So long as you can represent your problem on that grid, Excel will work wonders for you (maintainability issues aside), but most data doesn't present itself as just two axes. Bank software is guaranteed to have numerous dimensions of data that need to be worked with. Moreover, Excel is awful at systems building due to the lack of semantics around change and a tools for debugging/error reporting. It was really only meant to be a fancy ledger, and while people use it in much more interesting ways, it never really let go of that.
Openness and Accidental Complexity
When you program in a general purpose language like Java, or C++ or whatever, you can deal with accidental complexity, because these languages are like glue
That is actually exactly what we mean Eve to be. The language/semantics that underlie Eve come from the idea that programming is translating one format of data into another. Getting a drone to dance to an MP3 is translating the waveform of an MP3 into drone positions and those positions into motor movements. Bank software is about translating a series of transactions into a ledger under very strict invariants, translating user clicks and form submissions into those transactions, and so on. Eve is ultimately meant to be general purpose, the main difference is just that it has fewer data structures and so a much smaller set of inherent operations. I would argue it's a far more natural tool for gluing data from random sources together than either Java or C++ is.
“Closed” systems, on the other hand, force the world to play to their rules.
All programming languages force you to play by some set of rules, so I'm not sure that's the right axis to go down. The fundamental argument here is one of expressiveness - given some random string of bytes, what can I do and how hard is it given the rules? Eve definitely isn't going to be great at describing something like an operating system, or anything related to exactly how the machine works, but in terms of expressing domain-level data transformations it's hard for me to imagine something much better. It is amazingly expressive, e.g. in the talk I mentioned I wrote an IDE in 600 lines of code in less than a week. The guys at Berkeley re-implemented hadoop in orders of magnitude less code. The rules of Eve's world are far stricter than the rules in C++'s or Java's, but you don't sacrifice as much as you might think.
can we build open systems without importing accidental complexity?
That's an interesting question. Our answer is that we can help manage that complexity better than the alternatives, but we can't remove the inherent complexity that exists in dealing with disparate systems. Instead, we can design our language and tools to help us deal with it. A simple example of this is having complete provenance information for any value in the system. Knowing exactly what information from all these different systems feeds into a value is an incredibly valuable bit of information - one that no general purpose language will just give you. With a carefully applied reduction in "power" (e.g. not being able to talk about how a value is laid out in memory) we gain far more leverage for the set of tasks we're most interested: using computers as thinking tools. More "powerful" languages are strictly more capable, but that power comes at a cost in complexity. They aren't really any better at dealing with the accidental complexity of external systems, in fact, they just end up supplementing it with more of their own (citation: look at the code for any Java/C++ PDF parser).
Organisation vs Approachability
Global state is rightly vilified as being a source of errors in software: as soon as you scale your software endeavour, you run the risk of confusing pieces of global state.
Virtually every system eventually ends up dumping into a relational database of some kind. We need global sources of information. The fundamental issue with global state is unrestrained global mutation. The point at which any value can be changed by anything anywhere at anytime is the point at which we're screwed. To that end, Eve doesn't have the ability to programmatically change a value (at least at this point). A human has to manually update a value or something external to Eve has to put a new value into the system (e.g. if you build a UI, the UI "watcher" will stick new values in when a button is clicked). This is based on the ideas around "essential state" that papers like Out of the Tarpit talk about. The best way to think of Eve is as a set of views over an immutable log.
Another thing to keep in mind is when I talk about removing scope, I'm more talking about removing encapsulating scope; any value that I can see, I should be able to use anywhere. This doesn't preclude hierarchy or ownership, e.g. chris' age, it just means that if I have access to chris' age I can reference it in calculations arbitrarily. Per above, this is safe because all you can ever do is read a value.
I would argue that we have a far more powerful model for organization than anything strictly hierarchical (like classes) provides. Eve is a fully relational database, and so information (including "code" and the like) can be organized in arbitrary dimensions in any graph structure you'd like. You can even build arbitrary views over that information to create entirely new and specific organizations (e.g. "templates related to chris' age"). Ad hoc organizations turn out to be incredibly powerful.
Testing was all manual. Re-use was minimal. Revision control (a la git) was impossible.
By working at Eve's level, we get the ability to express invariants on the data directly, which we can then try to break automatically by trying inputs that could possibly invalidate those constraints. It's like generative testing on steroids. Moreover, without change, an entire class of bugs disappears. You're also working directly with real, or at least example data (which can be auto-generated), so it's a lot more obvious when things are going wrong.
What re-use means for us is an open question. Views naturally get you to express things in terms of sets that are always up to date, which means if I describe a view that translates my home movies into a different resolution, it will just go ahead and rerun when any new movie matches the search. So at some level, re-use is something you do accidentally just by using Eve. There's no reason why I couldn't take the same view and parameterize part of it either though, which gives you function-level reuse. From there, libraries and the like are the same as other languages.
We intend for Eve to have a more flexible and intuitive version control system than git. You can read about some of our thinking around that here:http://incidentalcomplexity.com/2015/04/22/version-control/
At some level all of these resolve down to issues with maintainability. I think one of the most exciting things about Eve for programmers is that it's a language built with observability from the ground up. Full data-provenance and a referentially transparent set of views give us complete information about why a value is what it is. This means that debugging is no longer a hunting and pecking exercise. You can ask "why is this 10?" and Eve will tell you all of the data that went into calculating it, the complete path it took... everything. You can even then start to ask questions about that, like where in this path did it get > 5?. Pair that with the ability to create arbitrary views of the system (e.g. "what templates would update if I modified chris' salary") and you end up with a ton of power to understand and navigate the system.
Focus vs Coverage
it’s better to be able to mix-and-match your technology choices to fit to the problem you have
I completely agree and that's why I like thinking of Eve as the glue between all those choices. The base language that Eve provides is one that is very specifically tuned for transforming data back and forth. It turns out a lot of programming really is just that. If you add in the notion of communication to the system (like Notes did) you account for a pretty incredible amount of functionality. All the examples we've collected over the years, from programmers and non-programmers alike, could be described as communicating information either to themselves, others, or other systems. So what we're building is a platform for doing exactly that - gathering information, transforming it, and communicating it to whomever or whatever you want to see it.
Where that will ultimately find a home is an open question to us and as you say we'll need to find focus. But wherever that ends up being, I don't think we'll really talk about it as programming. It doesn't look or feel like programming when using it. It's something else, something that just happens to accomplish the same tasks. In that way it really is analogous to Notes, which was just "something else" compared to what was out there at its inception. My pie in the sky dream is to take the Notes vision beyond an email workflow tool into a tool we could use to think and communicate with. Something that scientists could use to do their work to combat climate change and cure cancer. Something our teacher friends could use to automatically track their students and our lawyer friends could manage fundraising rounds with. Something with the power of python, but without the disasters that are virtualenv, unicode, and modern IDEs. I'm sure like most languages we'll find a niche, but there's no reason it can't still be generally useful.
There's no reason we can't shoot for getting our "bicycle of the mind."
Wow, that was extremely prompt and very thoughtful. Thanks.
What you've written here clears up a lot of my misconceptions, and is really useful in understanding where you're headed with this. Just picking up on three points here:
First, I don't want to appear down on Excel. I use it all the time and one of my github projects is applying Excel to functional testing, so if I came off as being anti-that, then that wasn't what I had in mind. But in any case, when you say:
.. that's exactly my point about approachability right there. If they'd put in all that other stuff - the public probably wouldn't like it as much, and I guess you'd also be moving in the direction of Improv... and towards mass rejection. That's the trade off: maybe, the answer is somehow to give people the impression that they have freedom and simplicity, but they end up "doing it right" anyway. There's probably a tonne of psychology in this.
Secondly, I can see the trade-off's you're making with Eve, from reading a load more of your documentation: pure functions, everything a relation. And yes, as you say, this doesn't necessarily make things any less powerful. As well as Notes, it seems conceptually close to Datomic or Apache Storm too, especially if implementing Hadoop in it is a thing. This follows a current trend towards immutability and distribution, which (I would argue) is down to us being unable to keep improving processors at Moore's-Law speed.
Honestly, if this is the case, I think a "bicycle of the mind" is selling it short. As programmers, we already have lots of those: Matlab, Wordpress, Evernote.. computers generally are this. But if you can distribute it, and have a team of people working on it, and you're on board with this trend towards distribution, then it's a bicycle which you can add wheels to, or extra drivers, and that really is something new and I've irreparably broken that analogy. Sorry. But, it does at least look like addressing some of the difficult software organisation issues I brought up in the post which I feel are key to that "computing in 10 years" question. (And also, that are mentioned in the post on revision control that you link to).
Lastly, I hope you're right on the glue/coverage thing... I don't have a good enough feel for Eve yet so I'll just accept what you're saying.
Anyway, it's very promising so far, keep up the good work.