According to Jonathan Blow, an ambitious project (or "deep work" as he calls it) is something that changes you.[1] Afterwards you are better at what you do, not in an incremental way, but in a substantial and qualitative way, you become a unique specialist in your field.
Blow spent 3.5 years designing the game Braid, and it taught him small puzzle design. He spent 7 years designing the game The Witness, and learned how to create long arcs of rich, non-verbal communication through puzzles. These were long, deep projects that changed who he was as a designer, and he carries their lessons forward in the work that he does.
An ambitious project is not necessarily a long project---though they are often correlated---but it is work you do without knowing the destination, led by a guiding compass. Bret Victor likens this guiding compass to the cause of a social activist.[2] It is motivating, personal, and subjective.
Victor's cause is giving creators immediate feedback as they craft their ideas. He has created several prototypes for designing games, electronic circuits, art, and animation. He has used this principle to guide many years of research and thought. The right idea can shape the future by revolutionizing health, transportation, or education. For Victor, it is a tragedy to see ideas die on the vine because creators lack the tools to express and shape them.
An ambitious project can be done from first principles, and it does not require any special tools, but it also requires a certain level of competency. You may need to attempt some long non-ambitious projects first to get to the place where you're ready for an ambitious project.
Rich Hickey was only ready to design Clojure after many years creating complex, real-time systems.[3] It was the pain he felt designing these systems that gave him a sense for the problems Clojure could address and the skill needed to do it.
Finally, you judge whether a project is ambitious or not. An ambitious project is defined by your guiding compass and the way it changes you. It may change the face on an industry, or the world, but few will support and encourage you even if they benefit from its fruit. In fact, according to Blow, most people don't even believe that deep work exists, and Victor's technical activist sees problems that others don't even perceive.
An ambitious project can change you as a person and professional and can be very rewarding. Why don't people do ambitious projects? There are two reasons: cultural pressure and cultural blindness.
As Blow says, people set their expectations low to be socially acceptable.[1] If you decide to do an ambitious project, and tell other people about it, you can get responses anywhere from ridicule to anger. This seems especially true as programmers.
Gary Bernhardt gave a talk "A Whole New World"[4] where (sorry to spoil it if you haven't seen it) he lied to the audience about creating a new editor and terminal that supported a wide array of amazing features. After confessing that he didn't actually create an editor (or terminal) he asked why people had responded with surprise and even anger that he might attempt such an ambitious thing. I think Bernhardt's discussion of why people might respond with surprise or anger is interesting, but for my purposes it is enough that people do, which creates cultural pressure to avoid ambitious projects.
A second reason that people don't do ambitious projects is cultural blindness. Victor did a talk "The Future of Programming"[5] that pointed out how the 60's and 70's were a fertile time for ideas about how humans would interact with computers and how computers would change the way we think, but we ended up with 40 years of one paradigm of interaction. The worst part for Victor is we don't think there are alternatives to the "way we've always done things."
Victor's technical activist sees problems that others don't see, because everyone else is blind to the possibilities. When you grow up with technology you tend to accept it without criticism. It defines the limits of your thinking so you don't consider the other ways to do things.
[1]: "Attempting Deep Work" by Jonathan Blow
[2]: "Inventing on Principle" by Bret Victor
[3]: "Effective Programs" by Rich Hickey
[4]: "A Whole New World" by Gary Bernhardt
[5]: "The Future of Programming" by Bret Victor