by "scientificninja" (Josh Petrie?)
To begin with, the term "engine" (specifically as it related to the game development world) has no strict definition. Therefore, in the interests of keeping everybody on the same page, I'll define the term as I intend to use it in this article. An "engine" is a collection of robust, reusable software subsystems (possibly including both code libraries and tools) designed to facilitate the development of actual games by addressing specific requirements. The requirements tend to be broadly-defined: rendering, audio, physics, et cetera. Particularly ambitious engines that address multiple broad requirement groups tend be to known as "game engines" rather than just "graphics engines" or "physics engines."
Now that that's out of the way, let's turn to the real issue: how to build engines, and more specifically, if you should even bother.
For any number of reasons, many neophyte game developers (and even some moderately experienced ones) seem to think that a game engine is a required, critical part of a game. They think that — in keeping with the mechanical analogy from which the term "engine" comes from in the first place — like a car, a game with no engine will simply not run. But that mechanical analogy starts to break down when you take it this far, because when you look at the reality of the situation, a game engine is about reusable components. Much like you can build a car with custom parts or generic ones, you can build a game with custom components or reusable ones. The fact that you must have an engine to build a non-trivial game is a fallacy, something perpetuated largely by people who don't know any better.
Nonetheless, every so often comes the obligatory post on GDNet by somebody who has decided its time for them to write their engine, so they can start making some real games. These posts usually consist of requests for resources about the engine design and development process, and in fact chances are that if you're reading this, it's because I caught you making just such a post and linked you here so that I could crush your dreams with a minimum of effort on my part.
So my advice to you, if you're trying to write an engine, is: Don't. No matter what your reasons are — it doesn't matter if you're writing an engine so you can write your dream game, or if you're writing an engine because you think it will be a good learning experience, or any number of similar reasons. They're all wastes of time. You can sit down and write a game without writing a pre-written engine, and in fact this is very often the better approach, regardless of why you want to write an engine. The entire development process goes much more smoothly if you are focused on writing a game instead: a game is much easier to identify requirements for, much narrower in focused, much more rewarding when finished, and much, much more useful.
Most hobby developers who "finish" an "engine" that was designed and built in isolation (with the goal of having an engine, not a game, upon completion) can't ever actually use it, and neither can anybody else. Since the project didn't have any goals, any boundaries, or any tangible applications, it essentially attempted to solve aspect of the chosen problem space and consequently failed miserably. These "engines" tend to be little more than an inconsistent and messy amalgamation of random functionality that isn't terribly practical, robust or reusable. Furthermore, the actual viewable end product of these "engines" tend to be a cute little map (loaded from one of the popular BSP formats) you can run around in and observe some cute little models (also loaded from a popular model format) rendered with some fairly basic lighting effects (maybe some per-pixel normal mapping or parallax mapping). Maybe you might even see some collision detection or animation. These projects are just tech demos, and not particularly compelling ones at that. There is nothing to them, and they feel rather flimsy and inflexible.
The solution, even if you really want to make an engine, is to make a game instead. I can hear you crying out in protest from here, but just bear with me for a minute. The game does not have to be an epic production. It does not have to be GTA5, Quake 17, the next Elder Scrolls game, or a WoW-killer. It just has to be a game, with well-defined (and ideally well-thought-out) gameplay and well-defined developmental scope. A game is much easier to identify requirements for, as I've already mentioned, and it is also a practical application of those requirements. Once you have made one game, make another. Then another. Each time you start a new project, you should identify functionality that could be used and pull it out into a common library of base code. You'll probably have to refactor some of your code to remove explicit dependencies on other code or data that is game specific, but this is a good thing. It will help you generalize what is generalizable in a way that you can still test the generalized functionality against your finished game and confirm that it still works (obviously you might have to modify some game-specific code to adapt to the increased generality, as well).
If you repeat this process long enough, after a few games you'll have the beginnings of a solid collection of reusable functionality that has been proven to have practical applications. You'll also have a set of far more interesting demonstration applications that could also double as test harnesses for your engine. This method of growing an engine (rather than manufacturing it from whole cloth) is superior because it helps you shape your goals, it forces you to actually think about the problems you will face and their practical implementations, it forces you to think about the separation between the domain-specific logic of the engine and the game. It makes you learn from your mistakes rather than pretend that mistakes cannot be made.
Great Point!