Instantly share code, notes, and snippets.

Embed
What would you like to do?
Why I think Immediate Mode GUI is way to go for GameDev tools

Why I think Immediate Mode GUI is way to go for GameDev tools

Prerequisites

Before you continue, if you don't know what IMGUI is don't bother reading this post, just ignore it, don't write anything in comments section, etc. If you're curious about IMGUI see bottom of this post, otherwise continue whatever you were doing, this post it's not for you. Thanks!

If you know what IMGUI is, for context read following presentations and blog posts:

The last link is not related to anything below but it's part of discussion above, putting it here just for people who are interested to see all opinions.

Disclaimers

  • I used extensively Qt in the past, I consider it to be one of the best designed C++ project, their documentation is amazing, there is huge community, etc. There is really nothing I dislike about it (not even moc'ing). All my experience with Qt is pre-version 5.0, but I don't think much changed about C++ side since they were focused to mobile and declarative UI side of things.

  • This post is not direct answer to Andreas, nor critique of their decision to use Qt, just my opinion why I think IMGUI is more suitable for GameDev tools.

  • When I say Qt below, I mean any traditional GUI library, like wxWidgets, WPF, etc.

  • When I say IMGUI, I mean dear-imgui since it's most popular one, but it could apply to any IMGUI library.

  • This is post only about tools UI, not game UI. I don't think those should be unified since constraints and requirements are completely different. I'm gonna keep writing tools UI in bold, because there will be always someone to ask what about game UI (it's NOT part of consideration here).

  • For gullible: Unity is doing it, they all the rage now, they make $$$, obviously you can attribute all of their success to decision to use IMGUI, End Of Discussion. See docs here: https://docs.unity3d.com/Manual/GUIScriptingGuide.html :)

Introduction

I never used Web Browser for building tools. I was curious when Insomniac announced their switch to web tech for tools. But Java Script always looked jenky to me for anything serious. To be honest, I think this about any interpreted code. ;) At the time I was hopping Native Client (NaCl) would take of and it would be fine to write tools in C++ and run them inside Web Browser. I shipped one game for NaCl, and that worked pretty well, but NaCl didn't really take of. Their postmortem explains in details why using web tech is bad if you only have C++ programmers working on it.

I used IMGUI style widgets for a long time. Most of use cases were for debugging or tweaking parameters in simple examples. Before I never considered it seriously for GameDev tools like world editor. When using Qt and similar frameworks there is a lot of utilities that come with it, but are unrelated to GUI widgets itself. IMGUIs traditionally don't have that part, and represent only widgets itself. For example, by just using Qt framework you get QDir or QFileInfo out of box. With IMGUI there is nothing like that, which usually mean you need to write your own, and it causes huge slow down early on.

Most of GameDev companies pick existing framework like Qt, WPF, etc. because they can get programmers productive ASAP, they don't need to ship tools on platforms other than Windows (which in case of Qt is not a problem, but it is in case of WPF). The reason why Unity didn't go this route is probably (guessing here) because at the time Qt wasn't licensed under LGPL (it was dual GPL / commercial license with $2K fee per programmer seat), and there wasn't any other viable cross platform solution for GUI, and they wanted to support OSX & Windows. Considering this, and ignoring Unity's case, it's pretty obvious route to choose existing framework from productivity point of view of manager or producer.

That all sounds good, what's wrong with it?

What is not obvious frameworks usually come with extra knowledge requirements of knowing how to use them, which makes person that knows it efficient, and person that doesn't know it very inefficient. People usually trivialize this with C# in the same way as they do with JavaScript, like everyone knows it, documentation is everywhere, etc. But that's not true, and C# requires people who spent enough time with to know it well. Just it appears that way because land mines in C# like in JavaScript are harder to trigger unlike in C++. Even if early on team is not split into two groups of those that know framework and those that don't, editor and engine teams, eventually there will be clear who knows more UI framework, and who knows more engine side... Which leads to another problem, person that knows the most about the system is not making the editor for it. In many cases this might not be huge issue, but it becomes major bottleneck for tools that require domain specific knowledge. When it comes to team, ideally whoever writes engine feature is also responsible for editor side of things, integration with world editor, user facing features, etc. Editor team with IMGUI could be much smaller and someone who provides editor infrastructure, undo-redo, load/save, docking, etc. not someone who writes specific editors.

If you used any framework before, and if you look pages 56-65 from Insomniac's slide deck, you'll realize that not many of those widgets are possible out of box. Most of them are custom widgets, or views of data that's also very engine specific. Custom widgets in Qt get really complicated pretty quick. If data is tree/list you could use QTreeView create delegates hook signals, etc. But when it gets really complicated you have to render it as QScrollArea or QPixmap. This is actually part where IMGUI is the strongest and why I feel it's ideal for writing GameDev tools. Code for custom widgets in IMGUI has hardware acceleration, you're not drawing pixel-by-pixel and blitting. Code for doing this with IMGUI is much more simple, and familiar to anyone working on the game. Just to illustrate this, here are hex editor widgets written in Qt qhexedit2, QHexEdit, there are million of these hex editors written in Qt you can search it. Then use that Qt code and compare it with hex editor written with dear-imgui https://gist.github.com/ocornut/0673e37e54aff644298b. Simpler? IMGUI widgets are highly interactive, everything can be updated every frame, with traditional UI framework this could be huge performance penalty since it's more traditional static UI solution.

Frameworks in general mean you don't embed framework to your engine, you embed your engine to theirs framework. Which complicates a lot of things, since engine is also framework itself. There are workarounds, grabbing window handle using it from other process, or using shared memory, or any other way of IPC, etc. But with IMGUI it's already in your engine, no need to figure out how to interop between renderer that your engine is using and framework's rendering API is using underneath.

Large frameworks have tendency that require large teams to support them. I lost count how many times Qt changed owner in last 10 years (Trolltech, Nokia, Digia, a few more?). Every time this happens you have to worry what new owner might do with it. You wonder is it going to be supported? Are they going to drop classic desktop widgets support and focus only on QtQuick in 2-3 years from now? With framework like WPF is the same, MS could just quietly stop working after some massive layoff and let it decay unsupported. MS is also famous for this hype-hype-then-abandon cycle for their frameworks (XNA, Silverlight, and others). Even if both companies drop everything on GitHub with permissive license on it, both code bases are massive and require huge investment to keep maintaining it and moving forward. IMGUI is simple enough that even small GameDev team could take control over it in case that original author decides they are done with it.

What IMGUI lacks?

As I said above utilities that come from frameworks are really huge boost in beginning. Nowadays is possible to use Qt's core library without the rest of Qt, so that might be good pick for anyone who doesn't have good base cross platform library.

All IMGUI libraries, that I know of, don't have well developed system for users to contribute their extensions to library. In Qt you could write very complex widgets, share it, without needing to change library or that Qt code knows anything about extension. The most popular IMGUI library dear-imgui has way to submit link to repo that contains extensions, new widgets, etc. but usually code doesn't conform to dear-imgui's standard, dependencies users choose are sometimes insane, and overall it's not easy way to integrate extensions with existing code. Or simply some people don't even choose to contribute anything back, because there is no easy obvious way to do it. This all could be solved with some more structured way of contributing to dear-imgui extensions.

IMGUI power usage is not good for mobile devices since it redraws whole screen all the time, that might be a problem and it could be solved, but we're talking here about GameDev tools UI, and my assumption is that GameDev tools are run on desktop PC.

IMGUI doesn't have native internationalization support, but your game should have that anyway, so you could use that system instead relying on frameworks for localization for tools and your own for game. Also if you're doing only internal company tools UI it's feasible to completely ignore localization, and just use the most common language company employees are using.

Examples

Here are few screenshots of GameDev tools made with IMGUI:

0 1 2 3 4 5

See other IMGUI screenshots here:
https://github.com/ocornut/imgui/issues/123
https://github.com/ocornut/imgui/issues/539
https://github.com/ocornut/imgui/issues/772
https://github.com/ocornut/imgui/issues/973
https://github.com/ocornut/imgui/issues/1269
https://github.com/ocornut/imgui/issues/1607

Summary

To summarize why I think Immediate Mode GUI is way to go for GameDev tools:

  • IMGUI is great for writing custom controls which are majority of controls you need anyway, but it comes with basic controls that big frameworks offer too.
  • IMGUI doesn't require domain specific knowledge of framework and it makes every programmer effective creating editors and tools to support their engine features.
  • IMGUI is simple enough that in case of abandonment by original author can be taken over trivially even by small team.

My prediction is that in a few years Insomniac's tools team will come up with another GDC postmortem explaining what's wrong with Qt, and why they switched from Qt to IMGUI. I'm just joking here, but you never know... :)

Extra

@nem0

This comment has been minimized.

nem0 commented Mar 6, 2017

Qt contains other stuff than UI but only highest common factor of feature on all platforms is usually implemented. So sometimes even though Qt has a feature I have to reimplement it. And chances are the feature is already in the engine.

Great gist.

@EmptyKeys

This comment has been minimized.

EmptyKeys commented Mar 6, 2017

Good thoughts, but you forget one simple thing.

There is no holy grail solution or "way to go for GameDev tools". Your decisions should be always based on your needs (requirements). Something which is way to go for you, will not be way to go for someone else.

@greggman

This comment has been minimized.

greggman commented Mar 6, 2017

IMGUI is awesome but it seems like 1/10th of the solution. Writing GUIs is hard and IMGUI makes that much easier but Tools generally require lots of robustness that a game does not. Tools need undo, games don't. Tools need not to crash when asked to load too much data, games should never be asked to load too much data. Tools need not to crash when a user selects the world, picks "copy" and then picks "paste" 5 times. Tools need to serialize much more than games (I'm sure some of you are using generic serialization libraries in your games but many games do not). Tools need to load older versions of data (games often don't). Tools usually need to load inefficient but editable formats, games often only load an optimized format.

My point is IMGUI has it's place but it's also seductive to use it to go down the wrong path. That path is hacking game editing tools into your game instead of writing real robust tools that are designed for editing and not hacking an editing GUI into a game engine which was never designed to handle editing.

A system like https://github.com/SonyWWS/ATF has more of that other support. I'm sure it might be better/easier to use if it was based on something like IMGUI but that the same time there are tools you can hack together in minutes or days from parts built into a larger "tool construction set".

@dougbinks

This comment has been minimized.

dougbinks commented Mar 6, 2017

The power issue is easily solved by not rendering when there is no input or no scene change. Using this I've reduced power so low during editor use that you can happily edit on a laptop with very low power consumption. You can check out the Power Save Mode option by downloading my Avoyd editor here.

@dougbinks

This comment has been minimized.

dougbinks commented Mar 6, 2017

Re @greggman response

Tools generally require lots of robustness that a game does not. Tools need undo, games don't. Tools need not to crash when asked to load too much data, games should never be asked to load too much data. Tools need not to crash when a user selects the world, picks "copy" and then picks "paste" 5 times. Tools need to serialize much more than games (I'm sure some of you are using generic serialization libraries in your games but many games do not). Tools need to load older versions of data (games often don't). Tools usually need to load inefficient but editable formats, games often only load an optimized format.

This has very little to do with the UI, and a lot too do with the engine changes needed for an editor, which will likely need to be done in C++ engine code anyway. So I don't think this is an argument against IMGUI.

@OndraVoves

This comment has been minimized.

OndraVoves commented Mar 7, 2017

What you think about combination QT+ IMGUI + HTML? I mean QT as main app + some widgets. Engine run standalone and window create to qt widget. IMGUI for GUI in engine window. HTML (QWebView) only for some widgets ex.: script editor . I use ace (https://ace.c9.io) editor because its better solution than QScintilla.

@DanielGibson

This comment has been minimized.

DanielGibson commented Mar 7, 2017

What you think about combination QT+ IMGUI + HTML? I mean QT as main app + some widgets. Engine run standalone and window create to qt widget. IMGUI for GUI in engine window. HTML (QWebView) only for some widgets ex.: script editor . I use ace (https://ace.c9.io) editor because its better solution than QScintilla.

Besides it's ease of use and flexibility, a huge advantage of (dear) imgui is how small it is and how easy it is to integrate.
Qt on the other hand is pretty bloated and creates a build dependency that causes further pain - if you add chromium-embedded or something for HTML it gets even worse.

The script editor case however is a good point, this is something dear imgui is still missing: A good widget for editing bigger amounts of text, ideally with syntax-highlighting (and maybe even the possibility to add more things like autocompletion etc).
This would be immensely useful for editing shaders, scripts and all kinds of config files one might use.
Of course it's also not trivial to implement and might introduce new external dependencies to Scintilla or something similar; also, as far as I know, doing this imgui-style would be pretty inefficient so this would probably need more explicit state on the user-side, giving it a more retained-mode feel.

@BobbyAnguelov

This comment has been minimized.

BobbyAnguelov commented Mar 7, 2017

So for me my concerns with IMGUI extend past obvious things like multi-monitor support, power draw, DPI scaling. Most of my tools experience comes from an enterprise background, in the past I've used the following frameworks: raw Win32, MFC, wxWidgets, QT 3, WinForms and most recently WPF.

When building complex editors, you often need multiple views on the same data. I.e. For a graph editor, you need the graph view, the outliner, the breadcrumb, the search dialog, node creation dialogs, etc... This often results in you creating a model abstraction of the data that all views will operate on. There also needs to be some mechanism for routing all events from the UI to run certain functions on the model.

With IMGUI approaches, you are additionally also explicitly managing the UI state e.g. what's selected, what's visible, view virtualization, etc. This UI state often needs to be shared across multiple controls i.e. selecting one object in one view needs to update other views (property windows, etc.). This implies any significant change to UI layouts requires changes to the model due to the UI state management code contained within. Now this can be ameliorated through the use of patterns like MVVM which I'm sure a lot of people are not familiar with and to be honest, is not an easily understood pattern and seems like more hassle than it's worth at first glance. Trying to convince people to do this will be difficult as it looks like a lot of boilerplate code, for little benefit. Though that approach is gaining traction in the web space where a few of the newer js UI frameworks are based around it. Furthermore due to IMGUI not offering any default bindings between model and data, it is up to the end user to try to build things in a reasonable way to allow future change to model or UI to not require extensive rework of the code. WPF + C# solves this in a very elegant way through the data context binding features and the property system on objects.

Other potential headaches is that we often have lots of standalone tools: content pipeline updaters, resource servers, data management tools, custom diff tools, etc... Some of these tools need to run on systems with no 3D acceleration hardware. People pointed out on twitter that you can use software renderers and so on for this, but it also means that you need a "game loop" and renderer code in all of your standalone tools. A game loop is a relatively pointless feature for tools as I mentioned tools are mostly event driven. That adds complexity and an additional maintenance burden.

Another concern is debug performance. Debug builds for games are often challenging enough to get running at a decent frame-rate without having the renderer also trying to render a ton of different UI widgets across potentially multiple windows. As most current IMGUI approaches are constantly rendering, this will severely impact the performance of a debug build with a full set of editor widgets open. Obviously you can try to only render widgets that are active, have, focus or have mouse over but this is not trivial as a lot of tools need to be updated as the game is running even if not focused, this again put all the performance management for every widget on the user instead of on the framework. Again not a hard problem but one that will potentially not scale easily across a large toolset or large team.

Another issue is reactivity of UI, having your UI only as reactive as the game is not in my opinion an ideal situation. Each time the game frame rate drops, so does the tool reactivity. I can only imagine the frustration of having your typing experience (latency, etc) affected by what's happening in the game at the current time. Furthermore in most UI frameworks, people are conscious of the concept of the UI thread vs. the logic threads. As such, it's good practice to have no work being done in the same thread as the UI. This means that we need to offload work to worker threads. UI frameworks manage that complexity for you. So if you want multiple long running or blocking operations active with progress bars that update and still have a super responsive UI this is easily achievable. With IMGUI you need to manage that complexity and naively interleaving potentially blocking UI work with your game work (using the engine's job system) can potentially block the whole engine or badly affect frame rates so you are left with creating a whole new set of "UI only" worker threads for those tasks and managing all the scheduling complexity yourself. IMGUI tends to unfortunately make it much easier to simply do the UI logic in place so inexperience programmers will be guided into doing that whereas other frameworks like WPF will try to offload work from the UI thread for you with dispatchers. Just look at UE4 slate to see all the potential performance and reactivity issues you can end up with if you are not careful.

Another argument is for hiring, obviously for smaller teams this is less of an issue but it is often easier to find people with experience in the larger standard UI frameworks than it is for things like IMGUI. Also tool development is an art, just like rendering or animation or game AI. There are domain experts in terms of tools architecture and implementation, there is a lot of experience with and for the existing frameworks and so it's easier to hire experts in them or to get help from the communities. A lot of tools developers that are focused on build systems, automation, bug report systems have no game engine experience and so forcing to build their tools with your IMGUI style approach is counterproductive as is supporting multiple tool frameworks for different tools. Never mind the fact that trying to keep a consistent UI design language across different frameworks will be a nightmare.

Now I feel a lot of people have been burnt by the massive out of the box complexity of UI frameworks and dont understand why all that is needed just to draw a few buttons. I felt the same way until I had to build large toolsets at that point you slowly start to understand why all that complexity is needed. It isn't just a bunch of bored engineers over-engineering everything pointlessly. Modern tool development is complex wherein multiple views on data need to operate simultaneously, people want unified copy/paste & undo/redo support, etc... It's dangerous to underestimate the skill needed to develop robust enterprise grade tools, especially responsive ones.

Just my 2c...

@nem0

This comment has been minimized.

nem0 commented Mar 7, 2017

@BobbyAnguelov what defines complex UI? Is it ammount of data it has to handle, ammount of different controls, or something else?

@BobbyAnguelov

This comment has been minimized.

BobbyAnguelov commented Mar 7, 2017

Lots of different controls all interacting with one another depending on context, operating on the same data, etc... Unified undo/redo, command systems, different visual representations of the exact same data based on view, etc...

Imagine I have a set of nodes, which are drawn in a visual node editor, but I also have the same nodes shown in the same time in an outliner but I can have multiple views for the nodes in the outliner (grouped by type, list, tree). Maybe selection is done programmatically or even through another dialog... It requires a lot of thought to build a system like that in a extensible way, especially if the UI/UX needs to change and adapt. A model definitely is necessary to abstract the actual data operations so that you can also automate UI operations without UI. You also need to abstract some of the purely UI related data that doesnt belong in the model. This is sometimes retained in the controls themselves or in the case of WPF split between the controls and the viewmodel, with IMGUI you would almost certainly need to think about how to effectively manage that data so that you have no dependencies between visual state data and model data.

I think I'm having a hard time explaining my thoughts without concrete examples but I dont think another wall of text will help anyone :P

@nem0

This comment has been minimized.

nem0 commented Mar 7, 2017

I'm in a unique position where I've completely reimplemented the exact same editor in Qt, C# and dear imgui and tried to implement in (but quit quite early) Gwen, CeGUI, Rocket and a webbrowser. I think I can consider the editor quite complex according to your definition. dearimg is by far the best. It really has a lot of problems you already mentioned (e.g. responsiveness), although some of them are related to dear imgui (multimonitor support, lot of missing features) rather than to the imgui concept and some of them are not related to imgui concept at all (DPI scaling, performance). For some stuff I have to implement "retain mode on top of imgui", on the other hand when I was using Qt I had to implement "imgui on top of retain mode", e.g. for property grid to display runtime data. Connecting all the events is pain in the ass and you can easily miss an event. What's worse, sometimes there are just no events. I found making RM on top of IM much easier than IM on top of RM, but that's just subjective.

@BobbyAnguelov

This comment has been minimized.

BobbyAnguelov commented Mar 7, 2017

My experience with QT was from 3 so I cant speak to that. I built live displays of runtime data with wxWidgets and didn't have any IM style code, same with WPF. Granted I tried using QT 5 over the weekend to try that list example you had and to be frank, it's gotten a lot more complex and confusing since the last time I used it. I cant help but think a lot of people's bad experiences are due to unfamiliarity with the frameworks and yeh, that is bad in itself but saying the approach is bad in general is too absolutist for me. IMGUI for me is a great tool for in-game debug widgets and visualizations and that's the role it will play for me.

But yeh, it seems I'm the only one that feels that way so I think I'm just gonna shut up :)

@nem0

This comment has been minimized.

nem0 commented Mar 7, 2017

I built live displays of runtime data with wxWidgets and didn't have any IM style code, same with WPF.

How did that work if there are no callbacks/events/delegates/whatever?

@bkaradzic

This comment has been minimized.

Owner

bkaradzic commented Mar 8, 2017

@BobbyAnguelov Qt3 was way more complicated than Qt4, but Qt4 changed to MVC architecture. It's quite elegant how Q*View widgets work even for very large number of items, just for things that are not pre-packaged in Qt, like animation timeline, spline editor, etc. things get really complex.

IMGUI for me is a great tool for in-game debug widgets and visualizations and that's the role it will play for me.

This is what I felt all the time, until one day it didn't just clicked. Qt was my choice for UI for a long time before that moment... :)

@cgbystrom

This comment has been minimized.

cgbystrom commented Mar 12, 2017

Thanks for sharing!

Multi-monitor support and not letting the UI interfere with game view could be solved using a remote viewer.
Something like https://github.com/JordiRos/remoteimgui/tree/master/src/app
Also addresses pointer grab and lack of mouse when developing on consoles.

@JSandusky

This comment has been minimized.

JSandusky commented May 1, 2017

When for whatever reason one must use a retained GUI it's worth pointing out that a builder style helper class can bring a lot of that down to an IMGUI style while keeping most of the retained advantages.

So sometimes even though Qt has a feature I have to reimplement it.

This is the norm. Whether Qt/MFC/Winforms/WPF you're going to encounter half-baked undo/redo, internationalization, and help that isn't helpful or sane to use. Realistically you always have to do these yourself (or reuse what you wrote ages ago). Even QAction is half-baked when it comes to global consistency and managing routing.

For some stuff I have to implement "retain mode on top of imgui", on the other hand when I was using Qt I had to implement "imgui on top of retain mode", e.g. for property grid to display runtime data.

Given Qt, this isn't surprising. Since Qt wants to own your data-model, in a "feed me first, then I'll feed you and that's it" fashion, it's not odd at all to implement an IM style widget in place of Qt's model based widgets. Thankfully Qt repaints pretty much constantly so it's not a lot of work, just painting and pumping tagged-rects for commands. I use an IM style "Repeater" instead of Qt's lists and trees that "just does it" as far as external changes go and gives me the info I need for triggering embedded edits and column info. Not having to rectify a Qt model with my actual data all of the time is a buddha-send.

Where is using something like Dear IMGUI or Nuklear never going to fly?

  • When you're developing a commercial product that is more than a "wizard" (looking at you CrazyBump). GUI in the renderer isn't going to fly for a commercial tool, we don't need more zBrush level atrocities.
@darkuranium

This comment has been minimized.

darkuranium commented Feb 24, 2018

@JSandusky Care to elaborate on your zBrush complaint? I've never used it, so I'm not sure what the problems are of having a GUI in the renderer.

@zwcloud

This comment has been minimized.

zwcloud commented Jul 25, 2018

I'm implementing a C# based ImGui named Hello ImGui.

Here are some of my thoughts on how to implement an ImGui. I will add more as the developement goes on.

UPDATE 2018-11-2
Yet another IMGUI I just found: https://github.com/rxi/microui

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment