Rethinking UI programming

What might the next competitor to WPF look like? How would I design one?

My overall goals would be:

  1. Easy to learn
  2. Favors composition over inheritance
  3. Great performance
  4. Encourages the use of patterns
  5. Cross platform and open source

I think a basic UI library would be composed of:

Diagram of the major components of a UI toolkit

  1. A markup language that programmers would use to design user interfaces
  2. An object model (like a DOM) that the markup is translated into
  3. A renderer, which sends the scene to the graphics card

What follows are some unordered thoughts about how I'd build the next UI library.

Markup

XAML and HTML are very similar. But if you ask most developers, they'll rate HTML as being much easier to learn than XAML. Markup languages are a fine way to describe a user interface, so I want to keep that concept, but there are some things about XAML that I don't quite like:

HTML is a markup format, while XAML is a serialization format. This meant that a lot of the implementation details leaked through. Property element syntax exists because the XAML parsers (which are really just serialization engines) would get confused and try to instantiate a ContextMenu object rather than setting the ContextMenu property. In contrast, with HTML, the interpreter would just know what ContextMenu meant in that particular context. Tags like DataTemplate would also be a thing of the past.

I would also want to avoid all of the "tax" that applies to XAML files. You know, XML namespaces, x:Class entries, and so on. I should be able to throw <strong>Hello</strong> into a .view file, hit F5, and see it.

My markup language would also be designed for programmers, not designers. HTML was designed to be hand-written, so it's easy to hand write. XAML was designed for tools to emit, so hand-writing XAML is an exercise in verboseness and RSI.

Composition

In WPF, the simplest controls have a huge hierarchy of inheritance, each layer of which adds different capabilities: Button : ButtonBase : ContentControl : Control : FrameworkElement : UIElement : Visual : DependencyObject : DispatcherObject : Object.

If you open the source code (in Reflector or via the recent source code debugging support) you'll find each of those are made up of thousands of lines of code, if not tens of thousands. And of course most of it is internal so you have no chance of changing any of it :)

The DependencyObject concept is one from WPF that I'd definitely keep. I think we also want to be able to write code to just new Button() and add it to a scene. I just want to cut out the middle layers of inheritance.

A button should look like this:

public class Button : DependencyObject 
{
    public Button()
    {
        AddCapability(new Visual());
        AddCapability(new Sizeable());
        AddCapability(new Clickable());
    } 
}

Each capability would be an aspect that hooks on to the element. Aspects that are commonly found together (most controls that support hit testing support probably also support MouseOver events) can be grouped into composite capabilities.

Performance

WPF is built on DirectX, which is a very welcome break from GDI+. But I suspect I'd go with OpenGL just to keep it open source. I think I'd like to combine some of the concepts though - my UI stack should allow me to mix a retained scene rendering mode (like WPF) with immediate mode rendering (GDI+). Concepts like UI virtualization should also be supported.

The other major area of performance I'd look at is having multiple UI threads. The low level graphics systems are capable of doing this, but most UI toolkits don't expose it. Ideally, I'd be able to demarcate a branch of the scene tree as being rendered by a different thread.

For example, in my version of Outlook, each of these panels could have its own rendering thread:

Outlook

Binding as law

In WPF you can do this:

label.Text = customer.FullName;

In my UI toolkit, you can only do this:

Bind(label.Text, customer.FullName);

By forcing binding to be used, we change from a push model (where the UI logic thread can throw things at us), to a pull model, where we only accept updates when ready.

When property changed events are fired, we can mark a UI property as having a pending change, but that is all. When the UI rendering thread is ready, it can read the new value. This means instead of having to lock and limit to one UI thread, we can have many, since each thread reads property values when it is ready.

Enforcing patterns

My UI toolkit wouldn't have the notion of code behind. Instead, it would be geared towards something that looks like Presentation Model/MVVM, and the infrastructure would make it easy to work with. For anything else, you would write a custom Capability to extend the behavior of the object.

The toolkit would also support concepts like DI and IOC, and have out of the box support for mediator, navigation and composition patterns.

Open source and cross platform

I think I'd write this UI toolkit on top of OpenGL, probably using C++ for the rendering portion.

C# is a fine language, but it is too closed to be a good way of writing UI code. The fact that automatic properties still don't support INotifyPropertyChanged tells me C# just isn't meant for UI coding. I'd probably consider a nice dynamic language, like Ruby, or maybe something on the JVM.

Conclusion

This is the point where someone will kindly tell me that [insert Java/Linux widget toolkit] already does all of this :)

A picture of me

Welcome, my name is Paul Stovell. I live in Brisbane and work on Octopus Deploy, an automated deployment tool for .NET applications.

Prior to founding Octopus Deploy, I worked for an investment bank in London building WPF applications, and before that I worked for Readify, an Australian .NET consulting firm. I also worked on a number of open source projects and was an active user group presenter. I was a Microsoft MVP for WPF from 2006 to 2013.

12 Jul 2010

Yeah, I think Lisp has had this since about 1978...

More seriously though, the multi-threaded thing is a problem on Windows (as far as I know) because windows makes some assumptions about each process only having one UI thread. Interacting with things like the clipboard is (apparently) problematic. Windows wouldn't necessarily have to change for this problem to go away, the programming model could hide it from us. Either that or we wait for whatever the UI model for Midori looks like....by all accounts it makes the early versions of WPF look well documented and easy to understand.

Why the lucky stiff's 'Shoes' sounded similar to this in some ways - Ruby lang, OpenGL rendering (among others) via Cairo, cross-platform. The only problem is that 'Why' pulled all his stuff off the internet not that long ago, so I'm not sure if you can even find the 'Shoes' code on-line anymore.

ross
ross
12 Jul 2010

It is interesting to think about such abstract things, and your ideas on the threading the UI are interesting but of course the devil is in the implementation detail, particularly given the the threads we think of are just simulated on the event loop. But that is just uninformed speculation on my part.

Here a few comments :

  1. I would have thought the boxes would be arranged different, eg the rendering engine would feed off the markup? Your diagram sort of implies that the rendering engine doesn't know about the markup, or gets the markup indirectly from the object model.

  2. If you are talking about a sound design, I don't think it really matters whether something is closed or open source.

  3. Having done a fair bit of both html/css and xaml programming, I much prefer xaml. Xaml is somewhat more complicated syntactically, but it is much less complicated to achieve sophisticated results and you can trust it. An example of this is a simple 3 column screen layout, where you want to split the screen into 3 even width columns that resize to the parent container (browser / window). In my opinion it is an ugly chore in html/css and a 1 minute job trivial job in xaml. The essence of this is that containers / panels are first class citizens in xaml and an afterthought / hack in html/css.

  4. I like your idea on binding only.

  5. I like the idea of a good navigation / composition framework.

  6. I think you can leave DI / IoC / NVVM out of the discussion. If you are building a separated presentation UI framework where binding is enforced, you shouldn't know or care about the engineering of the object model. Whist the engineering of the non-UI layer is interesting an important, it should be completely decoupled from the UI.

  7. I don't find the verboseness of xaml a problem, except in small projects. I think that styles / templates alleviate a lot of verboseness, and as a project grows in size this becomes a real strength. And secondly, whilst it might make purists cringe, ctrl-c -> ctrl-v is extremely useful and reduces the RSI :-)

  8. I don't really get the argument about c#. In xaml, the only thing you really need to do in c# is implement INotifyPropertyChanged and probably some commands. It is axiomatic to say that you need an ingoing and outgoing interface to your object model, so how can it be done any other way? Automatic INotifyPropertyChanged would be nice, but it is somewhat trivial really. Try out code snippets, or use an Aspect Oriented framework if it really is a productivity issue.

12 Jul 2010

For me, any future UI frameworks need to build on the separation of form and function that has been achieved through things like CSS, and evolved in WPF.

Obviously you are probably designing this UI framework for yourself :), but for me I reckon designers have to be first class citizens. The framework has to be designed around tooling - at the same time keeping it viable to write by hand. WPF does fairly well in this regard, but in many ways its definitely a 1st generation implementation (lots of ugly things you would do differently next time).

If your markup language is designed for programmers, that is presumably because it allows them to define a UI in a more natural, declarative, way? The essence of this though is you want an easier way to "design" or conceive the UI (a largely visual discipline). And as soon as start wanting to build something visual why optimize it for programmers instead of people who would be able to deliver the best visual "form"?

Jonx
Jonx
12 Jul 2010

C++ ? IRK ! As long as it's one of the existing .net language ;)

As for the binding... May I suggest that indeed, this would be the syntax for simple two way binding: label.Text = customer.FullName;

Now how is this gonna happen? No idea, you tell me ;)

12 Jul 2010

You lost me on Cross Platform. When will we learn that great UI is never cross platform?

12 Jul 2010

Hey Joseph, Shoes is still very much alive and kicking. We're gearing up for a 3.0 release on Whyday (Aug 19th). You can check it out at http://shoes.heroku.com/

12 Jul 2010

I generally agree with your ideas about composition, forced binding, and encouraging the use of patterns. However, some of your other ideas seem problematic.

Consider your thoughts on Xaml. In its most common Xml-based form, Xaml is, for the most part, as simple as it could reasonably be. The language designers didn't arbitrarily decide to make it more verbose than necessary. Take the 'property element' syntax as an example. Properties, which may be declared at various levels of an inheritance hierarchy, must support type qualification in order to guarantee successful runtime binding. A subclass might hide an inherited property of the same name, or a class might implement two interfaces with conflicting property names. It may be necessary to set the Control.Name property instead of the IHasMetadata.Name property. Your proposal of composing "Capabilities" or behaviors would likely result in this sort of name collision occurring more often. How does one override members of a capability? The short answer is probably "you don't". So, how then do I specify that I want to add a handler for the InputCapability.MouseOver event instead of, say, DataGridHeader.MouseOver? I would have to qualify it. In fact, I would always have to qualify it, because new capabilities could be added or removed at runtime. There's no way to know at design time that there will not be a collision.

There's still the problem of how the markup parser could differentiate between a child element and a property element: if you do not use a special syntax for property elements, then the only solution is for each element to have its own rules as to what nested elements are supported and how they are interpreted (i.e. a text element would know how to interpret the strong element). This would result in developers having to learn and remember a new grammar for every control.

How can you get rid of namespaces in the markup language? I need to be able to differentiate between my custom Button control and the standard Button control.

I suspect multi-threading thing is more problematic than you realize. In a multi-threaded UI system, the order of event delivery is not guaranteed. What happens when the MouseUp event arrives before the MouseDown event? This is why WPF and other UI frameworks use an event loop. Even if you only support multi-threading for the actual rendering/composition, there are still synchronization issues. I'm also not convinced that multiple threads would actually improve performance in most cases. My interpretation of your design is that the rendering thread may recompose a scene in response to a property invalidation. Well, what if there are lots of property invalidations occurring in short window of time? You'd likely end up with a lot of intermediate repaints between the first and last invalidation in a batch. This could definitely have a negative impact on performance, say nothing of the odd results that the user might perceive.

Just some food for thought.

Cheers, Mike

John Graham
John Graham
12 Jul 2010

Nice thoughts.

One remark though, maybe it's been made already.

XAML is not a WPF specific UI, but a technology to build any object graph. It's also used for WF for example. As such it is not a pure UI technology.

12 Jul 2010

Hi Mike,

Re: XAML, that's only the case when the XML is just a serialization format: "Every element is a class, every property is an attribute". But a plain old XML parser can apply a lot more context than that - "When you encounter a button with a contextMenu child element, you need to do this".

Consider how XML configuration works in .NET - your objects are given an XML tree and told to figure it out for themselves. They could map every element and attribute to objects and properties, or they could apply a lot more context to it. They can know that when they encounter ContextMenu it means to set the ContextMenu property rather than newing up an object.

Paul

Patrick Klug
Patrick Klug
13 Jul 2010

Great post! Very inspiring!

13 Jul 2010

Steve Klabnik - nice to see Shoes is still a going concern. There is too much "web" out there these days....client app development needs more love.

13 Jul 2010

From time to time, I think about how I would re-write Silverlight as well...

I like the idea of using composition over inheritance. The mixin approach via your addCapability reminds me of how Rails 3 was re-engineered.

Though I would not expose implementation nuisances such as DependencObject, INotifyProperty, or Dependency Properties. Bake those concepts (if they are even needed?) into the magic. Most of those, including the need for DI, are idiosyncratic artifacts of the framework being based on a staticly-typed system (language). If you use a dynamic solution at the core, a la Ruby or Python, you can alleviate most.

I am not a big fan of OpenGL, mainly due to it's painful, procedural approach. However, I agree it should be cross-platform; so it may as well be OpenGL.

13 Jul 2010

Hey Paul,

It's about 1 AM here so I'm not in a good state of mind to poke holes in your concepts...not sure I'd want to anyway because I generally like were you're going.

You KNOW I'm a big fan of Composition over Inheritance. I feel that if the WPF team had realized the true power of Attached Properties sooner (with the Attached Behavior pattern) we would have seen a flatter inheritance tree in the framework.

I think a good compromise between the current property syntax and absolutely no qualifier would be the simple introduction of a This notation for XAML so instead of

<myns:MyRidiculouslyLongControlName.SomeRandomProperty>
 Foo
</myns:MyRidiculouslyLongControlName.SomeRandomProperty>

I could just type

<This.SomeRandomProperty>
  Foo
</This.SomeRandomProperty>

I don't know...just a thought. Regardless, I love the general direction you're taking here.

--Mike

Patrick Klug
Patrick Klug
13 Jul 2010

Having worked with WPF extensively (by writing NovaMind 5 (www.novamind.com) completely in WPF) and seeing how the Mac version is developed (using Cocoa and Core Graphics) there are a couple of things that I think a 'perfect' UI framework should have:

  • Easy deployment/usage.
  • Excellent Typography
    • Subpixel rendering but also crisp looking text in any size.
    • Access to Glyphs, Kerning, Baseline Offsets, better Font meta data.
    • Abstraction and 'general support' for Bold, Italic and Subscript, Superscript.
  • Support for friggin images! (this is one thing I am soooo jealous of: On the Mac you just drop a vector image (or even PDF) in your user interface and it works and scales well - in WPF you have to convert everything to WPF and it just doesn't work well - also you can't support vector images dynamically)
  • Scalable and works well with higher DPI. (WPF is okay)
  • Easy to use Globalization. (WPF doesn't do a good job at this - For NovaMind we implemented our own. In XAML we use a markup extension {nm:Localize Text goes here} and in code behind a extension method "Text to localize".Localize("optional comments"). We have tools that scan our files and give us an updated and correct idea of what needs to be localized.
  • Support for Left-To-Right and Right-To-Left languages. (WPF does a good job here)
  • Support for Drag and Drop. (Trying to do good drag and drop with visual feedback, responsiveness and cross-application support is a nightmare on Windows)
  • Support for gestures (example: zoom, pan, 'flick')
  • Support for touch and ink (to be hype compatible)
  • Support for 3D
  • Access to the visual tree (paths, polygons, points etc.)
  • A solid (mouse and keyboard) focus system. (That one is extremely painful in WPF)
  • A solid popup system (also extremely painful in WPF)
  • Solid mouse and keyboard events.
  • Animations
    • Easy to use animations. (Way too complex in WPF without the help of some extensions such as http://artefactanimator.codeplex.com/)
    • Easy to use transitions. Like the general idea of a visual state manager but I think it is still painful to setup good transitions in WPF. I like the flexibility of Quartz composer on the Mac. Some of the transitions on the iPhone are also excellent - I am not sure how they are done.
  • Hardware accelerated effects.
    • Maybe even a way to write shader effects in a high level language. (see http://bling.codeplex.com/ for a WPF example of that)
  • Rockstar performance (WPF breaks down when you try to use a couple of hundred (that's not much) elements with data binding, animation and (hardware accelerated) effects.
  • Quick initialization of controls. (WPF is very slow in that respect)
  • Immediate drawing mode. (As Paul already said, the perfect UI framework should let you switch between retained and immediate drawing easily).
  • Plugin/Extensibility.
  • High quality controls (all supporting virtualization):
    • Canvas (autosizing, with pan/zoom, scroll features)
    • Panel
  • A bunch of high quality high level controls (this might seem like too much to ask but I think it is essential for any good framework - most of these controls seem simple enough but the devil is in the detail and currently too many products have to roll their own controls (badly) for the most basic things)
    • DateTime (pick date AND time visually - transitions). Probably also need to support Farsi and other calendar-types.
    • A color picker. (HSV, RGB and Alpha channel, color wheel, color matrix, named colors, a color grabber etc., color gallery) - If you think I am crazy for wanting this out of the box look at the Mac.
    • A Font picker.
    • A 'media library' allowing access to images, videos, documents etc. (see Mac)
    • A full blown rich text control.
    • A media player control.
    • ?A webbrowser control?
  • Common Dialogs (we should also be able to access this as a control instead of a dialog if necessary)
    • Print/Print Preview (
    • File/Folder open/save
    • A MessageBox or more generic command box. Enable multiple options, timer, don't show again feature etc.
    • Extensibility (on the Mac you can simply add your own controls to these dialogs to extend them. On Windows you can attempt some voodoo but I wouldn't recommend it)

I know that's lots and that I am mixing framework features with things that are built with the framework. The thing is that most users don't care how you build your software and as a developer of a commercial product I would recommend anyone to use 'the best supported' and native framework on your platform which ticks the most of the above boxes...

Eric
Eric
19 Jul 2010

You said:

my UI stack should allow me to mix a retained scene rendering mode (like WPF) with immediate mode rendering (GDI+).

Why would you want your immediate mode rendering not to be hardware-accelerated? What you really want (and what I want WPF 5 to include) is retained mode plus immediate mode rendering with Direct2D/DirectWrite.

Eric