Cabel.Cabel.

Apple's Next-Generation Themes

See, I was going to originally title this "Apple's Secret Theme Tool!!", in an effort to clutch your e-eyeballs dramatically. (I've gotta get this here vortal to be more sticky!) Unfortunately, a public patent filing isn't exactly "secret". And this is all about one of the most interesting Apple patent filings I've seen lately.

Well, interesting to me, anyway. My job has been heavily user interface design-tastic these days, and I'm also personally (and perhaps unhealthily) obsessed with the concept of resolution independence. So when Macsimum News posts a story about a Apple patent regarding a resolution independent user interface, my nerd-senses start tingling faster than an morning shower with Dr. Bronners. (Except without the crazy-ass text.)

First, if you're super curious, download patent filing itself, courtesy USPTO. I've made it into a nice PDF for you:


Apple Themes Patent.pdf (2 MB)

Resawhosa Indewhatnow?


Scaling in 10.4 is chunky. Not for long?
Time for a quick lesson in total deep nerd town. Oh, sweet resolution independence: you're almost upon us, and you go something like this: the pixel density of LCD displays is getting higher, while the physical screen sizes are largely the same. That means that, in the future, you'll probably have a good old 23" Cinema Display on your desk, but it'll pack a whole lot more pixels than it does today.

Now, if you were to use today's Mac OS X on that display, everything would be really tiny, because a button can only be drawn at one, fixed size. More pixels, with fixed (72 DPI) controls, equals tiny town, i.e. Aqua Villechaize.

A long term solution is needed, one that all OS developers are looking at: a user interface that can scale at will. A resolution independent interface, even. Have an Apple High-DPI MegaCinema Display? No problem. Just scale the user interface to twice its regular size. Every single button, title bar, piece of text and scroll widget draws itself at a new size. The result is silky smooth — beautiful buttons, beautifully clickable, and beautifully high-resolution. That's the idea, anyway. It just needs to, you know, be programmed.

If you were to do this today in Mac OS X 10.4, the user interface would look pixelated (and you can do this using Quartz Debug) — graphics are just doubled to twice their size. That's because the Mac OS X UI is still rooted in Mac OS 8 days, defined by a series of resource files (!) and bitmaps. There's just no space in the crusty old format for "bigger sizes", let alone "flexible sizes".

Apple's Fascinating Plan

Now, if you were to ask me how to solve this problem, I would have said: "just create a bunch of graphics for each of the different DPI settings! Done!". So you have your 72 DPI button, and your 288 DPI button, etc. Multiple copies of everything. (In fact, this is largely what developers will have to do, in many areas.) And this is why I'm not an Apple rocket doctor.

Here's the deal with this patent: Apple's approach seems is far more interesting, and ultimately more amazing. In essence, they're creating a user interface that's completely programmatically defined. I.e., not an exported series of graphic resources, but a series of instructions that define how the graphics should be drawn, from the ground up, just like you might create a graphic in Photoshop, all the way from "Create New Document" to "Add Drop Shadow". That means that talented Apple artists can sit down with a special internal tool — not Illustrator, not Photoshop — and craft the user interface of their dreams. And when it's time to scale it, meanmo! It just, theoretically, works, because each element is made of a series of layered, defined steps. New, higher resolution versions are instantly rendered based on the original (XML?) specification. Essentially, Apple has invented a scaleable hybrid vector/bitmap user interface that is created and edited by their own full-featured graphics tool. Phew! This is, as far as I know, a total first in the industry.

And did I mention this patent names Mark Zimmer as its inventor? You know, the guy who freakin' created Painter? Yow.

The Theme Tool

Far more interesting to me than the words in the patent were the pictures in the patent. Of course, we all know by now that Steve Jobs isn't too keen on allowing Mac OS X to switch themes, for a lot of reasons I can understand.

That makes these pictures all the more interesting — these crappy screenshots of Apple's internal theme creation tool, presumably being used right now to create the next great UI. Put simply, this may be the first and last time you'll ever see this thing.

Check it out — full-featured is an understatement. (Download the .pdf for more.)



So, this is some intense technology.

And if I were a betting man, I'd say we're going to see the fruits of this labor very, very shortly. It looks like Apple now has the tools to create a user interface for the future — all that's left is to see what the future actually looks like.

Bonus Macworld Note

If you're going to be at Macworld Expo on Tuesday at 4:30, stop by booth #408 in the South Hall, as famous Macintosh bundtcake — sorry, pundit — John Gruber of Daring Fireball has invited yours truly to do some kind of thing. It's at the Macworld Magazine booth. Are we talking? Debating? Jello wrestling? Probably talking. Come say hello!

42 Comments:

This is extremely interesting for us interface designers. Thanks for the pictures too!
Anonymous Fernando Lins 1/08/2007 8:46 AM  
This is really interesting, and seems to add major weight to the programmatic interfaces side of the debate. There was a lot of disagreement after I talked about resolution independence a couple months ago, with the bottom-line being that many designers don't believe you can sufficiently fine-tune programatically.

I wonder if Apple has engineered a solution sufficient to beat this limitation. From the PDF:

"...while other attribute values are specified for two or more resolutions. In general, those attributes associated with the physical location of geometry of an object may be associated with a plurality of values - one value for each specified resolution."
Anonymous Daniel Jalkut 1/08/2007 8:50 AM  
Oh, Dr. Bronner's rules.

Also, jello wrestling FTW! Post-wrestling clean-up with Dr. Bronner's. It's a win-win!b
Blogger Chris Foresman 1/08/2007 9:12 AM  
That's because the Mac OS X UI is still rooted in Mac OS 8 days, defined by a series of resource files

I think that's at least a tiny bit misleading. The average person might think we're all still using ResEdit and resource forks. In fact, it's possible for applications to draw correctly at higher resolutions today (at least in the places not defined by the OS), but it could be made a lot easier.

This tool looks really interesting, though.
Anonymous Scott Stevenson 1/08/2007 9:13 AM  
This is really interesting documents for me. Thanks Cabel!! :)
Blogger Kenichi Yoshida 1/08/2007 9:22 AM  
Either vector-based or high-res raster based would be slick in the OS. The screen shots are indeed interesting. Maybe bundled with Xcode? Who knows... maybe tomorrow we'll find out more.
Anonymous pedro 1/08/2007 9:28 AM  
This is a little troubling. Apple uses the actual UI sparingly these days, instead packing their applications with custom resources. Now, I don't necessarily think that is a bad thing. I am sure Apple will continue to do this, using this internal tool to build these resources. But, if the tool is not released to the public, the rest of us are left with either using bitmaps or the default buttons. As "Make Your App Look Nice" is the new HIG, that's going to be a problem for many of us. I guess developers will just have to create very high resolution resources and scale them appropriately?
Anonymous Steven Canfield 1/08/2007 10:35 AM  
"Essentially, Apple has invented a scaleable hybrid vector/bitmap user interface that is created and edited by their own full-featured graphics tool."

Sounds very similar to Flash or SVG.... Actually, I wonder if this tool does end up outputting SVG since that's open source.
Anonymous Anonymous 1/08/2007 10:50 AM  
Wow, this makes Apple's effort on Leopard resolution independence look even more monumental. I'm smelling high DPI screens, in 2007 if not tomorrow. Beautiful, and differentiating.

And for developers looking at supporting resolution independence and thinking, should I care? the answer is clearly yes.
Anonymous Duncan Wilcox 1/08/2007 12:20 PM  
"Aqua Villechaize" -- best visual ever!

//k
Blogger macfixer 1/08/2007 12:50 PM  
man... that interface tool kind of reminds me of my days working in Maya... yes, full featured is a major understatement. This is very very intriguing. I want to get my hands on it...
Anonymous gb 1/08/2007 2:43 PM  
I suspect this extreme level of customization will not be fully implemented. It is fairly common for patents to cover an extremely wide spectrum of technologies, as the inventors never quite know what part of the technology will be the most useful bit a few years down the line. They want to cover all the basics, plus as many aligned technologies as possible. So don't necessarily look for all these features in the next release of Interface Builder.
Anonymous Anonymous 1/08/2007 7:03 PM  
So, it's basically replacing all of the bitmap resources with SVG, but with a nifty developer tool to do it with?

While I'm not particularly impressed with scalable images (finally) being applied to the entire desktop environment, it is nice to see a major vendor finally realize that 72 DPI isn't for everyone.

Having been running at 120-something DPI for the last few years (14-15" screen at 1400x1050), it annoys me to no end how OS and monitor vendors don't seem to care that DPI is an absolute, object unit of measurement in Euclidean space.
Anonymous yangman 1/08/2007 7:10 PM  
So, Apple have created a tool that 'records' your actions in something akin to Photoshop/Illustrator, and at will re-do every step many times over to produce the ever elusive, and ever increasing, resolutions of OS X controls.

It could be used for other things, too, of course...

I'm just gonna throw this out there, because we all know that Steve has a flair for the dramatic.

MacWorld might be very consumer-oriented, but if Steve has anything - anything at all - to say about resolution independence tomorrow... Couldn't that explain the extreme proliferation of Space Odyssey -esque Apple-logo sinage?

I could very well imagine Steve saying something along the lines of "Just to show you how incredibly great our res-independence effort has been, know that every single Apple/Obelix sign you've seen, from website to 100 foot posters, were created using a single resource file."

I know, it won't happen. But wouldn't it be cool if it did? What? No show of hands?
Anonymous Andreas 1/08/2007 7:27 PM  
I'm not so sure this is going to be the last time we see it. Looks like something that could be part of the next version of interface builder.
Blogger Andrew 1/08/2007 8:52 PM  
That's so awesome! I actually had been thinking about this sort of thing before, but it turns out I was completely over thinking it. I was thinking of something where when you zoomed into an old image, a program would fill in the blanks, when it's really much simpler without the image file at all. Apple is awesome.
Blogger Jason 1/08/2007 11:02 PM  
Apple has invented a scaleable hybrid vector/bitmap user interface that is created and edited by their own full-featured graphics tool. Phew! This is, as far as I know, a total first in the industry.

Apple may have developed a way to use it as the main UI for an OS, but the credit goes to Microsoft (shock, horror) for implementing this long before Apple. Take a look at XAML and WPF (Windows Presentation Foundation) sometime, as well as the "Expression" design tool for XAML.
Anonymous David 1/09/2007 2:55 AM  
This is interesting, but I don't see what's so special about this tool. Aren't all graphics formats "programatically defining?" Obviously most existing vector formats wouldn't be idea for UI design, but that doesn't mean Apple couldn't use Illustrator or any other vector tool - they'd end up with "draw a circle and add a drop shadow" either way.

and to add on to what peter said: http://en.wikipedia.org/wiki/Microsoft_Expression_Blend has a bit of info on the topic.
Anonymous Anonymous 1/09/2007 5:59 AM  
Just want to go on record somewhere with this ... a prediction I haven't seen is that Apple might announce some sort of partnership with Adobe to incorporate the new Apollo Flash/PDF/rich media platform into either OS X or QuickTime. Flash-based widgets on the dashboard, ActionScript talking to AppleScript, playing FLVs in QuickTime without Perian ... even vaguely possible, do you think?
Anonymous conkyfilms 1/09/2007 5:59 AM  
With this added twist, and with modern browsers and their dpi-switching preferences, I'm guessing this is going to make web design a lot more complicated.

Or at the very least will force the issue of using only relative units...
Blogger Michael 1/09/2007 6:43 AM  
This should make Panic software look even better.

And Richard Stallman will think you *really* deserve punishment for that. Seriously - I was just reading one of his articles that someone had linked at Slashdot and he says:

"people who develop proprietary software that's attractive deserve a punishment."

http://fsfeurope.org/documents/rms-fs-2006-03-09.en.html

Well, Panic's software is some of the most attractive (in a number of senses) software out there. So Panic would obviously deserve a *lot* of punishment in Stallman's eyes. :-)
Anonymous Anonymous 1/09/2007 12:14 PM  
This is great. All very interesting stuff.

At first glance I don't see much difference between vector (SVG/PDF etc) and the procedural rendering based here. The one advantage might be that this hooks directly into Quartz, whereas SVG etc aren't based on those kinds of GPU-based filters.

Fig 4 does show some tweaking for the various UI sizes, which is crucial, even with 160+ DPI displays and fairly small icons.

The "recipes" look like a great way to design UI though. I'm all for it!
Blogger Marc Edwards 1/09/2007 5:05 PM  
Wow, a "user interface that's completely programmatically defined. I.e., not an exported series of graphic resources, but a series of instructions that define how the graphics should be drawn, from the ground up, just like you might create a graphic in Photoshop, all the way from "Create New Document" to "Add Drop Shadow"."? That could be, like, revolutionary. I mean, imagine if they were to choose postscript as the underlying language, they could create a kindof Display Postscript system where you specify all your drawing in real time in the postscript language and it gets rendered to whatever device is desired! Wow, and then not only could it draw to any screen, but exactly the same code could draw to a printer, a fax machine, whatever!

Could I borrow some trash from someone? I need to fuel up my Mr Fusion so I can get back to 1985 to file a patent on this.
Blogger leeg 1/10/2007 1:25 AM  
I wonder what the performance impact will be. Procedural rendering will take up a lot more resources than blitting ui elements to the framebuffer.
Blogger Nick 1/10/2007 8:07 AM  
"I wonder what the performance impact will be. Procedural rendering will take up a lot more resources than blitting ui elements to the framebuffer."

Since people generally don't change their screen resolution all the time, I bet the system would use bitmap caching of some sort -- i.e. create the proper-resolution bitmaps once and not re-render them every time (or has somebody already said this?).
Anonymous Stephen Rider 1/10/2007 12:11 PM  
Like a lot of people mentioned, there is nothing really new in the idea of defining a graphical element as 'series of layered, defined steps'. It's basically the idea behind any vector graphic format.

The idea of doing this for user interface is not new either (XAML being one example of this...).

What's supposedly new according to the patent is the method associating numerical values (essentially placement information) to a 'material map' which as I understand, is just a data structure containing the controls of a window... and a method to use this information to layout the said window.

Personally, I'm happy to see that they are actually working on something for resolution independence but I wouldn't call anything they are doing earth shattering.
Anonymous alexr.r. 1/10/2007 3:56 PM  
I'm ready for the iphone blog now Cabel. Plz.
Blogger Robert 1/11/2007 11:23 AM  
This isn't really new. The GTK2 toolkit used on Linux has had full support for hardware accelerated SVG rendering for a while now. It's common and expected that every piece of artwork you see, from icons in the toolbar to the about box, is now defined programmatically using SVG. Likewise, the most popular theme right now is defined entirely using vectors.

When I first started using OS X I was amazed you cannot simply wind up the screen font size system-wide like you can on Linux. I've been told this is because UIs on OS X are not defined using containment based layout, which I find hard to believe but would indeed explain this. Apple aren't really doing much new here, they're just catching up with what was state of the art 2 years ago on Linux.
Blogger Mike Hearn 1/13/2007 4:36 AM  
Matt Casamassini from IGN has stolen one of your pictures and used it to display Halo DS (psyche)
Anonymous Anonymous 1/13/2007 10:36 AM  
I meant psyche as in Halo DS, not him taking your picture.
Anonymous Anonymous 1/13/2007 11:57 AM  
I read last year that Haiku (a BeOS clone) uses a vector icon format. Just thought I'd mention it in the course of all this vector talk.

http://haiku-os.org/node/280
Anonymous Anonymous 1/13/2007 6:03 PM  
Cabel, I know that vector formats are the way to go but I'm still using pixel maps (though at 288 dpi) for Sound Studio. It's high-DPI ready, sort of. Bring it on, Leopard.
Blogger Lucius 1/18/2007 7:37 PM  
I may be getting the wrong end of the stick here, but isn't this: http://en.wikipedia.org/wiki/Image:Aqua.png a better version of your pictures?
Anonymous RB 1/30/2007 5:13 AM  
You've got the wrong end of the stick.
Anonymous Anonymous 2/05/2007 3:13 AM  
I was wondering: will this technique will be able to make a high-resolution screenmagnifier on a Mac (for partially sighted people?)
Anonymous pumbaa 2/17/2007 3:46 AM  
That's really cool, but wouldn't independent developers like, say, Panic, have to have some way of making their custom UI elements, like, say, Transmit's disconnect button, also resolution-independent? Wouldn't Apple therefore be forced to license and/or release for freesies their softwares for everyone? That would be niiice.
Blogger Flip 3/08/2007 5:30 PM  
Tiger has a screen magnifier built in... just go to "Universal Access" under the system preferences. This will just make everything look better (non-pixellated) when it is magnified.
Anonymous Anonymous 9/15/2007 4:25 AM  
we don't think it is reasonable to spend hundreds thousands dollars to buy a decorating

watch. you can use those money to invest in other industry which will return you good

profit.
watches ebayare made by the rating 1:1 according

to the original watches, and you can't distinguish the original and the fake watches when

you look at the surface of the watches.
batteries
Anonymous Anonymous 9/08/2008 7:00 PM  
I agree that Microsoft did it a long time ago. I'm a .net programmer and all my applications' interfaces are 100% vectorial and resolution-independent. Moreover, xaml (the xml file format used to describe those interfaces) is an open format and can be used independently from vb.net/c#.

I should add that in the next version of Windows (after 7), all the UI might be in xaml as an announcement stated that icons would be.
Blogger quentez 7/15/2009 7:21 AM  
Anonymous Anonymous 4/14/2010 2:03 AM  
Blogger deLi 5/03/2010 4:39 AM  
biber hapı ile lida satışı yaparız abi :p film izle
Blogger Adil 5/13/2010 5:14 AM  

Post a Comment

 
 
 
   

   
       
 
 
 
   
Name:Cabel Maxfield Sasser
Job:Co-Founder, Panic Inc.
Location:Portland, OR
Email: