Linux DevCenter    
 Published on Linux DevCenter (http://www.linuxdevcenter.com/)
 See this if you're having trouble printing code examples


Through Project Looking Glass with Hideya Kawahara

by John Littler
12/08/2005

3D has practically taken over video gaming. Lifelike, if not very pleasant, worlds exist aplenty--worlds that most users find easily navigable without any training whatsoever. Is the world of spreadsheets, word processors, and the like just unsuitable for 3D? Is it a case of "If it ain't broke, don't fix it"? Or is it that we've lacked imagination? John Littler recently talked to Hideya Kawahara about an open source 3D desktop project that he started and that Sun subsequently took under its wing.

John Littler: Let's maybe get a little history on Project Looking Glass project. How long has it been going and what was the original mission statement?

Hideya Kawahara: It has been open sourced for about one and a half years. We are grateful for the keen and growing numbers of community members we have around the technology.

The very first public demo of the technology using a proof-of-concept version was shown in LinuxWorld San Francisco back in August 2003. Time [flies].

Since then, we have spent a bit less than one year to reimplement the technology into a better code base for open source development and do further exploration of the 3D opportunity. The development is done by Sun's Advanced Development Group.

From the beginning, our mission statement has always been the following: "explore bringing a richer user experience to the desktop and applications via 3D windowing and visualization capabilities together with the open source community." So, I feel very happy whenever I see a new 3D application developed by community members on top of Project Looking Glass.

JL: Did that process mostly involve cleaning up and commentary, or was it more to do with the arrangement of the tree and like things?

Hideya: It was more like a complete rewrite. The proof-of-concept version used a Java to OpenGL binding for 3D rendering. We switched the platform from it to Java 3D, since Java 3D provides scalability and higher-level abstraction of 3D features, which helps development.

JL: Did it start out as an open source project?

Hideya: Actually, it started as a personal project, with open sourcing in mind. I spent almost one year developing the initial proof-of-concept demo. The demo generated some excitements among Sun engineers, and it rippled up to the management. Then, they proposed to evolve the technology as a company project.

Since Sun was supportive for the idea of open-sourcing the technology and creating a small team with excellent engineers with profound knowledge in 3D and window system technologies, I was happy to accept the offer. This was the beginning of Project Looking Glass.

And, as promised, Sun supported us to open-source the technology.

JL: 3D desktops seem like a cool idea just 'cuz, but most people will want functionality. What do you think are the functional uses--both now and potentially?

Hideya: In general, 3D provides us more freedom of visual expression than 2D. So, if we carefully use the additional opportunities brought by the 3D-ness in a meaningful way, I believe it has great potential. So, the issue really is to find out things that make sense.

In the project, we found some of the ideas that were received well by the users, like parking a window with angles in a 3D space, flipping a window to add notes or configure the window, a bookshelf view of all the windows, and panning and zooming out of a horizontally long virtual desktop space.

And we even started seeing similar 3D effects in major desktops, like flipping a widget in Apple Mac OS X's Dashboard, and a bookshelf-like view of windows when switching the window focus in Microsoft Windows Vista.

I think we've just begun to find out the usefulness of 3D UI features, and there are more undiscovered opportunities there.

JL: People have had trouble with this in the past, haven't they? We had VRML for the Web, for example, but it died partly because users needed powerful machines to cope with anything halfway decent, and the implementation wasn't too good either. I remember there were all sorts of problems with scaling pixmaps and the like, and the code was horrible and meant to be machine written. Did the Looking Glass project look at VRML for any ideas?

Hideya: We have many things we can learn from past approaches, including VRML, but we are doing it very carefully.

Most of research around VRML is virtual reality centric, not surprisingly. And I know many desktop environments proposed based on virtual reality. However, I didn't think it is a right approach as a first step toward a 3D desktop.

We think a desktop is a desktop, regardless of 2D or 3D. We avoided to bring the desktop into the 3D virtual reality world that could just confuse the user. Rather, we tried to bring useful 3D effects into the desktop space piece by piece. I think this focus is one of the reasons for the positive feedback we have received about our project.

On the other hand, as a 3D platform, Project Looking Glass is capable enough for us to create a desktop that provides a fully immersive virtual reality experience. This extensibility is one of strengths of this platform. So, in the future, we might choose to explore this direction too. But not right now.

JL: So there could be a "mod" community in time to come. That could be fun.

Hideya: Yes, definitely. This is one of the great things of being an open source project. I'm looking forward to seeing such communities around this technology.

JL: Performance probably isn't such an issue now that very powerful machines are in general circulation. I was impressed with the speed of the Looking Glass demos I've seen, especially when they have a Java heritage. Looking Glass can't possibly be running on top of an ordinary Java VM ... can it?! What sort of performance tweaks have you made?

Hideya: This is one of questions I'm often asked. Unfortunately the answer is a bit boring--we haven't done anything special. It is running on an ordinary Java VM that you can download from Sun. There are many people who think execution of a Java program is very slow, but it is not the case anymore. Thanks to the intensive exercise on the server side these days, Java VM has evolved a lot. Project Looking Glass is just leveraging that fact.

Linux Desktop Hacks

Related Reading

Linux Desktop Hacks
Tips & Tools for Customizing and Optimizing your OS
By Nicholas Petreley, Jono Bacon

JL: I understand that one thing you've done is to compile it so that it runs at least partially on the video card. Could you tell us about that?

Hideya: Actually, we don't perform any tricks like a special compilation in Project Looking Glass. The project uses Java 3D underneath for its rendering. Java 3D is a sophisticated 3D rendering engine designed to leverage the underlying 3D graphics capabilities. We are happy to see that Java 3D is doing a great job.

JL: Right. So does Java 3D do any card sensing or anything like that?

Hideya: Java 3D is implemented on top of OpenGL. As you know, most of today's graphics cards support accelerated 3D rendering via OpenGL. So, Java 3D simply benefits from the fact. In addition, Java 3D itself performs optimizations for rendering operations.

JL: Could you take us through some of the Looking Glass apps and tell us what they're about?

Hideya: The Project Looking Glass desktop itself can be considered as an application. One of the focuses here is to add value to the desktop environment and existing 2D applications by employing the additional Z dimension and 3D animation effects.

For example, now because a window is actually a 3D object, we can position it in the 3D space as we like. How can we leverage this to improve the desktop's usability? One of our attempts is to allow the user to slant it and park a window on a side. This is useful when the user is not really using the window but wants to keep an eye on it. This feature helps the user to utilize the limited screen real estate.

Thumbnail; click for full-size image
Figure 1. Slanting windows with Looking Glass--click for full-size image

We can also slant all the windows at one time so that the user can see all of the opened windows. We call it a bookshelf view.

By the way, have you ever seen the back side of a window? Since a window is now a 3D object, we can utilize its back side. We have implemented a prototype mechanism to put a sticky note on the back side of an application window. Also we are prototyping application configuration using the back side.

Thumbnail; click for full-size image
Figure 2. The front side of a terminal window--click for full-size image

Thumbnail; click for full-size image
Figure 3. The back side of a terminal window--click for full-size image

When you used up the current window, wouldn't it be nice to pan your desktop view toward left or right? We implemented such a feature too. This is similar to the virtual desktop switching, but by associating the virtual desktop with the panoramic background and using the panning motion, I think it can give better visual cues to the user. The panoramic desktop also supports the overview mode.

Thumbnail; click for full-size image
Figure 4. A panoramic view of multiple desktops--click for full-size image

Transparency effect is not a very 3D feature, but it can be achieved easily by leveraging a 3D graphics card. Imagine when we open up lots of windows; wouldn't it be nice if we could easily check how a window looks even if it is obscured by other windows? This desktop allows the user to do so by moving the mouse cursor onto the window thumbnail. The windows other than the one the user chose by selecting thumbnail become very close to transparent. This allows the user to see the selected window through the other windows. The user can move the mouse across the thumbnails next by next. This helps the user to find out the window of interest quickly.

Thumbnail; click for full-size image
Figure 5. Multiple transparent windows--click for full-size image

These are some of the examples that show what we are trying to achieve--improve the usability by leveraging the 3D graphics card.

Also, there are 3D applications created by the community members. All of them are still experimental, but here, let's check out a few of them.

The first one is an image viewer named Zoetrope.

Thumbnail; click for full-size image
Figure 6. The Zoetrope image viewer--click for full-size image

On the left side, it has thumbnails of images arranged in 3D space. It looks like a Ferris wheel. The mouse wheel motion is linked with this thumbnail wheel. This makes user navigation very convenient.

The second example is the background manager. This application allows the user to switch the desktop background.

Thumbnail; click for full-size image
Figure 7. The background manager--click for full-size image

As you can see, we have close to 50 background images. The background manager positions the thumbnails in the 3D space. It tries not to use up too much space but still provide good clues about each background to the user.

[Figure 8 shows] a file manager that [uses] the Z depth.

Thumbnail; click for full-size image
Figure 8. A file manager using Z-depth--click for full-size image

As the user opens subdirectories, the file manager grows toward the right. When it hits the right side of the screen, it starts slanting the least-used directory views, instead of dismissing them. The slanted view still gives a good visual cue to the user about the contents of the directory.

The last one [is] a challenging one. When we think about a scheduler application, we usually think about a calendar motif. Instead, the developers of this application wondered how they can make the scheduling process completely different and more fun. They then decided to use the solar system as the motif.

Thumbnail; click for full-size image
Figure 9. A solar system scheduling scheme--click for full-size image

Each orbit represents a kind of schedule, and a planet represents a schedule item. The more important the item is, the larger the planet becomes. It demonstrates an out-of-box thinking toward 3D applications.

Again, all of them are still under development, but I'm very happy to see that Project Looking Glass is serving as a platform of their exploration toward 3D applications.

JL: What sort of general problems have you had as the project has gone along?

Hideya: Initially we got lots of negative feedback like "3D would never work for desktop UI" or "it is just full of eye candies." I think it took more than one year until we start hearing opinions like "this project is not just an eye candy, but may have something real".

Also, the area we've been exploring was quite new. There were lots of trial-and-error situations, including the API design and the internal architecture. We find more issues as we go.

Although we have lots of keen supporters around the technology, development of this kind of system software requires a huge effort. We have a very long way to go. We are always looking for someone who shares the vision and is willing to contribute.

JL: What do you have coming up, and what further possibilities do you see for the functionality of the thing?

Hideya: One idea we've been discussing recently is Project Looking Glass as a 3D UI toolkit, in addition to being as a desktop system.

Many of community members suggested that the 3D API project significantly simplified writing 3D applications that involve user interaction. Some say that they could write a 3D application without a major trouble even without 3D knowledge. We project owners thought this is a great suggestion, and one of us has started working on this aspect.

In this scenario, the toolkit can be considered as a utility library on top of Java 3D. One significance is that such Java applications written using this 3D toolkit, together with Java 3D, will run on top of most of the desktop environments, Linux and Windows, without running Project Looking Glass desktop. We think this help broaden our audience and gain more interest from the community.

JL: Will this be able to link to native C or C++ apps? I'm just wondering about applications that need real-time performance--such as audio apps and the like.

Hideya: A typical way to deal with such a situation is to use a Java wrapper over native libraries. This can be done by using Java Native Interface (JNI). As for audio and video, Java has Java Media Framework (JMF) that provides Java API to control media resources.

3D Desktop-Related Projects

John Littler is chief gopher for Mstation.org.


Return to the Linux DevCenter.

Copyright © 2009 O'Reilly Media, Inc.