diff options
author | Matthieu Herrb <matthieu@cvs.openbsd.org> | 2010-12-05 15:36:12 +0000 |
---|---|---|
committer | Matthieu Herrb <matthieu@cvs.openbsd.org> | 2010-12-05 15:36:12 +0000 |
commit | a90ee792d96752ad1b71b9ada922fa6afe478c99 (patch) | |
tree | 6143b00f7646951d23dfe2a4fe2992ca40b77069 /xserver/hw/dmx/doc/scaled.xml | |
parent | bc97d4ecc0aa9e1b823565b07282f848700bd11a (diff) |
Upgrade to xorg-server 1.9.2.
Tested by ajacoutot@, krw@, shadchin@ and jasper@ on various configurations
including multihead with both zaphod and xrandr.
Diffstat (limited to 'xserver/hw/dmx/doc/scaled.xml')
-rw-r--r-- | xserver/hw/dmx/doc/scaled.xml | 725 |
1 files changed, 725 insertions, 0 deletions
diff --git a/xserver/hw/dmx/doc/scaled.xml b/xserver/hw/dmx/doc/scaled.xml new file mode 100644 index 000000000..48c83e076 --- /dev/null +++ b/xserver/hw/dmx/doc/scaled.xml @@ -0,0 +1,725 @@ +<?xml version="1.0" encoding="ISO-8859-1"?> +<!DOCTYPE article PUBLIC "-//OASIS//DTD DocBook XML V4.3//EN" + "http://www.oasis-open.org/docbook/xml/4.3/docbookx.dtd" [ +]> + <article> + + <articleinfo> + <!-- Title information --> + <title>Scaled Window Support in DMX</title> + <authorgroup> + <author><firstname>Kevin E.</firstname><surname>Martin</surname></author> + <author><firstname>Rickard E.</firstname><surname>Faith</surname></author> + </authorgroup> + <pubdate>15 October 2003 (created 19 September 2003)</pubdate> + <abstract> + <para> + This document investigates the possibility of adding scaled window + support to the DMX X server, thereby allowing a window or some + selected part of the logical DMX area to be displayed using a + scaling factor. For example, this might allow the contents of a + window to be magnified for easier viewing. In particular, scaling + for the VNC client is explored. <emphasis remap="it">Copyright 2003 + by Red Hat, Inc., Raleigh, North Carolina</emphasis> + </para> + </abstract> + </articleinfo> + + <!-- Begin the document --> + <sect1><title>Introduction</title> + <sect2><title>DMX</title> + <para> + The DMX X server (Xdmx) is a proxy server that is designed + to allow X servers on multiple machines to be combined into + a single multi-headed X server. Combined with Xinerama, + these heads can appear as a single very high-resolution + screen. Typical applications include the creation of a + video wall with 16 1280x1024 displays arranged in a + rectangle, for a total resolution of of 5120x4096. + </para> + </sect2> + <sect2><title>Problem Statement</title> + <para> + Applications displayed on a physically large video wall that + provides high pixel-resolution may be difficult to see, + especially if the application is designed for use on a + typical desktop computer with a relatively small display + located close to the human operator. The goal of this paper + is to describe and discuss solutions to this problem. + </para> + <para> + The original driving problem for this work is to provide + scaling for the <command>vncviewer</command> application when + displayed using DMX (VNC scaling is currently available only + with the Windows client, and there is no plan to extend that + capability to other clients). While this specific problem + will be addressed in this paper, the general solution space + will also be explored, since this may lead to a good + solution not only for <command>vncviewer</command> but also for + other applications. + </para> + </sect2> + <sect2><title>Task</title> + <para> + For reference, here is the original description of the task + this paper addresses: + <itemizedlist> + <listitem><para>Scaled window support (for VNC) + <itemizedlist> + <listitem><para> + Investigate possibility of implementing a "scaled + window" extension: + <itemizedlist> + <listitem><para> + Add XCreateScaledWindow call that could be used + in place of XCreateWindow + </para></listitem> + <listitem><para> + All primitives drawn to scaled window would be + scaled by appropriate (integral?) scaling factor + </para></listitem> + </itemizedlist> + </para></listitem> + <listitem><para> + Alternate approach: special case VNC support + </para></listitem> + </itemizedlist> + </para></listitem> + </itemizedlist> + </para> + </sect2> + </sect1> + + <sect1><title>Previous Work</title> + <para> + This section reviews relevant previous work. + </para> + <sect2><title>VNC</title> + <sect3><title>Scaling under VNC</title> + <para> + When using the <command>vncviewer</command> program for Windows, it + is possible to specify a scaling factor (as numerator and + denominator). When scaling is in effect, the viewer + software uses StretchBlt (instead of BitBlt) to display + the pixels for the user. When this call is made, the + viewer already has received all of the pixel information + (at full unscaled resolution). + </para> + <para> + The scaling in VNC is primitive. It does not conserve + bandwidth, it does not treat textual information + differently (i.e., by using a suitably scaled font), and + it does not provide any anti-aliasing other than that + provided by the underlying (Windows-only) system library. + </para> + </sect3> + </sect2> + <sect2><title>The X Video Extension</title> + <para> + The X Video Extension is a widely-available extension to the + X11 protocol that provides support for streaming video. + Integral to this support is the ability to arbitrarily scale + the output. In version 2.2 of the X Video specification, + support for scaled still images was provided, using both + shared memory and traditional transport. The API for this + support uses calls that are quite similar to XCreateWindow, + XPutImage, and XShmPutImage. Currently, most of the drivers + implemented in XFree86 only support data in various YUV + formats. However, several modern video adaptors support RGB + as well. + </para> + <para> + Note, though, that the target output for this scaling is an + overlay plane -- so X Video provides functionality that is + fundamentally different from that provided by the Windows + StrechBlt call. + </para> + </sect2> + </sect1> + + <sect1><title>Possible Solutions</title> + <para> + This section briefly discusses possible solutions, including + major advantages and disadvantages from both the + implementation and the end-user programmer standpoint. + </para> + <sect2><title>VNC-like Scaling</title> + <sect3><title>Software Scaling</title> + <para> + The <command>vncviewer</command> application could be modified to + provide software scaling. This is not a general solution, + but it does solve one of the goals of this work. + </para> + <para> + A prototype of this solution was implemented and a patch + against <filename>vnc-3.3.7-unixsrc</filename> is available in the + <filename>dmx/external</filename> directory. Because of limited time + available for this work, all of the edge cases were not + considered and the solution works well mainly for integer + scaling. + </para> + <para> + Currently, <command>vncviewer</command> writes to the X display + with XPutImage, XCopyArea, and XFillRectangle. All + instances of these calls have to be aware of scaling + and must round correctly. In the prototype solution, + rounding is incorrect and can cause artifacts. + </para> + <para> + A better solution would be to cache all updates to the + desktop image in <command>vncviewer</command> and only send the + damaged area to the X display with XPutImage. This would + allow the damaged area to be computed so that rounding + errors do not create artifacts. This method is probably + similar to what is used in the Window client. (The whole + VNC suite is being re-written in C++ and the forthcoming + version 4 has not been evaluated.) + </para> + </sect3> + <sect3><title>Scaling with the X Video Extension</title> + <para> + The scaling in the Windows <command>vncviewer</command> application + makes use of a scaled blit that is supplied by the + underlying system library. Several video cards currently + provide support for a scaled blit, and some X servers + (including XFree86) expose this capability to applications + via the XvPutImage interface of the X Video Extension. + The capability exposed by XvPutImage results in the scaled + image being drawn to an overlay plane. Most video cards + also provide support for a scaled blit into the normal + output planes, but this is not exposed via XvPutImage. + </para> + <para> + The <command>vncviewer</command> program could be modified to use + the X Video Extension to provide scaling under X11 that is + similar to the scaling currently provided under Windows. + Unfortunately, Xdmx does not currently export the X Video + Extension, so this would not provide an immediate solution + usable with DMX. + </para> + <para> + A very early-stage proof-of-concept prototype was + implemented and a preliminary patch against + <filename>vnc-3.3.7-unixsrc</filename> is available in the + <filename>dmx/external</filename> directory. This prototype was + implemented to better understand the problems that must be + solved to make this solution viable: + <itemizedlist> + <listitem><para> + As noted under the software scaling section above, + <command>vncviewer</command> writes to the X display with + several different calls. These calls write to the + normal output planes and are compatible with + XvPutImage, which writes to an overlay plane. To + eliminate artifacts caused by this problem, + <command>vncviewer</command> should be modified so that a cached + copy of the desktop is available, either as a + client-side image or a server-side off-screen pixmap, + so that XvPutImage would be the only method for + writing to the X display. + </para></listitem> + <listitem> + <para> + Although several modern graphics adaptors support + hardware scaling using an RGB format (e.g., ATI + Radeon, nVidia, etc.), XFree86 drivers typically + only implement YUV formats. YUV generally compress + the pixel information in some way. For example, two + commonly implemented formats, YUY2 and UYVY provide + intensity information for every RGB pixel, but only + provide chroma and luminance information for pairs + of horizontal pixels. Since VNC uses + pixel-resolution for communicating updates on the + wire, additional artifacts are introduced (because + there may not be enough information from the wire to + update a pair of pixels). + </para> + <para> + Further, the well-known problem with YUV encoding + is even more evident when the image is a desktop + instead of a movie. For example, consider a + 1-pixel-wide vertical window border. If the border + changes in color but not intensity (e.g., because a + window manager uses color to indicate focus), there + may or may not be a change in the YUY2 image, + depending on the algorithm used for RGB to YUV + conversion and on how the border pixel is ordered in + the pair of pixels used by the algorithm. + </para> + <para> + Many of these artifacts could be eliminated if + <command>vncviewer</command> cached a complete RGB image of + the desktop, and only did the conversion to YUV for + properly aligned areas of damage. The remaining artifacts + could be eliminated if an RGB format was used with X + Video (which may require the extension of existing + XFree86 drivers to support RGB). + </para> + </listitem> + <listitem><para> + Most modern video cards support exactly one overlay + plane that is suitable for use with X Video. + Therefore, only one application can use X Video at any + given time. This is a severe limitation in a desktop + environment. + </para></listitem> + </itemizedlist> + </para> + <sect4><title>Implementing the X Video Extension for DMX</title> + <para> + The user-level API for X Video is fairly simple, but the + underlying support required for the full specification + is large. However, since the API provides a method to + query supported capabilities, a usable subset of X + Video can be implemented that would support XvPutImage + and little else. This would require support for the + following: + <itemizedlist> + <listitem><para> + X Video Extension API calls, including the + following: + <itemizedlist> + <listitem><para>XvQueryExtension</para></listitem> + <listitem><para>XvQueryAdaptors</para></listitem> + <listitem><para>XvQueryPortAttributes</para></listitem> + <listitem><para>XvFreeAdaptorInfo</para></listitem> + <listitem><para>XvListImageFormats</para></listitem> + <listitem><para>XvGrabPort</para></listitem> + <listitem><para>XvCreateImage</para></listitem> + <listitem><para>XvPutImage</para></listitem> + <listitem><para>XvShmCreateImage</para></listitem> + <listitem><para>XvShmPutImage</para></listitem> + </itemizedlist> + </para></listitem> + <listitem><para> + Support for querying back-end X Video Extension + capabilities. + </para></listitem> + <listitem><para> + Support for sending the image to the back-ends. + Because X Video requires sending full images, there + may be a trade-off between bandwidth limitations and + additional complexity to divide the image up such + that is scales properly. + </para></listitem> + <listitem><para> + Possible support for a software fall-back. For + example, if all of the back-ends do not support the X + Video Extension, software scaling can be implemented + such that the image is sent to the back-end with + XPutImage. This pathway would have poor + performance. + </para></listitem> + </itemizedlist> + </para> + </sect4> + <sect4><title>Supporting RGB formats for the X Video Extension</title> + <para> + Assuming an XFree86 driver already supports the X Video + Extension, and assuming the target hardware supports an + RGB format, then adding support for that format is + relatively simple and straightforward. + </para> + </sect4> + </sect3> + <sect3><title>Scaling with an XPutImageScaled Extension</title> + <para> + Instead of (or in addition to) implementing the X Video + Extension in DMX, one obvious solution would be to + implement a new extension that provides access to + hardware-assisted scaled blits, similar to the StretchBlt + call available under Windows. This call would scale RGB + images and would not use the overlay plane (unlike the X + Video Extension). + </para> + <para> + This approach has many of the same advantages and + disadvantages as the XCopyAreaScaled Extension, discussed + in the next section. Discussion of XPutImageScaled is + deferred in favor of XCopyAreaScaled for the following + reasons: + <itemizedlist> + <listitem><para> + XPutImageScaled can be emulated with XCopyAreaScaled + by first using XPutImage to copy the image to an + off-screen pixmap, and then calling XCopyAreaScaled + between that off-screen pixmap and the target + drawable. + </para></listitem> + <listitem><para> + Since XCopyAreaScaled would copy between two areas of + on-screen or off-screen memory, it has additional uses + and can be viewed as efficiently providing a superset + of XPutImageScaled functionality. + </para></listitem> + </itemizedlist> + </para> + </sect3> + <sect3><title>Scaling with an XCopyAreaScaled Extension</title> + <para> + As noted in the previous section, because XCopyAreaScaled + provides a superset of the functionality provided by + XPutImageScaled, we will consider this extension instead. + </para> + <para> + First, XCopyAreaScaled would provide for RGB scaling + between pixmaps (i.e., on-screen or off-screen areas of + memory that reside on the video card). Unlike the X Video + Extension, which writes into an overlay plane, + XCopyAreaScaled would write into the non-overlay areas of + the screen. Key points to consider are as follows: + <itemizedlist> + <listitem><para> + Because different planes are involved, the two scaling + operations are usually implemented in hardware + differently, so an XCopyAreaScaled extension could be + added in a manner that would neither conflict with nor + interact with the X Video extension in any way. + </para></listitem> + <listitem><para> + The XCopyAreaScaled extension provides new + functionality that the X Video Extension does not + provide. Based on anecdotal feedback, we believe that + many people outside the DMX and VNC communities would + be excited about this extension. + </para></listitem> + <listitem><para> + The main drawback to this extension is that it is new + and needs to be implemented at the driver level in + XFree86 for each video card to be supported. At the + present time, it is more likely that the X Video + Extension will be implemented for a particular piece + hardware because the X Video extension has multimedia + uses. However, over time, we would expect the + XCopyAreaScaled extension to be implemented along with + the X Video extension, especially if it becomes + popular. + </para></listitem> + <listitem><para> + Another drawback is that not all modern cards provide + support for a simple scaled blit operation. However, + these cards usually do provide a 3D pipeline which + could be used to provide this functionality in a + manner that is transparent to the client application + that is using the XCopyAreaScaled extension. However, + this implementation pathway would make this extension + somewhat more difficult to implement on certain cards. + </para></listitem> + </itemizedlist> + </para> + </sect3> + <sect3><title>Scaling with OpenGL</title> + <para> + Another general solution to the scaling problem is to use + the texture scaling found in all 3D hardware. This + ability is already exposed through OpenGL and can be + exploited by clients without X server modification (i.e., + other than the ability to support OpenGL). An application + using OpenGL would transmit the non-scaled image to the X + server as a texture, and would then display a single + non-transformed rect using that texture. This also works + around the single overlay problem with the X Video + Extension as well as the need to implement additional + scaled primitive extensions. + </para> + <para> + The downside is that most OpenGL implementations require + power of 2 texture sizes and this can be very wasteful of + memory if, for example, the application needs to scale a + 1025x1025 image, which would require a 2048x2048 texture + area (even a 640x480 image would require a 1024x512 + texture). Another downside is that some OpenGL + implementations have a limited about of texture memory and + cannot handle textures that are very large. For example, + they might limit the texture size to 1024x1024. + </para> + </sect3> + </sect2> + <sect2><title>Application-transparent Scaling for DMX + </title><sect3><title>Back-end Scaling Without Disconnect/Reconnect</title> + <para> + VNC does scaling on the client side (in the + <command>vncviewer</command> application). Implementing a similar + solution for DMX would require support in the back-end X + servers and, therefore, is not a general solution. + </para> + <para> + XFree86 already implements some support for "scaling" that + could be used with DMX: if, in the XF86Config file, + multiple Modes are listed in the Display Subsection of the + Screen Section, then pressing Ctrl-Alt-Plus and + Ctrl-Alt-Minus can be used to iterate through the listed + modes. The display dimensions will change to the + dimensions in the Modes line, but the logical dimensions + of the X server (i.e., the dimensions that Xdmx knows + about) will not change. + </para> + <para> + Further, the dimensions of the XFree86 display are under + software control (via the XFree86-VidModeExtension), so + the Xdmx server could change the screen dimensions on a + per-display basis, thereby scaling the information on part + of that display. + </para> + <para> + However, this scaling appears to have limited use. For + example, assume a 4 by 4 display wall consisting of 16 + 1280x1024 displays. If all of the back-end servers were + simultaneously configured to display 640x480, the left + hand corner of each display would be magnified, but the + composite result would be unreadable. Magnifying one + display at a time could be usable, but could have limited + utility, since the result would still be no larger than a + single display. + </para> + </sect3> + <sect3><title>Back-end Scaling With Disconnect/Reconnect</title> + <para> + Disconnect and reconnect features are not currently + supported in DMX, but are scheduled to be implemented in + the future. These features, combined with the + XFree86-VidModeExtension Extension, would allow an + application to do the following: + <itemizedlist> + <listitem><para> + Disconnect a specific back-end server (via the DMX + Extension), + </para></listitem> + <listitem><para> + reconfigure the XFree86 back-end server resolution, + and + </para></listitem> + <listitem><para> + reconnect the back-end server to DMX -- at a new + origin with the new screen resolution. + </para></listitem> + </itemizedlist> + </para> + <para> + For example, consider a display wall consisting of 16 + 1280x1024 displays with a total resolution of 5120x4096. + All of the screens could be disconnected, repositioned, + and reconnected each at a resolution of 640x480. The + total resolution of the display wall would be 2560x1920, + allowing a view of a selected area approximately + one-fourth of the size of the DMX display. This change + would be completely application independent (except, + perhaps, for a DMX-aware window manager). When work at + the increased resolution was completed, the back-end + servers could be disconnected, reconfigured, and + reconnected for the original 5120x4096 view. + </para> + <para> + Support for this type of scaling can be implemented in a + DMX-aware X11 client assuming the DMX server support + arbitrary disconnect and reconnect semantics. Because + this application cannot be written before + disconnect/reconnect is implemented, this solution will + not be discussed further in this paper. + </para> + </sect3> + <sect3><title>Server-side Scaling</title> + <para> + In earlier versions of DMX, a frame buffer was maintained + on the server side, and XPutImage was used to move the + information from the server to the client (similar to some + early VNC implementations). The use of a server-side + frame buffer would allow the server to do scaling, but is + not a recommended solution because of overall performance + issues and server-side memory issues (i.e., the frame + buffer would be very large for large display walls). + </para> + <para> + Exploration of this path is not recommended. + </para> + </sect3> + </sect2> + <sect2><title>XCreateScaledWindow API</title> + <para> + The implementation of X Video Extension in DMX, and the use + of XvPutImage by applications requiring scaling requires + significant changes in DMX Further, XvPutImage is, + essentially a scaled blit, and it is only useful for + applications which are already using (or can be modified to + use) XPutImage. Therefore, a more general API will be + discussed as another possibility. + </para> + <para> + X applications typically create windows with the + XCreateWindow call. A new extension could provide an + XCreateScaledWindow call that could be used in place of the + XCreateWindow call and be otherwise transparent to the + application. This would allow applications, even those that + do not depend on XPutImage, to take advantage of window + scaling. In this section we describe how the call would + work, what transparency it provides, and how to solve the + potential problems that transparency creates. + </para> + <sect3><title>XCreateWindow</title> + <para> + The XCreateWindow call takes width and height as + parameters. An XCreateScaledWindow call could take all + the same parameters, with the addition of a scaling factor. + </para> + </sect3> + <sect3><title>XSetWindowAttributes</title> + <para> + An X11 window has several attributes that would have to be + scaled: + <itemizedlist> + <listitem><para>Background and border pixmaps</para></listitem> + <listitem><para>Border width</para></listitem> + <listitem><para>Cursor</para></listitem> + </itemizedlist> + </para> + </sect3> + <sect3><title>XGetWindowAttributes, XGetGeometry</title> + <para> + For transparency, calls that query the window attributes + should return unscaled information. This suggests that + all unscaled pixmaps and window attributes should be + cached. + </para> + <para> + Unfortunately, a window manager requires the scaled + geometry to properly decorate the window. The X server + can probably determine which client is acting as the + window manager (e.g., because that client will select + events that are used exclusively by the window manager). + However, other Scaled Window Extension aware clients may + also need to determine the scaled geometry. Therefore, at + least two additional extension calls should be + implemented: XGetScaledWindowAttributes and + XGetScaledGeometry. + </para> + </sect3> + <sect3><title>Popup and Child window positions</title> + <para> + Some applications may position popup and child windows + based on an unscaled notion of the main window geometry. + In this case, additional modifications to the client would + be required. + </para> + </sect3> + <sect3><title>Events</title> + <para> + Most events (e.g., for mouse motion) return information + about the coordinates at which the even occurred. These + coordinates would have to be modified so that unscaled + values were presented to the client. + </para> + </sect3> + <sect3><title>Implementation</title> + <para> + There are many implementation issues, some of which are + similar to the issues involved in implementing the X Video + Extension for DMX. The window contents must be scaled, + either by performing all operations to a frame buffer and + then writing the image to the display (perhaps using + hardware scaling support), or by modifying all of the + various drawing operations to perform scaling. Because of + the complexity involved, the frame buffer option is + recommended. + </para> + </sect3> + </sect2> + </sect1> + + <sect1><title>Conclusion and Recommendations + </title><para> + We recommend a three phase implementation strategy, based on + how an application could be written to take advantage of + scaling: + <orderedlist> + <listitem> + <para> + The XCopyAreaScaled extension should be implemented, since + this is the ideal solution for applications like VNC, and + since making use of this extension will require minimal + changes to applications that already use XPutImage or + XCopyArea. + </para> + <para> + The initial implementation work would include the design + of the X protocol extension, writing this up in the + usual format for extension documentation, implementation + of the protocol transport pieces in XFree86, + implementation of a software fall-back in XFree86 and + DMX, one example hardware implementation for XFree86, + and implementation of support for this extension in DMX. + </para> + <para> + We suggest implementing the extension first on the ATI + Radeon cards. However, since these cards do not provide + a 2D scaled blit primitive, the implementation would + have to make use of the 3D texture engine to emulate a + scaled blit. This is recommended, since other modern + graphics cards also do not provide a simple 2D scaled + blit operation and an example of the more difficult + implementation pathway would be helpful to others. + </para> + </listitem> + <listitem> + <para> + Until XCopyAreaScaled is widely supported, applications + that require scaling will have to fall back to another + scaling method. We suggest OpenGL as the first fall-back + method because it is widely available and supported by + DMX. + </para> + <para> + A project centered around OpenGL-based scaling would + implement this scaling in VNC as an example. This work + would include re-writing the <command>vncviewer</command> + rendering engine to cache a master copy of the desktop + image for all operations. + </para> + </listitem> + <listitem> + <para> + Since OpenGL is not implemented everywhere, and may not + provide hardware-assisted performance in every + implementation, an application that requires scaling + should also fall back to using the X Video Extension. + </para> + <para> + This project would add support for the X Video Extension + to DMX and would add support to VNC to take advantage of + this extension without introducing artifacts. This + would require modifying the <command>vncviewer</command> rendering + engine to cache a master copy of the desktop image for + all operations. This project should also add support + for the RGB format to at least one XFree86 driver (e.g., + ATI Radeon). + </para> + <para> + The X Video Extension is one of the few popular + extensions that DMX does not support. We recommend + implementing the X Video Extension even if scaling is + the specific goal of that work. + </para> + </listitem> + </orderedlist> + </para> + <para> + We do <emphasis>not</emphasis> recommend implementation of the + XCreateScaledWindow extension because of the complexity + involved. We do <emphasis>not</emphasis> recommend implementation of the + XPutImageScaled extension because it requires the same amount + of work as the XCopyAreaScaled extension, but provides less + functionality. Further, server-side scaling with a large + frame buffer is <emphasis>not</emphasis> recommended because of the + performance implications. + </para> + <para> + The back-end scaling, especially with disconnect/reconnect + support should be explored in the future after + disconnect/reconnect is implemented, but not at the present + time. + </para> + </sect1> + + </article> + <!-- Local Variables: --> + <!-- fill-column: 72 --> + <!-- End: --> |