Philipp Fechteler

Abstract: We present an efficient optimization method to determine the 3D pose of a human from a point cloud. The well-known ICP algorithm is adapted to fit a generic articulated template to the 3D points. Each iteration jointly refines the parameters for rigid alignment, uniform scale as well as all joint angles. In experimental results we demonstrate the effectiveness of this computationally efficient approach.

Abstract: We present a video streaming solution to provide fluent remote access to highly interactive 3D applications, such as games. To fulfill the very low delay and low complexity constraint for this class of applications, several optimizations have been developed. Image preprocessing is implemented on the graphics card to make efficient reuse of the rendered output, as well as the GPU's parallel processing capabilities. H.264/AVC video encoding is accelerated by extracting additional information from the rendering context, which allows for direct calculation of motion vectors and partitioning of macroblocks, thereby omitting the demanding search of generic video encoders. A highly optimized client software has been developed to provide very low delay playback of streamed video and audio, using minimum buffering. In experiments a hardly noticeable delay of less than 40 ms could be achieved.

Abstract: Künftig sollen Spiele gar nicht mehr installiert werden. Sie laufen auf riesigen Server-Farmen im Internet, die nur noch Bild und Ton zum Spieler streamen. Eine ganze Wolkenfront von Cloud-Diensten kommt auf Europa zu und nimmt Herstellern von Grafikkarten und aufgemotzten Gaming-PCs die Sonne.

Abstract: Will be added.

Abstract: Zur performanten Unterstützung unterschiedlichster Endgeräte sind dazu zwei Streaming-Methoden entwickelt worden. Beim Graphics-Streaming werden direkt die Grafikbefehle an Endgeräte mit Grafikprozessor übertragen, um dort das Bild dem Display angepasst zu rendern. Verschiedene in diesem Zusammenhang durchgeführte Optimierungen wie intelligentes Caching, Codierung und Emulation des Grafikkontextes führten zur Reduzierung der Datenrate um 80%. Alternativ, für Endgeräte ohne GPU, wird der visuelle Output als Video codiert übertragen. Basierend auf zusätzlichen Informationen aus dem Render-Kontext wurde dafür der rechenintensive Videocodierprozess optimiert: im Durchschnitt wurde eine Beschleunigungen von rund 25% erzielt. Im Artikel wird diese Plattform zum Streamen von interaktiven 3D-Computerspielen vorgestellt.

This article describes a streaming platform for interactive 3D computer gaming. Two streaming methods have been developed to support a wide range of devices. Graphics streaming is used to stream graphics commands directly to the graphics processor incorporated in each end device. The image is then rendered within the processor to match the connected display. Several optimisations have been developed, such as intelligent caching, entropy-coding and local emulation of graphic context, and an optimised encoding process that exploits additional render context information is used to achieve average accelerations of around 25 percent.

Abstract: In this paper, we present a method to speed up video encoding of GPU rendered 3D scenes, which is particularly suited for the efficient and low-delay encoding of 3D game output as a video stream. The main idea of our approach is to calculate motion vectors directly from the 3D scene information used during rendering of the scene. This allows the omission of the computationally expensive motion estimation search algorithms found in most of today's video encoders. The presented method intercepts the graphics commands during runtime of 3D computer games to capture the required projection information without requiring any modification of the game executable. We demonstrate that this approach is applicable to games based on Linux/OpenGL as well as Windows/DirectX. In experimental results we show an acceleration of video encoding performance of approximately 25% with almost no degradation in image quality.

Abstract: Computer game processing requirements for the CPU and graphics performance are growing all the time higher. At the same time many low cost and modest performance CE devices are gaining popularity. People are already used to mobile life style inside home and on the go and they want to enjoy entertainment everywhere. This paper describes an accelerated video streaming method used in a gaming system, called Games@Large, which enables heavy PC game playing on low cost CE devices without the need of game software modification. The key innovations of the Games@Large system are game execution distribution, game streaming, including graphics/ video and audio streaming and game control decentralization as well as network quality of service management. This paper concentrates on the advanced video streaming techniques used in Games@Large.

Abstract: A framework has been developed to support 3D computer gaming for a wide range of types of end devices, ranging from small mobile devices over settop boxes to high end computers. This system developed with the European project Games@Large targets at providing a platform for remote gaming in home, hotel, internet cafes and other local environments. The platform provides
- game execution on server without any local installation
- support of unmodified of-the-shelf computer games
- multiple parallel game sessions
- different real time streaming methods
The main focus of this presentation is on the streaming methods. To support as many end devices as possible with an enjoyable gaming experience, two streaming methods have been developed. For end devices with GPU, the graphics commands are transmitted and the client renders them optimized for its screen resolution. The advantage of this approach is that even large screens can be served without high bit rates. Additionally, the render commands are transmitted when the game emits them before the full scene is rendered which reduces overall delay. In order to achieve real time execution with very low delay, several optimizations have been developed:
- local emulation of server side state of graphics hardware (to answer frequent requests directly without network usage)
- caching server memory at client, e.g. display lists, vertex buffers ...
- encoding/compression of graphics stream
- cross-streaming by deploying an appropriate meta graphics language/protocol (to support games for DirectX/OpenGL independent of the client‘s graphics library)
In scenarios, where graphics streaming is not an option, the visual data is streamed as H264 video. This is the case where the end device does not contain a GPU, or when the game generates commands to complex to be handled with graphics streaming, or when the clients display is small so that the amount of video data is smaller in comparison to the corresponding graphics stream. Since encoding takes place in parallel to a running game, several optimizations have been developed to reduce the computational load of this typically very demanding encoding task. This is even more crucial by considering that there might by several video streaming sessions running in parallel on a single machine. These optimizations are
- intercepting and adapting graphics commands to generate frames ideally suited for the client (viewport - screensize etc.)
- capturing additional render context information, like depth map and projection information, in order to directly compute the macroblock partitioning and motion vectors used by H264 (in contrast to the exhaustive search which is performed by generic encoders)
- continuous intersperse of Intra macroblocks contrasting the usual approach of full Intra frames, in order to reduce the peaks in the bitrates
This framework supports a fully transparent way to provide full 3D game play from nearly every kind of end device, independent of hard- and software.

Abstract: In this paper we present a new cross-platform approach for video game delivery in wired and wireless local networks. The developed 3D streaming and video streaming approaches enable users to access video games on set top boxes and handheld devices that natively are not capable to run PC games. During the development of the distributed gaming system we have faced a number of challenges and problems posed by the hardware and network limitations. In order to solve these problems we have developed a multilevel testing methodology which is based on user assessment and technical measures for the system under development. In this paper we focus on the technical measures and instrumentation that we use for the system’s performance measurement and testing. The benefits of our testing methodology are demonstrated through examples from the development and testing work.

Abstract: Low cost networked consumer electronics (CE) are widely used. Various applications are offered, including IPTV, VoIP, VoD, PVR and games. At the same time the requirements of computer games by means of CPU and graphics performance are continuously growing. For pervasive gaming in various environments like at home, hotels, or internet cafes, it is beneficial to run games also on mobile devices and modest performance CE devices such as set top boxes. EU IST Games@Large project is developing a new cross-platform approach for distributed 3D gaming in local networks. It introduces novel system architecture and protocols used to transfer the game graphics data across the network to end devices. Simultaneous execution of video games on a central server and a novel streaming approach of the 3D graphics output to multiple end devices enable the access of games on low cost devices that natively lack the power of executing high-quality games.

Abstract: In this paper, we present a method to speed up video encoding of GPU rendered scenes. Modern video codecs, like H.264/AVC, are based on motion compensation and support partitioning of macroblocks, e.g. 16x16, 16x8, 8x8, 8x4 etc. In general, encoders use expensive search methods to determine suitable motion vectors and compare the rate-distortion score for possible macroblock partitionings, which results in high computational encoder load. We present a method to accelerate this process for the case of streaming graphical output of unmodified commercially available 3D games which use a Skybox or Skydome rendering technique. For rendered images, usually additional information from the render context of OpenGL resp. DirectX is available which helps in the encoding process. By incorporating the depth map from the graphics board, such regions can be uniquely identified. By adapting the macroblock partitioning accordingly, the computationally expensive search methods can often be avoided. Further reduction of encoding load is achieved by additionally capturing the projection matrices during the Skybox rending and using them to directly calculate a motion vector which is usually the result of expensive search methods. In experiential results, we demonstrate the reduced computational encoder load.

Abstract: The requirements of computer games by means of CPU and graphics performance are continuously growing. At the same time many low cost and modest performance CE devices are gaining popularity. People are already used to mobile life style inside home and on the go and want to enjoy entertainment everywhere. This paper describes a novel gaming system, called Games@Large, which enables heavy PC game play on low cost consumer electronic devices (CE) without the need of game software modification. The key innovations of the Games@Large system are distribution of game execution, streaming of graphics, video, audio, and game control as well as network quality of service management.

Abstract: The authors present an adaptive colour classification method as well as specialised low-level image processing algorithms. With this approach the authors achieve high-quality 3D reconstructions with a single-shot structured light system without the need of dark laboratory environments. The main focus of the presented work lies in the enhancement of the robustness with respect to environment illumination, colour cross-talk, reflectance characteristics of the scanned face etc. For this purpose the colour classification is made adaptive to the characteristics of the captured image to compensate for such distortions. Further improvements are concerned with enhancing the quality of the resulting 3D models. Therefore the authors replace the typical general-purpose image preprocessing with specialised low-level algorithms performing on raw photo sensor data. The presented system is suitable for generating high-speed scans of moving objects because it relies only on one captured image. Furthermore, due to the adaptive nature of the used colour classifier, it generates high-quality 3D models even under perturbing light conditions.

Abstract: Video games are typically executed on Windows platforms with DirectX API and require high performance CPUs and graphics hardware. For pervasive gaming in various environments like at home, hotels, or internet cafés it is beneficial to run games also on mobile devices and modest performance CE devices avoiding the necessity of placing a noisy workstation in the living room or costly computers/consoles in each room of a hotel. This paper presents a new cross-platform approach for distributed 3D gaming in wireless local networks. We introduce the novel system architecture and protocols used to transfer the game graphics data across the network to end devices. Simultaneous execution of video games on a central server and a novel streaming approach of the 3D graphics output to multiple local end devices enable the access of games on low cost set top boxes and handheld devices that natively lack the power of executing a game with high quality graphical output.

Abstract: Nach dem Structured-Light-Prinzip lässt sich ein kostengünstiges und zuverlässiges Verfahren für räumliche Scans erzielen. Voraussetzungen sind neben einem handelsüblichen Projektor und einer Digicam ein paar Kunstgriffe aus der mathematischen Algorithmen-Kiste.

Abstract: In this paper, we present a graphics streaming system for remote gaming in a local area network. The framework aims at creating a networked game platform for home and hotel environments. A local PC based server executes a computer game and streams the graphical output to local devices in the rooms, such that the users can play everywhere in the network. Since delay is extremely crucial in interactive gaming, efficient encoding and caching of the commands is necessary. In our system we also address the round trip time problem of commands requiring feedback from the graphics board by simulating the graphics state at the server. This results in a system that enables interactive game play over the network.

Abstract: We present a system to capture high accuracy 3D models of faces by taking just one photo without the need of specialized hardware, just a consumer grade digital camera and beamer. The proposed 3D face scanner utilizes structured light techniques: A colored pattern is projected into the face of interest while a photo is taken. Then, the 3D geometry is calculated based on the distortions of the pattern detected in the face. This is performed by triangulating the pattern found in the captured image with the projected one.
   The main focus of our work lies in the enhancement of the systems robustness with respect to environment illumination, color cross-talk, reflectance characteristics of the scanned face etc. For this purpose the color classification of the proposed system is made adaptive to the characteristics of the captured image to compensate for such distortions. Further improvements are concerned with enhancing the quality of the resultant 3D models. Therefore we replace the typical general-purpose image preprocessing with specialized low-level algorithms performing on raw CCD sensor data.
   The presented system is suitable for generating high speed scans of moving objects because it relies only on one captured image. Furthermore, due to the adaptive nature of the used color classifier, it generates high quality 3D models even under perturbing light conditions.

Abstract: In this paper, augmented reality techniques are used in order to create a Virtual Mirror for the real-time visualization of customized sports shoes. Similar to looking into a mirror when trying on new shoes in a shop, we create the same impression but for virtual shoes that the customer can design individually. For that purpose, we replace the real mirror by a large display that shows the mirrored input of a camera capturing the legs and shoes of a person. 3-D Tracking of both feet and exchanging the real shoes by computer graphics models gives the impression of actually wearing the virtual shoes. The 3-D motion tracker presented in this paper, exploits mainly silhouette information to achieve robust estimates for both shoes from a single camera view. The use of a hierarchical approach in an image pyramid enables real-time estimation at frame rates of more than 30 frames per second.

Abstract: In coming years we will see low cost networked consumer electronics (CE) devices dominating the living room. Various applications will be offered, including IPTV, VoIP, VoD, PVR and others. With regards to gaming, the need to compete with PlayStation and Xbox will require a radical change in system architecture. While traditional CE equipment suffers from having to meet low BOM (bill of materials) targets, dictated by highly competitive market and cable companies targeted costs, consoles enjoy superior hardware and software capabilities, being able to offset hardware and BOM costs with software royalties. Exent Technologies is leading the European FP6 Integrated Project Games@Large , whose mission is to research, develop and implement a new platform aimed at providing users with a richer variety of entertainment experience in familiar environments, such as their house, hotel room, and Internet Café. This will support low-cost, ubiquitous gameplay throughout such environments, while taking advantage of existing hardware and providing multiple members of the family and community the ability to play simultaneously and to share experiences.
   This paper focuses on one of the innovative aspects of the Games@Large project idea – the interactive streaming of graphical output to client devices. This is achieved by capturing the graphical commands at the DirectX API on the server and rendering them locally, resulting in high visual quality and enabling multiple game execution. In order to support also small handheld devices which lack hardware graphics support, an enhanced video method is additionally provided.

Abstract: In this work, we present a framework to capture 3D models of faces in high resolutions with low computational load. The system captures only two pictures of the face, one illuminated with a colored stripe pattern and one with regular white light. The former is needed for the depth calculation, the latter is used as texture. Having these two images a combination of specialized algorithms is applied to generate a 3D model. The results are shown in different views: simple surface, wire grid respective polygon mesh or textured 3D surface.

Abstract: In this paper, we present a system that enhances the visualization of customized sports shoes using augmented reality techniques. Instead of viewing yourself in a real mirror, sophisticated 3D image processing techniques are used to verify the appearance of new shoe models. A single camera captures the person and outputs the mirrored images onto a large display which replaces the real mirror. The 3D motion of both feet are tracked in real-time with a new motion tracking algorithm. Computer graphics models of the shoes are augmented into the video such that the person seems to wear the virtual shoes.

Abstract: In this paper, we present two techniques for streaming the output of computer games to an end device for remote gaming in a local area network. We exploit the streaming methods in the European project Games@Large, which aims at creating a networked game platform for home and hotel environments. A local PC based server executes a computer game and streams the graphical and audio output to local devices in the rooms, such that the users can play everywhere in the network. Dependent on the target resolution of the end device, different types of streaming are addressed. For small displays, the graphical output is captured and encoded as a video stream. For high resolution devices, the graphics commands of the game are captured, encoded, and streamed to the client. Since games require significant feedback from the user, special care has to be taken to achieve these constraints for very low delays.