Near Regular Texture Analsyis for Image-Based Texture Overlay
Image-based texture overlay or retexturing is the process of augmenting a surface in an image or a video sequence
with a new, synthetic texture. On the one hand, texture distortion
caused by projecting the surface into the image plane
should be preserved. On the other hand, only the texture
albedo should be altered but shading and reflection properties
should remain as in the original image. In many applications,
such as augmented reality applications for virtual
clothing, the surface material to be retextured is cloth.
In this case, high frequency details, representing e.g. selfshadowing
of the yarn structure, might also be a property
that should be preserved in the augmented result.
This work specifically addresses the decomposition of
images of deformed regular textures into its intrinsic parts which decompose the image into the appearance of the undeformed
regular texture, a deformation field, a shading map
representing lighting effects and additional high frequency
details (see figure below). This is closely related to intrinsic image decomposition methods which decompose an arbitrary
image into the product of an illumination component that
represents lighting effects and a reflectance component related
to the color of the observed material.
In our decomposition model, the deformed regular texture
(the estimated deformation field applied to the estimated appearance
of the regular texture) can be seen as the reflectance
part of our decomposition while the shading map and the
high frequency details can be seen as the illumination component.
Regular textures can be constructed by regularly tiling the
texture space with the same texture element, called texel in
the following. Textures that deviate geometrically
and photometrically from a regular congruent tiling are often
called near-regular textures (NRT) [Liu et al (2004)]. In
contrast to regular textures, the texture elements appear geometrically
and photometrically distorted in the image due
to variations in the viewing angle, lighting conditions and
partial occlusions. Nevertheless, they still exhibit
certain topological regularities and relations as regular
textures. We exploit this topological regularity
to estimate the intrinsic parts of the given image of
a near-regular texture. We first start by estimating the mean
appearance of a texel and candidate positions of the texel in
the image. From the estimated mean texel we synthetically
generate an image of the regular texture. This image is used
as reference in an image-based optimization method that registers
two images not only geometrically but also photometrically,
yielding a deformation field and a shading map. Finally,
as the mean texel does not contain any high-frequency
details, these can be estimated from the difference between
the original image and the warped and shaded synthetic reference
image.
Decomposition of the original texture (left) according into a regular texture, a deformation represented by a deformed mesh, an estimated shading map and estimated high frequency yarn structure (colors scaled).
Different texel appearance estimation results..
Different retexturing results. The most left images depict the original input image.
A. Hilsmann, D.C. Schneider and P. Eisert: "Warp-based Near-Regular Texture Analysis for Image-based Texture Overlay", Vision, Modeling, and Visualization Workshop 2011 , Berlin, Germany, October 2011. [pdf]
A. Hilsmann, D.C. Schneider and P. Eisert:"Image-based Retexturing of Deformed Surfaces from a Single Image", Eurographics 2011 Poster , Llandudno, UK, April 2011. Best Poster Award [pdf]