Research > Acquisition Representation > Navigation Segments
Enabling users to interactively navigate through different viewpoints of a static scene is a new interesting functionality in 3D streaming systems. While it opens exciting perspectives toward rich multimedia applications, it requires the design of novel representations and coding techniques to solve the new challenges imposed by the interactive navigation. In particular, the encoder must prepare a priori a compressed media stream that is flexible enough to enable the free selection of multiview navigation paths by different streaming media clients. Interactivity clearly brings new design constraints: the encoder is unaware of the exact decoding process, while the decoder has to reconstruct information from incomplete subsets of data since the server generally cannot transmit images for all possible viewpoints due to resource constrains.
Traditional prediction structures are not adapted for interactive navigation: for a given user navigation, more frames than requested are needed at the receiver side because of the heavy prediction structure.
We propose a novel multiview data representation that permits us to satisfy bandwidth and storage constraints in an interactive multiview streaming system. In particular, we partition the multiview navigation domain into segments, each of which is described by a reference image (color and depth data) and some auxiliary information. The auxiliary information enables the client to recreate any viewpoint in the navigation segment via view synthesis. The decoder is then able to navigate freely in the segment without further data request to the server; it requests additional data only when it moves to a different segment.
The navigation domain is partitioned into navigation segments, and each navigation segment is encoded and stored on a server. Users interact with the server to request the navigation segments needed for the navigation.
More precisely, the navigation domain (ND) is divided into sub-domains called navigation segments, which are transmitted to the decoder upon request. Upon reception of data of a navigation segment, the decoder can independently create any virtual view in this sub-domain without further request to the server. This provides flexible navigation capabilities to the receiver. But it also implies a complete change in the data representation in order to limit storage and bandwidth costs. Each navigation segment is thus represented with a reference frame and some auxiliary information. The auxiliary information carries, in a compact form, the innovation inherent to new viewpoints and permits to synthesize any view in the navigation segment with help of the reference frame. We further propose to optimize the partitioning of the navigation domain under rate and storage constraints.
Top-down illustration of the concept of navigation segment for a simple scene with one background (vertical plane) and two foreground objects (vertical rectangles).
More details in the following paper:
J3 – T. Maugey, I. Daribo, G. Cheung, P. Frossard, Navigation domain partitioning for interactive multiview imaging, in IEEE Transactions on Image Processing, Vol. 22, p. 3459-3472, 2013.