Record of activities

Two visits of one week each has been made by Thomas Maugey at EPFL in February and May. In February 2017, Thomas Maugey has been member of the jury for the Master thesis defense of Saleh Bagher Salimi at EPFL. In May, Francesca De Simone, Pascal Frossard and Thomas Maugey have developed the preliminary steps of the study of graph-based transform design for omnidirectional images. Additionally, a seminar has been given by Thomas Maugey at EPFL during his visit in May.

Mira Rizkallah is currently visiting EPFL for a period of two months between October and December 2017, in order to conduct the study of graph-based transform design for omnidirectional images. She will give a seminar at EPFL.

Scientific outcomes

As it was described in the GOP proposal, the objective of the first year was to work on the graph construction for 360° video, along with the acquisition and calibration of multi-view videos.
Here are the scientific outcomes of the project GOP for the year 2017 in line with the initial objectives:

– Acquisition of Omnidirectional Images: Cédric Le Cam (Inria) has first developed an algorithm to calibrate one omnidirectional video using a chessboard pattern. More precisely he has demonstrated that the unified spherical model holds for the Samsung Gear 360° used in our project. Secondly, he has worked on a two-view calibration algorithm estimating the relative position of two omnidirectional cameras. This work has led to one conference publication [1]. Finally, we have recorded a first sequence involving 40 omnidirectional cameras.

– Graph construction for one view: Saleh Bagher Salim (Master Student at EPFL, 2017) has worked on the graph representation and sampling for 360° images. More precisely, he has compared the well-adopted non-uniform sampling (panorama or equirectangular) of the sphere and the uniform one. For both of them he has build a graph-based transform based on the geodesic distance. Secondly, the student has developed a first version of sub-graph transform to circumvent the huge computation burden due to the big graph size [2].

On the similar objective, Mira Rizkallah (PhD student of Inria) is currently visiting the LTS4 laboratory at EPFL, to pursue the work on effective sub-graph construction.

– Navigation in Omndiriectional Image : an omnidirectional image does not contain interesting content everywhere. Some part of the images are indeed more likely to be looked at by some users than others. Knowing these regions of interest might be useful for 360° image compression, streaming, retargeting or even editing. In collaboration with Olivier Le Meur, we have modelled the user navigation within a 360° image, and detected which parts of an omnidirectional content might draw users’ attention. In particular, we have proposed a smooth navigation through the image to maximize saliency. This work has led to a conference publication in a Special Session on Omnidirectional Images organized by Francesca De Simone (EPFL) [3].

– Motion estimation in omnidirectional image sequence : Francesca De Simone and Pascal Frossard has developed an extension of block-based motion estimation for omnidirectional videos, based on a camera and translational object motion model that accounts for the spherical geometry of the imaging system. They use this model to design a new algorithm to perform block matching in sequences of panoramic frames that are the result of the equirectangular projection. Experimental results demonstrate that significant gains can be achieved with respect to the classical exhaustive block matching algorithm (EBMA) in terms of accuracy of motion prediction. This work has been published in [4].

Production

Several publications have been done in the context of the project GOP:

[1] T. Maugey, C. Le Cam, L. Guillo, Télévision à point de vue libre et système de capture à plusieurs caméra omnidirectionnelles GRETSI, Juan-les-Pins, France, Sep. 2017.

[2] Saleh Bagher Salimi, Omnidirectional Image Compression Using Graph-Fourier Transform, Master Thesis, EPFL, 2017

[3]  T. Maugey, O. Le Meur, Z. Liu, Saliency-based navigation in omnidirectional image
IEEE MMSP, London, UK, Oct. 2017.

[4] F. De Simone, N. Birkbeck, B. Adsumilli and P. Frossard. Deformable block based motion estimation in omnidirectional image sequences. IEEE MMSP, London, UK, Oct. 2017.

Next year work program

Next year, our effort will focus on the continuation of Mira Rizkallah’s work on graph-based transform design for 360° images. The purpose will be to find accurate models to evaluate the correlation between pixels through the graph connections as a function of the geometry. We will then focus on a rate allocation solutions with a 360° image, based on some learned statistics on multiple omnidirectional images.

Second, the study of multi-view camera systems will continue by enhancing the performance of the calibration algorithm and capturing more datasets. Based on these new tools, we will start working on the virtual view synthesis based on several reference omnidirectional views. The virtual view synthesis of omnidirectional video is indeed the last step before free viewpoint television in its more general definition and will be developed by Mattia Rossi at EPFL.