FascinatE Newsletter 1

Welcome from the Coordinator 1 Report from Third Plenary Meeting in London 1 Towards Gesture Recognition for User Interaction in the FascinatE System 2 Test Shoot at a Premier League Football Match 3 Announcements and Future Events 3 Welcome from the Coordinator h t t p : / / w w w . f a s c i n a t e - p r o j e c t . e u / Issue 1, November 2010 F a s c i n a t E N e w s l e t t e r I n s i d e t h i s i s s u e : Plenary Meeting, London, 6-7 October 2010 S p e c i a l p o i n t s o f i n t e r e s t : Successful capture of Chelsea vs Wolves game using OmniCam and Arri‟s Alexa camera Gesture control of home TV setup proposed using head and hands tracking FascinatE scripting development to allow user selection of scene down to mobile devices used by individuals are covered. Interaction methods tailored to the device, e.g. hand gestures for big devices and touch gestures for the smaller ones are investigated. Just recently we have finished our first test shoot, where we had the opportunity to capture a full range of content of the English Premier League soccer game Chelsea vs. Wolverhampton Wanderers (article on page 3). We will showcase our results at different stages of the project. The first demonstrations are planned to be given at IBC in September 2011. In order to be kept upto-date on the developments of the project please visit www.fascinate-project.eu or follow us on twitter "@Fascinate_Prjct". Georg Thallinger, Project Coordinator The project held its third plenary meeting at BBC R&D‟s premises in West London in October. The location of the meeting turned out to be rather convenient, as the venue for the test shoot (Chelsea‟s football ground, Stamford Bridge) was nearby. Around a dozen project participants took the opportunity to make a site visit there the day before the meeting, which was instrumental in getting permission to use the venue for the test shoot. The test shoot preparation was one of the main items discussed at the plenary meeting, including compiling lists of equipment needed, timing for setup and de-rigging, and configuration of cameras and microphones. Another main topic of discussion was the system specification, including the preparation of a forthcoming internal deliverable that will give an overview of the whole FascinatE system. The third main topic was „scripting‟ – the process of controlling which parts of the scene a viewer selects to see (or has suggested to him by the programme production team). This concept is one of the key novel aspects of the project. In addition to the main business of the meeting, the project partners enjoyed an evening boat trip along the Thames to a restaurant in the Docklands area of London, and a brief tour around BBC R&D, including a look into the dedicated R&D studio at Television Centre. I am happy to introduce you to FascinatE! Since February this year our consortium consisting of eleven partners spread over Europe is working with great enthusiasm to implement our idea of the ultra high resolution interactive television service of the future. A full system comprising appropriate capturing and analysis technology, networking components and various terminal devices will be implemented. The capturing side uses as base an ultra highdefinition panorama augmented with additional cameras and also 3D and ambient audio. This information is compiled into a layered scene representation together with metadata. The networking components will be able to interpret the layered scene representation and will adapt the content depending on the type of service or the capabilities of the target device. In respect to terminal devices the whole range from high resolution, immersive displays for a bigger audience, home environments with TV-sets Explaining the Omnicam to an official at Stamford Bridge 1 Knowing the camera parameters and the depth of the estimated position, a 3D estimate may be obtained. The size of the search zone is updated depending on the variance of the estimated head position and the matching score value. In a second step, a three dimensional bounding box is attached to the head position, in such a way that hands lie in the box when moved before the body. An estimate of the position of the hand(s) is obtained after segmenting and grouping the 3D points in the bounding box. The head+hands tracking module performs in real time at more than 20 frames per second, enabling many interesting applications. In the FascinatE Newsletter Towards Gesture Recognition for User Interaction in the FascinatE System Gesture recognition technologies are being widely applied to many applications related to the interaction between users and machines. There is a global tendency to replace external devices, such as remote controls, keyboards or mice, with device-less gesture recognition solutions. Indeed, the objective is to obtain device-less, but also marker-less, gesture recognition systems that allow users to interact as naturally as possible, providing a truly immersive experience. Within the FascinatE project, the Universitat Politècnica de Catalunya (UPC) is working in providing seamless user interaction with the system by detecting and recognizing user gestures. Therefore, a user of the FascinatE system will be able to interact with it from his/her couch without the need of any external device. Several gestures are being investigated in order to allow the user to interact with the system. The gestures allow the user to perform simple interactions, such as selecting different channels on their TVs, to more innovative interactions such as automatically following their choice of players in a football match, or navigating through high resolution panoramic views of the scene. A home setup consisting in a centered Time-Of-Flight (TOF) camera [1] and two lateral color cameras is proposed. In order to interpret user gestures, head and hands are tracked by exploiting TOF depth estimation. In such a context, an ellipse is resized to the estimated head size, depending on distance between the user and the camera. This ellipse is projected onto the TOF camera image plane, aiming to find the image zone which better matches the elliptical shape. A matching score is obtained at every image, the best score giving an estimate of the head position. Hand tracking using the Time-Of-Flight camera with visual feedback to the user figure below a user feed-back is presented, where the user can visualize the relative position of his/her hands on a TV screen. Eventually, the user could be able to point zones on the screen, navigate through menus or perform gestures to control some functionalities of FascinatE‟s TV-based home system. References [1] A Kolb, E Barth, R Koch, and R Larsen, “Time-of-Flight Cameras in Computer Graphics,” Computer Graphics Forum, vol. 29, no. 1, pp. 141–159, 2010. “In order to interpret user gestures, head and hands are tracked by exploiting TOF depth estimation” 2 On 23rd October 2010, the partners involved in “WP2 AV Capture and Processing” carried out the first FascinatE test shoot at a live event. The consortium had the opportunity to capture the UK premier league match Chelsea vs. Wolverhampton Wanderers. The aim of this shoot was to get a complete set of audiovisual material in order to research and develop the new concepts of format agnostic production. Therefore the omnidirectional high-resolution camera system by HHI, the new high-dynamic range camera Alexa from ARRI, an Eigenmike® from Technicolor and two Soundfield® mics from the University of Salford were brought to London and installed on different camera platforms in the stadium. Due to close cooperation between BBC and SIS Live, the outside broadcaster, we were able to get the recordings of several broadcast cameras, twelve shotgun microphones and several stereo microphones located around the pitch. Subscribe for updates at http://www.fascinate-project.eu/ Announcements You can access up-to-date information about the FascinatE project by accessing our web portal at http://www.fascinate-project.eu. FacinatE presented at the 2010 NEM Summit, 13-15 October 2010, 4th Towards Future Media Internet: http://nem-summit.eu/ 3 IS&T /SPIE, 23 – 27 January 2011, International Conference on Multimedia Content Access: Algorithms and Systems: http://spie.org/ei122 Issue 1, November 2010 Test Shoot of a Premier League Football Match During the setup, especially of the omni-directional camera system, the team faced several issues such as space problems – there were several discussions on obstructed views by the huge camera, particularly power issues and weather conditions. However, the team succeeded and was able to capture the whole match without any rain and a pretty nice cloudy sky. After the match, a complete 3D laser scan of the stadium was captured using the new 3D laser scanner Focus 3D by Faro. In this way we have been able to accurately register all the camera and microphone positions that are required for matching of visual and sound events. After an intensive preparation phase in the weeks before and a long day in the stadium itself, the test shoot team is pretty much satisfied with the result. However a detailed analysis especially of the video recordings has to be performed and relevant sequences of the match have to be selected in order to validate the success of the shooting. After that, the material will be distributed among the partners for further research and development. One final note: Chelsea won the match 2-0. Calibrating the OmniCam Chelsea vs Wolverhampton Wanderers at Stamford Bridge “…the team succeeded and was able to capture the whole match without any rain and a pretty nice cloudy sky.” Georg Thallinger Joanneum Research Graz, Austria [email protected] Contact Details and Project Office FascinatE is a €9.5m EU-funded project involving a group of 11 partners from across Europe. FascinatE stands for: Format-Agnostic SCript-based INterAcTive Experience and is looking at broadcasting live events to give the viewer a more interactive experience no matter what device they are using the view the broadcast. The FascinatE project will develop a system to allow end-users to interactively view and navigate around an ultra-high resolution video panorama showing a live event, with the accompanying audio automatically changing to match the selected view. The output will be adapted to their particular kind of device, covering anything from a mobile handset to an immersive panoramic display. At the production side, this requires the development of new audio and video capture systems, and scripting systems to control the shot framing options presented to the viewer. Intelligent networks with processing components will be needed to repurpose the content to suit different device types and framing selections, and user terminals supporting innovative interaction methods will be needed to allow viewers to control and display the content. Newsletter Editor Ben Shirley University of Salford Salford, UK [email protected] The FascinatE Project and Consortium FascinatE Newsletter FascinatE is funded by the European Union‟s Seventh Framework Programme © by the FascinatE consortium 4