Efficient compression of Fresnel fields for Internet transmission of three-dimensional images

Efficient compression of Fresnel fields for Internet transmission of three-dimensional images Thomas J. Naughton,1John B. Mc Donald,1Bahram Javidi2 1Department of Computer Science, National University of Ireland, Maynooth, County Kildare, Ireland 2Department of Electrical and Computer Engineering, University of Connecticut, U-157, Storrs, CT 06269, USA 1 Abstract. We compress phase-shift digital holograms (whole Fresnel fields) for the transmission of three-dimensional images. For real-time networking applications, the time required to compress can be as critical as compression rate. We achieve lossy compression through quantization of both the real and imaginary streams, followed by a bit packing operation. Compression losses in the reconstructed objects were quantified. We define a speedup metric that combines space gains due to compression with temporal overheads due to compression routine and transmission serialization. We empirically verify transmission speedup due to compression, using a special-purpose Internet-based networking application. c °2003 Optical Society of America OCIS codes: 90.1760 Computer holography, 100.6890 Three-dimensional image processing, 999.9999 Data compression, 100.2000 Digital image processing, 100.0100 Image processing 1. Introduction Digital holography1–9 is one of several possible techniques for three-dimensional (3D) imaging.10 Many existing 3D imaging and processing techniques are based on the explicit combination of several 2D perspectives (or light stripes, etc.) through digital image processing. Multiple perspectives of a 3D object can be combined optically, in parallel, and stored together as a single complex-valued digital hologram. Digital holography has seen renewed interest with the recent development of megapixel digital sensors with high spatial resolution and dynamic range. Their digital nature means that these holograms are in a suitable form for processing or transmission. 2 We record in-line digital holograms, recover the whole Fresnel field using a technique called phase-shift interferometry3,6,8 (PSI), and introduce a third step, that of digital compression and decompression.11 Each Fresnel field encodes multiple views of the object from a small range of angles. Different perspectives of the object can be reconstructed by extracting appropriate regions12, 13 from the field and applying a numerical propagation technique.7–9 Real-time optical reconstruction techniques have also been demonstrated.14,15 Our digital Fresnel fields have dimensions 2028 ×2044 pixels and in their native format store 8 bytes of amplitude information and 8 bytes of phase information for each pixel. We would like to compress16 these fields for more efficient storage and transmission. Compression of Fresnel fields (and digital holograms in general) differs to image compression principally because our fields store 3D information in complex-valued pixels, and secondly because of the inherent speckle content which gives them a white-noise appearance. It is not a straightforward procedure to remove the holographic speckle because it actually carries 3D information. The noisy appearance of digital Fresnel fields, and digital holograms in general, causes lossless data compression techniques (such as Lempel-Ziv-Welch, Huffman, and Burrows-Wheeler) to perform poorly.11 The use of lossy compression techniques seems essential. Digital hologram compression techniques based on Fourier-domain processing have been demonstrated.11,17 These block-based techniques tend to introduce localized noise at the boundaries of nonoverlapping blocks. A wavelet-based technique might be more effective. Ding et al.18 perform compression of digital holograms through wavelet decomposition and selection of the principal wavelet basis components appropriate for 3 their pattern recognition application. Liebling et al.19 have developed a wavelet-based reconstruction technique for digital holograms. A course-scale reconstruction or some wavelet-domain quantization in their scheme could also form the basis for a noise removal and/or compression technique. In this paper, we apply quantization directly to the complex-valued pixels. Quantization and phase quantization have been applied successfully to Fourier and holographic data in the past.11, 17, 20, 21 We apply a two-stage compression technique based on complex-domain quantization and bit packing. This introduces a third reason why compression of Fresnel fields (and digital holograms in general) differs to compression of digital images; a change locally in a Fresnel field will, in theory, affect the whole reconstructed object. We are not interested in how compression noise affects the decompressed Fresnel field itself, only how compression noise affects subsequent object reconstruction. In this paper, we use a reconstructed-object-plane RMS metric to quantify the quality of our decompressed Fresnel fields. Compression will permit Fresnel fields to be stored more efficiently. In terms of their transmission, however, there is at least one other property that should be evaluated when comparing compression strategies. We need to know the time it takes, relative to the transmission time, to compress and uncompress the field in order to decide on a compression mechanism for transmission. In particular, it might not even be advantageous to compress the data prior to transmission if the latency caused by the compression routine is significant relative to the average uncompressed transmission time. We consider the case where it is not possible to compress the data in advance, for example in a real-time imaging and transmission application. We use a 4 measure called speedup to quantify the effectiveness of our compression technique in terms of both space and time resources. Our data is obtained using a special-purpose freely-accessible Internet application that, through the integration of compression and transmission routines into a single application, was able to reliably measure compression time relative to transmission time. In Sect. 2, we describe how the fully-complex Fresnel fields are captured using PSI. The networking system is detailed in Sect. 3, and the compression algorithm and compression performance in Sect. 4. Finally, in Sect. 5, we present the results of speedup experiments performed with an implementation of the networking system. 2. Phase-shift digital holography We record Fresnel fields with an optical system based on a Mach-Zehnder interferometer (see Fig. 1). A linearly polarized Argon ion (514.5 nm) laser beam is expanded and collimated, and divided into object and reference beams. The object beam illuminates a reference object placed at a distance of approximately d= 350 mm from a 10-bit 2028 ×2044 pixel Kodak Megaplus CCD camera. Let U0(x, y) be the complex amplitude distribution immediately in front of the 3D object. The linearly polarized reference beam passes through half-wave plate RP1and quarter-wave plate RP2. This beam can be phase-modulated by rotating the two retardation plates. Through permutation of the fast and slow axes of the plates we can achieve phase shifts of 0, −π/2, −π, and −3π/2. The reference beam combines with the light diffracted from the object and forms an interference pattern in the plane of the camera. At each of the four phase shifts we record an interferogram. We use these four real-valued im5 ages to compute the camera-plane complex field by PSI.3,8 We call the camera-plane complex field the Fresnel field, and denote it H0(x, y). Fresnel fields captured using this architecture have themselves been referred to as digital holograms,9,22–24 given a generalized definition of the term hologram. A Fresnel field H0(x, y) contains sufficient amplitude and phase information to reconstruct the complex field U(x, y, z) in a plane in the object beam at any distance zfrom the camera. This can be calculated from the Fresnel approximation13 as U(x, y, z) = −i λz exp µi2π λz¶H0(x, y)?exp "iπ(x2+y2) λz #,(1) where λis the wavelength of the illumination and ?denotes a convolution operation. At z=d, and ignoring errors in digital propagation due to discrete space (pixelation) and rounding, the discrete reconstruction U(x, y, z) closely approximates the physical continuous field U0(x, y). Furthermore, as with conventional holography,12,13 a windowed subset of the Fresnel field can be used to reconstruct a particular view of the object. As the window explores the field a different angle of view of the object can be reconstructed. The range of viewing angles is determined by the ratio of the window size to the full CCD sensor dimensions. Our CCD sensor has approximate dimensions of 18.5×18.5 mm and so a 1024 ×1024 pixel window has a maximum lateral shift of 9 mm across the face of the sensor. With an object positioned d= 350 mm from the camera, viewing angles in the range ±0.74◦are permitted. Smaller windows will permit a larger range of viewing angles at the expense of image quality at each viewpoint. 6 3. Network We can evaluate compression algorithms in terms of both space and time resource usage by using a measure called speedup. Speedup sis defined as s=Pu/Pcwhere Pu is the time required to process and transmit the uncompressed Fresnel field and Pcis the time required to process, and transmit the compressed field. In order to measure speedup of a compression system that resides over a public wide-area network we have found that the following three requirements should be met. Firstly, due to the temporal fluctuations in bandwidth over wide-area networks, it is necessary to average over a large number of timing measurements. Secondly, in order to accurately measure compression and decompression times, the compression routines should be removed from their controlled prototyping environment and executed in a real-world setting. Finally, both the networking software and compression software should be integrated so that meaningful conclusions can be drawn from the relative performance of the transmission and compression components of the system. We have constructed an Internet-based Fresnel field compression application in order to measure reliably and accurately the interaction between compression times and transmission times. This client-server application and associated compression algorithms were written with JavaTM (Sun Microsystems, Inc). This allowed us to develop a platform-independent environment for experimentation over the Internet. Platform-independence ensures that the system supports any architecture that runs a Java virtual machine, and is suitable for heterogeneous environments (the server needs no knowledge of a client’s computer architecture or operating system to com7 municate). As such, we ensure as much as possible the repeatability and relevance of our results for a wide range of Internet set-ups. An overview of the operation of the networking application is shown in Fig. 2. Multiple clients, through their user interfaces, access the server and request particular views of 3D objects stored as Fresnel fields. The server responds by providing the appropriate window of pixels, and the clients reconstruct views of the 3D objects locally. To build the communication component for the client-server application we used Java’s remote method invocation (RMI) facilities. RMI allows applications running on different machines to communicate with each other in an efficient and transparent manner. Central to providing this ease of communication is Java serialization. The term serialization refers to the packaging of volatile data-structures into persistent bit-streams which can then be written to permanent storage or transmitted across a communications link. Our client-server system functions over any network (local, wide, wireless) that supports IP (Internet Protocol). The internal operation of the clients and server from Fig. 2 are shown in Fig. 3. A request for a particular view of a particular 3D object is passed from the user interface, through the client, to the server (stage 0 in Fig. 3). The server extracts the appropriate window from the Fresnel field stored on disk (stage 1), and formats the field for transmission (stage 2). Java is an object-oriented language. As such, Fresnel fields are stored as generalized hologram objects, in a data format that allows efficient manipulation by specialized complex-valued digital hologram processing algorithms, and complete with the functionality to read/write them from/to disk, display them, and reconstruct their 3D objects. Formatting is required to streamline the hologram 8 object for transmission, by removing all functionality and converting the field data into two compact 1D arrays of real and imaginary values. The field is compressed (stage 3) to a stream of bytes as explained in Sect. 4. The particular compression algorithm and compression parameters to be employed by the server are specified by the client at stage 0. In stages 4 and 5 the compressed Fresnel field data is serialized and transmitted. The server collects timing information (time to read from disk, format, and compress) and transmits this with the field. The server responds to the client through a dedicated communication channel that is set up through RMI, and closed immediately afterwards. The client, on receipt of the field data, performs a deserialization operation (stage 6) and decompresses the byte stream as explained in Sect. 4. The unformatting operation converts the separate real and imaginary arrays of field data into a hologram object that has the functionality to be propagated numerically and/or displayed as an intensity image at the user interface (stages 7 through 10). Once again, timings are taken by the client for each of its operations. The client-side interface of the timing application is shown in Fig. 4. Our Internet-based Fresnel field/hologram compression and timing application is accessible online.25 4. Compression In our experiments, the Fresnel field window was compressed by the server (stage 3 in Fig. 3) using a two-step process. The field data was first quantized at a particular resolution and then compressed using a bit packing technique. Each pixel of the field data required two data values (real, imaginary). Quantization levels were chosen to be 9 Technologies, Springer, Berlin, 2002. 11. T. J. Naughton, Y. Frauel, B. Javidi, and E. Tajahuerce, “Compression of digital holograms for three-dimensional object reconstruction and recognition,” Appl. Opt. 41, 4124–4132, 2002. 12. H. J. Caulfield, ed., Handbook of Optical Holography, Academic Press, New York, 1979. 13. J. W. Goodman, Introduction to Fourier Optics, McGraw-Hill, New York, second ed., 1996. 14. M. Sutkowski and M. Kujawinska, “Application of liquid crystal (LC) devices for optoelectronic reconstruction of digitally stored holograms,” Optics and Lasers in Engineering 33, 191–201, 2000. 15. O. Matoba, T. J. Naughton, Y. Frauel, N. Bertaux, and B. Javidi, “Real-time three-dimensional object reconstruction by use of a phase-encoded digital hologram,” Appl. Opt. 41, 6187–6192, 2002. 16. M. Rabbani, ed., Selected Papers on Image Coding and Compression, SPIE Milestone Series vol. MS48, SPIE Press, Bellingham, Washington, 1992. 17. T. Nomura, A. Okazaki, M. Kameda, Y. Morimoto, and B. Javidi, “Digital holographic data reconstruction with data compression,” Algorithms and Systems for Optical Information Processing V, Proceedings of SPIE vol. 4471, 235–242, (San Diego, California), 2001. 18. L. Ding, Y. Yan, Q. Xue, and G. Jin, “Wavelet packet compression for volume holographic image recognition,” Opt. Comm. 216, 105–113, 2003. 16 19. M. Liebling, T. Blu, and M. Unser, “Fresnelets: new multiresolution wavelet bases for digital holography,” IEEE Transactions on Image Processing 12, 29–43, 2003. 20. J. W. Goodman and A. M. Silvestri, “Some effects of Fourier domain phase quantization,” IBM J. Res. Develop. 14, 478–484, 1970. 21. W. J. Dallas and A. W. Lohmann, “Phase quantization in holograms,” Appl. Opt. 11, 192–194, 1972. 22. E. Tajahuerce, O. Matoba, and B. Javidi, “Shift-invariant three-dimensional object recognition by means of digital holography,” Appl. Opt. 40, 3877–3886, 2001. 23. Y. Frauel, E. Tajahuerce, M.-A. Castro, and B. Javidi, “Distortion-tolerant threedimensional object recognition with digital holography,” Appl. Opt. 40, 3887– 3893, 2001. 24. Y. Frauel and B. Javidi, “Neural network for three-dimensional object recognition based on digital holography,” Opt. Lett. 26, 1478–1480, 2001. 25. http://hologram.cs.may.ie/online/jao2003/hologram.html . 17 List of Figure Captions Fig. 1 Experimental setup for PSI: BE, beam expander; BS, beam splitter; RP, retardation plate; M, mirror. Fig. 2 Illustration of the network-independent multiple-client system; U.I., user interface. Fig. 3 Internal operation of (a) server and (b) client processes. Fig. 4 Screenshot of client-side of timings application. Top row (l to r): full Fresnel field with window indicated, uncompressed Fresnel field window data, uncompressed timings, and uncompressed reconstruction. Bottom row (l to r): control panel, compressed window, compressed timing information, and compressed reconstruction. Fig. 5 NRMS difference in the reconstructed intensity plotted against number of bits in each of the Fresnel field’s real and imaginary values, for various degrees of subsampling. Fig. 6 Reconstructed views (with 4×4 subsampling) from a 1024×1024-pixel window from the Fresnel field stored with different quantization resolutions: (a) no quantization, (b) 4 bits, (c) 3 bits, (d) 2 bits of resolution in each real and imaginary value. Fig. 7 Speedup as a function of increasing compression, for various Fresnel field window sizes. 18 λ/2 λ/4 BE M Ar laser CCD BS BS M M RP1RP2 d Fig. 1. Experimental setup for PSI: BE, beam expander; BS, beam splitter; RP, retardation plate; M, mirror. tjnF1.eps 19 Server ... Local, wireless, or wide-area network Client Client Client Hologram database U.I. U.I. U.I. Fig. 2. Illustration of the network-independent multiple-client system; U.I., user interface. tjnF2.eps 20 Format Compress Transmit Server process Deserialize Decompress Unformat Image Reconstruct Client process (b) 2 3 5 6 7 8 10 9 Request hologram window 0 (a) Read 1 Request from client Serialize 4 Fig. 3. Internal operation of (a) server and (b) client processes. tjnF3.eps 21 Fig. 4. Screenshot of client-side of timings application. Top row (l to r): full Fresnel field with window indicated, uncompressed Fresnel field window data, uncompressed timings, and uncompressed reconstruction. Bottom row (l to r): control panel, compressed window, compressed timing information, and compressed reconstruction. tjnF4.eps 22 2 3 4 5 6 7 8 9 10 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Quantization resolution (bits per value) Normalized RMS difference no subsampling, RM energy = 0.13394 [2 2], RM energy = 0.46764 [4 4], RM energy = 1.6359 [16 16], RM energy = 23.5195 Fig. 5. NRMS difference in the reconstructed intensity plotted against number of bits in each of the Fresnel field’s real and imaginary values, for various degrees of subsampling. tjnF5.eps 23 (a) (b) (c) (d) Fig. 6. Reconstructed views (with 4×4 subsampling) from a 1024×1024-pixel window from the Fresnel field stored with different quantization resolutions: (a) no quantization, (b) 4 bits, (c) 3 bits, (d) 2 bits of resolution in each real and imaginary value. tjnF6.eps 24 2345678 1 3 5 7 9 11 13 Quantization resolution (bits per value) Speedup in transmission 512x512 256x256 128x128 64x64 32x32 16x16 Fig. 7. Speedup as a function of increasing compression, for various Fresnel field window sizes. tjnF7.eps 25