US20110211040A1 - System and method for creating interactive panoramic walk-through applications - Google Patents

System and method of the invention provides for creating, storing and broadcasting interactive panoramic walk-through applications. The combination of images is determined by the user's choice of direction of displacement at each intersection point and from each view point or geographical coordinate, in order to provide a complete view from a first person's point of view. The system provides a visual perspective comparable to a human visual experience.

• This application claims the benefit of U.S. Provisional Application No. 61/111,346, entitled SYSTEM AND METHOD FOR CREATING AND BROADCASTING INTERACTIVE PANORAMIC WALK-THROUGH APPLICATIONS, filed Nov. 5, 2008.

FIELD OF THE INVENTION

• The present invention relates generally to virtual tours. More specifically, the present invention relates to virtual walk-through applications using panoramic images, 3D images or a combination of both.

BACKGROUND OF THE INVENTION

• A virtual tour (or virtual reality tour) is a virtual reality simulation of an existing location, which is usually built using contents consisting principally of 2D panoramic images, sequence of linked still images or video sequences, and/or image-based rendering (IBR) consisting of image-based models of existing physical locations, as well as other multimedia content such as sound effects, music, narration, and text. A virtual tour is accessed on a personal computer (typically connected to the Internet) or a mobile terminal. Although not replacing real travel, virtual tours aim at evoking an experience of moving through the represented space. Virtual tours can be especially useful for universities and in the real estate industry, looking to attract prospective students and tenants/buyers, respectively, eliminating for the consumer the cost of travel to numerous individual locations.

• The word panorama indicates an unbroken view, so essentially, a panorama in that respect could be either a series of photographic images or panning video footage. However, the terms ‘panoramic tour’ and ‘virtual tour’ are generally associated with virtual tours created using stills cameras. Such virtual tours created with still cameras are made up of a number of images taken from a single view point. The camera and lens are rotated around what is referred to as a nodal point (the exact point at the back of the lens where the light converges). These images are stitched together using specialist software to create a panorama representing a near 360 degree viewing angle, as viewed from a single “view point”; the panoramas are each resized and configured for optimal on-line use. Some ‘panographers’ will then add navigation features, such as hotspots (allowing the user to “jump” from one viewpoint or panorama to the next) and integrate geographic information such as plans or maps.

• Current virtual tours photographic techniques suffer from several limitations. A seamless panoramic image can not be created from still images whenever such still images are captured from different nodal points or, for two consecutive images, from a single nodal point but with different focal lengths or focus distances. Images captured from a single camera rotating on its nodal point can be stitched seamlessly but this solution can not be used for applications involving axial translation, where, for example, images are captured from a vehicle in motion.

• Catadioptric optical systems provide images having a 360° horizontal field of view and near 180° vertical field of view. The resulting panoramic images are of a annular shape and generally must be sliced and open and “unwarped” to create a panoramic image of a rectangular shape. The unwarping step causes image distortion which, together with the optical distortions caused by the Catadioptric optics having unevenly distributed angles along its radial axis (vertical axis of the view), must be compensated by specialised application software.

• Patent document US 2007/0211955 to Pan discloses a perspective correction method allowing e-panning without image distortion, wherein image correction step is performed on image slices (horizontal section of the wide-angle image) by repositioning each pixel to a corresponding point on a cylindrical surface. This method consumes significant processing power and bandwidth for respectively correcting and transmitting the images whenever fast user motion is involved during navigation, and is therefore not optimal for providing a seamless navigation experience at relatively high user directed panning speed.

• Also, with current image capture solutions, objects near the camera are responsible for occlusion on distant objects; “occlusion” meaning, with regard to 2D images, the non-projection of a surface to a point of observation, and with regard to a 3-D space, the effect of one object blocking another object from view. Limitations of current virtual tour technology, such as object occlusion, have had the detrimental result on virtual tours never materializing outside of the real estate industry.

• Virtual Walk-Through (“VWT”) applications constitute an evolution over virtual tours. This technology eliminates the occlusion limitation by enabling user to travel to a point where distant objects are no longer occluded.

• Commercial online walk-through products such as Google “Street View” provide virtual outdoor walk-through of cities using images captured from a camera mounted on a road vehicle which circulates on motorways and roads at speeds ranging from 30 kmh to 80 kmh. These products are limited to outdoor views wherein any two consecutive points of view are positioned at a relatively long distance from each other. Typically, StreetView and applications of the like provide visualisation at road-view level, that is, visiting a city as viewed from a car, wherein the user follows a pathway formed by a plurality of panoramas accessible along main streets, the user following the pathway by “jumping” (generally from a graphical interface allowing clicking of on screen icons) from a panorama or point of view to the next distant point of view. This application uses a series of standard photographic images, taken by multiple cameras systems mounted to produce images representative of multiple view angles; panoramic images are produced by computation (stitching) of still images from the multiple cameras. Such panoramic images can not provide accurate representation of geometric objects, of for example buildings, due to the inherent discontinuity (break) of such panoramic images, this discontinuity is due to the physical impossibility of superposing a single nodal point from multiple cameras and view angles. Furthermore, “STREET VIEW” products and the like provide images which suffer from trapezoidal distortion whenever the view angle is not pointing toward the horizon; this distortion is due to the perspective. Although geometrically correct, “STREET VIEW”'s images do not reflect human vision behaviours, which keep vertical lines mostly parallel whenever a viewer tilts its view gently above or below the horizon.

• Google “STREET VIEW” also creates ground plane distortion where planar ground seems to be inclined due to the unwanted motion of the cameras caused by the inertial force. Other current walk-through products, such as “EVERYSCAPE” (www.everyscape.com by Everyscape, Waltham, Mass.) and “EARTHMINE” (www.earthmine.com by Earthmine inc., Berkeley, Calif.) also produce trapezoidal distortion, which makes them unfit for applications requiring continuous undistorted images (i.e. images which more closely correspond to human vision), as for example, for virtual shopping.

• The trapezoidal distortion drawback is inherent also to virtual walk-through applications based on 3D virtual images, which can be used for example for visiting a virtual building using real-

  time

  3D engine, such as Second Life (www.secondlife.com by Linden Research Inc, San Francisco, Calif.) or video games.

• “EARTHMINE” provides commercial online walk-through for applications such as management of buildings and assets, telemetric measurement and other cadastral works. The product combines high resolution still images and 3D mesh information to provide pathways wherein user jumps from one distant view point to another.

• “EVERYSCAPE” provides commercial online panoramic products wherein motion between two consecutive view points is simulated by video postproduction effects. This product does not allow user to pan and tilt its viewing angle during its displacement along the travel path. During the travel motion effect, images are no longer panoramic unless the fields of view of images representative of the next fixed point, are constrained to the motion axis.

• In sum, current virtual walk-through applications and systems suffer several important limitations. Travel along pathways is achieved by jumping from one view point to another, instead of by a fluid travel motion. Views suffer from high occlusion rate, as lots of objects are never visible at all along pathways. Generally, images are not standard panoramic images but rather patchy assembly of 2D images where many discontinuities are found on each image.

• The prior art describes several techniques whose purpose is reducing the bandwidth associated with the transmission of panoramic images and applications between a server and the user remote terminal, allowing a user to navigate a walkthrough space while downloading data.

• The use of a predefined pathway has been widely adopted to prevent storage and transmission of redundant image data. Predefined pathways have the additional benefit of simplifying user navigation, notably by preventing the user from searching available paths or from hitting objects repetitively during motion, as would be the case when user tries to walk through walls or door images.

• U.S. Pat. Nos. 6,388,688 and 6,580,441, both to Schileru-Rey, disclose a computer system and method that allow interactive navigation and exploration of spatial environments, wherein pathways are represented by branches, and intersections in the real environment are represented by nodes. User selects which path to follow from the node. A branch represents a video sequence or animation played during motion between two adjacent view points. Virtual objects can be integrated to specific branches or nodes, without assigning a geographic coordinate to virtual objects; each object is linked to at least one branch or node and displayed when user is travelling on said branch or node.

• U.S. Pat. No. 7,103,232 to Kotake discloses an IBR system with improved broadcasting performance, where a panoramic image is created by stitching several images from several cameras, preferably video cameras, pointing to distinct points of view, the cameras being synchronised by use of a single time code. The '232 system provides panoramic images divided in six images sections of 60° horizontal field of view, and broadcast typically only two of the six sections' image (providing a 120° field of view) at any giving time with an aim to reduce processing power and communication bandwidth. The '232 solution is not optimized however for walk through applications allowing fast movement across the horizontal plane beyond 120°; moreover, the '232 patent does not disclose broadcasting images of different image resolution, meaning that it only covers broadcasting of image of the highest possible image resolution.

• U.S. Pat. No. 6,633,317 to Jiang Li discloses a data transfer scheme, dubbed spatial video streaming, allowing a client to selectively retrieve image segments associated with the viewer's current viewpoint and viewing direction, rather than transmitting the image data in the typical frame by frame manner. The method of patent 317' divides walkthrough space in a grid; each cell of the grid is assigned to at least one image of the surrounding scene as viewed from that cell; images are characterised similar to a concentric mosaic in that each cells is represented by a sequence of image columns. Method of patent 317' allows transmission of part of the images (compressed or not) needed in an attempt to anticipate viewer's change of view point within the walkthrough space, starting with image data corresponding to viewpoints immediately adjacent the current viewpoint, with subsequent image data associated with viewpoints radiating progressively out from the current viewpoint. Patent 317' is well suited to open walkthrough spaces where user can move in any direction using multiple source of image data (simple 2D image, panoramic image or concentric mosaic), such as in a typical 3D environment. However, this method is not suited to the optimal transmission of full panoramic images in situations where user travels along predefined pathways consisting of several view points in a linear arrangement within a network of pathways within the walkthrough space. Additionally, being view direction sensitive, this method is not optimized, in terms of response time, to allow user to change his travel plan, for example, by making a U-turn or to travel along another pathway. Finally as this method allows travel in any direction (along a predefined pathway) the amount of data download to represent a given view point is greater and therefore less suited for fast and responsive viewing experience on the Internet or other network media having limited bandwidth.

• Consequently, no system of the prior art provides a system optimized for seamless broadcasting of fluid motion where the user can orientate (pan and tilt) the field of view during motion and where the user can stop the motion anywhere along the travel path, in order to discover local objects in detail without occlusion.

• Integration of virtual objects (such as images, icons, etc) to panoramas has been limited to the integration of two dimensional objects in specific view point images wherein said objects are not visible from distant view points.

• U.S. Pat. No. 6,693,649 to Lipscomb discloses a solution for non-linear mapping between media and display allowing “hotspots”, defined as an outline of two points connected by straight lines, to be used in the context of panoramas. Such “hotspots” are referenced to each image using two angle's or two pixel's coordinates; such values are only valid for each distinct image.

• Allocation of a third dimension value for virtual objects and determination of precise geographic location information for each view point are prerequisite for the seamless integration of hotspots and other virtual objects in panoramas, where said object would be visible from any point having a direct line of sight to the object. Consequently, no prior art system provides a system and advanced features based on geographical information such as the ability to pin an element of information on any location in a view, such element staying spatially fixed to the point during the travel. The present method and system for creating and assembling interactive walkthrough applications overcomes these shortcomings as will now be described.

• Given the market need for immersive online applications, what is needed therefore is a system and method for providing seamless, quality, fluid walk-through navigation using any combination of 2D panoramic images, virtual 3D, and optionally, virtual objects or any images or a combination thereof.

• What is needed is an optimized system or method for the real-time construction and broadcasting of panoramic walkthrough applications, which allow the user, from each view point or geographical coordinate along a network of pathway, to have a complete view from a first person point of view, the view covering substantially 360° in field of view. What is needed is such a system or method that combines high rate fluid panoramic imaging broadcasting and the possibility of seamlessly providing higher quality images in which the visual perspective perception based on human vision is preserved.

• What is needed is a system or method for creating and broadcasting interactive panoramic walk-through applications that can combine still panoramic images that can be captured from either indoor or outdoor locations, and virtual scenes.

• Finally, what is needed is a system or method for creating and broadcasting interactive panoramic walk-through applications that can make genuinely interactive functions available and accessible in the images.

SUMMARY OF THE INVENTION

• A system, apparatus and method for creating interactive panoramic walk-through applications having a 2D image acquisition system is provided. The system includes, comprising a holding device such as a vehicle equipped with a camera connected to a catadioptric optical system providing near 360° field of view, a memory device adapted to store data including images and geographic coordinate information related to these images, communication device for transferring data from the memory device to a computer, a fixation and stabilisation system connecting the camera to the holding means for aligning the camera perpendicular to the horizontal plane; a processor and associated software for performing image modification steps; and optionally, 3D virtual images.

• The image capture system includes a location measurement device (GPS), a distance measurement device (odometer) and an inertial measurement unit (IMU) for measuring rate of acceleration and changes in rotational attributes (attitude) of the vehicle, the fixation device or camera.

• The stored data includes the image, date and time of image capture, geographical coordinate information, and other information, notably image reference and image group, so as to identify, for example, a particular district or street, as well as camera settings such as aperture, and image-related information such as camera model, speed and ISO reference.

• The image modification steps include the steps for providing panoramic images based on two points perspective, of unwarping images, vertical stretching of image proportional to the divergence of the field of view to the horizon of the optical system, and expanding horizontal edges. The image modification steps optionally include the step of automatic blurring of portion of images, such as faces or numerical car plates.

• Software operates a computer to perform image modification steps such as processes images in two resolutions, where low resolution is used for interactive walkthrough panoramic motion, and high resolution for interactive panoramic motion.

• According to another preferred embodiment of the present invention, preservation of visual perspective based on human vision is provided, notably by use of two points perspective that does not produce the trapezoidal distortion that is inherent with standard 360° environment interactive applications.

• It is an object of the invention to provide a system and method for providing immersive, interactive and intuitive walk-through applications using 2D panoramic true images, virtual 3D images or a combination of both, that can provide seamless quality walk-through navigation and high quality imaging.

• It is another object of the present invention to provide a system or method that combines high rate panoramic imaging broadcasting and the possibility of seamlessly providing higher quality images in which visual perspective based on human vision is preserved.

• It is another object of the present invention to provide a system or method for creating and broadcasting interactive panoramic walk-through applications that can combine indoor and outdoor images, based on 2D panoramic images and virtual 3D images.

• It is another object of the present invention to provide a system or method for creating and broadcasting interactive panoramic walk-through applications that can provide genuinely interactive functions accessible in the images.

BRIEF DESCRIPTION OF THE DRAWINGS
• FIG. 1

  is a schematic side view of a catadioptric (mirror-based) panoramic optical system.

• FIG. 2

  is a schematic view of a panoramic image made using the catadioptric based panoramic optical system of

  FIG. 1

  .

• FIG. 3

  is a schematic side view of a lens based panoramic optical system.

• FIG. 4

  is a schematic view of a panoramic image made using the lens based panoramic optical system of

  FIG. 3

  .

• FIG. 5

  is a flow chart showing the unwarping modification steps.

• FIG. 6A-6C

  are schematic views of a panoramic image, made from either systems of

  FIG. 1

  and

  FIG. 2

  , following each of the unwarping modification steps of

  FIG. 5

  .

• FIG. 7

  is a schematic view of a panoramic image before and after modification steps to compensate for vertical distortion.

• FIG. 8

  is a side view showing the vehicle, image capture system, measurement means and attached odometer over a schematic ruler representing distance of vehicle' travel along a road.

• FIG. 9

  is a schematic view of the data storage apparatus of the present invention.

• FIG. 10

  is a floor plan view showing the division of a panoramic virtual 3D image into four sections to be compatible with standard field of view (<179.9˜°) in standard 3D applications.

• FIG. 11

  is a schematic view showing four images resulting from the rendering of a 360° field of view in a standard 3D application by the rendering of section step, providing the division of a 360° panoramic field of view (FOV) on four 3D images of 90° of FOV of

  FIG. 10

  .

• FIG. 12

  is a schematic view showing the steps of assembly and panoramic distortion over the images of

  FIG. 11

  , wherein

  image

  250 shows an expansion of horizontal edges,

  image

  252 shows a distortion of image to compensate for standard perspective where image are projected on a plane, and

  image

  254 shows a cut of image bulb, discontinuous section of the image, at top and bottom of the image.

• FIG. 13

  is a schematic view of a panoramic image of

  FIG. 7

  after modification steps to expand horizontal edges. Same step is typically applied to image 254 of

  FIG. 12

  .

• FIG. 14

  is a flow chart showing the two points perspective distortion steps.

• FIG. 15

  is a schematic view showing the steps of modifying a panoramic image in order to provide a resulting image based on a two points perspective similar to human vision.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS Image Capture System

• Referring now to

  FIGS. 1-4

  and 8-9, an

  image capture system

  20 consists of a

  panoramic optic

  30, 30′, a

  camera

  40 and a

  memory device

  50 such as a computer, mounted on a

  vehicle

  70 or other portable holding device.

• The

  panoramic optic

  30, 30′ is a physical panoramic optic providing 2D panoramic images. The optic 30, 30′ including either, a “lens and mirror” based optic system (catadioptric system) 32, 38, 42 as shown in

  FIG. 1

  , or a physical optical panoramic system 33 (consisting of an ultra wide angle lens or fisheye system with lens providing more than 200° of continuous vertical field of view), without mirror, as shown in

  FIG. 3

  . Both

  systems

  30, 30′ are commercially available and reflect the substantially 360 degree panoramic field of view into the lens based optics connected to

  camera

  40.

• The mirror shape and lens used is specifically chosen and disposed such that the

  effective camera

  40 maintains a single viewpoint. Such a lens is available from Nikon, Canon, and other vendors for example Bellissimo Inc, Carlson Nev. (www.0-360.com). The single viewpoint means the complete panorama is effectively imaged or viewed from a single point in space. Thus, one can simply warp the acquired image into a cylindrical or spherical panorama.

• Optionally in the catadioptric system, the image capture system can use a panoramic lens optic (not shown) commercially available or specifically designed to be adapted to the

  image capture system

  20 of the present invention. This panoramic lens optic is composed of a lens assembly that distorts the field of view (FOV), wherein the FOV is expanded (on the geometric opposite) to cover an additional 90° in all directions, from the camera's original FOV, providing in total a FOV of at least 180° of vertical FOV, and ideally at least 240° of vertical FOV. Such a panoramic lens allows more vertical field of view than a catadioptric system, but accentuates chromatic aberration and is more vulnerable to dust, drops and lens flare artefacts.

• The most significant advantage of

  catadioptric systems

  32 and physical optical panoramic systems 33 (mono camera system) over multi-camera image capture systems, is to provide image almost free of chromatic aberrations or discontinuities (breaks). Moreover, since a complete panorama is obtained on each image shot, dynamic scenes can be captured.

• A first advantage of a physical optical panoramic and catadioptric system over multiple camera systems is that the former avoids the need to stitch multiple images to create a full panoramic image and image color and exposure are consistent inside one point of view over the 360° range. A second advantage is that the geometric nodal point does not need to be simulated, as is the case with stitched motion images. Moreover, with a physical optical panoramic or catadioptric system, the accuracy of an object's geometry in the image is not relative to it's distance from the camera. In a multiple camera system where the nodal point is simulated by software techniques, objects located proximate to the camera are discontinuous and produce ghosts images and artefacts over the resulting panoramic image.

• The

  camera

  40 used in the

  system

  30, 30′ can be any kind of imaging device (e.g., a conventional camera with chemical exposition film, video camera, etc), but is typically a high resolution digital camera, having CCD or CMOS captors of typically 12 Megapixel of resolution or more, with controllable aperture and fast response time of typically 4.5 images/second or more.

• Fast response time is required to obtain still images during image acquisition while the

  vehicle

  70 is in motion. Speed of displacement of the vehicle varies; it is typically of 10 km/h and 2.5 km/h for respectively outdoor and indoor image acquisition applications; this provides for a resolution of three to four or more images per meter (at 2.5 km/h) for indoor applications to one image per meter for outdoor applications (at 10 Km/h). Typical speed and number of image per meter disclosed in this document are provided by example and do not constitute a limitation to the applicable field of the 1.5 present invention. Images can be captured at higher vehicle velocity; during which, satisfactorily images can be captured using a camera with a better sensitivity or a lower image resolution or at a lower capture rate allowing fewer view points along a pathway. Identical or a higher number of view points may be captured using a faster capture device. Higher or lower density (images per meter) may be achieved based on requirements of the specific application field, and on hardware evolution.

• For a low light environment, image bracketing and high dynamic range (HDR) image processing is used to enhance the dynamic spectrum of the final image. Image bracketing and HDR is a technique used to mix multiple images of the same viewpoint, each image having different exposure time. Image bracketing and HDR require the immobility of the vehicle during image acquisition, the drawback being a slower image capture process.

• The

  digital camera

  40 is coupled to a

  catadioptric optic system

  32 by an

  optic apparatus

  42 such as is commercially available from manufacturers such as Nikon and Cannon, then by a

  standard connector

  38 provided by the catadioptric lens manufacturer for each proprietary optic mounting format. The

  digital camera

  40 could also be coupled with the

  panoramic lens

  33 by an

  optic apparatus

  43 that is commercially available from the above-mentioned manufacturers

• The memory device, such as

  computer

  50, receives and stores the images transferred from the

  camera

  40, together with other information received from

  measurement device

  60 such as geographic coordinates (including altitude) related to the images as well as geometric orientation, acceleration, rate of rotation on all three axes (attitude) and travel distance information of the capture vehicle and/or the measurement device.

  Memory device

  50 is typically a computer installed with an operating system, proprietary software and a logical device with multiple processing cores and/or CPU arrangement. The proprietary software manages the distribution of the load of image acquisition to multiple threads on multiple CPUs, processing cores or logical processing units. The distribution of load is achieved by attributing the processing work of each subsequent image to another CPU core or logical processing unit. In a

  computer

  50 having four logical processing units, a typical image capture sequence according to the present invention would be processed in sequential order as follows: (i) first image's acquisition processing work is performed on “

  logical processing unit

  1”, (ii) second image's acquisition processing work is performed on “

  logical processing unit

  2”, (iii) third image's acquisition processing work is performed on “

  logical processing unit

  3”, (iv) fourth image's acquisition processing work is performed on “

  logical processing unit

  4”, (v) fifth image acquisition processing work is performed on “

  logical processing unit

  1”, (vi) sixth image acquisition processing work is performed on “

  logical processing unit

  2”. The distribution of acquisition processing across multiple CPUs increases the rate of image acquisition in a given time, resulting in better image acquisition performance and accrued image acquisition reliability.

• Images are distributed from

  memory device

  50 to

  multiple storage devices

  52 to achieve the high data bandwidth required by the in-motion image capture method of the present invention.

  Memory device

  50 and

  multiple storage devices

  52 are located

  onboard vehicle

  70 or remote thereto. A

  communication device

  54 allows transfer of data from the

  memory device

  50 to a

  central computer

  80. Data is stored in a

  source database

  400 on the

  central computer

  80, wherein each image has a unique identification. Each unique image ID is associated with a specific time reference being the

  database

  400, the time reference is the time when the image is captured. Because the time reference needs high precision, the time reference is given as universal time reference, provided by the GPS unit, which is typically more precise than the internal computer clock. With the time reference, the image capture location can easily be retrieved.

Data Measurement Means

• The

  measurement device

  60 mounted on the

  vehicle

  70 comprises a

  GPS tracking device

  62 or a similar device able to determine geographic coordinate information from a satellite signal, radio signal or otherwise. Each image is recorded in the

  memory device

  50 or on a

  central computer

  80 with the associated geographic coordinate information, namely of the location of image capture, which is stored either on a dedicated recording device or on the

  memory device

  50 such as an on-board computer or on a remote

  central computer

  80. Data is transferred using a communication protocol such as USB, Bluetooth, Ethernet, WiFi, and stored on the destination apparatus in any standard database format.

• Geographic coordinates, also referred herein as “GPS data” 162, are stored with specific GPS universal time reference to images, allowing the determination of the exact geographic location at which each image was taken.

  Memory device

  50 is synchronised to the GPS clock to store universal time reference with any stored data. A system and method for the integration of virtual objects in an interactive panoramic walk through application, using a method for determining geographic coordinates with increased precision is described in PCT application entitled SYSTEM AND METHOD FOR THE PRECISE INTEGRATION OF VIRTUAL OBJECTS TO INTERACTIVE PANORAMIC WALK-THROUGH APPLICATIONS, by Lindemann et al. which is being concurrently filed with the instant application and is incorporated by reference hereto.

• Because GPS devices have limited precision in altitude tracking, other devices such as an altimeter or any altitude tracking device can be use in adjunction to GPS device to enhance the precision of altitude tracking of images. Further, because GPS devices have limited precision in direction tracking, direction can be obtained from an electronic compass or other direction tracking device, thereby enhancing the precision of the recorded path of image.

• An

  odometer

  66 is connected to

  vehicle

  70 for indicating distance traveled between any two image locations, thus improving the precision of the geographic coordinates associated with each image.

  Odometer

  66 may be an electronic or mechanical device. An Inertial Measurement Unit—

  IMU device

  67 on

  board vehicle

  70 detects the current rate of acceleration of the vehicle as well as changes in rotational attributes (attitude), including pitch, roll and yaw. Such data may be used to correct image inconsistencies caused by travel over uneven surfaces. It should be noted that the vehicle's acceleration or speed does not affect the capture density, number of image along a pathway, because successive images are automatically triggered as a function of the distance value between any two successive images, as provided, for example, by GPS data,

  odometer data

  166 and/or

  IMU data

  167.

Vehicle

• The

  host vehicle

  70, for image capture of outdoor locations, can be any vehicle adapted for circulation on roads, namely cars, trucks, or any vehicle adapted to limited circulation areas or indoor circulation such as golf carts, electric vehicles and mobility scooters (scooters for the disabled), etc.

  FIG. 8

  shows

  vehicle

  70 as a small car. For typically smaller, steep roads as well as image capture of indoor locations, remote controlled vehicles, unmanned vehicles, robots, and stair-climbing robots, can be used as the

  host vehicle

  70 for the

  image capture system

  20. In a miniaturized version, the

  Image capture system

  20 can also be carried by a human and for some special applications, small animals such as rats. Where the terrain is difficult, a flying machine can be used, in which case an odometer is emulated by the use of GPS data or/and triangulation using radio signals. Triangulation techniques using radio signals require that two or more emitters be located at known positions.

Logical Means for Performing Image Modification Steps

• According to a preferred embodiment of the present invention, source images 210 (2D panoramic images) from the

  image capture system

  20 are modified using a computer with a logical device such as

  central computer

  80. Images modification steps comprise the steps of

  unwarping

  312, compensation of

  vertical distortion

  322, expansion of

  horizontal edges

  332 and the resolution of two

  point perspective distortion

  342, in order to obtain

  release images

  280 that can be broadcast by

  web server

  82.

• As shown in

  FIGS. 1

  , 3, 5 and 6A-6C,

  source images

  210 obtained by a

  panoramic optic

  30, 30′ are typically circular in shape, referred in the art as annular images.

  Unwarping

  312 of

  source images

  210 is achieved using conventional software techniques, so as to form cylindrical images. The unwarping operation is typically performed in three consecutive operations: in a

  first operation

  313, the image is centered and aligned relative to a grid consisting typically of geographic direction (North-East-South-West axis) or corresponding degrees of the 360 degrees panorama, where the North axis direction is referred to as 0 degree for convenience; in a

  second operation

  314, the circular shaped image is opened up after “slicing” a section from the image center to an edge, typically the bottom point of the image on the “South” or 180 degree direction the resulting

  image

  211 is in the shape of a circular arc; and in

  third operation

  315, the circular arc is further unwarped to form an

  unwarped image

  212 of rectangular shape. For ease of understanding, images of

  FIGS. 6A-6C

  , 7 and 13 show section marks on the vertical axis where the “A” mark indicates the South or the 180 degree direction, “B” indicates the East or the 90 degree direction, “C” indicates the North or 0 degree direction, and “D” indicates West or the 270 degree direction.

• As shown in

  FIGS. 2 and 4

  , the field of

  view

  34 of catadioptric optic is typically un-evenly distributed along the vertical axis of the mirror, causing optical deformations, where the

  image

  212 appears compressed near the upper and lower edges of the image. To correct this vertical unevenness, compensation of

  vertical distortion

  322 is performed as shown in

  FIG. 7

  , and involves modifying each

  unwarped image

  212 by software operation to compensate for the compression of the upper and lower edges of the images, to provide a resulting

  image

  222 that is typically larger along the vertical axis, compared to the original

  unwarped image

  212.

• Compensation of

  vertical distortion step

  322 is performed by applying a function curve to affect pixel distribution along the vertical axis. In a first sub-step, an empty data buffer corresponding to a

  destination image

  222 is set-up, resulting in a blank image. In a second sub-step, correspondence between coordinates in the

  source image

  212 and

  destination image

  222 is determined using a function curve. In a third sub-step, pixel values (color, hue, intensity, CMYK or pixel RGB values) at given coordinates of

  source image

  212 are copied to pixel values at corresponding coordinates on

  destination image

  222 on a pixel per pixel basis. On the third sub-step, at least one image sampling method such as “median”, “summed area”, “bilinear” or “trilinear” known image sampling methods is applied to obtain sub pixel data whenever a pixel's coordinates, size and shape on the

  original image

  212 do not match resulting pixel's coordinates, size and shape on the

  destination image

  222. One resulting pixel can match a source pixel exactly or can be a variable portion of one or several source pixels), thereby allowing the destination image to have different vertical resolution (number of pixels) as compared to

  source image

  212.

• The function curve is determined by either measuring the curvature of different optic elements of the

  panoramic optic

  30, typically the main lens or mirror, or by measuring the vertical distortion produced by said optic elements. Vertical distortion is measured by acquiring the image of a calibration model (a 2D image) at a given distance with the image

  panoramic optic

  30 and

  camera

  40 in a calibration room having visible demarkations (for example, a geometrically precise grid painted on a room wall). This helps determine distance discrepancies between the image and the calibration model. In other words, on the

  destination image

  222, a measurement unit (for example 1 vertical meter) at a given distance (for illustration purpose a measurement unit of 1 vertical meter at the distance between a tree, 213 on both

  images

  212 and 222, and the position of the capture optic 30) fills the same number of pixel near the horizon and near the top or bottom of the image. This measurement unit which fills a number of pixels N near the horizon of the image, will fill the same number of pixels N at the top or bottom of the image. On the

  unwarped image

  212, same measurement unit at same distance will fill fewer pixels near the top and bottom of image compared as to the horizon (image vertical middle).

Use of Virtual 3D Images Instead of Real Camera Image.

• The

  system

  20 of the present invention also allows the use of 3D virtual images, alone or in combination with 2D (optical) panoramic images, for the purpose of creating and broadcasting interactive panoramic walk-through applications. Current commercially available 3D rendering software engines used in CAD (“Computer aided-design”), 3D modeling and gaming applications are not meant to provide panoramic images having near 360 degree horizontal field of view. As shown in

  FIGS. 10 to 15

  , the computation of a virtual 360 degree panoramic image is thus typically achieved by transforming a

  3D scene

  240, originating from a 3D modeling, for example, into a panoramic 254 image equivalent to the resulting

  image

  222. Because standard 3D software is not meant to render image that have an horizontal field of view larger than ˜179.9°, the computation of a virtual 360 degree panoramic image is thus achieved by dividing the 360° field of view in 4 sections of 90° fields of view, which in

  FIG. 10

  is represented by

  numbers

  1, 2, 3 and 4.

• According to a preferred embodiment of the present invention purpose is to increase image broadcasting speed and overall system performance, computing a virtual 360 degree panoramic image is performed, for each

  3D scene

  240, in a

  first step

  253 by rendering four images, called sections 244 (numbered 1 to 4 on each of

  FIGS. 10-12

  and 15), having a 90° horizontal field of view from a single

  nodal point

  241 and each

  section

  244 pointing toward one of the four right angle directions (front, left, back and right). From the

  nodal point

  241 each

  section

  244 is thus oriented, at a view angle of 90°, from the preceding and following sections. The combination of the four

  original sections

  244 can be represented by a single

  rectangular shape image

  250. In 3D space, the combination of the four 90° view sections can be used to define the inner faces of a box; consequently,

  image

  250 represents the juxtaposition over the same plane of the four inner faces of a box viewed from a central position inside this box.

• As disclosed by US 2007/0211.955 to Pan with respect to panoramic imaging, and as is known in 3D imaging, the projection of wide angle or panoramic images over a cylindrical field of view reduces the number of computing steps necessary to provide real-time compensation to image distortion, and thus eases the creation of interactive panoramic motion. This is due to the fact that, in this kind of image, each image section has the same proportion ratio independently of the angle of viewing angle.

• In a second step, according to a preferred embodiment of the present invention, as shown in

  FIG. 12 image

  250 is modified as if

  image

  250 was then projected over a continuous surface of cylindrical shape. This image modification is done by stretching vertically the resulting

  image

  250, section per section, along a sinusoidal curve using software which affects the sinusoidal stretching logical step 352, and resulting in modified

  image

  252. This sinusoidal stretching logical step 352 is described in detail below.

• In a first sub-step an empty data buffer corresponding to the

  destination image

  252 is setup (resulting in a blank image).

• In a second sub-step, the sinusoidal stretching logical step 352 is performed by evaluating resulting images pixels using a sinusoidal function curve that points on the original image pixels. This function curve computes the vertical coordinates in the

  destination image

  252 to determine the corresponding coordinates in the

  source image

  250. This sinusoidal function curve decreases its amplitude for vertical coordinates that are farther from the top edge of the section, the amplitude reaching a null value (0) at the vertical center of the section, then the amplitude continues to decrease as negative number and reaches the exact opposite value at the bottom edge of the section. This sinusoidal function curve is calculated from the start of an

  image section

  244 to the end of the same section. The same function curve is used horizontally for each section, which results in an

  image plane

  252 that contains 4 sections stretched in the same way regarding to the section local coordinate system.

• In a third sub-step, pixel values at a given coordinate of

  source image

  250 are copied according to pixel color value at corresponding coordinates (given by said sinusoidal function curve of the second sub step) on the

  destination image

  252, on a pixel per pixel basis. During the third step, at least one image sampling method such as a, “median”, “summed area”, “bilinear” or “trilinear” known image sampling method is applied to obtain sub pixel data, whenever the pixel's coordinates, size and shape, on the

  source image

  250 do not match the resulting pixel's coordinates, size and shape on the destination image 252 (one resulting pixel can match a source pixel exactly or can be a variable portion of one or many source pixels), allowing the

  destination image

  252 to have different vertical resolution (number of pixels per mm) compared to

  source image

  250. Pixel values can be either color, hue, intensity values, CMYK (Cyan, Magenta, Yellow, Key) values or pixel RGB (Red, Green, Blue) values.

• The sinusoidal stretching step 352 is performed to compensate the distance difference (i) between the view point and the pixels laid over a flat surface (the face of a box discussed above) and (ii) between the view point and same pixel laid over a cylindrical surface (the inner surface of a cylinder as viewed from within). In geometric perspective, the area, filled with the projection of an object, grows in the inverse proportional manner as the distance between said object and the view point. The center of an edge of a flat surface (any face of the box) is located closer to the viewpoint, compared to the center of an edge of an equivalent cylindrical surface (as if the box would be contained within the arc of a cylinder). In the geometric projection of a box to a cylinder, the upper and lower edges of each face of a box projected from the common center of the box and the cylinder become an arc that follow a portion of a sinus curve. Any pixel of the image is then positioned in the resulting

  image

  252 relative to its vertical distance between said pixel and top and bottom edge of the box, at the same relative distance to the top and bottom edge (arc) of the cylindrical projected box. Top and bottom edges of the projected faces of the box to a cylinder are comparable to top and bottom edges in the modified

  image

  252.

• In a third step, a

  logical step

  354 is performed which removes the circular area on the upper and lower edges of each image's section of

  image

  252, so that ultimately a single

  rectangular shape image

  254 is created.

• In order to increase image broadcasting speed and overall system performance, each image 222 (at this stage,

  image

  254 and

  image

  222 can be processed by the system in the same manner), which is either a panoramic image from the

  image capture system

  30 or an image resulting of the modification of a 3D image, is further modified by the expansion of horizontal edges

  logical step

  332 shown in

  FIG. 13

  .

  Image

  222 is typically divided in two sections. The left edge of

  image

  222 is cropped, copied and pasted on the opposite edge of the image to form expanded

  image

  232. In other words, the left border section of each image is sliced in a sub-section on the vertical axis, and the edge sub-section is cropped and displaced on the opposite edge of the image, so that two identical sub-sections of the image appear on both opposite sides of the image. The expansion of horizontal edges step 332 reduces the processing workload necessary for the real-time performance of the two points

  perspective distortion step

  342, allowing an output rate typically of 15 or more images per second. This rate is sufficiently fast to create the illusion of a seamless motion, without jerky movements, during displacement (image panoramic motion or rotation of the view) within the image. Increase of system performance is achieved because the expansion of

  horizontal edges

  332 avoids need for the image broadcasting system to manipulate two images simultaneously. Otherwise, this is the case every time the interactive panoramic reaches a vertical image split at 0° or 360° image borders. The system of the present invention achieves perpetual panoramic motion without slowdown: when the border of an image is reached, the image broadcasting system simply displays the other side of the same image, in an invisible manner to the user.

• A virtual walk-through application providing a convincing natural, immersive navigation environment should be visually as close as possible to human vision during panoramic motion (view point rotation, horizontal displacement inside one panoramic image), or translation motion (view point moving in space forward or backward on the pathway), irrespective of motion speed.

• According to a preferred embodiment of the present invention, a panoramic image closer to human vision during panoramic motion, with optimised calculation performance is provided by the use of two-point perspective distortion. During vertical panoramic motion, the image is vertically compressed depending on vertical elevation (distance to the horizon) in order to compensate for significant horizontal distortions which occur further from the horizon. Such distortion is systematically caused by the cylindrical projection of the image and is more severe toward the vertical lower and upper parts of the image. The present invention provides compensation of horizontal distortion similar to the projection of a cylindrical view to a planar view. In order to speed up image modification steps to real time, the present invention departs from the traditional projection of a plane inside a cylinder by assembling an image from the juxtaposition of several vertical image strips 235, each of them optionally having a different width, and by stretching said strips. This avoids the need to recalculate the position of each pixel of the image according to the projection of a plane inside a cylinder. Only a selected vertical and a horizontal portion of the image are displayed and visible to the user at any given time. The method of the present invention described below works in a “staircase-like” manner (visible on the zoomed

  image

  236 of

  FIG. 15

  ) whereby the width of each stripe is controlled and reduced and the number of stripes increased to provide the illusion of a progressive curvature that is a perspective closer to human vision.

• For this purpose, the

  panoramic images

  232 are further modified by the two points

  perspective distortion step

  342 in order to provide images based on a two points perspective. The two

  points perspective distortion

  342 step is achieved by applying two basic operations, as shown in

  FIG. 15

  , allowing vertical parallel lines in the real world to appear as parallel lines on screen during the panoramic motion and interactive navigation.

• In a first operation 343 (shown in

  FIG. 14

  ), the

  viewable portion

  234 of any

  image

  232 viewed by a user on screen, is divided into vertical slices 235 (stripes) in real time, which are each vertically stretched along sinusoidal lines in real time to recover a two points perspective. The sinusoidal virtual canvas (the vertical slices) is in a fixed position relative to the viewport which is the rectangular window by where the image is viewed by the user. As the user pans his view during navigation, the

  viewable portion

  234 travels across fixed

  vertical slices

  235, during which time only a portion of the panoramic image is visible on screen.

• In a

  second operation

  345, vertical panoramic interactive motion is achieved by simple vertical translation of

  vertical slices

  235 without affecting sinusoidal distortion (that is the relative position and relative size of each stripe regarding each other stripe). All the

  slices

  235 are then vertically compressed proportional to the angular divergence of the field of view to the horizon, in order to minimise vertical distortion of the two points perspective. The two

  points perspective distortion

  342 avoids trapezoidal distortion found on a regular three point perspective used in 3D computer graphics.

• The image modification steps of the present invention processes images in two resolutions; low resolution for interactive walkthrough panoramic motion (view point translation), and high resolution for interactive panoramic motion (view point rotation) applications.

• Additional image modification steps are possible, through the use of a logical processor.

• In order to ensure protection of privacy and to avoid infringement to privacy rights and right to images, the system of the present invention can include logical means for the automatic blurring, masking or removal of objects appearing on images, such as human faces and car plates. Automatic blurring of faces is achieved by a face recognition algorithm. Using image tracking software known in the art, each individual face is identified and tracked throughout any sequence of images. Based on the location of faces or image-based coordinates provided by the tracking software mechanism, software blurs the images on the coordinates corresponding to recognizable faces. Automatic blurring of car plates is achieved by a car plate recognition algorithm. Using image tracking software known in the art, each individual car plate is identified and tracked throughout any sequence of images. Based on the location of car plates or image-based coordinates provided by the tracking software mechanism, a software blurs car plates on the coordinates corresponding to recognizable car plates. Identical processes can be used to mask objects from images. The processes above are not limited to faces or car plates but can be used to blurred or mask any objects specified by the system's operator, with corresponding software adaptation.

Management of Images

• According to a preferred embodiment of the present invention, panoramic resulting

  images

  232 are stored on the

  central computer

  80, each image being associated with tridimensional (XYZ) geographic coordinates indicative of the image's location of capture and optionally other references such as date of capture, with day and time; project or name of location; image group, so as to identify for example a particular district or street; digital photography settings, such as aperture setting, speed, ISO (measure of the light sensitivity of the digital sensor), exposure, light measure, camera model; identified for mounted photos filters such as UV, neutral gray, etc.; vehicle information such as vehicle speed and model; and camera operator identification.

• Precise reference to the geographic coordinates indicative of the image's location of capture allows, among other things, the combination or the superposition of panoramic images of the present invention with any other heterogeneous geo-referenced digital information. Views of a street can therefore be combined with data of a GIS (“Geographic Information System”) application, such as, for example, layers of the utilities' services infrastructure (water, cable, electric distribution infrastructure), or contact information associated with commercial establishments located in the vicinity, PCT application No. ______, to Lindemann et al, entitled SYSTEM AND METHOD FOR CREATING AND BROADCASTING INTERACTIVE PANORAMIC WALK-THROUGH APPLICATIONS, filed concurrently with the instant application and incorporated by reference hereto, discloses a method for the assembly and broadcasting of interactive panoramic walk through applications, using an object-based database. Still further, PCT application No. ______, to Lindemann et al, entitled SYSTEM AND METHOD FOR THE PRECISE INTEGRATION OF VIRTUAL OBJECTS TO INTERACTIVE PANORAMIC WALK-THROUGH APPLICATIONS, filed concurrently with the instant application and incorporated by reference hereto, discloses a method for the integration of virtual objects to interactive panoramic walk through applications, using a method for determining geographic coordinates with increased precision. The foregoing PCT applications provide functionality which enhances the user experience or increases the efficiency of the

  system

  20 of the invention.

3D Computer Graphic Layering

• According to a preferred embodiment of the present invention, the system and method for creating panoramic walk-through applications herein provide image modification steps that enable the seamless and transparent integration of panoramic images (from a panoramic optic 20) with virtual (3D) images, wherein virtual images and panoramic images can be combined in a plurality of ways, for example, to mask or super-impose each other. It is therefore possible to combine panoramic images, virtual images and 3D objects without limitation in any panoramic view; the resulting view having a seamless appearance during walkthrough interactive motion and panoramic motion.

• The system of the present invention positions each image and virtual objects on a 3D space using absolute geographic coordinates. Any panoramic image is edited from either a

  source image

  210 or a

  virtual image

  240 has geographic coordinates in 3D space. Due to the use of absolute coordinates, the system of the present invention allows the addition (pining) of virtual 3D objects to any panoramic image or virtual image.

• Masking operations are achieved by the system in which a mask is defined, that is, the representation in shape and volume of a real object appearing on the panoramic images is defined. This technique is well known in the art and used to integrate 3D objects in photos in the real-

  estate

  3D industry, wherein said technique is limited to static view. Effectively, the mask position and orientation are correct for a given 3D object only for a given view point.

• According to an embodiment of the present invention, the use of absolute 3D space coordinates for referencing both panoramic images and virtual images inside the 3D space allows any given 3D mask to be used on any panoramic image in a pathway. The mask becomes visible from any view points that have a view on the mask's geographical coordinates.

• The 3D mask is designed to exist at a position and a shape identical to respectively, the 3D position and shape of a real physical object viewable on one or more panoramic images. When the 3D mask applied to cover a real physical object visible on a panoramic image is located closer to the viewpoint of the panoramic image, compared to the 3D object (i.e., located further from the viewpoint), the 3D mask pixels will occlude the pixel of the 3D object; in which case, pixels of the panoramic image representative of the physical object may become visible when the mask is rendered (the mask is “transparent” to an extent where pixels of the 3D object behind the mask pixels also become transparent, revealing the pixel of background panoramic image). In some cases, the mask can be displayed with a virtual shadow from, or cast virtual shadow to, other 3D objects to enhance the realism of the integration of 3D in the real image.

• A 3D mask can be occluded by a 3D object when the pixels representing said 3D object are located, relative to the view line, in front (closer to viewpoint) of pixels of said mask. As the masking process is determined on a pixel per pixel basis, complex situations may arise. For example, a 3D object can be partially occluded by a 3D mask that is partially occluded by another 3D object. The flexibility of the masking process allows a near visually perfect integration of 3D object inside real image used in interactive panoramic walk-through applications.

• According to a preferred embodiment of the present invention, 3D computer graphic layering using masking operations will produce new panoramic 3D layered images based on real world panoramic image composed of pixels from 3D objects and 3D masks. These panoramic 3D layered images are stored in the

  database

  400 with geographic coordinates of the 3D space equivalent to coordinates of the source panoramic image and tagged with a reference to the 3D computer graphic layering program. Said 3D computer graphic layering program reference can be a 3D application source, or a file name, a project name or an application name or both. 3D computer graphic layering can be used to mix 3D images from different 3D rendering packages, or to add objects in the 3D space without the need to re-render the full 3D panoramic images (typically necessary when inserting new objects in already rendered 3D panoramic images). 3D computer graphic layering can therefore be used for insertion and management of advertizing, wherein a virtual advertisement can be inserted by defining a 2D (flat) or 3D object and superimposing an advertising image on said object.

• Image is available in alternative version broadcasted depending on the context: typically, one panoramic representation of a location is edited without any advertising, and several other copies are made available to different advertisers; a software based management tool allows advertisers to select an advertising to be imported in the system; selection and display of advertising to be broadcast can be dependent upon, for example, available user information such as location and language, and the identity of the web portal from which the user is accessing the panoramic application.

• Other characteristics and modes of execution of the invention are described in the appended claims.

• Further, the invention should be considered as comprising all possible combinations of every feature described in the instant specification, appended claims, and/or drawing figures which may be considered new, inventive and industrially applicable.

• Multiple variations and modifications are possible in the embodiments of the invention described here. Although certain illustrative embodiments of the invention have been shown and described here, a wide range of modifications, changes, and substitutions is contemplated in the foregoing disclosure. While the above description contains many specifics, these should not be construed as limitations on the scope of the invention, but rather as exemplifications of one or another preferred embodiment thereof. In some instances, some features of the present invention may be employed without a corresponding use of the other features. Accordingly, it is appropriate that the foregoing description be construed broadly and understood as being given by way of illustration and example only, the spirit and scope of the invention being limited only by the claims which ultimately issue in this application.