Whoa - Apple Wins a 3D Display & Imaging System Patent Stunner

The US Patent and Trademark Office officially published a series of 13 newly granted patents for Apple Inc. today and one was a real stunner. Today's report focuses on this advanced 3D display and imaging system that packs one hell of a wallop. Apple's patent covers a wild 3D system that could generate an invisible space in front of the user that could allow them to work with holographic images or project their hands onto a screen in front of them to manipulate switches or move pieces of virtual paper or parts of a presentation. One could only image how this could be applied to 3D gaming, business or medical applications in the future. This is Apple's second major revelation about such an advanced 3D system and many supporting patent applications would suggest that the system is progressing quite well in Apple's research labs. The good news, is that future iOS devices will be one of the drivers behind this new beast. This is definitely one of Apple's coolest ideas to date.  

Background & Overview of Future 3D Displays & Devices

Modern three-dimensional ("3D") imaging and display technologies are employed in widely diverse environments and technologies. Examples include medical diagnostics, entertainment, flight simulation, robotics, education, animation, biomechanical studies, virtual reality, and so forth. There are numerous 3D input devices including, for example, variations on the computer mouse or touch pad. Ordinarily, these are manipulated in just two dimensions, the x-axis and the y-axis. However, through various computer programming artifices, these devices could provide a measure of control in the third dimension of the z-axis.

In 3D, however, such constructs could be indirect, time consuming, artificial, and could require considerable practice and training to do well. Similar observations could be made relative to joysticks, which in their original function were designed for input on but two angles (dimensions). Other more sophisticated means of 3D input are therefore preferred.

Passive 3D Input

One of the preferred methods of 3D input is passive input, wherein a device automatically detects and measures a target in 3D. A common technique for such 3D measurement uses the time of flight ("TOF") of a scanning light beam. The TOF technique measures the time or duration that lapses or accumulates from the moment of light pulse transmission to the moment of reception of the returning light pulse after reflection by a target object. The TOF is directly dependent on the distance the light pulse has traveled. TOF, however, requires very high-speed measuring equipment for accurate distance determination. Measurements at short distances could be inordinately difficult. Equipment costs and complexity are correspondingly high, making TOF unattractive for ordinary consumer applications.

3D Input via Light Attenuation

Another 3D method for measuring distance utilizes light attenuation. Light attenuation is based upon the fact that, in general, the farther a light beam or light pulse travels, the dimmer the reflected light is when it returns. The difficulty with the light attenuation technique, however, is that different objects are more or less reflective, so the measurement is very dependent upon the reflectivity of the target object. One way to manage the reflectivity is to standardize it, for example, by attaching standardized target reflectors to the target object. Another method is to modulate the light beam and to compare the reflected signals at different light intensity levels. Both techniques, however, are inconvenient and unnecessarily complicated, as well as costly.

What 3D Devices should Deliver for Consumer Products

A need thus remains for uncomplicated, economical, yet highly effective 3D input devices for computers. Such devices need to be able to detect, analyze, and measure objects located in a 3D volume, and to observe and track any motions thereof. The devices should therefore be well suited and designed compatibly for use with 3D graphically intensive activities. They need to be capable of operating by optically sensing object or human positions, orientations, and/or motions. For reasons of cost as well as user convenience, they should be compact and capable of incorporation into a simple, small, single housing or unit. They also need to be versatile, and thus capable of working effectively and beneficially with the full range of conventional consumer appliances.

To be effective, it is also important that such devices be capable, when desired or necessary, of economically but effectively and precisely providing feedback to the user. Thus, such devices should incorporate audio and/or visual mechanisms for reporting to the user the effects and results of the 3D input. And again, for reasons of cost as well as user convenience, such devices should incorporate such user feedback functionality while continuing to be compact and capable of incorporation into a simple, small, single housing or unit.

Thus, a need still remains for economical, small, portable, and versatile multi-functional 3D imaging and display systems that could both scan and display simultaneously. A need also remains for such devices that provide integral, immediate feedback. A further need remains for such devices that are particularly well adapted for use with other, small electronic devices, particularly hand-held and other portable devices.

Solutions to these problems have been long sought but prior developments have not taught or suggested any solutions and, thus, solutions to these problems have long eluded those skilled in the art – until now.

Apple's 3D System Overview

Apple's present invention covers a three-dimensional imaging and display system. User input is optically detected in an imaging volume by measuring the path length of an amplitude modulated scanning beam as a function of the phase shift thereof. Visual image user feedback concerning the detected user input is presented.

Apple's inventor Christoph Krah states that the three-dimensional ("3D") imaging apparatuses that are described in the invention are suitable for head tracking (also see), (hand) gesturing, presence detection, auxiliary display functions, and other capabilities and functionalities that could involve combinations including an assortment of the following components: A collimated high-speed infrared ("IR") or visible laser (e.g. such as used for fiber optic communications) A visible 3-color illumination source and driver circuit A high-speed photo detector (e.g., PIN diode based) A digital signal processor ("DSP") Dual axis scanning device (e.g., analog mirror) and driver circuit Analog subsystem Video subsystem.

Apple's 3D Imaging & Display System

Apple's patent FIG. 1 shown below presents us with embodiment 100 representing the 3D imaging and display system. The embodiment includes a projector 102 and a receiver 104. The projector projects a scanning beam 106 into an imaging volume or space108. The projector also projects a projection beam 110 onto a display area 112.

In one embodiment, the projection beam is utilized to project images onto the display area of various virtual objects that are characterized as being virtually located within the imaging volume or space. These virtual objects may include, for example, knobs, sliders, buttons, and so forth. Images of these virtual objects are then projected by the projection beam onto the display area producing, for example, corresponding knob images 114, a slider image 116, and button images 118.


 
It will be understood, of course, that these virtual elements have no real physical embodiment, and thus do not actually appear in the imaging volume or space. However, depending on the application it's applied to, they could. This would of course support Apple's other 3D patent which describes holographic imaging effects, a patent by the same Apple engineer.

Apple's Christoph Krah goes on to state that the imaging volume is configured to enable a user to interact with the various virtual elements located virtually therein. Thus, a user would place his or her left hand 120 and/or right hand 122 into the imaging volume or space. The hands would then be detected by the scanning beam. A left hand image 124 and a right hand image 126, respectively, would then be projected by the projection beam onto the display area to provide immediate feedback to the user concerning the relationship between the user's hands and the various virtual objects within the imaging volume or space. Thus, as depicted, the user could grasp one of the knob images with the left hand by moving the left hand until the left hand image indicates that the selected virtual knob (not shown) represented by the corresponding knob image has been engaged. The knob could then be grasped and manipulated such as by rotating the knob.

Similarly, a virtual button (not shown), represented by the button images 118, may be pressed by appropriately positioning and moving the right hand under the guidance of the feedback provided by the right hand image 126.

Accordingly, it will be understood that any virtual object could be grasped and manipulated within the virtual space of the imaging volume or space. Such objects, in addition to controls such as knobs, sliders, and buttons, can include virtually any kind of physical objects (e.g., a block of wood, a sheet of paper, hand tools, styli, virtual paint brushes, pencils, pens, grinders, knives, scissors, and so forth). When Apple mentions a "sheet of paper" or moving objects, I find that it fits in rather well with their latest projector system revelations of last month. Patent by patent Apple's vision of an advanced 3D projection system is coming together.

Invisible Light Beam

For the convenience and comfort of the user, the scanning beam in one embodiment consists of an invisible (e.g., IR) light beam. The projection beam is ordinarily in the visible light range, but could be invisible according to the application at hand. An example of the use of an invisible projection beam would be, for example, the projection of an ultra-violet ("UV") beam onto a fluorescent target. The projection beam may accordingly be mono- or polychromatic. In one embodiment, the projection beam would be a red-green-blue ("RGB") beam that would be appropriately modulated, as is known in the projection sciences, to enable the presentation of full color images in the display area.

The receiver 104 receives light reflected from the scanning beam by the user input, e.g., by the left hand and by the right hand of the user. This enables the system of the embodiment 100 to determine and display the exact configurations, positions, and movements of the physical objects (e.g. the left hand 120, the right hand 122, and so forth) introduced into and/or present within the imaging volume 108.

It will be understood, of course, that when the scanning beam isn't in the visible light range, it will not be expected to register true color information concerning the scanned object within the imaging volume or space. However, full color images could still be projected in the display area 112 utilizing, for example, pre-stored information concerning preferred color renditions for detected objects, according to their assigned configurations, and the various virtual objects, such as the virtual knobs, sliders, buttons, and so forth. Alternatively, complexity and costs can be reduced by utilizing monochromatic (e.g., gray scale) projection technologies in the projection beam.

Associated Audio & Feedback

In one embodiment, feedback to the user may also include sounds. For example, turning one of the knob images may be accompanied by a clicking sound corresponding to certain arcs of rotation of the knob. Similarly, appropriate sounds may accompany the pressing of the button images, thereby providing the user with additional feedback and confirmation that the virtual object has been engaged and manipulated as desired.

While the patent emphasizes a projection screen and/or display, it also describes it as being interactive with an iMac desktop display.

The Mirroring System

Apple's patent FIG. 6 shown below illustrates schematic 600 of the 3D imaging's internal mirroring system 302.


 
Apple's Krah states that when the mirror system is used for both the projection beam and the scanning beam, the projection beam 110 and the scanning beam are alternately generated and the mirror system 302 synchronized therewith.

Apple's 3D Imaging System: Unexpected Discoveries of Use and Value

In conclusion, Apple's Krah states that it has been unexpectedly discovered that the present invention has enormous versatility and value. An important utility and value of the invention resides particularly and unexpectedly in the great range of useful applications in which the invention could be exploited, as well as in the great range of devices with which the invention could be advantageously allied and employed. It has thus been discovered that the present invention could be advantageously, economically, and efficiently employed to extend the range of capabilities of numerous existing devices, as well as to afford entirely new functionalities in new applications and environments.

Interestingly, Apple's patent states and clarifies that the 3D imaging and display system would be used with an iPod (a personal music player), an iPod touch (a personal data assistant – or PDA ), other iOS devices (iPhone, iPad) to provide primary or extended display capability, thereby considerably enhancing the usability of the device, and substantially enhancing and improving the user experience.

Some of the examples that Apple provides have already worked through the system and have already appeared in some rather recent patent applications which bolster this patents claims. For example, Apple states that the system could be used a bar code reader or a scanner.  Coincidentally, sources at Apple tell 9to5Mac that an app called the "scanner" is in the works for iOS devices. Talk about timing.  Another application foreseen deals with "surveillance" – though Apple provides no examples of how that would work.

A Virtual Keyboard Application

Another future application could involve a virtual keyboard application. Apple states that some devices like the iPod touch or iPhone have a tiny virtual keyboard. Apple's invention states that it could solve that problem by generating an interactive virtual keyboard by detecting finger locations and movements, responding accordingly thereto, and, when desired, projecting an associated virtual keyboard image. The concept of the virtual keyboard and more is shown in the YouTube video below which came via Cult of Mac's Alex Heath's report.   

Painting a Moveable Object and Following It As It Moves

In another example, Apple points to an advanced Paint centric application. In one embodiment, the 3D imaging and display device could be used to draw 3D images. For example, the user could place a finger at a specific location within the imaging volume or space, which location then could be selected by the user as the vertex of a 3D shape. Similarly, the user could then select other vertices to complete the 3D shape. Both hands could be used to perform point and selection functions. For example, the index finger of the left hand could be used to point to certain locations inside the imaging volume or space, while the index finger of the right hand could be used to select those locations as vertices (e.g., by pressing a virtual button corresponding to a button image). During all these operations, the auxiliary display function would be providing immediate feedback. The right hand could also be used to select other functions, such as filling the 3D shape with a certain color or manipulating the shape by moving the vertices, or rotating or relocating (displacing) the shape, and so forth.

Think Gaming

When reviewing Apple's patent about moving objects and/or interacting with things like knobs and switches on a display, I just couldn't get over how Apple didn't utter a single word about how this could revolutionize some first-person 3D shooter-like games. Think about how some games require you to punch a code onto a security pad in order for you to get through a needed door; or require you to solve an interactive puzzle; or require you to type in information on a old typewriter or computer keyboard. With this type of 3D technology, you'd be able to reach into the invisible space with your hands and interact with the game on the screen. It could also change the way that you play board games on your HDTV. It's not hard to envision Apple's 3D technology being ideal for gaming. 

Another possible application could assist a surgeon discussing a patient with another doctor long distance. Having a holographic space to work in would allow a surgeon to map out a game plan for surgery with such a 3D modeling system.

Today's 3D Display and Imaging system patent has the potential of delivering multiple ground breaking applications in the future. One could only imagine what Apple's developer community will be able to dream up with these kinds of capabilities. While we're obviously not there yet, several off-shoot patent applications have already surfaced as we've pointed out earlier in our report. Other interrelated projector system patents could be found here.


Apple's patent FIG. 4 noted above is an illustration of a calibration, detection, and measurement procedure; Patent FIG. 5 is an illustration depicting the detection of an object; Patent FIG. 11 is a block diagram of a dual laser modulation-demodulation configuration.

Today's granted patent was published the USPTO and credits Apple's Christoph Krah as the sole inventor. The original filing was made in Q4 2005. To review all of Apple's forty claims, see granted patent 8,018,579.

Apple Wins their Second Design Patent for a Never Released iPod

The US Patent and Trademark Office have officially granted Apple a design patent for a never released iPod or related handheld. The first patent win for this design was reported on in July 2010. Unfortunately, design patents carry no description details, so we're left in the dark as to what Apple could be thinking of here. Although it may have simply been filed as a protective measure on Apple's part so as to ward off iPod copycat designers, anytime Steve Jobs is listed as one of the inventors, you have to take note that it could still be a viable future design. For instance, two of the handheld designs presented in the original granted patent noted above, carry some similarities to the "projector" and "receiver" devices noted in Apple's 3D Display and Imaging System described in today's report.

 
Like any design patent, the broken lines are noted as forming no part of the design. That means that the surface of this unit wouldn't come with a physical display window.

Apple credits CEO Steve Jobs, VP Industrial Design Jonathan Ive and team members Bartley Andre, Daniel Coster, Daniele De Iuliis, Richard Howarth, Duncan Kerr, Shin Nishibori, Matthew Dean Rohrbach, Douglas Satzger, Calvin Seid, Christopher Stringer, Eugene Whang and Rico Zorkendorfer as the inventors of Granted Patent D645,037. The latest filing is noted as July 2010, though the history of the design dates back to January 2009.

Notice: Patently Apple presents only a brief summary of granted patents with associated graphics for journalistic news purposes as each Granted Patent is revealed by the U.S. Patent & Trademark Office. Readers are cautioned that the full text of any Granted Patent should be read in its entirety for full details. About Comments: Patently Apple reserves the right to post, dismiss or edit comments.

Here are a Few Great Community Sites covering our Original Report

MacSurfer, HardOCP, Facebook, The University of Tennessee, Accelerate Your Mac, CNN Money, Business Insider, StockTwits, Investor Village, Twitter, Reddit, Apple Investor News, Google Reader, Gelearii Soup - British Indian Ocean Territory, Schubin Cafe, UpgradeOSX, Blues News, MacHash, Mac MajorGeeks, Engadget, TechWatching, Macnews, iPhone World Canada, CBS MarketWatch, iPhoneclub Netherlands, TidBITS, Yahoo! News, Techmeme, Gizmodo, 9to5 Mac, Cult of Mac, MacDailyNews, SlashGear, ModMyi, iPhoneItalia Italy, Golem Germany, iClarified, MacNN, Entertainment Technology Center, Geeky Gadgets, The Independent (Apple's Vision) UK, MacPlus France, Gizmodo Spain, Melamorsicata Italy, Gizmodo France, Apple Mobile Italy, Picopros, 20 Minutes France, MacZonen Denmark, The Mobile Indian India, MacNews Germany, Chip Hazard, Guomii China, and more.

Note: The sites that are linked to above offer Apple community members with an avenue to make comments about this report in many original languages. Additionally, many of these sites provide our guests with different takes on any given patent, concept or lawsuit that is presented in our reports to make it more fun, interesting and/or personal. If you have the time, join in!

Sphere: Related Content