Inside Apple's future iPhone camera
- 21 April, 2015 02:02
I've got an iPhone 6 Plus, and there's no getting around an obvious fact: The camera is pretty great.
I'm also hearing that the camera on the Samsung Galaxy S6 and Galaxy S6 Edge might be even better, although I haven't personally tried either of them.
Because camera superiority is often a major reason why people buy a company's highest-end phones, the industry is in a quandary: How can they dramatically improve what are already great phones, even as smartphones are getting thinner and thinner?
Apple this week revealed its direction: multi-aperture computational imaging. That's photography mumbo jumbo that means using more than one camera combined with software that creates a picture from the resulting data.
The revelation about Apple's direction came in the form of Apple's $20 million acquisition of an Israeli startup called LinX Computational Imaging. (LinX is pronounced "links.")
LinX has developed multiple products with two, three or four cameras -- each in a unique configuration and size.
(The deal wasn't just a technology acquisition; it was also an acqui-hire. Apple added several LinX employees to its team, including Ziv Attar, who worked as a senior optics specialist at Israeli defense company Rafael Advanced Defense Systems, and Andrey Tovchigrechko, who once headed a group of algorithm specialists at Samsung. Apple also got two brilliant engineers: Edwin Maria Wolterink and Chen Aharon-Attar.)
Why LinX tech is so much better
LinX has developed very small multi-aperture imaging technology that combines special-purpose sensors, optics and image processing. This approach brings two kinds of benefits to smartphone (and tablet) photography. The first is better quality, and the second is a long list of very cool features that wouldn't be possible with today's smartphone cameras (I'll go into those features below).
So called "array cameras," which combine inputs from multiple sensors and cameras, have been known for years to boost color fidelity, sensitivity and other features. But such technologies have long been plagued by problems, such as registration errors and issues resulting from the fact that not all cameras can see every point in the shot.
Of course, LinX technology has not been evaluated independently in a functioning smartphone, to the best of my knowledge. We have to rely on the company's claims and on the evidence it controls. But what LinX does claim on the image-quality front is that its cameras can match DSLR cameras, but in a tiny smartphone-compatible size -- in fact, LinX systems are smaller than existing smartphone camera electronics and, more specifically and more importantly, they weigh about half as much.
Before it was acquired by Apple, LinX published a 30-page presentation detailing its technology and, more tellingly, comparing LinX cameras with the camera in the iPhone 5S.
LinX has taken down that presentation, but the blog MacRumors had previously posted a copy on the digital library site Scribd. You can see it here.
What the presentation appears to demonstrate is that, in comparison with the iPhone 5S camera, LinX technology (which the company claims is comparable in both pixel density and cost to the iPhone model) has significantly better color fidelity, much sharper and cleaner images and, above all, very good performance at very low light levels with hardly any visual noise at all.
In addition to being able to produce superior 2D images, the LinX technology captures accurate depth information, which is the key to understanding all the smartphone camera features it can enable.
Now we get to the really good stuff
Because the LinX camera system is capturing images from more than one angle, and also mapping the distance of every object and surface in the shot, it can do amazing things far beyond simply taking great photos.
The LinX depth map can speed up and improve the quality of auto-focus. And if you don't like the image captured with auto-focus, you can refocus the image after it's been taken!
You could replace or remove the background of not only a photo, but also a live video -- in real time. It would work like a green screen, but no green screen would be required. Any background could be replaced.
It enables much better face recognition than current smartphone cameras, because all visual noise besides the face can be eliminated. And the face itself can be isolated for faster, more efficient and more accurate processing.
It's ideal for augmented reality, because the camera can selectively remove the background or the foreground of what you're seeing in real time.
It can be used for 3D object modeling, for copying physical objects for 3D printing or mapping the interior of a room.
A camera with LinX technology can even tell you the size of the objects you're shooting, and how far away they are.
We tend to think that the cameras in our smartphones are only good for taking cat photos and selfies. In fact, as we enter into the next phase of smartphones, we'll need our cameras to do much more: Live-streaming, face-recognition biometrics, augmented reality and 3D mapping. We'll also use them as data-harvesting super sensors.
LinX camera technology enables all this at low cost and in a tiny electronics package that's ideal for smartphones.
The LinX team claims boldly that they have solved all problems related to multi-aperture imaging, and that their technology is ready for production.
What that means is that LinX technology could be integrated into the iPhone 7 line. We could be using it by the end of the year!
So if you thought smartphone photography couldn't get any better, I'm here to tell you that it can -- and will.