The FCam API is
now available on the Nokia
N9 Linux-based smartphone. See below for details.
Computational photography refers broadly to sensing strategies and
algorithmic techniques that enhance or extend the capabilities of digital
photography. The output of these techniques is an ordinary photograph, but one
that could not have been taken by a traditional camera. Representative
techniques include high dynamic range imaging, flash-noflash imaging, coded
aperture and coded exposure imaging, photography under structured illumination,
multi-perspective and panoramic stitching, digital photomontage, all-focus
imaging, and light field imaging.
Although interest in computational photography has steadily increased among
graphics and vision researchers, progress has been hampered by the lack of a
portable, programmable camera platform with enough image quality and computing
power to be used for everyday photography. To address this problem, we have
been pursuing two subprojects:
After pursuing these two subprojects independently for two years, we decided
that the best approach was to define a single open architecture for
programmable cameras. Our architecture consists of a hardware specification, a
software stack based on Linux, and an API with bindings for C++. The
architecture permits control and synchronization of the sensor and image
processing pipeline at the microsecond time scale, as well as the ability to
incorporate and synchronize external hardware like lenses and flashes. To
demonstrate the viability of this architecture, we have built two reference
implementations: a Nokia N900 smartphone with a modified software stack, and
the Frankencamera F2 pictured above. Click here for our
SIGGRAPH 2010 paper about this architecture.
Computational Photography on Cell Phones.
Over the past five years, the cameras in cell phones have improved dramatically
in resolution, optical quality, and photographic functionality. (See 1st and
2nd images above.) Moreover, camera phones offer features that dedicated
cameras do not: wireless connectivity, a high-resolution display, 3D graphics,
and high-quality audio. Finally and perhaps most importantly, these platforms
run real operating systems, which vendors have begun opening to third-party
developers. We have therefore begun developing computational photography
applications for commerical cell phones. Our Eurographics 2008 paper on viewfinder
alignment is an example.
The Stanford Frankencamera.
Despite these encouraging trends, there are computational photography
experiments that cannot be implemented on today's cell phones. either because
the cameras' sensor or optics aren't good enough, the computing resources
aren't powerful enough, or the APIs connecting the camera to the computing are
too restrictive. We have therefore built a custom camera that runs Linux, is
fully programmable (including its digital signal processor) and connected to
the Internet, and accommodates SLR lenses and (soon) SLR-quality sensors. Our
current prototype (3rd and 4th images above) is constructed from off-the-shelf
parts, in some cases borrowed from dead cameras. It's also ugly
- hence the name.
On July 21, 2010 we released the source code for our implementation of the
Frankencamera architecture and
Nokia N900 smartphone.
A binary executable was also released, which can be downloaded to any retail
N900 without bricking the phone, thereby making its camera programmable. To
get started turning your N900 into a Frankencamera, click
here. FCam is now
also available on the Nokia N9 as part of the Meego/Harmattan 1.2 operating
system update. The Harmattan documentation is here.
The N9 is not sold by carriers in the U.S., but unlocked N9's can be
purchased on Amazon, and we believe they work with AT&T and T-Mobile.
In addition to our implementation on the N900, we plan to make the more
powerful custom-built Frankencamera F's available to the research and teaching
community. However, the Frankencamera F2 (pictured above) is not worth buying,
since its sensor is only cell-phone sized. (Hence, it has a 6x crop factor
relative to a 35mm sensor. The only Canon lens that makes sense given this
crop factor is a 10-22mm set to 10mm. This configuration is pictured on the
web page of our
SIGGRAPH 2010 paper.)
Instead, we are actively working on the Frankencamera F3, which will have a
24mm x 24mm sensor, hence a 1x crop factor (i.e. full-frame), at least in the
vertical direction. We don't have a release date for the F3, but we are
shooting for Spring 2012. Our plan is to distribute the camera "at cost"
(might be $8K) to anyone who wishes to purchase one.
Using FCam in university courses
Are you a university instructor teaching a course in computational photography?
If so, then we want to talk to you. We are currently assembling packages to
help teach computational photography at the university level. Each package
will consist of 20-30 lectures on computational photography written by
Marc Levoy (Stanford) and
Fredo Durand (MIT),
a box containing 10 Nokia N900 smartphones, and one Frankencamera F3. The
lecture material will be free, available to anyone and downloadable from the
Internet. The N900s will also be free, but as loaners only, and subject to
available supply. This loan program is supported by a generous grant from
Nokia. The Frankencamera F3s will be sold "at cost", as mentioned
As of Summer 2010, this package has been used in two courses - at Stanford
University (co-taught by Marc Levoy and Fredo Durand)
and at ETH Zurich (taught
In both courses we loaned Nokia N900s to every student, and assignment #1 was
to replace its autofocus algorithm. (The phone's camera, while small, has a
details look at the "Hello Camera" link on the
course web site.) We graded the assignment on the accuracy they could
focus on a test scene we provided, as well as focusing speed in milliseconds.
An assignment like this would have been impossible before FCam. Two of the
students submitted algorithms that were better than Nokia's; these were
presented in Finland at the conclusion of the course. The students also did
interesting projects for the course, some of which might be submitted for
publication over the coming year.
At present this program is designed mainly for universities in North America
and Europe. However, we will entertain all reasonable requests. In addition,
for universities in the United States, the Frankencamera F3s will be free,
supported by a grant from the National Science Foundation. Want to find out
more about this program?
If you're a U.S. university looking to join the NSF-supported program,
send email to
If you're any other university looking for loaner N900s,
Timo Ahonen at Nokia.
The Camera 2.0
project, which began as a collaboration
Stanford Computer Graphics Laboratory
Nokia Research Center Palo Alto Laboratory, has also received funding from
Adobe Systems, Kodak, Hewlett-Packard, The Walt Disney Company, Intel,
Texas Instruments, Google, NVIDIA, and Sony. The NSF portion of the project is
in collaboration with
Fredo Durand and
William Freeman of MIT.
- Abe Davis
< firstname.lastname@example.org >
Recent papers in this area:
Decoupling algorithms from schedules for easy optimization
of image processing pipelines
ACM Transactions on Graphics 31(4)
(Proc. SIGGRAPH 2012)
Experimental Platforms for Computational Photography
- IEEE Computer Graphics and Applications,
Vol. 30, No. 5, September/October, 2010, pp. 81-87.
An Experimental Platform for Computational Photography
Eino-Ville (Eddy) Talvala,
Sung Hee Park,
David E. Jacobs,
Henrik P.A. Lensch,
- Proc. SIGGRAPH 2010
- Reprinted in
CACM, November 2012, with an introductory
technical perspective by Richard Szeliski
Gaussian KD-Trees for Fast High-Dimensional Filtering
- Proc. SIGGRAPH 2009
Spatially Adaptive Photographic Flash
Rolf Adelsberger, Remo Ziegler,
- Technical Report 612, ETH Zurich, Institute of Visual Computing,
- Proc. Eurographics 2008
Publicity about the project
We'd love to hear from you, but first check out our list of
frequently asked questions.
Left: Marc Levoy.
Right: Andrew Adams with camera.
Behind them is the
Stanford Multi-Camera Array.
Another picture of the camera,
used in the July 2010 Stanford Report article.
Images by Linda Cicero.
Stanford Engineering Research Profile (August 2009), superceded by Stanford Report (below)
Stanford Report (August 31),
picked up by
Slashdot (September 3),
Hack A Day,
This Week in Photography (TWIP),
Tendencias Informáticas (Spanish),
Le monde de la photo (French),
ITC Online (Ukrainian),
La Repubblica (Italian),
Digital Focus (Italian),
Wissenschaft aktuell (German),
Chip Online (German),
Spiegel Online (German),
Internet Watch (Japanese),
United Daily News (Chinese),
- NPR's All Things Considered (October 11)
- New Scientist (November 11)
We have gaven up tracking these articles; there are too many of them.
Caveat: We have found articles illustrated
by "alternative Frankencameras" that are not related to our project. Some are
amusing, while others look like
Russian constructivist art projects.
The real Stanford Frankencamera is pictured at left
and at the top of this web page.
Left: The Frankencamera F2 with two Canon flash units attached.
Right: A picture taken by this Franken-arrangement.
The smaller 430EX flash was programmed using our
to strobe repeatedly, and the larger 580EX was programmed to fire once at the
end of the exposure.
Stanford Report (July 21, 2010), picked up by
Adorama Photo News. Here is a
of the photograph used in the Stanford Report article.
- Nokia Conversations (Nokia's official blog, July 21), picked up by
mobile blog.it (Italian),
- Nokia Conversations (July 27)
- MIT Technology Review (July 30)
- Imaging Executive Podcast (September 15 audio interview by Paul Worthington of the 6Sight Report), also available at
Daily Photo News
- Stanford Magazine (September/October issue)
- New York Times (December 18), reprinted in the San Francisco Chronicle (December 19) and elsewhere
- Stanford Daily (January 6, 2011)
MIT Technology Review (June 2011 issue)
- The Economist (September 3, 2011, see also this blog in the Economist on June 29)
- Science News (January 28, 2012)
November 19, 2012 01:13:15 AM