Assistance for the Artistically Challenged 
by Andrew Kerr
November 2006

Go to YouTube to see a demonstration of the projector-guided painting system.

While systematists would prefer that the world of thought be neatly partitioned into non-overlapping categories, K-12 teachers everywhere know that life is, by it's very nature, interdisciplinary. I encountered a wonderful example of this several months ago while working on an article about computer science researcher Jay Summet's research into Virtual Rear Projection (VRP). Throughout Dr. Summet's lab were canvases covered with paint. Oil painting in a computer lab? What's going on here?

Matt Flagg is a graduate student in Dr. Summet's lab who is using VRP as a novel means to teach painting. When I sat down with him on November 1, 2006 Mr. Flagg had just returned from a whirlwind tour of Europe and California (and was still recovering from the yo-yo of inhabiting multiple time zones). He had been demonstrating his research to various interested parties and at different conferences, including the User Interface Software Technology in Montreux, Switzerland, which chose to put his system (and a painterly rendering of Scarlett Johansson as the "Girl With a Pearl Earring") on the cover of their program.

"My wife is pretty into oil painting," Mr. Flagg says, "and I've always been impressed and kind of admired her ability. I wondered if there would be an interesting way to build a system to support painting for novices."

Taking inspiration from art projectors, which can be picked up in most ordinary art supply stores, and the camera obscura (believed by some to have been the primary tool Vermeer used to guide his works), Mr. Flagg decided to take things to the next level by incorporating VRP. While an art projector can cast the image of a painting onto a screen, it ends there. But a computer, in tandem with multiple multimedia projectors, can provide far more information, including a sequential presentation of the layers needed to build a painting, brush stroke orientation--anything that can be plugged into the computer's memory and projected onto a screen.

Coming up with a computerized painting instruction/guidance system required collaborating with artists. Fortunately, Mr. Flagg has one living right under his roof. "Fortunately since my wife paints I just take pictures of her painting and I use those as layers in the system," he explains. This is how the computer "knows" each successive layer needed in the sequence required to produce a good work of art. "In the painting literature, novices are known to kind of attack a very small area of a painting first and try to perfect that small area, like an eye in a face, until they move on," he explains. "There's a lot of evidence that novices don't organize the process, they just kinda go straight into it. By organizing the process they will get a better result."

There are two basic ways to use the painting system. One is the aforementioned layer-by-layer technique. Another is to input an image into the computer, have the computer render that image in a painterly style (you can select from a few different styles, including pointilism and other types of impressionism, thanks to groundbreaking research by Dr. Irfan Essa and James Hayes from a few years ago), and then allow the computer to interpret what would be required to reproduce that picture. At this point one cannot effectively input another finished painting into the system in this manner ("We need high frequency information, so we need a photograph because the system is designed for working on photographs."). So one cannot simply scan in Leonardo's "Mona Lisa" and whip out a perfect copy. At least not yet.

"We've had a lot of people say it reminds them of paint-by-numbers," Mr. Flagg says. "Our feeling about the paint by numbers thing is, first of all, paint-by-numbers can't do layers. And because you're turning a projector off and on and you're just kinda getting a hint as to what to do, we like to think it's a lot more free-form."

The sky is the limit with regards to the amount of additional data that could be inputted in future iterations of the program. "The system right now is capturing only a few dimensions of the whole painting experience. We could focus on sensing the pressure, speed, so many more things about the actual physics of painting. Another avenue is actually teaching techniques. So maybe animating the brush strokes, or showing more strategies, more styles."

We may not all become Monets with this program, but that's not the point. Mr. Flagg hopes it will encourage the majority of us who are unsure of our artistic ability to take the plunge and discover the joys of painting.