Copyright © 2016 Hail Science

Hail Science

Automating material-matching for movies and video games


Automating material-matching for movies and video games

A team led by researchers from MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) has developed an approach that they say can make texturing even less tedious, to the point where you can snap a picture of something you see in a store, and then go recreate the material on your home laptop.
“Imagine being able to take a photo of a pair of jeans that you can then have your character wear in a video game,”says PhD student Liang Shi, lead author of a paper about the new “MATch” project. “We believe this system would further close the gap between ‘virtual’ and ‘reality.’”
Shi says that the goal of MATch is to “significantly simplify and expedite the creation of synthetic materials using machine learning.” The team evaluated MATch on both rendered synthetic materials and real materials captured on camera, and showed that it can reconstruct materials more accurately and at a higher resolution than existing state-of-the-art methods.
A collaboration with researchers at Adobe, one core element is a new library called “DiffMat” that essentially provides the various building blocks for constructing different textured materials.
The team’s framework involves dozens of so-called “procedural graphs” made up of different nodes that all act like mini-Instagram filters: they take some input and transform it in a certain artistic way to produce an output.
“A graph simply defines a way to combine hundreds of such filters to achieve a very complex visual effect, like a particular texture,” says Shi. “The neural network selects the most appropriate combinations of filter nodes until it perceptually matches the appearance of the user’s input image.”
Moving forward, Shi says that Adobe is interested in incorporating the team’s work into future versions of Substance. In terms of next steps, the team would like to go beyond inputting just a single flat sample, and to instead be able to capture materials from images of curved objects, or with multiple materials in the image.
They also hope to expand the pipeline to handle more complex materials that have different properties depending on how they are pointed. (For example, with a piece of wood you can see lines going in one direction ”with the grain”; wood is stronger with the grain compared to ”against the grain”).
Shi co-wrote the paper with MIT Professor Wojciech Matusik, alongside MIT graduate student research scientist Beichen Li and Adobe researchers Miloš Hašan, Kalyan Sunkavali, Radomír Měch, and Tamy Boubekeur. The paper will be presented virtually this month at the SIGGRAPH Asia computer graphics conference.
The work is supported, in part, by the National Science Foundation.

Continue Reading

More in Technology

- Advertisement -

Most Popular

To Top