CLEMSON — A Clemson University researcher is letting her fingers do the talking with a new project that could lead to more realistic hand movements for animated movies, video games and a range of virtual reality simulations.
Sophie Joerg is leading five years of research aimed at finding new ways to make hands and fingers come to life on the screen. It currently is a tedious and time-intensive task and she plans to design algorithms that will automate the work.
The idea for her research came from her experience as an animator on the German TV series “Dragon’s Rock.” Joerg is doing the research as part of a $497,158 CAREER award from the National Science Foundation.
“We are reaching a point in computer graphics where awe-inspiring worlds and characters can be created,” Joerg said. “However, the processes involve an astonishing amount of manual work. Now is the time to develop more rigorous methods to create better algorithms at a faster pace.”
With her project, Joerg is joining a legion of researchers around the world who are chipping away at the technical challenges holding back virtual reality, even as falling hardware prices make it more affordable.
Her research could help professional animators make movies, games and learning simulations. It could also lead the way to hand motions for sign-language avatars and help hobbyists create avatars at home on motion-capture systems, such as the Kinect.
Further, Joerg’s research could answer some of the animation industry’s most basic questions about the importance of finger movements in making characters realistic.
Hand movements may be taken for granted, but they are crucial in making an animated or virtual world seem real. They help communicate information and conduct basic tasks, such as eating, writing and handling tools.
But their small size and complexity make it difficult to recreate them for the screen, Joerg said.
It is hard to measure hands with a motion-capture system and gaps are left in the data. For example, if you have sensors on your hands and fold your arms, some of those sensors are covered. The animator then has to go through the laborious task of manually filling in the gaps.
Joerg, an assistant professor in the School of Computing, hopes to change all that.
“The idea is to create a database of motions and find ways to reuse pieces of the motions over and over again,” Joerg said. “We want to automatically say, ‘Let’s take a half second from this motion, and then combine it with a half second from that motion.’ That’s the basic approach I want to use to synthesize motion automatically.”
The algorithm would choose which motions to pluck from the database to create the desired movement.
Joerg said she got the idea for the research while helping animate “Dragon’s Rock” when she was an undergraduate at the Hamburg University of Applied Sciences.
The sensor-covered gloves the team used to create the characters’ hand motions kept breaking.
“Every now and then, we would not have any CyberGloves or we would have only one,” Joerg said. “We would use one hand and plot it onto the motion of the other hand. We only had a limited amount of time to work on them, so we often would not manage to fix those errors in post-processing.”
Joerg noticed that when the producers saw a show’s near-final cut, they never complained about the hand motions. But in every scene they wanted changes, there were problems with the hands.
“My assumption is that in many cases people do not notice something wrong with the finger motions,” Joerg said. “But you’re so used to interpreting those subtle motions, that they still affect what you’re thinking.”
Joerg hopes that her research will help answer some of the key questions that her experience raised: If you see a virtual character and the hand or finger motions are incorrect, how important is that in the first place? Is it better to have no finger motion at all than to have wrong finger motions?
What makes Joerg’s research unique is that she has expertise in animating characters and studying how they are perceived, which are two separate but interconnected fields. She plans to conduct perceptual experiments and use the results to guide development of the new algorithms.
Eileen Kraemer, the C. Tycho Howle Director of the School of Computing, said that Joerg’s award is a reflection of her hard work and creative approach.
“The CAREER award places Dr. Joerg among the top junior faculty members in the country,” Kraemer said. “It’s a well-deserved honor. She has a strong and impressive vision for her research and a compelling educational plan.”
As part of her award, Joerg plans to use her research to enhance her work at R.C. Edwards Middle School in Central.
She has been working with a team of other researchers to teach computational thinking to students with VENVI, or Virtual Environment Interactions. The program is also aimed at shattering stereotypes by showing students that computer science researchers include women and minorities.
Back on campus, Joerg plans to offer summer research experiences for minority students through the NSF-supported Distributed Research Experiences for Undergraduates. To help recruit and retain people from underrepresented groups, she plans to create an inclusive and welcoming lab and classroom.
Further, Joerg plans to reach out to undergraduate honors students and integrate them into her team with master’s students. She is also developing a course for graduate students that tightly couples motion perception and character animation algorithms.
Congratulations on the award came from across campus, including from Anand Gramopadhye, dean of the College of Engineering, Computing and Applied Sciences.
“This is one of the NSF’s most prestigious awards in support of junior faculty who exemplify the role of teacher-scholar,” Gramopadhye said. “The award is richly deserved and a testament to the high caliber of research she is doing and the exceptional education experience she is offering to students.”