Sharp unit to license IP from U.S. labs

The venture's charter is to commercialize technologies developed at Sharp Laboratories of America Inc. that have languished here in the labs — "technologies that, for one reason or another, Sharp Corp. in Japan is not going to develop," said Jon Clemens, the leader of Sharp Technology Ventures. Clemens retired last year as director of Sharp Labs after getting permission from the $20 billion parent company in Osaka to form the tech venture company.

"There will be many advantages to users as we license these technologies, but for me it's about the people who created them," Clemens said. "You don't join Sharp Labs to write papers; you want your technologies to get out there."

One technology that could find a wide audience is Sharp's HiMpact Sports, which applies a set of algorithms that understand the semantics of baseball, football and soccer (for starters) and can boil down a three-hour game to 45 minutes without skipping a single play. The technology provides automatic indexing, random-access play-by-play navigation and annotated summaries for live or recorded video streams. Sharp will license HiMpact for video-on-demand, video-over-Internet, personal video recorders and handheld devices.

The Entertainment and Sports Programming Network will likely be the first licensee for HiMpact Sports summaries; according to Clemens, ESPN is interested in merging video feeds with its Sports Ticker service (a real-time wire service that offers play-by-play text).

But the company claims to have many other customers on the hook for the IP, including security experts seeking a HiMpact Security version for pinpointing and extracting "suspicious activity" highlights from long hours' worth of surveillance footage.

For ESPN, Sharp Labs created a prototype that segments game videos into plays, then merges in the SportsTicker commentary for each play. The result, which Sharp calls a summary, lets fans watch a game play-by-play with a textual index that will tell them what to watch for, which players will be featured and the outcome of the play.

Sharp Labs has also collaborated to create HiMpact prototypes for Virage Inc., a digital archive service for government proceedings that automates a formerly manual workflow; New Media Technology Inc., a provider of digital asset management software and video-logging productivity enhancements for sports footage; and Dixon Sports Computing, a provider of instant-replay and video coaching analysis technology.

"We think that HiMpact Sports will change the way people watch sports," said Ibrahim Sezan, director of information systems technologies at Sharp Labs. "Fans will still want to watch games like the Super Bowl live, if for no other reason than to have a party, but once they get used to the nonstop play-by-play action and the ability to jump ahead to the next play or back to the last play, they will want to watch most games that way."

HiMpact Sports recognizes an event in a sport in all its permutations. For example, a"base hit" in baseball could be an infield grounder or a line drive into center field. Understanding the semantics of major events in a game lets HiMpact Sports segment a game into plays, create summary views with random access to plays and provide annotation that lets a user view, for example, all of the home runs scored or all of a favorite player's hits during a given game.

How can Sharp Labs teach a computer to recognize a base hit regardless of whether it's a grounder, a line drive or a bunt? Traditional AI would extract features from the video stream, then use handwritten rules to infer the meaning (base hit) from the features. After extensive testing, however, Sharp Labs concluded that its requirement that HiMpact provide 100 percent accuracy could only be met by probabilistic methods that directly learn from experience.

"We believe that deterministic, rule-based inference from features extracted from video is too difficult to write rules for that cover every situation, but probabilistic methods that learn provide an elegant solution without getting into all the intricate problems associated with explicitly setting the rules of inference," said Sezan.

Playbooks for AI

The best probabilistic method Sharp Labs has tried thus far is the hidden Markov model (HMM), which has previously been successful in learning how to recognize spoken voices. Just as HMM is "taught" words by training it with samples of different people speaking the same word, Sharp Labs trained its HMM on video clips it categorized into a training set.

As a result, the researchers found a way to teach an HMM to interpret video sequences as either "a play" or "not a play." Thousands of video clips were flagged as either constituting or not constituting plays, and then HMM learned them, adjusting its internal parameters until it correctly classified the training set. After learning, the HMM was set to feedforward mode so it could be used to classify frames as either part of a play or not.

After much experimentation, Sezan said, Sharp Labs settled on the Baum-Welch variation on the HMM algorithm, with which it achieved 100 percent success for its Dixon Sports Computing prototype.

So far HiMpact has only conquered segmenting baseball, football and soccer into plays; other sports, such as basketball, have proofed tougher to dissect.

"Even sumo wrestling, with all the ceremony, is easier to segment than basketball," said Sezan.

Advanced face recognition

The system that achieved 100 percent accuracy employed a 2-GHz Pentium 4 processor with a video card capable of processing 180 frames/second, but at a resolution of only 120 x 160 pixels.

The video was first MPEG-compressed and then submitted to the previously parameterized HMM, which classified the frames as plays or nonplays.

The resulting metadata, classifying the frame, was then attached to an MPEG-7 file (MPEG-7 was created by the Moving Picture Experts Group to identify content for database retrieval, digital libraries, broadcast-channel selection, multimedia editing, surveillance and home entertainment). Also, Sharp Technology Ventures will offer an advanced face-recognition technology that allows real-time detection and tracking of faces at video rates for security and surveillance, teleconferencing, digital libraries and object-based video coding.

The tracker locks on to a face, ignoring the background, and can be interfaced with facial-expression recognition algorithms. For example, a system that recognizes a puzzled expression might automatically respond by offering the user access to a help menu. The face-tracking technology is supported by 3 U.S. patents.

Another IP offering is a scanner configuration that eliminates the nagging problems of image quality, especially the ghostly color fringes that appear around text as a result of color misregistration. Sharp's process uses a trilinear image scanner configuration that permits lower tolerances for sensor alignment while improving image quality.

The technology is designed to improve the quality of document scanning in photocopiers and other image-reproducing systems, but it also provides a method of refining color registration within finished images. It is supported by one U.S. patent.

Sharp will also license a cell phone-battery technology that saves power by disabling the receiver circuit when no data is to be sent or received, thereby extending battery life. The mobile unit initiates requests for data transfers and therefore does not need to constantly monitor the forward link signal. The technology is supported by one U.S. patent.