Content-Based Video Asynchronization Detection
2023 Cycle
SUMMARY
![A diagram illustrating a neural network sync predictor. The image includes a baseball player mid-swing, with a catcher and umpire in the background. A waveform labeled ‘[sound of bat hitting ball]’ connects to a box labeled ‘Neural Network Sync Predictor,’ which outputs ‘In-sync or out-of-sync?’](https://sciencehub.utexas.edu/wp-content/uploads/2023/10/Harwath-AV-Sync-Figure.png)
The goal of this project is to develop machine learning models capable of automatically detecting when the audio and visual streams of a live video broadcast become temporarily out-of-sync with one another. This type of synchronization error is commonly encountered in the real world and has a negative impact on the viewer’s experience. Models that can rapidly detect when these errors arise are essential tools in helping technicians correct the problem. The new benchmark dataset for this task is comprised of real-world sports and news broadcasts, as well as the development of novel deep neural network architectures for audio-visual asynchronization detection.