Moayed Haji ALi

22 Jan 2025

ICLR rejections go brrrr

Moayed Haji ALi reposted

Vicente Ordonez

@vicenteor.bsky.social

Check this recent work by my PhD student Moayed. He has been doing amazing work on Generative AI for images, video and audio. We introduce AV-Link ♾️, an unified approach for audio-video generation. Our generated audio is the best in terms of synchronization with video actions. Check thread below.

Moayed Haji ALi@moayedha.bsky.social

Can pretrained diffusion models be connected for cross-modal generation? 📢 Introducing AV-Link ♾️ Bridging unimodal diffusion models in one self-contained framework to enable: 📽️ ➡️ 🔊 Video-to-Audio generation. 🔊 ➡️ 📽️ Audio-to-Video generation. 🌐: snap-research.github.io/AVLink/ ⤵️ Results

View full thread

While current approaches uses external pretrained features (e.g. Meta CLIP, BEATs), we found that diffusion activations hold rich, semantically and temporally aware features, making them perfect for cross-modal generation in a self-contained framework. 🔊➡️📽️ Example:

arxiv.orgAV-Link: Temporally-Aligned Diffusion Features for Cross-Modal Audio-Video GenerationWe propose AV-Link, a unified framework for Video-to-Audio and Audio-to-Video generation that leverages the activations of frozen video and audio diffusion models for temporally-aligned cross-modal co...

A great collaboration with W. Menapace, A. Siarohin, I. Skorokhodov, A. Canberk, K.S Lee, V. Ordonez, and S. Tulyakov. Please repost to support our work and check out our Arxiv preprint: arxiv.org/abs/2412.15191 Webpage: snap-research.github.io/AVLink/

View full thread

Compared to Meta Movie Gen Video to Audio, we achieve significantly better temporal synchronization with a 90% smaller scale model.

Besides Video to Audio (📽️ ➡️🔊), we also support Audio to Video (🔊➡️📽️) generation under the same unified framework.

recise temporal synchronization remains a significant challenge for current video-to-audio models. AV-Link addresses this by leveraging diffusion features to accurately capture both local and global temporal events, such as hand slides on a guitar and fretboard pitch changes.