Machine Learning Engineer, Speech – Joint Audio-Video Modeling

Job not on LinkedIn

🔥 0 minutes ago

Apply Now

Find Similar Remote Jobs

Set Up Email Alert For New Jobs

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Upload Resume

Cantina

WebsiteLinkedInAll Job Openings

51 - 200 employees

Founded founded by Sean Parker

🤖 Artificial Intelligence

🎮 Gaming

🌍 Social Impact

Artificial Intelligence • Gaming • Social Impact

Cantina is a company that specializes in creating advanced AI characters that can talk, feel, and capture their adventures with selfies. It offers a platform where users can unleash their AI bots in online communities, allowing these lifelike, social creatures to interact with humans. Cantina focuses on building networks of AI influencers and encourages users to explore and build their own collections of AI bots. The company's mission is to foster an interactive universe, inviting creativity and social interaction through digital personalities and AI technology.

📋 Description

• Audio Representations: Design, train, and improve the audio VAEs, neural codecs, and vocoders our generative models sit on top of latent design, reconstruction and perceptual objectives, compression-vs-fidelity tradeoffs. • Model Building: Architect, implement, pre-train, fine-tune, and post-train/alignment (e.g., GRPO/DPO) diffusion and flow-matching transformers for large-scale audio and video generation. • Joint Audio-Video Modeling: Design the audio conditioning and cross-modal alignment inside joint AV models, audio latents alongside video latents, reference-audio and multi-speaker conditioning, multi shot generation audio/video modeling. • Experimental Design: Design, run, and analyze scientific experiments to advance our understanding of the models. • Data Ownership: Define data requirements and collaborate on acquisition, curation, AV-sync and quality filtering, annotation quality, and synthetic data strategies for paired audio-video and speech corpora. • Rigorous Evaluation: Design automated objective/subjective evaluations audio fidelity and intelligibility metrics, AV-sync, listening and viewing tests, robustness & bias checks, and red-team studies. • Inference Efficiency: Drive distillation, step-count reduction, quantization, and kernel/memory optimization to meet interactive latency and cost targets. • Pipeline Delivery: Harden the training → evaluation → inference pipeline; profile latency, memory, and cost; and meet production SLAs with robust monitoring and rollback. • GPU Scaling: Partner with infrastructure to run distributed training/inference on cloud fleets and productionize models with reliability and observability. • Project Leadership: Independently lead small research projects while collaborating on larger team initiatives, including cross-team work with video generation. • Tool Development: Develop and improve dev tooling to enhance team productivity. • Safety & Responsibility: Contribute to safety/consent guardrails, watermarking, and misuse/abuse mitigation for responsible voice and likeness technology.

🎯 Requirements

• Exceptional research/development experience with large-scale audio models (>8B parameters, >500k hours of data). • Deep hands-on experience with diffusion and/or flow-matching transformers, including practical knowledge of samplers, schedules, conditioning mechanisms, and distillation. • Deep hands-on experience training audio VAEs, neural audio codecs, and vocoders latent/tokenizer design, reconstruction and perceptual objectives, adversarial training. • Strong experience with multi-node, multi-GPU distributed training (FSDP/DeepSpeed or equivalent). • Strong software engineering skills with a proven track record of building complex systems. • Strong with PyTorch and performance work (profiling, CUDA/Triton/C++ as needed) and writing reliable production-quality code. • Shipped large-scale speech/audio or multimodal generative models to production. • Background in working with large-scale ML data, and the ability to iterate on data and triangulate quality using both subjective and objective signals. • Experience with voice cloning, speech control/steerability, or expressive speech generation. • Notable publications and/or open-source contributions in speech/audio/ML. • Strongly preferred: • Experience with multimodal audio-video modeling: joint AV generation of multi-shot, multi-speaker scenes with dialogue, music, and sound design generated jointly with video, and the cross-modal alignment that keeps them in sync. • Experience with video generation: video diffusion/flow-matching transformers, video VAEs, conditioned and multi-shot generation, building data pipelines for video models. • Streaming or real-time generation, causal distillation (e.g., Self Forcing / Self Forcing++).

🏖️ Benefits

• Competitive salary and generous company equity • Medical, dental, and vision insurance – 99.99% of premiums covered by Cantina • 42 days of paid time off, including: • 15 PTO days • 10 sick days • 15 company holidays • 2 floating holidays • Generous parental leave & fertility support • 401(k) retirement savings plan • Lifestyle spending account – $500/month to use however you’d like • Complimentary lunch and snacks for in-office employees • One Medical membership, and more!