Vision Sync
An innovative AI application that watches your facial expressions and surroundings through your webcam, then generates a real-time ambient soundtrack that matches your mood and environment.
How It Works
Mood Detection:
- Smile? → Upbeat electronic vibes
- Frown? → Melancholy jazz emerges
- Surprised? → Ethereal, atmospheric soundscapes
Core Features
- Real-Time Face Tracking: 468 facial points using MediaPipe Face Landmarker
- Object Detection: COCO-SSD for environmental awareness
- Music Generation: Google Lyria API + Procedural Web Audio Synth
- Cyberpunk UI: HUD with scanlines, vignette effects, live emotion sliders
Technical Architecture
Frontend:
- React + TypeScript for application logic
- Tailwind CSS for styling
- Framer Motion for animations
Computer Vision:
- TensorFlow.js + COCO-SSD for object detection
- MediaPipe Face Landmarker for facial tracking
Audio Generation:
- Google Lyria API for AI music composition
- Procedural Web Audio Synth for real-time playback
User Experience
Real-time mood detection with instant audio feedback Cyberpunk-themed HUD interface Live emotion sliders showing detected mood states Seamless music transitions based on expression changes
Stack
- Frontend: React, TypeScript, Tailwind CSS, Framer Motion
- Computer Vision: TensorFlow.js, COCO-SSD, MediaPipe
- Audio: Google Lyria API, Web Audio API
- Styling: Cyberpunk HUD with custom effects