Video Understand
Extract frames and transcribe video locally with ffmpeg and Whisper
What You Can Do
You can intelligently extract representative frames from videos using scene detection, keyframe analysis, or regular intervals, while simultaneously transcribing audio to text using OpenAI's Whisper model. This skill processes videos entirely offline, giving you frame-by-frame visual context and complete transcripts without relying on external APIs or cloud services.
Features
automatically identifies shot changes and extracts frames at transitions
isolates the most important frames based on visual content analysis
captures frames at regular time intervals for consistent sampling
converts video audio to text using offline speech recognition
control maximum frames extracted to balance detail vs. file size
choose from tiny to large models based on accuracy needs
extract visual content without transcription for faster processing
structured metadata with timestamps, frame paths, and transcription data
Example Output
Scene Detection with Transcription
{
"video": "interview.mp4",
"duration_seconds": 180,
"frames_extracted": 12,
"frames": [
{"timestamp": 0.0, "mode": "scene", "path": "frame_0000.jpg"},
{"timestamp": 15.3, "mode": "scene", "path": "frame_0001.jpg"}
],
"transcription": {
"full_text": "Welcome to the interview. Today we're discussing video production techniques...",
"segments": [
{"start": 0.0, "end": 5.2, "text": "Welcome to the interview."},
{"start": 5.2, "end": 12.1, "text": "Today we're discussing video production techniques..."}
]
}
}
Keyframe-Only Output
✓ Extracted 8 keyframes from video.mp4
✓ Identified critical visual moments
✓ Saved frames: frame_0000.jpg, frame_0001.jpg, ...
What's Included
- SKILL.md: complete setup and CLI reference
- understand_video.py: main script with scene detection, keyframe, and interval extraction modes
- Whisper integration: optional local transcription with configurable model sizes
- CLI flags & options: control frame limits, models, output formats, and processing modes
- JSON schema templates: structured output format for frame metadata and transcription segments
Who It's For
- Video editors — quickly understand video structure and content before detailed editing
- Content creators — extract key moments and auto-generate captions locally
- Researchers — analyze video datasets offline without API costs or privacy concerns
- Accessibility specialists — transcribe video audio to create subtitles and transcripts
- Post-production teams — preview shot sequences and identify editing points visually
Best For
- Transcribing video interviews and podcasts to searchable text
- Extracting thumbnail candidates and key frames for previews
- Analyzing shot composition and scene transitions
- Creating subtitle files from video audio tracks
- Cataloging video library content without external dependencies
You might also like
You can transform raw video footage into structured, professional content by leveraging AI to identify key moments, organize clips, transcribe audio, and augment with voiceovers, overlays, and subtitles. This skill guides you through a proven pipeline that emphasizes compression and smart structuring rather than AI generation—turning hours of footage into minutes of compelling content optimized for any platform.
by dliangthinks
You can break down video content into individual frames and aligned audio segments at any interval you specify. This skill generates PNG images for visual analysis, extracts corresponding audio clips in AAC format, and creates a JSON manifest that maps every frame to its timestamp and audio file—perfect for preparing video content for detailed review, transcription workflows, or handing off to other agents for automated processing.
by dliangthinks
Upload your asset specifications, render bottlenecks, or performance metrics, and receive a comprehensive optimization roadmap. Claude analyzes polygon topology, texture memory footprint, shader complexity, and LOD strategies specific to your target engine—whether game engines, live VFX systems, or AR/VR pipelines. You'll get concrete recommendations for polycount reduction, UV layout optimization, texture atlasing, and bake-versus-real-time lighting decisions, complete with documentation for handing off to animation and rigging teams.
by dliangthinks
Plan comprehensive grade structures for multi-scene sequences, extract and analyze color palettes from reference imagery, generate technical LUT specifications for custom creation, and validate your grades against broadcast and cinema standards (ITU-R BT.709, DCI-P3, Rec.2020). This skill handles the analytical and planning work that precedes timeline color correction, helping you make faster aesthetic decisions and ensure technical compliance across projects.
by dliangthinks
Transform raw video analytics into platform-specific optimization plans that boost reach and engagement. You'll identify why certain videos outperform others, uncover trending patterns in your niche, optimize metadata and technical specs for algorithmic distribution, and systematically test creative approaches. This skill bridges creative instinct with data-driven decision-making to help you scale social video performance.
by dliangthinks
You can instantly transform session details, creative decisions, and technical parameters into standardized mix notes, delivery specifications, and editorial documentation. Claude generates episode-consistent technical specs for dialogue, music, and effects stems across broadcast, streaming, and theatrical formats—capturing your creative intent without manual transcription.
by dliangthinks
This skill generates comprehensive audio design documentation that bridges creative vision and technical execution. You'll produce detailed mixing specifications with priority hierarchies, platform-specific audio budgets, middleware configurations for Wwise/FMOD, and implementation workflows tailored to your target engines (Unity, Unreal, custom). It accelerates pre-production planning, clarifies audio requirements for programmers, and reduces iteration cycles during integration.
by dliangthinks
This skill helps you architect complete color strategies before grading sessions begin. You'll generate detailed shot-by-shot grading maps that synchronize visual mood to dynamic audio, develop LUT strategies for specific music genres with documented creative reasoning, establish technical parameter ranges per scene or musical section, and create mood board documentation that aligns stakeholder expectations early. By treating color as a primary creative element rather than post-production cleanup, you reduce revision cycles and deliver grades that amplify both narrative and musical intent.
by dliangthinks