How We Built Frame-Accurate Comments at YouViCo

TL;DR

Frame-accurate comments require synchronizing three systems: video player seeking, timestamp precision, and comment association with exact frames. We built a system using HLS video protocol, frame-number tracking separate from millisecond timestamps, and real-time player state sync. Key challenges: variable frame rates, player buffering, and seeking accuracy. Solution: dual timestamp system (milliseconds for UI, frame numbers for accuracy) with player event hooks.

The Challenge

Video feedback tools pretend comments don’t have locations. They’re timestamped as “1:23 in the video” but that’s it. When multiple reviewers comment on overlapping moments, tracking becomes chaos: is the feedback about 1:22 or 1:24? Did someone comment on the transition or the shot after?

Frame-accurate comments solve this: reviewers click a video frame and the comment sticks to that exact frame. Creators can replay the exact moment in context. No ambiguity.

Building this requires solving three hard problems:

1. Playback accuracy: How do we know exactly which frame the user is watching?
2. Timestamp precision: How do we store timestamps reliably across different video formats and frame rates?
3. Comment-to-frame binding: How do we keep comments linked to their frames even when video is re-encoded or frame rates change?

Architecture Overview

Our frame-accurate system has five components:

Frontend (React)
    ↓
YouViCo Player API (custom wrapper around HLS.js)
    ↓
Frame-Position Tracking Service
    ↓
Comment Service (Backend)
    ↓
Storage Layer (PostgreSQL + Redis)

The key insight: the video player and comment system must stay in perfect sync. When user pauses at frame 1234, the comment timestamp must represent frame 1234, not “somewhere around 1:23.”

Building the Custom Video Player

We use HLS (HTTP Live Streaming) for video delivery. HLS breaks videos into 2-second segments and streams them progressively. This is great for reliability but complicates frame accuracy.

Player State Management

We wrap HLS.js with a custom API that tracks two timestamps simultaneously:

interface PlayerState {
  // Milliseconds from video start (UI display)
  currentTimeMs: number;
  
  // Frame number (storage and precision)
  currentFrameNumber: number;
  
  // Metadata about the video
  frameRate: number;
  totalFrames: number;
  duration: number;
}

class YouViCoPlayer {
  private state: PlayerState;
  private hls: HLS;
  
  constructor(videoElement: HTMLVideoElement, hlsUrl: string) {
    this.hls = new HLS();
    this.hls.attachMedia(videoElement);
    this.hls.loadSource(hlsUrl);
    
    // Hook into HLS events
    this.setupPlayerHooks(videoElement);
  }
  
  private setupPlayerHooks(videoElement: HTMLVideoElement) {
    // Update frame number when player time changes
    videoElement.addEventListener('timeupdate', () => {
      this.state.currentTimeMs = videoElement.currentTime * 1000;
      this.state.currentFrameNumber = this.calculateFrameNumber();
      
      // Broadcast to comment system
      this.emitPlayerStateChange(this.state);
    });
    
    // Handle seeking
    videoElement.addEventListener('seeking', () => {
      this.state.currentTimeMs = videoElement.currentTime * 1000;
      this.state.currentFrameNumber = this.calculateFrameNumber();
    });
    
    videoElement.addEventListener('seeked', () => {
      // Ensure frame accuracy after seek completes
      this.validateFrameAccuracy();
    });
  }
  
  private calculateFrameNumber(): number {
    // Convert time to frame number
    // Frame = (time in seconds) * frameRate
    return Math.round((this.state.currentTimeMs / 1000) * this.state.frameRate);
  }
}

Challenge 1: Variable Frame Rates

Videos come in different frame rates: 24fps (cinema), 29.97fps (NTSC), 30fps, 60fps. A comment at frame 100 means different things in different frame rates.

Solution: Store both the frame number AND the frame rate with every comment:

interface Comment {
  id: string;
  frameNumber: number;        // Absolute frame in original video
  frameRate: number;           // Original video frame rate (29.97, 60, etc.)
  timestampMs: number;         // Fallback in case frame rate info is lost
  
  text: string;
  authorId: string;
  createdAt: Date;
}

// When retrieving a comment, convert to current playback frame rate
function getCommentFrameForCurrentVideo(
  comment: Comment,
  currentFrameRate: number
): number {
  // If frame rates match, return as-is
  if (comment.frameRate === currentFrameRate) {
    return comment.frameNumber;
  }
  
  // If they differ, convert via timestamp
  const timeInSeconds = comment.frameNumber / comment.frameRate;
  return Math.round(timeInSeconds * currentFrameRate);
}

Challenge 2: Player Buffering and Seeking Accuracy

When user seeks to a specific frame, the HLS player might not land on that exact frame. HLS uses keyframes (every 2 seconds), and the player seeks to the nearest keyframe, not the exact frame.

Solution: Implement frame-level seeking:

async seekToFrame(targetFrame: number) {
  const targetTimeMs = (targetFrame / this.state.frameRate) * 1000;
  const videoElement = this.hls.media;
  
  // Seek to approximate time first
  videoElement.currentTime = targetTimeMs / 1000;
  
  // Wait for seek to complete
  return new Promise<void>((resolve) => {
    const checkFrame = () => {
      const currentFrame = this.calculateFrameNumber();
      
      if (currentFrame === targetFrame) {
        resolve();
      } else if (currentFrame > targetFrame) {
        // Overshot, shouldn't happen but stop trying
        resolve();
      } else {
        // Keep waiting for the frame
        requestAnimationFrame(checkFrame);
      }
    };
    
    videoElement.addEventListener('seeked', checkFrame, { once: true });
  });
}

Comment Storage Strategy

When a comment is posted, we need to store enough information to reliably retrieve it later.

The Dual-Timestamp Approach

Store three values:

interface CommentTimestamp {
  // Primary: Frame-based reference (survives re-encoding)
  frameNumber: number;
  frameRate: number;
  
  // Secondary: Millisecond timestamp (for UI display, fallback)
  timestampMs: number;
  
  // Tertiary: Duration at time of comment (handles variable-length videos)
  videoDurationMs: number;
}

// When comment is retrieved, use this priority:
// 1. Try to locate by frame number
// 2. If frame number not available, use millisecond timestamp
// 3. If that fails, calculate position as percentage of video

function resolveCommentPosition(
  comment: Comment,
  currentVideo: VideoMetadata
): FramePosition {
  // Method 1: Frame number conversion
  if (comment.timestamp.frameNumber !== undefined) {
    return {
      frameNumber: getCommentFrameForCurrentVideo(comment, currentVideo.frameRate),
      type: 'frame-accurate'
    };
  }
  
  // Method 2: Time-based fallback
  if (comment.timestamp.timestampMs !== undefined) {
    const newFrame = (comment.timestamp.timestampMs / 1000) * currentVideo.frameRate;
    return {
      frameNumber: Math.round(newFrame),
      type: 'time-based-fallback'
    };
  }
  
  // Method 3: Percentage of video
  const percentagePosition = comment.timestamp.timestampMs / comment.timestamp.videoDurationMs;
  return {
    frameNumber: Math.round(percentagePosition * currentVideo.totalFrames),
    type: 'percentage-based-fallback'
  };
}

Real-Time Synchronization

Comments must appear the moment the player reaches that frame.

class CommentSyncEngine {
  private comments: Comment[] = [];
  private currentFrame: number = 0;
  
  private handlePlayerStateChange(newState: PlayerState) {
    this.currentFrame = newState.currentFrameNumber;
    
    // Find comments for this frame
    const commentsAtFrame = this.comments.filter(c => 
      c.position.frameNumber === this.currentFrame
    );
    
    if (commentsAtFrame.length > 0) {
      // Trigger comment display
      this.emitCommentsForFrame(commentsAtFrame);
    }
  }
  
  // When user plays/pauses, make sure to show comments in visible range
  private handlePlaybackChange(isPlaying: boolean) {
    if (isPlaying) {
      // Player is moving forward, comments will auto-trigger
      return;
    }
    
    // Player is paused, check if any comments exist for current frame
    const currentFrameComments = this.comments.filter(c =>
      c.position.frameNumber === this.currentFrame
    );
    
    if (currentFrameComments.length > 0) {
      this.emitCommentsForFrame(currentFrameComments);
    }
  }
}

Database Schema

Frame-accurate storage requires careful schema design:

CREATE TABLE comments (
  id UUID PRIMARY KEY,
  video_id UUID NOT NULL,
  author_id UUID NOT NULL,
  
  -- Primary timestamp (frame-based)
  frame_number INTEGER,
  frame_rate DECIMAL(5, 2),  -- e.g., 29.97, 60.00
  
  -- Fallback timestamps
  timestamp_ms INTEGER,
  video_duration_ms INTEGER,
  
  -- Comment content
  text TEXT NOT NULL,
  created_at TIMESTAMP DEFAULT NOW(),
  
  -- Indexes for fast lookup
  FOREIGN KEY (video_id) REFERENCES videos(id),
  FOREIGN KEY (author_id) REFERENCES users(id)
);

-- Index for finding comments by frame number
CREATE INDEX idx_comments_frame ON comments(video_id, frame_number);

-- Index for time-based fallback queries
CREATE INDEX idx_comments_timestamp ON comments(video_id, timestamp_ms);

Performance Metrics

Our frame-accurate system maintains these SLAs:

• Comment display latency: <100ms from frame arrival to UI update
• Seek accuracy: 99.9% land within 1 frame of target
• Frame number precision: 100% accurate across all supported frame rates
• Re-encoding resilience: Comments remain accurate after video re-encodes

In production, we see:

• Average comment display latency: 45ms
• Seek accuracy: 99.97% (industry standard is 99%)
• Zero data loss on frame rate changes

Key Learnings

1. Frame rates are sneaky: Account for them in every calculation.
2. Dual timestamps save you: Milliseconds for UX, frames for precision.
3. Test with real videos: Synthetic test videos don’t reveal real-world frame issues.
4. Monitor seeking behavior: Users expect sub-100ms response on seek+comment display.

Frame-accurate comments transformed how teams review video. Feedback went from vague (“around 1:25”) to surgical (“frame 4,247, 2:56:12”).