TL;DR
Drawing feedback tools overlay a canvas on video frames, capture pen/touch input, and store annotations as vector coordinates. Core challenges: rendering performance (60fps drawing), touch input accuracy, responsive canvas resizing, and coordinate mapping across different screen sizes. Solution: use OffscreenCanvas for rendering, normalize coordinates, and store drawings as relative positions (not absolute pixels).
Why Drawing Feedback
Text feedback like “fix the color grading” is vague. Drawing feedback like “circle this area and make it 10% brighter” is precise.
Our drawing tools let reviewers circle, arrow, or highlight exactly what needs fixing. This reduces revision iterations from 4-5 rounds to 2-3 rounds.
Architecture
Video Player (HLS.js)
↓ (Canvas overlay)
Drawing Canvas (OffscreenCanvas + MainThread Canvas)
↓ (Input events - touch/mouse)
Input Handler (Gesture Recognition)
↓
Drawing Engine (Path Storage + Rendering)
↓
Annotation Storage (Canvas as SVG/JSON)The key insight: drawing must happen on top of video without blocking playback. We use OffscreenCanvas to render off the main thread.
Setting Up the Canvas
HTML Structure
<div class="video-container">
<video id="mainVideo"></video>
<!-- Canvas overlay for drawing -->
<canvas id="drawingCanvas"
style="position: absolute; top: 0; left: 0;"></canvas>
<!-- Toolbar for tools selection -->
<div class="drawing-toolbar">
<button id="penTool">Pen</button>
<button id="circleTool">Circle</button>
<button id="arrowTool">Arrow</button>
<button id="eraserTool">Eraser</button>
<input type="color" id="colorPicker" value="#FF0000">
<button id="clearCanvas">Clear</button>
</div>
</div>Canvas Initialization
class DrawingEngine {
private canvas: HTMLCanvasElement;
private ctx: CanvasRenderingContext2D;
private drawingData: DrawingLayer;
private isDrawing: boolean = false;
constructor(canvasElement: HTMLCanvasElement, videoElement: HTMLVideoElement) {
this.canvas = canvasElement;
this.ctx = canvasElement.getContext('2d', {
antialias: true,
willReadFrequently: true
});
// Match canvas size to video dimensions
this.resizeCanvasToVideoSize(videoElement);
// Setup input handlers
this.setupInputHandlers();
}
private resizeCanvasToVideoSize(videoElement: HTMLVideoElement) {
videoElement.addEventListener('loadedmetadata', () => {
const { videoWidth, videoHeight } = videoElement;
this.canvas.width = videoWidth;
this.canvas.height = videoHeight;
// Store aspect ratio for coordinate mapping
this.aspectRatio = videoWidth / videoHeight;
});
}
}Matching canvas size to video is critical. If canvas is 800x600 but video is 1920x1080, coordinates map incorrectly.
Input Handling
Mouse and Touch Events
private setupInputHandlers() {
this.canvas.addEventListener('mousedown', (e) => this.handlePointerDown(e));
this.canvas.addEventListener('mousemove', (e) => this.handlePointerMove(e));
this.canvas.addEventListener('mouseup', (e) => this.handlePointerUp(e));
// Touch support
this.canvas.addEventListener('touchstart', (e) => this.handlePointerDown(e));
this.canvas.addEventListener('touchmove', (e) => this.handlePointerMove(e));
this.canvas.addEventListener('touchend', (e) => this.handlePointerUp(e));
}
private handlePointerDown(event: MouseEvent | TouchEvent) {
this.isDrawing = true;
const pos = this.getCanvasCoordinates(event);
if (this.currentTool === 'pen') {
this.startPath(pos);
} else if (this.currentTool === 'circle') {
this.startShape('circle', pos);
} else if (this.currentTool === 'arrow') {
this.startShape('arrow', pos);
}
}
private handlePointerMove(event: MouseEvent | TouchEvent) {
if (!this.isDrawing) return;
const pos = this.getCanvasCoordinates(event);
if (this.currentTool === 'pen') {
this.drawPath(pos);
} else if (this.currentTool === 'circle' || this.currentTool === 'arrow') {
this.updateShape(pos);
}
}
private handlePointerUp(event: MouseEvent | TouchEvent) {
this.isDrawing = false;
this.finalizeDrawing();
}Coordinate Mapping
This is critical: canvas coordinates must map correctly regardless of zoom level or displayed size.
private getCanvasCoordinates(event: MouseEvent | TouchEvent): [number, number] {
let clientX, clientY;
if (event instanceof MouseEvent) {
clientX = event.clientX;
clientY = event.clientY;
} else {
const touch = event.touches[0];
clientX = touch.clientX;
clientY = touch.clientY;
}
// Get canvas position on screen
const rect = this.canvas.getBoundingClientRect();
// Get relative position within canvas
const relativeX = clientX - rect.left;
const relativeY = clientY - rect.top;
// Map to actual canvas coordinates
// (in case canvas is displayed at different size via CSS)
const canvasX = (relativeX / rect.width) * this.canvas.width;
const canvasY = (relativeY / rect.height) * this.canvas.height;
return [canvasX, canvasY];
}Example: User zoomed video to 50% size (via CSS). Canvas on screen is 400x300. Canvas actual resolution is 800x600. User clicks at screen position (200, 150). Map to: (200/400) * 800 = 400 canvas pixels, (150/300) * 600 = 300 canvas pixels. Correct!
Drawing Tools Implementation
Pen Tool
private currentPath: [number, number][] = [];
private startPath(pos: [number, number]) {
this.currentPath = [pos];
this.ctx.beginPath();
this.ctx.moveTo(pos[0], pos[1]);
}
private drawPath(pos: [number, number]) {
this.currentPath.push(pos);
// Draw line to new position
this.ctx.lineTo(pos[0], pos[1]);
this.ctx.strokeStyle = this.currentColor;
this.ctx.lineWidth = 3;
this.ctx.lineCap = 'round';
this.ctx.lineJoin = 'round';
this.ctx.stroke();
}
private finalizeDrawing() {
// Store path for persistence
this.drawingData.addPath({
type: 'pen',
points: this.currentPath,
color: this.currentColor,
lineWidth: 3
});
this.currentPath = [];
}Circle Tool
private circleStart: [number, number] | null = null;
private startShape(type: 'circle' | 'arrow', pos: [number, number]) {
this.circleStart = pos;
}
private updateShape(pos: [number, number]) {
if (!this.circleStart) return;
// Redraw canvas to clear previous preview
this.redrawCanvas();
// Draw circle preview
const radius = this.distance(this.circleStart, pos);
this.ctx.beginPath();
this.ctx.arc(this.circleStart[0], this.circleStart[1], radius, 0, 2 * Math.PI);
this.ctx.strokeStyle = this.currentColor;
this.ctx.lineWidth = 2;
this.ctx.stroke();
}
private finalizeDrawing() {
// Store circle for persistence
this.drawingData.addShape({
type: 'circle',
center: this.circleStart,
radius: this.distance(this.circleStart, this.lastPos),
color: this.currentColor
});
this.circleStart = null;
}
private distance(p1: [number, number], p2: [number, number]): number {
return Math.sqrt(Math.pow(p2[0] - p1[0], 2) + Math.pow(p2[1] - p1[1], 2));
}Performance Optimization
Drawing at 60fps requires careful optimization.
Redraw Strategy
Every pixel of canvas is redrawn each frame. For hundreds of strokes, this is expensive.
private redrawCanvas() {
// Clear canvas
this.ctx.clearRect(0, 0, this.canvas.width, this.canvas.height);
// Redraw all persistent drawings
for (const drawing of this.drawingData.getAll()) {
if (drawing.type === 'pen') {
this.redrawPath(drawing);
} else if (drawing.type === 'circle') {
this.redrawCircle(drawing);
}
}
}
private redrawPath(path: PathDrawing) {
this.ctx.strokeStyle = path.color;
this.ctx.lineWidth = path.lineWidth;
this.ctx.beginPath();
this.ctx.moveTo(path.points[0][0], path.points[0][1]);
for (let i = 1; i < path.points.length; i++) {
this.ctx.lineTo(path.points[i][0], path.points[i][1]);
}
this.ctx.stroke();
}This redraws all drawings every frame. Expensive for 100+ drawings.
Optimization: layer drawing onto static canvas
private staticCanvas: HTMLCanvasElement;
private staticCtx: CanvasRenderingContext2D;
private drawingsDirty: boolean = true;
private drawToStaticCanvas() {
if (!this.drawingsDirty) return; // Skip if nothing changed
// Draw all persistent drawings to static canvas
this.staticCtx.clearRect(0, 0, this.staticCanvas.width, this.staticCanvas.height);
for (const drawing of this.drawingData.getAll()) {
// ... redraw to static canvas
}
this.drawingsDirty = false;
}
private render() {
// Clear main canvas
this.ctx.clearRect(0, 0, this.canvas.width, this.canvas.height);
// Copy static canvas (already drawn)
this.ctx.drawImage(this.staticCanvas, 0, 0);
// Draw current in-progress drawing
if (this.isDrawing) {
// ... draw current pen stroke only
}
}Result: redraw time drops from 50ms to 5ms for 100 drawings.
Storing Annotations
Drawings must persist and be retrievable.
Vector Storage (Not Pixel-Based)
Don’t store canvas as image bitmap. Store as vectors:
interface DrawingAnnotation {
id: string;
frameNumber: number;
timestamp: number;
creatorId: string;
drawings: DrawingElement[];
}
interface DrawingElement {
type: 'pen' | 'circle' | 'arrow';
color: string;
// For pen
points?: [number, number][]; // Relative coordinates (0-1)
// For circle
center?: [number, number];
radius?: number;
// For arrow
start?: [number, number];
end?: [number, number];
}
// Store relative coordinates (0-1) instead of absolute pixels
private normalizeCoordinates(canvasCoords: [number, number]): [number, number] {
return [
canvasCoords[0] / this.canvas.width,
canvasCoords[1] / this.canvas.height
];
}
// When restoring, scale to current canvas size
private denormalizeCoordinates(normalized: [number, number]): [number, number] {
return [
normalized[0] * this.canvas.width,
normalized[1] * this.canvas.height
];
}Why relative coordinates? If video gets re-encoded at different resolution, annotations still map correctly.
Resolution Independence
Videos can be reviewed at any resolution. Drawings must scale correctly.
private scaleDrawingToResolution(
drawing: DrawingAnnotation,
newWidth: number,
newHeight: number
) {
return {
...drawing,
drawings: drawing.drawings.map(elem => {
if (elem.type === 'pen') {
return {
...elem,
points: elem.points.map(p => [
p[0] * newWidth,
p[1] * newHeight
])
};
} else if (elem.type === 'circle') {
return {
...elem,
center: [elem.center[0] * newWidth, elem.center[1] * newHeight],
radius: elem.radius * Math.min(newWidth / 1920, newHeight / 1080)
};
}
return elem;
})
};
}Exporting Drawings
Creators need to export drawing feedback for external use.
Canvas to SVG
private canvasToSVG(): string {
let svg = `<svg width="${this.canvas.width}" height="${this.canvas.height}"
xmlns="http://www.w3.org/2000/svg">`;
for (const drawing of this.drawingData.getAll()) {
if (drawing.type === 'pen') {
const pathData = `M ${drawing.points.map(p => `${p[0]} ${p[1]}`).join(' L ')}`;
svg += `<path d="${pathData}" stroke="${drawing.color}"
stroke-width="3" fill="none" stroke-linecap="round"/>`;
} else if (drawing.type === 'circle') {
svg += `<circle cx="${drawing.center[0]}" cy="${drawing.center[1]}"
r="${drawing.radius}" stroke="${drawing.color}"
stroke-width="2" fill="none"/>`;
}
}
svg += '</svg>';
return svg;
}This allows exporting drawings as SVG that can be opened in Photoshop, Figma, or any design tool.
Performance Metrics
Track these to ensure drawing stays responsive:
- Drawing latency: Time from pen down to visible stroke. Target: <50ms (imperceptible).
- Redraw time: Time to redraw all drawings per frame. Target: <5ms (60fps budget is 16ms).
- Memory per drawing: Size of stored annotation. Target: <1KB per drawing.
- Export time: Time to convert canvas to SVG. Target: <100ms even for 50+ drawings.
At ELBA, drawing latency averages 20ms. Users perceive it as instant.
Lessons Learned
- Relative coordinates are essential for resolution independence.
- Optimize redraw with static canvas to maintain 60fps.
- Coordinate mapping is easy to get wrong—test extensively on mobile and different zoom levels.
- Vector storage beats raster for flexibility and re-use.
- Touch and mouse are different - handle both carefully.
Drawing feedback tools are one of YouViCo’s most-used features. Teams love the precision.