Terminal Detection in Screen Recordings: How VidNo Reads Your Code

• Code editor: Identified by syntax highlighting, line numbers, and the characteristic layout of editor UIs (tab bar, sidebar, status bar).
• Terminal/console: Identified by monospace text, command prompts ($ or >), and the absence of syntax highlighting variety (usually single-color output with occasional ANSI colors).
• Browser: Identified by the URL bar, tab strip, and rendered HTML content. DevTools are identified separately by their characteristic panel layout.
• File explorer/sidebar: Identified by the tree structure, file icons, and narrow width.
• Dialog/popup: Identified by overlay characteristics -- drop shadows, distinct backgrounds, and smaller bounding boxes on top of other regions.

The detection model was trained on thousands of screenshots from VS Code, JetBrains IDEs, Vim/Neovim, terminal emulators (iTerm, Alacritty, Windows Terminal), and all major browsers.

Region-Specific OCR Processing

Once regions are identified, each gets specialized OCR processing:

Code editor regions:

• Line numbers are detected and stripped (they are metadata, not code)
• Syntax highlighting colors are used to improve character recognition accuracy (a blue word is likely a keyword, an orange word is likely a string)
• Indentation is preserved precisely -- in code, whitespace is semantic
• Special characters common in code (=>, !==, &&, ||) have enhanced recognition models

Terminal regions:

• Command prompts are identified to separate commands from output
• ANSI color codes are interpreted (red text usually indicates errors)
• Scrollback buffer changes are tracked to identify new output vs. already-processed text
• Common terminal patterns (file paths, URLs, error codes) have enhanced recognition

Browser regions:

• URL bar content is extracted (indicates what documentation or API the developer is referencing)
• Rendered page content is processed differently from code (proportional fonts, varied layouts)
• DevTools panels receive code-specific OCR treatment similar to editor regions

Temporal Tracking

Screen recording is not a series of independent frames. It is a continuous stream where most pixels do not change between frames. VidNo exploits this:

• Change detection: Only regions that changed between frames are re-processed. If the editor content did not change, the previous OCR result is reused.
• Incremental text tracking: When the developer types in the editor, the system tracks character-by-character additions rather than re-OCRing the entire file.
• Scroll tracking: When content scrolls, the system identifies it as a scroll (not new content) and adjusts the OCR context accordingly.
• Window switching: When the active window changes (alt-tab), the system recognizes this as a context switch and updates its region map.

Output: Structured Context for Script Generation

The OCR pipeline outputs structured data for each segment of the recording:

{
  "timestamp": "00:03:24",
  "regions": {
    "editor": {
      "language": "typescript",
      "file": "src/auth/middleware.ts",
      "visible_code": "...",
      "changes_since_last": ["added line 24-28"]
    },
    "terminal": {
      "last_command": "npm run test",
      "output_type": "error",
      "output_text": "TypeError: Cannot read property..."
    }
  }
}

This structured context feeds into the script generation pipeline, where it is combined with git diff data to produce narration that accurately describes what the developer is doing and seeing at each moment in the recording.

The OCR pipeline is not glamorous work. It is the foundation that makes everything else possible -- without accurate text extraction from developer screens, intelligent narration would be impossible.