# Video Interaction Mapper

Turn a UI recording into a static, annotated Figma storyboard. Extract the important
before/after states, infer what triggered each change, then place clean screenshots,
blue interaction markers, and concise annotations into a Figma Design file.

## Required Inputs and Tools

Start with:

- A local video file path (`.mp4`, `.mov`, `.webm`, or `.avi`).
- `ffmpeg` and `ffprobe` available on the machine.
- Python with Pillow installed.
- Figma MCP tools available: `create_new_file`, `use_figma`, `upload_assets`, and
  `get_screenshot` or `get_design_context`.

Run the bundled files in `scripts/` as executable workflow helpers. They are part
of the skill's implementation, not reference material. Read or modify them only
when debugging, adapting to an unusual environment, or changing the skill itself.

Use a Figma Design file (`figma.com/design/...`) as the target. The generated
Plugin API code creates pages and image-fill frames, which are Design-file
operations. If the user provides a FigJam or Slides URL, ask for a Design file or
create a new one.

Before any `use_figma` call, load the Figma API guidance skill if available
(`figma-use`). Before creating a new Figma file, load the file-creation guidance
skill if available (`figma-create-new-file`). Pass `skillNames:
"figma-use,video-interaction-mapper"` on `use_figma` calls when the client supports
that parameter.

## Workflow

### 1. Validate and Prepare

Confirm the video path exists, then check `ffmpeg -version`. If Pillow is missing,
install it in the active Python environment:

```bash
pip install Pillow --break-system-packages -q
```

Do local analysis before creating or modifying the Figma file. Create a Figma file
only after `key_moments.json`, `upload_manifest.json`, and generated Figma scripts
are ready, unless the user explicitly asks to create the file first. This avoids
leaving partial pages behind when local frame analysis changes.

### 2. Scout the Timeline Quickly

Start with a low-resolution scout pass and contact sheet:

```bash
python <SKILL_DIR>/scripts/extract_key_frames.py \
  --input "<video_path>" \
  --output /tmp/vim_frames_<timestamp>/ \
  --mode scout
```

The scout pass extracts at 1 fps by default, resizes frames to 640 px wide, scores
scene changes, writes `<output_dir>/frames_manifest.json`, and creates
`<output_dir>/contact_sheet.jpg`. Inspect the contact sheet first to identify likely
interaction moments.

Use the slower full extraction only when the scout pass is insufficient:

```bash
python <SKILL_DIR>/scripts/extract_key_frames.py \
  --input "<video_path>" \
  --output /tmp/vim_frames_<timestamp>/full_frames/ \
  --mode full \
  --max-width 1600
```

For very long videos, ask whether to focus on a specific time range before
processing the whole recording. For too many key frames, re-run extraction with a
higher `--scene-threshold`, such as `0.4`.

### 3. Analyze Key Moments

Read `frames_manifest.json` and inspect `contact_sheet.jpg` or relevant extracted
frames. For each meaningful transition, identify:

- Visual change: modal, drawer, dropdown, tooltip, snackbar, overlay, navigation,
  loaded content, selected state, validation state, input focus, or scroll position.
- Trigger: click, tap, keyboard input, hover, scroll, swipe, form submit, auto/timer,
  or unknown trigger when the cause is ambiguous.
- Interaction target and result coordinates when visible and useful.

Use normalized coordinates from `0` to `1` relative to the screenshot. Percent-like
values such as `45` or `94` are accepted by the generated Figma script and converted
to `0.45` or `0.94`. Coordinates are machine metadata for marker placement only.
Do not include coordinate values in visible annotation text, moment labels, or user
summaries.

Only add marker coordinates when the point lands on a clearly visible UI element or
state. If a target is inferred, between visible controls, or based mainly on cursor
position, mark it with `inferred: true`, `uncertain: true`, `visible: false`, or
`confidence: "medium"`/`"low"`. The generated script suppresses markers for those
points by default so the canvas does not show a confident pulse over empty UI. If
the target or result cannot be located usefully, set `interaction` to `null`.

Use `confidence: "high"` or omit `confidence` only when the marker should be drawn.
Use `show_marker: true` only when an uncertain point is still worth marking; the
generated marker uses an alternate style for these forced uncertain markers.

Aim for 5 to 15 key moments. Prioritize high scene scores and genuinely different
states over tiny visual shifts.

Write the analysis to `<output_dir>/key_moments.json` as an array. When using the
scout flow, provide timestamps first; resolve high-quality frame files in the next
step.

```json
[
  {
    "moment_index": 1,
    "timestamp_s": 3.4,
    "before_timestamp_s": 3.2,
    "after_timestamp_s": 3.4,
    "trigger": "Click",
    "trigger_emoji": "",
    "short_lab

…