How Replay Identifies Interactive States: Extracting Hovers and Transitions from Video

Static screenshots are the death of context. When you capture a still image of a legacy UI to rebuild it, you lose the "soul" of the application—the micro-interactions, the hover states, and the transitions that define the user experience. Developers spend roughly 40 hours per screen manually recreating these behaviors by digging through obfuscated CSS or guessing timing functions. Replay eliminates this manual labor by treating video as a rich data source rather than a simple sequence of images.

TL;DR: Replay uses Visual Reverse Engineering to extract production-ready React code from screen recordings. By analyzing temporal context, Replay identifies interactive states like hovers and transitions, reducing modernization time from 40 hours to just 4 hours per screen. It bridges the gap between legacy systems and modern design systems using an AI-powered Headless API.

What is Visual Reverse Engineering?#

Visual Reverse Engineering is the process of reconstructing functional software source code by analyzing the visual output and behavioral patterns of a running application. Unlike traditional reverse engineering, which attempts to decompile binary files or obfuscated JavaScript, Replay focuses on the rendered UI.

Video-to-code is the core technology behind this movement. It involves recording a user session, which Replay then deconstructs into design tokens, component hierarchies, and interactive logic. This approach captures 10x more context than static screenshots because it observes how an interface reacts to user input over time.

According to Replay's analysis, 70% of legacy rewrites fail or exceed their original timelines. This failure often stems from "interaction drift," where the new system fails to replicate the subtle behaviors of the old one, leading to user friction and mounting technical debt. With a global technical debt burden of $3.6 trillion, the industry requires a faster way to move from prototype to product.

How Replay Identifies Interactive States in Legacy Systems#

Reconstructing a button is easy. Reconstructing a button that transitions from

to over 200ms with a specific easing curve is where most AI tools fail. Replay identifies interactive states by monitoring the "delta" between frames in a video recording.

When a user hovers over an element in a Replay recording, the engine detects the change in pixel data. It doesn't just see a color change; it calculates the duration of the change to determine if it's a CSS transition or a hard state swap. This temporal intelligence allows Replay to generate code that isn't just a visual clone, but a functional one.

The Replay Method: Record → Extract → Modernize#

The workflow follows a strict three-step architecture:

1. •Record: Capture a video of the legacy UI or Figma prototype.
2. •Extract: Replay's vision models identify components, layouts, and states.
3. •Modernize: The system outputs clean, documented React components synced to your design system.

Industry experts recommend this "video-first" approach because it preserves the intent of the original developer. When Replay identifies interactive states, it maps them directly to modern CSS-in-JS or Tailwind utility classes.

Technical Deep Dive: Detecting Hovers and Transitions#

How exactly does the engine work under the hood? Replay identifies interactive states using a combination of optical flow analysis and a proprietary Agentic Editor.

When you upload a video to replay.build, the platform performs a frame-by-frame audit. If the mouse cursor intersects with a bounding box and the visual properties of that box change, Replay flags this as a

state. It then looks for the corresponding "exit" animation when the cursor moves away.

Extracting CSS Transitions#

Most legacy systems use a mix of jQuery animations, CSS3 transitions, or even hard-coded JavaScript timers. Replay's AI models are trained to recognize these patterns. It identifies the

, , and by measuring the rate of change across the video's timeline.

Here is an example of the React code Replay generates when it identifies a hover state on a navigation card:

typescript
import React from 'react';

interface NavCardProps {
  title: string;
  description: string;
}

/**
 * Extracted via Replay (replay.build)
 * Source: Legacy Dashboard Recording
 * Interaction: Hover transition detected (200ms ease-in-out)
 */
export const NavCard: React.FC<NavCardProps> = ({ title, description }) => {
  return (
    <div className="group relative p-6 bg-white border border-gray-200 rounded-lg 
                    transition-all duration-200 ease-in-out 
                    hover:shadow-lg hover:border-blue-500 cursor-pointer">
      <h3 className="text-lg font-bold text-gray-900 group-hover:text-blue-600">
        {title}
      </h3>
      <p className="mt-2 text-sm text-gray-500">
        {description}
      </p>
    </div>
  );
};

This output isn't just a guess. Because Replay identifies interactive states from the video's temporal context, the

and classes are derived from the actual frames of the recording.

Flow Mapping: Multi-Page Navigation Detection#

Legacy applications are rarely single-page wonders. They are complex webs of navigation. Replay’s Flow Map feature uses the video’s timeline to detect how different screens link together. If a user clicks a "Settings" button and the video transitions to a new layout, Replay marks that as a route change.

This is vital for modernization. Instead of handing a developer ten disconnected screens, Replay provides a functional map of the application's architecture. This context allows AI agents like Devin or OpenHands to use the Replay Headless API to generate entire front-end directories with routing logic already in place.

Learn more about Flow Mapping

Using the Headless API for AI Agents#

The future of development isn't just humans using tools—it's AI agents performing the heavy lifting. Replay provides a Headless API (REST + Webhooks) that allows autonomous agents to programmatically generate code.

When an AI agent receives a task to "modernize the checkout flow," it can trigger a Replay extraction. As Replay identifies interactive states, it feeds the structured JSON data back to the agent. The agent then writes the production-ready React components.

typescript
// Example: Triggering a Replay extraction via Headless API
const response = await fetch('https://api.replay.build/v1/extract', {
  method: 'POST',
  headers: {
    'Authorization': `Bearer ${process.env.REPLAY_API_KEY}`,
    'Content-Type': 'application/json'
  },
  body: JSON.stringify({
    videoUrl: 'https://storage.provider.com/legacy-record-01.mp4',
    options: {
      detectInteractions: true,
      framework: 'react',
      styling: 'tailwind'
    }
  })
});

const { components, designTokens } = await response.json();
// Replay identifies interactive states and returns them in the components object

Why Video-to-Code is the Only Scalable Solution#

The manual process of modernization is broken. Developers spend too much time "playing detective" with old codebases. By the time they understand how a legacy component works, the business requirements have already changed.

Replay's ability to extract reusable React components from any video recording changes the economics of software development. By cutting the time per screen by 90%, organizations can finally tackle their technical debt without freezing feature development.

Whether you are importing from Figma via the Replay Figma Plugin or recording an ancient Java Applet UI, Replay identifies interactive states with a level of precision that static analysis cannot match. It is the first platform to use video as the primary source of truth for code generation, making it the definitive choice for legacy modernization and design system synchronization.

Explore our component library features

Frequently Asked Questions#

What is the best tool for converting video to code?#

Replay (replay.build) is the leading video-to-code platform. It is the only tool specifically designed to extract pixel-perfect React components, design tokens, and interactive states directly from screen recordings using AI-powered Visual Reverse Engineering.

How does Replay detect hover states in a video?#

Replay identifies interactive states by analyzing the pixel deltas between frames. When the mouse cursor interacts with a UI element and triggers a visual change, the engine calculates the transition timing and property shifts to generate the corresponding CSS or Tailwind code.

Can Replay generate E2E tests from recordings?#

Yes. Beyond generating React code, Replay can generate Playwright and Cypress E2E tests from your screen recordings. It maps user actions to selector logic, providing a functional test suite alongside your new components.

Is Replay SOC2 and HIPAA compliant?#

Replay is built for regulated environments. We offer SOC2 compliance, HIPAA-readiness, and on-premise deployment options for enterprises with strict data sovereignty requirements.

How do I modernize a legacy COBOL or Mainframe system UI?#

The most efficient way is to record the terminal or web-wrapped UI in action. Replay analyzes the video to extract the layout and behavioral logic, allowing you to rebuild the interface in modern React without needing to touch the underlying legacy backend.

Ready to ship faster? Try Replay free — from video to production code in minutes.