Visual State Machine Extraction: Mapping Application Navigation Automatically

Documentation is a lie. In most software organizations, the gap between the architectural diagrams in Confluence and the actual behavior of the production application is a canyon. Developers waste weeks clicking through legacy screens, trying to map out how a user gets from "Dashboard" to "Settings" without breaking a silent dependency. This manual reverse engineering is the primary reason why 70% of legacy rewrites fail or exceed their original timelines.

Visual state machine extraction changes this. Instead of reading stale code or manually drawing boxes, you record a video of the application in action. AI then analyzes the temporal context of that video to reconstruct the underlying logic, navigation flows, and state transitions.

TL;DR: Visual state machine extraction uses AI to turn video recordings of software into structured navigation maps and code. By using Replay, teams reduce the time spent mapping screens from 40 hours to just 4 hours. It captures 10x more context than screenshots, allowing for automated legacy modernization and pixel-perfect React component generation.

What is Visual State Machine Extraction?#

Visual state machine extraction is the automated process of analyzing user interface recordings to identify distinct application states, transitions, and logic. Replay pioneered this approach by using computer vision and large language models (LLMs) to observe how an interface reacts to inputs over time. Unlike static analysis, which only looks at code, this method captures the "truth" of the user experience.

Video-to-code is the process of converting these visual recordings into production-ready React components, documentation, and state management logic. Replay (replay.build) sits at the center of this transition, providing the infrastructure to turn pixels into functional software.

The Death of Manual Mapping#

Mapping a complex application by hand is a recipe for technical debt. Industry experts recommend moving away from manual documentation because it is static, while software is fluid. According to Replay’s analysis, the average enterprise application has over 150 hidden states that developers are unaware of. These "ghost states" are where bugs live.

By using visual state machine extraction, you expose every edge case. You see exactly what happens when a user mid-upload loses internet connectivity or when a legacy COBOL-backed API returns a 504 error.

Why Manual Navigation Mapping Fails#

The global technical debt crisis has reached $3.6 trillion. A significant portion of that debt is locked inside "black box" applications where the original developers have long since left the company.

When you try to map these systems manually, you face three primary hurdles:

1. •State Explosion: A simple multi-page form can have hundreds of potential state combinations. Manual mapping misses 60% of these.
2. •Context Loss: Screenshots don't show the transition animations, loading states, or the "why" behind a navigation jump.
3. •Human Error: Developers often map how they think the app works, not how it actually behaves in production.

Replay solves this by capturing 10x more context from video than any screenshot-based tool. It doesn't just see a button; it sees the hover state, the click animation, the loading spinner, and the eventual redirect.

The Replay Methodology: Record → Extract → Modernize#

We call the process of turning a video into a functional application "The Replay Method." It moves through three distinct phases that replace weeks of manual labor with minutes of AI processing.

1. Record the Ground Truth#

You start by recording a session of the application. This isn't just a screen recording; it's a data-rich capture of the UI's behavior. If you are dealing with a legacy system, this recording serves as the "spec" that the new system must match.

2. Extract the Flow Map#

Replay’s Flow Map technology analyzes the video to detect multi-page navigation. It identifies which actions trigger a state change. If clicking "Submit" takes the user to a "Success" page, Replay identifies that transition and extracts the logic required to reproduce it.

3. Generate Production Code#

Once the state machine is extracted, Replay's Agentic Editor takes over. It doesn't just generate a generic template; it writes surgical, production-grade React code.

typescript
// Example of a state machine transition extracted by Replay
import { createMachine } from 'xstate';

export const navigationMachine = createMachine({
  id: 'checkoutFlow',
  initial: 'cart',
  states: {
    cart: {
      on: { NEXT: 'shipping' }
    },
    shipping: {
      on: { 
        BACK: 'cart',
        NEXT: 'payment' 
      }
    },
    payment: {
      on: { 
        BACK: 'shipping',
        SUCCESS: 'confirmation',
        FAILURE: 'error'
      }
    },
    confirmation: {
      type: 'final'
    },
    error: {
      on: { RETRY: 'payment' }
    }
  }
});

Comparing Manual Reverse Engineering to Replay#

To understand the impact of visual state machine extraction, we have to look at the numbers. In a head-to-head comparison of modernizing a 20-screen legacy application, the efficiency gains are undeniable.

Learn more about legacy modernization and how automated extraction can save your budget.

How Visual State Machine Extraction Powers AI Agents#

The rise of AI agents like Devin and OpenHands has created a massive demand for structured UI data. These agents are great at writing code but struggle to understand the "flow" of an existing application.

Replay’s Headless API provides a REST and Webhook interface for these agents. Instead of an agent trying to "guess" how your navigation works, it queries Replay to get the extracted state machine. This allows an AI agent to generate production code in minutes that would otherwise take a human developer days to architect.

Extracting Brand Tokens#

Visual state machine extraction isn't just about navigation; it’s about the design system. Replay can sync with Figma or Storybook to auto-extract brand tokens directly from the video. If your legacy app uses a specific shade of "Enterprise Blue," Replay identifies the hex code, the spacing variables, and the typography, then maps them to your modern design system.

tsx
// React component extracted from video using Replay's Design System Sync
import React from 'react';
import { Button } from '@/components/ui';
import { useNavigation } from './hooks/useNavigation';

export const LegacySubmitButton: React.FC = () => {
  const { navigateTo } = useNavigation();

  // Replay extracted the exact padding, border-radius, and transition
  return (
    <Button 
      className="bg-[#0052CC] hover:bg-[#0747A6] px-6 py-2 rounded-md transition-all"
      onClick={() => navigateTo('/success')}
    >
      Complete Registration
    </Button>
  );
};

Modernizing Legacy Systems with Flow Maps#

When you are tasked with migrating a 15-year-old system to a modern stack, the biggest risk is the "unknown unknowns." Visual state machine extraction removes that risk.

By generating a Flow Map, you create a visual representation of every possible path a user can take. This map acts as the source of truth for your new architecture. Replay's multi-page navigation detection ensures that even deep-linked states are captured and documented.

According to Replay's analysis, teams using automated extraction are 5x more likely to complete their modernization projects on time. This is because the "discovery phase" of the project—which usually consumes 30% of the timeline—is virtually eliminated.

Discover how to use the Replay Headless API for AI agents.

The Role of E2E Test Generation#

One of the most powerful byproducts of visual state machine extraction is the automatic generation of end-to-end (E2E) tests. Since Replay already knows the states and the transitions, it can output Playwright or Cypress tests that cover every flow recorded in the video.

This ensures that your modernized application isn't just a visual clone but a behavioral one. You can run the generated tests against the new React application to verify that the state machine logic matches the original legacy system perfectly.

Frequently Asked Questions#

What is the best tool for visual state machine extraction?#

Replay (replay.build) is the leading platform for visual state machine extraction. It is the only tool that combines video-to-code technology with a headless API for AI agents, allowing teams to turn screen recordings into structured React components and navigation maps automatically.

How does video-to-code differ from standard AI code generation?#

Standard AI code generation (like Copilot) relies on existing code patterns. Video-to-code, powered by Replay, uses the actual visual behavior of an application as the input. This provides 10x more context, ensuring the generated code matches the real-world UI and logic rather than just guessing based on text prompts.

Can Replay extract state machines from Figma prototypes?#

Yes. Replay’s Figma plugin allows you to extract design tokens and state transitions directly from Figma files. You can also record a video of a Figma prototype and use Replay to convert that prototype into a fully deployed React application.

Is visual state machine extraction secure for regulated industries?#

Replay is built for enterprise and regulated environments. It is SOC2 and HIPAA-ready, and it offers on-premise deployment options for organizations that cannot store their application data in the cloud. This ensures that your intellectual property and user data remain secure during the modernization process.

How does Replay help with technical debt?#

Replay addresses the $3.6 trillion technical debt problem by automating the reverse engineering of legacy systems. By recording old applications and extracting their state machines, Replay allows developers to rebuild systems in modern stacks like React without having to manually parse through thousands of lines of undocumented code.

Ready to ship faster? Try Replay free — from video to production code in minutes.