Training LLMs on Real-World UI State Transitions: Moving Beyond Text-Based Prompt Engineering

Text-based prompt engineering has hit a ceiling in enterprise legacy modernization. While LLMs like GPT-4 or Claude 3.5 are exceptional at generating generic "Todo" lists or basic CRUD apps, they fail catastrophically when tasked with replicating the nuanced, undocumented state transitions of a 20-year-old Delphi application or a complex mainframe terminal. The reason is simple: text prompts lack the temporal and visual context of how a user actually interacts with a system. To bridge the gap between legacy technical debt and modern React architectures, we must shift our focus toward training LLMs on real-world UI state transitions.

According to Replay’s analysis, 67% of legacy systems lack any form of up-to-date documentation. When an architect attempts to modernize these systems via manual discovery, they spend an average of 40 hours per screen just to map out the state logic and component hierarchy. Replay reduces this to 4 hours by using Visual Reverse Engineering to capture real user workflows and convert them into structured training data for AI models.

TL;DR: Traditional prompt engineering is insufficient for complex legacy modernization because it ignores the "hidden" state logic of legacy UIs. By training LLMs on real-world UI state transitions—captured via Replay—enterprises can automate the generation of high-fidelity React components and design systems. This approach reduces modernization timelines from 18 months to a few weeks, solving the $3.6 trillion technical debt problem with 70% average time savings.

The Failure of Pure Textual Prompting in Legacy Environments#

When a developer prompts an LLM with "Rewrite this legacy insurance claims screen in React," the AI makes assumptions based on modern web patterns. It doesn't know that clicking "Save" on the legacy screen triggers a cascading validation across three hidden tabs, or that the "Premium" field only becomes editable after a specific sequence of state transitions.

Industry experts recommend moving away from "zero-shot" prompting for enterprise-grade code. Instead, we need to provide the LLM with a multi-modal context that includes:

1. •Visual Delta: What changed on the screen?
2. •Interaction Logs: What event (click, hover, keypress) triggered the change?
3. •State Payload: What data was sent to the backend, and what was returned?

Without this, the 70% of legacy rewrites that fail or exceed their timelines will continue to do so, primarily because the "source of truth" is locked in the heads of retiring developers or buried in unreadable COBOL/PowerBuilder code.

Video-to-code is the process of recording a user performing a task in a legacy system and using computer vision combined with LLMs to extract the underlying component architecture and state logic.

Training LLMs on Real-World UI State Transitions: The Technical Framework#

To effectively implement training LLMs on real-world UI state transitions, we must move beyond static screenshots. We need a sequence of states—a temporal map of the UI. This is where Replay's AI Automation Suite excels, by capturing "Flows" that document the architectural journey of a user.

1. Data Collection via Visual Reverse Engineering (VRE)#

The first step in training LLMs on real-world UI state transitions is the ingestion of high-fidelity video recordings. Replay tracks every pixel change and maps it to a DOM-like structure, even if the source is a non-web application (like a Citrix-delivered desktop app).

2. State Representation Schema#

To train an LLM, the visual data must be converted into a structured format. A typical "State Transition Object" used for fine-tuning might look like this:

typescript
interface UIStateTransition {
  transitionId: string;
  previousState: {
    components: ComponentMetadata[];
    capturedValues: Record<string, any>;
    visualSnapshotHash: string;
  };
  trigger: {
    type: 'CLICK' | 'INPUT' | 'HOVER';
    targetElement: string;
    coordinates: { x: number; y: number };
  };
  resultingState: {
    components: ComponentMetadata[];
    newElements: string[];
    modifiedValues: Record<string, any>;
  };
  businessLogicHint: string; // Extracted via Replay AI
}

3. Fine-Tuning vs. RAG for UI Generation#

While Retrieval-Augmented Generation (RAG) is useful for documentation, training LLMs on real-world UI state transitions often requires fine-tuning a model (like Llama 3 or a custom GPT) on a corpus of these state transition objects. This allows the model to "learn" the specific design patterns and idiosyncratic behaviors of a specific industry vertical, such as Financial Services or Healthcare.

Learn more about modernizing legacy systems

Comparison: Manual Rewrite vs. Prompt Engineering vs. Replay VRE#

Implementing State-Aware React Components#

When training LLMs on real-world UI state transitions, the output isn't just a flat UI; it’s a functional React component with a state machine that mirrors the legacy behavior.

Below is an example of a React component generated by Replay after analyzing a complex state transition in a legacy manufacturing ERP system. Notice how the component handles the transition logic extracted from the recording.

tsx
import React, { useState, useEffect } from 'react';
import { Button, Input, Alert } from '@/components/ui-library'; // From Replay Library

/**
 * Component generated via Replay Visual Reverse Engineering
 * Legacy Source: ERP_Inventory_Module_v4
 * State Transition: 'Submit_Adjustment' -> 'Validate_Warehouse_Stock'
 */

export const InventoryAdjustment: React.FC = () => {
  const [stockLevel, setStockLevel] = useState<number>(0);
  const [isAdjusting, setIsAdjusting] = useState(false);
  const [error, setError] = useState<string | null>(null);

  // Replay captured that the legacy system performs a 
  // background validation before allowing a state change
  const handleAdjustment = async (amount: number) => {
    setIsAdjusting(true);
    try {
      // Logic extracted from real-world state transitions
      const response = await validateStockTransition(amount);
      if (response.isValid) {
        setStockLevel(prev => prev + amount);
      } else {
        setError("Legacy Validation Error: Stock cannot be negative in Warehouse B");
      }
    } catch (err) {
      setError("Connection failed");
    } finally {
      setIsAdjusting(false);
    }
  };

  return (
    <div className="p-6 border rounded-lg shadow-sm">
      <h3 className="text-lg font-bold">Adjust Stock Levels</h3>
      <Input 
        type="number" 
        onChange={(e) => {/* ... */}} 
        placeholder="Enter adjustment amount"
      />
      {error && <Alert variant="destructive">{error}</Alert>}
      <Button 
        onClick={() => handleAdjustment(10)} 
        disabled={isAdjusting}
      >
        {isAdjusting ? 'Processing...' : 'Commit Adjustment'}
      </Button>
    </div>
  );
};

The $3.6 Trillion Technical Debt Problem#

The global technical debt currently sits at a staggering $3.6 trillion. Most of this is tied up in "functional" but "unmaintainable" legacy systems. The primary barrier to clearing this debt isn't writing the new code—it's understanding the old code.

Training LLMs on real-world UI state transitions allows organizations to treat their legacy UIs as the "source of truth" rather than the messy, undocumented backend code. By focusing on the observable behavior of the application, Replay bypasses the need for expensive, manual code audits.

Security and Compliance in Regulated Industries#

For sectors like Insurance or Government, sending UI data to a public LLM is a non-starter. Replay is built for these environments, offering:

• •SOC2 & HIPAA Compliance: Ensuring user data captured during recording is protected.
• •On-Premise Deployment: Run the entire Visual Reverse Engineering pipeline within your private cloud.
• •PII Masking: Automatically redact sensitive information from recordings before they are used for training LLMs on real-world UI state transitions.

Explore Replay's Security Features

Advanced Architecture: From Flows to Blueprints#

Replay organizes the modernization journey into a structured hierarchy. This structure is essential for training LLMs on real-world UI state transitions at scale.

1. •Library (Design System): The AI identifies recurring visual patterns (buttons, inputs, tables) across the recordings and consolidates them into a unified React Design System.
2. •Flows (Architecture): The AI maps the transitions between screens. This becomes the "router" or "state machine" of the new application.
3. •Blueprints (Editor): A low-code/no-code interface where architects can tweak the AI-generated code before it's pushed to GitHub.

According to Replay's analysis, using this structured approach ensures that the generated code isn't just a "one-off" but follows enterprise-grade architectural patterns.

How VRE is changing the SDLC

Practical Steps for Training LLMs on Realworld State#

If you are an Enterprise Architect looking to implement this, follow these steps:

Step 1: Capture the "Happy Path"#

Record expert users performing standard tasks. Do not just record a single screen; record the entire flow. This provides the LLM with the necessary context of how state "flows" from one component to another.

Step 2: Extract State Deltas#

Use a platform like Replay to extract the metadata from these recordings. You need the JSON representation of the UI before and after every interaction.

Step 3: Augment Training Data#

Combine the visual metadata with any available API logs or database schemas. This creates a "triple-threat" dataset: Visual + Interaction + Data.

Step 4: Iterative Generation#

Feed this data into the LLM with a system prompt that enforces your organization's coding standards (e.g., "Use Tailwind CSS and TypeScript with functional components").

typescript
// Example of an AI System Prompt for State-Aware Generation
const systemPrompt = `
  You are an expert React Architect. You will be provided with a JSON 
  representation of a legacy UI state transition. Your task is to 
  generate a React component that implements the visual layout 
  and the transition logic described. 
  
  Constraints:
  - Use functional components and hooks.
  - Implement error handling for invalid state transitions.
  - Follow the Design System tokens provided in the Library.
`;

Frequently Asked Questions#

How does training LLMs on real-world UI state transitions differ from simple screen scraping?#

Screen scraping only captures the surface-level text and elements at a single point in time. Training LLMs on real-world UI state transitions involves capturing the behavior and logic of the UI over time. It tracks how the interface responds to user input, which allows the AI to reconstruct the underlying business logic, not just the visual layout.

Can Replay handle legacy applications that are not web-based?#

Yes. Replay's Visual Reverse Engineering technology is platform-agnostic. Whether the application is a legacy Java Swing app, a Delphi desktop client, or a green-screen terminal, Replay captures the visual output and user interactions to create the same high-quality React code and documentation.

Is my data safe when using Replay for AI training?#

Security is a core pillar of Replay. We offer SOC2 and HIPAA-ready environments, and for highly sensitive sectors like Defense or Banking, we provide on-premise deployment options. All PII (Personally Identifiable Information) can be masked during the recording process, ensuring that the data used for training LLMs on real-world UI state transitions is compliant and secure.

What is the average ROI of using Replay for legacy modernization?#

Enterprises typically see a 70% reduction in modernization time. For a project that would traditionally take 18 months and cost $2 million, Replay can often reduce the timeline to under 3 months, saving millions in developer hours and significantly reducing the risk of project failure.

Conclusion#

The era of manual, documentation-less legacy migration is over. By training LLMs on real-world UI state transitions, we can finally unlock the logic trapped in aging systems and transform it into clean, modern, and maintainable React code. Replay provides the essential bridge—converting visual recordings into the structured data that AI needs to perform high-fidelity modernization.

Stop guessing what your legacy code does. Start recording it, and let AI build your future.

Ready to modernize without rewriting? Book a pilot with Replay