Skip to main content

​
Overview

Once connected to Desktop, Qaptly Browser Pilot enables you to record user interactions, analyze web pages, and execute automated test sequences. This guide covers all major features and workflows.

​
Recording User Behavior

​
Starting Recording

  1. Navigate to the website you want to test
  2. Open Side Panel → Monitor tab
  3. Locate the Behavior Tracking section
  4. Toggle the “Enable tracking” switch to ON
  5. Begin interacting with the webpage

​
What Gets Recorded

When tracking is enabled, the extension captures: Click Events:
  • Mouse clicks on any element
  • Click coordinates (viewport-relative)
  • Target element details (tag, ID, classes, attributes)
  • Timestamp of each click
Input Events:
  • Text input in form fields
  • Element properties (placeholder, type, value)
  • Input field identifiers
Scroll Events:
  • Page scroll position changes
  • Viewport dimensions
  • Document dimensions
Toggle Events:
  • Checkbox state changes
  • Radio button selections
  • Switch/toggle interactions
Navigation Events:
  • Page URL changes
  • Navigation type (link click, form submit, etc.)
Mouse Move Events (when enabled via Desktop):
  • Cursor position tracking
  • Movement patterns

​
Viewing Statistics

Real-time event counters display in the Behavior Tracking section:
  • Mouse Move: Number of movement events captured
  • Click: Total clicks recorded
  • Input: Text input events
  • Scroll: Scroll actions
  • Toggle: Checkbox/radio changes

​
Stopping Recording

Toggle the “Enable tracking” switch to OFF to stop capturing events. Note: Recorded data is automatically streamed to Desktop in batches (max 50 events per batch) and cannot be viewed directly in the extension.

​
Page Analysis

​
Analyzing Page Elements

Page analysis identifies interactive elements and captures page structure:
  1. Navigate to the target page
  2. Open Side Panel → Monitor tab
  3. Click “Analyze” button
  4. Wait for analysis to complete (usually 1-3 seconds)
Analysis Results Include:
  • All interactive elements (buttons, links, inputs, forms)
  • Element positions and dimensions
  • Visibility and interactability status
  • Page framework detection (React, Vue, Angular)
  • Automatic page screenshot
  • Element hierarchy and relationships
Results are sent to Desktop for AI processing.

​
Marking Elements

After analysis, you can highlight elements on the page:
  1. Click “Mark” button
  2. Analyzed elements will be highlighted with colored overlays
  3. Helps verify what the extension detected

​
Clearing Highlights

Click “Clear” button to remove all element highlights from the page.

​
Screenshot Capture

​
Manual Screenshots

Capture the current page state anytime:
  1. Open Side Panel → Monitor tab
  2. Click “Take Snapshot” button
  3. Screenshot is captured and sent to Desktop immediately
Screenshot Details:
  • Full visible viewport
  • Includes device pixel ratio (retina displays)
  • PNG format, base64 encoded
  • Includes page title and URL metadata

​
Automatic Screenshots

Enable auto-screenshots during action execution:
  1. Switch to Command tab
  2. Toggle “Auto Screenshot” checkbox
  3. Screenshots will be captured at each execution step

​
Action Execution

​
Understanding Action Execution

When Desktop sends a test sequence to execute:
  1. Actions appear in Command tab’s Action List
  2. Each action shows:
    • Action type (click, input, scroll, navigate, etc.)
    • Target element information
    • Parameters (coordinates, text values, etc.)

​
Starting Execution

  1. Open Side Panel → Command tab
  2. Review the action list
  3. Enable “Auto Screenshot” if you want step-by-step captures
  4. Click “Execute” button
  5. Watch the automation run

​
Visual Feedback During Execution

The extension provides visual indicators during execution: Page Border Indicator:
  • Orange border: Appears around the entire page during action execution
  • Signals that automation is in progress
  • Automatically disappears when execution completes
Action Effects (per action type):
  • Click: Orange ripple effect at click location
  • Input: Blue ripple effect on input field
  • Toggle: Purple ripple effect on checkbox/radio
Action Position Indicator:
  • A position marker appears briefly at the action target location
  • Helps visualize where actions are being performed

​
Stopping Execution

If you need to interrupt execution:
  1. Click “Stop” button in Command tab
  2. Current action completes, then sequence stops
  3. Status returns to idle

​
Execution Status

The Action List shows status for each action:
  • Pending: Waiting to execute
  • In Progress: Currently executing
  • Completed: Successfully executed
  • Failed: Execution error occurred

​
Interactive Selection

​
When Selection is Required

Desktop may request you to identify elements or areas:
  • Target element for a new action
  • Verification area for test assertions
  • Region of interest for analysis
When this happens, the Side Panel automatically switches to Interaction tab.

​
Element Selection Mode

To select a specific UI element:
  1. Hover over elements on the page
  2. Elements highlight as you hover
  3. Click to select an element
  4. Element details display in the panel
  5. Click “Confirm” to submit selection
  6. Or click “Select Parent” to choose the parent element
Tips:
  • Use “Select Parent” if the child element is too specific
  • Element properties are shown for verification
  • Selection has a timeout (countdown shown)

​
Area Selection Mode

To select a rectangular screen region:
  1. Toggle Selection Mode switch to Area Selection
  2. Click and drag on the page to draw a rectangle
  3. Release to finalize the selection
  4. Click “Confirm” to submit
  5. Or click “Select Whole Page” to select the entire viewport
Use Cases:
  • Selecting regions with multiple elements
  • Defining verification areas
  • Capturing specific screen sections

​
Canceling Selection

  • Click “Cancel” to discard current selection and return to previous state
  • Click “Stop” to exit selection mode entirely

​
Execution Verification

​
AI Verification Process

After action execution, Desktop’s AI may verify the results:
  1. Side Panel switches to Verification tab automatically
  2. Shows message: “Verifying Execution - Please do not close this tab”
  3. AI analyzes page state, screenshots, and execution logs
  4. Verification completes automatically
  5. Panel returns to Monitor tab
Important: Keep the browser tab active during verification. Do not close or navigate away.

​
Data Streaming

​
Real-Time WebSocket Streaming

All captured data streams to Desktop via WebSocket: Behavior Events:
  • Batched in groups of up to 50 events
  • Sent every 2 seconds or when batch is full
  • Includes deduplication to prevent duplicates
Page Analysis Results:
  • Sent immediately after analysis completes
  • Includes element data and screenshot
  • Simplified format for efficiency
Screenshots:
  • Sent as base64-encoded PNG
  • Includes device pixel ratio
  • Throttled to prevent overwhelming network

​
Connection Status Monitoring

Monitor connection health:
  • Connected: Green indicator, data flowing
  • Disconnected: Red indicator, no data transmission
  • Reconnecting: Automatic retry in progress

​
Monitor Levels (Advanced)

Desktop can configure monitoring detail level: Level 1 (Minimal) includes:
  • Element positions and IDs
  • Interactivity flags (clickable, focusable, etc.)
  • Form attributes (disabled, required, etc.)
  • Excludes text content (visible in screenshots)
Level 2 (Standard) adds:
  • Text content
  • Placeholder text
  • Input values
  • Scroll events
Level 3 (Verbose) adds:
  • Mouse move events
  • Full class lists
  • Computed styles
  • Framework information
Note: Monitor level is set by Desktop and cannot be changed in the extension.

​
Best Practices

​
During Recording

  1. Focus on One Workflow: Record a single test scenario at a time
  2. Clear Actions: Pause briefly between steps for clarity
  3. Avoid Distractions: Close unrelated tabs and notifications
  4. Verify State: Ensure page is in expected state before recording
  5. Natural Pace: Don’t rush; perform actions at normal speed

​
During Execution

  1. Don’t Interfere: Avoid clicking or typing during automation
  2. Stay on Tab: Keep the test tab active (don’t switch tabs)
  3. Stable Network: Ensure stable internet connection
  4. Watch for Errors: Monitor execution progress for failures
  5. Keep Desktop Connected: Don’t close Desktop application

​
For Reliability

  1. Stable Page State: Wait for pages to fully load before actions
  2. Unique Identifiers: Use pages with stable IDs and data-testid attributes
  3. Avoid Dynamic Content: Be cautious with frequently-changing content
  4. Test Environment: Use consistent test environment (same browser, viewport size)

​
Limitations

​
Restricted Pages

Extension cannot operate on:
  • chrome:// internal Chrome pages
  • edge:// internal Edge pages
  • Chrome Web Store pages
  • Browser settings and configuration pages
  • PDF viewer pages (chrome-extension://)

​
Content Security Policy (CSP)

Some websites with strict CSP may limit functionality:
  • Certain script injections may be blocked
  • Visual indicators may not display
  • Fallback methods are used when possible

​
Complex Web Applications

Limitations with certain web technologies:
  • Shadow DOM: Limited access to shadow DOM internals
  • iframes: Cross-origin iframes have restricted access
  • Canvas/WebGL: Cannot inspect canvas-drawn content
  • Third-party widgets: Limited control over embedded widgets

​
Performance Considerations

  • Recording high-frequency mouse moves may impact performance
  • Analyzing complex pages (1000+ elements) may take longer
  • Continuous tracking increases memory usage over time

​
Keyboard Shortcuts

Currently, the extension does not support keyboard shortcuts. All actions must be performed via the Side Panel interface.

​
Next Steps

Related Resources: