VP Genie Usability Testing
Research Background
Supervisor Kathryn Brookshier
Stakeholder Mary Rauzi, Bernard Mendez
Team Brendan Sallee, Monica Zhang, Yuan Fang, Pooja Narain
Duration 2025.1 - 2025.3
My Role UX Researcher
Contribution Proposal, Study Kit, Interview, Data Log, Analysis, Assess and Define, Final Presentation
What is VP Genie?
VP Genie is a comprehensive Gen AI-integrated solution designed to support both the creative and production aspects of filmmaking.
We conducted this usability test on a beta version of the product.
We conducted this usability test on a beta version of the product.
Who is our target audience?
It targets filmmakers who have tight budgets to execute their projects, filling in the gaps in their teams and equipment and more.
Scope of Study
It targets filmmakers who have tight budgets to execute their projects, filling in the gaps in their teams and equipment and more.
Scope of Study
Pre-production Genie: Create scripts, storyboards, and pitch decks, leveraging generative AI to enhance creative processes
Script Creation – Generate detailed outlines, scripts, characters, locations, scenes, props, and gear needed for production.
Storyboarding – Visually map out scenes effectively.
Script Creation – Generate detailed outlines, scripts, characters, locations, scenes, props, and gear needed for production.
Storyboarding – Visually map out scenes effectively.
Research Goals
- Business Goal
🎯 Attract more users to try VP Genie’s paid subscriptionHow does the pre-production workflow using VP Genie compare to the established workflows of filmmakers? What does it do well? Where can it improve? How can it differentiate itself in the market?
🎯 Usability for All: Ensure VP Genie is effective for those still learning the craft of filmmaking.
The study focuses on gathering insights from students, early career professionals and hobbyists in filmmaking.
- Study Goal
🔍 Identify UX/Product Gaps in pre-production workflows
Map friction points in the current experience
Understand the pre-production workflows of user group
Understand pain points & missing features
Research Questions
Participants
We recruited 6 participants that met our requirements.
Research Methods
Standardized prompt given to participants to
generate a script
Pre-test interview questions
Task-based, moderated, within-subjects usability testing
Observed participants performing given tasks
Assessed task success/failure rate, task error (1-5 scale)
Post-test interview questions and System Usability Scale (SUS)
Task-based, moderated, within-subjects usability testing
Observed participants performing given tasks
Assessed task success/failure rate, task error (1-5 scale)
Post-test interview questions and System Usability Scale (SUS)
Study Overview
Scenario 1: Create a Script from a Prompt
Overview: The participant starts with an idea written as a prompt in a Word document and uses VP Genie to generate a script, scenes, characters, locations, and props.
![]()
Scenario 2: Edit or Regenerate Visual Elements
Overview: The participant has generated a script and now wants to edit specific characters and locations to align with their vision.
![]()
Scenario 3: Edit Storyboard
Overview: The participant refines a storyboard, making adjustments to scenes and shots.
![]()
Task Metrics
![]()
What Went Well?
![]()
Overview of Findings
![]()
Area of Improvement #1Unexpected AI Content
![]()
Area of Improvement #2 No error recovery or version control
![]()
Area of Improvement #3 Unintuitive content editing workflow
![]()
Area of Improvement #4 Unfamiliar Terminology
![]()
Area of Improvement #5 Difficulties finding essential CTAs
![]()
Area of Improvement #6 Incomplete and unstructured script export
![]()
SUS(System Usability Scale)
Scenario 1: Create a Script from a Prompt
Overview: The participant starts with an idea written as a prompt in a Word document and uses VP Genie to generate a script, scenes, characters, locations, and props.
Scenario 2: Edit or Regenerate Visual ElementsOverview: The participant has generated a script and now wants to edit specific characters and locations to align with their vision.
Scenario 3: Edit Storyboard
Overview: The participant refines a storyboard, making adjustments to scenes and shots.
Task Metrics
What Went Well?
Overview of Findings
Area of Improvement #1Unexpected AI Content
Area of Improvement #2 No error recovery or version control
Area of Improvement #3 Unintuitive content editing workflow
Area of Improvement #4 Unfamiliar Terminology
Area of Improvement #5 Difficulties finding essential CTAs
Area of Improvement #6 Incomplete and unstructured script export
SUS(System Usability Scale)
Our 6 participants reported they:
Did not see themselves using VP Genie often in its current state
Did not feel confident using VP Genie
Felt VP Genie was difficult to navigate
Yet did not think they would need a lot of help to learn to use VP Genie
Did not see themselves using VP Genie often in its current state
Did not feel confident using VP Genie
Felt VP Genie was difficult to navigate
Yet did not think they would need a lot of help to learn to use VP Genie
Higher values mean participants felt more positive about VP Genie in that aspect:
![]()
Takeaways
Due to the NDA, I cannot share more infos about the final design solutions, contact me for a more detailed portfolio deck!
