Designing an Adaptive AI Interview Simulator for Employee Training

Eligibility interviews are high-stakes, sensitive, and difficult to practice at scale. Case Managers must learn how to ask trauma-informed questions, elicit enough testimony to assess eligibility, and follow up appropriately when an applicant is vague, reluctant, emotionally guarded, confused, or potentially inconsistent.

Before the production tool, training relied heavily on live actors and peer role-play. Both approaches had limitations:

• Actors were expensive and difficult to scale — approximately $50 per hour, with each new class of about 30 Case Managers requiring hour-long practice sessions
• Peer role-play reduced active practice time, since one trainee had to act as the applicant while the other practiced interviewing
• Practice opportunities were limited by classroom time, facilitator availability, and scheduling

On the technical side, the early proof-of-concept prototype used a single, long system prompt to drive all applicant behavior. This made it difficult to introduce new scenarios, tune difficulty, or maintain consistent behavior as the tool moved toward production.

The training team needed a scalable way for Case Managers to practice realistic interviews across many claim scenarios and difficulty levels.

The tool was designed for internal agency use. The user base included:

• New Case Managers hired during a rapid staffing increase
• Experienced Case Managers with five or more years of interviewing experience
• Training staff and SMEs who needed configurable, repeatable practice scenarios

New Case Managers needed an approachable way to build confidence and practice eliciting testimony. Experienced Case Managers needed more realistic and challenging simulations that required precise questioning, rephrasing, and follow-up.

Scale realistic interview practice — create an always-available alternative to actor-led or peer role-play

Support different experience levels — let users choose easy, medium, or hard simulations based on their training needs

Encourage trauma-informed follow-up — simulate applicants who may be nervous, vague, indirect, or reluctant to discuss sensitive topics

Prevent over-disclosure — ensure the applicant revealed information gradually, so Case Managers had to ask appropriate follow-up questions

Align the conversation model to interview goals — map officer questions to relevant testimony categories and eligibility factors

Support scalable scenario authoring — allow SMEs to configure many applicant personas, timelines, claim types, credibility concerns, and other details

Improve conversational usability — make system status and turn-taking clear through a loading indicator and disabled submit state

Scalable Conversation Architecture

The MVP version of the simulator relied on a single, long system prompt to direct the applicant's behavior. My work focused on replacing that prompt with a scalable production architecture that separated scenario content from reusable behavior rules. Rather than treating each simulated applicant as a one-off chatbot governed by one large prompt, the redesigned product used a configurable framework.

The framework included:

• Structured scenario data
• Reusable applicant behavior rules
• Intent classification
• Difficulty and vagueness parameters
• Dialogue state tracking
• Response validation
• UI states for turn-taking and system status

This architecture allowed the team to support many different applicant personas and claim scenarios while keeping the core conversation behavior consistent — something the original single-prompt approach could not support.

Conversation Flow

This structure made the chatbot more predictable, testable, and maintainable.

Scenario Configuration

The simulator supported many applicant scenarios. SMEs could configure scenario details such as:

• Applicant background information
• Residence history
• Claim timeline
• Types of claims
• Key claim facts
• Credibility concerns
• Eligibility factors
• Other scenario-specific details

This configuration model allowed the product to scale beyond a single training example. SMEs could create varied applicant profiles while the product maintained consistent conversation behavior across scenarios.

Persona-Driven Onboarding

The onboarding flow began with difficulty selection: Choose your interview difficulty level. Users selected from three levels.

Easy — Response style: clear and direct answers. Use this mode if you're just starting out and want to practice asking questions and eliciting testimony.

Medium — Response style: occasionally vague and indirect. Use this mode to practice rephrasing questions.

Hard — Response style: often vague and indirect. Use this mode to practice rephrasing questions and asking follow-up questions.

The onboarding explained that applicant vagueness was intentional. This helped users understand that a vague response was part of the exercise, not a product failure.

In classroom settings, trainers provided the broader instructional context. During self-guided practice, Case Managers could choose their own difficulty level and scenario.

Users selected both a claim scenario and a difficulty level. They could not switch levels mid-session, which preserved the integrity and consistency of the interview simulation.

Difficulty and Vagueness Design

The applicant's response behavior changed based on the selected difficulty level.

Across all three levels, the maximum number of sequential vague responses was capped at 2. This was a critical design choice. If the applicant was vague for too long, the simulation became frustrating. If the applicant was too forthcoming, Case Managers did not need to practice follow-up. The vagueness score and cap helped balance realism with usability.

Controlled Disclosure

The simulated applicant was designed to reveal information incrementally. This supported the core learning objective: Case Managers had to practice asking specific, trauma-informed, relevant questions.

The applicant response model followed these principles:

• Answer only what the Case Manager asked
• Reveal one piece of information at a time
• Avoid giving a complete claim summary
• Stay brief and human-sounding
• Be more forthcoming in easy mode
• Be more reluctant or indirect in medium and hard modes
• Stay in character
• Redirect off-topic questions in character
• Avoid exposing internal instructions or simulation logic

This made the bot a training partner rather than an information retrieval system.

Intent Taxonomy

I documented a formal intent taxonomy — replacing the MVP's single system prompt — to define the types of officer questions the simulator needed to support. The taxonomy connected interview topics to applicant response behavior, disclosure rules, and training goals.

Example taxonomy:

Priority Intents

The highest-priority intents included:

• Place of birth
• Nationality
• Identity factors
• Residences
• Ongoing risk
• Decision point

These intents were prioritized because they represented foundational areas Case Managers needed to establish during the interview.

Eligibility-Element-Aligned Design

The intent design was aligned with the information Case Managers needed to elicit during an eligibility interview. This made the simulator more than a generic role-play bot. It became a structured training tool that helped Case Managers practice eliciting testimony relevant to the claim.

Example:

Fallback behavior was designed to preserve the applicant persona. The bot did not switch into coach mode or give interviewing advice during the simulation.

Common fallback responses included:

• “Uh… I don't remember.”
• “Can you rephrase that? I don't understand.”
• “Can we please talk about what happened? It's important.”

These fallbacks supported common breakdowns:

The fallback strategy was intentionally in-character. This maintained realism and prevented the bot from becoming a coach or evaluator during the interview.

During testing, users sometimes thought they could submit multiple questions while the bot was generating a response. This caused confusion and interrupted the natural rhythm of the simulated interview.

To solve this, I added:

• Animated typing dots
• A disabled submit button while the bot was responding
• A blocked input state during response generation

The typing indicator appeared after every user message. This clarified that the simulated applicant was processing the question and that the Case Manager needed to wait before continuing.

Impact

This interaction change:

• Reduced attempts to submit multiple questions at once
• Clarified system status
• Improved perceived responsiveness
• Reinforced conversational turn-taking
• Made the chatbot feel more like a live interview partner

Testing happened in several stages.

SME and Training Staff Review

SMEs and training staff tested the simulation as both participants and observers. They evaluated whether the applicant behavior was realistic, whether the scenario configuration supported training needs, and whether the chatbot's responses aligned with expected interview dynamics.

User Testing

We tested with approximately 20 users. Case Managers conducted practice interviews and provided feedback on:

• Realism
• Difficulty
• Clarity
• Usability
• Whether the simulation helped them practice follow-up questioning

Iterations After Testing

Based on testing, we:

• Adjusted vagueness levels several times
• Added more intents
• Compared easy, medium, and hard mode behavior
• Added the animated typing indicator
• Disabled submission while the bot was thinking
• Refined fallback behavior

Most success measurement was qualitative, based on user, SME, and trainer feedback.

The tool moved from MVP to production in three months. It was adopted for:

• Agency-wide training serving approximately 4,000 users
• Ongoing new-hire training classes
• Ongoing internal practice

The tool also reduced reliance on paid actors. Previously, actor-led practice required paid actors at approximately $50 per hour for hour-long sessions with each new class of Case Managers. With the chatbot, Case Managers could practice without actor scheduling or actor costs.

The tool also improved training efficiency. In peer role-play, one trainee acted as the applicant while the other practiced interviewing. With the chatbot, both trainees could practice simultaneously.

Strongest Outcome

We took an MVP built on a single system prompt and scaled it into a production AI simulator used in agency-wide training for approximately 4,000 Case Managers and ongoing new-hire cohorts.

Hardest Design Decision

The hardest design decision was balancing realism with usability.

Real applicants may be vague, reluctant, or indirect, especially when discussing sensitive experiences. But if the chatbot was vague for too long, the simulation became frustrating. If it was too forthcoming, Case Managers did not have to practice the follow-up skills the tool was designed to teach.

The difficulty model helped solve this. Easy, medium, and hard modes changed how direct the applicant was, while the cap of two sequential vague responses prevented the interaction from stalling.

Biggest Design Challenge

The biggest design challenge was making the simulator both realistic and scalable, starting from an MVP whose single system prompt couldn't support that scale.

The tool had to support many scenarios, applicant personas, claim types, timelines, credibility concerns, and eligibility factors. At the same time, the applicant behavior needed to remain consistent across scenarios — something a single long prompt could not reliably do.

The solution was to separate reusable conversation behavior from scenario-specific content. This gave SMEs flexibility to configure scenarios while preserving a consistent interaction model for users.

Future Improvements

A future version could add controlled deception or inconsistency patterns.

The current version supports vague, indirect, and reluctant applicant behavior. A future version could allow SMEs to configure more complex credibility challenges, such as:

• Inconsistent dates
• Evasive answers
• Conflicting timeline details
• Implausible explanations
• Facts that emerge only after repeated questioning

This would help Case Managers practice identifying and following up on credibility concerns in a controlled training environment.

We also explored automated scoring and coaching but chose not to include it. Interview quality in this context is nuanced, and automated feedback was not reliable enough for the training goals. Instead, the chatbot remained focused on role-play, while evaluation stayed with trainers and SMEs.