Hypothesis

LLM-based speaking coaches can provide scalable, affordable, and psychologically safe speaking practice for non-native English speakers, but may fall short of human experts on empathy and nuanced cognition.

Solution

• Why LLMs here: Traditional speaking improvement relies on human experts, which are often hard to access, schedule, and afford; many learners also fear judgment.
• What Comuniqa adds: An always-available mobile workflow that captures speech and generates structured feedback across multiple dimensions (vocab, fluency, pronunciation, grammar, coherence, emotion), aligned to what human tutors evaluate.
• Why human-centric evaluation: The work explicitly compares learning modalities—AI-only vs expert-only vs hybrid—to understand where LLM feedback helps and where it doesn't.

Product

Mobile app for English speaking practice with instant multi-parameter feedback.

Key feedback modules:

• Overall summary: numeric score + overall summary + "ideal response"
• Vocabulary: CEFR score + word distribution across CEFR levels
• Fluency: pace, pitch, filler words, awkward pauses
• Emotion: sentiment + confidence
• Pronunciation: mispronounced words + correct pronunciations
• Coherence: relevance, logical flow, completeness
• Grammar: error detection + corrected structures

Demo

• 👉 Watch Demo

• ACM COMPASS 2024 Presentation: Conference talk presenting Comuniqa at ACM COMPASS 2024.
  👉 Watch Presentation

Research & Validation

This work is formally documented and evaluated in a peer-reviewed ACM publication:

Paper: Comuniqa — AI-Powered Communication Coach Venue: ACM Link: https://dl.acm.org/doi/pdf/10.1145/3674829.3675082

Results

Quantitative

Study & engagement:

• Participants: 34 completed participants (36 recruited; 2 dropped out)
• Activity volume: 64 app tests and 70 expert sessions
• Engagement: avg 26m 26s per active user; avg 11m 54s per session; expert sessions ~29 min
• Feature attention: highest time on grammar (~2m 5s) and vocabulary (~1m 46s) reports; lowest on coherence/emotion screens

Ratings (Likert 1–5):

• Comuniqa overall experience: Group 1 (AI-only) 3.72 ± 0.88; Group 3 (hybrid) 4.58 ± 1.08
• Human experts overall experience: Group 2 (expert-only) 4.09 ± 0.60; Group 3 (hybrid) 4.5 ± 0.95

Qualitative

• AI's strength: perceived as judgment-free, helping learners practice without social anxiety; some participants preferred AI for this reason
• Human expert strengths: better at empathy and cognitive nuance, and can adapt instruction in more human ways
• Human expert limitations: scheduling/availability friction was a recurring drawback
• Overall theme: LLM feedback can be strong on structured, quantitative dimensions, but still lags humans on empathy and high-touch coaching

Takeaway

• LLM-driven speaking practice is promising for scale + affordability + psychological safety, especially where human tutoring access is limited
• Hybrid (AI + expert) can deliver the best perceived experience (higher AI rating in hybrid vs AI-only), suggesting complementarity rather than replacement
• Short-term studies may not show measurable proficiency gains; longer studies are needed to validate learning outcomes beyond engagement and perceptions