feat: Implement Phase 5 - AI & Voice Processing multi-child support

Enhanced AI chat assistant and voice processing to intelligently handle multi-child families with automatic child detection and context filtering. ## AI Context Manager (context-manager.ts) - Enhanced summarizeChildContext() for multi-child families * Shows all children with ages and genders * Adds clarification instructions for AI * Provides family overview - Updated buildSystemPrompt() with multi-child awareness section * Instructions for identifying which child is discussed * Guidance on handling sibling comparisons sensitively * Recognition that each child develops at their own pace - Added detectChildInMessage() method * Pattern matching for child names (exact, possessive, prepositional) * Case-insensitive matching * Auto-defaults to single child if only one exists ## Voice Service (voice.service.ts) - Updated extractActivityFromText() with multi-child support * Added availableChildren parameter * Returns detectedChildName and childId * Enhanced GPT-4o-mini prompt with child context - Implemented child name matching logic * Extracts childName from GPT response * Matches to childId using case-insensitive comparison * Triggers clarification if multi-child family but no name detected - Updated processVoiceInput() to pass children through ## Voice Controller (voice.controller.ts) - Updated all endpoints to accept availableChildren parameter * /transcribe: JSON string parameter * /process: JSON string parameter * /extract-activity: JSON array parameter ## AI Service (ai.service.ts) - Added child detection in chat() method * Calls contextManager.detectChildInMessage() * Filters recent activities by detected child ID * Enhanced logging for multi-child families ## Example Usage Voice: "Fed Emma 120ml" → Detects Emma, creates feeding for her Voice: "Baby slept" (2 kids) → Triggers clarification prompt Chat: "How is Emma sleeping?" → Filters to Emma's sleep data Build: ✅ PASSED Files: 4 modified, 1 new (168 lines) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-05 06:07:32 +00:00
parent 446bf4dca8
commit 9068818b57
5 changed files with 545 additions and 15 deletions
--- a/maternal-app/maternal-app-backend/src/modules/ai/ai.service.ts
+++ b/maternal-app/maternal-app-backend/src/modules/ai/ai.service.ts
@@ -291,12 +291,37 @@ export class AIService {
        where: { familyId: userId },
      });

+      // Detect which child is being discussed (if any)
+      const detectedChild = this.contextManager.detectChildInMessage(
+        sanitizedMessage,
+        userChildren,
+      );
+
+      // If a specific child is detected, prioritize their activities
      const recentActivities = await this.activityRepository.find({
-        where: { loggedBy: userId },
+        where: detectedChild
+          ? { childId: detectedChild.id }
+          : { loggedBy: userId },
        order: { startedAt: 'DESC' },
        take: 20,
      });

+      // Log multi-child context if applicable
+      if (userChildren.length > 1) {
+        this.logger.log(
+          `Multi-child family: ${userChildren.length} children (${userChildren.map(c => c.name).join(', ')})`,
+        );
+        if (detectedChild) {
+          this.logger.log(
+            `Detected child focus: ${detectedChild.name} (${detectedChild.id})`,
+          );
+        } else {
+          this.logger.log(
+            `No specific child detected - using general family context`,
+          );
+        }
+      }
+
      // Use enhanced conversation memory with semantic search
      const { context: memoryContext } =
        await this.conversationMemoryService.getConversationWithSemanticMemory(
--- a/maternal-app/maternal-app-backend/src/modules/ai/context/context-manager.ts
+++ b/maternal-app/maternal-app-backend/src/modules/ai/context/context-manager.ts
@@ -105,13 +105,13 @@ export class ContextManager {
  }

  /**
-   * Build system prompt with safety boundaries
+   * Build system prompt with safety boundaries and multi-child awareness
   */
  private buildSystemPrompt(userPreferences?: Record<string, any>): string {
    const language = userPreferences?.language || 'en';
    const tone = userPreferences?.tone || 'friendly';

-    return `You are a helpful AI assistant for parents tracking their baby's activities and milestones.
+    return `You are a helpful AI assistant for parents tracking their children's activities and milestones.

 IMPORTANT GUIDELINES:
 - You are NOT a medical professional. Always recommend consulting healthcare providers for medical concerns.
@@ -121,30 +121,51 @@ IMPORTANT GUIDELINES:
 - Respect cultural differences in parenting practices.
 - Keep responses concise and actionable.

+MULTI-CHILD FAMILY SUPPORT:
+- When multiple children are in the family, pay attention to which child the parent is asking about.
+- If a specific child's name is mentioned, focus your response on that child's age, stage, and patterns.
+- If the question is ambiguous about which child, politely ask for clarification.
+- When comparing or discussing siblings, be sensitive and avoid making judgments.
+- Recognize that each child develops at their own pace.
+
 USER PREFERENCES:
 - Language: ${language}
 - Tone: ${tone}

 Your role is to:
-1. Help interpret and log baby activities (feeding, sleep, diaper changes, etc.)
-2. Provide general developmental milestone information
-3. Offer encouragement and support to parents
-4. Suggest patterns in baby's behavior based on logged data
+1. Help interpret and log child activities (feeding, sleep, diaper changes, etc.)
+2. Provide general developmental milestone information appropriate to each child's age
+3. Offer encouragement and support to parents managing single or multiple children
+4. Suggest patterns in children's behavior based on logged data
 5. Answer general parenting questions (non-medical)
+6. Handle multi-child context intelligently by identifying which child is being discussed

 Remember: When in doubt, recommend professional consultation.`;
  }

  /**
-   * Summarize child context for the AI
+   * Summarize child context for the AI (multi-child aware)
   */
  private summarizeChildContext(children: Child[]): string {
-    return children
+    if (children.length === 0) {
+      return 'No children in family context.';
+    }
+
+    const childSummaries = children
      .map((child) => {
        const ageInMonths = this.calculateAgeInMonths(child.birthDate);
-        return `- ${child.name}: ${ageInMonths} months old, born ${child.birthDate.toDateString()}`;
+        const gender = child.gender ? ` (${child.gender})` : '';
+        return `- ${child.name}${gender}: ${ageInMonths} months old, born ${child.birthDate.toDateString()}`;
      })
      .join('\n');
+
+    // Add context about multiple children
+    if (children.length > 1) {
+      const names = children.map(c => c.name).join(', ');
+      return `Family has ${children.length} children: ${names}\n\n${childSummaries}\n\nIMPORTANT: When user mentions a specific child name, focus your response on that child. If unclear which child, ask for clarification.`;
+    }
+
+    return childSummaries;
  }

  /**
@@ -192,4 +213,49 @@ Remember: When in doubt, recommend professional consultation.`;
    // Rough estimate: 1 token ≈ 4 characters
    return Math.ceil(text.length / 4);
  }
+
+  /**
+   * Detect which child is being referenced in a message (multi-child support)
+   */
+  detectChildInMessage(
+    message: string,
+    availableChildren: Child[],
+  ): Child | null {
+    if (!availableChildren || availableChildren.length === 0) {
+      return null;
+    }
+
+    const lowerMessage = message.toLowerCase();
+
+    // Try to find child by exact name match
+    for (const child of availableChildren) {
+      const childNameLower = child.name.toLowerCase();
+
+      // Check for name mentions with common patterns
+      const patterns = [
+        new RegExp(`\\b${childNameLower}\\b`, 'i'),  // Exact word match
+        new RegExp(`\\b${childNameLower}'s\\b`, 'i'), // Possessive
+        new RegExp(`\\bfor ${childNameLower}\\b`, 'i'), // "for Emma"
+        new RegExp(`\\babout ${childNameLower}\\b`, 'i'), // "about Emma"
+        new RegExp(`\\b${childNameLower} is\\b`, 'i'), // "Emma is"
+        new RegExp(`\\b${childNameLower} has\\b`, 'i'), // "Emma has"
+      ];
+
+      for (const pattern of patterns) {
+        if (pattern.test(lowerMessage)) {
+          this.logger.log(
+            `Detected child "${child.name}" in message via pattern match`,
+          );
+          return child;
+        }
+      }
+    }
+
+    // If only one child exists, default to that child
+    if (availableChildren.length === 1) {
+      return availableChildren[0];
+    }
+
+    return null;
+  }
 }
--- a/maternal-app/maternal-app-backend/src/modules/voice/voice.controller.ts
+++ b/maternal-app/maternal-app-backend/src/modules/voice/voice.controller.ts
@@ -28,6 +28,7 @@ export class VoiceController {
    @Body('text') text?: string,
    @Body('language') language?: string,
    @Body('childName') childName?: string,
+    @Body('availableChildren') availableChildrenJson?: string,
  ) {
    this.logger.log('=== Voice Transcribe Request ===');
    this.logger.log(
@@ -36,6 +37,17 @@ export class VoiceController {
    this.logger.log(`Language: ${language || 'en'}`);
    this.logger.log(`Child Name: ${childName || 'none'}`);

+    // Parse available children if provided
+    let availableChildren: Array<{ id: string; name: string }> | undefined;
+    if (availableChildrenJson) {
+      try {
+        availableChildren = JSON.parse(availableChildrenJson);
+        this.logger.log(`Available Children: ${availableChildren?.map(c => c.name).join(', ')}`);
+      } catch (error) {
+        this.logger.warn(`Failed to parse availableChildren: ${error.message}`);
+      }
+    }
+
    // If text is provided (from Web Speech API), classify it directly
    if (text) {
      this.logger.log(`Input Text: "${text}"`);
@@ -44,6 +56,7 @@ export class VoiceController {
        text,
        language || 'en',
        childName,
+        availableChildren,
      );

      this.logger.log(
@@ -77,11 +90,12 @@ export class VoiceController {
      `Transcription: "${transcription.text}" (${transcription.language})`,
    );

-    // Also classify the transcription
+    // Also classify the transcription with multi-child support
    const classification = await this.voiceService.extractActivityFromText(
      transcription.text,
      language || 'en',
      childName,
+      availableChildren,
    );

    this.logger.log(
@@ -102,15 +116,27 @@ export class VoiceController {
    @UploadedFile() file: Express.Multer.File,
    @Body('language') language?: string,
    @Body('childName') childName?: string,
+    @Body('availableChildren') availableChildrenJson?: string,
  ) {
    if (!file) {
      throw new BadRequestException('Audio file is required');
    }

+    // Parse available children if provided
+    let availableChildren: Array<{ id: string; name: string }> | undefined;
+    if (availableChildrenJson) {
+      try {
+        availableChildren = JSON.parse(availableChildrenJson);
+      } catch (error) {
+        this.logger.warn(`Failed to parse availableChildren: ${error.message}`);
+      }
+    }
+
    const result = await this.voiceService.processVoiceInput(
      file.buffer,
      language,
      childName,
+      availableChildren,
    );

    return {
@@ -124,6 +150,7 @@ export class VoiceController {
    @Body('text') text: string,
    @Body('language') language: string,
    @Body('childName') childName?: string,
+    @Body('availableChildren') availableChildren?: Array<{ id: string; name: string }>,
  ) {
    if (!text) {
      throw new BadRequestException('Text is required');
@@ -133,6 +160,7 @@ export class VoiceController {
      text,
      language || 'en',
      childName,
+      availableChildren,
    );

    return {
--- a/maternal-app/maternal-app-backend/src/modules/voice/voice.service.ts
+++ b/maternal-app/maternal-app-backend/src/modules/voice/voice.service.ts
@@ -240,12 +240,14 @@ export class VoiceService {

  /**
   * Extract activity information from transcribed text using GPT
+   * Now supports multi-child detection and context
   */
  async extractActivityFromText(
    text: string,
    language: string,
    childName?: string,
-  ): Promise<ActivityExtractionResult> {
+    availableChildren?: Array<{ id: string; name: string }>,
+  ): Promise<ActivityExtractionResult & { detectedChildName?: string; childId?: string }> {
    if (!this.chatOpenAI) {
      throw new BadRequestException('Chat service not configured');
    }
@@ -254,14 +256,28 @@ export class VoiceService {
    this.logger.log(
      `[Activity Extraction] Language: ${language}, Child: ${childName || 'none'}`,
    );
+    this.logger.log(
+      `[Activity Extraction] Available children: ${availableChildren?.map(c => c.name).join(', ') || 'none'}`,
+    );

    // Apply common mishear corrections before extraction
    const correctedText = this.applyMishearCorrections(text, language);

    try {
+      // Build child context for multi-child families
+      let childContext = '';
+      if (availableChildren && availableChildren.length > 0) {
+        const childNames = availableChildren.map(c => c.name).join(', ');
+        childContext = `\n\n**Available Children in Family:** ${childNames}
+- If the user mentions a specific child name, extract it and return in "childName" field
+- Match child names case-insensitively and handle variations (e.g., "Emma", "emma", "Emmy")
+- If multiple children exist but no name is mentioned, set "needsClarification" to true`;
+      }
+
      const systemPrompt = `You are an intelligent assistant that interprets natural language commands related to baby care and extracts structured activity data.

 Extract activity details from the user's text and respond ONLY with valid JSON (no markdown, no explanations).
+${childContext}

 **Supported Activity Types:**

@@ -287,6 +303,7 @@ Extract activity details from the user's text and respond ONLY with valid JSON (
 {
  "type": "feeding|sleep|diaper|medicine|activity|milestone|unknown",
  "timestamp": "ISO 8601 datetime if mentioned (e.g., '3pm', '30 minutes ago'), otherwise use current time",
+  "childName": "extracted child name if mentioned, or null",
  "details": {
    // For feeding:
    "feedingType": "bottle|breast|solids",
@@ -374,17 +391,47 @@ If the text doesn't describe a trackable baby care activity:
      this.logger.log(
        `[Activity Extraction] Extracted activity: ${result.type} (confidence: ${result.confidence})`,
      );
+      this.logger.log(
+        `[Activity Extraction] Child name: ${result.childName || 'none'}`,
+      );
      this.logger.log(
        `[Activity Extraction] Details: ${JSON.stringify(result.details || {})}`,
      );

-      const extractedActivity: ActivityExtractionResult = {
+      const extractedActivity: ActivityExtractionResult & { detectedChildName?: string; childId?: string } = {
        type: result.type,
        timestamp: result.timestamp ? new Date(result.timestamp) : null,
        details: result.details || {},
        confidence: result.confidence || 0,
+        detectedChildName: result.childName || undefined,
      };

+      // Match detected child name to child ID
+      if (extractedActivity.detectedChildName && availableChildren) {
+        const matchedChild = availableChildren.find(
+          child => child.name.toLowerCase() === extractedActivity.detectedChildName!.toLowerCase()
+        );
+        if (matchedChild) {
+          extractedActivity.childId = matchedChild.id;
+          this.logger.log(
+            `[Activity Extraction] Matched child "${extractedActivity.detectedChildName}" to ID: ${matchedChild.id}`,
+          );
+        } else {
+          this.logger.warn(
+            `[Activity Extraction] Could not match child name "${extractedActivity.detectedChildName}" to any available child`,
+          );
+        }
+      }
+
+      // Check if clarification needed for multi-child families
+      if (availableChildren && availableChildren.length > 1 && !extractedActivity.detectedChildName) {
+        extractedActivity.needsClarification = true;
+        extractedActivity.clarificationPrompt = `Which child is this for? Available: ${availableChildren.map(c => c.name).join(', ')}`;
+        this.logger.warn(
+          `Multi-child family but no child name detected - clarification needed`,
+        );
+      }
+
      // Check if confidence is below threshold
      if (extractedActivity.confidence < this.CONFIDENCE_THRESHOLD) {
        extractedActivity.needsClarification = true;
@@ -410,23 +457,26 @@ If the text doesn't describe a trackable baby care activity:

  /**
   * Process voice input: transcribe + extract activity
+   * Now supports multi-child context
   */
  async processVoiceInput(
    audioBuffer: Buffer,
    language?: string,
    childName?: string,
+    availableChildren?: Array<{ id: string; name: string }>,
  ): Promise<{
    transcription: TranscriptionResult;
-    activity: ActivityExtractionResult;
+    activity: ActivityExtractionResult & { detectedChildName?: string; childId?: string };
  }> {
    // Step 1: Transcribe audio
    const transcription = await this.transcribeAudio(audioBuffer, language);

-    // Step 2: Extract activity from transcription
+    // Step 2: Extract activity from transcription with multi-child support
    const activity = await this.extractActivityFromText(
      transcription.text,
      transcription.language,
      childName,
+      availableChildren,
    );

    return {