maternal-app/docs/implementation-docs/azure-openai-integration-summary.md

Azure OpenAI Integration - Implementation Summary

Overview

The AI service has been updated to support both OpenAI and Azure OpenAI with automatic fallback, proper environment configuration, and full support for GPT-5 models including reasoning tokens.

---

Environment Configuration

✅ Complete Environment Variables (.env)

# AI Services Configuration
# Primary provider: 'openai' or 'azure'
AI_PROVIDER=azure

# OpenAI Configuration (Primary - if AI_PROVIDER=openai)
OPENAI_API_KEY=sk-your-openai-api-key-here
OPENAI_MODEL=gpt-4o-mini
OPENAI_EMBEDDING_MODEL=text-embedding-3-small
OPENAI_MAX_TOKENS=1000

# Azure OpenAI Configuration (if AI_PROVIDER=azure)
AZURE_OPENAI_ENABLED=true

# Azure OpenAI - Chat/Completion Endpoint (GPT-5)
# Each deployment has its own API key for better security and quota management
AZURE_OPENAI_CHAT_ENDPOINT=https://footprints-open-ai.openai.azure.com
AZURE_OPENAI_CHAT_DEPLOYMENT=gpt-5-mini
AZURE_OPENAI_CHAT_API_VERSION=2025-04-01-preview
AZURE_OPENAI_CHAT_API_KEY=your-chat-api-key-here
AZURE_OPENAI_CHAT_MAX_TOKENS=1000
AZURE_OPENAI_REASONING_EFFORT=medium

# Azure OpenAI - Whisper/Voice Endpoint
AZURE_OPENAI_WHISPER_ENDPOINT=https://footprints-open-ai.openai.azure.com
AZURE_OPENAI_WHISPER_DEPLOYMENT=whisper
AZURE_OPENAI_WHISPER_API_VERSION=2025-04-01-preview
AZURE_OPENAI_WHISPER_API_KEY=your-whisper-api-key-here

# Azure OpenAI - Embeddings Endpoint
AZURE_OPENAI_EMBEDDINGS_ENDPOINT=https://footprints-ai.openai.azure.com
AZURE_OPENAI_EMBEDDINGS_DEPLOYMENT=Text-Embedding-ada-002-V2
AZURE_OPENAI_EMBEDDINGS_API_VERSION=2023-05-15
AZURE_OPENAI_EMBEDDINGS_API_KEY=your-embeddings-api-key-here

Configuration for Your Setup

Based on your requirements:

AI_PROVIDER=azure
AZURE_OPENAI_ENABLED=true

# Chat (GPT-5 Mini) - Separate API key
AZURE_OPENAI_CHAT_ENDPOINT=https://footprints-open-ai.openai.azure.com
AZURE_OPENAI_CHAT_DEPLOYMENT=gpt-5-mini
AZURE_OPENAI_CHAT_API_VERSION=2025-04-01-preview
AZURE_OPENAI_CHAT_API_KEY=[your_chat_key]
AZURE_OPENAI_REASONING_EFFORT=medium  # or 'minimal', 'low', 'high'

# Voice (Whisper) - Separate API key
AZURE_OPENAI_WHISPER_ENDPOINT=https://footprints-open-ai.openai.azure.com
AZURE_OPENAI_WHISPER_DEPLOYMENT=whisper
AZURE_OPENAI_WHISPER_API_VERSION=2025-04-01-preview
AZURE_OPENAI_WHISPER_API_KEY=[your_whisper_key]

# Embeddings - Separate API key
AZURE_OPENAI_EMBEDDINGS_ENDPOINT=https://footprints-ai.openai.azure.com
AZURE_OPENAI_EMBEDDINGS_DEPLOYMENT=Text-Embedding-ada-002-V2
AZURE_OPENAI_EMBEDDINGS_API_VERSION=2023-05-15
AZURE_OPENAI_EMBEDDINGS_API_KEY=[your_embeddings_key]

Why Separate API Keys?

Each Azure OpenAI deployment can have its own API key for:

• Security: Limit blast radius if a key is compromised
• Quota Management: Separate rate limits per service
• Cost Tracking: Monitor usage per deployment
• Access Control: Different team members can have access to different services

---

AI Service Implementation

✅ Key Features

1. Multi-Provider Support

• Primary: Azure OpenAI (GPT-5)
• Fallback: OpenAI (GPT-4o-mini)
• Automatic failover if Azure unavailable

2. GPT-5 Specific Features

• ✅ Reasoning tokens tracking
• ✅ Configurable reasoning effort (minimal, low, medium, high)
• ✅ Extended context (272K input + 128K output = 400K total)
• ✅ Response metadata with token counts

3. Response Format

interface ChatResponseDto {
  conversationId: string;
  message: string;
  timestamp: Date;
  metadata?: {
    model?: string;                    // 'gpt-5-mini' or 'gpt-4o-mini'
    provider?: 'openai' | 'azure';
    reasoningTokens?: number;          // GPT-5 only
    totalTokens?: number;
  };
}

4. Azure GPT-5 Request

const requestBody = {
  messages: azureMessages,
  temperature: 0.7,
  max_tokens: 1000,
  stream: false,
  reasoning_effort: 'medium',  // GPT-5 specific
};

5. Azure GPT-5 Response

{
  choices: [{
    message: { content: string },
    reasoning_tokens: number,  // NEW in GPT-5
  }],
  usage: {
    prompt_tokens: number,
    completion_tokens: number,
    reasoning_tokens: number,  // NEW in GPT-5
    total_tokens: number,
  }
}

---

GPT-5 vs GPT-4 Differences

Reasoning Tokens

GPT-5 introduces reasoning_tokens:

• Hidden tokens used for internal reasoning
• Not part of message content
• Configurable via reasoning_effort parameter
• Higher effort = more reasoning tokens = better quality

Reasoning Effort Levels:

'minimal'  // Fastest, lowest reasoning tokens
'low'      // Quick responses with basic reasoning
'medium'   // Balanced (default)
'high'     // Most thorough, highest reasoning tokens

Context Length

GPT-5:

• Input: 272,000 tokens (vs GPT-4's 128K)
• Output: 128,000 tokens
• Total context: 400,000 tokens

GPT-4o:

• Input: 128,000 tokens
• Total context: 128,000 tokens

Token Efficiency

GPT-5 Benefits:

• 22% fewer output tokens vs o3
• 45% fewer tool calls
• Better performance per dollar despite reasoning overhead

Pricing

Azure OpenAI GPT-5:

• Input: $1.25 / 1M tokens
• Output: $10.00 / 1M tokens
• Cached input: $0.125 / 1M (90% discount for repeated prompts)

---

Implementation Details

Service Initialization

The AI service now:

1. Checks AI_PROVIDER environment variable
2. Configures Azure OpenAI if provider is 'azure'
3. Falls back to OpenAI if Azure not configured
4. Logs which provider is active

constructor() {
  this.aiProvider = this.configService.get('AI_PROVIDER', 'openai');

  if (this.aiProvider === 'azure') {
    // Load Azure configuration from environment
    this.azureChatEndpoint = this.configService.get('AZURE_OPENAI_CHAT_ENDPOINT');
    this.azureChatDeployment = this.configService.get('AZURE_OPENAI_CHAT_DEPLOYMENT');
    // ... more configuration
  } else {
    // Load OpenAI configuration
    this.chatModel = new ChatOpenAI({ ... });
  }
}

Chat Method Flow

async chat(userId, chatDto) {
  // 1. Validate configuration
  // 2. Get/create conversation
  // 3. Build context with user data
  // 4. Generate response based on provider:

  if (this.aiProvider === 'azure') {
    const response = await this.generateWithAzure(messages);
    // Returns: { content, reasoningTokens, totalTokens }
  } else {
    const response = await this.generateWithOpenAI(messages);
    // Returns: content string
  }

  // 5. Save conversation with token tracking
  // 6. Return response with metadata
}

Azure Generation Method

private async generateWithAzure(messages) {
  const url = `${endpoint}/openai/deployments/${deployment}/chat/completions?api-version=${apiVersion}`;

  const requestBody = {
    messages: azureMessages,
    temperature: 0.7,
    max_tokens: 1000,
    reasoning_effort: 'medium',  // GPT-5 parameter
  };

  const response = await axios.post(url, requestBody, {
    headers: {
      'api-key': this.azureApiKey,
      'Content-Type': 'application/json',
    },
  });

  return {
    content: response.data.choices[0].message.content,
    reasoningTokens: response.data.usage.reasoning_tokens,
    totalTokens: response.data.usage.total_tokens,
  };
}

Automatic Fallback

If Azure fails, the service automatically retries with OpenAI:

catch (error) {
  // Fallback to OpenAI if Azure fails
  if (this.aiProvider === 'azure' && this.chatModel) {
    this.logger.warn('Azure OpenAI failed, attempting OpenAI fallback...');
    this.aiProvider = 'openai';
    return this.chat(userId, chatDto);  // Recursive call with OpenAI
  }
  throw new BadRequestException('Failed to generate AI response');
}

---

Testing the Integration

1. Check Provider Status

GET /api/v1/ai/provider-status

Response:

{
  "provider": "azure",
  "model": "gpt-5-mini",
  "configured": true,
  "endpoint": "https://footprints-open-ai.openai.azure.com"
}

2. Test Chat with GPT-5

POST /api/v1/ai/chat
Authorization: Bearer {token}

{
  "message": "How much should a 3-month-old eat per feeding?"
}

Response:

{
  "conversationId": "conv_123",
  "message": "A 3-month-old typically eats...",
  "timestamp": "2025-01-15T10:30:00Z",
  "metadata": {
    "model": "gpt-5-mini",
    "provider": "azure",
    "reasoningTokens": 145,
    "totalTokens": 523
  }
}

3. Monitor Reasoning Tokens

Check logs for GPT-5 reasoning token usage:

[AIService] Azure OpenAI response: {
  model: 'gpt-5-mini',
  finish_reason: 'stop',
  prompt_tokens: 256,
  completion_tokens: 122,
  reasoning_tokens: 145,  // GPT-5 reasoning overhead
  total_tokens: 523
}

---

Optimizing Reasoning Effort

When to Use Each Level

Minimal (reasoning_effort: 'minimal'):

• Simple queries
• Quick responses needed
• Cost optimization
• Use case: "What time is it?"

Low (reasoning_effort: 'low'):

• Straightforward questions
• Fast turnaround required
• Use case: "How many oz in 120ml?"

Medium (reasoning_effort: 'medium') - Default:

• Balanced performance
• Most common use cases
• Use case: "Is my baby's sleep pattern normal?"

High (reasoning_effort: 'high'):

• Complex reasoning required
• Premium features
• Use case: "Analyze my baby's feeding patterns over the last month and suggest optimizations"

Dynamic Reasoning Effort

You can adjust based on query complexity:

// Future enhancement: Analyze query complexity
const effort = this.determineReasoningEffort(chatDto.message);

const requestBody = {
  messages: azureMessages,
  reasoning_effort: effort,  // Dynamic based on query
};

---

Future Enhancements

1. Voice Service (Whisper)

Implement similar pattern for voice transcription:

export class WhisperService {
  async transcribeAudio(audioBuffer: Buffer): Promise<string> {
    if (this.aiProvider === 'azure') {
      return this.transcribeWithAzure(audioBuffer);
    }
    return this.transcribeWithOpenAI(audioBuffer);
  }

  private async transcribeWithAzure(audioBuffer: Buffer) {
    const url = `${this.azureWhisperEndpoint}/openai/deployments/${this.azureWhisperDeployment}/audio/transcriptions?api-version=${this.azureWhisperApiVersion}`;

    const formData = new FormData();
    formData.append('file', new Blob([audioBuffer]), 'audio.wav');

    const response = await axios.post(url, formData, {
      headers: {
        'api-key': this.azureWhisperApiKey,  // Separate key for Whisper
      },
    });

    return response.data.text;
  }
}

2. Embeddings Service

For pattern recognition and similarity search:

export class EmbeddingsService {
  async createEmbedding(text: string): Promise<number[]> {
    if (this.aiProvider === 'azure') {
      return this.createEmbeddingWithAzure(text);
    }
    return this.createEmbeddingWithOpenAI(text);
  }

  private async createEmbeddingWithAzure(text: string) {
    const url = `${this.azureEmbeddingsEndpoint}/openai/deployments/${this.azureEmbeddingsDeployment}/embeddings?api-version=${this.azureEmbeddingsApiVersion}`;

    const response = await axios.post(url, { input: text }, {
      headers: {
        'api-key': this.azureEmbeddingsApiKey,  // Separate key for Embeddings
      },
    });

    return response.data.data[0].embedding;
  }
}

3. Prompt Caching

Leverage Azure's cached input pricing (90% discount):

// Reuse identical system prompts for cost savings
const systemPrompt = `You are a helpful parenting assistant...`; // Cache this

4. Streaming Responses

For better UX with long responses:

const requestBody = {
  messages: azureMessages,
  stream: true,  // Enable streaming
  reasoning_effort: 'medium',
};

// Handle streamed response

---

Troubleshooting

Common Issues

1. "AI service not configured"

• Check AI_PROVIDER is set to 'azure'
• Verify AZURE_OPENAI_CHAT_API_KEY is set (not the old AZURE_OPENAI_API_KEY)
• Confirm AZURE_OPENAI_CHAT_ENDPOINT is correct

2. "Invalid API version"

• GPT-5 requires 2025-04-01-preview or later
• Update AZURE_OPENAI_CHAT_API_VERSION

3. "Deployment not found"

• Verify AZURE_OPENAI_CHAT_DEPLOYMENT matches Azure deployment name
• Check deployment is in same region as endpoint

4. High token usage

• GPT-5 reasoning tokens are additional overhead
• Reduce reasoning_effort if cost is concern
• Use 'minimal' for simple queries

5. Slow responses

• Higher reasoning_effort = slower responses
• Use 'low' or 'minimal' for time-sensitive queries
• Consider caching common responses

Debug Logging

Enable debug logs to see requests/responses:

this.logger.debug('Azure OpenAI request:', {
  url,
  deployment,
  reasoning_effort,
  messageCount,
});

this.logger.debug('Azure OpenAI response:', {
  model,
  finish_reason,
  prompt_tokens,
  completion_tokens,
  reasoning_tokens,
  total_tokens,
});

---

Summary

✅ Fully Configured:

• Environment variables for all Azure endpoints
• Chat (GPT-5), Whisper, Embeddings separately configurable
• No hardcoded values

✅ GPT-5 Support:

• Reasoning tokens tracked and returned
• Configurable reasoning effort (minimal/low/medium/high)
• Extended 400K context window ready

✅ Automatic Fallback:

• Azure → OpenAI if Azure fails
• Graceful degradation

✅ Monitoring:

• Detailed logging for debugging
• Token usage tracking (including reasoning tokens)
• Provider status endpoint

✅ Production Ready:

• Proper error handling
• Timeout configuration (30s)
• Metadata in responses

---

Next Steps

1. Add your actual API keys to .env:

   AZURE_OPENAI_CHAT_API_KEY=[your_chat_key]
   AZURE_OPENAI_WHISPER_API_KEY=[your_whisper_key]
   AZURE_OPENAI_EMBEDDINGS_API_KEY=[your_embeddings_key]
   

2. Restart the backend to pick up configuration:

   npm run start:dev
   

3. Test the integration:

   • Check provider status endpoint
   • Send a test chat message
   • Verify reasoning tokens in response

4. Monitor token usage:

   • Review logs for reasoning token counts
   • Adjust reasoning_effort based on usage patterns
   • Consider cost optimization strategies

5. Implement Voice & Embeddings (optional):

   • Follow similar patterns as chat service
   • Use separate Azure endpoints already configured