Bài 4: Conversation Management — Session, Context Window & Memory Architecture

Kiến trúc Enterprise AI Chatbot Platform — Từ Prototype đến Production

Phần 2: Core Chatbot Engine

xdev.asia

1. Conversation Lifecycle

Mỗi conversation đi qua các trạng thái sau:

  ┌──────┐   user message   ┌────────┐   AI response   ┌──────────┐
  │ IDLE │──────────────────▶│ ACTIVE │────────────────▶│ WAITING  │
  └──────┘                   └───┬────┘                 └────┬─────┘
                                 │                           │
                            tool call                   user replies
                                 │                           │
                            ┌────▼─────┐                     │
                            │PROCESSING│─────────────────────┘
                            └────┬─────┘
                                 │
                    ┌────────────┼────────────┐
                    │            │            │
              ┌─────▼────┐ ┌────▼─────┐ ┌───▼──────┐
              │ ESCALATED│ │ RESOLVED │ │ TIMED_OUT│
              │(→ human) │ │  (done)  │ │ (30 min) │
              └──────────┘ └──────────┘ └──────────┘

type ConversationStatus =
  | 'idle'
  | 'active'
  | 'waiting_user'
  | 'processing'
  | 'escalated'
  | 'resolved'
  | 'timed_out';

interface ConversationManager {
  startConversation(params: StartParams): Promise<Conversation>;
  sendMessage(conversationId: string, message: UserMessage): Promise<AssistantMessage>;
  getHistory(conversationId: string, limit?: number): Promise<Message[]>;
  escalate(conversationId: string, reason: string): Promise<void>;
  close(conversationId: string): Promise<void>;
}

class ConversationService implements ConversationManager {
  constructor(
    private db: Database,
    private redis: Redis,
    private aiEngine: AIEngine,
    private memoryService: MemoryService,
    private eventBus: EventBus,
  ) {}

  async sendMessage(conversationId: string, userMessage: UserMessage): Promise<AssistantMessage> {
    const conversation = await this.db.conversation.findById(conversationId);
    if (!conversation) throw new NotFoundError('Conversation not found');

    // 1. Persist user message
    const savedMessage = await this.db.message.create({
      conversationId,
      role: 'user',
      content: userMessage.content,
      createdAt: new Date(),
    });

    // 2. Build context (messages + memory + knowledge)
    const context = await this.buildContext(conversation, savedMessage);

    // 3. Generate response via AI Engine
    const response = await this.aiEngine.generate(context);

    // 4. Persist assistant message
    const assistantMessage = await this.db.message.create({
      conversationId,
      role: 'assistant',
      content: response.content,
      metadata: {
        model: response.model,
        tokensUsed: response.usage,
        latencyMs: response.latencyMs,
        sources: response.citations,
      },
      createdAt: new Date(),
    });

    // 5. Update conversation state
    await this.db.conversation.update(conversationId, {
      status: 'waiting_user',
      lastMessageAt: new Date(),
    });

    // 6. Update memory
    await this.memoryService.updateFromMessage(conversation, savedMessage, assistantMessage);

    // 7. Emit events
    await this.eventBus.publish('conversation', {
      type: 'message.received',
      payload: { sessionId: conversationId, content: userMessage.content, role: 'user' },
    });

    return assistantMessage;
  }
}

2. Context Window Management

Thách thức lớn nhất: LLM có giới hạn context window (128K tokens cho GPT-4o). Conversation dài + RAG results + system prompt có thể vượt context limit.

┌──────────────────────── Context Window Budget ──────────────────────┐
│                                                                      │
│  ┌──────────┐ ┌────────────┐ ┌──────────┐ ┌────────┐ ┌──────────┐  │
│  │  System   │ │  Memory    │ │   RAG    │ │Messages│ │ Reserved │  │
│  │  Prompt   │ │  Summary   │ │ Results  │ │History │ │(response)│  │
│  │           │ │            │ │          │ │        │ │          │  │
│  │ ~500 tok  │ │ ~500 tok   │ │~2000 tok │ │ flex   │ │ 4096 tok │  │
│  └──────────┘ └────────────┘ └──────────┘ └────────┘ └──────────┘  │
│                                                                      │
│  Total Budget = model.maxContextTokens - reservedForResponse         │
│  Messages Budget = Total - systemPrompt - memory - ragResults        │
└──────────────────────────────────────────────────────────────────────┘

interface ContextBudget {
  totalTokens: number;          // Model's max context (e.g., 128000)
  reservedForResponse: number;  // Max output tokens (e.g., 4096)
  systemPromptTokens: number;   // ~500
  memoryTokens: number;         // ~500
  ragTokens: number;            // ~2000
  availableForMessages: number; // Calculated remaining
}

class ContextBuilder {
  constructor(
    private tokenCounter: TokenCounter,
    private memoryService: MemoryService,
    private ragPipeline: RAGPipeline,
  ) {}

  async buildContext(
    conversation: Conversation,
    currentMessage: Message,
    config: ContextConfig,
  ): Promise<BuiltContext> {
    const budget: ContextBudget = {
      totalTokens: config.modelMaxTokens,
      reservedForResponse: config.maxResponseTokens,
      systemPromptTokens: 0,
      memoryTokens: 0,
      ragTokens: 0,
      availableForMessages: 0,
    };

    // 1. System prompt (always included)
    const systemPrompt = config.systemPrompt;
    budget.systemPromptTokens = await this.tokenCounter.count(systemPrompt);

    // 2. Memory summary
    const memory = await this.memoryService.getSummary(conversation.id);
    budget.memoryTokens = await this.tokenCounter.count(memory);

    // 3. RAG results
    const ragResults = await this.ragPipeline.search(
      currentMessage.content,
      { tenantId: conversation.tenantId, topK: 5 },
    );
    const ragContent = this.formatRAGResults(ragResults);
    budget.ragTokens = await this.tokenCounter.count(ragContent);

    // 4. Calculate remaining budget for messages
    budget.availableForMessages =
      budget.totalTokens -
      budget.reservedForResponse -
      budget.systemPromptTokens -
      budget.memoryTokens -
      budget.ragTokens;

    // 5. Select messages within budget
    const messages = await this.selectMessages(
      conversation.id,
      budget.availableForMessages,
    );

    return {
      systemPrompt,
      memory,
      ragResults: ragContent,
      messages: [...messages, currentMessage],
      budget,
    };
  }

  private async selectMessages(
    conversationId: string,
    tokenBudget: number,
  ): Promise<Message[]> {
    // Strategy: Keep recent messages, summarize older ones
    const allMessages = await this.db.message.findByConversation(conversationId, {
      orderBy: 'createdAt',
      order: 'desc',
    });

    const selected: Message[] = [];
    let usedTokens = 0;

    for (const msg of allMessages) {
      const msgTokens = await this.tokenCounter.count(msg.content);
      if (usedTokens + msgTokens > tokenBudget) break;
      selected.unshift(msg);
      usedTokens += msgTokens;
    }

    return selected;
  }
}

3. Context Window Strategies

Strategy 1: Sliding Window

Giữ N messages gần nhất. Đơn giản nhưng mất context cũ.

Strategy 2: Summarization

Tóm tắt conversation cũ thành summary, chèn vào context.

class ConversationSummarizer {
  constructor(private llm: LLMProvider) {}

  async summarize(messages: Message[]): Promise<string> {
    const response = await this.llm.chat({
      model: 'gpt-4o-mini', // Cheap model for summarization
      messages: [
        {
          role: 'system',
          content: `Summarize the following conversation concisely in Vietnamese.
Focus on: key topics discussed, decisions made, pending questions, user preferences.
Keep under 200 words.`,
        },
        {
          role: 'user',
          content: messages.map(m => `${m.role}: ${m.content}`).join('\n'),
        },
      ],
      maxTokens: 300,
      temperature: 0.3,
    });

    return response.content;
  }

  // Auto-summarize when conversation gets too long
  async autoSummarize(
    conversationId: string,
    threshold: number = 20,
  ): Promise<void> {
    const messages = await this.db.message.findByConversation(conversationId);

    if (messages.length < threshold) return;

    // Summarize older messages (keep last 10)
    const toSummarize = messages.slice(0, -10);
    const summary = await this.summarize(toSummarize);

    await this.memoryService.updateConversationSummary(conversationId, summary);
  }
}

Strategy 3: Semantic Compression

Giữ messages có relevance cao với query hiện tại, remove irrelevant turns.

class SemanticCompressor {
  constructor(private embeddingService: EmbeddingService) {}

  async compress(
    messages: Message[],
    currentQuery: string,
    maxMessages: number,
  ): Promise<Message[]> {
    // Embed current query
    const queryEmbedding = await this.embeddingService.embed(currentQuery);

    // Score each message by relevance
    const scored = await Promise.all(
      messages.map(async (msg) => {
        const msgEmbedding = await this.embeddingService.embed(msg.content);
        const similarity = this.cosineSimilarity(queryEmbedding, msgEmbedding);
        return { message: msg, score: similarity };
      }),
    );

    // Always keep system messages and recent N messages
    const recent = messages.slice(-4);
    const older = scored
      .filter(s => !recent.includes(s.message))
      .sort((a, b) => b.score - a.score)
      .slice(0, maxMessages - recent.length)
      .map(s => s.message);

    // Merge and sort by timestamp
    return [...older, ...recent].sort(
      (a, b) => a.createdAt.getTime() - b.createdAt.getTime(),
    );
  }

  private cosineSimilarity(a: number[], b: number[]): number {
    let dot = 0, normA = 0, normB = 0;
    for (let i = 0; i < a.length; i++) {
      dot += a[i] * b[i];
      normA += a[i] * a[i];
      normB += b[i] * b[i];
    }
    return dot / (Math.sqrt(normA) * Math.sqrt(normB));
  }
}

4. Memory Architecture — Short-term vs Long-term

┌───────────────────────── MEMORY SYSTEM ─────────────────────────┐
│                                                                   │
│  ┌─────────────────── Short-term Memory ──────────────────────┐  │
│  │  • Message history (within conversation)                    │  │
│  │  • Working memory (current tool results, RAG results)       │  │
│  │  • Conversation summary (auto-generated)                    │  │
│  │  Storage: Redis (TTL: conversation duration)                │  │
│  └─────────────────────────────────────────────────────────────┘  │
│                                                                   │
│  ┌─────────────────── Long-term Memory ───────────────────────┐  │
│  │  • User preferences (e.g., "prefers Vietnamese")            │  │
│  │  • Past conversation summaries                              │  │
│  │  • Learned facts about user (e.g., "works at company X")    │  │
│  │  • Important decisions/actions taken                        │  │
│  │  Storage: PostgreSQL + Vector DB (permanent)                │  │
│  └─────────────────────────────────────────────────────────────┘  │
│                                                                   │
└───────────────────────────────────────────────────────────────────┘

interface MemoryService {
  // Short-term
  getConversationContext(conversationId: string): Promise<ShortTermMemory>;
  updateWorkingMemory(conversationId: string, data: unknown): Promise<void>;

  // Long-term
  getUserMemory(userId: string, query: string): Promise<MemoryEntry[]>;
  saveToLongTermMemory(userId: string, entry: MemoryEntry): Promise<void>;
  extractMemorableInfo(messages: Message[]): Promise<MemoryEntry[]>;
}

interface MemoryEntry {
  id: string;
  userId: string;
  type: 'preference' | 'fact' | 'decision' | 'summary';
  content: string;
  embedding: number[];
  confidence: number;
  source: { conversationId: string; messageId: string };
  createdAt: Date;
  lastAccessedAt: Date;
}

class MemoryServiceImpl implements MemoryService {
  async extractMemorableInfo(messages: Message[]): Promise<MemoryEntry[]> {
    const response = await this.llm.chat({
      model: 'gpt-4o-mini',
      messages: [
        {
          role: 'system',
          content: `Extract memorable information from this conversation.
Return JSON array with objects: { type, content, confidence }.
Types: "preference" (user likes/dislikes), "fact" (about user/company), "decision" (agreed actions).
Only extract high-confidence (>0.7) items. Be concise.`,
        },
        {
          role: 'user',
          content: messages.map(m => `${m.role}: ${m.content}`).join('\n'),
        },
      ],
      responseFormat: 'json',
    });

    return JSON.parse(response.content);
  }

  async getUserMemory(userId: string, query: string): Promise<MemoryEntry[]> {
    const queryEmbedding = await this.embeddingService.embed(query);

    // Search relevant memories by vector similarity
    const memories = await this.vectorDb.search({
      collection: 'user_memories',
      vector: queryEmbedding,
      filter: { userId },
      limit: 5,
      minScore: 0.7,
    });

    // Update last accessed (for memory decay)
    await Promise.all(
      memories.map(m =>
        this.db.memory.update(m.id, { lastAccessedAt: new Date() }),
      ),
    );

    return memories;
  }
}

5. Session Management

class SessionManager {
  private readonly SESSION_TTL = 30 * 60; // 30 minutes

  constructor(private redis: Redis) {}

  async getOrCreateSession(
    tenantId: string,
    channelType: string,
    channelUserId: string,
  ): Promise<Session> {
    const sessionKey = `session:${tenantId}:${channelType}:${channelUserId}`;
    const existing = await this.redis.get(sessionKey);

    if (existing) {
      const session = JSON.parse(existing) as Session;
      // Extend TTL on activity
      await this.redis.expire(sessionKey, this.SESSION_TTL);
      return session;
    }

    // Create new session
    const session: Session = {
      id: crypto.randomUUID(),
      tenantId,
      channelType,
      channelUserId,
      conversationId: crypto.randomUUID(),
      startedAt: new Date().toISOString(),
      metadata: {},
    };

    await this.redis.set(sessionKey, JSON.stringify(session), 'EX', this.SESSION_TTL);
    return session;
  }

  async invalidateSession(sessionKey: string): Promise<void> {
    await this.redis.del(sessionKey);
  }
}

Tổng kết Bài 4

• Conversation lifecycle: idle → active → processing → waiting → resolved/escalated
• Context window budget: system prompt + memory + RAG + messages + reserved = total tokens
• 3 strategies cho context management: Sliding Window (simple), Summarization (balanced), Semantic Compression (smart)
• Memory = Short-term (Redis, per-conversation) + Long-term (PostgreSQL + Vector DB, per-user)
• Auto-extract memorable info từ conversations cho personalization

Bài tiếp theo: RAG Pipeline — document ingestion, chunking strategies, hybrid search, re-ranking, và citation generation.