🤖 MLX Distributed AI Chat Interface

A production-ready distributed MLX inference system with multi-turn conversation support, optimized for Apple Silicon and 235B parameter thinking models.

✨ Features

🔄 Multi-Turn Conversations

• Full conversation history and context preservation
• Intelligent message cleaning for API compatibility
• Contextual reasoning that builds upon previous exchanges

💭 Real-Time Thinking Display

• Live streaming of model thinking process (<think> blocks)
• Intelligent text wrapping and formatting
• Context-aware status indicators

🎨 Professional UI

• Fixed 1200px chatbot width with comprehensive blue theme
• Enhanced Gradio interface with logo header
• Responsive design with proper text wrapping

⚡ Performance Optimized

• 5-minute timeouts for complex reasoning tasks
• Up to 5,000 tokens per response
• Efficient streaming with graceful fallbacks
• OpenAI-compatible API integration

Key Features

• 🔄 Real-time Streaming: Live text streaming from thinking models
• 🧠 Optimized for Reasoning: 5-minute timeouts, 5K token responses
• 🌐 Web Interface: Clean Gradio UI at http://127.0.0.1:7866
• ⚙️ Flexible Controls: Streaming toggle, model selection, response length
• 📖 Enhanced Display: Formatted thinking blocks, better readability
• 📁 Document Support: PDF, DOCX, TXT summarization

🏗️ Architecture

Infrastructure

• Primary Server: 172.16.0.1:80 (MLX API endpoint)
• API Format: OpenAI-compatible /v1/chat/completions
• Authentication: Bearer token via MLX_API_KEY environment variable
• Model: qwen3-235b-a22b-bf16 (235B parameter thinking model)

Components

1. gradio_ui.py - Main chat interface with streaming
2. gradio_rag_ui.py - Advanced RAG interface with document processing
3. test_api.py - API connectivity testing tool
4. Various pipeline scripts - Distributed processing utilities

🛠️ Setup & Usage

Prerequisites

# Set API key
export MLX_API_KEY="your_api_key_here"

# Ensure Python environment
source mlx-env/bin/activate  # if using virtual environment

Quick Start

# Start the main chat interface
./start_gradio_ui.sh

# Or start the RAG interface
./launch_rag_ui.sh

Testing

# Test API connectivity
python test_api.py "your_api_key"

📋 Interface Features

Main Chat Interface (Port 7866)

• Streaming Responses: Real-time text generation
• Model Selection: Choose from available ready models
• Response Control: 100-5000 token responses
• Document Upload: PDF/DOCX/TXT summarization
• Thinking Display: Formatted <think> blocks

Settings

• Max Tokens: 100-5000 (default: 2000)
• Timeout: 300 seconds (5 minutes)
• Streaming: Enabled by default
• Model: qwen3-235b-a22b-bf16

🔧 Configuration Files

Active Configuration

• API Base: http://172.16.0.1:80/v1
• Hosts: 172.16.0.1,172.16.0.2 (primary, secondary)
• Default Model: qwen3-235b-a22b-bf16

Key Files

• gradio_ui.py - Main interface (✅ Working)
• start_gradio_ui.sh - Launch script
• hosts.json - Distributed configuration
• hosts.txt - Host list for MPI

🎯 Performance Optimizations

Streaming Implementation

• Server-Sent Events: Proper SSE parsing
• Chunked Updates: Updates every 3 chunks for smooth display
• Fallback Support: Auto-switches to non-streaming if needed
• Progress Tracking: Real-time word count and status

Display Enhancements

• Thinking Blocks: <think> → 💭 Thinking: code blocks
• Text Formatting: Better line breaks and spacing
• Copy Support: Easy text copying for external use
• Increased Height: 500px chat area for better readability

📊 Working Models

Ready Models (as of last check)

1. qwen3-235b-a22b-bf16 ✅ (Primary - 1 ready replica)
2. qwen3-0.6b-gguf (0 ready replicas)
3. qwen3 (0 ready replicas)
4. qwen3-235b-a22b-q8_0 (0 ready replicas)
5. qwen3-235b-a22b-gguf (0 ready replicas)
6. DeepSeek-R1-0528-Q4_K_M (0 ready replicas)

🚨 Troubleshooting

Common Issues

1. Port 7866 in use: Change port in gradio_ui.py line 490
2. API Auth Error: Check MLX_API_KEY environment variable
3. No streaming: Toggle streaming off in interface
4. Timeout: Increase timeout for complex queries

Status Checks

# Check API status
curl -H "Authorization: Bearer ${MLX_API_KEY}" http://172.16.0.1:80/v1/models

# Check processes
ps aux | grep gradio

# Check ports
netstat -an | grep 7866

📝 Development Notes

Recent Improvements (Current Checkpoint)

1. Fixed streaming parser - Proper SSE handling
2. Enhanced readability - Formatted thinking blocks
3. Increased timeouts - 5 minutes for thinking models
4. Extended token limits - Up to 5K tokens per response
5. Better error handling - Graceful fallbacks
6. UI improvements - Model selector, streaming toggle

Next Potential Features

• Multi-model conversation comparison
• Advanced RAG with vector embeddings
• Conversation export/import
• Custom prompt templates
• Response regeneration
• Conversation branching

🔒 Security Notes

• API keys excluded from Git (see .gitignore)
• Local-only interface (127.0.0.1)
• Models and large files excluded from version control

📦 Dependencies

• gradio
• requests
• PyPDF2
• python-docx
• json (built-in)
• os (built-in)

---

Checkpoint Created: 2025-06-18 - Fully functional streaming interface with thinking model optimization