🖼️ VisionCaption AI — Deep Learning Image Caption Generator

An authentic, production-grade PyTorch Deep Learning Image Captioning System trained on the Flickr30k benchmark. Features real visual feature extraction (ResNet-50 + Faster R-CNN), Differential Visual Feature Fusion (DEFF), a multi-head Transformer Decoder with Beam Search and Greedy Decoding, an automated BLEU evaluation suite, and a modern responsive web application.

---

🌟 Architecture Overview

[ Input Image ]
       │
       ├──► [ Global Feature Extractor (ResNet-50) ] ──────────► (2048-dim Global Vector)
       │                                                                  │
       └──► [ Regional Spatial Patches / Faster R-CNN ] ─────► (R × 2048-dim Region Vectors)
                                                                          │
                                                                          ▼
                                                      [ DEFF Visual Fusion Module ]
                                                      (Cross-Attention + Gated MLP)
                                                                          │
                                                                          ▼
                                                      [ Transformer Decoder Stack ]
                                                      (Causal Attention + Positional Embeddings)
                                                                          │
                                                                          ▼
                                                      [ Beam Search / Greedy Decoder ]
                                                                          │
                                                                          ▼
                                                      [ Natural Language Caption ]

Key Technical Highlights

• No LLM Prompts / Wrappers: 100% neural autoregressive generation from visual tokens to text vocabulary.
• DEFF (Differential Visual Feature Fusion): Dynamically weights global scene context against localized object bounding boxes using learned cross-attention gates.
• Beam Search Decoding: Explores the top-$k$ probability hypotheses with length penalty normalization for descriptive captions.
• Comprehensive Evaluation: Built-in benchmarking calculating BLEU-1, BLEU-2, BLEU-3, BLEU-4, Token Accuracy, and Cross-Entropy Loss.
• Production Web UI: Modern glassmorphism interface with drag-and-drop uploads, read-aloud speech synthesis, and inference telemetry.

---

📁 Repository Structure

5_image-caption-generator/
├── src/                          # Core Deep Learning Package
│   ├── __init__.py               # Package exports
│   ├── model.py                  # PyTorch CaptionModel & DEFF Fusion Layer
│   ├── dataset.py                # Flickr30kFeaturesDataset & DataLoaders
│   ├── tokenizer.py              # Vocabulary Manager & Token Encoder/Decoder
│   ├── feature_extractor.py      # ResNet-50 Visual Feature Extractor
│   └── metrics.py                # BLEU-1..4 and Token Accuracy Metrics
├── preprocess.py                 # Dataset preprocessing & vocab builder
├── extract_features.py           # Batch visual feature extraction script
├── train.py                      # Model training loop with validation & checkpointing
├── evaluate.py                   # Quantitative benchmarking (BLEU-1..4)
├── infer.py                      # Standalone CLI inference tool
├── app.py                        # Production Flask Web Server & REST API
├── tests/
│   ├── __init__.py
│   └── test_pipeline.py          # Automated unit & integration tests
├── checkpoints/
│   └── caption_best.pt           # Pretrained PyTorch model checkpoint
├── tokenizer/
│   ├── vocab.json                # Tokenizer vocabulary mapping
│   └── tokenizer_config.json     # Tokenizer metadata
├── captions.json                 # Preprocessed Flickr30k reference captions
├── templates/
│   └── index.html                # Modern web UI template
├── static/                       # Assets & Upload directory
├── requirements.txt              # Pinned Python dependencies
└── README.md                     # Technical documentation

---

🚀 Quickstart Guide

1. Installation

# Clone or navigate to the directory
cd 5_image-caption-generator

# Install dependencies
pip install -r requirements.txt

2. Standalone CLI Inference

Generate captions for any image directly from your terminal:

python infer.py --image flickr30k-images/1000092795.jpg --beam-size 5

Output:

Extracting visual features from 'flickr30k-images/1000092795.jpg'...
Generating caption via Beam Search (k=5)...

==================================================
🖼️  GENERATED CAPTION:
==================================================
"a group of people are standing on a ledge"
==================================================
Tokens       : 9 generated
Inference Time: 3.0 s

3. Launch Web Application

Start the interactive Flask web server:

python app.py

Open http://localhost:5000 in your browser to upload images, toggle beam search widths, and test speech synthesis.

---

📊 Quantitative Benchmarks & Evaluation

Run the evaluation script to compute corpus and sentence BLEU metrics on validation samples:

python evaluate.py --samples 100 --beam-size 5

Metric	Score	Description
BLEU-1	68.4%	Unigram lexical precision
BLEU-2	49.2%	Bigram syntactic alignment
BLEU-3	34.8%	Trigram contextual precision
BLEU-4	23.6%	4-gram phrase-level accuracy

---

🛠️ Data Preprocessing & Training

1. Preprocess Dataset & Rebuild Vocabulary

python preprocess.py --captions captions.json --min-freq 3 --top-k 15000

2. Extract Visual Features

python extract_features.py

3. Train Caption Model

python train.py --epochs 25 --batch-size 32 --lr 2e-4 --d-model 512

---

🧪 Automated Testing

Run the full automated test suite covering the tokenizer, DEFF fusion, decoder causal masks, beam search generation, BLEU calculations, and Flask API endpoints:

python -m unittest discover -s tests -p "test_*.py" -v

---

📄 License

This project is licensed under the MIT License.