🧠 LLM Code Evaluation Framework

A modular framework for benchmarking Large Language Models (LLMs) on standardized code generation benchmarks.

The framework evaluates model-generated Python code using automated unit tests, computes evaluation metrics, categorizes common failure modes, generates JSON reports, and provides an interactive Streamlit dashboard for comparing models.

---

🚀 Live Demo

Try the interactive dashboard here:

The dashboard allows you to:

• 📊 Compare multiple LLMs across HumanEval and MBPP
• 📈 Visualize evaluation metrics and error distributions
• 🔍 Explore task-level evaluation results
• 🚀 Evaluate your own Groq-hosted model in real time

---

✨ Features

• 📊 Benchmark LLMs on multiple code generation datasets
• 🤖 Evaluate any Groq-hosted model
• ✅ Functional correctness evaluation using unit tests
• 📈 Automatic evaluation metrics
  • Accuracy
  • Passed / Failed tasks
  • Syntax Error Rate
  • Runtime Error Rate
  • Average Execution Time
• 🔍 Error categorization
  • Assertion Errors
  • Name Errors
  • Type Errors
  • Syntax Errors
• 💾 Export evaluation reports as JSON
• 📊 Interactive Streamlit dashboard
• 🚀 Live model evaluation from the dashboard

---

Why Code Evaluation?

Generated code may appear correct while still failing on hidden edge cases.

Instead of relying on manual inspection, this framework executes model-generated programs against standardized unit tests to objectively measure functional correctness.

This enables fair and reproducible comparison across different language models.

---

Supported Benchmarks

HumanEval

HumanEval is an OpenAI benchmark consisting of handwritten Python programming problems.

Each generated solution is evaluated using hidden unit tests, making it one of the most widely used benchmarks for measuring code generation capability.

---

MBPP (Mostly Basic Python Programming)

MBPP contains natural language programming tasks covering practical Python programming concepts including:

• Strings
• Lists
• Mathematics
• Recursion
• Searching
• Sorting

Solutions are evaluated using public unit tests.

---

Evaluation Pipeline

Prompt
   │
   ▼
Groq Model
   │
   ▼
Code Generation
   │
   ▼
Code Extraction
   │
   ▼
Python Executor
   │
   ▼
Benchmark Unit Tests
   │
   ▼
Evaluation Metrics
   │
   ▼
JSON Report
   │
   ▼
Interactive Dashboard

---

Project Structure

.
├── dashboard/
│   ├── app.py
│   └── dashboard_utils.py
│
├── results/
│   ├── humaneval/
│   └── mbpp/
│
├── src/
│   ├── benchmarks/
│   ├── evaluators/
│   ├── executors/
│   ├── metrics/
│   ├── models/
│   ├── reporters/
│   ├── utils/
│   └── main.py
│
├── requirements.txt
└── README.md

---

Supported Models

Currently tested using Groq-hosted models:

• Llama 3.3 70B Versatile
• GPT OSS 120B

The framework can evaluate any supported Groq model by providing the model name and API key through the Streamlit interface.

---

Evaluation Metrics

Metric	Description
Accuracy	Percentage of tasks successfully solved
Passed	Number of tasks passing all unit tests
Failed	Number of failed tasks
Syntax Error Rate	Percentage of syntax failures
Runtime Error Rate	Percentage of execution failures
Average Execution Time	Average runtime per task

---

Dashboard

The Streamlit dashboard provides:

• Benchmark comparison
• Model comparison
• Runtime analysis
• Error distribution
• Task explorer
• Live evaluation of user-provided Groq models

---

Live Evaluation

Users can benchmark their own Groq-hosted models directly from the dashboard.

Simply provide:

• Groq API Key
• Groq Model Name
• Benchmark
• Number of Tasks

The framework automatically:

• Runs evaluation
• Computes metrics
• Generates JSON reports
• Updates the benchmark dashboard

---

Installation

Clone the repository

git clone https://github.com/<username>/llm-code-evaluation-framework.git

cd llm-code-evaluation-framework

Install dependencies

pip install -r requirements.txt

Create a .env

GROQ_API_KEY=your_api_key_here

---

Running Evaluations

CLI

python src/main.py

Dashboard

streamlit run dashboard/app.py

---

Sample Results

Model	HumanEval	MBPP
GPT OSS 120B	92%	69%
Llama 3.3 70B	87%	54%

---

Technologies Used

• Python
• Streamlit
• Plotly
• Hugging Face Datasets
• Groq API

---

Future Work

• Support additional benchmarks (e.g., TACO)
• Hugging Face Inference support
• Ollama integration
• CSV export
• Pass@k evaluation
• Additional visualization and benchmarking metrics

---