LUNA - 100M Parameter LLM from Scratch

Custom ~100M parameter GPT model (Pythia-like architecture) pretrained on 4.5B tokens of clean English text.

Quick Start (RunPod / Cloud GPU)

1. Clone & Install (one command)

git clone https://huggingface.co/spaces/ASTERIZER/LUNA /workspace/LUNA && \
cd /workspace/LUNA && \
pip install -q -r requirements.txt

2. Get Dataset + Train (one command)

The dataset (~4.5B tokens) is hosted as a zip at ASTERIZER/Luna_Dataset. The script downloads, extracts, and starts training automatically.

From HuggingFace (recommended):

bash setup_and_train.sh huggingface ASTERIZER/Luna_Dataset

From Google Drive:

bash setup_and_train.sh gdrive YOUR_GDRIVE_FOLDER_ID

Smoke test (10M tokens only):

bash setup_and_train.sh huggingface ASTERIZER/Luna_Dataset 10000000

That's it. The script auto-detects your GPU, VRAM, RAM, CPU cores and configures everything for maximum utilization.

---

How It Works

Auto vs Manual Config

All hyperparameters live in train_config.yaml:

auto_config: true   # auto-detect everything from hardware
auto_config: false  # use exact values below, no overrides

When auto_config: true (default), the trainer:

• Probes VRAM via binary search to find max micro_batch_size (82% safety)
• Sets grad_accum to hit the target global_batch_size
• Picks precision (bf16 on Ampere+, fp16 otherwise)
• Scales workers to half your CPU cores, capped by RAM
• Enables torch.compile if Triton is available (Linux)

When auto_config: false, every value in the YAML is used exactly as-is.

CLI Overrides

Any config value can be overridden from the command line:

python train.py --config train_config.yaml --data_path /data/litdata --max_tokens 100000000

Priority: CLI args > train_config.yaml > auto-detection

---

Dataset

• 4,515,286,950 tokens (4.5B) in 270 binary chunks
• Sources: Wikipedia, FineWeb-Edu, OpenWebText (deduplicated, cleaned)
• Format: LitData binary (int32, block_size=1025, TokensLoader)
• Tokenizer: EleutherAI/pythia-160m (50,254 vocab)

Model Architecture

File Structure

LUNA/
  train.py              # Main training script (config-driven, auto-detects hardware)
  train_config.yaml     # All hyperparameters (auto_config: true/false)
  fetch_data.py         # Downloads dataset from HuggingFace / GDrive
  setup_and_train.sh    # One-command cloud entrypoint
  benchmark_runpod.py   # Local performance benchmark
  requirements.txt      # Python dependencies
  Base/
    checkpoints/EleutherAI/pythia-160m/   # Tokenizer files
    configs/             # Legacy litgpt YAML configs (reference only)
    scripts/             # Data preprocessing scripts

Training Performance Notes

Throughput varies by GPU — measure yours with benchmark_runpod.py.

Resume Training

Training auto-saves latest.pt every save_interval steps. If interrupted, just re-run the same command -- it picks up where it left off.

---

Verified Configs (What Worked)

These are the exact configurations that produced the current LUNA 100M model. Do NOT change them unless you know what you're doing — they are proven and validated.

---

1. Pretraining — 4.5 Billion Tokens

The pretraining ran in two phases on an RTX 4060 Ti 16GB.

Phase 1: Bulk pretraining on 3B general web tokens

Phase 2: Continued pretraining on clean English (Wikipedia + FineWeb-Edu)

Final combined dataset used for the production run:

---

2. SFT Fine-Tuning — ~145 Million Tokens

Supervised fine-tuning on the pretrained LUNA 100M checkpoint.

Model (frozen architecture — matches pretrain exactly):

Training hyperparameters:

Prompt format (used during training — must be matched at inference):

### Instruction:
{instruction}

### Response:

With optional input field:

### Instruction:
{instruction}

### Input:
{input}

### Response:

Loss masking: Only the response tokens (after ### Response:\n) contribute to the loss. The prompt tokens are masked out (loss_mask=0). EOS token (id=0) is appended to every response.

---

3. SFT Inference / Chat — Loaded Configs

These are the exact generation parameters loaded when running chat.py or validate_sft.py. They match the training eval config from sft_train.py.

python chat.py --ckpt "Base\out\sft\model.pth"

Model loading:

Generation parameters:

Prompt template (must match training exactly):

def format_prompt(instruction, context=""):
    if instruction and context:
        return f"### Instruction:\n{instruction}\n\n### Input:\n{context}\n\n### Response:\n"
    else:
        return f"### Instruction:\n{instruction}\n\n### Response:\n"

Critical notes:

• There is NO Alpaca preamble text (e.g., "Below is an instruction...") — the model was never trained with one
• EOS token is id=0 (pythia), not 50276 (GPT-NeoX) — using the wrong EOS causes the model to never stop
• Generation stops when EOS is produced OR max_new_tokens is reached
• For longer responses in chat, you can override: --max_new 512
• For less repetition in production, add: --rep_pen 1.05

Validation results with these configs (100 complex examples):

---

License

Private / ASTERIZER 2026