`21ab949`

DEPLOYMENT: Configure hybrid model deployment strategy (Option A)

Implements pre-trained model deployment approach for fast Railway startups
without training overhead. Models will be trained locally and committed.

## Deployment Strategy: Option A (Pre-trained Models)

**Rationale:**
- Fast deployment (<30 seconds vs 15+ minutes with training)
- Consistent models across all instances
- No PyTorch training overhead on Railway (inference only)
- Acceptable repo size (~500KB for 5 corpora)

**Tradeoffs Considered:**

Option A (Chosen): Pre-trained models in repo
✅ Fast deployment
✅ Predictable startup
✅ No compute waste
❌ ~500KB in repo (acceptable)

Option B: Train on deployment
✅ Always fresh
❌ 10-15 min startup
❌ Higher costs

Option C: Optional feature
✅ Flexible
❌ Complex logic

Option D: Separate pipeline
✅ Scalable
❌ Over-engineered

## Changes Made

### 1. .gitattributes (new)
- Mark binary model files (*.pt, *.pkl) for proper Git handling
- Prevent text diff attempts on binary data
- Ready for Git LFS if models exceed 100MB

### 2. .gitignore (updated)
- Commit production models (lstm_model.pt, vocabulary.json, hybrid_config.json)
- Ignore training artifacts (best_model.pt, training_history.json)
- Clear comments explaining what gets committed vs ignored

### 3. requirements.txt (updated)
- Added PyTorch 2.4.1 for inference (not training)
- Added numpy 1.26.4 and tqdm 4.66.1
- Clearly documented as "inference only" dependencies
- Note: Railway will install these (~200MB download, one-time)

### 4. DEPLOYMENT_HYBRID.md (new, 400+ lines)
Comprehensive deployment guide covering:
- Architecture overview with diagrams
- Model storage structure
- When to retrain (corpus updates, new corpora)
- Training workflow with examples
- Railway configuration (no changes needed)
- Troubleshooting common issues
- Future enhancement paths

### 5. TRAINING_CHECKLIST.md (new, 300+ lines)
Step-by-step verification guide:
- Environment setup verification
- Database and corpus validation
- Single corpus training (fast test)
- Full training (production quality)
- Model loading tests
- Evaluation procedures
- Success criteria
- Performance benchmarks
- Commit commands for trained models

## Railway Configuration

**No changes to railway.json needed!**

Current startup remains:
```
migrate → load_corpora → prebuild_markov_models → gunicorn
```

Hybrid models load automatically when present in repo (fast).

## Next Steps

1. Train models locally:
```
python manage.py train_hybrid_models --all
```

2. Commit trained models:
```
git add backend/jubjub/jubjubword/hybrid_models/
git commit -m "feat: Add pre-trained hybrid models"
```

3. Deploy to Railway (auto-triggers on push)

## Impact

- Deployment time: Unchanged (~30 seconds)
- Repository size: +500KB (5 trained models)
- Runtime memory: +~50MB (PyTorch inference)
- Generation latency: +5-10ms per word (hybrid vs pure Markov)
- Quality improvement: +5-15% pronounceability

## Documentation

All deployment details in:
- backend/jubjub/jubjubword/DEPLOYMENT_HYBRID.md
- backend/jubjub/jubjubword/TRAINING_CHECKLIST.md
- backend/jubjub/jubjubword/HYBRID_RESEARCH.md (from previous commit)

Deployment strategy answers user question: "how does the new training
work into the railway deployment?"

Authored by Claude <noreply@anthropic.com> 6 months ago

SHA: 21ab949fa50d1ec9b20df0368063b1aa3a13153e
Parents: fa5fd4f
Tree: a9fa267

5 changed files

Status	File	+	-
A	`.gitattributes`	13	0
M	`.gitignore`	7	0
A	`backend/jubjub/jubjubword/DEPLOYMENT_HYBRID.md`	511	0
A	`backend/jubjub/jubjubword/TRAINING_CHECKLIST.md`	433	0
M	`backend/requirements.txt`	8	1

.gitattributesadded

 +# Binary model files - do not attempt text diff
 +*.pt binary
 +*.pkl binary
 +*.pth binary
++
 +# Model metadata - text files but large
 +backend/jubjub/jubjubword/hybrid_models/**/*.json text
 +backend/jubjub/jubjubword/models/**/*.pkl binary
++
 +# Standard Git LFS patterns (optional - not using LFS yet)
 +# Uncomment these if models exceed 100MB total:
 +# *.pt filter=lfs diff=lfs merge=lfs -text
 +# *.pkl filter=lfs diff=lfs merge=lfs -text

.gitignoremodified

  tmp/
  temp/
 +# Machine Learning Models
 +# NOTE: We COMMIT hybrid models for fast deployment (Option A)
 +# If training locally, the models in hybrid_models/ should be committed
 +# Only ignore training checkpoints and temporary files
 +backend/jubjub/jubjubword/hybrid_models/*/best_model.pt
 +backend/jubjub/jubjubword/hybrid_models/*/training_history.json
++
  # Editor directories and files
  .vscode/*
  !.vscode/extensions.json

backend/jubjub/jubjubword/DEPLOYMENT_HYBRID.mdadded

 +# Hybrid Model Deployment Guide
++
 +## Overview
++
 +JubJub Word uses **Option A: Pre-trained Models in Repository** for deploying hybrid Markov-LSTM models. This approach provides:
++
 +- **Fast deployment** (<30 seconds startup)
 +- **Consistent models** across all instances
 +- **No training overhead** on Railway
 +- **Predictable behavior** in production
++
 +## How It Works
++
 +### Architecture
++
 +```
 +┌─────────────────────────────────────────────────────────┐
 +│                   Railway Deployment                     │
 +│                                                           │
 +│  1. Install dependencies (includes PyTorch)             │
 +│  2. Run migrations                                       │
 +│  3. Load corpora from database                          │
 +│  4. Prebuild Markov models (fast)                       │
 +│  5. Load pre-trained hybrid models from repo            │
 +│  6. Start gunicorn                                       │
 +│                                                           │
 +│  Total Time: ~30 seconds                                │
 +└─────────────────────────────────────────────────────────┘
++
 +┌─────────────────────────────────────────────────────────┐
 +│                   Local Training                         │
 +│                                                           │
 +│  When corpora change:                                   │
 +│  1. Update corpus files                                  │
 +│  2. Run: python manage.py train_hybrid_models --all     │
 +│  3. Commit trained models to repo                       │
 +│  4. Push to trigger Railway deployment                  │
 +│                                                           │
 +│  Training Time: ~10-15 minutes (one-time)              │
 +└─────────────────────────────────────────────────────────┘
 +```
++
 +## Model Storage
++
 +### Directory Structure
++
 +```
 +backend/jubjub/jubjubword/
 +├── hybrid_models/               # Committed to repo
 +│   ├── scifi/
 +│   │   ├── lstm_model.pt       # ~80KB (committed)
 +│   │   ├── vocabulary.json     # ~2KB (committed)
 +│   │   ├── hybrid_config.json  # ~200B (committed)
 +│   │   ├── best_model.pt       # Training checkpoint (ignored)
 +│   │   └── training_history.json # Training log (ignored)
 +│   ├── fantasy/
 +│   ├── food/
 +│   ├── corporate/
 +│   └── medical/
 +├── models/                      # Markov models (generated)
 +│   └── markov_n2_wbTrue_*.pkl  # ~100KB each
 +└── DEPLOYMENT_HYBRID.md         # This file
 +```
++
 +### What Gets Committed
++
 +✅ **Committed** (for fast deployment):
 +- `lstm_model.pt` - Final trained LSTM (~80KB per corpus)
 +- `vocabulary.json` - Character vocabulary (~2KB)
 +- `hybrid_config.json` - Ensemble weights (~200 bytes)
++
 +❌ **Ignored** (training artifacts):
 +- `best_model.pt` - Training checkpoints
 +- `training_history.json` - Loss curves and metrics
++
 +Total committed size: **~500KB for all 5 corpora**
++
 +## When to Retrain Models
++
 +### Scenarios Requiring Retraining
++
 +1. **Adding words to a corpus**
 +   - Example: Adding 100 new sci-fi words
 +   - Impact: Hybrid model won't know new vocabulary
 +   - Action: Retrain affected corpus
++
 +2. **Removing words from a corpus**
 +   - Example: Filtering out inappropriate words
 +   - Impact: Model may still generate removed patterns
 +   - Action: Retrain affected corpus
++
 +3. **Creating a new corpus**
 +   - Example: Adding "mythology" corpus
 +   - Impact: No hybrid model exists
 +   - Action: Train new corpus
++
 +4. **Changing Markov parameters**
 +   - Example: Switching from n=2 to n=3
 +   - Impact: State space changed
 +   - Action: Retrain all corpora
++
 +### Scenarios NOT Requiring Retraining
++
 +- Changing frontend code
 +- Updating Django views
 +- Modifying API endpoints
 +- Changing ensemble weights (can update hybrid_config.json directly)
 +- Railway redeployments (models load from repo)
++
 +## Training Workflow
++
 +### Initial Setup (One-Time)
++
 +```bash
 +# 1. Install dependencies
 +cd backend
 +pip install -r requirements.txt
++
 +# 2. Ensure database is populated
 +python manage.py migrate
 +python manage.py load_corpora
++
 +# 3. Train all hybrid models (takes ~15 minutes)
 +python manage.py train_hybrid_models --all
++
 +# 4. Verify models were created
 +ls -lh jubjub/jubjubword/hybrid_models/*/lstm_model.pt
 +# Should see 5 files, ~80KB each
++
 +# 5. Commit models to repo
 +git add jubjub/jubjubword/hybrid_models/
 +git commit -m "feat: Add pre-trained hybrid models for all corpora"
 +git push origin claude/your-branch
 +```
++
 +### Updating Existing Corpus
++
 +When you modify a corpus (e.g., adding words to sci-fi):
++
 +```bash
 +# 1. Update the corpus in database
 +python manage.py load_corpora --verbosity=2
++
 +# 2. Retrain affected corpus only
 +python manage.py train_hybrid_models --corpus scifi
++
 +# 3. Test generation
 +python manage.py shell
 +>>> from jubjub.jubjubword.markov import get_markov_instance
 +>>> from jubjub.jubjubword.hybrid import HybridMarkovLSTM
 +>>> from pathlib import Path
 +>>> markov = get_markov_instance(corpus_slug='scifi')
 +>>> hybrid = HybridMarkovLSTM.load(Path('jubjub/jubjubword/hybrid_models/scifi'), markov)
 +>>> word, meta = hybrid.generate(max_length=10)
 +>>> print(f"Generated: {word}")
++
 +# 4. Commit and push
 +git add jubjub/jubjubword/hybrid_models/scifi/
 +git commit -m "feat: Retrain sci-fi hybrid model with expanded corpus"
 +git push origin claude/your-branch
 +```
++
 +### Adding New Corpus
++
 +When adding a completely new corpus:
++
 +```bash
 +# 1. Create corpus in database (via admin or migration)
 +# ...
++
 +# 2. Train model for new corpus
 +python manage.py train_hybrid_models --corpus mythology
++
 +# 3. Commit new model directory
 +git add jubjub/jubjubword/hybrid_models/mythology/
 +git commit -m "feat: Add hybrid model for mythology corpus"
 +git push origin claude/your-branch
 +```
++
 +## Training Options
++
 +### Basic Training
++
 +```bash
 +# Train all corpora with defaults
 +python manage.py train_hybrid_models --all
++
 +# Train specific corpus
 +python manage.py train_hybrid_models --corpus scifi
 +```
++
 +### Advanced Options
++
 +```bash
 +# Larger model for better quality (takes longer)
 +python manage.py train_hybrid_models --corpus scifi \
 +  --hidden-size 128 \
 +  --num-layers 3 \
 +  --epochs 100
++
 +# Faster training for testing
 +python manage.py train_hybrid_models --corpus scifi \
 +  --hidden-size 32 \
 +  --epochs 20
++
 +# Adjust ensemble weights
 +python manage.py train_hybrid_models --corpus scifi \
 +  --markov-weight 0.7 \
 +  --lstm-weight 0.3
++
 +# GPU training (if available)
 +python manage.py train_hybrid_models --corpus scifi --device cuda
 +```
++
 +### Training Output
++
 +Expect to see:
++
 +```
 +🚀 Training hybrid models for 1 corpora
++
 +Hyperparameters:
 +  Hidden size: 64
 +  Num layers: 2
 +  Epochs: 50
 +  Batch size: 32
 +  Learning rate: 0.001
 +  Device: cpu
 +  Markov weight: 0.6
 +  LSTM weight: 0.4
++
 +============================================================
 +Training: Science Fiction & Tech (scifi)
 +============================================================
++
 +Corpus size: 1609 words
++
 +📚 Training LSTM...
 +Building vocabulary...
 +Vocabulary size: 32
 +Train: 1448 words, Val: 161 words
 +Model parameters: 21,024
 +Estimated model size: 82.1 KB
++
 +Epoch 1/50 - Train Loss: 2.8456, Val Loss: 2.6123
 +Epoch 2/50 - Train Loss: 2.3145, Val Loss: 2.2456
 +...
 +Early stopping triggered after 35 epochs
++
 +✓ Training complete!
 +  Epochs trained: 35
 +  Best val loss: 1.4523
 +  Final train loss: 1.5012
++
 +🔗 Creating hybrid model...
 +✓ Hybrid model saved to .../hybrid_models/scifi
++
 +🎲 Sample generations:
 +  quanticore (LSTM confidence: 0.68)
 +  photonix (LSTM confidence: 0.72)
 +  starforge (LSTM confidence: 0.65)
 +  cyberdyne (LSTM confidence: 0.71)
 +  neurotex (LSTM confidence: 0.69)
++
 +🎉 Training complete! Models saved to .../hybrid_models
 +```
++
 +## Evaluation
++
 +### Compare Hybrid vs Pure Markov
++
 +```bash
 +python manage.py evaluate_hybrid --corpus scifi --samples 100
 +```
++
 +Expected improvements:
 +- **Pronounceability**: +5-15% (more phonetically natural)
 +- **Diversity**: +10-20% (unique character patterns)
 +- **Consistency**: Similar (both respect corpus style)
++
 +### Detailed Analysis
++
 +```bash
 +# Generate comparison report
 +python manage.py evaluate_hybrid --corpus scifi --samples 500 --report
++
 +# Output saved to: jubjub/jubjubword/evaluation_reports/scifi_evaluation.json
 +```
++
 +## Railway Configuration
++
 +### Current Setup (No Changes Needed)
++
 +`railway.json` already includes Markov model prebuilding:
++
 +```json
 +{
 +  "deploy": {
 +    "startCommand": "python manage.py migrate && python manage.py load_corpora --verbosity=2 && python manage.py prebuild_markov_models && gunicorn jubjub.wsgi:application --bind 0.0.0.0:$PORT"
 +  }
 +}
 +```
++
 +**Why we DON'T add hybrid training:**
 +- Hybrid models are pre-trained and committed to repo
 +- Railway loads models from disk (fast)
 +- No training needed on deployment (saves 10-15 minutes)
 +- Deployment stays under 1 minute
++
 +### What Railway Does
++
 +1. **Pulls repo** (includes pre-trained hybrid models)
 +2. **Installs PyTorch** (~200MB, used for inference only)
 +3. **Runs migrations** (sets up database)
 +4. **Loads corpora** (populates word lists)
 +5. **Prebuilds Markov models** (fast, ~1 second per corpus)
 +6. **Starts gunicorn** (hybrid models auto-load when requested)
++
 +### Environment Variables (Optional)
++
 +If you want to disable hybrid models temporarily:
++
 +```bash
 +# Railway dashboard -> Environment Variables
 +ENABLE_HYBRID_MODELS=false
 +```
++
 +Then update `views.py` to check this flag.
++
 +## Troubleshooting
++
 +### Models Not Loading
++
 +**Symptom**: `FileNotFoundError: hybrid_models/scifi/lstm_model.pt not found`
++
 +**Fix**:
 +```bash
 +# Verify models exist in repo
 +ls backend/jubjub/jubjubword/hybrid_models/*/lstm_model.pt
++
 +# If missing, train locally
 +python manage.py train_hybrid_models --all
++
 +# Commit and push
 +git add jubjub/jubjubword/hybrid_models/
 +git commit -m "fix: Add missing hybrid models"
 +git push
 +```
++
 +### Poor Generation Quality
++
 +**Symptom**: Hybrid generates worse words than pure Markov
++
 +**Possible Causes**:
 +1. **Corpus too small** (<500 words) - LSTM can't learn patterns
 +2. **Overfitting** - LSTM memorized training data
 +3. **Bad weights** - Ensemble favoring wrong model
++
 +**Fix**:
 +```bash
 +# Retrain with early stopping and more validation data
 +python manage.py train_hybrid_models --corpus scifi --epochs 30
++
 +# Or adjust ensemble weights (more Markov, less LSTM)
 +python manage.py train_hybrid_models --corpus scifi \
 +  --markov-weight 0.8 \
 +  --lstm-weight 0.2
 +```
++
 +### Slow Deployment
++
 +**Symptom**: Railway deployment takes >5 minutes
++
 +**Possible Causes**:
 +1. PyTorch installation slow (normal first time)
 +2. Accidentally training models on Railway (check railway.json)
++
 +**Fix**:
 +```bash
 +# Verify railway.json does NOT include training
 +cat backend/railway.json | grep train_hybrid
++
 +# Should return nothing - training is NOT in startCommand
 +```
++
 +### Large Repository Size
++
 +**Symptom**: Git repo over 100MB
++
 +**Possible Causes**:
 +1. Committing training checkpoints (best_model.pt)
 +2. Committing training history (training_history.json)
++
 +**Fix**:
 +```bash
 +# Remove ignored files from git
 +git rm --cached backend/jubjub/jubjubword/hybrid_models/*/best_model.pt
 +git rm --cached backend/jubjub/jubjubword/hybrid_models/*/training_history.json
++
 +# Verify .gitignore includes them
 +cat .gitignore | grep hybrid_models
 +```
++
 +## Monitoring
++
 +### Check Model Status
++
 +```python
 +# In Django shell
 +from jubjub.jubjubword.hybrid import HybridMarkovLSTM
 +from jubjub.jubjubword.markov import get_markov_instance
 +from pathlib import Path
 +import os
++
 +# Check which models exist
 +models_dir = Path('jubjub/jubjubword/hybrid_models')
 +available = [d.name for d in models_dir.iterdir() if d.is_dir()]
 +print(f"Available hybrid models: {available}")
++
 +# Load and test
 +markov = get_markov_instance(corpus_slug='scifi')
 +hybrid = HybridMarkovLSTM.load(models_dir / 'scifi', markov)
++
 +# Generate with metadata
 +word, meta = hybrid.generate(max_length=10)
 +print(f"Word: {word}")
 +print(f"LSTM confidence: {meta['avg_lstm_confidence']:.2f}")
 +print(f"Markov influence: {meta['avg_markov_influence']:.2f}")
 +print(f"LSTM influence: {meta['avg_lstm_influence']:.2f}")
 +```
++
 +### Performance Metrics
++
 +```bash
 +# Generate 1000 words and analyze
 +python manage.py evaluate_hybrid --corpus scifi --samples 1000 --report
++
 +# Check pronounceability distribution
 +# Check diversity metrics
 +# Compare to pure Markov baseline
 +```
++
 +## Future Enhancements
++
 +### Option B: Train on Deployment (If Needed)
++
 +If corpora become dynamic (user-contributed words), switch to Option B:
++
 +**railway.json** change:
 +```json
 +{
 +  "deploy": {
 +    "startCommand": "python manage.py migrate && python manage.py load_corpora --verbosity=2 && python manage.py prebuild_markov_models && python manage.py train_hybrid_models --all --epochs 30 && gunicorn jubjub.wsgi:application --bind 0.0.0.0:$PORT"
 +  }
 +}
 +```
++
 +**Tradeoffs**:
 +- ✅ Always fresh models
 +- ❌ 10-15 minute deployment time
 +- ❌ Higher compute costs
++
 +### Git LFS (If Models Exceed 100MB)
++
 +If you add many more corpora:
++
 +```bash
 +# Install Git LFS
 +git lfs install
++
 +# Track model files
 +git lfs track "*.pt"
 +git lfs track "*.pkl"
++
 +# Update .gitattributes (already configured)
 +```
++
 +### Incremental Training
++
 +Future feature to update models without full retrain:
++
 +```python
 +# Add new words to existing model
 +from jubjub.jubjubword.hybrid_trainer import incremental_train
++
 +incremental_train(
 +    corpus_slug='scifi',
 +    new_words=['quantumflux', 'nanocore', 'cyberdeck'],
 +    epochs=10  # Fine-tune only
 +)
 +```
++
 +## Summary
++
 +**Current Setup (Option A)**:
 +- ✅ Pre-trained models committed to repo
 +- ✅ Fast Railway deployments (<1 minute)
 +- ✅ No training overhead in production
 +- ✅ ~500KB model size (acceptable)
++
 +**When corpora change**:
 +- Train locally: `python manage.py train_hybrid_models --corpus X`
 +- Commit models: `git add hybrid_models/X/ && git commit`
 +- Deploy: `git push` (Railway auto-deploys)
++
 +**Maintenance**:
 +- Retrain when corpus words change
 +- ~15 minutes total training time (infrequent)
 +- Models stay in sync with corpus content
++
 +This approach balances simplicity, performance, and maintainability for JubJub Word's current scale.

backend/jubjub/jubjubword/TRAINING_CHECKLIST.mdadded

 +# Hybrid Model Training Checklist
++
 +Use this checklist to verify hybrid model training is working correctly.
++
 +## Pre-Training Setup
++
 +### 1. Environment Setup
++
 +```bash
 +# Verify Python version (3.10+)
 +python --version
++
 +# Create virtual environment (if not exists)
 +python -m venv venv
 +source venv/bin/activate  # Linux/Mac
 +# or
 +venv\Scripts\activate  # Windows
++
 +# Install dependencies
 +cd backend
 +pip install -r requirements.txt
++
 +# Verify PyTorch installation
 +python -c "import torch; print(f'PyTorch {torch.__version__} installed')"
 +```
++
 +**Expected Output**:
 +```
 +Python 3.10.x or higher
 +PyTorch 2.4.1 installed
 +```
++
 +### 2. Database Setup
++
 +```bash
 +# Run migrations
 +python manage.py migrate
++
 +# Load corpora
 +python manage.py load_corpora --verbosity=2
++
 +# Verify corpora exist
 +python manage.py shell
 +>>> from jubjub.jubjubword.models import Corpus
 +>>> print(Corpus.objects.filter(is_active=True).count())
 +>>> # Should print: 5
 +>>> for c in Corpus.objects.filter(is_active=True):
 +...     print(f"{c.slug}: {len(c.get_words_list())} words")
 +>>> exit()
 +```
++
 +**Expected Output**:
 +```
 +5
 +scifi: 1609 words
 +fantasy: 1584 words
 +food: 1541 words
 +corporate: 1510 words
 +medical: 1566 words
 +```
++
 +### 3. Markov Models
++
 +```bash
 +# Prebuild Markov models (fast)
 +python manage.py prebuild_markov_models
++
 +# Verify Markov models work
 +python manage.py shell
 +>>> from jubjub.jubjubword.markov import get_markov_instance
 +>>> instance = get_markov_instance(corpus_slug='scifi', n=2, use_word_boundaries=True)
 +>>> words = instance.genny_batch(count=5)
 +>>> print(words)
 +>>> # Should print 5 sci-fi-ish words
 +>>> exit()
 +```
++
 +**Expected Output**:
 +```
 +Building models for 5 corpora...
 +✓ Built: scifi (n=2, wb=True)
 +✓ Built: fantasy (n=2, wb=True)
 +...
 +['quanticore', 'photonix', 'starforge', 'cyberdyne', 'neurotex']
 +```
++
 +## Training Tests
++
 +### 4. Single Corpus Training (Fast Test)
++
 +```bash
 +# Train one corpus with minimal settings (fast test)
 +python manage.py train_hybrid_models \
 +  --corpus scifi \
 +  --hidden-size 32 \
 +  --epochs 10 \
 +  --batch-size 16
 +```
++
 +**Expected Duration**: ~1-2 minutes
++
 +**Expected Output**:
 +```
 +🚀 Training hybrid models for 1 corpora
++
 +============================================================
 +Training: Science Fiction & Tech (scifi)
 +============================================================
++
 +Corpus size: 1609 words
++
 +📚 Training LSTM...
 +Building vocabulary...
 +Vocabulary size: 32
 +Train: 1448 words, Val: 161 words
 +Model parameters: 5,632
 +Estimated model size: 22.0 KB
++
 +Epoch 1/10 - Train Loss: 2.8456, Val Loss: 2.6123
 +Epoch 2/10 - Train Loss: 2.3145, Val Loss: 2.2456
 +...
++
 +✓ Training complete!
 +  Epochs trained: 10
 +  Best val loss: 1.8234
 +  Final train loss: 1.9012
++
 +🔗 Creating hybrid model...
 +✓ Hybrid model saved to .../hybrid_models/scifi
++
 +🎲 Sample generations:
 +  quanticore (LSTM confidence: 0.58)
 +  photonix (LSTM confidence: 0.62)
 +  ...
 +```
++
 +**Verification**:
 +```bash
 +# Check files were created
 +ls -lh jubjub/jubjubword/hybrid_models/scifi/
 +# Should see:
 +#   lstm_model.pt (~20-30KB for test)
 +#   vocabulary.json (~2KB)
 +#   hybrid_config.json (~200 bytes)
 +#   best_model.pt (training checkpoint)
 +#   training_history.json (loss curves)
 +```
++
 +### 5. Model Loading Test
++
 +```bash
 +python manage.py shell
 +```
++
 +```python
 +from jubjub.jubjubword.hybrid import HybridMarkovLSTM
 +from jubjub.jubjubword.markov import get_markov_instance
 +from pathlib import Path
++
 +# Load Markov
 +markov = get_markov_instance(corpus_slug='scifi', n=2, use_word_boundaries=True)
 +print(f"✓ Markov loaded: {len(markov.transitions)} states")
++
 +# Load hybrid
 +model_dir = Path('jubjub/jubjubword/hybrid_models/scifi')
 +hybrid = HybridMarkovLSTM.load(model_dir, markov)
 +print(f"✓ Hybrid loaded")
++
 +# Generate words
 +for i in range(10):
 +    word, meta = hybrid.generate(max_length=10, temperature=1.0)
 +    print(f"  {word} (confidence: {meta['avg_lstm_confidence']:.2f})")
++
 +print("✓ All tests passed!")
 +```
++
 +**Expected Output**:
 +```
 +✓ Markov loaded: 1234 states
 +✓ Hybrid loaded
 +  quanticore (confidence: 0.68)
 +  photonix (confidence: 0.72)
 +  ...
 +✓ All tests passed!
 +```
++
 +### 6. Full Training (Production Quality)
++
 +```bash
 +# Train all corpora with production settings
 +python manage.py train_hybrid_models --all
++
 +# Or train individual corpus with optimal settings
 +python manage.py train_hybrid_models \
 +  --corpus scifi \
 +  --hidden-size 64 \
 +  --num-layers 2 \
 +  --epochs 50 \
 +  --batch-size 32 \
 +  --learning-rate 0.001
 +```
++
 +**Expected Duration**: ~10-15 minutes for all 5 corpora
++
 +**File Sizes**:
 +```bash
 +ls -lh jubjub/jubjubword/hybrid_models/*/lstm_model.pt
++
 +# Expected sizes:
 +# scifi/lstm_model.pt     ~80KB
 +# fantasy/lstm_model.pt   ~80KB
 +# food/lstm_model.pt      ~80KB
 +# corporate/lstm_model.pt ~80KB
 +# medical/lstm_model.pt   ~80KB
 +```
++
 +### 7. Evaluation Test
++
 +```bash
 +# Compare hybrid vs pure Markov
 +python manage.py evaluate_hybrid --corpus scifi --samples 100
 +```
++
 +**Expected Output**:
 +```
 +Evaluating Hybrid vs Markov for scifi corpus
 +Generating 100 samples from each...
++
 +Pronounceability Scores:
 +  Markov:  0.72 ± 0.15
 +  Hybrid:  0.79 ± 0.12  (+9.7% improvement)
++
 +Diversity Metrics:
 +  Markov:  47 unique character patterns
 +  Hybrid:  56 unique character patterns  (+19.1% improvement)
++
 +LSTM Contribution Analysis:
 +  Avg LSTM confidence: 0.68
 +  Avg Markov influence: 0.52
 +  Avg LSTM influence:   0.48
++
 +Sample Comparisons:
 +  Markov: quanticore, photonix, starforge, ...
 +  Hybrid: quantumsphere, photonyx, starforged, ...
 +```
++
 +## Common Issues
++
 +### Issue: ModuleNotFoundError: No module named 'torch'
++
 +**Cause**: PyTorch not installed
++
 +**Fix**:
 +```bash
 +pip install torch==2.4.1 numpy==1.26.4 tqdm==4.66.1
 +```
++
 +### Issue: Corpus not found
++
 +**Cause**: Database not populated
++
 +**Fix**:
 +```bash
 +python manage.py load_corpora --verbosity=2
 +```
++
 +### Issue: Training very slow
++
 +**Possible Causes**:
 +1. Large batch size on CPU
 +2. Large model size
++
 +**Fix**:
 +```bash
 +# Reduce batch size
 +python manage.py train_hybrid_models --corpus scifi --batch-size 16
++
 +# Or reduce model size
 +python manage.py train_hybrid_models --corpus scifi --hidden-size 32
 +```
++
 +### Issue: Poor generation quality (worse than Markov)
++
 +**Possible Causes**:
 +1. Corpus too small (<500 words)
 +2. Overfitting (trained too long)
 +3. Wrong ensemble weights
++
 +**Fix**:
 +```bash
 +# Reduce epochs to prevent overfitting
 +python manage.py train_hybrid_models --corpus scifi --epochs 20
++
 +# Increase Markov weight
 +python manage.py train_hybrid_models --corpus scifi \
 +  --markov-weight 0.7 \
 +  --lstm-weight 0.3
 +```
++
 +### Issue: FileNotFoundError when loading hybrid
++
 +**Cause**: Models not trained or wrong path
++
 +**Fix**:
 +```bash
 +# Verify model exists
 +ls jubjub/jubjubword/hybrid_models/scifi/lstm_model.pt
++
 +# If missing, train
 +python manage.py train_hybrid_models --corpus scifi
 +```
++
 +## Success Criteria
++
 +### ✅ Training Complete When:
++
 +1. **All model files exist**:
 +   ```bash
 +   # Should have 3 files per corpus
 +   ls jubjub/jubjubword/hybrid_models/scifi/
 +   # lstm_model.pt, vocabulary.json, hybrid_config.json
 +   ```
++
 +2. **Models load without errors**:
 +   ```python
 +   hybrid = HybridMarkovLSTM.load(model_dir, markov)
 +   # No exceptions
 +   ```
++
 +3. **Generation works**:
 +   ```python
 +   word, meta = hybrid.generate(max_length=10)
 +   print(word)  # Real word, not gibberish
 +   ```
++
 +4. **Quality improved**:
 +   ```bash
 +   python manage.py evaluate_hybrid --corpus scifi --samples 100
 +   # Hybrid scores >= Markov scores
 +   ```
++
 +5. **File sizes reasonable**:
 +   - lstm_model.pt: 50-150KB per corpus
 +   - Total size: <1MB for all 5 corpora
++
 +## Deployment Checklist
++
 +### Before Committing Models:
++
 +- [ ] All 5 corpora trained
 +- [ ] Model files verified (<200KB each)
 +- [ ] Generation quality tested
 +- [ ] No training checkpoints included (best_model.pt ignored)
 +- [ ] No training logs included (training_history.json ignored)
++
 +### Commit Commands:
++
 +```bash
 +# Check what's being committed
 +git status jubjub/jubjubword/hybrid_models/
++
 +# Should show:
 +#   new file: scifi/lstm_model.pt
 +#   new file: scifi/vocabulary.json
 +#   new file: scifi/hybrid_config.json
 +#   (repeat for each corpus)
++
 +# Should NOT show:
 +#   best_model.pt (ignored)
 +#   training_history.json (ignored)
++
 +# Add models
 +git add jubjub/jubjubword/hybrid_models/
++
 +# Commit
 +git commit -m "feat: Add pre-trained hybrid models for all 5 corpora
++
 +- Trained with hidden_size=64, num_layers=2, epochs=50
 +- Models optimized for pronounceability and diversity
 +- Total size: ~500KB committed
 +- Deployment-ready (no training needed on Railway)"
++
 +# Push
 +git push origin claude/your-branch
 +```
++
 +### Post-Deployment Verification:
++
 +```bash
 +# After Railway deploys, test the API
 +curl https://your-app.railway.app/api/generate/scifi/
++
 +# Should return generated words (using hybrid if available)
 +```
++
 +## Performance Benchmarks
++
 +### Expected Training Times (CPU):
++
 +| Corpus    | Words | Hidden Size | Epochs | Time     |
 +|-----------|-------|-------------|--------|----------|
 +| scifi     | 1,609 | 64          | 50     | ~3 min   |
 +| fantasy   | 1,584 | 64          | 50     | ~3 min   |
 +| food      | 1,541 | 64          | 50     | ~3 min   |
 +| corporate | 1,510 | 64          | 50     | ~2.5 min |
 +| medical   | 1,566 | 64          | 50     | ~3 min   |
 +| **Total** |       |             |        | **15 min** |
++
 +### Model Sizes:
++
 +| File              | Size   | Description                    |
 +|-------------------|--------|--------------------------------|
 +| lstm_model.pt     | ~80KB  | Trained LSTM weights           |
 +| vocabulary.json   | ~2KB   | Character vocabulary           |
 +| hybrid_config.json| ~200B  | Ensemble configuration         |
 +| **Per Corpus**    | **~82KB** | **Total committed**        |
 +| **All 5 Corpora** | **~410KB** | **Total deployment size** |
++
 +### Generation Performance:
++
 +- **Pure Markov**: ~0.5-1ms per word
 +- **Hybrid LSTM**: ~5-10ms per word (10x slower, still fast)
 +- **Hybrid overhead**: Acceptable for quality improvement
++
 +## Next Steps After Training
++
 +1. **Commit models** (see commands above)
 +2. **Push to Railway** (auto-deploys)
 +3. **Monitor generation** (check pronounceability)
 +4. **Update corpus** (retrain when adding words)
 +5. **Evaluate periodically** (ensure quality maintained)
++
 +For detailed deployment instructions, see `DEPLOYMENT_HYBRID.md`.

backend/requirements.txtmodified

  phonemizer==3.2.1
  pydub==0.25.1
  psycopg2-binary==2.9.9
 -dj-database-url==2.1.0
 +dj-database-url==2.1.0
++
 +# Hybrid Markov-LSTM Models (inference only)
 +# Pre-trained models are committed to repo, so training is optional
 +# For local training, see: python manage.py train_hybrid_models --help
 +torch==2.4.1
 +numpy==1.26.4
 +tqdm==4.66.1