swadhin / opengrammar public

swadhinbiswas

[email protected]
2026-02-28 02:10:11 +0600 committed NaN years ago
f9cd753
commit f9cd753
1 file changed
+362 -0
Showing 1 changed file+362-0
opengrammar/SELF_HOSTING.md
▼opengrammar/SELF_HOSTING.md
+362-0
@@ -0,0 +1,362 @@
1
+# 🐳 OpenGrammar Self-Hosting Guide
2
+
3
+Run OpenGrammar locally with **local LLM support** (Qwen, Phi-4-mini, Llama) using Docker and Ollama.
4
+
5
+---
6
+
7
+## 🎯 Options Overview
8
+
9
+| Option | Best For | Cost | Privacy |
10
+|--------|----------|------|---------|
11
+| **Cloud APIs** (OpenAI, Groq, Together) | Production, best quality | $0.01-0.10/request | ⭐⭐⭐ |
12
+| **Local Ollama** (Qwen, Phi-4-mini) | Privacy, free, offline | Free (your hardware) | ⭐⭐⭐⭐⭐ |
13
+| **Hybrid** (Rules local + AI cloud) | Balance | Free-$5/mo | ⭐⭐⭐⭐ |
14
+
15
+---
16
+
17
+## 🚀 Quick Start
18
+
19
+### Option 1: Docker Compose (Recommended)
20
+
21
+```bash
22
+cd opengrammar
23
+
24
+# Start backend only (use cloud APIs)
25
+docker-compose up -d opengrammar-backend
26
+
27
+# Start backend + Ollama with local models (requires GPU for good performance)
28
+docker-compose --profile local-llm up -d
29
+
30
+# Check status
31
+docker-compose ps
32
+
33
+# View logs
34
+docker-compose logs -f
35
+```
36
+
37
+### Option 2: Manual Docker
38
+
39
+```bash
40
+# Build backend image
41
+cd opengrammar/backend
42
+docker build -t opengrammar-backend .
43
+
44
+# Run container
45
+docker run -d \
46
+ -p 8787:8787 \
47
+ --name opengrammar \
48
+ opengrammar-backend
49
+```
50
+
51
+### Option 3: Direct Node.js
52
+
53
+```bash
54
+cd opengrammar/backend
55
+npm install
56
+npm run dev # Runs on http://localhost:8787
57
+```
58
+
59
+---
60
+
61
+## 🤖 Local LLM Setup with Ollama
62
+
63
+### Step 1: Install Ollama
64
+
65
+**Linux:**
66
+```bash
67
+curl -fsSL https://ollama.com/install.sh | sh
68
+```
69
+
70
+**macOS:**
71
+```bash
72
+brew install ollama
73
+```
74
+
75
+**Windows:**
76
+Download from [ollama.com](https://ollama.com)
77
+
78
+### Step 2: Pull Grammar-Focused Models
79
+
80
+```bash
81
+# Small & fast (good for grammar checking)
82
+ollama pull qwen2.5:0.5b # 0.5B parameters, ~400MB
83
+ollama pull qwen2.5:1.5b # 1.5B parameters, ~1GB
84
+
85
+# Medium quality
86
+ollama pull phi4-mini:3.8b # 3.8B parameters, ~2.5GB
87
+ollama pull llama3.2:3b # 3B parameters, ~2GB
88
+
89
+# High quality (requires GPU)
90
+ollama pull llama3.2:7b # 7B parameters, ~4GB
91
+ollama pull mistral:7b # 7B parameters, ~4GB
92
+```
93
+
94
+### Step 3: Test Ollama
95
+
96
+```bash
97
+# Test the model
98
+ollama run qwen2.5:0.5b "Fix this: me and him went to store"
99
+
100
+# Should respond with corrected grammar
101
+```
102
+
103
+### Step 4: Configure OpenGrammar
104
+
105
+In extension options:
106
+- **Provider:** Ollama (Local)
107
+- **Base URL:** `http://localhost:11434/v1`
108
+- **Model:** `qwen2.5:0.5b` or `qwen2.5:1.5b`
109
+- **API Key:** (leave empty - not needed)
110
+
111
+---
112
+
113
+## ☁️ Cloud API Setup
114
+
115
+### Supported Providers
116
+
117
+| Provider | Speed | Quality | Cost | Best Models |
118
+|----------|-------|---------|------|-------------|
119
+| **Groq** | ⚡⚡⚡ | ⭐⭐⭐⭐ | Free tier | `llama-3.1-70b-versatile` |
120
+| **Together** | ⚡⚡ | ⭐⭐⭐⭐ | $0.01/1K | `Meta-Llama-3.1-70B` |
121
+| **OpenRouter** | ⚡⚡ | ⭐⭐⭐⭐⭐ | Varies | `claude-3.5-sonnet` |
122
+| **OpenAI** | ⚡ | ⭐⭐⭐⭐⭐ | $0.015/1K | `gpt-4o-mini` |
123
+
124
+### Get API Keys
125
+
126
+1. **Groq:** [console.groq.com](https://console.groq.com) (Free tier: 100 req/day)
127
+2. **Together:** [api.together.ai](https://api.together.ai) ($25 free credit)
128
+3. **OpenRouter:** [openrouter.ai](https://openrouter.ai) (Pay-as-you-go)
129
+4. **OpenAI:** [platform.openai.com](https://platform.openai.com) ($5 free credit)
130
+
131
+### Configure in Extension
132
+
133
+1. Click extension icon → Settings
134
+2. Select provider
135
+3. Enter API key
136
+4. Choose model
137
+5. Test connection
138
+
139
+---
140
+
141
+## 🔧 Configuration
142
+
143
+### Environment Variables (Backend)
144
+
145
+```bash
146
+# .env file in backend/
147
+PORT=8787
148
+NODE_ENV=production
149
+
150
+# Optional: Default API keys (users can override in extension)
151
+OPENAI_API_KEY=sk-...
152
+GROQ_API_KEY=gsk_...
153
+TOGETHER_API_KEY=...
154
+OPENROUTER_API_KEY=...
155
+
156
+# Ollama URL (for local LLM)
157
+OLLAMA_BASE_URL=http://localhost:11434/v1
158
+```
159
+
160
+### Recommended Models by Use Case
161
+
162
+| Use Case | Model | Provider | RAM Required |
163
+|----------|-------|----------|--------------|
164
+| **Basic grammar** | qwen2.5:0.5b | Ollama | 1GB |
165
+| **Better accuracy** | qwen2.5:1.5b | Ollama | 2GB |
166
+| **Professional writing** | gpt-4o-mini | OpenAI | N/A |
167
+| **Fast + free** | llama-3.1-70b | Groq | N/A |
168
+| **Best quality** | claude-3.5-sonnet | OpenRouter | N/A |
169
+
170
+---
171
+
172
+## 📊 Performance Benchmarks
173
+
174
+### Local LLM (RTX 4090)
175
+
176
+| Model | Tokens/sec | Grammar Accuracy | RAM |
177
+|-------|------------|------------------|-----|
178
+| qwen2.5:0.5b | ~100 | 75% | 1GB |
179
+| qwen2.5:1.5b | ~80 | 82% | 2GB |
180
+| phi4-mini:3.8b | ~50 | 88% | 4GB |
181
+| llama3.2:7b | ~30 | 90% | 8GB |
182
+
183
+### Cloud APIs
184
+
185
+| Model | Latency | Grammar Accuracy | Cost/1K requests |
186
+|-------|---------|------------------|------------------|
187
+| gpt-4o-mini | ~1s | 95% | $0.15 |
188
+| llama-3.1-70b (Groq) | ~200ms | 92% | Free (tier) |
189
+| claude-3.5-sonnet | ~2s | 96% | $3.00 |
190
+
191
+---
192
+
193
+## 🔒 Security Considerations
194
+
195
+### Local Deployment
196
+- ✅ All data stays on your machine
197
+- ✅ No API calls
198
+- ✅ Full privacy
199
+- ⚠️ Your hardware cost
200
+
201
+### Cloud APIs
202
+- ✅ Text sent only to chosen provider
203
+- ✅ No logging by OpenGrammar
204
+- ⚠️ API key required
205
+- ⚠️ Data leaves your machine
206
+
207
+### Best Practices
208
+1. Use API keys with spending limits
209
+2. Rotate keys periodically
210
+3. Never commit keys to git
211
+4. Use HTTPS in production
212
+
213
+---
214
+
215
+## 🛠️ Troubleshooting
216
+
217
+### Backend Won't Start
218
+
219
+```bash
220
+# Check logs
221
+docker-compose logs opengrammar-backend
222
+
223
+# Test health endpoint
224
+curl http://localhost:8787/health
225
+
226
+# Check port conflict
227
+lsof -i :8787
228
+```
229
+
230
+### Ollama Connection Failed
231
+
232
+```bash
233
+# Check Ollama is running
234
+ollama list
235
+
236
+# Test Ollama directly
237
+curl http://localhost:11434/api/tags
238
+
239
+# Restart Ollama
240
+sudo systemctl restart ollama # Linux
241
+brew services restart ollama # macOS
242
+```
243
+
244
+### Docker GPU Support (NVIDIA)
245
+
246
+```bash
247
+# Install NVIDIA Container Toolkit
248
+sudo apt-get install -y nvidia-container-toolkit
249
+
250
+# Configure Docker
251
+sudo nvidia-ctk runtime configure --runtime=docker
252
+sudo systemctl restart docker
253
+
254
+# Run with GPU
255
+docker-compose --profile local-llm up -d
256
+```
257
+
258
+### Models Not Downloading
259
+
260
+```bash
261
+# Check internet connection
262
+ping ollama.com
263
+
264
+# Try alternative mirror
265
+OLLAMA_ORIGINS=https://ollama.ollama.com ollama pull qwen2.5:0.5b
266
+
267
+# Manual download
268
+wget https://ollama.com/download/qwen2.5:0.5b
269
+```
270
+
271
+---
272
+
273
+## 📈 Scaling
274
+
275
+### Production Deployment
276
+
277
+```bash
278
+# Deploy to cloud (Railway, Render, Fly.io)
279
+railway up # Auto-deploys from GitHub
280
+
281
+# Or use Kubernetes
282
+kubectl apply -f k8s/
283
+
284
+# Load balancer setup
285
+nginx -c nginx.conf
286
+```
287
+
288
+### High Availability
289
+
290
+- Run multiple backend instances
291
+- Use Redis for caching
292
+- Enable request queuing
293
+- Monitor with Prometheus + Grafana
294
+
295
+---
296
+
297
+## 💡 Tips & Tricks
298
+
299
+### 1. Hybrid Mode (Best of Both Worlds)
300
+
301
+Configure rules to run locally, AI only for complex issues:
302
+
303
+```json
304
+{
305
+ "provider": "ollama",
306
+ "model": "qwen2.5:0.5b",
307
+ "fallbackProvider": "groq",
308
+ "fallbackModel": "llama-3.1-70b-versatile"
309
+}
310
+```
311
+
312
+### 2. Custom System Prompt
313
+
314
+Tailor AI behavior for your needs:
315
+
316
+```
317
+You are a grammar expert. Be concise. Only suggest changes that improve clarity.
318
+```
319
+
320
+### 3. Batch Processing
321
+
322
+For long documents, split into chunks:
323
+
324
+```javascript
325
+const chunks = text.split(/\n\n/);
326
+const results = await Promise.all(
327
+ chunks.map(chunk => analyze(chunk))
328
+);
329
+```
330
+
331
+### 4. Model Comparison
332
+
333
+Test multiple models on same text:
334
+
335
+```bash
336
+curl http://localhost:8787/analyze \
337
+ -d '{"text": "...", "provider": "ollama", "model": "qwen2.5:0.5b"}'
338
+
339
+curl http://localhost:8787/analyze \
340
+ -d '{"text": "...", "provider": "groq", "model": "llama-3.1-70b"}'
341
+```
342
+
343
+---
344
+
345
+## 🎓 Learning Resources
346
+
347
+- [Ollama Documentation](https://ollama.com/docs)
348
+- [Cloudflare Workers](https://developers.cloudflare.com/workers/)
349
+- [Hono Framework](https://hono.dev/docs)
350
+- [OpenAI API Reference](https://platform.openai.com/docs)
351
+
352
+---
353
+
354
+## 🤝 Support
355
+
356
+- **Issues:** [GitHub Issues](https://github.com/your-repo/opengrammar/issues)
357
+- **Discussions:** [GitHub Discussions](https://github.com/your-repo/opengrammar/discussions)
358
+- **Discord:** [Join our server](https://discord.gg/your-server)
359
+
360
+---
361
+
362
+**Happy self-hosting! 🎉**
0
363