Add Fox Desktop implementation plan

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
main
boksbc 2026-05-11 21:48:22 +02:00
parent c6ddbc3dc9
commit 6cf9cac79b
1 changed files with 859 additions and 0 deletions

View File

@ -0,0 +1,859 @@
# Fox Desktop Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Electron-App als native Mac-Hülle, die `https://fox.pxo.at/hud` in einem transparenten Fullscreen-Overlay lädt und native Mac-Fähigkeiten hinzufügt (Tray, Wakeword, AppleScript, LaunchAgent).
**Architecture:** Electron lädt `fox.pxo.at/hud` als remote URL in einem rahmenlosen transparenten BrowserWindow. `preload.js` injiziert `window.foxBridge` in die HUD-Seite für native Mac-Befehle via IPC. Ein Python-Subprocess mit OpenWakeWord erkennt "hey jarvis" und löst `showHud()` aus. Kein eigener WebSocket in Electron — der Reverb-Kanal läuft komplett in der HUD-Seite.
**Tech Stack:** Electron 28, Node.js (child_process, IPC), Jest 29, Python 3 + openwakeword + pyaudio + onnxruntime
**Spec:** `docs/superpowers/specs/2026-05-11-fox-desktop-design.md`
---
## Dateien
| Datei | Aufgabe |
|---|---|
| `package.json` | App-Konfig, Scripts, Dependencies |
| `.gitignore` | node_modules, dist etc. |
| `electron/main.js` | App-Lifecycle, IPC-Handler, globaler Shortcut |
| `electron/hud.js` | BrowserWindow (remote URL, transparent, always-on-top) |
| `electron/tray.js` | Menu Bar Icon + Kontextmenü |
| `electron/wakeword.js` | Python-Subprocess starten/auto-restarten |
| `electron/mac-agent.js` | AppleScript-Dispatch (resolveCommand + executeMacCommand) |
| `electron/preload.js` | `window.foxBridge` in HUD-Seite injizieren |
| `python/wakeword.py` | OpenWakeWord Loop, schreibt "WAKE" auf stdout |
| `assets/fox-tray.png` | 16×16 Platzhalter-Icon |
| `__tests__/mac-agent.test.js` | Unit-Tests für mac-agent.js |
| `launchagent/at.pxo.fox.plist` | LaunchAgent-Vorlage für Auto-Start beim Login |
| `docs/hud-bridge-integration.md` | Anleitung für Laravel HUD Anpassungen |
---
### Task 1: Projekt-Scaffolding
**Files:**
- Create: `package.json`
- Create: `.gitignore`
- [ ] **Step 1: package.json erstellen**
`package.json`:
```json
{
"name": "fox-desktop",
"version": "1.0.0",
"description": "Fox — Persönlicher KI-Assistent",
"main": "electron/main.js",
"scripts": {
"start": "electron .",
"test": "jest"
},
"dependencies": {
"axios": "^1.6.0"
},
"devDependencies": {
"electron": "^28.0.0",
"jest": "^29.0.0"
},
"jest": {
"testEnvironment": "node",
"testMatch": ["**/__tests__/**/*.test.js"]
}
}
```
- [ ] **Step 2: .gitignore erstellen**
`.gitignore`:
```
node_modules/
dist/
*.log
.env
```
- [ ] **Step 3: Verzeichnisse anlegen**
```bash
mkdir -p electron python assets __tests__ launchagent docs
```
Expected: kein Fehler.
- [ ] **Step 4: Dependencies installieren**
```bash
npm install
```
Expected: `node_modules/` wird erstellt, keine Fehler.
- [ ] **Step 5: Commit**
```bash
git add package.json package-lock.json .gitignore
git commit -m "feat: project scaffolding"
```
---
### Task 2: Platzhalter-Icon
**Files:**
- Create: `assets/fox-tray.png`
- [ ] **Step 1: 16×16 PNG via Python generieren**
```bash
python3 -c "
import struct, zlib
def mkpng(w, h, rgb):
def chunk(t, d):
c = struct.pack('>I', len(d)) + t + d
return c + struct.pack('>I', zlib.crc32(t + d) & 0xffffffff)
ihdr = struct.pack('>IIBBBBB', w, h, 8, 2, 0, 0, 0)
row = b'\x00' + bytes(rgb) * w
idat = zlib.compress(row * h)
return b'\x89PNG\r\n\x1a\n' + chunk(b'IHDR', ihdr) + chunk(b'IDAT', idat) + chunk(b'IEND', b'')
open('assets/fox-tray.png', 'wb').write(mkpng(16, 16, [255, 165, 0]))
print('Icon erstellt')
"
```
Expected: `Icon erstellt``assets/fox-tray.png` existiert.
- [ ] **Step 2: Commit**
```bash
git add assets/fox-tray.png
git commit -m "feat: add placeholder tray icon"
```
---
### Task 3: mac-agent.js — TDD
**Files:**
- Create: `__tests__/mac-agent.test.js`
- Create: `electron/mac-agent.js`
- [ ] **Step 1: Fehlschlagenden Test schreiben**
`__tests__/mac-agent.test.js`:
```javascript
const { resolveCommand } = require('../electron/mac-agent');
describe('resolveCommand', () => {
test('bekannter Shortcut → applescript', () => {
expect(resolveCommand('spotify_play')).toEqual({
type: 'applescript',
script: 'tell application "Spotify" to play',
});
});
test('script: Präfix → applescript mit direktem Script', () => {
expect(resolveCommand('script:tell application "Finder" to activate')).toEqual({
type: 'applescript',
script: 'tell application "Finder" to activate',
});
});
test('open: Präfix → URL öffnen', () => {
expect(resolveCommand('open:https://example.com')).toEqual({
type: 'open',
url: 'https://example.com',
});
});
test('app: Präfix → App öffnen', () => {
expect(resolveCommand('app:Spotify')).toEqual({
type: 'app',
name: 'Spotify',
});
});
test('save: Präfix → Datei auf Desktop speichern', () => {
expect(resolveCommand('save:{"filename":"test.txt","content":"hallo"}')).toEqual({
type: 'save',
filename: 'test.txt',
content: 'hallo',
});
});
test('unbekannter Befehl → unknown', () => {
expect(resolveCommand('gibts_nicht')).toEqual({ type: 'unknown' });
});
test('volume_up ist bekannter Shortcut', () => {
const result = resolveCommand('volume_up');
expect(result.type).toBe('applescript');
expect(result.script).toContain('output volume');
});
});
```
- [ ] **Step 2: Test ausführen — erwartet FAIL**
```bash
npm test
```
Expected: `Cannot find module '../electron/mac-agent'`
- [ ] **Step 3: mac-agent.js implementieren**
`electron/mac-agent.js`:
```javascript
const { exec, execSync } = require('child_process');
const util = require('util');
const fs = require('fs');
const execAsync = util.promisify(exec);
const SCRIPTS = {
spotify_play: 'tell application "Spotify" to play',
spotify_pause: 'tell application "Spotify" to pause',
spotify_next: 'tell application "Spotify" to next track',
spotify_prev: 'tell application "Spotify" to previous track',
music_play: 'tell application "Music" to play',
music_pause: 'tell application "Music" to pause',
music_next: 'tell application "Music" to next track',
mail_open: 'tell application "Mail" to activate',
mail_unread: 'tell application "Mail" to return (count of (messages of inbox whose read status is false))',
calendar_open: 'tell application "Calendar" to activate',
notes_open: 'tell application "Notes" to activate',
volume_up: 'set volume output volume (output volume of (get volume settings) + 10)',
volume_down: 'set volume output volume (output volume of (get volume settings) - 10)',
volume_mute: 'set volume output muted true',
volume_unmute: 'set volume output muted false',
dark_mode_on: 'tell app "System Events" to tell appearance preferences to set dark mode to true',
dark_mode_off: 'tell app "System Events" to tell appearance preferences to set dark mode to false',
screenshot: 'do shell script "screencapture -x ~/Desktop/fox-screenshot-$(date +%Y%m%d-%H%M%S).png"',
};
function resolveCommand(command) {
if (SCRIPTS[command]) {
return { type: 'applescript', script: SCRIPTS[command] };
}
if (command.startsWith('script:')) {
return { type: 'applescript', script: command.slice(7) };
}
if (command.startsWith('open:')) {
return { type: 'open', url: command.slice(5) };
}
if (command.startsWith('app:')) {
return { type: 'app', name: command.slice(4) };
}
if (command.startsWith('save:')) {
return { type: 'save', ...JSON.parse(command.slice(5)) };
}
return { type: 'unknown' };
}
async function executeMacCommand(command) {
try {
const resolved = resolveCommand(command);
if (resolved.type === 'applescript') {
const { stdout } = await execAsync(`osascript -e '${resolved.script}'`);
return { success: true, output: stdout.trim() };
}
if (resolved.type === 'open') {
execSync(`open "${resolved.url}"`);
return { success: true, output: 'Geöffnet' };
}
if (resolved.type === 'app') {
execSync(`open -a "${resolved.name}"`);
return { success: true, output: `${resolved.name} geöffnet` };
}
if (resolved.type === 'save') {
const filePath = `${process.env.HOME}/Desktop/${resolved.filename}`;
fs.writeFileSync(filePath, resolved.content, 'utf8');
return { success: true, output: `Gespeichert: ${filePath}` };
}
return { success: false, output: 'Unbekannter Befehl' };
} catch (err) {
return { success: false, output: err.message };
}
}
module.exports = { executeMacCommand, resolveCommand, SCRIPTS };
```
- [ ] **Step 4: Tests ausführen — erwartet PASS**
```bash
npm test
```
Expected:
```
PASS __tests__/mac-agent.test.js
resolveCommand
✓ bekannter Shortcut → applescript
✓ script: Präfix → applescript mit direktem Script
✓ open: Präfix → URL öffnen
✓ app: Präfix → App öffnen
✓ save: Präfix → Datei auf Desktop speichern
✓ unbekannter Befehl → unknown
✓ volume_up ist bekannter Shortcut
Tests: 7 passed, 7 total
```
- [ ] **Step 5: Commit**
```bash
git add electron/mac-agent.js __tests__/mac-agent.test.js
git commit -m "feat: mac-agent with AppleScript dispatch and unit tests"
```
---
### Task 4: preload.js — foxBridge
**Files:**
- Create: `electron/preload.js`
- [ ] **Step 1: preload.js erstellen**
`electron/preload.js`:
```javascript
const { contextBridge, ipcRenderer } = require('electron');
contextBridge.exposeInMainWorld('foxBridge', {
sendMessage: (text) => ipcRenderer.send('send-message', text),
hideHud: () => ipcRenderer.send('hide-hud'),
macCommand: (cmd) => ipcRenderer.invoke('mac-command', cmd),
onFoxAction: (cb) => ipcRenderer.on('fox-action', (_, data) => cb(data)),
onHudShow: (cb) => ipcRenderer.on('hud-show', () => cb()),
onHudHide: (cb) => ipcRenderer.on('hud-hide', () => cb()),
onStartListening: (cb) => ipcRenderer.on('start-listening', () => cb()),
onPlayAudio: (cb) => ipcRenderer.on('play-audio', (_, url) => cb(url)),
});
```
- [ ] **Step 2: Commit**
```bash
git add electron/preload.js
git commit -m "feat: preload — injects window.foxBridge into remote HUD"
```
---
### Task 5: hud.js — BrowserWindow
**Files:**
- Create: `electron/hud.js`
- [ ] **Step 1: hud.js erstellen**
`electron/hud.js`:
```javascript
const { BrowserWindow, screen } = require('electron');
const path = require('path');
const HUD_URL = process.env.FOX_HUD_URL || 'https://fox.pxo.at/hud';
let hudWindow = null;
function createHud() {
const { width, height } = screen.getPrimaryDisplay().workAreaSize;
hudWindow = new BrowserWindow({
width,
height,
x: 0,
y: 0,
frame: false,
transparent: true,
alwaysOnTop: true,
show: false,
skipTaskbar: true,
hasShadow: false,
webPreferences: {
preload: path.join(__dirname, 'preload.js'),
contextIsolation: true,
nodeIntegration: false,
webSecurity: false,
allowRunningInsecureContent: true,
},
});
hudWindow.setAlwaysOnTop(true, 'screen-saver');
hudWindow.setVisibleOnAllWorkspaces(true, { visibleOnFullScreen: true });
hudWindow.loadURL(HUD_URL);
hudWindow.on('closed', () => { hudWindow = null; });
return hudWindow;
}
function showHud() {
if (!hudWindow) createHud();
hudWindow.show();
hudWindow.focus();
hudWindow.webContents.send('hud-show');
}
function hideHud() {
if (!hudWindow) return;
hudWindow.webContents.send('hud-hide');
setTimeout(() => {
if (hudWindow) hudWindow.hide();
}, 600);
}
module.exports = { createHud, showHud, hideHud };
```
- [ ] **Step 2: Commit**
```bash
git add electron/hud.js
git commit -m "feat: HUD BrowserWindow — loads fox.pxo.at/hud with preload injection"
```
---
### Task 6: tray.js — Menu Bar Icon
**Files:**
- Create: `electron/tray.js`
- [ ] **Step 1: tray.js erstellen**
`electron/tray.js`:
```javascript
const { Tray, Menu, nativeImage } = require('electron');
const path = require('path');
let tray = null;
function createTray({ onShow, onHide, onQuit }) {
const iconPath = path.join(__dirname, '../assets/fox-tray.png');
const icon = nativeImage.createFromPath(iconPath);
icon.setTemplateImage(true);
tray = new Tray(icon);
tray.setToolTip('Fox');
tray.setContextMenu(Menu.buildFromTemplate([
{ label: 'Fox öffnen', accelerator: 'Cmd+Shift+F', click: onShow },
{ label: 'Fox verstecken', click: onHide },
{ type: 'separator' },
{ label: 'Beenden', click: onQuit },
]));
tray.on('click', onShow);
return tray;
}
module.exports = { createTray };
```
- [ ] **Step 2: Commit**
```bash
git add electron/tray.js
git commit -m "feat: tray — menu bar icon with context menu"
```
---
### Task 7: Wakeword — Python + Electron Subprocess
**Files:**
- Create: `python/wakeword.py`
- Create: `electron/wakeword.js`
- [ ] **Step 1: Python-Abhängigkeiten installieren**
```bash
pip install openwakeword pyaudio numpy onnxruntime
```
Expected: alle Pakete installiert ohne Fehler. Falls `pyaudio` auf macOS fehlschlägt:
```bash
brew install portaudio && pip install pyaudio
```
- [ ] **Step 2: python/wakeword.py erstellen**
`python/wakeword.py`:
```python
#!/usr/bin/env python3
import sys
import numpy as np
import pyaudio
from openwakeword.model import Model
CHUNK = 1280 # 80ms bei 16kHz — von openwakeword erwartet
RATE = 16000
THRESHOLD = 0.5
def main():
model = Model(wakeword_models=["hey_jarvis"])
pa = pyaudio.PyAudio()
stream = pa.open(
rate=RATE,
channels=1,
format=pyaudio.paInt16,
input=True,
frames_per_buffer=CHUNK,
)
print("Wakeword aktiv", flush=True)
try:
while True:
data = stream.read(CHUNK, exception_on_overflow=False)
audio = np.frombuffer(data, dtype=np.int16)
prediction = model.predict(audio)
for mdl, score in prediction.items():
if score > THRESHOLD:
print("WAKE", flush=True)
model.reset()
break
except KeyboardInterrupt:
pass
finally:
stream.stop_stream()
stream.close()
pa.terminate()
if __name__ == "__main__":
main()
```
- [ ] **Step 3: Python-Script manuell testen**
```bash
python3 python/wakeword.py
```
Expected: `Wakeword aktiv` — dann "Hey Jarvis" sagen.
Expected: `WAKE` erscheint in der Ausgabe.
Mit `Ctrl+C` beenden.
- [ ] **Step 4: electron/wakeword.js erstellen**
`electron/wakeword.js`:
```javascript
const { spawn } = require('child_process');
const path = require('path');
let wakewordProcess = null;
let running = false;
function startWakeword({ onWake }) {
if (running) return;
running = true;
const scriptPath = path.join(__dirname, '../python/wakeword.py');
wakewordProcess = spawn('python3', [scriptPath], {
stdio: ['pipe', 'pipe', 'pipe'],
});
wakewordProcess.stdout.on('data', (data) => {
if (data.toString().trim() === 'WAKE') {
onWake();
}
});
wakewordProcess.stderr.on('data', (data) => {
console.error('[Wakeword]', data.toString().trim());
});
wakewordProcess.on('close', (code) => {
running = false;
wakewordProcess = null;
if (code !== 0) {
setTimeout(() => startWakeword({ onWake }), 2000);
}
});
}
function stopWakeword() {
if (wakewordProcess) {
wakewordProcess.kill();
wakewordProcess = null;
running = false;
}
}
module.exports = { startWakeword, stopWakeword };
```
- [ ] **Step 5: Commit**
```bash
git add python/wakeword.py electron/wakeword.js
git commit -m "feat: wakeword — OpenWakeWord subprocess with hey_jarvis model"
```
---
### Task 8: main.js — Alles verkabeln
**Files:**
- Create: `electron/main.js`
- [ ] **Step 1: main.js erstellen**
`electron/main.js`:
```javascript
const { app, ipcMain, globalShortcut, nativeTheme } = require('electron');
const { createTray } = require('./tray');
const { createHud, showHud, hideHud } = require('./hud');
const { startWakeword, stopWakeword } = require('./wakeword');
const { executeMacCommand } = require('./mac-agent');
const axios = require('axios');
const FOX_URL = process.env.FOX_URL || 'https://fox.pxo.at';
app.dock?.hide();
if (!app.requestSingleInstanceLock()) {
app.quit();
}
app.whenReady().then(() => {
nativeTheme.themeSource = 'dark';
createHud();
createTray({
onShow: showHud,
onHide: hideHud,
onQuit: () => app.quit(),
});
globalShortcut.register('CommandOrControl+Shift+F', showHud);
startWakeword({ onWake: showHud });
ipcMain.on('hide-hud', () => hideHud());
ipcMain.on('send-message', async (_, text) => {
try {
await axios.post(`${FOX_URL}/fox/message`, { text });
} catch (err) {
console.error('[Fox] send-message fehlgeschlagen:', err.message);
}
});
ipcMain.handle('mac-command', async (_, command) => {
return await executeMacCommand(command);
});
});
app.on('will-quit', () => {
globalShortcut.unregisterAll();
stopWakeword();
});
app.on('window-all-closed', (e) => {
e.preventDefault();
});
```
- [ ] **Step 2: App erstmalig starten**
```bash
npm start
```
Expected:
- Kein Dock-Icon
- Oranges Icon in der Menu Bar
- Terminal zeigt `[Wakeword] Wakeword aktiv`
- [ ] **Step 3: Tray-Klick testen**
Auf das Tray-Icon klicken.
Expected: BrowserWindow öffnet sich und lädt `https://fox.pxo.at/hud`.
- [ ] **Step 4: Tastaturkürzel testen**
`Cmd+Shift+F` drücken (HUD muss vorher geschlossen sein).
Expected: HUD öffnet sich.
- [ ] **Step 5: Wakeword testen**
"Hey Jarvis" sagen.
Expected: HUD öffnet sich automatisch, Terminal zeigt nichts Ungewöhnliches.
- [ ] **Step 6: Beenden via Tray-Menü testen**
Rechtsklick auf Tray → "Beenden".
Expected: App beendet sich sauber, kein hängender Prozess.
- [ ] **Step 7: Commit**
```bash
git add electron/main.js
git commit -m "feat: main.js — wires tray, HUD, wakeword, shortcuts and IPC"
```
---
### Task 9: LaunchAgent
**Files:**
- Create: `launchagent/at.pxo.fox.plist`
- [ ] **Step 1: plist-Vorlage erstellen**
`launchagent/at.pxo.fox.plist`:
```xml
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN"
"http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>Label</key>
<string>at.pxo.fox</string>
<key>ProgramArguments</key>
<array>
<string>/Applications/Fox.app/Contents/MacOS/Fox</string>
</array>
<key>RunAtLoad</key>
<true/>
<key>KeepAlive</key>
<true/>
<key>StandardOutPath</key>
<string>/tmp/fox-desktop.log</string>
<key>StandardErrorPath</key>
<string>/tmp/fox-desktop.log</string>
<key>EnvironmentVariables</key>
<dict>
<key>FOX_URL</key>
<string>https://fox.pxo.at</string>
</dict>
</dict>
</plist>
```
Aktivieren (erst nach `electron-builder`-Build und Kopieren nach `/Applications`):
```bash
cp launchagent/at.pxo.fox.plist ~/Library/LaunchAgents/
launchctl load ~/Library/LaunchAgents/at.pxo.fox.plist
```
- [ ] **Step 2: Commit**
```bash
git add launchagent/at.pxo.fox.plist
git commit -m "feat: LaunchAgent plist for auto-start on login"
```
---
### Task 10: Laravel HUD Anpassungen dokumentieren
**Files:**
- Create: `docs/hud-bridge-integration.md`
- [ ] **Step 1: Integrations-Dokument erstellen**
`docs/hud-bridge-integration.md`:
```markdown
# Fox HUD — Electron Bridge Integration
Änderungen in `~/fox` (Laravel App). Alle Punkte sind **optional**
ohne `window.foxBridge` (normaler Browser) läuft das HUD unverändert weiter.
## 1. ESC → HUD schließen
In `fox-hud.blade.php` oder dem HUD-JavaScript:
```javascript
document.addEventListener('keydown', (e) => {
if (e.key === 'Escape') window.foxBridge?.hideHud();
});
```
## 2. Mac-Aktionen in onAction()
Im bestehenden `onAction()` Handler am Anfang ergänzen:
```javascript
if (payload.action === 'spotify_play') { await window.foxBridge?.macCommand('spotify_play'); return; }
if (payload.action === 'spotify_pause') { await window.foxBridge?.macCommand('spotify_pause'); return; }
if (payload.action === 'spotify_next') { await window.foxBridge?.macCommand('spotify_next'); return; }
if (payload.action === 'spotify_prev') { await window.foxBridge?.macCommand('spotify_prev'); return; }
if (payload.action === 'mail_open') { await window.foxBridge?.macCommand('mail_open'); return; }
if (payload.action === 'calendar_open') { await window.foxBridge?.macCommand('calendar_open'); return; }
if (payload.action === 'notes_open') { await window.foxBridge?.macCommand('notes_open'); return; }
if (payload.action === 'volume_up') { await window.foxBridge?.macCommand('volume_up'); return; }
if (payload.action === 'volume_down') { await window.foxBridge?.macCommand('volume_down'); return; }
if (payload.action === 'screenshot') { await window.foxBridge?.macCommand('screenshot'); return; }
if (payload.action === 'open_url' && payload.url) {
await window.foxBridge?.macCommand('open:' + payload.url);
return;
}
if (payload.action === 'open_app' && payload.app) {
await window.foxBridge?.macCommand('app:' + payload.app);
return;
}
if (payload.action === 'save_desktop' && payload.filename) {
await window.foxBridge?.macCommand('save:' + JSON.stringify({
filename: payload.filename,
content: payload.content || '',
}));
return;
}
```
## 3. Nachrichten senden (optional)
Wenn Nachrichten direkt über Electron statt Livewire gesendet werden sollen:
```javascript
window.foxBridge?.sendMessage(text);
```
## 4. Laravel Route für /fox/message
In `routes/api.php` oder `routes/web.php`:
```php
Route::post('/fox/message', function (\Illuminate\Http\Request $request) {
$text = $request->input('text', '');
if (empty(trim($text))) {
return response()->json(['error' => 'Kein Text'], 400);
}
$userMessage = app(\App\Services\FoxAgent::class)
->recordUserMessage($text, ['source' => 'electron']);
dispatch(fn() => app(\App\Services\FoxAgent::class)
->processUserMessage($userMessage));
return response()->json(['status' => 'ok']);
})->middleware('api');
```
## 5. Electron erkennen
```javascript
const isElectron = typeof window.foxBridge !== 'undefined';
```
```
- [ ] **Step 2: Commit**
```bash
git add docs/hud-bridge-integration.md
git commit -m "docs: HUD bridge integration guide for Laravel app"
```