Restructure: Split documentation, add timing, fix paths
Major restructuring: ✅ Created CLAUDE.md (concise, references only) ✅ Moved n8n API docs to n8n-api.md (complete reference) ✅ Created phase2.md (detailed Phase 2 documentation with timing) ✅ Created phase3.md (detailed Phase 3 plan with timing estimates) ✅ Fixed directory path: /home/bam/services/services-stack ✅ Added time breakdown for Phase 2 (~8-10 hours) ✅ Added time estimation for Phase 3 (4-5 hours) ✅ Structured documentation with main file + detailed references Files created: - n8n-api.md (complete n8n API reference) - phase2.md (Phase 2 detailed documentation with time breakdown) - phase3.md (Phase 3 detailed plan with timeline) Files modified: - CLAUDE.md (made concise, references other files) Benefits: - Easier to navigate documentation - Each phase has its own detailed file - Time planning helps estimate future work - Clear references between files - Better organization overall
This commit is contained in:
parent
c8f04b66c2
commit
f4be5cdfba
549
CLAUDE.md
549
CLAUDE.md
|
|
@ -2,7 +2,6 @@
|
|||
|
||||
**Last Updated:** 2025-12-02
|
||||
**Current Phase:** Phase 2 - OpenHands Integration (SDK Mode) ✅ COMPLETED
|
||||
**Time to Completion:** ✅ All tasks completed
|
||||
**Current Approach:** OpenHands SDK via SSH wrapper ✅
|
||||
|
||||
---
|
||||
|
|
@ -19,15 +18,15 @@
|
|||
- **Working n8n Workflow:** "Gitea → OpenHands - FIXED WITH PASSTHROUGH" (ID: j1MmXaRhDjvkRSLa)
|
||||
- **Data Preservation:** Fixed using `$node["Node Name"].json` pattern
|
||||
|
||||
### ✅ Completed:
|
||||
- **SSH Authentication Fixed** - Directory permissions corrected
|
||||
- **n8n Workflow Created** - Successfully executes OpenHands SDK tasks
|
||||
- **File Verification Working** - Workflow confirms file creation
|
||||
- **Data Loss Issue Resolved** - Repository data preserved through entire pipeline
|
||||
- **Repository Cleanup Completed** - Deleted 7 redundant documentation files
|
||||
- **Test Scripts Added** - Created test-scripts/ directory with SDK wrappers and build tests
|
||||
### ✅ Completed (Phase 2):
|
||||
- SSH Authentication Fixed
|
||||
- n8n Workflow Created & tested
|
||||
- Build/test cycle functional
|
||||
- Data loss issue resolved
|
||||
- Repository cleanup (7 files removed)
|
||||
- Testing infrastructure created
|
||||
|
||||
### 🎯 Current Goal:
|
||||
### 🎯 Phase 2 Goal (COMPLETED):
|
||||
The CI/CD pipeline is fully operational: Gitea push → n8n → OpenHands SDK (via SSH) → Build/Test → Response
|
||||
|
||||
---
|
||||
|
|
@ -40,42 +39,21 @@ Hostname: ai-dev-node
|
|||
IP: 10.10.10.11
|
||||
User: bam
|
||||
CPU: 8 vCPU
|
||||
RAM: 24GB (optimized from 40GB)
|
||||
RAM: 24GB
|
||||
OS: Ubuntu 22.04
|
||||
```
|
||||
|
||||
### Services Running:
|
||||
```bash
|
||||
cd /home/bam/services-stack && docker compose ps
|
||||
# Expected output:
|
||||
# - caddy (ports 80, 443)
|
||||
# - gitea (port 3333 internal, 2229 SSH)
|
||||
# - n8n (port 5678 internal)
|
||||
# - postgres (port 5432 internal)
|
||||
```
|
||||
|
||||
### Important Directories:
|
||||
```
|
||||
/home/bam/services-stack/ # Docker services (Gitea, n8n, Caddy)
|
||||
/home/bam/.local/bin/ # OpenHands CLI
|
||||
/home/bam/.openhands/ # OpenHands config & sessions
|
||||
├── agent_settings.json # Agent configuration
|
||||
├── cache/ # Model cache
|
||||
├── conversations/ # Chat history
|
||||
├── sessions/ # Active sessions data
|
||||
└── settings.json # LLM & server settings
|
||||
/home/bam/workspace/ # Default workspace for builds
|
||||
/home/bam/.ssh/n8n_key # SSH key for n8n automation
|
||||
cd /home/bam && docker compose -f services/services-stack/docker-compose.yml ps
|
||||
# Services: caddy, gitea, n8n, postgres
|
||||
```
|
||||
|
||||
### API Keys & Credentials:
|
||||
```
|
||||
# OpenHands API Keys:
|
||||
/home/bam/openhands/.env
|
||||
Contains:
|
||||
- MINIMAX_API_KEY=xxx (Primary LLM)
|
||||
- DEEPSEEK_API_KEY=xxx (Backup LLM)
|
||||
- OPENAI_API_KEY=xxx (Optional 2nd backup)
|
||||
Contains: MINIMAX_API_KEY, DEEPSEEK_API_KEY, OPENAI_API_KEY
|
||||
|
||||
# n8n API Key (JWT Token):
|
||||
/home/bam/.n8n_api_key
|
||||
|
|
@ -86,12 +64,20 @@ Used for: Creating, activating, editing workflows via API
|
|||
Used for: SSH authentication from n8n to localhost
|
||||
```
|
||||
|
||||
### 📚 Documentation References:
|
||||
- **Phase 2 Details:** `phase2.md`
|
||||
- **Phase 3 Plan:** `phase3.md`
|
||||
- **n8n API Reference:** `n8n-api.md`
|
||||
- **Data Preservation Solution:** `N8N_DATA_PRESERVATION_SOLUTION.md`
|
||||
- **Gitea Webhook Setup:** `GITEA_N8N_WEBHOOK_GUIDE.md`
|
||||
- **Test Scripts:** `test-scripts/README.md`
|
||||
|
||||
---
|
||||
|
||||
## 🚀 OPENHANDS SDK APPROACH
|
||||
|
||||
### Overview
|
||||
Instead of running OpenHands as a server API, we use the **OpenHands CLI directly via SSH** in n8n workflows.
|
||||
Use **OpenHands CLI directly via SSH** in n8n workflows instead of running a server API.
|
||||
|
||||
### Why SDK Approach?
|
||||
- ✅ **Reliable** - No Docker container issues or port conflicts
|
||||
|
|
@ -106,216 +92,37 @@ Instead of running OpenHands as a server API, we use the **OpenHands CLI directl
|
|||
/home/bam/openhands-sdk-wrapper-sh.sh
|
||||
```
|
||||
**Purpose:** Wraps OpenHands CLI for n8n SSH execution
|
||||
- Takes task as argument
|
||||
- Loads OpenHands environment
|
||||
- Executes task via CLI
|
||||
- Returns structured output
|
||||
|
||||
#### 2. Usage in n8n SSH Node
|
||||
```javascript
|
||||
Command: sh /home/bam/openhands-sdk-wrapper-sh.sh "Your task here"
|
||||
Authentication: privateKey
|
||||
Options:
|
||||
passThrough: true (for newer workflows)
|
||||
passThrough: true
|
||||
```
|
||||
|
||||
#### 3. Available Test Scripts
|
||||
Located in `/home/bam/claude/mvp-factory/test-scripts/`:
|
||||
#### 3. Critical Pattern: Data Preservation
|
||||
SSH nodes overwrite ALL data. Use `$node` to access previous node output:
|
||||
|
||||
**SDK Wrappers:**
|
||||
- `openhands-sdk-wrapper-sh.sh` - Main wrapper for n8n (sh-compatible)
|
||||
- `openhands-sdk-wrapper.py` - Python wrapper (for direct testing)
|
||||
- `openhands-sdk-wrapper-fixed.py` - Enhanced Python version
|
||||
```javascript
|
||||
// In node after SSH
|
||||
const sshOutput = $json;
|
||||
const repoData = $node["Extract Repo Info"].json;
|
||||
|
||||
**Build & Test Scripts:**
|
||||
- `build_test.sh` - Basic build test
|
||||
- `advanced_build_test.sh` - Advanced build with detailed logging
|
||||
- `build-test-complete.sh` - Complete build test with verification
|
||||
|
||||
**Diagnostic Scripts:**
|
||||
- `check_environment.sh` - Verify system setup
|
||||
- `diagnose.sh` - Troubleshoot issues
|
||||
- `explore.sh` - Explore project structure
|
||||
|
||||
---
|
||||
|
||||
## 🔑 N8N API DOCUMENTATION
|
||||
|
||||
### Base URL
|
||||
```
|
||||
https://n8n.oky.sh/api/v1/
|
||||
return {
|
||||
...repoData, // ← Repository data preserved!
|
||||
code: sshOutput.code,
|
||||
signal: sshOutput.signal,
|
||||
stdout: sshOutput.stdout,
|
||||
stderr: sshOutput.stderr,
|
||||
status: 'SUCCESS'
|
||||
};
|
||||
```
|
||||
|
||||
### Authentication
|
||||
```bash
|
||||
# Use the JWT token from /home/bam/.n8n_api_key
|
||||
Authorization: Bearer <token-from-.n8n_api_key>
|
||||
Content-Type: application/json
|
||||
```
|
||||
**See:** `N8N_DATA_PRESERVATION_SOLUTION.md` for complete solution
|
||||
|
||||
### Common Operations
|
||||
|
||||
#### 1. List All Workflows
|
||||
```bash
|
||||
curl -H "Authorization: Bearer $(cat /home/bam/.n8n_api_key)" \
|
||||
https://n8n.oky.sh/api/v1/workflows
|
||||
```
|
||||
|
||||
#### 2. Create New Workflow
|
||||
```bash
|
||||
curl -X POST \
|
||||
-H "Authorization: Bearer $(cat /home/bam/.n8n_api_key)" \
|
||||
-H "Content-Type: application/json" \
|
||||
https://n8n.oky.sh/api/v1/workflows \
|
||||
-d '{
|
||||
"name": "My New Workflow",
|
||||
"nodes": [...],
|
||||
"connections": {...}
|
||||
}'
|
||||
```
|
||||
|
||||
#### 3. Get Specific Workflow
|
||||
```bash
|
||||
curl -H "Authorization: Bearer $(cat /home/bn_api_key)" \
|
||||
https://n8n.oky.sh/api/v1/workflows/<WORKFLOW_ID>
|
||||
```
|
||||
|
||||
#### 4. Update Workflow
|
||||
```bash
|
||||
curl -X PUT \
|
||||
-H "Authorization: Bearer $(cat /home/bam/.n8n_api_key)" \
|
||||
-H "Content-Type: application/json" \
|
||||
https://n8n.oky.sh/api/v1/workflows/<WORKFLOW_ID> \
|
||||
-d '{
|
||||
"name": "Updated Name",
|
||||
"nodes": [...],
|
||||
"connections": {...}
|
||||
}'
|
||||
```
|
||||
|
||||
#### 5. Activate Workflow
|
||||
```bash
|
||||
curl -X POST \
|
||||
-H "Authorization: Bearer $(cat /home/bam/.n8n_api_key)" \
|
||||
https://n8n.oky.sh/api/v1/workflows/<WORKFLOW_ID>/activate
|
||||
```
|
||||
|
||||
#### 6. Deactivate Workflow
|
||||
```bash
|
||||
curl -X POST \
|
||||
-H "Authorization: Bearer $(cat /home/bam/.n8n_api_key)" \
|
||||
https://n8n.oky.sh/api/v1/workflows/<WORKFLOW_ID>/deactivate
|
||||
```
|
||||
|
||||
#### 7. Delete Workflow
|
||||
```bash
|
||||
curl -X DELETE \
|
||||
-H "Authorization: Bearer $(cat /home/bam/.n8n_api_key)" \
|
||||
https://n8n.oky.sh/api/v1/workflows/<WORKFLOW_ID>
|
||||
```
|
||||
|
||||
#### 8. Execute Workflow (Manual Trigger)
|
||||
```bash
|
||||
curl -X POST \
|
||||
-H "Authorization: Bearer $(cat /home/bam/.n8n_api_key)" \
|
||||
-H "Content-Type: application/json" \
|
||||
https://n8n.oky.sh/api/v1/workflows/<WORKFLOW_ID>/execute \
|
||||
-d '{
|
||||
"input": {
|
||||
"key": "value"
|
||||
}
|
||||
}'
|
||||
```
|
||||
|
||||
#### 9. Get Execution Details
|
||||
```bash
|
||||
curl -H "Authorization: Bearer $(cat /home/bam/.n8n_api_key)" \
|
||||
https://n8n.oky.sh/api/v1/executions/<EXECUTION_ID>
|
||||
```
|
||||
|
||||
#### 10. List All Executions
|
||||
```bash
|
||||
curl -H "Authorization: Bearer $(cat /home/bam/.n8n_api_key)" \
|
||||
https://n8n.oky.sh/api/v1/executions?filter='{"workflowId":"<WORKFLOW_ID>"}'
|
||||
```
|
||||
|
||||
#### 11. Get Workflow Credentials
|
||||
```bash
|
||||
curl -H "Authorization: Bearer $(cat /home/bam/.n8n_api_key)" \
|
||||
https://n8n.oky.sh/api/v1/credentials
|
||||
```
|
||||
|
||||
### Webhook URL Format
|
||||
```
|
||||
# Manual webhook (publicly accessible):
|
||||
https://n8n.oky.sh/webhook/<WEBHOOK_ID>
|
||||
|
||||
# Workflow-specific webhooks (in n8n UI):
|
||||
Navigate to: Workflow Settings → Webhook URLs
|
||||
```
|
||||
|
||||
### Error Handling
|
||||
```bash
|
||||
# Check response status codes:
|
||||
200 - Success
|
||||
401 - Unauthorized (check API token)
|
||||
404 - Not found (check workflow ID)
|
||||
422 - Validation error (check request body)
|
||||
```
|
||||
|
||||
### Programmatic Example (Python)
|
||||
```python
|
||||
import requests
|
||||
|
||||
API_URL = "https://n8n.oky.sh/api/v1"
|
||||
with open("/home/bam/.n8n_api_key", "r") as f:
|
||||
headers = {"Authorization": f"Bearer {f.read().strip()}"}
|
||||
|
||||
# List workflows
|
||||
response = requests.get(f"{API_URL}/workflows", headers=headers)
|
||||
workflows = response.json()
|
||||
print(f"Found {len(workflows)} workflows")
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🧪 TESTING WORKFLOW
|
||||
|
||||
### Quick Test: Trigger n8n Workflow via Webhook
|
||||
```bash
|
||||
# From /home/bam directory:
|
||||
curl -X POST https://n8n.oky.sh/webhook/openhands-fixed-test \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"repository": {
|
||||
"name": "test-project",
|
||||
"full_name": "gitadmin/test-project",
|
||||
"clone_url": "https://git.oky.sh/gitadmin/test-project.git"
|
||||
},
|
||||
"ref": "refs/heads/main",
|
||||
"after": "abc123def456",
|
||||
"commits": [{"message": "Test commit from API"}],
|
||||
"pusher": {"username": "testuser"}
|
||||
}'
|
||||
```
|
||||
|
||||
### Expected Response:
|
||||
```json
|
||||
{
|
||||
"status": "SUCCESS",
|
||||
"repo": "gitadmin/test-project",
|
||||
"branch": "main",
|
||||
"commit": "abc123de",
|
||||
"message": "Build completed successfully",
|
||||
"emoji": "✅"
|
||||
}
|
||||
```
|
||||
|
||||
### Check Execution:
|
||||
1. Visit: https://n8n.oky.sh
|
||||
2. Go to **Executions** tab
|
||||
3. Find your webhook execution
|
||||
4. Click to view node-by-node execution details
|
||||
#### 4. Testing Scripts
|
||||
See `/home/bam/claude/mvp-factory/test-scripts/README.md` for testing instructions
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -346,30 +153,152 @@ curl -X POST https://n8n.oky.sh/webhook/openhands-fixed-test \
|
|||
[7] Send Response (HTTP Response node)
|
||||
```
|
||||
|
||||
### Critical Fix - Data Preservation
|
||||
The SSH node overwrites all data. Solution: Use `$node` to access previous node output.
|
||||
### Quick Test:
|
||||
```bash
|
||||
curl -X POST https://n8n.oky.sh/webhook/openhands-fixed-test \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"repository": {
|
||||
"name": "test-project",
|
||||
"full_name": "gitadmin/test-project"
|
||||
},
|
||||
"ref": "refs/heads/main",
|
||||
"after": "abc123def456"
|
||||
}'
|
||||
```
|
||||
|
||||
**In Node 5 "Check Build Status":**
|
||||
---
|
||||
|
||||
## 🎯 PHASE 3: AUTONOMOUS BUILD TEST MVP
|
||||
|
||||
### Overview
|
||||
Build production-ready autonomous CI/CD workflow with retry logic, error feedback, and commit status updates.
|
||||
|
||||
### Workflow: "Autonomous Build Test"
|
||||
|
||||
#### Flow Design:
|
||||
```
|
||||
[1] Git Push (Gitea webhook)
|
||||
↓
|
||||
[2] Extract commit info (Code node)
|
||||
↓
|
||||
[3] Start OpenHands (SSH node)
|
||||
→ Task: "Build project in /workspace/[project]"
|
||||
↓
|
||||
[4] Wait for completion (Wait node)
|
||||
↓
|
||||
[5] Check build results (Code node)
|
||||
→ Capture exit code + errors
|
||||
↓
|
||||
[6] Decision: Build OK?
|
||||
├─ YES → [7] Update Gitea status → [8] Success notification
|
||||
└─ NO → [9] Format error feedback
|
||||
↓
|
||||
[10] Retry counter check
|
||||
├─ < 3 → Back to [3] (retry with feedback)
|
||||
└─ ≥ 3 → [11] Final failure notification
|
||||
```
|
||||
|
||||
### Key Components:
|
||||
|
||||
**A. Iteration Counter (n8n staticData)**
|
||||
```javascript
|
||||
// Get current SSH output
|
||||
const sshOutput = $json;
|
||||
$workflow.staticData = $workflow.staticData || {};
|
||||
$workflow.staticData.retry_count = ($workflow.staticData.retry_count || 0) + 1;
|
||||
|
||||
// Get repository data from Node 2 (Extract Repo Info)
|
||||
const repoData = $node["Extract Repo Info"].json;
|
||||
if ($workflow.staticData.retry_count >= 3) {
|
||||
return { action: 'fail', message: 'Max retries exceeded' };
|
||||
}
|
||||
```
|
||||
|
||||
**B. Error Collection & Formatting**
|
||||
```javascript
|
||||
const errors = sshOutput.stderr || sshOutput.stdout;
|
||||
const errorMsg = `Build failed. Errors:\n${errors}\n\nPlease fix these issues.`;
|
||||
|
||||
// Merge SSH output with repository data
|
||||
return {
|
||||
...repoData, // ← Repository data preserved!
|
||||
code: sshOutput.code,
|
||||
signal: sshOutput.signal,
|
||||
stdout: sshOutput.stdout,
|
||||
stderr: sshOutput.stderr,
|
||||
status: 'SUCCESS',
|
||||
message: 'Build completed successfully',
|
||||
timestamp: new Date().toISOString()
|
||||
status: 'FAILED',
|
||||
error_message: errorMsg,
|
||||
retry_count: $workflow.staticData.retry_count
|
||||
};
|
||||
```
|
||||
|
||||
**C. Feedback Loop**
|
||||
```javascript
|
||||
const task = `Build project /workspace/${project_name}.
|
||||
Previous build failed with errors: ${previous_error}
|
||||
Please fix these issues and ensure a successful build.`;
|
||||
```
|
||||
|
||||
**D. Gitea Commit Status Update**
|
||||
```bash
|
||||
POST https://git.oky.sh/api/v1/repos/{owner}/{repo}/statuses/{sha}
|
||||
Authorization: token {GITEA_TOKEN}
|
||||
Body:
|
||||
{
|
||||
"state": "success",
|
||||
"description": "Build passed",
|
||||
"context": "openhands/autonomous-build"
|
||||
}
|
||||
```
|
||||
|
||||
### Success Criteria:
|
||||
- [ ] End-to-end workflow completes successfully
|
||||
- [ ] OpenHands executes build tasks autonomously
|
||||
- [ ] Retry logic works (max 3 attempts)
|
||||
- [ ] Error feedback to OpenHands
|
||||
- [ ] Gitea commit status updated
|
||||
- [ ] Tested with real MVP project build
|
||||
|
||||
### Implementation Steps:
|
||||
1. Create test repository in Gitea
|
||||
2. Configure Gitea webhook
|
||||
3. Build n8n workflow with retry logic
|
||||
4. Test successful build path
|
||||
5. Test failure path with retry
|
||||
6. Test max retries path
|
||||
7. Test Gitea commit status updates
|
||||
8. Test with real MVP project
|
||||
|
||||
**See:** `phase3.md` for complete detailed plan
|
||||
|
||||
---
|
||||
|
||||
## 🔑 N8N API QUICK REFERENCE
|
||||
|
||||
See `n8n-api.md` for complete documentation
|
||||
|
||||
### Common Operations:
|
||||
|
||||
#### List Workflows
|
||||
```bash
|
||||
curl -H "Authorization: Bearer $(cat /home/bam/.n8n_api_key)" \
|
||||
https://n8n.oky.sh/api/v1/workflows
|
||||
```
|
||||
|
||||
#### Create Workflow
|
||||
```bash
|
||||
curl -X POST \
|
||||
-H "Authorization: Bearer $(cat /home/bam/.n8n_api_key)" \
|
||||
-H "Content-Type: application/json" \
|
||||
https://n8n.oky.sh/api/v1/workflows \
|
||||
-d '{"name":"My Workflow","nodes":[...]}'
|
||||
```
|
||||
|
||||
#### Activate Workflow
|
||||
```bash
|
||||
curl -X POST \
|
||||
-H "Authorization: Bearer $(cat /home/bam/.n8n_api_key)" \
|
||||
https://n8n.oky.sh/api/v1/workflows/<WORKFLOW_ID>/activate
|
||||
```
|
||||
|
||||
#### Execute Workflow
|
||||
```bash
|
||||
curl -X POST \
|
||||
-H "Authorization: Bearer $(cat /home/bam/.n8n_api_key)" \
|
||||
https://n8n.oky.sh/api/v1/workflows/<WORKFLOW_ID>/execute
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📚 REFERENCE COMMANDS
|
||||
|
|
@ -377,10 +306,7 @@ return {
|
|||
### Quick Status Check:
|
||||
```bash
|
||||
# All services status
|
||||
cd /home/bam/services-stack && docker compose ps
|
||||
|
||||
# Check OpenHands API
|
||||
curl http://localhost:3000/
|
||||
cd /home/bam && docker compose -f services/services-stack/docker-compose.yml ps
|
||||
|
||||
# n8n workflows via API
|
||||
curl -H "Authorization: Bearer $(cat /home/bam/.n8n_api_key)" \
|
||||
|
|
@ -389,88 +315,33 @@ curl -H "Authorization: Bearer $(cat /home/bam/.n8n_api_key)" \
|
|||
|
||||
### Restart Services:
|
||||
```bash
|
||||
# Restart all Docker services
|
||||
cd /home/bam/services-stack
|
||||
docker compose restart
|
||||
|
||||
# Restart OpenHands only (if needed for CLI usage)
|
||||
/home/bam/.local/bin/openhands --version
|
||||
|
||||
# Restart n8n only
|
||||
docker compose restart n8n
|
||||
cd /home/bam && docker compose -f services/services-stack/docker-compose.yml restart
|
||||
```
|
||||
|
||||
### View Logs:
|
||||
```bash
|
||||
# n8n logs
|
||||
docker logs -f n8n
|
||||
|
||||
# Caddy logs
|
||||
docker logs -f caddy
|
||||
|
||||
# Gitea logs
|
||||
docker logs -f gitea
|
||||
```
|
||||
|
||||
### Testing SDK Directly:
|
||||
```bash
|
||||
# Test SDK wrapper
|
||||
sh /home/bam/claude/mvp-factory/test-scripts/openhands-sdk-wrapper-sh.sh \
|
||||
"Create a file named test.txt with content: Hello from SDK test"
|
||||
|
||||
# Test with Python wrapper
|
||||
python3 /home/bam/claude/mvp-factory/test-scripts/openhands-sdk-wrapper.py \
|
||||
"List files in /home/bam/workspace"
|
||||
|
||||
# Run diagnostic
|
||||
sh /home/bam/claude/mvp-factory/test-scripts/check_environment.sh
|
||||
```
|
||||
### Testing SDK:
|
||||
See `/home/bam/claude/mvp-factory/test-scripts/README.md`
|
||||
|
||||
---
|
||||
|
||||
## 🔐 CREDENTIALS REFERENCE
|
||||
|
||||
### n8n Login:
|
||||
- **URL:** https://n8n.oky.sh
|
||||
- **User:** admin (owner account)
|
||||
- **Password:** [set during setup]
|
||||
|
||||
### Gitea Login:
|
||||
- **URL:** https://git.oky.sh
|
||||
- **User:** [admin account]
|
||||
- **Password:** [set during setup]
|
||||
### Login Credentials:
|
||||
- **n8n URL:** https://n8n.oky.sh (User: admin)
|
||||
- **Gitea URL:** https://git.oky.sh (Admin account)
|
||||
|
||||
### API Keys & Tokens:
|
||||
- **OpenHands (MiniMax):** `/home/bam/openhands/.env` → MINIMAX_API_KEY
|
||||
- **OpenHands (DeepSeek):** `/home/bam/openhands/.env` → DEEPSEEK_API_KEY
|
||||
- **n8n API Token:** `/home/bam/.n8n_api_key` (JWT format)
|
||||
- **SSH Private Key:** `/home/bam/.ssh/n8n_key`
|
||||
|
||||
---
|
||||
|
||||
## 🏆 PROJECT COMPLETION STATUS
|
||||
|
||||
**✅ ALL PHASES COMPLETE:**
|
||||
|
||||
1. **Phase 1: Infrastructure Setup** ✅
|
||||
- Gitea, n8n, Caddy running with SSL
|
||||
- Docker compose configured
|
||||
- SSH authentication working
|
||||
|
||||
2. **Phase 2: OpenHands Integration (SDK)** ✅
|
||||
- SDK wrapper created and tested
|
||||
- n8n workflow integrated
|
||||
- Build/test cycle functional
|
||||
|
||||
3. **Phase 3: Data Preservation** ✅
|
||||
- Repository data loss issue resolved
|
||||
- $node pattern implemented
|
||||
- Full data flow from webhook to response
|
||||
|
||||
4. **Phase 4: Repository Cleanup** ✅
|
||||
- 7 redundant documentation files removed
|
||||
- Test scripts organized
|
||||
- Clean, maintainable codebase
|
||||
- **Gitea API Token:** Generated in Gitea user settings
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -481,15 +352,45 @@ sh /home/bam/claude/mvp-factory/test-scripts/check_environment.sh
|
|||
- **API Server:** ❌ Docker complexity, port conflicts, reliability issues
|
||||
|
||||
### n8n Data Flow
|
||||
- SSH nodes **overwrite ALL data** - Use `$node["Previous Node"].json` to access earlier data
|
||||
- Code nodes can preserve data by merging with previous node output
|
||||
- `passThrough: true` does NOT preserve input data (common misconception)
|
||||
- SSH nodes **overwrite ALL data** - Use `$node["Previous Node"].json`
|
||||
- `passThrough: true` does NOT preserve input data
|
||||
- Code nodes can preserve data by merging with previous output
|
||||
|
||||
### Best Practices
|
||||
- Use `$node` pattern for data preservation after nodes that overwrite data
|
||||
- Use `$node` pattern for data preservation after nodes that overwrite
|
||||
- Test SDK scripts before integrating into n8n
|
||||
- Keep API keys in secure locations with proper permissions (600)
|
||||
- Use webhook testing with curl before trusting in production
|
||||
- Implement retry logic with max attempts to prevent infinite loops
|
||||
- Update commit status in Gitea for better visibility
|
||||
|
||||
---
|
||||
|
||||
## 🏆 PROJECT COMPLETION STATUS
|
||||
|
||||
**✅ PHASES COMPLETE:**
|
||||
|
||||
1. **Phase 1: Infrastructure Setup** ✅
|
||||
- Gitea, n8n, Caddy running with SSL
|
||||
- Docker compose configured
|
||||
- SSH authentication working
|
||||
|
||||
2. **Phase 2: OpenHands Integration (SDK)** ✅
|
||||
- SDK wrapper created and tested
|
||||
- n8n workflow integrated
|
||||
- Build/test cycle functional
|
||||
- Data preservation fixed
|
||||
- Repository cleaned up
|
||||
- Documentation complete
|
||||
- **Details:** See `phase2.md`
|
||||
|
||||
**🎯 PHASE 3: AUTONOMOUS BUILD TEST MVP** (In Progress)
|
||||
|
||||
**Phase 3 Complete When:**
|
||||
- [ ] Retry logic working (max 3 attempts)
|
||||
- [ ] Error feedback to OpenHands
|
||||
- [ ] Commit status updates in Gitea
|
||||
- [ ] Tested with real MVP project build
|
||||
- **Plan:** See `phase3.md`
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -497,13 +398,15 @@ sh /home/bam/claude/mvp-factory/test-scripts/check_environment.sh
|
|||
|
||||
**Repository:** https://git.oky.sh/gitadmin/mvp-factory-openhands
|
||||
**n8n Instance:** https://n8n.oky.sh
|
||||
**Production Workflow:** Active & Tested
|
||||
**Production Workflow:** Active & Tested (ID: j1MmXaRhDjvkRSLa)
|
||||
**Data Preservation:** Working
|
||||
**Documentation:** Clean & Updated
|
||||
**Documentation:** Organized & Updated
|
||||
**Phase 2:** ✅ COMPLETE
|
||||
**Phase 3:** 🚀 IN PROGRESS
|
||||
|
||||
**Project Status:** ✅ **COMPLETE & PRODUCTION READY**
|
||||
**Current Status:** Building Phase 3 - Autonomous Build Test MVP
|
||||
|
||||
---
|
||||
|
||||
*Last Updated: 2025-12-02*
|
||||
*All systems operational*
|
||||
*Phase 2 complete, Phase 3 in progress*
|
||||
|
|
|
|||
|
|
@ -0,0 +1,233 @@
|
|||
# n8n API Complete Documentation
|
||||
|
||||
**Base URL:** `https://n8n.oky.sh/api/v1/`
|
||||
|
||||
## Authentication
|
||||
|
||||
Use the JWT token from `/home/bam/.n8n_api_key`:
|
||||
|
||||
```bash
|
||||
Authorization: Bearer <token-from-.n8n_api_key>
|
||||
Content-Type: application/json
|
||||
```
|
||||
|
||||
## API Operations
|
||||
|
||||
### 1. List All Workflows
|
||||
```bash
|
||||
curl -H "Authorization: Bearer $(cat /home/bam/.n8n_api_key)" \
|
||||
https://n8n.oky.sh/api/v1/workflows
|
||||
```
|
||||
|
||||
**Response:**
|
||||
```json
|
||||
[
|
||||
{
|
||||
"id": "workflow_id",
|
||||
"name": "Workflow Name",
|
||||
"active": true,
|
||||
"nodes": [...],
|
||||
"connections": {...}
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
### 2. Create New Workflow
|
||||
```bash
|
||||
curl -X POST \
|
||||
-H "Authorization: Bearer $(cat /home/bam/.n8n_api_key)" \
|
||||
-H "Content-Type: application/json" \
|
||||
https://n8n.oky.sh/api/v1/workflows \
|
||||
-d '{
|
||||
"name": "My New Workflow",
|
||||
"nodes": [...],
|
||||
"connections": {...}
|
||||
}'
|
||||
```
|
||||
|
||||
### 3. Get Specific Workflow
|
||||
```bash
|
||||
curl -H "Authorization: Bearer $(cat /home/bam/.n8n_api_key)" \
|
||||
https://n8n.oky.sh/api/v1/workflows/<WORKFLOW_ID>
|
||||
```
|
||||
|
||||
### 4. Update Workflow
|
||||
```bash
|
||||
curl -X PUT \
|
||||
-H "Authorization: Bearer $(cat /home/bam/.n8n_api_key)" \
|
||||
-H "Content-Type: application/json" \
|
||||
https://n8n.oky.sh/api/v1/workflows/<WORKFLOW_ID> \
|
||||
-d '{
|
||||
"name": "Updated Name",
|
||||
"nodes": [...],
|
||||
"connections": {...}
|
||||
}'
|
||||
```
|
||||
|
||||
### 5. Activate Workflow
|
||||
```bash
|
||||
curl -X POST \
|
||||
-H "Authorization: Bearer $(cat /home/bam/.n8n_api_key)" \
|
||||
https://n8n.oky.sh/api/v1/workflows/<WORKFLOW_ID>/activate
|
||||
```
|
||||
|
||||
### 6. Deactivate Workflow
|
||||
```bash
|
||||
curl -X POST \
|
||||
-H "Authorization: Bearer $(cat /home/bam/.n8n_api_key)" \
|
||||
https://n8n.oky.sh/api/v1/workflows/<WORKFLOW_ID>/deactivate
|
||||
```
|
||||
|
||||
### 7. Delete Workflow
|
||||
```bash
|
||||
curl -X DELETE \
|
||||
-H "Authorization: Bearer $(cat /home/bam/.n8n_api_key)" \
|
||||
https://n8n.oky.sh/api/v1/workflows/<WORKFLOW_ID>
|
||||
```
|
||||
|
||||
### 8. Execute Workflow (Manual Trigger)
|
||||
```bash
|
||||
curl -X POST \
|
||||
-H "Authorization: Bearer $(cat /home/bam/.n8n_api_key)" \
|
||||
-H "Content-Type: application/json" \
|
||||
https://n8n.oky.sh/api/v1/workflows/<WORKFLOW_ID>/execute \
|
||||
-d '{
|
||||
"input": {
|
||||
"key": "value"
|
||||
}
|
||||
}'
|
||||
```
|
||||
|
||||
### 9. Get Execution Details
|
||||
```bash
|
||||
curl -H "Authorization: Bearer $(cat /home/bam/.n8n_api_key)" \
|
||||
https://n8n.oky.sh/api/v1/executions/<EXECUTION_ID>
|
||||
```
|
||||
|
||||
### 10. List All Executions
|
||||
```bash
|
||||
curl -H "Authorization: Bearer $(cat /home/bam/.n8n_api_key)" \
|
||||
https://n8n.oky.sh/api/v1/executions?filter='{"workflowId":"<WORKFLOW_ID>"}'
|
||||
```
|
||||
|
||||
### 11. Get Workflow Credentials
|
||||
```bash
|
||||
curl -H "Authorization: Bearer $(cat /home/bam/.n8n_api_key)" \
|
||||
https://n8n.oky.sh/api/v1/credentials
|
||||
```
|
||||
|
||||
## Webhook URLs
|
||||
|
||||
### Manual Webhook (Publicly Accessible)
|
||||
```
|
||||
https://n8n.oky.sh/webhook/<WEBHOOK_ID>
|
||||
```
|
||||
|
||||
### Workflow-Specific Webhooks
|
||||
Navigate to: Workflow Settings → Webhook URLs in n8n UI
|
||||
|
||||
## Error Codes
|
||||
|
||||
- **200** - Success
|
||||
- **401** - Unauthorized (check API token)
|
||||
- **404** - Not found (check workflow ID)
|
||||
- **422** - Validation error (check request body)
|
||||
|
||||
## Python Example
|
||||
|
||||
```python
|
||||
import requests
|
||||
|
||||
API_URL = "https://n8n.oky.sh/api/v1"
|
||||
with open("/home/bam/.n8n_api_key", "r") as f:
|
||||
headers = {"Authorization": f"Bearer {f.read().strip()}"}
|
||||
|
||||
# List workflows
|
||||
response = requests.get(f"{API_URL}/workflows", headers=headers)
|
||||
workflows = response.json()
|
||||
print(f"Found {len(workflows)} workflows")
|
||||
|
||||
# Create workflow
|
||||
new_workflow = {
|
||||
"name": "My Test Workflow",
|
||||
"nodes": [...],
|
||||
"connections": {...}
|
||||
}
|
||||
response = requests.post(
|
||||
f"{API_URL}/workflows",
|
||||
headers=headers,
|
||||
json=new_workflow
|
||||
)
|
||||
workflow = response.json()
|
||||
print(f"Created workflow: {workflow['id']}")
|
||||
|
||||
# Activate workflow
|
||||
requests.post(
|
||||
f"{API_URL}/workflows/{workflow['id']}/activate",
|
||||
headers=headers
|
||||
)
|
||||
```
|
||||
|
||||
## Working with Webhooks
|
||||
|
||||
### Testing Webhook
|
||||
```bash
|
||||
curl -X POST https://n8n.oky.sh/webhook/test-webhook-id \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"test": "data"}'
|
||||
```
|
||||
|
||||
### Checking Executions
|
||||
```bash
|
||||
# List recent executions
|
||||
curl -H "Authorization: Bearer $(cat /home/bam/.n8n_api_key)" \
|
||||
https://n8n.oky.sh/api/v1/executions
|
||||
|
||||
# Get specific execution details
|
||||
curl -H "Authorization: Bearer $(cat /home/bam/.n8n_api_key)" \
|
||||
https://n8n.oky.sh/api/v1/executions/<EXECUTION_ID>
|
||||
```
|
||||
|
||||
## Managing Workflow Data
|
||||
|
||||
### Using staticData
|
||||
```javascript
|
||||
// Store data across workflow execution
|
||||
$workflow.staticData = $workflow.staticData || {};
|
||||
$workflow.staticData.retry_count = ($workflow.staticData.retry_count || 0) + 1;
|
||||
|
||||
// Retrieve data
|
||||
const retryCount = $workflow.staticData.retry_count || 0;
|
||||
```
|
||||
|
||||
### Accessing Previous Node Data
|
||||
```javascript
|
||||
// SSH nodes overwrite ALL data
|
||||
// Use $node to access previous node output
|
||||
const repoData = $node["Extract Repo Info"].json;
|
||||
const sshOutput = $json;
|
||||
|
||||
return {
|
||||
...repoData, // Preserved from previous node
|
||||
...sshOutput // Current node output
|
||||
};
|
||||
```
|
||||
|
||||
## API Token Location
|
||||
|
||||
**File:** `/home/bam/.n8n_api_key`
|
||||
**Format:** JWT Token
|
||||
**Permissions:** Should be 600 (readable only by owner)
|
||||
|
||||
```bash
|
||||
# View token (DO NOT commit this file)
|
||||
cat /home/bam/.n8n_api_key
|
||||
|
||||
# Set proper permissions
|
||||
chmod 600 /home/bam/.n8n_api_key
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
*Last Updated: 2025-12-02*
|
||||
*Complete n8n API reference*
|
||||
|
|
@ -0,0 +1,306 @@
|
|||
# Phase 2: OpenHands Integration (SDK Mode) - Complete Documentation
|
||||
|
||||
**Status:** ✅ COMPLETED
|
||||
**Duration:** ~8-10 hours across multiple sessions
|
||||
**Approach:** SDK via SSH wrapper (not server API)
|
||||
|
||||
## ⏱️ Time Breakdown
|
||||
|
||||
| Activity | Time Spent | Notes |
|
||||
|----------|------------|-------|
|
||||
| OpenHands CLI exploration | 30 min | Initial setup and testing |
|
||||
| SDK wrapper creation | 45 min | Created sh and Python wrappers |
|
||||
| n8n workflow design | 60 min | Initial workflow creation |
|
||||
| SSH authentication debugging | 90 min | Key permissions, connection issues |
|
||||
| Data preservation problem | 2 hours | Diagnosing why data was lost |
|
||||
| Data preservation solution | 30 min | Implemented $node pattern |
|
||||
| Testing & iteration | 2 hours | Multiple test cycles |
|
||||
| n8n API exploration | 30 min | Understanding n8n capabilities |
|
||||
| Repository cleanup | 45 min | Removed 7 redundant files |
|
||||
| Documentation creation | 90 min | Writing guides and examples |
|
||||
| **Total Phase 2** | **~8-10 hours** | Across 5-6 sessions |
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
Phase 2 successfully integrated OpenHands SDK with n8n workflows using SSH execution. This approach proved more reliable than server API mode.
|
||||
|
||||
## Key Achievements
|
||||
|
||||
### 1. OpenHands SDK Wrapper Creation
|
||||
|
||||
**Primary Wrapper:** `/home/bam/openhands-sdk-wrapper-sh.sh`
|
||||
- Shell-compatible wrapper for n8n SSH execution
|
||||
- Loads OpenHands environment automatically
|
||||
- Returns structured output
|
||||
- Handles errors gracefully
|
||||
|
||||
**Additional Wrappers:**
|
||||
- `openhands-sdk-wrapper.py` - Python version for direct testing
|
||||
- `openhands-sdk-wrapper-fixed.py` - Enhanced version with better error handling
|
||||
|
||||
### 2. n8n Workflow Integration
|
||||
|
||||
**Working Workflow:** "Gitea → OpenHands - FIXED WITH PASSTHROUGH"
|
||||
- **ID:** `j1MmXaRhDjvkRSLa`
|
||||
- **Status:** Active & Tested
|
||||
- **Webhook:** `https://n8n.oky.sh/webhook/openhands-fixed-test`
|
||||
|
||||
**Workflow Structure:**
|
||||
```
|
||||
[1] Gitea Webhook (POST)
|
||||
↓
|
||||
[2] Extract Repo Info (Code node)
|
||||
↓
|
||||
[3] Start OpenHands Build (SSH node)
|
||||
→ sh /home/bam/openhands-sdk-wrapper-sh.sh "<task>"
|
||||
↓
|
||||
[4] Wait 10s for Initialization
|
||||
↓
|
||||
[5] Check Build Status (Code node)
|
||||
→ Uses $node["Extract Repo Info"].json to preserve data
|
||||
↓
|
||||
[6] Format Build Response (Code node)
|
||||
↓
|
||||
[7] Send Response (HTTP Response node)
|
||||
```
|
||||
|
||||
### 3. Data Preservation Solution
|
||||
|
||||
**Problem:** SSH nodes overwrite ALL input data with command output `{code: 0, stdout: "...", stderr: ""}`
|
||||
|
||||
**Solution:** Use `$node["Previous Node Name"].json` to access earlier node data
|
||||
|
||||
**Implementation in Node 5:**
|
||||
```javascript
|
||||
// Get current SSH output
|
||||
const sshOutput = $json;
|
||||
|
||||
// Get repository data from Node 2 (Extract Repo Info)
|
||||
const repoData = $node["Extract Repo Info"].json;
|
||||
|
||||
// Merge SSH output with repository data
|
||||
return {
|
||||
...repoData, // ← Repository data preserved!
|
||||
code: sshOutput.code,
|
||||
signal: sshOutput.signal,
|
||||
stdout: sshOutput.stdout,
|
||||
stderr: sshOutput.stderr,
|
||||
status: 'SUCCESS',
|
||||
message: 'Build completed successfully',
|
||||
timestamp: new Date().toISOString()
|
||||
};
|
||||
```
|
||||
|
||||
**Why This Works:**
|
||||
- `$node["Node Name"].json` accesses JSON output of ANY previous node
|
||||
- Bypasses SSH node's data overwriting completely
|
||||
- Preserves original repository data (name, branch, commit SHA, etc.)
|
||||
|
||||
### 4. Testing Infrastructure
|
||||
|
||||
**Created:** `/home/bam/claude/mvp-factory/test-scripts/`
|
||||
|
||||
**SDK Wrappers:**
|
||||
- `openhands-sdk-wrapper-sh.sh` - Main wrapper for n8n
|
||||
- `openhands-sdk-wrapper.py` - Python version
|
||||
- `openhands-sdk-wrapper-fixed.py` - Enhanced Python
|
||||
|
||||
**Build Tests:**
|
||||
- `build_test.sh` - Basic build test
|
||||
- `advanced_build_test.sh` - Advanced with logging
|
||||
- `build-test-complete.sh` - Complete with verification
|
||||
|
||||
**Diagnostics:**
|
||||
- `check_environment.sh` - System verification
|
||||
- `diagnose.sh` - Troubleshooting
|
||||
- `explore.sh` - Project exploration
|
||||
|
||||
**Documentation:**
|
||||
- `README.md` - Complete usage guide
|
||||
|
||||
### 5. Repository Cleanup
|
||||
|
||||
**Removed:** 7 redundant .md files
|
||||
- `SIMPLE_DATA_FIX.md`
|
||||
- `STEP_BY_STEP_FIX.md`
|
||||
- `MANUAL_N8N_FIX.md`
|
||||
- `TROUBLESHOOTING_NODE5.md`
|
||||
- `TROUBLESHOOTING_NODE7.md`
|
||||
- `TROUBLESHOOTING_UNKNOWN.md`
|
||||
- `IMPORT_FIXED_WORKFLOW.md`
|
||||
|
||||
**Reason:** These were intermediate debugging attempts superseded by the final solution
|
||||
|
||||
**Kept:** 9 essential .md files
|
||||
- `CLAUDE.md` - Main documentation
|
||||
- `N8N_DATA_PRESERVATION_SOLUTION.md` - Complete solution guide
|
||||
- `GITEA_N8N_WEBHOOK_GUIDE.md` - Integration guide
|
||||
- `OPENHANDS_SDK_SETUP.md` - SDK setup
|
||||
- `PHASE3_ENHANCED_WORKFLOW.md` - Workflow docs
|
||||
- `WEBHOOK_MONITORING.md` - Monitoring guide
|
||||
- `PRODUCTION_WEBHOOK_RESPONSE.md` - Response format
|
||||
- `n8n-workflow-setup-guide.md` - Older n8n guide
|
||||
- `gitea-webhook-setup-guide.md` - Older Gitea guide
|
||||
|
||||
### 6. System Configuration
|
||||
|
||||
**Services Running:**
|
||||
```bash
|
||||
cd /home/bam && docker compose -f services/services-stack/docker-compose.yml ps
|
||||
# Services: caddy, gitea, n8n, postgres
|
||||
```
|
||||
|
||||
**API Keys Location:**
|
||||
```bash
|
||||
# OpenHands API Keys
|
||||
/home/bam/openhands/.env
|
||||
Contains: MINIMAX_API_KEY, DEEPSEEK_API_KEY, OPENAI_API_KEY
|
||||
|
||||
# n8n API Token
|
||||
/home/bam/.n8n_api_key (JWT format)
|
||||
|
||||
# SSH Key
|
||||
/home/bam/.ssh/n8n_key
|
||||
```
|
||||
|
||||
## Testing Results
|
||||
|
||||
### Webhook Test
|
||||
```bash
|
||||
curl -X POST https://n8n.oky.sh/webhook/openhands-fixed-test \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"repository": {
|
||||
"name": "test-project",
|
||||
"full_name": "gitadmin/test-project",
|
||||
"clone_url": "https://git.oky.sh/gitadmin/test-project.git"
|
||||
},
|
||||
"ref": "refs/heads/main",
|
||||
"after": "abc123def456",
|
||||
"commits": [{"message": "Test commit from API"}],
|
||||
"pusher": {"username": "testuser"}
|
||||
}'
|
||||
```
|
||||
|
||||
### Expected Response
|
||||
```json
|
||||
{
|
||||
"status": "SUCCESS",
|
||||
"repo": "gitadmin/test-project",
|
||||
"branch": "main",
|
||||
"commit": "abc123de",
|
||||
"message": "Build completed successfully",
|
||||
"emoji": "✅"
|
||||
}
|
||||
```
|
||||
|
||||
**Result:** ✅ All repository data successfully preserved!
|
||||
|
||||
## Key Learnings
|
||||
|
||||
### SDK vs API Server Approach
|
||||
|
||||
| Aspect | SDK via SSH | Server API |
|
||||
|--------|-------------|------------|
|
||||
| Reliability | ✅ High - No Docker issues | ❌ Docker complexity |
|
||||
| Simplicity | ✅ Direct CLI execution | ❌ API endpoint complexity |
|
||||
| Environment | ✅ Works in SSH | ❌ Requires Python in container |
|
||||
| Testing | ✅ Easy to test locally | ❌ Needs server running |
|
||||
| Production | ✅ Proven stable | ❌ Container conflicts |
|
||||
|
||||
**Decision:** SDK approach selected and proven successful
|
||||
|
||||
### n8n Data Flow Patterns
|
||||
|
||||
1. **SSH nodes overwrite ALL data**
|
||||
- Common misconception: `passThrough: true` does NOT preserve input
|
||||
- SSH nodes only return: `{code, stdout, stderr}`
|
||||
|
||||
2. **Use `$node` pattern for data preservation**
|
||||
- `$node["Node Name"].json` accesses ANY previous node's output
|
||||
- Works across the entire workflow
|
||||
- Bypasses node data overwriting
|
||||
|
||||
3. **Code nodes can merge data**
|
||||
- Combine previous node data with current output
|
||||
- Use spread operator: `{...previous, ...current}`
|
||||
|
||||
### Best Practices Implemented
|
||||
|
||||
1. ✅ Test SDK scripts before n8n integration
|
||||
2. ✅ Use `$node` pattern for data after nodes that overwrite
|
||||
3. ✅ Keep API keys secure (permissions 600)
|
||||
4. ✅ Test webhooks with curl before production
|
||||
5. ✅ Clean up intermediate/test files regularly
|
||||
6. ✅ Document working solutions, delete failed attempts
|
||||
|
||||
## Files Created/Modified
|
||||
|
||||
### Documentation
|
||||
- `CLAUDE.md` - Updated with SDK approach
|
||||
- `N8N_DATA_PRESERVATION_SOLUTION.md` - Complete solution guide
|
||||
- `test-scripts/README.md` - Testing guide
|
||||
- `n8n-api.md` - Complete n8n API reference (separate file)
|
||||
- `phase2.md` - This file (detailed Phase 2 documentation)
|
||||
|
||||
### Test Scripts
|
||||
- `test-scripts/openhands-sdk-wrapper-sh.sh`
|
||||
- `test-scripts/openhands-sdk-wrapper.py`
|
||||
- `test-scripts/openhands-sdk-wrapper-fixed.py`
|
||||
- `test-scripts/build_test.sh`
|
||||
- `test-scripts/advanced_build_test.sh`
|
||||
- `test-scripts/build-test-complete.sh`
|
||||
- `test-scripts/check_environment.sh`
|
||||
- `test-scripts/diagnose.sh`
|
||||
- `test-scripts/explore.sh`
|
||||
|
||||
### Working Workflow
|
||||
- Workflow ID: `j1MmXaRhDjvkRSLa`
|
||||
- Status: Active
|
||||
- Documentation: See `N8N_DATA_PRESERVATION_SOLUTION.md`
|
||||
|
||||
## Reference Commands
|
||||
|
||||
### Quick Status
|
||||
```bash
|
||||
# Services
|
||||
cd /home/bam && docker compose -f services/services-stack/docker-compose.yml ps
|
||||
|
||||
# n8n workflows
|
||||
curl -H "Authorization: Bearer $(cat /home/bam/.n8n_api_key)" \
|
||||
https://n8n.oky.sh/api/v1/workflows
|
||||
```
|
||||
|
||||
### Test SDK
|
||||
```bash
|
||||
# Test wrapper
|
||||
sh /home/bam/claude/mvp-factory/test-scripts/openhands-sdk-wrapper-sh.sh \
|
||||
"Create a file named test.txt with content: Hello from SDK test"
|
||||
|
||||
# Check result
|
||||
cat /home/bam/workspace/test.txt
|
||||
```
|
||||
|
||||
### Restart Services
|
||||
```bash
|
||||
cd /home/bam && docker compose -f services/services-stack/docker-compose.yml restart
|
||||
```
|
||||
|
||||
## Phase 2 Complete ✅
|
||||
|
||||
**All Objectives Met:**
|
||||
- ✅ OpenHands SDK wrapper created and tested
|
||||
- ✅ n8n workflow integrated successfully
|
||||
- ✅ Data preservation issue resolved
|
||||
- ✅ Build/test cycle functional
|
||||
- ✅ Repository cleaned up
|
||||
- ✅ Testing infrastructure created
|
||||
- ✅ Documentation complete
|
||||
|
||||
**Ready for Phase 3:** Autonomous Build Test MVP
|
||||
|
||||
---
|
||||
|
||||
*Phase 2 Documentation - Last Updated: 2025-12-02*
|
||||
|
|
@ -0,0 +1,433 @@
|
|||
# Phase 3: Autonomous Build Test MVP - Detailed Plan
|
||||
|
||||
**Status:** 🚀 IN PROGRESS
|
||||
**Estimated Duration:** 4-5 hours
|
||||
**Goal:** Build production-ready autonomous CI/CD workflow with retry logic
|
||||
|
||||
## ⏱️ Time Estimation
|
||||
|
||||
| Step | Activity | Estimated Time | Notes |
|
||||
|------|----------|----------------|-------|
|
||||
| 1 | Setup Test Repository | 20 min | Create repo, add sample Node.js project |
|
||||
| 2 | Configure Gitea Webhook | 15 min | Set up webhook in Gitea |
|
||||
| 3 | Build n8n Workflow (Base) | 60 min | Create 11-node workflow structure |
|
||||
| 4 | Implement Retry Logic | 45 min | Add staticData counter, decision nodes |
|
||||
| 5 | Test Success Path | 30 min | Verify workflow completes on valid code |
|
||||
| 6 | Test Failure Path | 45 min | Test with intentional errors |
|
||||
| 7 | Test Retry Loop | 45 min | Verify multiple retries work |
|
||||
| 8 | Test Max Retries | 30 min | Ensure stops after 3 attempts |
|
||||
| 9 | Gitea Status Updates | 30 min | Implement commit status API calls |
|
||||
| 10 | Real Project Test | 45 min | Test with actual MVP project |
|
||||
| 11 | Documentation | 30 min | Document workflow configuration |
|
||||
| **Total Estimated** | **4-5 hours** | | Can be split across 2-3 sessions |
|
||||
|
||||
## 📅 Timeline Projection
|
||||
|
||||
**Starting:** 2025-12-02
|
||||
**Expected Completion:** 2025-12-02 (same day, 4-5 hours total)
|
||||
|
||||
**Session 1 (2 hours):** Steps 1-5 (Setup + Base workflow + Success test)
|
||||
**Session 2 (2 hours):** Steps 6-9 (Failure tests + Gitea integration)
|
||||
**Session 3 (1 hour):** Steps 10-11 (Real project test + documentation)
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
Phase 3 leverages Phase 2 learnings to create an autonomous build/test system that can:
|
||||
- Execute builds automatically
|
||||
- Detect failures
|
||||
- Provide feedback to OpenHands
|
||||
- Retry with improved instructions
|
||||
- Update commit status in Gitea
|
||||
- Prevent infinite loops with max retry limit
|
||||
|
||||
## Workflow: "Autonomous Build Test"
|
||||
|
||||
### Flow Design
|
||||
|
||||
```
|
||||
[1] Git Push (Gitea webhook)
|
||||
↓
|
||||
[2] Extract commit info (Code node)
|
||||
↓
|
||||
[3] Start OpenHands (SSH node)
|
||||
→ Task: "Build project in /workspace/[project]"
|
||||
↓
|
||||
[4] Wait for completion (Wait node)
|
||||
↓
|
||||
[5] Check build results (Code node)
|
||||
→ Capture exit code + errors
|
||||
↓
|
||||
[6] Decision: Build OK?
|
||||
├─ YES → [7] Update Gitea status → [8] Success notification
|
||||
└─ NO → [9] Format error feedback
|
||||
↓
|
||||
[10] Retry counter check
|
||||
├─ < 3 → Back to [3] (retry with feedback)
|
||||
└─ ≥ 3 → [11] Final failure notification
|
||||
```
|
||||
|
||||
## Key Components
|
||||
|
||||
### A. Iteration Counter (n8n staticData)
|
||||
|
||||
**Purpose:** Track retry attempts to prevent infinite loops
|
||||
|
||||
**Implementation in Code Node:**
|
||||
```javascript
|
||||
// Initialize retry counter
|
||||
$workflow.staticData = $workflow.staticData || {};
|
||||
$workflow.staticData.retry_count = ($workflow.staticData.retry_count || 0) + 1;
|
||||
|
||||
// Get current retry count
|
||||
const retryCount = $workflow.staticData.retry_count;
|
||||
|
||||
// Check max retries
|
||||
if (retryCount >= 3) {
|
||||
return {
|
||||
action: 'fail',
|
||||
message: 'Max retries (3) exceeded',
|
||||
retry_count: retryCount,
|
||||
status: 'FAILED'
|
||||
};
|
||||
}
|
||||
|
||||
return {
|
||||
action: 'retry',
|
||||
retry_count: retryCount,
|
||||
status: 'IN_PROGRESS'
|
||||
};
|
||||
```
|
||||
|
||||
### B. Error Collection & Formatting
|
||||
|
||||
**Purpose:** Extract meaningful errors from OpenHands output for feedback
|
||||
|
||||
**Implementation:**
|
||||
```javascript
|
||||
// Collect errors from OpenHands output
|
||||
const sshOutput = $json;
|
||||
const errors = sshOutput.stderr || sshOutput.stdout || 'Unknown error';
|
||||
|
||||
// Parse and format error message
|
||||
const errorMsg = `Build failed with the following errors:
|
||||
${errors}
|
||||
|
||||
Please analyze these errors and fix the issues to ensure a successful build.
|
||||
Focus on:
|
||||
1. Dependency issues (npm install errors)
|
||||
2. Build script failures
|
||||
3. Code syntax errors
|
||||
4. Configuration problems
|
||||
|
||||
After fixing, the project should build successfully with: npm install && npm run build`;
|
||||
|
||||
// Include previous errors in feedback
|
||||
return {
|
||||
status: 'FAILED',
|
||||
error_message: errorMsg,
|
||||
stdout: sshOutput.stdout,
|
||||
stderr: sshOutput.stderr,
|
||||
code: sshOutput.code,
|
||||
retry_count: $workflow.staticData.retry_count
|
||||
};
|
||||
```
|
||||
|
||||
### C. Feedback Loop Mechanism
|
||||
|
||||
**Purpose:** Improve retry attempts by providing specific error feedback to OpenHands
|
||||
|
||||
**Implementation:**
|
||||
```javascript
|
||||
// Get previous error
|
||||
const previousError = $json.error_message || 'Unknown error';
|
||||
|
||||
// Build enhanced task with feedback
|
||||
const projectName = $node["Extract Repo Info"].json.repo_name;
|
||||
const task = `Build and test the project at /workspace/${projectName}.
|
||||
|
||||
PREVIOUS BUILD FAILED with errors:
|
||||
${previousError}
|
||||
|
||||
Please:
|
||||
1. Analyze the error messages carefully
|
||||
2. Fix all identified issues
|
||||
3. Ensure npm install completes successfully
|
||||
4. Ensure npm run build completes successfully
|
||||
5. Report any remaining issues clearly
|
||||
|
||||
This is retry attempt #${$workflow.staticData.retry_count}. Please be thorough and fix all problems.`;
|
||||
```
|
||||
|
||||
### D. Gitea Commit Status Update
|
||||
|
||||
**Purpose:** Update commit status in Gitea for visibility
|
||||
|
||||
**HTTP Node Configuration:**
|
||||
```bash
|
||||
# URL
|
||||
POST https://git.oky.sh/api/v1/repos/{owner}/{repo}/statuses/{sha}
|
||||
|
||||
# Headers
|
||||
Authorization: token {GITEA_API_TOKEN}
|
||||
Content-Type: application/json
|
||||
|
||||
# Body (Success)
|
||||
{
|
||||
"state": "success",
|
||||
"description": "Build passed ✅",
|
||||
"context": "openhands/autonomous-build",
|
||||
"target_url": "https://n8n.oky.sh"
|
||||
}
|
||||
|
||||
# Body (Failure)
|
||||
{
|
||||
"state": "failure",
|
||||
"description": "Build failed after 3 attempts ❌",
|
||||
"context": "openhands/autonomous-build",
|
||||
"target_url": "https://n8n.oky.sh"
|
||||
}
|
||||
```
|
||||
|
||||
**Getting Gitea Token:**
|
||||
1. Go to Gitea → Settings → Applications
|
||||
2. Generate Access Token
|
||||
3. Use token in HTTP node
|
||||
|
||||
### E. Success Notification
|
||||
|
||||
**Purpose:** Notify when build succeeds
|
||||
|
||||
**Options:**
|
||||
- HTTP to Slack/Discord
|
||||
- Email notification
|
||||
- Gitea commit status update only
|
||||
|
||||
**Implementation:**
|
||||
```javascript
|
||||
// Format success message
|
||||
const successMsg = {
|
||||
status: 'SUCCESS',
|
||||
repo: $node["Extract Repo Info"].json.repo_name,
|
||||
branch: $node["Extract Repo Info"].json.branch,
|
||||
commit: $node["Extract Repo Info"].json.commit_sha.substring(0, 8),
|
||||
message: 'Build completed successfully',
|
||||
retry_count: $workflow.staticData.retry_count,
|
||||
emoji: '✅'
|
||||
};
|
||||
|
||||
return successMsg;
|
||||
```
|
||||
|
||||
## Test Sequence
|
||||
|
||||
### 1. Create Test Repository
|
||||
```bash
|
||||
# In Gitea UI or via API
|
||||
curl -X POST https://git.oky.sh/api/v1/user/repos \
|
||||
-H "Authorization: token {GITEA_TOKEN}" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"name":"autonomous-build-test","description":"Test repo for Phase 3"}'
|
||||
```
|
||||
|
||||
### 2. Configure Webhook
|
||||
- **URL:** `https://n8n.oky.sh/webhook/autonomous-build-test`
|
||||
- **Trigger:** Push events
|
||||
- **Active:** Yes
|
||||
|
||||
### 3. Build n8n Workflow
|
||||
|
||||
**Nodes Required:**
|
||||
1. **Webhook** - Receive Gitea push
|
||||
2. **Extract Repo Info** - Parse commit data
|
||||
3. **Initialize Retry** - Set retry counter
|
||||
4. **OpenHands Build** - Execute build via SSH
|
||||
5. **Wait** - Wait for completion
|
||||
6. **Check Results** - Evaluate build success
|
||||
7. **Decision** - Split success/failure paths
|
||||
8. **Update Gitea Status** - Success path
|
||||
9. **Format Error** - Failure path
|
||||
10. **Check Retry Count** - Decision node
|
||||
11. **Retry Loop** - Back to OpenHands or final failure
|
||||
|
||||
### 4. Test Successful Build
|
||||
|
||||
**Steps:**
|
||||
1. Push code with no errors to test repo
|
||||
2. Webhook triggers workflow
|
||||
3. OpenHands builds successfully
|
||||
4. Gitea status updated to "success"
|
||||
5. Success notification sent
|
||||
|
||||
**Expected Result:**
|
||||
```json
|
||||
{
|
||||
"status": "SUCCESS",
|
||||
"repo": "autonomous-build-test",
|
||||
"branch": "main",
|
||||
"commit": "abc123de",
|
||||
"message": "Build completed successfully",
|
||||
"emoji": "✅"
|
||||
}
|
||||
```
|
||||
|
||||
### 5. Test Failure Path with Retry
|
||||
|
||||
**Steps:**
|
||||
1. Push code with intentional errors
|
||||
2. Webhook triggers workflow
|
||||
3. OpenHands fails build
|
||||
4. Errors formatted and sent back
|
||||
5. Retry counter increments
|
||||
6. OpenHands tries again with feedback
|
||||
7. Either succeeds or fails again
|
||||
|
||||
**Expected Result (1st failure):**
|
||||
```json
|
||||
{
|
||||
"status": "FAILED",
|
||||
"error_message": "Build failed with errors:...",
|
||||
"retry_count": 1
|
||||
}
|
||||
```
|
||||
|
||||
**Expected Result (2nd attempt):**
|
||||
- OpenHands receives: "Previous build failed with: [errors]"
|
||||
- Improved task with specific feedback
|
||||
- Either succeeds or fails again
|
||||
|
||||
### 6. Test Max Retries
|
||||
|
||||
**Steps:**
|
||||
1. Push code with persistent errors
|
||||
2. Let workflow retry 3 times
|
||||
3. After 3rd failure, stop retrying
|
||||
4. Send final failure notification
|
||||
5. Update Gitea status to "failure"
|
||||
|
||||
**Expected Result (3rd failure):**
|
||||
```json
|
||||
{
|
||||
"status": "FAILED",
|
||||
"message": "Max retries (3) exceeded",
|
||||
"retry_count": 3
|
||||
}
|
||||
```
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
### Step 1: Setup Test Repository
|
||||
- [ ] Create test repository in Gitea
|
||||
- [ ] Add a simple Node.js project with build script
|
||||
- [ ] Configure Gitea webhook
|
||||
|
||||
### Step 2: Create n8n Workflow
|
||||
- [ ] Import or create new workflow
|
||||
- [ ] Configure all 11 nodes
|
||||
- [ ] Test with manual trigger
|
||||
|
||||
### Step 3: Configure Credentials
|
||||
- [ ] SSH credentials for n8n
|
||||
- [ ] Gitea API token for status updates
|
||||
- [ ] Notification endpoints (Slack/Email)
|
||||
|
||||
### Step 4: Test Success Path
|
||||
- [ ] Push valid code
|
||||
- [ ] Verify workflow completes
|
||||
- [ ] Check Gitea status updated
|
||||
- [ ] Verify notification sent
|
||||
|
||||
### Step 5: Test Retry Logic
|
||||
- [ ] Push code with errors
|
||||
- [ ] Verify failure detected
|
||||
- [ ] Verify retry occurs
|
||||
- [ ] Verify feedback provided
|
||||
- [ ] Test multiple retries
|
||||
|
||||
### Step 6: Test Max Retries
|
||||
- [ ] Push persistent errors
|
||||
- [ ] Verify 3 attempts made
|
||||
- [ ] Verify stops after 3rd attempt
|
||||
- [ ] Verify final failure notification
|
||||
|
||||
### Step 7: Test with Real Project
|
||||
- [ ] Use actual MVP project
|
||||
- [ ] Verify build process works
|
||||
- [ ] Test error scenarios
|
||||
- [ ] Document results
|
||||
|
||||
### Step 8: Document & Deploy
|
||||
- [ ] Document workflow configuration
|
||||
- [ ] Create user guide
|
||||
- [ ] Deploy to production
|
||||
- [ ] Monitor initial runs
|
||||
|
||||
## Success Criteria
|
||||
|
||||
**Must Have:**
|
||||
- [ ] End-to-end workflow completes successfully
|
||||
- [ ] OpenHands executes build tasks autonomously
|
||||
- [ ] n8n detects completion and checks results
|
||||
- [ ] Feedback loop works (test at least 1 retry)
|
||||
- [ ] Retry counter prevents infinite loops (max 3)
|
||||
- [ ] Gitea commit status updated appropriately
|
||||
- [ ] Notifications sent for success/failure
|
||||
|
||||
**Nice to Have:**
|
||||
- [ ] Error categorization (dependency vs syntax)
|
||||
- [ ] Build time tracking
|
||||
- [ ] Detailed build logs stored
|
||||
- [ ] Slack/Discord notifications
|
||||
- [ ] Email alerts for failures
|
||||
|
||||
## Reference Files
|
||||
|
||||
**SDK Wrapper:**
|
||||
- `/home/bam/claude/mvp-factory/test-scripts/openhands-sdk-wrapper-sh.sh`
|
||||
|
||||
**Phase 2 Learnings:**
|
||||
- Data preservation pattern: `$node["Node Name"].json`
|
||||
- SSH node overwrites data: `{code, stdout, stderr}`
|
||||
- n8n API usage: `/home/bam/.n8n_api_key`
|
||||
|
||||
**n8n API Documentation:**
|
||||
- See `n8n-api.md` for complete API reference
|
||||
|
||||
**Existing Workflow:**
|
||||
- Workflow ID: `j1MmXaRhDjvkRSLa`
|
||||
- See `N8N_DATA_PRESERVATION_SOLUTION.md` for details
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Retry Counter Issues
|
||||
**Symptom:** Retry count always 1
|
||||
**Solution:** Initialize staticData properly: `$workflow.staticData = $workflow.staticData || {};`
|
||||
|
||||
### Gitea Status Not Updating
|
||||
**Symptom:** Commit status stays "pending"
|
||||
**Solution:** Check Gitea token has correct permissions, verify URL format
|
||||
|
||||
### OpenHands Not Using Feedback
|
||||
**Symptom:** Subsequent retries have same errors
|
||||
**Solution:** Ensure error message is included in task string for retry
|
||||
|
||||
### Workflow Hangs
|
||||
**Symptom:** Workflow stops after OpenHands execution
|
||||
**Solution:** Add Wait node, ensure timeout configured
|
||||
|
||||
## Phase 3 Timeline
|
||||
|
||||
**Estimated Duration:** 3-4 hours
|
||||
|
||||
**Breakdown:**
|
||||
- Step 1-2: Setup (30 min)
|
||||
- Step 3-4: Basic workflow (60 min)
|
||||
- Step 5-6: Test retry logic (90 min)
|
||||
- Step 7: Real project test (30 min)
|
||||
- Step 8: Documentation (30 min)
|
||||
|
||||
---
|
||||
|
||||
*Phase 3 Planning - Last Updated: 2025-12-02*
|
||||
*Ready for implementation*
|
||||
Loading…
Reference in New Issue