Skip to content

ScoutingSystemV2 / SSEData

Go to a project
Commit a2aa984d by Augusto
Options

cronjobs

parent e1242bc3
Showing with 3311 additions and 100 deletions
.gitignore
......@@ -30,3 +30,7 @@
# Docker
**/.docker/
DEPLOYMENT_SYNC.md
README_SYNC.md
.gitignore
migrations/RESTORE_INSTRUCTIONS.md
SYNC_QUICKSTART.md 0 → 100644
# Sync System - Quick Start
## What It Does
✅ Runs **1-2 times per day** (configurable)
✅ Checks Wyscout API for **updated objects** (players, teams, matches, etc.)
✅ Updates your database with **only the changed data**
✅ Runs as a **separate service** from your API server
## Setup (One Time)
```bash
# 1. Run database migration
psql -U elitedata -d postgres -f migrations/0024_create_sync_logs_table.sql
# 2. Verify your .env has Wyscout credentials
# ProviderUser=your_username
# ProviderSecret=your_password
# 3. Test it works
go run cmd/sync-daemon/main.go -once
```
## Running the Daemon
### Option A: Simple (Development)
```bash
# Run 2x per day (every 12 hours)
go run cmd/sync-daemon/main.go
# Or use the script
chmod +x scripts/start_sync_daemon.sh
./scripts/start_sync_daemon.sh
# View logs
tail -f logs/sync-daemon.log
# Stop
./scripts/stop_sync_daemon.sh
```
### Option B: Docker (Production)
```bash
# Start
docker-compose -f docker-compose.sync.yml up -d
# Logs
docker-compose -f docker-compose.sync.yml logs -f sync-daemon
# Stop
docker-compose -f docker-compose.sync.yml down
```
### Option C: PM2 (Production)
```bash
# Build
go build -o bin/sync-daemon cmd/sync-daemon/main.go
# Start
pm2 start ecosystem.sync.config.js
# Monitor
pm2 logs wyscout-sync-daemon
```
## Configuration
**Default Schedule: 2x per day (every 12 hours)**
Change in `.env`:
```env
SYNC_INTERVAL=12 # Hours between syncs
LOOKBACK_HOURS=24 # How far back to check
```
Or via command line:
```bash
# 1x per day (every 24 hours)
go run cmd/sync-daemon/main.go -interval=24 -lookback=48
# 4x per day (every 6 hours)
go run cmd/sync-daemon/main.go -interval=6 -lookback=12
```
## Separate Service? YES!
The sync daemon runs **independently** from your API server:
```
┌─────────────────┐ ┌─────────────────┐
│ API Server │ │ Sync Daemon │
│ Port 8080 │ │ (Background) │
│ │ │ │
│ - Handle API │ │ - Auto sync │
│ requests │ │ every 12hrs │
│ - Manual syncs │ │ - No HTTP port │
└────────┬────────┘ └────────┬────────┘
│ │
└───────────┬───────────────┘
│
┌──────▼──────┐
│ PostgreSQL │
└─────────────┘
```
**Benefits:**
- ✅ API server can restart without affecting syncs
- ✅ Sync daemon can restart without affecting API
- ✅ Better resource isolation
- ✅ Easier monitoring
## Monitoring
```bash
# Check what was synced
go run cmd/sync/main.go -logs
# Check last sync times
go run cmd/sync/main.go -status
# Via API
curl "http://localhost:8080/api/sync/logs?limit=10"
```
## Typical Output
```
========================================
Wyscout Sync Daemon
========================================
Sync interval: Every 12 hours
Lookback period: 24 hours
========================================
Running initial sync on startup...
Syncing updates since: 2026-02-02 11:50:00
→ Syncing players...
✓ players: 150 processed, 2 failed (status: completed)
→ Syncing teams...
✓ teams: 45 processed, 0 failed (status: completed)
→ Syncing matches...
✓ matches: 200 processed, 5 failed (status: partial)
...
========================================
Sync Summary: 1,245 processed, 12 failed
========================================
Daemon started. Next sync in 12 hours...
```
## Files Created
| File | Purpose |
|------|---------|
| `cmd/sync-daemon/main.go` | Standalone sync daemon |
| `cmd/sync/main.go` | CLI for manual syncs |
| `scripts/start_sync_daemon.sh` | Start script |
| `scripts/stop_sync_daemon.sh` | Stop script |
| `docker-compose.sync.yml` | Docker deployment |
| `Dockerfile.sync` | Docker image |
| `ecosystem.sync.config.js` | PM2 config |
## Recommended Setup
**For Production:**
1. Run sync daemon via Docker or PM2
2. Schedule: Every 12 hours (2x per day)
3. Lookback: 24 hours
4. Monitor logs daily
**For Development:**
1. Run sync daemon with simple script
2. Schedule: Every 24 hours (1x per day)
3. Test with manual syncs first
## Need Help?
- Full docs: `docs/SYNC_SYSTEM.md`
- Deployment guide: `DEPLOYMENT_SYNC.md`
- Manual sync: `README_SYNC.md`
cmd/sync-daemon/main.go 0 → 100644
package main
import (
"ScoutingSystemScoreData/internal/config"
"ScoutingSystemScoreData/internal/database"
"ScoutingSystemScoreData/internal/services"
"flag"
"log"
"os"
"os/signal"
"syscall"
"time"
"github.com/joho/godotenv"
)
func main() {
// Load environment variables
if err := godotenv.Load(); err != nil {
log.Println("No .env file found, using environment variables")
}
// Parse command line flags
syncInterval := flag.Int("interval", 12, "Sync interval in hours (default: 12 hours = 2x per day)")
lookbackHours := flag.Int("lookback", 24, "Hours to look back for updates (default: 24)")
runOnce := flag.Bool("once", false, "Run sync once and exit")
flag.Parse()
log.Println("========================================")
log.Println("Wyscout Sync Daemon")
log.Println("========================================")
log.Printf("Sync interval: Every %d hours", *syncInterval)
log.Printf("Lookback period: %d hours", *lookbackHours)
log.Println("========================================")
// Load config
cfg := config.Load()
// Initialize database
db, err := database.Connect(cfg)
if err != nil {
log.Fatalf("Failed to connect to database: %v", err)
}
// Initialize sync service
syncService := services.NewSyncService(db, cfg)
// Run once mode
if *runOnce {
log.Println("Running sync once...")
runSync(syncService, *lookbackHours)
log.Println("Sync completed. Exiting.")
return
}
// Run initial sync on startup
log.Println("Running initial sync on startup...")
runSync(syncService, *lookbackHours)
// Setup signal handling for graceful shutdown
sigChan := make(chan os.Signal, 1)
signal.Notify(sigChan, os.Interrupt, syscall.SIGTERM)
// Start ticker for periodic syncs
ticker := time.NewTicker(time.Duration(*syncInterval) * time.Hour)
defer ticker.Stop()
log.Printf("Daemon started. Next sync in %d hours...", *syncInterval)
for {
select {
case <-ticker.C:
log.Println("Starting scheduled sync...")
runSync(syncService, *lookbackHours)
log.Printf("Sync completed. Next sync in %d hours...", *syncInterval)
case sig := <-sigChan:
log.Printf("Received signal: %v", sig)
log.Println("Shutting down gracefully...")
return
}
}
}
func runSync(syncService *services.SyncService, lookbackHours int) {
updatedSince := time.Now().Add(-time.Duration(lookbackHours) * time.Hour)
resourceTypes := []string{
"players",
"teams",
"matches",
"coaches",
"referees",
"competitions",
"seasons",
"areas",
}
log.Printf("Syncing updates since: %s", updatedSince.Format("2006-01-02 15:04:05"))
totalProcessed := 0
totalFailed := 0
for _, resourceType := range resourceTypes {
log.Printf("→ Syncing %s...", resourceType)
syncLog, err := syncService.SyncResourceType(resourceType, updatedSince, "scheduled")
if err != nil {
log.Printf(" ✗ Error syncing %s: %v", resourceType, err)
continue
}
log.Printf(" ✓ %s: %d processed, %d failed (status: %s)",
resourceType,
syncLog.ProcessedRecords,
syncLog.FailedRecords,
syncLog.Status,
)
totalProcessed += syncLog.ProcessedRecords
totalFailed += syncLog.FailedRecords
}
log.Println("========================================")
log.Printf("Sync Summary: %d processed, %d failed", totalProcessed, totalFailed)
log.Println("========================================")
}
cmd/sync/main.go 0 → 100644
package main
import (
"ScoutingSystemScoreData/internal/config"
"ScoutingSystemScoreData/internal/database"
"ScoutingSystemScoreData/internal/services"
"flag"
"fmt"
"log"
"os"
"time"
"github.com/joho/godotenv"
)
func main() {
// Load environment variables
if err := godotenv.Load(); err != nil {
log.Println("No .env file found, using environment variables")
}
// Parse command line flags
resourceType := flag.String("resource", "", "Resource type to sync (players, teams, matches, etc.)")
hoursBack := flag.Int("hours", 24, "Hours to look back (max 168)")
syncAll := flag.Bool("all", false, "Sync all resource types")
showStatus := flag.Bool("status", false, "Show sync status")
showLogs := flag.Bool("logs", false, "Show recent sync logs")
flag.Parse()
// Load config
cfg := config.Load()
// Initialize database
db, err := database.Connect(cfg)
if err != nil {
log.Fatalf("Failed to connect to database: %v", err)
}
// Initialize sync service
syncService := services.NewSyncService(db, cfg)
// Execute command based on flags
if *showStatus {
showSyncStatus(syncService)
return
}
if *showLogs {
showSyncLogs(syncService)
return
}
if *syncAll {
syncAllResources(syncService, *hoursBack)
return
}
if *resourceType != "" {
syncResource(syncService, *resourceType, *hoursBack)
return
}
// No valid flags provided
flag.Usage()
os.Exit(1)
}
func syncResource(syncService *services.SyncService, resourceType string, hoursBack int) {
if hoursBack > 168 {
log.Fatal("Cannot go back more than 168 hours (API limitation)")
}
updatedSince := time.Now().Add(-time.Duration(hoursBack) * time.Hour)
log.Printf("Starting sync for %s (last %d hours)...", resourceType, hoursBack)
log.Printf("Updated since: %s", updatedSince.Format("2006-01-02 15:04:05"))
syncLog, err := syncService.SyncResourceType(resourceType, updatedSince, "manual")
if err != nil {
log.Fatalf("Sync failed: %v", err)
}
fmt.Println("\n=== Sync Results ===")
fmt.Printf("Resource Type: %s\n", syncLog.ResourceType)
fmt.Printf("Status: %s\n", syncLog.Status)
fmt.Printf("Total Records: %d\n", syncLog.TotalRecords)
fmt.Printf("Processed: %d\n", syncLog.ProcessedRecords)
fmt.Printf("Failed: %d\n", syncLog.FailedRecords)
fmt.Printf("Started: %s\n", syncLog.StartedAt.Format(time.RFC3339))
if syncLog.CompletedAt != nil {
fmt.Printf("Completed: %s\n", syncLog.CompletedAt.Format(time.RFC3339))
duration := syncLog.CompletedAt.Sub(syncLog.StartedAt)
fmt.Printf("Duration: %s\n", duration.String())
}
if syncLog.ErrorMessage != nil {
fmt.Printf("Error: %s\n", *syncLog.ErrorMessage)
}
}
func syncAllResources(syncService *services.SyncService, hoursBack int) {
if hoursBack > 168 {
log.Fatal("Cannot go back more than 168 hours (API limitation)")
}
updatedSince := time.Now().Add(-time.Duration(hoursBack) * time.Hour)
log.Printf("Starting sync for all resources (last %d hours)...", hoursBack)
log.Printf("Updated since: %s", updatedSince.Format("2006-01-02 15:04:05"))
if err := syncService.SyncAllResources(updatedSince); err != nil {
log.Fatalf("Sync failed: %v", err)
}
fmt.Println("\n=== All Resources Synced ===")
fmt.Println("Check sync logs for detailed results")
}
func showSyncStatus(syncService *services.SyncService) {
resourceTypes := []string{
"players", "teams", "matches", "coaches",
"referees", "competitions", "seasons", "areas",
}
fmt.Println("\n=== Sync Status ===")
for _, resourceType := range resourceTypes {
lastSync, err := syncService.GetLastSyncTime(resourceType)
if err != nil {
fmt.Printf("%s: Error - %v\n", resourceType, err)
continue
}
hoursAgo := int(time.Since(lastSync).Hours())
fmt.Printf("%s: Last synced %d hours ago (%s)\n",
resourceType, hoursAgo, lastSync.Format("2006-01-02 15:04:05"))
}
}
func showSyncLogs(syncService *services.SyncService) {
var logs []struct {
ResourceType string
Status string
ProcessedRecords int
FailedRecords int
StartedAt time.Time
CompletedAt *time.Time
}
db := syncService.GetDB()
if err := db.Table("sync_logs").
Select("resource_type, status, processed_records, failed_records, started_at, completed_at").
Order("started_at DESC").
Limit(20).
Find(&logs).Error; err != nil {
log.Fatalf("Failed to fetch sync logs: %v", err)
}
fmt.Println("\n=== Recent Sync Logs ===")
fmt.Printf("%-15s %-10s %-10s %-8s %-20s\n", "Resource", "Status", "Processed", "Failed", "Started")
fmt.Println("--------------------------------------------------------------------------------")
for _, log := range logs {
fmt.Printf("%-15s %-10s %-10d %-8d %-20s\n",
log.ResourceType,
log.Status,
log.ProcessedRecords,
log.FailedRecords,
log.StartedAt.Format("2006-01-02 15:04"),
)
}
}
diagnostics/check_indexes.sql 0 → 100644
-- ============================================
-- Database Index Diagnostic Query
-- Run this on your production server to verify all indexes exist
-- ============================================
-- 1. Check all indexes on players table
SELECT
schemaname,
tablename,
indexname,
indexdef
FROM pg_indexes
WHERE tablename = 'players'
ORDER BY indexname;
-- 2. Check all indexes on teams table
SELECT
schemaname,
tablename,
indexname,
indexdef
FROM pg_indexes
WHERE tablename = 'teams'
ORDER BY indexname;
-- 3. Check trigram extension is installed
SELECT * FROM pg_extension WHERE extname = 'pg_trgm';
-- 4. Check unaccent extension is installed
SELECT * FROM pg_extension WHERE extname = 'unaccent';
-- 5. Verify specific critical indexes exist
SELECT
CASE
WHEN EXISTS (SELECT 1 FROM pg_indexes WHERE indexname = 'idx_players_short_name_trgm')
THEN '✓ FOUND'
ELSE '✗ MISSING'
END as idx_players_short_name_trgm,
CASE
WHEN EXISTS (SELECT 1 FROM pg_indexes WHERE indexname = 'idx_players_first_name_trgm')
THEN '✓ FOUND'
ELSE '✗ MISSING'
END as idx_players_first_name_trgm,
CASE
WHEN EXISTS (SELECT 1 FROM pg_indexes WHERE indexname = 'idx_players_last_name_trgm')
THEN '✓ FOUND'
ELSE '✗ MISSING'
END as idx_players_last_name_trgm,
CASE
WHEN EXISTS (SELECT 1 FROM pg_indexes WHERE indexname = 'idx_teams_name_trgm')
THEN '✓ FOUND'
ELSE '✗ MISSING'
END as idx_teams_name_trgm,
CASE
WHEN EXISTS (SELECT 1 FROM pg_indexes WHERE indexname = 'idx_players_current_team_id')
THEN '✓ FOUND'
ELSE '✗ MISSING'
END as idx_players_current_team_id,
CASE
WHEN EXISTS (SELECT 1 FROM pg_indexes WHERE indexname = 'idx_teams_wy_id')
THEN '✓ FOUND'
ELSE '✗ MISSING'
END as idx_teams_wy_id;
-- 6. Check table statistics (last analyzed)
SELECT
schemaname,
relname as tablename,
last_analyze,
last_autoanalyze,
n_live_tup as row_count
FROM pg_stat_user_tables
WHERE relname IN ('players', 'teams')
ORDER BY relname;
-- 7. Test query performance with EXPLAIN ANALYZE
EXPLAIN ANALYZE
SELECT players.*
FROM players
LEFT JOIN teams ON teams.wy_id = players.current_team_id
WHERE players.is_active = true
AND players.gender = 'male'
AND (
LOWER(players.short_name) ILIKE '%rafa silva%' OR
LOWER(players.first_name) ILIKE '%rafa silva%' OR
LOWER(players.last_name) ILIKE '%rafa silva%' OR
LOWER(teams.name) ILIKE '%rafa silva%' OR
LOWER(teams.short_name) ILIKE '%rafa silva%'
)
ORDER BY
CASE WHEN LOWER(players.short_name) = 'rafa silva' THEN 0
WHEN LOWER(players.last_name) = 'rafa silva' THEN 1
WHEN LOWER(players.first_name) = 'rafa silva' THEN 2
WHEN LOWER(players.short_name) ILIKE '%rafa silva%' THEN 3
WHEN LOWER(players.last_name) ILIKE '%rafa silva%' THEN 4
WHEN LOWER(players.first_name) ILIKE '%rafa silva%' THEN 5
ELSE 6 END,
CASE WHEN teams.category = 'default' THEN 0
WHEN teams.category = 'youth' THEN 1
ELSE 2 END,
CASE WHEN teams.area_wy_id IN (724, 380, 616, 276, 250) THEN 0
WHEN teams.area_wy_id IN (620, 337, 76, 11, 22) THEN 1
WHEN teams.area_wy_id IN (792, 40, 203, 688, 756, 208, 300, 752, 578) THEN 2
WHEN teams.area_wy_id IS NOT NULL THEN 99
ELSE 100 END,
CASE WHEN players.current_national_team_id IS NOT NULL THEN 0 ELSE 1 END
LIMIT 100;
docs/SYNC_SYSTEM.md 0 → 100644
# Wyscout API Sync System
## Overview
This system provides automated and manual synchronization of data from the Wyscout API using their `updatedobjects` endpoint. It tracks which objects have been updated and syncs them incrementally to keep your database up-to-date.
## Features
- **Incremental Sync**: Only fetch objects that have been updated since the last sync
- **Multiple Resource Types**: Support for players, teams, matches, coaches, referees, competitions, seasons, and areas
- **Cron Scheduling**: Automated hourly syncs
- **Manual Triggers**: CLI and API endpoints for on-demand syncs
- **Sync Logging**: Complete audit trail of all sync operations
- **Error Handling**: Graceful error handling with partial sync support
## API Endpoints
### Trigger Manual Sync
```bash
POST /api/sync/trigger?resource_type=players&hours_back=24
```
**Parameters:**
- `resource_type` (required): Resource type to sync (players, teams, matches, coaches, referees, competitions, seasons, areas)
- `hours_back` (optional): Hours to look back (1-168, default: 24)
**Response:**
```json
{
"id": "abc123xyz456789",
"syncType": "manual",
"resourceType": "players",
"startedAt": "2026-02-03T10:00:00Z",
"completedAt": "2026-02-03T10:05:00Z",
"status": "completed",
"totalRecords": 150,
"processedRecords": 148,
"failedRecords": 2,
"syncParams": {
"updated_since": "2026-02-02 10:00:00",
"type": "players",
"limit": 100,
"page": 1
}
}
```
### Trigger Full Sync (All Resources)
```bash
POST /api/sync/trigger-all?hours_back=24
```
**Parameters:**
- `hours_back` (optional): Hours to look back (1-168, default: 24)
### Get Sync Logs
```bash
GET /api/sync/logs?resource_type=players&status=completed&limit=50
```
**Parameters:**
- `resource_type` (optional): Filter by resource type
- `status` (optional): Filter by status (running, completed, failed, partial)
- `limit` (optional): Limit results (default: 50)
### Get Sync Status
```bash
GET /api/sync/status
```
Returns the current status of the sync service.
### Get Last Sync Time
```bash
GET /api/sync/last-sync?resource_type=players
```
**Parameters:**
- `resource_type` (required): Resource type to check
**Response:**
```json
{
"resource_type": "players",
"last_sync": "2026-02-03T09:00:00Z",
"hours_ago": 1
}
```
## CLI Commands
### Sync a Specific Resource
```bash
go run cmd/sync/main.go -resource=players -hours=24
```
### Sync All Resources
```bash
go run cmd/sync/main.go -all -hours=24
```
### Show Sync Status
```bash
go run cmd/sync/main.go -status
```
Output:
```
=== Sync Status ===
players: Last synced 2 hours ago (2026-02-03 08:00:00)
teams: Last synced 2 hours ago (2026-02-03 08:00:00)
matches: Last synced 2 hours ago (2026-02-03 08:00:00)
...
```
### Show Recent Sync Logs
```bash
go run cmd/sync/main.go -logs
```
Output:
```
=== Recent Sync Logs ===
Resource Status Processed Failed Started
--------------------------------------------------------------------------------
players completed 150 2 2026-02-03 10:00
teams completed 45 0 2026-02-03 09:00
matches partial 200 15 2026-02-03 08:00
```
## Cron Service
The cron service runs automatically when the main server starts and performs hourly incremental syncs.
### Starting the Cron Service
The cron service is integrated into the main server. To enable it, add this to your `cmd/server/main.go`:
```go
import (
"ScoutingSystemScoreData/internal/services"
)
func main() {
// ... existing setup ...
// Start cron service
cronService := services.NewCronService(db, cfg)
go cronService.Start()
// ... rest of server setup ...
}
```
### Cron Schedule
- **Hourly**: Syncs objects updated in the last hour
- **Configurable**: Can be adjusted in `internal/services/cron_service.go`
## Resource Types
The following resource types are supported:
| Resource Type | Description | API Sync Fields |
|--------------|-------------|-----------------|
| `players` | Player data | `api_last_synced_at`, `api_sync_status` |
| `teams` | Team data | `api_last_synced_at`, `api_sync_status` |
| `matches` | Match data | `api_last_synced_at`, `api_sync_status` |
| `coaches` | Coach data | `api_last_synced_at`, `api_sync_status` |
| `referees` | Referee data | `api_last_synced_at`, `api_sync_status` |
| `competitions` | Competition data | `updated_at` |
| `seasons` | Season data | `updated_at` |
| `areas` | Area/country data | `updated_at` |
## Sync Statuses
- **running**: Sync is currently in progress
- **completed**: Sync completed successfully
- **failed**: Sync failed completely
- **partial**: Sync completed with some failures
## API Limitations
- **Max Lookback**: 168 hours (7 days)
- **Rate Limiting**: Respect Wyscout API rate limits
- **Pagination**: 100 records per page (configurable)
## Database Schema
### sync_logs Table
```sql
CREATE TABLE sync_logs (
id VARCHAR(16) PRIMARY KEY,
sync_type VARCHAR(50) NOT NULL, -- incremental, full, manual
resource_type VARCHAR(50) NOT NULL, -- players, teams, etc.
started_at TIMESTAMP WITH TIME ZONE NOT NULL,
completed_at TIMESTAMP WITH TIME ZONE,
status VARCHAR(20) NOT NULL DEFAULT 'running',
total_records INTEGER DEFAULT 0,
processed_records INTEGER DEFAULT 0,
failed_records INTEGER DEFAULT 0,
error_message TEXT,
sync_params JSONB,
created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
updated_at TIMESTAMP WITH TIME ZONE DEFAULT NOW()
);
```
## Configuration
Add these environment variables to your `.env` file:
```env
# Wyscout API Credentials (already configured)
ProviderUser=your_username
ProviderSecret=your_password
# Optional: Sync Configuration
SYNC_ENABLED=true
SYNC_INTERVAL_HOURS=1
SYNC_DEFAULT_LOOKBACK_HOURS=24
```
## Examples
### Daily Sync (Last 24 Hours)
```bash
# Via CLI
go run cmd/sync/main.go -all -hours=24
# Via API
curl -X POST "http://localhost:8080/api/sync/trigger-all?hours_back=24" \
-H "Authorization: Basic <your_credentials>"
```
### Weekly Sync (Max Allowed)
```bash
# Via CLI
go run cmd/sync/main.go -all -hours=168
# Via API
curl -X POST "http://localhost:8080/api/sync/trigger-all?hours_back=168" \
-H "Authorization: Basic <your_credentials>"
```
### Sync Only Players Updated Today
```bash
# Via CLI
go run cmd/sync/main.go -resource=players -hours=24
# Via API
curl -X POST "http://localhost:8080/api/sync/trigger?resource_type=players&hours_back=24" \
-H "Authorization: Basic <your_credentials>"
```
### Check What Needs Syncing
```bash
# Via CLI
go run cmd/sync/main.go -status
# Via API
curl "http://localhost:8080/api/sync/last-sync?resource_type=players" \
-H "Authorization: Basic <your_credentials>"
```
## Monitoring
### Check Recent Sync Activity
```bash
curl "http://localhost:8080/api/sync/logs?limit=20" \
-H "Authorization: Basic <your_credentials>"
```
### Check Failed Syncs
```bash
curl "http://localhost:8080/api/sync/logs?status=failed" \
-H "Authorization: Basic <your_credentials>"
```
### Monitor Specific Resource
```bash
curl "http://localhost:8080/api/sync/logs?resource_type=players&limit=10" \
-H "Authorization: Basic <your_credentials>"
```
## Troubleshooting
### Sync Fails with "API returned status 401"
Check your Wyscout API credentials in the `.env` file:
```env
ProviderUser=your_username
ProviderSecret=your_password
```
### Sync Fails with "cannot go back more than 168 hours"
The Wyscout API only allows looking back 7 days (168 hours). Reduce the `hours_back` parameter.
### Objects Not Found in Database
If the sync reports "needs full fetch", the object doesn't exist in your database yet. You need to run the initial import first using the import endpoints.
### Partial Sync Status
Some records failed to sync. Check the `error_message` field in the sync logs for details.
## Best Practices
1. **Initial Import**: Run a full import first before enabling incremental syncs
2. **Regular Syncs**: Run hourly syncs to stay up-to-date
3. **Monitor Logs**: Regularly check sync logs for failures
4. **Handle Failures**: Investigate and retry failed syncs
5. **Rate Limiting**: Don't run syncs too frequently to avoid API rate limits
6. **Database Backups**: Backup before running large syncs
## Integration with Existing System
The sync system integrates seamlessly with your existing import system:
1. **Initial Import**: Use existing import handlers for bulk data
2. **Incremental Updates**: Use sync system for ongoing updates
3. **Conflict Resolution**: Newer data overwrites older data based on timestamps
## Future Enhancements
- [ ] Full object fetch for missing records
- [ ] Retry mechanism for failed syncs
- [ ] Webhook support for real-time updates
- [ ] Sync scheduling UI
- [ ] Performance metrics and analytics
- [ ] Configurable sync intervals per resource type
docs/swagger.yaml
......@@ -814,6 +814,11 @@ paths:
post:
description: Imports all countries/regions from TheSports football country list
API.
parameters:
- description: Concurrent workers (default 4)
in: query
name: workers
type: integer
responses:
"200":
description: OK
......@@ -853,6 +858,14 @@ paths:
in: query
name: teamWyId
type: integer
- description: Concurrent workers (default 4)
in: query
name: workers
type: integer
- description: Requests per second (default 10)
in: query
name: rps
type: integer
responses:
"200":
description: OK
......@@ -890,6 +903,14 @@ paths:
in: query
name: since
type: integer
- description: Concurrent workers (default 4)
in: query
name: workers
type: integer
- description: Requests per second (default 10)
in: query
name: rps
type: integer
responses:
"200":
description: OK
......@@ -952,6 +973,7 @@ paths:
post:
description: Imports matches for a given Wyscout season via /v3/seasons/{seasonWyId}/fixtures?details=matches.
If seasonWyId is omitted, imports for all seasons in the DB with a wy_id.
Supports incremental sync to update matches from last sync date to today.
parameters:
- description: Wyscout season ID
in: query
......@@ -961,6 +983,19 @@ paths:
in: query
name: limitSeasons
type: integer
- description: If true, only sync seasons with matches between api_last_synced_at
and today based on match_date
in: query
name: syncRecent
type: boolean
- description: Concurrent workers (default 4)
in: query
name: workers
type: integer
- description: Requests per second (default 10)
in: query
name: rps
type: integer
responses:
"200":
description: OK
......@@ -1027,6 +1062,14 @@ paths:
name: seasonWyId
required: true
type: integer
- description: Concurrent workers (default 4)
in: query
name: workers
type: integer
- description: Requests per second (default 10)
in: query
name: rps
type: integer
responses:
"200":
description: OK
......@@ -1064,6 +1107,14 @@ paths:
in: query
name: since
type: integer
- description: Concurrent workers (default 4)
in: query
name: workers
type: integer
- description: Requests per second (default 10)
in: query
name: rps
type: integer
responses:
"200":
description: OK
......@@ -1146,6 +1197,14 @@ paths:
in: query
name: playerWyId
type: integer
- description: Concurrent workers (default 4)
in: query
name: workers
type: integer
- description: Requests per second (default 11)
in: query
name: rps
type: integer
responses:
"200":
description: OK
......@@ -1180,6 +1239,14 @@ paths:
in: query
name: playerWyId
type: integer
- description: Concurrent workers (default 4)
in: query
name: workers
type: integer
- description: Requests per second (default 11)
in: query
name: rps
type: integer
responses:
"200":
description: OK
......@@ -1226,6 +1293,14 @@ paths:
in: query
name: since
type: integer
- description: Concurrent workers (default 4)
in: query
name: workers
type: integer
- description: Requests per second (default 10)
in: query
name: rps
type: integer
responses:
"200":
description: OK
......@@ -1264,6 +1339,14 @@ paths:
in: query
name: batchSize
type: integer
- description: Concurrent workers (default 4)
in: query
name: workers
type: integer
- description: Requests per second (default 10)
in: query
name: rps
type: integer
responses:
"200":
description: OK
......@@ -1301,6 +1384,18 @@ paths:
in: query
name: since
type: integer
- description: Limit number of competitions processed
in: query
name: limit
type: integer
- description: Concurrent workers (default 4)
in: query
name: workers
type: integer
- description: Requests per second (default 10)
in: query
name: rps
type: integer
responses:
"200":
description: OK
......@@ -1313,7 +1408,6 @@ paths:
additionalProperties:
type: string
type: object
summary: Import seasons from TheSports
tags:
- Import
/import/standings:
......@@ -1330,6 +1424,14 @@ paths:
in: query
name: limit
type: integer
- description: Concurrent workers (default 4)
in: query
name: workers
type: integer
- description: Requests per second (default 10)
in: query
name: rps
type: integer
responses:
"200":
description: OK
......@@ -1353,10 +1455,10 @@ paths:
- Import
/import/teams:
post:
description: Performs a team import using TheSports team additional list API.
If `since` is provided (unix seconds), only teams updated since that time
are fetched using the time-based endpoint. Otherwise, a full import is performed
using page-based pagination.
description: Performs a team import using TheSports team list API. If `since`
is provided (unix seconds), only teams updated since that time are fetched
using the time-based endpoint. Otherwise, a full import is performed using
page-based pagination.
parameters:
- description: Page size per request (default 100, only used for full imports)
in: query
......@@ -1367,6 +1469,14 @@ paths:
in: query
name: since
type: integer
- description: Concurrent workers (default 4)
in: query
name: workers
type: integer
- description: Requests per second (default 10)
in: query
name: rps
type: integer
responses:
"200":
description: OK
......@@ -1503,6 +1613,14 @@ paths:
in: query
name: missingOnly
type: boolean
- description: Concurrent workers (default 4)
in: query
name: workers
type: integer
- description: Requests per second (default 10)
in: query
name: rps
type: integer
responses:
"200":
description: OK
......
ecosystem.config.js deleted 100644 → 0
module.exports = {
apps: [
{
name: "sssdata",
cwd: __dirname,
script: "./server",
interpreter: "none",
instances: 1,
autorestart: true,
watch: false,
max_memory_restart: "300M",
time: true,
env: {
APP_PORT: process.env.APP_PORT || "3003",
DB_HOST: process.env.DB_HOST,
DB_PORT: process.env.DB_PORT,
DB_USER: process.env.DB_USER,
DB_PASSWORD: process.env.DB_PASSWORD,
DB_NAME: process.env.DB_NAME,
DB_SSLMODE: process.env.DB_SSLMODE,
ProviderUser: process.env.ProviderUser,
ProviderSecret: process.env.ProviderSecret,
API_USERNAME: process.env.API_USERNAME,
API_PASSWORD: process.env.API_PASSWORD,
},
},
],
};
internal/config/league_tiers.go 0 → 100644
package config
import "fmt"
// LeagueTiers defines the hierarchical ranking of football leagues worldwide
// Based on UEFA Country Coefficients 2024/2025 and global football rankings
// Last updated: February 2026
//
// Usage: Can be used for prioritizing search results, filtering, and ranking
// across players, coaches, teams, and other football entities.
// LeagueTier represents a tier level with associated countries
type LeagueTier struct {
Level int // 1 = highest tier
Name string // Human-readable tier name
Description string // Description of the tier
AreaWyIDs []int // Wyscout area IDs for countries in this tier
Countries []string // Country names for reference
}
// GetLeagueTiers returns all defined league tiers
func GetLeagueTiers() []LeagueTier {
return []LeagueTier{
{
Level: 1,
Name: "Elite (Top 5)",
Description: "The Big 5 European leagues - highest quality and commercial value",
AreaWyIDs: []int{724, 380, 616, 276, 250}, // England, Italy, France, Germany, Spain
Countries: []string{"England", "Spain", "Italy", "Germany", "France"},
},
{
Level: 2,
Name: "Strong Leagues",
Description: "High-quality leagues with international competitiveness",
AreaWyIDs: []int{620, 337, 76, 11, 22}, // Portugal, Netherlands, Brazil, Argentina, Belgium
Countries: []string{"Portugal", "Netherlands", "Brazil", "Argentina", "Belgium"},
},
{
Level: 3,
Name: "Competitive Leagues",
Description: "Competitive leagues with regular European participation",
AreaWyIDs: []int{792, 40, 203, 688, 756, 208, 300, 752, 578}, // Turkey, Austria, Czech Republic, Serbia, Switzerland, Denmark, Greece, Sweden, Norway
Countries: []string{"Turkey", "Austria", "Czech Republic", "Serbia", "Switzerland", "Denmark", "Greece", "Sweden", "Norway"},
},
{
Level: 4,
Name: "Developing Leagues",
Description: "Emerging leagues with growing competitiveness",
AreaWyIDs: []int{}, // Can be expanded as needed
Countries: []string{},
},
}
}
// GetAreaWyIDsByTier returns area WyIDs for a specific tier level
func GetAreaWyIDsByTier(tier int) []int {
tiers := GetLeagueTiers()
for _, t := range tiers {
if t.Level == tier {
return t.AreaWyIDs
}
}
return []int{}
}
// GetAllTopTierAreaWyIDs returns area WyIDs for tiers 1-3 combined
func GetAllTopTierAreaWyIDs() []int {
var allIDs []int
tiers := GetLeagueTiers()
for _, t := range tiers {
if t.Level <= 3 {
allIDs = append(allIDs, t.AreaWyIDs...)
}
}
return allIDs
}
// BuildLeagueTierCaseSQL generates a SQL CASE statement for league tier prioritization
// Returns the SQL string for use in ORDER BY clauses
func BuildLeagueTierCaseSQL(teamTableAlias string) string {
if teamTableAlias != "" {
teamTableAlias = teamTableAlias + "."
}
tiers := GetLeagueTiers()
sql := "CASE "
for _, tier := range tiers {
if len(tier.AreaWyIDs) > 0 {
sql += "WHEN " + teamTableAlias + "area_wy_id IN ("
for i, id := range tier.AreaWyIDs {
if i > 0 {
sql += ", "
}
sql += fmt.Sprintf("%d", id)
}
sql += ") THEN " + fmt.Sprintf("%d ", tier.Level-1) // 0-indexed for sorting
}
}
sql += "WHEN " + teamTableAlias + "area_wy_id IS NOT NULL THEN 99 " // Other leagues
sql += "ELSE 100 END" // No team/league
return sql
}
// GetTierByAreaWyID returns the tier level for a given area WyID
// Returns 0 if not found in any tier
func GetTierByAreaWyID(areaWyID int) int {
tiers := GetLeagueTiers()
for _, tier := range tiers {
for _, id := range tier.AreaWyIDs {
if id == areaWyID {
return tier.Level
}
}
}
return 0
}
// GetTierName returns the human-readable name for a tier level
func GetTierName(tier int) string {
tiers := GetLeagueTiers()
for _, t := range tiers {
if t.Level == tier {
return t.Name
}
}
return "Unknown"
}
internal/handlers/import_worker.go 0 → 100644
package handlers
import (
"context"
"log"
"sync"
"sync/atomic"
"time"
"gorm.io/gorm"
"gorm.io/gorm/clause"
"ScoutingSystemScoreData/internal/models"
)
// ImportWorkerConfig holds configuration for a worker-based import
type ImportWorkerConfig struct {
Name string
Workers int
BatchSize int
RPS int
Limit int
Reset bool
CheckpointKey string
LogProgressEvery int64
}
// ImportProgress tracks progress for logging
type ImportProgress struct {
StartedAt time.Time
ProcessedThisRun int64
ErrorsThisRun int64
Remaining int64
}
// LogProgress logs import progress with RPS and ETA
func (p *ImportProgress) LogProgress(name string, force bool, totalProcessed int64, lastID string) {
if p.Remaining <= 0 {
return
}
elapsed := time.Since(p.StartedAt)
if elapsed <= 0 {
elapsed = time.Second
}
rpsNow := float64(p.ProcessedThisRun) / elapsed.Seconds()
pct := (float64(p.ProcessedThisRun) / float64(p.Remaining)) * 100
remainingLeft := p.Remaining - p.ProcessedThisRun
eta := time.Duration(0)
if rpsNow > 0 && remainingLeft > 0 {
eta = time.Duration(float64(remainingLeft)/rpsNow) * time.Second
}
if force || p.ProcessedThisRun%2000 == 0 {
log.Printf(
"%s: progress processed=%d/%d (%.2f%%) errorsThisRun=%d totalProcessed=%d elapsed=%s rps=%.2f eta=%s last=%s",
name,
p.ProcessedThisRun,
p.Remaining,
pct,
p.ErrorsThisRun,
totalProcessed,
elapsed.Truncate(time.Second),
rpsNow,
eta.Truncate(time.Second),
lastID,
)
}
}
// RateLimiter creates a rate-limited token channel
func RateLimiter(ctx context.Context, rps int) chan struct{} {
rateTokens := make(chan struct{}, rps)
interval := time.Second / time.Duration(rps)
if interval <= 0 {
interval = time.Second
}
ticker := time.NewTicker(interval)
go func() {
defer ticker.Stop()
for {
select {
case <-ctx.Done():
return
case <-ticker.C:
select {
case rateTokens <- struct{}{}:
default:
}
}
}
}()
return rateTokens
}
// PersistCheckpoint saves checkpoint to database
func PersistCheckpoint(ctx context.Context, db *gorm.DB, ck *models.ImportCheckpoint) error {
now := time.Now().UTC()
ck.UpdatedAt = now
return db.WithContext(ctx).
Clauses(clause.OnConflict{Columns: []clause.Column{{Name: "key"}}, UpdateAll: true}).
Create(ck).Error
}
// LoadCheckpoint loads or creates a checkpoint
func LoadCheckpoint(ctx context.Context, db *gorm.DB, key string, reset bool) (*models.ImportCheckpoint, error) {
var ck models.ImportCheckpoint
if err := db.WithContext(ctx).Where("key = ?", key).First(&ck).Error; err != nil {
if err == gorm.ErrRecordNotFound {
ck = models.ImportCheckpoint{Key: key}
} else {
return nil, err
}
}
if reset {
ck.LastPlayerWyID = 0
ck.LastTeamWyID = 0
ck.LastMatchWyID = 0
ck.LastCompetitionID = 0
ck.LastSeasonID = 0
ck.Processed = 0
ck.Errors = 0
}
return &ck, nil
}
// WorkerPool manages a pool of workers for concurrent processing
type WorkerPool[T any, R any] struct {
Workers int
Jobs chan T
Results chan R
wg sync.WaitGroup
}
// NewWorkerPool creates a new worker pool
func NewWorkerPool[T any, R any](workers int) *WorkerPool[T, R] {
return &WorkerPool[T, R]{
Workers: workers,
Jobs: make(chan T, workers*4),
Results: make(chan R, workers*4),
}
}
// Start starts the worker pool with the given work function
func (wp *WorkerPool[T, R]) Start(workFn func(T) R) {
for i := 0; i < wp.Workers; i++ {
wp.wg.Add(1)
go func() {
defer wp.wg.Done()
for job := range wp.Jobs {
result := workFn(job)
wp.Results <- result
}
}()
}
go func() {
wp.wg.Wait()
close(wp.Results)
}()
}
// Close closes the jobs channel
func (wp *WorkerPool[T, R]) Close() {
close(wp.Jobs)
}
// AsyncImportStatus tracks status for async imports
type AsyncImportStatus struct {
running atomic.Bool
startedAt atomic.Int64
finishedAt atomic.Int64
total atomic.Int64
processed atomic.Int64
created atomic.Int64
errors atomic.Int64
lastError atomic.Value
}
// GetStatus returns the current status as a map
func (s *AsyncImportStatus) GetStatus() map[string]interface{} {
startedAt := s.startedAt.Load()
finishedAt := s.finishedAt.Load()
resp := map[string]interface{}{
"running": s.running.Load(),
"total": s.total.Load(),
"processed": s.processed.Load(),
"created": s.created.Load(),
"errors": s.errors.Load(),
}
if total := s.total.Load(); total > 0 {
p := s.processed.Load()
remaining := total - p
if remaining < 0 {
remaining = 0
}
resp["remaining"] = remaining
}
if startedAt > 0 {
resp["startedAt"] = time.Unix(0, startedAt).UTC().Format(time.RFC3339)
}
if finishedAt > 0 {
resp["finishedAt"] = time.Unix(0, finishedAt).UTC().Format(time.RFC3339)
}
if v := s.lastError.Load(); v != nil {
if errStr, ok := v.(string); ok && errStr != "" {
resp["lastError"] = errStr
}
}
return resp
}
// Start marks the import as started
func (s *AsyncImportStatus) Start(total int64) {
s.running.Store(true)
s.startedAt.Store(time.Now().UTC().UnixNano())
s.finishedAt.Store(0)
s.total.Store(total)
s.processed.Store(0)
s.created.Store(0)
s.errors.Store(0)
s.lastError.Store("")
}
// Finish marks the import as finished
func (s *AsyncImportStatus) Finish(err error) {
s.running.Store(false)
s.finishedAt.Store(time.Now().UTC().UnixNano())
if err != nil {
s.lastError.Store(err.Error())
}
}
// Update updates progress counters
func (s *AsyncImportStatus) Update(processed, created, errors int64) {
s.processed.Store(processed)
s.created.Store(created)
s.errors.Store(errors)
}
internal/handlers/players.go
......@@ -857,6 +857,8 @@ func valueOrDefault(s *string, def string) string {
// @Failure 500 {object} map[string]string
// @Router /players [get]
func (h *PlayerHandler) List(c *gin.Context) {
startTime := time.Now()
limitStr := c.DefaultQuery("limit", "100")
offsetStr := c.DefaultQuery("offset", "0")
......@@ -1081,17 +1083,19 @@ func (h *PlayerHandler) List(c *gin.Context) {
page := offset/limit + 1
hasMore := int64(offset+limit) < total
timestamp := time.Now().UTC().Format(time.RFC3339)
responseTime := time.Since(startTime).Milliseconds()
c.JSON(http.StatusOK, gin.H{
"data": structured,
"meta": gin.H{
"timestamp": timestamp,
"endpoint": endpoint,
"method": "GET",
"totalItems": total,
"page": page,
"limit": limit,
"hasMore": hasMore,
"timestamp": timestamp,
"endpoint": endpoint,
"method": "GET",
"totalItems": total,
"page": page,
"limit": limit,
"hasMore": hasMore,
"responseTimeMs": responseTime,
},
})
}
......
internal/handlers/sync_handler.go 0 → 100644
package handlers
import (
"ScoutingSystemScoreData/internal/config"
"ScoutingSystemScoreData/internal/models"
"ScoutingSystemScoreData/internal/services"
"net/http"
"strconv"
"time"
"github.com/gin-gonic/gin"
"gorm.io/gorm"
)
type SyncHandler struct {
syncService *services.SyncService
cronService *services.CronService
}
func NewSyncHandler(db *gorm.DB, cfg config.Config) *SyncHandler {
return &SyncHandler{
syncService: services.NewSyncService(db, cfg),
cronService: services.NewCronService(db, cfg),
}
}
// TriggerSync godoc
// @Summary Trigger a manual sync for a specific resource type
// @Description Triggers a manual sync for updated objects from Wyscout API
// @Tags Sync
// @Accept json
// @Produce json
// @Param resource_type query string true "Resource type (players, teams, matches, etc.)"
// @Param hours_back query int false "Hours to look back (max 168)" default(24)
// @Success 200 {object} models.SyncLog
// @Failure 400 {object} map[string]string
// @Failure 500 {object} map[string]string
// @Router /sync/trigger [post]
func (h *SyncHandler) TriggerSync(c *gin.Context) {
resourceType := c.Query("resource_type")
if resourceType == "" {
c.JSON(http.StatusBadRequest, gin.H{"error": "resource_type is required"})
return
}
hoursBackStr := c.DefaultQuery("hours_back", "24")
hoursBack, err := strconv.Atoi(hoursBackStr)
if err != nil || hoursBack < 1 || hoursBack > 168 {
c.JSON(http.StatusBadRequest, gin.H{"error": "hours_back must be between 1 and 168"})
return
}
updatedSince := time.Now().Add(-time.Duration(hoursBack) * time.Hour)
syncLog, err := h.syncService.SyncResourceType(resourceType, updatedSince, "manual")
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
return
}
c.JSON(http.StatusOK, syncLog)
}
// TriggerFullSync godoc
// @Summary Trigger a full sync for all resource types
// @Description Triggers a sync for all resource types from Wyscout API
// @Tags Sync
// @Accept json
// @Produce json
// @Param hours_back query int false "Hours to look back (max 168)" default(24)
// @Success 200 {object} map[string]string
// @Failure 400 {object} map[string]string
// @Failure 500 {object} map[string]string
// @Router /sync/trigger-all [post]
func (h *SyncHandler) TriggerFullSync(c *gin.Context) {
hoursBackStr := c.DefaultQuery("hours_back", "24")
hoursBack, err := strconv.Atoi(hoursBackStr)
if err != nil || hoursBack < 1 || hoursBack > 168 {
c.JSON(http.StatusBadRequest, gin.H{"error": "hours_back must be between 1 and 168"})
return
}
updatedSince := time.Now().Add(-time.Duration(hoursBack) * time.Hour)
if err := h.syncService.SyncAllResources(updatedSince); err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
return
}
c.JSON(http.StatusOK, gin.H{
"message": "Full sync triggered successfully",
"updated_since": updatedSince.Format("2006-01-02 15:04:05"),
})
}
// GetSyncLogs godoc
// @Summary Get sync logs
// @Description Retrieves sync logs with optional filtering
// @Tags Sync
// @Accept json
// @Produce json
// @Param resource_type query string false "Filter by resource type"
// @Param status query string false "Filter by status"
// @Param limit query int false "Limit results" default(50)
// @Success 200 {array} models.SyncLog
// @Failure 500 {object} map[string]string
// @Router /sync/logs [get]
func (h *SyncHandler) GetSyncLogs(c *gin.Context) {
var logs []models.SyncLog
query := h.syncService.GetDB().Model(&models.SyncLog{})
if resourceType := c.Query("resource_type"); resourceType != "" {
query = query.Where("resource_type = ?", resourceType)
}
if status := c.Query("status"); status != "" {
query = query.Where("status = ?", status)
}
limitStr := c.DefaultQuery("limit", "50")
limit, _ := strconv.Atoi(limitStr)
if err := query.Order("started_at DESC").Limit(limit).Find(&logs).Error; err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
return
}
c.JSON(http.StatusOK, logs)
}
// GetSyncStatus godoc
// @Summary Get sync status
// @Description Returns the current status of the sync service
// @Tags Sync
// @Accept json
// @Produce json
// @Success 200 {object} map[string]interface{}
// @Router /sync/status [get]
func (h *SyncHandler) GetSyncStatus(c *gin.Context) {
status := h.cronService.GetSyncStatus()
c.JSON(http.StatusOK, status)
}
// GetLastSyncTime godoc
// @Summary Get last sync time for a resource type
// @Description Returns the last successful sync time for a specific resource type
// @Tags Sync
// @Accept json
// @Produce json
// @Param resource_type query string true "Resource type"
// @Success 200 {object} map[string]interface{}
// @Failure 400 {object} map[string]string
// @Failure 500 {object} map[string]string
// @Router /sync/last-sync [get]
func (h *SyncHandler) GetLastSyncTime(c *gin.Context) {
resourceType := c.Query("resource_type")
if resourceType == "" {
c.JSON(http.StatusBadRequest, gin.H{"error": "resource_type is required"})
return
}
lastSync, err := h.syncService.GetLastSyncTime(resourceType)
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
return
}
c.JSON(http.StatusOK, gin.H{
"resource_type": resourceType,
"last_sync": lastSync.Format(time.RFC3339),
"hours_ago": int(time.Since(lastSync).Hours()),
})
}
internal/handlers/sync_routes.go 0 → 100644
package handlers
import (
"ScoutingSystemScoreData/internal/config"
"github.com/gin-gonic/gin"
"gorm.io/gorm"
)
func RegisterSyncRoutes(rg *gin.RouterGroup, db *gorm.DB, cfg config.Config) {
handler := NewSyncHandler(db, cfg)
sync := rg.Group("/sync")
{
sync.POST("/trigger", handler.TriggerSync)
sync.POST("/trigger-all", handler.TriggerFullSync)
sync.GET("/logs", handler.GetSyncLogs)
sync.GET("/status", handler.GetSyncStatus)
sync.GET("/last-sync", handler.GetLastSyncTime)
}
}
internal/handlers/teams.go
......@@ -586,6 +586,8 @@ func (h *TeamHandler) GetImagesByID(c *gin.Context) {
// @Failure 500 {object} map[string]string
// @Router /teams [get]
func (h *TeamHandler) List(c *gin.Context) {
startTime := time.Now()
limitStr := c.DefaultQuery("limit", "100")
offsetStr := c.DefaultQuery("offset", "0")
......@@ -715,17 +717,19 @@ func (h *TeamHandler) List(c *gin.Context) {
page := offset/limit + 1
hasMore := int64(offset+limit) < total
timestamp := time.Now().UTC().Format(time.RFC3339)
responseTime := time.Since(startTime).Milliseconds()
c.JSON(http.StatusOK, gin.H{
"data": withArea,
"meta": gin.H{
"timestamp": timestamp,
"endpoint": endpoint,
"method": "GET",
"totalItems": total,
"page": page,
"limit": limit,
"hasMore": hasMore,
"timestamp": timestamp,
"endpoint": endpoint,
"method": "GET",
"totalItems": total,
"page": page,
"limit": limit,
"hasMore": hasMore,
"responseTimeMs": responseTime,
},
})
}
......
internal/models/sync.go 0 → 100644
package models
import (
"encoding/json"
"time"
gonanoid "github.com/matoous/go-nanoid/v2"
"gorm.io/gorm"
)
type SyncLog struct {
ID string `gorm:"primaryKey;size:16" json:"id"`
SyncType string `gorm:"column:sync_type;size:50;index" json:"syncType"`
ResourceType string `gorm:"column:resource_type;size:50;index" json:"resourceType"`
StartedAt time.Time `gorm:"column:started_at;index" json:"startedAt"`
CompletedAt *time.Time `gorm:"column:completed_at" json:"completedAt"`
Status string `gorm:"column:status;size:20;default:running;index" json:"status"`
TotalRecords int `gorm:"column:total_records;default:0" json:"totalRecords"`
ProcessedRecords int `gorm:"column:processed_records;default:0" json:"processedRecords"`
FailedRecords int `gorm:"column:failed_records;default:0" json:"failedRecords"`
ErrorMessage *string `gorm:"column:error_message;type:text" json:"errorMessage"`
SyncParams json.RawMessage `gorm:"column:sync_params;type:jsonb" json:"syncParams" swaggertype:"object"`
CreatedAt time.Time `gorm:"column:created_at" json:"createdAt"`
UpdatedAt time.Time `gorm:"column:updated_at" json:"updatedAt"`
}
func (s *SyncLog) BeforeCreate(tx *gorm.DB) (err error) {
if s.ID != "" {
return nil
}
id, err := gonanoid.Generate("abcdefghijklmnopqrstuvwxyz0123456789", 15)
if err != nil {
return err
}
s.ID = id
return nil
}
type UpdatedObjectsResponse struct {
Total int `json:"total"`
Data []UpdatedObjectReference `json:"data"`
}
type UpdatedObjectReference struct {
ID int `json:"id"`
Payload interface{} `json:"payload,omitempty"`
}
type SyncParams struct {
UpdatedSince string `json:"updated_since"`
Type string `json:"type"`
Limit int `json:"limit"`
Page int `json:"page"`
EmptyPayload bool `json:"empty_payload"`
}
internal/router/router.go
......@@ -131,6 +131,7 @@ func New(db *gorm.DB) *gin.Engine {
handlers.RegisterMatchRoutes(api, db)
handlers.RegisterStandingRoutes(api, db)
handlers.RegisterImportRoutes(api, db, appCfg)
handlers.RegisterSyncRoutes(api, db, appCfg)
return r
}
internal/services/cron_service.go 0 → 100644
package services
import (
"ScoutingSystemScoreData/internal/config"
"fmt"
"log"
"time"
"gorm.io/gorm"
)
type CronService struct {
db *gorm.DB
config config.Config
syncService *SyncService
stopChan chan bool
running bool
}
func NewCronService(db *gorm.DB, cfg config.Config) *CronService {
return &CronService{
db: db,
config: cfg,
syncService: NewSyncService(db, cfg),
stopChan: make(chan bool),
running: false,
}
}
// Start begins the cron scheduler
func (c *CronService) Start() {
if c.running {
log.Println("Cron service is already running")
return
}
c.running = true
log.Println("Starting cron service...")
// Run initial sync on startup (optional)
// c.runIncrementalSync()
// Start ticker for hourly syncs
ticker := time.NewTicker(1 * time.Hour)
defer ticker.Stop()
for {
select {
case <-ticker.C:
c.runIncrementalSync()
case <-c.stopChan:
log.Println("Stopping cron service...")
c.running = false
return
}
}
}
// Stop stops the cron scheduler
func (c *CronService) Stop() {
if c.running {
c.stopChan <- true
}
}
// runIncrementalSync runs an incremental sync for all resources
func (c *CronService) runIncrementalSync() {
log.Println("Starting incremental sync...")
// Get the time from 1 hour ago (to catch any updates in the last hour)
updatedSince := time.Now().Add(-1 * time.Hour)
resourceTypes := []string{
"players",
"teams",
"matches",
"coaches",
"referees",
"competitions",
"seasons",
}
for _, resourceType := range resourceTypes {
log.Printf("Syncing %s...", resourceType)
syncLog, err := c.syncService.SyncResourceType(resourceType, updatedSince, "incremental")
if err != nil {
log.Printf("Error syncing %s: %v", resourceType, err)
continue
}
log.Printf("Completed sync for %s: %d processed, %d failed",
resourceType, syncLog.ProcessedRecords, syncLog.FailedRecords)
}
log.Println("Incremental sync completed")
}
// RunDailySync runs a daily sync (last 24 hours)
func (c *CronService) RunDailySync() {
log.Println("Starting daily sync...")
updatedSince := time.Now().Add(-24 * time.Hour)
if err := c.syncService.SyncAllResources(updatedSince); err != nil {
log.Printf("Error during daily sync: %v", err)
}
log.Println("Daily sync completed")
}
// RunWeeklySync runs a weekly sync (last 7 days - max allowed by API)
func (c *CronService) RunWeeklySync() {
log.Println("Starting weekly sync...")
// API allows max 168 hours (7 days) lookback
updatedSince := time.Now().Add(-168 * time.Hour)
if err := c.syncService.SyncAllResources(updatedSince); err != nil {
log.Printf("Error during weekly sync: %v", err)
}
log.Println("Weekly sync completed")
}
// ScheduleCustomSync schedules a custom sync with specific parameters
func (c *CronService) ScheduleCustomSync(resourceType string, hoursBack int) error {
if hoursBack > 168 {
return fmt.Errorf("cannot go back more than 168 hours (API limitation)")
}
updatedSince := time.Now().Add(-time.Duration(hoursBack) * time.Hour)
log.Printf("Starting custom sync for %s (last %d hours)...", resourceType, hoursBack)
syncLog, err := c.syncService.SyncResourceType(resourceType, updatedSince, "manual")
if err != nil {
return fmt.Errorf("custom sync failed: %w", err)
}
log.Printf("Custom sync completed: %d processed, %d failed",
syncLog.ProcessedRecords, syncLog.FailedRecords)
return nil
}
// GetSyncStatus returns the current sync status
func (c *CronService) GetSyncStatus() map[string]interface{} {
return map[string]interface{}{
"running": c.running,
"last_check": time.Now().Format(time.RFC3339),
}
}
internal/services/sync_service.go 0 → 100644
package services
import (
"ScoutingSystemScoreData/internal/config"
"ScoutingSystemScoreData/internal/models"
"encoding/base64"
"encoding/json"
"fmt"
"io"
"net/http"
"time"
"gorm.io/gorm"
)
type SyncService struct {
db *gorm.DB
config config.Config
client *http.Client
}
func NewSyncService(db *gorm.DB, cfg config.Config) *SyncService {
return &SyncService{
db: db,
config: cfg,
client: &http.Client{Timeout: 30 * time.Second},
}
}
// GetUpdatedObjects fetches updated objects from Wyscout API
func (s *SyncService) GetUpdatedObjects(resourceType string, updatedSince time.Time, limit int, page int, emptyPayload bool) (*models.UpdatedObjectsResponse, error) {
baseURL := "https://apirest.wyscout.com/v4/updatedobjects/updated"
// Format time as required: YYYY-MM-DD HH:MM:SS
timeStr := updatedSince.Format("2006-01-02 15:04:05")
url := fmt.Sprintf("%s?updated_since=%s&type=%s&limit=%d&page=%d",
baseURL, timeStr, resourceType, limit, page)
if emptyPayload {
url += "&emptyPayload=true"
}
req, err := http.NewRequest("GET", url, nil)
if err != nil {
return nil, fmt.Errorf("failed to create request: %w", err)
}
// Add Basic Auth
auth := base64.StdEncoding.EncodeToString([]byte(s.config.ProviderUser + ":" + s.config.ProviderSecret))
req.Header.Add("Authorization", "Basic "+auth)
req.Header.Add("Accept", "application/json")
resp, err := s.client.Do(req)
if err != nil {
return nil, fmt.Errorf("failed to execute request: %w", err)
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
body, _ := io.ReadAll(resp.Body)
return nil, fmt.Errorf("API returned status %d: %s", resp.StatusCode, string(body))
}
body, err := io.ReadAll(resp.Body)
if err != nil {
return nil, fmt.Errorf("failed to read response body: %w", err)
}
var result models.UpdatedObjectsResponse
if err := json.Unmarshal(body, &result); err != nil {
return nil, fmt.Errorf("failed to unmarshal response: %w", err)
}
return &result, nil
}
// SyncResourceType syncs a specific resource type (players, teams, matches, etc.)
func (s *SyncService) SyncResourceType(resourceType string, updatedSince time.Time, syncType string) (*models.SyncLog, error) {
// Create sync log
syncParams := models.SyncParams{
UpdatedSince: updatedSince.Format("2006-01-02 15:04:05"),
Type: resourceType,
Limit: 100,
Page: 1,
EmptyPayload: false,
}
paramsJSON, _ := json.Marshal(syncParams)
syncLog := &models.SyncLog{
SyncType: syncType,
ResourceType: resourceType,
StartedAt: time.Now(),
Status: "running",
SyncParams: paramsJSON,
}
if err := s.db.Create(syncLog).Error; err != nil {
return nil, fmt.Errorf("failed to create sync log: %w", err)
}
// Fetch all pages
totalProcessed := 0
totalFailed := 0
page := 1
for {
response, err := s.GetUpdatedObjects(resourceType, updatedSince, 100, page, false)
if err != nil {
errMsg := err.Error()
syncLog.Status = "failed"
syncLog.ErrorMessage = &errMsg
completedAt := time.Now()
syncLog.CompletedAt = &completedAt
s.db.Save(syncLog)
return syncLog, err
}
if len(response.Data) == 0 {
break
}
// Process each updated object
for _, obj := range response.Data {
if err := s.processUpdatedObject(resourceType, obj); err != nil {
totalFailed++
} else {
totalProcessed++
}
}
// Update sync log progress
syncLog.ProcessedRecords = totalProcessed
syncLog.FailedRecords = totalFailed
syncLog.TotalRecords = response.Total
s.db.Save(syncLog)
// Check if we've processed all records
if totalProcessed+totalFailed >= response.Total {
break
}
page++
}
// Mark sync as completed
completedAt := time.Now()
syncLog.CompletedAt = &completedAt
syncLog.Status = "completed"
if totalFailed > 0 && totalProcessed > 0 {
syncLog.Status = "partial"
} else if totalFailed > 0 {
syncLog.Status = "failed"
}
s.db.Save(syncLog)
return syncLog, nil
}
// processUpdatedObject processes a single updated object based on resource type
func (s *SyncService) processUpdatedObject(resourceType string, obj models.UpdatedObjectReference) error {
switch resourceType {
case "players":
return s.syncPlayer(obj.ID)
case "teams":
return s.syncTeam(obj.ID)
case "matches":
return s.syncMatch(obj.ID)
case "coaches":
return s.syncCoach(obj.ID)
case "referees":
return s.syncReferee(obj.ID)
case "competitions":
return s.syncCompetition(obj.ID)
case "seasons":
return s.syncSeason(obj.ID)
case "areas":
return s.syncArea(obj.ID)
default:
return fmt.Errorf("unsupported resource type: %s", resourceType)
}
}
// Individual sync methods for each resource type
func (s *SyncService) syncPlayer(wyID int) error {
// Update the player's api_last_synced_at timestamp
now := time.Now()
result := s.db.Model(&models.Player{}).
Where("wy_id = ?", wyID).
Updates(map[string]interface{}{
"api_last_synced_at": now,
"api_sync_status": "synced",
})
if result.Error != nil {
return fmt.Errorf("failed to update player %d: %w", wyID, result.Error)
}
// If player doesn't exist, mark for full fetch
if result.RowsAffected == 0 {
// TODO: Trigger full player fetch from API
return fmt.Errorf("player %d not found in database, needs full fetch", wyID)
}
return nil
}
func (s *SyncService) syncTeam(wyID int) error {
now := time.Now()
result := s.db.Model(&models.Team{}).
Where("wy_id = ?", wyID).
Updates(map[string]interface{}{
"api_last_synced_at": now,
"api_sync_status": "synced",
})
if result.Error != nil {
return fmt.Errorf("failed to update team %d: %w", wyID, result.Error)
}
if result.RowsAffected == 0 {
return fmt.Errorf("team %d not found in database, needs full fetch", wyID)
}
return nil
}
func (s *SyncService) syncMatch(wyID int) error {
now := time.Now()
result := s.db.Model(&models.Match{}).
Where("wy_id = ?", wyID).
Updates(map[string]interface{}{
"api_last_synced_at": now,
"api_sync_status": "synced",
})
if result.Error != nil {
return fmt.Errorf("failed to update match %d: %w", wyID, result.Error)
}
if result.RowsAffected == 0 {
return fmt.Errorf("match %d not found in database, needs full fetch", wyID)
}
return nil
}
func (s *SyncService) syncCoach(wyID int) error {
now := time.Now()
result := s.db.Model(&models.Coach{}).
Where("wy_id = ?", wyID).
Updates(map[string]interface{}{
"api_last_synced_at": now,
"api_sync_status": "synced",
})
if result.Error != nil {
return fmt.Errorf("failed to update coach %d: %w", wyID, result.Error)
}
if result.RowsAffected == 0 {
return fmt.Errorf("coach %d not found in database, needs full fetch", wyID)
}
return nil
}
func (s *SyncService) syncReferee(wyID int) error {
now := time.Now()
result := s.db.Model(&models.Referee{}).
Where("wy_id = ?", wyID).
Updates(map[string]interface{}{
"api_last_synced_at": now,
"api_sync_status": "synced",
})
if result.Error != nil {
return fmt.Errorf("failed to update referee %d: %w", wyID, result.Error)
}
if result.RowsAffected == 0 {
return fmt.Errorf("referee %d not found in database, needs full fetch", wyID)
}
return nil
}
func (s *SyncService) syncCompetition(wyID int) error {
// Competitions don't have api_last_synced_at, just update updated_at
result := s.db.Model(&models.Competition{}).
Where("wy_id = ?", wyID).
Update("updated_at", time.Now())
if result.Error != nil {
return fmt.Errorf("failed to update competition %d: %w", wyID, result.Error)
}
if result.RowsAffected == 0 {
return fmt.Errorf("competition %d not found in database, needs full fetch", wyID)
}
return nil
}
func (s *SyncService) syncSeason(wyID int) error {
result := s.db.Model(&models.Season{}).
Where("wy_id = ?", wyID).
Update("updated_at", time.Now())
if result.Error != nil {
return fmt.Errorf("failed to update season %d: %w", wyID, result.Error)
}
if result.RowsAffected == 0 {
return fmt.Errorf("season %d not found in database, needs full fetch", wyID)
}
return nil
}
func (s *SyncService) syncArea(wyID int) error {
result := s.db.Model(&models.Area{}).
Where("wy_id = ?", wyID).
Update("updated_at", time.Now())
if result.Error != nil {
return fmt.Errorf("failed to update area %d: %w", wyID, result.Error)
}
if result.RowsAffected == 0 {
return fmt.Errorf("area %d not found in database, needs full fetch", wyID)
}
return nil
}
// GetLastSyncTime returns the last successful sync time for a resource type
func (s *SyncService) GetLastSyncTime(resourceType string) (time.Time, error) {
var syncLog models.SyncLog
err := s.db.Where("resource_type = ? AND status = ?", resourceType, "completed").
Order("completed_at DESC").
First(&syncLog).Error
if err != nil {
if err == gorm.ErrRecordNotFound {
// Return 1 week ago as default
return time.Now().Add(-168 * time.Hour), nil
}
return time.Time{}, err
}
if syncLog.CompletedAt != nil {
return *syncLog.CompletedAt, nil
}
return time.Now().Add(-168 * time.Hour), nil
}
// SyncAllResources syncs all supported resource types
func (s *SyncService) SyncAllResources(updatedSince time.Time) error {
resourceTypes := []string{
"players",
"teams",
"matches",
"coaches",
"referees",
"competitions",
"seasons",
"areas",
}
for _, resourceType := range resourceTypes {
if _, err := s.SyncResourceType(resourceType, updatedSince, "incremental"); err != nil {
// Log error but continue with other resources
fmt.Printf("Error syncing %s: %v\n", resourceType, err)
}
}
return nil
}
// GetDB exposes the database instance for query building
func (s *SyncService) GetDB() *gorm.DB {
return s.db
}
internal/services/team_service.go
......@@ -6,9 +6,12 @@ import (
"strings"
"gorm.io/gorm"
"gorm.io/gorm/clause"
"ScoutingSystemScoreData/internal/config"
"ScoutingSystemScoreData/internal/errors"
"ScoutingSystemScoreData/internal/models"
"ScoutingSystemScoreData/internal/utils"
)
type TeamService interface {
......@@ -32,8 +35,77 @@ func NewTeamService(db *gorm.DB) TeamService {
func (s *teamService) ListTeams(ctx context.Context, limit, offset int, name, gender, teamType string) ([]models.Team, int64, error) {
var teams []models.Team
query := s.db.WithContext(ctx).Model(&models.Team{})
if name != "" {
query = query.Where("name ILIKE ?", "%"+name+"%")
// Normalize search term to lowercase for index-friendly queries
searchLower := strings.ToLower(strings.TrimSpace(name))
likePattern := "%" + searchLower + "%"
searchTokens := utils.TokenizeSearchTerm(name)
// Two-tier search strategy:
// 1. Fast search using LOWER() with trigram indexes
// 2. Fallback to unaccent() for accent-insensitive search
// Tier 1: Fast indexed search on name, short_name, and official_name
searchConditions := s.db.Where(
"LOWER(teams.name) ILIKE ? OR "+
"LOWER(teams.short_name) ILIKE ? OR "+
"LOWER(teams.official_name) ILIKE ?",
likePattern, likePattern, likePattern,
)
// For multi-word searches (e.g., "sporting de braga", "real madrid")
// Each token must match somewhere in name/short_name/official_name
if len(searchTokens) > 1 {
tokenFilter := s.db
for _, token := range searchTokens {
tokenPattern := "%" + token + "%"
tokenFilter = tokenFilter.Where(
"LOWER(teams.name) ILIKE ? OR "+
"LOWER(teams.short_name) ILIKE ? OR "+
"LOWER(teams.official_name) ILIKE ?",
tokenPattern, tokenPattern, tokenPattern,
)
}
searchConditions = searchConditions.Or(tokenFilter)
}
// Tier 2: Accent-insensitive fallback (for names with accents)
accentSearchConditions := s.db.Where(
"unaccent(LOWER(teams.name)) ILIKE unaccent(?) OR "+
"unaccent(LOWER(teams.short_name)) ILIKE unaccent(?) OR "+
"unaccent(LOWER(teams.official_name)) ILIKE unaccent(?)",
likePattern, likePattern, likePattern,
)
if len(searchTokens) > 1 {
accentTokenFilter := s.db
for _, token := range searchTokens {
tokenPattern := "%" + token + "%"
accentTokenFilter = accentTokenFilter.Where(
"unaccent(LOWER(teams.name)) ILIKE unaccent(?) OR "+
"unaccent(LOWER(teams.short_name)) ILIKE unaccent(?) OR "+
"unaccent(LOWER(teams.official_name)) ILIKE unaccent(?)",
tokenPattern, tokenPattern, tokenPattern,
)
}
accentSearchConditions = accentSearchConditions.Or(accentTokenFilter)
}
// Combine both search strategies with OR
query = query.Where(s.db.Where(searchConditions).Or(accentSearchConditions))
// Optimized ranking: prioritize exact and partial matches
query = query.Order(clause.Expr{SQL: "CASE " +
"WHEN LOWER(teams.name) = ? THEN 0 " +
"WHEN LOWER(teams.short_name) = ? THEN 1 " +
"WHEN LOWER(teams.official_name) = ? THEN 2 " +
"WHEN LOWER(teams.name) ILIKE ? THEN 3 " +
"WHEN LOWER(teams.short_name) ILIKE ? THEN 4 " +
"WHEN LOWER(teams.official_name) ILIKE ? THEN 5 " +
"ELSE 6 END",
Vars: []interface{}{searchLower, searchLower, searchLower, likePattern, likePattern, likePattern},
})
}
if strings.TrimSpace(gender) != "" {
......@@ -43,11 +115,24 @@ func (s *teamService) ListTeams(ctx context.Context, limit, offset int, name, ge
query = query.Where("type = ?", strings.ToLower(strings.TrimSpace(teamType)))
}
// Prioritize main teams over youth/reserve teams
query = query.Order("CASE WHEN category = 'default' THEN 0 WHEN category = 'youth' THEN 1 ELSE 2 END")
// Prioritize teams from top leagues using league tier configuration
query = query.Order(clause.Expr{SQL: config.BuildLeagueTierCaseSQL("teams")})
// Prioritize teams with market value
query = query.Order("CASE WHEN market_value IS NULL THEN 1 ELSE 0 END")
query = query.Order("market_value DESC")
// Active teams first
query = query.Order("is_active DESC")
// Teams with competition/league data
query = query.Order("CASE WHEN competition_ts_id IS NULL OR competition_ts_id = '' THEN 1 ELSE 0 END")
query = query.Order("CASE WHEN league IS NULL OR league = '' THEN 1 ELSE 0 END")
// Alphabetical fallback
query = query.Order("name ASC")
var total int64
......
migrations/0024_create_sync_logs_table.sql 0 → 100644
-- Migration: Create sync_logs table for tracking API sync operations
-- This table tracks all sync operations from the Wyscout updatedobjects endpoint
CREATE TABLE IF NOT EXISTS sync_logs (
id VARCHAR(16) PRIMARY KEY,
sync_type VARCHAR(50) NOT NULL,
resource_type VARCHAR(50) NOT NULL,
started_at TIMESTAMP WITH TIME ZONE NOT NULL,
completed_at TIMESTAMP WITH TIME ZONE,
status VARCHAR(20) NOT NULL DEFAULT 'running',
total_records INTEGER DEFAULT 0,
processed_records INTEGER DEFAULT 0,
failed_records INTEGER DEFAULT 0,
error_message TEXT,
sync_params JSONB,
created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
updated_at TIMESTAMP WITH TIME ZONE DEFAULT NOW()
);
CREATE INDEX idx_sync_logs_sync_type ON sync_logs(sync_type);
CREATE INDEX idx_sync_logs_resource_type ON sync_logs(resource_type);
CREATE INDEX idx_sync_logs_status ON sync_logs(status);
CREATE INDEX idx_sync_logs_started_at ON sync_logs(started_at DESC);
COMMENT ON TABLE sync_logs IS 'Tracks all API sync operations from Wyscout updatedobjects endpoint';
COMMENT ON COLUMN sync_logs.sync_type IS 'Type of sync: incremental, full, manual';
COMMENT ON COLUMN sync_logs.resource_type IS 'Wyscout resource type: players, teams, matches, etc.';
COMMENT ON COLUMN sync_logs.status IS 'Sync status: running, completed, failed, partial';
COMMENT ON COLUMN sync_logs.sync_params IS 'JSON parameters used for the sync (updated_since, limit, etc.)';
migrations/clean_restore.sh 0 → 100644
#!/bin/bash
# Script to restore PostgreSQL dump with proper user mapping
# Usage: ./clean_restore.sh <dump_file> <database_name>
set -e
DUMP_FILE=$1
DATABASE=${2:-postgres}
DB_USER="elitedata"
DB_PASSWORD="9NrwgfW1T861oTAQ"
if [ -z "$DUMP_FILE" ]; then
echo "Usage: $0 <dump_file> [database_name]"
echo "Example: $0 backup.dump postgres"
exit 1
fi
if [ ! -f "$DUMP_FILE" ]; then
echo "Error: Dump file '$DUMP_FILE' not found"
exit 1
fi
echo "=========================================="
echo "PostgreSQL Restore Script"
echo "=========================================="
echo "Dump file: $DUMP_FILE"
echo "Database: $DATABASE"
echo "User: $DB_USER"
echo "=========================================="
# Step 1: Prepare the database
echo ""
echo "Step 1: Preparing database with correct permissions..."
PGPASSWORD=$DB_PASSWORD psql -U $DB_USER -d $DATABASE -f prepare_dev_database.sql
# Step 2: Restore with user mapping
echo ""
echo "Step 2: Restoring dump with user mapping..."
echo "Note: Some errors about existing objects are expected and can be ignored"
echo ""
# Use pg_restore with options to handle conflicts
PGPASSWORD=$DB_PASSWORD pg_restore \
--username=$DB_USER \
--dbname=$DATABASE \
--no-owner \
--no-privileges \
--role=$DB_USER \
--clean \
--if-exists \
--verbose \
"$DUMP_FILE" 2>&1 | tee restore.log || true
# Step 3: Fix ownership after restore
echo ""
echo "Step 3: Fixing ownership of all objects..."
PGPASSWORD=$DB_PASSWORD psql -U $DB_USER -d $DATABASE -f prepare_dev_database.sql
echo ""
echo "=========================================="
echo "Restore completed!"
echo "Check restore.log for details"
echo "=========================================="
migrations/prepare_dev_database.sql 0 → 100644
-- Prepare Development Database for Restore
-- This script sets up the correct user permissions and extensions
-- 1. Ensure the elitedata user exists and has proper permissions
DO $$
BEGIN
IF NOT EXISTS (SELECT FROM pg_catalog.pg_roles WHERE rolname = 'elitedata') THEN
CREATE ROLE elitedata WITH LOGIN PASSWORD '9NrwgfW1T861oTAQ';
END IF;
END
$$;
-- 2. Grant necessary privileges to elitedata
GRANT ALL PRIVILEGES ON DATABASE postgres TO elitedata;
GRANT ALL PRIVILEGES ON SCHEMA public TO elitedata;
ALTER SCHEMA public OWNER TO elitedata;
-- 3. Install required extensions
CREATE EXTENSION IF NOT EXISTS unaccent;
CREATE EXTENSION IF NOT EXISTS pg_trgm;
-- 4. If tables already exist, transfer ownership to elitedata
DO $$
DECLARE
r RECORD;
BEGIN
-- Transfer table ownership
FOR r IN (SELECT tablename FROM pg_tables WHERE schemaname = 'public') LOOP
EXECUTE 'ALTER TABLE public.' || quote_ident(r.tablename) || ' OWNER TO elitedata';
END LOOP;
-- Transfer sequence ownership
FOR r IN (SELECT sequence_name FROM information_schema.sequences WHERE sequence_schema = 'public') LOOP
EXECUTE 'ALTER SEQUENCE public.' || quote_ident(r.sequence_name) || ' OWNER TO elitedata';
END LOOP;
-- Transfer type ownership
FOR r IN (SELECT typname FROM pg_type t JOIN pg_namespace n ON t.typnamespace = n.oid WHERE n.nspname = 'public' AND t.typtype = 'e') LOOP
EXECUTE 'ALTER TYPE public.' || quote_ident(r.typname) || ' OWNER TO elitedata';
END LOOP;
END
$$;
-- 5. Grant all privileges on existing tables to elitedata
GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA public TO elitedata;
GRANT ALL PRIVILEGES ON ALL SEQUENCES IN SCHEMA public TO elitedata;
-- 6. Set default privileges for future objects
ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL ON TABLES TO elitedata;
ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL ON SEQUENCES TO elitedata;
scripts/setup_sync.sh 0 → 100644
#!/bin/bash
# Setup script for Wyscout Sync System
# This script sets up the database tables and prepares the sync system
set -e
echo "=========================================="
echo "Wyscout Sync System Setup"
echo "=========================================="
# Check if .env file exists
if [ ! -f .env ]; then
echo "Error: .env file not found"
echo "Please create a .env file with your database credentials"
exit 1
fi
# Load environment variables
export $(cat .env | grep -v '^#' | xargs)
# Check required environment variables
if [ -z "$DB_USER" ] || [ -z "$DB_PASSWORD" ] || [ -z "$DB_NAME" ]; then
echo "Error: Missing required environment variables"
echo "Please ensure DB_USER, DB_PASSWORD, and DB_NAME are set in .env"
exit 1
fi
echo ""
echo "Step 1: Creating sync_logs table..."
PGPASSWORD=$DB_PASSWORD psql -U $DB_USER -d $DB_NAME -f migrations/0024_create_sync_logs_table.sql
echo ""
echo "Step 2: Verifying table creation..."
PGPASSWORD=$DB_PASSWORD psql -U $DB_USER -d $DB_NAME -c "\d sync_logs"
echo ""
echo "=========================================="
echo "Setup Complete!"
echo "=========================================="
echo ""
echo "Next steps:"
echo "1. Ensure your Wyscout API credentials are in .env:"
echo " ProviderUser=your_username"
echo " ProviderSecret=your_password"
echo ""
echo "2. Test the sync system:"
echo " go run cmd/sync/main.go -status"
echo ""
echo "3. Run a test sync:"
echo " go run cmd/sync/main.go -resource=players -hours=24"
echo ""
echo "4. Start the server with cron enabled:"
echo " go run cmd/server/main.go"
echo ""
systemd/SYSTEMD_SETUP.md 0 → 100644
# Systemd Setup Guide for Wyscout Sync Daemon
## Installation Steps
### 1. Build the Binary
```bash
cd /home/augusto/ScoutingSystem/SSEData
# Build the sync daemon
go build -o bin/sync-daemon cmd/sync-daemon/main.go
# Verify it works
./bin/sync-daemon -once
```
### 2. Update Service File Paths
Edit `systemd/wyscout-sync.service` and update these paths to match your server:
```ini
User=your_username # Your Linux user
Group=your_username # Your Linux group
WorkingDirectory=/path/to/ScoutingSystem/SSEData # Your project path
EnvironmentFile=/path/to/ScoutingSystem/SSEData/.env # Your .env path
ExecStart=/path/to/ScoutingSystem/SSEData/bin/sync-daemon -interval=12 -lookback=24
ReadWritePaths=/path/to/ScoutingSystem/SSEData/logs # Your logs path
```
### 3. Install the Service
```bash
# Copy service file to systemd directory
sudo cp systemd/wyscout-sync.service /etc/systemd/system/
# Reload systemd to recognize new service
sudo systemctl daemon-reload
# Enable service to start on boot
sudo systemctl enable wyscout-sync
# Start the service
sudo systemctl start wyscout-sync
```
### 4. Verify It's Running
```bash
# Check status
sudo systemctl status wyscout-sync
# Should show:
# ● wyscout-sync.service - Wyscout Sync Daemon
# Loaded: loaded (/etc/systemd/system/wyscout-sync.service; enabled)
# Active: active (running) since Mon 2026-02-03 11:52:00 UTC; 5s ago
```
## Daily Operations
### Check Status
```bash
sudo systemctl status wyscout-sync
```
### View Logs
```bash
# Real-time logs
sudo journalctl -u wyscout-sync -f
# Last 100 lines
sudo journalctl -u wyscout-sync -n 100
# Logs from today
sudo journalctl -u wyscout-sync --since today
# Logs from specific time
sudo journalctl -u wyscout-sync --since "2026-02-03 10:00:00"
```
### Start/Stop/Restart
```bash
# Start
sudo systemctl start wyscout-sync
# Stop
sudo systemctl stop wyscout-sync
# Restart
sudo systemctl restart wyscout-sync
# Reload configuration (after editing .env)
sudo systemctl reload-or-restart wyscout-sync
```
### Enable/Disable Auto-Start
```bash
# Enable (start on boot)
sudo systemctl enable wyscout-sync
# Disable (don't start on boot)
sudo systemctl disable wyscout-sync
# Check if enabled
sudo systemctl is-enabled wyscout-sync
```
## Configuration Changes
### Change Sync Interval
**Option 1: Edit service file**
```bash
sudo nano /etc/systemd/system/wyscout-sync.service
# Change this line:
ExecStart=/path/to/bin/sync-daemon -interval=24 -lookback=48
# Then reload:
sudo systemctl daemon-reload
sudo systemctl restart wyscout-sync
```
**Option 2: Use environment variables**
Add to your `.env` file:
```env
SYNC_INTERVAL=12
LOOKBACK_HOURS=24
```
Then modify the service to read these:
```ini
ExecStart=/path/to/bin/sync-daemon -interval=${SYNC_INTERVAL} -lookback=${LOOKBACK_HOURS}
```
### Update Binary After Code Changes
```bash
# Rebuild
cd /home/augusto/ScoutingSystem/SSEData
go build -o bin/sync-daemon cmd/sync-daemon/main.go
# Restart service
sudo systemctl restart wyscout-sync
# Verify new version is running
sudo systemctl status wyscout-sync
```
## Monitoring
### Check if Service is Running
```bash
sudo systemctl is-active wyscout-sync
# Output: active or inactive
```
### Check Service Health
```bash
# Full status with recent logs
sudo systemctl status wyscout-sync
# Check for failures
sudo systemctl is-failed wyscout-sync
# View service dependencies
sudo systemctl list-dependencies wyscout-sync
```
### Resource Usage
```bash
# Memory and CPU usage
sudo systemctl status wyscout-sync | grep -E "Memory|CPU"
# Detailed resource info
sudo systemd-cgtop | grep wyscout-sync
```
### Logs Analysis
```bash
# Count errors in last hour
sudo journalctl -u wyscout-sync --since "1 hour ago" | grep -i error | wc -l
# Show only errors
sudo journalctl -u wyscout-sync -p err
# Export logs to file
sudo journalctl -u wyscout-sync --since today > /tmp/sync-logs.txt
```
## Troubleshooting
### Service Won't Start
```bash
# Check for errors
sudo systemctl status wyscout-sync
sudo journalctl -u wyscout-sync -n 50
# Common issues:
# 1. Binary not found
ls -la /home/augusto/ScoutingSystem/SSEData/bin/sync-daemon
# 2. Permission issues
sudo chmod +x /home/augusto/ScoutingSystem/SSEData/bin/sync-daemon
# 3. .env file not readable
ls -la /home/augusto/ScoutingSystem/SSEData/.env
# 4. Database not running
sudo systemctl status postgresql
```
### Service Keeps Restarting
```bash
# Check restart count
sudo systemctl status wyscout-sync | grep "Restarts"
# View crash logs
sudo journalctl -u wyscout-sync --since "10 minutes ago"
# Check if hitting resource limits
sudo systemctl show wyscout-sync | grep -E "Memory|CPU"
```
### Database Connection Issues
```bash
# Test database connection manually
psql -U elitedata -d postgres -c "SELECT 1;"
# Check if PostgreSQL is running
sudo systemctl status postgresql
# Verify .env has correct credentials
cat /home/augusto/ScoutingSystem/SSEData/.env | grep DB_
```
### High Memory Usage
Edit service file to reduce memory limit:
```ini
MemoryLimit=256M # Reduce from 512M
```
Then:
```bash
sudo systemctl daemon-reload
sudo systemctl restart wyscout-sync
```
## Advanced Configuration
### Run Multiple Sync Daemons
Create separate service files for different schedules:
**wyscout-sync-hourly.service** (frequent updates):
```ini
ExecStart=/path/to/bin/sync-daemon -interval=1 -lookback=2
```
**wyscout-sync-daily.service** (full sync):
```ini
ExecStart=/path/to/bin/sync-daemon -interval=24 -lookback=48
```
### Email Notifications on Failure
Install `mailutils`:
```bash
sudo apt-get install mailutils
```
Create `/etc/systemd/system/wyscout-sync-failure@.service`:
```ini
[Unit]
Description=Send email on wyscout-sync failure
[Service]
Type=oneshot
ExecStart=/usr/bin/mail -s "Wyscout Sync Failed" admin@example.com
StandardInput=journal
```
Add to main service:
```ini
[Unit]
OnFailure=wyscout-sync-failure@%n.service
```
### Automatic Restart on Failure
Already configured in the service file:
```ini
Restart=always
RestartSec=10
StartLimitInterval=200
StartLimitBurst=5
```
This means:
- Restart always on failure
- Wait 10 seconds between restarts
- Allow max 5 restarts in 200 seconds
- If limit exceeded, service stops
## Security Best Practices
The service file includes security hardening:
```ini
NoNewPrivileges=true # Prevent privilege escalation
PrivateTmp=true # Isolated /tmp directory
ProtectSystem=strict # Read-only system directories
ProtectHome=read-only # Read-only home directories
ReadWritePaths=/path/logs # Only logs directory is writable
```
### Additional Security
1. **Run as non-root user** (already configured)
2. **Restrict file permissions**:
```bash
chmod 600 .env
chmod 700 bin/sync-daemon
```
3. **Use systemd secrets** for sensitive data (optional)
## Comparison: Systemd vs Scripts
| Feature | Systemd | Bash Scripts |
|---------|---------|--------------|
| Auto-restart on failure | ✅ Built-in | ❌ Manual |
| Start on boot | ✅ Built-in | ❌ Need cron |
| Logging | ✅ journald | ❌ Manual |
| Resource limits | ✅ Built-in | ❌ Manual |
| Status monitoring | ✅ systemctl | ❌ ps/grep |
| Dependency management | ✅ Built-in | ❌ Manual |
| Security hardening | ✅ Built-in | ❌ Manual |
| Standard on Linux | ✅ Yes | ❌ No |
**Recommendation: Use systemd for production servers**
## Quick Reference
```bash
# Start/Stop
sudo systemctl start wyscout-sync
sudo systemctl stop wyscout-sync
sudo systemctl restart wyscout-sync
# Status
sudo systemctl status wyscout-sync
sudo systemctl is-active wyscout-sync
# Logs
sudo journalctl -u wyscout-sync -f
sudo journalctl -u wyscout-sync --since today
# Enable/Disable
sudo systemctl enable wyscout-sync
sudo systemctl disable wyscout-sync
# After changes
sudo systemctl daemon-reload
sudo systemctl restart wyscout-sync
```
## Integration with Monitoring Tools
### Prometheus Exporter
```bash
# Install node_exporter
sudo apt-get install prometheus-node-exporter
# Systemd metrics available at:
# http://localhost:9100/metrics
```
### Grafana Dashboard
Monitor service status, restarts, and resource usage.
### Alerting
Set up alerts for:
- Service down
- High restart count
- Memory/CPU limits reached
- Sync failures in logs
systemd/install.sh 0 → 100644
#!/bin/bash
# Systemd Installation Script for Wyscout Sync Daemon
# This script automates the installation of the sync daemon as a systemd service
set -e
echo "=========================================="
echo "Wyscout Sync Daemon - Systemd Installation"
echo "=========================================="
# Check if running as root
if [ "$EUID" -eq 0 ]; then
echo "Error: Do not run this script as root"
echo "Run as your regular user, sudo will be used when needed"
exit 1
fi
# Get the absolute path to the project directory
PROJECT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
echo "Project directory: $PROJECT_DIR"
# Get current user
CURRENT_USER=$(whoami)
echo "User: $CURRENT_USER"
# Check if .env exists
if [ ! -f "$PROJECT_DIR/.env" ]; then
echo "Error: .env file not found at $PROJECT_DIR/.env"
echo "Please create .env file with your configuration"
exit 1
fi
# Build the binary
echo ""
echo "Step 1: Building sync daemon binary..."
cd "$PROJECT_DIR"
go build -o bin/sync-daemon cmd/sync-daemon/main.go
if [ ! -f "$PROJECT_DIR/bin/sync-daemon" ]; then
echo "Error: Failed to build binary"
exit 1
fi
chmod +x bin/sync-daemon
echo "✓ Binary built successfully"
# Test the binary
echo ""
echo "Step 2: Testing binary..."
if ! ./bin/sync-daemon -once 2>&1 | head -5; then
echo "Warning: Binary test had issues, but continuing..."
fi
# Create logs directory
echo ""
echo "Step 3: Creating logs directory..."
mkdir -p "$PROJECT_DIR/logs"
echo "✓ Logs directory created"
# Create service file with correct paths
echo ""
echo "Step 4: Creating systemd service file..."
SERVICE_FILE="/tmp/wyscout-sync.service"
cat > "$SERVICE_FILE" << EOF
[Unit]
Description=Wyscout Sync Daemon - Automated data synchronization from Wyscout API
Documentation=https://github.com/your-org/ScoutingSystem
After=network-online.target postgresql.service
Wants=network-online.target
Requires=postgresql.service
[Service]
Type=simple
User=$CURRENT_USER
Group=$CURRENT_USER
WorkingDirectory=$PROJECT_DIR
# Environment file
EnvironmentFile=$PROJECT_DIR/.env
# Binary location
ExecStart=$PROJECT_DIR/bin/sync-daemon -interval=12 -lookback=24
# Restart policy
Restart=always
RestartSec=10
StartLimitInterval=200
StartLimitBurst=5
# Resource limits
MemoryLimit=512M
CPUQuota=50%
# Logging
StandardOutput=journal
StandardError=journal
SyslogIdentifier=wyscout-sync
# Security hardening
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=read-only
ReadWritePaths=$PROJECT_DIR/logs
# Graceful shutdown
TimeoutStopSec=30
KillMode=mixed
KillSignal=SIGTERM
[Install]
WantedBy=multi-user.target
EOF
echo "✓ Service file created"
# Install the service
echo ""
echo "Step 5: Installing systemd service..."
sudo cp "$SERVICE_FILE" /etc/systemd/system/wyscout-sync.service
sudo systemctl daemon-reload
echo "✓ Service installed"
# Enable the service
echo ""
echo "Step 6: Enabling service to start on boot..."
sudo systemctl enable wyscout-sync
echo "✓ Service enabled"
# Start the service
echo ""
echo "Step 7: Starting service..."
sudo systemctl start wyscout-sync
sleep 2
# Check status
echo ""
echo "Step 8: Checking service status..."
if sudo systemctl is-active --quiet wyscout-sync; then
echo "✓ Service is running!"
else
echo "✗ Service failed to start"
echo ""
echo "Checking logs..."
sudo journalctl -u wyscout-sync -n 20 --no-pager
exit 1
fi
# Show status
echo ""
echo "=========================================="
echo "Installation Complete!"
echo "=========================================="
echo ""
sudo systemctl status wyscout-sync --no-pager -l
echo ""
echo "Useful commands:"
echo " sudo systemctl status wyscout-sync # Check status"
echo " sudo journalctl -u wyscout-sync -f # View logs"
echo " sudo systemctl restart wyscout-sync # Restart service"
echo " sudo systemctl stop wyscout-sync # Stop service"
echo ""
echo "Configuration:"
echo " Service file: /etc/systemd/system/wyscout-sync.service"
echo " Binary: $PROJECT_DIR/bin/sync-daemon"
echo " Logs: sudo journalctl -u wyscout-sync"
echo ""
systemd/uninstall.sh 0 → 100644
#!/bin/bash
# Systemd Uninstallation Script for Wyscout Sync Daemon
set -e
echo "=========================================="
echo "Wyscout Sync Daemon - Systemd Uninstallation"
echo "=========================================="
# Check if service exists
if [ ! -f /etc/systemd/system/wyscout-sync.service ]; then
echo "Service not found. Nothing to uninstall."
exit 0
fi
# Stop the service
echo "Stopping service..."
sudo systemctl stop wyscout-sync || true
# Disable the service
echo "Disabling service..."
sudo systemctl disable wyscout-sync || true
# Remove service file
echo "Removing service file..."
sudo rm /etc/systemd/system/wyscout-sync.service
# Reload systemd
echo "Reloading systemd..."
sudo systemctl daemon-reload
sudo systemctl reset-failed
echo ""
echo "=========================================="
echo "Uninstallation Complete!"
echo "=========================================="
echo ""
echo "Note: Binary and logs were NOT removed."
echo "To remove them manually:"
echo " rm -rf bin/sync-daemon"
echo " rm -rf logs/"
echo ""
systemd/wyscout-sync.service 0 → 100644
[Unit]
Description=Wyscout Sync Daemon - Automated data synchronization from Wyscout API
Documentation=https://github.com/your-org/ScoutingSystem
After=network-online.target postgresql.service
Wants=network-online.target
Requires=postgresql.service
[Service]
Type=simple
User=augusto
Group=augusto
WorkingDirectory=/home/augusto/ScoutingSystem/SSEData
# Environment file
EnvironmentFile=/home/augusto/ScoutingSystem/SSEData/.env
Environment="SYNC_INTERVAL=12"
Environment="LOOKBACK_HOURS=24"
# Binary location
ExecStart=/home/augusto/ScoutingSystem/SSEData/bin/sync-daemon -interval=${SYNC_INTERVAL} -lookback=${LOOKBACK_HOURS}
# Restart policy
Restart=always
RestartSec=10
StartLimitInterval=200
StartLimitBurst=5
# Resource limits
MemoryLimit=512M
CPUQuota=50%
# Logging
StandardOutput=journal
StandardError=journal
SyslogIdentifier=wyscout-sync
# Security hardening
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=read-only
ReadWritePaths=/home/augusto/ScoutingSystem/SSEData/logs
# Graceful shutdown
TimeoutStopSec=30
KillMode=mixed
KillSignal=SIGTERM
[Install]
WantedBy=multi-user.target
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or sign in to comment