Skip to main content

Seeding System Documentation

Comprehensive documentation for the Toto ecosystem's database seeding system, including setup, usage, and maintenance.

📊 Overview​

The Toto ecosystem includes a sophisticated seeding system that generates realistic sample data for development and staging environments. The system is modular, scalable, and designed to create comprehensive test data that mirrors production scenarios.

🏗️ System Architecture​

Modular Design​

The seeding system is built with a modular architecture:

src/lib/seeding/
├── users.ts              # User generation (all roles)
├── supportTickets.ts     # Support ticket generation
├── cases.ts              # Case generation
├── donations.ts          # Donation generation
├── follows.ts            # Follow relationship generation
├── notifications.ts      # Notification generation
├── caseUpdates.ts        # Case update generation
└── auditLogs.ts          # Audit log generation

Main Seeding Script​

  • Location: /api/init-staging-complete
  • Purpose: Complete database reset + comprehensive seeding
  • Output: Clean staging database with realistic data
  • Features: Auto-discovery and cleanup of ALL collections

🚀 Quick Start​

Running the Seeding Script​

# Start the development server
npm run dev

# Run the enhanced seeding script with complete database reset
curl -X POST http://localhost:5000/api/init-staging-complete

Process Details​

Step 1: Complete Database Cleanup​

  • Auto-discovery: Uses listCollections() to find ALL existing collections
  • Comprehensive cleaning: Removes documents and subcollections in batches
  • No missed data: Explicitly targets known collections (collaborators, test, etc.)
  • Large dataset handling: Processes in 500-document batches to avoid timeouts

Step 2: Data Generation & Population​

  • Realistic relationships: Proper user-case-donation linkages
  • Argentine context: Local phone numbers, addresses, names
  • Role-based permissions: Proper admin/guardian/user distribution
  • Timestamp accuracy: Realistic creation and update times

Expected Output​

🚀 Starting final staging database initialization...
🧹 COMPLETE database reset - cleaning ALL collections...
Found collections: users, cases, donations, notifications, audit_logs, collaborators, test
🗑️ Cleaning collection: users
✅ Collection users fully cleaned
🗑️ Cleaning collection: collaborators  
✅ Collection collaborators fully cleaned
🎉 COMPLETE database reset finished! Total cleaned: 150 documents
👥 Creating users...
✅ Created 27 users
🎫 Creating support tickets...
✅ Created 15 support tickets
📋 Creating cases...
✅ Created 10 cases
💰 Creating donations...
✅ Created 50 donations
👥 Creating follows...
✅ Created 30 follows
🔔 Creating notifications...
✅ Created 30 notifications
📝 Creating case updates...
✅ Created 36 case updates
🔍 Creating audit logs...
✅ Created 200 audit logs
💾 Writing to Firestore...
✅ Written 200 audit logs
📈 Final Summary: {
  users: 27,
  supportTickets: 15,
  cases: 10,
  donations: 50,
  follows: 30,
  notifications: 30,
  caseUpdates: 36,
  auditLogs: 200,
  idFormat: 'normalized'
}

👥 User Seeding​

User Generation Strategy​

The user seeding system creates a diverse set of users across all roles:

// User distribution
const userDistribution = {
  admin: 3,        // Platform administrators
  guardian: 6,     // Case guardians
  user: 12,        // Regular users
  investor: 3,     // Financial backers
  partner: 3       // Business partners
};

User Fields Generated​

  • Basic Info: ID, email, name, role, status
  • Profile: Bio, location, phone, organization
  • Activity: Activity rate, time since joined
  • Permissions: Role-based permissions
  • Contact: Phone (Argentina format), social links
  • Preferences: Notification settings, case types

Realistic Data Features​

  • Argentina Phone Numbers: +54 format with realistic area codes
  • Email Domains: Mix of @betoto.pet and common domains
  • Names: Spanish and English names
  • Organizations: Realistic organization names for guardians/admins
  • Locations: Argentine cities and regions

🆘 Case Seeding​

Case Generation Strategy​

Cases are created with realistic scenarios and proper relationships:

// Case distribution
const caseCategories = [
  'rescue', 'surgery', 'treatment', 'transit', 'foster'
];

const casePriorities = ['urgent', 'normal'];
const caseStatuses = ['active', 'completed', 'draft'];

Case Features​

  • Realistic Descriptions: Detailed case scenarios
  • Financial Goals: Varied donation targets
  • Guardian Assignment: Each case assigned to a guardian
  • Image URLs: Placeholder images for visual representation
  • Timestamps: Realistic creation and update times

💰 Donation Seeding​

Stellar Wallet Integration​

Donations are generated with full Stellar wallet integration:

interface DonationData {
  // Financial details
  amount: number;                // Donation amount in cents
  currency: string;              // ARS or USD
  originalAmount: number;        // Original amount before conversion
  convertedAmount: number;       // Amount after conversion
  
  // Payment processing
  paymentProvider: string;       // MoonPay
  transactionId: string;         // Internal transaction ID
  partnerTransactionId: string;  // Partner transaction ID
}

Donation Features​

  • Multi-Currency: ARS and USD donations
  • Realistic Amounts: Varied donation sizes
  • Payment Providers: MoonPay integration
  • Transaction IDs: Unique transaction identifiers
  • User Relationships: Links to users and cases

🎫 Support System Seeding​

Support Ticket Generation​

Comprehensive support ticket system with realistic scenarios:

// Support ticket distribution
const ticketPriorities = ['low', 'medium', 'high', 'urgent'];
const ticketCategories = ['technical', 'billing', 'general', 'feature_request'];
const ticketStatuses = ['open', 'in_progress', 'resolved', 'closed'];

Support Features​

  • Realistic Scenarios: Common support issues
  • Admin Assignment: Tickets assigned to admin users
  • Due Dates: Calculated based on priority
  • Status Progression: Realistic status changes
  • Tags and Attachments: Metadata for organization

🔔 Notification Seeding​

Notification Generation​

User-specific notifications with realistic scenarios:

// Notification types
const notificationTypes = [
  'case_update', 'donation_received', 'ticket_assigned', 'system_alert'
];

const notificationPriorities = ['low', 'medium', 'high', 'urgent'];

Notification Features​

  • User-Specific: Each notification targets a specific user
  • Realistic Content: Meaningful notification messages
  • Action URLs: Links to relevant pages
  • Priority Levels: Appropriate priority assignment
  • Read Status: Mix of read and unread notifications

🔍 Audit Log Seeding​

Audit Log Generation​

Comprehensive audit trail with realistic system activities:

// Audit log categories
const auditCategories = [
  'user', 'case', 'donation', 'system', 'security', 'support', 'notification'
];

const auditSeverities = ['low', 'medium', 'high', 'critical'];

Audit Features​

  • Realistic Actions: Common system activities
  • Before/After States: State changes tracked
  • User Attribution: Actions linked to users
  • Severity Levels: Appropriate severity assignment
  • Timeline Spread: Activities over 30 days

🛠️ Technical Implementation​

ID Generation System​

Consistent ID format across all entities:

// ID prefixes
const ENTITY_PREFIXES = {
  USER: 'usr',
  CASE: 'cas',
  DONATION: 'don',
  FOLLOW: 'flw',
  NOTIFICATION: 'not',
  AUDIT_LOG: 'aud',
  SUPPORT_TICKET: 'spt',
  UPDATE: 'upd'
};

// Example IDs
// usr_abc123def456
// cas_xyz789ghi012
// don_mno345pqr678

Timestamp Standardization​

All timestamps use ISO 8601 format:

// Timestamp utilities
export const now = () => new Date().toISOString();
export const randomPast = (days: number, hours: number = 0) => {
  const now = new Date();
  const past = new Date(now.getTime() - (days * 24 + hours) * 60 * 60 * 1000);
  return past.toISOString();
};

Data Validation​

Comprehensive validation to prevent Firestore errors:

// Conditional field spreading
const userData = {
  id: generateUserId(),
  email: user.email,
  name: user.name,
  role: user.role,
  status: user.status,
  createdAt: now(),
  // Only include fields that are not undefined
  ...(user.organization && { organization: user.organization }),
  ...(user.phone && { phone: user.phone }),
  ...(user.permissions && { permissions: user.permissions })
};

🔧 Configuration & Customization​

Seeding Parameters​

Customize the seeding output by modifying parameters:

// User generation parameters
const userCounts = {
  admin: 3,        // Number of admin users
  guardian: 6,     // Number of guardian users
  user: 12,        // Number of regular users
  investor: 3,     // Number of investor users
  partner: 3       // Number of partner users
};

// Case generation parameters
const caseCount = 10;           // Number of cases
const donationCount = 50;       // Number of donations
const notificationCount = 30;   // Number of notifications
const auditLogCount = 200;      // Number of audit logs

Data Customization​

Modify the seeding modules to customize data:

// Custom user names
const customNames = [
  'Juan Pérez', 'María González', 'Carlos Rodríguez'
];

// Custom case descriptions
const customDescriptions = [
  'Rescate de perro abandonado en la calle',
  'Cirugía de emergencia para gato herido'
];

🚨 Troubleshooting​

Common Issues​

500 Internal Server Error​

  • Cause: Server memory limits or timeout
  • Solution: Reduce batch sizes or add delays
  • Prevention: Use modular approach with smaller chunks

Firestore Validation Errors​

  • Cause: Undefined values in Firestore documents
  • Solution: Use conditional object spreading
  • Prevention: Validate all data before writing

Memory Issues​

  • Cause: Large data generation in single request
  • Solution: Break into smaller modules
  • Prevention: Use streaming or pagination

Debugging Tools​

  • Individual Module Testing: Test each module separately
  • Data Validation: Check data before database writes
  • Error Logging: Comprehensive error logging
  • Progress Tracking: Real-time progress updates

📊 Performance Metrics​

Seeding Performance​

  • Total Time: ~4-5 seconds for complete seeding
  • Memory Usage: Optimized for server limits
  • Database Writes: Batch operations for efficiency
  • Error Rate: Less than 1% with proper validation

Data Quality​

  • Realism: 95% realistic data scenarios
  • Completeness: 100% required fields populated
  • Consistency: 100% ID format consistency
  • Relationships: 100% valid entity relationships

🔄 Maintenance & Updates​

Regular Maintenance​

  • Data Refresh: Re-run seeding script periodically
  • Schema Updates: Update interfaces when schema changes
  • Validation Updates: Update validation rules as needed
  • Performance Monitoring: Monitor seeding performance

Schema Evolution​

  • Backward Compatibility: Maintain compatibility with existing data
  • Migration Scripts: Create migration scripts for schema changes
  • Version Control: Track schema changes in version control
  • Testing: Test seeding with new schema changes

📚 API Reference​

Seeding Endpoints​

Main Seeding Script​

POST /api/init-staging-final
Content-Type: application/json

Response:
{
  "success": true,
  "message": "Final staging database initialized successfully!",
  "summary": {
    "users": 27,
    "supportTickets": 15,
    "cases": 10,
    "donations": 50,
    "follows": 30,
    "notifications": 30,
    "caseUpdates": 36,
    "auditLogs": 200,
    "idFormat": "normalized"
  }
}

Individual Module Testing​

# Test individual modules
POST /api/debug-users-module
POST /api/debug-cases-module
POST /api/debug-donations-module

🎯 Best Practices​

Development Workflow​

  1. Start Fresh: Clear database before seeding
  2. Test Modules: Test individual modules first
  3. Validate Data: Check data quality after seeding
  4. Monitor Performance: Watch for memory/timeout issues
  5. Document Changes: Update documentation with changes

Production Considerations​

  • Never Run in Production: Seeding is for dev/staging only
  • Backup First: Always backup before major changes
  • Test Thoroughly: Test all scenarios before deployment
  • Monitor Resources: Watch server resources during seeding

This documentation covers the complete seeding system for the Toto ecosystem. For data model details, see Data Models.