Data Migration Guide

Comprehensive guide for handling database schema changes and data migrations in the Toto ecosystem.

📋 Overview

This guide covers how to safely evolve your Firestore database schema while maintaining data integrity and minimizing downtime.

🚨 Migration Principles

Backward Compatibility

• Never remove fields without deprecation period
• Add new fields as optional initially
• Use default values for missing data
• Maintain API compatibility during transitions

Data Safety

• Always backup before major migrations
• Test migrations on staging data first
• Use transactions for related changes
• Monitor migration progress in production

Performance Considerations

• Batch operations for large datasets
• Use background functions for heavy migrations
• Implement progressive migration for large collections
• Monitor Firestore costs during migrations

🔧 Migration Strategies

1. Additive Changes (Safest)

// Adding new optional fields
interface User {
  // ... existing fields
  newField?: string;        // Optional new field
  preferences?: {           // New nested object
    notifications: boolean;
    emailUpdates: boolean;
  };
}

Implementation:

// No immediate migration needed
// New users get the field, existing users don't
// Handle missing fields gracefully in code
const userPreferences = user.preferences || {
  notifications: true,  // Default value
  emailUpdates: true
};

2. Field Renaming (Medium Risk)

// Renaming a field (e.g., 'pet' → 'case')
interface Case {
  // ... existing fields
  // OLD: petName: string;
  caseName: string;         // NEW field name
}

Implementation:

// 1. Add new field alongside old one
await updateDoc(doc(db, 'cases', caseId), {
  caseName: existingCase.petName,  // Copy old value
  // Keep old field for backward compatibility
});

// 2. Update code to use new field
const caseName = case.caseName || case.petName; // Fallback

// 3. Remove old field after migration period
await updateDoc(doc(db, 'cases', caseId), {
  petName: deleteField()  // Remove old field
});

3. Type Changes (High Risk)

// Changing field types (e.g., string → number)
interface Case {
  // ... existing fields
  // OLD: donationGoal: string;  // "$50.00"
  donationGoal: number;          // 5000 (cents)
}

Implementation:

// 1. Create migration function
async function migrateDonationGoals() {
  const casesSnapshot = await getDocs(collection(db, 'cases'));
  
  const batch = writeBatch(db);
  
  casesSnapshot.docs.forEach(doc => {
    const caseData = doc.data();
    if (typeof caseData.donationGoal === 'string') {
      // Convert "$50.00" to 5000
      const amount = parseFloat(caseData.donationGoal.replace('$', ''));
      const cents = Math.round(amount * 100);
      
      batch.update(doc.ref, {
        donationGoal: cents,
        // Keep old field for rollback
        _oldDonationGoal: caseData.donationGoal
      });
    }
  });
  
  await batch.commit();
}

// 2. Update code with fallback
const donationGoal = typeof case.donationGoal === 'number' 
  ? case.donationGoal 
  : parseFloat(case.donationGoal.replace('$', '')) * 100;

4. Collection Restructuring (Highest Risk)

// Moving from flat structure to nested
// OLD: /cases/{caseId}/updates/{updateId}
// NEW: /cases/{caseId}/updates/{updateId}

Implementation:

// 1. Create new structure
async function restructureUpdates() {
  const casesSnapshot = await getDocs(collection(db, 'cases'));
  
  for (const caseDoc of casesSnapshot.docs) {
    const updatesSnapshot = await getDocs(
      collection(db, 'updates'),
      where('caseId', '==', caseDoc.id)
    );
    
    // Move updates to subcollection
    const batch = writeBatch(db);
    
    updatesSnapshot.docs.forEach(updateDoc => {
      const updateData = updateDoc.data();
      
      // Create in new location
      const newUpdateRef = doc(
        collection(db, 'cases', caseDoc.id, 'updates')
      );
      batch.set(newUpdateRef, updateData);
      
      // Mark old document for deletion
      batch.update(updateDoc.ref, { _migrated: true });
    });
    
    await batch.commit();
  }
}

// 2. Clean up old documents
async function cleanupOldUpdates() {
  const oldUpdatesSnapshot = await getDocs(
    query(collection(db, 'updates'), where('_migrated', '==', true)
  );
  
  const batch = writeBatch(db);
  oldUpdatesSnapshot.docs.forEach(doc => {
    batch.delete(doc.ref);
  });
  
  await batch.commit();
}

🛠️ Migration Tools

Migration Function Template

interface MigrationConfig {
  version: string;
  description: string;
  collections: string[];
  batchSize: number;
  dryRun: boolean;
}

class DataMigrator {
  constructor(private config: MigrationConfig) {}
  
  async migrate() {
    console.log(`Starting migration: ${this.config.description}`);
    
    for (const collectionName of this.config.collections) {
      await this.migrateCollection(collectionName);
    }
    
    console.log('Migration completed successfully');
  }
  
  private async migrateCollection(collectionName: string) {
    const snapshot = await getDocs(collection(db, collectionName));
    const docs = snapshot.docs;
    
    console.log(`Migrating ${docs.length} documents in ${collectionName}`);
    
    // Process in batches
    for (let i = 0; i < docs.length; i += this.config.batchSize) {
      const batch = docs.slice(i, i + this.config.batchSize);
      await this.processBatch(collectionName, batch);
      
      console.log(`Processed ${Math.min(i + this.config.batchSize, docs.length)}/${docs.length}`);
    }
  }
  
  private async processBatch(collectionName: string, docs: DocumentSnapshot[]) {
    if (this.config.dryRun) {
      // Just log what would be changed
      docs.forEach(doc => {
        console.log(`Would update ${collectionName}/${doc.id}`);
      });
      return;
    }
    
    const batch = writeBatch(db);
    
    docs.forEach(doc => {
      const updatedData = this.transformDocument(doc.data());
      batch.update(doc.ref, updatedData);
    });
    
    await batch.commit();
  }
  
  private transformDocument(data: any): any {
    // Implement your transformation logic here
    return data;
  }
}

Usage Example

const migration = new DataMigrator({
  version: '1.2.0',
  description: 'Migrate pet names to case names',
  collections: ['cases'],
  batchSize: 500,
  dryRun: false
});

await migration.migrate();

📊 Migration Monitoring

Progress Tracking

interface MigrationProgress {
  version: string;
  startTime: Date;
  totalDocuments: number;
  processedDocuments: number;
  errors: string[];
  status: 'running' | 'completed' | 'failed';
}

// Store progress in Firestore
async function updateProgress(progress: MigrationProgress) {
  await setDoc(doc(db, 'migrations', progress.version), progress);
}

Error Handling

async function safeMigration(migrationFn: () => Promise<void>) {
  try {
    await migrationFn();
    console.log('Migration completed successfully');
  } catch (error) {
    console.error('Migration failed:', error);
    
    // Log error details
    await addDoc(collection(db, 'migrationErrors'), {
      timestamp: serverTimestamp(),
      error: error.message,
      stack: error.stack,
      version: '1.2.0'
    });
    
    throw error;
  }
}

🔄 Rollback Strategies

Field-Level Rollback

// Keep old field during migration
interface Case {
  caseName: string;         // New field
  petName?: string;         // Old field (kept for rollback)
  _migrationVersion: string; // Track migration state
}

// Rollback function
async function rollbackCaseNames() {
  const casesSnapshot = await getDocs(
    query(collection(db, 'cases'), 
    where('_migrationVersion', '==', '1.2.0')
  );
  
  const batch = writeBatch(db);
  
  casesSnapshot.docs.forEach(doc => {
    const caseData = doc.data();
    if (caseData.petName) {
      batch.update(doc.ref, {
        caseName: caseData.petName,  // Restore old value
        _migrationVersion: 'rolled_back'
      });
    }
  });
  
  await batch.commit();
}

Collection Rollback

// Restore from backup collection
async function rollbackCollection(collectionName: string, backupCollectionName: string) {
  const backupSnapshot = await getDocs(collection(db, backupCollectionName));
  
  const batch = writeBatch(db);
  
  backupSnapshot.docs.forEach(doc => {
    const backupData = doc.data();
    batch.set(doc(db, collectionName, doc.id), backupData);
  });
  
  await batch.commit();
}

📋 Migration Checklist

Pre-Migration

• Backup production data
• Test on staging environment
• Notify team of downtime
• Prepare rollback plan
• Set up monitoring

During Migration

• Monitor progress
• Check for errors
• Validate data integrity
• Update application code

Post-Migration

• Verify all data migrated
• Test application functionality
• Clean up old fields/collections
• Update documentation
• Monitor performance

🚀 Best Practices

Planning

• Version your migrations clearly
• Document all changes thoroughly
• Estimate migration time accurately
• Plan for rollback scenarios

Execution

• Use transactions for related changes
• Process in batches to avoid timeouts
• Monitor costs during migration
• Test thoroughly before production

Maintenance

• Keep migration history in Firestore
• Clean up old data after migration period
• Update application code to use new schema
• Train team on new data structure

---

For specific migration scenarios, refer to the Data Models documentation and consult with the development team.