Summary

This RFC proposes elevating transactions to first-class citizens in TanStack Optimistic by introducing explicit transaction support through a new mutation factory API. This change addresses limitations in the current implementation that prevent developers from implementing common patterns like cross-collection updates and multi-step operations. By making transactions explicit and providing direct control over their lifecycle, we enable more complex workflows while maintaining the optimistic UI experience that makes the library valuable. The proposed API is designed to be intuitive and flexible, supporting both simple cross-collection operations and complex multi-step workflows.

Background

TanStack Optimistic is a library for building optimistic UI experiences, where user interface updates happen immediately while data changes are persisted asynchronously in the background. This approach provides a responsive user experience while ensuring data consistency.

The current architecture of TanStack Optimistic includes several key components:

1. Collections: Represent sets of data with the same structure, similar to database tables. Each collection has its own state management and transaction handling.

2. TransactionManager: Each collection has its own transaction manager that handles the lifecycle of transactions for that collection. The transaction manager is responsible for creating, persisting, and tracking transactions.

3. Transactions: Currently an internal implementation detail, transactions represent groups of mutations (inserts, updates, deletes) that are processed together. Transactions have a lifecycle (pending, persisting, completed, failed) and are used to track the status of optimistic updates.

4. MutationFn: Each collection can define a mutation function that is responsible for persisting changes to the backend. This function is called when a transaction is ready to be persisted.

5. Sync: Collections also have a sync mechanism that handles incoming changes from the backend, ensuring that the local state stays in sync with the server state.

In the current implementation, transactions are created implicitly when mutation operations (insert, update, delete) are called on a collection. Mutations that occur within the same event loop tick are grouped into a single transaction. The transaction is then processed by the collection's transaction manager, which calls the collection's mutation function to persist the changes.

This approach works well for simple cases where mutations are isolated to a single collection. However, it doesn't address more complex scenarios where related data needs to be updated across multiple collections as part of a single logical operation.

Other libraries and frameworks have addressed similar challenges:

• Database ORMs: Libraries like Knex.js, Prisma, and Drizzle ORM provide explicit transaction APIs that allow grouping multiple operations across tables into a single atomic unit.
• Client-side data stores: Libraries like Dexie.js (for IndexedDB) and Replicache provide transaction capabilities for local data stores.
• State management libraries: Libraries like Redux and MobX support middleware and actions that can group related state changes.

These examples demonstrate the value of explicit transaction control for ensuring data consistency across related entities and providing a better developer experience for complex workflows.

Problem

The current transaction system in TanStack Optimistic has several limitations that prevent developers from implementing common patterns found in real-world applications:

1. Transactions are hidden implementation details: Transactions exist in the library but are not exposed as first-class objects that developers can interact with directly. This limits the ability to build more complex workflows on top of the transaction system.

2. No cross-collection transactions: Each collection manages its own transactions independently, making it impossible to atomically update related data across multiple collections. This is a common requirement in real-world applications, such as adding a user to a team which requires updating both the users collection and the team_members collection.

3. Mutation functions are tied to collections: The mutationFn is defined at the collection level, which doesn't accommodate scenarios where a single API endpoint handles changes to multiple related entities. For example, a POST to /api/users might create both a user record and team membership records.

4. No explicit transaction lifecycle control: Developers cannot explicitly control when a transaction is created, accumulated, committed, or rolled back. The transaction lifecycle is managed automatically by collections, with mutations within a single tick being grouped together.

5. Limited flexibility for multi-step operations: Some workflows require accumulating changes over time before committing them, such as multi-step forms or wizard interfaces. The current system doesn't provide a way to build up a transaction across multiple user interactions.

These limitations force developers to create workarounds or avoid using TanStack Optimistic for more complex scenarios, reducing the library's utility in real-world applications.

Proposal

We propose making transactions first-class citizens in TanStack Optimistic by introducing explicit transaction support through a new mutation factory API. This will allow developers to create, control, and persist transactions that span multiple collections.

API Design

The new API will provide two main patterns for working with transactions:

1. Immediate transactions - For simple cross-collection operations that should be committed immediately:

const addTodo = useOptimisticMutation({ 
  mutationFn: async ({ mutations }) => {
    // Custom logic to persist all changes in a single API call
    await api.createTodo(mutations[0].modified)
  }
})

// Call as many times as you want
addTodo.mutate(() => {
  todosCollection.insert(data)
  todoStatsCollection.update(stats => {
    stats.totalCount += 1
  })
})

2. Explicit transactions - For multi-step operations where changes need to be accumulated over time:

const multiStepForm = useOptimisticMutation({ 
  mutationFn: async ({ mutations }) => {
    // Custom logic to persist all changes in a single API call
    await api.createComplexEntity({
      user: mutations.find(m => m.collection === 'users').modified,
      profile: mutations.find(m => m.collection === 'profiles').modified,
      preferences: mutations.find(m => m.collection === 'preferences').modified
    })
  }
})

// Create an explicit transaction
const transaction = multiStepForm.createTransaction({ id: 'user-signup-flow' })

// Add first step changes
transaction.mutate(() => {
  usersCollection.insert({ id: 'new-user', name: 'John Doe' })
})

// Later, add second step changes
transaction.mutate(() => {
  profilesCollection.insert({ userId: 'new-user', bio: 'Software developer' })
})

// Finally, add third step changes and commit
transaction.mutate(() => {
  preferencesCollection.insert({ userId: 'new-user', theme: 'dark' })
})

// Commit all changes when ready
await transaction.commit()

// Or discard changes if needed
// transaction.rollback()

Implementation Details

1. Transaction Registry: Create a global transaction registry to track active transactions. When a transaction's mutate method is called, it registers itself as the active transaction for the duration of the callback.

2. Collection Integration:

• Modify collection mutation methods (insert, update, delete) to check if there's an active transaction. If so, add the mutation to that transaction instead of creating a new implicit transaction.
• remove mutationFn from the Collection API.

3. Transaction Manager Refactoring: Refactor the existing TransactionManager to work with external transactions rather than being tied to a specific collection. This will involve:

   • Moving transaction creation logic out of collections
   • Adding support for cross-collection mutations
   • Implementing dependency tracking between transactions

4. Mutation Factory: Implement the useOptimisticMutation hook that creates mutation functions with optional transaction creation capabilities.

5. Transaction Object: Create a new Transaction class with methods for:

   • mutate(callback): Execute a function where collection operations are automatically linked to the transaction
   • commit(): Finalize the transaction and persist changes
   • rollback(): Discard optimistic changes and cancel the transaction
   • getState(): Get the current state of the transaction (pending, committing, completed, failed)

6. Dependency Tracking: Implement tracking of transaction dependencies to ensure that if a transaction fails, any dependent transactions are also rolled back.

Transaction Behavior

1. Building on Optimistic Changes: Each new transaction will build on top of existing optimistic changes from other active transactions, as well as synced changes from the backend.

2. Concurrent Transactions: Multiple transactions can be active simultaneously, with the understanding that conflicts will be resolved by the backend or through defensive programming. The library will not enforce strict isolation between transactions.

3. Error Handling: Transaction errors will be exposed through the promise returned by transaction.commit(). If a transaction fails, its optimistic changes will be rolled back, along with any dependent transactions.

4. Collection Awareness: Collections will be informed when they are part of a transaction, but they will not own or manage the transaction lifecycle.

5. Mutation Function: The transaction's mutationFn will receive an array of all mutations across all collections involved in the transaction, allowing for flexible backend integration.

Definition of Success

The explicit transaction feature will be considered successful if it:

1. Makes transactions first-class citizens: Developers can create, control, and interact with transactions directly, building more complex workflows on top of the transaction system.

2. Enables cross-collection operations: Developers can atomically update related data across multiple collections as part of a single transaction.

3. Provides flexible mutation persistence: The transaction's mutation function can handle persisting changes to multiple collections in the most appropriate way for the backend, whether that's a single API call or multiple coordinated calls.

4. Supports explicit lifecycle control: Developers can explicitly control when a transaction is created, accumulated, committed, or rolled back.

5. Enables multi-step operations: Developers can build up a transaction across multiple user interactions, such as multi-step forms or wizard interfaces.

6. Provides a clean, intuitive API: The API is easy to understand and use, with minimal boilerplate and clear patterns for common use cases.

7. Passes comprehensive tests: The implementation passes tests for:

   • Cross-collection mutations being correctly tied to the right transaction
   • Multiple calls to .mutate() accumulating optimistic updates correctly
   • Transaction dependencies being properly tracked and handled
   • Error handling and rollback working correctly

8. Maintains performance: The feature does not significantly impact performance compared to the existing transaction system.

9. Solves real-world use cases: The feature enables developers to implement common patterns like the user-team example without workarounds or compromises.