Distributed Transactions from the PHP SDK

Distributed ACID Transactions in Couchbase SDKs

This document presents a practical HOWTO on using Couchbase transactions, following on from our transactions documentation.

Requirements

Couchbase Server 6.6.1 or above.
Couchbase PHP SDK 4.0.0 or above.
NTP should be configured so nodes of the Couchbase cluster are in sync with time.
The application, if it is using extended attributes (XATTRs), must avoid using the XATTR field txn, which is reserved for Couchbase use.

If using a single node cluster (for example, during development), then note that the default number of replicas for a newly created bucket is 1. If left at this default, then all Key-Value writes performed at with durability will fail with a {durability-exception}. In turn this will cause all transactions (which perform all Key-Value writes durably) to fail. This setting can be changed via GUI or command line. If the bucket already existed, then the server needs to be rebalanced for the setting to take affect.

Getting Started

Couchbase transactions require no additional components or services to be configured. Simply install the most recent version of the SDK.

Configuration

Transactions can optionally be globally configured when configuring the Cluster. For example, if you want to change the level of durability which must be attained, this can be configured as part of the connect options:

$options = new ClusterOptions();
$options->credentials($CB_USER, $CB_PASS);

$transactions_configuration = new TransactionsConfiguration();
$transactions_configuration->durabilityLevel($durabilityLevel); // Couchbase\DurabilityLevel::PERSIST_TO_MAJORITY etc.
$options->transactionsOptions($transactions_configuration);

$cluster = new Cluster($CB_HOST, $options);

The default configuration will perform all writes with the durability setting Majority, ensuring that each write is available in-memory on the majority of replicas before the transaction continues. There are two higher durability settings available that will additionally wait for all mutations to be written to physical storage on either the active or the majority of replicas, before continuing. This further increases safety, at a cost of additional latency.

A level of None is present but its use is discouraged and unsupported. If durability is set to None, then ACID semantics are not guaranteed.

Creating a Transaction

A core idea of Couchbase transactions is that an application supplies the logic for the transaction inside a lambda, including any conditional logic required, and the transaction is then automatically committed. If a transient error occurs, such as a temporary conflict with another transaction, then the transaction will rollback what has been done so far and run the lambda again. The application does not have to do these retries and error handling itself.

Each run of the lambda is called an attempt, inside an overall transaction.

try {
  $cluster->transactions()->run(
    function (TransactionAttemptContext $ctx) {
    // `$ctx` is a TransactionAttemptContext which permits getting, inserting,
    // removing and replacing documents, performing N1QL queries, etc.
    // … Your transaction logic here …
    // Committing is implicit at the end of the lambda.
    });
}
catch (\Couchbase\Exception\TransactionOperationFailedException $e) {
    echo "Transaction did not reach commit point: $e\n";
}
catch (\Couchbase\Exception\TransactionException $e) { 
// TODO check is this equivalent to TransactionCommitAmbiguousError as per Node examples?
  echo "Transaction possibly committed: $e\n";
}

The lambda gets passed {lambda-attempt-ctx} object, generally referred to as ctx here.

Since the lambda may be rerun multiple times, it is important that it does not contain any side effects. In particular, you should never perform regular operations on a Collection, such as {collection-insert}, inside the lambda. Such operations may be performed multiple times, and will not be performed transactionally. Instead such operations must be done through the ctx object, e.g. {ctx-insert}.

Examples

Here is a quick summary of the main transaction operations. They are described in more detail below.


try {
  $cluster->transactions()->run(
    function (TransactionAttemptContext $ctx) use ($collection) {
      // Inserting a doc:
      $ctx->insert($collection, 'doc-a', []);

      // Getting documents:
      $docA = $ctx->get($collection, 'doc-a');

      // Replacing a doc:
      $docB = $ctx->get($collection, 'doc-b');
      $content = $docB->content;
      $newContent = array_merge(
        ["transactions" => "are awesome"],
        $content);
      
      $ctx->replace($docB, $newContent);

      // Removing a doc:
      $docC = $ctx->get($collection, 'doc-c');
      $ctx->remove($docC);

      // Performing a SELECT N1QL query against a scope:
      $qr = $ctx->query('SELECT * FROM hotel WHERE country = $1',
        [ "scope" => "inventory",
          "parameters" => ["United Kingdom"]
      ]);
      echo $qr->rows();

      $ctx->query('UPDATE route SET airlineid = $1 WHERE airline = $2',
        [ "scope" => "inventory",
          "parameters" => ['airline_137', 'AF'] ] );
    });
}
catch (\Couchbase\Exception\TransactionOperationFailedException $e) {
    echo "Transaction did not reach commit point: $e\n";
}
catch (\Couchbase\Exception\TransactionException $e) { 
// TODO check is this equivalent to TransactionCommitAmbiguousError as per Node examples?
  echo "Transaction possibly committed: $e\n";
}

Transaction Mechanics

While this document is focussed on presenting how transactions are used at the API level, it is useful to have a high-level understanding of the mechanics. Reading this section is completely optional.

Recall that the application-provided lambda (containing the transaction logic) may be run multiple times by Couchbase transactions. Each such run is called an attempt inside the overall transaction.

Active Transaction Record Entries

The first mechanic is that each of these attempts adds an entry to a metadata document in the Couchbase cluster. These metadata documents:

Are named Active Transaction Records, or ATRs.
Are created and maintained automatically.
Begin with "_txn:atr-".
Each contain entries for multiple attempts.
Are viewable, and they should not be modified externally.

Each such ATR entry stores some metadata and, crucially, whether the attempt has committed or not. In this way, the entry acts as the single point of truth for the transaction, which is essential for providing an 'atomic commit' during reads.

Staged Mutations

The second mechanic is that mutating a document inside a transaction, does not directly change the body of the document. Instead, the post-transaction version of the document is staged alongside the document (technically in its extended attributes (XATTRs)). In this way, all changes are invisible to all parts of the Couchbase Data Platform until the commit point is reached.

These staged document changes effectively act as a lock against other transactions trying to modify the document, preventing write-write conflicts.

Cleanup

There are safety mechanisms to ensure that leftover staged changes from a failed transaction cannot block live transactions indefinitely. These include an asynchronous cleanup process that is started with the creation of the Transactions object, and scans for expired transactions created by any application, on all buckets. These include an asynchronous cleanup process that is started with the first transaction, and scans for expired transactions created by any application, on the relevant collections.

Note that if an application is not running, then this cleanup is also not running.

The cleanup process is detailed below in [Asynchronous Cleanup].

Committing

Only once the lambda has successfully run to conclusion, will the attempt be committed. This updates the ATR entry, which is used as a signal by transactional actors to use the post-transaction version of a document from its XATTRs. Hence, updating the ATR entry is an 'atomic commit' switch for the transaction.

After this commit point is reached, the individual documents will be committed (or "unstaged"). This provides an eventually consistent commit for non-transactional actors.

Key-Value Mutations

Replacing

Replacing a document requires a $ctx→get() call first. This is necessary so the transaction can check that the document is not involved in another transaction. If it is, then the transaction will handle this at the $ctx→replace() point. Generally, this involves rolling back what has been done so far, and retrying the lambda. Handling errors should be done through try/catch as in the example above.

$cluster->transactions()->run(
  function (TransactionAttemptContext $ctx) use ($collection) {
    $doc = $ctx->get($collection, "doc-b");
    $content = $doc->content();
    $newContent = array_merge(
      ["transactions" => "are awesome"],
      $content);
    
    $ctx->replace($doc, $newContent);
});

Removing

As with replaces, removing a document requires a $ctx→get() call first.

$cluster->transactions()->run(
  function (TransactionAttemptContext $ctx) use ($collection) {
    $doc = $ctx->get($collection, "docId");
    $ctx->remove($doc);
});

Inserting

$cluster->transactions()->run(
  function (TransactionAttemptContext $ctx) use ($collection) { 
    $adoc = $ctx->insert($collection, "docId", []);
});

Key-Value Reads

From a transaction context you may get a document:

$cluster->transactions()->run(
  function (TransactionAttemptContext $ctx) use ($collection) { 
    $docA = $ctx->get($collection, "doc-a");
  });

get will cause the transaction to fail with TransactionOperationFailedException (after rolling back any changes, of course).

Gets will 'read your own writes', e.g. this will succeed:

$cluster->transactions()->run(
  function (TransactionAttemptContext $ctx) use ($collection) { 
  $docId = 'docId';
  $ctx->insert($collection, $docId, []);

  $doc = $ctx->get($collection, $docId);
});

N1QL Queries

As of Couchbase Server 7.0, N1QL queries may be used inside the transaction lambda, freely mixed with Key-Value operations.

BEGIN TRANSACTION

There are two ways to initiate a transaction with Couchbase 7.x: via a transactions library, and via the query service directly using BEGIN TRANSACTION. The latter is intended for those using query via the REST API, or using the query workbench in the UI, and it is strongly recommended that application writers instead use the transactions library. There are two ways to initiate a transaction with Couchbase 7.x: via the SDK, and via the query service directly using BEGIN TRANSACTION. The latter is intended for those using query via the REST API, or using the query workbench in the UI, and it is strongly recommended that application writers instead use the SDK. This provides these benefits:

It automatically handles errors and retrying.
It allows Key-Value operations and N1QL queries to be freely mixed.
It takes care of issuing BEGIN TRANSACTION, END TRANSACTION, COMMIT and ROLLBACK automatically. These become an implementation detail and you should not use these statements inside the lambda.

Supported N1QL

The majority of N1QL DML statements are permitted within a transaction. Specifically: INSERT, UPSERT, DELETE, UPDATE, MERGE and SELECT are supported.

DDL statements, such as CREATE INDEX, are not.

Using N1QL

If you already use N1QL from the PHP SDK, then its use in transactions is very similar. it returns the same QueryResult you are used to, and takes most of the same options.

You must take care to write $ctx→query() inside the lambda however, rather than $cluster→query() or $scope→query().

An example of SELECTing some rows from the travel-sample bucket:

$inventory = $cluster->bucket('travel-sample')->scope('inventory');

$cluster->transactions()->run(
  function (TransactionAttemptContext $ctx) { 
    $st = "SELECT * FROM `travel-sample`.inventory.hotel WHERE country = $1";
    $qr = $ctx->query(
      $st,
      TransactionQueryOptions::build()
        ->positionalParameters(["United Kingdom"]));

    foreach ($qr->rows() as $row) {
      // do something
    }
  }
);

Here is an example combining SELECTs and UPDATEs. It’s possible to call regular PHP functions from the lambda, as shown here, permitting complex logic to be performed. Just remember that since the lambda may be called multiple times, so may the method.

$cluster->transactions()->run(
  function (TransactionAttemptContext $ctx) use ($hotelChain, $country) { 
    // Find all hotels of the chain
    $qr = $ctx->query(
      'SELECT reviews FROM `travel-sample`.inventory.hotel WHERE url LIKE $1 AND country = $2',
      TransactionQueryOptions::build()
        ->positionalParameters([$hotelChain, $country]));
        // ->scopeQualifier("`travel-sample`.`inventory`"));

    // This function (not provided here) will use a trained machine learning model to provide a
    // suitable price based on recent customer reviews.
    function priceFromRecentReviews(Couchbase\QueryResult $qr) {
        // this would call a trained ML model to get the best price
        return 99.98;
    }
    $updatedPrice = priceFromRecentReviews($qr);

    // Set the price of all hotels in the chain
    $ctx->query(
      'UPDATE `travel-sample`.inventory.hotel SET price = $1 WHERE url LIKE $2 AND country = $3',
      TransactionQueryOptions::build()
        ->positionalParameters([$updatedPrice, $hotelChain, $country]));
        // ->scopeQualifier("`travel-sample`.`inventory`"));
  }
);

Read Your Own Writes

As with Key-Value operations, N1QL queries support Read Your Own Writes.

This example shows INSERTing a document and then SELECTing it again.

$cluster->transactions()->run(
  function (TransactionAttemptContext $ctx) { 
    $ctx->query("INSERT INTO `travel-sample`.inventory.airline VALUES ('doc-c', {'hello':'world'})"); (1)
    $st = "SELECT `default`.* FROM `travel-sample`.inventory.airline WHERE META().id = 'doc-c'"; (2)
    $qr = $ctx->query($st);
});

1	The inserted document is only staged at this point. as the transaction has not yet committed. Other transactions, and other non-transactional actors, will not be able to see this staged insert yet.
2	But the SELECT can, as we are reading a mutation staged inside the same transaction.

Mixing Key-Value and N1QL

Key-Value operations and queries can be freely intermixed, and will interact with each other as you would expect.

In this example we insert a document with Key-Value, and read it with a SELECT.

$cluster->transactions()->run(
  function (TransactionAttemptContext $ctx) {
    $qr = $ctx->query("UPDATE `travel-sample`.inventory.hotel SET price = 99.00 WHERE name LIKE \"Marriott%\"");
    if ($qr->metaData()->metrics()["mutationCount"] != 2) {
      // throw new \Exception("Mutation count not the expected amount.");
      echo "WARN: Mutation count not the expected amount.\n";
    }
  }
);

1	As with the 'Read Your Own Writes' example, here the insert is only staged, and so it is not visible to other transactions or non-transactional actors.
2	But the SELECT can view it, as the insert was in the same transaction.

Query Options

Query options can be provided via TransactionsQueryOptions, which provides a subset of the options in the PHP SDK’s QueryOptions.


$cluster->transactions()->run(
  function (TransactionAttemptContext $ctx) {
    $txQo = TransactionQueryOptions::build()
      ->readonly(false)
      ->positionalParameters(["key", "value"]);
      
    $ctx->query(
      "UPSERT INTO `travel-sample`.inventory.airline VALUES ('docId', {\$1:\$2})",
      $txQo);
});

See the QueryOptions documentation for full details on the supported parameters.

Query Performance Advice

This section is optional reading, and only for those looking to maximize transactions performance.

After the first query statement in a transaction, subsequent Key-Value operations in the lambda are converted into N1QL and executed by the query service rather than the Key-Value data service. The operation will behave identically, and this implementation detail can largely be ignored, except for this caveat:

These converted Key-Value operations are likely to be slightly slower, as the query service is optimised for statements involving multiple documents. Those looking for the maximum possible performance are recommended to put Key-Value operations before the first query in the lambda, if possible.

Single Query Transactions

This section is mainly of use for those wanting to do large, bulk-loading transactions.

The query service maintains where required some in-memory state for each document in a transaction, that is freed on commit or rollback. For most use-cases this presents no issue, but there are some workloads, such as bulk loading many documents, where this could exceed the server resources allocated to the service. Solutions to this include breaking the workload up into smaller batches, and allocating additional memory to the query service. Alternatively, single query transaction, described here, may be used.

Single query transactions have these characteristics:

They have greatly reduced memory usage inside the query service.
As the name suggests, they consist of exactly one query, and no Key-Value operations.

You will see reference elsewhere in Couchbase documentation to the tximplicit query parameter. Single query transactions internally are setting this parameter. In addition, they provide automatic error and retry handling.

Single query transactions may be initiated like so:

try {
  $cluster->transactions()->run(
    function (TransactionAttemptContext $ctx) {
      $bulkLoadStatement = "..."; // a bulk-loading N1QL statement not provided here

      $ctx->query($bulkLoadStatement);
  });
}
catch (\Couchbase\Exception\TransactionOperationFailedException $e) {
  echo "Transaction did not reach commit point\n";
}
catch (\Couchbase\Exception\TransactionException $e) { 
  echo "Transaction possibly committed\n";
}

Committing

Committing is automatic: If no exception is thrown, the transaction will be committed at the end of the code block with the transaction context. If you want to rollback the transaction, simply throw an exception. Transactions may rollback from the transaction logic itself, various failure conditions, or from your application logic by throwing an exception.

As soon as the transaction is committed, all its changes will be atomically visible to reads from other transactions. The changes will also be committed (or "unstaged") so they are visible to non-transactional actors, in an eventually consistent fashion.

Commit is final: after the transaction is committed, it cannot be rolled back, and no further operations are allowed on it.

An asynchronous cleanup process ensures that once the transaction reaches the commit point, it will be fully committed - even if the application crashes.

A Full Transaction Example

Let’s pull together everything so far into a more real-world example of a transaction.

This example simulates a simple Massively Multiplayer Online game, and includes documents representing:

Players, with experience points and levels;
Monsters, with hitpoints, and the number of experience points a player earns from their death.

In this example, the player is dealing damage to the monster. The player’s client has sent this instruction to a central server, where we’re going to record that action. We’re going to do this in a transaction, as we don’t want a situation where the monster is killed, but we fail to update the player’s document with the earned experience.

(Though this is just a demo - in reality, the game would likely live with the small risk and limited impact of this, rather than pay the performance cost for using a transaction.)

function playerHitsMonster($damage, $playerId, $monsterId, $cluster, $collection) {
  try {
    $cluster->transactions()->run(
      function (TransactionAttemptContext $ctx) use ($damage, $playerId, $monsterId, $collection) {
        $monsterDoc = $ctx->get($collection, $monsterId);
        $monsterContent = $monsterDoc->content();
        $playerDoc = $ctx->get($playerId, $monsterId);
        $playerContent = $playerDoc->content();

        $monsterHitpoints = $monsterContent["hitpoints"];
        $monsterNewHitpoints = $monsterHitpoints - $damage;

        if ($monsterNewHitpoints <= 0) {
          // Monster is killed. The remove is just for demoing, and a more realistic
          // example would set a "dead" flag or similar.
          $ctx->remove($monsterDoc);

          // The player earns experience for killing the monster
          $experienceForKillingMonster = $monsterContent["experienceWhenKilled"];
          $playerExperience = $playerContent["experience"];
          $playerNewExperience = $playerExperience + $experienceForKillingMonster;
          $playerNewLevel =
            calculateLevelForExperience($playerNewExperience);

          $playerContent['experience'] = $playerNewExperience;
          $playerContent['level'] = $playerNewLevel;

          $ctx->replace($playerDoc, $playerContent);
      }
    });
  }
  catch (\Couchbase\Exception\TransactionOperationFailedException $e) {
    echo "Transaction did not reach commit point\n";
    // The operation failed. Both the monster and the player will be untouched.
    //
    // Situations that can cause this would include either the monster
    // or player not existing (as get is used), or a persistent
    // failure to be able to commit the transaction, for example on
    // prolonged node failure.
  }
  catch (\Couchbase\Exception\TransactionException $e) { 
    // TODO check is this equivalent to TransactionCommitAmbiguousError as per Node examples?
    // Indicates the state of a transaction ended as ambiguous and may or
    // may not have committed successfully.
    //
    // Situations that may cause this would include a network or node failure
    // after the transactions operations completed and committed, but before the
    // commit result was returned to the client
    echo "Transaction possibly committed\n";
  }
}

Concurrency with Non-Transactional Writes

This release of transactions for Couchbase requires a degree of co-operation from the application. Specifically, the application should ensure that non-transactional writes are never done concurrently with transactional writes, on the same document.

This requirement is to ensure that the strong Key-Value performance of Couchbase was not compromised. A key philosophy of our transactions is that you 'pay only for what you use'.

If two such writes do conflict then the behaviour is undefined: either write may 'win', overwriting the other. This still applies if the non-transactional write is using CAS.

Note this only applies to writes. Any non-transactional reads concurrent with transactions are fine, and are at a Read Committed level.

Rollback

If an exception is thrown, either by the application from the lambda, or by the transactions library, then that attempt is rolled back. The transaction logic may or may not be retried, depending on the exception.

The application can use this to signal why it triggered a rollback, as so:

try {
  $cluster->transactions()->run(
    function (TransactionAttemptContext $ctx) use ($collection, $costOfItem) {
      $customer = $ctx->get($collection, "customer-name");

      if ($customer->content()["balance"] < $costOfItem) {
        throw new \InsufficientBalanceError("Transaction failed, customer does not have enough funds.");
      }
      // else continue transaction
  });
}
catch (\Couchbase\Exception\TransactionOperationFailedException $e) {
  // This exception can only be thrown at the commit point, after the
  // BalanceInsufficient logic has been passed
}
catch (InsufficientBalanceError $e) {
  echo "user had insufficient balance";
}

After a transaction is rolled back, it cannot be committed, no further operations are allowed on it, and the system will not try to automatically commit it at the end of the code block.

Error Handling

As discussed previously, Couchbase transactions will attempt to resolve many errors for you, through a combination of retrying individual operations and the application’s lambda. This includes some transient server errors, and conflicts with other transactions.

But there are situations that cannot be resolved, and total failure is indicated to the application via errors. These situations include:

Any error thrown by your transaction lambda, either deliberately or through an application logic bug.
Attempting to insert a document that already exists.
Attempting to remove or replace a document that does not exist.
Calling {ctx-get} on a document key that does not exist (if the resultant exception is not caught).

Once one of these errors occurs, the current attempt is irrevocably failed (though the transaction may retry the lambda to make a new attempt). It is not possible for the application to catch the failure and continue (with the exception of {ctx-get} raising an error). Once a failure has occurred, all other operations tried in this attempt (including commit) will instantly fail.

Transactions, as they are multi-stage and multi-document, also have a concept of partial success or failure. This is signalled to the application through the {error-unstaging-complete}, described later.

Full Error Handling Example

Pulling all of the above together, this is the suggested best practice for error handling:

try {
  $result = $cluster->transactions()->run(
    function (TransactionAttemptContext $ctx) use ($collection, $costOfItem) {
      // ... transactional code here ...
    }
  );

  // The transaction definitely reached the commit point. Unstaging
  // the individual documents may or may not have completed

  if (! $result->unstagingComplete) {
      // In rare cases, the application may require the commit to have
      // completed.  (Recall that the asynchronous cleanup process is
      // still working to complete the commit.)
      // The next step is application-dependent.
  }
}
catch (\Couchbase\Exception\TransactionOperationFailedException $e) {
  echo "Transaction did not reach commit point\n";
}
catch (\Couchbase\Exception\TransactionException $e) { 
  echo "Transaction possibly committed\n";
}

Monitoring Cleanup

To monitor cleanup, increase the verbosity on the logging.

Logging

To aid troubleshooting, raise the log level on the SDK.

Please see the PHP SDK logging documentation for details.

Concurrent Operations

The API allows operations to be performed concurrently inside a transaction, which can assist performance. There are two rules the application needs to follow:

The first mutation must be performed alone, in serial. This is because the first mutation also triggers the creation of metadata for the transaction.
All concurrent operations must be allowed to complete fully, so the transaction library can track which operations need to be rolled back in the event of failure. This means the application must 'swallow' the error, but record that an error occurred, and then at the end of the concurrent operations, if an error occurred, throw an error to cause the transaction to retry.

Distributed Transactions from the PHP SDK

Requirements

Getting Started

Configuration

Creating a Transaction

Examples

Transaction Mechanics

Active Transaction Record Entries

Staged Mutations

Cleanup

Committing

Key-Value Mutations

Replacing

Removing

Inserting

Key-Value Reads

N1QL Queries

BEGIN TRANSACTION

Supported N1QL

Using N1QL

Read Your Own Writes

Mixing Key-Value and N1QL

Query Options

Query Performance Advice

Single Query Transactions

Committing

A Full Transaction Example

Concurrency with Non-Transactional Writes

Rollback

Error Handling

Full Error Handling Example

Monitoring Cleanup

Logging

Concurrent Operations

Further Reading