Migrating from SDK2 to SDK3 API

The 3.0 API breaks the existing 2.0 APIs in order to provide a number of improvements. Collections and Scopes are introduced. The Document class and structure has been completely removed from the API, and the returned value is now Result. Retry behaviour is more proactive, and lazy bootstrapping moves all error handling to a single place. Individual behaviour changes across services are explained here.

Unresolved include directive in modules/project-docs/pages/migrating-sdk-code-to-3.n.adoc - include::7.0@sdk:shared:partial$migration.adoc[]

As an example here is a KeyValue document fetch:

const result = await collection.get(key);
document = result.value;

Compare this to a N1QL query:

async function queryNamed() {
  const query = `
    SELECT airportname, city FROM \`travel-sample\` 
    WHERE type=$TYPE 
      AND city=$CITY;
  `
  const options = { parameters: { TYPE: 'airport', CITY: 'Reno' } }

  try {
    let result = await cluster.query(query, options)
    console.log("Result:", result)
    return result
  } catch (error) {
    console.error('Query failed: ', error)
  }
}

Unresolved include directive in modules/project-docs/pages/migrating-sdk-code-to-3.n.adoc - include::7.0@sdk:shared:partial$migration.adoc[]

Installation and Configuration

The Node.js SDK 3.x is available through npm, just like the previous generation. Please see the Release Notes for up-to-date information.

SDK 3 depends on the following ones instead:

  "dependencies": {
    "bindings": "^1.5.0",
    "debug": "^4.1.1",
    "nan": "^2.14.0",
    "parse-duration": "^0.1.1",
    "prebuild-install": "^5.3.2",
    "qs": "^6.9.0"
  }

Connection to the Cluster

const options = { username:"Administrator", password:"password"};
cluster = await couchbase.connect( "http://127.0.0.1", options);
bucket = cluster.bucket("travel-sample");
collection = bucket.scope("inventory").collection("airport");

Similar to SDK 2, if you create your own ClusterEnvironment the SDK will not shut it down for you — you need to do this manually at the end of the program lifetime:

cluster.close();

Connection String Url Query Parameters

const options = { username:"Administrator", password:"password"};
cluster = await couchbase.connect( "http://127.0.0.1/?query_timeout=2000", options);

Authentication

Connecting with certificate-based authentication.

// see sdk-examples/etc for creation of certificates
const here = process.cwd();
const truststorepath = here + "/" + '../etc/x509-cert/SSLCA/clientdir/trust.pem';
const certpath = here + "/" + '../etc/x509-cert/SSLCA/clientdir/client.pem';
const keypath = here + "/" + '../etc/x509-cert/SSLCA/clientdir'; // gets /client.key from cluster.bucket()

// Setup Cluster Connection Object
const options = {username: 'testuser', password: 'password'};
var cluster = new couchbase.Cluster(
    'couchbases://127.0.0.1/travel-sample' +
    '?truststorepath=' + truststorepath +
    '&certpath=' + certpath +
    '&keypath=' + keypath, options);

Please see the documentation on certificate-based authentication for detailed information on how to configure this properly.

Connection Lifecycle

From a high-level perspective, bootstrapping and shutdown is very similar to SDK 2. One notable difference is that the Collection is introduced and that the individual methods like bucket immediately return, and do not throw an exception. Compare SDK 2: the openBucket method would not work if it could not open the bucket.

The reason behind this change is that even if a bucket can be opened, a millisecond later it may not be available any more. All this state has been moved into the actual operation so there is only a single place where the error handling needs to take place. This simplifies error handling and retry logic for an application.

In SDK 2, you connected, opened a bucket, performed a KV op, and disconnected like this:

const cluster = new couchbase.Cluster("127.0.0.1");
cluster.authenticate("user", "pass");
const bucket = cluster.openBucket("travel-sample");

const getResult = bucket.get("airline_10");

cluster.close();

Here is the SDK 3 equivalent:

const options = { username:"Administrator", password:"password"};
cluster = await couchbase.connect( "http://127.0.0.1", options);
bucket = cluster.bucket("travel-sample");
collection = bucket.scope("inventory").collection("airport");

const getResult = await collection.get("airport_1254");

cluster.close();

Collections is generally available from Couchbase Server 7.0 release, but the SDK already encoded it in its API to be future-proof. If you are using a Couchbase Server version which does not support Collections, always use the defaultCollection() method to access the KV API; it will map to the full bucket.

You’ll notice that bucket(String) returns immediately, even if the bucket resources are not completely opened. This means that the subsequent get operation may be dispatched even before the socket is open in the background. The SDK will handle this case transparently, and reschedule the operation until the bucket is opened properly. This also means that if a bucket could not be opened (say, because no server was reachable) the operation will time out. Please check the logs to see the cause of the timeout (in this case, you’ll see socket connect rejections).

Also note, you will now find Query, Search, and Analytics at the Cluster level. This is where they logically belong. If you are using Couchbase Server 6.5 or later, you will be able to perform cluster-level queries even if no bucket is open. If you are using an earlier version of the cluster you must open at least one bucket, otherwise cluster-level queries will fail.

Serialization and Transcoding

In SDK 2 the main method to control transcoding was through providing different Document instances (which in turn had their own transcoder associated), such as the JsonDocument. This only worked for the KV APIs though — Query, Search, Views, and other services exposed their JSON rows/hits in different ways. All of this has been unified in SDK 3 under a single concept: serializers and transcoders.

By default, all KV APIs transcode to and from JSON — you can also provide Javascript objects which you couldn’t in the past.

const  upsertResult = await collection.upsert("mydoc-id", { myvalue: "me"});
const getResult = await collection.get("mydoc-id");
console.log(getResult);

If you want to write binary data, you can use a new RawBinaryTranscoder():

const content = Buffer.from("some data to become binary");
const  upsertResult = await collection.upsert(
  "mydoc-id",
  content,
  {transcoder:new RawBinaryTranscoder()}
);
const getResult = await collection.get("mydoc-id");
console.log(getResult);

Exception Handling

How to handle exceptions is unchanged from SDK 2. You should still use try/catch on the blocking APIs and the corresponding reactive/async methods on the other APIs. There have been changes made in the following areas:

Exception hierachy and naming.
Proactive retry where possible.

Exception hierachy

The exception hierachy is now flat and unified under a CouchbaseError. Each CouchbaseError has an associated ErrorContext which is populated with as much info as possible and then dumped alongside the stack trace if an error happens.

Here is an example of the error context if a N1QL query is performed with an invalid syntax (i.e. select 1= from):

Exception in thread "main" com.couchbase.client.core.error.ParsingFailedException: Parsing of the input failed {"completed":true,"coreId":1,"errors":[{"code":3000,"message":"syntax error - at from"}],"idempotent":false,"lastDispatchedFrom":"127.0.0.1:62253","lastDispatchedTo":"127.0.0.1:8093","requestId":3,"requestType":"QueryRequest","retried":11,"retryReasons":["ENDPOINT_TEMPORARILY_NOT_AVAILABLE","BUCKET_OPEN_IN_PROGRESS"],"service":{"operationId":"9111b961-e585-42f2-9cab-e1501da7a40b","statement":"select 1= from","type":"query"},"timeoutMs":75000,"timings":{"dispatchMicros":15599,"totalMicros":1641134}}

Proactive Retry

One reason why the APIs do not expose a long list of exceptions is that the SDK now retries as many operations as it can if it can do so safely. This depends on the type of operation (idempotent or not), in which state of processing it is (already dispatched or not), and what the actual response code is if it arrived already. As a result, many transient cases — such as locked documents, or temporary failure — are now retried by default and should less often impact applications. It also means, when migrating to the new SDK API, you may observe a longer period of time until an error is returned by default.

Operations are retried by default as described above with the default BestEffortRetryStrategy.

Logging and Events

Migrating Services

The following section discusses each service in detail and covers specific bits that have not been covered by the more generic sections above.

Key Value

The Key/Value (KV) API is now located under the Collection interface, so even if you do not use collections, the defaultCollection() needs to be opened in order to access it.

The following table describes the SDK 2 KV APIs and where they are now located in SDK 3:

Table 1. SDK 2.x KV API vs. SDK 3.x KV API
SDK 2	SDK 3
`bucket.upsert`	`collection.upsert`
`bucket.get`	`collection.get`
`bucket.exists`	`collection.exists`
`bucket.getFromReplica`	`collection.getAnyReplica` and `collection.getAllReplicas`
`bucket.getAndLock`	`collection.getAndLock`
`bucket.getAndTouch`	`collection.getAndTouch`
`bucket.insert`	`collection.insert`
`bucket.upsert`	`collection.upsert`
`bucket.replace`	`collection.replace`
`bucket.remove`	`collection.remove`
`bucket.unlock`	`collection.unlock`
`bucket.touch`	`collection.touch`
`bucket.lookupIn`	`collection.lookupIn`
`bucket.mutateIn`	`collection.mutateIn`
`bucket.counter`	`binarycollection.increment` and `binarycollection.decrement`
`bucket.append`	`binarycollection.append`
`bucket.prepend`	`binarycollection.prepend`

In addition, the datastructure APIs have been renamed and moved:

Table 2. Datastructure API Changes
SDK 2	SDK 3
`bucket.mapAdd`	`collection.map`
`bucket.mapGet`	`collection.map`
`bucket.mapRemove`	`collection.map`
`bucket.mapSize`	`collection.map`
`bucket.listGet`	`collection.list`
`bucket.listAppend`	`collection.list`
`bucket.listRemove`	`collection.list`
`bucket.listPrepend`	`collection.list`
`bucket.listSet`	`collection.list`
`bucket.listSize`	`collection.list`
`bucket.setAdd`	`collection.set`
`bucket.setContains`	`collection.set`
`bucket.setRemove`	`collection.set`
`bucket.setSize`	`collection.set`
`bucket.queuePush`	`collection.queue`
`bucket.queuePop`	`collection.queue`

There are two important API changes:

On the request side, overloads have been reduced and moved under a Options block
On the response side, the return types have been unified.

The signatures now look very similar. The concept of the Document as a type is gone in SDK 3 and instead you need to pass in the properties explicitly. This makes it very clear what is returned, especially on the response side.

Thus, the get method does not return a Document but a GetResult instead, and the upsert does not return a Document but a MutationResult. Each of those results only contains the field that the specific method can actually return, making it impossible to accidentally try to access the expiry on the Document after a mutation, for example.

Instead of having many overloads, all optional params are now part of the Option block. All required params are still part of the method signature, making it clear what is required and what is not (or has default values applied if not overridden).

The timeout can be overridden on every operation and now takes a Duration from java 8. Compare SDK 2 and SDK 3 custom timeout setting:

// SDK 2 custom timeout
bucket.get("mydoc-id", 5000);

// SDK 3 custom timeout
const getResult = await collection.get( "airport_1254", {timeout : 2000});

In SDK 2, the getFromReplica method had a ReplicaMode argument which allowed to customize its behavior on how many replicas should be reached. We have identified this as a potential source of confusion and as a result split it up in two methods that simplify usage significantly. There is now a getAllReplicas method and a getAnyReplica method.

getAllReplicas asks the active node and all available replicas and returns the results as a stream.
getAnyReplica uses getAllReplicas, and returns the first result obtained.

Unless you want to build some kind of consensus between the different replica responses, we recommend getAnyReplica for a fallback to a regular get when the active node times out.

Operations which cannot be performed on JSON documents have been moved to the binarycollection, accessible through Collection.binary(). These operations include append, prepend, increment, and decrement (previously called counter in SDK 2). These operations should only be used against non-json data. Similar functionality is available through mutateIn on JSON documents.

Query

N1QL querying is now available at the Cluster level instead of the bucket level, because you can also write N1QL queries that span multiple buckets. Compare a simple N1QL query from SDK 2 with its SDK 3 equivalent:

// SDK 2 simple query
queryResult = await bucket.query(
      couchbase.N1qlQuery.fromString('SELECT * FROM `travel-sample` WHERE city=$1 LIMIT 10'),
      [ 'Paris' ]);

queryResult.rows.forEach((row)=>{
      console.log(row);
   });

// SDK 3 simple query
const queryResult = await cluster.query("SELECT * FROM `travel-sample` WHERE city=$1 LIMIT 10", { parameters: ['Paris']});
queryResult.rows.forEach((row)=>{
   console.log(row);
});

Note that there is no N1qlQuery.fromString any more — and query parameters argument has been moved to the options parameter for consistency reasons. The following shows how to do named and positional parameters in SDK 3:

// SDK 3 named parameters
const queryResultNamed = await cluster.query("SELECT * FROM `travel-sample` WHERE city=$CITY LIMIT 10", { parameters: {CITY:'Paris'}});
queryResultNamed.rows.forEach((row)=>{
   console.log(row);
});
// SDK 3 positional parameters
const queryResultPositional = await cluster.query("SELECT * FROM `travel-sample` WHERE city=$1 LIMIT 10", { parameters: ['Paris']});
queryResultPositional.rows.forEach((row)=>{
   console.log(row);
});

Much of the non-row metadata has been moved into a specific QueryMetaData section:

It is no longer necessary to check for a specific error in the stream: if an error happened during processing it will throw an exception at the top level of the query. The reactive streaming API will terminate the rows' Flux with an exception as well as soon as it is discovered. This makes error handling much easier in both the blocking and non-blocking cases.

While in SDK 2 you had to manually check for errors (otherwise you’d get an empty row collection):

const queryResult = bucket.query(N1qlQuery.simple("select 1="));
if (!queryResult.errors.isEmpty()) {
  // errors contain [{"msg":"syntax error - at end of input","code":3000}]
}

In SDK 3 the top level query method will throw an exception:

Parsing of the input failed {"completed":true,"coreId":1,"errors":[{"code":3000,"message":"syntax error - at end of input"}],"idempotent":false,"lastDispatchedFrom":"127.0.0.1:51703","lastDispatchedTo":"127.0.0.1:8093","requestId":5,"requestType":"QueryRequest","retried":0,"service":{"operationId":"1c623a77-196a-4890-96cd-9d4f3f596477","statement":"select 1=","type":"query"},"timeoutMs":75000,"timings":{"dispatchMicros":13798,"totalMicros":70789}}
	at com.couchbase.client.java.AsyncUtils.block(AsyncUtils.java:51)
	at com.couchbase.client.java.Cluster.query(Cluster.java:225)

Not only does it throw a CouchbaseError, it also tries to map it to a specific exception type and include extensive contextual information for a better troubleshooting experience.

Analytics

Analytics querying, like N1QL, is also moved to the Cluster level: it is now accessible through the Cluster.analyticsQuery method. As with the Query service, parameters for the Analytics queries have moved into the AnalyticsOptions:

// SDK 3 simple analytics query
const analyticsResult = await cluster.analyticsQuery("select * from `travel-dataset` LIMIT 10");
analyticsResult.rows.forEach((row)=>{
   console.log(row);
});

// SDK 3 named parameters for analytics
const analyticsResult1 = await cluster.analyticsQuery(
  "select * from `travel-sample` where city = $CITY LIMIT 10",
  { parameters : { CITY : "Paris"}}
).catch((e)=>{console.log(e); throw e;});
analyticsResult1.rows.forEach((row)=>{
   console.log(row);
});

// SDK 3 positional parameters for analytics
const analyticsResult2 = await cluster.analyticsQuery(
  "select * from `travel-sample` where city = $1 LIMIT 10",
  { parameters : [ "airport" ] }
);
analyticsResult2.rows.forEach((row)=>{
   console.log(row);
});

Also, errors will now be thrown as top level exceptions and it is no longer necessary to explicitly check for errors:

// SDK 2 error check
AnalyticsQueryResult analyticsQueryResult = b1.query(AnalyticsQuery.simple("select * from foo"));
if (!analyticsQueryResult.errors().isEmpty()) {
  // errors contain [{"msg":"Cannot find dataset foo in dataverse Default nor an alias with name foo! (in line 1, at column 15)","code":24045}]
}

Search

The Search API has changed a bit in SDK 3 so that it aligns with the other query APIs. The type of queries have stayed the same, but all optional parameters moved into SearchOptions. Also, similar to the other query APIs, it is now available at the Cluster level.

Here is a SDK 2 Search query with some options, and its SDK 3 equivalent:

//  SDK 2 search query
const searchResult = bucket.query(
  "airports"",
  { indexName : "airport_view", limit:5, fields : [ "a", "b", "c"]}
  2000,
);
searchResult.rows.forEach((row)=>{
      console.log(row);
   });
}

    // SDK 3 search query
    const ftsQuery =  couchbase.SearchQuery.match("airport");
    const searchResult = await cluster.searchQuery(
      "hotels",
      ftsQuery,
      { timeout:2000,
        limit:5,
        fields : ["a", "b", "c"] },
      (err, res) => {
if(err) console.log(err);
if(res) console.log(res);
      }
    );
    searchResult.rows.forEach((row)=>{
       console.log(row);
    });

If you want to be absolutely sure that you didn’t get only partial data, you can check the error map:

const ftsQuery =  couchbase.SearchQuery.match("airport");
const searchResult = await cluster.searchQuery(
  "hotels",
  ftsQuery,
  { timeout:2000,
    limit:5}
).catch((e)=>{console.log(e); throw e;});
console.log(searchResult);
if (searchResult.meta.status.failed == 0) {
  searchResult.rows.forEach((row)=>{
     console.log(row);
  });
}

Views

Views have stayed at the Bucket level, because it does not have the concept of collections and is scoped at the bucket level on the server as well. The API has stayed mostly the same, the most important change is that staleness is unified under the ViewConsistency enum.

Table 3. View Staleness Mapping
SDK 2	SDK 3
`Stale.TRUE`	`QueryScanConsistency.NotBounded`
`Stale.FALSE`	`QuerywScanConsistency.RequestPlus`
`Stale.UPDATE_AFTER`	`removed`

Compare this SDK 2 view query with its SDK 3 equivalent:

// SDK 2 view query
const query = async bucket.query(
  "design", "view", {limit:5, skip:2},
  10000
);
query.rows.forEach((row)=>{
      console.log(row);
   });
}

// SDK 3 view query
const viewResult = await bucket.viewQuery(
  "dev_airports",
  "airport_view",
  { limit:5, skip:2, timeout:10000 }
).catch((e)=>{console.log(e); throw e;});
viewResult.rows.forEach((row)=>{
   console.log(row);
});

Management APIs

In SDK 2, the management APIs were centralized in the clustermanager at the cluster level and the bucketmanager at the bucket level. Since SDK 3 provides more management APIs, they have been split up in their respective domains. So for example when in SDK 2 you needed to remove a bucket you would call clustermanager.removeBucket you will now find it under bucketmanager.dropBucket. Also, creating a N1QL index now lives in the queryindexmanager, which is accessible through the cluster.

The following table provides a mapping from the SDK 2 management APIs to those of SDK 3:

Table 4. SDK 2.x vs SDK 3.x clustermanager
SDK 2	SDK 3
`clustermanager.info`	removed
`clustermanager.getBuckets`	`bucketmanager.getAllBuckets`
`clustermanager.getBucket`	`bucketmanager.getBucket`
`clustermanager.hasBucket`	removed
`clustermanager.insertBucket`	`bucketmanager.createBucket`
`clustermanager.updateBucket`	`bucketmanager.updateBucket`
`clustermanager.removeBucket`	`bucketmanager.dropBucket`
`clustermanager.upsertUser`	`usermanager.upsertUser`
`clustermanager.removeUser`	`usermanager.dropUser`
`clustermanager.getUsers`	`usermanager.getAllUsers`
`clustermanager.getUser`	`usermanager.getUser`
`clustermanager.apiClient`	removed

Table 5. SDK 2.x vs SDK 3.x bucketmanager
SDK 2	SDK 3
`bucketmanager.info`	removed
`bucketmanager.flush`	`bucketmanager.flushBucket`
`bucketmanager.getDesignDocuments`	`viewindexmanager.getAllDesignDocuments`
`bucketmanager.getDesignDocument`	`viewindexmanager.getDesignDocument`
`bucketmanager.insertDesignDocument`	`viewindexmanager.upsertDesignDocument`
`bucketmanager.upsertDesignDocument`	`viewindexmanager.upsertDesignDocument`
`bucketmanager.removeDesignDocument`	`viewindexmanager.dropDesignDocument`
`bucketmanager.publishDesignDocument`	`viewindexmanager.publishDesignDocument`
`bucketmanager.listN1qlIndexes`	`queryindexmanager.getAllIndexes`
`bucketmanager.createN1qlIndex`	`queryindexmanager.createIndex`
`bucketmanager.createN1qlPrimaryIndex`	`queryindexmanager.createPrimaryIndex`
`bucketmanager.dropN1qlIndex`	`queryindexmanager.dropIndex`
`bucketmanager.dropN1qlPrimaryIndex`	`queryindexmanager.dropPrimaryIndex`
`bucketmanager.buildN1qlDeferredIndexes`	`queryindexmanager.buildDeferredIndexes`
`bucketmanager.watchN1qlIndexes`	`queryindexmanager.watchIndexes`