Error code 2. The service is currently unavailable. #2555

Using saveAll within a Parse Job to insert a batch of new objects into a remote mLab DB fails randomly with error object {"code":2,"message":"the service is currently unavailable"}. This error arose when when saving 6,000 new objects from this Job.

In an attempt to diagnose the issue, I separated the 6,000 objects into batches of objects and used saveAll on each batch. I've tested with batches as small as 10 objects and as many as 250 objects. Although the result is the same, this provided more information...

Most of these batches would be processed successfully, while one or more batches would occasionally fail with this message. Sometimes none fail. As a whole, the save process only succeeds without failure only about 50% of the time. The batches which fail seem completely random, even when using identical input (in this case, a CSV file) for each attempt.

The folks at mLab have investigated on their end for network connectivity issues, failed connection pooling, and other server resource limits. They've been very responsive and helpful. So far, there is no indication of issue on their end. We've tried reproducing this issue against a different, dedicated database cluster with more horsepower and yet the problem remains.

The error object described above appears to be an Internal Error Code originating from within Parse, although no specific details are available to troubleshoot this error code.

Steps to reproduce

The code below retrieves a CSV file, parses it, then creates objects for inserting. However, another Parse user has reproduced this issue 8 months ago and requested assistance with it via StackOverflow. Their code is much simpler, yet it still produces the error (accessible here: http://stackoverflow.com/questions/33932963/parse-com-background-job-fail-the-service-is-currently-unavailable )

My code is as follows...

1. Option Why?? #1: Save all records in one saveAll.
2. Option Add JS Syntax Highlighting to README.md code block #2: Save objects in small batches.

Expected Results

Objects should be saved to the DB without issue.

Actual Outcome

Process occasionally fails with error object {"code":2,"message":"the service is currently unavailable"}

Environment Setup

• Server
  • parse-server version: Hosted on Parse.com (running in production mode w/ 600req/sec.)
  • Operating System: Hosted on Parse.com (DB migrated to mLab)
  • Hardware: Hosted on Parse.com (DB migrated to mLab)
  • Localhost or remote server?: Hosted on Parse.com (DB migrated to mLab)
• Database
  • MongoDB version: 3.2.8
  • Storage engine: Unknown.
  • Hardware: Reproduced on Shared Cluster and Dedicated Cluster.
  • Localhost or remote server? mLab

Logs/Trace

You can turn on additional logging by configuring VERBOSE=1 in your environment.

Is this available on the Parse hosted environment? If so, where?