Repository navigation
[Colossus] storage failing to query data objects #5002
Description
Activity
This seems to be happening on >= v3.9.0
I tested on 3.8.1 and it's not happening- changed the title
[-]storage failing to query data objects[/-][+][Colossus] storage failing to query data objects[/+]on Dec 19, 2023 This error is coming from response validation failing, we use Ajv package - the schema for the response is defined in:
Somehow, the map is having a key set that is a number, not an string. Not sure how that is possible at least at compile time since the map is strongly typed.
I inspected all code paths that update the map and couldn't identify where we might be mistakenly setting a number.
Only thing I can think of is if somehow the spread operator used to generate the array:
const ids = [...idCache.keys()]could be doing something funny on the node that has this issue?Not sure if we wan't to investigate this further if we plan on removing this api endpoint see: #5008 ?
Finally managed to find the fault.
When HTTP GET or HEAD request is made to the node which does not have the object id, we hit these lines of code:For the GET request, since the catch clause was executed because the file was not found, the request has already been handled and passed to the next handler, and the
req.params.idis nowundefined-> this pollutes theidCachewith anundefinedkey, which results in response validation when fetching the data-objects endpoint. eg..response[565943] should be stringthe number565943is the index into the array not the actual object id.For the HEAD request, the behavior is different, during the unpinning the data object is now persisted in the
idCachewith a negative pin count.It is easy to re-produce this by simply starting a test setup and doing GET and HEAD request to the node that does not have a particular object id.
Better checks as suggested in #4991 should make any future issues easier to detect.
How I found the issue: Adding
assert(typeof dataObjectId === 'string')statements in the functions that add and/or update entries.2023-12-27 21:49:41:4941 debug: Query - storageBucketsConnection 2023-12-27 21:49:41:4941 http: HTTP POST /api/v1/files?dataObjectId=204&storageBucketId=1&bagId=static:council 2023-12-27 21:49:41:4941 http: HTTP GET /api/v1/files/204 2023-12-27 21:49:41:4941 error: GET /api/v1/files/204: Error 404: ENOENT: no such file or directory, open '/data/uploads/204' /joystream/storage-node/src/services/caching/localDataObjects.ts:93 assert(typeof dataObjectId === 'string') ^ AssertionError [ERR_ASSERTION]: false == true at unpinDataObjectIdFromCache (/joystream/storage-node/src/services/caching/localDataObjects.ts:93:9) at processTicksAndRejections (node:internal/process/task_queues:95:5) at async getFile (/joystream/storage-node/src/services/webApi/controllers/filesApi.ts:65:5) { generatedMessage: true, code: 'ERR_ASSERTION', actual: false, expected: true, operator: '==' }- added a commit that references this issue
on Dec 28, 2023
Issue: when query data objects intermittently get error response
detail:
Product: colossus
Version: 3.9.0, 3.9.1