getObjects not returning all objects in collection

Third Party Coronium
#1

Hi

Both client side calls…

coronium:getObjects( collection, { where = { objectId = { ["$exists"] = true } }, limit = 99999 } )

…and server side…

coronium.mongo:getObjects( collection, nil, 99999 )

…only return a fraction of the objects in a collection. getObjectCount returns the correct number. I’m getting this on all collections I create, no matter how simple each object is. Here are the results of some tests:

count : returned

10000 : 9899

9000 : 8899

5000 : 4899

1000 : 899

500 : 399

200 : 101

100 : 100

The only theory I have is that we changed the object size with this command…

sudo sed 's/client\_body\_buffer\_size 32k/client\_body\_buffer\_size 64k/' -i /usr/local/openresty/nginx/conf/nginx.conf

…as a result of this discussion.

Any help would be hugely appreciated  :slight_smile:

(AWS Coronium 1.93.1)

#2

Hi,

I remember this coming up before. Let me think about it a bit. Additionally, this is not an uncommon behavior for Mongo, the solution is to use a “cursor” to bring in sets at a time.

#3

Thanks Chris!

#4

Hi Chris,

Just wondering if you’ve had any thoughts on this issue as I’m a little bit crippled at the moment. Is there a way to reset the object size in case that’s the root of the problem? I thought to maybe create a new instance but the AWS image seems to be disabled at present.

Many thanks

Andrzej

#5

Andrrzej,

What is the result for coronium.mongo:getObjectCount( collection, { where = { objectId = { ["$exists"] = true } } ). Just curious if you see the same count.

Also, just for a sanity check you may want to check coronium.mongo:getObjectCount( collection, { where = { objectId = { ["$exists"] = false } } ). You might somehow have a few objects where objectId was not set.

#6

Hi Steve,

Many thanks for your reply. I tried the following code:

local answer1 = coronium.mongo:getObjectCount( collection, { where = { objectId = { ["$exists"] = true } } } ) local answer2 = coronium.mongo:getObjectCount( collection, { where = { objectId = { ["$exists"] = false } } } ) local answer3 = coronium.mongo:getObjectCount( collection ) local out = { hasObjectId = answer1.result, noObjectId = answer2.result, allObjects = answer3.result } coronium.output( coronium.answer( out ) ) 

But only allObjects returns the object count, the other two queries return 0. I can see that at least some of if not all the objects do have objectIds from the cloud admin panel.

#7

Hi,

I remember this coming up before. Let me think about it a bit. Additionally, this is not an uncommon behavior for Mongo, the solution is to use a “cursor” to bring in sets at a time.

#8

Thanks Chris!

#9

Hi Chris,

Just wondering if you’ve had any thoughts on this issue as I’m a little bit crippled at the moment. Is there a way to reset the object size in case that’s the root of the problem? I thought to maybe create a new instance but the AWS image seems to be disabled at present.

Many thanks

Andrzej

#10

Andrrzej,

What is the result for coronium.mongo:getObjectCount( collection, { where = { objectId = { ["$exists"] = true } } ). Just curious if you see the same count.

Also, just for a sanity check you may want to check coronium.mongo:getObjectCount( collection, { where = { objectId = { ["$exists"] = false } } ). You might somehow have a few objects where objectId was not set.

#11

Hi Steve,

Many thanks for your reply. I tried the following code:

local answer1 = coronium.mongo:getObjectCount( collection, { where = { objectId = { ["$exists"] = true } } } ) local answer2 = coronium.mongo:getObjectCount( collection, { where = { objectId = { ["$exists"] = false } } } ) local answer3 = coronium.mongo:getObjectCount( collection ) local out = { hasObjectId = answer1.result, noObjectId = answer2.result, allObjects = answer3.result } coronium.output( coronium.answer( out ) ) 

But only allObjects returns the object count, the other two queries return 0. I can see that at least some of if not all the objects do have objectIds from the cloud admin panel.