Problem
In this node application, there is a page where we show all the user's data. It has some filters, sorting and paginations. All of these data fetching, sorting and filtering are done by node API. Also, there are a large amount of data in the database. The problem is when the page loads API request failed. But in the localhost, it will load without any issue.
Debugging
I have done hours of debugging to recognize the issue. The application has three tabs for show all the users using status. For example, statuses are like "Joined", "Invited", and "Rejected". When I move to a "Rejected" status tab user loads without any issue. But when I was in the other two tabs request failed. But I'm surprised because the tab where all the data loads ("Rejected") have the highest no of records other than the other two tabs.
The reason for loading all the data in the "Rejected" status tab is there is no sorting applied under "Rejected" status data. But other data have a sorting functionality.
After hours of debugging, I'll recognize it as a temporary disk space issue that came up with MongoDB. When I was run the app locally temporary disk space apply as needed. But when it comes to the ubuntu server, when the doc exceeds 16MB limit in aggregation data sorting can't be done. Due to that reason, the request failed.
Solution
Use allowDiskUse() to either allow or prohibit writing temporary files on disk when a pipeline stage exceeds the 100 megabyte limit. This is the solution for this problem. allowDiskUse is unrelated to the 16MB result size limit. That setting controls whether pipeline steps such as $sort or $group can use some temporary disk space if they need more than 100MB of memory. In theory, for an arbitrary pipeline, this could be a very large amount of disk space.
If your result is going to be more than 16MB then you need to use the $out pipeline stage to output the data to a collection or use a pipeline API that returns a cursor to results instead of returning all the data inline (for some drivers this is a separate method, for others, it is a flag passed to the same method {allowDiskUse: true}).
Thank You