cancel
Showing results for 
Search instead for 
Did you mean: 
Announcements
Update: Find information on Dropbox support during COVID-19 here
Close
cancel
Showing results for 
Search instead for 
Did you mean: 
Highlighted

Http 504 Gateway Timeout when trying to get upto 1000 or more files in c#

New member | Level 2
New member | Level 2

I am trying to fetch a folder using C# dropbox api and I am paginating results as well but still I get timeout on it as it contains more than 1000 files.

 

Here is my pagination code:

 

var children = await GetDropboxClient().Files.ListFolderAsync(dbxItem.PathLower == DROPBOX_ROOT_PATH ? string.Empty : dbxItem.PathLower);
						
while (children.HasMore)
{
      children = await GetDropboxClient().Files.ListFolderContinueAsync(children.Cursor);				
}
return children.Entries;

 

9 Replies 9
Highlighted

Re: Http 504 Gateway Timeout when trying to get upto 1000 or more files in c#

Super User II
Super User II
*moves to API area*

 


- - -


Heart Did this post help you? If so please mark it for some Kudos below. 


 Did this post fix your issue/answer your question? If so please press the 'Accept as Solution' button to help others find it.


 Did this post not resolve your issue? If so please give us some more information so we can try and help - please remember we cannot see over your shoulder so be as descriptive as possible! 


 

Highlighted

Re: Http 504 Gateway Timeout when trying to get upto 1000 or more files in c#

Dropboxer
Dropboxer
Thanks for the report. To help us look into this, please share:

- the full error output
- the relevant cursor(s)
- the affected user IDs

Thanks in advance!
Highlighted

Re: Http 504 Gateway Timeout when trying to get upto 1000 or more files in c#

New member | Level 2
New member | Level 2

I want to attach a fiddler trace but I don't see an option to upload other than image files. How can I do that?

Highlighted

Re: Http 504 Gateway Timeout when trying to get upto 1000 or more files in c#

New member | Level 2
New member | Level 2

But here the call: 

 

POST https://connectoreastsftest.sharefileconnect.com/dropbox/v3/Items(e797f2aKVXZaWW5kkUdydma2IxQEFpXbp-UXmRunpCbrWhibpOiaGBoa6iYamKTpphgbW6YmW1iYGiWaKmSUlBQUW-nrJ-fn5aUml-QXFRvrFafpJhZklqQWl-gl5-fqAwA_)/Children?%24select=Owner%2FId%2CCreator%2FId%2CId%2CCreationDate%2CCreatorNameShort%2CDescription%2CFileName%2CName%2CFileCount%2CFileSizeBytes%2CHasMultipleVersions%2CPreviewStatus%2CState%2CVirusStatus%2CUri%2Curl%2CESignatureDocument%2CExpirationDate%2CMaxPreviewSize%2CStreamID%2CFileLockInfo%2CInfo%2CInfo%2FCanDeleteCurrentItem%2CProperties%2CProperties%2F%2A%2CInfo%2FCanDownload%2CInfo%2FCanSend%2CInfo%2FCanManagePermissions%2CInfo%2FCanAddNode%2CDlpInfo%2CDlpInfo%2FStatus%2CHasPendingAsyncOp%2CFavorite%2CFavorite%2F%2A%2CBundle%2CBundle%2F%2A&%24expand=ESignature%2CFileLockInfo%2CDlpInfo%2CInfo%2CFavorite%2CFavorite%2FItem%2CBundle%2COwner%2CCreator&includeDeleted=false&orderingMode=UseFolderOptions HTTP/1.1
Host: connectoreastsftest.sharefileconnect.com
Connection: keep-alive
Content-Length: 2
Accept: application/json, text/plain, */*
Origin: https://connectors3.sharefiletest.com
X-SF-App: ShareFileWeb
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36
X-HTTP-Method-Override: GET
Content-Type: application/json
Referer: https://connectors3.sharefiletest.com/app/
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.8,hi;q=0.6
Cookie: DbxConnector_AuthId=477b3691becf491d9139fdeae81f4104
{}

And response: 

HTTP/1.1 504 GATEWAY_TIMEOUT
Content-Length: 0
Connection: keep-alive
 

 

Highlighted

Re: Http 504 Gateway Timeout when trying to get upto 1000 or more files in c#

New member | Level 2
New member | Level 2

But here is the request that fails: 

 

POST https://connectoreastsftest.sharefileconnect.com/dropbox/v3/Items(e797f2aKVXZaWW5kkUdydma2IxQEFpXbp-UXmRunpCbrWhibpOiaGBoa6iYamKTpphgbW6YmW1iYGiWaKmSUlBQUW-nrJ-fn5aUml-QXFRvrFafpJhZklqQWl-gl5-fqAwA_)/Children?%24select=Owner%2FId%2CCreator%2FId%2CId%2CCreationDate%2CCreatorNameShort%2CDescription%2CFileName%2CName%2CFileCount%2CFileSizeBytes%2CHasMultipleVersions%2CPreviewStatus%2CState%2CVirusStatus%2CUri%2Curl%2CESignatureDocument%2CExpirationDate%2CMaxPreviewSize%2CStreamID%2CFileLockInfo%2CInfo%2CInfo%2FCanDeleteCurrentItem%2CProperties%2CProperties%2F%2A%2CInfo%2FCanDownload%2CInfo%2FCanSend%2CInfo%2FCanManagePermissions%2CInfo%2FCanAddNode%2CDlpInfo%2CDlpInfo%2FStatus%2CHasPendingAsyncOp%2CFavorite%2CFavorite%2F%2A%2CBundle%2CBundle%2F%2A&%24expand=ESignature%2CFileLockInfo%2CDlpInfo%2CInfo%2CFavorite%2CFavorite%2FItem%2CBundle%2COwner%2CCreator&includeDeleted=false&orderingMode=UseFolderOptions HTTP/1.1
Host: connectoreastsftest.sharefileconnect.com
Connection: keep-alive
Content-Length: 2
Accept: application/json, text/plain, */*
Origin: https://connectors3.sharefiletest.com
X-SF-App: ShareFileWeb
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36
X-HTTP-Method-Override: GET
Content-Type: application/json
Referer: https://connectors3.sharefiletest.com/app/
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.8,hi;q=0.6
Cookie: DbxConnector_AuthId=477b3691becf491d9139fdeae81f4104
{}
Highlighted

Re: Http 504 Gateway Timeout when trying to get upto 1000 or more files in c#

Dropboxer
Dropboxer
That seems to be the POST to your own site. Can you share the information I requested for the Dropbox API call itself? Thanks in advance!
Highlighted

Re: Http 504 Gateway Timeout when trying to get upto 1000 or more files in c#

New member | Level 2
New member | Level 2

@Greg :I think I found the issue. Our servers hosted on AWS EC2 were timing out as the default idle time was 300s. What should be the idle connection time set to avoid timing out for files upto 1000?


@Greg K. wrote:
That seems to be the POST to your own site. Can you share the information I requested for the Dropbox API call itself? Thanks in advance!

 

Highlighted

Re: Http 504 Gateway Timeout when trying to get upto 1000 or more files in c#

New member | Level 2
New member | Level 2

Also, if I had paginated results using the code mentioned initially in my post, then shouldn't it get the results in batches instead of getting all at once?  I think that's what is happening and its timing out eventually.

Highlighted

Re: Http 504 Gateway Timeout when trying to get upto 1000 or more files in c#

Dropboxer
Dropboxer
How long each call takes, as well as how long it takes to list everything, will vary from account to account, e.g., based on the number of files/activity, so I can't offer a guarantee on how much time it would take.

Each call to a ListFolderAsync or ListFolderContinueAsync method makes one call to the Dropbox API, with each call returning one page of results. If you have that entire code snippet, including the while loop, blocking the HTTP response for your own page, that can cause your own HTTP call to time out, since the number of Dropbox API calls/pages is unbounded. (I.e., per the above, if the user has many files, there will be many calls to make). I don't have full context on your app of course, but it sounds like perhaps you want to only return one page at a time per HTTP call to your app itself, or perhaps move this into an asynchronous code path, retrieved via AJAX call or some other mechanism.

Work Smarter with Dropbox

The way we work is changing. Share and discover new ways to work smarter with Dropbox in our community.

Sound good? Let's get started.
Who's talking

Top contributors to this post

What do Dropbox user levels mean?
Need more support?