Dropbox API Support & Feedback
Find help with the Dropbox API from other developers.
Hi,
We've designed an app and found that if we create more than 5000 team folders under team root (eg. 1st level folder). The cursor returned by list folder api (eg. https://api.dropboxapi.com/2/files/list_folder) will be around 90000 bytes long.
If we use this cursor to send list folder continue api (eg. https://api.dropboxapi.com/2/files/list_folder/continue), then it will always be responded with 504 gateway timeout.
We've found that the cursor may grow with the number of team folders (eg. see https://www.dropboxforum.com/t5/Dropbox-API-Support-Feedback/Cursor-max-size/td-p/253505). But we didn't expect that there should always be timeout if there are more than 5000 team folders. (eg. We know that there a soft limit related to this, see https://help.dropbox.com/zh-tw/teams-admins/admin/member-space-limits. But more than 20 seconds still takes too long). Is there any way to list folders recursively faster without timeout?
Thank you very much.
Best regards,
Ethan Fang
Thanks for the report. It looks this error may be related to the large number of files and/or a large amount of file activity in the affected account. That can vary over time due to account activity and other such factors though. We'll look into it.
You may be able to workaround this issue by setting a smaller limit
and/or setting recursive:false
on /2/files/list_folder (though that may require updating your implementation accordingly), if you aren't doing so already.
Hi,
We've attempted to set the limit to a low number (eg. 20), it still leads to timeout after 1m30s.
However, we really need to set recursive as true. Or there will be lots of api calls. (eg. 5000 folders, so at least 5000 calls)
Thank you very much~!
Best regards,
Ethan Fang
Hi @ethanfang,
The problem in your case is that you have lots of namespaces. Every single namespace gets locked by self. Since you need to lock all namespaces together, looks like the task takes too long. You need to split list processing for different namespaces to avoid such a deadlock. On other side, as you mentioned too many calls take too long to establish connections (Dropbox SDKs doesn't support multi-call on single connection). As a workaround you can perform such a request serie on you own on a single connection and so speed up the processing. I just created a simple example in Python:
#!/usr/bin/python3
from http.client import HTTPSConnection, _CS_IDLE
from urllib.parse import urlparse, urlunparse
from json import JSONDecoder, JSONEncoder
token = '<your access token here>'
paths = ['', '/path1', 'id:..', ...] # id's or paths to all your namespaces
def requestListResponse(conn, path):
body = JSONEncoder().encode({"path": path,
"include_mounted_folders": False,
"recursive": True}).encode()
headers = {'Host': conn.host,
'Content-Length': len(body),
'Connection': 'Keep-Alive',
'Authorization': 'Bearer ' + token,
'Content-Type': 'application/json'}
conn._HTTPConnection__state = _CS_IDLE
conn.request("POST", "/2/files/list_folder", body, headers)
return conn.response_class(conn.sock, method=conn._method)
def requestListContResponse(conn, cursor):
body = '{"cursor":"' f"{cursor}" '"}'
headers = {'Host': conn.host,
'Content-Length': len(body),
'Connection': 'Keep-Alive',
'Authorization': 'Bearer ' + token,
'Content-Type': 'application/json'}
conn._HTTPConnection__state = _CS_IDLE
conn.request("POST", "/2/files/list_folder/continue", body, headers)
return conn.response_class(conn.sock, method=conn._method)
def handleResponse(resp):
resp.begin()
if 200 != resp.status:
raise RuntimeError(f"Status: {resp.status}\nHeader info:\n{resp.info()}\n"
f"\n{resp.read().decode()}")
return JSONDecoder().decode(resp.read().decode())
def listFolders(paths):
conn = HTTPSConnection("api.dropboxapi.com")
responses = []
for path in paths:
responses.append(requestListResponse(conn, path))
while len(responses) > 4:
resp = responses.pop(0)
result = handleResponse(resp)
for entry in result['entries']:
yield entry
if result['has_more']:
responses.append(requestListContResponse(conn, result['cursor']))
while len(responses):
resp = responses.pop(0)
result = handleResponse(resp)
for entry in result['entries']:
yield entry
if result['has_more']:
responses.append(requestListContResponse(conn, result['cursor']))
for e in listFolders(paths):
print(f"{e['.tag']}: {e['name']} with path -> {e['path_lower']}")
Hope the above demonstrates the idea clearly.
Hi,Greg
We've tried to create 6000 folders under a team folder under team space root (eg./team_folder1/folder1,......,/team_folder1/folder6000). In this case, we can use list_folder and list_folder_continue api to list until the end within seconds.
But if all the 6000 folders are created under team space root, then it will takes long and 504 timeout. I guess the problem is not totally with large amount of files, but with 1st level team space folder.
We've attempted adding "limit" parameter, but it still timeout, and we can't set recursive to false, since we want to list all the folders with 1 api. If we do so, then 6000 folders would require 6000 api.
Is there any other way to solve the large number team folders performance issue?
Thank you,
Best regards,
Ethan Fang
@ethanfang Thanks for following up. No, unfortunately I don't have any other workarounds to suggest for this.
Hi there!
If you need more help you can view your support options (expected response time for a ticket is 24 hours), or contact us on X or Facebook.
For more info on available support options for your Dropbox plan, see this article.
If you found the answer to your question in this Community thread, please 'like' the post to say thanks and to let us know it was useful!