If the cursor is too long, list_folder_continue api always encounter 504 timeout

Thanks for the report. It looks this error may be related to the large number of files and/or a large amount of file activity in the affected account. That can vary over time due to account activity and other such factors though. We'll look into it.

You may be able to workaround this issue by setting a smaller limit and/or setting recursive:false on /2/files/list_folder (though that may require updating your implementation accordingly), if you aren't doing so already.

Hi,
We've attempted to set the limit to a low number (eg. 20), it still leads to timeout after 1m30s.
However, we really need to set recursive as true. Or there will be lots of api calls. (eg. 5000 folders, so at least 5000 calls)

Thank you very much~!
Best regards,

Ethan Fang

Hi ethanfang,

The problem in your case is that you have lots of namespaces. Every single namespace gets locked by self. Since you need to lock all namespaces together, looks like the task takes too long. You need to split list processing for different namespaces to avoid such a deadlock. On other side, as you mentioned too many calls take too long to establish connections (Dropbox SDKs doesn't support multi-call on single connection). As a workaround you can perform such a request serie on you own on a single connection and so speed up the processing. I just created a simple example in Python:

#!/usr/bin/python3

from http.client import HTTPSConnection, _CS_IDLE
from urllib.parse import urlparse, urlunparse
from json import JSONDecoder, JSONEncoder

token = '<your access token here>'
paths = ['', '/path1', 'id:..', ...] # id's or paths to all your namespaces

def requestListResponse(conn, path):
  body = JSONEncoder().encode({"path": path,
                               "include_mounted_folders": False,
                               "recursive": True}).encode()
  headers = {'Host': conn.host,
             'Content-Length': len(body),
             'Connection': 'Keep-Alive',
             'Authorization': 'Bearer ' + token,
             'Content-Type': 'application/json'}
  conn._HTTPConnection__state = _CS_IDLE
  conn.request("POST", "/2/files/list_folder", body, headers)
  return conn.response_class(conn.sock, method=conn._method)

def requestListContResponse(conn, cursor):
  body = '{"cursor":"' f"{cursor}" '"}'
  headers = {'Host': conn.host,
             'Content-Length': len(body),
             'Connection': 'Keep-Alive',
             'Authorization': 'Bearer ' + token,
             'Content-Type': 'application/json'}
  conn._HTTPConnection__state = _CS_IDLE
  conn.request("POST", "/2/files/list_folder/continue", body, headers)
  return conn.response_class(conn.sock, method=conn._method)

def handleResponse(resp):
  resp.begin()
  if 200 != resp.status:
    raise RuntimeError(f"Status: {resp.status}\nHeader info:\n{resp.info()}\n"
                       f"\n{resp.read().decode()}")
  return JSONDecoder().decode(resp.read().decode())

def listFolders(paths):
  conn = HTTPSConnection("api.dropboxapi.com")
  responses = []
  for path in paths:
    responses.append(requestListResponse(conn, path))
    while len(responses) > 4:
      resp = responses.pop(0)
      result = handleResponse(resp)
      for entry in result['entries']:
        yield entry
      if result['has_more']:
        responses.append(requestListContResponse(conn, result['cursor']))
  while len(responses):
    resp = responses.pop(0)
    result = handleResponse(resp)
    for entry in result['entries']:
      yield entry
    if result['has_more']:
      responses.append(requestListContResponse(conn, result['cursor']))

for e in listFolders(paths):
  print(f"{e['.tag']}: {e['name']} with path -> {e['path_lower']}")

Hope the above demonstrates the idea clearly.

Hi，Greg

We've tried to create 6000 folders under a team folder under team space root (eg./team_folder1/folder1,......,/team_folder1/folder6000). In this case, we can use list_folder and list_folder_continue api to list until the end within seconds.

But if all the 6000 folders are created under team space root, then it will takes long and 504 timeout. I guess the problem is not totally with large amount of files, but with 1st level team space folder.

We've attempted adding "limit" parameter, but it still timeout, and we can't set recursive to false, since we want to list all the folders with 1 api. If we do so, then 6000 folders would require 6000 api.

Is there any other way to solve the large number team folders performance issue?

Thank you,

Best regards,

Ethan Fang

ethanfang Thanks for following up. No, unfortunately I don't have any other workarounds to suggest for this.