Re: Dropbox API not returning all files

Nothing looks clearly wrong in the code. Can you add some extra logging to see:

• how many calls are being made to files_list_folder and files_list_folder_continue, and when? e.g., how many of that repeated folder entry do you see per call?
• what the value of result.cursor is each time to make sure it is changing each time?

Alternatively, you can step through with a debugger to inspect this interactively.

The cursor is definitely changing:

/MarketingCreative/External/Bidalgo_Zynga_CSR2
AAE0hozJExL3S0MSePeqiKCWxFrYSYHlKaVrjL587setM0OKa4pyMp363SxU-HkSHEqgiOcrqKrvzc_LNfxuLMCzqnJBhdZlFdiywbVrO1LtvaZQ8_K-YDSCvZhEGXE4WLubSEivzHFm8BzLTzLJGWbIi2Qd8EBibzeVpJB_1DtqqOPJ0Wz-SUaMHnGsisFikWW5Gbtl6FBjHHFcuhPClt9adQobe0JxtU5doi8Mw-gu-QmiUd_UyNl-_XlItsPjWg11psAhW-F1_HmkWl8-CEOb3EkXzwEwtf-TNiScpgtzrLf2NGxmDyQ6smjrIwqxVbJsEmzsfuTonlOdlz7_PtGusXpcrDlqzXSi4XWqS4x47haSKyC9te0OkhpZu3XKJvIdTaiet96q6aB7q9PigxZYnZ6NuXPgChUQByeiqp0Jn5fn5bBr9zkKEhUM79uKWfO-yXHjthbk-WFMVh9a7cS_pNI1LmE6tYmGvvSsDuMEFWqf0ociMs5TSWrbjxtv0BlGdq9UlUNiWAhr_n1LljsRjKlj-m-W4oc8guyw_GRhYQ
/MarketingCreative/External/Bidalgo_Zynga_CSR2
AAFiWZcpPkiqoVEmJeEbko1CcKgixZTulhCZWqw-H_M6XyBJ6ajzNcERE6nCtgQFbk7ve5Cv8wwRuWSuXyV6eNfaq-SIAudJtZNnPJSKrvU_4efEXJL4j_SJonn8YMMnCwVU7NL73Be2BSqY9sNVub_siDt6hwz-7738zfSDDmkpQ3D01_HmOgfqxx9igwQ9S-PQBnOhh0UKK17dhtpw2Oh0BZOam20ss_ZCyJ5tdSjs05qgTR5yNZPQD_2BZA-bA709MBU6ca1szjRUgAVkFM4GK622uUmtpHxNGE-WMHm8jPN-sKQqqe1fI1j56d5EHERdxnKpmjI6zORoGQC2g6qhni4z9VHq6EiGNLaD3oLsZV2dl5ysQxBkc58MXeNSuqHQ0ddfX3WyXGI974JYq3ntJl4__5PFvPeacX3Iv9Xk3pQ9aso-SW7-OEETnlPNMN_vRU1qYhQmHcC4yJ-K1KK7ivMCpoHfQ0wI7O4f8-JeYV2EtyOdtS7d6vQf0i8520QgdFf1L92xZvCaXIprL65dP7y8khGg40mArGDE3ubhfw
/MarketingCreative/External/Bidalgo_Zynga_CSR2
AAHOafBCzEArLBqdMJgftkMA569bX7IkSVOV2V0If9_yQZgEDMy05naPaonoosRVkPlMNL9xfXw98d6lm2PQ5Kz7vsNX4orStx1o4oVqJfNR6zF41yuB5IC8RGa1_7gnfJUv9zKkDYxAtSNTv8pzgLG5aNl7yUifQt6KtPTAdqcIUXSJHo7lBQMjosHOpUjiXnflf-Lz51HR0EWM9vmaLy6f6VKnU_4u9iiwifWnE2DrZlXvvjDMvKmJ9tx5L_yqCJ-Ybc8ZCuI3WxbnVdwQXJkKT-cKXLXKOCnScDVBZDMU2EEAudT1wDEtTYmY9wAnOCO3keU4cF9pxYnQMpgWkWdjcZ2lGi5aGb4bgaGJ5i2981rb0X1ajdfBVt6-48amwM8kepBKsoQuhp-vw-IFe6BSZhUeBk4kB1lfkGR8yMwFaHNTuIhD0v-rwCdMzs83XdIrLS3csoKg2rSoKGijXO-9hIGb5thQGfKH-jsxc6zLvd4ogOJDIofYzDZDE2142psEQy9NbhITeim46Ts-ZjW5wdDoZ0ceBCYRbQAr1X_8kg
/MarketingCreative/External/Bidalgo_Zynga_CSR2
AAFuVVTjXZdm1hzaN3XqyesuicqXo8kvKkLAdA_rqPsr9XJ6t-AtDw8bFzups3vNFK6iqo3e2cJzZnfQ0B-eUh6edxRY79EDFkf0SKCe1zuTgOfzCOe2yh4nSabe3Qvs2VSh0yuL9sgzp317QU9JwzDTcNlHj5xrb9u-vqket-4CjuQv_xlXdmLcCwAhDHIrY_L8WDsnNQy4S5Mai0NMxgXmjorGjhm-mz7GJndBCQj_pKskBm6-F9VVgV3_YoDvrGxXIFLMvzKzIWqNYVREGw5FCaWGb5Nk1Yn-3BLVh3y4UKIEtdbsXEBeqyeEoaRvuAFIeN-BcE1Fb34ZsfCJUKc7dCPKwvNVKVgvZcWp0CK8zoPYVuMh-1vKReWwUwFSiz1Exy1drEKPj3rf-vC61o6G4D3sZOk0Lo9ZNiiHJkt72zfuGAGmd4f_7ilrasgR8KVytHN0af6HL6IiGH8YHlITwBJJwG2Dq6GsU_cFO9qKz328NhvOGTFj5-CWMu9P4kr5JZdxuAZX45r9I-2j_Lp7LxaiDrbsL7xZ-lk48eNpNg
/MarketingCreative/External/Bidalgo_Zynga_CSR2
AAFDMBZSLW3NspymEotwvGV3AhYgpBju_cN-GLAHdWXs9gXtDZOiivkOap19ELrZ3bZSxWNx0RE-ZxsmVfahTQyHx-z0gs1LnEzarkC2vL0UM1tgUSkoo1ZpFzh3LnZx1Cee90QfxCY2x6fjWNBCxv_5gjv4-2wAibpf9tWfy2JfOVoDYg2DzyvyUEMExiCnFoR2VSPC-POk11NJOuZI2bDCOFqyW9qu2zQv6w9FIkGlZCR9751erkfIErO8Ho8WrWhYf1PiqlnHG7KmM7EY_kkGXWbPyCmZWqeNsLmS9bj4uuO8nKRPOkjyjs_U5-o31FrnssLKaOAnmhnCv25BIfmkFll76TXb3ZDV_H1-049u1SD-4SWTZEQbG2Xy5vonNqlgpcDNJi0phbjMYpoQLcq52KbmcJXsPmffVcST4KB7UTUM2FkfHD2u0qwy68RJ-w6G7nhNs0b8FQ7LLda2Rgy8qcIEWemnquqw1bu3aMfdPqTIXEmEk9ICMG7TARsJ-Oj9StHXVMGXCOzJsvMDfYXjaDLjAG9qO2EZDovNqVEdpw
/MarketingCreative/External/Bidalgo_Zynga_CSR2
AAF1kRHSusb9bNO6BvgAuPxPWISkGl46eWyCv_RJuMiEXGsc6On94JEpC97INNYpxo6e2sNaaQxxqCrbBr8anGbj8-erf5KDDsDXCm5Wfh3aDD7A7vWxrTPKB33oPwtLb5hKUQQTpIOjja8s5D1QRewD0OrrAPjnqIpQD7ik48r9BwMBcRd8PkCg2aRRLxQaS4NHk04l9JsHB10JarYJbyOlltVQ3lWq0BwRMVaRtSCgOW1NQZbpoYnrXMHOxyhcWev4NlvTso1OXBUmrZfwiZxvtO0qTyprDRdhjxYNcc29A_KpAuad-SebhNef3ISWCLB2URC5vBoy3sZZMe4UL9_9IGjxxNjQKO0TCME2WN1J4qPLkknmbzz5sgjL155JBTo8WWXfb2BoCtOLBPbUHVimAQkPd60iNdPjdwKmRCkg_MQrvP2XXXWwK2duOE4NMZRcBIbUv-FeY53GFec2skndu6bP5-NMNOhP2snYWW_EnirExR64xWiVLypZhJv0YDJO3cFA6wd6d8cxzvM2u4rWQXIUcy4pRNSDJsLRlG28jA
/MarketingCreative/External/Bidalgo_Zynga_CSR2
AAG4CVIxqMGl_h1RMvaaCWvnCwrob8GlthSZsXeHHrP9y9Eyp_eHdJvO7XBsOHEh4Fhg-31UaHYCaDYz8pd91eyJZUcIpTc4YP1-wxHRrxewVds_Fyk0NUzeXEAMIApUweRbybG-IDCugHirdEJLuQdLX0fBHihXPDOOSnekCR6pWDmzKiP0lFjHMA6bPUy1A-jaw2zg_ven5_oAXixbdpE7ka5y308LPSD4UM-w6a5eSsHcVHFz1fQlNDtcBG3r1zO2v3hsi-lpS7MhpWe7ti8dT1seujZPrw9axDO50wvEPFHvkvi7nFa09M7qXUal3GWNOcwP1dfKiLey1fo__JVFQTMqyGKriR84pR6wyXkAGQqjhr_qkRnZ3Rf60fvuKH2fwXP5KpP_-7jWy7BSt92QDnZpIcK_PaEJBEvjN798-rNwf_J9wgeMzN2jcavhURKapPE5y9pA4R4TzeCLdnXnaQ6I24gYGX9KSYnLKnDrNnxP9tvsZo04Zir5cqWYEe7TmCg4uB-sXQjy-qWbBCwJKTui0zSDg_tLaJ3jCFDAjQ

How would I go about finding how many calls are being made to files_list_folder and files_list_folder_continue (other than what already exists)?

It seems like the script just resorts to /MarketingCreative/External/Bidalgo_Zynga_CSR2 and stays there.

I've let it run for 10k+ iterations a couple of times and the same situation.

Thanks :)

Thanks! Can you share the updated code with this extra logging too? That would help illustrate what exactly is producing this output.

As for the number of calls being made, you don't need an actual counter; I just meant to illustrate each call to files_list_folder and files_list_folder_continue, which your logging appears to be doing now. That is, there are apparently an unbounded number of calls, with each one returning a list with just one entry. That can happen, but it shouldn't repeat, as long as you pass the returned cursor back into files_list_folder_continue. You appear to be doing so, so we'll need to debug that further. Seeing where your logging occurs may help. 

Sure thing, code is here:

root_path = '/MarketingCreative/External/Bidalgo_Zynga_CSR2/'

result = dbx.files_list_folder(root_path, recursive=True)


def process_entries(entries):
    for entry in entries:
      print(entry.path_display)
      print(result.cursor)
      if isinstance(entry, dropbox.files.FileMetadata):
        file = entry.path_display.rsplit('/', 1)[1]
        if file.startswith('zas'):
          print(file)

process_entries(result.entries)

while result.has_more:
  result = dbx.files_list_folder_continue(result.cursor)
  process_entries(result.entries)

Thing is, I'm trying to get all the files that start with 'zas', and I can do this, if I specify the root folder one level below what I currently have, but that's no good to me as I need all the files within the current root folder.

But again, it just seems to be looping infinitely. :/

Thanks again for your help :)

Thanks, that's helpful. That confirms you're getting one identical entry per page, and a different cursor for each page. 

This doesn't reproduce for me, so we'll need to investigate further on the server-side to find out why this is happening for you. I'll follow up here once I have an update for you.

Hey Greg :) Any luck on this?

This is currently open with engineering, but I don't have an update on it yet. I'll follow up here once I do.

Engineering let me know they're looking into it, but unfortunately we don't have a timeline for a fix right now.

That being the case, you may want to use files_search_v2/files_search_continue_v2 as a workaround for now. 

Hi where are you with this issue, as we backup our files to dropbox business, and we can no longer do an integrity check to verify our backup from synology.  If you cannot resolve this issue, we will have to go else where as we cannot verify that our files are being backed up, and we cannot afford to loose any data if we have to restore from backup.

IanHarker This issue is still open with engineering, but I don't have an update to share yet.