delete_batch returns internal error consistently when deleting more than 3425 files

[skysync]

I was using the delete API call to remove a folder that contained more than 10k files when I saw the too_many_files error.  I switched over to the delete_batch API call in order to bypass this error, but received the same error at 10,001 files.  I decided to send smaller chunks, but when trying to use delete_batch with more than 3425 files, I consistently get an internal error reported.  When I reduced the number of files to be sent to the delete_batch API call to 3422, I was successful in deleting 3422 files the first go around, then received the internal error for any subsequent calls to delete_batch.  I'm having a hard time writing reliable code to this endpoint without fully understanding it's limits and don't want to resort to deleting files one at a time since this will take much longer.  This code was being used for performance testing and the issue first documented when deleting a folder with 100k files.  

[Greg-DB]

Thanks for the report! We do unfortunately currently return internal_error when a more specific error would be more helpful. I'm sending this along to the team, but I can't make any promises as to if/when this would be improved.

If you share the job ID though, I may be able to look into it and share some information as to the specific issue. (Or, feel free to open an API ticket here if you'd prefer to share privately: https://www.dropbox.com/developers/contact )

[skysync]

Hi, here's a job ID to look into: dbjid:AABmns5UvrGc8_P9u_aHxWgCafbeZL1ejUfh1LaMShLO2s-gLpFA8wKeNJ_CtWUufwrTWwQ6d1m4ylYsIOvR5Mq6

[Greg-DB]

Thanks! I'm looking into it. When you called /2/files/delete_batch/check for this one in particular though, can you confirm if you got a response body containing a internal_error, or did you actually just get a 500 on the call itself?

[skysync]

This is the result I received:

{
"error_summary": "internal_error/..",
"error": {
".tag": "internal_error"
}
}

However, the jobID below, returned nothing:

dbjid:AABvdNk9LAAlE1U8IZVO2Ak3vawOviEm72PYHhVaDzQKPdV5t-Clc26PwhDLTr3EHGbAFVTqplZprCcMm9KsWs7A

[Greg-DB]

Thanks! These two jobs that returned an empty response actually completed. If you check the file listing, you should see the changes reflected. They're just hitting a different issue where the call to check the job status itself failed (with a 500 error). We're looking into that.

Are you still seeing jobs explicitly returning internal_error now though? If so, please share one of those job IDs so I can check on it for you. Thanks!

If you're not seeing them anymore, they were likely just transient server-side issues. You should have your app retry the jobs (i.e., run them again, don't just check the status again) when it fails with internal_error.

[skysync]

These jobs never removed the files.  I've been running additional testing using the same directory and adding and removing files.  I assumed the job ID was being used to look at logs, etc.  Sorry I didn't leave the directory in tact for troubleshooting.  My code is setup to retry whenever I hit those errors, but the errors are fairly frequent and I rairly received a success when I sent 7000+ files at a time.  I received the most consistent results around 3000 or 4000 files per request.  

[Greg-DB]

Thanks! I'll need to investigate further. Can you supply the latest job ID that explicitly failed with internal_error? Thanks in advance!