Downloading all the files in a folder using CoreSDK

There isn't any particular limit on our side around 172 files. The -1001 error code corresponds to NSURLErrorTimedOut though, which can happen for a variety of reasons, e.g., if the network connection fails, or if your connection is too congested. How many simultaneous connections are you allowing? I've seen other developers choosing a value of 4 for that.

I'm doing it 1 at a time, in essence its:

for var m in metadata.contents

{

var req = client?.createLoadFileRequest(m.path, atRev:m.rev, intoPath:local.stringByAppendingPathComponent(m.filename))

addToQueue(req)

}

Where createLoadFileRequest can be seen here:

https://github.com/skela/dbcore/blob/master/DropboxCore/DBCRestClient.m#L180

I can't for the life of me figure out why its always 172 files, without fail, even with or without a queue.

Just wanted to make sure that this isn't an issue on your end before I spend too much time trying to solve this.

What's considered best practice if I have to download 10k very small files in a folder using CoreSDK?

There certainly shouldn't be anything on our side causing this. If the API was going to intentionally fail a request, it would respond with an actual HTTP error code, but -1001 is just a client side error code.

The best practice is to only download files if/when you need them, but if you do need to download many files, the best practice is just to loop through and download them individually, limiting the number of simultaneous connections.

Is it always the same file that this occurs on? Are you able to see how long the request is in fact taking before that error occurs?

Thing is, this is a replacement for the Dropbox Datastores (thanks again /s), which were always synced to the device, so yes, I do need all the files in that folder ;)

The gist of what I'm doing and plan on doing is:

1) App starts up

2) App loads the metadata for 1 particular folder where these 10k json files are stored

3) LoadMetadata finishes, with 1 metadata object (called metadata) that has a member called contents, which has 10k items in it.

4) for var m in metadata.contents { addToLoadFileQueueIfNecessary(m) } 

5) 1 by 1 go through the queue, loading each file.

6) poll to check if there are any changes and repeat.

Its a bit odd that it always happens at file number 172, so perhaps there's something wrong with that file. Weird if there is because loadMetadata is called right before really.

Part of me was wondering whether because I'm creating all of the DBRequests but not starting them. That perhaps once the queue has reached further down the list, they timeout immediately when I try to start them, because they were created X seconds ago. But its still too odd that its always 172 files.

I'll take a closer look tomorrow though, thanks for your help Gregory.

After playing around with the timeouts, and moving some code around , it seems more robust. So thanks Gregory.

Obviously it will still timeout from time to time, whats your preferred strategy to retry failed downloads? I assume you had to build some kind of logic like that into the Sync SDK. 

I believe that the Sync SDK employed capped exponential backoff. Pseudocode:

failCount = 0
while true
  if attemptDownload().success
    break
  sleep(min(maxSleep, minSleep * (2**failCount)))
  failCount += 1

Thanks!