cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Announcements
We just wanted to say thank you! Check out our customer appreciation video here.

Dropbox API Support & Feedback

Find help with the Dropbox API from other developers.

cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 

Re: Threads, tasks, and the Objective-C SDK

Threads, tasks, and the Objective-C SDK

Robert S.138
Helpful | Level 7

Having completed an Android version of my app, I was hoping that the Objective-C version for iOS would be similar.  But looking at the DBRoulette example, there seems to be a big difference regarding threading.  Is it correct that all the code shown in the four .m source files runs in the UI thread?  This may be fine for transferring one file at a time, when the only thing the UI thread needs to do is respond when the background operation is done.  But in my app, I need to transfer an entire folder, and recursively, any subfolders.  In Android I did this in an Async Task, where my doInBackground method did all the heavy lifting, handling folder recursion, etc.  The UI thread got periodic postings of progress through the onProgressUpdate method. And the UI thread could cause an early termination of the Async Task the usual way.

In the iOS version, I would like to use the same pattern, doing all the work in a worker thread.  But I don't see how to implement that pattern using the functions shown in the DBRoulette example.  Is there some other example that uses a worker thread to sequence through a tree of files and transfer them one at a time?

 

9 Replies 9

Stephen C.14
Dropbox Staff

Thanks for reaching out. Are you looking to specify a custom queue on a route-by-route basis?

If you're looking to simply set a custom queue for all API calls, you can do that by supplying a custom `DBTransportClient` with a custom `delegateQueue` when you setup your app. You can read about it more on the GitHub tutorial page here. Do this will mean that all response handler code is executed on a thread other than the main thread.

If you want to specify a queue for reach route, however, the SDK doesn't currently offer that functionality. But we're thinking about implementing it. If you think being able to specify the queue/thread for each API call (in the response handler) would help your workflow a lot, we can try and get the new feature out quickly to you.

Robert S.138
Helpful | Level 7

What I need to do is copy an entire directory tree, one file at a time.  The easiest way to do that is to write a method that handles one folder, and then make that method call itself recursively when a subfolder is found.  But such code is not suitable for the UI thread, which must be event-driven and non-blocking.  If I were to do it using only UI code, I would have to implement a state machine, and recursion would be much harder.  But if I could call a blocking (but interruptable) API function for each file access, I could do the whole thing in one procedural thread.

Robert S.138
Helpful | Level 7

I have been studying these API calls, and even with a custom queue, I don't see how to do what I want.  First of all, here is the pseudo-code for what I have implemented in Android (omitting all error-handling), and what I want to do in iOS.  It downloads all the files in Dropbox and runs in a single background thread - not the UI thread.  I have also omitted all interactions with the main UI thread:

Code for thread:
  processWithRecursion(");        //..start with the root of Dropbox
end of execution of thread.

And here is the recursively-called function, processWithRecursion():

  processWithRecursion( currentDirectory )
  {
    form list of currentDirectory
    for each element in that list, do:
    {
      if(element is a directory)
      {
        create local destination directory of that name
        processWithRecursion(currentDirectory + subdirectory name);
      }
      else  //..element is a file
      {
        create destination file and download from Dropbox to that file
      }
    }
  }

As you can see, the recursive call nicely steps through all the files, including any nested subdirectories.  The call stack performs the function of keeping track of where we are in the search for files.  This could be performed by an elaborate state machine, but then I would have to create a separate push-down stack data structure to traverse the directory tree.

This process is best implemented with API functions that block.  But it seems the API functions available to me do not block, so there is no easy way to execute a series of these API calls by simply writing them one after another in code.  For example, I would need to call listFolder:searchPath to get the list of files in one directory, and then for each of the items returned in the list I would need to call downloadData:filePath to actually do the download.  But both of these API calls are implemented as non-blocking calls that return immediately, leaving the completion block to handle the completion of the action.  So I could not write listFolder:searchPath followed immediately by downloadData:filePath, because the downloadData: method would start executing before the searchPath: method was completed.  In fact, I couldn't even form a filePath parameter for downloadData: without examining the completion of searchPath:

That was just for downloading one file.  When you throw in the need for traversing the entire directory tree with recursion, all without any UI input, I don't see a neat way of doing this.

Robert S.138
Helpful | Level 7

OK, I figured out how to make the API calls blocking.  Here is my worker thread function, processWithRecursion (in pseudo-code):

processWithRecursion( currentDirectory )
{
  set waiting = YES;  //..a volatile, atomic variable
  call listFolder: currentDirectory, with completion block:
  {
    get list of entries;
    set waiting = NO;
  }
  do forever
  {
    sleep 80 milliseconds
    if( not waiting )  break;
    if( threadMustStop ) return;  //..set by "Cancel" button
  }
  for each element in that list, do:
  {
    if(element is a directory)
    {
      create local destination directory of that name
      processWithRecursion(currentDirectory + subdirectory name);
    }
    else  //..element is a file
    {
      set waiting = YES;
      call downloadData: elementPathName, with completion block
      {
        get the data downloaded and write to local file
        set waiting = NO;
      }
      do forever
      {
        sleep 80 milliseconds
        if( not waiting )  break;
        if( threadMustStop ) return;
      }
    }
  }
}

Is this an advisable way to the SDK to download an entire directory?

Stephen C.14
Dropbox Staff

@Robert S: I wouldn't recommend using `sleep` like that. I'd instead maintain references to each task object and then `cancel` each task object when the user cancels. I took a bit of time and wrote up what a solution might look like. It took me longer than I thought, which might be an indication that the SDK needs richer functionality.

Try this out and let me know how it works. I think it might be easier pulling the file directly: DownloadFolder.m

 

@interface MyClass ()

@property (atomic) DropboxClient * _Nullable client;
@property (atomic) NSOperationQueue * _Nullable responseQueue;
@property (atomic) NSFileManager * _Nullable fileManager;

@property (atomic) BOOL shouldContinue;
@property (atomic) NSLock *shouldContinueLock;

@property (atomic) NSMutableDictionary<NSString *, DBRpcTask *> *rpcTasks;
@property (atomic) NSMutableDictionary<NSString *, DBDownloadUrlTask *> *downloadUrlTasks;
@property (atomic) NSLock *tasksLock;

@end

 

@implementation MyClass

- (void)initAndLaunch {
  NSLog(@"\n\n\nInitializing on: %@", [NSThread currentThread]);

  _client = [DropboxClientsManager authorizedClient];
  _responseQueue = [NSOperationQueue new];
  [_responseQueue setMaxConcurrentOperationCount:1];
  _fileManager = [NSFileManager defaultManager];

  _shouldContinue = YES;
  _shouldContinueLock = [NSLock new];

  _rpcTasks = [NSMutableDictionary new];
  _downloadUrlTasks = [NSMutableDictionary new];
  _tasksLock = [NSLock new];

  NSURL *outputUrl = [_fileManager URLsForDirectory:NSDocumentDirectory inDomains:NSUserDomainMask][0];
  outputUrl = [outputUrl URLByAppendingPathComponent:[NSString stringWithFormat:@"%@.%@", @"MyOutputFolder", [NSUUID UUID].UUIDString]];

  [self downloadFolder:@" outputUrl:outputUrl];

  // cancel the download after 5 seconds
  double delayInSeconds = 5.0;
  dispatch_time_t popTime = dispatch_time(DISPATCH_TIME_NOW, (int64_t)(delayInSeconds * NSEC_PER_SEC));
  dispatch_after(popTime, dispatch_get_main_queue(), ^(void){
    [self cancelDownload];
  });
}

- (void)downloadFolder:(NSString *)inputPath outputUrl:(NSURL *)outputUrl {
  NSLog(@"Downloading all files from: \"%@\" to \"%@\", inputPath, outputUrl);
  [_responseQueue addOperationWithBlock:^{
    [self downloadFolderHelper:inputPath outputUrl:outputUrl];
  }];
}

- (void)downloadFolderHelper:(NSString *)inputPath outputUrl:(NSURL *)outputUrl {
  NSLog(@"Commencing folder download on: %@", [NSThread currentThread]);

  NSString *listFolderTaskKey = [NSString stringWithFormat:@"ListFolderTask.%@", [NSUUID UUID].UUIDString];

  [self.shouldContinueLock lock];

  if (!_shouldContinue) {
    NSLog(@"No longer continuing...");
    [self.shouldContinueLock unlock];
    return;
  }

  DBRpcTask *listFolderTask = [self listFolder:inputPath outputUrl:outputUrl listFolderTaskKey:listFolderTaskKey];

  [self.rpcTasks setObject:listFolderTask forKey:listFolderTaskKey];
  [self.shouldContinueLock unlock];
}

-(DBRpcTask *)listFolder:(NSString *)inputPath outputUrl:(NSURL *)outputUrl listFolderTaskKey:(NSString *)listFolderTaskKey {
  DBRpcTask *listFolderTask = [[_client.filesRoutes listFolder:inputPath]
   response:_responseQueue response:^(DBFILESListFolderResult *listFolderResult, DBFILESListFolderError *routeError, DBError *error) {
     NSLog(@"Handling list folder response on: %@", [NSThread currentThread]);
     if (listFolderResult) {
       NSArray<DBFILESMetadata *> *entries = listFolderResult.entries;

       for (DBFILESMetadata *entry in entries) {
         if ([entry isKindOfClass:[DBFILESFileMetadata class]]) {
           DBFILESFileMetadata *fileMetadata = (DBFILESFileMetadata *)entry;
           NSString *filePath = fileMetadata.pathDisplay;
           NSURL *fileOutputUrl = [outputUrl URLByAppendingPathComponent:fileMetadata.name];
           NSString *fileOutputPath = [fileOutputUrl path];
           NSLog(@"Downloading \"%@\" to \"%@\", filePath, fileOutputPath);
           NSString *downloadTaskKey = [NSString stringWithFormat:@"DownloadTask.%@", [NSUUID UUID].UUIDString];

           [self.shouldContinueLock lock];

           if (!_shouldContinue) {
             NSLog(@"No longer continuing...");
             [self.shouldContinueLock unlock];
             return;
           }

           DBDownloadUrlTask *downloadTask = [self downloadFile:filePath fileOutputUrl:fileOutputUrl downloadTaskKey:downloadTaskKey];
           [self.downloadUrlTasks setObject:downloadTask forKey:downloadTaskKey];
           [self.shouldContinueLock unlock];
         } 
         else if ([entry isKindOfClass:[DBFILESFolderMetadata class]]) {
           DBFILESFolderMetadata *folderMetadata = (DBFILESFolderMetadata *)entry;
           NSURL *directoryOutputUrl = [outputUrl URLByAppendingPathComponent:folderMetadata.pathDisplay];
           NSString *directoryOutputPath = [directoryOutputUrl path];

           if (![_fileManager fileExistsAtPath:directoryOutputPath]) {
             NSError *error;
             [_fileManager createDirectoryAtPath:directoryOutputPath withIntermediateDirectories:@(YES) attributes:nil error:&error];
             if (error) {
               NSLog(@"Error creating directory: %@", error);
             }
           }

           [self downloadFolderHelper:[NSString stringWithFormat:@"%@/%@", inputPath, folderMetadata.name] outputUrl:directoryOutputUrl];
         } else if ([entry isKindOfClass:[DBFILESDeletedMetadata class]]) {
           continue;
         }
       }
       [self.tasksLock lock];
       [_downloadUrlTasks removeObjectForKey:listFolderTaskKey];
       [self.tasksLock unlock];
     } else {
       NSLog(@"Error listing folder \"%@\":", inputPath);
       if (routeError) {
         NSLog(@"RouteError: %@", routeError);
       }
       NSLog(@"Error: %@", error);
     }
   }]; 

  return listFolderTask;
}

-(DBDownloadUrlTask *)downloadFile:(NSString *)filePath fileOutputUrl:(NSURL *)fileOutputUrl downloadTaskKey:(NSString *)downloadTaskKey {
  DBDownloadUrlTask *downloadTask = [[_client.filesRoutes downloadUrl:filePath overwrite:@(YES) destination:fileOutputUrl]
   response:_responseQueue response:^(DBFILESFileMetadata *downloadResult, DBFILESDownloadError *routeError, DBError *error, NSURL *destination) {
     NSLog(@"Handling download response on: %@", [NSThread currentThread]);
     if (downloadResult) {
       NSLog(@"Successfully downloaded \"%@\" to \"%@\", downloadResult.pathDisplay, [destination path]);
       [self.tasksLock lock];
       [self.downloadUrlTasks removeObjectForKey:downloadTaskKey];
       [self.tasksLock unlock];
     } else {
       NSLog(@"Error downloading \"%@\" to \"%@\", filePath, [fileOutputUrl path]);
       if (routeError) {
         NSLog(@"RouteError: %@", routeError);
       }
       NSLog(@"Error: %@", error);
     }
   }];

  return downloadTask;
}

-(void)cancelDownload {
  [self.shouldContinueLock lock];
  self.shouldContinue = NO;
  [self.shouldContinueLock unlock];

  [self.tasksLock lock];
  [self cancelAndClearTasks];
  [self.tasksLock unlock];
}

 

- (void)cancelAndClearTasks {
  for (NSString *key in _rpcTasks) {
    DBRpcTask *task = _rpcTasks[key];
    [task cancel];
    NSLog(@"Canceled task: %@", @(task.task.taskIdentifier));
  }

  for (NSString *key in _downloadUrlTasks) {
    DBDownloadUrlTask *task = _downloadUrlTasks[key];
    [task cancel];
    NSLog(@"Canceled task: %@", @(task.task.taskIdentifier));
  }

  [_rpcTasks removeAllObjects];
  [_downloadUrlTasks removeAllObjects];
}

Robert S.138
Helpful | Level 7

Thank you, Stephen, for all that work.  I will study this and incorporate as much of this into my design as I can understand.  There are a few simplifications I will make, though.  In my application the download only takes place when a certain view controller is active and the user is involved, if only as an observer.  Instead of a 5-second time-out on operations, there will be a "Cancel" button in the UI that the user can use to terminate anything that seems to be taking too long.  Also, I found that I don't really need a separate NSResponseQueue.  The main UI queue serves me just fine, and it makes it possible for me to access UI elements directly from within a response block.

I am a little curious why you found it necessary to put locks around accesses to boolean flags that are only set by one thread.  I think you need to have both threads modifying the variable to need a lock.

Stephen C.14
Dropbox Staff

No worries! Hopefully it will be of some help. I would just make sure that you weigh whether it's important to track the response `DBTask` objects that you receive back, so that if you're downloading a lot of files from a folder, you have a way of canceling them all at once (if you feel like that's important).

As far as the lock around the boolean, I did that to make sure that the following were one atomic transaction: 1. check if should continue 2. create a download request 3. store the download task object for later use (to cancel). Otherwise, if the cancel function was called, a request might be made but not stored in the task data structure, and therefore would not be cancelled when iterated through. Does that make sense? Probably could have used an `NSOperationQueue` rather than a lock.

Let me know if you run into any issues down the road. And please, if you have any additional feedback for the SDK, do let me know!

Robert S.138
Helpful | Level 7

Stephen,

Having worked a little more with the SDK and gotten more familiar with it, I don't think it needs to be changed.  Having API that block carries its own problems of interruptibility, etc.  It is only a couple of lines of code to add a custom wait for completion.

As far as the locked flags, I see why your implementation needs them. But mine is simpler.  I only have one single task that performs all the downloads sequentially.  I realize this probably means lower performance, but I think it is worth it for the simplicity.  Each download can potentially create an error requiring user response to "Skip or Abort".  The most logical thing for the user to expect if he Aborts is that none of the files after that file have been transferred.  But if there are simultaneous downloads, there may be several successful downloads that occur after the one that had an error, and by the time the user decides to Abort, those downloads have already completed.  So with only one task that is manually started, there is no question about which tasks need to be cancelled.

Stephen C.14
Dropbox Staff

@Robert S.: That makes sense! There certainly is a balance to be struck between simplicity of the code and performance. Good luck with your implementation, and please reach out if you run into anything else or have any more general feedback about the SDK.

Need more support?