Problem with GetMetadataAsync

[donaldp]

Hi,

I'm trying to test for the existence of a folder so as not to repeat myself. My understanding is that you use GetMetadataAsync and treat the folder as not there if you get an exception. (1) I'm getting exceptions at times that I shouldn't - i.e. there IS a folder there - and (2) I'm also having trouble getting further info from the exception. This is in C#/.NET in VS2017 on Win10.

What I've found is when I set a beakpoint and step through the code, it works perfectly! But when I remove the breakpoint I sometimes get exceptions when I shouldn't (so apparently some kind of timing issue). The relevant code is (note Dataservice is a Dropbox instance, accessed through an Interface)...

public async Task<Metadata> GetFolderMetadata(string path)
{
if (DropboxDebug) { log.SMAdebug("GetFolderMetadata has started");};
using (DropboxClient DxClient=new DropboxClient(AccessToken)){
    try {
        Metadata metadata =await DxClient.Files.GetMetadataAsync(path);
        if (DropboxDebug) { log.SMAdebug("GetFolderMetadata has ended normally");};
        return metadata;
        }
    catch {
        if (DropboxDebug) {log.SMAdebug("Exception in GetFolderMetadata");};
        return null;
        }
    }
}

    var folderMetadata=await DataService.GetFolderMetadata(thisFolder);
    if (folderMetadata!=null) {
        if (VMdebug) {
            System.Diagnostics.Debug.WriteLine(string.Format("{0} already exists - skipping",thisFolder));
            };
        continue;
    } else {
        if (VMdebug) {log.SMAdebug("Creating folder",thisFolder);};
        await DataService.CreateFolder(thisFolder,false);
        };
 

I did try to get further info from the exception by putting "catch (ApiException e) {...", but it wanted further info, and since I was getting "ApiException'1", I tried "catch (Apiexception<1>) {...", but VS didn't like that either. Not sure what syntax I need to put to get further info from this exception (which, as noted, being returned when it shouldn't when no breakpoints).

Not sure if there's an intentional limit to how often you can query the API, or if I'm just running into a glitch? i.e. not sure if I need to be dealing with it (presumably by sticking in a delay, though note I still would like to know how to get further info from the exception to confirm what it's complaining about), or if maybe there's something you need to look at on your end.

thanks,

  Donald.

[Greg-DB]

Hi Donald, there is a rate limiting system on the Dropbox API, but it's relatively generous. If you are hitting it though, the Dropbox API will return an explicit rate limiting error. It's possible you're missing that since you're not inspecting what kind of exception you're getting, as you mentioned. Likewise, there are other exceptions that can occur on GetMetadataAsync other than just 'not_found'.

Here's an example of how you can break out some of the various error cases:

try
{
    Metadata res = await client.Files.GetMetadataAsync(path);
    Console.WriteLine("{0}", res);
}
catch (ApiException<GetMetadataError> e)
{
    if (e.ErrorResponse.IsPath) {
        if (e.ErrorResponse.AsPath.Value.IsNotFound) {
            Console.WriteLine("Path not found!");
        } else if (e.ErrorResponse.AsPath.Value.IsMalformedPath) {
            Console.WriteLine("Malformed path!");
        } // and so on for the other .Is* methods as desired
        else {
            Console.WriteLine("LookupError: {0}", e.ErrorResponse.AsPath.Value);
        }
    } else {
        Console.WriteLine("GetMetadataError: {0}", e);
    }
} catch (Exception e)
{
    Console.WriteLine("Some other error: {0}", e);
}

[donaldp]

Hi Greg,

catch (ApiException<GetMetadataError> e)

 Thanks for that! That was the missing piece of the puzzle. I can confirm now that I'm getting "path/not_found/..." for a folder that has definitely already been created (FYI I'm dealing with input that has duplicates, and editing them out is not practical, so I'm just looking to skip processing the duplicates. i.e. test if that folder has already been created).

I'm also getting a lot of "Exception thrown: 'Dropbox.Api.RetryException' in Dropbox.Api.dll" messages, though it's not crashing the program, and also not being caught by my general catch statement for this line of code (perhaps because it's coming from somewhere in Dropbox.Api?). Note: I already had a 500ms delay due to part of the process being a Bing image search - which is limited to 3 per second - however I doubled it to 1000, and STILL getting this "not found" and retry exceptions for a folder that was already created 1000ms previously. 2000ms, same thing. 5000ms, same thing. Becoming ridiculous now. As noted, works perfectly with breakpoints, just not working at normal speed. Not sure how to fix this?

[donaldp]

My apologies. I disovered I'd forgotten to await the delay, so it wasn't actually waiting for it. Oops! Now that I've rectified that, the code is working correctly... sometimes. I originally had a 500ms wait between each item (since Bing image search limited to 3 per second), and I still get issues at that speed (500ms later, a Dropbox folder which has been created is still being reported as not existing yet), but I decided to try having a 350ms delay between each image, and the code is working correctly at that slower speed (though otherwise that is un-necessarily slow, since I should only need to wait for each item, not each image). Also got rid of most, but not all, of the retry exceptions at the slower speed. You said there was a generous API call limit - what is it? (I've not seen it mentioned) I can try pacing my code to that and see how it goes. So, there's still an issue, but I have a work-around for the time being.

[Greg-DB]

Thanks for following up. You shouldn't be getting 'not_found' for a path that already exists, but if you are calling to get the metadata immediately after the item is created (e.g., in the case of a duplicate in your input list), there is a chance that can happen. The CreateFolderV2Async method (as well as others, such as the upload methods) return the metadata for the new item, so I recommend keeping track of those results and de-duplicating locally, as that would likely be faster and less error-prone in this scenario. 

Alternatively, you can just allow CreateFolderV2Async (assuming that's what you're using) to run whenever you are getting 'not_found', and catch the CreateFolderError on that. I.e., to ignore it if it contains a WriteConflictError.Folder.

Anyway, a RetryException isn't actually explicit rate limiting. (That would be RateLimitException.) That just indicates an issue on the Dropbox servers so you can have your app retry the request if you wish.

[donaldp]

"Alternatively, you can just allow CreateFolderV2Async (assuming that's what you're using) to run whenever you are getting 'not_found', and catch the CreateFolderError on that. I.e., to ignore it if it contains a WriteConflictError.Folder"

I think that's the best solution. Thanks for the suggestion!

In the meantime, working on implementing that, I'm now hitting the ratelimit exceptions, so can you please let me know what the allowed rate is so that I can pace my code appropriately? Thanks.

[Greg-DB]

We don't document any specific numbers for the rate limit. Apps should be written to dynamically catch and handle any rate limiting errors instead of hard coding limits. You should use the returned RateLimitError.RetryAfter value to see how long to wait before retrying.

Also, note that not all RateLimitException indicate explicit rate limiting exactly. You should check the RateLimitReason in RateLimitError.Reason to see why the request couldn't be serviced at that time. RateLimitReason.TooManyRequests indicates explicit rate limiting, but RateLimitReason.TooManyWriteOperations just indicates "lock contention". You can read more about lock contention in the Data Ingress guide.