Announcements
Do you have an idea for Dropbox Paper? Share it with us!

Internal server error when sharing folders

Explorer | Level 3

Hi, is there some kind of limitation with sharing folders? I'm trying to share 5.4k folders and I'm getting 

PollError('internal_error', None))

 

consistently. Sometimes the files share, sometimes they don't. I can manually share them no problem and usually I can share them as one offs in a python REPL using the dropbox sdk.

There are no error details.

My code is functionallt simialr to the snippet below:

import dropbox

#censored
access_token = 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'
dbx = dropbox.Dropbox(access_token)

def get_entries(folder_result_list):
    '''using this so we don't try and share the same folder twice'''
    entries = []
    for entry in folder_result_list.entries:
        if not hasattr(entry,'shared_folder_id') and not entry.shared_folder_id:
            entries.append(entry)
    return entries

folder_result_list = dbx.files_list_folder(parent_folder,include_mounted_folders=False,limit=2000)
entries = get_entries(folder_result_list)

while folder_result_list.has_more:
    folder_result_list = dbx.files_list_folder_continue(folder_result_list.cursor)
    entries.extend(get_entries(folder_result_list))

#filter for only FolderMetadata so we don't try and share DeletedFolder's or individual files    
entry_dict  = {entry.id:entry for entry in entries if isinstance(entry,dropbox.files.FolderMetadata)}

for index,keyval in enumerate(entry_dict.items()):
    entry_id = keyval[0]
    entry = keyval[1]
    try:
        #forcing async so we get consistent results. checking the job status and doing stuff with the shared_folder_id later on in the script
        entry_id_to_job_id[entry_id] = dbx.sharing_share_folder(entry.path_lower,force_async=True).get_async_job_id()
    except dropbox.exceptions.ApiError as e:
        #do some logging of errors here
        pass

The full error looks something like this:

ApiError('0ed9d04d8f01449b5dedea33b7445fee', PollError('internal_error', None))

 

 

1 Accepted solution

Accepted Solutions

Re: Internal server error when sharing folders

Dropboxer

Thanks for the post, and apologies that the error message isn't more helpful in this case. For this `internal_error` case, I looked it up and I can confirm that it's actually due to "lock contention". 

That's a result of how Dropbox works on the backend. It's a technical inability to make multiple modifications, such as sharing a folder, at the same time. The simultaneous activity could be coming from your app itself, or elsewhere, e.g., from the user's desktop client. It can come from the same user, or another member of a shared folder. You can find more information about lock contention here:

https://www.dropbox.com/developers/reference/data-ingress-guide

In short, to avoid this error, you should avoid making multiple concurrent state modifications. E.g., don't issue multiple such requests at a time, and use batch endpoints whenever possible. That won't guarantee that you won't run in to this error though, as contention can still come from other sources, so you may also want to implement automatic retries in your app. For this particular scenario of sharing multiple folders, I also recommend adding a delay between the consecutive share operations, which can help in certain cases.

I'll send this along as a request to the team to improve the error reporting.

Reply
Loading...
3 Replies

Re: Internal server error when sharing folders

Dropboxer

Thanks for the post, and apologies that the error message isn't more helpful in this case. For this `internal_error` case, I looked it up and I can confirm that it's actually due to "lock contention". 

That's a result of how Dropbox works on the backend. It's a technical inability to make multiple modifications, such as sharing a folder, at the same time. The simultaneous activity could be coming from your app itself, or elsewhere, e.g., from the user's desktop client. It can come from the same user, or another member of a shared folder. You can find more information about lock contention here:

https://www.dropbox.com/developers/reference/data-ingress-guide

In short, to avoid this error, you should avoid making multiple concurrent state modifications. E.g., don't issue multiple such requests at a time, and use batch endpoints whenever possible. That won't guarantee that you won't run in to this error though, as contention can still come from other sources, so you may also want to implement automatic retries in your app. For this particular scenario of sharing multiple folders, I also recommend adding a delay between the consecutive share operations, which can help in certain cases.

I'll send this along as a request to the team to improve the error reporting.

Reply
Loading...

Re: Internal server error when sharing folders

Explorer | Level 3

Thanks for the links.

This makes sense since I was trying to share thousands of folders. However, I avoided multi-threading and no one else is using this folder since it's a test folder.

It seems that namespace locking is the culprit since all of the folders are in the same name space and I'm using force_asyn=True leaving when the folder actually gets shared up to dropbox.

I know that there are batch endpoints for uploading data, but are there batch endpoints for sharing folders?

 

Reply
Loading...

Re: Internal server error when sharing folders

Dropboxer

No, unfortunatelty there isn't a batch option for sharing folders in particular, but I'll pass this along as a feature request. I can't promise if or when that might be implemented though. 

Reply
Loading...
API Support

Have a question? Our Dropbox Community is here to help you find an answer!
You won’t have to wait too long for a reply, but why not learn more about the Community while you’re here.

Not the right discussion for you? Find out what else our Community has to say about API Support.

Or, search our Help Center for even more.

Who's talking
Top contributors to this post