cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Announcements
Want to learn some quick and useful tips to make your day easier? Check out how Calvin uses Replay to get feedback from other teams at Dropbox here.

Dropbox API Support & Feedback

Find help with the Dropbox API from other developers.

cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 

Re: Internal server error when sharing folders

Internal server error when sharing folders

Levi2
Explorer | Level 3
Go to solution

Hi, is there some kind of limitation with sharing folders? I'm trying to share 5.4k folders and I'm getting 

PollError('internal_error', None))

 

consistently. Sometimes the files share, sometimes they don't. I can manually share them no problem and usually I can share them as one offs in a python REPL using the dropbox sdk.

There are no error details.

My code is functionallt simialr to the snippet below:

import dropbox

#censored
access_token = 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'
dbx = dropbox.Dropbox(access_token)

def get_entries(folder_result_list):
    '''using this so we don't try and share the same folder twice'''
    entries = []
    for entry in folder_result_list.entries:
        if not hasattr(entry,'shared_folder_id') and not entry.shared_folder_id:
            entries.append(entry)
    return entries

folder_result_list = dbx.files_list_folder(parent_folder,include_mounted_folders=False,limit=2000)
entries = get_entries(folder_result_list)

while folder_result_list.has_more:
    folder_result_list = dbx.files_list_folder_continue(folder_result_list.cursor)
    entries.extend(get_entries(folder_result_list))

#filter for only FolderMetadata so we don't try and share DeletedFolder's or individual files    
entry_dict  = {entry.id:entry for entry in entries if isinstance(entry,dropbox.files.FolderMetadata)}

for index,keyval in enumerate(entry_dict.items()):
    entry_id = keyval[0]
    entry = keyval[1]
    try:
        #forcing async so we get consistent results. checking the job status and doing stuff with the shared_folder_id later on in the script
        entry_id_to_job_id[entry_id] = dbx.sharing_share_folder(entry.path_lower,force_async=True).get_async_job_id()
    except dropbox.exceptions.ApiError as e:
        #do some logging of errors here
        pass

The full error looks something like this:

ApiError('0ed9d04d8f01449b5dedea33b7445fee', PollError('internal_error', None))

 

 

1 Accepted Solution

Accepted Solutions

Greg-DB
Dropbox Staff
Go to solution

Thanks for the post, and apologies that the error message isn't more helpful in this case. For this `internal_error` case, I looked it up and I can confirm that it's actually due to "lock contention". 

That's a result of how Dropbox works on the backend. It's a technical inability to make multiple modifications, such as sharing a folder, at the same time. The simultaneous activity could be coming from your app itself, or elsewhere, e.g., from the user's desktop client. It can come from the same user, or another member of a shared folder. You can find more information about lock contention here:

https://www.dropbox.com/developers/reference/data-ingress-guide

In short, to avoid this error, you should avoid making multiple concurrent state modifications. E.g., don't issue multiple such requests at a time, and use batch endpoints whenever possible. That won't guarantee that you won't run in to this error though, as contention can still come from other sources, so you may also want to implement automatic retries in your app. For this particular scenario of sharing multiple folders, I also recommend adding a delay between the consecutive share operations, which can help in certain cases.

I'll send this along as a request to the team to improve the error reporting.

View solution in original post

3 Replies 3

Greg-DB
Dropbox Staff
Go to solution

Thanks for the post, and apologies that the error message isn't more helpful in this case. For this `internal_error` case, I looked it up and I can confirm that it's actually due to "lock contention". 

That's a result of how Dropbox works on the backend. It's a technical inability to make multiple modifications, such as sharing a folder, at the same time. The simultaneous activity could be coming from your app itself, or elsewhere, e.g., from the user's desktop client. It can come from the same user, or another member of a shared folder. You can find more information about lock contention here:

https://www.dropbox.com/developers/reference/data-ingress-guide

In short, to avoid this error, you should avoid making multiple concurrent state modifications. E.g., don't issue multiple such requests at a time, and use batch endpoints whenever possible. That won't guarantee that you won't run in to this error though, as contention can still come from other sources, so you may also want to implement automatic retries in your app. For this particular scenario of sharing multiple folders, I also recommend adding a delay between the consecutive share operations, which can help in certain cases.

I'll send this along as a request to the team to improve the error reporting.

Levi2
Explorer | Level 3
Go to solution

Thanks for the links.

This makes sense since I was trying to share thousands of folders. However, I avoided multi-threading and no one else is using this folder since it's a test folder.

It seems that namespace locking is the culprit since all of the folders are in the same name space and I'm using force_asyn=True leaving when the folder actually gets shared up to dropbox.

I know that there are batch endpoints for uploading data, but are there batch endpoints for sharing folders?

 

Greg-DB
Dropbox Staff
Go to solution

No, unfortunatelty there isn't a batch option for sharing folders in particular, but I'll pass this along as a feature request. I can't promise if or when that might be implemented though. 

Need more support?