We Want to Hear From You! What Do You Want to See on the Community? Tell us here!
Forum Discussion
Retrieve thousands of files and folders without throttling
I'm trying to generate a nice big array for each user that contains all the files and folders the user owns as well as the users that each file/folder is shared with (e.g. users: ["John Doe": [{"name":"DemoDoc", "type": "file", "isShared": true, "sharedWithUsers": "bob@company.com", "joe@company.com"},{"name":"DemoFolder", "type": "folder", "isShared": false},...]]). I'll need to do this for a few hundred users. How should I do this with as few calls as possible?
The following is one approach per user:
1) Get Dropbox users (POST request): https://api.dropboxapi.com/2/team/members/list
2) Loop through each member and grab their team_member_id
3) For each member (currently 364 members) get all their folders and files recursively: https://api.dropboxapi.com/2/files/list_folder
4) For each member folder, if it's shared get the members and add folder to user object if user is the owner of shared folder: https://api.dropboxapi.com/2/sharing/list_folder_members
5) If folder isn't shared then add folder to user object
6) If file check to see if user is owner of file: https://api.dropboxapi.com/2/sharing/list_file_members
7) If owner of file add file to user object
This approach could be thousands of http calls just for one user, which leads me to think this could be a very time expensive approach as well as the possibility of throttling.
7 Replies
- Greg-DB
Dropbox Community Moderator
[Cross-linking for reference: https://stackoverflow.com/questions/74813769/retrieve-thousands-of-files-and-folders-without-throttling ]
It sounds like you have the right idea here, though be sure to use batch options/endpoints whenever possible. For instance, use recursive:true on /2/files/list_folder and then page through with /2/files/list_folder/continue instead of recursing for each folder yourself. Also, you can use /2/sharing/list_file_members/batch to get the members of multiple files at once.
Also, you may want to start from /2/team/namespaces/list and /2/team/namespaces/list/continue to iterate over and list unique namespaces (such as team member folders, shared folders, etc.) as that may be more efficient than working from the member list.
And for reference, the Dropbox API does have a rate limiting system, but if/when you do hit it, the back-off period is generally only on the scale of seconds/minutes, so that hopefully wouldn't be too disruptive if you do run in to that. You can find more information in the error documentation and Error Handling Guide.
If I used the /2/team/namespaces/list and /2/team/namespaces/list/continue endpoints how would I know who owns the folder and the files within the folder (and their owners)?
- Greg-DB
For team member folders, the /2/team/namespaces/list and /2/team/namespaces/list/continue results include the relevant team_member_id in the NamespaceMetadata object.
For shared folders, you can still use /2/sharing/list_folder_members and /2/sharing/list_folder_members/continue to list the shared folder's members and owner. (And content within a shared folder that isn't itself specifically shared doesn't have an owner distinct from its parent shared folder.)
On initial glance, the namespaces array is returning objects like the following:
{"name": "*****","namespace_id": "*****27","namespace_type": {".tag": "shared_folder"}},{"name": "*** Test","namespace_id": "*****92","namespace_type": {".tag": "team_folder"}}The exception is when "name": "Root". Is there a reason the other objects don't include the team_member_id?Also for /2/sharing/list_file_members/batch, if I understand correctly this will only list members that are different from the inherited sharing of the parent folder. If I call the batch endpoint I'm getting an array of empty results, like so:
"file": "id:****","result": {".tag": "result","members": {"users": [],"groups": [],"invitees": []},"member_count": 0}However, if I call the /2/sharing/list_file_members endpoint (using just one file id, of course) I get each member and their access type. The problem with the batch approach is how do I know who are the members and owner of each file then?- Greg-DB
Other types of namespaces like that aren't specific to a single team member. Shared folders have multiple members (and the owner can change over time), for instance, and team folders are owned by the team.
And that's correct, /2/sharing/list_file_members/batch does work differently than /2/sharing/list_file_members. Refer to the documentation for more information.
Thanks Greg,
Seems like the only way to get info on who the owner of each file is to call the https://api.dropboxapi.com/2/sharing/list_file_members endpoint then. Batching only tells me if members are different than the inherited parent folder, and https://api.dropboxapi.com/2/files/list_folder doesn't give me that information.
About Dropbox API Support & Feedback
Find help with the Dropbox API from other developers.
