You might see that the Dropbox Community team have been busy working on some major updates to the Community itself! So, here is some info on what’s changed, what’s staying the same and what you can expect from the Dropbox Community overall.
Forum Discussion
Beatso
5 years agoExplorer | Level 4
File uploading slow
Hi,
I'm using the javscript sdk, and trying to upload multiple files (one after another) with dbx.filesUpload. Due to the nature of my projectm I have to upload multiple files when the user sends a request. Each file is taking around a second, which results in long waiting times for the user when they need, say, 50 files.
My code:
async function uploadMultipleFiles (storageFilePaths,packFilePaths,packRoot) { try { for (i in storageFilePaths) { console.log(await dbx.filesUpload({ path: packRoot+packFilePaths[i], contents: fs.readFileSync(storageFilePaths[i])})) } } catch (err) { throw err } }
Full code is here. I have to use await; see this thread. Would there be a quicker way to do this without getting too many requests? Would it be quicker to copy the files from another place in my dropbox rather than upload them like this?
- Greg-DBDropbox Staff
There are a number of variables that will affect overall performance of transfers like this, some of which can't be controlled unfortunately. There are some other options that may be helpful though, depending on the specifics of your scenario.
If the files do already exist in the Dropbox account and you just need to make copies of them, using filesCopyBatchV2 may be faster.
Or, if the files already exist in another Dropbox account, and you have authorization to both of them, you may want to try using filesCopyReferenceGet and filesCopyReferenceSave instead.
Otherwise, if you have a significant number of files to upload, you may want to use the "Batch Upload" method described in the Performance Guide. That way, you can upload the file data for several files at a time in parallel, using filesUploadSessionStart and filesUploadSessionAppendV2 as needed, and then commit them all at once when those are done using filesUploadSessionFinishBatch.
- BeatsoExplorer | Level 4
Thanks for your help.
Using the batch upload method, how to I "group files that are being uploaded into one or more batches"? Also, do I use filesUploadSessionStart for each file, or do I use it once, and append for every other file? Is there anywhere where I could find an example of uploading multiple files with a batch upload?Edit: I found your example here. My code is as follows:var entries = [] for (i in storageFilePathsToUpload) { let fileData = fs.readFileSync(storageFilePathsToUpload[i]) dbx.filesUploadSessionStart({ contents: fileData, close: true, }) .then(function (response) { entries.push({cursor:{session_id:response.session_id,offset:fileData.length},commit:{path:packPath+packFilePathsToUpload[i]}}) console.log("entriesinprog") }) .catch(function (err) { console.log(err); }); } console.log(entries) dbx.filesUploadSessionFinishBatch({entries:entries}) .then(function(response){ dbx.filesUploadSessionFinishBatchCheck({async_job_id: response.async_job_id}) .then(function(response){ console.log(response) }) .catch(function(err){ console.log(err) }) }).catch(err => { console.log(err) })It's not working because it's not executing in the right order. The filesUploadSessionFinishBatchCheck is logging its response before the filesUploadSessionStart has finished uploading. Why is this, and how could I only finish the batch once the for loop has finished? (I'm not in an async function). It might sort of relate to this thread, but I obviously can't use .then on a for loop.Thanks once again for your help.Edit 2: I got the above working, and now my code looks like this:
async function startSessions () { entries = [] for (let [index,val] of storageFilePathsToUpload.entries()) { let fileData = fs.readFileSync(val) await dbx.filesUploadSessionStart({ contents: fileData, close: true, }) .then(function (response) { entries.push({cursor:{session_id:response.session_id,offset:fileData.length},commit:{path:packPath+packFilePathsToUpload[index]}}) }) .catch(function (err) { console.error(err) }) } } startSessions().then(()=>{ console.log(entries) dbx.filesUploadSessionFinishBatch({entries:entries}) .then(function(response){ dbx.filesUploadSessionFinishBatchCheck({async_job_id: response.async_job_id}) .then(function(output){ console.log(output) // THIS RETURNS { '.tag': 'in_progress' } // get share link and send it getShareLink("/test").then((response)=>{res.send(response)}).catch((error)=>{res.send("error");console.error(error)}) }) .catch(function(err){ console.error(err) }) }).catch(err => { console.error(err) }) }).catch((err)=>{console.error(err)})
When the output of filesUploadSessionFinishBatchCheck is logged, it gives { '.tag': 'in_progress' }. Is there a way to not run the code until the batch is finished?
- Greg-DBDropbox Staff
Your app needs to call filesUploadSessionFinishBatchCheck occasionally to determine when the batch has finished, and only then proceed with whatever other operations it needs to run based on the result (similar to your other thread, except that you may need to call a few times). That batch operation is happening on the Dropbox servers, so your app won't know by itself when that's done without checking with the servers via that method.
So, you should have your app poll filesUploadSessionFinishBatchCheck occasionally, say, every few seconds, until FilesUploadSessionFinishBatchJobStatus is no longer 'in_progress'. Then you can check the FilesUploadSessionFinishBatchResult in FilesUploadSessionFinishBatchJobStatus.complete to see the results and continue from there.
About Dropbox API Support & Feedback
Find help with the Dropbox API from other developers.
5,894 PostsLatest Activity: 2 hours agoIf you need more help you can view your support options (expected response time for an email or ticket is 24 hours), or contact us on X or Facebook.
For more info on available support options for your Dropbox plan, see this article.
If you found the answer to your question in this Community thread, please 'like' the post to say thanks and to let us know it was useful!