Massive Data Upload and Sync Process - 20TB

[ehcropydoc]

Hi everyone,

I am writing here to explain the process of our most recent data upload of 20TB to Dropbox via a 2019 Mac Pro connected to a 10Gbps Fiber line. The goal is to help those understand some of the issues we experienced as well as setting expectations to those who plan on uploading massive amounts of data at once. Below are the steps:

1. Procure a 10Gb fiber PCIe card for the 2019 Mac Pro - Solo10G SFP+ 10GbE PCIe Card with SFP+ Module - after working closely with our Apple business rep, we found the this card which would work for our requirements. Yes, the 2019 Mac Pro has 10Gb ethernet ports, but we only had Fiber lines in our setup, not Ethernet. Not to mention adapters usually have 1Gb limits so this was not an acceptable solution either. We wanted to experience max speeds so we opted for a dedicated 10GbE fiber card.
2. Speed Tests - we conducted speed tests after card installation and confirmed the speeds were around 4.5Gbps to 5Gbps upload and download. We could not figure out if the 50% falloff was on the 2019 Mac Pro side or on the ISP side. Still however, 5Gbps is no joke so we continued onward. (PS: We could not conduct speed tests via the Speedtest.net when on 4GB+ fiber. We had to download software to the PC/Mac to get true internet speeds.)
3. Upload - We already had 20TB on the Dropbox Folder moved and was paused & ready to sync. We simply turned the computer on, ensured it had a good connection, and then turned Sync on via the Dropbox for Mac application. Thats it. The data automatically began to upload just as it would any other sync.
   • Realized Speeds - Even with the 4.5 to 5Gbps up|down speeds, we only saw between 600Mbps to 900Mbps being uploaded to the Dropbox cloud. We hoped to see at least 2Gbps or 3Gbps being sent to Dropbox but this was not the case unfortunately. Still however, the ~750Mbps speed was enough to upload the 20TB in about three days which we were ok with.
   • Note: At the beginning of the sync, the speeds were extremely slow which we contributed to the initial file indexing. About 3 hours in to the main sync is when the speeds started to get to their highest levels. Also, we checked our ISP speeds twice per day and at its LOWEST point we saw internet speeds of 3.5Gbps.
4. Online-Only (Spoiler Warning - massive bottleneck!) - After about 70 hours of upload time, the full 20TB was on the cloud. Our Dropbox account confirmed the newly uploaded data storage amount. Our next step was to get this 20TB off of our 2019 Mac Pro external RAID (Pegasus R4i). Again, the process was just like any other file(s). We simply highlighted thew newly uploaded files and right-clicked, then selected "Make online-only". Simple right? Yeah, not so much. This is where we noticed lots of issues with our system. Essentially this 20TB "indexing?" caused us to essentially have to run Dropbox completely via the browser. There was no possibility of doing other things like downloading files via the Dropbox Mac application or Double clicking and a small JPEG opening up. The application simply said, "NO" and only had one task in mind, get these 20TB worth of files OFF of this hard drive. The good news? It did so, only taking a whopping 500 hours to remove these files off of the drive.
   • A few notes: We were able to pause syncing during the "file-removal" / online-only process), restart the computer, resume syncing all without any error messages which was very impressive. I expected the application to crash, hang up, etc. Not at all. Note: We did NOT pause syncing during the main 20TB upload.
   • Why did it take so long? I am not a software engineer but clearly it did not enjoy this task and it could be due to a number of reasons. Maybe simply the application was not built to handle 20TB? Maybe our computer did not have the power? Maybe it was due to the main Dropbox drive being on a RAID array? Who knows.

Thats about all I wanted to say here. Again, the purpose of this small write up is to show how the software/cloud performed under extreme tasking. I would say overall very well considering. Hopefully this helps someone out there who's considering uploading massive amounts of data at once.

TLDR: Uploading massive amounts of data via fiber internet = quick and easy. Removing this data off the drive / Making files "Online-only" = very time-consuming and not for the faint of heart.

2019 Mac Pro specs:

[Graham]

Hi @ehcropydoc,

I hope this message finds you well today. I just wanted to take a moment to say a big thank you for taking the time to share this write up with the Dropbox Community. Feedback like this in invaluable to both us and to our Community members and we're sure that this will be a huge help to others who plan to do something similar in the future.

By the sounds of it, this was a mammoth task, meticulously planned and executed with aplomb! So, hats off to everyone who was involved. We're delighted to hear that you thought Dropbox performed well too, even if it was a bit slower than expected. It's hard to say what might have caused this to work at the pace it did, given the sheer volume of the task at hand, but the important thing is you got there in the end without too many hiccups along the way.  

If there's anything else at all that you would like to share about your experience with Dropbox, good or bad, we're always here and happy to listen!

Thanks again, and have a great day!

Graham

[DanijelD]

Hi,

Thank you for this! We also tried to upload 7TB worth of data to Dropbox via the same way you did this. Our issue was we used an older Mac with an old OS. That resulted in some unsupported file issues and Dropbox-app indexing issues. We we want to try again with a newer Mac.

One question I do have is how did you manage to add an external drive to the indexing mapping for the Dropbox-app? It seems the data needs to be on the HD where the app is installed on. Or did I miss some setting?

Thank you!

[ehcropydoc]

Maybe we were able to do this before the recent changes? The drive to which our Dropbox folder resides on is technically an "external" drive although it is physically located internal to the Mac Pro chassis. But as I type this, our folder has not yet moved.

Note: We are still on 12.4 Monterey.