Need to see if your shared folder is taking up space on your dropbox 👨💻? Find out how to check here.
Forum Discussion
Dropbox Python API and Google Cloud Composer
I am attempting to orchestrate downloads using the Dropbox Python API and Google Cloud's Cloud Composer. I am trying to download the files off my dropbox account, unzip them and them upload them to m...
I updated the code as follows and ran it both as a standalone script and in my airflow environment:
for entry in files.entries:
print(entry.path_lower)
dbx.files_download_to_file(os.path.join(os.path.expanduser("~"), entry.name), entry.path_lower)
print('downloaded file')
It works when I ran it as part of my standalone script. In the Google Cloud Composer Environment, no error log was produced i.e. it ran successfully. However I cannot find the files which were supposed to download to my computer. Is it because I am using a cloud environment to run the script?
This is the complete log produced from my Google Cloud Composer Environment (i.e. Airflow)
*** Reading remote log from gs://us-central1-fitnessdata-3bac80e2-bucket/logs/fitness_tracker/list/2022-07-05T19:35:17.822704+00:00/1.log.
[2022-07-05 19:35:34,499] {taskinstance.py:671} INFO - Dependencies all met for <TaskInstance: fitness_tracker.list 2022-07-05T19:35:17.822704+00:00 [queued]>
[2022-07-05 19:35:34,566] {taskinstance.py:671} INFO - Dependencies all met for <TaskInstance: fitness_tracker.list 2022-07-05T19:35:17.822704+00:00 [queued]>
[2022-07-05 19:35:34,567] {taskinstance.py:881} INFO -
--------------------------------------------------------------------------------
[2022-07-05 19:35:34,568] {taskinstance.py:882} INFO - Starting attempt 1 of 1
[2022-07-05 19:35:34,568] {taskinstance.py:883} INFO -
--------------------------------------------------------------------------------
[2022-07-05 19:35:34,626] {taskinstance.py:902} INFO - Executing <Task(PythonOperator): list> on 2022-07-05T19:35:17.822704+00:00
[2022-07-05 19:35:34,647] {standard_task_runner.py:54} INFO - Started process 81707 to run task
[2022-07-05 19:35:35,863] {standard_task_runner.py:77} INFO - Running: ['airflow', 'run', 'fitness_tracker', 'list', '2022-07-05T19:35:17.822704+00:00', '--job_id', '1961', '--pool', 'default_pool', '--raw', '-sd', 'DAGS_FOLDER/exercise.py', '--cfg_path', '/tmp/tmp3j0kew8z']
[2022-07-05 19:35:35,870] {standard_task_runner.py:78} INFO - Job 1961: Subtask list
[2022-07-05 19:35:36,807] {logging_mixin.py:120} INFO - Running <TaskInstance: fitness_tracker.list 2022-07-05T19:35:17.822704+00:00 [running]> on host airflow-worker-7b8cd9bbf6-ptgxv
[2022-07-05 19:35:36,927] {dropbox_client.py:471} INFO - Request to files/list_folder
[2022-07-05 19:35:37,358] {logging_mixin.py:120} INFO - /apps/rungap/export/2022-06-29_7-20-04 pm_hk_1656544804.zip
[2022-07-05 19:35:37,359] {dropbox_client.py:471} INFO - Request to files/download
[2022-07-05 19:35:38,096] {logging_mixin.py:120} INFO - downloaded file
[2022-07-05 19:35:38,098] {logging_mixin.py:120} INFO - /apps/rungap/export/2022-06-26_4-34-36 pm_hk_1656275676.zip
[2022-07-05 19:35:38,098] {dropbox_client.py:471} INFO - Request to files/download
[2022-07-05 19:35:38,599] {logging_mixin.py:120} INFO - downloaded file
[2022-07-05 19:35:38,604] {logging_mixin.py:120} INFO - /apps/rungap/export/2022-06-25_12-00-07 pm_hk_1656172807.zip
[2022-07-05 19:35:38,604] {dropbox_client.py:471} INFO - Request to files/download
[2022-07-05 19:35:39,114] {logging_mixin.py:120} INFO - downloaded file
[2022-07-05 19:35:39,115] {logging_mixin.py:120} INFO - /apps/rungap/export/2022-06-16_7-52-59 pm_hk_1655423579.zip
[2022-07-05 19:35:39,115] {dropbox_client.py:471} INFO - Request to files/download
[2022-07-05 19:35:39,523] {logging_mixin.py:120} INFO - downloaded file
[2022-07-05 19:35:39,524] {logging_mixin.py:120} INFO - /apps/rungap/export/2022-05-23_6-08-26 pm_hk_1653343706.zip
[2022-07-05 19:35:39,524] {dropbox_client.py:471} INFO - Request to files/download
[2022-07-05 19:35:39,980] {logging_mixin.py:120} INFO - downloaded file
[2022-07-05 19:35:39,980] {logging_mixin.py:120} INFO - /apps/rungap/export/2022-05-10_7-38-44 pm_hk_1652225924.zip
[2022-07-05 19:35:39,981] {dropbox_client.py:471} INFO - Request to files/download
[2022-07-05 19:35:41,998] {logging_mixin.py:120} INFO - downloaded file
[2022-07-05 19:35:41,999] {logging_mixin.py:120} INFO - /apps/rungap/export/2022-04-20_6-32-35 pm_hk_1650493955.zip
[2022-07-05 19:35:42,000] {dropbox_client.py:471} INFO - Request to files/download
[2022-07-05 19:35:42,447] {logging_mixin.py:120} INFO - downloaded file
[2022-07-05 19:35:42,448] {logging_mixin.py:120} INFO - /apps/rungap/export/2022-04-08_4-33-54 pm_hk_1649450034.zip
[2022-07-05 19:35:42,449] {dropbox_client.py:471} INFO - Request to files/download
[2022-07-05 19:35:43,070] {logging_mixin.py:120} INFO - downloaded file
[2022-07-05 19:35:43,071] {logging_mixin.py:120} INFO - /apps/rungap/export/2022-03-31_7-59-07 pm_hk_1648771147.zip
[2022-07-05 19:35:43,071] {dropbox_client.py:471} INFO - Request to files/download
[2022-07-05 19:35:43,570] {logging_mixin.py:120} INFO - downloaded file
[2022-07-05 19:35:43,570] {logging_mixin.py:120} INFO - /apps/rungap/export/2022-03-24_1-42-20 pm_hk_1648143740.zip
[2022-07-05 19:35:43,571] {dropbox_client.py:471} INFO - Request to files/download
[2022-07-05 19:35:43,983] {logging_mixin.py:120} INFO - downloaded file
[2022-07-05 19:35:43,984] {logging_mixin.py:120} INFO - /apps/rungap/export/2022-03-23_4-51-28 pm_hk_1648068688.zip
[2022-07-05 19:35:43,985] {dropbox_client.py:471} INFO - Request to files/download
[2022-07-05 19:35:44,400] {logging_mixin.py:120} INFO - downloaded file
[2022-07-05 19:35:44,401] {logging_mixin.py:120} INFO - /apps/rungap/export/2022-03-01_6-02-44 pm_hk_1646175764.zip
[2022-07-05 19:35:44,401] {dropbox_client.py:471} INFO - Request to files/download
[2022-07-05 19:35:44,884] {logging_mixin.py:120} INFO - downloaded file
[2022-07-05 19:35:44,885] {logging_mixin.py:120} INFO - /apps/rungap/export/2022-01-25_4-47-32 pm_hk_1643147252.zip
[2022-07-05 19:35:44,885] {dropbox_client.py:471} INFO - Request to files/download
[2022-07-05 19:35:45,329] {logging_mixin.py:120} INFO - downloaded file
[2022-07-05 19:35:45,330] {logging_mixin.py:120} INFO - /apps/rungap/export/2021-12-23_5-44-12 pm_hk_1640299452.zip
[2022-07-05 19:35:45,331] {dropbox_client.py:471} INFO - Request to files/download
[2022-07-05 19:35:45,852] {logging_mixin.py:120} INFO - downloaded file
[2022-07-05 19:35:45,853] {logging_mixin.py:120} INFO - /apps/rungap/export/2021-12-19_3-36-59 pm_hk_1639946219.zip
[2022-07-05 19:35:45,854] {dropbox_client.py:471} INFO - Request to files/download
[2022-07-05 19:35:46,495] {logging_mixin.py:120} INFO - downloaded file
[2022-07-05 19:35:46,495] {logging_mixin.py:120} INFO - /apps/rungap/export/2021-12-16_4-19-31 pm_hk_1639689571.zip
[2022-07-05 19:35:46,496] {dropbox_client.py:471} INFO - Request to files/download
[2022-07-05 19:35:47,121] {logging_mixin.py:120} INFO - downloaded file
[2022-07-05 19:35:47,140] {python_operator.py:114} INFO - Done. Returned value was: None
[2022-07-05 19:35:47,205] {taskinstance.py:1058} INFO - Marking task as SUCCESS.dag_id=fitness_tracker, task_id=list, execution_date=20220705T193517, start_date=20220705T193534, end_date=20220705T193547
[2022-07-05 19:35:48,918] {local_task_job.py:102} INFO - Task exited with return code 0
Greg-DB
Dropbox Community Moderator
Dropbox Community ModeratorIt looks like there wasn't an error, so the files should have been successfully saved to the home folder. I can't offer help with interacting with the filesystem in your environment though, as that's made by Dropbox, so I recommend referring to the documentation for your environment for that.
For reference, you can see how the SDK writes to the specified location on the local filesystem here.
About Dropbox API Support & Feedback
Find help with the Dropbox API from other developers.
