download/upload and short term tokens

Any endpoint using "User Authentication" (or "Team Authentication") does require an access token. That includes both the download and the "upload session" endpoints, for uploading large files. The access token can be either a legacy long-lived access token, or a short-lived access token.

So, if you're using a short-lived access token to call any of those endpoints and get back an 'expired_access_token' error, that indicates that that short-lived access token has expired, in which case you'll need to get a new one and retry the call with the new short-lived access token.

Our multiplart xfer sizes are typically 20-200 GB, would you expect any issues other then tokens?

That should be fine. The upload session endpoints support uploading files up to 350 GB in size.