When an upload fails

A submission that DataPipe has accepted but not yet written to your storage provider gets HTTP 202 instead of 201. The body carries error: null.

Treat a 202 as success and don't resubmit. DataPipe already has the data and the retry is scheduled. Sending it again would either store the participant's data twice under two filenames, or fail the second time on a duplicate name. If your experiment code retries on anything other than 201, change it to treat any 2xx as success.

Five situations produce a 202, on both /api/data and /api/base64:

• Your stored credential failed to resolve. An expired or revoked refresh token, a failed token-refresh request, or an expired static token (Dataverse). DataPipe treats this as recoverable: reconnecting the account, or a transient refresh problem clearing up, is enough for the next automatic retry to succeed, so the submission is queued instead of rejected. Having no connection at all is different. With nothing to retry against, that still comes back as a rejection (PROVIDER_NOT_CONNECTED in the API docs' error codes table).
• The provider write failed. The most common case: an outage, a rate limit, a credential that expired mid-write, or a Zenodo record that is already full.
• DataPipe couldn't check the filename. Its filename record (the list of names this experiment has already used) couldn't be rebuilt, usually because your Drive folder, Dataverse dataset, or Zenodo deposition is missing or access was revoked. Rather than risk overwriting existing data, DataPipe queues the submission.
• The filename record was being rebuilt. Another request was already rebuilding it, which takes at most 60 seconds.
• An archive merge was in progress. A merge was rearranging your Zenodo record, so writes were queued rather than rejected. These entries carry the code CONTENTION and clear in about a minute.

A queued upload still counts toward your session limit. It's a real session, just not stored yet.

A background worker wakes up every five minutes and takes up to 25 queued uploads whose next attempt is due, oldest first. It spreads them across accounts, one per account per pass, so one researcher's broken connection can't hold up everyone else's data.

Each queued upload gets five attempts. The wait doubles after each failure, starting at one hour:

So an upload that keeps failing runs out of attempts about 31 hours after it was queued. The doubling is capped at 24 hours, and if your provider sends a Retry-After header, DataPipe honors that instead, clamped to the same 24 hours.

Running out of retries is not the same as losing the file. DataPipe keeps the file for the rest of its retention window, so a permanently failed upload stays downloadable from your dashboard for days after the last attempt.

How long the retention window is, and when the clock starts. Retention

Failures that are handled faster

Two kinds of failure clear much sooner than an outage does, so they get their own schedules.

• Write contention (CONTENTION): another write to the same folder, dataset, or deposition is already in flight, which clears in seconds. The first attempt is 60 seconds out and the waits are 2, 4, 8, 16, and 30 minutes, so all five attempts happen within about 31 minutes of queueing.
• Expired credentials and provider blips (AUTH_EXPIRED, UNAVAILABLE): these look the same as a problem that has already fixed itself, so DataPipe takes one look after 60 seconds instead of waiting an hour. If that attempt fails, it falls back to the hourly chain above, giving five attempts across about 30 hours from queueing.

Because the worker runs every five minutes, a 60-second wait means the attempt lands at the next five-minute tick.

One case costs nothing at all. If an archive merge is running on your Zenodo record when the retry comes due, the upload is rescheduled 60 seconds later without using up an attempt.

Uploads that never reported back

If a request is cut off before it can finish (a server restart, or a memory limit on a very large submission), the participant's data has already been written to DataPipe's storage, so it's still there. A recovery sweep runs every fifteen minutes, picks up anything that has been sitting for more than fifteen minutes, and moves it into the upload queue, where it shows up on your dashboard and retries like any other queued file.

One limitation: a session recovered this way gets its raw data file, but Psych-DS metadata and derived tables aren't regenerated for it. The raw file is the source of truth, and the next live submission re-merges the dataset description.

Whenever an experiment has queued uploads, its dashboard shows a status line: “N upload(s) waiting to be stored” when nothing has been attempted yet, “N upload(s) failed” once any one of them has used up its attempts, and a plain (uncoloured) badge in between if everything queued is simply waiting its turn -- DataPipe reserves the orange warning for uploads it has actually tried and not yet landed. The panel below it lists the files.

Not every queued file has been attempted. A file can be held before DataPipe has tried to store it at all -- recovered from a session that did not finish, kept after a metadata problem, or paused while DataPipe reorganizes your provider's storage -- and the panel says so plainly (“Waiting to be stored”) rather than describing it as a failed or retrying upload.

Each row tells you:

• Filename: the name the submission would be stored under.
• Status: waiting to be stored, retrying, being uploaded/stored right now, or failed, plus when the next (or first) attempt is due.
• Reason: a plain-language description of why the file is here -- not always a failure.
• Kept for another: how much of the retention window is left.

The panel also has a “What is happening to these files?” section explaining the three states above, and the download controls described below.

You never have to wait for a retry to succeed to get your data. Every row in the queued files panel has a download button that gives you that one file, decrypted, under the filename it was submitted with. Text files come back as CSV or JSON. Base64 submissions are decoded back to the original binary.

Download all as ZIP collects every queued, in-flight, and failed file for the experiment into a single archive named after the experiment ID. Use it as soon as any file shows as failed. It costs nothing, and it means you hold a copy no matter what the retries do.

If a download fails, nothing is lost. DataPipe still holds the file for the rest of its retention window. If the ZIP won't build, try the single-file buttons.

The endpoint behind these buttons, if you want to script it. Queue status

After the fifth failed attempt, the upload is marked failed and isn't tried again. Download it from the dashboard and add it to your Drive folder, Dataverse dataset, or Zenodo deposition by hand. Don't put it off: the file is deleted when its retention window ends, whether it failed or not.

Some failures are permanent from the start, because retrying could never help. DataPipe marks these failed on the first pass rather than spending five attempts on them:

• The experiment was finalized while the upload was queued. Finalizing seals your Zenodo record permanently, so the file can't be added to it. The data isn't lost. Download it from the queued files panel.
• The experiment or the owning account no longer exists.
• The stored copy couldn't be read. Rare, and it still appears in the panel with a reason rather than vanishing silently.

If uploads are failing on every submission rather than now and then, the cause is usually your connection to the storage provider rather than the provider being down.

Expired credentials, and how to restore them. When a token expires