Filenames, archives and your storage

Every file DataPipe stores must have a name no other submission has used. That's what stops a participant who submits twice, or a duplicate request from a flaky connection, from overwriting data you've already collected.

DataPipe does the checking, not your storage provider. Most providers will happily accept the same name twice, each in its own way, so DataPipe keeps its own filename record, a list of the names an experiment has used, and checks that first.

The filename record doesn't contain your actual filenames. Each name is hashed with a secret unique to the experiment, and only the hash is stored, so DataPipe doesn't hold a list of the names your participants' files were given. Entries last 90 days from the submission that created them.

What your experiment sees:

• The name is already taken: 400 FILE_EXISTS. Nothing is stored, and the participant's submission is rejected. Give each submission a name you know is unique, such as one built from a random ID.
• The record is being rebuilt right now: 202. Another request is already rebuilding it, which takes at most 60 seconds. The submission is queued and lands shortly after.
• The record can't be rebuilt: 202. DataPipe couldn't list your Drive folder, Dataverse dataset, or Zenodo deposition, usually because it was deleted or access was revoked. Rather than risk overwriting real data, it queues the submission and retries.

A name reserved by a submission that never completed is released fifteen minutes after that submission arrived, so an interrupted request doesn't block the name forever.

DataPipe keeps its own filename record so a participant who submits twice can't overwrite existing data. Files that appear without DataPipe writing them aren't in that record, so the next submission that happens to use the same name may be quietly renamed, or may overwrite what you added.

What that looks like depends on the provider, and none of them gives you a clean error:

Files you added by hand do become visible to DataPipe eventually. If the filename record ever has to be rebuilt from your storage, it reads whatever is there and treats those names as taken. You can't trigger that yourself, though, so it isn't a fix. It only means a hand-added file is invisible for a while rather than forever.

Usually the name your experiment sent. Two rules can change it.

Slashes in a filename don't create folders. When Psych-DS metadata (the optional description DataPipe writes alongside your data) is on, a name like condition-A/abc.json is flattened with hyphens before the path is built, so it's stored at data/raw/condition-A-abc~a145753b.json. The prefix is kept rather than dropped, so two submissions that share a name after the last slash still don't collide. The short code after ~ is derived from the name you sent and stays the same if you resubmit. It keeps condition-A/abc.json distinct from a file literally named condition-A-abc.json. Names without a slash are stored exactly as sent and get no code.

Zenodo can't store a slash at all. It holds every file under a flat name, with no folders, so every remaining / becomes an underscore: data/raw/subject-1.json is stored as data_raw_subject-1.json. If you open a metadata-enabled Zenodo deposition mid-study and see a flat list of underscore names instead of a data/raw/ tree, this is why. Nothing has gone wrong. The real Psych-DS folder structure exists inside the archives described below, where DataPipe controls the paths.

On Google Drive, slashes do create real folders when metadata is off: condition-A/abc.json lands in a condition-A folder. Two submissions that share a name after the last slash but sit in different folders are different files, and both are kept.

What the metadata files contain, and how to turn them on. Psych-DS metadata

If you open a Zenodo deposition partway through a study and find files named datapipe-batch-0001.zip, nothing has gone wrong and nothing has been lost. Your sessions are inside them.

Zenodo allows 100 files per record, and it's the only provider with a file-count ceiling DataPipe has to work around. A study that writes several files per session would hit that ceiling long before it finished, so once a record reaches 80 of its 100 files, DataPipe merges older sessions into a zip and removes the originals. The five most recent sessions are left loose so you can still open and spot-check recent data in Zenodo's own interface. Each archive holds at most 95 files or 150 MB, whichever comes first.

Two files are never archived: dataset_description.json, which describes your dataset and its variables and should stay visible on the record, and .psychds-ignore, which is rewritten on every submission anyway.

Nothing is deleted until the archive is verified. DataPipe uploads the zip, compares the checksum Zenodo reports back against the one it computed, records the names now held inside the archive, and only then deletes the originals. If the checksum can't be verified, the originals stay where they are.

There's no schedule for this. A merge is triggered by your record growing, not by a timer, so it runs the moment it's needed. While a merge is running, submissions are queued rather than rejected. Your participants still get a success response, and the queued files land about a minute later.

The archives are ordinary zip files. Unzip one and you get the folder tree back, complete with the data/raw/ paths that Zenodo itself can't represent. Zenodo also previews zip contents on the record page, so you can see what's in an archive without downloading it.

If a Zenodo record fills up completely, DataPipe needs one free slot to upload the archive that would make room. It gets that slot by temporarily removing .psychds-ignore, a file whose contents never change, and writes it back afterwards. An experiment with metadata off has no such file to give up, so archiving stops and needs your help: remove one file from the record yourself, and it will proceed on the next submission. Files you add by hand count toward the limit, which is one more reason not to edit the record during collection.

At the end of collection, every batch is merged into one final archive. Finishing a study