Psych-DS metadata

With metadata on, DataPipe writes a dataset_description.json file alongside your data. It describes the dataset and every variable in it, in the Psych-DS format, and DataPipe rewrites it after each session. Turn it on from your experiment dashboard if you plan to share or publish your data. It makes the dataset much easier for someone else to read and reuse.

Turning it on also changes where your files land. Instead of one file per session at the top of your Drive folder, Dataverse dataset, or Zenodo deposition, each session produces:

• data/raw/<your filename>

  The submission exactly as your experiment sent it, byte for byte. This is the file that matters. Everything else is derived from it.

• data/<name>_data.csv

  The session's main data table. A CSV submission keeps its original bytes here. A JSON submission is written out as a table.

• data/<name>_measure-<column>_data.csv

  One extra table for each column that held nested objects or arrays (survey responses, mouse-tracking samples, and the like), so the main table stays flat.

• dataset_description.json

  At the top level. The description of the dataset and every variable in it, rewritten after each session.

• .psychds-ignore

  At the top level. Tells Psych-DS validators to skip data/raw/, which holds your originals rather than Psych-DS tables.

With metadata off, none of this happens. DataPipe stores each submission at the top level under the filename you sent, and creates no other files.

That difference in where files land is why the setting locks after the first submission. Changing it partway through would leave the sessions you already collected in one place and every later session in another, and DataPipe's duplicate detection would no longer recognize the earlier files.

You won't get subfolders inside your dataset, whatever filenames you send. A Psych-DS dataset keeps a flat data/ folder, so a filename with a slash in it is flattened before the path is built, and Zenodo flattens the data/raw/ path itself because it has no folders at all.

Exactly how a name is flattened, and why a short code is added to it. What your files are named

For each variable, DataPipe records its data type and, when it can, a human-readable description taken from the documentation of the jsPsych plugin that produced it.

“When it can” is the important part. DataPipe doesn't ship with a table of descriptions. When a session arrives, it fetches the source of each jsPsych plugin from unpkg.com and reads the descriptions out of that source's documentation comments. So descriptions exist only for plugins and extensions published to npm under the official jsPsych names.

Two things follow from that, and neither one announces itself:

• Custom, private, renamed, and unpublished plugins have no source to fetch. DataPipe describes their variables as "unknown" but stores your data normally. The variables still appear in the metadata. Only the prose descriptions are missing.
• An unpkg outage does the same thing, for that session only. DataPipe makes the fetch while it handles the submission. If the fetch fails, that session's new variables come out as "unknown". DataPipe doesn't go back later to fill them in, but a variable that was already described keeps its description.

dataset_description.json is an ordinary file in your own storage, so you can write the missing descriptions in yourself. Do it after collection ends. DataPipe rewrites that file from its own copy of the metadata after every session, so the next submission would overwrite any edit you made mid-study.

DataPipe also combines information across sessions, such as the numeric ranges and categorical values it has seen. The description file is never rebuilt from just the newest session. Instead, DataPipe merges each session into what's already there:

• new values seen for a categorical variable are added to its list of levels;
• a numeric variable's minimum and maximum widen to cover the new session;
• variables that appear for the first time are added, so a condition that only some participants see is still described;
• variables already described are kept, and nothing is dropped just because a later session didn't contain it.

Four jsPsych bookkeeping variables (trial_type, trial_index, time_elapsed, and internal_node_id) are written once and then left alone, since their meaning doesn't change from session to session.

Metadata is a description of your data, and DataPipe treats it that way. The derived tables and .psychds-ignore are uploaded after your raw file has landed. If one of them fails, DataPipe queues and retries it on its own, and the submission still succeeds. The metadataMessage field on every data response reports what happened. It never decides whether a submission is accepted.

There is one case where metadata affects the response. If DataPipe can't produce metadata from a submission at all, that request comes back as a 400 with METADATA_ERROR. The usual cause is a submission that parses as JSON but isn't an array of trials, because metadata needs the trial array that jsPsych produces. Anything that doesn't parse as JSON is treated as CSV instead.

Even then, the data itself isn't lost. DataPipe keeps the copy it took when the submission arrived, and a scheduled sweep picks that copy up once it's more than fifteen minutes old and sends it to your storage provider. A session recovered this way arrives as the raw file only, with no derived tables, so fix the shape of your data rather than relying on the sweep.

What DataPipe does with a submission it has accepted but not yet delivered. When an upload fails