Saving as you go

Send each trial as it happens, so a participant who drops out partway through still leaves data behind.

Why save as you go

Without saving as you go, DataPipe sees a session's data exactly once, when the experiment finishes. If a participant closes the tab, loses their connection, or their browser crashes at trial 199 of 200, all 199 trials are lost. DataPipe never saw any of them. On an online panel, that happens often.

With it on, each trial is sent to DataPipe as it's produced. A participant who finishes is stored exactly as before. A participant who doesn't leaves behind a partial file with everything they did up to the point they stopped.

Turning it on

In jsPsych, the @jspsych/extension-pipe extension does this by default. Registering it, as shown below, is the whole setup. To turn streaming off and submit once at the end instead, add stream: false to its params.

Plain JavaScript can stream too, with datapipe-client's createSession. Unlike the extension, the library doesn't stream by default. Your experiment opts in by starting a session. The Save as you go tab under JavaScript in the panel below has the code.

Load the extension and register it. That is the whole integration — there is no save trial to add.

<script src="https://unpkg.com/@jspsych/extension-pipe"></script>
const jsPsych = initJsPsych({
  extensions: [
    {
      type: jsPsychExtensionPipe,
      params: {
        experiment_id: "YOUR_EXPERIMENT_ID",
        filename: () => `${subject_id}.csv`
      }
    }
  ]
});

const subject_id = jsPsych.randomization.randomID(10);

const timeline = [];
// ...add your trials to the timeline...

jsPsych.run(timeline);

Each trial is sent as it happens, so a participant who closes the tab partway through does not take all of their data with them: their completed trials arrive as a separate file ending in .partial.json, and do not count toward your session limit. A participant who finishes produces one ordinary file.

Add format: "json" to save JSON instead of CSV. Add stream: false to send only at the end.

One thing to get right in plain JavaScript: call flush() before you read sessionId. A session starts in the background, and until it has, the ID is an empty string. If you submit without it, DataPipe can't match your file to the staged copy, so it recovers that copy separately and you end up with a stray .partial.json next to a complete file.

The rest of the code, for jsPsych and plain JavaScript, and what each response means. Sending data from your experiment

What to expect

Three things to know before you rely on it:

  • A completed session is unchanged. Whatever your experiment submits at the end, your whole dataset in your chosen format under the filename you gave it, is the file that lands in your storage. What DataPipe held during the session is deleted as soon as that submission lands.
  • An abandoned session becomes a second kind of file. DataPipe assembles the trials it received and stores them as <your filename>-<id>.partial.json. It's JSON even if your experiment submits CSV, because it's rebuilt from individual trials rather than from the string your experiment would have sent. The short ID keeps two participants who happened to use the same filename from colliding. Plan for these files in your analysis, and treat a partial file as a participant who didn't finish. Partial sessions don't count toward your session limit, and they skip your validation rules, since those run on completed submissions.
  • It can't break your experiment. If a session can't be started, because the experiment is switched off or the participant is offline, the experiment runs and submits exactly as it would without it. The same goes for every individual trial write.

While a study is running, your experiment's dashboard shows how many participants are partway through and how long each has been going, updating as they start, finish, or lose their connection. A participant whose connection drops is shown as Connection lost — may resume for 10 minutes, then Stopped — being recovered once DataPipe starts turning what they did into a partial file.

What it means for privacy

Saving as you go changes what happens to a participant's data in ways your consent form and your IRB may care about. Four things are different from a study that submits once at the end:

  • Data leaves the browser during the session. Each trial is written straight from the participant's browser to a private database DataPipe operates, over a connection separate from the one that carries the final submission.
  • Quitting no longer discards the data. A participant who closes the tab partway through leaves a partial file in your storage. DataPipe can't tell a withdrawal from a dropped connection. If your consent process treats closing the tab as withdrawal, say so in your protocol and plan to delete partial files, or turn saving as you go off.
  • Staged trials aren't encrypted by DataPipe. The browser that writes them has no key. They're protected by Google's platform encryption at rest and by access rules that let no browser read them, and they're deleted the moment the participant's final submission arrives, or as soon as an abandoned session has been turned into a partial file.
  • The filename you pass at session start is stored with the session until the session ends, so keep participant identifiers out of it.

A participant who finishes is unaffected. Their final submission is handled exactly as it would be without saving as you go.

The same differences in the privacy page's terms, with a sentence you can adapt for an IRB protocol. If your experiment saves as it goes

Limits

Saving as you go enforces the limits below on every request. None of them can be changed, and hitting one never breaks your experiment. Streaming carries on, and a completed submission is unaffected. What a limit costs is the partial-file safety net for a participant who never finishes, not the data your experiment collects.

  • 16 KiB per trial. The database refuses a trial larger than that. The write for that one trial fails, and streaming continues with the next one. A completed session still sends your whole dataset in its final submission, so the trial is only missing from the partial file DataPipe would recover if the participant never finished.
  • 1,000 trials per session. The 1,001st trial and every one after it are refused the same way an oversized trial is. Again, only the partial-file safety net is affected. The final submission isn't built from staged trials.
  • 10 minutes to reconnect, 24 hours to finish. If a participant's connection drops and DataPipe sees no reconnect and no further trial from them for 10 minutes, it treats the session as abandoned and turns it into a partial file the next time the sweep runs. Reconnecting, or getting even one more trial through, within that window keeps the session going as if nothing happened. Separately, every session expires 24 hours after it started and is recovered the same way, whether or not a disconnect was ever recorded.
  • 20 disconnects and 20 reconnects per session. After a participant's connection has dropped and recovered 20 times, further drops aren't recorded. The 10-minute clock then runs from the last recorded drop rather than the most recent real one. The session is still recovered eventually, at the 24-hour expiry if nothing else, but the fast path may miss it.
  • 500 sessions open per experiment at once. A participant who asks for a session while 500 are already open for your experiment doesn't get one. That's the same response as when incremental upload is switched off, and their experiment runs and submits exactly as it would without it. Nothing about their data is different.
  • 24 MiB per recovered file, 200-character filenames. A recovered partial file stops growing at 24 MiB. Trials beyond that point are left out of the file DataPipe assembles. The filename you give when starting a session is capped at 200 characters and is quietly shortened past that when it's used to name a recovered file. It never affects the filename you submit on a clean completion.

The endpoint behind this, for anyone writing their own client. Start an incremental session