Blog · Data

Every frame accounted for: inside Paddy's first open dataset

Today we're releasing our first open dataset: 25 episodes of bimanual cable sorting, recorded by teleoperation on The Harvester and published on Hugging Face under CC BY 4.0.

It's small on purpose. It's a spec sheet you can download: every episode, every camera, every sensor stream, and every number we measured about how well the rig recorded it, including the ones that don't flatter us.

If you buy or train on teleop data, you've probably received a dataset that looked great in the demo video and fell apart in the dataloader. Missing frames, unlabeled failures, cameras that drift out of sync, force readings nobody checked. If there are problems, we'd rather you find them in a sample than in a contract.

Explore the sample on Hugging Face →

Overhead · D435
Scene · D405
Left wrist · D405
Right wrist · D405
Episode 15, all four cameras on one clock. Play any episode in the episode explorer →

The task and the rig

The task is simple to say and annoying to do: pick a cable out of one bin, hand it between arms and sort it by color. Cables are floppy, they tangle, and they don't sit where you left them. That's exactly why they're good training data.

Every episode is labeled with three timestamped subtasks:

SubtaskEpisodesSucceededMedian time
Pick up a cable with the left arm, using the right wrist camera for a better view252420.6 s
Hand the cable from left arm to right arm24238.8 s
Place green cables in the left bin, black in the right23226.4 s

Step 1PickLeft arm picks, right wrist camera gives the view. Median 20.6 s.

Step 2PassLeft arm hands the cable to the right. Median 8.8 s.

Step 3PlaceGreen left, black right. Median 6.4 s.

Each step from episode 15, shown in all four views.

We recorded across two scenes, our normal office setup and a second bin-wall layout in low light, over three sessions in one five-hour window on Nov 13.

The rig is The Harvester's default build: two UFactory xArm 7 arms (7-DOF each), driven from a Meta Quest. It carries four Intel RealSense cameras (a D435 overhead, a D405 on each wrist and a D405 scene camera) plus a 6-axis force-torque sensor on the right wrist. Joints and force-torque log at 100 Hz, the arms servo at 200 Hz, and cameras record at 30 fps.

What's in the box

The same episodes ship as two datasets, both in LeRobot v3.0 format so they load with tooling you already use.

cable_sorting_samplecable_sorting_sample_full
Episodes25 (22 success, 3 failure)4 (episodes 10, 13, 14, 17)
Frames / duration27,953 / 15.5 min4,143 / 2.3 min
CamerasOverhead + both wrists, color, 1280×720All four, overhead at 1920×1080
DepthNoAligned and raw 16-bit depth from every camera, plus the overhead IR stereo pair
CompressionCompressedLossless
Features3953
Size3.24 GB19.4 GB
Full release: overhead color with aligned depth, episode 17 (0.3–1.3 m).

The full release runs about 8.4 GB per minute of recording. That's the honest cost of capturing everything losslessly, and why future datasets can be trimmed to what a customer actually trains on.

The Hugging Face Space has three views: an overview of the task, scenes and failures; a full spec covering every column with its unit and frame, camera calibration and file layout; and an episode explorer that plays all four cameras in sync with the subtask lanes and the gripper, speed, force-torque and joint charts.

Failures are data

Three of the 25 episodes failed, and we kept all three. Each one is labeled with the subtask where it broke and a plain note on what happened:

EpisodeFailed atWhat happened
3PickLeft arm couldn't reach the cable
14HandoverDropped the cable mid-pass
18PlaceBlack cable landed partly hanging out the front of the bin

Episode 3 · PickOut of reachLeft arm couldn't reach the cable.

Episode 14 · HandoverDroppedThe cable slipped mid-pass.

Episode 18 · PlaceHalf in the binBlack cable landed hanging out the front.

All three failures, kept and labeled.

That's a 12% failure rate by episode, about 10.5% of recorded time. Teleop datasets often ship successes only. We think that's a mistake: a policy that has never seen a dropped cable has no idea what to do after one. Labeled failures let you choose to filter them, weight them, or train recovery on them. Throwing them away should be your call, not ours.

One more thing the data shows: episodes in the third session averaged about 33 seconds, versus about 41 in the first two, even though that session was in low light. Most likely that's the operator getting faster at the task, and you can see the curve in the per-episode record.

The numbers we measured

Every quality figure in this post is computed from the datasets themselves, not from a spec sheet. These are the ones we'd want to see before buying anyone's data:

MeasureResult
Camera sync across views9.5 ms median per episode (best 5.5, worst 14.7)
Held (repeated) camera frames196 of 83,859, or 0.23%
Force-torque readings valid99.94%
Peak contact force9.3 N median per episode, 21.9 N max
Command-to-motion lagAbout 115 ms (range 102 to 150 ms)
Arm tracking error, p99 after lag5.6 mm

Two of these deserve context. The 115 ms lag is the delay between the commanded target and the arm's measured motion, most of it in the arm's own controller. It's consistent, and we publish it so you can align actions to observations rather than guess.

The actions also carry a deadman flag. The operator only engages an arm while actively moving it, so the left arm is engaged about 61% of the time and the right about 45%: in this task the right arm often waits while the left one picks. Rows without a live command are flagged in paddy.action_valid, so you know exactly which actions are real commands.

Known limits

The dataset card lists these openly, and so do we:

  • Grippers out of the overhead frame. In episodes 1 and 2 both grippers were out of the overhead camera's view for several seconds, up to 11.2 s in episode 2; across the sample that's 2.4% of rows. The overhead video itself is complete, and the scene camera, recorded but not in this sample, saw the grippers throughout.
  • Joint effort isn't validated. The columns are there, but we haven't checked them against ground truth, so treat them as unverified.
  • Two depth images are missing out of 16,572 in the full release.
  • Episode 14 is offset in the full release. It starts 79 rows (2.6 s) later there than in the sample.
  • People may appear in the background of some camera views.
  • One operator, one task, 15.5 minutes. This is a sample of how we record, not a training set on its own.

None of these are dealbreakers. All of them are things you'd want to know before you train.

What's next

This sample is the standard every Paddy dataset ships to: labeled subtasks, labeled failures, synchronized multi-camera video, force-torque, and a measured quality report with its limits stated. The next step is scale, on more tasks, more scenes and more operators.

If you're training manipulation policies or post-training a model for deployment, we'd like to hear what you need. We collect to spec on our own rigs, and we sell the rigs too.