Cookies help us deliver our services. You can find more information in our Privacy Policy. Learn more




The 20BN-something-something Dataset V1


Attention: This is the outdated V1 of the dataset. V2 is available here.

Introduction

The 20BN-SOMETHING-SOMETHING dataset is a large collection of densely-labeled video clips that show humans performing pre-defined basic actions with everyday objects. The dataset was created by a large number of crowd workers. It allows machine learning models to develop fine-grained understanding of basic actions that occur in the physical world. It is available free of charge for academic research. Commercial licenses are available upon request.

A paper with supplementary material can be found here.

Poking a stack of cans so the stack collapses
Plugging cable into charger
Closing dishwasher

Data format

The video data is provided as one large TGZ archive, split into parts of 1 GB max. The total download size is 25.2 GB. The archive contains directories numbered from 1 to 108499. Each directory corresponds to one video and contains JPG images with height 100px and variable width. The JPG images were extracted from the orginal videos at 12 frames per seconds. The filenames of the JPGs start at 00001.jpg. The number of JPGs varies as the length of the original videos varies.

Terms of use

This dataset be used for academic research free of charge under the below license agreement. If you seek to use the data for commercial purposes please contact us.

Download Dataset

Please register or log in to download the dataset.


20BN-SOMETHING-SOMETHING-DATASET
Total number of videos
108,499
Training Set
86,017
Validation Set
11,522
Test Set (w/o labels)
10,960
Labels
174
986
Holding something
979
Turning something upside down
924
Turning the camera left while filming something
914
Stacking number of something
914
Turning the camera right while filming something
888
Opening something
885
Approaching something with your camera
877
Picking something up
873
Pushing something so that it almost falls off but doesn't
864
Folding something
863
Moving something away from the camera
858
Closing something
850
Moving away from something with your camera
845
Turning the camera downwards while filming something
841
Pushing something so that it slightly moves
839
Turning the camera upwards while filming something
838
Pretending to pick something up
838
Showing something to the camera
833
Moving something up
830
Plugging something into something
830
Unfolding something
828
Putting something onto something
827
Showing that something is empty
825
Pretending to put something on a surface
825
Taking something from somewhere
824
Putting something next to something
821
Moving something towards the camera
820
Showing a photo of something to the camera
815
Pushing something with something
808
Throwing something
802
Pushing something from left to right
801
Something falling like a feather or paper
801
Throwing something in the air and letting it fall
796
Throwing something against something
793
Lifting something with something on it
788
Taking one of many similar things on the table
785
Showing something behind something
781
Putting something into something
780
Tearing something just a little bit
779
Moving something away from something
778
Tearing something into two pieces
777
Holding something next to something
777
Pushing something from right to left
776
Putting something, something and something on the table
775
Moving something closer to something
775
Pretending to take something from somewhere
774
Pretending to put something next to something
773
Uncovering something
772
Pouring something into something
772
Putting something and something on the table
772
Something falling like a rock
769
Moving something down
769
Pulling something from right to left
767
Throwing something in the air and catching it
763
Tilting something with something on it until it falls off
762
Putting something in front of something
760
Pretending to turn something upside down
759
Putting something on a surface
757
Pretending to throw something
756
Covering something with something
756
Showing something on top of something
753
Squeezing something
752
Putting something similar to other things that are already on the table
751
Lifting up one end of something, then letting it drop down
749
Taking something out of something
747
Moving part of something
745
Pulling something from left to right
744
Lifting something up completely without letting it drop down
743
Attaching something to something
743
Holding something in front of something
743
Moving something and something closer to each other
743
Putting something behind something
742
Pushing something so that it falls off the table
735
Holding something over something
734
Pretending to open something without actually opening it
732
Removing something, revealing something behind
729
Hitting something with something
727
Moving something and something away from each other
727
Touching (without moving) part of something
724
Pretending to put something into something
724
Showing that something is inside something
721
Lifting something up completely, then letting it drop down
720
Pretending to take something out of something
709
Holding something behind something
707
Laying something on the table on its side, not upright
700
Poking something so it slightly moves
699
Pretending to close something without actually closing it
698
Putting something upright on the table
690
Dropping something in front of something
687
Dropping something behind something
685
Lifting up one end of something without letting it drop down
682
Rolling something on a flat surface
677
Throwing something onto a surface
671
Showing something next to something
668
Dropping something onto something
668
Stuffing something into something
662
Dropping something into something
662
Piling something up
660
Letting something roll along a flat surface
658
Twisting something
643
Spinning something that quickly stops spinning
636
Putting number of something onto something
634
Moving something across a surface without it falling down
634
Putting something underneath something
628
Plugging something into something but pulling it right out as you remove your hand
627
Dropping something next to something
606
Poking something so that it falls over
593
Spinning something so it continues spinning
588
Poking something so lightly that it doesn't or almost doesn't move
585
Wiping something off of something
582
Moving something across a surface until it falls down
580
Pretending to poke something
570
Putting something that cannot actually stand upright upright on the table, so it falls on its side
566
Pulling something out of something
565
Scooping something up with something
562
Pretending to be tearing something that is not tearable
543
Burying something in something
542
Tipping something over
533
Tilting something with something on it slightly so it doesn't fall down
528
Pretending to put something onto something
522
Bending something until it breaks
512
Letting something roll down a slanted surface
509
Trying to bend something unbendable so nothing happens
505
Bending something so that it deforms
503
Digging something out of something
502
Pretending to put something underneath something
497
Putting something on a flat surface without letting it roll
479
Putting something on the edge of something so it is not supported and falls down
471
Pretending to put something behind something
471
Spreading something onto something
466
Sprinkling something onto something
463
Something colliding with something and both come to a halt
462
Pushing something off of something
453
Putting something that can't roll onto a slanted surface, so it stays where it is
451
Lifting a surface with something on it until it starts sliding down
433
Pretending or failing to wipe something off of something
433
Trying but failing to attach something to something because it doesn't stick
427
Pulling something from behind of something
423
Pushing something so it spins
420
Pouring something onto something
416
Pulling two ends of something but nothing happens
413
Moving something and something so they pass each other
413
Pretending to sprinkle air onto something
405
Putting something that can't roll onto a slanted surface, so it slides down
395
Something colliding with something and both are being deflected
386
Pretending to squeeze something
367
Pulling something onto something
362
Putting something onto something else that cannot support it so it falls down
358
Lifting a surface with something on it but not enough for it to slide down
358
Pouring something out of something
346
Moving something and something so they collide with each other
341
Tipping something with something in it over, so something in it falls out
339
Letting something roll up a slanted surface, so it rolls back down
318
Pretending to scoop something up with something
311
Pretending to pour something out of something, but something is empty
294
Pulling two ends of something so that it gets stretched
290
Failing to put something into something because something does not fit
288
Pretending or trying and failing to twist something
282
Trying to pour something into something, but missing so it spills next to it
277
Something being deflected from something
273
Poking a stack of something so the stack collapses
267
Spilling something onto something
245
Pulling two ends of something so that it separates into two pieces
229
Pouring something into something until it overflows
220
Pretending to spread air onto something
219
Twisting (wringing) something wet until water comes out
217
Poking a hole into something soft
207
Spilling something next to something
206
Poking a stack of something without the stack collapsing
183
Putting something onto a slanted surface but it doesn't glide down
170
Pushing something onto something
141
Poking something so that it spins around
121
Spilling something behind something
77
Poking a hole into some substance

Leaderboard

If you have been successful in creating a classification model based on the training set and it performs well on the validation set, we encourage you to run your model on the test set (which is published without any class labels, as you might have noticed). Please prepare a .csv file with the video's id in the first column and your predicted class label (as a string matching the wording used in the training and validation sets). As a separator, please use a semicolon. You can then upload your .csv file here (user login required) to be ranked in the leaderboard and to benchmark your approach against that of other machine learners. We are looking forward to your submission.

Rank
Name
Approach
Score
1
Anonymous
4 months ago

Camera, Two-stream, ResNet-50

50.72%
2
Anonymous
4 months ago

TSM ResNet-50 8f, RGB+Flow

50.72%
3
Anonymous
12 days ago

GSM

50.42%
4
Anonymous
9 months ago

2stream 101

49.87%
5
Anonymous
9 months ago

rgb ensemble

49.85%
6
Anonymous
6 months ago

49.73%
7
sandy stream
about 1 month ago

ipcsn 5_crop G_sim

49.39%
8
Sandy Stream
about 1 month ago

ipcsn, 5_crop

49.25%
9
Anonymous
5 months ago

two stream 1+1 12F

48.98%
10
Anonymous
7 months ago

single RGB model only

48.93%
11
Over
9 months ago

single RGB model, I3D50 bachbone using high order blocks

48.18%
12
Lee
about 1 month ago

resnet50/two-stream

48.03%
13
Anonymous
11 days ago

GSM: trained on train split

47.3%
14
Anonymous
9 months ago

rgb ensemble

46.82%
15
Anonymous
9 months ago

46.67%
16
Lucas Lee
about 1 month ago

46.61%
17
Anonymous
7 months ago

two stream 1+1

46.12%
18
Bark
3 months ago

3+10

46.05%
19
any
6 months ago

any4

46.0%
20
ANY
6 months ago

ANY5

46.0%
21
Maker
10 months ago

two stream

45.92%
22
Anonymous
9 months ago

tscv

45.66%
23
Anonymous
7 months ago

test2

45.66%
24
any
6 months ago

any3

45.6%
25
Any
6 months ago

any1

45.57%
26
Anonymous
9 months ago

101rgb

45.46%
27
claire
3 months ago

en11

45.4%
28
Anonymous
7 months ago

two stream diff

45.38%
29
Phoenix Lee
about 1 month ago

META, 8 frame, revaluation (work in progress, the last submitted results are wrong.)

45.32%
30
SKY
8 months ago

PTnet: only RGB 43.39%

45.26%
31
Anonymous
almost 2 years ago

45.04%
32
Anonymous
over 1 year ago

Submission only for adding description for our previous submission.
The results is still the same as in 03/13/2018:

RGB only, Non-local ResNet-50 + GCN.
Model is pre-trained with:
https://github.com/facebookresearch/video-nonlocal-net

45.04%
33
R50_Test
8 months ago

ResNet-50 backbone, enhanced residual units, RGB only model.

45.01%
34
Anonymous
7 months ago

44.96%
35
Tao
10 months ago

two stream

44.9%
36
AnonymousLiu
3 months ago

16f-3crop-10clips

44.74%
37
Anonymous
9 months ago

Full

44.63%
38
Anonymous_hlwc
11 months ago

update

44.53%
39
ESTNET
22 days ago

16frame_twicesample_fullres

44.53%
40
Tony Maker
10 months ago

rgb

44.26%
41
MCG
10 months ago

44.12%
42
Anonymous
7 months ago

test1

44.01%
43
AnonymousLiu2
3 months ago

16f-1crop-1clip

43.89%
44
ECO
over 1 year ago

Validation results-(RGB): 46.4
Validation results-(RGB+Flow): 49.5
Test results-(RGB): 42.3
%
"ECO: Efficient Convolutional Network for Online Video Understanding"

https://github.com/mzolfaghari/ECO-efficient-video-understanding

43.88%
45
Debidatta Dwibedi
over 1 year ago

43.87%
46
Anonymous
11 months ago

43.8%
47
Elaine
9 months ago

RGB Only

43.74%
48
Anonymous
9 months ago

50rgb

43.69%
49
Anonymous
9 months ago

p2 2s

43.61%
50
any
6 months ago

any2

43.5%
51
Anonymous
5 months ago

Mutli-crop rgb 50 layer

43.49%
52
Anonymous
9 months ago

16f

43.06%
53
anonymous
9 months ago

42.95%
54
Run
9 months ago

32f

42.94%
55
Anonymous
7 months ago

only rgb

42.81%
56
Anonymous
9 months ago

8+16f

42.66%
57
Anonymous
9 months ago

24f

42.66%
58
Anonymous
about 1 year ago

TVB

42.58%
59
Anonymous
3 months ago

42.42%
60
Anonymous
7 months ago

42.31%
61
Anonymous
over 1 year ago

Bug fixes (Apr. 16 2018) - Previous Results: 41.96%

42.22%
62
Anonymous
almost 2 years ago

41.96%
63
Anonymous
about 1 year ago

41.93%
64
Dio Brando
9 months ago

the world

41.82%
65
ANY
6 months ago

A

41.81%
66
Any
2 months ago

i3dbs test

41.81%
67
Anonymous
almost 2 years ago

DRX3D

41.67%
68
Anonymous
about 1 year ago

41.66%
69
Anonymous
9 months ago

RGB 16 o

41.62%
70
Anonymous
9 months ago

16f test

41.62%
71
Anonymous
7 months ago

check overfitting

41.44%
72
ESTnet
25 days ago

41.23%
73
Anonymous
7 months ago

RGB-only

40.75%
74
Anonymous
over 1 year ago

2stream TRN

40.71%
75
Anonymous
7 months ago

RGB-only

40.66%
76
Anonymous
almost 2 years ago

flowTN

40.56%
77
Li Yan
about 1 month ago

40.55%
78
Yan Li
about 1 month ago

META (work in progress)

40.55%
79
Anonymous
over 1 year ago

Previous BSL: 39%

40.5%
80
PT M
10 months ago

two stream

40.5%
81
Anonymous
9 months ago

40.35%
82
Yana Hasson
almost 2 years ago

40.15%
83
Anonymous
about 1 year ago

39.84%
84
Steven
about 1 year ago

39.16%
85
Anonymous
over 1 year ago

3D CNN Architecture

39.15%
86
Anonymous
9 months ago

38.98%
87
Anonymous
9 months ago

resnet50 rgb 8

37.84%
88
Anonymous
about 1 year ago

37.81%
89
Anonymous
almost 2 years ago

Motion Feature Network (MFNet)

37.48%
90
iAnonymous
9 months ago

3D-Resnet18, 32 frames (val: 0.42)

37.18%
91
Anoymous
4 months ago

36.5%
92
Anonymous
almost 2 years ago

36.13%
93
Anonymous
over 1 year ago

R50D

35.97%
94
Donald Welling
11 months ago

zju

35.65%
95
Donald Dod
11 months ago

35.63%
96
HOME FREE
11 months ago

2

35.46%
97
Anonymous
almost 2 years ago

DIN

34.11%
98
Farzaneh Mahdisoltani
over 1 year ago

Two-channel M(256-256)
https://arxiv.org/pdf/1804.09235.pdf

33.3%
99
TZ M
10 months ago

rgb

33.06%
100
Anonymous
10 months ago

A lightweight method
rgb

32.06%
101
Anonymous
10 months ago

final test
label+string

32.03%
102
Anonymous
almost 2 years ago

R50 2_1D coco TH

31.9%
103
Eren Gölge
about 2 years ago

Besnet

31.66%
104
Noname
almost 2 years ago

TwistStream

31.22%
105
R34_coco_1
almost 2 years ago

30.62%
106
cheng wei
2 months ago

Zhou, Bolei, et al. "Temporal relational reasoning in videos." Proceedings of the European Conference on Computer Vision (ECCV). 2018.

30.49%
107
Guillaume Berger
about 2 years ago

30.48%
108
Anonymous
almost 2 years ago

29.59%
109
Aengus Tran
almost 2 years ago

29.18%
110
Dang Dinh Ang Tran
almost 2 years ago

29.18%
111
小虎 黄
about 1 month ago

TSM segment8 crop1

28.86%
112
huang xiaohu
about 1 month ago

seg8_crop1_epoch25

28.86%
113
Raghav Goyal
about 2 years ago

27.23%
114
Harrison.AI
over 2 years ago

26.38%
115
Valentin (esc) Haenel
about 2 years ago

https://github.com/esc/smth-smth-baseline/tree/train_004

smth-smth-baseline with dropout

24.55%
116
Melinda
almost 2 years ago

Two Stream LSTM

24.14%
117
Anonymous
almost 2 years ago

Motion Feature Network (MFNet)

22.4%
118
Xiu-Shen Wei
about 2 years ago

22.13%
119
Peng
about 2 years ago

21.93%
120
Can
over 1 year ago

21.03%
121
Peter_CV
about 2 years ago

19.68%
122
Anonymous(:
over 1 year ago

17.28%
123
chenzhaomin
over 1 year ago

tsn

17.28%
124
Anonymous
11 months ago

2.03%
125
Anonymous
11 months ago

testing

2.03%
126
Anonymous
over 1 year ago

1.67%
127
Eduardo Monauni
over 2 years ago

1.43%
128
Aaryahi Roshan
over 2 years ago

1.39%
129
Anonymous
over 1 year ago

0.85%
130
Anonymous
almost 2 years ago

0.0%
131
JZ
9 months ago

0.0%



Feedback