J. Quesada, M. Prabhushankar, and G. AlRegib
Label uncertainty is known to be detrimental to machine learning development, particularly in the seismic interpretation domain. Training with single low-quality samples can lead to noise overfitting, and using multi-label aggregation strategies has been shown to yield suboptimal results in domains where the root cause of the disagreements is unknown. In this context, two potential axis from which seismic uncertainty can be studied are the degrees of expertise and confidence that annotators bring to the labeling process. We present here a seismic fault segmentation dataset compiled across 3 different levels of confidence and expertise. The dataset is comprised of images from the Netherlands F3 block, each of which has been annotated for seismic faults by a total of 38 annotators using 3 confidence categories. We collected these labels using the Amazon Mechanical Turk (MTurk) crowdsourcing platform, and we provide an analysis of several statistics of the dataset across these axes.
We generated labels for a total of 400 images taken from the Netherlands F3 block. Each image was labeled by 38 annotators: 29 novices (N), 8 practitioners (P) and 1 expert (E), leading to over 15000 image annotations. The acquired data from each annotatorincludes the detected fault regions, the associated confidence level, and the time taken for the labeling of each image. We also designated 40 quality assessment (QA) images within the dataset to evaluate annotator self-consistency by showing each of these images 3 times at random points in the labeling process.
