Author: Paula Ramos (Senior DevRel and Applied AI Research Advocate at Voxel51)

Getting Started with FiftyOne: Actionable Insights for Your Data

As the second blog in my series on FiftyOne, this post aims to guide you on an exciting adventure of using this open-source toolkit to bridge the gap between your data, models, and deployments. Drawing from my experience, I’ll demonstrate how FiftyOne can unlock real value in your dataset, whether you’re working with 2D images, 3D point clouds, or any other type of visual data.

From my early work with coffee datasets in Colombia to modern machine learning applications, I’ve faced many challenges with dataset curation, annotation, and processing. In this blog, I’ll walk you through how FiftyOne can transform your workflow, expedite the AI production process, and help you extract actionable insights from your data.

The Problem: Time-Intensive AI Workflows and Unseen Complexity

When I began working in AI nearly two decades ago, it wasn’t called AI, it was machine learning, image processing, robotics, and smart devices. The lines between these fields have since blurred, but one constant remains: developing AI solutions has always been time-intensive.

A significant challenge in these workflows is that we often don’t truly know our data. We collect datasets without fully understanding the patterns, biases, or noise embedded within them. Even less do we comprehend how this complexity influences the AI equation and how acquisition conditions, annotations, or dataset distribution can profoundly affect the model’s performance.

In my early days working with coffee growers in Colombia, I spent years cycling through data collection, annotation, and model evaluation. Fast forward to today, FiftyOne has made it possible to reduce these cycles from months to weeks or even days for data management and model evaluation, allowing me to connect data and models. Who doesn’t want to expedite the AI production process and improve the performance of your models?

A Coffee Lover’s Dataset

o make this blog actionable, I’ll use one of my datasets (see the link at the end of the blog), a collection of coffee branch images captured with a cellphone camera. The goal? Count coffee fruits in a branch using a semantic segmentation task.

Here’s a quick overview of the dataset:

• Control conditions: No
• Capture device: Cell phone camera
• Format: Video and video-frames
• Curation: Manually curated (initially)
• Computer Vision Task: Semantic Segmentation (SS)
• Annotations: I annotated four maturation stages (immature, semimature, mature, and overmature).

Like many of you, I initially spent months manually curating the data, finding undesired images, re-annotating frames, and training

Why FiftyOne?

FiftyOne is a Python-based API (SDK) and GUI (APP) that allows you to explore, curate, and analyze datasets with actionable commands. Its features make it indispensable. For more information, see the user guide. I will show you some FiftyOne features, and you can replicate this on your side using my notebook.

1. Shuffle Your Dataset

How often have you struggled to make sense of your dataset? Whether it’s navigating folder structures or parsing JSON files, it’s a headache. FiftyOne makes this a thing of the past with its visualization options.

import fiftyone as fo

from fiftyone.utils.coco import COCODetectionDatasetImporter

dataset = fo.Dataset.from_dir(
   dataset_type=fo.types.COCODetectionDataset,
   dataset_dir="./colombian_coffee",
   data_path="images/default",
   labels_path="annotations/instances_default.json",
   label_types="segmentations",
   label_field="categories",
   name="coffee",
   include_id=True,
   overwrite=True
)

view = dataset.shuffle()

Once installed, you can quickly load your dataset, shuffle it, and view random samples with annotations (or simple raw data) in the APP.

Understand Your Dataset with fiftyone brain

One of the most transformative features of FiftyOne is its ability to help you understand your dataset. It’s the secret sauce for improving AI applications, from identifying potential biases to visualizing experimental relationships.

In the past, I relied on PCA plots to detect patterns in my data, or I needed to create my own vector representation. FiftyOne furthers this with a package called fiftyone. brain that provides a collection of machine learning techniques to help you gain insight into your datasets and models and curate your data with measurable techniques.

Discover Uniqueness

Finding unique cases in your dataset, like outliers or bad actors, has always been challenging. FiftyOne Brain allows you to calculate an index of uniqueness to identify these anomalies and adjust your data.

Example Use Case:
While working on coffee fruit detection, I noticed specific images were captured under poor focus or leaves in front of the lens, causing significant bias. FiftyOne enabled me to isolate and review these cases for potential elimination and curate the dataset properly.

import fiftyone.brain as fob

fob.compute_uniqueness(dataset)
# Explore most unique samples
session.view = dataset.sort_by("uniqueness", reverse=True)
# Explore the least unique samples
session.view = dataset.sort_by("uniqueness")

Let’s use the compute_uniqueness() function to index the samples in our dataset according to their visual uniqueness:

Each sample is populated with a scalar-valued uniqueness field that ranks its uniqueness (a higher value means more unique). this value is normalized to [0, 1], with the most unique sample in the collection having a uniqueness value of 1.

Leverage Similarity Search

Once you’ve identified problematic samples, FiftyOne’s similarity search allows you to find more instances like them. Whether you’re detecting duplicate samples or filtering undesired data, this feature is a game-changer for dataset curation.

# Index images by similarity
fob.compute_similarity(
    dataset,
    model="clip-vit-base32-torch",
    brain_key="img_sim",
)

Once you’ve indexed a dataset by similarity, you can use the sort_by_similarity() view stage to programmatically sort your dataset by similarity to any image(s) or object patch(es) of your choice in your dataset. In addition, the App provides a convenient point-and-click interface for sorting by similarity with respect to an index on a dataset.

Additional Help: Text similarity

You can prompt your dataset and use text prompts to find images within your dataset. You can create a similarity index powered by the CLIP model and use natural language to find images. The FiftyOne Brain provides a compute_similarity() method that you can use to index the images or object patches in a dataset by similarity.

Embedding Visualizations

One of FiftyOne’s most exciting features is its ability to visualize embeddings. This lets you understand the relationships between data points, clusters, and categories in your dataset. By projecting embeddings into 2D or 3D space, you can uncover new insights and refine your data.

# Image embeddings
fob.compute_visualization(dataset, brain_key="img_viz")

Challenges in Computer Vision and ML

No matter how big our actual advancements in computer vision and machine learning are, challenges still persist:

• Data Quality: Poorly annotated or biased datasets hinder model performance. Bad data causes 30% of model errors.
• Time to Production: Lengthy cycles for model development and deployment slow innovation. ML teams wrangle data 65% of the time.
• Scalability: Manual curation doesn’t scale as datasets grow in size and complexity. 85% of AI projects fail in production.

FiftyOne addresses these challenges by providing a data-centric AI approach, focusing on improving data quality and making workflows scalable.

Additional resources:

1. First blog of this series: “Journey into Visual AI: Exploring FiftyOne Together — Part I Introduction.”
2. Coffee Dataset: https://huggingface.co/datasets/pjramg/colombian_coffee
3. FiftyOne Repos: FiftyOne, FiftyOne Examples, FiftyOne Documentation.
4. Notebook for running these code snippets.
5. Are you a beginner and want to start from scratch? Take a look at the readme file of the FiftyOne-Example repo.

Just wrapping up! 😀

After our hands were all dirty on the basics of FiftyOne, the next step was more challenging for all of us. In my next post, I will give you some ideas about advanced features like Integrations, Plugins, Dataset /Model Zoo, Video, and 3D.

I would love to hear about your experiences! Please Share Your Thoughts, Ask Questions, and Provide Testimonials. Your insights might help others in our next posts.

Together, we can learn more effectively and contribute to improving FiftyOne!

Stay tuned for the next post, in which we’ll explore FiftyOne’s essential features, manipulate the dataset, and evaluate the model.

Let’s make this journey with FiftyOne a collaborative and enriching experience. Happy coding!

Stay Connected:

• Follow me on Medium: https://medium.com/@paularamos_phd
• Follow Me on LinkedIn: https://www.linkedin.com/in/paula-ramos-phd/
• Join the Conversation: Discord Fiftyone-community

What is next?

I’m excited to share more about my journey at Voxel51! 🚀 If you’d like to follow along as I explore the world of AI and grow professionally, feel free to connect or follow me on LinkedIn. Let’s inspire each other to embrace change and reach new heights!

You can find me at some Voxel51 events (https://voxel51.com/computer-vision-events/), or if you want to join this amazing team, it’s worth taking a look at this page: https://voxel51.com/jobs/