Editor’s note: Ivan Lee is a speaker for ODSC East 2025 this May 13th to 15th in Boston. Be sure to check out his talk, “The 2025 Shift to Smaller Models: Why Specialized AI Will Win,” there!

In the evolving field of natural language processing (NLP), data labeling remains a critical step in training machine learning models. While the demand for high-quality, labeled data continues to grow, the last two years have prompted (pun intended) a notable shift from manual annotation to automated methods. Traditional manual labeling is often time-consuming, expensive, and prone to inconsistencies, making it difficult to scale projects efficiently.

To address these challenges, organizations are increasingly turning to LLM-automated annotation—leveraging Large Language Models (LLMs) to automate and streamline the labeling process. Datasaur’s LLM Labs provides a powerful solution, allowing users to experiment with different LLMs, optimize their configurations, and deploy them for efficient, high-quality data annotation. This guide explores how to use Datasaur’s LLM Labs to automate data labeling, experiment with multiple models, and utilize robo-labeling to achieve consensus between AI and human annotators.

Introduction to LLM-Automated Data Annotation

Traditional data annotation methods rely heavily on manual labeling, requiring teams of annotators to painstakingly tag data—a slow, expensive, and error-prone process. As machine learning applications grow more complex, the demand for high-quality, labeled data has skyrocketed, making fully manual annotation increasingly unsustainable.

With the advent of Large Language Models (LLMs), it’s now possible to automate 50-80% of the labeling process, reducing costs and time-to-insight. Datasaur’s LLM Labs allows teams to integrate LLMs into their annotation workflows, leveraging LLM-automated labeling to handle the bulk of the work while humans review and refine the results.

Keeping the Human in the Loop (HITL)

Instead of relying solely on automated AI labeling or fully manual annotation, the most effective approach is a Human-in-the-Loop (HITL) system: utilizing human labelers and reviewers while using LLMs for automated labeling. HITL ensures:

• Efficiency & Scalability: LLMs generate initial labels, significantly reducing the manual workload.
• Accuracy & Control: Human reviewers validate and correct AI-generated labels, ensuring quality and eliminating systematic errors.
• Consensus & Refinement: By comparing multiple LLM outputs and human corrections, organizations can achieve high-confidence, consensus-driven labels.

So how do we leverage LLM Assisted labeling with a HITL process to ensure both efficiency and quality? Let’s jump in!

A) How to Use LLMs to Automate Data Labeling with Datasaur

Datasaur’s LLM Sandbox allows users to experiment with and compare multiple foundational models before deploying them for assisted labeling. This provides a controlled environment where you can test models such as Claude, ChatGPT, Llama, and more to determine which best suits your annotation task.

Choose and Configure a Model

There are many LLMs on the market and every week we see new innovations. LLM Labs enables users to compare and contrast any LLM on the market. To get started with LLM-automated labeling, select a foundational model from OpenAI, AWS Bedrock, Microsoft Azure, HuggingFace, or other providers available in Datasaur’s LLM Labs. Each model has unique strengths—some may be better at understanding context, while others might be optimized for speed or cost.

Once you’ve chosen a model, configure it by providing clear user and system instructions, including examples of correctly labeled data. Research shows that the more examples an LLM has, the higher the quality of its outputs. This step ensures that the model generates structured and relevant labels for your dataset.

Evaluate Model Performance

Rather than relying on a single LLM, experiment with multiple models to compare their performance. Use prompt testing to see how different models label the same data. Consider key factors such as:

• Accuracy: Does the model correctly categorize or extract information?
• Consistency: Does it produce reliable results across different samples?
• Cost & Speed: How quickly does the model process requests, and what are the associated API costs?

Using Datasaur, you can leverage all these LLMs without needing to create individual accounts for each one, and Datasaur does not charge for this – you will have the same cost using Datasaur for all LLMs as you would creating individual accounts for each.

Deploy and Integrate the Best Model

Once you’ve identified the most effective LLM for your use case, deploy it for your NLP project. This model will now function as an automated annotator, assisting with data labeling while allowing human reviewers to validate and refine its outputs.

By using this LLM-automated labeling workflow, teams can automate up to 80% of their annotation tasks, saving time, reducing costs, and improving labeling accuracy and consistency—all while keeping human oversight where it’s most needed.

For detailed steps, refer to Datasaur’s LLM Sandbox documentation.

Now let’s go to the NLP workspace to connect our configured LLM to automatically label our data.

B) QA the Results

With our LLM applications configured and ready to label our data, we can upload our dataset. From there, we can connect our LLM to label our data. When the LLM returns labels, you will be able to truly analyze the performance of your application’s ability to label your data. For example, if we were running a NER labeling project and were attempting to execute the labels Organization and Date within our dataset – we can easily see how much of your dataset was covered, and which instances of dates and organizations were not captured. However, beyond seeing the LLM’s coverage, we can execute a number of strategies to ensure the quality of the output.

Deploy Multiple Configured LLMs

Each configured LLM can function as an independent labeler, allowing users to compare their outputs directly in Datasaur’s Reviewer Mode. Users can set a consensus threshold to easily find disagreements between different LLMs.

This has a few benefits. It enables you to create a multipass workflow for data labeling. This means each LLM application will independently apply labels to the same exact dataset. Meaning you are able to review the differences and agreements between the models. Datasaur is the only platform on the market with these multipass capabilities.

Consensus Building

By employing multiple models to label the same dataset, users can identify areas of agreement and disagreement. You can also apply a LLM application and a human labeler (in their own labeler mode) to the same dataset so you can compare model vs human as the Reviewer. Either way, this consensus-driven approach enhances the reliability of annotations and highlights instances that may require human intervention. Reviewers can evaluate the conflicts with consensus and correct the labels. They can also use this information to improve the instructions to the LLM application (going back to Step 1).

Utilizing Analytic Reports for Performance Evaluation:

Users can leverage the Inter-Annotator Agreement (IAA) table in Datasaur’s analytics to assess how well their configured LLM performs against human labelers. This provides a data-driven approach to selecting the most effective LLM for their annotation needs. You will clearly know how well your LLM application has historically performed against the QA process: how many labels have been rejected, accepted, labeled. These insights are critical to understanding the effectiveness of your model.

By utilizing multiple configured LLMs and assessing their performance through Datasaur’s Reviewer Mode and utilizing the platform’s QA reports, users can iteratively configure their models for optimal annotation accuracy: ensuring a balance between automation and human expertise.

Conclusion

Datasaur’s LLM Labs streamlines the data annotation landscape by integrating advanced LLMs into the workflow. By employing an array of different models for automatic labeling, teams can achieve efficient, accurate, and scalable annotation. This synergy between artificial intelligence and human expertise not only accelerates project timelines but also enhances the overall quality of NLP projects.

For a comprehensive guide on setting up and utilizing LLM Labs, refer to Datasaur’s official documentation.

About the Author