RLHF and alternatives: Overview

Introduction

This is a series of blog posts related to alternatives to Reinforcement Learning by Human Feedback, created as a joint effort between Argilla and MantisNLP teams. Please make sure you have gone through the first, second, third, fourth, fifth, sixth, seventh, eighth and ninth entries in the series to fully understand the context and progression of the discussion before moving on to this segment.

In our previous blog posts, we've delved into various preference alignment algorithms, many of which have demonstrated promising results. However, a critical question arises: how do we select the most appropriate algorithm for our specific needs? Given that most evaluations in their respective studies were benchmarked against RLHF and DPO, this blog post aims to comprehensively analyze all previously discussed algorithms by examining their diverse attributes, advantages, and drawbacks in a side-by-side comparison.

A Brief Recap

Before we delve into the comparison, let's briefly recap the preference alignment algorithms we've discussed so far:

What Does the Input Data Look Like?

Most of the mentioned methods follow two defined stages instruction tuning and preference alignment. The first one implies SFT and requires instruction data, that is, a dataset containing an instruction and a response. The second one requires preference data, which is a collection of alternatives to a prompt rated by human annotators from the best to the worst. Then, these ranked responses are normally organized in pairs of chosen/rejected, e.g. A > B > C > D will create the following pairs (A > B), (A > C), (A > D), (B > C), (B > D). This is the case for PPO, DPO, IPO, ORPO and SimPO.

For DOVE, instead of comparing responses based on identical instructions, it compares pairs of instruction-response.

However, preparing preference data for alignment algorithms can be challenging. As a result, some methods autonomously generate their own preference data to enhance alignment throughout the RL process, allowing for continuous refinement of the model's alignment with desired outcomes without requiring extensive pre-existing datasets of human preferences. This is the case for SPIN. Initially, it only relies on a high-quality instruction dataset, where the completion column is considered as real/chosen and the new generations/rejected are generated at each iteration by the model to get a preference dataset (you can check some examples here). On the other hand, RLAIF directly uses as input the prompt with the possible responses (previously generated or generated by the same LLM) without any mark, as it is the LLM who is in charge of labeling them to be fed to the reward model.

In the case of KTO, although it has proved to benefit from preference data, its main advantage is that it doesn’t need it. Instead, it can work with binary signal data, that is, each annotation consists of a positive signal (+1) if the instruction is useful or acceptable, or negative (-1) if it is not. This way KTO also avoids the need for a pair-balanced dataset, as it can handle a desirable:undesirable ratio of, for instance, 1:10.

For CoH, the input data is a single sequence that includes the model outputs with feedback about their correctness, similar to how a person would explain that an answer that one answer is preferable over the other.

So, as mentioned in previous posts, many preference alignment methods require high-quality human preference data, which can be both time-consuming and costly to acquire. Conversely, approaches that only rely on prompts and responses are easier to get, as well as the case of DOVE which uses pairs of prompt-response to rank them or KTO which doesn’t need paired preference data so annotation is smoother. Besides, to reproduce similar experiments, we can use the same datasets as in the original papers, as most of them are available to the public.

The table below shows the datasets used during experimentation and their original size. As we can observe Anthropic-HH is the most common and relatively large dataset. In general, while individual datasets are roughly similar in size, when combined their volume is substantial, once again highlighting the challenges in acquiring a significant well-curated dataset. The case of SPIN is noteworthy since most of the data is generated during the process the initial data size is minor.

Can We Reproduce Them?

Considering the importance of these research studies, it is essential to reproduce them and continue experimenting with those approaches.

In this context, Transformer Reinforcement Learning is an open-source library from Hugging Face, offering a straightforward framework to train transformer language models with Reinforcement Learning. As of the time this post is being written, TRL can be used for SFT, RM, PPO, DPO, IPO (by setting loss_type="ipo" in the DPOTrainer), KTO, and ORPO. Axoltl and LLaMA-Factory, two tools to fine-tune LLM models, also implemented SFT, RM, PPO, DPO, and ORPO. For SPIN, we also released the distilabel-spin-dibt repository with code to reproduce the experiments using distilabel. In the other cases, useful code and information can be found on the original repositories mentioned in the recap section, although it slightly hinders reproducibility when using a different dataset or base model.

The alignment handbook of Hugging Face also provides us with some recipes to perform these experiments, e.g., Constitutional AI or the comparison of the DPO, IPO, and KTO methods, and to understand how some models were trained, such as the case of Zephyr-7b-beta, Zephyr 7B Gemma or Zephyr-141B-A35B with ORPO.

Regarding the base models, the most recent studies use open models such as the Mistral-7B models (SPIN, KTO, SimPO), Llama (RLAIF, KTO, ORPO, SimPO), Phi-2 (ORPO), or the OPT ones (CoH, ORPO), among others. This is quite encouraging as it allows us to easily replicate the paper research for verification or even improvement. However, in the case of RLAIF, it relies on GPT-3.5/4 or claude-v1, two closed-source models to label the data, which means that it needs some investment to use their API.

In our case, we have also conducted research and performed experiments with these algorithms. Using argilla and distilabel, we generated some meaningful datasets to align the models. For instance, you can find our collection of datasets as preference data for DPO or ORPO among others (highlighting some of them as ultrafeedback-binarized, distilabel-orca or capybara-preferences), prepared for KTO or for SPIN. During the process, we could reaffirm that well-curated data not only enhances the results but also allows for the reduction of the dataset size without compromising, and potentially even improving, the results. That was the case for the replication of SPIN with only 1.8K prompts in comparison to the 50K used in the original paper (you can find the datasets and models here). We could also observe this in ORPO, where only 7K instances of the capybara dataset were used to fine-tune the model, achieving comparable results using Zephyr, although we will still scale the experiments.

Instruction tuning, Preference tuning, or Combining Both?

SFT is key in RLHF as most of the approaches rely on it to perform the preference alignment. However, the process of preference tuning can vary, involving several steps or being unified in a single one. In the standard approach, RLHF requires three steps: SFT, RW, and PPO. Similarly, RLAIF follows the same stages, as although avoids using human-annotated data, once is labeled the same stages as in RLHF are performed.

As a result, to reduce costs and time, new methods arose such as DPO or CoH, which combined RW and PPO into a single one. Those were followed by IPO to address the DPO overfitting issue, KTO that fixes the need for preference data, leading to comparable results, or DOVE that changes the type of preference data. SimPO also makes training faster as it gets rid of the reference policy in its reward model. SPIN also avoids the implementation of a reward model, but, even if the results were promising, we should note that it needs several iterations.

On the other side, ORPO only relies on the base model as the preference alignment is performed during the SFT leading to a more efficient computation and resource savings.

AI Feedback: Yes or No?

Using AI feedback can be beneficial, especially if we want to obtain results quickly and fairly accurately. Such is the case of RLAIF, which directly uses an LLM as a labeler, or SPIN which during its iterations generates its own preference data until it is unable to discern between the ground truth and the generation.

Most of the experiments reported in the original papers use datasets that have been previously annotated and curated by humans. However, there are also others like WebGPT or UltraFeedback, that were generated using AI Feedback, and even in those cases, the results were significant. As mentioned in the KTO paper, both human and artificial feedback can be quite noisy and contradictory.

In this context, using AI feedback to generate the necessary preference datasets is now a widely accepted practice, offering substantial resource savings. However, it is advisable to ensure that such processes are consistently overseen by a human to maintain quality and reliability.

Conclusions

As we could observe, there isn't a single method that addresses all aspects effectively. Each approach has its advantages and disadvantages, and their effectiveness varies across different scenarios. Thus, the suitability of a method depends on the specific requirements at hand. However, this still proves that there are remaining aspects to be investigated.

Want to know more?

This is the seventh entry of a series of blog posts dedicated to alternatives to RLHF. The first, second, third, fourth, fifth, sixth, seventh, eighth, tenth, and eleventh posts of the series can be found on our website too.

Argilla and Mantis NLP teams are happy to help with any question you may have about preparation steps for training a LLM using Supervised fine-tuning, Reinforcement Learning, or Direct Preference Optimization.

All the data curation steps are currently supported by Argilla’s Data Platform for LLM, and from Mantis NLP we offer end-to-end support for the whole process.