Harmlessness Finetuning

An independent research project where I am implementing a full post-training pipeline from scratch. I am taking a pre-trained-only base Llama 3.2 1B model from HuggingFace, and instead of backpropagating through all of the base model's 1 billion parameters, I am freezing the base model and utilizing two sets of LoRA adapters to implement the intended behaviors.

The first set is used to supervised fine-tune (SFT) the model via the databricks-dolly-15k dataset to encourage general assistant-like prompt-response behavior. The second set of adapters is used to align the model's responses away from harmful requests via direct preference optimization (DPO) on Anthropic's hh-rlhf harmless-base dataset.

After the two fine-tuning steps, I am conducting a general capabilities evaluation on a random set from the Center for AI Safety's MMLU Benchmark and a safety evaluation on the AdvBench benchmark.