Orca 2’s Design and Size. Orca 2 is available in two sizes, 7 billion and 13 billion parameters, and is based on the LLAMA 2 models. It’s designed to perform complex reasoning tasks efficiently, a feature usually associated with larger models having 5-10 times more parameters.

Training Smaller Models.  The concept behind Orca 2 is to utilize the knowledge gained from larger LMs, such as GPT-4, to enhance the reasoning abilities of smaller models. Traditionally, smaller models lacked the advanced reasoning skills exhibited by their larger counterparts.

Diverse Reasoning Techniques.  Orca 2 employs various reasoning techniques like step-by-step processing and recall-reason-generate, adapting its approach based on the task at hand. This versatility allows it to tackle a range of complex tasks more effectively than before.

Performance and Evaluation.  To assess Orca 2’s capabilities, it was tested across 15 diverse benchmarks involving language understanding, common-sense reasoning, and multi-step problem-solving, among others. The results indicate that Orca 2 matches or even surpasses larger models in performance, especially in zero-shot reasoning tasks.

Future Potential and Limitations.  While Orca 2 has shown impressive results, it retains some limitations of the base models it was trained on. However, its development signifies a substantial stride in enhancing reasoning, specialization, control, and safety of smaller models. The use of tailored synthetic data for training has been a key factor in these achievements.