Top Open-Source LLMs for 2024

An LLM, or large language model, is a general-purpose AI text generator. It's what's behind the scenes of all AI chatbots and AI writing generators. The current generative AI revolution wouldn’t be possible without it. Based on transformers, a powerful neural architecture, LLMs are AI systems used to model and process human language. They are called “large” because they have hundreds of millions or even billions of parameters, which are pre-trained using a massive corpus of text data.

LLMs are the foundation models of popular and widely-used chatbots, like ChatGPT and Google Bard. In particular, ChatGPT is powered by GPT-4, a LLM developed and owned by OpenAI, while Google Bard is based on Google’s PaLM 2 model. ChatGPT and Bard, as well as many other popular chatbots, have in common that their underlying LLM are proprietary. This article aims to explore the top open-source LLMs available in 2024.

Mistral LLM refers to the Large Language Model developed by Mistral AI. Mistral 7B is the first dense model released. At the time of the release, it matched the capabilities of models up to 30B parameters. Here are some of its features:

1. Advanced Architecture: Mistral LLM employs a Mixture of Experts (MoE) architecture, which allows it to achieve the performance of a 12 billion parameter-dense model while being significantly more efficient in terms of cost and latency.

2. Reasoning and Knowledge: Mistral LLM exhibits powerful reasoning abilities, surpassing other top-leading LLM models on commonly used benchmarks for reasoning and knowledge, such as MMLU, HellaSwag, WinoGrande, Arc Challenge, and TriviaQA.

3. Multilingual Capabilities: Mistral LLM possesses strong multilingual skills, performing exceptionally well in languages like French, German, Spanish, and Italian, particularly in HellaSwag, Arc Challenge, and MMLU benchmarks.

4. Maths & Coding: Mistral LLM excels in coding and math tasks, achieving top performance across various popular benchmarks, including HumanEval pass@1, MBPP pass@1, Math maj@4, GSM8K maj@8 (8-shot), and GSM8K maj@1 (5 shot).

5. Function Calling: Mistral LLM supports function calling, allowing developers to interact with the model more naturally and extract information in a structured way, which can be useful when working with internal code, APIs, or databases.

6. JSON Format: Mistral LLM supports JSON format, which enables developers to force the language model output to be valid JSON, facilitating easier integration with external tools.

7. Flexibility: Mistral LLM comes in different versions, including Mistral Large and Mistral Small, tailored to meet specific needs in terms of performance, cost, and latency.

8. Open Source: Mistral LLM is open source, allowing researchers and developers to access its codebase and modify it according to their requirements.

9. Real-World Applications: Mistral LLM is designed for real-world applications, providing both efficiency and high performance to enable practical use cases.

10. Instruction-Fine-Tuned Model: Mistral LLM includes an instruction-fine-tuned model, Mistral-7B-Instruct-v0.1, which can be used for chat-based inference.

Mixtral is an advanced large language model developed by Mistral AI, which stands out due to its unique architecture. Mixtral implements a technique called "Mixture of Experts" (MoE), which replaces some feed-forward layers with a sparse MoE layer. This layer contains a router network that selects two expert models to process each token, enabling the model to decode at the speed of a 12 billion parameter-dense model while having 4 times the number of effective parameters.

Key features of Mixtral include:

1. Performance: Mixtral outperforms Llama 2 70B on most benchmarks and matches or beats GPT-3.5 on most standard benchmarks, demonstrating exceptional performance.

2. Multilingual capabilities: Mixtral masters English, French, German, Spanish, and Italian, making it suitable for multilingual tasks.

3. Code generation: Mixtral shows strong performance in code generation, making it a valuable tool for programming tasks.

4. Instruction-following model: Mixtral can be fine-tuned into an instruction-following model that achieves a score of 8.3 on MT-Bench, indicating its ability to follow instructions accurately.

5. Sparse architecture: Mixtral is a sparse mixture-of-experts network, which allows it to increase the number of parameters while controlling cost and latency, making it more efficient than dense models.

6. Training: Mixtral is pre-trained on data extracted from the open Web, ensuring that it has a diverse and comprehensive understanding of various topics.

7. Deployment: Mixtral can be deployed with an open-source deployment stack, making it easily accessible for users who want to leverage its capabilities.

Llama 2 is an open-source large language model (LLM) developed by Meta, formerly known as Facebook. It is a response to OpenAI's GPT models and Google's AI models like PaLM 2. Llama 2 is made available for research and commercial purposes, unlike many other large language models that are closed source.

Here are some key features of Llama 2:

1. Family of LLMs: Llama 2 is a family of LLMs, like GPT-3 and PaLM 2. It utilizes the same transformer architecture and development ideas as other large language models.

2. Text Prediction: When given a text prompt or other text input, Llama 2 attempts to predict the most plausible follow-on text using its neural network, employing a cascading algorithm with billions of parameters and incorporating a small amount of randomness to generate human-like responses.

3. Versions: Llama 2 offers several versions with varying parameter counts: 7B, 13B, and 70B. Smaller versions like 7B and 13B are optimized for speed and can run efficiently on lower-spec hardware, albeit with slightly less effectiveness in generating plausible or accurate text compared to larger models.

4. Optimized for Dialogue: Some versions of Llama 2, such as Llama-2-chat, are fine-tuned specifically for chatbot-like dialogue, similar to how ChatGPT is optimized for conversational interactions.

5. Availability: Llama 2 is available through various platforms, including AWS, Hugging Face, and others, allowing developers to integrate it into their projects and utilize its capabilities.

Gemma is a family of four large language models developed by Google, based on the Gemini models. It includes two sizes, 2B and 7B parameters, and both come in base and instruction-tuned versions. The base models include Gemma-7B and Gemma-2B, while the instruction-tuned versions are Gemma-7B-IT and Gemma-2B-IT. These models are capable of running on various types of consumer hardware, including CPUs and GPUs, without requiring quantization. They have a context length of 8K tokens and are designed for use in text generation and specialized tasks through model tuning.

Some notable features of Gemma include:

1. Continuous batching: Gemma supports continuous batching, which allows for more efficient use of hardware resources and faster inference.

2. Token streaming: Gemma supports token streaming, which allows for the processing of large inputs in a memory-efficient manner.

3. Tensor parallelism: Gemma supports tensor parallelism for fast inference on multiple GPUs.

4. Production-ready: Gemma is designed to be production-ready, with features such as automatic mixed precision and model quantization.

5. Compatibility: Gemma is compatible with TensorFlow, JAX, and PyTorch, making it accessible to a wide range of developers.

6. Familiar API: Gemma is offered with a familiar KerasNLP API and a super-readable Keras implementation, making it easy to use for developers familiar with Keras.

Mistral 7B is a 7.3 billion parameter model known for its impressive performance on various benchmarks. It outperforms the Llama 2 13B in all benchmarks and is on par with the Llama 34B. It also approaches the coding performance of CodeLlama 7B while maintaining proficiency on English tasks. The model uses Grouped Query Attention (GQA) for faster inference and Sliding Window Attention (SWA) to handle longer sequences at less cost. One of the main advantages of the Mistral 7B is its adaptability. It can also be used locally via a reference implementation provided by the developer. Additionally, the Mistral 7B can be easily fine-tuned for any task.

Llama 2 is part of a collection of pretrained and fine-tuned generative text models ranging in size from 7 billion to 70 billion parameters. The Llama 2 family of large language models (LLMs) was developed by Meta and optimized for conversational use cases. The fine-tuned LLM (called Llama-2-Chat) outperforms open-source chat models on most test benchmarks and is comparable to popular closed-source models such as ChatGPT and PaLM in terms of usefulness and security. Llama 2 is intended for English-language business and research use. The tuned model is designed for assistant-style chat, while the pre-trained model can be applied to a variety of natural language generation tasks.

Mixtral 8x7B, a cutting-edge sparse model mixture of experts (SMoE) with open weights. This new model is a significant leap forward, outperforming Llama 2 70B on most benchmarks while delivering 6x faster inference. Mixtral 8x7B is licensed under the open and permissive Apache 2.0 and is the most powerful open-weight model available. It sets a new standard in terms of price/performance, with Mixtral matching or outperforming Llama 2 70B as well as GPT-3.5 in most benchmarks. Mixtral enables faster inference at small batch sizes and higher throughput at large batch sizes.

The Gemma model is the first open source LLM launched by Google built using the same research and technology as the Gemini model. This series of models currently comes in two sizes, 2B and 7B, and provides a basic chat version and a command version. It learned the advantages of Llama 2 and Mistral 7B, using more Tokens and words to train a better 7B (8.5B) model. Gemma 2B and 7B were trained on 2 trillion and 6 trillion tokens respectively. This means that Gemma 7B accepts 3 times more tokens than Llama 2. Gemma 7B looks like a good competitor to Mistral 7B, but let's not forget that it also has 1 billion more parameters than Mistral 7B.

There are multiple short-term and long-term benefits to choosing open-source LLMs instead of proprietary LLMs. Below, you can find a list of the most compelling reasons:

Enhanced data security and privacy

One of the biggest concerns of using proprietary LLMs is the risk of data leaks or unauthorized access to sensitive data by the LLM provider. Indeed, there have already been several controversies regarding the alleged use of personal and confidential data for training purposes. By using open-source LLM, companies will be solely responsible for the protection of personal data, as they will keep full control of it.

Cost savings and reduced vendor dependency

Most proprietary LLMs require a license to use them. In the long term, this can be an important expense that some companies, especially SME ones, may not be able to afford. This is not the case with open-source LLMs, as they are normally free to use. However, it’s important to note that running LLMs requires considerable resources, even only for inference, which means that you will normally have to pay for the use of cloud services or powerful infrastructure.

Code transparency and language model customization

Companies that opt for open-source LLMs will have access to the workings of LLMs, including their source code, architecture, training data, and mechanism for training and inference. This transparency is the first step for scrutiny but also for customization. Since open-source LLMs are accessible to everyone, including their source code, companies using them can customize them for their particular use cases.

Active community support and fostering innovation

The open-source movement promises to democratize the use and access of LLM and generative AI technologies. Allowing developers to inspect the inner workings of LLMs is key for the future development of this technology. By lowering entry barriers to coders around the world, open-source LLMs can foster innovation and improve the models by reducing biases and increasing accuracy and overall performance.

When choosing the best open-source language model (LLM), there are several factors to consider, including the model's performance, adaptability, and compatibility with your specific needs. It's important to note that the open-source LLM landscape is rapidly evolving, and new models and versions may be released. Stay updated with the latest research and developments to make an informed choice based on the most recent information available.