Evaluating Multimodal Models

In today's episode of the Daily AI Show, Brian, Andy, Eran, and Jyunmi discussed the evaluation of multimodal models. They explored the importance of assessment prompts and models, why evaluations are necessary, and highlighted the work of REKA.ai in this space.

Key Points Discussed:

• Overview of Evaluation Models: Andy broke down the types of evaluation models, such as perplexity, GLUE (General Language Understanding Evaluation), and BLU (Bilingual Evaluation Understudy). He also touched on benchmarks like MMLU (Massive Multitask Language Understanding) and the challenges of training models to game leaderboards.
• Multimodal Evaluations and RECA: The team introduced REKA.ai's Vibe-Eval, which helps measure progress in multimodal models. This suite includes 269 image-text prompts with ground truth responses to evaluate models' capabilities. They praised the system's ability to assess nuanced image features and text.
• GitHub and Leaderboards: Brian showcased REKA's GitHub page, where Vibe-Eval and a leaderboard are available. REKA Core ranks third on its own leaderboard but maintains a prominent seventh place among 95 models on LMSYS's comprehensive leaderboard.
• Independent Evaluations and Bias: The importance of independent evaluations to avoid bias was raised, noting that benchmarks could be tailored to favor certain models. The group stressed the need for varied testing to ensure unbiased and comprehensive results.
• Tool Recommendations: The team recommended platforms like Poe, Respell, and PromptMetheus to conduct prompt testing across various models. They highlighted the value of experimenting with different models to achieve optimal results.