AI IdeasMissionMissionAboutAboutContact ↗Contact ↗ (opens in a new tab)
GitHubGitHub (opens in a new tab)
  • Homepage
  • AI Vision
    • Accountability
    • Agentic Workflows Vision
      • Agentic Frameworks Vision
      • Agents Vision
    • Ai Safety Vision
    • Alignment
    • Build Llm
    • Evaluation
    • Foundations of Ai
    • Integrations
    • JavaScript
    • Lms Vision
    • Opensource Tools Ai Vision
    • Responsible Ai
    • Safety
    • Techstack
    • Training
    • Trends & Expectations
  • Product
    • Collaboration
      • GitHub Organization
    • Communication
      • Discord
      • GitHub Discussions
    • Deployment
    • Development
      • Back End
      • Considerations
        • Devcontainers Considerations
          • Best Practices
          • CI CD Enrivonments
          • Containers Tooling
          • Deployment
          • Devcontainers Definition
          • Development Workflow
          • Getting Started
          • Hardware
          • Mounting
          • Platform Specific Node Modules
          • Setup
          • Team Members Ondoarding
          • Troubleshooting
      • Data Persistence
      • front-end.en-Us
    • Distribution
      • Linux
      • Rust Multi Platform
      • Tutorial Kit
    • Finance
      • Hardware Rent Cloud
      • Local Hardware Costs
      • Operational Costs
      • Ownership Costs
      • Tokens as a Service
    • Legal
      • Contributor License Agreement
      • License
    • Marketing
      • Market Research
    • Product Viability
      • Choosing Oss Tools
      • Llm Empowered Features
    • Product Vision
      • Product Hardware Vision
    • Quality
    • Release Strategy
    • Roadmap
    • Safe Portal
      • Competition
      • Considerations
      • Development
      • Marketing
      • Mvp
      • Personas
          • Poc 1
          • Poc 2
          • Poc 3 1
          • Poc 3 2
          • Poc 4
          • Poc 5
          • Poc 6
          • Poc 7
          • Poc 1
          • Poc 1
    • Security
    • User Interface
      • Ai Safety
      • Education Mode
      • Execution Mode
      • Enterprise
      • Portfolio Vision
      • Release
      • Roadmap
      • Strategic Themes
      • Ai Data Registry
      • Browser Automation
      • Continuous Delivery Platform
      • Continuous Integration Platform
      • Continuous Training Pipeline
      • Devops Platform
      • Domain Specific Perspective
      • Facilitate Scaling
      • Finance Ai Assistant
      • Flavors
      • Generate Entities
      • Knowledge Base
        • Knowledge Base Tools
        • Mintlify
      • Model Switcher
      • Network Intelligence
      • Nx Ai Generators
      • Os Control
      • Solution Intenter
      • Status Page System
      • Terminal Automation
      • Ticket System
        • Backlog
        • Release
        • Sprint
        • Ticket
            • Bdd Testing Report
            • Code
            • E2e Testing Report
            • Execution Logs
            • Unit Testing Report
            • Visual Regression Testing Report
            • Capability
            • Enabler Capability
            • Enabler Epic
            • Enabler Feature
            • Enabler Story
            • Epic
            • Feature
            • Story
        • Ticket Sequence
      • User Input Context Builder
      • Web Containers
        • Ai Techstack Planner
        • Codebase Maintainer Ai
        • Codegen Templates
        • Data
          • Changelog Generation
          • Data Commit Types
          • Releasing
          • Tooling Datasets
          • Versioning
        • Devcontainers
        • Entities
        • Integrator
        • Language Model
        • Manual Evaluation
        • Prompt Execution
        • Release Distribution Options
        • Software Engineer Ai
        • UI Prototyper
        • Use Cases Codification
        • User Input
  • AI Tools
    • Agentic Frameworks
      • Crew Ai
      • Griptape
      • Langchain
      • Vercel Ai Sdk
    • Ai Agents
      • Agentic Vs Non
      • Benefits of Agents
      • Best Practices
      • Discovery Agents
      • How Ai Agents Work
      • Reasoning Paradigms
      • Relates Resources
      • Risks Limitations
      • Types of Agents
      • Use Cases
        • Chain of Thought
        • Program of Thought
    • Ai Platforms
    • Dspy
      • Mental Model
    • Sdk
    • Vector Databases
    • Vision
    • Voice
    • Web Access
      • Alignment
      • Benchmark
        • Osq Bench
      • Evaluation
        • Allenai Eval Frameworks
        • Analysis360
        • Autoarena
        • Big Bench
        • Bigcode Evaluation Harness
        • Challenges
        • Eleuther Ai Harness
        • Elk
        • Gaia
        • Helm
        • Lighteval
        • Olmes
        • Opik
        • Paloma
        • Savety Eval
      • Icl Vs Rag
      • In Context Learning
      • Interpretability
      • Llm Data Processing
      • Llm Toolkits
      • Local
        • Steps
          • Bitnet
          • Huggingface
          • Jan
          • Litgpt
            • Models
          • Llama Cpp
          • Llamafile
          • Lmstudio
          • Ollama
          • Torchchat
          • Torchtune
          • Vllm
          • Whisper Cpp
      • Self Refine
      • Tracing
      • Training
        • Cookbook
        • Fine Tuning
        • Settings
  • AI Ideas
    • Ai Developer Hub
    • Ai University
    • Commit Verifiable Changes
    • Data Enabler
    • Economics of Ai
    • Future Companies
    • Future Jobs
    • Generate Sites
    • Gui Lm Evaluation
    • Org Wide Chat
    • Python JS Monorepo
    • Vscode Lm Extension
  • Foundation
  • Hardware
    • Ai Accelerators
    • Ai Server
    • Cluster
    • Compute Platform
    • Considerations
    • Diy Hardware
      • Cameras
      • Compute Blade
      • Coral
      • Esp32
      • Hailo
      • Lite Rt
      • Llms on Diy Hardware
      • Raspberry Pi
      • Raspberrypi Dgpu
      • Resources
      • Rpi Ai Kit
      • Single Board Computers
      • Turing Pi
        • Pineboards
    • Hardware Evaluation
    • Hardware Monitoring
      • Apple Hardware Monitoring
    • Mlx
    • Racks
    • Requirements
  • Library
  • Models
    • Allenai Models
    • Bloom
    • Coding
    • Granite
    • Llama 3 1
    • Llama 3 2
    • Llm 360 Models
    • Maths Models
  • Open Source
  • Software Engineering
    • Bolt New
    • Changelog
    • CLI
    • Code Intelligence
    • Conda
    • Containers
      • Dev Containers
      • Devpod
      • Docker
    • Continuous Releases
    • Data Persistence
    • Devops Platform
    • E Mails
    • GitHub Profile Generator
    • Iam Sso
    • Link Management
    • Nextjs
    • Python Monorepo
    • Status Page System
    • Syntax Highlighter
    • Tauri
    • UI Libraries
    • Vscode
    • Web Containers
    • Web Scrapers
      • Rust
    • Data Analysis
    • Datasets
      • Allenai Open Data
      • Allenai S2orc S2ag
      • Code Pile
      • Common Crawl
      • Dolma
      • Downloaders
      • Maths
      • The Pile
      • The Stack V2
    • Ai News
    • Courses
      • Aman Ai
    • Funding
    • Glossary
    • Research
Question? Give us feedback → (opens in a new tab)Edit this page
AI Tools
LLM
Evaluation
Helm

HELM

  • A holistic framework for evaluating foundation models. (opens in a new tab)
    • 10 scenarios
      • Core scenarios
      • NarrativeQA
      • NaturalQuestions (open-book)
      • NaturalQuestions (closed-book)
      • OpenbookQA
      • MMLU (Massive Multitask Language Understanding)
      • MATH
      • GSM8K (Grade School Math)
      • LegalBench
      • MedQA
      • WMT 2014
    • 72 models
      • ...
    • github (opens in a new tab)
GaiaLighteval
LICENSE: ...