{"payload":{"feedbackUrl":"https://github.com/orgs/community/discussions/53140","repo":{"id":290909192,"defaultBranch":"main","name":"lm-evaluation-harness","ownerLogin":"EleutherAI","currentUserCanPush":false,"isFork":false,"isEmpty":false,"createdAt":"2020-08-28T00:09:15.000Z","ownerAvatar":"https://avatars.githubusercontent.com/u/68924597?v=4","public":true,"private":false,"isOrgOwned":true},"refInfo":{"name":"","listCacheKey":"v0:1727380690.0","currentOid":""},"activityList":{"items":[{"before":"9af298939c54281c0b473320bf9d28e2aa14687a","after":null,"ref":"refs/heads/openai","pushedAt":"2024-09-26T19:58:10.000Z","pushType":"branch_deletion","commitsCount":0,"pusher":{"login":"haileyschoelkopf","name":"Hailey Schoelkopf","path":"/haileyschoelkopf","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/65563625?s=80&v=4"}},{"before":"00f5537abc673d63937c44ccd4975ce5e925a44c","after":"1bc6c93394d0a27f1672695e017f84b97427197e","ref":"refs/heads/main","pushedAt":"2024-09-26T19:58:09.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"haileyschoelkopf","name":"Hailey Schoelkopf","path":"/haileyschoelkopf","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/65563625?s=80&v=4"},"commit":{"message":"openai: better error messages; fix greedy matching (#2327)\n\n* better error message; fix greedy matching\r\n\r\n* Update lm_eval/models/openai_completions.py\r\n\r\nCo-authored-by: Hailey Schoelkopf <65563625+haileyschoelkopf@users.noreply.github.com>\r\n\r\n* Update lm_eval/models/openai_completions.py\r\n\r\nCo-authored-by: Hailey Schoelkopf <65563625+haileyschoelkopf@users.noreply.github.com>\r\n\r\n* pre-commit\r\n\r\n---------\r\n\r\nCo-authored-by: Hailey Schoelkopf <65563625+haileyschoelkopf@users.noreply.github.com>","shortMessageHtmlLink":"openai: better error messages; fix greedy matching (#2327)"}},{"before":"056566165670e28ad194af6d87f79a2b60892fbe","after":null,"ref":"refs/heads/mmlureadme","pushedAt":"2024-09-26T19:54:27.000Z","pushType":"branch_deletion","commitsCount":0,"pusher":{"login":"haileyschoelkopf","name":"Hailey Schoelkopf","path":"/haileyschoelkopf","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/65563625?s=80&v=4"}},{"before":"deb4328771e180e086cf9390d527529fadc1d357","after":"00f5537abc673d63937c44ccd4975ce5e925a44c","ref":"refs/heads/main","pushedAt":"2024-09-26T19:54:26.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"haileyschoelkopf","name":"Hailey Schoelkopf","path":"/haileyschoelkopf","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/65563625?s=80&v=4"},"commit":{"message":"add mmlu readme (#2282)","shortMessageHtmlLink":"add mmlu readme (#2282)"}},{"before":"558d0d71aff9d9d2c72143a3cb96be48ca16c527","after":"deb4328771e180e086cf9390d527529fadc1d357","ref":"refs/heads/main","pushedAt":"2024-09-26T19:27:56.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"haileyschoelkopf","name":"Hailey Schoelkopf","path":"/haileyschoelkopf","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/65563625?s=80&v=4"},"commit":{"message":"Added TurkishMMLU to LM Evaluation Harness (#2283)\n\n* Added TurkishMMLU to LM Evaluation Harness\r\n\r\n* Fixed COT name\r\n\r\n* Fixed COT name\r\n\r\n* Updated Readme\r\n\r\n* Fixed Test issues\r\n\r\n* Completed Scan for changed tasks\r\n\r\n* Updated Readme\r\n\r\n* Update README.md\r\n\r\n* fixup task naming casing + ensure yaml template stubs aren't registered\r\n\r\n---------\r\n\r\nCo-authored-by: Hailey Schoelkopf <65563625+haileyschoelkopf@users.noreply.github.com>\r\nCo-authored-by: haileyschoelkopf ","shortMessageHtmlLink":"Added TurkishMMLU to LM Evaluation Harness (#2283)"}},{"before":"e03dcf3f9c27df031642e745c81c54b9fff03b1e","after":"9af298939c54281c0b473320bf9d28e2aa14687a","ref":"refs/heads/openai","pushedAt":"2024-09-26T19:21:58.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"baberabb","name":"Baber Abbasi","path":"/baberabb","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/92168766?s=80&v=4"},"commit":{"message":"pre-commit","shortMessageHtmlLink":"pre-commit"}},{"before":"b1c1c696e29f7bd85113c8b5d9a1ef942f2f94c5","after":"e03dcf3f9c27df031642e745c81c54b9fff03b1e","ref":"refs/heads/openai","pushedAt":"2024-09-26T19:19:33.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"baberabb","name":"Baber Abbasi","path":"/baberabb","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/92168766?s=80&v=4"},"commit":{"message":"Update lm_eval/models/openai_completions.py\n\nCo-authored-by: Hailey Schoelkopf <65563625+haileyschoelkopf@users.noreply.github.com>","shortMessageHtmlLink":"Update lm_eval/models/openai_completions.py"}},{"before":"623727a4c257545b7f076927069ce2839956bec1","after":"b1c1c696e29f7bd85113c8b5d9a1ef942f2f94c5","ref":"refs/heads/openai","pushedAt":"2024-09-26T19:18:54.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"baberabb","name":"Baber Abbasi","path":"/baberabb","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/92168766?s=80&v=4"},"commit":{"message":"Update lm_eval/models/openai_completions.py\n\nCo-authored-by: Hailey Schoelkopf <65563625+haileyschoelkopf@users.noreply.github.com>","shortMessageHtmlLink":"Update lm_eval/models/openai_completions.py"}},{"before":"a7ce62f68c9bc7666740390a36cffa4cddc71ef5","after":null,"ref":"refs/heads/mmlupro_","pushedAt":"2024-09-26T19:08:09.000Z","pushType":"branch_deletion","commitsCount":0,"pusher":{"login":"haileyschoelkopf","name":"Hailey Schoelkopf","path":"/haileyschoelkopf","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/65563625?s=80&v=4"}},{"before":"7d242381c0aeca89a2bea94c4c849ddd2a4bec35","after":"558d0d71aff9d9d2c72143a3cb96be48ca16c527","ref":"refs/heads/main","pushedAt":"2024-09-26T19:08:07.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"haileyschoelkopf","name":"Hailey Schoelkopf","path":"/haileyschoelkopf","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/65563625?s=80&v=4"},"commit":{"message":"mmlu-pro: add newlines to task descriptions (not leaderboard) (#2334)\n\n* add newlines to task descriptions; increment versions\r\n\r\n* fix task tests (with groups)\r\n\r\n* Apply suggestions from code review\r\n\r\n---------\r\n\r\nCo-authored-by: Hailey Schoelkopf <65563625+haileyschoelkopf@users.noreply.github.com>","shortMessageHtmlLink":"mmlu-pro: add newlines to task descriptions (not leaderboard) (#2334)"}},{"before":"ff00d56a58f95e422acd9c1337d2ff8965ebf16b","after":"a7ce62f68c9bc7666740390a36cffa4cddc71ef5","ref":"refs/heads/mmlupro_","pushedAt":"2024-09-26T19:07:40.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"haileyschoelkopf","name":"Hailey Schoelkopf","path":"/haileyschoelkopf","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/65563625?s=80&v=4"},"commit":{"message":"Apply suggestions from code review","shortMessageHtmlLink":"Apply suggestions from code review"}},{"before":"eda27563ad2cfd77b86dfa915c5af7f1315959b1","after":null,"ref":"refs/heads/glia","pushedAt":"2024-09-26T18:57:40.000Z","pushType":"branch_deletion","commitsCount":0,"pusher":{"login":"haileyschoelkopf","name":"Hailey Schoelkopf","path":"/haileyschoelkopf","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/65563625?s=80&v=4"}},{"before":"af92448e4e9b763ced7ea1ddce8432b2a174c1e2","after":"7d242381c0aeca89a2bea94c4c849ddd2a4bec35","ref":"refs/heads/main","pushedAt":"2024-09-26T18:57:39.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"haileyschoelkopf","name":"Hailey Schoelkopf","path":"/haileyschoelkopf","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/65563625?s=80&v=4"},"commit":{"message":"change glianorex to test split (#2332)\n\n* change glianorex to test set\r\n\r\n* nit\r\n\r\n* fix test; doc_to_target can be str for multiple_choice\r\n\r\n* nit","shortMessageHtmlLink":"change glianorex to test split (#2332)"}},{"before":"7bc960c4f57eb0f762f45ffebe23b552458c270b","after":null,"ref":"refs/heads/tag_g","pushedAt":"2024-09-26T18:56:32.000Z","pushType":"branch_deletion","commitsCount":0,"pusher":{"login":"haileyschoelkopf","name":"Hailey Schoelkopf","path":"/haileyschoelkopf","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/65563625?s=80&v=4"}},{"before":"b2bf7bc4a601c643343757c92c1a51eb69caf1d7","after":"af92448e4e9b763ced7ea1ddce8432b2a174c1e2","ref":"refs/heads/main","pushedAt":"2024-09-26T18:56:31.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"haileyschoelkopf","name":"Hailey Schoelkopf","path":"/haileyschoelkopf","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/65563625?s=80&v=4"},"commit":{"message":"change group to tags in task `eus_exams` task configs (#2320)","shortMessageHtmlLink":"change group to tags in task eus_exams task configs (#2320)"}},{"before":"6fb8dde6f31caa94139c7dba8444a2577447cb85","after":"8582a380b17ce3e08ac28fa88c00ae174261bfdf","ref":"refs/heads/cost","pushedAt":"2024-09-26T15:08:12.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"baberabb","name":"Baber Abbasi","path":"/baberabb","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/92168766?s=80&v=4"},"commit":{"message":"nit","shortMessageHtmlLink":"nit"}},{"before":null,"after":"6fb8dde6f31caa94139c7dba8444a2577447cb85","ref":"refs/heads/cost","pushedAt":"2024-09-26T14:56:34.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"baberabb","name":"Baber Abbasi","path":"/baberabb","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/92168766?s=80&v=4"},"commit":{"message":"fix `cost_estimate`","shortMessageHtmlLink":"fix cost_estimate"}},{"before":"72d619ffdb075be0261e60d3bcd2d91e155b5580","after":"b2bf7bc4a601c643343757c92c1a51eb69caf1d7","ref":"refs/heads/main","pushedAt":"2024-09-26T14:03:33.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"haileyschoelkopf","name":"Hailey Schoelkopf","path":"/haileyschoelkopf","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/65563625?s=80&v=4"},"commit":{"message":"Treat tags in python tasks the same as yaml tasks (#2288)\n\n* Treat python tasks same as yaml tasks.\r\n\r\n* Add tests.\r\n\r\n* Re-add fixture decorators.\r\n\r\n* Fix typing specification error for Python 3.9.","shortMessageHtmlLink":"Treat tags in python tasks the same as yaml tasks (#2288)"}},{"before":"2d33465bbba3254a6ccbba80366e286a72971f1b","after":null,"ref":"refs/heads/writeout","pushedAt":"2024-09-26T13:36:04.000Z","pushType":"branch_deletion","commitsCount":0,"pusher":{"login":"haileyschoelkopf","name":"Hailey Schoelkopf","path":"/haileyschoelkopf","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/65563625?s=80&v=4"}},{"before":"f378f30605e0c3afef65d795370d220e6bfc92c4","after":"72d619ffdb075be0261e60d3bcd2d91e155b5580","ref":"refs/heads/main","pushedAt":"2024-09-26T13:36:02.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"haileyschoelkopf","name":"Hailey Schoelkopf","path":"/haileyschoelkopf","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/65563625?s=80&v=4"},"commit":{"message":"fix writeout script (#2350)","shortMessageHtmlLink":"fix writeout script (#2350)"}},{"before":"920beede135562547d6a42bb6b22627b764f5820","after":null,"ref":"refs/heads/dsev","pushedAt":"2024-09-26T02:04:09.000Z","pushType":"branch_deletion","commitsCount":0,"pusher":{"login":"lintangsutawika","name":"Lintang Sutawika","path":"/lintangsutawika","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/5774558?s=80&v=4"}},{"before":"bc50a9aa6d6149f6702889b4ef341b83d0304f85","after":"f378f30605e0c3afef65d795370d220e6bfc92c4","ref":"refs/heads/main","pushedAt":"2024-09-26T02:04:06.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"lintangsutawika","name":"Lintang Sutawika","path":"/lintangsutawika","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/5774558?s=80&v=4"},"commit":{"message":"load metric with `evaluate` (#2351)","shortMessageHtmlLink":"load metric with evaluate (#2351)"}},{"before":"f096d1925e713ad27346a8a0793bad7423abb5a7","after":"f5b068628bcc9ad9e1ba49104a7dd3e3e505e497","ref":"refs/heads/automodel","pushedAt":"2024-09-25T17:38:25.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"baberabb","name":"Baber Abbasi","path":"/baberabb","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/92168766?s=80&v=4"},"commit":{"message":"nit","shortMessageHtmlLink":"nit"}},{"before":"b0d6d8c9d08348c79d04d17359933b17d3797ed0","after":"f096d1925e713ad27346a8a0793bad7423abb5a7","ref":"refs/heads/automodel","pushedAt":"2024-09-25T17:32:07.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"baberabb","name":"Baber Abbasi","path":"/baberabb","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/92168766?s=80&v=4"},"commit":{"message":"nit","shortMessageHtmlLink":"nit"}},{"before":null,"after":"b0d6d8c9d08348c79d04d17359933b17d3797ed0","ref":"refs/heads/automodel","pushedAt":"2024-09-25T17:26:52.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"baberabb","name":"Baber Abbasi","path":"/baberabb","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/92168766?s=80&v=4"},"commit":{"message":"switch conditional checks to `self.backend`","shortMessageHtmlLink":"switch conditional checks to self.backend"}},{"before":null,"after":"920beede135562547d6a42bb6b22627b764f5820","ref":"refs/heads/dsev","pushedAt":"2024-09-25T12:33:46.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"baberabb","name":"Baber Abbasi","path":"/baberabb","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/92168766?s=80&v=4"},"commit":{"message":"load metric with `evaluate`","shortMessageHtmlLink":"load metric with evaluate"}},{"before":null,"after":"2d33465bbba3254a6ccbba80366e286a72971f1b","ref":"refs/heads/writeout","pushedAt":"2024-09-25T12:24:41.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"baberabb","name":"Baber Abbasi","path":"/baberabb","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/92168766?s=80&v=4"},"commit":{"message":"fix script","shortMessageHtmlLink":"fix script"}},{"before":"15e3f5c5390e1408a4c14e07e93d0b745774d159","after":"3453b1286aacf8a1de80678367ce6ca379724990","ref":"refs/heads/bjudge","pushedAt":"2024-09-25T11:52:12.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"baberabb","name":"Baber Abbasi","path":"/baberabb","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/92168766?s=80&v=4"},"commit":{"message":"add todos","shortMessageHtmlLink":"add todos"}},{"before":"d7734d1923fe74265045169ca6fd1031046d646d","after":"bc50a9aa6d6149f6702889b4ef341b83d0304f85","ref":"refs/heads/main","pushedAt":"2024-09-24T14:12:57.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"baberabb","name":"Baber Abbasi","path":"/baberabb","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/92168766?s=80&v=4"},"commit":{"message":"add a note for missing dependencies (#2336)","shortMessageHtmlLink":"add a note for missing dependencies (#2336)"}},{"before":"a6b17509eeaa177cd7adb9ffa05b16a182de985d","after":"56f40c535c213f6b2c71e6a78aaaf8f55da66270","ref":"refs/heads/mathvista","pushedAt":"2024-09-24T13:29:25.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"baberabb","name":"Baber Abbasi","path":"/baberabb","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/92168766?s=80&v=4"},"commit":{"message":"add target delimiter","shortMessageHtmlLink":"add target delimiter"}}],"hasNextPage":true,"hasPreviousPage":false,"activityType":"all","actor":null,"timePeriod":"all","sort":"DESC","perPage":30,"startCursor":"Y3Vyc29yOnYyOpK7MjAyNC0wOS0yNlQxOTo1ODoxMC4wMDAwMDBazwAAAATB6HHF","endCursor":"Y3Vyc29yOnYyOpK7MjAyNC0wOS0yNFQxMzoyOToyNS4wMDAwMDBazwAAAAS_MXzQ"}},"title":"Activity ยท EleutherAI/lm-evaluation-harness"}