Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add SWEBench-docker eval #2085

Merged
merged 24 commits into from
Jun 10, 2024
Merged

Add SWEBench-docker eval #2085

merged 24 commits into from
Jun 10, 2024

Conversation

xingyaoww
Copy link
Contributor

@xingyaoww xingyaoww commented May 27, 2024 •
edited
Loading

Attempt to address: #2072

Integrating https://github.com/aorwall/SWE-bench-docker for more reliable SWE-Bench evaluation

@xingyaoww xingyaoww added the evaluation Related to running evaluations with OpenHands label May 28, 2024
This was referenced Jun 8, 2024
Closed
Closed
Merged
Closed
@tobitege tobitege mentioned this pull request Jun 9, 2024
Merged
@xingyaoww xingyaoww marked this pull request as ready for review June 10, 2024 17:17
@xingyaoww xingyaoww mentioned this pull request Jun 10, 2024
Merged
@xingyaoww
Copy link
Contributor Author

This should be ready to review! The new eval harness (our fork here: https://github.com/OpenDevin/SWE-bench-docker, with some improvements on patch applying), from my observation, improves our task success rate by roughly 0.5-1%.

@xingyaoww xingyaoww mentioned this pull request Jun 10, 2024
Closed
neubig
neubig approved these changes Jun 10, 2024
View reviewed changes
Copy link
Contributor

@neubig neubig left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great, just a few comments! Happy to merge after these are resolved as you see fit.

evaluation/swe_bench/README.md Outdated Show resolved Hide resolved
evaluation/swe_bench/scripts/docker/pull_all_eval_docker.sh Outdated Show resolved Hide resolved
evaluation/swe_bench/scripts/eval/prep_eval.sh Outdated Show resolved Hide resolved
@xingyaoww xingyaoww enabled auto-merge (squash) June 10, 2024 19:08
@xingyaoww
Copy link
Contributor Author

@neubig done address those points! I've enabled auto-merge

@xingyaoww xingyaoww merged commit a6ba6c5 into main Jun 10, 2024
2 checks passed
@xingyaoww xingyaoww deleted the xw/support-swe-bench-docker branch June 10, 2024 19:30
@xingyaoww xingyaoww mentioned this pull request Jun 10, 2024
Merged
@xingyaoww xingyaoww mentioned this pull request Jun 26, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@neubig neubig neubig approved these changes

Assignees

@xingyaoww xingyaoww

Labels
evaluation Related to running evaluations with OpenHands
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@xingyaoww @neubig