[SWE-bench] Util: Compare files modified between gold patches and OpenDevin patches

[li-boxuan]

https://aider.chat/2024/05/22/swe-bench-lite.html says it finds correct files to edit in 70.3% of benchmark tests. This PR adds a utility script that computes the number for OpenDevin.

Usage Example:

poetry run python evaluation/swe_bench/scripts/setup/compare_patch_filename.py --od_output_file /Users/liboxuan/workspace/OpenDevin/evaluation/evaluation_outputs/outputs/swe_bench_lite/CodeActAgent/claude-3-5-sonnet@20240620_maxiter_30_N_v1.8-no-hint/output.jsonl

claude-3-5-sonnet@20240620_maxiter_30_N_v1.8-no-hint result says OpenDevin found correct files to edit in 64% of benchmark tests. The real number might be even lower because I consider success as long as OpenDevin modifies the correct file even though it might have modified other files erroneously and caused the test to fail.

[li-boxuan]

I didn't add a README file coz I feel this script is not very useful for general audience. I created this PR mainly to convey my findings. Feel free to merge or close this PR.

[tobitege]

I didn't add a README file coz I feel this script is not very useful for general audience. I created this PR mainly to convey my findings. Feel free to merge or close this PR.

I don't mind having this added, in case anyone else might find it useful.
How about adding your usage example to the docstring in this file itself, though?

[xingyaoww]

Thanks for this script! LGTM!