[Agent] Improve edits by adding back append_file

[xingyaoww]

What is the problem that this fixes or functionality that this introduces? Does it fix any open issues?

Dependency: #2805

This is a follow-up of #2685.

Give a brief summary of what the PR does, explaining any non-trivial design decisions

Per discussion with @tobitege in #2685, this PR:

• add back edit_file_by_line and make it co-exists with edit_file_by_replace it shows degraded performance ([Agent] Improve edits by adding back append_file #2722 (comment))
• add back append_file so that agent can easily append content to the end due to concern mentioned here.
• add '(this is the end of the file)\n' and '(this is the beginning of the file)\n' to tell the agent when it has scrolled to the bottom/top of the file
• fix the extra newline produced by insert_content_at_line introduced in [Agent] (Potentially) improve Editing using diff #2685

TODO:

• This requires [Agent] (Potentially) improve Editing using diff #2685 to be merged first
• Run SWE-Bench Lite to validate performance.

Other references

[xingyaoww]

@neubig Got the claude-3.5 results out.. There's slight degradation on claude-3.5 results (69/300=23%) compared to 1.7 (73/300=24.3%). I suspect it might be caused by the addition of edit by line number in addition to edit by replace.

Do you think we should remove the edit by line and run eval again, or we can just merge as is, and then try to remove it further in the next PR?

EDIT: I plan to try removing edit_file_by_line and see if it improves things - if so, we can just do it with this PR.

[neubig]

I agree that if accuracy has gone down we should probably re-try the eval.

[xingyaoww]

Improved results on this PR - after merging #2846 that removes edit_file_by_line (this does hurt!).

74 resolved out of 300-27=273 inference instances (~27% solve rate - will be 24.6% if we consider 300 instances).
Those 27 instances are unable to run on claude-3.5-sonnet due to the following error:

litellm.BadRequestError: litellm.ContentPolicyViolationError: VertexAIException ContentPolicyViolationError - 
{"type":"error","error":{"type":"invalid_request_error","message":"Output blocked by content filtering policy"}}

Anthropic updated their article last week (https://support.anthropic.com/en/articles/9205721-why-am-i-receiving-an-output-blocked-by-content-filtering-policy-error), suggesting that '.... they generally arise from Anthropic’s efforts to prevent Claude from being used to replicate or regurgitate pre-existing materials...'

Maybe claude is trained on some repository in the SWE-Bench test set, and managed to reproduce them exactly during inference, which is blocked by anthropic's latest update feature?

Nevertheless, 74 resolved, compared to the previous run, 73 resolved, which is still a minor improvement, which I think worth merge this in.

[xingyaoww]

Let me know if people have more concerns here (cc @tobitege) -- if not, i plan to merge this sometime tomorrow (in the next 12-18 hours) so we can go ahead test #2489 with the updated "edit" function.

[yufansong]

LGTM

[tobitege]

WINDOW should be 300?

[xingyaoww]

Good catch! It is actually 100 - i experimented earlier with 300 and it didn't go well so i reverted back to 100, but forgot to revert the doc back :(

[tobitege]

There are 2 more locations of 300, I think

[tobitege]

Let me know if people have more concerns here (cc @tobitege) -- if not, i plan to merge this sometime tomorrow (in the next 12-18 hours) so we can go ahead test #2489 with the updated "edit" function.

Of course, go ahead. 😃