Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Agent] Improve edits by adding back append_file #2722

Merged
merged 61 commits into from
Jul 11, 2024
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
61 commits
Select commit Hold shift + click to select a range
d3a2d10
add replace-based block edit & preliminary test case fix
xingyaoww Jun 28, 2024
b03fd2c
further fix the insert behavior
xingyaoww Jun 28, 2024
3700478
make edit only work on first occurence
xingyaoww Jun 28, 2024
8138371
bump codeact version since we now use new edit agentskills
xingyaoww Jun 28, 2024
45283e1
update prompt for new agentskills
xingyaoww Jun 28, 2024
790b94f
update integration tests
xingyaoww Jun 28, 2024
8e630c5
make run_infer.sh executable
xingyaoww Jun 28, 2024
693a53f
remove code block for edit_file
xingyaoww Jun 28, 2024
7fcba43
update integration test for prompt changes
xingyaoww Jun 28, 2024
22de134
default to not use hint for eval
xingyaoww Jun 29, 2024
4ec06f0
fix insert emptyfile bug
xingyaoww Jun 29, 2024
2c85f43
throw value error when `to_replace` is empty
xingyaoww Jun 29, 2024
621a158
make `_edit_or_insert_file` return string so we can try to fix some l…
xingyaoww Jun 29, 2024
4cb8bdb
add todo
xingyaoww Jun 29, 2024
20cdd32
update integration test
xingyaoww Jun 29, 2024
db14924
fix sandbox test for this PR
xingyaoww Jul 1, 2024
00f57e9
fix inserting with additional newline
xingyaoww Jul 1, 2024
a2f0916
rename to edit_file_by_replace
xingyaoww Jul 1, 2024
cbf2298
add back `edit_file_by_line`
xingyaoww Jul 1, 2024
fb0b9de
update prompt for new editing tool
xingyaoww Jul 1, 2024
cf9eef9
fix integration tests
xingyaoww Jul 1, 2024
a251122
bump codeact version since there are more changes
xingyaoww Jul 1, 2024
0aa476c
add back append file
xingyaoww Jul 1, 2024
ab47585
fix current line for append
xingyaoww Jul 1, 2024
eba37d5
fix append unit tests
xingyaoww Jul 1, 2024
22274be
change the location where we show edited line no to agent and fix tests
xingyaoww Jul 1, 2024
a45953d
update integration tests
xingyaoww Jul 2, 2024
f85c847
fix global window size affect by open_file bug
xingyaoww Jul 2, 2024
62480e5
fix global window size affect by open_file bug
xingyaoww Jul 2, 2024
8968e80
increase window size to 300
xingyaoww Jul 2, 2024
84b3358
add file beginning and ending marker to avoid looping
xingyaoww Jul 2, 2024
53ef016
Merge commit '41ddba84bd0d62f8a4bc48d08addde4b4269a687' into HEAD
xingyaoww Jul 2, 2024
26d9c90
expand the editor window to better display edit error for model
xingyaoww Jul 2, 2024
a4e41ed
refractor to breakdown edit to internal functions
xingyaoww Jul 2, 2024
eec2aa4
reduce window to 200
xingyaoww Jul 2, 2024
31ae269
move window to 100
xingyaoww Jul 2, 2024
53c36e1
refractor to cleanup some logic into _calculate_window_bounds
xingyaoww Jul 2, 2024
ef80308
Merge branch 'main' into xw/further-improve-edit
neubig Jul 3, 2024
e426298
fix integration tests
xingyaoww Jul 4, 2024
1d6a39e
fix sandbox test on new prompt
xingyaoww Jul 4, 2024
6bba835
update demonstration with new changes
xingyaoww Jul 4, 2024
738c657
fix integration
xingyaoww Jul 4, 2024
8fe5034
initialize llm inside process_instance to circumvent "AttributeError:…
xingyaoww Jul 5, 2024
d2a49aa
update kwargs
xingyaoww Jul 5, 2024
64938f5
retry for internal server error
xingyaoww Jul 5, 2024
21246fb
fix max iteration
xingyaoww Jul 5, 2024
f9cfecd
override max iter from config
xingyaoww Jul 5, 2024
4c6fa51
Merge commit 'adf1a0d55654d0065b615d0b4a6534cc5478b732' into xw/furth…
xingyaoww Jul 7, 2024
5247bfa
fix integration tests
xingyaoww Jul 7, 2024
949b38d
remove edit file by line
xingyaoww Jul 7, 2024
b4ac713
fix integration tests
xingyaoww Jul 7, 2024
06fd2c5
add instruction to avoid hanging
xingyaoww Jul 8, 2024
e690830
Revert "add instruction to avoid hanging"
xingyaoww Jul 9, 2024
57823b3
handle content policy violation error
xingyaoww Jul 10, 2024
5be8c05
Merge commit 'e6908307b436b07ede7ff3c093aa79db2ce6ee9c' into xw/furth…
xingyaoww Jul 10, 2024
7c67798
Merge commit '57823b3846cc297933282b181d6eed2c7511bcdc' into xw/furth…
xingyaoww Jul 10, 2024
8df6970
Merge commit '456690818c94a266935888f1e56e0afa2c4d5219' into xw/furth…
xingyaoww Jul 10, 2024
b474bbd
fix integration tests
xingyaoww Jul 10, 2024
f02c145
fix typo in prompt - the window is 100
xingyaoww Jul 11, 2024
5473bf8
update all integration tests
xingyaoww Jul 11, 2024
1204fe1
Merge branch 'main' into xw/further-improve-edit
xingyaoww Jul 11, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
fix integration tests
  • Loading branch information
xingyaoww committed Jul 1, 2024
commit cf9eef93571614aaaa612ff8d5518fd31d526b7a
Original file line number Diff line number Diff line change
Expand Up @@ -54,11 +54,11 @@ create_file(filename: str) -> None:
Args:
filename: str: The name of the file to create.

edit_file(file_name: str, to_replace: str, new_content: str) -> None:
edit_file_by_replace(file_name: str, to_replace: str, new_content: str) -> None:
Edit a file. This will search for `to_replace` in the given file and replace it with `new_content`.
Every *to_replace* must *EXACTLY MATCH* the existing source code, character for character, including all comments, docstrings, etc.
Include enough lines to make code in `to_replace` unique. `to_replace` should NOT be empty.
`edit_file` will only replace the *first* matching occurrences.
`edit_file_by_replace` will only replace the *first* matching occurrences.
For example, given a file "/workspace/example.txt" with the following content:
```
line 1
Expand All @@ -67,7 +67,7 @@ edit_file(file_name: str, to_replace: str, new_content: str) -> None:
line 3
```
EDITING: If you want to replace the second occurrence of "line 2", you can make `to_replace` unique:
edit_file(
edit_file_by_replace(
'/workspace/example.txt',
to_replace='line 2
line 3',
Expand All @@ -83,7 +83,7 @@ edit_file(file_name: str, to_replace: str, new_content: str) -> None:
line 3
```
REMOVAL: If you want to remove "line 2" and "line 3", you can set `new_content` to an empty string:
edit_file(
edit_file_by_replace(
'/workspace/example.txt',
to_replace='line 2
line 3',
Expand All @@ -94,6 +94,30 @@ edit_file(file_name: str, to_replace: str, new_content: str) -> None:
to_replace: str: The content to search for and replace.
new_content: str: The new content to replace the old content with.

edit_file_by_line(file_name: str, start: int, end: int, new_content: str) -> None:
Edit a file.
Replaces in given file `file_name` the lines `start` through `end` (inclusive) with the given text `content`.
If a line must be inserted, an already existing line must be passed in `content` with new content accordingly!
For example, given a file "/workspace/example.txt" with the following content (line numbers are included for clarity):
```
1|line 1
2|line 2
3|line 3
```
If you want to replace the second line, you can call:
edit_file_by_line('/workspace/example.txt', 2, 2, 'new line')
This will give the following result:
```
1|line 1
2|new line
3|line 3
```
Args:
file_name: str: The name of the file to edit.
start: int: The start line number. Must satisfy start >= 1.
end: int: The end line number. Must satisfy start <= end <= number of lines in the file.
content: str: The content to replace the lines with.

insert_content_at_line(file_name: str, line_number: int, content: str) -> None:
Insert content at the given line number in a file.
This will NOT modify the content of the lines before OR after the given line number.
Expand Down Expand Up @@ -153,7 +177,7 @@ parse_pptx(file_path: str) -> None:
Args:
file_path: str: The path to the file to open.

Please note that THE `edit_file` and `insert_content_at_line` FUNCTIONS REQUIRE PROPER INDENTATION. If the assistant would like to add the line ' print(x)', it must fully write that out, with all those spaces before the code! Indentation is important and code that is not indented correctly will fail and require fixing before it can be run.
Please note that THE `edit_file_by_replace`, `edit_file_by_line` and `insert_content_at_line` FUNCTIONS REQUIRE PROPER INDENTATION. If the assistant would like to add the line ' print(x)', it must fully write that out, with all those spaces before the code! Indentation is important and code that is not indented correctly will fail and require fixing before it can be run.

Responses should be concise.
The assistant should attempt fewer things at a time instead of putting too many commands OR too much code in one "execute" block.
Expand Down Expand Up @@ -341,7 +365,7 @@ USER:
ASSISTANT:
I should edit the file to display the numbers in a table format. I should include correct indentation. Let me update the file:
<execute_ipython>
edit_file(
edit_file_by_replace(
'app.py',
to_replace=" return str(numbers)",
new_content=" return '<table>' + ''.join([f'<tr><td>{i}</td></tr>' for i in numbers]) + '</table>'",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -54,11 +54,11 @@ create_file(filename: str) -> None:
Args:
filename: str: The name of the file to create.

edit_file(file_name: str, to_replace: str, new_content: str) -> None:
edit_file_by_replace(file_name: str, to_replace: str, new_content: str) -> None:
Edit a file. This will search for `to_replace` in the given file and replace it with `new_content`.
Every *to_replace* must *EXACTLY MATCH* the existing source code, character for character, including all comments, docstrings, etc.
Include enough lines to make code in `to_replace` unique. `to_replace` should NOT be empty.
`edit_file` will only replace the *first* matching occurrences.
`edit_file_by_replace` will only replace the *first* matching occurrences.
For example, given a file "/workspace/example.txt" with the following content:
```
line 1
Expand All @@ -67,7 +67,7 @@ edit_file(file_name: str, to_replace: str, new_content: str) -> None:
line 3
```
EDITING: If you want to replace the second occurrence of "line 2", you can make `to_replace` unique:
edit_file(
edit_file_by_replace(
'/workspace/example.txt',
to_replace='line 2
line 3',
Expand All @@ -83,7 +83,7 @@ edit_file(file_name: str, to_replace: str, new_content: str) -> None:
line 3
```
REMOVAL: If you want to remove "line 2" and "line 3", you can set `new_content` to an empty string:
edit_file(
edit_file_by_replace(
'/workspace/example.txt',
to_replace='line 2
line 3',
Expand All @@ -94,6 +94,30 @@ edit_file(file_name: str, to_replace: str, new_content: str) -> None:
to_replace: str: The content to search for and replace.
new_content: str: The new content to replace the old content with.

edit_file_by_line(file_name: str, start: int, end: int, new_content: str) -> None:
Edit a file.
Replaces in given file `file_name` the lines `start` through `end` (inclusive) with the given text `content`.
If a line must be inserted, an already existing line must be passed in `content` with new content accordingly!
For example, given a file "/workspace/example.txt" with the following content (line numbers are included for clarity):
```
1|line 1
2|line 2
3|line 3
```
If you want to replace the second line, you can call:
edit_file_by_line('/workspace/example.txt', 2, 2, 'new line')
This will give the following result:
```
1|line 1
2|new line
3|line 3
```
Args:
file_name: str: The name of the file to edit.
start: int: The start line number. Must satisfy start >= 1.
end: int: The end line number. Must satisfy start <= end <= number of lines in the file.
content: str: The content to replace the lines with.

insert_content_at_line(file_name: str, line_number: int, content: str) -> None:
Insert content at the given line number in a file.
This will NOT modify the content of the lines before OR after the given line number.
Expand Down Expand Up @@ -153,7 +177,7 @@ parse_pptx(file_path: str) -> None:
Args:
file_path: str: The path to the file to open.

Please note that THE `edit_file` and `insert_content_at_line` FUNCTIONS REQUIRE PROPER INDENTATION. If the assistant would like to add the line ' print(x)', it must fully write that out, with all those spaces before the code! Indentation is important and code that is not indented correctly will fail and require fixing before it can be run.
Please note that THE `edit_file_by_replace`, `edit_file_by_line` and `insert_content_at_line` FUNCTIONS REQUIRE PROPER INDENTATION. If the assistant would like to add the line ' print(x)', it must fully write that out, with all those spaces before the code! Indentation is important and code that is not indented correctly will fail and require fixing before it can be run.

Responses should be concise.
The assistant should attempt fewer things at a time instead of putting too many commands OR too much code in one "execute" block.
Expand Down Expand Up @@ -341,7 +365,7 @@ USER:
ASSISTANT:
I should edit the file to display the numbers in a table format. I should include correct indentation. Let me update the file:
<execute_ipython>
edit_file(
edit_file_by_replace(
'app.py',
to_replace=" return str(numbers)",
new_content=" return '<table>' + ''.join([f'<tr><td>{i}</td></tr>' for i in numbers]) + '</table>'",
Expand Down
36 changes: 30 additions & 6 deletions tests/integration/mock/CodeActAgent/test_edits/prompt_001.log
Original file line number Diff line number Diff line change
Expand Up @@ -54,11 +54,11 @@ create_file(filename: str) -> None:
Args:
filename: str: The name of the file to create.

edit_file(file_name: str, to_replace: str, new_content: str) -> None:
edit_file_by_replace(file_name: str, to_replace: str, new_content: str) -> None:
Edit a file. This will search for `to_replace` in the given file and replace it with `new_content`.
Every *to_replace* must *EXACTLY MATCH* the existing source code, character for character, including all comments, docstrings, etc.
Include enough lines to make code in `to_replace` unique. `to_replace` should NOT be empty.
`edit_file` will only replace the *first* matching occurrences.
`edit_file_by_replace` will only replace the *first* matching occurrences.
For example, given a file "/workspace/example.txt" with the following content:
```
line 1
Expand All @@ -67,7 +67,7 @@ edit_file(file_name: str, to_replace: str, new_content: str) -> None:
line 3
```
EDITING: If you want to replace the second occurrence of "line 2", you can make `to_replace` unique:
edit_file(
edit_file_by_replace(
'/workspace/example.txt',
to_replace='line 2
line 3',
Expand All @@ -83,7 +83,7 @@ edit_file(file_name: str, to_replace: str, new_content: str) -> None:
line 3
```
REMOVAL: If you want to remove "line 2" and "line 3", you can set `new_content` to an empty string:
edit_file(
edit_file_by_replace(
'/workspace/example.txt',
to_replace='line 2
line 3',
Expand All @@ -94,6 +94,30 @@ edit_file(file_name: str, to_replace: str, new_content: str) -> None:
to_replace: str: The content to search for and replace.
new_content: str: The new content to replace the old content with.

edit_file_by_line(file_name: str, start: int, end: int, new_content: str) -> None:
Edit a file.
Replaces in given file `file_name` the lines `start` through `end` (inclusive) with the given text `content`.
If a line must be inserted, an already existing line must be passed in `content` with new content accordingly!
For example, given a file "/workspace/example.txt" with the following content (line numbers are included for clarity):
```
1|line 1
2|line 2
3|line 3
```
If you want to replace the second line, you can call:
edit_file_by_line('/workspace/example.txt', 2, 2, 'new line')
This will give the following result:
```
1|line 1
2|new line
3|line 3
```
Args:
file_name: str: The name of the file to edit.
start: int: The start line number. Must satisfy start >= 1.
end: int: The end line number. Must satisfy start <= end <= number of lines in the file.
content: str: The content to replace the lines with.

insert_content_at_line(file_name: str, line_number: int, content: str) -> None:
Insert content at the given line number in a file.
This will NOT modify the content of the lines before OR after the given line number.
Expand Down Expand Up @@ -153,7 +177,7 @@ parse_pptx(file_path: str) -> None:
Args:
file_path: str: The path to the file to open.

Please note that THE `edit_file` and `insert_content_at_line` FUNCTIONS REQUIRE PROPER INDENTATION. If the assistant would like to add the line ' print(x)', it must fully write that out, with all those spaces before the code! Indentation is important and code that is not indented correctly will fail and require fixing before it can be run.
Please note that THE `edit_file_by_replace`, `edit_file_by_line` and `insert_content_at_line` FUNCTIONS REQUIRE PROPER INDENTATION. If the assistant would like to add the line ' print(x)', it must fully write that out, with all those spaces before the code! Indentation is important and code that is not indented correctly will fail and require fixing before it can be run.

Responses should be concise.
The assistant should attempt fewer things at a time instead of putting too many commands OR too much code in one "execute" block.
Expand Down Expand Up @@ -341,7 +365,7 @@ USER:
ASSISTANT:
I should edit the file to display the numbers in a table format. I should include correct indentation. Let me update the file:
<execute_ipython>
edit_file(
edit_file_by_replace(
'app.py',
to_replace=" return str(numbers)",
new_content=" return '<table>' + ''.join([f'<tr><td>{i}</td></tr>' for i in numbers]) + '</table>'",
Expand Down
48 changes: 35 additions & 13 deletions tests/integration/mock/CodeActAgent/test_edits/prompt_002.log
Original file line number Diff line number Diff line change
Expand Up @@ -54,11 +54,11 @@ create_file(filename: str) -> None:
Args:
filename: str: The name of the file to create.

edit_file(file_name: str, to_replace: str, new_content: str) -> None:
edit_file_by_replace(file_name: str, to_replace: str, new_content: str) -> None:
Edit a file. This will search for `to_replace` in the given file and replace it with `new_content`.
Every *to_replace* must *EXACTLY MATCH* the existing source code, character for character, including all comments, docstrings, etc.
Include enough lines to make code in `to_replace` unique. `to_replace` should NOT be empty.
`edit_file` will only replace the *first* matching occurrences.
`edit_file_by_replace` will only replace the *first* matching occurrences.
For example, given a file "/workspace/example.txt" with the following content:
```
line 1
Expand All @@ -67,7 +67,7 @@ edit_file(file_name: str, to_replace: str, new_content: str) -> None:
line 3
```
EDITING: If you want to replace the second occurrence of "line 2", you can make `to_replace` unique:
edit_file(
edit_file_by_replace(
'/workspace/example.txt',
to_replace='line 2
line 3',
Expand All @@ -83,7 +83,7 @@ edit_file(file_name: str, to_replace: str, new_content: str) -> None:
line 3
```
REMOVAL: If you want to remove "line 2" and "line 3", you can set `new_content` to an empty string:
edit_file(
edit_file_by_replace(
'/workspace/example.txt',
to_replace='line 2
line 3',
Expand All @@ -94,6 +94,30 @@ edit_file(file_name: str, to_replace: str, new_content: str) -> None:
to_replace: str: The content to search for and replace.
new_content: str: The new content to replace the old content with.

edit_file_by_line(file_name: str, start: int, end: int, new_content: str) -> None:
Edit a file.
Replaces in given file `file_name` the lines `start` through `end` (inclusive) with the given text `content`.
If a line must be inserted, an already existing line must be passed in `content` with new content accordingly!
For example, given a file "/workspace/example.txt" with the following content (line numbers are included for clarity):
```
1|line 1
2|line 2
3|line 3
```
If you want to replace the second line, you can call:
edit_file_by_line('/workspace/example.txt', 2, 2, 'new line')
This will give the following result:
```
1|line 1
2|new line
3|line 3
```
Args:
file_name: str: The name of the file to edit.
start: int: The start line number. Must satisfy start >= 1.
end: int: The end line number. Must satisfy start <= end <= number of lines in the file.
content: str: The content to replace the lines with.

insert_content_at_line(file_name: str, line_number: int, content: str) -> None:
Insert content at the given line number in a file.
This will NOT modify the content of the lines before OR after the given line number.
Expand Down Expand Up @@ -153,7 +177,7 @@ parse_pptx(file_path: str) -> None:
Args:
file_path: str: The path to the file to open.

Please note that THE `edit_file` and `insert_content_at_line` FUNCTIONS REQUIRE PROPER INDENTATION. If the assistant would like to add the line ' print(x)', it must fully write that out, with all those spaces before the code! Indentation is important and code that is not indented correctly will fail and require fixing before it can be run.
Please note that THE `edit_file_by_replace`, `edit_file_by_line` and `insert_content_at_line` FUNCTIONS REQUIRE PROPER INDENTATION. If the assistant would like to add the line ' print(x)', it must fully write that out, with all those spaces before the code! Indentation is important and code that is not indented correctly will fail and require fixing before it can be run.

Responses should be concise.
The assistant should attempt fewer things at a time instead of putting too many commands OR too much code in one "execute" block.
Expand Down Expand Up @@ -341,7 +365,7 @@ USER:
ASSISTANT:
I should edit the file to display the numbers in a table format. I should include correct indentation. Let me update the file:
<execute_ipython>
edit_file(
edit_file_by_replace(
'app.py',
to_replace=" return str(numbers)",
new_content=" return '<table>' + ''.join([f'<tr><td>{i}</td></tr>' for i in numbers]) + '</table>'",
Expand Down Expand Up @@ -388,19 +412,17 @@ Fix typos in bad.txt. Do not ask me for confirmation at any point.

----------

Sure! Let's start by opening the `bad.txt` file to identify the typos.
Sure! Let's start by opening the `bad.txt` file to identify and fix the typos.
<execute_ipython>
open_file('bad.txt')
find_file('bad.txt')
</execute_ipython>

----------

OBSERVATION:
[File: /workspace/bad.txt (4 lines total)]
1|This is a stupid typoo.
2|Really?
3|No mor typos!
4|Enjoy!
[Found 1 matches for "bad.txt" in ./]
./bad.txt
[End of matches for "bad.txt" in ./]


ENVIRONMENT REMINDER: You have 8 turns left to complete the task.
Loading