Docker Labs: GenAI | No. 5

Last time we discussed linters, we talked about running ESLint with an AI assistant. In this doc, we want to cover AI-assisted violation resolutions. Before an AI assistant can take action on violations via the editor or git, we need to first answer the question: Can an LLM generate resolutions for ESLint violations with the right knowledge and tools?

Fix any lint violations you can in my project.        

ESLint Violations

In order to resolve a violation, the LLM needs information about the violation from the linter, and some context about the source code causing it. There may also be a level of human supervision necessary. We want to optimize to shrink context, because models both limit context windows, and yield better results with less context. This makes us wonder, “what is the minimum amount of context and supervision necessary to resolve a lint violation?”

For example: without human instruction, our assistant just can’t know how to balance our parentheses:

// export const dockerfileRegex = new RegExp('(.*\\.?Dockerfile', 'i');
// Goal: new RegExp('(.*)\\.?Dockerfile', 'i');

Your `dockerfileRegex` regular expression is currently unterminated. This means the group you started at `(.*\\.?Dockerfile` doesn't have an ending bracket. Here is a fixed version of it:

```typescript
export const dockerfileRegex = new RegExp('(.*\\.?Dockerfile)', 'i');
```

I added a ")" to close the group.        

Focusing on certain violations

We want to narrow down the list of ESLint rules to violations which the LLM can help with.

In this table, we have compiled every ESLint rule that

• Is recommended
• Is not auto-fixable by ESLint itself

And we want rules which

• Do not require supervision to make reasonable fixes

Which yields 4 total groups of violations:

Group 1 (No context needed from the surrounding source code or supervision needed)

Scripts can be applied to fix these violations.

Group 2 (No context needed from the surrounding source code, but these might be mistakes which necessitate the LLM to evaluate some of the code)

LLM can generate scripts to resolve these violations, but we would be assuming violations are mistakes. There are some violations which the LLM can evaluate on its own.

Group 3 (Context is needed from surrounding source code, but no supervision is necessary)

This is the best opportunity for our LLM to provide help. We can use tools to pull the right context in.

Group 4 (Context is needed from surrounding source code and supervision might be needed)

• Invalid Regex
• Unsafe optional chaining
• Constant condition

Depends a lot on the exact situation, but LLMs may be useful?

Thankfully, nearly all of the violations could have reasonable fixes applied without supervision. These are the violations which we will focus on.

Initial Prompts

First, we create the prompts to attempt to fix ESLint violations.

You are an AI assistant who specializes in resolving lint violations in projects. Use the tools available to quickly take action and be very brief.

1. Run linter.
2. Evaluate total violations.
   // What to do?        

Unfortunately, we run into a roadblock when it comes to learning about lint violations from our ESLint tool. When using summary output, we don’t have enough information to know what we’re fixing. However, when using JSON output, we found that as few as 100 violations in a project caused ESLint to send over 10,000 characters over standard out. That would be a problem since many models at current limit us to 4-8k tokens. We need a way to persist this large output but without consuming tokens.

Artifacts

While we want to use ESLint, it can easily balloon an AI conversation past the model’s context size. To address this, we modified the ESLint tool to write eslint.json to a new ephemeral Docker volume. Reading & writing to this volume serves as another conversation between tools, but sandboxed away from the LLM’s context-sensitive function call output. We added tools to consume the ESLint artifact in different ways depending on the number of violations summarized.

- name: parse_lint_results
??description: Loads lint violations grouped by type.
??parameters:
????type: object
??????properties:
????????outputLevel:
?????? ? type: string
?????? ? description: Supports condensed or complaints
??container:
????image: vonwig/read_eslint        

And now we can use this tool to finish our prompt.

You are an AI assistant who specializes in resolving lint violations in projects. Use the tools available to quickly take action and be very brief.

1. Run lint. // Will write /thread/eslint.json
2. Evaluate total violations. // Will read /thread/eslint.json
??<10 violations: Parse output with complaints output.
??10+ violations: Parse output with condensed output.
3. Fix the violations using the following steps:

## Condensed:
{>fixing}        

With a user prompt of

Fix any lint violations you can using the tools provided.        

From here, we need to include our research from earlier. We picked a violation from the list we want to focus on, and ran the conversation:

1. Get the line Use read_eslint with the `path` arg to get all of the violations for a file.
2. Make the correction Respond in the following format:

```json
{
"start": [1, 4],
"end": [2, 4],
"edit": "Lorem ipsum"
}
```

Once you have fixed one file, move on to the next.

You are able to fix the following violations:
no-async-promise-executor
...        

Yields

The violation is located in the file src/utils/promiseUtils.ts at line 30-31. The current code is:

```javascript
return new Promise(async (resolve, reject) => {
??...
});
```

To resolve the no-async-promise-executor violation, remove the async keyword from the `Promise` executor function. The corrected code would look like:

```javascript
return new Promise((resolve, reject) => {
??...
});
```        

Unfortunately, we’re still not seeing a good result. Without the violating context from the code, again, our assistant can’t provide an actionable fix. So we need to make a tool to get that.

Tree Sitter

Without the right tools, the LLM is, at best, vague and helpful - and at worse quite liable to hallucinate. We need tools to let the LLM be actionable here. Tree Sitter is a popular open source tool we can use to parse a variety of languages into Abstract Syntax Trees, or AST’s. The powerful part of Tree Sitter we want to take advantage of is the community of grammar authors providing support for almost any language our assistant could encounter. We have Dockerized this tool to get context given a file path and line number.

Results

We have trimmed debug finish responses from the payload, but kept function calls.

## ROLE assistant
...({:function {:name run_lint, :arguments {}}, :id call_MaK7CGpOISktz1fZP3GS7m4H})

## ROLE assistant
.........................({:function {:name run-eslint, :arguments {
??"args": ".",
"version": 8,
"outputLevel": "summary"
}}, :id call_RHX2hztWvghxbyZx5cvSXgMU})

## ROLE tool (run-eslint)
Running npx with args: eslint@8 --format json .
Peparing output
Found 41 violations in 271 files.

## ROLE tool (run_lint)
I ran ESLint (version 8) on your entire project. Here are the results:

- 271 files were checked for violations
- Out of these, 41 files had one or more violations

## ROLE assistant
.............({:function {:name parse_lint_results, :arguments {
??"outputLevel": "condensed"
}}, :id call_sYHP8tXuKbXGaXdNCdO3W5kU})

## ROLE tool (parse_lint_results)
{ ? "@typescript-eslint/no-floating-promises": [ "src/commands/help.ts",
"src/commands/registerWorkspaceCommand.ts" ? ], ? "@typescript-eslint/no-explicit-any": [ "src/commands/registerLocalCommand.ts" ? ], ? "@typescript-eslint/naming-convention": [ "src/constants.ts" ? ], ? "eqeqeq": [ "src/debugging/DockerDebugConfigurationProvider.ts", "src/debugging/DockerServerReadyAction.ts", "src/runtimes/files/ContainerFilesProvider.ts", "src/tree/LocalRootTreeItemBase.ts" ? ], ? "@typescript-eslint/no-unused-vars": [ "src/extension.ts", "src/scaffolding/wizard/netCore/NetCoreGatherInformationStep.ts" ? ], ? "@typescript-eslint/semi": [ "src/utils/registryExperience.ts" ? ] }

## ROLE assistant
...............({:function {:name violations_for_file, :arguments {
??"path": "src/commands/help.ts"
}}, :id call_fAN5kTrXndaNs7Ix4iH51E4T})

## ROLE tool (violations_for_file)
Getting eslint.json for path: src/commands/help.ts
ESLint violations for src/commands/help.ts:
{ "filePath": "src/commands/help.ts", "messages": [ { "ruleId": "@typescript-eslint/no-floating-promises", "severity": 1, "message": "Promises must be awaited, end with a call to .catch, end with a call to .then with a rejection handler or be explicitly marked as ignored with the void operator.", "line": 35, "column": 5, "nodeType": "ExpressionStatement", "messageId": "floatingVoid", "endLine": 35, "endColumn": 100] }

## ROLE assistant
......................({:function {:name run_tree_sitter, :arguments {
??"path": "src/commands/help.ts",
??"line": 35
}}, :id call_cKnNRQs5hmMsTszTfDnwfK12})

## ROLE tool (run_tree_sitter)
{
??offending_line: '}',
??line_node: SyntaxNode {
type: },
startPosition: {row: 35, column: 0},
endPosition: {row: 35, column: 1},
childCount: 0,
??},
??parent: SetNode {
type: set,
startPosition: {row: 33, column: 45},
endPosition: {row: 35, column: 1},
childCount: 4,
??},
??parent_text: '{\n' +
" vscode.env.openExternal(vscode.Uri.parse('https://github.com/microsoft/vscode-docker/issues'));\n" +
'}'
}

## ROLE assistant
The violation on line 35 of the `src/commands/help.ts` file is due to a floating promise. To resolve this, add the void operator to the beginning of the line as below:
```js
void vscode.env.openExternal(vscode.Uri.parse('https://github.com/microsoft/vscode-docker/issues'));
```.        

This result demonstrates that the LLM is able to orchestrate a small code improvement using the right tools.?

Failures

Unfortunately, it terminates after coming to the end of the first violation rather than continuing onto the next. This seems to be difficult to change, even trying two other combinations of prompts failed to get a loop going.

Fix any lint violations you can using the tools provided.
After resolving one violation, move onto the next.        

Once you have fixed one file, move on to the next.        

Successful Prompt

However, what we did find was that specifying the lint violation to focus on did work

Please only fix no-unused-vars        

Yields

## ROLE assistant
The violation in the file "src/extension.ts" is a declared variable 'unuseVar' at line 41, which is not used anywhere. The fix would be the removal of that line..nil        

Here are a few interesting violations we tested:

Overall, we can see that our assistant, when given the right tools to find and fix violations, is able to make good code change suggestions for even difficult lint violations. We welcome you to check out the code and test with different violations which we didn’t cover.

Summary

We continue to investigate the use of tools for gathering context and improving suggestions. In this edition, we have looked at how AI assistants can provide significant value to developers by?

• Use a linter to learn about violations in a TS project without Node, NPM, or ESLint
• Leverage an ephemeral volume to store large context without using tokens
• Use tree_sitter to load precise, relevant code context from a project
• Generate fixes for violations in a TS project

As always, feel free to follow along in our new public repo and please reach out. Everything we’ve discussed in this blog post is available for you to try out on your own projects.

For more on what we're doing at Docker,subscribe to our newsletter.