Add semantic highlighting support via the LSP

[rcjsuen]

This is a preliminary implementation of semantic highlighting for Dockerfiles. This pull request intends to address #1840 which in a way is itself a collection of many long standing bug reports about malformed syntax highlighting in Dockerfiles.

To test this feature, simply enable semantic highlighting via the settings (or if you're using the JSON settings editor, the key is "editor.semanticHighlighting.enabled").

Known issues:
There are a few issues that will be fixed in the next release of the language server. You can copy/paste the Dockerfile below into VS Code to see what I mean.

1. parameter that precedes an escaped string does not get coloured.
2. quoted string values of a variable do not get highlighted as a string
3. variables within quoted strings are not coloured

FROM alpine
# second abc is white compared to the first abc that is coloured correctly
RUN echo abc"e"
RUN echo abc\'e\'
# variable value in an enclosed string does not get coloured
ENV abc="def"
# $abc as a variable reference should be coloured differently inside the string
RUN echo "$abc"

Q&A:

1. How do I enable/disable semantic highlighting only for Dockerfiles?

   From what I understand, you must use the JSON settings editor for this. Use the following JSON snippet to toggle the setting as you please. I'm not sure how you can achieve this with the GUI editor.

   "[dockerfile]": {
       "editor.semanticHighlighting.enabled": true
   }

2. Why does your online Dockerfile editor seem to colour more things than VS Code?

   I specifically made changes to my own editor so that it would provide custom colours for various situations. Based on the semantic highlighting wiki page, there appears to be a way to add custom colour mappings though I may be misunderstanding what is written on the page. I think this will require more investigation and can be addressed in a different pull request so I hope that is not a blocker for this pull request.

3. What is the ETA for the next release of the Dockerfile language server that will address the aforementioned bugs?

   I don't have a timeline to share at the moment.

[bwateratmsft]

I tested out some scenarios, it looks good. A couple of questions for @rcjsuen:

1. Not sure how solvable this is, but images from not-Docker-Hub offer a link that will 404 when you open it, for example, the following links to https://hub.docker.com/r/dotnet/core/aspnet/ (which will 404)
   

2. "bar" is just a string here, right? Shouldn't it get the orange-ish highlighting? (Or is that the bug # 2 you mention?)
   

3. Specifically in your example Dockerfile showing some of the bugs, tab completion seems to stop working at the bottom; for example, HEALTHCHECK doesn't offer tab-completion until the entire HEALTHCHECK word is typed in. In other Dockerfiles, it does still work.
   
   (You can see the spaces at the end where I hit tab)

[rcjsuen]

@bwateratmsft Thanks for taking a look.

1. Yeah, I haven't really decided what the best course of action is here as some people just use public images on Docker Hub whereas some others only use private container registries.
2. Yes, that is the same bug that I've pointed out.
3. Good catch! That seems to be a parsing error related to line 4 (RUN echo abc\'e\'). Removing that line seems to fix the problem. I'll have to do some poking around to see what the issue is there as other features like "hovering over a keyword" or "highlighting referenced variables" seems to be unaffected. 🤔

[rcjsuen]

@bwateratmsft Another interesting thing is that if you type HEALTH and then use Control+Space to trigger suggestions, it does pop up!

Looks like there might be something else that may be amiss...?

[bwateratmsft]

I guess also, # 1 is not a new issue, and nobody has complained yet...

[rcjsuen]

@bwateratmsft I was able to reproduce the issue with a simple piece of code using the Monaco Editor so I think it may be related to the grammar of file? The system seems convinced that the text cursor is inside a string so it doesn't appear to want to invoke code completion.

1. Open the Monaco Editor Playground.
2. Copy/paste the code below on the left hand side.
3. Click the 'Run' button.
4. Open the debugger and dock it to the side. Open up the console.
5. Follow the instructions typing on the 4th, 6th, and 7th line. Observe the differences in behaviour with regards to when something gets printed to the console or not.

const content = 
`# type on line 4, it works
# type on line 6, it fails
# type on line 7, it works

RUN \\'e\\'

`;
const language = "dockerfile";
monaco.languages.registerCompletionItemProvider(language, {
    provideCompletionItems: function(_model, position) {
        console.log(`provideCompletionItems called on line ${position.lineNumber} character ${position.column}`)
        return {
            suggestions: []
        };
    }
});

monaco.editor.create(document.getElementById("container"), {
    value: content,
    language: language
});

[rcjsuen]

I guess also, # 1 is not a new issue, and nobody has complained yet...

Correct, it's not a new issue and it has been like that since I first implemented Docker Hub linking support.

Regarding the code completion issue, I have opened microsoft/vscode#95924 and microsoft/monaco-editor#1935.

[rcjsuen]

@bwateratmsft I decided to just pull the trigger and cut a release so the three aforementioned semantic highlighting bugs should be fixed now.

[bwateratmsft]

Awesome, thanks! I verified and all three are fixed.

[rcjsuen]

@bwateratmsft FYI regarding the code completion bug you discovered.

[bwateratmsft]

@rcjsuen Thanks!