Highlighting spaces and tabs

[thaliaarchi]

I'd like to add Linguist support for the Whitespace programming language. It should meet the 200-repo requirement and there is precedent for including other esolangs in Linguist (e.g. Brainfuck, Befunge, LOLCODE).

The only characters in Whitespace are space, tab, and line feed, but since they are non-printable, there is no text to highlight. There are two existing TextMate grammars for Whitespace: veniversum/language-whitespace and bvisness/whitespace. @veniversum appears to solve this by replacing space, tab, and line feed with s, t, and ¬, respectively, then coloring those. @bvisness creates a theme that colors the background color of those tokens. Is it possible to set background colors for GitHub syntax highlighting?

[bvisness]

It's one thing to support syntax highlighting for esolangs, but another thing to build new features specifically for esolangs. That seems like a clear waste of time.

Besides, the only reason I did anything with Whitespace was because I was in college, it was on the Wikipedia page for esolangs, and it sounded funny. I never even got it running. Nothing I have done is a worthwhile precedent for serious work.

And besides, isn't whitespace funnier when you can't see the contents of the file? 😛

[thaliaarchi]

Well as it turns out, I've seen several schools that use Whitespace in their compilers courses, so it's at least useful for pedagogical reasons.

[Nixinova]

Here's the CSS classes that have a background color on GitHub:

• .pl-ii {color:#fafbfc;background-color:#b31d28} - #b31d28
• .pl-c2 {color:#fafbfc;background-color:#d73a49} - #d73a49
• .pl-md {color:#b31d28;background-color:#ffeef0} - #ffeef0
• .pl-mi1{color:#22863a;background-color:#f0fff4} - #f0fff4
• .pl-mc {color:#e36209;background-color:#ffebda} - #ffebda
• .pl-mi2{color:#f6f8fa;background-color:#005cc5} - #005cc5

So you (or I) could create a grammar that uses .pl-ii (token invalid.illegal) and .pl-mi2 (not sure what token) and display well on GitHub.

[Nixinova]

Here you go, a simple grammar for highlighting WhiteSpace files:

• Lightshow
  

[thaliaarchi]

I checked each of the other tokens. For, carriage-return, it renders a ^M for each instance. The best case for this grammar would be to have a middle dot · rendered for space and an arrow → for tab.

.pl-ii  invalid.illegal
.pl-c2  carriage-return ^M
.pl-md  markup.deleted
.pl-mi1 markup.inserted
.pl-mc  markup.changed
.pl-mi2 markup.ignored

[Nixinova]

This is all GitHub gives you. The only class that adds a character is carriage-return, '^M'.

[thaliaarchi]

It's probably too much to ask for two more classes then, right?

[Alhadis]

Bookmark this, and bookmark it well. This is an exhaustive list not just of the TextMate-scope/CSS-class mappings, but also of their current styling (which IS subject to wanton, random changes at the whims of GitHub's designer team, JSYK).

So if these were to be used, the grammar tokens couldn't be keyword.control.space and keyword.control.tab, but rather invalid.illegal and <token corresponding to .pl-mi2>?

There's actually a trick you can use if you're willing to complicate your grammar a little. It exploits the fact that TextMate scope selectors are order dependent; i.e., unlike CSS selectors, a TextMate scope of keyword.operator isn't the same as a CSS class selector for .operator.keyword (although something like keyword.operator.whatever will work).

Hence, you can "hide" a token from GitHub's highlighter by prefixing it with a scope it doesn't recognise. For example:

tab:
	name: "markup.inserted.whitespace"
	match: "\t"
	captures:
		0: name: "_.keyword.control.tab.whitespace"

space:
	name: "markup.changed.space.whitespace"
	match: " "
	captures:
		0: name: "_.keyword.control.space.whitespace"

Here's a preview with an unabridged example.

Show grammar source

name: "Whitespace"
scopeName: "source.whitespace"
fileTypes: [".ws"]
patterns: [include: "#main"]

repository:
	main:
		patterns: [
			{include: "#tab"}
			{include: "#space"}
		]

	tab:
		name: "markup.ignored.whitespace"
		match: "\t"
		captures:
			0: name: "_.keyword.control.tab.whitespace"

	space:
		name: "invalid.illegal.space.whitespace"
		match: " "
		captures:
			0: name: "_.keyword.control.space.whitespace"

[lildude]

I'd like to add Linguist support for the Whitespace programming language. It should meet the 200-repo requirement and there is precedent for including other esolangs in Linguist (e.g. Brainfuck, Befunge, LOLCODE).

Keep in mind that we now have minimum usage requirements which may not have been in place when other esolangs were added. Given Whitespace has nothing to differentiate it content-wise you're only going to be able to rely on the extension or specific filename to prove usage.

[thaliaarchi]

There actually are some distinguishing features of Whitespace syntax: most lines in Whitespace have tabs after spaces, whereas tabs in other languages almost exclusively appear at the beginning of a line, only rarely after spaces, such as in coding styles that uses tabs for aligning tables. There's also very low prevalence of symbols, unless the program is a polyglot, in which case, it would make more sense to highlight the other language anyways.

[lildude]

Sorry, I should have made clear that you won't be able to demonstrate this with GitHub's Search.

[thaliaarchi]

I was afraid that would be the case. Is there a way to download all .ws files for local processing?

[lildude]

Unfortunately, not any more. The old internal method we used to use has been removed so you'd need to craft something yourself using the API. Warning: you're almost certain to hit rate limiting so this could take a very very long time.