{
"posts": [
{
"id": "86",
"title": "Whence \n",
"date": "9.17.2024",
"tags": ["computers", "programming"],
"content": "<p>If you do <code>just foo</code>, the following <a href="https://github.com/casey/just/">justfile</a>
will write a single byte <code>0x0A</code> to a file named bar:</p>
<pre><code>x := "\n"
foo:
printf '{{x}}' > bar
</code></pre>
<p>Let's find out where that <code>0x0A</code> byte comes from.</p>
<p><code>just</code> is written in Rust, and the <code>just</code> parser has a function called
<code>cook_string</code>, which transforms a <code>just</code> string token containing escape
sequences into a UTF-8 string.</p>
<p>The code is here
<a href="https://github.com/casey/just/blob/ab7105afceabf7c3346e72eeafdaaf53ecbc0aa6/src/parser.rs#L731">here</a>.</p>
<p>With some irrelevant details elided, it looks like this:</p>
<pre><code>for c in text.chars() {
match state {
…
State::Backslash => {
match c {
'n' => cooked.push('\n'),
…
}
…
}
…
}
}
</code></pre>
<p>So <code>just</code> asks <code>rustc</code> to insert the result of evaluating the <em>Rust</em> <code>'\n'</code>
character escape. Let's take a look at how <code>rustc</code> handles <code>'\n'</code>.</p>
<p><code>rustc</code>'s escape code handling is in the lexer, in a function called
<code>scan_escape</code>, which is
<a href="https://github.com/rust-lang/rust/blob/e2dc1a1c0f97a90319181a721ab317210307617a/compiler/rustc_lexer/src/unescape.rs#L240">here</a>.</p>
<p>With some details removed:</p>
<pre><code>let res: char = match chars.next().ok_or(EscapeError::LoneSlash)? {
…
'n' => '\n',
…
};
</code></pre>
<p><code>rustc</code> is written in Rust and compiles itself, so somehow <code>rustc</code> is
delegating to <code>rustc</code> to figure out what <code>'\n'</code> means, which seems odd, to say
the least, and we still haven't seen the naked <code>0x0A</code> byte we're looking for.</p>
<p><code>rustc</code> wasn't always written in Rust though. Before it was self-hosted, early
versions were written in OCaml.</p>
<p>GitHub has old versions of the OCaml version of <code>rustc</code>, which handled
character escapes in the lexer
<a href="https://github.com/rust-lang/rust/blob/ef75860a0a72f79f97216f8aaa5b388d98da6480/src/boot/fe/lexer.mll#L342">here</a>.</p>
<pre><code>and char_escape = parse
…
| 'n' { end_char (Char.code '\n') lexbuf }
…
</code></pre>
<p>So <code>rustc</code> asks the OCaml compiler to insert the result of evaluating the
<em>OCaml</em> character escape <code>'\n'</code>. Which is totally reasonable, but still not a
<code>0x0A</code> in sight.</p>
<p>Going one step deeper, let's look the OCaml lexer
<a href="https://github.com/ocaml/ocaml/blob/4d6ecfb5cf4a5da814784dee7363a15ea278f324/lex/lexer.mll#L37">here</a>.</p>
<p>And finally, some clarity:</p>
<pre><code>let char_for_backslash = function
'n' -> '\010'
…
</code></pre>
<p>When the OCaml compiler sees <code>\n</code>, it inserts the result of evaluating the
OCaml character escape <code>\010</code>, which is a decimal character escape, and since
<code>0x0A</code> is 10, we finally have our byte value.</p>
<p>So when have a <code>\n</code> character escape in your justfile, the <code>just</code> binary
contains a <code>0x0A</code> byte in some form, which it will then write to your final
string.</p>
<p>That <code>0x0A</code> byte was put there by <code>rustc</code>, which contained it's <em>own</em> <code>0x0A</code>
byte somewhere in the binary, which was stuffed there by its <code>rustc</code> progenitor.</p>
<p><code>rustc</code> is currently at version 1.81.0, so this has happened at least 81 times
since <code>rustc</code> 1.0 was first released, and probably many more times than that
before 1.0, with <code>rustc</code>s furtively smuggling <code>0x0A</code> bytes from one to the
other, all the way back to when it was written in OCaml, when finally the first
<code>0x0A</code> byte was stuffed into a <code>rustc</code> binary by the OCaml compiler, which
evaluated it from a decimal character escape <code>'\010'</code>.</p>
<p><em>This post was inspired by another post about exactly the same thing. I
couldn't find it when I looked for it, so I wrote this. All credit to the
original author for noticing how interesting this rabbit hole is.</em></p>"
}
]
}