hygiene

gistfile1.js

var random = function(seed) { /* ... */ } 
let m = macro { 
  rule {()} => { 
    var n = random(42); // ... 
  } 
}

So, in that example, I understand why n needs to be renamed, to not conflict/overlap with any other identifier of same name in the whole program. But I don't understand why random needs to get renamed here. Can you illustrate why?

Even when we consider:

m();

function foo() {
   var random = function() { return 42; };
   m();
}

I still don't understand why random has to be renamed? IIUC, the above would be re-written to:

var n$1 = random(42);

function foo() {
   var random = function() { return 42; };
   var n$2 = random(42);
}

It's quite clear why n gets renamed, but why does random identifier need to be renamed here? Lexical scoping rules take care of making sure the correct random is referenced.

Ah! I think my formatting has confused things. Let me try and make it clearer:

var random = function(seed) { /* ... */ }
let m = macro { 
  rule {()} => { 
    var n = random(42); 
    // ... 
  } 
}
function foo() {
  var random = "a random string";
  m()
}

if we do no renaming this expands to:

var random = function(seed) { /* ... */ }
function foo() {
  var random = "a random string";
  var n = random(42); // bound to the string instead of the function!
  // ...
}

The issue is that the random referenced by the macro needs to refer to the top-level random function definition. But in the scope of the foo function it refers to the random string.

Really? That's quite bizarre. That's the opposite of the way lexical scoping works. I would expect that wherever I expand a macro, the code in it takes on the lexical scoping characteristics of the place in which I expanded it. I would not expect it to inherit scoping from where the macro was declared. That's super strange and unexpected behavior to me.

But at least that explains my misunderstanding of your post.

I think it makes perfect sense when you think about it. Macros exist in their own "lexical scope" just like a function. Think about it from the perspective of a macro author. When writing the macro definition you don't want to think about whether or not the calling context has rebound something you are using. That completely breaks any hope of creating a macro as an abstraction.

The key idea here is that lexical scope must be extended to include macro definitions. It's not just a runtime thing...also needs to hold at compile/macro-expansion time.

Or the reverse thinking, from the world of prototypes, behavior delegation and this re-assignment, which is that I want to author a macro that allows the user of the macro to be able to provide their own definition for a function, if they didn't want to use the otherwise "default" one provided in the containing scope.

I just associate macros in my mind with find-n-replace type actions, code expansions, not as creating and using their own scopes as functions.

Not saying your way is "wrong", just saying it's a total surprise. Perhaps you might want to make clear how these macros do associate to their own scope rather than acting like find-n-replace.

What would you think about having both "lexical macros" (which work like the current ones) and "non-lexical macros" (like I assumed, which don't keep any of their own lexical scope and just act like a find-n-replace expansion)? You could just add a flag or indicator of some sort to your macro to indicate "non-lexical" behavior or whatever? I think that kind of flexibility would make macros even more broadly useful.

So what you are asking for ("non-lexical macros") are unhygienic macros. You can build them if you'd like. Sweet.js has features that let you specifically write them (see the homepage for a mini-tutorial on breaking hygiene, it shouldn't be very difficult to define a macroUnhygienic if you really want). But in general they are very very bad and I don't think it's a good idea to include a macro form that does them "by default" with sweet.js.

Here's the thing. Hygiene (of the sort I've described here) is essential in building macro abstractions that play nice together and work in any context. Breaking hygiene and doing a "find-n-replace" expansion means that the macro author must reason about every possible context in which the macro could be invoked. This is bad in ways you might not have thought about yet. function could be rebound, var could be rebound. How can you write a robust macro when you can't even be sure that function means what you thought it meant?

This isn't equivalent to late-binding this. When you late-bind this there is an implicit contract on what is expected to go on the object/prototype chain. You are explicitly calling out the delegation that is occurring by using this and prototypes. Not so with an unhygienic macro because the potential delegation is everything in scope. This is impossible to reason about.

A hygienic macro system like sweet.js by design and by default protects the abstraction capability of macros. It provides escape hatches when you really need to break hygiene but this should only be done when you absolutely must.

@disnet

I suppose what I had in my mind was a mixture between the two. I was thinking a macro whose internal declarations were in fact hygenic (didn't leak out), but whose references to non-internal (aka, external) variables were not closured and instead just adopted the scope where they were expanded.

That model would prevent a macro that you use from accidentally overwriting something in your scope that is unexpected, but still let you provide in your own scope any other scoped bindings for any external references it may have.

To put it in the parlance of your post about variable renaming, you'd rename any variables that are declared inside a macro, but you wouldn't rename any other references which weren't declared there.

That would sort of turn var into a let that bound itself to the scope of the macro's (implicit) block, while still letting in scope from outside the macro for external references.

I can think of quite a few cases where I would use such a model.

In fact, instead of renaming macro-declared variables to prevent overlap, you could just wrap (while expanding) the body of the macro code in a stand-alone { .. } block and change vars to lets, and that would create hygiene (aka, prevent any inner-to-outer leakage) in the same way as renaming, but with way less work.