PART 3: Iterator block implementation details: auto-generated state machines

Question

private bool MoveNext()
{
    switch (this.<>1__state)
    {
        case 0:
            this.<>1__state = -1;
            Console.WriteLine(Test.Padding + "First line of GetNumbers()");
            Console.WriteLine(Test.Padding + "Just before yield return 0");
            this.<>2__current = 10;
            this.<>1__state = 1;
            return true;

        case 1:
            this.<>1__state = -1;
            Console.WriteLine(Test.Padding + "Just after yield return 0");
            Console.WriteLine(Test.Padding + "Just before yield return 1");
            this.<>2__current = 20;
            this.<>1__state = 2;
            return true;

        case 2:
            this.<>1__state = -1;
            Console.WriteLine(Test.Padding + "Just after yield return 1");
            break;
    }
    return false;
}

This is a very simple case, of course - we just start off in state 0, then 1, then 2, then -1 (although we're briefly in -1 while the code executes each time, of course). Let's take a look at our first example again. Here's the original code:

using System;
using System.Collections;

class Test
{
    static IEnumerator GetCounter()
    {
        for (int count = 0; count < 10; count++)
        {
            yield return count;
        }
    }
}

and here's the MoveNext() method, just as we saw before:

private bool MoveNext()
{
    switch (this.<>1__state)
    {
        case 0:
            this.<>1__state = -1;
            this.<count>5__1 = 0;
            while (this.<count>5__1 < 10)
            {
                this.<>2__current = this.<count>5__1;
                this.<>1__state = 1;
                return true;
            Label_004B:
                this.<>1__state = -1;
                this.<count>5__1++;
            }
            break;

        case 1:
            goto Label_004B;
    }
    return false;
}

Yes, it's using a goto statement to leap from one case to half-way through another. Ouch. Don't forget that this is only generated code though. When you think about it, for loops and while loops are really just nice wrappers round comparisons and jumps. We don't really care how nasty this code is in terms of readability, so long as it works and performs well. The simplest way for the C# compiler team to achieve those two goals was to model it all with switch and goto.

I don't intend to explain how all the different transformations take place. I'll look at the handling of finally blocks later on, and it's interesting to note that you can't yield (either to return or break) from a try block which has a catch block, or from any catch or finally block. Importantly, you can yield from a try block which only has a finally block. That means you can still use using statements, which can be very handy indeed. It can be quite interesting to experiment with different ways of making the code branch and loop, and then see what happens to the generated MoveNext() method. However, I couldn't do that exhaustively whereas you can experiment very easily. The simple examples above show the principle, along with the states involved. Let's move on to the next piece of state.

Local variables

Normal local variables are very simple in iterator blocks. They become instance variables in the iterator type, and are assigned meaningful values in the same way (and at the same point) that they'd be initialized in normal code. Of course, being instance variables there's no longer any meaningful idea of them being definitely assigned, but the normal compilation rules will prevent you from seeing their default values (unless you deliberately mess with the state described above, using reflection). All of this can be seen in the earlier example with the <count>5__1 variable.

The nature of local variables means that creating an iterator instance doesn't require any extra information about the variables themselves - any initial value will be set within the course of the code. That initial value may rely on non-local variables of course, which brings me to the final type of state.

Parameters and this

Methods implemented with iterator blocks can take parameters, and if they're instance methods they can use this as well. Any reference to an instance variable of the type containing the iterator block is effectively just using this and then navigating from that reference to the variable. Here's an example containing both a method parameter (max and a reference to an instance variable min - I've qualified the instance variable with this just to make it clear.

using System;
using System.Collections.Generic;

class Test
{
    int min;
    
    public Test(int min)
    {
        this.min = min;
    }
    
    public IEnumerator<int> GetCounter(int max)
    {
        for (int count = this.min; count < max; count++)
        {
            yield return count;
        }
    }
}

Note that it's returning an IEnumerator rather than an IEnumerable. This makes more of a difference when using parameters and this than it does otherwise, as we'll see soon. Here are the interesting bits of the generated code:

internal class Test
{
    private int min;

    public Test(int min)
    {
        this.min = min;
    }

    public IEnumerator<int> GetCounter(int max)
    {
        <GetCounter>d__0 d__ = new <GetCounter>d__0(0);
        d__.<>4__this = this;
        d__.max = max;
        return d__;
    }

    private sealed class <GetCounter>d__0 : IEnumerator<int>, IEnumerator, IDisposable
    {
        private int <>1__state;
        private int <>2__current;
        public Test <>4__this;
        public int <count>5__1;
        public int max;

        public <GetCounter>d__0(int <>1__state)
        {
            this.<>1__state = <>1__state;
        }

        private bool MoveNext()
        {
            switch (this.<>1__state)
            {
                case 0:
                    this.<>1__state = -1;
                    this.<count>5__1 = this.<>4__this.min;
                    while (this.<count>5__1 < this.max)
                    {
                        this.<>2__current = this.<count>5__1;
                        this.<>1__state = 1;
                        return true;
                    Label_0050:
                        this.<>1__state = -1;
                        this.<count>5__1++;
                    }
                    break;

                case 1:
                    goto Label_0050;
            }
            return false;
        }

        // Other methods as normal
    }
}

We've gained two extra fields in the iterator. One is just called max, and the other is <>4__this. Where the original code accesses min, the generated code accesses<>4__this.min - which it can do despite Test.min being private due to the fact that it's in a nested type.

The interesting and (to my mind) somewhat counterintuitive part is the way these extra fields are initialised. Personally, I would have added them as constructor parameters, making both of them private and <>4__this readonly too. Eric Lippert, from the C# team, has explained that the code which is responsible for this is the same code which hoists captured variables from closures - and those really do need to be public so that the original method can still get at them. So basically it's a code reuse issue rather than there being a sneaky reason why it couldn't have been done my preferred way. There's no real harm here, but I find this sort of thing fascinating :)

As it happens, our iterator block doesn't change the value of max - but it could do. Now suppose that instead of IEnumerator we were to return IEnumerable. Given that we want each of the iterators generated by calls to GetEnumerator() to use the original value of max, how does the compiler keep things in check? Here's the interesting subset of generated code (the source is the same as it was before, just with a change to the return type).

PART 4: Continuous soon