Buddhists were not only aware of the Upanishadic teachings, but they were consciously providing a critique of them

A Geneology of the "Three States" and the "Three Realms"
from Gaia Community: kelamuni's Blog

Shankara clearly sides with the Brhad Up on this matter: consciousness is no more capable of full reflexivity than a juggler can stand on his own shoulders or a knife cut itself. He does, however, admit a kind of reflexivity by saying that in enlightenment, or brahman-jnana, there is a "fruit" or effect of release, and this is the reflexive knowledge that one is released once brahman-jnana occurs. Among the the Buddhists, the Madhyamikas agree with Shankara. The Yogacharins, however, hold that consciousness is capable of being aware of itself, and Mandana Mishra, Shankara's great Advaitin contemporary, agrees with them.

Thus, we find a fault line running through both traditions, with some Buddhists holding one position and others another, and some Advaita Vedantins holding one position and others the counterposition. I think this fault line can ultmately be traced back to the difference between the teachings of the Brhadaranyaka and Chandogya Upanishads, and to their respective soteriological orientations: one toward transcendence and the other toward immanence.

With the notion that the third formless state is somehow "inadequate", the door is opened for a "fourth" state. But no such "fourth" is mentioned in the Chandoya Upanishad, at least explicitly. It does however mention a "pure Self" or Purusha that in some sense stands beyond the third state. Later commentators, including Shankara, will take this as referring to the "Fourth" state of the self, to Turiya. We can say then that the teaching of this "Purusha" is the teaching of Turiya in some sort of nascient form.

The "fourth", or Turiya, is explicitly mentioned for the first time in the Mandukya Upanishad, a very late Upanishad that is far removed from the Brhadaranyaka and Chandogya Up. Indeed, it is so late that some commentators do not even take it as an Upanishad. While the teaching of the Chandogya Up prefigures that of Mandukya Up, there is a historical and hermeneutic context that I believe must be supplied in order to understand what the Mandukya is actually saying with respect to this "fourth"; that context is the Buddhist tradition.

In a parallel prefiguring of the Mandukya Upanishad, the Potthapada Sutta --- an early Buddhist text --- refers to, and rejects, three "selves" that are related to three "states". The first self is "with form" (rupa) and "made up" of the four elements, and of "food." This latter formulation is similar to that of the Taittiriya Upanishad, which had referred to the first kosha or sheath of the self as being made up of "food". The Potthapada Sutra then describes a second self made up of form (rupa) but "consisting of mind," (mano-maya) which is precisely the same terminology used in the Taittiriya Upanishad. Last, it describes, and rejects, a third "formless" (arupa) self made up of consciousness (samjna). The grounds for the rejection of these three selves is the fact that they "come and go", that is, that each is transitory. Interestingly, transitoriness is also the reason given by the Advaitin Gaudapada, in his commentary upon the Mandukya, for the rejection of the first three selves as not "ultimate": the three states are not ultimate because they come and go (ie, we go into sleep then come out of it).

These features of commonality show two things: One, that the Buddhists were not only aware of the Upanishadic teachings, but that they were consciously providing a critique of them; and two, that the Advaitins were not only aware of the Buddhist critique, but that they were in turn consciously responding to it. That the author of the Mandukya Upanishad, and not just Gaudapada who is several centuries later, is himself aware of Buddhist doctrine is apparent in the fact, noted by Nakamura, that it refers to the "non-dual" fourth (Turiya) as "prapanca-upashama" (quieting of discursive proliferation) which is a technical term of Mahayana Buddhism used specifically by Nagarjuna. [...]

To return to my initial line of thought, what I would like to suggest is that the Potthapada Sutta, with its description of the "three states," is the context for this later doctrine of the three lokas, as well as background against which was written the Mandukya Upanishad. The Mandukya, like later Buddhism and the Potthapada Sutta, also gives us a description of three "structures" of consciousness. It also describes a "fourth" that somehow transcends the three. As we noted, the textual precedent for the "fourth" can be found in the Chandogya Up, with its critique of the "third" state as somehow "inadequate." But it is the Buddhist tradition, I would suggest, that is the actual context for the composition of the Mandukya Up. The most important feature of this context, I think, is the idea that the three states are somehow "imperfect." And as we noted, this inadequacy, viz., the transitoriness of the three states is taken over in toto by Gaudapada.

What the Vedantins required was a "response" to the Buddhists. They needed an account that "transcended and included" that of the Buddhists. That account was provided by the Mandukya Upanishad. In effect, the Vedantins answered the critique of the Potthapada Sutta by saying the following: The three selves are indeed transitory. But beyond the three states is the true Self that never comes and goes, that is transcendent of the three and yet always already present as their "truth" or immanent basis. In other words they said, "Yes we have described three states and three selves, but we also have a Fourth, which is beyond the three."

And in the process of giving this response they had started the ball rolling in a new game: the game called "let's ratchet up the terminology," a game that will lead to the doctrine of Turiyatita, that which is "beyond the fourth," which is found in Kashmiri Shaivism and Tantricized Advaita, as well as to Da's collapsing of certain features of the fourth into the third, the "causal."