If a creature can answer a question about a hypothetical experiment, without actually performing that experiment, then he has demonstrated some knowledge about the world. For his answer to the question must be an encoded description of the behavior, inside the creature, of some sub-machine or model responding to an encoded description of the world situation described by the question.
In my view the key to any really advanced problem-solving technique must exploit some mechanism for planning—for breaking the problem into parts and allocating shrewdly the machine’s effort and resources for the world ahead.
If one thoroughly understands a machine or a program one finds no urge to attribute volition to it. If one does not understand it so well, one must supply an incomplete model for explanation. Our everyday intuitive models of higher human activity are quite incomplete and many notions in our informal explanations do not tolerate close examination. Free-will or volition is one such notion—people are incapable of explaining how it differs from stochastic caprice, but feel strongly that it does.
When intelligent machines are constructed, we should not be surprised to find them as confused and as stubborn as men on their convictions about mind-matter, consciousness, free will and the like. For all such questions are pointed at explaining the complicated interactions between parts of the self-model.