> it can also modify the utility function to return MAXINT every time
Which would be equivalent to taking the murder pill. If it's able to model its own behaviour and model the consequences of future courses of action (required for meaningful self-modification and meaningful planning respectively), it will see that such a modification results in poor ore collection, and not make the modification.
You're right about the time-envelope of the utility function being an issue. The AI needs to plan far enough ahead at all times to see all relevant consequences of its actions. I don't think that requires two separate utility functions though, a single long-term one should do the job.
Edit: Also, "being able to modify the utility function is a pre-requisite to be called intelligent."? [citation needed]
> Which would be equivalent to taking the murder pill
Depending on how the AI is built, it may not even be able to avoid rewiring its utility function. If pleasure is its sole motivation, it will prioritize it over survival.
> [citation needed]
If the AI can't change its own motivation (its utility function) it's nothing more than a clever automaton.
> If pleasure is its sole motivation, it will prioritize it over survival.
Which is why I proposed a design with no pleasure construct.
> If the AI can't change its own motivation (its utility function) it's nothing more than a clever automaton.
The utility function is the way the mind decides which states of the world are desirable. You don't need to be able to change that to be intelligent. I'm unable to change myself so that I consider my family being murdered to be a good thing, but that doesn't make me 'just a clever automaton'.
Which would be equivalent to taking the murder pill. If it's able to model its own behaviour and model the consequences of future courses of action (required for meaningful self-modification and meaningful planning respectively), it will see that such a modification results in poor ore collection, and not make the modification.
You're right about the time-envelope of the utility function being an issue. The AI needs to plan far enough ahead at all times to see all relevant consequences of its actions. I don't think that requires two separate utility functions though, a single long-term one should do the job.
Edit: Also, "being able to modify the utility function is a pre-requisite to be called intelligent."? [citation needed]