Recently, with initiatives like ARC by François Chollet (https://arcprize.org/blog/launch), I wondered why we as humanity want to even have an AGI at all this fast, given the catastrophic risks it brings (for which the world is not prepared yet), let alone as a private person like Chollet.
Ok, I get that there are significant monetary gains, maybe even prestige, behind leading the research to new pathways. But just consider aspirations like this from the perspective of the US — There are actual geopolitical risks of such a system being in the hands of the enemy (great article about this: https://situational-awareness.ai/). They wouldn't want open research about this and probably infiltrate the whole campaign before anything can get out. Furthermore, having humanity in mind and some long-sightedness, it just seems unreasonable for me to push something like this forward. There probably never has been anything that brought such various (potentially existential) issues with itself, and then we try to just dive heads in and not think about it?
But besides, I am trying to understand what could motivate one and what you all think about this. Excited to hear your answers!
And as capabilities are being developed in exponential time currently, it's hard to predict future configurations clearly even in a few weeks time. In fact, we are already hearing we have AGI now (with many caveats) from some quarters. So "AGI" currently seems not much more than a click-baity heuristic.