By default, if somebody without extraordinary competence and paranoia tried to make an AGI that would implement government laws, in practice you'd just get something equivalent to a paperclip maximizer (https://wiki.lesswrong.com/wiki/Paperclip_maximizer).

It doesn't matter much whether it's the government accidentally destroying the universe, or a private company or a university research team accidentally destroying the universe. Pointing an AGI in a particular direction is a hard technical problem that we haven't yet seen much interest in solving as a technical problem, in part because people have been distracted over arguing who gets to destroy the universe. This is easier to argue about and activates a lot of fun human political instincts where you get to valorize or devalorize particular tribes, so it's a very dangerous distraction.

You would need to make a lot more progress on the alignment problem, as a technical problem, before it made any difference to actual outcomes who'd built the thing or what they'd thought their intentions were.

Absolutely true. You could substitute "organization" in place of "government" in my comment and it would still hold. And to some extent, funding efforts like these are (hopefully) so far away from the actual problems of an AGI that arguing about which one is more likely to create a paperclipper is somewhat moot. The concept that any such organization could get an AGI close enough to right that it isn't just a paperclipper, while being just wrong enough to create the kind of dystopia imagined in fiction rather than in the imaginations of people who think about AGI failure modes, is almost an uninterestingly small target, and not what I was aiming for with the preceding comment; rather, I was thinking specifically about fail-by-paperclip.

But regardless of the failure mode, my concern is precisely that anyone or any organization with a sufficiently large (and sunk) investment into itself, regardless of what the nature of that organization is, will have a temptation to include their own interests in a value function. And because it's hard to reason about the consequences of errors on that scale, such organizations might also tend to not make a sufficient distinction between "fail by destroying the universe (or worse)" and "'fail' by not preserving whatever principles this organization (thinks it) holds dear". Not distinguishing those two cases would then tend to lead the organization to discount the risk of meddling in the value function, even in the face of warnings that doing so would destroy the universe, because they might view their own obsolescence as little better.

So, among the large number of things that have to go exactly right, one of many will be the need for the organization developing it to very specifically have no goal other than getting AGI right, for a definition of right in terms of sentient life rather than anything distinguishing that particular organization.