FYI: though EFF articles have individual named authors, they go through an extensive collective editing process. Every post will have had at least one domain-specific lawyer reviewer who signs off on it.
I got early access to the pre-ChatGPT OpenAI API (actually by pinging someone from OpenAI who posted about it on HN). At work, we were setting up to play a livestreamed JackBox game for a charity event. This would have been in 2019.
In a previous life, I'd been a writer for the original You Don't Know Jack game (the UK variant), where the job was to crank out as many funny quips about a topic as you could, and then use a handful of them in the recording of the game itself. Some of the later JackBox games are like that, but for the players -- you're given a set piece, have to come up with little funny improvisations within a time limit.
As an experiment, I tried the set-up lines with the OpenAI API, and see whether it could come up with some responses. Of course, 90% of them were unfunny or incoherent, but 1/10 were not bad, or even pretty good.
I'm not sure that would have been impressive to anyone else -- but remember, I'd had this as a job, and sat in a writer's room, where everyone did this, for hours. In that environment, you expect a large proportion to be duds: the discipline is keep pumping them out, and not flagging creatively until you find a rich vein. I realised that this was a tool that would have been the perfect complement to that work -- and it was a pretty good JackBox player too.
> When low-frequency sound becomes strong enough to be heard or otherwise felt, it can cause annoyance, discomfort, and sleep disruption like any other normal noise pollution.
So which is it? Sure, I don’t really believe that there is magical super special harmful noise from a datacenter, but are these monster datacenters emitting disruptive amounts of low frequency sound or are they not?
Ad homniems aside, is the accusation even accurate? So far as I can tell he doesn't obviously have "a massive financial incentive to do so", like he's a VC investor in anthropic or whatever. He does seem to be bullish on AI in general, but I'm not sure why that'd be a disqualification for someone on the pro-ai camp any more than someone who's interested in retaining their property values or whatever would be a disqualifier for the anti-ai camp.
Ad hominem would be if shimman had said something like "don't post rebuttals from people who are stupid meanyheads". Identifying a characteristic of the posters that affects their incentives is a perfectly legitimate reason to discredit their posts, or at least call their impartiality into question.
>Ad hominem (Latin for 'to the person'), short for argumentum ad hominem ('an argument to the person'), refers to when a speaker attacks the character, motive, or some other attribute of the person making an argument rather than the substance of the argument itself.
I think people sometimes misunderstand Daniel's point here, though it's clearer when taken in context of the rest of his article. The tools in general are getting a lot better at finding security bugs, it was unclear to Daniel based on his usage whether Mythos in particular is a huge step, but the Mythos generation of LLMs definitely are. Note though that Daniel was using Mythos somewhat indirectly. One thing I've taken away from the whole Mythos debate is that a) I suspect that Anthropic's GPU crunch meant that they felt they had to ration Mythos access anyway, so the calculus of whether they would release it generally was probably influenced by that, and b) finding bugs with Mythos or a similar model is still expensive -- a $20K or $100K Mythos run on Curl might have shown the same level of issues as other projects like Firefox, but Daniel didn't get that kind of access.
He posted a general update today on LinkedIn which I think gives the wider context:
> Not even half-way through this hashtag#curl release cycle we are already at 11 confirmed vulnerabilities - and there are three left in the queue to assess and new reports keep arriving at a pace of more than one/day.
> 11 CVEs announced in a single release is our record from 2016 after the first-ever security audit (by Cure 53).
> This is the most intense period in hashtag#curl that I can remember ever been through.
For me, a modern descendant of METAFONT is probably Iosevka's build system, which has its own internal DSL, PatEL, for defining its font forms, based on decomposed sub-functions. PatEL's a Lisp-with-infixes-and-indentation that compiles to JS[1].
See the definitions of "O" and related glyphs for a good example[2].
So, this is not quite right: Alexander contributed to the report, but his personal opinion is more like the mid-2030s[1]. Freddie feels like this is him backing down from the original statement, but in fact he said this at the time the report was published, and in fact pointed out a graf below the quote that Freddie claims does tie him to 2027:
> Do we really think things will move this fast? Sort of no - between the beginning of the project last summer and the present, Daniel’s median for the intelligence explosion shifted from 2027 to 2028. We keep the scenario centered around 2027 because it’s still his modal prediction (and because it would be annoying to change). Other members of the team (including me) have medians later in the 2020s or early 2030s, and also think automation will progress more slowly. So maybe think of this as a vision of what an 80th percentile fast scenario looks like - not our precise median, but also not something we feel safe ruling out. [2]
I don't think this changes your observation that he is "personally invested" (i.e. believes this trendline will continue), but I'm pretty sure when AGI doesn't appear in 2027, many people will believe that this invalidates the arguments being made here (or in the report). The actual report was intended to give a feel for what a near-future "disaster" AGI scenario, and settled on a date to give that some concrete immediacy. The collective review that gave that as a possible, but not inevitable date is still ongoing (they originally pushed their best estimate out a bit further, but now they think, judging by the goals that are being hit, their scenario was a little too conservative). [3]
LLMs are nothing close to AGI and not going to lead to it, they can’t distinguish right from wrong, they can’t count, they can’t reason, they generate plausible text from a vast databank of connected text.
Apparently that is enough to fool many people but it’s nothing close to AGI which would require internal models of the world, reasoning etc.
We are nowhere close to AGI and the fools who predicted we were will unfortunately keep lying about their stated timelines when it inevitably doesn’t arrive. You’re already hedging and trying to caveat previous predictions, as OpenAI did with their AGI predictions which they’re now furiously back-pedalling on.
This is all speculative. We don't understand intelligence, so you literally have no idea whether what we recognize as intelligence is some suitable arrangement of "statistical token generation", especially once you add feedbacks loops.
> "We don't understand intelligence, so you literally have no idea whether what we recognize as intelligence is some suitable arrangement of "statistical token generation""
Do you mean "token" as in the LLM sense?
Or are you thinking that thoughts in the human brain are also constructed out of some sort of underlying "token" even though the abstract thought happens and is held before any words are used to try to communicate that thought to an external party?
LLMs also don't run on tokens internally, they're just the inputs and outputs. The reasoning models do operate (partially) in the token space, but then so do I.
> the thought happens and then the words are generated to reasonably describe that thought.
Thoughts don't happen in a vacuum, they are triggered by external or internal stimuli, and these stimuli/thought precursors could very easily be tokens (dense info packets), which then map to latent space vectors, which very well could be thoughts.
Claims like "humans don't operate the same way" has no basis. Not only do we literally not know how humans operate mechanistically, and so we literally don't know the logical structure of human thought, but any system that is Turing complete is so easy to create that many wildly different mechanistic systems are fundamentally equivalent/interconvertible.
> Thoughts don't happen in a vacuum, they are triggered by external or internal stimuli, and these stimuli/thought precursors could very easily be tokens (dense info packets), which then map to latent space vectors, which very well could be thoughts.
Yes, possible, that's why I asked you above if that's what you meant by "token". Someone else responded and I didn't notice it wasn't you.
> Claims like "humans don't operate the same way" has no basis. Not only do we literally not know how humans operate mechanistically, and so we literally don't know the logical structure of human thought, but any system that is Turing complete is so easy to create that many wildly different mechanistic systems are fundamentally equivalent/interconvertible.
I think this position is too extreme, we do have some information.
We know how LLM's work when generating a sequence of words and I know that my brain does not work the same way for word generation because I am fully aware of the complete thought in advance of any words getting generated by me externally or internally.
I know prior to generating words that my thought is X and the words I'm about to produce need to express that thought.
But with LLM's we know that the essence of what they produce is not known in advance, that it must complete the word generation process to fully realize the end result and that multiple different end results are possible.
What I'm saying is that this is incorrect. An "idea" exists within a model before it generates tokens. This property does not distinguish humans from LLMs.
Additionally "from learned stats" doesn't disambiguate between a wider variety of things. I'm not aware of any other way to acquire knowledge from measurements. I'd bet that humans do this differently, based on the fact the humans can get further with less training data and that they learn actively during operation, but not so differently that 'learning stats' would be an inaccurate description.
> What I'm saying is that this is incorrect. An "idea" exists within a model before it generates tokens.
If that were the case, then the systems would generate words based on the fully resolved idea, but that is not how the LLM systems currently work (per vendors descriptions).
They choose words sequentially and both the specifics of the input as well as the chosen output words significantly impacts not just the rest of the output but the very correctness of the output.
> but not so differently that 'learning stats' would be an inaccurate description.
Agreed, humans are generalizing using some mechanism that can be modeled with math.
But the execution of our reasoning and thought processes is not obviously similar to LLM's next word generation based on probabilities.
>that is not how the LLM systems currently work (per vendors descriptions)
Anthropic says of the their model[0]:
"""Claude sometimes thinks in a conceptual space that is shared between languages, suggesting it has a kind of universal “language of thought.”
{...}
Claude will plan what it will say many words ahead, and write to get to that destination. We show this in the realm of poetry, where it thinks of possible rhyming words in advance and writes the next line to get there. This is powerful evidence that even though models are trained to output one word at a time, they may think on much longer horizons to do so."""
Anthropic also created 'golden gate claude'[1] by identifying the region of its architecture that corresponded to the concept of the golden gate bridge and activating it. What would such a region exist for if claude could only think one token at a time?
>the execution of our reasoning and thought processes is not obviously similar to LLM's
"Not obviously similar" I can agree with. I don't think you've identified a way in which they are obviously different, though.
We understand it enough to see the obvious massive deficiencies in LLMs.
They can predict likely sentences but not evaluate truth or logic. They can fairly reliably record facts about the world but not construct internal models of the world.
> They can predict likely sentences but not evaluate truth or logic.
They do probabilistically. So do humans as a matter of fact. The best of us are better at it than LLMs, but that's not persuasive evidence of anything meaningful really.
> They can fairly reliably record facts about the world but not construct internal models of the world.
You don't know that, unless your presuppose a very specific definition of world model that necessarily precludes emergent ones.
Humans do not reason by guessing the next most likely token/word. They use logic, morality and systems of thought they have constructed and shared to help them reason and don’t in any way predict tokens in a sequence - we use words to represent our thoughts and feelings about the world, not to construct them.
You’re constructing a post-hoc fantasy of human thought based on how LLMs work because you are desperate for some reason to believe that they are thinking like humans, but they are not. The process is very different and the results are also different.
> LLMs are nothing close to AGI and not going to lead to it, they can’t distinguish right from wrong, they can’t count, they can’t reason, they generate plausible text from a vast databank of connected text.
Argument?
Are LLMs close to being able to significantly help AGI researchers?
I think it's worth linking to the original Agile Manifesto[1], because that's pretty much all the consensus you're ever going to get on what's "agile" and "what's not".
Lewis is right that most of these principles were described before the manifesto, but I can vouch for the near-impossibility in many contexts of convincing anyone who wasn't a coder (and a lot of coders too) why these might be sensible defaults.
For every person burned by a subsequent maladaptive formalization of these principles, there was someone horribly scarred before the agile manifesto by being forced to go through a doomed waterfall process.
Ask anyone with 30 years in the industry whether "agile", for all its problems, was a force for good or bad, and the answer will be an emphatic Good!
If nothing else, it gave us ammunition to argue against the impossibility of delivering a fixed thing in a fixed amount of time - which was the universal view from senior stakeholders of what competent software delivery looked like.
No, you are not going to get that consensus, because middle management and hire ups don't want to know what agile actually means, but want to continue believing, that the processes they impose are agile, and they have probably never even seen that page. In a truly agile team, the team takes many if not all of their work and responsibilities, so that even the jobs of PMs and low middle management would be on the line. As we know it is difficult to get someone to understand something, when their income depends on not understanding it.
I worked at EFF during that time, and this is a weird story that I’ve not heard before. EFF doesn’t let interns write blog posts (at least not with a lot of supervision) and certainly wouldn’t sack someone for getting something wrong — partly because that’s a terrible lesson to teach someone just starting out in law or activism, but also and more pragmatically it risks being a PR nightmare.
I concede it might be a mangled version of some other incident — EFF’s network neutrality policy during that time was /extremely/ subtle and we often struggled to express it without annoying some colleague organization or another. Do you remember any other details, or link to coverage of it?
yes it was this, not the EFF but the Trump admin, it was a surprisingly normal and level headed policy take, and I was pleasantly surprised, but then it turned out it wasn't their official stance, it was removed and replaced with a statement and stance that was nearly the opposite. But for the life of me I can't find it again, but I swear I didn't imagine it.
reply