Childish as in... A newish artist who is drawing a concept rather than light / forms (Which is something artists typically do as they understand drawing more and more).
The rose in the vase specifically - some models understood that there was supposed to be shading, reflections, the concept of refraction - others just drew "blue = glass" and "green = stem" and "red = rose".
Really odd to look at, considering if I saw any of these drawings from a human kid, I would say "good job buddy" and put it on the fridge. I'm expecting these to get better as models improve, and perhaps the artistic progression will be there along with it...
A lot of art that someone smarter than me told me to appreciate seems to follow the pattern of hitting the space and/or experience while minimizing the use of symbols. Impressionistic paintings esp avoid symbols IMHO, while bizzaro picassos abuse symbols outright and still hit the experience they are going for.
The Grok ones in particular gave me that thought. Most of them really look like what a kid would do when given the same tools, while the other models' output have distinctly more "AI-ness" to them (for lack of a better term.)
The fact that it has clearly iconified these concepts in its mind and is tracing the outlines of the things it thinks/expects to go where is very human.
[deleted]
At work, even if Fable is technically better I much prefer Sol because it is so much faster and concise.
Seriously, what's going on there ? Why is it so different from others? Is it just behind technologically/training wise or it's using something fundamentally different?
It performs much better than composer2.5 (while being as fast). It's not Opus, but I think it's not that far off. Definitely better than sonnet for what I've been doing.
On the other hand, I think they probably heavily adapted the training data so that it really is extremely focused on code. I just recently ran my personal "poetry benchmark" on it (where I give it ~850 poems I've written over my life and ask it to comment the corpus as a whole), and it's whack. It tries to write in portuguese (most of the poems are portuguese) and code-switches constantly and mixes up words to the point of making what it writes almost unreadable (e.g. it writes stuff like "You can't QoS that that look for beast poems", in portuguese, all messed up). The quality of the analysis is also quite bad (I'd say it's definitely behind Sonnet).
So I really think they either threw away data that wasn't tied to coding so that they could fine-tune it to that, or somehow they've got such an unbalanced dataset that coding ends up dominating either way. To me, its disastrous performance in this drawing "competition" fits this narrative.
Give Opus access to an image model, and it will use that to create images just fine just like ChatGPT.
I think using this skill, Claude drew it better than the examples given. https://claude.ai/share/bc0867dc-034f-4865-bb71-acb83bfa36a1
The rectangular smudge tool is a weird tool in the first place, but it's cute to see the models try to use it.
Which in turn kinda jives with my experience of using these models for code: to some extent they only seem to have a concept of "forward", which invariably leads to "write more code to fix previous problems created", rather than taking a step back and removing broken things entirely.
https://www.tryai.dev/?error=server_error&error_code=unexpec...
I made a lower effort but similar scaffold for LLMs to do iterative drawing in Nov 2024, with Sonnet 3.5 as the artist: https://paritybits.me/llm-drawing-with-eyes-open/
Quite a difference.
Alternatively, I think much better results could be had by computing the cosine similarly using DINOv2 ONNX (@xenova/transformers).
My guess is that the results are greatly limited by relying on the current metrics.
Even better, attach toModelOutput directly to drawTool so it returns the rendered canvas image immediately upon drawing.
- the best image related stuff I've seen is where the harness is constantly cropping and looking closer at things (likely helps a lot for computer vision in general) - also it would be interesting because the harness could almost have its own "palette" as if it could play with blank squares and different strokes over lapping or blending before applying to the main canvas
.-""-.
/ \
| _ _ |
| (o)(o) |
\ /\ /
| -- |
| \/ |
| |
/ -- \
/ / \ \
/ / \ \
(__/ \__)
[dead]
[dead]
First of all, this is a marketing piece, made for engagement.
Second, in production, you would adjust your approach to the model's strengths and weaknesses. For example, if you found that a model does not respond to the tools you are offering, then you would find a way to make it more responsive. Also, we know that different models handle prompts and system prompts differently. So comparing several models from different providers against exactly the same prompt is quite a naive way of exploring the capabilities of those models or comparing them to each other. You may have just happened to “speak the language“ of one model better than that of another. Why would that be the model’s fault and not yours?
Third, none of these results mean anything if you do not run each process on each model several times to check for consistency. Your model might just have gotten lucky. This point is even more true when you're talking about a progressive task where one step builds on the previous one. The model might just have painted itself into a corner.
How do we know the difference?
These SOTA LLMs aren't trying to mimic existing children's drawings, but interestingly they're following somewhat similar progression that human children do as they develop.
AI labs are improving the ML techniques used to build better models; it’s a big difference.
It’s like saying that a new M-Apple chip developed like a child. There is a clear tendency to give human like attributes to LLMs, probably caused by the terms used in ML: training, learning, etc.
reminds me of those jokes?
what is difference between a parot and a human? silence rly? u cant tell the difference between a parot and a human? :')
It felt a bit surreal to see a machines rendition of something that very closely maps to a younger human as they explore + understand more about the world. Scary, even (to me). This was the first time I’ve seen LLM output and thought “wow, maybe it is learning.”
For some reason, OpenAI models, Gemini (and apparently Grok too), love to latch onto this and obsess over this idea that it's "programming poetry" or "poetry for the IT crowd". Often OpenAI and Gemini try to write the "equations of my poetry" (granted, I do write about a cyclical relationship between thinking, feeling and writing a lot, and I do have ONE poem which ends with a Q.E.D.).
I'm giving this context to say that it is very bizarre. It's as if they latch onto it and act as if it's a core or highly distinguished part of the poetry, when it really isn't. Anthropic models, on the other hand, absolutely do not do this, and have never done it.
I really don't understand why this happens. Maybe it's because it has a lot of portuguese, I don't know. And even though the "QoS" is clearly the wrong token being generated, I have had situations where gemini spoke of some phase of my poetry as the "Q&A part" (really, no joke...)
In any case, it's why it's my personal benchmark after all :D
It still somewhat affects the way the AI looks at the poems, especially if it has to make lists of the “best”, but it is not a very pronounced effect nowadays. Maybe it gives some preference for earlier and latter poems but not a lot.
Back when I started doing it, these holes in the context were super obvious, just like you described. It would mixup entire periods and also hallucinate or forget about specific sections. That’s precisely why I started randomizing them as part of my experiments.
Nowadays the frontier models are much better and don’t really mix anything up.
Edit: I just don't see the point of redacting the Mona Lisa
HN, the only environment in the world where people will scientifically falsify a joke.
And I like how its starry night decided to accentuate the church/steeple in the village backdrop.
Pretty cool how it did these with a scrappy little tool palette. What a fun idea.
There's likely not training data for this and Claude seemed to spend a lot of time blending the background.
Citation needed - my company is paying more than ever for code generation. I have no reason to believe (given anecdotes) that anyone finds themselves in the opposite situation.
I get a lot more use (read: cheaper per interaction) and much better quality results from the $20 that I spend on this stuff every month than I did several years ago.
(And several years before that, it was all essentially unobtanium.)
For me, it was the same genre of comment as e.g. comments that call out how vision models tend to be structured surprisingly similarly to how the visual cortex is structured, even without specifically attempting to make that happen, and within the boundaries of our understanding of either things of course.
So not so much anthropomorphization, and more a recognition of the unintentional, and surprising, biomimicry.
Though to be honest, I'll be damned if any of the pictures in the post look anything like any child drawing I've ever seen, Grok's or any of the other models.
The parent wasn't referring to individual LLMs developing, but that overall new SOTA LLMs models are increasing in capabilities in ways similar to how human children do.
[deleted]
A specific model develops as it passes through training.
AI labs change the architecture between models to allow them to surpass the previous models' best scores.
It's also going to keep being a hard sell to say "stop anthropomorphizing LLMs" when the models anthropomorphise themselves.
But besides that: Yeah, sure, they're not human, they're a cargo-cult mimicry of by and of minds that popped out of evolution doing gradient decent on intergenerational survivability. So what? Still interesting when the result of such cargo-culting incidentally echoes what our natural-selection-not-engineered moist electrochemistry happens to do.
Edit: And not a great example to use to prove human-ness considering humans didn't disprove the conjecture?
Why do some people get very defensive when you suggest AI isn’t comparable to humans?
Like, where is it coming from?
Why do people seem to want to ascribe human traits on the tech? It’s just technology. Why do we need to draw parallels to biology? Is it actually helpful? My position is “no” because the parallels only seem similar, but fundamentally aren’t.
Allow me to explain the theory behind the humor...
Grok's art had heavy use of what strongly resembles a permanent marker with lateral-ish scribbles similar to what one might see when someone is trying to mark something as discarded. Documents, especially those procured by the US government when responding to FOIA requests, contain many 'strike-through' style black marks resembling bold permanent-marker lines, which obfuscate the words they wish the reader to remain ignorant of. In many cases, this is the entire document, especially where unredacted words might otherwise serve in any way to clarify the nature of the FOIA inquiry, i.e. answer the question.
Grok's art, resembled to me, and others potentially appreciating the joke, redactions more than art. In some forms of humor, the presenter of the joke feigns obtuseness, or a mildly self-deprecating, stupid perspective. But the stupidity of the perspective actually reveals a relevant concept, or idea.
Often, such jokes can be deciphered using common sense, or various razors, including both occam and hanlon, the former self-suggesting that the joker is probably not so stupid as to think Grok was really redacting anything, and the latter self-suggesting that there was probably no malice involved.
Hence, I can't be bothered to read any of this.
Look at the sequence: You inquired. I answered. Not finding something funny is not really a good reason to fake a question you never intended to really have answered. You are kind of admitting that you knew it was humor but decided to approach it deceptively, or manipulatively, as if I am the only one ever to make a joke that wasn't universally appreciated.
I think that makes you a prick, with a lot of karma-tokens to play with.
But you didn't do that, did you? You rattled off a goofy cliché about parrots that we've all heard a hundred times before. If that's all you have to contribute to the conversation, go contribute it somewhere else.