hckrnws
Samsung is expected to more than double output of its HBM4 and HBM4E DRAM
(en.sedaily.com)
561
459
by giuliomagnifico
by giuliomagnifico
"Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway."
When I worked on satellites we had some electronics which used a radiation hardened FPGA which cost something like $500k per chip. Carrying that around was nerve wracking.
I had to remind myself a few times the practical value to anyone was about $22.
What stayed with me from that talk was that all the data that was captured from those massive antennas was packaged in boxes, loaded in a cargo van and travel 1.600 kilometers to Santiago to be uploaded into the systems.
[0] https://aws.amazon.com/blogs/aws/aws-snowmobile-move-exabyte...
[1] https://www.cnbc.com/2024/04/17/aws-stops-selling-snowmobile...
It’s not a step I’ve seen discussed very much, so it was kind of neat to see it discussed front and center in a lay article like this.
https://resources.pcb.cadence.com/blog/2023-the-planarizatio...
Not dissimilar to lens or mirror making, the process of grinding to a precise flatness. The key step here is bonding it to a glass carrier, presumably because otherwise it gets too thin to support itself.
See also the "reduced kerf diamond wire saw": technology critical to cheap solar panels is the ability to saw the boule into thinner and thinner pieces with less waste at each cut, like a big ham slicer.
The lack of ASML EUV machines certainly hurts, and pushing DUV so hard results in abysmal yields of good chips, but you can compensate by running more wafers or making smaller chips, and the net result is that Huawei's Ascend production volume is limited by CXMT's HBM capacity not processor dies.
The problem is that HBM manufacture requires many steps (die thinning, via drilling, plating, alignment) where the equipment used by everyone else (Samsung, SK Hynix, Micron) is also blocked by sanctions, so the Chinese are having to develop all of this themselves too, which they have, but yields are currently low, even when using shorter HBM stacks.
He alleged memory maker's public announcements of the shortage lasting throughout the end of 2027 are attempts to covertly signal the others on pricing strategy.
With additional production capacity ramping up this year and next, this seems somewhat plausible.
If manufacturer's inventories are filling since they advanced orders when prices were rising, there could be an oversupply of memory when they don't place new orders at the higher current price.
This could quickly result in a price war while capacity expansion and yield improvements create irreversible oversupply.
And the worst thing is that is the best possible strategy for them. It's essentially win-win for everyone but consumers.
- Buyer (AI) has crazy money, so will pay whatever
- Seller doesn't have to build anything new, as the buyer is willing to pay whatever
- Generate ridiculous profits from crazy money
- No oversupply risk in case of reversal
- Return from producing HBM to DDR5 in a single quarter if reversal does happen.
Hold on to your existing hardware, people, and be on lookout for your local deals.
But someday, that demand will stop rising.
At some point, most people will have enough RAM. And they just won't need to buy more.
The inventories of RAM will become overfull, in warehouses and stores. And suddenly the manufacturers, middlemen, and even retailers will all find the price they can sell at keeps dropping.
But you can't shut down a huge manufacturing facility overnight. You can't shut down a years-in-the-making retail distribution system overnight, either. You can't even shut down a facility that is halfway finished with construction. They'll keep producing and selling RAM at lower and lower profit margins, until the day comes they finally accept they're losing money with each sale.
And when that happens, we'll all have more RAM than we know what to do with. It'll be cheaper than we have all ever experienced in our lifetimes.
I don't expect this to happen soon. But demand cannot keep exponentially rising forever. There will come a point, probably in the next 5 years, when demand at least levels off, even as production capacity is still rising.
And it will be interesting to see what's possible at that point. We'll be able to create software that is not doable right now, because it will no longer be constrained by memory. And even cheap laptops will massive amounts.
I wonder how many layers theirs is. So many layers (ok, 8, 12, or 16 for different fabs). So much less dram for the rest of us.
The big problem is that this will not depress the consumer DRAM market and mean we can all afford DRAM again
[dead]
[dead]
[dead]
Which governments, and what exactly would you want these governments to do? The demand is global. There are no levers a single government can pull to meaningfully influence the global demand without fully committing to an protectionist economic policy, in which case the U.S. doesn't have the facilities to magically pop up world class fabs overnight, and South Korea and Taiwan don't have the market demand that the U.S. generates to justify their investments in making these chips and China lacks the IP to be able to build anything comparable to Nvidia's silicon at the moment.
No one has all the cards and no one controls all the levers.
I used to work for a financial institution small enough to sometimes need someone from the back office to help fill in as a warm body for dual-control when we received cash shipments.
It only took a couple sessions in a vault to get entirely bored with having a half million dollars in front of you.
The truck has multiple drivers and private security.
Security is an issue, though, these are treated as a high value load and typically would be followed by unmarked, plain clothes armed security guards tailing around the truck. (That's how Apple often does warehouse deliveries of trucks full of iPhones. It wouldn't surprise me if there was going to be some rolling lead soon following some trucks full of Duos...)
https://www.wired.com/story/the-worst-ive-ever-seen-cargo-th...
Likely ending up on a container to China, export bans be damned.
https://www.supplychain247.com/article/how_do_you_get_all_of...
It's just crazy how far down the scale we're tickling atoms these days.
Zero chance you could do it if a human had to touch the die/chip directly during those steps.
But clean room + vacuum pick and place machine? A matter of fine tuning, and it should work just fine.
Edit:
And, that, my fellow keyboard high-minded thinker but powerless corp peons, was also part of the China's plan all along.
Western sanctions have succeeded in achieving China's self-sufficiency goals where their own government failed.
A friend of a friend is married to a member of the national academy of sciences in China. I heard at dinner a few years ago that he had been lobbying for more onshoring of chip production for years before the Nvidia restrictions, and always heard "you're right, maybe next year we'll be able to fund that". Then the restrictions came in and everything changed.
It's a possibility. It is also possible that they will steal yesterday's technology and still pretend to have moral high ground.
I do not wish China and Chinees wrong but most probably communism will work as it always do - by faking thing and bringing self-harm to their own citizens.
Was the Great Leap Forward part of the plan too?
No, that's generally what happens when countries stop being poor. Taiwan, Singapore, Korea, all have similar fertility rates.
And it’s not a given that they’ll catch up. I think it’s fair to say that their development of jet engines is on track to be obsolete before they’re competitive.
They may in principle have established the technologies to be self sustained but that’s only relevant and sustainable if they develop an economy based on domestic consumption rather than exports. Which they are struggling with as well.
Whereas China is developing a powerful network of rapidly developing trading partners like Africa and the Middle East?
[dead]
Prefill (input tokens) is heavily compute bound. And the ratio of input to output continues to rise, as typically in agentic sessions you have a few tokens output for a tool call and (many) thousands of input from the tool result.
Then you have cached input tokens, which is a totally different issue, system RAM or NVMe bound.
Obviously output tokens is VRAM memory bandwidth bound, but this is less and less of the bottleneck these days for overall agentic speed.
[dead]
If your yield is low and you still want the volume, then run more wafers, but then you need access to more DUV machines and more wafers. If your defect density is too high, then design smaller chips with a higher chance of avoiding defects, which is what Hauwei are doing - split processor into multiple chiplets (NVidia do this too, to raise yields and reduce cost).
HBM3E usually has 5-6000 vias, but doing this with multi-pattern DUV without defects is tough, so CXMT currently drop that to 3000, at a cost of some loss of thermal and voltage stability.
HBM:processor production ratios aren't what Huawei would like, so they mitigate this at system level by putting an optical memory bus on the GPU chip and sharing memory across the system.
Not everything is a huge LLM - smaller models like recommendation systems don't have so many parameters and need so much memory, so why waste HBM on them? ByteDance use their SeedChip acelerator for this, currently made by TSMC (using an older sanctions-approved 28mm process), which instead etches dense RRAM on die beside the processor.
There is also a time dynamic to this, with China still using pre-sanctions equipment and chips as their domestic alternatives ramp up to replace them. One interesting part of this is HBM ... HBM is very demanding to make, and everyone had problems with it, with initially only SK Hynix being successful. Samsung and Micron took a year or so to catch up, and during this time Samsung had made a ton of HBM that didn't meet NVidia's specifications, so ended up, pre-sanctions, selling it to China, where it has acted as a stockpile to carry them over as CXMT's domestic capacity ramps up.
China seems to be doing fine. Of course they would like sanctions lifted, moreso for HBM than anything else, but all that sanctions have really achieved is accelerating their semiconductor independence. They are still building 1T+ SOTA models, standing up 100K clusters of domestic AI accelerators, etc, etc.
If it could be made to work, you could run a fable-grade model in tens of watts.
[deleted]
I gather a practical max ceiling today is a stack of 16 chips in height yielding 64GB?
These chips have a massive bus size of 2048 bits, instead of the 64 or 128 bits (dual channel) used by DDR5. That's what gives them their order-of-magnitude bandwidth speedup. But even though they technically pack in more capacity per square millimeter of motherboard, I gather they take up more space than older technologies once you account for the vias and interconnects to route all those signals.
HBM is meant to be integrated into the same package as the CPU, so no more DIMM sockets. It also has higher latency apparently.
> These chips have a massive bus size of 2048 bits, instead of the 64 or 128 bits (dual channel) used by DDR5.
The advantage of HBM over regular non-stacked DRAM is memory bandwidth, which also requires a super-wide memory bus - 2048 bits wide for HBM4. Compare that to the 128 bit wide bus of a modern CPU.
So to take advantage of it on the desktop, or anywhere else, you need that 2048 bit wide bus, and a processor capable of consuming 2-3 TB of data per second!
These are not normal requirements, other than for a GPU.
Or 256-512 bits on medium to high end consumer CPUs if you're apple.
At least DDR6 is probably widening things 50%.
But in reality we also already have unified memory architecture systems, integrated graphics etc.
The advantage in going wide is transferring cache lines rapidly, not the CPU bus interface.
Buy the way you win with CPUs is with latency, and not bandwidth, which is why Apple M series actually uses DDR with lower latency because of the stacking.
Running 17 chrome tabs doesn't benefit at all from that HBM and all the additional hardware+software complexities that come with it. You want a specialized coprocessor to handle specialized workloads. The GPU exists separately from the CPU for a reason.
You make HBM instead of DDR and the number of bits you're making goes down.
It's really that simple.
People will have to get used to buying a fixed amount of RAM with their CPU but thats unlikely to be a problem.
They have managed to pull this sort of thing off many many times. https://en.wikipedia.org/wiki/Reality_distortion_field
No, Mac laptops use LPDDR, currently LPDDR5X.
More HBM4 will drive the price of it down. That will make it less profitable to produce. That will mean firms shift some production back to more profitable "consumer" RAM. Increased supply will lead to lower prices.
A shortage is always followed by a glut.
Supply is currently moving the wrong way. Supply and demand for consumer DRAM will get even worse, even if it will "definitely fix this" in the long run.
And the fact that production is moving the wrong way suggests that in the short term supply and demand is actually doing more harm than good. HBM is so profitable it's radioactive.
I don't know where you got the idea in another comment that all memory will get "very cheap quite soon". They're not increasing total memory production. (much/yet)
Everything in their statement can be true and it be a bad thing for consumers of non-HBM RAM.
"HBM capacity to expand to 250,000 wafers a month"
So let's say their current HBM capacity is 100k wafers/month (pure speculation/random number for illustration), and their total RAM capacity (including HBM( is 300k wafers/month, then non-HBM capacity reduces from 200k to 50k.
SK Hynix's new Cheongju M15x fab was originally intended to be for DDR, but mid-construction they shifted it to HBM. Similarly, the Icheon M10F packaging line has been retooled to HBM, allowing it to assist in processing the output of other fabs.
And this was news early last year, so who knows how far it has progressed since then.
[deleted]
All supply is supply, despite what certain folks seem to believe.
There are no privileged consumers that have "dedicated" amount of production, no matter the demand.
If someone can dump 100x the money to get something I want, they're not "just like me".
It's all GDDR6X or 7X
oof.
It absolutely is not a great situation at all. This won't make things better.
I.e., you get a locked down device with access to their AI and maybe a way to run third party apps. Of course the developers of those apps will have to pay a percentage of their revenue.
Sound familiar?
It may turn out that demand for SOTA frontier models isn’t so limitless after all.
Long-term contracts, not building new fabs - they're in full on hedging mode right now.
I grew up in a country where there was a structural shortage of say, shoes and cars. You didn't really even get to choose which shoe you wanted to get, when there was a new shipment, folks went to the "department store" and got whatever size they could get their hands on and later trading it for the right one. To get any car, you would submit an application and 5-8 years down the line you would get a letter that your car would now be ready to be picked up, in a random color.
Everything is moving to edge compute. From smart rings and watches, to robotics and drones, iphones and humanoids, and no end in sight to the scaling improvements in AI... It does seem that you can continue to get more emergent behavior the more compute you expend in pre/ post training. There's no reason we can't scale up to a million data centers in space, as long as you get dramatically improved intellegence.
Yes, only a small number of companies produce memory because of capital requirements and complexity. When prices go up both of those things become less of a problem for investors who can see better return for lower risk on that high capital investment. They will then invest in production to capture those returns. This is literally the exact reason Samsung are doubling supply of this particular type of memory.
High prices will fix the shortage.
https://en.wikipedia.org/wiki/DRAM_industry_price_fixing?use...
Never has been, but what makes you think the major players won't collude to keep prices high?
If you look over all such lawsuits, they extremely rarely succeed.
Especially not for consumer goods.
I'm as fanatical a free-market fundamentalist as you'll ever meet, but if someone were to argue that government has a role in preventing bullshit like OpenAI's unilateral 40% attack on the entire DRAM market, backed by nothing but funny money, I would have a hard time coming up with defensible counterarguments.
One might say "well what specific law have they broken" and I don't know and I don't know if they have, but at the very least the power of the State _could_ be used to force behavioral changes in some ways or another.
[0]: https://en.wikipedia.org/wiki/DRAM_industry_price_fixing
Can that share of production be allocated to consumers, and the AI fights over the rest?
You can’t demand anything with this mentality.
“The current situation is messing up the economy for a very large portion of us” is universally a good enough reason
https://news.skhynix.com/en/fab-facility-investment-2026/
It's not an overnight thing obviously.
Bastards.
Everything is affected by the endless thirst for RAM by AI companies and the accompanying hyperscalers. I'm honestly disappointed governments aren't doing more to keep the price of consumer goods in check.
I guess the AI companies telling the world how good they are at accidental cybercrime and fraud is creating enough FUD to let the AI industry run wild.
I definitely know what those small brown boxes which ship UPS ground look like. Very normal.
And, to the point I was responding to originally, we can check those documents! They don't speak of "annihilating" other countries' industrial capacities.
Why would they have to?
When China talked about their 5 and 10 year plans on solar panels there was no need to talk about putting the rest of the competitors out of business. That was a highly probable end goal of what they were doing, no need to state it.
It would be like Walmart stating "We're going to put out massive loss leaders in sporting goods to put X out of business". Instead they'll talk about growing market share and long term customer loyalty.
[deleted]
Next you'll be telling me fairy tales like "laws are real".
This is all nonsense. Capitalism isn't an ideology, it's just a slight formalisation of what naturally arises. People are free to choose; they can own things; they can make agreements; they must honour the agreements or pay a penalty. Capitalism in no way is about stealing.
[deleted]
Which incentivizes pretending you need a ton, and can give nothing. You think being useful at work in Capitalism is bad? Wait until you are accidentally useful in a communist/socialist economy.
There is usually less pretending under capitalism, but people are people, so it all the same stories still play out.
People are shitty, if given the chance and influenced in specific ways. Capitalism or not. We’re dealing with the current bullshit in the west because somehow people bought the line that everyone is a good person - and we totally can’t actually see or do anything if we see someone doing something bad! - and we need to always follow ‘the rules’ even when someone bad clearly isn’t.
Until we find our balls, good luck.
[dead]
[deleted]
On the one hand, you're proving my point. On the other hand, if you think that China's headed for a demographic disaster, so are all of those countries, and your thesis of blaming the CCP for everything... Falls a bit flat.
Developed countries tend to have lower fertility rates (i.e. less kids per family) than undeveloped ones. Comfortably off, well educated people tend to put thought into whether to have kids or not. In poor countries people tend to have large families so they can send the kids out to work.
Having a high fertility rate isn't "high status" and a sign of a country's progress, but rather the exact opposite.
Of course this is separate from this issue of having a shrinking population, which is a sign that your citizens are not optimistic about the future, not a positive thing.
The one child policy is gone now but birth rates haven't really moved upwards.
It was insanely disruptive.
i dont think it will - they still have a chokehold on battery production (which implies dominance in the EV sector), and they also control the vast majority of all rare earths. The chinese gov't can subsidize chip research/development with these other sectors that are dominating.
The west is the one that need to watch out for their own economic dividend running out from the past century.
but... not the western govs
The Chinese domestic EV market is the largest in the world.
Chinese factories are full of Chinese robotics.
Companies like Huawei are struggling to meet the domestic demand for AI accelerators, while growing volume very fast year over year.
The Chinese domestic AI market is massive, but we just don't hear much about it outside of the models that are also sold here. Can you even name the models that ByteDance (China's Meta) makes?
As far as exports go, China is a low cost producer for many things. If the US doesn't want to buy high quality EV's costing 1/2 the price of a Tesla, then there are plenty of other countries happy to do so. The US is only 5% of the global population - there are lots of other consumers out there.
Is this really true? Can't they just keep exporting "forever"? What requires them to find a domestic audience? I can think of some other countries that have done quite well mostly without finding a domestic audience (eg, Switzerland comes to mind, and frankly Europe might largely be in the same bucket).
And a teeny-tiny country like Switzerland can't be usefully compared to something as big as China - the rest of the world can easily consume whatever the Swiss produce, but there are limits to how much Chinese can produce until consumption will have to shift to domestic consumption because the world doesn't have infinite ability to pay for its demand.
That said don't expect any big change anytime soon - there are very strong political incentives to keep the status quo with China focusing a bit too much on exports.
As someone who is myself 'old af' by tech standards (I'll be 53 very soon) the main problem is that the policymakers are all 70+ years old.
Their concept of what China even is is forever stuck in pre-2004 thinking from back when they still had significant neuroplasticity.
Vote these fucking fossils out, kids, before it is entirely too late.
(And, yeah, before someone inevitably brings it up, some exceptions-that-prove-the-rule older policymakers/politicians do avoid this trap of getting stuck in the past, but most don't)
Over the last 20 years the economy has become dysfunctional. It no longer really resembles a free market; monopolies have established barriers to entry everywhere. And the biggest investors are awash with helicopter money that's been doled out for favors by the political class.
So those investors have tons of cash to burn and surprisingly few opportunities. Even within Silicon Valley/VC there are surprisingly few who seem to really understand the fundamental economics of software. Or perhaps those economics just aren't that important when you have billions of dollars on hand and cash is obviously not going to get you a return. Any whisper of possible exponential growth is worth throwing money at. Crypto? Why not. AI? Why not. Datacenters? Why not. Tulips? Why not.
This is by all definitions an empire in decline. Everything is broken or fake. Everyone is afraid to do what needs to be done. Power forbids it. So we're all just waiting for the other shoe to drop. Our secret police aren't as bad as the late stage USSR's yet, but hold Uncle Sam's beer...
(That's not a recommendation to try and time the nadir, by the way, as it could easily be 50 years away.)
Overfinancialisation of the economy while removing safeguards to keep markets functional, like anti-trust enforcement, are the main forces behind this erosion. Through finance the focus of companies is completely shifted away from producing good products and services, and into how to extract more paper wealth from existing structures to the detriment of products.
Not enforcing anti-trust and letting behemoths to form which cannot be competed against since with their amount of capital they can either buy their competition outright or just price dump for long enough to make competition non-viable.
It's the failure of neoliberalism, and that agenda has been pushed into Western countries since the 1980s-1990s, it made enormous wealth for the few at the top while eroding whole societies, economically and socially, it's all dysfunctional.
[edit] The packaging needed to support HBM is also much more expensive too. If demand for current HBM applications tanks and the manufacturing lines need filled, then maybe. Currently, the price point would make it very very niche.
Soldering the DRAM onto a PCB is such a reliable process that there is almost zero risk of defects and even if a defect occurs the damage is limited. If the DRAM is soldered onto a DIMM the risk of a defect on the non memory hardware is non-existent. If the DRAM is soldered straight onto an SBC or GPU, then the DRAM can be removed to save the precious SoC or GPU chips.
Meanwhile HBM is the ultimate nightmare scenario. You stack up to 16 DRAM wafers on top of each other. One defect and the whole stack is worthless and that was actually the easy part.
In stage two things get even worse. You now have your accelerator chip and you must place the HBM on that chip. E.g. Blackwell GB300 has eight HBM stacks and the accelerator chip has a bigger area than the HBM. You must get the packaging right eight times in a row or you have wasted not only the DRAM silicon, but also the accelerator silicon because HBM cannot be removed and defects are permanent.
The issue here isn't just the yield of the HBM (which is obviously lower if you have taller stacks) but rather the yield of the combined HBM-based product, which is why doesn't make sense to say it needs more area but it is completely correct to say that HBM leads to more silicon being consumed. Hence it doesn't make sense to talk about yield of the HBM itself, because it is always part of an integrated product.
Those two combined already result in a huge reduction in bytes per mm2, so with the same wafer processing capacity you're producing far less byte of memory. Add to that a complicated chain of HBM-specific packaging steps, and you're now also losing a decent bunch of perfectly-fine dies because rather than putting it into DDR you tried making a HBM sandwich and screwed up.
Even if the memory cells are the same and have an absolutely identical yield, HBM will always end up having a significantly lower output. That's just the cost of stacking, but some people are willing to pay the per-gigabyte price penalty in return for the higher bandwidth.
This might not be far off the mark. You are irreversibly linking the fates of these devices after a certain stage of manufacturing. If something goes wrong at final packaging time, you lose all dies instead of one.
And memory is already expensive. It's downright hard to even get it though - you frequently would prefer not what's cheapest, but whatever is in largest scale production.
The only way I see HBM becoming competitive for consumer CPUs is if they solve the yield issues. Or if AI crashes so hard that people are putting those GPUs on fire sale and salvaging mass quantities of HBM off of them.
I’m assuming that it’s mounted in the same unit as the GPU, not in a handy-dandy socket.
It could be cracked open and extracted no doubt but if it’s glued in that’s going to be nigh on impossible to extract.
I had been pinning my hopes on HBM coming onto the second hand markets after there three years or so of use, perhaps I’m wrong.
Consider the 'MMA N matrices' primitive modern CPUs are starting to support. For the current generation of CPUs, N is a constant like 16 or 32, but there's nothing preventing it from being 1024 or larger if we have more memory bandwidth.
All this with a single instruction.
If you have infinite memory bandwidth you just move the bottleneck back to compute so both have to grow simultaneously in lockstep.
What you should have said is that CPUs have so much compute headroom for matrix vector multiplication that simply adding more memory bandwidth would make them faster so every improvement in memory bandwidth is welcome.
The "move the bottleneck back to compute" bit is changing rapidly though. The first time a major hardware company ships a PIM chip, you can push for a few orders of magnitude more data through without being compute bound.
We'll have to figure out how to read and right to the surface of a black hole to get speeds that high.
This distinction doesn't change what the performance numbers look like today, but it does inform what changes would be necessary for those numbers to look different tomorrow. E.g. Apple Silicon isn't fundamentally orders of magnitude more efficient than x86, they just used smaller features. Newer Intel and AMD chips made on equivalent processes _also_ get similar efficiency gains.
And honestly, they have historically had different markets.
When the design is for only one customer, you don't need to generalize things, and those things you generalize to give different customers different options has costs.
AMD will soon be a larger customer for TSMC than Apple (NVIDIA is already there) so Apple's pre-booking new processes is likely to be gone in the near future.
If I get 10% more performance for 50% more cost it really depends on one's needs, for example.
Keep in mind that non-Pro threadripper is still only 256 bits wide and Pro is 512. And the memory is 30% slower than with an M5. So an M5 Ultra has 3x the memory bandwidth of the best threadripper.
It's the address space that's unified, not always the physical hardware.
The data movement (when needed) is handled transparently in the background by page faults and other tricks.
DDR is optimized for latency and stability at the cost of bandwidth whilst GDDR is optimized for bandwidth at the cost of latency and stability. GDDR is pushed so hard these days that a small percentage of errors is expected and corrected because this is still faster than running it slower but more accurate.
GDDR7 often has 10-20x the total bandwidth but 3x the latency of DDR5. Graphical workloads want as much bandwidth as possible but care relatively little for latency. Conversely, applications love low latency but don't really see any performance benefit from higher bandwidth.
So basicallyt you have workloads that are diametrically opposed and running unified memory forces you to compromise.
This is really not a limit because of unified memory -- in principle, PCIe GPUs could read/write main memory without the CPU. But it's a limit for /fast/ unified memory, because fast means close.
So unified memory is great as long as the integrated GPU is strong enough. Then it has two advantages: a) probably faster transfer CPU<->GPU (but that's an implementation choice for the non-unified case b) If you either need a lot of memory for the CPU or the GPU, but not for both at the same time, you pay for memory only once.
The reality distortion is that people seem to believe it's HBM, or somehow it gives you extraordinary amounts of vram. Neither are really true.
[deleted]
Ram always goes through these boom and bust cycles. They're going to increase supply but not so much that they're eventually left with huge overproduction as usual.
Of course if Google researchers come up with more ram reduction tricks for newer AI architectures that could make a huge difference as well
Micron's NY fab had a press release two months ago that they started pouring the first concrete, so that's far from coming online.
If AI demand sticks around and buildouts stay cautious it could take much much longer than usual to build up enough capacity to push prices back down below $3/GB.
More efficient AI memory use risks even more demand for memory because you're now able to get even more performance per dollar; the benefit of extra memory has to be sufficiently far into diminishing returns for demand to drop overall.
You said:
> the market doesn't fit the issue.
I presume you meant "fix" not "fit". You are wrong that the market will not fix the issue. There are literally millions of observed cases across many markets of increased demand (or decreased supply) leading to higher prices leading to increased supply leading to lower prices, which is why it's a fundamental law of economics.
The price of RAM will fall.
Long-term, sure. But the article we are discussing is about next year, non-HBM supply going from 60% of their output to 20% (HBM 40%→80%).
Yes. Non-HBM Samsung supply going from 60% to 20% will cause prices to increase, which will cause increased production of non-HBM, and then prices will fall (below where they currently are).
Micron has been working on expanding in Boise since around 2023. They are predicting that the first new chips will start rolling out around 2027.
I don't think we'll see any relief until at least 2028 or even later. And even then, probably not without antitrust laws being enforced. Because you know memory manufacturers won't just simply drop prices, they have a long history of price fixing.
if I'm ordering 1 RAM and demanding some special customization, even mom-and-pop stores will just ignore me
but if I'm buying 100000000000 RAM, things change
You can own stock, but other than that?
Infinite demand isn't a thing. Even if they'd offer free compute forever (not likely possible) I and many others would still use local models that we have full control over and that do not harvest our personal data.
We're not seeing the progress in those "frontier models" that we have previously seen. There's certainly still gas left in tank tank, but we're way into the diminishing returns by now.
Cloud inference still beats hardware investments by orders of magnitude of course, but that's only if your data doesn't really matter to you.
But I suspect returns may have already diminished into negative territory for at least some other use cases. One of my least favorite job responsibilities in this brave new era is figuring out how to avoid performance and behavior regressions when an older model were using for some application reaches end of life. It’s getting uncommon for me to look at our benchmark results and say, “Oh, good, it does better on one of the newer models!”
I agree that datacenters are not going to go away, but I have doubts that the buildup that has happened is really going to pay off for most operators.
But I also don't believe things just because I want them to be true.
[deleted]
[deleted]
Why? You can't just assert it. There are very good reasons to think it won't happen soon, and you've given no reasons to think it will happen soon.
Also, the frontier can’t keep advancing at this rate forever. Eventually the low-hanging fruit will all be gone and advances will slow down.
Even over their many years, projects like the Manhattan Project, the Apollo Program, or the U.S. Interstate Highway System never added up to that. [2]
Is your argument that there is insufficient AI spending at present as they aren’t also taking on building their own fabs?
[1] https://www.pwc.com/gx/en/news-room/press-releases/2026/glob...
[2] https://www.aljazeera.com/news/2026/2/19/visualising-ai-spen...
Their choice, their consequences.
No. Please take a look at the unit price for hard drive storage 10 years ago as a percentage of what it is today.
What Altman did, and the way he did it, was not OK. Unless you believe that humanity's adoption of AI-based intellectual work is best served by concentrating an enormous amount of power in one company at the expense of everyone else, turning the market into a literal zero-sum game. Do you?
[deleted]
Alternatively, the money invested can be rational because it prices out competitors and establishes a monopoly, after which point the monopolist earns their absurd investment back with complete control of the market. This is also bad.
The problem you have here is hundreds of different companies are doing the "gambling" in a non collusionary manner so it's going to take decades in court to prove it. Are you saying governments should do an authoritarian take over of RAM allotment?
[deleted]
[deleted]
More of, China is smart enough to use the benefits of both capitalism and authoritarianism to grow their country while western governments are busy stealing the furniture and fixtures off the walls.
Of course they're trying to take care of themselves while the US implodes, how can you blame them for that? The whole world has a serious oil problem right now, because of US actions, and of course each country is dealing with it in their own way. Why is that a problem?
"Problem" is a relative term for those dealing with the consequences of it. It is one if you are America means a whole lot less economic freedom than you had in the past. If this is a good thing or not depends on your position.
> Capitalism isn't an ideology
I don't believe that even the CCP itself, in hindsight, considers the one-child policy to have been a good decision. At the very least it should have been ended some 15 years earlier.
The problem with low birth rates is assumed to be that there are less workers to take care of the pensioners. But right now it seems like even the few young workers available often aren't needed anymore, thanks to increasing automation of industries. I predict youth unemployment will be made even worse by AI soon enough. So, why do we need children, if capitalist economies don't need workers? Unemployed people do not pay much taxes to sustain the non-working age population.
I believe low birth rates might become a blessing soon enough. It just means less people for governments to take care of when robots and AI take away most jobs.
There is a handful of developed countries where this is not an issue.
Different countries are handling this in different ways, with encouraging immigration being the major one. Japan is facing a major lack of workers, with many businesses having to shut down as a result, but have decided to tackle it in a different way relying on advanced robotics as an alternative to immigration.
There are very few developed countries NOT having this problem. Most of the population growth is in poorer parts of the world like Africa.
There were forced abortions (sometimes very late), there was forced contraception, forced sterilization (I think), and lots of punishments for people who broke the policy.
There were also others who got around it fairly easily by paying a bribe and by not registering their "surplus" child/children (especially if they were girls). That of course caused some problems down the road.
That still doesn't mean that it affected China's birth rate much. It was already low (and had fallen steadily) when the policy was instituted. It also had plenty of exceptions for rural China and ethnic minorities.
It's difficult to near impossible here to determine which counterfactuals would actually hold true. Based on other world wide trends it's very likely that Chinese family sizes now would only be slightly higher than if the policy didn't exist. In the modern world there are just a huge number of different factors working together to decrease family sizes.
Comparing this to crypto or turnips makes me doubt the merit of any other opinions your comment offers.
Just to give some rough numbers in the past 5 years the amount of GPU compute installed in FLOPS is somewhere over 5 times all the CPU flops that have ever existed.
This has nothing to do with AI being good or bad or being able to produce things and economic value. It is by far the largest and fastest growth of any technology ever and we have zero clue about the economic stability of this grand experiment we're performing.
Some differences to the market at that time seems to be that during the dotcom bubble, many of the companies had little to no revenue.
Leading AI labs already generate enormous revenue. The investments into capacity are needed to address the current demand.
The situation seems somewhat less speculative.
That said, I cannot predict how AI capabilities will develop and how demand will respond.
Should capabilities plateau hard and soon, maybe the demand will not be there for the compute investments.
If it does not, and instead AI applications in robotics, science, and self driving expand, chances seem reasonably good that the demand will be there, no?
And as for the economic stability, much of the investment comes from existing giants like Microsoft, Alphabet, Amazon, and Meta, who have the necessary cash flow.
These companies are less likely to collapse than some of the ones during the dotcom bubble.
LPDDR6 PIM would primarily help the low end and mid range accelerator market. E.g. embedded models running on SBCs can be up to 1 GiB in size with acceptable performance, small models at 8 GiB become really easy to run at reasonable speeds on a smartphone and PIM enabled laptops or mini PCs make it possible to run 32 GiB models locally without compromise.
Of course this also assumes that the associated accelerators (NPUs) will catch up too, but the general point is that you won't need a 5090 or a 4090 anymore.
[deleted]
There are so few players in this space they can coordinate and collude pretty easily.
That wasn't me.
Also they're talking about the short to medium term. On that time scale they're not wrong. The market is so far from fixing it right now, it's going in the opposite direction. The market is chasing HBM money which is outbidding everything else.
> The price of RAM will fall.
And if that might take 5+ years, that's not much comfort.
> higher prices leading to increased supply leading to lower prices
So far higher prices are causing barely any supply increases. (Note that the article we're commenting on is about a supply decrease.) Companies are barely trying to expand, and even if they do start trying hard it'll take a very long time.
Maybe I'd be willing to pay $10 for product A if I had other options. But if there's a product B for $3 that's not quite as nice but still ticks all my boxes, then product instantly becomes a lot less attractive.
Part of the reason harnesses work well is you can run a lot of agents in parallel. That doesn't slow down demand.
LLMs are amazing tech, but they're terrible without oversight. More agents faster just makes reality collapse on them quicker.
But yeah, you're right, temporarily, this will still push demand. But the topic was about "diminishing returns" as in "tech getting better". Not as in "customer spending".
New models keep being able to use more and more agents on longer time frames. Your hypothesis doesn't look like what we're measuring.
Is it just that the providers are generating tons of synthetic datasets on coding tasks so that the models get more exposure to the right thing to do? Every time someone points out an LLM stupidity they add some training data to patch over the weakness (trivial to generate "there are two 'l's in llama")?
https://scholar.google.com/scholar?hl=en&as_sdt=0%2C23&q=llm...
https://proceedings.iclr.cc/paper_files/paper/2026/hash/3b4e...
It’s also poor reasoning to you only pick one thing you think supports your view, and ignore vastly more things not supporting it, all from the same useful criteria.
Now if only you’d carefully read the report you chose, and spend equal time looking at ample presented evidence, you’d develop a more accurate understanding.
Before that, we had 0. After that, we had more than 1.
A leap as far as that is hard to recreate.
But that wasn't my point. That's just trolling.
The actual point is that LLMs aren't gaining new capabilities anymore. They just get more reliable at the ones they already have; turning what was a coin flip to some higher probability.
That's (intuitively speaking, not strictly mathematically speaking) kinda the mathematical definition of diminishing returns.
Coorperate people likely just want “the best” and are happy to pay a lot for it to make their workforce more efficient.
It’s hard for me to guess where the data center buildout will go, eventually there will be an oversupply, but we haven’t hit that yet, and at the moment the companies owning these high capacity data centers appear to be making lots of very real money in leases to hyper scalers, hard to imagine this premium continuing, but we need to see where demand for cloud based LLMs tops out.
Who is so emotionally invested into random comment sections being purely positive about their pet.. uuuuuuuh.. tech?
Very weird.
I expect something similar for AI. It's useful tech... just not very profitable tech, and certainly not to the tune of trillions of dollars worth of public demand. The promises of superintelligence, replacing everyone, and the rest of the hype will die with the companies who made the promises, but the tech will survive and thrive.
The hype will die but there are many reasons why super intelligence and robots are a separate entity from said hype.
Look, back a few decades and tell someone our gdp would be in the trillions and it's likely they'd have a hard time believing you. A huge portion of our products that we use day to day would be complete science fiction to them.
All we're negotiating at this point is the timescale it will take.
But in my mind robots and superintelligence are inevitable unless there is a massive setback in humanity before then. Nature already did it once. We're not inventing something totally new. Add in every new invention and bit of intelligence we build on and actualize pushes us that much closer to the goal. Information technology allows this to speed up even further with easy sharing of information and testing.
Superintelligence is not like faster than light travel. We have many frameworks that show us FTL is impossible. I don't believe there is a single widely accepted framework that shows there is some limit to intelligence and we're near it. Intelligence isn't even a singular thing. My calculator is a super intelligent adder compared to me. It would seem highly improbable that somehow nature random walked into making brains the most efficient general intelligence device in all dimensions and scales.
If you're fired up by nationalism but your nation is sucking, getting mad at the Chinese isn't the answer. And that's before we even get into relative levels of activism, Chinese political ideology is much more biased towards "neutral, just trade" than American politics.
Near as I can tell, they fundamentally believe that the US actually can’t do better (despite protestations to the contrary), at least not with them there doing there thing, which is why they get so manipulative and shitty. They are unwilling to make other decisions than the ones leading to these outcomes.
Yes people, it is actually possible to do better. It requires making better choices, however.
Going purely by the evolution theory and natural selection, having so few children that there is a serious risk a population will disappear in a century or two, means losing. Needless to say, this is stupid. From the entire American continent, to Europe, India more recently, Thailand, China, the Koreas, Japan.
Africans, Arabs, and some non-Arab Muslim populations will carry the torch. Much respect for not falling into the trap.
I'm not saying fast fall in birth rates isn't a problem, it obviously is. However, nearly all countries that don't have this problem offer much worse quality of life than those which do.
There is a saying. More is better. Morocco has issues with tourists and diaspora pricing out locals from houses to food even, but Algeria is an honestly good country all around. With its issues of course. But there is more to life than earning $$$.
Westerners forget the tiny detail that these countries are extremely young unlike them, hence why it feels like there are so "many" people.
Edit: oops, I glossed over the part about Israel. Trust me, it's still a billion times better than nothing, which is the case for the entire West.
Edit 2: Oh my oh my, I missed this too. I "loathe" European countries? Do I? I mildly not respect European populations for wiping themselves out. I cannot loathe land.
I don't know the exact correlation but even in Morocco, total fertility rate is declining very rapidly, down from 6 children in 1980s to now around 1.7 in urban areas. UAE (Dubai) has similar TFR to Belgium or Portugal.
Honestly you're reading the data completely wrong. You cannot look at it at the country level. In Israel for example the populations that are producing the most population are highly religious and less educated than those that are producing less population.
It's more about memes. Religion is a meme driven breeding cult. This is why said religions tend to have very long term staying power. The particular problem here is these religions come at a cost of having to ignore a lot of reality to get to that point. Because they are faith based anything that presents a direct challenge to that faith must be setup as an enemy or it may risk the religion. Yes, religions can incorporate new information but it tends to be a slow process as it needs to involve slowly changing peoples mind to avoid instability.
What this looks like in the long term is large populations of highly religious that have a fundamental disconnect from the technology that's keeping them alive. A new Late Bronze Age collapse.
And if that fails, we've got "Idiocracy" to save the human species as the dumb-asses pump out babies as fast as you like.
It's the total amount of money in the economy that's been invested toward a potential outcome. AI represents the largest amount of money, and largest fractional part of the economy invested ever.
Because of this AI could be the biggest economic boon ever, yet still not recover the full amount invested. This will have deep economic impacts that affect everyone and everything. At this point AI must achieve all its stated economic impacts or there will still be a huge economic crash that kills off any company that is over invested and cannot make a profit.
Worse, the many of the perceived economic impacts of AI are not for humans like you or I, but the huge companies you listed. Even if they economically win, everyone else made out of meat could still lose.
They say history doesn't repeat, but it does rhyme. This, at least to me, sounds like a mixtape of "internet" + "tulips" + "1920s financial world leading to global political instability".
Every potential outcome I see occurring pushes us closer to further instability, even if the economics on it work out on paper.
I had a look at the numbers.
The investment into data centers is estimated around $800B/year currently.
OpenAI + Anthropic combined had ARR of >$100B in July.
Global labor income is around $65T/year, the US GDP $32T, the global one $126T.
Global software spending: $1.4T/year.
I am no financial analyst, but I don't think all the stated AI impacts have to be met just to recover the investments.
A 1% productivity gain on global labor income represents $660B/year. At 5% we would look at $3.3T/year.
They don't need to cure cancer to justify the investment, even though Amodei hopes to cure most major disease in the next 5-10 years.
>Global software spending: $1.4T/year.
Out of this number how much of this goes to payroll that goes to humans versus how much of this goes to non-human infrastructure costs. The numbers suggest this is anywhere from 60 to 90%, with 70% being a reasonable average figure. Just under a trillion dollars paid directly to humans would disappear if AI somehow captured all of this market.
Moreso the AI industry needs to capture these gains very fast or they will be crushed by interest payments on the massive debts they've accrued.
>ven though Amodei hopes to cure most major disease in the next 5-10 years
The average development time for medicine is 10 to 15 years before a single dollar is earned from said medicine. AI will shave very little off that as human testing and our stupidly complex bodies introduce all kinds of problems. Furthermore running head long into blindly using AI medicine is how you produce X risks from super intelligence AI.
The speed at which AI has to develop in which to get profitability is the biggest risk, a potentially catastrophic risk at that.
The reality is that so far there has been no sign of GDP growth picking up. It is basically flat at 2.5% +/- over the last few years.
In a similar vein one might have expected that the internet (think of all the e-commerce and efficiencies!) might have shifted GDP growth into a higher gear, but it did not, although in that case there was at least a significant boost in the 1996-2000 "dot com" era when the build out was happening (then to be followed by the crash and all the unused dark fiber etc).
So, maybe the hoped-for AI boom will be just as much of a dud (as it appears to be so far) as the internet boom. New day, different tools, same growth.
It's perhaps odd that we're not even seeing datacenter/etc build out register on GDP, but perhaps the scale of it is not as large as the internet build out?
[dead]
Uh... Meme-based? This is such an unserious argument I don't even know what I can possibly say to that.
How are they disconnected with the technology keeping them alive? All people, in GCC countries especially, are working to make 1st world development happen.
Implying that religious Jews or Muslims are stuck in the Bronze Age is ignoring reality. Most European development itself came from men of science who were Christian, I thought this apparent conflict of ideas was something taught in schools.
GCC countries are resource economies mostly. They must weave a very narrow path to success to non-oil based means of income or the entire region collapses. I will say they do realize this is a problem and are using oil money to diversify, but resource curses are very hard to break and collapse can happen very quickly.
"My grandfather rode a camel, my father rode a camel, I drive a Mercedes, my son drives a Land Rover, his son will drive a Land Rover, but his son's son will ride a camel." --Sheikh Rashid bin Saeed Al Maktoum
>Most European development itself came from men of science who were Christian
This is just stating that that in 1700 you were a Christian or you got hung (hell, pre-independence America was founded by people that thought that too few people were being hung). As we see when you take out the hanging or social shame part of the religion people tend to flee them as they are oppressive structures. You could say them being men of science either had no bearing to their religion, or was in spite of their religion.
I am saying, for example, that the Abrahamic religions are bronze age (ok, sorry, Iron age) social structures. I don't think studied scholars are going to disagree on that point very much. People saw how people worked and could be sheppereded in societies without formal education and turned it into a series of stories that were effective in modifying peoples behaviors. Kings later came along and added their stories so they could further control the population. Over time a semi-coherent story about how people worked was formed below a set of rather incoherent stories regarding things that didn't happen that people are supposed to accept with faith.
(Comic book superheros, cough cough. The references to Canon in what are essentially liturgical disagreements - I mean power scaling arguments - are particularly hilarious.)
Not me of course, I’m different and unique. Just like everyone else.
That's 0.6%. Subtract last year's investment and then we're talking yearly growth.
Turns out it's not the earth shattering capital investment the comment before mine was making it out to be.
And about productivity gains, AI became good at software development in spring this year.
I'm not sure what productivity miracles reflecting in GDP growth you already expect to see by now.
In the past 5 years 5 times as much GPU compute capacity has been installed as total CPU capacity that has ever existed. When it comes to capital intensity the growth of AI has been one of, if not the largest capital expenditure for a class of project in this short of time frame. Things like the Manhattan project which was absolutely monstrous expense at the time is dwarfed by this.
Of course measuring GDP growth is difficult when you have an idiot in chief trying to destroy the global economy with the dumbest set of actions contrived by a human ever. Disentangling this factors will take a lot of work on someones part.