This is Uber/AirBnB/Wework/literally every VC subsidized hungry-hungry-hippos market grab all over again. If you’re falling in love because the prices are so low, that is ephemeral at best and is not a moat. Someone try calling an Uber in SF today and tell me how much that costs you and how much worse the experience is vs 2017.

OpenAI is the undisputed future of AI… for timescales 6 months and less. They are still extremely vulnerable to complete disruption and as likely to be the next MySpace as they are Facebook.

It's a race to the bottom on pricing on the provider/infra side. It seems very unlikely that any single LLM provider will achieve a sustained and durable advantage enough to achieve large margins on their product.

Consumers can swap between providers with relative ease, and there is very little stickiness to these LLM APIs, since the interfaces are general and operate on natural language. Versus something like building out a Salesforce integration and then trying to update that to a competitor. Or migrating from Mongo to DynamoDB.

Building the LLMs is where the cool tech lives, but surprised so many are seeing that as a compelling investment opportunity.

But, let's see!

(If you can train your internally deployed LLM on data none of your competitors have, that's an advantage).

This could be one of the more interesting privacy fights of the next decade.

I’m sure there are easy cynical takes about how they will just shrink wrap the EULA, and maybe they will. But in a good privacy environment, users should never be surprised and have control over how their data is used. And I think we’ve made some progress there.

If there's one company that I don't think cares about user permissions or the law, it'd be Twitter.

The EU officially warned Elon about DSA fines and the response was less than serious.

https://www.cnn.com/2023/10/10/tech/x-europe-israel-misinfor...

Cloud infra may be a comparable market, since computation is a big share of AI costs. Did consumers win big from competition between AWS, Azure, and GCP? Not sure. I see an uptick in write ups saying “We switched off cloud and reduced costs by 2/3rds.” Not a scientific sample but may leave the question open.

Several very big profile names have recently begun moving back to self-hosted, hybrid, or dedicated hosting solutions.

Cloud computing never was good in terms of value, however, it was only good in terms of scalability. AI solutions built on top of 'the cloud' will always be even worse.

Note that I pay a certain "third tier" cloud provider less than $100/mo total for hosting large websites that would cost me more than 10 grand a month on AWS/Azure/Google...while having better uptime. (the biggest differences? the complete lack of IO and bandwidth charges, and much lower storage charges)

That should tell you all you need about these types of bubbles, but then again, most of us that watched the entire tech field unfold since the pre-internet phase already knew this.

The prime selling point for cloud for large enterprises was (and still is):

- a signatory that shares blame on several core security issues (iso stuff) - high amount of flexibility for individual teams used to asking for a vm then waiting four weeks for the itops dept to bring it online

Now the vast majority of cloud moves for large enterprises ends up as a shitshow due to poor implementation sure, but the key points for getting it sold are still there.

And ofc you CAN still cost optimize with cloud, its just harder.

Context: Worked in post and pre sales over some years in MSFT in the enterprise segment.

I got out before the downturn and everyone talking about cost, but my approach in selling azure to the c-suite would be fairly similar today I reckon.

Could you provide some examples?

Some random survey from ESG found 60% of respondents repatriated at least some workloads. Who knows what the N was though.

Source: https://www.sdxcentral.com/articles/thinking-about-cloud-rep...

Frankly, it makes sense that companies are deciding what works best and where. But the death knell of cloud providers is simply not happening.

Margin is a function of stickiness/cost of switching (among other things).

I suspect eventually we will enter a world where migrating cloud providers is mostly a click of a button, but we're a long ways off from that. Requires vendor agnostic and portable apis/containers/WASM runtimes everywhere.

Swapping an LLM, at least in the current state, is about as close to updating a pointer as you can get.

This will almost certainly lead to a prompt rebuild, to better accommodate new model idiosyncrasies.

If you are unlucky- your use case may be one where Evals require human review.

Unless you are YOLOing it without evals. In that case this is relevant.

Especially given that many orgs use managed or cloud specific solutions that have no 1:1 mapping between vendors

As someone who worked on a similar project recently, I'm getting that idea that you obviously don't know what you're talking about.

You need to plan ahead, sure. Portability was one of my main concerns (including the possibility to go self-hosted). But it's definitely not impossible, nor too hard to do.

And Kubernetes? It goes way beyond containers, you can replace many cloud provider-specific resources with Kubernetes resources. What alternative gives you that?

(Probably also less reliable than "cloud", but who cares if you're a startup.)

Learning aws vs learning how to operate a vps are of comparable complexity

It took me less then a day to setup infra for my startup (more than 10 years ago)

But tell me what third tier cheap provider has managed scale-to-zero-or-infinity functions? Managed storage with S3-like API? Where can I get an API gateway cheaper than Amazon? What about managed databases? These tools allow me to develop insanely scalable software incredibly easily.

Agreed - all the big clouds are very expensive VPS hostings. Don't use it for that.

You can scale from 1 thread 512MB RAM, to 500 threads and 12TB of RAM (off the shelf). Which is good enough for almost everyone who isn't planet scale.

Auto scaling also comes with auto billing. Oops, your accidental infinite loop spawning functions has bankrupted your company. You don't have that risk starting with a VPS.

What's third tier here? Pretty much all of them offer S3 compatibility it's basically table stakes

I try to not use anything else if I can avoid it on a new project.

Ingress gives you the ability to load balance and is threaded and will scale with network transfer and cpu. Database should scale with a bump in VM specs as well, CPU and disk IOPS.

If you keep your app server stateless you can simply multiply it for the number of copies you need for your load.

Systemd can keep your app server running, it you docker it up and use that

This right here so many times over.

I'm not going to spend time worrying about scaling, I'm going spend time figuring out how to make stuff stateless.

For new small systems I suggest you start with a cloud VPS. If traffic is low, cost of downtime is low, and system requirements are high then a cheap mini PC ($150) at the home or office can keep your bill microscopic. If your app server and database are small then you can just throw them on the VPS too.

I run light traffic stuff at home in a closet so it doesn't occupy more costly cloud RAM. Production ready saas offerings I'm trying to sell right now are all in the cloud. Hosting all my stuff in the cloud would cost me hundreds per month. My home SLA is fine for the extras. I don't need colocation at this time, but I have spoken with data centers to understand my upgrade path.

You can run a live backup server at a second location and have pretty good redundancy should the primary lose power or connectivity.

When system requirements elevate (SLA, security, etc) you probably want to move into a data center for better physical security, reliable power and network. Bigger VPS is fine if it is big enough. Can also do a colocation if you don't want to rent, and you contract directly with a data center. I wouldn't look at colocation until your actual hosting needs exceed at least $100/mo and you're ready for a year long commitment.

> Cloud infra may be a comparable market, since computation is a big share of AI costs. Did consumers win big from competition between AWS, Azure, and GCP? Not sure. I see an uptick in write ups saying “We switched off cloud and reduced costs by 2/3rds.” Not a scientific sample but may leave the question open.

If the cloud had not existed, those that claim they saved money switching away from cloud might never have been in business in the first place.

Some will scream in horror but I wanted Netflix to be a monopoly. A single place and app and account with all the content I need.

"competition" in streaming space has been nothing but disastrous for me as a consumer. It led to greedy heterogeneous islands of content, with proliferation of crappy apps and pointless restrictions and return to cable package mentality.

Again, Possibly irrationally and ignorantly, my fear is that 5 years from now I'll need a dozen subscriptions to less good services which will hoard their source data and models and be specialized based on which content they got licenses to. I. E. There'll be ai1 with new York times and Wikipedia, and ai2 with Washington post and encyclopedia Britannica, and ai3 with I don't know fox news and RT, and ai4 with mit and Harvard business libraries, and ai5 focused on math with extra subscription to wolfram, and ai6 with rights to stack overflow and JavaScript and so on.

There are many scenarios various writers have posited where we are actually in local maxima lf ll, with data being increasingly closed and or poisoned, and possibly segregated in the near future. :-/

If you run it you get some money and higher balances mean better treatment when it takes over.

Distributed evil AI evaluation in exchange for protection from it. You can trade the protection on the free market.

If other vendors LLMs become good enough it will actually be easily to interchange and then the race for the best UX and integration will be upon us (which the other commenter alluded to).

"if other llms are good enough" assumes that in principle they have same opportunities, access to same data, or content. My fear is precisely that this assumption may be taken away - I. E. That news paper publishers or encyclopedia owners or big websites (stack overflow, web Md, etc) will enter into arrangement with specific llm companies - just like Netflix Disney prime etc aren't competing on their app or price or flexibility, but on exclusive underlying content. Nobody WANTS to subscribe to Paramount+ or cbs access... But if they hold enough material hostage some people will 'have to'. I can see a future, not far off, where different llm organizations selling feature is not how good their technology is - to your and overvodys point, THAT moat is likely to even out - but what underlying training data they have legal access to.

We're not socially ready for one AI winner. The resulting giant would be too powerful and too influential.

I am convinced that's how they will end up.

AI models have some GPU constraints but could easily reach a state where the cost to opperate falls and becomes relatively trivial with almost no lowerbound, for most use cases.

You are correct there is a race for marketshare. The crux in this case will be keeping it. Easy come, easy go. Models often make the worst business model.

I'm willing to bet dollars to doughnuts that Google and Facebook have at least one, possibly 2 or more orders of magnitude more latent training data to work with - not including Googles search index.

My uninformed opinion is that Google and Meta's ML efforts are fragmented - with lots of serious effort going into increasing existing revenue streams with LLMs and the like being treated as a hobby or R&D projects. OpenAI is putting all its effort into a handful of projects that go into a product they sell. The dynamic and headcounts will change if the LLM market grows into billions

It seems more likely that at Google at least they just fell into the classic innovator's dilemma in which they were stuck trying to apply innovation to their current business models in an attempt at incremental innovation instead of seeking an entirely different customer and market.

Both have a large graveyard of failed innovation attempts. Remember Google Buzz? Google Plus?

They're not as good at innovating, that's why they acquire startups all the time. It's a blood transfusion

If OpenAI survives all the legal challenges, they'll just click "Go" and be in business in weeks to months.

If OpenAI gets smacked down, they haven't lost much.

There are probably also some submarine operations that are already doing/have done the training. If OpenAI gets bankrupted for copyright violations, we'll just never hear from those.

Not sure any startup would be ok with sitting on a potentially gamechanging model.

Google/Facebook? Maybe?

Amazon's LLM efforts are pretty much in shambles

For what reason do you doubt that?

They trained a custom model for this. Better accuracy, sure, but I was a little surprised to watch how much faster it is than GPT4.

Based on their testing, they’ve become believers in domain specific smaller models, especially for performance.

However, I am skeptical that smaller and more focused models will do well for more general tasks. I’ve found gpt-4 to just be flatly excellent for tasks that involve emitting a custom DSL and customer-provided business context (the latter being something we cannot train for, not without immense expense).

That is a beautiful metaphor, seriously.

I just took Lyft again to the airport earlier this month same location and I was billed $49 USD, and a $1.30 “Texas Surcharge”.

An inflation calculator says that $38 usd in 2015 is equivalent to $49 in 2023. Color me surprised. I thought the prices had significantly increased since I signed up but it looks like actually no they didn’t.

Trawling back through those old emails I do see constant “50% off all weekday rides” offers from the time I signed up until about March 2016, at which point they stopped. So there were some subsidized incentives when they were early in Austin but it looks like they stopped sometime in early 2016. So if the money train existed, it happened before that, at least in Austin.

It is for the consumers of these models, is there even a point to train your own or experiment with OSS!

If you need chatGPT4-quality responses they aren't close yet, but it'll happen.

I hate this. Not the best will win, but the one with the biggest pockets. Nothing that helps with technofeudalism. Proper competition would be good.

I like Uber; it’s convenient and fairly reliable. Five milliseconds after Lyft creates a better experience, I can switch.

Both are B2C and have network effects.

I will happily take a 1:1 bet on them not going the way of MySpace in, say, 5 years!

If anything is going to be based on how good the tech is, it's this.

https://arxiv.org/abs/2305.08377

and shows how LLM technology has a lot more to offer than "ChatGPT". The real takeaway is that by training LLMs with real training data (even with a "less powerful" model) you can get an error rate more than 10x less than you get with the "zero shot" model of asking ChatGPT to answer a question for you the same way that Mickey Mouse asked the broom to clean up for him in Fantasia. The "few-shot" approach of supplying a few examples in the attention window was a little better but not much.

The problem isn't something that will go away with a more powerful model because the problem has a lot to do with the intrinsic fuzziness of language.

People who are waiting for an exponentially more expensive ChatGPT-5 to save them will be pushing a bubble around under a rug endlessly while the grinds who formulate well-defined problems and make training sets will actually cross the finish line.

Remember that Moore's Law is over in the sense that transistors are not getting cheaper generation after generation, that is why the NVIDIA 40xx series is such a disappointment to most people. LLMs have some possibility of getting cheaper from a software perspective as we understand how they work and hardware can be better optimized to make the most of those transistors, but the driving force of the semiconductor revolution is spent unless people find some entirely different way to build chips.

But... people really want to be like Mickey in Fantasia and hope the grinds are going to make magic for them.

You say that training datasets will win, but this is where OpenAI is currently have a big leg up: Everyone is dumping tons of real data into them, while the LocalLLM crowd is using GPT-4 to try to keep up.

We will see who is faster.

I am unconvinced by the idea of trying to redefine Moore's Law to be about MSRP. The NVIDIA H100 has twice the FLOPS of the A100 on a smaller die. That's Moore's Law, full stop. When NVIDIA has useful competition in the AI space, they'll be forced to cut prices, as has reliably been the case for every semiconductor vendor for the last 60 years.

Neural nets don't need fully precise digital computing. Especially with quantization we're seeing that losing a bit of precision in the weights isn't impactful. Now that we're serving huge foundation models with static weights there's an enormous incentive to develop analog hardware to run them.

Mark my words, this will lead to a renaissance in analog computing, and in the future we will be shocked at the enormous waste of having run huge models on digital chips.

Just think, how many multiplications per second is the light refracting through your window right now clocking? More or less than is required to ChatGPT do you think? If only the crystals were configured correctly and the patterns of light coming through could be interpreted...

It really depends where you draw the lines, because you could also say that one single transistor in my electrical CPU is doing a kerjillion calculations for all of the atoms and electrons involved.

Fascinating stuff - a synchronous distributed system allows treating 1000 chips as one, knowing exactly when data will arrive cycle-for-cycle and which network paths are open. The compiler can balance loads. No more indeterminism or complexity in optimizing performance (high compute utilization). A few basic operations suffice, with the compiler handling optimization, instead of 100 kernel variants of CONV for all shapes. Of course, it integrates with Pytorch and other frameworks.

https://youtu.be/hk4QGpQAvSY?t=57

You have to create a prompt/function that for a wide set of inputs, generates a token sequence that will perpetually expand in a manner that corresponds to an externally observed truth.

Way too often it feels like you have to shove a universal decoding sequence into a prompt.

“Talk your steps, list your clues, etc.”

Just trying to luck into a prompt that keeps decompressing the model/ generating the next token that ensures the next token is true.*

I recall there was a paper with a relevant title recently… https://arxiv.org/abs/2309.10668

Basically - LLMs don’t reason , they regurgitate. If they have the right training data, and the right prompt, they can decompress the training data into something that can be validated as true

——-

* Also this has to be done in a limited context window, there is no long term memory, and there is no real underlying model of thought.

Better beyond a certain point is unlikely to be competitive with the cheaper models.

For example, I struggle to see DALL-E winning over Firefly if Firefly is integrated into a very rich environment, whereas DALL-E is basically prompt UI only (while DALL-E 3 is a better model IMO).

In that case you might be able to get linguistic competence with a much smaller model that you end up training with a smaller, cleaner, and probably partially synthetic data set.

Add all the downloadable material at archive.org and you've got a formidable corpus.

https://commoncrawl.org/

What I would really want though is an uncensored LLM. OpenAI is basically unusable now, most of its replies are like "I'm only a dumb AI and my lawyers don't want me to answer your question". Yes I work in cyber. But it's pretty insane now.

I'm not sure, there are 100+ yo companies like Kodak, Nikon, IBM, Panasonic, GSK, Merck, etc. With that in mind, it's not hard for me to imagine that some of the tech giants will also have a presence in hundred years, maybe not as dominant as today, but they could survive.

Coca-Cola, Nestle, and Unilever. Easy bet. Coca-Cola is already 137.

Apple did good with the M processors though

The point of surge pricing is to rebalance supply and demand when demand for rides outstrips supply of drivers, whether by attracting more drivers or by discouraging price sensitive riders.

Some riders at the airport will strongly prefer Uber, for whatever reason, so they're less price sensitive than you. Because you're happy to substitute an Uber for a taxi, you decrease the demand as a response to the surge pricing, preserving the limited supply of drivers for the riders who really want them (or are at least are price insensitive enough to pay for that privilege).

It's not clear there's any more of a long-term market for this any more than there is for compilers. I'm kind of scratching my head trying to figure out where this assumption there is one comes from.

I've talked to a good amount of businesses and 90% of custom use cases would also have negligible AI costs. In my opinion, unless you're in a super regulated industry or doing genuinely cutting edge stuff, you should probably just be using the best that's available (OpenAI).

I don’t understand how people are surprised by this anymore.

So yeah, it’s the best option right now, when the company is burning through cash, but they’re planning on getting that money back from you eventually.

Genuine question, what are some examples of companies in that "hiking the price" camp?

I can think of tons of tech companies that sold or sell stuff at a loss for growth, but struggling to find examples where the companies then are able to turn dominant market share into higher prices.

To be clear, I'm definitely not implying they are not out there, just looking for examples.

One thing I'll add is that it's not always that this ends with higher prices in an absolute sense, but that the tech company is able to essentially cut the knees out of their competitors until they're a shell of their former selves. Then when the prices go "up", they're in a way a return to the "norm", only they have a larger and dominant market share because of their crazy pricing in the early stages.

But yeah the movie thing is not at all unrealistic here. Though at night I usually wave the torch on my phone to attract attention because they don't always see a raised hand.

At the busiest time it's a bit harder but at that time the ride-sharing services are also overloaded so it's still faster to just wait for a green light (free taxi). We don't get Uber as far as I know but we do have a similar thing called Cabify. But it's useless if you need something quick and they've put the prices up too much. I now only use them for scheduled stuff like airport dropoffs.

There's no way I want to ever wait on hold for a dispatcher again, or be mystified as to when my cab is arriving (this always seems to involve standing outside in the snow or pouring rain).

If the cab companies have apps comparable to Uber and Lyft, sure, I'll give them a shot.

[1] https://kobedigital.com/google-maps-api-changes

- Google is showing more and more ads over time to power high revenue growth YoY

- Unity has just tried to increase its prices

Uber/Lyft did raise prices, but interestingly (at least to me) is that if the strategy was the smother the competition with low prices, it didn't seem to work.

Unity is interesting too, though I'm not sure it would make a good poster child for this playbook. It raised prices but seems to be suffering for it.

As for Unity, they're certainly dealing with a bunch of underperforming PE and IPO-enabled M&A on the one hand (really should have considered that AppLovin offer, folks), but also just a failure to extract reasonable income from their flagship product on the other; I don't think their problems come from raising prices per se (game devs pay for a lot already, an engine fee is nothing new to them) as much as how they chose to do it and the original pricing model they tried to force on their clients. What they chose to do and the way they handled it wasn't just bad, it was "HBS case study bad."

At some point, someone else will make a competitive model, if it’s Facebook then it might even be open source, and the industry will see price competition downwards.

I don't know how they could _not_ incorporate customer usage to improve their models.

There is no such thing as an open source Google because Google’s value is in its vast data centers. Search is hard to train and hard to run.

GPT4 is not that big. It’s about 220B parameters, if you believe geohot, or perhaps more if you don’t.

One hard drive.

Whereas the underlying algorithms behind all these GPTs so far are broadly same. Yes, OpenAI does probably have better data, model finetuning and other engineering techniques now, but I don't feel it's anything special that'll allow themselves to differentiate themselves from competitors in the long run.

(If the data collected from a current LLM user in improving model proves very valuable, that's different. I personally think that's not the case now but who knows).

rephrasing this for LLMs instead of search: "you can create your own model architecture/training method, but you can't crawl the web and serve language query results to billions of worldwide users in a few milliseconds."

that checks out, right? Google/search == """Open"""AI/LLMs still seems like a decent metaphor to me.

That is the moat. For developer platforms, it's all about building mindshare and adoption. The more people who know how to use OpenAI, the stronger OpenAI's position on the market. It doesn't matter if there's equivalent or slightly better models unless they start to fall significantly behind (and they're currently well in the lead).

People will figure out what OpenAI is doing and duplicate it. There’s many people working at OpenAI, it’s going to leak out.

e.g. As they have a fixed model which they know they would get billions of request to, they could even work with analogue chip which is significantly cheaper and faster for inference. [1] could achieve 10-100x flops/watt for fixed models compared to nvidia for their first gen chip.

[1]: https://www.nature.com/articles/s41586-023-06337-5

However, what will be interesting is if the price of delivering ChatGPT-style experiences drops as the industry matures/advances, and their pricing moat erodes.

Unlike Uber - where prices are dictated by factors unlikely to move significantly (Labour / Vehicles / Fuel / etc), The LLM space doesn't have these types of overheads.

Microsoft is a behemoth at dealing with enterprises and has been doing it for decades. Even old school enterprises are OK with uploading data to Azure.

Sure, if you want to let a monopoly have all the added value while you get to keep the rest you can do that.

Just make sure you're never successful enough to inspire them though, otherwise you're dead the next minute. Oops.

I suspect that OpenAI doesn't have the bandwidth to build most uses of ai and so is in the bill gates platform land: the ecosystem should be pocketing more money than the owner of the platform is.

It's possible gpt next or next++ ends up making whatever work you do trivial, but it's likely you'll still have customers

Apple may be leading the way here, with Apple Silicon prioritizing AI processing and built into all their devices. These capabilities are free (or at least don't require an extra sub), and just used to sell more hardware.

OpenAI is clearly going to compete in that market with its upcoming smart phone or device [1]. But what revenue model can OpenAI use to compete with Apple's and not get undercut by it? I suppose hardware + free GPT3.5, and optional subscription to GPT4 (or whatever their highest end version is). Maybe that will be competitive.

I also wonder what mobile OS OpenAI will choose. Probably not Android, otherwise they would have partnered with Google. A revamped and updated Microsoft mobile OS maybe, given their MS partnership? Or something new and bespoke? I could imagine Johnny Ive demanding something new, purpose-built, and designed from scratch for a new AI-oriented UI/UX paradigm.

A market for increasingly sophisticated AI that can only be done in huge GPU datacenters will exist, and that's probably where the margins will be for a long time. I think that's what OpenAI, Microsoft, Google, and the others will be increasingly competing for.

[1]:https://www.reuters.com/technology/openai-jony-ive-talks-rai...

Excuse me, I'm not an english native, you mean like a smart phone? Or do you mean some sort of other new business direction? Where did you get the info thtat they're planning to launch a phone?

https://www.nytimes.com/2023/09/28/technology/openai-apple-s...

They will keep pricing the off-the-shelf AI at-cost to keep competitors at bay.

As for competitors, Anthropic is the most similar to OpenAI both in capabilities and business model. I am not sure what Google is up to, since historically their focus has been in using AI to enhance their products rather than making it a product. The "dark horses" here are Stability and Mistral which both are OSS and European and will try to make that their edge as they give the models for _free_ but to institutional clients that are more sensitive to the models being used and where is the data being handled.

Amazon and Apple are probably catching up. Apple likely thinks that all of this just makes their own hardware more attractive. It's not clear to me what Meta's end goal is.

So eventually you could be running decent sized models locally (iOS could even provide an API with fine tuning etc)

In my experience apple's ML on iphones is seamless. Tap and hold on your dog in a picture and it'll cut out the background, your photos are all sorted automatically including by person (and I think by pet).

OCR is seamless - you just select text in images as if it was real text.

I totally understand these aren't comparable to LLMs - rumor has it apple is working on an llm - if their execution is anything like their current ML execution it'll be glorious.

(Siri objectively sucks although I'm not sure it's fair to compare siri to an LLM as AFAIK siri does not do text prediction but is instead a traditional "manually crafted workflow" type of thing that just uses S2T to navigate)

Wasn't that solved about a decade ago. Does anyone suck at that?

Does android even have native OCR? Last I checked everything required an OCR app of varying quality (including windows/linux).

On ios/macos you can literally just click on a picture and select the text in it as if it wasn't a picture. I know for sure on iOS you don't even open an app to do it, just any picture you can select it.

Last I checked the Opensource OCR tools were decent but behind the closed source stuff as well.

Random google result of OCR on android (could be outdated) - https://www.reddit.com/r/androidapps/comments/10te5et/why_oc...

To use, go to the app switcher and just start selecting text.

It's funny trying to select text in a video while the video is playing...

AFAIK this is a feature only available on Google Pixel devices. so not on 'Android' in general.

For one, my Oppo device running Android 12 doesn't have it.

Yes, Android fragmentation sucks.

That at least works for Pixel phones. Not sure about Samsung and others.

Tesseract? https://github.com/tesseract-ocr/tesseract

I'm not sure this is true, even in the short term. For some things yes, that's definitely true. But for other things that are real-time or near real-time where network latency would be unacceptable, we're already there. For example, Google's Pixel 8 launch includes real-time audio processing/enhancing which is made possible by their new Tensor chip.

I'm no fan of Apple, but I think they're on the right path with local AI. It may even be possible that the tendency of other device makers to put AI in the cloud might give Apple a much better user experience, unless Google can start thinking local-first which kind of goes against their grain.

Agreed. Something else I wonder is if local AI in mobile devices might be better able to learn from its real-time interactions with the physical world than datacenter-based AI.

It's walking around in the world with a human with all its various sensors recording in real-time (unless disabled) - mic, camera, GPS/location, LiDAR, barometer, gyro, accelerometer, proximity, ambient light, etc. Then the human uses it to interact with the world too in various ways.

All that data can of course be quickly sent to a datacenter too, and integrated into the core system there, so maybe not. But I'm curious about this difference and wonder what advantages local AI might eventually confer.