Apple is going all-in on AI in 2024. The company is expected to introduce several artificial intelligence features during the WWDC keynote that will work with existing and upcoming devices, from iPhones to iPads and Macs. Here’s everything we know about Apple’s AI efforts.
Latest rumors
According to Haitong International Securities analyst Jeff Pu, iOS 18 could finally bring generative AI features to the iPhone in late 2024. He suggested Apple built a few hundred AI servers in 2023 and will build even more in 2024. TrendForce has also hinted at Apple buying AI servers, as AI server firm Supermicro wants to “aggressively expand its efforts to secure AI orders from clients, including Apple and Meta.”
By combining cloud-based AI and on-device data processing, Apple will finally roll out its generative AI to iPhone and iPad by late 2024, which means they will be part of the iOS 18 cycle.
Tim Cook confirms AI is coming – but core features might be left for later
During Apple’s first earnings call of the year, the company’s CEO, Tim Cook, did not mention iOS 18 by name. But he did address AI twice, teasing the upcoming announcements. He did so during his remarks right after reminding those on the call about the Vision Pro launch.
Cook said Apple will spend a “tremendous amount of time and effort” on AI, with details coming later this year:
Moments like these [the Vision Pro launch] are what we live for at Apple. They’re why we do what we do. They’re why we’re so unflinchingly dedicated to groundbreaking innovation and why we’re so focused on pushing technology to its limits as we work to enrich the lives of our users.
As we look ahead, we will continue to invest in these and other technologies that will shape the future. That includes artificial intelligence, where we continue to spend a tremendous amount of time and effort, and we’re excited to share the details of our ongoing work in that space later this year.
Still, Tim Cook’s words weren’t enough to convince Haitong Securities analyst Jeff Pu. He believes $AAPL stock is overvalued, and investors should be cautious about the iPhone 15 cycle, AirPods, Mac, and Apple Watch, in addition to “potentially muted iPhone 16” sales.
Even though generative AI features will start to appear later this year on the iPhone, Pu expects it to be limited with some AI-related hardware upgrades for the iPhone 16 with the A18 and A18 Pro chips. “We expect a better-functioned Gen AI to be in 2025, likely with iPhone 17, a time to lift Apple’s product/service businesses.”
Features
Apple’s AI efforts could include several new features. Even though the company already uses artificial intelligence in several of its products, the company usually refers to them as powered by “machine learning” or the Neural Engine chip.
Among the new features, Bloomberg‘s Mark Gurman expects the following:
- Auto-summarizing and auto-complete features for core apps and productivity software (Pages, Keynote). This was also rumored by leaker yeux1122, which says code on iOS 18 suggests this function.
- Better playlist creation in Apple Music
- A big Siri overall overhaul with a focus on AI
- Code completion in a new version of Xcode for developers
- AppleCare tools to assist employees in helping customers with troubleshooting
X user Nicolás Álvarez also found a Safari Browsing Assistant feature with iOS 18 code. It will use iCloud Private Relay’s structure to send relevant data to Apple in a privacy-focused manner.
Ajax GPT: Will Apple use its own chatbot?
The Information says Apple is spending millions of dollars daily to train its large-language models – LLM for short. While the publication said most of this investment would focus on AppleCare support, the Siri team plans to incorporate these language models to make complex shortcut integrations much more accessible.
The Apple team believes its most advanced language model, Ajax GPT (which Bloomberg reported on), might be better than OpenAI’s GPT-3.5. In all, Apple is testing four AI models, according to iOS 17.4 code. These include AjaxGPT, ChatGPT, and Google’s FLAN-T5.
Ultimately, Apple aims to improve and integrate its LLMs into iOS, so it compares its test results with ChatGPT and FLAN-T5 with both on-device and online processing. Still, a report says Apple won’t make Ajax GPT available to its users. Instead, Cupertino is in talks with Google to license Gemini after having previously considered OpenAI’s ChatGPT.
A report also claims Apple will use Baidu for its generative AI functions in China. Baidu’s Ernie Bot is a ChatGPT rival and one of the more than 40 AI models from China that local regulators have approved. A partnership with Apple would be a big win for Baidu, considering the growing competition in the region.
Apple papers suggest where its AI efforts are at
AI model for instruction-based image editing
In February, Apple released a revolutionary AI model for instruction-based image editing. According to a paper published by Apple researchers, instruction-based image editing improves the controllability and flexibility of image manipulation via natural commands without elaborate descriptions or regional masks. The study shows “promising capabilities in cross-modal understanding and visual-aware response generation via LM” as they investigated how MLLMs facilitate edit instructions and MLLM-guided image editing.
This image editing AI model made by Apple can produce concise and clear instructions for the editing process, create Photoshop-style modifications, optimize photo quality, and edit specific elements of a picture, such as faces, eyes, hair, clothes, and accessories.
MM1: Apple’s AI model
In March, Apple researchers published a paper highlighting how they’re training a new large language model (LLM).
Called MM1, this LLM can integrate text and visual information simultaneously. The paper offers an interesting look at the importance of various architectural components and data choices. The researchers say they were able to “demonstrate that for large-scale multimodal pre-training using a careful mix of image-caption, interleaved image-text, and text-only data is crucial for achieving state-of-the-art (SOTA) few-shot results across multiple benchmarks, compared to other published pre-training results.”
In addition, they showed that “the image encoder together with image resolution and the image token count has a substantial impact, while the vision-language connector design is of comparatively negligible importance.”
Apple’s MM1 AI model uses a family of multimodal models with up to 30 billion parameters, consisting of both dense models and mixture-of-experts (MoE) variants, that are state-of-the-art in pre-training metrics and achieve competitive performance after supervised fine-tuning on a range of established multimodal benchmarks.
ReALM could be better than OpenAI’s GPT-4
Apple researchers have published a paper about a new AI model. According to the company, ReALM is a language model that can understand and successfully handle contexts of different kinds. With that, users can ask about something on the screen or run in the background, and the language model can still understand the context and give the proper answer.
This is the third paper regarding AI that Apple has published in the past few months. These studies only tease the upcoming AI features of iOS 18, macOS 15, and Apple’s newest operating systems. In the paper, Apple researchers say, “Reference resolution is an important problem, one that is essential to understand and successfully handle context of different kinds.
One example is a user asking for pharmacies near them. After a list is presented, something Siri could do, the user could ask, “Call the one on Rainbow Rd.,” “Call the bottom one,” or “Call this number (present onscreen).” Siri can’t perform this second part, but with ReALM, this language model could understand the context by analyzing on-device data and completing the query.
Ferret LLM
This paper explains how a multimodal large language model can understand user interfaces of mobile displays. The researchers say they have advanced in MLLM usage but still “fall short in their ability to comprehend and interact effectively with user interface (UI) screens.”
This assistive assistant is still far from being released. But once Apple masters it, it could be integrated alongside ReALM model.
Compatible devices
At this moment, it’s unclear which devices will use Apple’s newest AI features, although many of the current gadgets may support these features.
Still, some of the products are more likely to take advantage of the latest features, such as:
BGR will update this guide as we learn more about Apple’s AI efforts.
