Foundry AI Local: A Game Changer for Local AI on Windows 11
When people talk about Artificial Intelligence (AI), minds often drift towards cloud-based models like ChatGPT or Google Gemini. However, Microsoft has introduced an incredibly simple method for running local AI on personal computers (PCs), and its ease of use demands attention.
At the recent Microsoft Build Developer Conference, Microsoft unveiled Microsoft Foundry AI Local. This is a local AI Large Language Model (LLM) tool aimed at developers, but it holds significant implications for anyone looking to explore the potential of local AI. Let’s delve into this tool and examine why it’s causing such a stir.
Foundry AI Local: The Dawn of Local AI
Microsoft Foundry AI Local is essentially a command-line tool that allows you to run LLMs locally, right on your machine. While targeted initially towards developers, it’s one of the easiest ways to experiment with local AI today, as it handles practically everything for you. More importantly, it also optimizes your PC’s performance.
Imagine you want to install a new application. Traditionally, you’d navigate to a website or the Microsoft Store, find the download link, and then tell Windows where everything should be placed. Now, with “winget,” the process is significantly streamlined. “winget” is like DoorDash for applications. You simply open the command line and type the name of the application you want, and “winget” automatically handles the rest. You don’t even need to log in to third-party websites.
Foundry AI Local works much the same way. You can get started immediately in two commands by copying and pasting from this article. While Microsoft doesn’t explicitly state that you need a dedicated GPU or NPU, they definitely help. You’ll essentially need Windows 10 or 11, at least 8GB of RAM, and 3GB of storage (16GB RAM and 15GB of disk space are recommended). Copilot+ PCs are optional, but you will enjoy a better experience if you have a device equipped with a Qualcomm Snapdragon X Elite processor or an Nvidia RTX 2000 series or AMD Radeon 6000 series GPU or newer.
Getting Started: Steps to Use Foundry AI Local
Here’s a detailed walkthrough on how to start using Foundry AI Local:
Open a Command-Line Terminal: Press the Windows key and start typing “terminal.” You should see the Windows Terminal application listed in the suggestions, or it may show as Windows PowerShell. It doesn’t particularly matter – either will work.
Enter the Commands: At the prompt, enter the following command:
1 | winget install Microsoft.Foundry.AI.Local |
This command will download and install the Foundry AI Local tool. Winget is the package manager for Windows and facilitates the installation process significantly. You should see progress as the tool downloads; if winget isn’t set up on your system before, it will also be installed.
- Install the Model: Once the installation is complete, the next step is to install a local AI model. Again, this is accomplished through the command line. Enter the following command:
1 | C:\Program Files\Microsoft\FoundryAI\Local\AI.Local.exe install-model Microsoft.Phi-3-mini |
This particular command installs the Microsoft Phi-3-mini model. Phi-3 is a series of open-source language models developed by Microsoft. The mini variant is optimized for running on a range of devices, including PCs, and it offers a good balance between performance and resource usage. Other models may become available over time, so keep an eye out for options suitable for different use cases.
- Verify the Installation: Once the installation is finished, you can verify that the model is set up correctly with the following command:
1 | C:\Program Files\Microsoft\FoundryAI\Local\AI.Local.exe list-models |
This command will output a list of installed local AI models. If the operation completes sucessfully you should see Microsoft.Phi-3-mini in the list.
- Configure your application: You have successfully installed a local LLM, now you will need to configure your application to use it. The configuration parameters are below:
- API endpoint:
http://localhost:8080/v1/chat/completions - Model Name:
Microsoft.Phi-3-mini
You can now connect your applications to this local Phi-3-mini model. This allows for offline AI experimentation, development of private applications and reduced reliance on cloud-based AI services. You can connect applications such as LM Studio or continue in a notebook with Python and Langchain.
Why Foundry AI Local Matters
Foundry AI Local isn’t just another tool; it represents a fundamental shift in how we interact with AI. Here are several reasons why it’s significant:
Accessibility: It brings AI development and usage within reach of more people by simplifying the setup and management of local LLMs. The previous hurdles of configuring environments, installing dependencies, and understanding intricate backend processes are dramatically reduced.
Privacy and Security: Running AI models locally means your data stays on your machine. This is crucial for scenarios where privacy is paramount, such as processing sensitive information or developing applications that must comply with strict data protection regulations. The local execution prevents reliance on external servers, which can be vulnerable to interception or breaches.
Offline Capabilities: When a local AI model is installed, your application can run autonomously without an active internet connection. This is vital for scenarios where internet connectivity is unreliable, restricted or nonexistent. Think of remote locations, air travel or offline work environments which now can still leverage AI capabilities.
Reduced Latency: Running AI models locally eliminates network latency, resulting in faster response times. This is particularly important for applications where instantaneous interactions are required, such as real-time language translation, responsive chatbots, or interactive data analysis. Reduced latency enhances the user experience and enables more sophisticated applications.
Cost Efficiency: Executing AI models locally avoids the costs associated with cloud-based AI services. Organizations can reduce their reliance on expensive APIs and infrastructure. The initial investment in hardware is a one-time cost versus ongoing cloud service fees, making it more economical for high-volume or long-term AI usage.
Customization and Control: Running AI locally empowers developers with greater control over their AI environments. You can fine-tune models, optimize performance, and integrate AI into your applications efficiently. Complete customization allows tailored AI experiences to specific scenarios with precision like enterprise security systems or specialized gaming applications.
Considerations for Using Foundry AI Local
While Foundry AI Local offers many advantages, it’s essential to consider the limitations:
Hardware Requirements: Running LLMs locally require powerful hardware, including a capable CPU, sufficient RAM, and a dedicated GPU or NPU. Performance will depend heavily on your machine’s capabilities, and older or underpowered devices may struggle to produce satisfying results.
Model Selection: The availability of optimized LLMs for local execution is still limited compared to the vast selection of models available in the cloud. You may need to experiment with different models to find the best fit for your specific use case and hardware configuration. Microsoft is starting with Phi-3-mini, but expanding this will be crucial to the project’s success.
Resource Management: Running AI models locally can consume significant system resources, potentially impacting the performance of other applications. Careful monitoring and optimization are necessary to ensure seamless and efficient multitasking. It’s worth observing the load on the CPU and GPU while testing and tweaking settings to find the optimal balance.
Updates and Maintenance: Ensuring your local AI models are up-to-date with the latest improvements and security patches requires active maintenance. You’ll need to manage updates manually, as opposed to cloud-based AI services, where updates are typically handled automatically.
Complexity: Setting up and configuring Foundry AI Local still requires some technical expertise, particularly when integrating with other tools and applications. While Microsoft has greatly simplified the process, be prepared to invest some time in understanding the underlying concepts and troubleshooting potential issues.
Foundry AI Local and the Future of AI
Foundry AI Local is not just a technological advancement; it’s a strategic move by Microsoft to empower developers and users. By providing a seamless way to run AI models locally, Microsoft is accelerating the adoption of AI across various industries and applications. Here are some potential implications of this technology:
Edge Computing: The shift towards local AI enables edge computing, processing data closer to the source, reducing the need to transmit large amounts of data to the cloud. This allows for real-time analysis and decision-making in scenarios where low latency and high bandwidth are critical, such as autonomous vehicles and IoT devices.
Democratization of AI: By making AI more accessible to a wider audience, Foundry AI Local democratizes AI development and usage. Smaller businesses and individual developers can now leverage the power of AI without needing substantial resources or expertise. It opens up opportunities for innovation and creativity across diverse sectors.
Hybrid AI Architectures: Foundry AI Local supports the creation of hybrid AI architectures combining local and cloud-based AI capabilities. This approach allows developers to leverage the strengths of both environments, optimizing performance, cost, and security. For example, a local model can handle real-time tasks, while a cloud-based model performs complex analytics.
AI in Embedded Systems: The ability to run AI models on resource-constrained devices enables the use of AI in embedded systems, such as smart appliances, wearable devices, and industrial equipment. This opens up new possibilities for intelligent automation and remote monitoring.
New AI Applications: Foundry AI Local can inspire new AI applications not previously feasible due to limitations related to latency, bandwidth, or data privacy. These applications could range from personalized AI assistants to advanced cybersecurity systems.
Other Approaches for Offline LLMs
While Foundry AI Local simplifies the installation and configuration, there are also other methods to run LLMs offline for those who like to have even more control and customisation:
LM Studio: LM Studio allows you to discover, download, and run LLMs on your local machine. It provides a graphical interface for easy management and interaction with various models.
Ollama: Ollama is similar to Docker for LLMs is that it can run the most recent and popular LLMs, such as Llama 3, locally. The installation is easy and using the command-line is simple.
Python with PyTorch or Tensorflow: This approach involves installing the models within a Python environment, giving you a lot of control, but also requiring more technical knowledge. Once you have familiarised yourself with any tool above, this method may also be viable.
Ultimately, Microsoft’s initiative represents a transformative step towards the future of AI. By embracing local AI, Microsoft is helping to unlock new possibilities, empower developers, and make AI a more integral part of our daily lives. As the tool evolves and the ecosystem around it grows, expect Foundry AI Local to play an increasingly important role in shaping the landscape of AI and software development.
```