Databricks on Tuesday announced Databricks Apps, a suite of features aimed at making it easy for users to build customized data and AI applications.
Databricks already offers Mosaic AI, an environment that allows customers to integrate systems such as large-scale language models (LLMs) with a company’s own data. However, it lacked the ability to develop real-world conversational applications, such as generative AI chatbots that utilize a combination of AI systems mixed with proprietary data.
Databricks Apps adds the ability to develop applications on top of the tools previously available in Mosaic AI, allowing developers to perform the entire development and deployment process within the secure Databricks environment.
The new tool set is important because Databricks Apps expands on what users can do with Mosaic AI and allows them to develop applications without the need for a third-party platform, said TreeHive Strategy founder and principal Donald Farmer. .
“Very interesting news from Databricks,” he said. “The new features in Databricks Apps remove some pesky obstacles, such as the need to spin up separate infrastructure for development and deployment. Now you can deploy and manage your apps directly in Databricks. This should be pretty easy.”
BARC US analyst Kevin Petrie similarly said that Databricks Apps is an important addition for the vendor’s customers, given that it is in addition to the functionality that was previously able to be developed with Mosaic AI. I am.
“Companies cannot differentiate themselves competitively simply by implementing AI/ML models,” he said. “Rather, you need to create differentiated applications that leverage your unique datasets. Databricks Apps helps AI adopters take that important step.”
San Francisco-based Databricks is a data platform vendor founded in 2013 that pioneered the data lakehouse storage format, which combines the structured data storage capabilities of a data warehouse with the unstructured data storage capabilities of a data lake. There was one.
Over the past two years, the vendor has focused on AI and expanded its platform to include environments for deploying and managing traditional AI, generative AI, and machine learning applications.
Databricks’ $1.3 billion acquisition of MosaicML in June 2023 was an important part of creating an environment where MosaicML’s technology now serves as the foundation for Databricks’ AI and machine learning capabilities. Subsequent acquisitions and product development efforts, including the launch of Databricks’ proprietary large-scale language model, DBRX, have added functionality.
Databricks Apps (now available in public preview on AWS and Azure) further advances the vendor’s AI development capabilities.
New features
Business interest in AI is growing, driven by the potential of generative AI to aid data management and analysis.
One of its promises is to enable true natural language processing (NLP), allowing non-technical employees to use analytics to make decisions. Another of its possibilities is that it can be used to generate code and automate processes, thereby increasing the efficiency of data professionals.
However, developing generative AI applications, such as chatbots that allow users to query and analyze data, or tools that use machine learning to perform tasks such as monitoring data quality, is challenging.
Databricks Apps is designed to simplify application development by giving developers choice when building data and AI applications while allowing them to do all their work in a secure Databricks environment. Masu.
Prior to Databricks Apps, Databricks customers were required to use third-party vendor platforms to complete the development of generative AI chatbots, AI-powered analytical applications, and other intelligent capabilities.
However, mixing proprietary data, AI systems like LLM, and third-party development platforms ran the risk of accidental data beaches. Plus, it was expensive.
One of the things that makes developing data and AI applications so difficult, risky, and expensive is all the behavior they require. To train an application, relevant data must be discovered and moved from the data storage platform. Applications must be developed in an integrated development environment or other data science platform. Next, you need to move your application to a hosted environment for deployment and management.
Databricks Apps eliminates the need for these labor-intensive, expensive, and risky moves.
Instead, developers will be able to build applications natively on Databricks using development frameworks such as Dash, Flask, Gradio, Shiny, and Streamlit. Additionally, it comes with pre-built Python templates designed to speed up the development process.
However, if developers prefer to work in integrated development environments such as Visual Studio Code or PyCharm, Databricks Apps supports that as well.
According to the vendor, Databricks Apps eliminates the need to build infrastructure to deploy and run applications by running them on automatically allocated serverless compute storage within Databricks after development. Masu. Management, on the other hand, includes security measures and governance features such as access control and data lineage that are accessed through the Unity Catalog.
“There are some potentially very impactful features here,” Farmer said. “For example, support for popular developer frameworks allows application developers to choose and work with the tools they are familiar with, expanding the Databricks ecosystem to new markets for application developers.”
Additionally, he continued, it is worth noting that there is no longer a need to develop infrastructure to manage applications.
“Automated provisioning of serverless computing is important because it allows developers to focus on building applications rather than the complex process of deploying data architectures, which has been a barrier for developers who are not data specialists. ” he said.
From a competitive perspective, Databricks has differentiated itself from other data platform vendors by actively developing an environment for building, deploying, and managing AI and machine learning tools over the past several years. said Farmer.
AWS, Google Cloud, Microsoft and Snowflake are all similarly putting AI at the center of their product development, but the tools for developing and managing AI models and applications aren’t as integrated as what Databricks has built, he said. continued. Databricks Apps promotes isolation between Databricks and its peers.
“While Snowflake is catching up or at least continuing to catch up, this continued development by Databricks is very attractive,” Farmer said. “Of course, Microsoft Fabric aims to be an integrated platform similar to Databricks, but it’s still an inferior product. Google Cloud Platform and AWS have a wide range of AI and ML services, but they don’t integrate as deeply into all systems. Not a platform.
Despite the additional capabilities of Databricks Apps, Petrie cautioned that the applications that customers can develop, especially generative AI applications, do not suddenly give everyone in their organization the freedom to manipulate data. did.
Databricks aims to help businesses extend the reach of data and AI beyond a small number of users, but also to use data and AI to inform and act on decisions. It still requires training and expertise to wake up.
“Like many vendors, Databricks aims to ‘democratize’ the use of data, analytics and AI,” said Petrie. “However, I think users of these applications will still require significant data, AI, and business domain expertise, depending on the use cases involved.”
Databricks Apps extends what customers can do with Mosaic AI and demonstrates Databricks’ continued focus on improving the AI and machine learning development environment, according to Shank Niyogi, vice president of the vendor. Customer feedback was the driving force behind the new features. Product management.
He noted that developing and deploying internal applications is always complex. But enterprise interest in AI is rapidly increasing, increasing the need for vendors like Databricks to simplify the development and deployment of AI applications.
“Customers share that building and deploying internal data apps has traditionally been a complex and time-consuming process,” Niyogi said. “They specifically wanted an easy way to test new features while maintaining a secure environment. With the explosion of AI, this need will only grow.”
Looking to the future
Niyogi said Databricks Apps does not end Databricks’ focus on enabling application development and deployment.
The vendor’s goal is to make data and AI available to a wide range of users, he said. To that end, Databricks is investing in the development of new Mosaic AI capabilities and plans to add other capabilities through partnerships.
“Databricks will continue to make AI more accessible to organizations,” Niyogi said. “This includes further ways to simplify the app development process, new Mosaic AI capabilities that help teams build, deploy, and measure complex AI systems, and continued investment in our collaborative AI partner ecosystem. Included.
Meanwhile, Farmer said it’s appropriate for Databricks to focus on improving AI and machine learning workflows. In particular, we suggested that vendors improve application development for non-technical users and support for new AI technologies such as multimodal models.
“Multimodal is going to be important in the next few years,” Farmer said. “I think we should see more development for non-technical users. This release contains the first attempt at that, and it’s definitely the start of a new direction for Databricks, which is very welcome.” That’s what you should do.”
Eric Avidon is a senior news writer for TechTarget Editorial and a journalist with over 25 years of experience. He is responsible for analysis and data management.
