© IE Online Media Services Pvt Ltd
Tags:
Journalism of Courage
Google has rolled out VaultGemma, a new AI model, developed from the ground up with privacy-preserving techniques to keep training data confidential.
VaultGemma, the small language model (SLM), boasts one billion parameters, and is touted as the largest model ever trained with differential privacy (DP). The model has been developed by applying a new set of scaling laws derived by Google researchers in partnership with its DeepMind AI unit.
The weights of the model can be downloaded for free from Hugging Face and Kaggle. “VaultGemma represents a significant step forward in the journey toward building AI that is both powerful and private by design. By developing and applying a new, robust understanding of the scaling laws for DP, we have successfully trained and released the largest open, DP-trained language model to date,” the tech giant said in a blog post on September 12.
Data privacy remains one of the biggest challenges in AI development today. Large language models (LLMs) like ChatGPT and Gemini pose threats to user privacy, both in how they are trained and how they handle user data. Privacy experts have pointed out that personal data can be extracted from LLMs by submitting the right prompts. For instance, in its lawsuit against OpenAI, The New York Times had said that it came across examples of ChatGPT reproducing its articles verbatim.
Rather than fine-tuning an LLM with user-level privacy protections, Google said it has integrated differential privacy in VaultGemma at the deeper, pre-training stage by adding calibrated noise to prevent the model from memorising and reproducing its own training data.
However, applying differential privacy to LLMs comes with certain trade-offs, such as a decline in training stability and an increase in batch sizes, as well as computation costs. To strike a balance between the compute-privacy-utility trade-offs, Google has said it has established new scaling laws to determine “optimal training configuration changes based on different constraints.”
“A key finding is that one should train a much smaller model with a much larger batch size than would be used without DP,” the company said.
In terms of performance, Google said that VaultGemma achieved scores comparable to an older, similar-sized GPT-2 model across a range of standard academic benchmarks such as HellaSwag, BoolQ, PIQA, SocialIQA, TriviaQA, ARC-C, and ARC-E.
To test its privacy-preserving capabilities, Google said it prompted VaultGemma with partial text from a training document to see if the AI model would respond with the corresponding text. While the model did not return the corresponding text, Google also said that “if many training sequences contain information relevant to a particular fact, then in general VaultGemma will be able to provide that information.”
The company further said that more research on DP training is needed to close the utility gap between DP-trained and present-day, non-DP-trained models.
