Generative artificial intelligence (GenAI) models, such as generative adversarial networks (GANs) and transformer-based large language models (LLMs), are developing at an accelerated pace and positioned to be integrated into clinical workflows and healthcare systems across the world. However, this rapid rise of GenAI in medicine and healthcare presents not just unprecedented opportunities, but also systemic risks in the integration of this new technology and critical vulnerabilities in terms of safety, governance and regulatory oversight. GenAI and LLMs are non-deterministic in nature, possess broad generalist functionalities, and display evolving capabilities. These characteristics challenge conventional regulatory frameworks designed for deterministic, task-specific artificial intelligence (AI) models, such as those for Software as a Medical Device (SaMD).