État de l'art: Modélisation Monomodale du Agentic Workflow pour la Gestion des Services de Diagnostic Médical Intelligent

1. Introduction to Agentic Workflows in Medical Diagnostics

- Overview of Agentic Systems

Agentic systems are transforming the landscape of automated task execution by leveraging the power of Large Language Models (LLMs) [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_0]. These sophisticated systems are designed to orchestrate a diverse array of specialized tools, enabling them to tackle complex challenges across numerous domains. The core principle behind agentic systems lies in their ability to autonomously plan, execute, and adapt to dynamic environments, making them particularly well-suited for tasks that require intricate problem-solving and decision-making capabilities. By integrating LLMs, these systems can understand natural language instructions, reason about the task at hand, and coordinate the actions of various tools to achieve specific goals. This integration allows for a seamless and efficient workflow, where AI agents act as intelligent intermediaries between users and the underlying tools and resources.

However, while agentic systems have demonstrated considerable success across various sectors, their application in the medical field presents unique challenges [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_2]. The medical domain is characterized by its stringent requirements for accuracy, reliability, and safety, necessitating specialized tools and workflows that are tailored to the specific nuances of medical diagnostics and treatment. The development and integration of these specialized tools can be a complex and resource-intensive process, often requiring deep domain expertise and a thorough understanding of medical protocols and regulations. Furthermore, the ethical considerations surrounding the use of AI in healthcare, such as data privacy and algorithmic bias, add another layer of complexity to the deployment of agentic systems in this field. Despite these challenges, the potential benefits of agentic systems in medical diagnostics are substantial, ranging from improved efficiency and reduced costs to enhanced accuracy and personalized treatment approaches.

M3Builder represents a significant advancement in the field, offering a novel agentic framework specifically designed to automate machine learning workflows in medical imaging [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_3]. This innovative system addresses the unique requirements of medical diagnostics by incorporating a multi-agent collaboration framework, a structured medical imaging ML workspace, and automated self-correction mechanisms. By leveraging these components, M3Builder aims to streamline the development and deployment of AI-powered tools for medical imaging analysis, ultimately bridging the gap between clinical needs and tailored AI solutions. The system's ability to autonomously manage complex workflows, from data preparation to model training and deployment, holds immense promise for improving the efficiency and effectiveness of medical diagnostics.

- Application of Agentic Workflows in Medical Imaging

Agentic workflows hold immense potential for revolutionizing the development of AI tools specifically tailored for medical imaging analysis [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_4]. These workflows leverage the capabilities of Large Language Models (LLMs) to automate and streamline the intricate processes involved in creating and deploying AI models for various medical imaging tasks. By employing agentic systems, researchers and clinicians can significantly reduce the time and resources required to develop these tools, while also enhancing their accuracy and reliability. The application of agentic workflows in medical imaging encompasses a wide range of tasks, including image segmentation, anomaly detection, disease diagnosis, and report generation, each of which can benefit from the automation and intelligent decision-making capabilities of AI agents.

M3Builder exemplifies the power of agentic workflows in medical imaging by employing a multi-agent collaboration framework, where specialized LLM agents work in concert to execute complex workflows [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_3]. This framework is meticulously designed to manage a comprehensive set of tasks, including task requirement understanding, procedural planning, data processing, environment configuration, automated self-correction, and model training execution. Each agent within the framework is assigned a specific role and responsibilities, allowing for a division of labor that optimizes efficiency and effectiveness. The agents communicate and collaborate with each other, sharing information and coordinating their actions to achieve the overall goal of developing a high-performing AI model for medical imaging analysis.

The multi-agent collaboration framework within M3Builder is particularly well-suited for addressing the complex and multi-faceted nature of medical imaging tasks [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_3]. By breaking down the overall workflow into smaller, more manageable sub-tasks, the system can leverage the specialized expertise of each agent to ensure that each step is executed with precision and accuracy. For example, the Task Manager agent is responsible for understanding the specific requirements of the task and generating a comprehensive plan, while the Data Engineer agent focuses on preparing and processing the raw data. The Module Architect agent designs and implements the model architecture, and the Model Trainer agent optimizes the model's performance through training and debugging. This collaborative approach not only enhances the efficiency of the development process but also ensures that the resulting AI model is robust and reliable.

- Challenges and Opportunities in Medical Diagnostics

One of the most significant hurdles in applying agentic workflows to the medical domain is the existing shortage of well-prepared tools [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_2]. This scarcity arises from the inherent complexity of medical workflows, which encompass a vast array of diseases, imaging modalities, and task requirements. Developing tools that can effectively address this diversity demands substantial resources, specialized expertise, and a deep understanding of the intricacies of medical practice. Moreover, the stringent regulatory requirements and ethical considerations surrounding the use of AI in healthcare further complicate the tool development process. Ensuring that these tools are not only accurate and reliable but also compliant with data privacy regulations and free from algorithmic bias is paramount.

The complexity of medical workflows stems from several factors [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_2]. First, the sheer number of diseases and conditions that can be diagnosed using medical imaging techniques presents a significant challenge. Each disease may require a unique set of imaging protocols, data processing steps, and model architectures. Second, the variety of imaging modalities, such as X-ray, CT, MRI, and ultrasound, each with its own strengths and limitations, adds another layer of complexity. Developing tools that can effectively handle data from different modalities requires specialized knowledge and expertise. Third, the task requirements for medical imaging analysis can vary widely, ranging from simple image segmentation to complex disease diagnosis and report generation. Meeting these diverse requirements necessitates a flexible and adaptable toolset that can be customized to the specific needs of each task.

Despite these challenges, agentic workflows present significant opportunities to bridge the gap between clinical needs and tailored AI solutions [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_3]. By automating many of the manual and time-consuming tasks involved in developing AI-powered tools, agentic systems can accelerate the development process and reduce the overall cost. Furthermore, the ability of agentic systems to learn from data and adapt to changing conditions makes them particularly well-suited for addressing the dynamic nature of medical practice. As new diseases emerge, new imaging techniques are developed, and new clinical guidelines are established, agentic systems can be continuously updated and refined to ensure that they remain accurate and relevant. The potential for agentic workflows to transform medical diagnostics is immense, offering the promise of more efficient, accurate, and personalized healthcare.

2. Multi-Agent Collaboration Frameworks

- Core Components of Multi-Agent Systems

Multi-agent systems represent a paradigm shift in AI, composed of multiple LLM agents with distinct roles that collaboratively tackle complex problems [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_7]. These systems adopt a divide-and-conquer strategy, decomposing intricate tasks into smaller, more manageable sub-tasks that can be addressed by individual agents with specialized expertise. This approach not only enhances efficiency but also allows for a more flexible and adaptable problem-solving process. Each agent operates autonomously, making decisions based on its own knowledge and goals, while also coordinating its actions with other agents to achieve the overall objectives of the system. The interactions between agents can range from simple communication to complex negotiation and cooperation, enabling the system to effectively address a wide range of challenges.

The core strength of multi-agent systems lies in their ability to iteratively perform code generation, editing, and execution using tools defined by toolset descriptions until a functional AI model is successfully produced [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_7]. This iterative process allows the system to continuously refine its approach, learn from its mistakes, and adapt to changing conditions. The agents work together to generate code, test its functionality, identify and correct errors, and optimize its performance. This collaborative approach ensures that the final AI model is robust, reliable, and well-suited for its intended purpose. The use of toolset descriptions provides a standardized framework for the agents to interact with the environment, ensuring that they have access to the necessary resources and capabilities to perform their tasks effectively.

M3Builder's multi-agent collaboration framework exemplifies the power of this approach by decomposing the AI task into four distinct sub-tasks and assigning them to specialized role-playing LLM agents [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_13]. These agents, namely the Task Manager, Data Engineer, Module Architect, and Model Trainer, work in concert to manage the complexities of medical imaging model development. Each agent is responsible for a specific aspect of the workflow, leveraging its unique expertise to ensure that the overall process is executed with precision and efficiency. The Task Manager is responsible for understanding the task requirements and generating a comprehensive plan, while the Data Engineer prepares and processes the data. The Module Architect designs and implements the model architecture, and the Model Trainer optimizes the model's performance through training and debugging. This collaborative approach not only streamlines the development process but also ensures that the resulting AI model is of high quality.

- Role Specialization in Agentic Workflows

In the context of M3Builder, role specialization is a critical component of the multi-agent collaboration framework, with each agent assigned a unique set of responsibilities and expertise [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_13]. The Task Manager plays a pivotal role in selecting the most suitable dataset for the given task and generating a comprehensive planning document that serves as a roadmap for the entire workflow. This agent must possess a deep understanding of the available datasets, their characteristics, and their suitability for different types of medical imaging tasks. The planning document outlines the steps that need to be taken, the resources that need to be allocated, and the timelines that need to be met to successfully complete the project.

The Data Engineer is responsible for transforming raw data into a format that is suitable for model training [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_14]. This involves a variety of tasks, including data cleaning, data normalization, feature extraction, and data augmentation. The Data Engineer must be proficient in data manipulation techniques and have a strong understanding of the specific requirements of the AI model. The agent also interacts with the external compiler environment, generating, editing, and refining code until it executes successfully. This iterative process ensures that the dataset preparation code is robust and functional.

The Module Architect is tasked with integrating essential components into the training pipeline, including developing dataloader scripts, designing appropriate model architecture, initializing the entrance function for training, and selecting other components, such as loss functions, data augmentation strategies, and training utilities [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_15]. This agent must have a strong understanding of machine learning principles and techniques, as well as experience in designing and implementing AI models. The Module Architect iteratively validates the dataloader to ensure that it outputs batches with correct shapes and formats.

The Model Trainer is responsible for finalizing the debugging and optimizing of the training procedure [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_16]. Building upon the pipeline established by the Module Architect, the Model Trainer verifies the completeness and correctness of the training framework. This agent selects hyperparameters to meet the model's specific training requirements while retaining the authority to modify any part of the code, including model code, dataloader code, and training scripts, based on errors encountered during training. The Model Trainer must have a strong understanding of optimization algorithms and techniques, as well as experience in debugging and troubleshooting AI models.

- Advantages of Multi-Agent Collaboration

Multi-agent collaboration has a significant impact on the overall performance of agentic systems, with its absence leading to a considerable performance gap [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_22]. By leveraging the specialized expertise of multiple agents, these systems can achieve higher levels of accuracy, efficiency, and robustness compared to single-agent systems. The ability to divide complex tasks into smaller, more manageable sub-tasks allows each agent to focus on its area of expertise, resulting in a more thorough and effective problem-solving process. Furthermore, the collaborative nature of these systems enables agents to share information, coordinate their actions, and learn from each other, leading to continuous improvement and adaptation.

M3Builder's multi-agent system exemplifies the benefits of this approach, achieving a higher average success rate while requiring fewer action steps and execution iterations compared to single-agent systems [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_22]. This demonstrates the efficiency and effectiveness of the multi-agent collaboration framework in streamlining the development and deployment of AI-powered tools for medical imaging analysis. The system's ability to manage complex workflows with minimal human intervention highlights the potential of agentic systems to transform the medical diagnostics field.

The multi-agent framework demonstrates remarkable robustness and adaptability in completing assigned task executions [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_24]. Despite the strict requirements for code organization, data preprocessing, and achieving error-free training within limited iterations, the majority of agents successfully completed or nearly completed all assigned tasks. This showcases the system's ability to handle challenging and dynamic conditions, making it well-suited for real-world medical applications. The collaborative nature of the framework allows agents to support each other, overcome obstacles, and adapt to changing circumstances, ensuring that the overall system remains resilient and effective.

3. Medical Imaging ML Workspace

- Structure and Functionality of the Workspace

The Medical Imaging ML workspace serves as a central hub for agentic systems, providing agents with free-text descriptions of datasets, training codes, and interaction tools to enable seamless communication and task execution [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_1]. This structured environment is designed to facilitate the development and deployment of AI-powered tools for medical imaging analysis by providing agents with the necessary resources and information to perform their tasks effectively. The workspace acts as a shared repository of knowledge, allowing agents to access and utilize relevant data, code, and tools in a coordinated manner.

The workspace includes several key elements that support the multi-agent collaboration framework, including data cards, toolset descriptions, and code templates [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_6]. Data cards provide agents with detailed information about the available datasets, including their characteristics, provenance, and suitability for different types of medical imaging tasks. Toolset descriptions define the available tools and their functionalities, allowing agents to understand how to interact with the environment and perform specific tasks. Code templates provide agents with a starting point for developing AI models, streamlining the development process and ensuring consistency across projects.

The functionality of the workspace extends beyond simply providing access to resources and information [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_1]. It also facilitates communication and coordination between agents, allowing them to share information, request assistance, and monitor progress. The workspace provides a common platform for agents to interact, ensuring that they are all working towards the same goals and following the same protocols. This coordinated approach enhances the efficiency and effectiveness of the overall workflow, leading to faster development times and higher quality AI models.

- Key Elements: Datacards, Toolsets, and Code Templates

Datacards are a crucial component of the Medical Imaging ML workspace, containing key information such as the dataset name, a concise summary of its contents, and detailed metadata about its characteristics [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_10]. This information is essential for agents to understand the available datasets and select the most appropriate one for a given task. The dataset name provides a unique identifier for the dataset, while the summary provides a brief overview of its contents, including the type of images, the anatomical regions covered, and the diseases or conditions represented. The metadata provides more detailed information about the dataset, such as the number of images, the image resolution, the imaging modality, and the patient demographics.

The toolset is another essential element of the workspace, including functions such as list_files, read_files, copy_files, write_files, edit_files, and run_script [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_12]. These functions provide agents with the ability to interact with the environment, access and manipulate files, and execute code. The list_files function allows agents to view the contents of a directory, while the read_files function allows them to read the contents of a file. The copy_files function allows agents to copy files from one location to another, while the write_files function allows them to write data to a file. The edit_files function allows agents to modify the contents of a file, and the run_script function allows them to execute a script.

Code templates are pre-defined code structures tailored to primary medical imaging tasks such as disease diagnosis, organ segmentation, anomaly detection, and report generation [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_11]. These templates are designed to streamline the development process by providing agents with a starting point for building AI models. Each template is implemented as a modular package with configurable components, such as the main forward architecture and network backbone options (2D/3D models). Shared features across tasks include a unified selection of loss functions, data augmentation strategies, training utilities, and architectural frameworks. By offering these templates, the workspace reduces the complexity of free-form coding while preserving adaptability for diverse AI tasks.

- Importance of a Structured Environment

The structured environment of the ML workspace plays a pivotal role in guiding the automatic AI workflow and ensuring that the development process is efficient, consistent, and reliable [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_6]. By providing agents with a well-defined set of resources, tools, and protocols, the workspace reduces the complexity of the development process and allows agents to focus on their specific tasks. The structured environment also promotes collaboration and coordination between agents, ensuring that they are all working towards the same goals and following the same procedures.

The workspace provides a standardized starting point for agents, demonstrating a typical AI model training pipeline [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_8]. This allows agents to quickly understand the overall workflow and identify the steps that need to be taken to develop a successful AI model. The standardized starting point also promotes consistency across projects, ensuring that all models are developed using the same best practices. The workspace defines and describes all tools available to the agents, restricting their action space to a predefined, complete set [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_8]. This ensures that agents are not overwhelmed by the vast array of possible actions and that they focus on the most relevant and effective tools.

The structured environment also facilitates the integration of new datasets and tools into the workflow [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_10]. By providing a clear and consistent framework for organizing and accessing resources, the workspace makes it easy to add new datasets, tools, and code templates. This allows the system to continuously adapt to changing conditions and incorporate new advances in the field. The structured environment also promotes the sharing of resources and knowledge between agents, fostering a collaborative and innovative culture.

4. Automation of Machine Learning Workflows

- End-to-End Automation in M3Builder

M3Builder represents a paradigm shift in the automation of machine learning workflows, autonomously managing the entire development process, from data preparation to model construction, training, and deployment [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_4]. This end-to-end automation eliminates the need for manual intervention at various stages of the workflow, significantly reducing the time and resources required to develop and deploy AI-powered tools for medical imaging analysis. The system is designed to handle a wide range of tasks, from data cleaning and preprocessing to model selection, hyperparameter tuning, and performance evaluation.

Given a medical imaging-specific ML task and raw training data, M3Builder can perform the entire process without requiring human assistance [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_4]. This includes understanding the task requirements, selecting the appropriate dataset, preparing the data for training, designing and implementing the model architecture, training the model, evaluating its performance, and deploying it for use in clinical practice. The system is designed to be flexible and adaptable, allowing it to handle a variety of different types of medical imaging tasks and datasets.

The system integrates user requirements with a workspace containing candidate data, tools, and code templates, providing a comprehensive and well-organized environment for AI model development [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_10]. The workspace acts as a central hub for all the resources and information needed to develop and deploy AI models, ensuring that agents have access to the necessary tools and data to perform their tasks effectively. The integration of user requirements with the workspace ensures that the resulting AI model is aligned with the specific needs of the user, maximizing its value and impact.

- Workflow from User Request to Model Delivery

The M3Builder workflow begins with a user's free-text request, which serves as the starting point for the entire AI model development process, and culminates in the delivery of a fully trained and deployable model [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_10]. This seamless and automated workflow is designed to minimize human intervention and accelerate the development cycle, allowing users to quickly and easily create AI-powered tools for medical imaging analysis. The workflow is orchestrated by a team of specialized agents, each with its own unique responsibilities and expertise.

The Task Manager plays a crucial role in analyzing the user's request and selecting the appropriate data for the task [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_2]. This agent must have a deep understanding of the available datasets and their suitability for different types of medical imaging tasks. The Data Engineer is responsible for screening and processing the data, ensuring that it is clean, consistent, and ready for training. This agent must be proficient in data manipulation techniques and have a strong understanding of the specific requirements of the AI model.

The Module Architect is tasked with writing the code to complete the pipeline, including designing the model architecture, implementing the training loop, and defining the evaluation metrics [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_2]. This agent must have a strong understanding of machine learning principles and techniques, as well as experience in designing and implementing AI models. The Model Trainer is responsible for training the model and auto-debugging any errors that may arise during the training process. This agent must have a strong understanding of optimization algorithms and techniques, as well as experience in debugging and troubleshooting AI models. A sample log tracks the Model Trainer agent's activities during diagnosis model development, providing valuable insights into the training process.

- Self-Correction and Debugging Mechanisms

M3Builder incorporates automated self-correction and debugging mechanisms to ensure that the AI models are developed with high accuracy and reliability [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_3]. These mechanisms are designed to identify and correct errors in the code, data, and model architecture, minimizing the need for manual intervention and reducing the risk of human error. The self-correction and debugging mechanisms are integrated into the workflow at various stages, ensuring that errors are detected and corrected as early as possible.

Auto-debugging is a crucial component of the system, proving essential for successful training [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_22]. The system employs a variety of techniques to automatically identify and correct errors in the code, including static analysis, dynamic analysis, and fault injection. Static analysis involves examining the code without executing it, looking for potential errors such as syntax errors, type errors, and logical errors. Dynamic analysis involves executing the code and monitoring its behavior, looking for errors such as runtime exceptions, memory leaks, and performance bottlenecks. Fault injection involves intentionally introducing errors into the code to test the system's ability to detect and correct them.

The agents iteratively generate, edit, and refine code, incorporating feedback from the compiler until the code executes successfully [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_14]. This iterative process ensures that the code is robust, reliable, and free from errors. The agents use a variety of tools to assist them in this process, including debuggers, profilers, and code linters. The agents also collaborate with each other to identify and correct errors, leveraging their combined expertise to ensure that the code is of high quality.

5. Evaluation Benchmarks for Medical Imaging AI

- Introduction to M3Bench

M3Bench is a comprehensive benchmark designed to evaluate the capabilities of agentic systems in automated machine learning within the medical imaging domain [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_1]. This benchmark comprises four general tasks performed on 14 training datasets, spanning five anatomical regions and three distinct imaging modalities, encompassing both 2D and 3D data. M3Bench aims to provide a standardized and rigorous framework for assessing the performance of AI systems in medical imaging, enabling researchers and practitioners to compare different approaches and identify areas for improvement.

M3Bench provides a standardized foundation for assessing the capabilities of agentic systems in automated ML in medical imaging, addressing a critical need in the field [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_5]. The benchmark is designed to be challenging and representative of real-world medical imaging tasks, ensuring that the results are meaningful and applicable to clinical practice. The use of multiple datasets, anatomical regions, and imaging modalities ensures that the benchmark is comprehensive and captures the diversity of medical imaging applications.

M3Bench comprises four general tasks: organ segmentation, anomaly detection, disease diagnosis, and report generation, covering a wide range of medical imaging applications [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_4]. These tasks are designed to be challenging and require a combination of skills, including image processing, machine learning, and medical knowledge. The benchmark also includes a set of evaluation metrics that are used to assess the performance of the AI systems, ensuring that the results are objective and comparable.

- Tasks and Datasets in M3Bench

The tasks included in M3Bench are carefully selected to represent the key challenges and opportunities in medical imaging analysis [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_4]. Organ segmentation involves identifying and delineating specific anatomical structures within medical images, which is essential for a variety of clinical applications, such as surgical planning and radiation therapy. Anomaly detection involves identifying unusual or abnormal patterns in medical images, which can be indicative of disease or injury. Disease diagnosis involves classifying medical images based on the presence or absence of specific diseases or conditions. Report generation involves automatically generating textual reports that summarize the findings from medical images, providing clinicians with a concise and informative overview of the patient's condition.

These tasks are associated with 14 detailed training datasets, spanning five anatomical regions (head & neck, chest, abdomen & pelvis, limb, spine) and three primary imaging modalities (X-ray, CT, MRI), encompassing both 2D and 3D models [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_4]. The use of multiple datasets, anatomical regions, and imaging modalities ensures that the benchmark is comprehensive and captures the diversity of medical imaging applications. The datasets are carefully curated to ensure that they are of high quality and representative of real-world clinical data.

The datasets included in M3Bench are diverse and cover a wide range of medical imaging applications [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_18]. These datasets include ADNI, KneeMRI, CC-CCII, CT-Kidney, BTCV, MSDPancreas, VerSe, L2R-OASIS, COVID-19, CT-RATE, INSTANCE2022, ChestX-Det10, RadGenome-Brain-MRI, and IU-Xray. Each dataset is accompanied by a datacard with metadata from their original publications, providing detailed information about the data and its characteristics. The inclusion of these datasets ensures that the benchmark is relevant to a wide range of medical imaging applications and that the results are generalizable to real-world clinical practice.

- Metrics for Assessing System Performance

The effectiveness of the systems under evaluation is assessed through a comprehensive analysis of task completion rates, framework superiority, and agent role-specification [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_17]. These metrics provide a holistic view of the system's performance, capturing its ability to successfully complete medical imaging tasks, its overall superiority compared to other approaches, and the effectiveness of its agent-based architecture. The evaluation process is designed to be rigorous and objective, ensuring that the results are meaningful and comparable across different systems.

Task completion is defined as the successful training of a model with performance on the test set falling within an acceptable range [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_19]. This metric assesses the system's ability to develop AI models that are both accurate and reliable, ensuring that they can be used with confidence in clinical practice. The acceptable range for performance on the test set is determined based on the specific requirements of the task and the characteristics of the dataset.

The performance of each role-specific agent is evaluated using distinct success criteria, providing insights into the effectiveness of the multi-agent collaboration framework [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_23]. The Task Manager is assessed on its ability to select appropriate training datasets, ensuring that the system is using the most relevant and informative data for each task. The Data Engineer is assessed on generating valid data index files with correct structures and paths, ensuring that the data is properly prepared for training. The Module Architect is assessed on producing executable scripts for data loading, ensuring that the system can efficiently access and process the data. The Model Trainer is assessed on successfully completing model training, ensuring that the system can develop AI models that achieve high performance.

6. Comparative Analysis with State-of-the-Art Systems

- Benchmarking Against Existing Agentic Systems

M3Builder undergoes rigorous benchmarking against a range of existing agentic systems, including MLAgent-Bench, Aider, Cursor Composer, Windsurf Cascade, and Copilot Edits, to comprehensively evaluate its performance and capabilities [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_21]. This comparative analysis provides valuable insights into the strengths and weaknesses of M3Builder relative to other state-of-the-art approaches, highlighting its unique contributions and potential for advancing the field of automated machine learning in medical imaging. The benchmarking process is designed to be fair and objective, ensuring that the results are meaningful and reliable.

Each system performed each task twice on the workspace under their built-in framework, providing a standardized and controlled environment for comparison [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_21]. This ensures that the results are not influenced by variations in the environment or the implementation details of the different systems. The use of a common workspace and a standardized set of tasks allows for a direct comparison of the performance of the different systems, highlighting their relative strengths and weaknesses.

M3Builder consistently achieves superior performance across a range of metrics, demonstrating its effectiveness in automating machine learning workflows for medical imaging analysis [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_5]. This superior performance is attributed to its unique combination of a multi-agent collaboration framework, a structured medical imaging ML workspace, and automated self-correction mechanisms. The system's ability to manage complex workflows with minimal human intervention highlights its potential to transform the medical diagnostics field.

- Performance Comparison Across Radiology Tasks

Across a range of radiology tasks, including Organ Segmentation, Anomaly Detection, Disease Diagnosis, and Report Generation, MLAgent-Bench exhibited suboptimal performance, primarily attributed to its limited data structure understanding capabilities [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_21]. This highlights the importance of having a robust and flexible data structure understanding mechanism in order to effectively process and analyze medical imaging data. The ability to understand the structure and relationships within the data is essential for accurate and reliable performance on these tasks.

Other frameworks achieved only moderate success rates, with a maximum of 39.29%, due to the inherent limitations of single-agent architectures [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_21]. This underscores the benefits of multi-agent collaboration in tackling complex medical imaging tasks, where the combined expertise and capabilities of multiple agents can lead to significantly improved performance. The single-agent systems were also limited by the need for human-in-the-loop confirmation, increasing operational complexity and max iteration constraints.

M3Builder demonstrated superior performance with a 42.85% higher average success rate while requiring fewer action steps and execution iterations [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_22]. This highlights the efficiency and effectiveness of M3Builder's multi-agent collaboration framework and automated self-correction mechanisms. The system's ability to achieve higher success rates with fewer resources demonstrates its potential to reduce the cost and time required to develop and deploy AI-powered tools for medical imaging analysis.

- Advantages of M3Builder over Existing Systems

M3Builder's multi-agent collaboration framework and well-crafted example instructions significantly impact performance, leading to improved accuracy and efficiency [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_22]. The multi-agent architecture allows for a division of labor, with each agent focusing on its area of expertise, while the well-crafted example instructions provide agents with clear guidance on how to perform their tasks. This combination of factors contributes to the system's superior performance compared to existing systems.

M3Builder achieves a 94.29% model building success rate with Claude-3.7-Sonnet standing out among seven SOTA LLMs, demonstrating its effectiveness in automating the development of high-quality AI models for medical imaging analysis [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_25]. The use of state-of-the-art LLMs as agent cores ensures that the system has access to the latest advances in natural language processing and machine learning. The system's ability to achieve such a high success rate highlights its potential to transform the medical diagnostics field.

The experimental results demonstrate that M3Builder possesses impressive and robust capabilities for medical imaging model training automation, making it a valuable tool for researchers and practitioners in the field [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_21]. The system's ability to automate the entire development process, from data preparation to model deployment, significantly reduces the time and resources required to develop and deploy AI-powered tools for medical imaging analysis. The system's robust capabilities ensure that the resulting AI models are accurate, reliable, and well-suited for use in clinical practice.

7. The Role of Large Language Models (LLMs)

- LLMs as Agent Cores

Seven state-of-the-art large language models serve as agent cores for the system, such as Claude series, GPT-4o, and DeepSeek-V3, demonstrating the versatility and adaptability of M3Builder [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_1]. These LLMs provide the intelligence and reasoning capabilities necessary for the agents to perform their tasks effectively. The use of multiple LLMs allows for a comparison of their performance and the identification of the best LLM for each specific task.

The evaluation employs 7 leading LLMs as agent cores: GPT-4 , Claude-3.7-Sonnet, Claude-3.5-Sonnet , DeepSeek-v3 , Gemini-2.0-flash , Qwen-2.5-max , and Llama-3.3-70B , providing a comprehensive assessment of their suitability for medical imaging AI tasks [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_17]. This comprehensive evaluation allows for a thorough understanding of the strengths and weaknesses of each LLM, enabling researchers and practitioners to select the most appropriate LLM for their specific needs. The use of a diverse set of LLMs ensures that the results are generalizable and not specific to any particular LLM architecture.

The performance of different LLMs exhibits significant variation, highlighting the importance of selecting the right LLM for the task at hand [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_19]. This variability is due to differences in the LLMs' architectures, training data, and fine-tuning strategies. The evaluation process helps to identify the LLMs that are best suited for medical imaging AI tasks, enabling researchers and practitioners to develop more effective and efficient AI systems.

- Performance of Different LLMs in M3Builder

Claude-3.7-Sonnet achieves the highest completion rate of 94.29%, while Gemini2.0 and Llama3.3 only reach 4.29%, indicating a significant performance gap among different LLMs [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_19]. This highlights the importance of carefully selecting the LLM that is used as the agent core, as the choice of LLM can have a significant impact on the overall performance of the system. The superior performance of Claude-3.7-Sonnet suggests that it is particularly well-suited for medical imaging AI tasks.

The Task Manager demonstrated exceptional accuracy in task analysis, with stable token usage across all tasks, highlighting its efficiency and effectiveness in understanding and planning medical imaging tasks [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_23]. This is crucial for ensuring that the system is able to correctly interpret the user's request and select the appropriate dataset and tools for the task. The stable token usage indicates that the Task Manager is able to perform its tasks efficiently, without wasting resources.

The majority of agents successfully completed or nearly completed all assigned task executions, showcasing the robustness and adaptability of the multi-agent framework [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_24]. This demonstrates the effectiveness of the multi-agent architecture in distributing the workload and ensuring that the system is able to handle a variety of different medical imaging tasks. The robustness and adaptability of the framework make it well-suited for real-world clinical practice, where conditions can be unpredictable and challenging.

- Impact of LLM Selection on Task Completion

LLM selection has a significant impact on task completion rates and overall system performance [chunk_23cf0170e733818337c330db2b9dfedfbd9a84799dc9fcb957fe7a7036e65bd1_1]. The choice of LLM can affect the system's ability to understand the user's request, select the appropriate dataset and tools, and generate accurate and reliable results. The evaluation process helps to identify the LLMs that are best suited for medical imaging AI tasks, enabling researchers and practitioners to develop more effective and efficient AI systems.

The choice of LLM affects the system's ability to precisely and reliably answer questions based on EHRs, highlighting the importance of selecting an LLM that is well-trained on medical data [chunk_23cf0170e733818337c330db2b9dfedfbd9a84799dc9fcb957fe7a7036e65bd1_18]. The LLM must be able to understand the complex relationships between medical entities and concepts in order to accurately answer questions about patient health. The system's accuracy rate improves with the few-shot strategy, demonstrating the effectiveness of providing the LLM with a small number of examples to guide its reasoning [chunk_23cf0170e733818337c330db2b9dfedfbd9a84799dc9fcb957fe7a7036e65bd1_36]. This suggests that LLMs can be effectively fine-tuned for medical imaging AI tasks with a relatively small amount of training data.

The system's accuracy rate improves with the few-shot strategy [chunk_23cf0170e733818337c330db2b9dfedfbd9a84799dc9fcb957fe7a7036e65bd1_36]. On average, the accuracy rate increased by 11.1% after incorporating few-shot examples into the prompt. This result indicates that LLMs can learn efficiently from a limited number of examples, making them practical for use in medical settings where large labeled datasets may be scarce.

8. Monomodal vs. Multimodal Approaches

- Focus on Monomodal Data

The current implementation of M3Builder primarily focuses on monomodal data, specifically leveraging textual data and associated metadata for its operations [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_6]. This monomodal approach simplifies the system's design and reduces the complexity of data processing, allowing for a more efficient and streamlined workflow. By focusing on textual data, the system can leverage the power of LLMs to understand and reason about medical concepts, relationships, and tasks.

The system operates using textual data from Electronic Health Records (EHRs) to create patient profiles and guide the AI model development process [chunk_23cf0170e733818337c330db2b9dfedfbd9a84799dc9fcb957fe7a7036e65bd1_39]. This textual data provides valuable information about the patient's medical history, symptoms, diagnoses, and treatments, which can be used to train AI models for a variety of medical imaging tasks. The use of EHR data ensures that the AI models are trained on real-world clinical data, making them# État de l'art: Modélisation Monomodale du Agentic Workflow pour la Gestion des Services de Diagnostic Médical Intelligent

1. Introduction to Agentic Workflows in Medical Diagnostics

- Overview of Agentic Systems

Agentic systems are transforming numerous fields by using Large Language Models (LLMs) to automate intricate tasks [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_0]. These systems operate by orchestrating a variety of specialized tools to achieve specific goals [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_0]. While agentic systems have seen success in various sectors, their deployment in the medical field presents unique challenges due to the necessity for highly specialized tools and domain-specific knowledge [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_2]. The medical field requires tools that can handle the complexities of medical data and workflows [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_2]. M3Builder represents an innovative agentic framework specifically designed to automate machine learning workflows within medical imaging, addressing these challenges [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_3].

- Application of Agentic Workflows in Medical Imaging

Agentic workflows hold considerable promise for the development of advanced AI tools tailored for medical imaging analysis [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_4]. These workflows can automate various processes, from data preparation to model training and deployment, enhancing the efficiency and accuracy of medical diagnostics [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_4]. M3Builder leverages a multi-agent collaboration framework, where specialized LLM agents collaborate to execute complex workflows involved in medical imaging tasks [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_3]. This framework effectively manages task requirement understanding, procedural planning, data processing, environment configuration, automated self-correction, and model training execution, streamlining the entire process [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_3].

- Challenges and Opportunities in Medical Diagnostics

A primary challenge in deploying agentic workflows within the medical domain is the existing shortage of well-prepared, domain-specific tools [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_2]. This scarcity arises from the inherent complexity of medical workflows, which encompass diverse diseases, varied imaging modalities, and a broad spectrum of task requirements [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_2]. Creating and integrating tools that can effectively handle this complexity is a significant hurdle [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_2]. However, agentic workflows also present significant opportunities to bridge the gap between clinical needs and the creation of tailored AI solutions [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_3]. By automating and streamlining the development process, agentic systems can facilitate the creation of AI tools that precisely address specific clinical challenges [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_3].

2. Multi-Agent Collaboration Frameworks

- Core Components of Multi-Agent Systems

Multi-agent systems are characterized by the use of multiple LLM agents, each with distinct roles, working together to address complex AI tasks [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_7]. These agents employ a divide-and-conquer strategy, breaking down large problems into smaller, more manageable sub-tasks [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_7]. The agents then iteratively perform code generation, editing, and execution using tools defined by toolset descriptions until a functional AI model is successfully produced [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_7]. M3Builder's multi-agent collaboration framework exemplifies this approach by decomposing the AI task into four distinct sub-tasks and assigning them to specialized role-playing LLM agents [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_13].

- Role Specialization in Agentic Workflows

Role specialization is a key feature of effective multi-agent systems. In M3Builder, the Task Manager is responsible for selecting the most suitable dataset for the given task and generating a comprehensive planning document to guide the other agents [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_13]. The Data Engineer then transforms raw data into a format suitable for model training, ensuring data quality and compatibility [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_14]. Following data preparation, the Module Architect integrates essential components into the training pipeline, including the design of the model architecture and selection of appropriate training parameters [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_15]. Finally, the Model Trainer finalizes the debugging and optimization of the training procedure, ensuring the model performs effectively [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_16].

- Advantages of Multi-Agent Collaboration

Multi-agent collaboration offers significant advantages over single-agent systems, particularly in complex domains like medical imaging. The absence of multi-agent collaboration can result in a considerable performance gap, highlighting the importance of this approach [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_22]. M3Builder's multi-agent system achieves a higher average success rate while requiring fewer action steps and execution iterations compared to single-agent systems, demonstrating its efficiency [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_22]. The multi-agent framework also demonstrates robustness and adaptability in completing assigned task executions, showcasing its ability to handle diverse and challenging tasks [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_24].

3. Medical Imaging ML Workspace

- Structure and Functionality of the Workspace

The Medical Imaging ML workspace provides agents with a structured environment that includes free-text descriptions of datasets, training codes, and interaction tools, facilitating seamless communication and task execution [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_1]. This workspace is designed to support the entire machine learning workflow, from data ingestion to model deployment [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_1]. It includes key elements such as data cards, toolset descriptions, and code templates, which provide the necessary context and resources for the agents to perform their tasks effectively [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_6]. The workspace equips the multi-agent framework with the necessary resources for dataset preparation, a coding foundation, and a well-defined action space, ensuring that the agents have everything they need to succeed [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_10].

- Key Elements: Datacards, Toolsets, and Code Templates

Datacards are a critical component of the workspace, containing key information such as the dataset name, a concise summary of its contents, and detailed metadata about the data [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_10]. This information helps the agents understand the characteristics of the available datasets and select the most appropriate one for the task at hand [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_10]. The toolset includes a range of functions, such as list_files, read_files, copy_files, write_files, edit_files, and run_script, which allow the agents to interact with the environment and manipulate files and data [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_12]. Code templates are tailored to primary medical imaging tasks, such as disease diagnosis, organ segmentation, anomaly detection, and report generation, providing a starting point for the agents to develop their models [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_11].

- Importance of a Structured Environment

The structured environment of the ML workspace plays a crucial role in guiding the automatic AI workflow [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_6]. It provides a standardized starting point for agents, ensuring consistency and reproducibility across different tasks and experiments [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_8]. The workspace also defines and describes all tools available to the agents, restricting their action space to a predefined, complete set, which helps to simplify the development process and reduce the risk of errors [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_8].

4. Automation of Machine Learning Workflows

- End-to-End Automation in M3Builder

M3Builder is designed to autonomously manage the entire development process, from initial data preparation to final model construction, training, and deployment [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_4]. This end-to-end automation significantly reduces the need for human intervention and accelerates the development of AI models for medical imaging [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_4]. Given a medical imaging-specific ML task and raw training data, M3Builder can perform the entire process without requiring extensive manual configuration or coding [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_4]. The system integrates user requirements with a workspace containing candidate data, tools, and code templates, providing a complete environment for automated model development [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_10].

- Workflow from User Request to Model Delivery

The M3Builder workflow begins with a user's free-text request, describing the desired medical imaging task, and culminates in the delivery of a trained AI model ready for deployment [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_10]. The Task Manager analyzes the user's request and selects the appropriate data, while the Data Engineer screens and processes the data to ensure its quality and suitability [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_2]. The Module Architect then writes code to complete the pipeline, designing the model architecture and training procedure, and the Model Trainer trains the model and auto-debugs any errors that arise during the training process [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_2]. A sample log tracks the Model Trainer agent's activities during diagnosis model development, providing valuable insights into the training process [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_10].

- Self-Correction and Debugging Mechanisms

M3Builder incorporates automated self-correction and debugging mechanisms to ensure the robustness and reliability of the generated AI models [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_3]. Auto-debugging proves crucial for successful training, allowing the system to identify and fix errors without human intervention [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_22]. The agents iteratively generate, edit, and refine code, incorporating feedback from the compiler, until the code executes successfully and the model achieves the desired performance [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_14].

5. Evaluation Benchmarks for Medical Imaging AI

- Introduction to M3Bench

M3Bench is a specialized benchmark designed to evaluate the performance of agentic systems in the context of automated machine learning for medical imaging [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_1]. It comprises four general tasks performed on 14 different training datasets, covering five anatomical regions and three different imaging modalities, including both 2D and 3D data [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_1]. This benchmark provides a standardized foundation for assessing the capabilities of agentic systems, ensuring that they meet the specific requirements of medical imaging applications [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_5]. M3Bench includes four general tasks: organ segmentation, anomaly detection, disease diagnosis, and report generation, which represent common and important applications of AI in medical imaging [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_4].

- Tasks and Datasets in M3Bench

The tasks included in M3Bench are designed to cover a range of common and important applications of AI in medical imaging. These tasks include organ segmentation, which involves identifying and delineating specific organs in medical images, anomaly detection, which aims to identify unusual or abnormal features in the images, disease diagnosis, which focuses on using the images to diagnose specific diseases or conditions, and report generation, which involves automatically generating reports based on the information extracted from the images [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_4]. These tasks are associated with 14 detailed training datasets, spanning five anatomical regions (head & neck, chest, abdomen & pelvis, limb, spine) and three primary imaging modalities (X-ray, CT, MRI), encompassing both 2D and 3D models [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_4]. The datasets include ADNI, KneeMRI, CC-CCII, CT-Kidney, BTCV, MSDPancreas, VerSe, L2R-OASIS, COVID-19, CT-RATE, INSTANCE2022, ChestX-Det10, RadGenome-Brain-MRI, and IU-Xray [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_18].

- Metrics for Assessing System Performance

The effectiveness of agentic systems evaluated using M3Bench is assessed through analysis of task completion rates, framework superiority compared to other systems, and the performance of individual agents in their assigned roles [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_17]. Task completion is defined as the successful training of a model that achieves an acceptable level of performance on a held-out test set [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_19]. The performance of each role-specific agent is evaluated using distinct success criteria, such as the accuracy of dataset selection by the Task Manager and the quality of code generated by the Module Architect [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_23].

6. Comparative Analysis with State-of-the-Art Systems

- Benchmarking Against Existing Agentic Systems

M3Builder's performance and capabilities have been rigorously compared against those of other state-of-the-art agentic systems, including MLAgent-Bench, Aider, Cursor Composer, Windsurf Cascade, and Copilot Edits [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_21]. This benchmarking process involves each system performing the same set of tasks within the M3Bench environment, using their respective built-in frameworks and tools [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_21]. These comparisons provide valuable insights into the strengths and weaknesses of different approaches to automated machine learning in medical imaging [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_21]. M3Builder consistently achieves superior performance across a range of metrics, demonstrating its effectiveness and efficiency in this domain [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_5].

- Performance Comparison Across Radiology Tasks

Across a range of radiology tasks, including Organ Segmentation, Anomaly Detection, Disease Diagnosis, and Report Generation, M3Builder has demonstrated significant advantages over other agentic systems. MLAgent-Bench, for example, performed poorly due to insufficient data structure understanding capabilities [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_21]. Other frameworks achieved only moderate success rates (39.29% max) due to limitations associated with single-agent architectures and the need for human intervention [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_21]. M3Builder, with its multi-agent collaboration framework and automated debugging mechanisms, demonstrated superior performance, achieving a higher average success rate while requiring fewer action steps and execution iterations [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_22].

- Advantages of M3Builder over Existing Systems

M3Builder's superior performance can be attributed to several key factors, including its multi-agent collaboration framework and the use of well-crafted example instructions [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_22]. The multi-agent approach allows for the division of complex tasks into smaller, more manageable sub-tasks, with each agent specializing in a specific role [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_22]. This specialization leads to greater efficiency and accuracy in the overall workflow [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_22]. Additionally, M3Builder achieves a 94.29% model building success rate with Claude-3.7-Sonnet standing out among seven SOTA LLMs, highlighting the importance of selecting an appropriate LLM for the task [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_25]. The experimental results demonstrate that M3Builder possesses impressive and robust capabilities for medical imaging model training automation, making it a valuable tool for researchers and clinicians [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_21].

7. The Role of Large Language Models (LLMs)

- LLMs as Agent Cores

Large Language Models (LLMs) serve as the core intelligence driving the agents within the M3Builder framework, enabling them to understand and execute complex tasks related to medical imaging [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_1]. These LLMs provide the agents with the ability to process natural language, generate code, and reason about complex medical concepts [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_1]. Seven state-of-the-art large language models serve as agent cores for the system, such as Claude series, GPT-4o, and DeepSeek-V3 [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_1]. The evaluation employs 7 leading LLMs as agent cores: GPT-4 , Claude-3.7-Sonnet, Claude-3.5-Sonnet , DeepSeek-v3 , Gemini-2.0-flash , Qwen-2.5-max , and Llama-3.3-70B [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_17].

- Performance of Different LLMs in M3Builder

The performance of different LLMs within M3Builder can vary significantly, highlighting the importance of selecting the most appropriate LLM for the specific task and dataset [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_19]. Claude-3.7-Sonnet achieves the highest completion rate of 94.29%, demonstrating its effectiveness in this context, while Gemini2.0 and Llama3.3 only reach 4.29%, indicating that they may not be as well-suited for these tasks [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_19]. The Task Manager demonstrated exceptional accuracy in task analysis, with stable token usage across all tasks, suggesting that it is effectively leveraging the capabilities of the LLM to understand and respond to user requests [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_23]. The majority of agents successfully completed or nearly completed all assigned task executions, showcasing the robustness and adaptability of the multi-agent framework, regardless of the specific LLM used [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_24].

- Impact of LLM Selection on Task Completion

The selection of the LLM has a significant impact on the overall task completion rate and the accuracy of the generated AI models [chunk_23cf0170e733818337c330db2b9dfedfbd9a84799dc9fcb957fe7a7036e65bd1_1]. The choice of LLM affects the systems ability to precisely and reliably answer questions based on EHRs, highlighting the importance of selecting an LLM that is well-suited for the specific data and task [chunk_23cf0170e733818337c330db2b9dfedfbd9a84799dc9fcb957fe7a7036e65bd1_18]. The system's accuracy rate improves with the few-shot strategy, suggesting that LLMs can benefit from being provided with examples of how to perform the task [chunk_23cf0170e733818337c330db2b9dfedfbd9a84799dc9fcb957fe7a7036e65bd1_36].

8. Monomodal vs. Multimodal Approaches

- Focus on Monomodal Data

The current implementation of M3Builder primarily focuses on monomodal data, which means it primarily processes and analyzes data from a single source or modality at a time [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_6]. Specifically, the system operates using textual data and associated metadata, such as dataset descriptions and code templates [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_6]. The system operates using textual data from EHRs to create patient profiles [chunk_23cf0170e733818337c330db2b9dfedfbd9a84799dc9fcb957fe7a7036e65bd1_39]. The data cards in the workspace are represented in natural language, providing a human-readable description of the available datasets [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_6].

- Limitations of Monomodal Modeling

The exclusive reliance on textual data limits the richness and comprehensiveness of the simulation experience [chunk_23cf0170e733818337c330db2b9dfedfbd9a84799dc9fcb957fe7a7036e65bd1_39]. While textual data provides valuable information about the characteristics of the datasets and the structure of the code, it does not capture the full complexity of medical imaging data [chunk_23cf0170e733818337c330db2b9dfedfbd9a84799dc9fcb957fe7a7036e65bd1_39]. The absence of visual processing limits the system's ability to approximate clinical expertise, as clinicians often rely on visual cues and patterns in medical images to make diagnoses and treatment decisions [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_25]. The system could benefit from the integration of medical images such as ECGs, X-rays, MRIs, and CT scans, which would provide a more complete and realistic representation of the patient's condition [chunk_23cf0170e733818337c330db2b9dfedfbd9a84799dc9fcb957fe7a7036e65bd1_40].

- Potential for Multimodal Integration

Future iterations of the system could integrate medical images such as ECGs, X-rays, MRIs, and CT scans, allowing for a richer and more holistic patient simulation experience [chunk_23cf0170e733818337c330db2b9dfedfbd9a84799dc9fcb957fe7a7036e65bd1_40]. This multimodal approach would allow the system to leverage the complementary strengths of different data modalities, such as the detailed anatomical information provided by MRI scans and the functional information provided by ECGs [chunk_23cf0170e733818337c330db2b9dfedfbd9a84799dc9fcb957fe7a7036e65bd1_40]. The advent of multimodal large language models (MLLMs) that can process a variety of data types opens new possibilities for integrating medical images and other non-textual data into the system [chunk_23cf0170e733818337c330db2b9dfedfbd9a84799dc9fcb957fe7a7036e65bd1_39]. By incorporating these modalities, the system could offer more realistic and comprehensive medical investigations, where both clinical notes and imaging data are used in diagnosis and treatment planning [chunk_23cf0170e733818337c330db2b9dfedfbd9a84799dc9fcb957fe7a7036e65bd1_40].

9. Applications in Medical Service Management

- Enhancing Medical Education

The system has the potential to significantly enhance medical education by providing a realistic and interactive simulation environment for medical students [chunk_23cf0170e733818337c330db2b9dfedfbd9a84799dc9fcb957fe7a7036e65bd1_6]. It could benefit medical education and research applications, such as serving as simulated patients in medical student education, allowing students to practice their diagnostic and treatment skills in a safe and controlled setting [chunk_23cf0170e733818337c330db2b9dfedfbd9a84799dc9fcb957fe7a7036e65bd1_6]. It facilitates patient-focused evaluation on AI models and integrates as a patient agent in multi-agent AI systems, providing a platform for evaluating the performance of AI models in realistic clinical scenarios [chunk_23cf0170e733818337c330db2b9dfedfbd9a84799dc9fcb957fe7a7036e65bd1_7]. The system is potentially useful for both medical student training and medical system integrations, making it a valuable tool for improving the quality of medical care [chunk_23cf0170e733818337c330db2b9dfedfbd9a84799dc9fcb957fe7a7036e65bd1_37].

- Improving Clinical Decision-Making

Simulated patient systems are designed to enhance integrative learning and evaluation by incorporating basic science objectives, simulating the outcomes of clinical decisions, and including diverse cases to improve cultural competency [chunk_23cf0170e733818337c330db2b9dfedfbd9a84799dc9fcb957fe7a7036e65bd1_3]. By providing clinicians with access to a wide range of simulated patient scenarios, the system can help them to improve their diagnostic and treatment skills and make more informed decisions [chunk_23cf0170e733818337c330db2b9dfedfbd9a84799dc9fcb957fe7a7036e65bd1_3]. The system supports medical investigation in an effective and trustworthy manner, providing clinicians with access to reliable and accurate information [chunk_23cf0170e733818337c330db2b9dfedfbd9a84799dc9fcb957fe7a7036e65bd1_32]. By incorporating these modalities, the system could offer more realistic and comprehensive medical investigations, where both clinical notes and imaging data are used in diagnosis and treatment planning, leading to more accurate diagnoses and more effective treatment plans [chunk_23cf0170e733818337c330db2b9dfedfbd9a84799dc9fcb957fe7a7036e65bd1_40].

- Facilitating Patient-Focused Care

The AIPatient system delivers verified information in a natural language format tailored to the user's needs, particularly aligning responses with patient personalities [chunk_23cf0170e733818337c330db2b9dfedfbd9a84799dc9fcb957fe7a7036e65bd1_6]. This personalized approach can help to improve patient engagement and satisfaction, as patients are more likely to trust and follow the advice of a system that understands and responds to their individual needs [chunk_23cf0170e733818337c330db2b9dfedfbd9a84799dc9fcb957fe7a7036e65bd1_6]. It ensures continuity in the interaction by summarizing and updating the conversation history throughout the process, allowing for a more seamless and natural interaction between the patient and the system [chunk_23cf0170e733818337c330db2b9dfedfbd9a84799dc9fcb957fe7a7036e65bd1_6]. The system successfully balances the need for medical accuracy with the necessity for clear communication, making it potentially useful for both medical student training and medical system integrations [chunk_23cf0170e733818337c330db2b9dfedfbd9a84799dc9fcb957fe7a7036e65bd1_37].

10. Future Directions and Research Gaps

- Expanding Dataset Diversity

Future work includes extending beyond medical imaging to broader medical tasks, developing more robust tool-building agent systems, implementing automated dataset preparation capabilities, and incorporating visual processing to better approximate clinical expertise [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_25]. Expanding the diversity of the datasets used to train and evaluate the system is a critical area for future research [chunk_2b2422a31813af12956ea95279bc3c0277fcf49eed441622fb9f4f4ecd54705d_25]. The current dataset only covers a homogenous population, which limits the generalizability of the study's results and may introduce biases into the system [chunk_23cf0170e733818337c330db2b9dfedfbd9a84799dc9fcb957fe7a7036e65bd1_39]. Future work plans to enrich the KG by incorporating more medical entities and relationships and expand the current patient population to a broader population to enhance system inclusiveness, ensuring that the system is representative of the diverse patient populations that it will be used to serve [chunk_23cf0170e733818337c330db2b9dfedfbd9a84799dc9fcb957fe7a7036e65bd1_39].

- Enhancing System Robustness and Stability

Future research could include multi-round evaluations, designed with input from professionals across various medical disciplines to better reflect the nuances of their respective fields [chunk_23cf0170e733818337c330db2b9dfedfbd9a84799dc9fcb957fe7a7036e65bd1_41]. Multi-round evaluations would allow for a more comprehensive assessment of the system's ability to handle complex and evolving clinical scenarios [chunk_23cf0170e733818337c330db2b9dfedfbd9a84799dc9fcb957fe7a7036e65bd1_41]. The system maintains its stability across different simulated personalities, ensuring the integrity and consistency of the medical information presented, regardless of personality variations [chunk_23cf0170e733818337c330db2b9dfedfbd9a84799dc9fcb957fe7a7036e65bd1_31]. The system delivers human-like, accessible, and context-aware generation while maintaining high system robustness and personality stability in responses to medical investigations, demonstrating its ability to provide reliable and consistent information across a range of different scenarios [chunk_23cf0170e733818337c330db2b9dfedfbd9a84799dc9fcb957fe7a7036e65bd1_36].

- Addressing Ethical and Practical Considerations

Future work is needed to understand the comfort and concerns of trainees, physicians, and patients regarding the implementation of such generative AI systems in clinical education and practice [chunk_23cf0170e733818337c330db2b9dfedfbd9a84799dc9fcb957fe7a7036e65bd1_42]. As AI systems become more prevalent in healthcare, it is essential to address the ethical, psychological, and professional considerations that arise [chunk_23cf0170e733818337c330db2b9dfedfbd9a84799dc9fcb957fe7a7036e65bd1_42]. Ethical, psychological, and professional considerations must be addressed when integrating AI in sensitive healthcare contexts, ensuring that the systems are used responsibly and ethically [chunk_23cf0170e733818337c330db2b9dfedfbd9a84799dc9fcb957fe7a7036e65bd1_42]. The system's safety and reliability in real-world applications must be ensured by addressing issues such as appropriate socio-cultural contexts and realistic case scenarios, minimizing the risk of errors and ensuring that the systems are used in a way that is consistent with the values and beliefs of the patients and communities they serve [chunk_23cf0170e733818337c330db2b9dfedfbd9a84799dc9fcb957fe7a7036e65bd1_42].