Operational maturity areas represent organizational functions that impact the implementation of AI capabilities.
While each area is treated as a discrete capability for maturity evaluation, they generally depend on one another. The operational maturity areas are:
- PeopleOps: Recruit, develop, retain, and organize an AI-ready workforce.
- CloudOps: Provide and allocate storage, compute, and other resources in the cloud.
- SecOps: Ensure secure deployment of code, access to systems and data, and identity resolution across storage, compute, and data assets.
- DevOps: Deploy and manage software throughout development, test, and production environments.
- DataOps: Maximize data discovery, access, and use throughout its lifecycle.
- MLOps: Test, experiment, and deploy AI or ML models.
- AIOps: Identify and resource AI initiatives within the organization.
Each operational maturity area below is supported by related key activities and key questions to focus the conversation on the current organizational state.
PeopleOps
People are at the core of every organization. PeopleOps covers not only the defining skills required to build and deploy AI capabilities, but also creating a culture that promotes the behavior necessary to successfully create intelligent systems.
Organizations must be able to explicitly identify skill sets and traits that make a strong AI team. We focus intentionally on continuously developing skill sets, technical acumen, and dynamic team management. We also identify talent gaps and provide training. Strong PeopleOps practices include the structure and program support to organize high-functioning teams and manage them throughout the entire talent lifecycle.
Organizational Maturity for PeopleOps
Initial / Ad Hoc Individual Project
- Self-directed identification and acquisition of AI related skillsets
Repeatable Team Project
- Identify types of AI talent for Lifecycle
- Employee journey created
- KSA’s identified
- Active talent acquisition for AI related skills begin
Defined Program
- PeopleOps Program established
- Conduct talent gaps analysis
- Needs documented
- Talent map created
- Training identified
Managed Portfolio
- PeopleOps program linked to measurable performance objectives and active efforts to create a culture around AI is a central organizational theme
Optimized Enterprise
- Uses engagement tools for employee growth
- Integrates an ownership culture throughout organization
Key Activities for PeopleOps
- Understand the AI talent requirements for the organization (needs, skills, and training assessments, use case identification)
- Develop the AI talent lifecycle talent pipeline, staffing plans (FTE + contractors), learning paths, project team composition
- Enhance the AI talent lifecycle (automate the talent pipeline, develop internal best practices, expand employee engagement throughout the organization)
Key Questions for PeopleOps
- Can you identify the barriers that prevent your agency from building AI talent, or the precursors necessary to do so?
- Are you able to identify AI talent within your organization?
- How do you align culture with mission objectives to create an engaging, creative workplace?
- Do you understand staff’s training and resource needs?
- How can you identify ideal AI practitioner candidates?
- How can you ensure that AI practitioners within your organization are able to succeed?
CloudOps
Modern infrastructure and platform management best practices have consolidated in the cloud, so the CoE assumes that mature AI organizations will be cloud-capable.
CloudOps functionally manages development teams’ virtual onboarding, and provides compute and storage resources, as well as services and tools to enable development teams. Organizations must have a repeatable process to safely and securely stand up and scale development environments for a wide range of AI activities.
Organizational Maturity for CloudOps
Initial / Ad Hoc Individual Project
- Minimal Cloud Resources or Individual User Account
Repeatable Team Project
- Innovation Sandbox in the cloud
Defined Program
- Dev, Test, and Prod environments available, but manual resources allocation
Managed Portfolio
- Self-service or templated cloud resource allocation
Optimized Enterprise
- Balanced cloud resource sharing across the organization with robust cost/benefit/usage metrics
Key Activities for CloudOps
- Assess existing cloud capabilities in the organization (accounts, environments, service providers, etc.)
- Provide environments for development, test, and production-level activities, ensuring secure access controls within and between environments.
- Enhance the provided environments’ use through secured and automated pipelines.
- Continually monitor and optimize cloud resources to manage usage rates and compliance.
Key Questions for CloudOps
- Which cloud resources or platforms are available in your environment?
- Is there a systematic, enterprise-wide process by which to allocate cloud resources?
- How do AI practitioners get access to cloud resources?
- Are you able to track resource utilization across the enterprise?
DevOps
A mature AI organization must be able to securely move software into a production environment and provide continuous integration and delivery of software updates.
DevSecOps best practices enable organizations to shrink the time to deliver new software while maintaining the security of highly reliable and available AI-enabled capabilities. DevSecOps allow organizations to produce microservices in hardened containers; move them throughout development, test, and production environments; and scale services based on user demand.
Organizational Maturity for DevOps
Initial / Ad Hoc Individual Project
- Development on local workstation
Repeatable Team Project
- Process exists for moving locally developed tools into production, but some parts are manual
- Test Driven Development (TDD)
Defined Program
- Established secure process for containerizing tools and moving into production environment
Managed Portfolio
- Increasingly automated process for deploying secure software with emphasis on reducing iteration and delivery timelines
Optimized Enterprise
- Fully-managed secure software container orchestration
- CI/CD/CATO
Key Activities for DevOps
- Understand where tools are currently developed within the organization.
- Create and secure development pipelines from local to cloud environments.
- Contain and isolate dependencies for tools to effectively use and scale resources.
- Reduce delivery and iteration timelines through development pipeline automation and full containerization.
Key Questions for DevOps
- What processes exist for moving new tools into production?
- When are security concerns resolved during development?
- How quickly can any new tool be deployed in a containerized environment?
SecOps
Secure access to code repositories, infrastructure, and platform resources, and data assets depends on understanding how person and non-person entities (NPEs) operate.
SecOps unifies best practices in security for software development, system administration, data access, and protection of AI techniques from attacks. This function also supports the ability to audit and account for user activity, while defending against security threats across the system.
Organizational Maturity for SecOps
Initial / Ad Hoc Individual Project
- Code security/validation is manually accomplished in the Test environment
- Container security is default to orchestration
Repeatable Team Project
- Code security/validation is manually accomplished within the pipeline
- Container security is baselined at orchestration
Defined Program
- Code/Container security and validation is automated within the pipeline and manually approved
Managed Portfolio
- Code/Container security and validation software is automated and automatically approved
- Software rollouts are “trusted”
Optimized Enterprise
- Pipeline security software feeds central Security Data Lake
- Automation embedded at the Pipeline Orchestration layers
- Automated code rollbacks
- Ongoing Authorization
Key Activities for SecOps
- Threat detection: mean time to detect
- Threat resolution: mean time to respond
- Threat impact: mean failure cost
Key Questions for SecOps
- Are corrective actions fully integrated into the overarching security framework?
- Is mean time to detection (MTTD) near real-time?
- Does automated resolution implementation occur throughout pipelines, including automatic rolling back suspect code/version control?
- Is mean time to response (MTTR) near real-time?
- Are impact analyses to emerging threats automated and integrated into organizational strategic decision making?
- Are threat impact: mean failure costs projected?
DataOps
Effective AI capabilities require highly tailored data to train and develop models. To deliver data to development teams, DataOps enables data discovery, access, and use for AI development.
This includes support for batch and streaming data sources, the pre- and post-processing of data, and managing data at all lifecycle phases. Effective DataOps includes a thorough inventory asset catalog, dynamic methods of accessing data, and necessary tools to manipulate data to conduct documented AI experiments. Datasets are described and subjected to version control, creating a history of how the data was sourced and transformed throughout the development process.
Organizational Maturity for DataOps
Initial / Ad Hoc Individual Project
- “Shoeboxes” of data stored locally, not discoverable
- Copied from one machine to another
Repeatable Team Project
- Routine data sources available and well documented but new data discovery is ad hoc
- Exploratory data analysis (EDA) initiated
Defined Program
- Engineering support for data management activities is explicit
- Data pipeline exists
Managed Portfolio
- Self-service for adding new data sources, preparing datasets, and curating data for ML projects
Optimized Enterprise
- Intelligent, secure data discovery, access, and use across all organizations with metrics on business usage and compliance
Key Activities for DataOps
- Determine the locations and method of access for data sources within the organization
- Make tools and resources available to actively manage data
- Align data sources within a data lifecycle management framework
- Optimize use of data sources within the data management framework through automation and self-service data curation
- Continually monitor creation and usage of data to ensure business alignment and compliance
Key Questions for DataOps
- How is data transferred within the organization?
- Are there enterprise-wide data governance policies in place?
- What resources are available to ensure high-quality data management?
- Who is able to access data sources?
MLOps
MLOps is the selection, application, interpretation, deployment, and maintenance of machine learning models within an AI-enabled system.
Through these functions, AI development teams conduct data analysis and feature engineering to create a model that achieves a threshold level of performance. Over time, MLOps will enable deployed models to become more accurate as long as they are properly maintained. Another key function in MLOps is to create an audit trail of experiments to document design decisions throughout experimentation and create transparent, explainable AI-enabled capabilities.
Organizational Maturity for MLOps
Initial / Ad Hoc Individual Project
- Models and methods are selected ad hoc and not documented
Repeatable Team Project
- Implement methods for documenting experiments and model selection
- Utilization of GPUs initiated
Defined Program
- Model and methods catalog exists
- Model to use case matching leverages previous historical knowledge
- Model measures model accuracy and speed prediction
Managed Portfolio
- Increased use of infrastructure/Server based GPU acceleration for model development
- Automated selection, testing, and evaluating ML models using AutoML
Optimized Enterprise
- Use of Hyper-scale GPU acceleration
- Feedback from ML tools captured and models are continuously updated and improved
Key Activities for MLOps
- Build resources to document machine learning model and method selection
- Use historical documentation to develop and enhance machine learning models
- Monitor tool usage and feedback to continuously enhance tools
Key Questions for MLOps
- How are machine learning use cases identified and documented?
- Are practitioners able to access an enterprise knowledge-base of existing machine learning models?
- Are models tested, evaluated, and optimized?
AIOps
AIOps support AI capability development as a discipline of systems engineering and product management.
Initially, organizations may identify processes to create a funnel of AI pilots and move them to operations. Over time, effective AIOps will allow organizations to put institutional structure around AI activities through project management, product development, and systems integration.
AIOps focus squarely on integrating AI into the larger enterprise from a technical point of view as well as planning technology roadmaps for future investment. Program evaluation and support functions are also included in AIOps to show the operational impact of AI investments. AIOps include user-centered activities that organize tasks into iterative atomic units of work, manage allocating resources and personnel to work efforts, and set long-term objectives for AI-enabled capabilities.
Organizational Maturity for AIOps
Initial / Ad Hoc Individual Project
- Reactionary and ad hoc AI capability identification
Repeatable Team Project
- Established process to capture AI product requirements and user workflows
Defined Program
- Formal AI Product Management team, Strategy, and roadmap, including test and evaluation plan
Managed Portfolio
- AI Product Management goals are linked to organizational performance objectives
- T&E synthesis and evaluation included for each AI product
Optimized Enterprise
- AI Capability dependencies are mapped across org boundaries and linked to an enterprise strategy with measurable objectives
- Retrospective analysis of past efforts to continuously modify business objectives related to AI investment
- Dedicated T&E efforts to optimize AI cap
Key Activities for AIOps
- Establish processes by which to determine AI use cases
- Develop AI capabilities through AI product teams
- Align AI products with organizational objectives and enterprise-wide dependencies
- Continuously evaluate AI products to optimize business usage, resource allocation, and compliance
Key Questions for AIOps
- How are AI products identified within the organization?
- Who manages AI products?
- Are product dependencies mapped to both organization objectives and data dependencies?
