System Audits
Independent review and examination of records and activities to assess the adequacy of system controls for compliance with established policies and procedures
System Audits, Reviews, and Assessments
There are several methods at CMS for ensuring that information system controls, records, and activities are in compliance with security and privacy requirements. These methods are summarized below.
System Audits
System Audits evaluate CMS's information technology infrastructure, applications, data management, policies, and procedures. Auditors collect evidence, or artifacts, from CMS systems and compare it to recognized standards and established federal laws and policies. System Audits evaluate if implemented controls protect system security and privacy, and safeguard beneficiary personally identifiable information (PII) and protected health information (PHI). If deficiencies are found, Auditors provide Findings and Recommendations to CMS staff. CMS then can use the information provided within the specific Finding or Recommendation to correct deficiencies and improve system security and privacy posture.
Reviews
Reviews are an internal management evaluation that asserts that a financial statement is free from material misstatement. In this activity, the Reviewers perform the same action of collecting evidence or artifacts and reporting on the result of testing.
Assessments
An Assessment is defined as the testing and/or evaluation of the management, operational, and technical security controls in an information system to determine the extent to which the controls are implemented correctly, operating as intended, and producing the desired outcome with respect to meeting the security requirements for the system. Assessments are not Audits.
Audits and Reviews conducted specifically for CMS or as part of an HHS audit, are conducted by independent auditors under the guidelines established by the Government Accountability Office (GAO) and HHS’s Office of Inspector General (OIG). These tests examine and report on CMS’s ability to provide reasonable assurances of achievement for the following control objectives:
- Confidentiality, availability, and integrity of information
- Compliance with policies, plans, procedures, laws, and regulations
- Safeguarding assets
- Economical and efficient use of resources
- Facilitating communication
The Federal Information Security Modernization Act (FISMA) requires federal agencies to perform both internal and external audits, reviews, and assessments. Additionally, the National Institute for Standards and Technology (NIST) provides their own audit policies and procedures that are designed to keep Federal systems secure.
Who manages Audits at CMS?
Within CMS, two roles take the lead in managing the audit process, the Information System Security Officer (ISSO), and the Audit Liaison.
Information System Security Officer (ISSO)
The ISSO's role in an audit is to provide the evidence and artifacts requested by the auditors. The ISSO acts as the representative for the system being audited and works with the auditors and the Audit Liaison to address any IT security-related issues. When audit Findings and Recommendations are released, the ISSO works with the system team to correct Findings and address any resulting Plan of Action and Milestones (POA&Ms) created. The ISSO’s role in any audit is to ensure audit compliance. If you have specific questions about the audit or the system being audited, it’s best to direct your questions to the Audit Liaison or the OIT Audit Team.
Audit Liaison
An audit liaison is a CMS employee or designated contractor that acts as the intermediary between a CMS component and an audit-related entity, such as the Office of Inspector General (OIG) or the Government Accountability Office (GAO). The liaison coordinates activities, including:
- Audit request broadcasts
- Scheduling walkthrough dates
- Negotiating on behalf of the stakeholders,
- Correspondence between the auditors and the ISSOs who represent the system in the scope of the audits
The OIT Audit Team is the audit liaison for the Information Security and Privacy Group (ISPG). Not every component within CMS has an audit liaison; to find yours, contact the OIT Audit Team or your Division Director.
Types of Audits at CMS
Many different types of audits occur within the CMS ecosystem, but all audits can be classified as either Scheduled Audits and Reviews or Ad Hoc Audits and Reviews. Scheduled Audits and Reviews occur at a regular cadence, while Ad Hoc Audits and Reviews are conducted at the request of Federal agencies and can occur any time during the year.
Scheduled Audits and Reviews
Annual FISMA Audit
The Office of Management and Budget (OMB) and the Department of Homeland Security (DHS) work with an interagency working group to develop Chief Information Officer (CIO) FISMA metrics to track agencies' progress in implementing cybersecurity capabilities. OMB and DHS also collaborate with the Inspector General (IG) community to ensure that the IG FISMA metrics provide independent assessments of agency information security programs under FISMA requirements.
As part of the FISMA Audit, the OIG requires agencies to conduct a self-assessment of the information security program of the organization. Since 2017, the Office of Inspector General has used the information gathered during the self-assessment as the basis for the metrics in the FISMA Audit. Responses made by CMS provide a foundation for FISMA Audit requests to systems to determine the overall maturity level of the CMS Information Security Program. The FISMA Audit then assigns a maturity model level to each agency. The maturity model is aligned with the NIST Cybersecurity Framework. There are five levels of maturity:
Policies, procedures, and strategies are not formalized; activities are performed in an ad-hoc, reactive manner.
Policies, procedures, and strategies are formalized and documented but not consistently implemented.
Policies, procedures, and strategies are consistently implemented, but quantitative and qualitative effectiveness measures still need to be improved.
Quantitative and qualitative measures of the effectiveness of policies, procedures, and strategies are collected across the organization and used to assess them and make necessary changes.
Policies, procedures, and strategies are fully institutionalized, repeatable, self-generating, consistently implemented, and regularly updated based on a changing threat and technology landscape and business/mission needs.
The auditors give an agency an Overall Rating Score after reviewing the maturity model self-assessment responses and the artifacts provided in the audit. The score is based on the lowest rating identified. An Overall Rating Score is categorized as either an Information Security Program being “Effective or Not Effective.” Level 4 (Managed and Measurable) or Level 5 (Optimized) are the only ratings that result in an agency’s Information Security Program being labeled Effective.
Quarterly CIO Metrics FISMA Report
These quarterly audits focus on assessing an agency’s progress toward achieving outcomes that strengthen federal cybersecurity. Agency progress is measured using guidance provided by FISMA. The FISMA Report:
- Ensures that agencies implement the Administration's priorities and best practices
- Provide the Office of Management and Budget (OMB) with performance data to monitor progress toward implementing the Administration's priorities
It is important to note that as a result of FISMA, regulations or guidance published throughout the year may affect the metrics within the report. For example, with the publication in Fiscal Year 2022 of Executive Order (EO) 14028, the report added metrics specifically asking to identify requirements in EO 14028.
Annual Senior Agency Official for Privacy (SAOP) Report
On an annual basis, OMB issues guidance instructing each SAOP to review the administration of the agency’s privacy program and report compliance data to the OMB. This report is completed simultaneously with the Quarter 4 FISMA CIO Metrics Report.
OMB collects the annual Senior Agency Official for Privacy (SAOP) FISMA Metrics according to the authority in the Federal Information Security Modernization Act of 2014, the Privacy Act of 1974, the Paperwork Reduction Act of 1995, the E-Government Act of 2002, Executive Order 13719, OMB Circular A-130,6 OMB Circular A-108, as well as other laws, regulations, and policies.
A-123 Review
OMB Circular A-123 issued under the authority of Federal Managers' Financial Integrity Act (FMFIA) prescribes the process to assess the effectiveness of an agency's internal control program. It provides specific requirements for federal agencies to:
- Establish internal controls
- Assess internal controls
- Correct internal control deficiencies
- Report on internal controls
OMB Circular A-123 also directs agencies to mature an Enterprise Risk Management (ERM) Program (as of FY22, ERM is not tested in the A-123 Review but may change in the future).
The A-123 Audit is conducted annually. The OIT Audit Team supports the efforts of the A-123 Reviewers, but the team does not lead this type of audit. Instead, A-123 Reviewers will directly interface ISSOs related to audit support. Systems designated as financial systems (Chief Financial Officer systems) are within the scope of the A-123 Review. The primary focus is on how Findings might impact financial statements.
Both the A-123 Review and the CFO Audit (detailed below) focus on internal controls of financial systems and oversight. Think of the A 123 Review (or reviews in general) of “Hey, we are providing assurance to you that CMS meets the requirements of implementing the control.”
Chief Financial Officer (CFO) Audit
CFO Audits occur annually. They establish a leadership structure, provide for long-range planning, and strengthen accountability reporting by requiring audited financial statements. The CFO Audit, as in the case of the A-123 Review, is focused on systems identified within CMS as CFO systems. As with the A-123 Review test results, there is also a focus on how Findings affect financial statements.
CFO Audits are governed by the Chief Financial Officer Act of 1990, and the Federal Information System Controls Audit Manual (FISCAM). FISCAM provides the methodology for auditing information system controls in federal and other governmental entities.
IRS Safeguards Review
The Internal Revenue Service (IRS) Safeguards Review is conducted annually. It ensures that agencies and contractors that have access to federal tax returns and privileged information from the IRS maintain adequate safeguards for the protection of such information. Only those CMS systems that collect Federal Tax Information (FTI) are within the scope of the Review.
IRS Special Publication 1075 serves as guidance for all US government agencies that access Federal Tax Information (FTI) to ensure that they use policies, practices, and controls to protect its confidentiality.
Ad Hoc Audits and Reviews
Ad-hoc, or unscheduled audits, are not scheduled until the OIT Audit Team receives a notification from the Office of Legislation (OL) about the need for an audit. The Office of Inspector General (OIG) or the General Accountability Office (GAO) typically leads these audits. The OL acts as the CMS liaison between the OIT Audit Team and the OIG/GAO during these activities. There are two types of Ad Hoc audits:
CISO-Owned Privacy and Security Audits
These audits occur at the request of OIG or GAO. They assess the security and/or privacy compliance of CMS FISMA systems. The OIT Audit Team staff lead these audits when the Office of Legislation (OL) identifies that ISPG is the lead component (primary focus) for the audit.
CISO-Supported Privacy and Security Audits
These audits are completed at the request of another federal agency, such as the Government Accountability Office (GAO) or the Office of Inspector General (OIG). The purpose of these audits is to assess the security and/or privacy compliance of FISMA systems. The OIT Audit Team’s staff leads these audits when the Office of Legislation (OL) identifies that ISPG is not the primary focus of the audit.
Audit process
Beginning your audit
System Audits are initiated when the OIT Audit Team receives a notification from the auditor that conducts the audit directly, or from the CMS Office of Legislation. Once notification has been received, the OIT Audit Team identifies the points of contact or your component's audit liaison and informs them of the audit via email.
Kick-off meeting
The auditors and the OIT Audit Team work with the ISSO and Audit Liaison to schedule a kick-off meeting to initiate the audit. The kick-off meeting identifies the system being audited, determines the scope of the audit, and sets deadlines for audit activities.
Audit artifacts provided
Based on the audit scope determined during the kick-off meeting, Auditors request a list of artifacts and ask questions about the system. During this time, the Auditors may also perform a walkthrough if required by the scope of the audit. The ISSO then works (either alone or with the Audit Liaison) to return the requested artifacts to the Auditors within 5 business days.
Feedback and Findings Received
Auditors review the artifacts provided and the responses to Auditor questions, and provide feedback to the ISSO and Audit Liaison. They may also issue Findings to correct any deficiencies.
Evidence and artifacts acquired by Auditors during an Audit lead to test results. The test result occurs after evaluating a security control, privacy control, control enhancement, standard, or regulation. An audit’s test result can lead to a Finding if the identified test result does not meet the evaluated requirement. Any Findings are returned to the ISSO and Audit LResponse
Depending on the nature of the feedback, ISSOs and/or Audit Liaisons have an opportunity to submit new artifacts and evidence or even request a meeting with the auditors to address the concerns of the stakeholders notified of the Finding. The type of audit and the timeline outlined during the kick-off meeting determine the time frame that will be used to review and fix any Findings.
Remediation
The test result that shows a requirement is Not Satisfied (Finding) will result in the creation of a Corrective Action Plan (CAP) known as a Plan of Action and Milestone (POA&M). A POA&M shows the progress of how a Finding is remediated. The POA&M is stored in the CMS FISMA Controls Tracking System (CFACTS) application and is generated either via the CMS Assessment, and Audit Tracker Sheet (CAATS) file manual methodology or the OIT Audit Team’s CFACTS Audit Module automated process.
CAAT created
The OIT Audit Team will create the CAAT after Findings from an audit have been approved by the stakeholders who own the Finding and the auditors (note that this does sometimes mean that Final Report has yet to be issued at that time). OIT Audit Team will submit the CAAT for review to system stakeholders, including the Information System Security Officer (ISSO), Cyber Risk Adviser (CRA), Business Owner (BO), and Audit Liaison (where applicable). Once the system stakeholder approves the CAAT, the OIT Audit Team will continue the process of the CAAT submission and subsequent POA&M creation.
Draft report released
The Auditors then release a draft report that outlines the specific Findings and remediation efforts. ISPG management and those stakeholders within the scope of the audit can agree or disagree with the report. If everyone accepts the draft report, it is sent to CMS management for final review and approval. When someone disagrees with the draft report, the comments are made to the OL or the auditor to identify the reasons for not approving the draft report. The draft report is then updated, reviewed, and approved by all stakeholders and the Auditors.
Final report
A final report is the published analysis of the audit or review that occurred. The final report identifies the scope and purpose of the audit, details any Findings, and provides recommendations to remediate any Findings.
The final report can be public or private. A public report is published online and released to the general public. Private reports are the result of audits that contain sensitive information and are released only to CMS and the stakeholders within the scope of the audit. System teams will know from the start of the audit whether their audit is sensiti
Follow up
Auditors will follow up on the progress of the Findings and any related POA&Ms within six months during the Final Management Decision (FMD) process. During the FMD, CMS management must review and approve the evidence supplied to the auditors.
For the scheduled audits and reviews, the POA&Ms are reviewed by the auditor/reviewer during the following year’s scheduled cycle to determine the feasibility and approval to close the POA&Ms of any identified Findings.
Related documents and resources
A corrective action plan roadmap to address system weaknesses and the resources required to fix them
CFACTS is a CMS database that tracks application security deficiencies and POA&Ms, and supports the ATO process
FISMA is federal legislation that defines a framework of guidelines and security standards to protect government information and operations
Information about NIST and how the agency's policies and guidance relate to security and privacy at CMS
Procedures to help CMS staff and contractors implement federal policies and standards for information security and privacy
The IS2P2 defines how CMS protects and controls access to its information and systems. It outlines compliance activities and defines roles and responsibilities.