Algorithmic Impact Assessment (AIA) - Leveraging Artificial Intelligence (AI) for backlog reduction: AI assistant prototype

Version 0.10.0

On this page

Project Details

1. Name of Respondent
Human Capital Management – Data Modernization

2. Department
Public Works and Government Services (Department of)

3. Branch
Data Modernization

4. Project Title
Leveraging Artificial Intelligence (AI) for backlog reduction: AI assistant prototype

5. Project Phase
Design [Points 0]

6. Please provide a project description:
PSPC Human Capital Management aims to address and eliminate the existing backlog of pay cases by implementing an AI assistant designed to support compensation agents (CA). The primary objective of the AI assistant is to offer advice and necessary information to CAs and support them in expediating case closure. Currently, the tool is applied to synthetic data on select case types (actings) that are over 365 days old.

Adopting a hybrid-by-design approach, this initiative aims to harness the strengths of both human insight and machine efficiency. This approach enables rapid data consumption and recommendations by machines, while CAs retain decision-making authority over how the data is used, ensuring a balanced and effective resolution process.

The project will focus on aggregating relevant case data and information, assessing it for accuracy and completeness, and validating it against authoritative sources such as approved standard operating procedures (SoPs), job aids, directives, and policy documents. This comprehensive strategy aims to enhance the efficiency and accuracy of CAs work, facilitate faster case resolutions, and elevate the quality of service.

About the System

7. Please check which of the following capabilities apply to your system.
Image and object recognition: Analyzing very large data sets to automate the recognition, classification, and context associated with an image or object.
Content generation: Analyzing large data sets to categorize, process, triage, personalize, and serve specific content for specific contexts
Risk assessment: Analyzing very large data sets to identify patterns and recommend courses of action and in some cases trigger specific actions
Process optimization and workflow automation: Analyzing large data sets to identify and anomalies, cluster patterns, predict outcomes or ways to optimize; and automate specific workflows.

Section 1: Impact Level: 2

Current Score: 46
Raw Impact Score: 54
Mitigation Score: 37

Section 2: Requirements Specific to Impact Level 2

Peer review

Consult at least one of the following experts and publish the complete review or a plain language summary of the findings on a Government of Canada website:

OR

Publish specifications of the automated decision system in a peer-reviewed journal. Where access to the published review is restricted, ensure that a plain language summary of the findings is openly available.

Gender-based Analysis Plus

Ensure that the Gender-based Analysis Plus addresses the following issues:

Notice

Plain language notice posted through all service delivery channels in use (Internet, in person, mail or telephone).

Human-in-the-loop for decisions

Decisions may be rendered without direct human involvement.

Explanation

In addition to any applicable legal requirement, ensure that a meaningful explanation is provided to the client with any decision that results in the denial of a benefit or service, or involves a regulatory action. The explanation must inform the client in plain language of:

Explanations must also inform clients of relevant recourse options, where appropriate.

A general description of these elements must also be made available through the Algorithmic Impact Assessment and discoverable via a departmental website.

Training

Documentation on the design and functionality of the system.

IT and business continuity management

None

Approval for the system to operate

None

Other requirements

The Directive on Automated Decision-Making also includes other requirements that must be met for all impact levels.

Link to the Directive on Automated Decision-Making

Contact your institution's ATIP office to discuss the requirement for a Privacy Impact Assessment as per the Directive on Privacy Impact Assessment.

Section 3: Questions and Answers

Section 3.1: Impact Questions and Answers

Reasons for Automation

1. What is motivating your team to introduce automation into this decision-making process? (Check all that apply)
Existing backlog of work or cases
Improve overall quality of decisions
Use innovative approaches

2. What client needs will the system address and how will this system meet them? If possible, describe how client needs have been identified.
This project has two clients: 1) The CAs working on pay cases, 2) public servants who have been impacted by pay issues.

Since October 2023, consultations have taken place with CAs working on the backlog of cases in order to identify opportunities to reduce the backlog of pay issues. As a result of the consultations, we heard that a main barrier to backlog reduction was the time spent manually looking up information. The AI will aid CAs by identifying the correct information they need to make decisions related to case resolution. The AI tool eliminates the need to look up information manually, providing CAs with additional time to complete more complex tasks and address more cases in the backlog.

As a result, this will help improve the number of cases addressed for public servants which supports the Minister of Public Service Procurement Mandate Letter (2021) commitments.

3. Please describe any public benefits the system is expected to have.
Addressing the backlog of pay issues will directly impact both public servants, and members of the public. The need to address issues caused by the Phoenix Pay system is a commitment in the Minister of Public Service Procurement Mandate Letter (2021).

Public Servants will benefit as the tool will allow CAs to move through cases quicker, allowing the compensation community to resolve backlog items for thousands of public servants.

4. How effective will the system likely be in meeting client needs?
Very effective [ Points: +0 ]

5. Please describe any improvements, benefits, or advantages you expect from using an automated system. This could include relevant program indicators and performance targets.
The AI assistant is expected to yield significant improvements by reducing the burden of manual and repetitive tasks in the case resolution process. Manual tasks result in high case touch time which makes it difficult to resolve cases in a timely manner. By automating specific parts of the process, a substantial reduction in case touch time will be seen, contributing to more efficient and streamlined operations.

To date we have seen an average of 22% touch time reduction in treating acting cases in the backlog. As the AI efficiently handles more information and automates repetitive tasks, it frees up CAs to focus on more complex and strategic aspects of their roles, such as decision-making about cases, and ultimately contributes to a more agile and responsive organization.

These improvements align with our commitment to enhance client satisfaction, optimizing our service delivery processes, and meeting mandate letter commitments.

6. Please describe how you will ensure that the system is confined to addressing the client needs identified above.
To ensure the project remains focused on addressing the specific client needs identified, several strategies have been implemented.

Publicly available information has been built for quicker data digestion. This allows the AI to efficiently analyze and process vast amounts of information enabling it to identify and prioritize cases that align with the needs of an account. The client data is evaluated using existing SoPs that are approved by PSPC.

A case selection process is in place where there are clear criteria and guidelines for choosing cases based on their relevance. Currently, the tool uses two case selection criteria for processing synthetic data: 1) case type (acting) and 2) the date the case was created, which identifies acting cases that are over 365 days old. This selection approach ensures that the AI is directed towards cases that require its specialized assistance and fall within the backlog.

In addition, there are robust user and workload management tools in place. They help monitor and regulate the CAs interactions with cases, ensuring that the system is confined to addressing the client needs identified above.

Transparency is built into the tool. For each recommendation presented to the CAs, the tool identifies steps that were taken to arrive at this recommendation. The CAs have full oversight on this information including the ability to review the analysis, check for errors, and make amendments as required.

By incorporating these measures, the solution will effectively focus on and address the specific use cases that have been chosen, ultimately contributing to a more streamlined and responsive case resolution process.

7. Please describe any trade-offs between client interests and program objectives that you have considered during the design of the project.
We thoroughly analyzed the trade-offs between CAs’ interests and program objectives to ensure a balanced approach. We acknowledge that speed, accuracy, and quality are of utmost importance and are committed to optimizing these three factors.

To achieve this, we will empower CAs with decision-making opportunities while allowing them to retain necessary control over specific processes to effectively achieve case closure.

Our goal is to expedite case resolution, enhance service standards, and ensure fair compensation, all while upholding high standards of transparency, accuracy, and quality. By navigating these trade-offs, our intention is to develop a system that serves both dedicated CAs and public servants impacted with pay issues.

8. Have alternative non-automated processes been considered?
Yes [ Points: +0 ]

9. If non-automated processes were considered, why was automation identified as the preferred option?
AI was identified as the preferred option due to an increase in intake and growing backlog. The current manual processes and interventions are not keeping pace with the volume of cases and as such, the backlog continues to grow. AI allows room for efficiencies to be gained in areas where large volumes of information are analyzed prior to taking action on a given case.

10. What would be the consequence of not deploying the system?
Service costs are too high
Service quality is not as high
Service delivery cannot achieve performance targets

Risk Profile

11. Is the project within an area of intense public scrutiny (e.g. because of privacy concerns) and/or frequent litigation?
Yes [ Points: +3 ]

12. Are clients in this line of business particularly vulnerable?
Yes [ Points: +3 ]

13. Are stakes of the decisions very high?
No [ Points: +0 ]

14. Will this project have major impacts on staff, either in terms of their numbers or their roles?
No [ Points: +0 ]

15. Will the use of the system create or exacerbate barriers for persons with disabilities?
No [ Points: +0 ]

Project Authority

16. Will you require new policy authority for this project?
Yes [ Points: +2 ]

About the Algorithm

17. The algorithm used will be a (trade) secret
No [ Points: +0 ]

18. The algorithmic process will be difficult to interpret or to explain
No [ Points: +0 ]

About the Decision

19. Please describe the decision(s) that will be automated.
There are no decisions that will be automated. The CAs will review the information generated by the AI, conduct their own assessment, and make a decision on each case.

Impact Assessment

20. Which of the following best describes the type of automation you are planning?
Partial automation (the system will contribute to administrative decision-making by supporting an officer through assessments, recommendations, intermediate decisions, or other outputs) [ Points: +2 ]

21. Please describe the role of the system in the decision-making process.
The type of AI being planned aligns with partial automation, where the system will actively contribute to administrative decision-making by supporting CAs through assessments, offering recommendations, generating intermediate decisions, and providing other relevant outputs without ever taking action itself.

The AI uses programming algorithms, which are sets of rules and instructions it follows to analyze account data. Think of these algorithms as a recipe the AI follows, where each step needs to be done in a certain order to achieve the correct result. These algorithms help the AI sort through all the information.

To ensure accuracy, the AI relies on knowledge sets, which are collections of information, such as job aids, policies, and directives. These knowledge sets act like a reference book, helping the AI check if the data matches what's expected or if something is out of place. By comparing the information details against these knowledge sets, the AI can spot any mismatches or anomalies and ensure everything lines up correctly. These knowledge sets are existing SoPs that are approved by PSPC and the Government of Canada.

The AI plays an important role in enhancing the decision-making process, however the decision making is left in hands of the CAs.

The algorithms and data analysis capabilities allow it to provide CAs information that is easily accessible which empowers them to make more informed and strategic decisions.

Essentially, the AI gathers data for the CAs, contributing to more efficient and effective outcomes. This collaborative approach ensures that decisions are well informed, data-driven, and aligned with the best steps to take in order to resolve a case.

22. Will the system be making decisions or assessments that require judgement or discretion?
Yes [ Points: +4 ]

23. Please describe the criteria used to evaluate client data and the operations applied to process it.
The AI system works with detailed instructions and guidelines, such as SoPs, job aids, directives, and policies, which outline how to correctly close a case.

Here's how it functions:

Identifying Data: The AI starts by looking at account information. This includes basic account details to specific client cases, such as whether someone is being promoted into a higher paid position.

Analysis: With this data, the AI then compares it against the set instructions (SoPs, job aids, directives, and policies) to check for accuracy, completeness, relevance, and whether it meets the required standards.

Recommendations: Based on its analysis, the AI identifies patterns or important details. The advice or recommendations the AI produces are then given to CAs. For example, this might include necessary salary information for the CAs to resolve an issue or steps to ensure a backlog case is properly addressed.

Human Oversight: For each recommendation presented to the CAs, the tool identifies steps that were taken to arrive at this conclusion. The CAs have full oversight on this information including the ability to review the analysis, check for errors, and make amendments as required.

Decision-making: CAs review the AI suggestions, combining the AI's insights with their own knowledge and experience. They make the final decisions on what actions to take, ensuring each decision is informed, consistent, and tailored to individual client backlog case.

This process allows for a blend of advanced AI analysis and human expertise, ensuring that every decision is well-informed and aligns with existing guidelines, while also considering the specific context of each case.

24. Please describe the output produced by the system and any relevant information needed to interpret it in the context of the administrative decision.
The AI crafts a detailed summary for CAs, through its analytical abilities. This analysis involves examining account data and aligning it with the same job aids, procedures and policies that CAs use, ensuring the AI's advice is grounded and operating within existing processes. These knowledge sets are existing SoPs that are approved by PSPC and the Government of Canada.

Summary: The summary from the AI equips CAs with a full picture of the case at hand, assessing what's needed within the scope of the decision to be made. The summary isn't just a collection of data; it's a review that highlights crucial information like salary details and offers specific advice by referencing the relevant guidelines. For instance, if there's a question about when to start union dues for someone who began their job mid-month, the AI taps into its understanding of the Union Dues Directive.

Interpreting the Summary: To effectively use this summary, CAs need a solid grasp of the pay knowledge base, including compensation policies, relevant laws, and SoPs. This background allows CAs to evaluate the AI's recommendations accurately within the context of each unique administrative scenario.

The Collaborative Approach: By merging the AI's data driven insights with the CAs' expertise, the system ensures the advice it gives is not just informative but also directly applicable. This partnership enables CAs to make decisions that are not only well-informed but also tailored to the specific requirements of each case, thereby elevating the quality and efficiency of their service.

25. Will the system perform an assessment or other operation that would not otherwise be completed by a human?
No [ Points: +0 ]

26. Is the system used by a different part of the organization than the ones who developed it?
No [ Points: +0 ]

27. Are the impacts resulting from the decision reversible?
Likely reversible [ Points: +2 ]

28. How long will impacts from the decision last?
Impacts are most likely to be brief [ Points: +1 ]

29. Please describe why the impacts resulting from the decision are as per selected option above.
Currently, if an incorrect transaction is processed, an amendment can be made to rectify the error. The CAs have full oversight and total control of the systems in order to review the AI analysis, check for errors, and make amendments as required.

30. The impacts that the decision will have on the rights or freedoms of individuals will likely be:
Little to no impact [ Points: +1 ]

31. Please describe why the impacts resulting from the decision are as per selected option above.
No impacts to freedoms of individuals in enhancing internal pay processing capacity using AI and synthetic data.

32. The impacts that the decision will have on the equality, dignity, privacy, and autonomy of individuals will likely be:
Little to no impact [ Points: +1 ]

33. Please describe why the impacts resulting from the decision are as per selected option above.
No impacts to equality, dignity, privacy and autonomy of individuals in enhancing internal pay processing capacity using AI and synthetic data.

34. The impacts that the decision will have on the health and well-being of individuals will likely be:
Moderate impact [ Points: +2 ]

35. Please describe why the impacts resulting from the decision are as per selected option above.
The impacts resulting from the decision will be moderate due to several key factors. Implementation will facilitate the streamlining of repetitive tasks within operations. By automating these tasks, one can significantly reduce the burden of manual and repetitive work on CAs. This, in turn, will contribute to improved job satisfaction and reduced stress levels among employees, thereby positively affecting their overall well-being. This also impacts the overall retention of CAs.The focus on eliminating manual and repetitive tasks, such as retrieving information from one screen and pasting it into another, is a proactive step toward creating an engaging and fulfilling work environment.

CAs will have the opportunity to redirect their efforts towards more meaningful and higher value tasks, which can lead to increased job satisfaction and a sense of accomplishment.

By considering these factors, it is anticipated that the decision's impact on the health and well-being of individuals will fall within the moderate range, bringing improvement to both the work environment and the quality of worklife for CAs.

36. The impacts that the decision will have on the economic interests of individuals will likely be:
Little to no impact [ Points: +1 ]

37. Please describe why the impacts resulting from the decision are as per selected option above.
The use of AI to enhance pay administration using synthetic data may lead to decisions to scale the solution using real data that could improve operational efficiency resulting in cost savings. Additionally improved services could mean greater resolution of cases that could positively impact clients.

38. The impacts that the decision will have on the ongoing sustainability of an environmental ecosystem, will likely be:
Little to no impact [ Points: +1 ]

39. Please describe why the impacts resulting from the decision are as per selected option above.
N/A

About the Data - A. Data Source

40. Will the Automated Decision System use personal information as input data?
Yes [ Points: +4 ]

41. Have you verified that the use of personal information is limited to only what is directly related to delivering a program or service?
Yes [ Points: +0 ]

42. Is the personal information of individuals being used in a decision-making process that directly affects those individuals?
Yes [ Points: +2 ]

43. Have you verified if the system is using personal information in a way that is consistent with:
(a) the current Personal Information Banks (PIBs) and Privacy Impact Assessments (PIAs) of your programs or
(b) planned or implemented modifications to the PIBs or PIAs that take new uses and processes into account?
No [ Points: +1 ]

44. What is the highest security classification of the input data used by the system? (Select one)
Protected B / Protected C [ Points: +3 ]

45. Who controls the data?
Federal government [ Points: +1 ]

46. Will the system use data from multiple different sources?
Yes [ Points: +4 ]

47. Will the system require input data from an Internet- or telephony-connected device? (e.g. Internet of Things, sensor)
Yes [ Points: +4 ]

48. Will the system interface with other IT systems?
Yes [ Points: +4 ]

49. Who collected the data used for training the system?
Your institution [ Points: +1 ]

50. Who collected the input data used by the system?
Your institution [ Points: +1 ]

51. Please describe the input data collected and used by the system, its source, and method of collection.
The input data collected and utilized by the system primarily consists of synthetic account data. This data is created to resemble datasets from the Case Management Tool (CMT) and Phoenix. The data is not collected from any live source.

About the Data - B. Type of Data

52. Will the system require the analysis of unstructured data to render a recommendation or a decision?
Yes [ Points: 0 ]

53. What types of unstructured data? (Check all that apply)
Audio and text files [ Points: +2 ]
Images and videos [ Points: +4 ]

Section 3.2: Mitigation Questions and Answers

Consultations

1. Internal Stakeholders (federal institutions, including the federal public service)
Yes [ Points: +1 ]

2. Which Internal Stakeholders will you be engaging?
Legal Services
Communications services
TBS Office of the Chief Human Resources Officer
TBS Office of the Chief Information Officer
Client Experience / Client Relationship Management
Access to Infromation and Privacy Office
Other (describe)

3. Please describe
HCM - Pay Administration Branch
PSPC Privacy
Shared Services Canada
Other government departments testing AI

4. External Stakeholders (groups in other sectors or jurisdictions)
Yes [ Points: +1 ]

5. Which External Stakeholders will you be engaging?
Bargaining Agents

De-Risking and Mitigation Measures - Data Quality

6. Will you have documented processes in place to test datasets against biases and other unexpected outcomes? This could include experience in applying frameworks, methods, guidelines or other assessment tools.
Yes [ Points: +2 ]

7. Will you be making this information publicly available?
Yes [ Points: +1 ]

8. Will you be developing a process to document how data quality issues were resolved during the design process?
Yes [ Points: +1 ]

9. Will you be making this information publicly available?
No [ Points: +0 ]

10. Will you undertake a Gender Based Analysis Plus of the data?
Yes [ Points: +1 ]

11. Will you be making this information publicly available?
Yes [ Points: +1 ]

12. Have you assigned accountability in your institution for the design, development, maintenance, and improvement of the system?
Yes [ Points: +2 ]

13. Will you have a documented process to manage the risk that outdated or unreliable data is used to make an automated decision?
Yes [ Points: +2 ]

14. Will you be making this information publicly available?
No [ Points: +0 ]

15. Will the data used for this system be posted on the Open Government Portal?
No [ Points: +0 ]

De-Risking and Mitigation Measures - Procedural Fairness

16. Will the audit trail identify the authority or delegated authority identified in legislation?
Yes [ Points: +1 ]

17. Will the system provide an audit trail that records all the recommendations or decisions made by the system?
Yes [ Points: +2 ]

18. Will all key decision points be identifiable in the audit trail?
Yes [ Points: +2 ]

19. Will all key decision points within the automated system's logic be linked to the relevant legislation, policy or procedure?
No [ Points: +0 ]

20. Will you maintain a log detailing all of the changes made to the model and the system?
Yes [ Points: +2 ]

21. Will the audit trail clearly set out all decision points made by the system?
Yes [ Points: +1 ]

22. Could the audit trail generated by the system be used to help generate a notification of the decision (including a statement of reasons or other notification) where required?
Yes [ Points: +1 ]

23. Will the audit trail identify precisely which version of the system was used for each decision it supports?
Yes [ Points: +2 ]

24. Will the audit trail show who the authorized decision-maker is?
Yes [ Points: +1 ]

25. Will the system be able to produce reasons for its decisions or recommendations when required?
Yes [ Points: +2 ]

26. Will there be a process in place to grant, monitor, and revoke access permission to the system?
Yes [ Points: +1 ]

27. Will there be a mechanism to capture feedback by users of the system?
Yes [ Points: +1 ]

28. Will there be a recourse process planned or established for clients that wish to challenge the decision?
No [ Points: +0 ]

29. Will the system enable human override of system decisions?
Yes [ Points: +2 ]

30. Will there be a process in place to log the instances when overrides were performed?
Yes [ Points: +1 ]

31. Will the audit trail include change control processes to record modifications to the system's operation or performance?
Yes [ Points: +2 ]

32. Will you be preparing a concept case to the Government of Canada Enterprise Architecture Review Board?
Yes [ Points: +1 ]

De-Risking and Mitigation Measures - Privacy

33. If your system uses or creates personal information, will you undertake or have you undertaken a Privacy Impact Assessment, or updated an existing one?
Yes [ Points: +1 ]

34. Will you design and build security and privacy into your systems from the concept stage of the project?
Yes [ Points: +1 ]

35. Will information be used within a closed system (i.e. no connections to the Internet, Intranet or any other system)?
No [ Points: +0 ]

36. If the sharing of personal information is involved, has an agreement or arrangement with appropriate safeguards been established?
Yes [ Points: +1 ]

37. Will you de-identify any personal information used or created by the system at any point in the lifecycle?
No [ Points: +0 ]

Related Links

Report a problem or mistake on this page
Date modified: