2. Methodology

This section describes the process of compiling the 250 contract records into the following steps: Identified Constraints, Sample Attributes, Data Sourcing, OCDS Mapping and Consolidation, and Data Cleansing and Normalization.

2.1 Identified Constraints

In the pursuit of compiling the sample data, there are constraints put in place for completing this project as directed by Procurement Assistance Canada (PAC) senior management. PAC also does not create data, nor do they manipulate data in accordance with the PSPC policy and guidelines.

2.2 Sample Attributes

The outlined 250 contract records were chosen from a filtered pool of contracts based on the distribution of five specific attributes: High/Low Dollar Value, Size of Enterprise, Goods/Services, Closed Contracts, and Contracting Type.

2.2.1 High/Low Dollar Value

The pilot data’s dollar values were evenly distributed over four bands:

• less than $25k
• $25k to $499k
• $500k to $999k
• $1 Million and above

2.2.2 Size of Enterprise

The sizes of the enterprises were also evenly distributed among the following categories:

• self-employed
• 1 to 4 employees
• 5 to 9 employees
• 10 to 19 employees
• 20 to 49 employees
• 50 to 99 employees
• 100 to 499 employees
• 500 to 999 employees
• greater than 1000 employees

2.2.3 Goods/Services

PSPC has chosen the types of commodities to be represented at a two-to-one ratio of Services to Goods.

2.2.4 Closed Contracts

Only closed contracts were used for the sample records. A closed contract was defined as the tenders having been awarded no later than 2019.

2.2.5 Contracting Type

With the inclusion of varied Contracting Types, the nature of having subsequent callups on Standing Offers or Supply Arrangements that can span multiple years has been limited to any and all contract records that have been published in 2020 and older.

2.3 Data Sourcing

The OCDS has a standard set of data fields to be populated by the Government of Canada’s data collected as part of the procurement process. Most of the data used in the 250 contract records, save the planning and implementation data, is actively published on Buyandsell.gc.ca (BAS) and is easily accessible through the data repository on Open.Canada.ca. The data is sourced primarily from BAS and our procurement systems.

PSPC has identified that the systems did not have all the fields needed to fulfil the OCDS. As mentioned in the constraints section above, for the purposes of Milestone 2.3, PSPC will not create data fields to fulfil the OCDS. The OCDS fields that we do not have available from our data sources are shown in Appendix A. In some cases, the current Government of Canada’s procurement data is not captured as “Structured” data in the procurement systems. This exercise identified the need to standardize the way data elements are captured across PSPC procurement systems. Future OCDS efforts will greatly benefit from such an initiative.

PSPC has also sourced data from the OPI in the sectors and regions directly.

2.4 OCDS Mapping and Consolidation

This project is an exercise in best efforts and no data was created for the completion of the 250 sample OCDS records. PSPC mapped the procurement data to the Open Contracting Standard by exploring the data collected in the systems and documents and contacting the OPI for the data. The data was mapped on best effort basis, if the data was not available, those fields were not included for publishing and justification was provided. When possible, PSPC took the steps to complete the field sections that were highly recommended in OCDS. However, due to the discrepancy and unavailability between the OCDS data fields, quality and/or imprecise purpose of data, PSPC has omitted some highly recommended fields from the data standard after taking extensive efforts to acquire the data. The highly recommended fields omitted include the following:

• project Title
• project ID
• planning Documents (Includes ID, title, type, description…)
• planning Milestones (Includes ID, title, type, description…)

The disparate nature of the data across government agencies has presented issues in trying to unify the data. The main identifier for data is the Requisition Number which is comprised of the Ordering Office, the year of procurement and a serial number. The Requisition Number is persistent throughout the procurement phases with modifiers added as the process matures as seen the following table.

When consolidating the published requisition data in Buyandsell.gc.ca with the purchase order and invoice data in the PSPC Financial system from Financial Accounting (FI postings), we found that the linkage was not fully standardized. The Purchase Order Number (PO) in the Financial System, in most cases, uses an internal identification scheme and, in some cases, the PO uses aspects of the Requisition Number in varying ways. An example of the most common Financial system’s purchase order number format variation can be seen in the following table.

This record does not provide a direct link to a specific awarded contract as it does not include the “001” portion of the award number as seen in the second column. The link was then further strengthened with the financial coding data found in the procurement requisition. The OCDS aims at an itemized invoice level of reporting. In order to meet the standard, we had to further link the PO data to the invoice data.

2.5 Data Cleansing and Normalization

Data cleansing involves the correction of values that have been entered in error and is only done when there is certainty of input error (e.g., Canadian Provinces, Cities, and letters in number fields). Normalization is the conforming of data to a set uniform structure or syntax. The data required cleansing and normalization in order to conform to OCDS.

In general, the sample records have been normalized and/or cleaned for all fields related to:

• regions
• postal codes
• emails
• name of contacts
• phone numbers
• currency codes and values
• boolean (True and False) values
• unit names
• free text fields:
  • bidder name