The Image Problem in Data
In a pension administration system implementation or upgrade project, image transfers from the legacy system are often a key component to ensuring that historical “proof” data doesn’t get lost. Sometimes, the image data is migrated during the implementation of the new system and sometimes it’s a stand-alone project. In either case, the data that’s being handled requires the highest security as it is essentially PII (Personally Identifiable Information) in physical form.
The Solution
Data migration projects should begin by studying the target system’s behavior and requirements and the source system’s extensive data profiling to understand critical components, usual business practices, and data structure, including corner case management. Data profiling, cleansing, and migration via data mapping sessions and data issue resolution should also be reviewed with your integration partner. Information to be gathered includes the total file storage needed. i.e., total disk space used for both data and images.
The conversion or transfer of images should involve three distinct parts that include physical image files, metadata, file path, folder, and file name as part of the metadata to marry the file. The steps involved in image conversion may vary depending on the requirements of the project and the landing place of the image data. Here’s a general overview of the process:
Step 1: Acquiring the Images
The first step is to acquire the images that need to be converted. Most of the time the data is already in its final format. That is, it has already been scanned and stored in a location accessible to the users. If this is not the case, there is an additional layer that may involve scanning physical documents and storing them in a central location. The converted image data can then be stored in a file system or transmitted over a network, depending on the project requirements. The file can be saved to a local disk, uploaded to a cloud storage service, or integrated into a database.
Typically, format conversion is done in the source system. If there is one agreed-upon format, such as PDF, then all the images will be converted from the original format to the desired PDF. In other cases, your migration partner should perform the conversion as a part of the process. Depending on the project requirements and the target format, compression techniques may be applied to reduce the file size to store and access the data more efficiently.
The physical images are then delivered via an agreed-upon protocol. They can be sent via Secure File Transfer Protocol (SFTP) or placed on a dedicated drive or folder. Some clients, to ensure the security of the data, deliver the data in person on a hard drive. Your migration partner should understand the sensitivity and privacy that is being entrusted with the possession of this data and ensures that the decision on the delivery of the data is something with which both our client and the system vendor are comfortable.
It’s worth noting that image conversion projects can be more complex depending on the specific requirements. Look for a partner with extensive experience with staging annotations, which is a tedious and difficult process. The ultimate goal of image conversion is to get the data to where it needs to go and for the user to be able to seamlessly use the data in their day-to-day processes within the applicable applications.
Step 2: Metadata
Metadata conversion in a data conversion project involves converting the descriptive information and attributes associated with the files, such as file names, file sizes, creation dates, author information, keywords, tags, and other relevant details.
The process of metadata conversion typically involves the following steps:
- The metadata is extracted from the source files or systems. This can be done through various methods depending on the file types and systems involved.
- Once the metadata is extracted, it is mapped and transformed to fit the desired target system. This step involves following a mapping schema that identifies how the source metadata attributes correspond to the attributes in the target format or system.
- During the conversion process, it is crucial to validate and clean the metadata to ensure its accuracy, consistency, and integrity. This may involve checking for errors, duplicate entries, missing values, or incompatible formats. Data validation rules can be applied to identify and rectify any inconsistencies or errors in the metadata.
In some cases, additional cleansing of metadata may be required, like adding new attributes or deriving values based on existing metadata. Once the metadata is converted and validated, it can be loaded into the target system and integrated with the converted files.
Step 3: Reconciliation of Metadata to Physical
The reconciliation of metadata to physical files involves aligning the metadata information with the corresponding physical files to ensure consistency and accuracy. We identify the metadata attributes that link or describe the physical files. The actual reconciliation process can be tedious as it involves opening the files and cross-referencing the metadata attributes with the file itself. If any discrepancies are found during the comparison, corrective actions can be taken to resolve the inconsistencies. It is key to maintain proper documentation of the reconciliation process, including any modifications or corrections made to the metadata. This documentation serves as a reference for future audits or inquiries regarding the accuracy and integrity of the metadata and file relationships.
Finding a Solution That Works
This process of image conversion is a proven method that treats every step of the migration process as a checkpoint. It promotes collaboration between all parties as it brings stakeholders from the client (and the vendor, if necessary) and the migration partner to the virtual table to review the results of each step. Input from the client is encouraged and included in the decision-making. The process of reconciliation of metadata and the physical file is crucial to ensure the consistency and integrity of the migrated data. By aligning all aspects of the images and data, documenting the process used to correct discrepancies, and extensive quality assurance tests, the client can be assured that the overall results of an integrated pension administration system implementation project are one of reliable and useful converted data and images. If your migration partner can’t deliver on the processes outlined above, you should be looking for one that will.
