Technology Vetting

What does this special interest workgroup do?

This workgroup was developed to ensure that every technology listed in our Actionable Patient Safety Solutions (APSS) has been vetted by an unbiased group before being published. This workgroup has been operating since 2018 building a rubric for vetting technologies and will have used the rubric for the first time in 2019 for APSS published in March of 2020.

Who developed the rubric?

The Technology Vetting Workgroup developed the rubric and sought out feedback from workgroup members who serve on other workgroups.

What does the scoring rubric look like?

The goal for the rubric is to provide a consistent scoring guide. This will be used to evaluate the elements to be acknowledged by PSMF beginning with the March 2020 publication of APSS.

The technology vetting workgroup will use the Analytic Rubric approach and will be asked to “score” all submissions in the same format. The workgroup recognizes that supporting the PSMF in this effort will also include an adaptive learning approach, therefore, the workgroup will revisit the rubric and its effectiveness impact to inform additional items or phases annually.

The following process (Figure 1) has been adopted by the workgroup regarding how technologies are vetted. The appeals process in Figure 1.0 can be started by emailing clinical@patientsafetymovement.org. Individuals who wish to appeal may appeal a total of one time per submission.

Figure 1: Protocol for vetting technology

The criteria are for an equitable, utilization & measurable impact score with the purpose to support the safe delivery of care and clinician burden reduction. Table 1 represents the rubric that will be used by the workgroup. The transparent approach and available score is meant for consideration in adopting a product/technology within the care coordination process. It is understood that the product/technology (not necessarily the brand) meeting the “excellent” therefore superior technology and product specifications will be adapted into the APSS for 2020.

In addition, future development for the PSM Tech Vetting rubric will consider the social-technical complex adaptive model, interoperability, security and end-user impact. The goal is to build upon the current recommended model in alignment with the PSM mission and goals.

Table 1: Criteria for scoring technologies submitted

Criteria Not meeting – 0 point Adequate- 1 point Good – 2 points Excellent – 3 points
Demonstration for “live” product/ technology implemented and outcomes achieved None listed. Not live for evaluation Prototype device demonstrated in simulated patient conditions – “Proof of concept” Production version demonstrated in human volunteers; results verified by independent measures Production devices used in patients under clinical conditions. Results are shown to improve outcomes, examples provided as to how.
Description of product/technology Not listed Provides high-level overview that is vaguely understandable for evaluation.

Does not list the impact on end-user burden.

Does not include benefits outcomes for the patient, that without this product/ technology would be in greater harms way then if adapted into the system of care.

Provides examples that demonstrate the impact for improving the process that is clearly understandable for evaluation.

Must include both intended and unintended consequences of implementation for patient safety.

Provides examples of how their product/ technology has been successfully applied in real-world complex adaptive settings.

Able to provide examples that demonstrate reduction and/or impact to end-user burden.

Able to provide clear benefits outcomes for the patient that without this product/ technology would be in greater harm’s way then if adapted into the system of care.

Provides examples that demonstrate positive/ high impact for improving a process within the coordination of care.

Provides an evaluation for both intended and unintended consequences of implementation with effects on the end-user.

Provides a specific model design that addresses the socio-technical challenges involved in design, development, implementation, use, and evaluation within a complex adaptive healthcare system.

Effects on the end-user and benefits outcomes for the patient.

Able to clearly list cognitive knowledge management support and/or technology-aided decision support that without adoption of this product/technology solution would clearly demonstrate added risk to patient care coordination.

Product/technology has a data model that is semantically and syntactically identified Not listed References data model but not included. Provides data model. Provides data model for process impact on the system of care that it’s intended use will improve patient safety if adopted.

In addition, it provides workflow impact and system flow design and outcome measurement goals.

Feedback on the use of technology None listed.

Not live for feedback evaluation.

Includes hospitals or care environments that the product is “live”.

Implemented at 1-2 sites able to list the vision for the impact of technology adoption.

Includes hospitals or care environments that the product is “live” with outcomes measurements achieved.

Implemented at 3-5 sites, no adoption or end-user feedback included reflecting outcomes achieved.

Includes hospitals or care environments that the product is “live” with outcomes measurements achieved.

Implemented at 3-5 sites and able to include user feedback that indicates workflow impact improvements for patient safety.

Peer-reviewed literature supporting the inclusion of the product/ technology in APSS. The quality of evidence for benefit, efficacy, effectiveness should be explicit. None listed 1-2 publications within the past 3-5 years. 3-5 publications within the past 3-5 years. 6+ publications within the past 3-5 years.

 

 

APSS to which product/technology is applicable Evaluation by submitter does not indicate Submitter does not identify within what workflow or care environment that adoption of product/ technology would be applicable Submitter identifies within what workflow or care environment that adoption of product/ technology would be applicable. Submitter identifies within what workflow or care environment that adoption of product/ technology would be applicable.

This is a scalable solution for more than one use case scenario.

TOTAL SCORE

Scoring Criteria

≥12 points = list category in APSS if not already listed.
CONSIDERATION: If peer-reviewed literature supports the specific product/technology over another then list out the product/tech specifications that make that technology superior.

<11 points = category does not have enough points to be listed in the APSS

Want to join the workgroup?

Join here

Ready to submit your technology for vetting?

Submit here