Position: Regular, Full-Time
Location: 200 Granville St, Vancouver, BC. (NOTE: A combination of home-office (remote) and office-based work is expected.)
NCAS is looking for an experienced psychometrician to support its current and future assessments. This Vancouver-based position provides psychometric and data flow management expertise to the ongoing refinement and evolution of NCAS’ multi-pronged, multi-dimensional assessments that involve varied assessment tools such as Objective Structured Clinical Examinations (OSCE), oral interviews, and selected-response measures.
NCAS is disrupting the assessment space with its innovative approach to assessment. This position offers an exciting opportunity for an exceptional individual interested in developing and implementing new competency assessment models, strengthening reliability and validity approaches, and applying foundational psychometric and statistical research to improve the quality of health care across Canada.
Who we are
The Nursing Community Assessment Service (NCAS) is an initiative of the BC College of Nurses and Midwives. BCCNM is empowered under the Health Professions Act to regulate the practice of all licensed practical nurses, nurse practitioners, registered midwives, registered nurses, and registered psychiatric nurses in British Columbia. Regulation helps to protect the public by ensuring that professional care or service received by the public is competent, ethical, and meets the standards that society views as acceptable. NCAS operates as a unique and independent unit within BCCNM.
BCCNM launched NCAS in January 2017 to assess the competencies of those seeking licensure to enter nursing practice in BC and other parts of Canada. Our high-stakes assessment is comprised of a computer-based test that is administered at testing sites all over the world, as well as a simulation-lab and oral assessment, which are delivered either in Vancouver or Halifax. The NCAS assessment represents a consistent, rigorous, and defensible approach for determining the extent to which individuals possess the competencies required to enter practice safely.
As a member of our small, creative, and talented team, you can expect to be treated in a respectful and professional manner. While your workload will be busy, and the environment is dynamic, the college supports staff in achieving a healthy work-life balance.
Our office rests on the unceded and incomparably beautiful territories of the Musqueam, Squamish and Tsleil-Waututh nations. Surrounded by ocean and mountain views, steps from the SkyTrain, and with running paths and kayaking routes at your doorstep, BCCNM welcomes you to the heart of Vancouver’s vibrant downtown core.
What you'll be doing
Reporting to the Director of Assessment, Innovation, and Operations, the Manager of Assessments and Psychometrics divides their time between three principal accountabilities: providing psychometric and data management expertise to support the delivery of existing assessments; leading the development of new assessment tools; and, creating and/or evolving the data flow pipelines to optimize data integrity and organization. The manager establishes the scoring methodologies and design of assessment instruments for the production of results and analysis of item and test functioning. The Manager provides oversight to assessment development staff while also playing a critical role in designing and ensuring the quality of data management pipelines.
- In consultation with the Director of Assessment, Innovation, and Operations (Dir, AIO), manage the development, implementation and assessment of the tools and processes for the assessment of health care competencies.
- Lead the development, refinement, and analysis of performance-based and selected-response instruments, including pilot and field testing, standard setting, and validation activities needed for the assessment of health care competencies.
- Provide oversight to assessment development staff including an Exam Developer and Junior Psychometrician.
- Create and/or refine, manage, and troubleshoot data flow pipelines (in R and SQL) that enable assessment scoring and reporting.
- Generate performance reports, with associated quality checks (include importing data via Scantron, Excel, XML, and SQL in R)
- Identify risks and offer recommendations as appropriate
- Interface with multiple stakeholders and technology providers to ensure accurate and seamless data flows.
- Collaborate with Director, AIO to understand the purpose of each new assessment in order to define required content and outcomes.
- Create and implement quality-control procedures for assessment delivery, including rater scoring.
- Develop, implement, and evaluate standard-setting and product-improvement processes, including the facilitation of discussions with subject matter experts and stakeholders.
- Provide regular psychometric consulting, problem-solving and innovation in support of assessment operations and new assessment builds, including item analysis, forms construction and equating, reliability and validity analyses, etc.
- Work closely with the Exam Developer to create and launch training for content writers and certification tools for raters, and to conduct analysis in support of content development.
- Draft public and technical reports, as well as documentation on the validity and reliability of the assessments.
- Identify and carrying out independent research in support of assessment activities.
- Present research findings, representing the organization and sharing innovations at professional conferences and/or through publication of research articles
- Foster and maintain an organizational culture that promotes mutual respect, teamwork, and service excellence.
Education & skills
- Minimum Masters (Ph.D. preferred) in Measurement and Evaluation, Data Science, Psychology, Statistics, or educational equivalent in a closely related field.
- At least five years of experience in an applied measurement or data science environment.
- At least five years in designing and conducting statistical and/or psychometric analyses.
- Expertise in a variety of psychometric models (e.g., classical test theory, generalizability theory, factor analysis, item response theory, differential item functioning, dimensionality analysis, interrater reliability analysis).
- Expertise in assessment, statistical, and database software (RStudio and R Markdown (required), and other programming languages an asset; Microsoft SQL Server Management Studio, XML file processing, SQLite, Remark OMR, Qualtrics, Excel) and the ability to write user-defined R functions.
- Familiarity with professional assessment standards (e.g., INACSL, AERA/APA/NCME Standards)
- Experience managing testing programs with small volumes is preferred
- Experience managing staff is preferred
- Strong project management background, preferably in technical education, certification, or credentialing services. Aptitude to quickly come up to speed on new concepts.
- Exceptional research and writing skills.
- Demonstrated initiative-taker, with strong organizational skills and the ability to meet deadlines.
- Demonstrated ability to prepare analytic plans, monitor progress, and track timelines.
- Highly collaborative, with strong interpersonal skills to work with various stakeholders in the assessment development and reporting process.
- Ability to work independently and collaboratively within a team.
- Demonstrated ability to prepare effective visual presentations of complex data.
- Bilingualism in French and English is an asset
- Ability to thrive and take initiative in an environment of ambiguity and change.
- Displays a passion for doing meaningful work
- Utilizes available networks and resources to stay current in the field of competency assessment
- Demonstrated understanding of diversity, inclusion, and cultural humility as they apply to nursing and midwifery practice and health care is an asset.