Center for Applied Linguistics
Manager, Writing Test Development
Center for Applied Linguistics, Washington, District Of Columbia, United States, 20001
Manager, Writing Test Development
The Center for Applied Linguistics is a nonprofit organization promoting access, equity, and mutual understanding for linguistically and culturally diverse people around the world. The mission of the Center for Applied Linguistics (CAL) is to promote language learning and cultural understanding by serving as a trusted source for research, resources, and policy analysis. Through its work, CAL seeks solutions to issues involving language and culture as they relate to access and equity in education and society around the globe. CAL is located in Washington, D.C. For this position, the employee may choose to work in their preferred combination of in person at the D.C. office in a hoteling capacity, or remotely (up to 100% remote). CAL does not compensate or reimburse candidates for relocation expenses. This position is not open to candidates residing in New York, New Jersey, Pennsylvania, Texas, California, or the U.S. Territories. Job Description
The Manager, Writing Test Development, is responsible for overseeing the development of Writing test items and ancillary materials for the WIDA suite of assessments, which are large-scale English language proficiency assessments for multilingual learners in grades 1-12. This position involves reviewing, revising, and approving test items (linguistic content, graphics, scripts, audio files) that align to the WIDA English Language Development Standards Framework, with a focus on the language domain of Writing. The Manager, Writing Test Development, oversees the test development process from start to finish. This position develops test refreshment plans, timelines, and procedures; provides comprehensive feedback to test developers; and signs off on test items content at critical points in the development process. This position also collaborates with the Associate Manager, Rater Training, to oversee the creation of scoring materials for the Writing test. Essential Duties: Manage test development staff who write, review, revise, and select Writing test items based on item specifications, test maps, and blueprints Evaluate test items for appropriate task setup and design, linguistic complexity, grade-level appropriateness, content correctness, universal accessibility, appropriate scaffolding, fairness, sensitivity, and adherence to style guides Give clear, constructive, and timely feedback to test developers on test item text and graphics using test item specifications Sign off on test item content at the end of each phase of the item development cycle Work in close collaboration with managers and directors across functional teams and partner organizations to coordinate efforts Collaborate on the production of project plans and timelines for test item development in coordination with client and other vendors and oversee their implementation Oversee the development and revision of item specifications and sign off on revisions Oversee the creation of training materials to prepare item writers and reviewers Ensure test development staff provide content, administrative, and technical support for item writers and reviewers Lead meetings during which stakeholders review the content, fairness, and grade-appropriateness of test items and ensure stakeholder feedback on test items is resolved Work with test development staff and audio recording studio to cast actors and produce recordings for Writing test items, overseeing recording sessions and quality monitoring Write, review, and edit written reports, procedures, and foundational documents for internal and external stakeholders Update and maintain style guides and item development checklists Ensure test development staff follow quality control and test security procedures Pilot Testing and Field Testing: Conduct annual pilot testing of Writing tasks, including obtaining IRB approval and overseeing recruitment and logistics Coordinate the analysis of pilot test data and report on findings Coordinate, conduct, and sign off on quality checks of field test QTI packages and ancillary materials Review and update field test functional rules Oversee the qualitative analysis of Writing field test responses and report findings to internal teams and external stakeholders Rater Training and Scoring: Collaborate on research and development work for scoring materials related to the Writing test, including rubrics, rater training presentations and courses, and other scoring guidance Select anchor papers and training papers Review and approve score justifications, as well as qualifying, recalibration, and validity sets Synthesize information from the scoring of Writing tasks for use in future development Operational Test and Database Management: Review test maps, test forms, ancillary documents, test practice items, and sample items for the public Write, review, and maintain test and item design plans Write, review, and maintain the Writing Test Construct Framework document Ensure accuracy of data in test item databases Supervision: Set functional team goals and expectations for Writing test development Monitor progress and performance of staff completing Writing test development tasks Mentor test development staff in their job duties and professional development Fulfill administrative responsibilities related to hiring/retention and performance reviews Other duties as assigned. Qualifications
Education and Experience: A Master's degree from an accredited college or university, in a relevant field, with at least two years of relevant experience. Expertise in the written English language development of K-12 multilingual learners, or in the development of written English language proficiency assessments for K-12 multilingual learners. The ideal candidate for this role has: Deep familiarity with the written language that K-12 multilingual learners must be able to produce to be successful in U.S. school settings. Experience teaching K-12 multilingual learners in U.S. school contexts. Experience developing constructed response test items for K-12 students, and preferably for English language proficiency assessments. Experience supervising the development of test items in a high-stakes context. Experience managing complex assessment projects. Knowledge, Skills, and Abilities: Must be able to apply their understanding of multilingual learners' English language development in writing, and of K-12 education standards and practices to the development of language proficiency test items. Coordinate remote staff across multiple tasks to meet deadlines and quality control expectations. Provide constructive, clear feedback that both improves test items and builds staff understanding and skills. Understand complex test designs with multiple interconnected parts. Work independently with minimal support/supervision. Prioritize tasks appropriately, meet deadlines, and work well within a collaborative, full remote team. Attend closely to details without losing sight of the bigger picture. Demonstrate flexibility in responding to feedback and evolving project priorities. Show leadership to attain team goals. Bring both analytical and creative approaches to problem solving. The ideal candidate for this role has: Exemplary writing, editing, and proofreading skills in English. Excellent organization, planning, and time management skills. Strong oral and interpersonal communication skills. Proficiency in a language other than English. At least intermediate proficiency with Microsoft Office and Smartsheet, or other work management platforms. Travel Requirements: This position requires occasional travel (1-3 times per year). Additional Information
Compensation: Minimum Salary: $75,000; Benefits eligible Full-time Physical Demands: The physical demands described here are representative of those that must be met by an employee to successfully perform the essential functions of this job. While performing the duties of this job, the employee is regularly required to effectively communicate. The employee may be required to sit or stand in a stationary position for 50% of the time. This position constantly operates a computer and other office productivity equipment. CAL adheres to ADA compliance and guidelines as a common practice. Additional Information: This job description summarizes the main duties of the job. It neither prescribes nor restricts
The Center for Applied Linguistics is a nonprofit organization promoting access, equity, and mutual understanding for linguistically and culturally diverse people around the world. The mission of the Center for Applied Linguistics (CAL) is to promote language learning and cultural understanding by serving as a trusted source for research, resources, and policy analysis. Through its work, CAL seeks solutions to issues involving language and culture as they relate to access and equity in education and society around the globe. CAL is located in Washington, D.C. For this position, the employee may choose to work in their preferred combination of in person at the D.C. office in a hoteling capacity, or remotely (up to 100% remote). CAL does not compensate or reimburse candidates for relocation expenses. This position is not open to candidates residing in New York, New Jersey, Pennsylvania, Texas, California, or the U.S. Territories. Job Description
The Manager, Writing Test Development, is responsible for overseeing the development of Writing test items and ancillary materials for the WIDA suite of assessments, which are large-scale English language proficiency assessments for multilingual learners in grades 1-12. This position involves reviewing, revising, and approving test items (linguistic content, graphics, scripts, audio files) that align to the WIDA English Language Development Standards Framework, with a focus on the language domain of Writing. The Manager, Writing Test Development, oversees the test development process from start to finish. This position develops test refreshment plans, timelines, and procedures; provides comprehensive feedback to test developers; and signs off on test items content at critical points in the development process. This position also collaborates with the Associate Manager, Rater Training, to oversee the creation of scoring materials for the Writing test. Essential Duties: Manage test development staff who write, review, revise, and select Writing test items based on item specifications, test maps, and blueprints Evaluate test items for appropriate task setup and design, linguistic complexity, grade-level appropriateness, content correctness, universal accessibility, appropriate scaffolding, fairness, sensitivity, and adherence to style guides Give clear, constructive, and timely feedback to test developers on test item text and graphics using test item specifications Sign off on test item content at the end of each phase of the item development cycle Work in close collaboration with managers and directors across functional teams and partner organizations to coordinate efforts Collaborate on the production of project plans and timelines for test item development in coordination with client and other vendors and oversee their implementation Oversee the development and revision of item specifications and sign off on revisions Oversee the creation of training materials to prepare item writers and reviewers Ensure test development staff provide content, administrative, and technical support for item writers and reviewers Lead meetings during which stakeholders review the content, fairness, and grade-appropriateness of test items and ensure stakeholder feedback on test items is resolved Work with test development staff and audio recording studio to cast actors and produce recordings for Writing test items, overseeing recording sessions and quality monitoring Write, review, and edit written reports, procedures, and foundational documents for internal and external stakeholders Update and maintain style guides and item development checklists Ensure test development staff follow quality control and test security procedures Pilot Testing and Field Testing: Conduct annual pilot testing of Writing tasks, including obtaining IRB approval and overseeing recruitment and logistics Coordinate the analysis of pilot test data and report on findings Coordinate, conduct, and sign off on quality checks of field test QTI packages and ancillary materials Review and update field test functional rules Oversee the qualitative analysis of Writing field test responses and report findings to internal teams and external stakeholders Rater Training and Scoring: Collaborate on research and development work for scoring materials related to the Writing test, including rubrics, rater training presentations and courses, and other scoring guidance Select anchor papers and training papers Review and approve score justifications, as well as qualifying, recalibration, and validity sets Synthesize information from the scoring of Writing tasks for use in future development Operational Test and Database Management: Review test maps, test forms, ancillary documents, test practice items, and sample items for the public Write, review, and maintain test and item design plans Write, review, and maintain the Writing Test Construct Framework document Ensure accuracy of data in test item databases Supervision: Set functional team goals and expectations for Writing test development Monitor progress and performance of staff completing Writing test development tasks Mentor test development staff in their job duties and professional development Fulfill administrative responsibilities related to hiring/retention and performance reviews Other duties as assigned. Qualifications
Education and Experience: A Master's degree from an accredited college or university, in a relevant field, with at least two years of relevant experience. Expertise in the written English language development of K-12 multilingual learners, or in the development of written English language proficiency assessments for K-12 multilingual learners. The ideal candidate for this role has: Deep familiarity with the written language that K-12 multilingual learners must be able to produce to be successful in U.S. school settings. Experience teaching K-12 multilingual learners in U.S. school contexts. Experience developing constructed response test items for K-12 students, and preferably for English language proficiency assessments. Experience supervising the development of test items in a high-stakes context. Experience managing complex assessment projects. Knowledge, Skills, and Abilities: Must be able to apply their understanding of multilingual learners' English language development in writing, and of K-12 education standards and practices to the development of language proficiency test items. Coordinate remote staff across multiple tasks to meet deadlines and quality control expectations. Provide constructive, clear feedback that both improves test items and builds staff understanding and skills. Understand complex test designs with multiple interconnected parts. Work independently with minimal support/supervision. Prioritize tasks appropriately, meet deadlines, and work well within a collaborative, full remote team. Attend closely to details without losing sight of the bigger picture. Demonstrate flexibility in responding to feedback and evolving project priorities. Show leadership to attain team goals. Bring both analytical and creative approaches to problem solving. The ideal candidate for this role has: Exemplary writing, editing, and proofreading skills in English. Excellent organization, planning, and time management skills. Strong oral and interpersonal communication skills. Proficiency in a language other than English. At least intermediate proficiency with Microsoft Office and Smartsheet, or other work management platforms. Travel Requirements: This position requires occasional travel (1-3 times per year). Additional Information
Compensation: Minimum Salary: $75,000; Benefits eligible Full-time Physical Demands: The physical demands described here are representative of those that must be met by an employee to successfully perform the essential functions of this job. While performing the duties of this job, the employee is regularly required to effectively communicate. The employee may be required to sit or stand in a stationary position for 50% of the time. This position constantly operates a computer and other office productivity equipment. CAL adheres to ADA compliance and guidelines as a common practice. Additional Information: This job description summarizes the main duties of the job. It neither prescribes nor restricts