Testing Consortium Helps Students and Educators Across the Nation Succeed with Updated Reporting Tool

Challenge

The test reporting system that was built for Smarter Balanced Assessment Consortium (Smarter Balanced), an educational assessment provider that provides common core tests to public schools in California and across the country, was struggling to meet the needs of the teachers and administrators Smarter Balanced serves. Smarter Balanced creates interim and summative tests for the consortium’s member states. Summative tests are important end-of-the-year assessment tests, some of which are used to determine students’ readiness for college-level coursework, while interim tests are given throughout the year to help teachers assess where students need assistance.

Although the reporting tools Smarter Balanced provides are complimentary, California schools use the reports to provide guidance to teachers so that students can succeed on the important end-of-year summative tests. The reporting system that Smarter Balanced used was built by a third-party contractor in 2015. From the teacher and school districts’ perspective, reports that were generated did not reflect their needs. Teachers and administrators could only see students’ scores and could not drill down to see which questions students had missed. Also, teachers could not access the scores of their own classrooms. Instead, they had to sift through the scores of results for the entire school. All of these factors led teachers and school districts to invest money on proprietary reporting software that could have been spent elsewhere.

The reporting system also presented challenges for Smarter Balanced’s IT team. Instabilities in the system led to crashes and necessitated debugging on a regular basis. When teachers and administrators were not able to find test results in the system, Smarter Balanced’s IT team would track down results manually. Consequently, the team was spending unnecessary time maintaining the system. To add to the intensity of the situation, educators, community members, and a California legislator reported that interim tests and reports were not meeting the state’s goal of providing tests that would improve teaching and learning. As Smarter Balanced’s largest partner, California paid large sums of money for its services, administering millions of tests across the state a year. Smarter Balanced needed to fix its reporting system in order to meet the needs of its partners and ensure the success of the consortium.

challenge_sb_people
challenge_sb_screen

Although the reporting tools Smarter Balanced provides are complimentary, California schools use the reports to provide guidance to teachers so that students can succeed on the important end-of-year summative tests. The reporting system that Smarter Balanced used was built by a third-party contractor in 2015. From the teacher and school districts’ perspective, reports that were generated did not reflect their needs. Teachers and administrators could only see students’ scores and could not drill down to see which questions students had missed. Also, teachers could not access the scores of their own classrooms. Instead, they had to sift through the scores of results for the entire school. All of these factors led teachers and school districts to invest money on proprietary reporting software that could have been spent elsewhere.

The reporting system also presented challenges for Smarter Balanced’s IT team. Instabilities in the system led to crashes and necessitated debugging on a regular basis. When teachers and administrators were not able to find test results in the system, Smarter Balanced’s IT team would track down results manually. Consequently, the team was spending unnecessary time maintaining the system. To add to the intensity of the situation, educators, community members, and a California legislator reported that interim tests and reports were not meeting the state’s goal of providing tests that would improve teaching and learning. As Smarter Balanced’s largest partner, California paid large sums of money for its services, administering millions of tests across the state a year. Smarter Balanced needed to fix its reporting system in order to meet the needs of its partners and ensure the success of the consortium.

Solutions

Fairway Technologies rebuilt Smarter Balanced’s reporting system, transforming an application that did not meet the needs of California educators into a useful reporting tool that was met with widespread satisfaction. Fairway was hired to fix the existing Reporting Data Warehouse (RDW). After extensive analysis, Fairway reported to Smarter Balanced that it would be less expensive and require less time to rebuild the entire system. To create a functional system, data had to flow successfully between Smarter Balanced and a number of third-parties involved in administering and scoring the tests, as well as teachers and school districts. Fairway coordinated with all stakeholders including Smarter Balanced, third-party vendors, and the California state department that handles interim testing, employing focus and user groups to ensure that everyone’s needs and requirements were met. The system needed to provide teachers, school administrators, and the state of California with information that would help students succeed on important end-of-year tests.

Fairway built a reporting system that allows teachers to find scores faster, view the scores of their classroom, filter their results, and drill down into student answers, saving time and enabling them to help their students succeed. School district administrators have access to entirely new functionality that includes aggregate reporting and longitudinal reports across schools and districts, demographic breakdowns by several categories, and custom reports that allow administrators to filter by types of content.

solutions_sb
soultion_sb_people

Technology and Tools Used

RDW is implemented as a suite of Spring Boot services. Technologies used include Java, MySQL/Aurora, Redshift, Flyway (database migration), RabbitMQ, Angular, Bootstrap, D3, RESTful web services, transaction management, caching (simple local and shared), scheduled and asynchronous processing, OAuth2 and SSO/SAML security, actuator monitoring and management, centralized configuration, and containerized executable jars. For testing, Fairway used: JUnit, Mockito, Protractor, Karma, and Python (for data generation). For continuous integration, Fairway used Gradle, TeamCity, and Docker Compose. Deployment is in AWS using Kubernetes (with kops), and leverages the following AWS services Aurora, Redshift, ElastiCache (Redis), S3, ELB, Route 53, Certificate Manager.

With the new tool, educators no longer have to wait 24 hours or more to get test scores; they can see scores in near real-time. The reporting tool Fairway built also seamlessly integrates with other Smarter Balanced properties, including an administration and registration tool, a test delivery system, a digital library, and a single sign-on portal. The RDW is divided into sub-systems based on data, performance and scalability requirements. These include:

  • Data Warehouse – Data enters the system here. The warehouse is a collection of all assessment definitions, institutions, students, student groups, test results, and supplemental reference data. 
  • Reporting – The main reporting data mart and web user interface. Fairway optimized test result reporting to allow teachers and administrators to quickly view results.
  • Aggregate Reporting – This is the system for viewing test results through the lens of aggregation, trends, and more. Fairway optimized test results to be analyzed in a variety of ways.
  • Administration – This is the system for performing application administrative tasks including management of student groups, instructional resource links, and embargo settings.

To make deployment easier, all the RDW applications have been containerized, and use Kubernetes for orchestration. RDW is deployed in the cloud via Amazon Web Services (AWS). The cloud-based, containerized architecture provides a robust, scalable solution for Smarter Balanced. Additionally, the RDW is an open-source initiative. Combined with commercial support, this provides all the advantages of open source for the educational testing community and also acts as training material for Smarter Balanced’s team.

Benefits

The reporting system that Fairway built for Smarter Balanced has not only been met with educator approval, it has also been adopted by other states and its uses expanded. Fairway came into a challenging situation and produced a solution that exceeded expectations. The new reporting system allows teachers and administrators to access and understand scores quickly and effectively. The easy-to-use reporting tools help educators to pinpoint those sections of the tests where students are having trouble. Because schools are happy with the new complimentary reporting tool that Smarter Balanced provides, they are not spending time and money to create proprietary reporting systems, allowing public schools to invest elsewhere. Other states have also begun adopting the reporting system due to its success. Prior to Fairway’s involvement, Smarter Balanced’s reporting system covered Math and English Language Arts (ELA) interim testing. As of 2019, Smarter Balanced will also be administering Science exams and the reporting tool will be used for crucial end-of-year summative tests as well.

Smarter Balanced’s IT team has also benefited from the new system. Prior to Fairway’s new reporting system, the team had spent unnecessary time on maintenance. The new system works so seamlessly that when it was deployed, Fairway received a message from Smarter Balanced’s IT team wondering if anyone was using the system. Overall, the Fairway reporting system has helped Smarter Balanced to not only meet the needs of California’s public school students and educators, it has also helped to serve students across the nation more effectively. In 2017, over three million students took Smarter Balanced assessment tests in California alone.

benefits_sb

Company Overview

Smarter Balanced Assessment Consortium is a public agency currently supported by its members. Through the work of thousands of educators, Smarter Balanced created an online assessment system aligned to the Common Core State Standards (CCSS), as well as tools for educators to improve teaching and learning. Smarter Balanced is housed at the University of California Santa Cruz Silicon Valley Extension. With 13 member states along with the U.S. Virgin Islands and the Bureau of Indian Education, Smarter Balanced’s work is guided by the belief that a high-quality assessment system can provide information and tools for teachers and schools to improve instruction and help students succeed—regardless of disability, language, or subgroup.

Technical Highlights

RDW is architected to allow appropriate scaling of services based on their specific performance characteristics. Message queues distribute the workload to services which can be horizontally scaled to meet demand. Data is distributed between a data warehouse optimized for reliability and resilience, and reporting data stores optimized for highly-concurrent report generation. Centralized configuration of containerized services running in a robust orchestration environment simplifies deployment and minimizes maintenance effort. The result is a system that can process millions of test results per day and provide sub-second response times for individual reports, all with low operational cost.

About Fairway Tech

Fairway Technologies provides strategic consulting services and end-to-end technology solutions to solve the world’s toughest business challenges. From Fortune 50 multinational companies to startups, our innovative engagement model focuses on your company’s needs, helping to cut costs and freeing your team to innovate. Fairway’s 100% US-based team has decades of experience crafting elegant and effective technology solutions to make your business thrive.

At Fairway, providing genuine support and being attentive to our clients’ needs is a core value. Rather than applying a one-size-fits-all approach, we dedicate our attention and focus to create the right solution for each client.

Fairway Tech

Some of the top reasons people come to us are because they:

  • Can’t complete a project—either on their own or with another consultant who provides less value
  • Do not have the required skill set within their internal team
  • Want to outsource development so that their team can focus on operations
  • Have run out of time to complete a project

Need help with your project?

 

Let’s Chat – hi@fairwaytech.com