Publisher’s Response to EdReports Score

EdReports.org recently published their review of Green Ninja’s middle school science program. Their report and Green Ninja’s publisher response can be found here. In our publisher response to the EdReports score (see below), we outline the shortcomings of the EdReports rubric and why our program scored low using their methodology.

Original EdReports Publisher Response:

Green Ninja shares EdReports’ goal of improving education. We both agree that the NRC Framework and resulting NGSS standards provide a valuable roadmap to create an effective and engaging science program that is relevant and meaningful to all students. The rubric EdReports developed to evaluate science programs, however, only includes a portion of the essential components that the NRC authors envisioned and that are supported by science education research. While 3D instruction is a critical component of the NGSS, student motivation and interest are still the key elements of any learning experience. Green Ninja prioritizes making material relevant to students’ lives and their futures. This priority particularly benefits students that have been traditionally left out from science learning by making strong connections to their home life and problems facing their community. Our main concern with EdReports’ evaluation is that it not only ignores these components of education, but it also actively penalizes our curriculum for spending time addressing them.

This document outlines how the EdReports rubric is too narrow, and how it fails to capture Green Ninja's biggest strengths relating to student motivation and real-world learning.

“We are being penalized for spending time on building student buy-in and engagement.”

EdReports Gateway 1 rubric is too narrow.

Imagine a perfect lesson with a perfect EdReports rating — it masterfully has students employ the three dimensions to do sense-making around a phenomenon. The only problem is that the students have absolutely no interest in the phenomenon — maybe that’s because they are asked to explain something as esoteric as the foraging habits of the lesser white-throated warblers of the outer islands of Greenland. EdReports Gateway 1 rubric prioritizes three-dimensionality over all other factors, and that is not the design approach taken by Green Ninja.

Green Ninja’s top priority is to empower students to become environmental problem-solvers. We want them to be motivated to make improvements in the world around them, and mastering the three-dimensional science and engineering tools in the NGSS helps give them the confidence to do that. While the three dimensions are a key part of the NRC Framework on which the NGSS were built, we point to the guiding principles in the Framework which include curiosity (“Children are born investigators”), relevance (“Connecting to students’ interests and experiences”), and equity — non-cognitive factors that are completely ignored by the EdReports analysis of our curriculum.

Figure 1. An outline illustrating how the Green Ninja curriculum was developed using the NRC Framework Guiding Principles and how that translated to the Green Ninja Framework.

During our early conversations with the EdReports team, they agreed that these principles were important but that they were simply too hard to measure and capture with a consistent rubric. Trying to evaluate the Green Ninja curriculum with the EdReports Gateway 1 rubric is therefore like evaluating a meal only by its appearance while ignoring its taste. As we show in the diagram below, the EdReports rubric only captures a portion of the intention of the NRC Framework and NGSS, and it fails to address factors that contribute to student motivation and interest — the key elements to student learning.

Figure 2. An outline of how the EdReports Gateway 1 Rubric only partially aligns with the NRC Framework Guiding Principles.

Effective 3D instruction begins with the real world.

Green Ninja does value three-dimensionality, and EdReports identified 80 lesson sequences in our curriculum where students engage in sense-making using all three dimensions (constituting a majority of the days of instruction in our curriculum series). EdReports penalizes us for lesson sequences that we claim are a time investment essential for the success of the curriculum — lessons that introduce the problems, have students draw connections to their everyday life, and establish the importance of solving the problem for their future. Of the example lessons that EdReports cites that “do not meet expectations” in criteria 1.AI, 5 of the 7 are these introductory lessons that lay motivational groundwork for a unit. For example, in Grade 8, Unit 6: Future Energy, Lessons 6.1-6.3, students use Google Maps to study local infrastructure in their community. They can literally see the infrastructure that brings electricity to their school and their home, and they draw a diagram showing how electricity gets to their house. While this lesson sequence does not examine a specific DCI, it invites students to look deeper into the world around them and creates buy-in for the investigations of renewable energy solutions that follow throughout the unit by showing the connection between electricity production and their everyday lives.

“The real-world context is the lever for activating prior knowledge.”

Leveraging prior knowledge is more than just explicit prompts.

EdReports does consider the ways in which a curriculum elicits and leverages prior experience and knowledge, but their rubric seems narrowly defined around explicit cognitive uses of prior knowledge. It does not recognize the value of the motivational aspects of making connections to students’ lives. Our curriculum is filled with home-school connections where students try to make changes in their own energy and resource use. For example, EdReports cites Grade 6, Unit 2: Home Energy, Lessons 2.26-2.28 as a lesson sequence that “does not elicit or leverage prior knowledge,” but EdReports fails to credit the fact that students are analyzing energy use in their own school building — the context is the leverage. The very nature of solving a problem in a familiar environment is that students naturally consider their personal experience at their school and make these connections without extra prompts. The problem is relevant to the students because it builds students’ agency as problem-solvers in their local community. We do agree that providing more opportunities to explicitly reflect on the connections could help some teachers and students get the maximum value of their local context, and we will make improvements based on the EdReports feedback. But the rubric is misleading because it only looks for explicit prompts about prior knowledge and ignores the fact that the problems in the Green Ninja curriculum are framed around the students’ direct experience and everyday lives.

Figure 3. The Green Ninja curriculum focus on both non-cognitive (heart) and cognitive (brain) aspects of science learning.

In summary, the EdReports rubric is simply too narrow. Our curriculum is penalized because we spend time at the beginning of each unit creating motivation and buy-in while the rubric is singularly focused on time allotted to three-dimensional learning. Our curriculum receives no credit for focusing on phenomena or problems that are relevant to students’ lives.