## [Testing Testing] SBAC Grade 11 Practice Test #4

Each “Testing Testing” post analyzes a released test item, focusing on both the mathematics and interface involved in the new breed of exams. Our goal is to help improve the quality of these exams, especially if they may be used to inform student graduation or teacher merit pay. The mathematical analysis here is from Al Cuoco, director of the Center for Mathematics Education at EDC, from the Trevi Fountain in Rome. I provide the interface analysis.

Here’s Problem 4 from the SBAC Grade 11 Math Practice Test.

There are several issues with this item. By far the greatest issue is that it doesn’t assess a Common Core standard. The closest standard is HSA-APR.B.3, and read it carefully:

• HSA-APR.B.3: Identify zeros of polynomials when suitable factorizations are available, and use the zeros to construct a rough graph of the function defined by the polynomial.

This problem doesn’t ask students to do that. It asks students to use a rough sketch of a graph to build a factorization, identifying the linear factors of the underlying polynomial.

Besides this, there are some serious mathematical issues with this item as presented. The correct answer to this type of problem should start out in one’s mind as a multiple of $(x+3)(x+1)(x-2)^2(x-4)$, applying the Remainder Theorem (HSA-APR.B.2). Then, you check a sixth point to find the multiple, making sure the polynomial agrees with the nonzero points on the graph (a polynomial function of degree n is determined by n+1 function values, not just its n roots). But no coordinates of a sixth point are given. A student could simply not think of this and get the problem right, or estimate the value of the function at 0 and see if the graph is approximately correct (and get it right), or assume that this is a test and read the minds of the test writers (and get it right). It’s a good thing the graph seems to pass through (0, -48), or there would be no way to give a correct answer!

Even if you could get the exact coordinates of f(0), there are other polynomials functions whose graphs contain these same points and that have an extra x-intercept off the picture. The phrase “the function for the graph” is not accurate: there is more than one. Better wording: “Find a polynomial function that could have this graph.” The phrase “Create the function” should also be avoided, since it’s not clear what it means; Common Core has students “define” a function or “build” a function for modeling or transformation.

Still, the core issue is that this problem does not directly address a standard. The problem would be much better if it assessed HSA-APR.B.3 directly: Give a function defined by a polynomial that has been partially factored, revealing some zeros, allowing for complete factorization. For example:

$g(x) = 3(x^2-1)(x-2)^2$

Students who can sketch the graph of g (or answer questions about the graph) have met several HSA-APR standards.

Separate from the mathematics, there are a few major interface issues with this problem. Students are only allowed to drag linear factors to the right, so it is not possible for a student to enter $(x-2)^2$. They must instead enter the clumsy $(x-2)(x-2)$. Some students will wonder why they have to do it this way, others will think the interface is broken. It’s an unnecessary hurdle that will prevent some students from answering correctly even though they are capable of completing the task on paper.

Like #11, the interface does a poor job of overwriting when new objects are placed. Dragged objects “snap” into one of five positions, and if an object is already there, conflict:

We can picture a student trying to do this on purpose, attempting to get the $(x-2)^2$ term. It easily happens by accident and the interface for “deleting” objects is not obvious. Would the answer above be marked correct or incorrect? It’s not clear, and that is a big problem.

The SBAC Practice Tests are available for public viewing, and we are grateful to have these problems available for public comment.

## [Testing Testing] SBAC Grade 11 Practice Test #11

Each “Testing Testing” post will analyze a released test item, focusing on both the mathematics and interface involved in the new breed of SBAC and PARCC exams. Interfaces can have huge effects on how problems are posed, read, answered, and scored. Posts will also provide suggestions on how to improve these exams, in terms of the mathematics presented, and also in terms of the interface students will use when taking exams.

Here’s Problem 11 from the SBAC Grade 11 Math Practice Test.  (A colleague suggests I have chosen Problems 7 and 11 as the first examples primarily because I love Slurpees.)

This problem targets HSA-REI-D.10 and HSA-REI.D.11, knowing that the points on a graph are the solutions to the corresponding equation, and that the x-coordinates of the points where the graphs of y = f(x) and y = g(x) intersect are the solutions of the equation f(x) = g(x).

There is one mathematical error in the problem. When asking for “a solution for f(x) = g(x)”, the answer is an x-coordinate, not a point — something HSA-REI.D.11 states explicitly. This placement of points should be correct:

The problem interface will not allow the third point to be placed here. A placed point “snaps” to the nearest marked point on the grid. These interface behaviors need to be described in the problem, or the results can be surprising or frustrating. For example, a student might want to place their solution to y = g(x) at (0,0). The problem doesn’t exclude them from doing this, but the interface does.

The interface also causes trouble when points are placed atop one another. It looks like this:

Note that the newly placed point is not in the same position as the original. I can’t tell whether or not this solution would be accepted on the exam, and fixing the problem requires moving away the new point, then the old point, then replacing the new point. It’s a mess. If the exam requires all three points to be different from one another, this needs to be part of the problem statement, and the interface should “push” old points back to the gray area at the bottom if a new point is placed in the same location.

These interface issues can be fixed. Eliminate the dragging, label each target point in the grid, and ask the questions about the specific points. This also allows more explicit asking about HSA-REI.D.11 rather than the previous “solution for f(x) = g(x)”.

This new version keeps alive what makes the original question better than multiple-choice: there are several correct answers to each part. One might consider changing this version (or the original) to require the student to provide points that are on the graph of one function and not the other: “Name a point that is a solution to y = f(x) but not y = g(x)”.  Otherwise, a student can answer all three questions correctly with “C” without assurance that they have mastered the standard.

The SBAC Practice Tests are available for public viewing, and we are grateful to have these problems available for public comment.

## [Testing Testing] SBAC Grade 11 Practice Test #7

We’re starting a new category of post here called “Testing Testing”. Each post will analyze a released test item. While this would normally focus just on the mathematics of the problem, the SBAC and PARCC exams are also promising new formats and interfaces for testing, which can potentially have effects on how problems are posed, read, answered, and scored. Writing assessment items for these new interfaces will be especially challenging, and it may take more revisions than usual to build a high-quality test item.

Recently, SBAC (short for Smarter Balanced Assessment Consortium) released a set of Practice Tests, giving everyone the chance to study the testing environment and suggest improvements. I took the 11th grade mathematics exam, with 23 problems, trying to get a sense of the quality of the mathematics and the quality of the interface. Several posts will talk about items from the SBAC Practice Test, along with suggestions to SBAC on how to improve both the mathematics of the exam and the interface students will use when taking it.

Here’s Problem 7 from the SBAC Grade 11 Math Practice Test.

At first glance, this problem looks pretty innocent, and has a useful testing target, specifically HSA-REI.D.11, finding approximate solutions to f(x) = g(x) when f and g are given by tables. The test designers were even careful to use function types prescribed by the standard: there’s a quadratic rule matching f and an absolute value rule matching g. This could have been a good item, expecting students to interpret the table in a meaningful way.

However, there are two serious errors in the problem. First, the problem is never stated. The problem asks to “show the interval for x in which the solution to this system of equations lies”. But there is no system of equations. There’s not even an equation at all, just a table. To get this question right, the student must figure out the intended question, the interval for x that contains the solution to f(x) = g(x). Note that this still isn’t a system of equations, just a single equation with a single variable.

Second, and even more serious, there is more than one correct answer to the problem. The only thing we are told about functions f and g is that they are continuous. We are not told that they are quadratic, absolute value, or otherwise. We cannot assume that the functions do not intersect in other intervals: in fact, they can easily intersect. These two functions match the given tables and intersect many more times:

$f(x) = x^2 - 5 + 100 \sin (2 \pi x) \text{ and } g(x) = |120-17x|$

Because f and g are continuous, we know they must intersect somewhere between x=5 and x=6, but we do not know for certain that they will not intersect elsewhere.

These issues are very serious and require correction. There are other minor issues: “f(x)” isn’t a function, it’s an output of the function f (see HSF-IF.A.1). Functions don’t have points, they have ordered pairs; graphs of functions have points. Neither of these minor issues spells doom for the problem, but these details can be corrected.

On the plus side, the interface for the problem is clean and the task for the student is clear (“Click the number line”). Clicking marks a unit interval in red, and clicking anywhere else changes the interval to reflect the new click.

This problem can be repaired simply by changing the text that appears on the left: “The table shows several inputs and outputs for continuous functions f and g. [table] The graphs of f and g intersect exactly once. Click the number line to show the unit interval for x in which the solution to the equation f(x) = g(x) must lie.”

I encourage you to take the SBAC Practice Test. I’ll have more to say about other problems. My hope is that through the work of the community we can improve these exams while they are still in their pilot stage.

## We Got A Problem #12: Tile Shufflin’

Online word games, such as Scrabble, now let you “shuffle” your tiles with the touch of a button.  As far as I can tell, when you click the “shuffle” button the 7 tiles are randomly rearranged into any of the 7! = 5040 different orderings.

And there’s an animation of the tiles moving.  Well… sort of.  Because, sometimes one or more of the tiles doesn’t move, because they are in the same position before and after the shuffling.

How does the probability of having one stuck tile compare to the probability of having no stuck tiles? How does the probability of having one stuck tile compare to the probability of having two stuck tiles? Three? Four?

A Scrabble variant called Lexulous gives players eight tiles instead of seven.  What happens then?

What is the approximate probability of having no stuck tiles?

## The Gambling Machine

The NY Times “Numberplay” blog recently posted an interesting question called “The Gambling Machine”.  The problem was based on a problem found in the NCTM 2011 slides from “Mathematics of Game Shows”, from the PCMI 2007 materials, and originally from the (thankfully cancelled) game show National Bingo Night.

The Numberplay article asks about the expected return of proper play in the game, and whether or not a player can get \$40 return from the game.  A reasonable guess is for the player to guess higher when the first number is half or less, and guess lower when the first number is more than half, but this is not the best possible strategy.

The article provides some analysis and solutions, and I’ve provided my solution as this PDF.

## We Got A Problem #11: Rock Rock Paper Scissors

Most people know Rock-Paper-Scissors (or its more fun cousin, Ninja-Cowboy-Bear).  Kate and Ogden are locked in a long battle of online Rock-Paper-Scissors, with a winner scoring 1 point, and a tie scoring nothing for either player.  Game’s to 100.

Ogden won the first series 100-77 by constantly looking for patterns in Kate’s choices. For the second series, Kate tries something interesting: she has no pattern.  She randomly selects rock, paper, or scissors with equal probability.  She even tells Ogden that she is doing this, and he can do nothing about it but sigh and accept that he cannot beat this “strategy”.  Kate wins the second series 100-95 by luck.

For the third series, Ogden changes a rule: if rock beats scissors, the winner scores 2 points instead of 1.  Other victories (scissors over paper, paper over rock) are still worth 1.

Now Kate’s panicked: if she still randomly selects, is she beatable in the long run, or not?  Can she alter the probabilities to make a new “strategy” that Ogden can’t beat?  It seems like it would be bad to play scissors so often, and she should probably play more rocks… or not?

Let us know whether you find any ways that Kate can play to force Ogden into equilibrium.

## We Got A Problem #10: Cliff Hangers!

My wife was recently playing the Cliff Hangers slot machine, based on a great game from The Price Is Right.  Here’s how the bonus game works:

The goal is to climb exactly to the 25th and final step on the mountain (while yodeling, of course).  You keep taking turns, picking from 5 doors, one each with the numbers 3, 5, 10, 15, and 20 on them.  Whatever pick you make, that’s the number of steps taken on that turn.  The doors are randomized, so you don’t know what you’re going to get, and you can get the same number more than once.

If after any number of turns your total is exactly 25 steps, you win.  If you go over 25, you lose.

There are clearly some ways to win the game with lucky picking, such as 5 + 20 or 10 + 10 + 5.  My wife won even though her first pick was a 3.

What is the probability of hitting 25 and winning this game?  Which of these three goals gives the greatest chance of winning: 18, 55, or 58?  If there were no stopping point, what is the long-term probability of hitting a high number like 507?

We’ll provide a solution next week that involves a single mathematical calculation.  It’s pretty awesome.