Earlier this month, CNN ran a back-to-school story on homework with the headline, “Kids Have Three Times Too Much Homework, Study Finds; What’s the Cost?” Homework is an important topic, especially for parents, but unfortunately, CNN’s story misleads rather than informs. The headline suggests American parents should be alarmed because their kids have too much homework. Should they? No, CNN has ignored the best evidence on that question, which suggests the opposite. The story relies on the results of one recent study of homework—a study that is limited in what it can tell us, mostly because of its research design. But CNN even gets its main findings wrong. The study suggests most students have too little homework, not too much.
The study that piqued CNN’s interest was conducted during four months (two in the spring and two in the fall) in Providence, Rhode Island. About 1,200 parents completed a survey about their children’s homework while waiting in 27 pediatricians’ offices. Is the sample representative of all parents in the U.S.? Probably not. Certainly CNN should have been a bit leery of portraying the results of a survey conducted in a single American city—any city—as evidence applying to a broader audience. More importantly, viewers are never told of the study’s significant limitations: that the data come from a survey conducted in only one city—in pediatricians’ offices by a self-selected sample of respondents.
The survey’s sampling design is a huge problem. Because the sample is non-random there is no way of knowing if the results can be extrapolated to a larger population—even to families in Providence itself. Close to a third of respondents chose to complete the survey in Spanish. Enrollment in English Language programs in the Providence district comprises about 22 percent of students. About one-fourth (26 percent) of survey respondents reported having one child in the family. According to the 2010 Census, the proportion of families nationwide with one child is much higher, at 43 percent. [i] The survey is skewed towards large, Spanish-speaking families. Their experience with homework could be unique, especially if young children in these families are learning English for the first time at school.
The survey was completed by parents who probably had a sick child as they were waiting to see a pediatrician. That’s a stressful setting. The response rate to the survey is not reported, so we don’t know how many parents visiting those offices chose not to fill out the survey. If the typical pediatrician sees 100 unique patients per month, in a four month span the survey may have been offered to more than ten thousand parents in the 27 offices. The survey respondents, then, would be a tiny slice, 10 to 15 percent, of those eligible to respond. We also don’t know the public-private school break out of the respondents, or how many were sending their children to charter schools. It would be interesting to see how many parents willingly send their children to schools with a heavy homework load.
I wish the CNN team responsible for this story had run the data by some of CNN’s political pollsters. Alarm bells surely would have gone off. The hazards of accepting a self-selected, demographically-skewed survey sample as representative of the general population are well known. Modern political polling—and its reliance on random samples—grew from an infamous mishap in 1936. A popular national magazine, the Literary Digest, distributed 10 million post cards for its readers to return as “ballots” indicating who they would vote for in the 1936 race for president. More than two million post cards were returned! A week before the election, the magazine confidently predicted that Alf Landon, the Republican challenger from Kansas, would defeat Franklin Roosevelt, the Democratic incumbent, by a huge margin: 57 percent to 43 percent. In fact, when the real election was held, the opposite occurred: Roosevelt won more than 60% of the popular vote and defeated Landon in a landslide. Pollsters learned that self-selected samples should be viewed warily. The magazine’s readership was disproportionately Republican to begin with, and sometimes disgruntled subjects are more likely to respond to a survey, no matter the topic, than the satisfied.
Here’s a very simple question: In its next poll on the 2016 presidential race, would CNN report the results of a survey of self-selected respondents in 27 pediatricians’ offices in Providence, Rhode Island as representative of national sentiment? Of course not. Then, please, CNN, don’t do so with education topics.
The Providence Study’s Findings
Let’s set aside methodological concerns and turn to CNN’s characterization of the survey’s findings. Did the study really show that most kids have too much homework? No, the headline that “Kids Have Three Times Too Much Homework” is not even an accurate description of the study’s findings. CNN’s on air coverage extended the misinformation. The online video of the coverage is tagged “Study: Your Kids Are Doing Too Much Homework.” The first caption that viewers see is “Study Says Kids Getting Way Too Much Homework.” All of these statements are misleading.
In the published version of the Providence study, the researchers plotted the average amount of time spent on homework by students’ grade. [ii] They then compared those averages to a “10 minutes per-grade” guideline that serves as an indicator of the “right” amount of homework. I have attempted to replicate the data here in table form (they were originally reported in a line graph) to make that comparison easier. [iii]
Click to enlarge
Contrary to CNN’s reporting, the data suggest—based on the ten minute per-grade rule—that most kids in this study have too little homework, not too much. Beginning in fourth grade, the average time spent on homework falls short of the recommended amount—a gap of only four minutes in fourth grade that steadily widens in later grades.
A more accurate headline would have been, “Study Shows Kids in Nine out of 13 Grades Have Too Little Homework.” It appears high school students (grades 9-12) spend only about half the recommended time on homework. Two hours of nightly homework is recommended for 12th graders. They are, after all, only a year away from college. But according to the Providence survey, their homework load is less than an hour.
So how in the world did CNN come up with the headline “Kids Have Three Times Too Much Homework?” By focusing on grades K-3 and ignoring all other grades. Here’s the reporting:
The study, published Wednesday in The American Journal of Family Therapy, found students in the early elementary school years are getting significantly more homework than is recommended by education leaders, in some cases nearly three times as much homework as is recommended.
The standard, endorsed by the National Education Association and the National Parent-Teacher Association, is the so-called “10-minute rule”— 10 minutes per-grade level per-night. That translates into 10 minutes of homework in the first grade, 20 minutes in the second grade, all the way up to 120 minutes for senior year of high school. The NEA and the National PTA do not endorse homework for kindergarten.
In the study involving questionnaires filled out by more than 1,100 English and Spanish speaking parents of children in kindergarten through grade 12, researchers found children in the first grade had up to three times the homework load recommended by the NEA and the National PTA.
Parents reported first-graders were spending 28 minutes on homework each night versus the recommended 10 minutes. For second-graders, the homework time was nearly 29 minutes, as opposed to the 20 minutes recommended.
And kindergartners, their parents said, spent 25 minutes a night on after-school assignments, according to the study…
CNN focused on the four grades, K-3, in which homework exceeds the ten-minute rule. They ignored more than two-thirds of the grades. Even with this focus, a more accurate headline would have been, “Study Suggests First Graders in Providence, RI Have Three Times Too Much Homework.”
Homework is a controversial topic. People hold differing points of view as to whether there is too much, too little, or just the right amount of homework. That makes it vitally important that the media give accurate information on the empirical dimensions to the debate. The amount of homework kids should have is subject to debate. But the amount of homework kids actually have is an empirical question. We can debate whether it’s too hot outside, but the actual temperature should be a matter of measurement, not debate. It’s impossible to think of a rational debate that can possibly ensue on the homework issue without knowing the empirical status quo in regards to time. Imagine someone beginning a debate by saying, “I am arguing that kids have too much [substitute “too little” here for the pro-homework side] homework but I must admit that I have no idea how much they currently have.”
Data from the National Assessment of Educational Progress (NAEP) provide the best evidence we have on the amount of homework that kids have. NAEP’s sampling design allows us to make inferences about national trends, and the Long-Term Trend (LTT) NAEP offers data on homework since 1984. The latest LTT NAEP results (2012) indicate that the vast majority of nine-year-olds (83 percent) have less than an hour of homework each night. There has been an apparent uptick in the homework load, however, as 35 percent reported no homework in 1984, and only 22 percent reported no homework in 2012. MET Life also periodically surveys a representative sample of students, parents, and teachers on the homework issue. In the 2007 results, a majority of parents (52 percent) of elementary grade students (grades 3-6 in the MET survey) estimated their children had 30 minutes or less of homework.
The MET Life survey found that parents have an overwhelmingly positive view of the amount of homework their children are assigned. Nine out of ten parents responded that homework offers the opportunity to talk and spend time with their children, and most do not see homework as interfering with family time or as a major source of familial stress. Minority parents, in particular, reported believing homework is beneficial for students’ success at school and in the future. [iv]
That said, just as there were indeed Alf Landon voters in 1936, there are indeed children for whom homework is a struggle. Some bring home more than they can finish in a reasonable amount of time. A complication for researchers of elementary age children is that the same students who have difficulty completing homework may have other challenges—difficulties with reading, low achievement, and poor grades in school. [v] Parents who question the value of homework often have a host of complaints about their child’s school. It is difficult for researchers to untangle all of these factors and determine, in the instances where there are tensions, whether homework is the real cause. To their credit, the researchers who conducted the Providence study are aware of these constraints and present a number of hypotheses warranting further study with a research design supporting causal inferencing. That’s the value of this research, not CNN’s misleading reporting of the findings.
— Tom Loveless
This first appeared on the Brown Center Chalkboard
Notes:[i] Calculated from data in Table 64, U.S. Census Bureau, Statistical Abstract of the United States: 2012, page 56. http://www.census.gov/compendia/statab/2012/tables/12s0064.pdf.[ii] The mean sample size for each grade is reported as 7.7 percent (or 90 students). Confidence intervals for each grade estimate are not reported.[iii] The data in Table I are estimates (by sight) from a line graph incremented in five percentage point intervals.[iv] Met Life, Met Life Survey of the American Teacher: The Homework Experience, November 13, 2007, pp. 15.[v] Among high school students, the bias probably leans in the opposite direction: high achievers load up on AP, IB, and other courses that assign more homework.
New York University
Computer Science Department
Courant Institute of Mathematical Sciences
Homework #1 – Traditional Application Server Technology
Due Date: 09/17/01
Course Title: XML for Java DevelopersCourse Number: g22.3033-011
Instructor: Jean-Claude FranchittiSession: 1
1. Ongoing Project Background
As part of the course on-going project, you will develop a framework-based enterprise application of your choice and deploy it on top of various types of application server environments. You will also enrich your application, and tune its operational environment as allowed by the underlying application server facilities. You will need to develop and deploy your application as a set of application-level services and common facilities on top of various application server frameworks suggested as part of the homework specifications. Your applications should keep the processing, rendering, querying, secure messaging, and persistence aspects of your application well componentized to help study the migration of these capabilities across various types of application server platforms. A meaningful example of a framework-based enterprise application is provided by the family of “multi-channel online community applications” that were developed in the Summer 2001 course project for the XML for Java Developers course (i.e., g22-3033.002). As you transition your enterprise application from one platform to another, you will study and develop model-based migration and interoperability tools that leverage off of the cutting edge concepts subsumed by modern Model Driven Architectures (MDA).
You should follow the "real-life" application architecture development approach suggested in the Session1 Sub-Topics 1-3 slide sets. In real projects, the application, application infrastructure, and technology infrastructure are usually analyzed, designed, and developed in parallel. You should always select a technology infrastructure (e.g., Windows 98/NT/2000 or Solaris 5.8 on i5.nyu.edu, and Sun’s JDK 1.3, or 1.4) compatible with the application infrastructure set forth by the corresponding homework.
2. Introduction to Traditional Application Server Technology
The first generation application servers were spawned from the requirement to improve upon basic CGI connections between a database and a Web server. To relate to this initial requirement, this first homework studies a simple Application Server configuration based on an extension of basic CGI technology.
The simple Application Server is made up of the following components (see also attached diagrams)
1. A small CGI client program which could be written in any language
2. An Application Server which is multithreaded and is written in Java
3. Templates which contain the presentation logic of the screens. By keeping the presentation logic detached from the actual code, there is no need to recompile the Application Server to create new screens allowing for rapid development and maintenance. In addition, these templates can contain logic for remote method calls or queries to an underlying database.
2.1. Software Infrastructure Provided:
1. Perl archive (to be downloaded from http://www.perl.com/pub/a/language/info/software.html)
2. Apache Web Server archive to be downloaded from http://www.apache.org. Other web servers can be used as alternatives (e.g., JWS, IIS, etc.)
3. Browser (Internet Explorer, or Netscape)
4. J2SE 1.4
5. A Java IDE of your choice
6. An OOAD environment of your choice (e.g., Rational Rose, Softera, etc.)
7. Other development tools best suited for legacy application server development environments (as identified in homework #1a)
Templates (these files must be placed in the same directory where AppServer has been started)
Additional Sample Applications
2.2. Installing and Running the Homework Software Infrastructure:
1. Install Perl
2. Run the AppServer
ü Make sure to place all templates in the same directory as AppServer
ü Run the AppServer as follows: AppServer port#
3. Run the cgimodule
ü Make sure to place the cgimodule.pl in the Web server cgi-bin directory
ü Make changes to the script so that the correct host and port are set
ü Run the cgimodule as follows from your web browser:
1. Preparation phase:
- Select a technology infrastructure that is compatible with the application infrastructure components provided for this homework.
- Install the application infrastructure software, and experiment with it.
- Come up with a framework-based enterprise application based on your own interest and the requirements set forth in the first section of this handout.
- Install the additional sample applications, and experiment with them.
2. Prepare a short report documenting your framework-based enterprise application (using software engineering standards), and explaining its motivation.
3. Prepare a short report including functional diagrams and screenshots (as needed) to demonstrate your understanding of the infrastructure software. Explain the infrastructure software differences between the additional sample applications and the software you are using for the homework.
4. Develop and deploy your framework-based enterprise application on top of the infrastructure software provided. Document the benefits and deficiencies of the approach on which the infrastructure software is based, and explain (as needed) how it limits your ability to develop the various application components you have envisioned for your enterprise application. Note that you do not need to provide a complete implementation of your application in this first homework. You should restrict yourself to what you feel is feasible based on time and the level of support provided by the infrastructure software. Your application should be tuned for efficiency as allowed by the underlying infrastructure software, and you should document your performance engineering approach. You should conclude your report by suggesting, and implementing (as time allows) an improved Application Server model.
5. Explain how you would build an “analyzer” tool that can capture information about your deployed application and redeploy it as an XML-based web application such as the “spyweb” application provided as support material under demo programs on the course website. Note that the target application should maintain a strict separation between content, style, and logic. Your analyzer should extract and represent a generic model of your application using a suitable markup language.
6. Extra Credit: Implement a prototype of the analyzer tool described in question 5.
Please provide an electronic copy of your homework submission as one zip archive by sending it to the course TA by the beginning of next class. The archive should include your application software, any modified infrastructure software, a readme file describing the environment setup required to run your application, and your homework report (in word or text format). You should name your archive using the following convention for all homeworks: name_hw?_fa01.zip, where name may include additional underscore characters as needed. You are also required to provide a hard copy of your homework reports at the beginning of each session or according to the homework submission scheduled spelled out in the homework specification.
All assignments are graded on a maximum scale of 10 points. Your grade will be based equally on:
a. The overall quality of your documentation.
b. The understanding and appropriate use of application server related technologies.
c. Your ability to submit working and well-commented code.
d. Extra credit may be granted for solutions that are particularly creative.
Teaming is allowed for all the homework and projects in this course. No more than two members per team are allowed. As the course homework is based on an ongoing project, the same team members will be required work together on all of the homework assignments. Please let the TA know as soon as possible about teaming arrangements. You should only submit one report/archive per team for each assignment. To balance things out, the final grading in the course will take into account the fact that you are working as a team instead of individually, so you should feel free to work individually as well. Note that the final take home examination will require individual work.