Jon R Bond. Presidential Studies Quarterly. Volume 49, Issue 1. March 2019.
CQ’s 2017 Presidential Support Vote Study proclaims, “He divided, but conquered: full control of government gives trump a record success rate” (Bennett, 26). The data show that President Donald Trump prevailed on 100 percent of House votes and 98 percent of Senate votes on which he expressed a clear position. These are indeed record high scores, besting the first‐year success rates of the eleven previous presidents in the House and tying Obama’s record in the Senate (see Figure 1).
CQ’s annual Presidential Support studies have long provided key indicators in the scientific study of presidential success in Congress. Support on roll‐call votes, of course, is not an indication of presidential influence, but understanding the conditions that promote presidential success is important. Richard Fleisher and I concluded that CQ’s studies of how often the president’s position prevails on roll‐call votes provide a valid and reliable measure of an important part of legislative success (Bond and Fleisher, 60-71).
This conclusion assumes, however, that the president’s position‐taking behavior is valid and reliable. Determining how often the president’s position prevails on roll‐call votes rests on a presumption that the president generally expresses truthful and consistent positions. Consider Trump’s scores in light of his interactions with Congress. If the president merely endorses policies developed solely by congressional leaders but remains ignorant of basic details of policies he purportedly supports, misunderstands Congress and what congressional leaders can and cannot do, and incessantly spews out tweets riddled with errors that often contradict previously stated positions, then interpreting the outcomes of roll‐call votes as record high presidential success is akin to giving an A to a student who turned in a plagiarized paper. This erratic behavior makes it difficult to reliably determine whether Trump has a clear personal preference for a particular legislative outcome. Although this unique threat to the reliability of roll‐call‐based measures may be temporary, the experience highlights the fundamental importance of reliably observing presidential position taking. Thus, this article reconsiders the validity and reliability of CQ’s presidential support studies and compares CQ’s list of floor votes on which Trump expressed a position in 2017 to the list identified at the data journalism website FiveThirtyEight.com.
The Scientific Study of Presidential Success in Congress
Scientific analysis of presidential success in Congress requires valid and reliable measures of both the president’s position (yea/nay) and how often each member’s votes or the aggregate outcomes of the roll calls correspond to the president’s preference. Determining how members of Congress vote is highly reliable because they regularly show up for roll calls, and their votes are recorded in the public record. Members take this part of their job seriously. Mean participation rates on roll‐call votes were around 86 percent from the 1950s to the 1970s and have exceeded 94 percent since the 1980s (calculated from data in the Congressional Quarterly Almanac 2015, B‐10).
Presidents, in contrast, do not express a position on all roll calls, and when they do, their position is not part of the public record. Absent an authoritative source, scholars have relied on journalists at Congressional Quarterly to identify votes on which the president expressed a position and what that position was.
Validity of Roll‐Call Votes
To be a valid measure, the “activity observed must bear a close relation to what we ordinarily mean by…legislative success, and…accurately reflect major components of decision making in Congress” (Bond and Fleisher, 55). The floor vote, of course, is not the only (or necessarily the most important) decision point in the legislative process. Nonetheless, it is one of the essential points and one that is readily observable over time.
Major issues seldom fail to show up on the House and Senate floor for resolution by roll‐call vote. Members of Congress go on record for or against issues. Interest groups construct scores based on members’ votes to identify supporters and opponents. Presidents use votes as a basis for distributing rewards (Covington). If we seek to understand how elected policy makers interact in the legislative process, analyzing behavior that is observable to the participants and to voters can contribute to that understanding. Furthermore, with observations spanning 65 years and 12 presidents, the number of roll calls is large enough to provide a basis for valid generalizations. From 1953 to 2017, I found 4,406 conflictual presidential roll calls (at least 10 percent opposed) in the House and 4,752 in the Senate. The number per year ranges from fewer than 20 to more than 100. The means are 68 in the House and 73 in the Senate. Yet with only 12 presidents on which to base generalizations about presidential behavior, we still have a small‐N problem, and an erratic outlier can distort patterns revealed by analysis of decades of data.
In addition to validity, reliability is essential to a meaningful measure of presidential positions.
Reliability of Identifying Presidential Positions
The reliability of a measure refers to the extent to which repeated observations of the same phenomenon are consistent and uniform across different researchers and over time. Because the president is not a member of Congress who is expected to vote on all roll calls, measuring presidential positions requires observing whether the president expressed a position, in addition to the preferred result (yea/nay).
CQ has published their Presidential Support study annually since 1953, using a methodology that can produce a comprehensive and reliable list of presidential positions on roll‐call votes. CQ editors published detailed “Ground Rules” describing how they identified presidential positions and how they dealt with borderline cases, important votes excluded, motions, special rules on House votes, appropriations, failure to vote, weighting of votes (equal), changed positions (the one at the time of the vote that members could know before they voted), and changes in the legislation.
The basic approach is to search all messages, press conferences, and public statements to determine whether the president expressed a clear position prior to the vote. Some studies indicate that they searched backward from the vote: “CQ checked the background of [all] roll‐call votes [to determine if the president] was definitely ‘for’ or ‘against’ on each of these roll calls” (Congressional Quarterly Almanac 1954, 56). Either way, the position is observed before the vote. A presidential position was defined as what the president “does or does not want in the way of legislative action” (Congressional Quarterly Almanac 1959, 125). Before 1959, the wording was “whether he would have voted ‘yea’ or ‘nay’ if he had been a member of Congress” (Congressional Quarterly Almanac 1955, 70). Personally, I prefer this wording because it corresponds to members’ choices, but the revision is close.
The editors at CQ were professional “informed political observers,” and CQ had “an excellent reputation for making sound…judgments about congressional politics” (Bond and Fleisher, 71, 62). This professionalism and assurances that studies of presidential support “have been conducted in a consistent manner since 1953” (Congressional Quarterly Almanac 2009, B‐3) persuaded me that the data provided reliable measures. Presidency scholars did not accept CQ’s scores at face value, however. There was enough detail in the methods description that we could adjust the scores (e.g., analyzing “Key Votes,” no penalty for not voting) to increase their usefulness in our research. As we coded the presidential roll calls included in the studies, it was reassuring that we found (and corrected) only a few minor errors (Edwards; Bond and Fleisher). We were concerned that presidents might rush to endorse sure winners and shirk expressing a position on difficult votes to artificially inflate their success rate. To minimize potential distortions from such posturing, we eliminated votes on which there was near consensus support for the president’s position. Moreover, Peterson (chap. 2) presents evidence that presidents’ publicly stated preferences are generally sincere and they do not duck difficult issues. Thus, CQ’s Presidential Support studies appear to provide a valid and reliable indicator of presidential support and success in Congress.
A Reevaluation of the Reliability of Identifying Presidential Positions
Nevertheless, despite assurances that methods used to identify presidential positions have been consistent since 1953, nagging reliability concerns persist.
A Closer Look at the Reliability of CQ’s Presidential Support Studies
CQ reports some minor inconsistencies that pose little threat to reliability. For example, CQ has consistently weighted votes equally, but in 1964, “CQ arbitrarily chose 10 of the 116 Senate roll calls on civil rights…to avoid overweighting by the civil rights issue” (Congressional Quarterly Almanac 1964, 730). However, CQ recorded presidential positions in the description of each vote, so we can include them if appropriate. Including them has only a small effect on Lyndon B. Johnson’s Senate success rate in 1964 (Bond and Fleisher, 63).
Other inconsistencies are more troubling. Although the basic approach of identifying the president’s position based on public statements before the vote occurs has remained consistent, what counts as a presidential position has changed. These changes raise doubts about whether a presidential position has a consistent meaning since 1953.
Initially, only “sharply defined pronouncements by [the president] himself qualified as evidence” (Congressional Quarterly Almanac 1954, 56). That is, a presidential position was one that the president “personally, as distinct from other Administration spokesmen, does or does not want” (Congressional Quarterly Almanac 1956, 108). In 1974 and 1975, CQ added “and official White House spokesmen to determine what he personally, as distinct from other administration officials, does or does not want” (Congressional Quarterly Almanac 1974, 1008). Statements of White House staff rather than, say, cabinet secretaries likely reveals the president’s “personal” position. CQ eventually clarified, “Statements by individual cabinet members are not taken as positions unless endorsed by the president” (Congressional Quarterly Almanac 1987, 27‐C).
From 1976 to 1986, the “official White House spokesmen” language was replaced by “public statements and documents.” CQ does not specify which documents or if they are part of the public record. Without this information, we cannot assess whether presidential positions for an entire decade are comparable to those in other years.
CQ revised the description in 1987 and 1988 to “or by his authorized spokesmen, including spokesmen for the Office of Management and Budget [OMB]” (Congressional Quarterly Almanac 1987, 27‐C). The mention of OMB might refer to Statements of Administration Policy (SAP), but this is not explicitly stated.
From 1989 forward, CQ has used consistent language, “statements by the president or…authorized spokesmen,” to describe the method used to identify presidential positions. Yet the discussion in several studies suggests that SAPs may have become a recurring, if not routine, source of presidential positions. In 2006, for example, CQ implies that SAPs were the primary source: “In most cases, positions are based on official ‘statements of administration policy,’ issued by” OMB. The traditional method apparently was secondary: “Occasionally, a position is derived from speeches or news conferences” (Congressional Quarterly Almanac 2006, B‐8). The discussion in 2007 equates an SAP with a “willingness to take strong positions on legislation” (Congressional Quarterly Almanac 2007, B‐3). This analysis observes that President Bush issued many more SAPs in 2007 with Democrats in control of Congress than he did in 2003 when Republicans were in charge (Congressional Quarterly Almanac 2007, B‐3). The 2004 study observes that Bush issued fewer SAPs in 2004 than President Clinton did in 2000 (Congressional Quarterly Almanac 2004, B‐4). The 2013 study also mentions that President Obama issued SAPs (Congressional Quarterly Almanac 2013, B‐6‐7). Although this language suggests that CQ may have used SAPs in several studies since 2000, I found no documentation of when SAPs were used or the extent to which they were privileged over traditional sources. The section documenting the method used to identify the president’s position contains only the general language, “statements by the president or…authorized spokesmen.” The references to SAPs appear in the discussion and analysis of presidential support.
SAPs constitute an authoritative source in the public record to reliably identify presidential positions. Yet, SAPs are a relatively recent source. The earliest versions focused on budget issues and distribution was limited to key members of the appropriations committees. SAPs expressing general policy positions did not appear until the Reagan administration and were not publicly available on the White House website until the Clinton administration. SAPs created before the Clinton administration are difficult to find (Stuessy). Fortunately, Sam Kernell has collected SAPs from 1985 forward and deposited them at the American Presidency Project (http://www.presidency.ucsb.edu/saps.php). This archive provides information (e.g., bill number and date of issue) that might permit scholars to determine which votes in CQ’s Presidential Support studies relied on an SAP and perhaps identify votes that CQ may have omitted, at least back to 1985. However, it is not clear that presidential positions from SAPs are comparable to those based solely on public statements.
Thus, CQ has made a number of changes in the methods used to identify presidential positions. Although lack of consistency and transparency about these changes raises concerns about reliability, these are common challenges in conducting empirical research. President Trump’s erratic behavior, in contrast, presents a more fundamental threat to reliability that has not been seen before.
Trump’s Unique Threat to Reliability
Winning support on roll‐call votes is not an indication of presidential influence, although CQ has occasionally suggested this. The president may influence the outcome, but winning roll‐call votes results largely from shared partisan and ideological preferences (Edwards; Bond and Fleisher).
Even so, to interpret the outcome of a vote as a presidential victory requires at a minimum that: (1) the president’s positions must be observed independently of the outcome, (2) the expressed preferences must be generally honest and consistent, and (3) the president must make some contribution to the development of the policy and its movement through the legislative process. CQ has consistently observed the president’s position before the vote occurs so that it is credible to infer that the outcome agrees with the president’s position rather than the reverse. And although presidents, like all politicians, are notorious prevaricators, there has been “a presumption that politicians tell the truth most of the time, so the things they say should be treated with a basic level of respect” (Waldman). In view of this presumption, professional journalists at CQ respected that clear statements of what the president wants in the way of legislative action were sincere, and they reported how often members of Congress and the outcomes of votes supported the president’s position. Trump belies the presumption that presidential statements are generally honest and that he has facilitated movement of his preferred action through Congress.
Fundamentally, honesty is a general absence of demonstrably false statements. False statements from Trump, however, are routine—in his first 500 days in office, nearly 70 percent of his statements were false or mostly false (PolitiFact), an average of 6.5 per day (Kessler, Rizzo, and Kelly). False statements are neither valid nor reliable indicators of presidential preferences.
A statement expressing a preference for some legislative action may be sincere, but flip‐flops on legislative positions prevent members of Congress (and presidency scholars) from reliably determining what the president actually wants. Consider Trump’s actions on behalf of repealing the Affordable Care Act (Obamacare). Repealing Obamacare has been congressional Republicans’ top legislative priority since it was enacted in 2010. Trump expressed strong support for this Republican priority during the campaign and for bills introduced in the House in 2017. On the day that the House finally passed a bill to repeal and replace Obamacare, the president held an early signing ceremony in the Rose Garden, an event typically reserved for legislation that has passed both chambers. He bragged, “it’s a great plan” and took credit for the win, “Coming from a different world and only being a politician for a short period of time, how am I doing? Am I doing OK? …Hey, I’m president. Can you believe it?” (McCaskill). Yet after expressing confidence that the bill would pass the Senate, he undermined the effort a few weeks later, calling the House bill “mean.” Senate aides indicated that the flip‐flops resulted in growing indifference to Trump’s expressed preferences (Kaplan, Steinhauer, and Pear).
Furthermore, Trump does not understand what congressional leaders can and cannot do. Recall the ham‐handed effort to “persuade” Senator Lisa Murkowski (R‐AK) to vote for the Obamacare repeal: Interior Secretary Ryan Zinke called to threaten that the Trump administration would withdraw support for policies related to energy exploration in Alaska if she did not support repeal. Uncowed, Murkowski provided a key nay vote. Then she instructed the White House on the power of a committee chair by postponing indefinitely a nomination that the Interior Department badly needed approved. Denying the delay was retribution, a Murkowski spokeswoman attributed it to uncertainty of the Senate schedule (Hohmann). Or consider Trump’s Twitter barrage berating Senator Mitch McConnell (R‐KY) for failure to prevent the defections: “that should have been very easy to handle, whether it’s through the fact that can take away a committee chairmanship or do whatever you have to do” (quoted in Bolton). President Trump apparently is unaware that the job of Senate majority leader has been aptly compared to “herding cats” (Address by Howard H. Baker, Jr., July 14, 1998).
Another aspect of honesty is the assumption that the president contributed to and understands the policies on which he takes positions. Until Trump, there was no reason to question presidential involvement in the legislative arena. Edwards and Barrett presented evidence that presidents from Eisenhower to Clinton were actively engaged in the development of major polices. During unified government, about half of potentially significant bills were presidential as opposed to congressional initiatives. To be sure, Congress did not pass all presidential initiatives—69 percent of House bills and 65 percent of Senate bills became law (Edwards and Barrett, 124-29).
Because even majority presidents do not get everything they want from Congress, CQ’s interpretation crediting Trump with a “record success rate” in 2017—”winning” 100 percent of House votes and 98 percent of Senate votes—does not seem credible (valid). Interpreting the outcome on these votes as Trump victories assumes that he contributed to and understands basic provisions of the policies on which he expressed positions. Although Trump’s knowledge and involvement in developing legislation is not readily observable, the absence of nay positions is consistent with just endorsing policies already on the Republican agenda.
The proportion of yea and nay positions is at best indirect evidence of presidential involvement, however. A nay position does not always indicate opposition to a policy. For example, opposing a killer amendment or a procedural motion to table or recommit a bill indicates support for the policy. Nonetheless, a nay position normally indicates the president’s opposition. We would expect presidents to express fewer nay positions under unified government because the president’s party controls the agenda and can keep unacceptable bills and procedural threats off the floor. We might also expect fewer nay positions during the honeymoon year when Congress may be more deferential to the new president. Thus, if Trump were just endorsing policies developed by congressional leaders, we would expect him to express many fewer nay positions than did other majority presidents in their first year.
Trump is an extreme outlier in the near complete absence of nay positions (Table 1). On roll calls in CQ’s 2017 study, Trump expressed zero nay positions in the House and one in the Senate (the single nay was support for the nuclear option, a parliamentary move to invoke cloture with a simple majority and confirm Neil Gorsuch to the Supreme Court). The means are 38.2 percent nay in the House and 46.7 percent in the Senate.
Table 1. Percent of Nay Positions of Majority Presidents in Their Honeymoon Year
|
President |
House | N | Senate | N |
| Eisenhower | 41.4% | 29 | 59.1% | 44 |
| Kennedy | 27.3% | 55 | 54.8% | 115 |
| Johnson | 40.7% | 54 | 79.8% | 203 |
| Carter | 50.0% | 68 | 40.8% | 71 |
| Reagan | 48.4% | 95 | ||
| Clinton | 44.2% | 95 | 13.4% | 82 |
| G. W. Bush | 41.0% | 39 | 33.3% | 12 |
| Obama | 41.5% | 65 | 30.6% | 49 |
| Trump | 0.0% | 35 | 1.1% | 94 |
| Mean | 38.2% | 55 | 46.7% | 85 |
Majority only through May 26, 2001, when Senator James Jeffords’ (I‐VT) party switch gave Democrats control. Bush took positions on an additional 19 roll calls in 2001; 68.4 percent nay positions on votes when in minority.
The innovative use of Twitter is a highly visible tool in Donald Trump’s unique style of presidential politics. Trump is not the first president to tweet, but he is the first to put such a large share of his public statements in blocs of 140 (280) characters that he earned the title “Tweeter in Chief”. Should tweets be considered official public statements to identify presidential positions? The answer is not obvious. On one hand, the president’s tweets are presidential records that must be preserved under the Presidential Records Act as historical records (Laird). On the other hand, tweets do not have the same authority and permanence as other official presidential records. Positions expressed in tweets are not the product of a rigorous vetting process that official public statements and SAPs typically go through. For this unembarrassable president in particular, excessive grammatical and factual errors in his tweets raise doubts about professionalism and veracity. Furthermore, even if tweets are presidential records, a president who has so little respect for the rule of law can easily delete them. Trump routinely rips up letters and memos that are supposed to be retained. There are staff members assigned to tape the torn documents back together (Karni). But because Trump has sent many tweets from a private, insecure phone (Waddell), it is not clear that all of these deleted tweets could be retrieved. Yet, it is not tweets per se but rather the disregard for truth and accuracy that pose the greatest threat to reliability. Twitter just facilitates a more rapid and targeted dissemination of misinformation.
Reliability Check of Identifying Trump’s Positions in 2017
Although I cannot offer a definitive conclusion about whether tweets are an appropriate source to identify presidential positions, practices of professional observers of presidential behavior may offer some guidance about how tweets have been used, if at all. I found no indication that CQ used tweets in the 2017 Presidential Support study. Fortunately, another media organization has begun systematically observing presidential position taking.
FiveThirtyEight (http://fivethirtyeight.com/) started publishing “The Trump Score” (Bycoffe) in 2017. They report two indicators for each member of Congress. The Trump Score is analogous to CQ’s presidential support score—how often each member votes yea or nay in agreement with the president’s position. “Trump plus‐minus” is the difference between actual support and predicted support based on Trump’s vote margin in the member’s state or district. Because FiveThirtyEight is a “data journalism” website, the journalists there are more transparent in explaining the methods.
FiveThirtyEight’s general approach to identifying presidential positions is similar, but not identical, to CQ’s. They “look for a clear statement of support or opposition made by [Trump] or by someone on his behalf.” On judgment calls, they adopt “a conservative approach, focusing on cases in which the Trump administration has provided fairly explicit guidance about its position.”
However, FiveThirtyEight provides additional information that facilitates assessment of reliability. First, the description is explicit that the primary source for determining whether Trump has expressed a clear position is a SAP if one is available. The description links to Kernell to document that “[p]revious administrations have taken clear positions on bills by issuing ‘statements of administration policy’ through the [OMB].” Absent a SAP, they look to “the White House for the president’s position…and look for media reports about Trump’s stance.” Second, they provide links to the source of Trump’s position. Thus, we can check their work and decide if the case is appropriate for our research.
Most links lead to a SAP, the White House Press Office, or a media source (e.g., CNN, New York Times, Washington Post, The Hill, and others). The description does not mention Twitter specifically, but a few links lead to a tweet. Links to the White House Press Office are dead. I could verify the source and date from the error message (HTTP/1.0 404) but not the contents. Twenty‐six votes had no link. Twenty‐five were Senate votes on nominations. The methods description has a detailed discussion of how they code votes on nominations. One was a House vote to table Articles of Impeachment against President Trump. FiveThirtyEight identified three House votes and two Senate votes that occurred on January 5 and 13, 2017 before Trump was inaugurated. I exclude these votes from this analysis because we have only one president at a time, and that was Obama when these votes occurred.
CQ and FiveThirtyEight observing the same phenomenon—identifying Trump’s positions—provides an opportunity to check the intercoder reliability. Table 2 shows the overlap. Only 66 percent of House votes and 42 percent of Senate votes were on both lists. Although overlap is lower in the Senate, a closer look at the inconsistent cases reveals that the threat to reliability is mostly in identifying House votes.
Table 2. 2017 Presidential Position Votes from CQ Weekly & 538.com
|
Votes |
House Roll Calls | Senate Roll Calls | ||
| % | N | % | N | |
| On both lists | 66.0% | 35 | 41.6% | 42 |
| On CQ list not on 538 | 0.0% | 0 | 51.5% | 52 |
| On 538 list not on CQ | 34.0% | 18 | 6.9% | 7 |
| 100.0% | 53 | 100.0% | 101 |
Looking first at votes in CQ’s study that FiveThirtyEight omitted, we see that this type of inconsistency occurred exclusively in the Senate. Of the 52 CQ votes omitted from FiveThirtyEight’s list, 51 were nominations below the level of secretary. FiveThirtyEight includes only top cabinet‐level and Supreme Court nominations whereas CQ includes nominations to lower‐level cabinet positions, independent agencies, and lower federal courts. Including only high‐level nominations is a reasonable research decision, but success rates are not comparable to studies that include the lower‐level appointments.
The other CQ vote missing from FiveThirtyEight’s list was on the Fiscal 2018 Budget Resolution (HCONRES 71). CQ used the Motion to Proceed (Senate Vote 219) that occurred on October 17. The source of Trump’s position is unknown, but CQ’s guidelines indicate that it was an official statement expressed before the vote. FiveThirtyEight used the vote on adoption (Senate Vote 245). The source of Trump’s position was a tweet sent on October 19, 2017 at 3:54 a.m. before senators cast votes at 5:18 p.m. Compared to language in official presidential statements, the language in the tweet is juvenile and unprofessional: “Republicans are going for the big Budget approval today, first step toward massive tax cuts. I think we have the votes, but who knows?” To be fair, one can say only so much with a limit of 180 characters (this one has 134 characters). Trump’s position prevailed on both votes by a similarly close margin. Thus, although FiveThirtyEight’s analysis contains less than half of the Senate votes identified by CQ, the methods are sufficiently transparent that we can make adjustments (e.g., retrieve lower‐level nominations from CQ, decide whether to use one or both votes on the same issue) to improve reliability.
Other inconsistencies pose a more serious threat to reliability. FiveThirtyEight identified 18 House votes and 7 Senate votes that were not on CQ’s list. Especially troubling are three Senate votes and one House vote that Trump lost. Most important are three losses on votes to impose sanctions on Russia. The Senate passed a sanctions bill (S 722) by a vote of 97-2 on June 14. The source for Trump’s opposition is a New York Times story on June 13 (Sanger and Flegenheimer). The story does not quote Trump directly. It refers to an interview with Trump in 2016 (before the election) and quotes Secretary of State Rex Tillerson expressing reservations. In light of Trump’s public rebukes of this cabinet secretary, whether Tillerson’s view qualifies as a “clear statement of support or opposition…by someone on his behalf” is a judgment call. Nonetheless, the tone of the story is that this legislative action confronts Trump with a dilemma about whether to veto sanctions against Russia for interfering in the election. With such overwhelming bipartisan support, an all but certain veto override would be a humiliating defeat.
A broader sanctions bill that included sanctions on Russia (HR 3364) passed the House by a vote of 419-3 on July 25 and the Senate by a vote of 98-2 on July 27. The source of Trump’s position on this bill was the signing statement on August 2 expressing constitutional objections. Second‐hand media reports and signing statements after passage apparently did not provide adequate documentation for CQ to conclude that Trump expressed a clear preference before the vote occurred. Yet it seems evident that Trump opposed and lost all three votes on this key issue for his presidency. Failure to identify a position on sanctions in particular raises questions about CQ’s finding of no losses in the House.
FiveThirtyEight also included a Senate vote to disapprove a regulation on methane gas release that Trump lost. The source of Trump’s position was a White House press release regarding a House vote several months earlier. Both CQ and FiveThirtyEight included this House vote. Trump won the House vote, but the Motion to Proceed failed in the Senate. On the issue of disapproving the rule on methane gas regulation, CQ identified Trump’s victory on the House vote but overlooked the loss in the Senate. Because CQ found that Trump had a record‐tying success rate in the Senate, missing this loss, in addition to two probable losses on Russia sanctions, is especially troubling.
The remaining FiveThirtyEight votes missing from CQ’s study are all victories. FiveThirtyEight’s transparency including links to the source of the president’s position, however, permits an assessment of their judgment in identifying presidential positions. Four votes that link to a SAP seem to be obvious positions. Although CQ has not explained fully how they use SAPs, comments in several studies noted above suggest that CQ considers them an authoritative source. Why CQ omitted them is unknown.
Other votes on FiveThirtyEight’s list link to press statements and various media sources. The timing of several statements is problematical. The date of several statements is after the vote. Several others, however, link to a campaign statement made long before election. A campaign statement is the source of Trump’s position on votes on HJRES 43 to repeal of a rule requiring state and local governments to distribute federal funds to health centers that perform abortions. Trump won in the House (vote 99) on final passage. He also won in the Senate (vote 101), with Vice President Pence voting yea to break a tie. A vice president’s vote likely reflects a clear presidential position expressed before the vote, yet CQ did not include it. In general, however, interpreting the outcome of a roll‐call vote as a presidential win or loss is problematical if the position is announced after the vote occurs or before the position taker is president.
FiveThirtyEight also includes a vote to table Articles of Impeachment against President Trump. There is no link to a source for Trump’s position. Although it is obvious that Trump would support killing this premature (more than two‐thirds of Democrats voted with 100 percent of Republicans to table) resolution, CQ may include impeachment votes only if they can find an explicit statement. In the 1998 Presidential Support study, CQ included five impeachment votes with President Clinton’s position as nay (he lost on four). There were three votes to impeach Vice President Cheney in 2007 and two votes to impeach President Bush in 2008, but CQ did not list a presidential position for any of these.
Thus, FiveThirtyEight identified a number of votes on which Trump expressed a position that were missing from CQ’s 2017 study. Although the timing and source of some cases deviate from long‐standing practices of looking for clear statements before the vote occurs, several votes, including four losses, raise questions about CQ’s procedure for identifying presidential positions. FiveThirtyEight also used tweets to identify Trump’s position in a handful of cases, but there is no discussion justifying the use of tweets or whether they looked for multiple tweets expressing contradictory positions. Nonetheless, combining votes from both studies does not greatly change Trump’s success rate in 2017—he still has a record high score in the House (98.1 percent) but not in the Senate (95.0 percent) (see Table 3).
Table 3. 2017 Trump Success Rate
| Success Rate | N | Wins | |
| CQ House | 100.0% | 35 | 35 |
| 538 House | 94.4% | 18 | 17 |
| CQ+538 House | 98.1% | 53 | 52 |
| CQ Senate | 97.9% | 94 | 92 |
| 538 Senate | 89.8% | 49 | 44 |
| CQ+538 Senate | 95.0% | 101 | 96 |
Based on conflictual votes (at least 10 percent opposed). Success rates including 18 House votes and 7 Senate with CQ’s list.
Discussion
Students of presidential-congressional relations have long relied on data from CQ’s annual Presidential Support studies to assess presidential support and success on votes on which he expressed a clear position. President Trump’s innovative way of relating his preferences to Congress, however, raises concerns about the validity and reliability of roll‐call‐vote‐based measures. Beginning in 2017, FiveThirtyEight introduced The Trump Score to assess how often members of Congress vote with or against the president. This new source identifying presidential positions on roll‐call votes provides the first opportunity to measure intercoder reliability. It also provides the incentive to look more closely at the reliability of more than six decades of CQ’s Presidential Support studies.
CQ claims that the studies have used consistent methods over more than six decades. The general approach of searching all public statements to determine whether the president expressed a clear position prior to the vote appears to have been consistent. A comprehensive review, however, reveals that CQ has revised the sources used to identify presidential positions at several points in the time series. For example, from 1976 to 1986, the description added “public statements and documents.” In 1987 and 1988, the wording was revised to “spokesmen for the Office of Management and Budget.” Some studies in the 2000s confirm that SAPs were a source of presidential positions. Regrettably, the methods descriptions do not clarify what documents CQ used, whether “spokesmen for OMB” refers to SAPs, or exactly when and to what extent they used SAPs.
Although this review reveals inconsistencies in CQ’s Presidential Support studies over time, the limited documentation is not sufficient to determine precisely the extent to which they threaten reliability. The publication of FiveThirtyEight’s Trump Score in 2017 does provide an opportunity to measure the consistency of different researchers identifying presidential positions at one point in time. Only 66 percent of House votes and 42 percent of Senate votes appeared on both lists. The discrepancy in the Senate is almost exclusively due to FiveThirtyEight’s decision not to include nominations below the level of secretary while CQ has consistently included lower‐level nominations. Researchers can adjust for this discrepancy, so it does not present a major threat to reliability. However, 18 House votes and 7 Senate votes on FiveThirtyEight’s list that were not on CQ’s list are more troublesome, especially four losses that CQ omitted. FiveThirtyEight provided links to the source used to identify Trump’s position. This transparency allows researchers to evaluate whether the source is adequate to determine a clear presidential preference. We cannot make this determination for CQ’s studies.
FiveThirtyEight’s data, however, would be more useful for academic research if they were more transparent about methods. In particular, FiveThirtyEight reveals that they occasionally use tweets but provide no documentation about how and under what circumstances they relied on tweets. Such information might help clarify whether these presidential records are a legitimate source of official presidential positions.
Furthermore, they provide no documentation about whether they checked for later statements expressing a contrary position and how they resolved any inconsistencies discovered. A 2018 vote on Foreign Intelligence Surveillance Act amendments reauthorization illustrates the importance of checking for contradictions. The White House issued a SAP supporting passage on January 9, 2018. Both CQ and FiveThirtyEight recorded yea as the president’s position. At 7:33 a.m., January 11, 2018 (four hours before the House voted on the bill), Trump tweeted, “House votes on controversial FISA ACT today. This is the act that may have been used, with the help of the discredited and phony Dossier, to so badly surveil and abuse the Trump Campaign by the previous administration and others?” The Lawfare blog responded that “the president here seemed to be at least implicitly opposing reauthorization of 702—and doing so…when the outcome of that vote is uncertain” (Hennessey and Wittes). Although “implicit opposition” might not be adequate to impute a position under CQ’s stated criteria, there can be no doubt that the contradictory statement produces uncertainty about the president’s preference.
The analysis of presidential support and success on roll‐call votes should be interpreted in light of concerns about comparability to presidential positions identified using different criteria. Going forward, we may be able to leverage the additional transparency that FiveThirtyEight brings to the identification of presidential positions to evaluate whether to adjust which votes to use in academic research. Perhaps the competition will encourage CQ to be more transparent in describing their methods. In addition, the archive at the American Presidency Project is a resource that may permit the verification of votes tied to a SAP to improve the reliability of past CQ studies, at least since 1985. Nevertheless, the analysis here raises questions about whether presidential support scores and success rates in recent years are comparable to those in previous decades.
The greater and immediate threat to the study of presidential support and success on roll‐call votes, however, is a president whose erratic behavior is neither valid nor reliable. With such a large number of false statements and statements that contradict earlier positions, how can members of Congress or presidency scholars know with confidence what the president actually wants in the way of legislative action? Presidency scholars have long understood that support on roll‐call votes is not an indication of presidential influence. Yet, if we cannot reliably observe the president’s position, then the outcome of roll‐call votes is not even an indication of presidential success.