A Deep Learning Model to Predict Congressional Roll Call Votes from Legislative Texts

Machine Learning and Applications: An International Journal (MLAIJ) Vol.7, No.3/4, December 2020 DOI:10.5121/mlaij.2020.7402 15 A DEEP LEARNING MODEL TO PREDICT CONGRESSIONAL ROLL CALL VOTES FROM LEGISLATIVE TEXTS Jonathan Wayne Korn and Mark A. Newman Department of Data Science, Harrisburg University, Harrisburg, Pennsylvania ABSTRACT Developments in natural language processing (NLP) techniques, convolutional neural networks (CNNs), and long-shortterm memory networks (LSTMs) allow for a state-of-the-art automated system capable of predicting the status (pass/fail) of congressional roll call votes. The paper introduces a custom hybrid model labeled "Predict Text Classification Network" (PTCN), which inputs legislation and outputs a prediction of the document's classification (pass/fail). The convolutional layers and the LSTM layers automatically recognize features from the input data's latent space. The PTCN's custom architecture provides elements enabling adaptation to the input's variance from adjustment to the kernel weights over time. On the document level, the model reported an average evaluation of 67.32% using 10-fold crossvalidation. The results suggest that the model can recognize congressional voting behaviors from the associated legislation's language. Overall, the PTCN provides a solution with competitive performance to related systems targeting congressional roll call votes. KEYWORDS Deep Learning (DL), Convolutional Neural Networks (CNNs), Long-Short-Term Memory Networks (LSTMs), Natural Language Processing (NLP), Congressional Roll Call Votes 1. INTRODUCTION Predicting the status (pass/fail) of congressional roll call votes has been political scientists' goal for decades. There are patterns of congressional voting behavior captured in the legislative text, which has shown significance when predicting congressional votes' status. Understanding the future status of legislation provides vital insights into government and industry matters. Analyzing roll-call data allows insight into information detailing the legislation's vote status and can predict future votes [1]. Other approaches using quantitative roll call data and legislative text have shown success in the past, however only under certain conditions. Their success expresses limitations due to the dimensionality of the data and unforeseen conditions in Congress's complex social environment. For instance, using two separate datasets diminishes a model's flexibility to predict and adapt to the event space's changing conditions. Political environments are complex social networks that often create noisy data. The scale of topics that the government considers in legislation is a diverse subject matter, and the language is sparse. Most past approaches rely on an attempt to use extra dimensionality from both text and quantitative data. However, using extra dimensionality produces limitations, or predictive bottleneck, in approaches using text and quantitative congressional data. Many situations occur when congressional roll call data is not available or represents uncontrolled conditions creating Machine Learning and Applications: An International Journal (MLAIJ) Vol.7, No.3/4, December 2020 16 issues predicting the event. For instance, a pure expression of mixed but non-partisan support in the voting data could limit existing models' overall accuracy. A more robust model is required to address the current limitations expressed in prior models addressing the problem discussed above. Statistical modeling seeks to learn the joint probability function from words contained in the texts [2]. However, it is difficult to obtain this goal because of the "... curse of dimensionality" [2]. However, the rise of deep learning makes it possible for computer systems to recognize patterns in complex text representations. Advancements in natural language processing techniques allow text to convert into tokenized word vector spaces. The conversion provides the proper dimensions to embed the texts into different deep neural networks. A custom hybrid architecture provides the ability to input texts and provide accurate outputs from recognized patterns. Convolutional neural networks (CNNs) and longshort term memory neural networks (LSTMs) can recognize these intricate patterns within the data's dimensions. Each layer in the network provides benefits in recognizing temporal and spatial features from abstract data representations. Mainly, CNNs reflect successful results in identifying abstract data patterns mainly because of development in max-pooling layers. However, due to long lag periods from the data's complexity, the CNN architecture cannot alone capture the text's patterns. A primary reason for implementing LSTM layers in the model's architecture is to overcome the long lag time problem that occurs when processing high dimensional data. Overcoming long lag periods allows for the neural network's depth to continue, enabling minute features to be recognized. Combining CNN and LSTM algorithms provide an architecture capable of recognizing features in sparse word vector spaces over long periods, known as a Convolutional Long-Short Term Memory Neural Network (C-LSTM). Adaptable layers during kernel initialization help filter out the non-significant features from the inputs dimensional space. The adaptability in each layer of the network provides stability in the predictions and robustness against the dynamic social environment creating the data. The custom architecture ensures the network depth is suitable for accurate prediction from highly sparse text samples. The paper presents a custom deep learning solution to accomplish the binary classification of legislative texts. 1.1. Organizational Structure The remainder of the paper is organized into the following sections, including Section 2. Literature Review, Section 3. Methods and Materials, Section 4. Results, Section 5. Conclusion, and Section 6. Future Work. Section 3. Methods and Materials includes two sections including 3.1. The Data, and 3.2. The Deep Learning Approach. Section 3.1. includes Section 3.1.1. Preprocessing text and 3.1.2. Equal Distribution of Samples. Section 3.2. includes 3.2.1. The Custom PTCN Architecture and 3.2.2. The PTCN Modelling Process. 2. LITERATURE REVIEW A focus for quantitative political scientists is using roll-call data to understand legislative voting behaviors better. Few models have attempted to use supplementary data such as the text of legislation to understand congressional voting trends better. In 1991, research expressed that spatial positions captured in roll call data is stable and contains reliable features to recognize voting patterns [3]. The party discipline is present in the spatial Machine Learning and Applications: An International Journal (MLAIJ) Vol.7, No.3/4, December 2020 17 dimensions of roll-call votes of the legislator's ideal points captured in the data [4]. Spatial dimensions in the context of the domain refer to the probabilistic word occurrence contained across the text samples. Researchers utilized a method using Bayesian simulation models to capture the ideal points of the legislators. The approach allows researchers to ensure the beliefs incorporated into the inputs' dimensional space for roll call analysis [5]. The method enables researchers to handle the increasing complexity in higher-dimensional contexts [5]. Policy ideas are standard features in legislation that provide insight into legislators' behavior similar to quantitative roll call data [6]. In 2011, Gerrish and Blei introduced a probabilistic model capable of inferring a person's political position for specific topics [7]. The model focuses on capturing the individual representative's view on specific political interests from the text alone. The authors have used 12 years of congressional legislative data in their experiment to capture significant patterns. The patterns express the lawmakers' vote behavior and on which type of document [7]. Gerrish and Blei integrated the analysis of text into quantitative models such as the ideal point model with success inferring "... ideal points and bills' locations..." from roll call data leading to the prediction of legislative vote status (pass/fail) [8]. The integration of the bill texts into the ideal point model helped mitigate a limitation of only predicting on vote data alone, which may, at times, be inconsistent in its availability. The authors developed a supervised ideal point topic model capable of predicting pending bills using votes. It also is a method of exploring the connection between language and political support [8]. Other models developed by Gerrish and Blei to predict the "...inferred ideal points using different forms of regression on phrase counts." [8]. However, lead existing models cannot predict when mixed but non-partisan support is present in the data. Many existing models cannot expand beyond one-dimensional limitations, such as the ideal point model [8]. The authors also based their model's performance on the baseline that 85% of all votes are 'yea', limiting the model's performance results. The authors reported their ideal topic model predicted 89% of the votes with 64 topics, and their L2 model predicted 90% [8]. In sequential predictions, both their models predicted 87% and 88.1% accurate at predicting future votes, respectively [8]. Their study only attempts to understand a few topics with a one-dimensional political space, which creates a predictive bottleneck [8]. Interestingly, the ideal point model reflects representatives' preferences, constituency preferences, or any other feature indicating a preference for particular legislation [1]. In 2013, Spatio-temporal modeling expressed success in using text to predict congressional roll call votes with relative success to the ideal points model [9]. In 2015, ideal points were redefined as two characterizations, including word and vote choice [10]. These characterize the ideal points and the dimensions of the policy. The researchers utilize Sparse Factor Analysis to combine both votes and textual data to estimate the ideal points [10]. In 2016, Yang and others introduced hierarchical attention networks for document classification [11], the results outperformed previous models by a large margin, which can be indicated in the author's results using the Yelp 2013, Yelp 2014, Yelp 2015, IMBD review, yahoo Answer, and Amazon review datasets to test their algorithm and compare it to other methods [11]. On average, the HN-ATT performed with about a 70% accuracy rate across the tasks. They were generalizing the network to address multiple types of tasks that limit their ability to identify the text's critical features, such as the political ideology in the congressional legislative texts across spatial and temporal dimensions. A similar approach that uses multi-dimensional bill texts to predict roll calls' status is Kraft, Jain, and Rush's approach established in 2016. Using an embedding model with prepared bill texts, they competed with Gerrish and Blei's approach. Mainly the approach utilized ideal vectors rather than ideal points [12]. The approach relies on quantitative data that leads to the same predictive bottlenecks as prior models due to the complex event space. Machine Learning and Applications: An International Journal (MLAIJ) Vol.7, No.3/4, December 2020 18 In 2016, the Gov2Vec method expressed success in capturing opinions from legal documents. The text's transformation allows words to be embedded in a model to learn the representations of individuals' opinions [13]. In 2017 it became possible to use quantitative data and text to predict if a bill will become law. The researcher states that they always perform better than using text or content individually [14]. Interestingly, the author conducted three experiments, including "...text-only, text, and context, or context-only..." to test the predictive power of each type of model [14]. Nay's approach uses a language model that provides a prediction on the text's sentence level, providing a probability of the sentence contributing to the bill text's (pass/fail) status [14]. The data included a few performance measures and two data conditions spanning 14 years.[14]. Nay concluded that a text alone approach is fundamental for better results. For newer data, the use of bill text outperforms bill context-only. Context-only only outperforms bill text alone for older data [14]. The most important finding is that text adds predictive power to the model. However, the approach relies on two data sources, which creates a predictive bottleneck. As the most successful approach to date using text alone, the model can predict a 65% accuracy [14]. A hybrid C-LSTM Neural Networks architecture can help mitigate the predictive bottleneck in existing models by capturing more features from the bill texts alone. In statistical language modeling, the primary goal is learning. The main objective to learn is the "... joint probability function ..." of each sequence of words contained in the language [2]. However, there is difficulty in this type of task because of the curse of dimensionality. By learning distributed representations of language, the curse mitigates. The main issue is addressing the variance from the training data to the testing data. Past research discovered "... the similarities between words to obtain generalization from training sequences to new sequences..." [15] [16] [17] [18]. Much of this type of work is thanks to contributions by Schutze, 1993, where vectorspace representations for words can be learned based on the probability of the word co-occurring in documents [19]. Most of the experimental workings can be summed up from learning distributed feature vectors to represent their similarities between words, which is discussed in [15] [17] [20]. Past research has shown that the hybridization of CNNs and LSTMs tasked to solve text classification problems is successful [21]. The combination of the two different types of neural networks builds a C-LSTM [21]. In 2015, Zhou et al. introduced a successful CLSTM in sentence and document modeling [21]. The CNN extracts a sequence of high-level phrase representations, which are in-return fed through LSTM layers set to obtain the sentencerepresentations [21]. Overall, the hybrid model performed better than existing systems in classification tasks. The results indicated that the local features of phrases and the sentence's global/temporal semantics are recognizable by their model [21]. In 2019, a team of researchers took advantage of convolutional recurrent neural networks to tackle text classification tasks [22]. Their experiments showed that the method of using a C-LSTM achieves better success than other networks. 3. METHODS AND MATERIALS The following sections discuss key components to developing the PTCN and the associated results referred to in Section IV. The methodology is summarized below in Figure 1. Machine Learning and Applications: An International Journal (MLAIJ) Vol.7, No.3/4, December 2020 19 Fig. 1: Visualization of the methodology. 3.1. The Data The original data is extracted and organized from (https://www.Govtrack.us) consisting of legislative texts and associated quantitative roll call data. The original data contained samples from the year 2000 to 2019, including 3668 samples from the house and senate. Refer to Figure 2 for an example of original legislative texts. Not all the samples from the original population will be included in the experiment due to limiting factors as discussed below. Fig. 2: Legislative text samples. Machine Learning and Applications: An International Journal (MLAIJ) Vol.7, No.3/4, December 2020 20 3.1.1. Pre-Processing the Text A series of NLP techniques are required to supervise the custom C-LSTM training to classify the legislative text based on voting status. The text samples contain noise, which is in the form of stop words, uppercasing, special characters, NA values, punctuation, numbers, and whitespace. The removal of noise from the text samples helps mitigate the C-LSTM from recognizing patterns within the texts that create bias predictions. The C-LSTM is a deep learning algorithm that automatically extracts features from an input vector. Each text undergoes augmentation using the following conditions, including converting data types from character to a string, lower case conversion, stop-words removal using the “SMART” function, punctuation removal, number removal, white-space removal, and document stemming. The above augmentation resulted in Figure 3. Fig. 3: Sample of pre-processed Legislative texts. After pre-processing, the text goes through a tokenization and vectorization step resulting in each word, symbol, or any other character represented as a unique number. For instance, the word ‘bill’ is represented by the value of 3109 across all the documents. Note the text is limited to 10000 max features during the tokenization process. Refer to Figure 4 for an example of the vectorized vocabulary from the legislative documents. Fig. 4: Sample of tokenized and vectorized Vocabulary. The pre-processing of the text resulted in vectorized legislative documents, as seen in Figure 5. A method of padding is implemented to transform all the texts to the same length. The dimensions of the text are converted into a tokenized, vectorized, and padded format with a max length of 10,000. Machine Learning and Applications: An International Journal (MLAIJ) Vol.7, No.3/4, December 2020 21 Fig. 5: Sample of tokenized, vectorized, and padded text. 3.1.2. Equal Distribution of Samples The labeled legislative text samples are balanced into an equal distribution of each voting status to reduce bias. It is essential to mitigate bias in the data. Deep neural networks transform inputs over extensive credit assignment pathways (CAPs). The more complicated the dimensions of an input, then the more complex the CAPs. Each vote’s status was captured by determining the number of “yea” or “Aye” responses reached for each legislation. Each translates into a value of 1, representing a vote to pass the document. All other responses are considered a vote against the document, labeled as a value of 0. Interestingly, the number of documents labeled one is greater than 0 labeled documents. Note that some votes require a special condition to pass, which is more than 2/3 of the votes. The study ignores the special voting condition because it only focuses on an equal distribution of (pass/fail) text representations. An equal distribution of the voting statuses in the sample is necessary to mitigate bias in the network for either classification. The text samples are randomly selected from each class to represent an equal distribution, which reduces the original number of samples. Each class of vote is represented equally by 98 randomly selected legislative texts. 98 is the maximum number of 0 labeled samples available in the data due to the behavior of congress. Each text is a max length of 10,000 features. It should be noted that most of the legislative texts are close or higher to the maximum number of features. Custom features in the PTCN’s architecture are implemented to deal with the small number of samples during training as discussed below. 3.2. The Deep Learning Approach The following section discusses the custom deep learning architecture and modelling process implemented in the study. 3.2.1. The Custom PTCN Architecture The sequential deep learning model is a stack of different layers set with several parameters, including dropout rate, hidden convolutional nodes, LSTM hidden nodes, L1 regularization rate, L2 regularization rate, batch size, input max length, max features, embedding dimensions, leaky Relu rate, kernel size, epochs, max-pooling size, learning rate, and validation split. In Figure 6, an example of the PTCN models architecture: Machine Learning and Applications: An International Journal (MLAIJ) Vol.7, No.3/4, December 2020 22 Fig. 6: PTCN architecture The first layer in the model is the embedding layer, which embeds the tokenized words. At the next layer, the inputs pipe into a 1-dimensional convolutional layer with only a 4th of the set convolutional hidden nodes. The inputs are representations of lengthy and highly sparse word vector spaces, making the model weight's critically important. The model requires a method to adjust to the high variance between samples due to the low number of samples. The control of the kernels will help the model steadily learn significant features contained in the text. Throughout the convolutions, the model can explore deeper to recognize more significant features. A variance scaling kernel initializer provides an “initializer capable of adapting its scale to the shape of weights. ” [23]. Variance scaling is a kernel initialization strategy that encodes an object without knowing the shape of a variable [23]. The Convolutional layers’ initializer makes the deep network adapt to the input weights. It is important to regularize the kernels when utilizing an adaptive initializer. Setting a duel regularization technique that deploys L1 and L2 regularization helps mitigate high fluctuations while batching samples through the layers. L1 is a Lasso Regression (LR). L2 is Ridge Regression. The main difference between the two methods is the penalty term. The layer includes strides set at 1L through the convolutions are an “...an integer or list of a single integer, specifying the stride length of the convolution” [24]. The convolutional layer is activated using a Leaky Relu function. The leaky Relu function allows for a “... small gradient when the unit is not active ...”, providing “... slightly higher flexibility to the model than traditional Relu. ” [25]. Machine Learning and Applications: An International Journal (MLAIJ) Vol.7, No.3/4, December 2020 23 The first convolutional layer extracts lower level features from the inputs due to the decrease in the hidden nodes. The reduction of the number of transformations provides control to the adaptable features initializing the weights. The second layer pipes the inputs through another 1dimensional convolutional layer with the same parameters set, except the number of hidden nodes set to 32. Increasing the number of hidden nodes provides more transformations extracting higher-level features from the inputs. The second convolutional layer is activated using the leaky Relu function. The next layer in the stack is another convolutional layer set at half the set number of hidden nodes. Reducing the number of nodes and following with a max-pooling layer helps mitigate overfitting during training. In the study, the max-pooling layer is set to 4. The following are two more layers of 1-dimensional convolutional layers set to half the set hidden nodes and a 4th of the set hidden nodes. All parameters are set the same as the prior convolutional layers. A second max-pooling layer, batch normalization, and dropout layer help mitigate overfitting further. The next layer is an LSTM layer set at 32 hidden nodes. Variance Scaling kernel initializers and L1 and L2 kernel regularizers control the model's exploration of the feature space. The LSTM layer is activated using a Leaky Relu function. A dropout layer of 0.5 is in the stack before the output layer. The output layer is activated using a sigmoid function. The model compiles using a loss function of binary cross-entropy. The PTCN uses a stochastic gradient descent (SGD) optimizer. The hypertuning sessions determine the learning rate. 3.2.2. The PTCN Modelling Process To ensure the model is producing the best results, the parameters of the PTCN are hyper-tuned. The data is split 80/20% for training and validation of the model during the hyper-tuning sessions. The hyper-tuning model’s performance evaluates a random selection of 100 samples. The best parameters are captured and set for the final model training. Once the final parameters are identified, the PTCN is trained for a final training session. The final training session of the PTCN uses the best parameters identified during the hyper-tuning sessions, and 10-fold crossvalidation is implemented to evaluate the model’s performance. 4. RESULTS After implementing the model using 10-fold cross-validation, the PTCN Model averaged 67.32% evaluation accuracy with a standard deviation of 9.11. The best model performed at 76.32% evaluation on fold 6, as depicted below in Figure 7. Fig. 7: Sample of PTCN training and validation performance including early stopping.