Building the Model

Now, let us dive straight in and build our model. We use the following Python libraries to build the model:

• Requests
• Beautiful Soup
• Pattern

Step 1: Create a list of the news section URL of the component companies

We identify the component companies of the NIFTY Pharma index, and create a dictionary in python which contains the company names as the keys, while the dictionary values comprise the respective company abbreviation used by the financial portal site to form the news section URL. Using this dictionary we create a python list of the news section URLs for the all components companies.

import csv import time import requests from bs4 import BeautifulSoup from pattern.en import ngrams   Base_url = "http://www.moneycontrol.com"   # Build a dictionary of companies and their abbreviated names companies = {'cadilahealthcare':'CHC','piramalenterprises':'PH05', 'glenmarkpharma':'GP08','glaxosmithklinepharmaceuticals':'GSK', 'sunpharmaceuticalindustries':'SPI','lupinlaboratories':'LL', 'cipla':'C','aurobindopharma':'AP', 'drreddyslaboratories':'DRL','divislaboratories':'DL03'} # Create a list of the news section urls of the respective companies url_list = ['http://www.moneycontrol.com/company-article/{}/news/{}#{}'.format(k,v,v) for k,v in companies.iteritems()] print url_list  

Step 2: Extract the relevant news articles web-links from the company’s news section page

Using the Python list of the news section URLs, we run a Python For loop which pings the portal with every URL in our Python list. We use the requests.get function from the Python requests library (which is a simple HTTP library). The requests module allows you to send HTTP/1.1 requests. One can add headers, form data, multipart files, and parameters with simple Python dictionaries, and also access the response data in the same way.

The text of the response object is then applied to create a Beautiful Soup object. Beautiful Soup is a Python library for pulling data out of HTML and XML files. It works with a given parser to provide for ways of navigating, searching, and modifying the parse tree.

HTML parsing basically means taking in the HTML code and extracting relevant information like the title of the page, paragraphs in the page, headings, links, bold text etc.

The news section webpage on the financial portal site contains 20 news articles per page. We target only the first page of the news section, and our objective is to extract the links for all the news articles that appear on the first page using the parsed HTML. We inspect the HTML, and use the find_all method in the code to search for a tag that has the CSS class name as “arial11_summ”. This enables us to extract all the 20 web-links.

Fortunes of the R&D intensive Indian Pharma sector are driven by sales in the US market and by approvals/rejections of new drugs by US Food and Drug Administration (USFDA). Hence, we will select only those news articles pertaining to the US Food and Drug Administration (USFDA) and the US market. Using keywords like “US”, “USA”, and “USFDA” in a If statement which is nested within the Python For Loop, we get us our final list of all the relevant news articles.

# Create an empty list which will contain the selected news articles List_of_links = []   # Extract the relevant news articles weblinks from the news section of selected companies for urls in url_list: html = requests.get(urls) soup = BeautifulSoup(html.text,'html.parser') # Create a BeautifulSoup object   # Retrieve a list of all the links and the titles for the respective links word1,word2,word3 = "US","USA","USFDA" sub_links = soup.find_all('a', class_='arial11_summ') for links in sub_links: sp = BeautifulSoup(str(links),'html.parser') # first convert into a string tag = sp.a if word1 in tag['title'] or word2 in tag['title'] or word3 in tag['title']: category_links = Base_url + tag["href"] List_of_links.append(category_links) time.sleep(3) # Print the select list of news articles weblinks #for p in List_of_links: print p

Step 3: Remove the duplicate news articles based on news title

It may happen that the financial portal publishes important news articles pertaining to the overall pharma sector on every pharma company’s news section webpage. Hence, it becomes necessary to weed out the duplicate news articles that appear in our Python list before we run our sentiment analysis model. We call the set function on our Python list which we generated in Step 2 to give us a list with no duplicate news articles.

# Remove the duplicate news articles based on News Title unique_links = list(set(List_of_links)) for q in unique_links: print q   # Create a dictionary of positive/negative words related to the Pharma Sector reader = csv.reader(open('dict.csv', 'r')) pharma_dict = dict((rows[0],rows[1]) for rows in reader)   # Creating an empty list which will be filled later with news article links, and Polarity values (pos/neg) df =[] print df  

Step 4: Extract the main text from the selected news articles

In this step we run a Python For Loop and for every news article URL, we call the requests.get() on the URL, and then convert the text of response object into a Beautiful Soup object. Finally, we extract the main text using the find and get_text methods from the Beautiful Soup module.

# Open the choosen news articles and extract the main text for selected_links in unique_links: results_url = selected_links #print results_url results = requests.get(results_url) results_text = BeautifulSoup(results.text) extract_text = results_text.find(class_='arti_cont') final_text = extract_text.get_text()

Step 5: Pre-processing the extracted text

We will use the n-grams function from the Pattern module to pre-process our extracted text. The ngrams() function returns a list of n-grams (i.e., tuples of n successive words) from the given string. Since we are building a simple model, we use a value of one for the n argument in the n-grams function. The Pattern module contains other useful functions for pre-processing like parse, tokenize, tag etc. which can be explored to conduct an in-depth analysis.

# Pre-processing the extracted text using ngrams function from the pattern package final_text1 = ngrams(final_text, n=1, punctuation=".,;:!?()[]{}`''\"@#$^&*+-|=~_", continuous=False) #print final_text1

Step 6: Compute the Sentiment analysis score using a simple dictionary approach

To compute the overall polarity of a news article we use the dictionary method. In this approach a list of positive/negative words help determine the polarity of a given text. This dictionary is created using the words that are specific to the Pharma sector. The code checks for positive/negative matching words from the dictionary with the processed text from the news article.

# Checking if any of the words in the news article text matches with the words in the Pharma dictionary(pos/neg) new_dict = {} positive_score,negative_score = 0,0 for x in final_text1: if pharma_dict.has_key(x[0]): new_dict[x[0]] = pharma_dict[x[0]] positive_list = [] ; negative_list = []; for key, value in new_dict.items(): if value == 'positive': positive_list.append(key) if value == 'negative': negative_list.append(key) # Compute the positive score, the negative score for each news articles positive_score = len(positive_list) ; negative_score = len(negative_list); #print positive_list ; print negative_list ; #print positive_score ; print negative_score;

Step 7: Create a Python list of model output

The final output from the model is populated in a Python list. The list contains the URL, positive score and the negative score for each of the selected news articles on which we conducted sentiment analysis.

# Appending the empty list to create the final Sentiment analysis output var_nos = [results_url,positive_score, negative_score] df.append(var_nos) # Print the final list of the Sentiment Analysis for item in df: print(item)

Final Output

📷

Step 8: Plot NIFTY vs NIFTY Pharma returns

Shown below is a plot of NIFTY vs NIFTY Pharma for the months of October-November 2016. In our NIFTY Pharma plot we have drawn arrows highlighting some of the press releases on which we ran our sentiment analysis model. The impact of the uncertainty regarding the US Presidential election results, and the negative news for the Indian Pharma sector emanating from the US is clearly visible on NIFTY Pharma as it fell substantially from the highs made in late October’2016. Thus, our attempt to gauge the direction of the Pharma Index using the Sentiment analysis model in Python programming language is giving us accurate results (more or less)

Link: Article Forecasting corporate financial performance using sentiment ...