1. To convert the unicode string into a normal string format use the string.decode("utf-8") method. For example if string extracted from the webpage is assigned to the variable called extracted_string. the decoded string is extracted_string_decoded = extracted_string.decode("utf-8")

2.For this one, I would reccomend using a regex expression. The following code should work. It will give you a list of the "p" tags that have the "a" tag in it.

_texts_w_a = []

For _text in a_text:

if _text.find(b'' ) != -1:

_texts_w_a.append(_text)

Note: putting b before a string as I have done, allows python to understand you are passing it a byte as opposed to a regular string. This allows you to manipulate strings which are encoded.

Hope this helps

from bs4 import BeautifulSoup

import requests

import re

html = requests.get("http://thehill.com/blogs/blog-briefing-room/365407-sean-diddy-combs-wants-to-buy-carolina-panthers-and-sign-kaepernick").content

#1 Recoding

unicode_str = html.decode("utf8")

encoded_str = unicode_str.encode("ascii",'ignore')

news_soup = BeautifulSoup(encoded_str, "html.parser")

a_text = news_soup.find_all('p')

#2 Removing

y=[re.sub(r'',r'',str(a)) for a in a_text]

I recently used BeautifulSoup in Python to scrape a large data set of website URLs and the following tutorial helped me through it:

https://medium.freecodecamp.org/how-to-scrape-websites-with-python-and-beautifulsoup-5946935d93fe