Plan ¶
- Setting up Twitter API $\checkmark$
- Gathering tweets $\checkmark$
- Cleaning the tweets $\checkmark$
- Creating word-graph $\checkmark$
- Exploring small-world properties $\checkmark$
Setting up an API¶
- Needs accreditation from Twitter: need to be a developer
- Using the previous knowledge from the dataexp course
- Authentication and gathering is done with the $\texttt{tweepy}$ package
Properties of the tweets ¶
In [141]:
len(tweets)
Out[141]:
In [142]:
tweets[0].keys()
Out[142]:
The ratio of tweets where there was no need for a 'full_text' option:
In [291]:
texts = []
cnt = 1
for i, tweet in enumerate(tweets):
try:
texts.append(tweet['extended_tweet']['full_text'])
except KeyError:
texts.append(tweet['text'])
cnt += 1
print(f"{cnt/i}")
DataFrame of the tweets ¶
In [147]:
df.head()
Out[147]:
Location of the tweets ¶
Two dots denote the bounding box ¶
In [148]:
mymap = folium.Map(location=[52.809865,-2.118092],zoom_start=5.4,tiles='cartodbpositron')
for i in range(1000):
marker = folium.Marker(location=df.locs[i][0],popup=df.text[i])
marker.add_to(mymap)
folium.Popup(parse_html=True)
mymap
Out[148]:
Cleaning from unnecessary things ¶
In [150]:
print(f"Text of tweet no.8: \n\n{df.text[8]}")
In [151]:
clean = re.sub(r'http\S+', '', df.text[8])
print(f"\nAfter cleaning:\n\n{clean}")
In [152]:
clean_tknzd = [snow.stem(word) for word in re.findall('\w+',clean.lower())]
print(f"After tokenizing:\n\n{clean_tknzd}")
Network of tweets ¶
Tool: $\texttt{MultiGraph}$ by $\texttt{netwotrkx}$,
Connection = same tweet $\rightarrow$ the more tweet = greater weight,
Paralell edges possible, but no self-loops.
In [287]:
print(f"Num of nodes and edges: {len(g.nodes), len(g.edges())}")
print(f"Largest connected component consists of {lcc} nodes, which is {np.round(lcc/len(g.nodes()), 3)} of the all")
In [160]:
plotly.offline.init_notebook_mode(connected=True)
plotly.offline.iplot(fig1, filename='degdist')
The words with the highest degree:
In [164]:
plotly.offline.init_notebook_mode(connected=True)
plotly.offline.iplot(fig2, filename='mostcommnodes')
Below you can see the 20 most common occurrences ('most numerous paralell edge').
In [168]:
plotly.offline.init_notebook_mode(connected=True)
plotly.offline.iplot(fig, filename='simple-3d-scatter')
In [292]:
print(f"Num of nodes and edges in the multie, non-w: {gnodes}, {gedges}")
print(f"Num of nodes and edges in the non-m, weighted: {G2nodes}, {G2edges}")
print(f"Ratio between the nodes and edges: {G2nodes/gnodes}, {G2edges/gedges}")
In [263]:
plotly.offline.init_notebook_mode(connected=True)
plotly.offline.iplot(fig, filename='degdistnew')
In [200]:
plotly.offline.init_notebook_mode(connected=True)
plotly.offline.iplot(fig10, filename='betw')
Communities ¶
Community detection with Greedy Modularity algorithm:
$$ {\displaystyle Q={\frac {1}{2m}}\sum \limits _{ij}{\bigg [}A_{ij}-{\frac {k_{i}k_{j}}{2m}}{\bigg ]}\delta (c_{i},c_{j}),} $$where
- $A_{ij}$ edge weight between node $i,j$,
- $k_{i}$ sum of weights of the edges of $i$,
- $m$ normalization due to weights (sum of all weights),
- $c_{i}$ and $c_{j}$ are the communities of node $i$ and $j$.
Goal is to minimize the modularity.
In [294]:
plotly.offline.init_notebook_mode(connected=True)
plotly.offline.iplot(fig10, filename='betw')
In [226]:
print(ones_twos)
In [230]:
for i in c:
rtz = len(list(i))
if 5 <= rtz <=21 :
print(rtz, list(i)[:5])
In [233]:
for i in c[:10]:
rtz = len(list(i))
print(rtz, list(i)[:10])
In [235]:
print(list(c[5]))
In [261]:
szavak = ["meg", "vagy", "van", "vagyok", "vannak", "vagytok", "volt", "már", "kell",
"még", "és", "mint", "azt", "az", "akkor", "sem", "lehet", "mert", "minden", "olyan",
"szerint", "pedig", "ezt", "ez", "így", "után", "úgy", "nagy", "fel", "majd", "két",
"nem", "nagyon", "aki", "akik", "akit", "kit", "kik", "most", "több", "lesz", "itt",
"magyar", "ami", "amik", "amit", "mit", "első", "között", "amely", "hanem", "nincs",
"más", "illetve", "alatt", "egyik", "volna", "arra", "kft", "ilyen", "azonban"] #deleted "a"
In [82]:
not_in = []
for szó in szavak:
stemd = snow.stem(szó)
if g.has_node(stemd):
print(szó, stemd)
print(list(G2.edges(stemd))[:7])
print()
else:
not_in.append(stemd)
In [80]:
for i in sorted(c):
for cou, szó in enumerate(szavak):
stemd = snow.stem(szó)
if stemd in i:
print(cou, stemd, len(i))
Conclusions ¶
- Managed to gather tweets
- Familiarized myself with NLP
- Created word graph
- Showed small-world effect
- Communities found different languanges
