\n", "\n", "\n", "
![\"tw_api\"](\"tw_dev_app.png\")
\n",
" \n",
"\n",
"\n",
" \n",
"
\n",
"
"
],
"text/plain": [
" id date \\\n",
"0 1181300835122327552 Mon Oct 07 20:10:41 +0000 2019 \n",
"1 1181300835239759872 Mon Oct 07 20:10:41 +0000 2019 \n",
"2 1181300835348815874 Mon Oct 07 20:10:41 +0000 2019 \n",
"3 1181300836519075841 Mon Oct 07 20:10:41 +0000 2019 \n",
"4 1181300838846926848 Mon Oct 07 20:10:42 +0000 2019 \n",
"\n",
" source likes retweets \\\n",
"0 "
],
"text/plain": [
"| \n", " | id | \n", "date | \n", "source | \n", "likes | \n", "retweets | \n", "name | \n", "locs | \n", "text | \n", "len | \n", "
|---|---|---|---|---|---|---|---|---|---|
| 0 | \n", "1181300835122327552 | \n", "Mon Oct 07 20:10:41 +0000 2019 | \n", "<a href=\"http://twitter.com/download/android\" ... | \n", "0 | \n", "0 | \n", "Peter Dudley | \n", "[[51.543815, 0.010398], [51.626165, 0.010398],... | \n", "Side saw good chance thwarted by last ditch ta... | \n", "147 | \n", "
| 1 | \n", "1181300835239759872 | \n", "Mon Oct 07 20:10:41 +0000 2019 | \n", "<a href=\"http://twitter.com/download/iphone\" r... | \n", "0 | \n", "0 | \n", "Damian Wawrzyniak | \n", "[[52.603205, -0.202653], [52.620535, -0.202653... | \n", "@KarenWhiteFood @tweetertucker @AdamHandling I... | \n", "322 | \n", "
| 2 | \n", "1181300835348815874 | \n", "Mon Oct 07 20:10:41 +0000 2019 | \n", "<a href=\"http://twitter.com/download/iphone\" r... | \n", "0 | \n", "0 | \n", "deli Muru | \n", "[[54.543241, -6.036116], [54.648497, -6.036116... | \n", "@maggimccvil Congratulations Caroline! ⭐️ | \n", "41 | \n", "
| 3 | \n", "1181300836519075841 | \n", "Mon Oct 07 20:10:41 +0000 2019 | \n", "<a href=\"http://twitter.com/download/android\" ... | \n", "0 | \n", "0 | \n", "Jordan Carroll | \n", "[[53.438332, -3.058666], [53.501369, -3.058666... | \n", "@Eric_Toffee1878 Original mate. Disturbing but... | \n", "57 | \n", "
| 4 | \n", "1181300838846926848 | \n", "Mon Oct 07 20:10:42 +0000 2019 | \n", "<a href=\"http://twitter.com/download/iphone\" r... | \n", "0 | \n", "0 | \n", "sarah guerra | \n", "[[51.417277, -0.259465], [51.486036, -0.259465... | \n", "You were great | \n", "14 | \n", "
\n",
" \n",
" \n",
" \n",
" \n",
"
"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"plotly.offline.init_notebook_mode(connected=True)\n",
"plotly.offline.iplot(fig2, filename='mostcommnodes')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The figure above shows the most common words (words with the highest degree) with blue bars. The measure for these are their relative occurrence which is the number of edges connecting to a word divided by the total number of edges. Red lines are those, which words originate from [this website](http://www.cs.cmu.edu/~cburch/words/top.html) and are in accordance with [Zipf's law](https://en.wikipedia.org/wiki/Zipf%27s_law). The measure here is the occurrence in a million words. What is interesting to see here is -- despite the fact that literature is very different for the sets -- 15 out of the 20 words match, and often thier positions, too. Maybe if we were to measure the 50 most frequent word, the relative mismatche would decrease. It is a great checkpoint to make sure, our measurement is based on real world data. "
]
},
{
"cell_type": "code",
"execution_count": 165,
"metadata": {
"ExecuteTime": {
"end_time": "2019-11-19T14:28:53.709487Z",
"start_time": "2019-11-19T14:28:49.601429Z"
},
"slideshow": {
"slide_type": "skip"
}
},
"outputs": [],
"source": [
"edgs = list(g.edges())\n",
"\n",
"edgs_d = dict(Counter(edgs))\n",
"\n",
"sorted_edgs = sorted(edgs_d.items(), key=operator.itemgetter(1), reverse=True)"
]
},
{
"cell_type": "code",
"execution_count": 166,
"metadata": {
"ExecuteTime": {
"end_time": "2019-11-19T14:28:53.811981Z",
"start_time": "2019-11-19T14:28:53.713727Z"
},
"slideshow": {
"slide_type": "skip"
}
},
"outputs": [],
"source": [
"n = 20\n",
"\n",
"toplot = [[e[0], e[1]] for c, e in enumerate(sorted_edgs) if c<30]\n",
"nums = [100*z[1]/len(edgs) for z in toplot]\n",
"labels = [str(z[0][0]+' - '+ z[0][1]) for z in toplot]"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"Below you can see the 20 most common occurrences ('most numerous paralell edge')."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"ExecuteTime": {
"end_time": "2019-11-19T14:28:54.143050Z",
"start_time": "2019-11-19T14:28:53.815554Z"
},
"scrolled": false,
"slideshow": {
"slide_type": "skip"
}
},
"outputs": [],
"source": [
"\n",
"fig = go.Figure(go.Bar(x = labels,y = nums, name = 'Occurrence'), layout=layout )\n",
"\n",
"fig.update_xaxes(title_text = \"Connection pairs\", tickangle=315)\n",
"fig.update_yaxes(title_text = \"% of all the connections\")\n",
"fig.update_layout(title_text=\"Relative occurrence of the most common edges\", font=dict(\n",
" family='Courier New, monospace',\n",
" size=14,\n",
" color='black'\n",
" ))"
]
},
{
"cell_type": "code",
"execution_count": 168,
"metadata": {
"ExecuteTime": {
"end_time": "2019-11-19T14:28:54.369801Z",
"start_time": "2019-11-19T14:28:54.147037Z"
},
"scrolled": false,
"slideshow": {
"slide_type": "fragment"
}
},
"outputs": [
{
"data": {
"text/html": [
" \n",
" "
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.plotly.v1+json": {
"config": {
"linkText": "Export to plot.ly",
"plotlyServerURL": "https://plot.ly",
"showLink": false
},
"data": [
{
"name": "Occurrence",
"type": "bar",
"x": [
"to - the",
"of - the",
"and - the",
"the - a",
"i - the",
"the - in",
"to - a",
"to - i",
"and - to",
"it - the",
"i - a",
"for - the",
"is - the",
"and - a",
"you - the",
"to - it",
"of - a",
"to - you",
"and - i",
"to - of",
"i - it",
"it - a",
"the - on",
"to - in",
"that - the",
"a - in",
"you - a",
"to - for",
"and - of",
"for - a"
],
"y": [
0.17705228969782566,
0.1335221551176907,
0.13275735041057266,
0.1318650782522683,
0.12103034490142942,
0.11656898410990754,
0.11070548135533592,
0.09840487231585415,
0.09668406172483857,
0.09139416250060547,
0.08693280170908359,
0.08049569542417344,
0.08049569542417344,
0.08030449424739393,
0.07654420443739691,
0.07335751815773842,
0.07055323423163895,
0.06908735854299605,
0.06857748873825069,
0.06781268403113265,
0.06583693853774439,
0.06507213383062635,
0.06379745931876295,
0.0632875895140176,
0.06296892088605174,
0.06099317539266348,
0.05627687969876892,
0.05474727028453284,
0.05449233538216017,
0.05340886204707628
]
}
],
"layout": {
"autosize": false,
"bargroupgap": 0.3,
"font": {
"color": "black",
"family": "Courier New, monospace",
"size": 14
},
"height": 480,
"margin": {
"b": 100,
"l": 50,
"pad": 4,
"r": 50,
"t": 100
},
"template": {
"data": {
"bar": [
{
"error_x": {
"color": "#2a3f5f"
},
"error_y": {
"color": "#2a3f5f"
},
"marker": {
"line": {
"color": "#E5ECF6",
"width": 0.5
}
},
"type": "bar"
}
],
"barpolar": [
{
"marker": {
"line": {
"color": "#E5ECF6",
"width": 0.5
}
},
"type": "barpolar"
}
],
"carpet": [
{
"aaxis": {
"endlinecolor": "#2a3f5f",
"gridcolor": "white",
"linecolor": "white",
"minorgridcolor": "white",
"startlinecolor": "#2a3f5f"
},
"baxis": {
"endlinecolor": "#2a3f5f",
"gridcolor": "white",
"linecolor": "white",
"minorgridcolor": "white",
"startlinecolor": "#2a3f5f"
},
"type": "carpet"
}
],
"choropleth": [
{
"colorbar": {
"outlinewidth": 0,
"ticks": ""
},
"type": "choropleth"
}
],
"contour": [
{
"colorbar": {
"outlinewidth": 0,
"ticks": ""
},
"colorscale": [
[
0,
"#0d0887"
],
[
0.1111111111111111,
"#46039f"
],
[
0.2222222222222222,
"#7201a8"
],
[
0.3333333333333333,
"#9c179e"
],
[
0.4444444444444444,
"#bd3786"
],
[
0.5555555555555556,
"#d8576b"
],
[
0.6666666666666666,
"#ed7953"
],
[
0.7777777777777778,
"#fb9f3a"
],
[
0.8888888888888888,
"#fdca26"
],
[
1,
"#f0f921"
]
],
"type": "contour"
}
],
"contourcarpet": [
{
"colorbar": {
"outlinewidth": 0,
"ticks": ""
},
"type": "contourcarpet"
}
],
"heatmap": [
{
"colorbar": {
"outlinewidth": 0,
"ticks": ""
},
"colorscale": [
[
0,
"#0d0887"
],
[
0.1111111111111111,
"#46039f"
],
[
0.2222222222222222,
"#7201a8"
],
[
0.3333333333333333,
"#9c179e"
],
[
0.4444444444444444,
"#bd3786"
],
[
0.5555555555555556,
"#d8576b"
],
[
0.6666666666666666,
"#ed7953"
],
[
0.7777777777777778,
"#fb9f3a"
],
[
0.8888888888888888,
"#fdca26"
],
[
1,
"#f0f921"
]
],
"type": "heatmap"
}
],
"heatmapgl": [
{
"colorbar": {
"outlinewidth": 0,
"ticks": ""
},
"colorscale": [
[
0,
"#0d0887"
],
[
0.1111111111111111,
"#46039f"
],
[
0.2222222222222222,
"#7201a8"
],
[
0.3333333333333333,
"#9c179e"
],
[
0.4444444444444444,
"#bd3786"
],
[
0.5555555555555556,
"#d8576b"
],
[
0.6666666666666666,
"#ed7953"
],
[
0.7777777777777778,
"#fb9f3a"
],
[
0.8888888888888888,
"#fdca26"
],
[
1,
"#f0f921"
]
],
"type": "heatmapgl"
}
],
"histogram": [
{
"marker": {
"colorbar": {
"outlinewidth": 0,
"ticks": ""
}
},
"type": "histogram"
}
],
"histogram2d": [
{
"colorbar": {
"outlinewidth": 0,
"ticks": ""
},
"colorscale": [
[
0,
"#0d0887"
],
[
0.1111111111111111,
"#46039f"
],
[
0.2222222222222222,
"#7201a8"
],
[
0.3333333333333333,
"#9c179e"
],
[
0.4444444444444444,
"#bd3786"
],
[
0.5555555555555556,
"#d8576b"
],
[
0.6666666666666666,
"#ed7953"
],
[
0.7777777777777778,
"#fb9f3a"
],
[
0.8888888888888888,
"#fdca26"
],
[
1,
"#f0f921"
]
],
"type": "histogram2d"
}
],
"histogram2dcontour": [
{
"colorbar": {
"outlinewidth": 0,
"ticks": ""
},
"colorscale": [
[
0,
"#0d0887"
],
[
0.1111111111111111,
"#46039f"
],
[
0.2222222222222222,
"#7201a8"
],
[
0.3333333333333333,
"#9c179e"
],
[
0.4444444444444444,
"#bd3786"
],
[
0.5555555555555556,
"#d8576b"
],
[
0.6666666666666666,
"#ed7953"
],
[
0.7777777777777778,
"#fb9f3a"
],
[
0.8888888888888888,
"#fdca26"
],
[
1,
"#f0f921"
]
],
"type": "histogram2dcontour"
}
],
"mesh3d": [
{
"colorbar": {
"outlinewidth": 0,
"ticks": ""
},
"type": "mesh3d"
}
],
"parcoords": [
{
"line": {
"colorbar": {
"outlinewidth": 0,
"ticks": ""
}
},
"type": "parcoords"
}
],
"scatter": [
{
"marker": {
"colorbar": {
"outlinewidth": 0,
"ticks": ""
}
},
"type": "scatter"
}
],
"scatter3d": [
{
"line": {
"colorbar": {
"outlinewidth": 0,
"ticks": ""
}
},
"marker": {
"colorbar": {
"outlinewidth": 0,
"ticks": ""
}
},
"type": "scatter3d"
}
],
"scattercarpet": [
{
"marker": {
"colorbar": {
"outlinewidth": 0,
"ticks": ""
}
},
"type": "scattercarpet"
}
],
"scattergeo": [
{
"marker": {
"colorbar": {
"outlinewidth": 0,
"ticks": ""
}
},
"type": "scattergeo"
}
],
"scattergl": [
{
"marker": {
"colorbar": {
"outlinewidth": 0,
"ticks": ""
}
},
"type": "scattergl"
}
],
"scattermapbox": [
{
"marker": {
"colorbar": {
"outlinewidth": 0,
"ticks": ""
}
},
"type": "scattermapbox"
}
],
"scatterpolar": [
{
"marker": {
"colorbar": {
"outlinewidth": 0,
"ticks": ""
}
},
"type": "scatterpolar"
}
],
"scatterpolargl": [
{
"marker": {
"colorbar": {
"outlinewidth": 0,
"ticks": ""
}
},
"type": "scatterpolargl"
}
],
"scatterternary": [
{
"marker": {
"colorbar": {
"outlinewidth": 0,
"ticks": ""
}
},
"type": "scatterternary"
}
],
"surface": [
{
"colorbar": {
"outlinewidth": 0,
"ticks": ""
},
"colorscale": [
[
0,
"#0d0887"
],
[
0.1111111111111111,
"#46039f"
],
[
0.2222222222222222,
"#7201a8"
],
[
0.3333333333333333,
"#9c179e"
],
[
0.4444444444444444,
"#bd3786"
],
[
0.5555555555555556,
"#d8576b"
],
[
0.6666666666666666,
"#ed7953"
],
[
0.7777777777777778,
"#fb9f3a"
],
[
0.8888888888888888,
"#fdca26"
],
[
1,
"#f0f921"
]
],
"type": "surface"
}
],
"table": [
{
"cells": {
"fill": {
"color": "#EBF0F8"
},
"line": {
"color": "white"
}
},
"header": {
"fill": {
"color": "#C8D4E3"
},
"line": {
"color": "white"
}
},
"type": "table"
}
]
},
"layout": {
"annotationdefaults": {
"arrowcolor": "#2a3f5f",
"arrowhead": 0,
"arrowwidth": 1
},
"colorscale": {
"diverging": [
[
0,
"#8e0152"
],
[
0.1,
"#c51b7d"
],
[
0.2,
"#de77ae"
],
[
0.3,
"#f1b6da"
],
[
0.4,
"#fde0ef"
],
[
0.5,
"#f7f7f7"
],
[
0.6,
"#e6f5d0"
],
[
0.7,
"#b8e186"
],
[
0.8,
"#7fbc41"
],
[
0.9,
"#4d9221"
],
[
1,
"#276419"
]
],
"sequential": [
[
0,
"#0d0887"
],
[
0.1111111111111111,
"#46039f"
],
[
0.2222222222222222,
"#7201a8"
],
[
0.3333333333333333,
"#9c179e"
],
[
0.4444444444444444,
"#bd3786"
],
[
0.5555555555555556,
"#d8576b"
],
[
0.6666666666666666,
"#ed7953"
],
[
0.7777777777777778,
"#fb9f3a"
],
[
0.8888888888888888,
"#fdca26"
],
[
1,
"#f0f921"
]
],
"sequentialminus": [
[
0,
"#0d0887"
],
[
0.1111111111111111,
"#46039f"
],
[
0.2222222222222222,
"#7201a8"
],
[
0.3333333333333333,
"#9c179e"
],
[
0.4444444444444444,
"#bd3786"
],
[
0.5555555555555556,
"#d8576b"
],
[
0.6666666666666666,
"#ed7953"
],
[
0.7777777777777778,
"#fb9f3a"
],
[
0.8888888888888888,
"#fdca26"
],
[
1,
"#f0f921"
]
]
},
"colorway": [
"#636efa",
"#EF553B",
"#00cc96",
"#ab63fa",
"#FFA15A",
"#19d3f3",
"#FF6692",
"#B6E880",
"#FF97FF",
"#FECB52"
],
"font": {
"color": "#2a3f5f"
},
"geo": {
"bgcolor": "white",
"lakecolor": "white",
"landcolor": "#E5ECF6",
"showlakes": true,
"showland": true,
"subunitcolor": "white"
},
"hoverlabel": {
"align": "left"
},
"hovermode": "closest",
"mapbox": {
"style": "light"
},
"paper_bgcolor": "white",
"plot_bgcolor": "#E5ECF6",
"polar": {
"angularaxis": {
"gridcolor": "white",
"linecolor": "white",
"ticks": ""
},
"bgcolor": "#E5ECF6",
"radialaxis": {
"gridcolor": "white",
"linecolor": "white",
"ticks": ""
}
},
"scene": {
"xaxis": {
"backgroundcolor": "#E5ECF6",
"gridcolor": "white",
"gridwidth": 2,
"linecolor": "white",
"showbackground": true,
"ticks": "",
"zerolinecolor": "white"
},
"yaxis": {
"backgroundcolor": "#E5ECF6",
"gridcolor": "white",
"gridwidth": 2,
"linecolor": "white",
"showbackground": true,
"ticks": "",
"zerolinecolor": "white"
},
"zaxis": {
"backgroundcolor": "#E5ECF6",
"gridcolor": "white",
"gridwidth": 2,
"linecolor": "white",
"showbackground": true,
"ticks": "",
"zerolinecolor": "white"
}
},
"shapedefaults": {
"line": {
"color": "#2a3f5f"
}
},
"ternary": {
"aaxis": {
"gridcolor": "white",
"linecolor": "white",
"ticks": ""
},
"baxis": {
"gridcolor": "white",
"linecolor": "white",
"ticks": ""
},
"bgcolor": "#E5ECF6",
"caxis": {
"gridcolor": "white",
"linecolor": "white",
"ticks": ""
}
},
"title": {
"x": 0.05
},
"xaxis": {
"automargin": true,
"gridcolor": "white",
"linecolor": "white",
"ticks": "",
"zerolinecolor": "white",
"zerolinewidth": 2
},
"yaxis": {
"automargin": true,
"gridcolor": "white",
"linecolor": "white",
"ticks": "",
"zerolinecolor": "white",
"zerolinewidth": 2
}
}
},
"title": {
"text": "Relative occurrence of the most common edges"
},
"width": 720,
"xaxis": {
"tickangle": -45,
"title": {
"text": "Connection pairs"
}
},
"yaxis": {
"title": {
"text": "% of all the connections"
}
}
}
},
"text/html": [
"\n",
" \n",
" \n",
" \n",
" \n",
"
"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"plotly.offline.init_notebook_mode(connected=True)\n",
"plotly.offline.iplot(fig, filename='simple-3d-scatter')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This figure shows the most weighted edges aka. the edge word-pairs that occured the most. The measure is the number of occurrence divided by the number of total edges. As it could have been predicted, the word \"the\" is present in many pairs and the two most frequent nodes' pair is significantly higher."
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"# \n",
" \n",
" \n",
" \n",
" \n",
"
"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"plotly.offline.init_notebook_mode(connected=True)\n",
"plotly.offline.iplot(fig, filename='degdistnew')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Here one can see the degree distribution for both of the networks, and it is clear that they do show a scaling property. For the modified network, a $\\gamma = 2.226$ also shows that this is a real-world network. The title also shows the average degree, which also decreased by renorming the network, which is not surprising, and tells us that the network is densely connected. \n",
"\n",
"Unfortunately, computing the $d_{max}$ even only for the giant component would take more energy resources than I have, which I skipped therefore. "
]
},
{
"cell_type": "code",
"execution_count": 182,
"metadata": {
"ExecuteTime": {
"end_time": "2019-11-19T14:49:13.281288Z",
"start_time": "2019-11-19T14:48:47.834105Z"
},
"slideshow": {
"slide_type": "skip"
}
},
"outputs": [],
"source": [
"r = nx.centrality.betweenness_centrality(G2, weight='weight')\n",
"\n",
"sorted_x = sorted(r.items(), key=operator.itemgetter(1), reverse=True)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"ExecuteTime": {
"end_time": "2019-11-19T15:04:31.927047Z",
"start_time": "2019-11-19T15:04:31.670084Z"
},
"slideshow": {
"slide_type": "skip"
}
},
"outputs": [],
"source": [
"num_to_show = 30\n",
"\n",
"x = [i[0] if i[0] == j else i[0] + \" (\" + j + \")\" for i,j in zip(sorted_x[:num_to_show], node[:num_to_show])]\n",
"y = [i[1] for i in sorted_x[:num_to_show]]\n",
"\n",
"fig10 = go.Figure([go.Bar(x = x,y = y, name = 'betweenness', )\n",
" ], layout=layout )\n",
"\n",
"\n",
"fig10.update_xaxes(title_text = \"Node name\", tickangle=315)\n",
"fig10.update_yaxes(title_text = \"Betweenness\")\n",
"#fig10.update_layout(xaxis_type=\"log\", yaxis_type=\"log\")\n",
"fig10.update_layout(title_text=\"Edges w highest betweenness\", font=dict(\n",
" family='Courier New, monospace',\n",
" size=14,\n",
" color='black'\n",
" ))\n",
"\n",
"fig10.update_layout({'hovermode': 'x',})\n"
]
},
{
"cell_type": "code",
"execution_count": 200,
"metadata": {
"ExecuteTime": {
"end_time": "2019-11-19T16:46:44.922279Z",
"start_time": "2019-11-19T16:46:44.488859Z"
},
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [
{
"data": {
"text/html": [
" \n",
" "
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.plotly.v1+json": {
"config": {
"linkText": "Export to plot.ly",
"plotlyServerURL": "https://plot.ly",
"showLink": false
},
"data": [
{
"name": "betweenness",
"type": "bar",
"x": [
"the",
"to",
"a",
"i",
"and",
"of",
"it",
"in",
"you",
"for",
"is",
"that",
"on",
"be",
"have (this)",
"this (have)",
"my (with)",
"with (my)",
"we",
"but (not)",
"not (are)",
"are (but)",
"was (at)",
"at (was)",
"so",
"all (as)",
"as (all)",
"get (your)",
"can (get)",
"me"
],
"y": [
0.41866890917612753,
0.3390368414692356,
0.29626519819123986,
0.2711888260762572,
0.2610320181098136,
0.2157065466046208,
0.2070755588574412,
0.20008891663832432,
0.1842731614628166,
0.16525426966436302,
0.1506775885213314,
0.13104623355084305,
0.11847692396753813,
0.11015628447593091,
0.1034316272262872,
0.0982646925616323,
0.09526642876344371,
0.09094715795472227,
0.08880627421364824,
0.07924054794308322,
0.07864938274991609,
0.07722239639143426,
0.07479790051175335,
0.0688618821933494,
0.06802277090274199,
0.06421841926289698,
0.06415704149820713,
0.06115666538131763,
0.05969984714272515,
0.059456578094802634
]
}
],
"layout": {
"autosize": false,
"bargroupgap": 0.3,
"font": {
"color": "black",
"family": "Courier New, monospace",
"size": 14
},
"height": 480,
"hovermode": "x",
"margin": {
"b": 100,
"l": 50,
"pad": 4,
"r": 50,
"t": 100
},
"template": {
"data": {
"bar": [
{
"error_x": {
"color": "#2a3f5f"
},
"error_y": {
"color": "#2a3f5f"
},
"marker": {
"line": {
"color": "#E5ECF6",
"width": 0.5
}
},
"type": "bar"
}
],
"barpolar": [
{
"marker": {
"line": {
"color": "#E5ECF6",
"width": 0.5
}
},
"type": "barpolar"
}
],
"carpet": [
{
"aaxis": {
"endlinecolor": "#2a3f5f",
"gridcolor": "white",
"linecolor": "white",
"minorgridcolor": "white",
"startlinecolor": "#2a3f5f"
},
"baxis": {
"endlinecolor": "#2a3f5f",
"gridcolor": "white",
"linecolor": "white",
"minorgridcolor": "white",
"startlinecolor": "#2a3f5f"
},
"type": "carpet"
}
],
"choropleth": [
{
"colorbar": {
"outlinewidth": 0,
"ticks": ""
},
"type": "choropleth"
}
],
"contour": [
{
"colorbar": {
"outlinewidth": 0,
"ticks": ""
},
"colorscale": [
[
0,
"#0d0887"
],
[
0.1111111111111111,
"#46039f"
],
[
0.2222222222222222,
"#7201a8"
],
[
0.3333333333333333,
"#9c179e"
],
[
0.4444444444444444,
"#bd3786"
],
[
0.5555555555555556,
"#d8576b"
],
[
0.6666666666666666,
"#ed7953"
],
[
0.7777777777777778,
"#fb9f3a"
],
[
0.8888888888888888,
"#fdca26"
],
[
1,
"#f0f921"
]
],
"type": "contour"
}
],
"contourcarpet": [
{
"colorbar": {
"outlinewidth": 0,
"ticks": ""
},
"type": "contourcarpet"
}
],
"heatmap": [
{
"colorbar": {
"outlinewidth": 0,
"ticks": ""
},
"colorscale": [
[
0,
"#0d0887"
],
[
0.1111111111111111,
"#46039f"
],
[
0.2222222222222222,
"#7201a8"
],
[
0.3333333333333333,
"#9c179e"
],
[
0.4444444444444444,
"#bd3786"
],
[
0.5555555555555556,
"#d8576b"
],
[
0.6666666666666666,
"#ed7953"
],
[
0.7777777777777778,
"#fb9f3a"
],
[
0.8888888888888888,
"#fdca26"
],
[
1,
"#f0f921"
]
],
"type": "heatmap"
}
],
"heatmapgl": [
{
"colorbar": {
"outlinewidth": 0,
"ticks": ""
},
"colorscale": [
[
0,
"#0d0887"
],
[
0.1111111111111111,
"#46039f"
],
[
0.2222222222222222,
"#7201a8"
],
[
0.3333333333333333,
"#9c179e"
],
[
0.4444444444444444,
"#bd3786"
],
[
0.5555555555555556,
"#d8576b"
],
[
0.6666666666666666,
"#ed7953"
],
[
0.7777777777777778,
"#fb9f3a"
],
[
0.8888888888888888,
"#fdca26"
],
[
1,
"#f0f921"
]
],
"type": "heatmapgl"
}
],
"histogram": [
{
"marker": {
"colorbar": {
"outlinewidth": 0,
"ticks": ""
}
},
"type": "histogram"
}
],
"histogram2d": [
{
"colorbar": {
"outlinewidth": 0,
"ticks": ""
},
"colorscale": [
[
0,
"#0d0887"
],
[
0.1111111111111111,
"#46039f"
],
[
0.2222222222222222,
"#7201a8"
],
[
0.3333333333333333,
"#9c179e"
],
[
0.4444444444444444,
"#bd3786"
],
[
0.5555555555555556,
"#d8576b"
],
[
0.6666666666666666,
"#ed7953"
],
[
0.7777777777777778,
"#fb9f3a"
],
[
0.8888888888888888,
"#fdca26"
],
[
1,
"#f0f921"
]
],
"type": "histogram2d"
}
],
"histogram2dcontour": [
{
"colorbar": {
"outlinewidth": 0,
"ticks": ""
},
"colorscale": [
[
0,
"#0d0887"
],
[
0.1111111111111111,
"#46039f"
],
[
0.2222222222222222,
"#7201a8"
],
[
0.3333333333333333,
"#9c179e"
],
[
0.4444444444444444,
"#bd3786"
],
[
0.5555555555555556,
"#d8576b"
],
[
0.6666666666666666,
"#ed7953"
],
[
0.7777777777777778,
"#fb9f3a"
],
[
0.8888888888888888,
"#fdca26"
],
[
1,
"#f0f921"
]
],
"type": "histogram2dcontour"
}
],
"mesh3d": [
{
"colorbar": {
"outlinewidth": 0,
"ticks": ""
},
"type": "mesh3d"
}
],
"parcoords": [
{
"line": {
"colorbar": {
"outlinewidth": 0,
"ticks": ""
}
},
"type": "parcoords"
}
],
"scatter": [
{
"marker": {
"colorbar": {
"outlinewidth": 0,
"ticks": ""
}
},
"type": "scatter"
}
],
"scatter3d": [
{
"line": {
"colorbar": {
"outlinewidth": 0,
"ticks": ""
}
},
"marker": {
"colorbar": {
"outlinewidth": 0,
"ticks": ""
}
},
"type": "scatter3d"
}
],
"scattercarpet": [
{
"marker": {
"colorbar": {
"outlinewidth": 0,
"ticks": ""
}
},
"type": "scattercarpet"
}
],
"scattergeo": [
{
"marker": {
"colorbar": {
"outlinewidth": 0,
"ticks": ""
}
},
"type": "scattergeo"
}
],
"scattergl": [
{
"marker": {
"colorbar": {
"outlinewidth": 0,
"ticks": ""
}
},
"type": "scattergl"
}
],
"scattermapbox": [
{
"marker": {
"colorbar": {
"outlinewidth": 0,
"ticks": ""
}
},
"type": "scattermapbox"
}
],
"scatterpolar": [
{
"marker": {
"colorbar": {
"outlinewidth": 0,
"ticks": ""
}
},
"type": "scatterpolar"
}
],
"scatterpolargl": [
{
"marker": {
"colorbar": {
"outlinewidth": 0,
"ticks": ""
}
},
"type": "scatterpolargl"
}
],
"scatterternary": [
{
"marker": {
"colorbar": {
"outlinewidth": 0,
"ticks": ""
}
},
"type": "scatterternary"
}
],
"surface": [
{
"colorbar": {
"outlinewidth": 0,
"ticks": ""
},
"colorscale": [
[
0,
"#0d0887"
],
[
0.1111111111111111,
"#46039f"
],
[
0.2222222222222222,
"#7201a8"
],
[
0.3333333333333333,
"#9c179e"
],
[
0.4444444444444444,
"#bd3786"
],
[
0.5555555555555556,
"#d8576b"
],
[
0.6666666666666666,
"#ed7953"
],
[
0.7777777777777778,
"#fb9f3a"
],
[
0.8888888888888888,
"#fdca26"
],
[
1,
"#f0f921"
]
],
"type": "surface"
}
],
"table": [
{
"cells": {
"fill": {
"color": "#EBF0F8"
},
"line": {
"color": "white"
}
},
"header": {
"fill": {
"color": "#C8D4E3"
},
"line": {
"color": "white"
}
},
"type": "table"
}
]
},
"layout": {
"annotationdefaults": {
"arrowcolor": "#2a3f5f",
"arrowhead": 0,
"arrowwidth": 1
},
"colorscale": {
"diverging": [
[
0,
"#8e0152"
],
[
0.1,
"#c51b7d"
],
[
0.2,
"#de77ae"
],
[
0.3,
"#f1b6da"
],
[
0.4,
"#fde0ef"
],
[
0.5,
"#f7f7f7"
],
[
0.6,
"#e6f5d0"
],
[
0.7,
"#b8e186"
],
[
0.8,
"#7fbc41"
],
[
0.9,
"#4d9221"
],
[
1,
"#276419"
]
],
"sequential": [
[
0,
"#0d0887"
],
[
0.1111111111111111,
"#46039f"
],
[
0.2222222222222222,
"#7201a8"
],
[
0.3333333333333333,
"#9c179e"
],
[
0.4444444444444444,
"#bd3786"
],
[
0.5555555555555556,
"#d8576b"
],
[
0.6666666666666666,
"#ed7953"
],
[
0.7777777777777778,
"#fb9f3a"
],
[
0.8888888888888888,
"#fdca26"
],
[
1,
"#f0f921"
]
],
"sequentialminus": [
[
0,
"#0d0887"
],
[
0.1111111111111111,
"#46039f"
],
[
0.2222222222222222,
"#7201a8"
],
[
0.3333333333333333,
"#9c179e"
],
[
0.4444444444444444,
"#bd3786"
],
[
0.5555555555555556,
"#d8576b"
],
[
0.6666666666666666,
"#ed7953"
],
[
0.7777777777777778,
"#fb9f3a"
],
[
0.8888888888888888,
"#fdca26"
],
[
1,
"#f0f921"
]
]
},
"colorway": [
"#636efa",
"#EF553B",
"#00cc96",
"#ab63fa",
"#FFA15A",
"#19d3f3",
"#FF6692",
"#B6E880",
"#FF97FF",
"#FECB52"
],
"font": {
"color": "#2a3f5f"
},
"geo": {
"bgcolor": "white",
"lakecolor": "white",
"landcolor": "#E5ECF6",
"showlakes": true,
"showland": true,
"subunitcolor": "white"
},
"hoverlabel": {
"align": "left"
},
"hovermode": "closest",
"mapbox": {
"style": "light"
},
"paper_bgcolor": "white",
"plot_bgcolor": "#E5ECF6",
"polar": {
"angularaxis": {
"gridcolor": "white",
"linecolor": "white",
"ticks": ""
},
"bgcolor": "#E5ECF6",
"radialaxis": {
"gridcolor": "white",
"linecolor": "white",
"ticks": ""
}
},
"scene": {
"xaxis": {
"backgroundcolor": "#E5ECF6",
"gridcolor": "white",
"gridwidth": 2,
"linecolor": "white",
"showbackground": true,
"ticks": "",
"zerolinecolor": "white"
},
"yaxis": {
"backgroundcolor": "#E5ECF6",
"gridcolor": "white",
"gridwidth": 2,
"linecolor": "white",
"showbackground": true,
"ticks": "",
"zerolinecolor": "white"
},
"zaxis": {
"backgroundcolor": "#E5ECF6",
"gridcolor": "white",
"gridwidth": 2,
"linecolor": "white",
"showbackground": true,
"ticks": "",
"zerolinecolor": "white"
}
},
"shapedefaults": {
"line": {
"color": "#2a3f5f"
}
},
"ternary": {
"aaxis": {
"gridcolor": "white",
"linecolor": "white",
"ticks": ""
},
"baxis": {
"gridcolor": "white",
"linecolor": "white",
"ticks": ""
},
"bgcolor": "#E5ECF6",
"caxis": {
"gridcolor": "white",
"linecolor": "white",
"ticks": ""
}
},
"title": {
"x": 0.05
},
"xaxis": {
"automargin": true,
"gridcolor": "white",
"linecolor": "white",
"ticks": "",
"zerolinecolor": "white",
"zerolinewidth": 2
},
"yaxis": {
"automargin": true,
"gridcolor": "white",
"linecolor": "white",
"ticks": "",
"zerolinecolor": "white",
"zerolinewidth": 2
}
}
},
"title": {
"text": "Edges w highest betweenness"
},
"width": 720,
"xaxis": {
"tickangle": -45,
"title": {
"text": "Node name"
}
},
"yaxis": {
"title": {
"text": "Betweenness"
}
}
}
},
"text/html": [
"\n",
" \n",
" \n",
" \n",
" \n",
"
"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"plotly.offline.init_notebook_mode(connected=True)\n",
"plotly.offline.iplot(fig10, filename='betw')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This figure shows the 30 nodes with the highest betweenness. For those nodes, where there is another word in the parentheses the latter shows the word with the same rank in the frequency table. For example, on this figure on the 15th place \"have\" is present (it is the 15th node with the highest betweenness) but the \"this\" word was the 15th most frequent. Mostly there only small changes meaning that high betweenness correlates with high frequency also."
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"# \n",
" \n",
" \n",
" \n",
" \n",
"
"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"plotly.offline.init_notebook_mode(connected=True)\n",
"plotly.offline.iplot(fig10, filename='betw')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In this plot one can see the distribution of the cardinality of the communities found by the algorithm. What is hardly seen here is a community consisting of $\\sim 4400$ nodes at the rightmost side. We also can see that there are many node which could not be clustered."
]
},
{
"cell_type": "code",
"execution_count": 224,
"metadata": {
"ExecuteTime": {
"end_time": "2019-11-19T17:28:20.541743Z",
"start_time": "2019-11-19T17:28:20.534692Z"
},
"slideshow": {
"slide_type": "skip"
}
},
"outputs": [],
"source": [
"ones_twos = []\n",
"for i in c:\n",
" rtz = len(list(i))\n",
" if rtz <= 2 :\n",
" ones_twos.append(list(i)[0])"
]
},
{
"cell_type": "code",
"execution_count": 226,
"metadata": {
"ExecuteTime": {
"end_time": "2019-11-19T17:29:26.438702Z",
"start_time": "2019-11-19T17:29:26.433366Z"
},
"scrolled": false,
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"['xrmjoel', 'katecrosby92', 'littlevix', 'kaydm49', 'viners_uk', 'bigmondo77', 'pwsplash', 'okwu_id', 'officialj_ramm', 'ste51510874', 'biseysolucem', 'alanweir85', 'shotonmo', 'alexxisfalcon', 'отлично', 'iainm97', 'ספרד', 'racisminfootbal', 'jokerlaugh', 'jackhosk', 'bewareofspock', 'impeachtrump', 'malisetweet', 'sleepmastereu', 'honestlyyyy', 'mqasem', 'classicshirt', 'حبيبي', 'mariacaulfield', 'occasion', 'fcroyals10', 'yhg_partington', 'riotbadg', 'ritirati', 'paulgarvey4', 'haww', 'aliiigriiiiaaa', 'loool', 'darpross', 'aesfet', 'wingyipstor', 'calumlawson_27', 'verbaldiahrreoa', 'pemalu', 'balang', 'shirlvemb', 'ditchthetori', 'hhclubnit', 'goddess', 'lxuren2000', 'teambori', 'امين', 'infoalishba', 'edie_raffi', 'anfieldedit', 'amirulhafiiiiz', 'stewood', 'fancy_that123', '_sammarriott', 'gawg', 'hollyyybain', 'يانجد', 'epsteinsuicidecoverup', 'saints1890', 'teamliam', 'rfitzgibbon_', 'mrprowsemath', 'jayneybarn', 'leedsandyorkpft', 'ollieoswin', 'j_cartwrightn0t', 'jwphxtographi', 'dup_onlin', 'dubiinbus', '한복낙서', 'adamlatchford', 'pgarrittypaul', 'izzycridland', 'fackkkk', 'za13aimbot', 'damianbaker1', 'buzzziiiinnnnnnnnn', 'helss19', 'رايك', 'tvcritic', 'toolzbab', 'hebridesnew', 'bradley_safc', 'xgabrielphoenix', 'fairwarning101', 'everymindmatt', 'markyholtend', 'valery_violet', 'riksredguard', 'lanaswond', 'abelarana', 'tedxdoncast', 'thehoundmma', 'perrycourtpri', 'chynafemdom', 'priceless', 'nor_edu', 'zaraali2k19', 'rolake_ba', 'barbervill', 'hcsarah', 'thundercat8366', 'melreylaw', 'fanfuck', 'kevvief', 'southcoastblok', 'jwhitehurst1989', '_bradleybutch', 'safiyya_db', 'stephiearnett', 'astonishingtrev', 'enjoyinglif', 'ryantwm', 'jackratuno', 'anaglogsdaughtr', 'asechat', 'gecfrost', 'londonahp', 'hawkpadilla', 'goalz', 'dbeasleyharl', 'driverfarms1', 'احم', 'sundaygirl74', 'marky_boy1968', 'mdrrr', 'tvtonightau', 'anfellyy', 'wassup', 'theumilk99', 'scollanangela', 'vamplacey', 'savestpeterward', 'grumpnow', 'katie_cak', 'tdunne888', 'sho_yah', 'jackcaudwel', 'overthink', 'oscarcain', 'passthepod', 'pranksnpup', 'skinny_fatblok', 'tomcampbel', 'seanwleach', 'johnjhammond', 'renad_alshareef', 'yike', 'adrianofcind', 'nrstirzak', 'michaelroseny', 'dusk', 'suzukicarsuk', 'moridura', 'destuhnym', 'adilabdalmahdi', 'readliverpoolfc', 'vgagymnast', 'heronbank_cat', 'lynnecampbell5', 'messyjessy85', 'revmaryhaw', 'bighairygrowl', 'doong', 'ambrob64', 'loooooooooool', 'momonobyy', 'tessabunney', 'sgtstockport', 'ra62156', 'wimbart', 'suzanne02190370', 'maashaallah', 'cathygremin', '07818114120', 'catalansfory', 'bathynotcathi', 'thepoliticalbe2', 'stevepith', 'kineticyouthltd', 'patchmcscratchi', 'fucker', 'sigh', 'wahaaga', 'nononononknon', 'vasazz', 'nl_vossi', 'nayimfromthehalfwaylin', 'therealsexpectr', 'leonagraham', 'こっちのおっさん達本当笑い声大きくて草', 'matttreaci', 'krystal_888', 'chrismouldink', 'pineappleslut', 'rgrodger', 'nottmbantam', 'itsjusdyl', 'ryanreynold', 'daus_sss', 'ilovem', 'thisiswhatticklesm', 'richydisney', 'joannablythman', 'finalfantasyxiv', 'sadietrent_92', 'elainedyson1', 'nani', 'theanxiousteac2', 'awesomofpl', 'hertscricket', 'collaredboi', 'drvbate', 'latemotivcero', 'justinemiller_', 'russiareport', 'mrgregblackman', 'kate_sir', 'katelawl', 'tiinawild', 'kulganofcryde', 'percochewin', 'oswtcatg', 'bigracek', 'k1ngfii', 'eeeek', 'tiredddd', 'allot50cornwal', 'sades023', 'ksjsksjs', 'broooooo', 'helpdip', 'jakeyl0l', 'dog_feel', 'fiyah', 'mrsclart', 'ohsinnerman', 'annhardey', 'ahmadjasem_', 'sirstevotimothi', 'silvioaquinog', 'saywhat', 'joolsd', 'vibin', 'deersh', 'سالخير', 'georgiarice_x', 'paulminor4r', 'akkitwt', 'heminguay_', 'alexcraftlfc', 'drinkinthemovi', 'simplelampoon', 'nuriyegülmenserbestbırakılsın', 'xxp123', 'notcapnamerica', 'harvkudo', 'openbritainshrp', 'bigspenderr1690', 'surbiton2012', 'bemoremariah', 'markpit35296344', 'itslibdemtim', 'deirdre_flynn', 'daffodilnebula', 'rhylhighp', 'dailymirror', 'pringster78', 'james_orrey', 'tescokingstonmk', 'marcrebillet', 'hantshighway', 'raahh', 'yayyy', 'aestheticsgroup', 'peteregan6', 'thesilentgirl6', 'tenorbenjohnson', 'bringiton28', 'being_ajay_', 'beaumontsduff', 'listerrd', 'followstevi']\n"
]
}
],
"source": [
"print(ones_twos)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Above one can see those nodes which were not clustered at all or were clustered in pairs. Despite some interesting nodes (\"epsteinsuicidecoverup\" and \"こっちのおっさん達本当笑い声大きくて草\" -- meaning \"This uncle is really laughing loud and grass\") there were a few normal words also (\"occasion\", \"overthink\" or \"priceless\") most of the words are meaningless or usernames/hashtags."
]
},
{
"cell_type": "code",
"execution_count": 230,
"metadata": {
"ExecuteTime": {
"end_time": "2019-11-19T17:43:20.283646Z",
"start_time": "2019-11-19T17:43:20.269999Z"
},
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"21 ['द', 'आजक', 'स', 'क', 'ल']\n",
"20 ['brenmac01', 'themuddlersclub', 'fuckri', 'so', 'westindi']\n",
"18 ['b_isto__ar', 'indianregista', 'nycfc', 'hollybeang', 'towel']\n",
"18 ['28s', 'suemurray14', 'snatch', 'thekophq', 'mall']\n",
"16 ['ryan_kenny1', 'clivegreenwd', 'jasbar', 'cihfutur', 'eviecopland']\n",
"16 ['pcso3580ruth', 'alicel', 'emilylinka', 'have', 'shirleysetia']\n",
"14 ['эплу', 'хороший', 'очень', 'выпуск', 'iamvasabi']\n",
"14 ['ଆଙ', 'ସ', 'ନ', 'ପ', 'ଳନ']\n",
"14 ['samozřejmě', 'bambilionnásobně', 'platit', 'croplus', 'podl']\n",
"13 ['debbiezimmer54', 'jilliemari', 'newworlddd555', 'djdebster', 'suzannelepage1']\n",
"13 ['maureenb2b', 'burgessbev', 'joel_b2beditor', 'nailedit', 'elise_amil']\n",
"13 ['eileenbwyatt', '7th_layer', 'lovemypir', 'c_licar', 'yulvazquez']\n",
"13 ['boy1010tori', 'toxicambassador', 'sarzboogi', 'we', 'onejasonkayley']\n",
"13 ['brockleymax', 'jimiadefirany', 'catcornucopia', 'brockleybreweri', 'thebrockleybuzz']\n",
"12 ['misssdoherti', 'div', 'bravotv', 'scott_riley', 'button']\n",
"12 ['มต', 'ดเข', 'ท', 'ยย', 'วก']\n",
"11 ['rosefarmpreserv', 'godminsterfarm', 'pdevonian', 'jacobscreek', 'oliveset']\n",
"11 ['brexiteerjacob', 'pole', 'stepto', 'ageofblhyt', 'deal']\n",
"11 ['rabbi_matondo', 'kinnock', 'bentaleb', 'be', 'greedi']\n",
"10 ['allezamani', 'napaul_', 'queen_hon', 'iamdahmmi', 'toluwanikri']\n",
"9 ['私がチケットを手配したんですか', 'どなたか経由か言えないですかね', 'ツイッターだと', 'ところで', '気になります']\n",
"9 ['millbrookafc', 'hafc125', 'ilfcombeafc', 'swsportsnew', 'devon_fa']\n",
"9 ['they', 'rockcak', 'hughbarnett6', 'fir', 'rachamuffin']\n",
"9 ['untuk', 'aimanhadiii', 'tidak', 'amirabdlatif', 'u2']\n",
"8 ['itsbeenadayforcompoundword', 'jamlondon', 'jonnyburch', 'clearleft', 'intercomdesign']\n",
"7 ['pbuckley05', 'for', 'sophdoy', 'tyler_campbeii', 'profootballtalk']\n",
"6 ['الغلط', 'عليك', 'احـب', 'بالغلط', 'اسفه']\n",
"6 ['つまりサブスタンスこそ肝心', '英語はあくまでツール', '初めて英語は生きてくる', 'そのツールで何に取り組むか', '英語だけ話せてバカ騒ぎしても気持ちが良いだけで何も産まない']\n",
"6 ['هالشوف', 'يا', 'ويا', 'سعد', 'هالرجعه']\n",
"6 ['نواياكم', 'hamza_tekin2023', 'وينصر', 'ربنا', 'الحسنة']\n",
"6 ['bowmancourtney_', 'sixthemus', 'perryobre', 'westendcov', 'daniellest']\n",
"6 ['ministerktr', 'ani_digit', 'zcwz_ghmc', 'arvindkumar_ia', 'commissionrghmc']\n",
"6 ['passagechar', 'pathwayuk', 'itsgroundswel', 'lnnmhomeless', 'danabeal']\n",
"6 ['lacnetworkuk', 'westheathlac', 'maffpott', 'catdre', 'bemorepir']\n",
"6 ['kaan', 'wkwkwkwk', 'udaahh', 'silviahani_tj', 'eike']\n",
"5 ['learningfromphilosophi', 'dumbarton_acad', 'strathfamili', 'rmps', 'rme']\n",
"5 ['exeterchief', 'southwestcomm', 'moon_p1g', 'realaxelphim', 'backingben']\n",
"5 ['hayleylev', 'hannahport2', 'northpowerwomen', 'greatersport', 'gmmove']\n",
"5 ['kazimiergarden', 'thekazimi', 'spqrband', 'harvest_sun', 'smmmgmt']\n",
"5 ['mattbeasty24', 'chrispiper4', 'bleacherreport', 'p1p3s', 'jcmack03']\n",
"5 ['snoffl', 'garethjennings3', 'ldpierre1', 'pepere_06', 'samhepworth14']\n"
]
}
],
"source": [
"for i in c:\n",
" rtz = len(list(i))\n",
" if 5 <= rtz <=21 :\n",
" print(rtz, list(i)[:5])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Communties above are listed their cardinality was between 5 and 21 and 5 of their members. Here there are interesting clusters: for example, not only the cluster with 21 nodes is probably the cluster of the hindi words but all the communities with a cardinality with 14 are distinct languanges (russian, hindi maybe(?) and czech, respectively). One can also find japanese characters (which are whole sentences by the way) and many arabic communities. An arabic group of six is about something bad, since the meanings of the words listed are the following: ['Mistake', 'You', 'Love', 'Mistake', 'Sorry']. Such a cheap drama. "
]
},
{
"cell_type": "code",
"execution_count": 233,
"metadata": {
"ExecuteTime": {
"end_time": "2019-11-19T17:45:36.629544Z",
"start_time": "2019-11-19T17:45:36.614219Z"
},
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"4419 ['beginn', 'improv', 'temperatur', 'loncon19', 'xcase_', 'russtnuttz', 'davidhall75', 'wildearth', 'bbcnickrobinson', 'mike_lions_71']\n",
"2962 ['constitu', 'regain', 'counter', 'shock', 'goodnewshackney', 'popsicle_____', 'penyrheolgerrig', 'lack', 'unlikeli', 'uklabour']\n",
"2042 ['habit', 'gerad', 'slept', 'pusscat', 'ellarinajohn', 'wish', 'waveoflight2019', 'safeti', 'yoga', 'fat']\n",
"1996 ['rasa', 'pásalo', 'mahez', 'marialal', 'illustriousg40', 'botham_sam', 'gedl', 'tywydd', 'tien', 'farklı']\n",
"1331 ['quid', 'dvd', 'fu', 'profil', 'gibsidehotel', 'wildathearthq', 'th3gasman', 'merseaisland', 'lx', 'waywardhu']\n",
"930 ['حرفيا', 'monetari', 'بعد', 'ربيع', 'يوم', 'سادگی', 'richardlatto', 'womeninart', 'المباركة', 'قسومنا']\n",
"914 ['feather', 'angelahilleri', 'darzi', 'mindless', 'finest', 'twofussyblok', 'drinkwat', 'vicky_mcclur', 'tegen', '070906']\n",
"559 ['bhamcitycouncil', 'sarashamma', 'referendumnin', 'royalfamili', 'gadaviman', 'gabe', 'chelseafran', 'nickybenedetti', 'audibleuk', 'vch_shropshir']\n",
"378 ['lubnan', 'cleveley', 'bill', 'dancludlow', 'sore', 'wouldent', 'eleaonr', '104k', 'ptal', 'jamesaflanagan1']\n",
"297 ['9b051e07df75489', 'cee', 'timolarch', 'limegrovecar', 'hollywood', 'davidlammi', 'other', 'gmail', 'cooi_daddi', 'sherrymanning85']\n"
]
}
],
"source": [
"for i in c[:10]:\n",
" rtz = len(list(i))\n",
" print(rtz, list(i)[:10])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Here you can see the 10 largest communitites with 10 members of that. Some interesting things:\n",
"\n",
"- the second largest group may be based on political tweets (according to these words),\n",
"- the third probably is about fitness ('habit', 'wish', 'yoga', 'fat'),\n",
"- the group 930 is probably the largest non-english group, which is listed totally below."
]
},
{
"cell_type": "code",
"execution_count": 235,
"metadata": {
"ExecuteTime": {
"end_time": "2019-11-19T17:49:45.341137Z",
"start_time": "2019-11-19T17:49:45.329173Z"
},
"scrolled": false,
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"['حرفيا', 'monetari', 'بعد', 'ربيع', 'يوم', 'سادگی', 'richardlatto', 'womeninart', 'المباركة', 'قسومنا', 'ومراته', 'الأميرات', 'spay', 'الخاطفة', 'انا', 'الله', 'آکھے', 'بمشي', 'ثبات', 'حرام', 'georgia', 'favourit', 'بیغیرت', 'cleansman', 'الأساسية', 'قالتلي', 'الساعه', 'ایتھے', 'ph', 'غير', 'فخاري', 'فالدنيا', 'الرز', 'announc', 'joalsubai', 'طبية', 'الكوارث', 'العصر', 'بس', 'الشيخ', 'بنت', 'يفيدها', 'خبراء', 'الرياض', 'بالضرورة', 'تمنى', 'كأس', 'حساب', 'وسيوارى', 'العزاء', 'الانستقرام', 'كتاب', 'كلمة', 'مع', 'مررره', 'غيمة', 'زاد', 'karimahamed9', 'اريد', 'wigan', 'alfaskara7', 'travel', 'بندہ', 'انت', 'خربوك', 'احب', 'ونعوضها', 'لحضور', 'سوق', 'صار', 'لاحول', 'نجيب', 'ف', 'ترهبني', 'منزل', 'شيخة', 'complimentari', 'نحمد', 'م', 'مازرت', 'حاجة', 'اسهل', 'جعلنى', 'الغدا', 'classicbritcom', 'أجامل', 'الفردوس', 'اضبطها', 'الصغيرة', 'دار', 'الخرينج', '٧', 'المتعبة', 'roaa_alsabban', 'ghumman', 'ليه', 'عمرى', 'والديكتاتورية', 'والشيخة', 'آگ', 'أيامك', 'ق', 'المقربه', 'ظلمه', 'اطلع', 'ويتناقشه', 'ومصطلحات', 'کھوتی', 'كله', 'دمتما', 'يتعايش', 'ا', 'الرفيق', 'باذن', 'رايحة', 'واحد', 'jurgen', 'وإن', 'ابو', 'hadizonouzi', 'وهالزمن', 'المعدي', 'واقعا', 'ها', 'قبل', 'كامل', 'فيه', 'عزیز', 'مبالغ', 'رحم', 'الخريف', 'يلقى', 'ومنتفضين', 'إلا', 'بالساعات', 'الفاتحة', 'ويوفقنا', 'انها', 'انتقل', 'الذي', 'د', 'بالدنيا', 'فرحا', 'من', 'باللي', 'اس', 'تركب', 'destin', 'فضله', 'أو', 'وتفاؤلا', 'كانت', 'alirabiei_ir', 'منطقه', 'كورية', 'پھر', 'صديقة', 'سياسة', 'الويوو', 'thelifeofcoko', 'مختوم', 'أهواه', '٢٠٠', 'عبدالله', 'فيفى', 'girlfriend', 'نسى', 'walkden', 'عن', 'هذه', 'قتل', 'مصداقيتها', 'halquraan14', 'بالسياسة', 'عون', 'عمق', 'leicscipd', 'الخطير', 'اوقف', 'ما', 'مما', 'rush', 'نشوفكم', 'الجميل', 'خلينا', 'رياض', 'هذا', 'الف_رحمه_ونور', 'بالستر', 'كنا', 'كل', 'الشخص', 'dakota', 'لمن', 'حكي', 'قطعة', 'تسمح', 'يابيونه', 'کر', 'احسن', 'دعواه', 'کھان', 'العقلية', 'قلوبهم', 'والوفاء', 'jfk_man', 'او', 'صديقتها', 'مش', 'وقت', 'سواك', 'وكانوا', 'فيا', 'صباحا', 'مبارك', 'له', 'مضى', 'موثر', 'الكورية', 'أرجو', 'الهادي', 'يشعل', 'ن', 'وأصلح', 'جيت', 'تلتقي', 'اعتمادها', 'مثالية', 'اكتر', 'ناس', 'تظلم', 'rouhani_ir', 'اني', 'عايشين', 'مو', 'مصدرها', 'الثورة', 'بمين', 'rezgo9', 'ر', 'والفساد', 'الان', 'اضطراباتنا', 'وجهاتنا', 'ظهر', 'أنت', 'اكثر', 'بمناسبة', 'منها', 'نه', 'قولي', 'لبنان_يتنفض', 'كثير', 'ليا', 'العطر', 'قدوم', 'مقبرة', 'حديق', 'خبر', 'مصرف', 'در', '١١', 'الشهرة', 'بالحسبان', 'تسال', 'شفا', 'پر', 'mysmuttylif', 'المفروض', 'مشكلتى', 'lancashir', 'نفسيه', 'العالمي', 'التي', 'وعلشان', 'التأجيل', 'أكثر', 'السالفة', 'ثقة', 'وأخرجني', 'cipd', 'وامان', 'الخانقة', 'خبرية', 'الرجال', 'اتعجب', 'تمت', 'aldresehadiy', 'tantanman_46', 'حدث', 'stopmisus', 'ي', 'محلللك', 'جدى', 'linajululian', 'اور', 'تتكلمون', 'السياسة', 'اتغدا', 'المؤجلة', 'قوى', 'طا', 'تخليك', 'هل', 'liesssssss', 'unitedinfocus', 'الجنة', 'نتشارك', 'بالخير', 'منك', 'مصاحبه', 'ومننشرها', 'باقات', 'الشهداء', 'تو', 'أصدق', 'دھرنے', 'يقول', 'أصعب', 'وأخواته', 'كان', 'أشووفه', 'حسبنا', 'لبنان', 'تھے', 'الجديدة', 'الزحمة', 'وسياسته', 'یوتھئیو', 'اجيب', 'وتقريبا', 'جيران', 'عنوان', 'نشرت', 'کے', 'وبالغصب', 'me3j', 'لندن', 'لو', 'تعالى', 'يجعل', 'الكلاس', 'كلها', 'مجاديفه', 'bee', 'غلاك', 'ناسب', 'ينحل', 'القلب', 'عرفتى', 'ھموطنوں', 'rynivv', 'عنى', 'بفرقتين', 'الخيال', 'نفسي', 'نصر', 'رني', 'السلطة', 'لذلك', 'النية', 'a_rrz', 'الناس', 'لغة', 'حال', 'قلبك', 'مفوض', '٩', 'شكر', 'النقدية', 'قصص', 'manchest', 'ماكسبنا', 'سيزون', 'سلامة', 'ان', 'كنت', 'mnvll5', 'فارس', 'ليس', 'ولكل', 'longhair', 'المالية', 'يستطيع', 'zad_15', 'تويتر', 'الا', 'نشئ', 'إني', 'شاور', 'اوصلك', 'بمدينة', 'وشو', 'مصغ', 'كدرت', 'الثاني', 'كبيرة', 'بداية', 'وضعي', 'طرفة', 'نفس', 'فطری', 'عمي', 'اخت', 'بود', 'کو', 'حياة', 'm_oman00', 'وهذا', 'دیجیئے', 'منع', 'دياركم', 'رہے', 'يعبر', 'وعن', 'alsammak85', 'الموافق', 'مين', 'league10', 'وآل', 'هنعوضها', 'و', 'كيف', 'ابتسم', 'کہ', 'سعادة', 'فيما', 'أنا', 'ہے', 'لأندية', 'ودام', 'الأنتيم', '855', 'محد', 'جانب', 'الأميرة', 'موسم', 'firewithin', 'والود', 'جیسے', 'يسمع', 'الن', 'bno0oo', 'الكلام', 'بانيه', 'این', 'فينا', 'ماذا', 'جثمانه', 'يرحمه', 'رحمتك', 'العالم', 'بھی', 'مركزين', 'crossedthelin', 'استفسار', 'ألف', 'سليمان_بن_حسين', 'قرأ', 'حالي', 'إليه', 'فى', '١٠', 'ghislaine_howard', 'أقول', 'جريح', 'المنيو', '١', 'والله', 'بكره', 'silwoodsocialseminar', 'يعانق', 'غدا', 'ماحقق', 'حق', 'يقدر', 'وصلتنا', 'اللي', 'تیرے', 'seemegraja74', 'عطر', 'مصرالجديدة', 'تعالی', 'وعهدي', 'زعزع', 'کردار', 'تهد', 'الاشياء', 'omar_alomair', 'سررررررر', 'گئی', 'ثالث', 'خالة', 'يعني', 'بالله', 'في', 'محمد', 'بنخاف', 'حدودك', 'وان', 'مصر', 'ibrahimrrz', '07445', 'نجاة', 'northenrailf', 'جانتی', 'وحسن', 'ردي', 'ومنشرها', 'مساعد', 'أمورنا', 'sneak', 'اربع', 'يفرق', 'زمان', 'للعبة', 'where', 'الشمولية', 'businesskfu', 'افتقد', 'وطيبا', 'هي', '311', 'تتشهد', 'عشان', 'ہوتی', 'lolwootbox', 'تشوفو', 'mackiemayor', 'لغيره', 'ofteneatingchocol', 'stephani', 'لولو', 'حسينية', 'لما', 'واحب', 'الأحلام', 'أحسن', 'يسألني', 'fiscal', 'هوشمندانه', 'اقتصادي', 'حبيبته', 'tehran422', 'لترسم', 'کرے', 'jordan', 'جميل', 'بريك', 'ستأتيك', 'قباني', 'ذلك', 'manc', 'بقا', 'فحسب', 'قتلهم', 'ادخل', 'يجابل', 'مصطلحاتنا', 'النظم', 'القمررررر', 'صلاح', 'کامل', 'hepplestonefineart', 'اطالع', 'نیست', 'أعطاه', 'خواهد', 'حبيتهم', 'grey', 'راض', 'seiz', 'الم', 'busineessckfu', 'السفر', 'الذكر', 'نموذج', 'الفورمولا', 'وبصراحه', 'أجبر', 'جنبي', 'ضعف', 'عطرا', 'أن', 'بصراحه', 'sept', 'بأكره', 'ياأجمل', 'فلا', 'العطلات', 'كلنا', 'الصامل', 'قررت', '١٢٠٠', 'جاء', 'درجات', 'معروف', 'الصلاة', 'تبادل', 'الدنيا', 'الأردني', 'أقذر', 'المجامل', 'تعرف', 'أبن', 'لي', 'حيث', 'عونك', 'لهما', 'غروب', 'الت', 'وجدتى', 'قوة', 'يسهل', 'belong', 'للعام', 'لعبي', 'petergraystock', 'ظرة', 'watt', 'تكون', 'جاءه', 'وأنا', 'cipdace19', 'أحيان', 'ثانيه', 'اللقاء', 'للأسف', 'اقتصاد', 'كسر', 'جہنم', 'يالله', 'منى', 'نقد', 'بچے', 'وشوووو', 'تسبقها', 'الخاص', 'والدى', 'تكن', 'أعلنا', 'بسباق', 'sourdough', 'هو', 'وابتسامك', 'الجوية', 'علاقتها', 'اششراء', 'ونحو', 'بغاها', 'میمو', 'وإنما', 'السناب', 'alwayslearn', 'واصحابه', 'الوافين', 'الملك', 'r_creedbalboa76', 'تبريرات', 'خزينة', 'القائمين', 'وكل', 'بشاير', 'نواياها', 'basmah_omar', 'أهداف', 'مرور', 'أ', 'المشكلة', 'صالح_الدلال', 'المرء', 'بــ', 'مصدر', 'كلن_يعني_كلن', 'ہن', 'نظيف', 'صفحة', 'الخاصة', 'اللهم', 'تواضعا', 'lolarabia', 'فرح', 'غیر', 'pathogen', 'اسمع', 'بکاو', 'النارا', 'سنوات', 'شخص', 'مانشستر', 'ه', 'مبروك', 'تابعوني', 'نفتح', 'salford', 'ودلوقتى', 'بمكة', 'يعو', 'يناديني', 'بهذه', 'صح', 'ياأميرة', 'قلب', 'الجميلة', 'whatawast', 'گل', 'للمعرض', 'الطايرة', 'عطا', 'هيلعب', 'شيء', 'behalf', 'اننا', 'ومش', 'الاوادم', '_jimmydx', 'النشمي', 'الموضوع', 'نحتاج', 'ماخذين', 'إبعادها', 'لاتعقدهة', 'اول', 'ومكالمات', 'وجدى', 'بتتعمل', 'بصوت', 'كانوا', 'rdmgough', 'شو', 'emanshaband', 'دقيقة', 'وهل', 'ياربي', 'ودامت', 'النادي', 'وعنده', 'احد', 'نعمه', 'يفشل', 'تصویر', 'اشراء', 'نیا', 'به', 'تتوقع', 'راجعون', 'إلى', 'القادم', 'عبد', 'اشوفكم', 'المحيط', 'لنهرب', 'مناسبين', 'انجليزية', 'يخيب', 'لگائی', 'اهو', 'البيت', 'سپورٹ', 'عندما', 'كسبنا', 'طموحاته', 'ibuttar', 'للبقاء', 'formula', 'عمري', 'ولا', 'خبير', 'جميع', 'لا', 'عين', 'جدا', 'الاربعاء', 'الأمور', 'يلي', 'جمالك', 'زيارات', 'حرکت', 'خيراته', 'دام', 'lulufalmulhim', 'المتظاهرين', 'عامرة', 'فرماۓ', 'لم', 'كده', 'بعض', 'کی', 'تيانا', 'الدوام', 'اللہ', 'بعضهم', 'عني', 'منين', 'نزار', 'شهيد', 'الخير', 'يكسر', 'القلوب', 'حمد', 'لعبة', 'بأن', 'تحقيقات', 'هذة', 'سودمند', '٥', 'الرشيدي', 'باللكاعة', 'اطوف', 'symbol', 'اليوم', 'اختبار', 'خسرنا', 'الوقت', 'بتواجدي', 'اپنے', 'جميعا', 'ساعه', 'انسانی', 'البشر', 'بالعكس', 'الvoic', 'صبحان', 'اذا', 'علشان', 'على', 'بتدخل', 'كاراباو', 'تتدف', 'إن', 'ولی', 'الطبية', 'جواز_العصمة_بيد_المرأة', 'اي', 'سنين', 'مروه', 'ليفربول', 'انه', 'لن', 'ضو', 'وأنت', 'شارع٥٠٦', 'alrayah', 'مظبوطة', 'منسية', 'مشكله', 'معه', 'حفظ', 'ظنك', 'بهامة', 'قديم', 'أدانيه', 'منسدقها', 'الروح', 'المساء', 'وقصص', 'يادبلن', 'المخلوع', 'البطولتين', 'وزاده', 'افلامهم', 'قاعده', 'صاحب', 'هنا', 'thefirewithin', 'وعزوتي', 'لينا', 'ترة', 'حتى', 'کیاے', 'چ', 'روحه', 'cheaper', 'منطقة', 'كلمتان', 'إسمها', 'منا', 'رحمة', 'sayyed3li', 'الخوه', 'لمشاركتنا', 'ماخسرنا', 'ياااارب', '1sheryfa', 'ماعرفت', 'لحربنا', 'سورة', 'بالتوفيق', 'تملا', 'قلبين', 'وفرحة', 'two', 'وحضورك', 'إليكم', 'إله', 'عائشة', 'اخر', 'microchip', 'عائلة', 'alrayahuk_i', 'تجربتها', 'الهجر', 'وحشتيني', 'ديز', 'هم', 'غضبانين', 'asmi_malik_', 'يمكن', 'bluebadg', 'يسعنا', 'حبيبتي', 'وربي', 'التغطية', 'وانا', 'تكلني', 'aishaahmedjug', 'المقتطفات', 'سعيد', 'peek', 'pizza', 'كيوت', 'شاء', 'الكوريين', 'بيخافوا', 'تدري', 'مقابلة', 'شکل', 'آمین', 'فعله', 'شأني']\n"
]
}
],
"source": [
"print(list(c[5]))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Furthermore, I looked for Hungarian words. Based on [Wiktionary](https://en.wiktionary.org/wiki/Wiktionary:Frequency_lists/Hungarian_frequency_list_1-10000), in a code cell below, one can see the 50 most frequent words."
]
},
{
"cell_type": "code",
"execution_count": 261,
"metadata": {
"ExecuteTime": {
"end_time": "2019-11-19T18:29:20.737910Z",
"start_time": "2019-11-19T18:29:20.728645Z"
},
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [],
"source": [
"szavak = [\"meg\", \"vagy\", \"van\", \"vagyok\", \"vannak\", \"vagytok\", \"volt\", \"már\", \"kell\", \n",
" \"még\", \"és\", \"mint\", \"azt\", \"az\", \"akkor\", \"sem\", \"lehet\", \"mert\", \"minden\", \"olyan\",\n",
" \"szerint\", \"pedig\", \"ezt\", \"ez\", \"így\", \"után\", \"úgy\", \"nagy\", \"fel\", \"majd\", \"két\", \n",
" \"nem\", \"nagyon\", \"aki\", \"akik\", \"akit\", \"kit\", \"kik\", \"most\", \"több\", \"lesz\", \"itt\",\n",
" \"magyar\", \"ami\", \"amik\", \"amit\", \"mit\", \"első\", \"között\", \"amely\", \"hanem\", \"nincs\",\n",
" \"más\", \"illetve\", \"alatt\", \"egyik\", \"volna\", \"arra\", \"kft\", \"ilyen\", \"azonban\"] #deleted \"a\""
]
},
{
"cell_type": "code",
"execution_count": 82,
"metadata": {
"ExecuteTime": {
"end_time": "2019-11-19T12:57:04.300175Z",
"start_time": "2019-11-19T12:57:04.271256Z"
},
"scrolled": false,
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"van van\n",
"[('van', 'and'), ('van', 'i'), ('van', 'of'), ('van', 'is'), ('van', 'the'), ('van', 'me'), ('van', 'my')]\n",
"\n",
"mint mint\n",
"[('mint', 'by'), ('mint', 'last'), ('mint', 'follow'), ('mint', 'it'), ('mint', 'this'), ('mint', 'the'), ('mint', 'absolut')]\n",
"\n",
"az az\n",
"[('az', 'is'), ('az', 'via'), ('az', 'bekliyoruz'), ('az', 'sunayakin'), ('az', 'ağabey'), ('az', 'kaldı'), ('az', '19')]\n",
"\n",
"sem sem\n",
"[('sem', 'do'), ('sem', 'a'), ('sem', 'como'), ('sem', 'de'), ('sem', 'se'), ('sem', 'o'), ('sem', 'que')]\n",
"\n",
"ez ez\n",
"[('ez', 'is'), ('ez', 'via'), ('ez', 'az'), ('ez', 'te'), ('ez', 'nagyon'), ('ez', 'megérintett'), ('ez', 'ügi')]\n",
"\n",
"fel fel\n",
"[('fel', 'i'), ('fel', 'un'), ('fel', 'y'), ('fel', 'n'), ('fel', 'w'), ('fel', 'r'), ('fel', 'yn')]\n",
"\n",
"nagyon nagyon\n",
"[('nagyon', 'is'), ('nagyon', 'via'), ('nagyon', 'az'), ('nagyon', 'te'), ('nagyon', 'megérintett'), ('nagyon', 'ez'), ('nagyon', 'ügi')]\n",
"\n",
"aki aki\n",
"[('aki', 'by'), ('aki', 'and'), ('aki', 'to'), ('aki', 'save'), ('aki', 'up'), ('aki', 'work'), ('aki', 'of')]\n",
"\n",
"kit kit\n",
"[('kit', 'good'), ('kit', 'and'), ('kit', 'just'), ('kit', 'to'), ('kit', 'over'), ('kit', 'bar'), ('kit', 'i')]\n",
"\n",
"most most\n",
"[('most', 'good'), ('most', 'chanc'), ('most', 'by'), ('most', 'last'), ('most', 'and'), ('most', 'now'), ('most', 'just')]\n",
"\n",
"ami ami\n",
"[('ami', 'good'), ('ami', 'and'), ('ami', 'to'), ('ami', 'your'), ('ami', 'you'), ('ami', 'have'), ('ami', 'a')]\n",
"\n",
"más más\n",
"[('más', 'me'), ('más', 'a'), ('más', 'london'), ('más', 'porqu'), ('más', 'como'), ('más', 'un'), ('más', 'de')]\n",
"\n"
]
}
],
"source": [
"not_in = []\n",
"for szó in szavak:\n",
" stemd = snow.stem(szó)\n",
" if g.has_node(stemd):\n",
" print(szó, stemd)\n",
" print(list(G2.edges(stemd))[:7])\n",
" print()\n",
" else:\n",
" not_in.append(stemd)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Above are those \"Hungarian\" words which were present in the network. The listing is the following; in the first line one can see the pure word and the stemmed one (which appeared to be the same for all) and below the 7 of its connections. Unfortunately many Hungarian words have an English meaning, too (\"van\", \"mint\", \"kit\", \"most\"). There are two words which have either Spanish and/or Portuguese links (\"sem\", \"más\") and one with Turkish connections (\"az\"). However, it did find some Hungarian connections for \"ez\" and \"nagyon\". Maybe, do these words make up a whole cluster? Let us check for the cardinality of their communities."
]
},
{
"cell_type": "code",
"execution_count": 80,
"metadata": {
"ExecuteTime": {
"end_time": "2019-11-19T12:46:29.017626Z",
"start_time": "2019-11-19T12:46:28.882578Z"
},
"scrolled": false,
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"11 mint 4441\n",
"13 az 4441\n",
"15 sem 4441\n",
"23 ez 4441\n",
"28 fel 4441\n",
"32 nagyon 4441\n",
"36 kit 4441\n",
"52 más 4441\n",
"2 van 2874\n",
"33 aki 2874\n",
"38 most 657\n",
"43 ami 139\n"
]
}
],
"source": [
"for i in sorted(c):\n",
" for cou, szó in enumerate(szavak):\n",
" stemd = snow.stem(szó)\n",
" if stemd in i:\n",
" print(cou, stemd, len(i))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The listing here means the following: (rank in frequency, word, cardinality of its cluster). Unfortunately, the words which had Hungarian link appear to be in the largest \"quite meaningless\" group. Therefore we can say that the algorithm did not find a Hungarian cluster. "
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## Conclusions\n",
"\n",
"\n",
"During this project I was not only able to utilize all the learnt abilities from the dataexp course but also got results which are in accordance with the literature. Gathering the tweets and preparing the data might have been the most difficult part of all the work, yet, measuring the properties of the network was undoubtedly the most time-consuming task. Thus, there are many improvements, including for example thresholding for number of occurrences or filtering for those tweets which were reposted or the user was disturbingly active posting. Furthermore, other stemmers probably would have stemmed the (foreign) words in other which would then create a slightly different network. Finally, if I had had the resources to conduct a measurement involving more complex community findig algorithms, comparing the outcoming clusters would have said a lot about the robustness about each method.\n",
"\n",
"\n",
"Nevertheless, proving scale-free property and finding different communities not only based on the language but topic also are great achievements to notice. In my opinion I did everything I could to do my based on my time schedule. This project unambiguously helped my professional career as it seems to me, many companies do greet having knowledge about natural language processing. "
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.4"
},
"toc": {
"base_numbering": 1,
"nav_menu": {},
"number_sections": true,
"sideBar": true,
"skip_h1_title": false,
"title_cell": "Table of Contents",
"title_sidebar": "Contents",
"toc_cell": false,
"toc_position": {},
"toc_section_display": true,
"toc_window_display": false
}
},
"nbformat": 4,
"nbformat_minor": 2
}