Example of Natural Language Processing (NLP) in Python for

finance
Saeed Amen / Founder of Cuemacro

https://www.cuemacro.com / [email protected] / @saeedamenfx / All material is copyright Cuemacro / 2021

In this example section, we'll show how to do topic modelling. We'll use news articles from BBC News online, and see if we can identify
groups of articles using LDA, without using predefined classifications.

If you have time you can try to repeat the exercise, but with other news websites. You'll need to look up the topic URLs and will likely
need to change the extract_story_links function later if you do this.

Extract URLs from main topic pages

Our first step is to find a few URLs of the main topic pages on BBC News Online. We'll just choose a selection of the topic pages.

In [1]: # Create a dictionary with the topics and their respective URLs on BBC News Online

Let's write a function which goes through the main BBC News topic pages and grabs the URLs for the articles (but ignores the links in
the header, footer etc.).

We will be using regular expressions to match the text. Regular expressions can be pretty difficult to understand, but can be very
useful when processing text for use in NLP. It's recommend you have a look at the official Python reference for regular expressions
https://docs.python.org/3/howto/regex.html

In [2]: # Do all the imports

# Grab webpage

# Get HTML links which start with /news/havesomeletters-andfinishwithnumbers

# eg. /news/business-12686570 (using regular expressions)
# Then add https://www.bbc.co.uk at the front

# Remove duplicated links

Now download the story links for each topic page using extract_story_links function. Show the extracted links for the Family &
Education topic. In some cases, we might observe duplicated articles. A further step could be to remove duplicates.

In [3]: # For every topic page, extract the story links

# Show the links for family & education pages

['https://www.bbc.co.uk/news/world-europe-isle-of-man-60765283',
Out[3]:
'https://www.bbc.co.uk/news/education-60785871',
'https://www.bbc.co.uk/news/education-60683839',
'https://www.bbc.co.uk/news/uk-northern-ireland-60787327',
'https://www.bbc.co.uk/news/uk-60557186',
'https://www.bbc.co.uk/news/world-asia-60689267',
'https://www.bbc.co.uk/news/uk-northern-ireland-60768948',
'https://www.bbc.co.uk/news/uk-england-london-60733528',
'https://www.bbc.co.uk/news/health-60698414',
'https://www.bbc.co.uk/news/uk-england-cumbria-60791262',
'https://www.bbc.co.uk/news/uk-scotland-scotland-business-60780953',
'https://www.bbc.co.uk/news/uk-scotland-60575939',
'https://www.bbc.co.uk/news/uk-england-suffolk-60783162',
'https://www.bbc.co.uk/news/uk-england-london-60769462',
'https://www.bbc.co.uk/news/uk-england-60793837',
'https://www.bbc.co.uk/news/uk-england-60707381',
'https://www.bbc.co.uk/news/science-environment-60708712',
'https://www.bbc.co.uk/news/uk-northern-ireland-60785082',
'https://www.bbc.co.uk/news/education-60769468',
'https://www.bbc.co.uk/news/uk-england-york-north-yorkshire-60774056',
'https://www.bbc.co.uk/news/world-africa-60603226',
 'https://www.bbc.co.uk/news/world-europe-isle-of-man-60788290',
'https://www.bbc.co.uk/news/uk-england-beds-bucks-herts-60695965',
'https://www.bbc.co.uk/news/health-58741536',
'https://www.bbc.co.uk/news/uk-england-derbyshire-60515276',
'https://www.bbc.co.uk/news/uk-wales-60684874',
'https://www.bbc.co.uk/news/science-environment-56837908',
'https://www.bbc.co.uk/news/world-europe-60632147',
'https://www.bbc.co.uk/news/uk-scotland-60587499',
'https://www.bbc.co.uk/news/technology-60759050',
'https://www.bbc.co.uk/news/uk-england-somerset-59418678',
'https://www.bbc.co.uk/news/uk-england-bristol-60793364',
'https://www.bbc.co.uk/news/business-60757928',
'https://www.bbc.co.uk/news/uk-wales-60779911',
'https://www.bbc.co.uk/news/world-60525350',
'https://www.bbc.co.uk/news/uk-england-devon-60480724',
'https://www.bbc.co.uk/news/health-60377339',
'https://www.bbc.co.uk/news/uk-england-manchester-60792926']

Download text of each news article

Here, we create a function that will get the body text of a URL using the Newspaper library and ignore all other text on the page. Note
that there can be issues where Newspaper doesn't seem to fully download articles from BBC News (possibly due to a site redesign).

In [4]:

Using the grabbed links, we'll get the body text from those pages using get_story_text . Let's take a look at the articles
downloaded for Family & Education.

In [5]: # Show the links for family & education pages

['The document was drafted after a Tynwald committee highlighted social and economic issues relating to the
Out[5]:
inadequacy of provision and will be put out for consultation before being debated in July.',
'Saint said: "I don\'t think it really surprises me at all. As celebrities, for example, have decided who
they are, it\'s become easier for young people to look at the people they admire and think it\'s OK to expr
ess myself.',
'Primary maths is one of the subjects where there are shocking regional differences: the average learning
loss was 0.5 months in the South West, 0.9 months in London, four months in the North East and 5.3 months i
n Yorkshire and the Humber.',
'"The increase in the proportion of entrants continuing in higher education (HE) after their first year ca
nnot be directly attributed to the Covid-19 pandemic but there is often a trend for increased HE enrolments
in periods of economic uncertainty," HESA said.',
'Ukraine conflict: How to help yourself, your kids, and others By Lauren Potts\n\nBBC News Published 1 Mar
ch\n\nImage source, Constantinis/Getty Images Image caption, It\'s normal to feel upset about what\'s happe
ning in Ukraine, according to experts\n\nIf you woke up this morning, looked at the news, and felt increasi
ngly worried about the war in Ukraine, you are not alone. After a two-year pandemic, it\'s a lot to absorb,
and experts agree that feeling overwhelmed is normal.\n\nHere is their advice on how you can take care of y
ourself, your kids - and others.\n\nWhat you can do to help yourself\n\nWhile it\'s right to think first an
d foremost about the impact on those caught up in the conflict, it\'s also completely normal to feel upset
from afar by what we\'re seeing in Ukraine, says Alex Bushill, from the mental health charity Mind.\n\n"It
\'s very natural to be distressed by what we\'re seeing, you wouldn\'t be human if you didn\'t," he says.\n
\nThis doesn\'t always lead to anxiety, but the NHS and Anxiety UK agree on some key ways to avoid it: eat
well, get outside, put your phone down, connect with people, rest. These are all pretty basic pieces of adv
ice, but when you\'re stressed, they can be difficult to do consistently.\n\nAlex says breaking it down int
o a two-step process can help. The first is to remove yourself from triggers so that you can practice mindf
ulness.\n\n"Get yourself into a place where you can be in the moment - whether that\'s sport or walking the
dog," he says.\n\nThe second step is to focus on self-discipline around specific techniques that work for y
ou. "Create the space to take a lunch break or play squash on a Tuesday with your mate - and to make sure y
ou do that."\n\n🧵 There\'s no question that the news is difficult to read and absorb right now. But we\'ve
got some things that might help you feel a bit less stressed. Settle in for a thread. (1/12) — Mind (@MindC
harity) February 25, 2022 The BBC is not responsible for the content of external sites. View original tweet
on Twitter\n\nExperts agree that if you\'re feeling anxious, avoid doomscrolling - the act of spending an e
xcessive amount of time consuming negative news.\n\nClinical psychologist Dr Emma Hepburn says it\'s someth
ing we often do when faced with uncertainty, but instead of filling in the gaps with useful information, we
can end up catastrophising.\n\n"The way we seek clarity is to look for more information, but that often cre
ates less clarity, cause we\'re seeing the same information again and again and it doesn\'t allow us to ste
p back."\n\nShe also points out that you\'re allowed to feel safe and be concerned. "The two aren\'t mutual
ly exclusive."\n\nMind advises only looking at news and social media at certain times of the day, for a lim
ited duration, and then doing something relaxing afterwards.\n\nAlex also recommends being conscious of whe
re you get your information: rely on trusted news sources, focus on facts rather that alarmist speculation,
don\'t engage with graphic content.\n\n"When I do it, I step away, and it clears my mind, [it works] becaus
e you\'re not thinking about things you can\'t control," he says.\n\nImage source, Getty Images Image capti
on, Stepping away from the constant news cycle can help manage anxiety\n\nSarah Kendrick, clinical director
with text support service Shout, says she is particularly concerned about young people, who are typically o
n their phones a lot and accessing news 24/7.\n\n"Children and young people are texting in saying they\'re
worried about war, some say it\'s keeping them awake at night," she says. "They\'re up at night with access
to their phones and with news alerts going off."\n\nThe guidance that Shout has been giving out sounds simp
le - turn off your phone at night and don\'t feel like you have to keep looking at the news - but it can be
hard to put into practice.\n\n"We always say, try to remember what we can control, and you can certainly co
ntrol the amount of media you consume and how much sleep you have and how much you reach out for suppor
t."\n\nWhat you can do for your kids\n\nThe first thing to know is that avoiding the topic can make childre
n feel scared, says Ane Lemche, a psychologist and child counsellor with Save the Children.\n\nProf Vivian
Hill, from the British Psychological Society, recommends making the conversation age appropriate. For young
er children, she suggests, this could be as simple as showing them where Ukraine is on a map, so they under
stand it\'s happening somewhere far away from them. For older children, it might include providing some con
text.\n\nBut at any age, the key thing is to reassure them.\n\n"Keep it as matter-of-fact as you can: expla
in that there have been wars all over the world in the past few years and we\'ve been lucky that most of th
em haven\'t had any impact on families in the UK - it\'s a good way of helping them to have a sense of the
degree of risk that\'s posed to them," says Prof Hill.\n\n"And that\'s what children want reassurance about
- that they\'re safe."\n\nIf they are worried about people in Ukraine, she suggests talking about the measu
res they are taking to protect themselves. "You talk to them about how most people are in air raid shelters
and away from the bombs, or you may talk about how people are moving to other countries where it\'s safe
r."\n\nImage source, Getty Images Image caption, It\'s important to talk to children about the conflict so
they feel less scared\n\nIf parents themselves are feeling anxious, children can pick up on that, so it\'s
important adults get support too, she adds.\n\n"If you\'re feeling really upset and your child asks you a q
uestion, say \'can we talk about this a little bit later\'.\n\n"And if you don\'t feel up to it, find someo
ne else who can do it - you might have a partner or a member of extended family who\'s better at coping wit
h that."\n\nShe says it\'s normal for adults to feel "wobbly" post-pandemic and to wonder when there might
be a period of stability. She recommends making sure there are "protected times" where there is no exposure
to news, such as switching off the radio during the school run.\n\n"The advice I\'d give to children is jus
t as much relevant to the adults. You have to try hard to guard time where you just have fun and escape all
of the bleakness. It\'s about having compassion for everyone including yourself."\n\nWhat you can do for ot
hers\n\nHelping others is one of the five steps recommended by the NHS to improve mental wellbeing and ther
e are countless ways to help with the Ukraine crisis.\n\nA number of charities have launched appeals, inclu
ding the British Red Cross, the UNHCR refugee agency and Unicef, while many individuals across the UK are r
aising money and collecting donations independently.\n\nDr Hepburn says that watching war unfold can make y
ou feel out of control, and focusing on ways to help can manage that.\n\n"There\'s lots of evidence that do
ing something for other people has a really beneficial impact on your own mental health. If we perceive tha
t we made someone else feel good, we feel good. Our brains are designed to be social and have connections a
nd creating a connection with something bigger than you is really beneficial to your wellbeing."\n\nImage s
ource, Dr Emma Hepburn\n\nDr Hepburn "draws psychology" and posts the images on her Instagram page, The Psy
chology Mum. Her most recent illustration is about how people can help with the Ukraine crisis.\n\n"It\'s a
bout making sense of a complex situation. And I guess it was me trying to do something as well."\n\nShout,
which is looking for volunteers, believes there are benefits for people on both ends of the phone.\n\n"We
\'ve all found ourselves talking about this war to friends and family and there\'s a really big reason for
that and that\'s because those conversations are comforting," explains Sarah.\n\n"One of the great things a
bout being involved in a service that helps other people is the sense of satisfaction or accomplishment you
get, and that\'s what people are looking for.\n\n"People can feel helpless in the face of war or big world
events, and taking it back to the small things we can do is hugely enriching for people."\n\nProf Hill says
"kids absolutely love" finding positive ways to help others and recommends getting them involved.\n\n"Injus
tice is something kids feel very strongly about, so get them to think about ways they can help the Ukrainia
n community," she says.\n\n"There are people collecting donations, things to send to refugee centres, do so
mething like that. Sometimes doing something constructive is really powerful."',
"Watch as the BBC follows some of the thousands of Indian students who've been evacuated from Ukraine - wi
th one so traumatised that he is unable to speak.\n\nIndia's government is evacuating the final group of mo
re than 17,000 Indian students who were stranded.\n\nThe BBC’s Divya Arya is on the Polish-Ukrainian border
hearing some of their stories.\n\nVideo by Neha Sharma",
'"The trust takes very seriously the areas identified as requiring improvement in Lakeview Hospital and ha
s demonstrated its willingness to take whatever actions are necessary to ensure the service is compliant wi
th the relevant standards and practices."',
'Meanwhile the Race Review found: "Significant events, such as the spotlight on Black Lives Matter, led to
an increased awareness of racism but the continued denial of the racism and the invisibility of the issue a
s part of the school environment led to a dichotomy between those for whom this was very important and thos
e who felt that they had \'done awareness\' - so there is no need to \'keeping talking about it\'."',
'"What we will be doing over the next few months is trying to make sense of all the information that is av
ailable, as well as seeing if we can plug any of the gaps in the research," she writes.',
'Mr Hayes said: "Our aim at QKS is to provide the best and broadest possible educational experience for al
l of our young people, and having Sandgate students as part of our community is a key part of that philosop
hy."',
'Craig Johnstone, who will remain at the company, said: "This private equity-backed acquisition comes as w
e look to accelerate our growth across the UK, Ireland and internationally and at a time when the edtech ma
rket has never been more buoyant."',
'She continued: "Whilst we have many centres of excellence, unfortunately there are whole geographic areas
of Scotland that are not properly served and whole areas of the population who feel either that they are co
mpletely overlooked or the services don\'t cater for their particular type of eating disorders."',
'Mrs Grimes said the last two years had been challenging but they had also "probably been two very exiting
years because we\'ve come a long way, we\'ve changed our curriculum, and we\'ve thought really carefully ab
out what the children need for their wellbeing," she said.',
'He apologised on behalf of the Met Police "to the child concerned, her family and the wider community", a
nd said it was "wholly right that the actions of officers are held to account".',
 'Birmingham councillor Sharon Thompson, who chairs the West Midlands Combined Authority homelessness taskf
orce, said: "Whilst this is great news, the devil is in the detail and I will be working to ensure that the
package of measures set out in [the] announcement is implemented in a way that brings the real change that
is so desperately needed."',
'The focus will be to admit large numbers of students on free school meals, from disadvantaged postcodes o
r who would be the first in their families to go to university, the partners said.',
'It finds that just 21 universities out of the 93 that signed up for the award hold a bronze Race Equality
Charter award, run by the higher education charity Advance HE. None have received a silver or gold award, d
espite the scheme being launched in 2016. Similarly, 37% of FTSE 100 companies have no representation of mi
nority ethnicities on their board, despite a target set by an independent review, also in 2016, to have one
director from a minority ethnic background on every board by 2021.',
'Glenveagh also said it would make a decision on Monday about the rest of the week, but told parents they
may need to make their own transport arrangements and send a packed lunch for their child if they were comi
ng into school.',
'"We don\'t want to see the government giving with one hand to ethnic minority communities and taking away
with the other, but instead action from the whole of government is needed to address racial disparities in
our society."',
'"We have made repeated attempts to correct the facts before the report was published, we do not lightly s
tand up to our regulator but in this instance the injustice cannot be allowed to stand."',
'She said they had begged the university to allow them to leave but, she added, the students were told the
y would be fined if they missed their classes, which is standard policy in some Ukrainian universities, so
"we didn\'t have a choice".',
'In a letter to parents, Ms Findlater said staff would inform the 158 pupils affected of the work that sho
uld be completed at home on Friday and Monday "where possible", but that would not happen in all cases "wit
h such a significant amount of the teaching workforce absent".',
'A drummer who was part of rock group Status Quo for 15 years has been giving drumming lessons for pupil
s.\n\nJeff Rich met and performed for young musicians at Sir Herbert Leon Academy in Bletchley, Buckinghams
hire, to help them sharpen their skills.\n\nHe decided to swap the stage for the classroom to try to inspir
e them into a career in music.\n\n"Hopefully they\'ve taken on board what I\'ve been saying about practisin
g and working hard. I really enjoyed it today," he said.',
"The NHS Digital report authors said it was difficult to work out how much of the change in children's men
tal health was because of the pandemic - but they found those with a probable mental health disorder were m
ore likely to be worse off.",
'An international grime artist has swapped the stage for the classroom to teach college students how to ra
p.\n\nElijah Branford, aka Eyez, has started teaching music at Derby College while completing his teacher t
raining.\n\nThe rapper, who is a former student at the college, has performed at the O2 in London and his m
usic has been streamed by millions online.\n\nDerby College said it hoped the lessons, which cover the genr
es of grime, drill and hip-hop, will help more teenagers in the city to engage with higher education.\n\n"I
really wanted to give back to the youth and I really wanted to be that change that I want to see in educati
on, especially when it comes to music," Mr Branford said.\n\nVideo journalist: Alex Thorp\n\nFollow BBC Eas
t Midlands on Facebook, Twitter, or Instagram. Send your story ideas to [email protected].',
"CPR training is set to be offered to all students at Cardiff University in a bid to save more lives.\n\nS
tudent medics are teaching their fellow students in a scheme that could be rolled out at other universitie
s.\n\nJust one in 20 people in Wales survive cardiac arrest - one of the worst rates in Europe.\n\nIf all u
niversities in Wales offered CPR training, up to 130,000 people would learn skills that could save someon
e's life.",
'A new bill is intended to address concerns about unsold products being destroyed or going to landfill.\n
\nRead more next',
'"This was a challenge for me as a mother because I have to choose between how to explain the truth, but a
t the same time not to scare them too much," she says. "So I was just telling them that we are under attac
k, and at the moment we are safe, but when we feel we have to move, they have to listen."',
"Some wanted to integrate more with local life and felt the university wasn't doing enough to facilitate t
hat, being too ready to house them in large exclusively Chinese blocks of residence - like the city's West
Village development - and not doing enough to develop their English language skills.",
'"Providing age-appropriate and safe experiences for teens while also empowering them to explore in VR is
a balancing act", Oculus, the Meta-owned company that makes the Quest, blogged.',
'She said it was really helpful to have a self-care package with toiletries in to help her look after hers
elf at a time when she was not able to.',
'It also said: "The retention of the War Room for employment purposes, as either a commercial or community
use, would overall be beneficial as it would retain some job opportunities in the area."',
'"That\'s why we are calling on the UK government to bring benefits in line with the forecast rate of infl
ation as a bare minimum in the upcoming Spring Statement, to prevent thousands more people being forced int
o debt and through the doors of food banks."',
'"My daughter\'s with me, my mum and my nan so all of us are here and it\'s been really nice because they
\'ve seen how much I\'ve had to work and how difficult it\'s been."',
"BBC Copyright: BBC A group of young refugees from Ukraine take part in a karate class in eastern Poland I
mage caption: A group of young refugees from Ukraine take part in a karate class in eastern Poland\n\nIn th
e early stages of Russia's invasion of Ukraine, most of the Ukrainians fleeing the war went to stay with fa
mily and friends already living in Poland. As time has passed, the proportion of refugees with nowhere to s
tay has grown.\n\nThe number of refugees fleeing Ukraine for Poland has now reached two million and at leas
t 1.1 million of them have stayed. And although thousands of Poles have opened their own homes to them, lar
ge Polish cities including Warsaw and Krakow that are attracting the most are saying they have reached thei
r capacity to accommodate them.\n\nTemporary accommodation centres set up in sports halls and conference ce
ntres across the country are designed to cater to people for a few days only. Authorities urgently need to
make housing available that can accommodate the refugees for several months. Both the government and Presid
ent Andrzej Duda are calling for greater international assistance, recognising the response to the crisis w
ill not be a sprint but a marathon.\n\nThere’s still broad support across the political spectrum for the re
 fugees. Only the small far-right Confederation party is complaining about Ukrainians receiving benefits tha
t Poles do not, such as free travel on the trains. Journalists boycotted their most recent news conference
in disgust.",
'"That is simply because the minute something is out there in the public domain, you can almost guarantee
that the agent will have viewings that day, and if you can\'t make that viewing, well tough and there are m
aybe 10 other people lined up."',
"A review of pregnancy loss services is England is currently investigating the lack of support services fo
r partners after miscarriage. It’s expected to make recommendations later this year. Partners often report
feeling helpless, ignored and isolated during and after a miscarriage.\n\nIn the BBC's latest special repor
t on pregnancy loss, global health correspondent Tulip Mazumdar has been to meet two dads who started a pod
cast to discuss their experiences.\n\nWatch their story.\n\nIf you've been affected by issues in this vide
o, help and support is available via BBC action line.\n\nReporting by Tulip Mazumdar\n\nFilmed by Rachel Pr
ice\n\nProduced by Gabby O'Donnell\n\nEdited by Joshua Falcon",
'"As well as meet a clear housing need, our proposals will create a brand-new residential community, boost
the local economy and create a place that will that we believe people will want to call home for the long t
erm."']

We'll now combine the articles into one big list, thus removing any predefined topic tags. Later, we'll see if LDA can group the articles
even if it doesn't know the topics beforehand.

In [6]: # Extract the articles from the dictionary and combine into a big list

['The document was drafted after a Tynwald committee highlighted social and economic issues relating to the
inadequacy of provision and will be put out for consultation before being debated in July.', 'Saint said:
"I don\'t think it really surprises me at all. As celebrities, for example, have decided who they are, it
\'s become easier for young people to look at the people they admire and think it\'s OK to express mysel
f.', 'Primary maths is one of the subjects where there are shocking regional differences: the average learn
ing loss was 0.5 months in the South West, 0.9 months in London, four months in the North East and 5.3 mont
hs in Yorkshire and the Humber.', '"The increase in the proportion of entrants continuing in higher educati
on (HE) after their first year cannot be directly attributed to the Covid-19 pandemic but there is often a
trend for increased HE enrolments in periods of economic uncertainty," HESA said.']

Remove stopwords
Next, we'll remove the stop words, that is words that don't add meaning to the text. We'll use NLTK, and also add a few extra ones
(we've gone through the text a few times and topics to help).

In [7]: # Imports for the stop words from NLTK

# Add our own stop words too

# For each text in the corpus

# Split it up into tokens
# Then strip the tokens which match the stopwords

# Put it back together the tokens into a full text

We can now look at the filtered_corpus , to see the stop words have been removed.

In [8]:

['document drafted Tynwald committee highlighted social economic issues relating inadequacy provision put c
Out[8]:
onsultation debated July .',
"Saint : `` I n't think really surprises . As celebrities , example , decided , 's become easier young peo
ple look people admire think 's OK express .",
'Primary maths one subjects shocking regional differences : average learning loss 0.5 months South West ,
0.9 months London , four months North East 5.3 months Yorkshire Humber .',
"`` increase proportion entrants continuing higher education ( HE ) first year directly attributed Covid-1
9 pandemic often trend increased HE enrolments periods economic uncertainty , '' HESA .",
"Ukraine conflict : How help , kids , others By Lauren Potts BBC News Published 1 March Image source , Con
stantinis/Getty Images Image caption , It 's normal feel upset 's happening Ukraine , according experts If
woke morning , looked news , felt increasingly worried war Ukraine , alone . After two-year pandemic , 's l
ot absorb , experts agree feeling overwhelmed normal . Here advice take care , kids - others . What help Wh
ile 's right think first foremost impact caught conflict , 's also completely normal feel upset afar 're se
eing Ukraine , Alex Bushill , mental health charity Mind . `` It 's natural distressed 're seeing , would
n't human n't , '' . This n't always lead anxiety , NHS Anxiety UK agree key ways avoid : eat well , get ou
tside , put phone , connect people , rest . These pretty basic pieces advice , 're stressed , difficult con
sistently . Alex breaking two-step process help . first remove triggers practice mindfulness . `` Get place
moment - whether 's sport walking dog , '' . second step focus self-discipline around specific techniques w
ork . `` Create space take lunch break play squash Tuesday mate - make sure . '' 🧵 There 's question news
difficult read absorb right . But 've got things might help feel bit less stressed . Settle thread . ( 1/12
) — Mind ( @ MindCharity ) February 25 , 2022 BBC responsible content external sites . View original tweet
Twitter Experts agree 're feeling anxious , avoid doomscrolling - act spending excessive amount time consum
 ing negative news . Clinical psychologist Dr Emma Hepburn 's something often faced uncertainty , instead fi
lling gaps useful information , end catastrophising . `` way seek clarity look information , often creates
less clarity , cause 're seeing information n't allow us step back . '' She also points 're allowed feel sa
fe concerned . `` two n't mutually exclusive . '' Mind advises looking news social media certain times day
, limited duration , something relaxing afterwards . Alex also recommends conscious get information : rely
trusted news sources , focus facts rather alarmist speculation , n't engage graphic content . `` When I , I
step away , clears mind , [ works ] 're thinking things ca n't control , '' . Image source , Getty Images I
mage caption , Stepping away constant news cycle help manage anxiety Sarah Kendrick , clinical director tex
t support service Shout , particularly concerned young people , typically phones lot accessing news 24/7 .
`` Children young people texting saying 're worried war , say 's keeping awake night , '' . `` They 're nig
ht access phones news alerts going . '' guidance Shout giving sounds simple - turn phone night n't feel lik
e keep looking news - hard put practice . `` We always say , try remember control , certainly control amoun
t media consume much sleep much reach support . '' What kids first thing know avoiding topic make children
feel scared , Ane Lemche , psychologist child counsellor Save Children . Prof Vivian Hill , British Psychol
ogical Society , recommends making conversation age appropriate . For younger children , suggests , could s
imple showing Ukraine map , understand 's happening somewhere far away . For older children , might include
providing context . But age , key thing reassure . `` Keep matter-of-fact : explain wars world past years
've lucky n't impact families UK - 's good way helping sense degree risk 's posed , '' Prof Hill . `` And
's children want reassurance - 're safe . '' If worried people Ukraine , suggests talking measures taking p
rotect . `` You talk people air raid shelters away bombs , may talk people moving countries 's safer . '' I
mage source , Getty Images Image caption , It 's important talk children conflict feel less scared If paren
ts feeling anxious , children pick , 's important adults get support , adds . `` If 're feeling really upse
t child asks question , say 'can talk little bit later ' . `` And n't feel , find someone else - might part
ner member extended family 's better coping . '' She 's normal adults feel `` wobbly '' post-pandemic wonde
r might period stability . She recommends making sure `` protected times '' exposure news , switching radio
school run . `` advice I 'd give children much relevant adults . You try hard guard time fun escape bleakne
ss . It 's compassion everyone including . '' What others Helping others one five steps recommended NHS imp
rove mental wellbeing countless ways help Ukraine crisis . A number charities launched appeals , including
British Red Cross , UNHCR refugee agency Unicef , many individuals across UK raising money collecting donat
ions independently . Dr Hepburn watching war unfold make feel control , focusing ways help manage . `` Ther
e 's lots evidence something people really beneficial impact mental health . If perceive made someone else
feel good , feel good . Our brains designed social connections creating connection something bigger really
beneficial wellbeing . '' Image source , Dr Emma Hepburn Dr Hepburn `` draws psychology '' posts images Ins
tagram page , Psychology Mum . Her recent illustration people help Ukraine crisis . `` It 's making sense c
omplex situation . And I guess trying something well . '' Shout , looking volunteers , believes benefits pe
ople ends phone . `` We 've found talking war friends family 's really big reason 's conversations comforti
ng , '' explains Sarah . `` One great things involved service helps people sense satisfaction accomplishmen
t get , 's people looking . `` People feel helpless face war big world events , taking back small things hu
gely enriching people . '' Prof Hill `` kids absolutely love '' finding positive ways help others recommend
s getting involved . `` Injustice something kids feel strongly , get think ways help Ukrainian community ,
'' . `` There people collecting donations , things send refugee centres , something like . Sometimes someth
ing constructive really powerful . ''"]

Topic modelling with sklearn and pyLDAvis

We'll now setup an LDA model to do the topic classification using sklearn . We've chosen to identify 5 topics.

In [9]: # We'll create a document term matrix for each text in the corpus
# converting the text into a numerical representation)

# Create an LDA model, with 10 topics

LatentDirichletAllocation(n_components=5, random_state=123)
Out[9]:

Run the model and display the results with pyLDAvis (see https://nlp.stanford.edu/events/illvi2014/papers/sievert-illvi2014.pdf for
more discussion of this visualization). We can hover over the circles to identify the important keywords for the topics.

In [10]: # Imports

# Generate the topics

# Display visualisation in Jupyter

C:\Anaconda3\envs\py38class\lib\site-packages\sklearn\utils\deprecation.py:87: FutureWarning: Function get_

feature_names is deprecated; get_feature_names is deprecated in 1.0 and will be removed in 1.2. Please use
get_feature_names_out instead.
warnings.warn(msg, category=FutureWarning)
Out[10]: Selected Topic: 0 Previous Topic Next Topic Clear Topic Slide to adjust relevance metric:(2)
λ=1
 Intertopic Distance Map (via multidimensional scaling) To
0 10 20
PC2
3 booster
law
dose
new
scotland
children
covid
refugees
employees
russia
bill
employment
workers
over
PC1 doses
staff
two
second
1 variants
feel
ve
4 months
bbc
consultation
5
would
going
connolly