Embed Size (px)
Citation preview
Multi-Task Learning for Mental Healthusing Social Media Text
Adrian BentonJohns Hopkins University
Margaret MitchellMicrosoft Research∗
Dirk HovyUniversity of Copenhagen
Abstract
We introduce initial groundwork for esti-mating suicide risk and mental health ina deep learning framework. By model-ing multiple conditions, the system learnsto make predictions about suicide risk andmental health at a low false positive rate.Conditions are modeled as tasks in a multi-task learning (MTL) framework, with gen-der prediction as an additional auxiliarytask. We demonstrate the effectivenessof multi-task learning by comparison toa well-tuned single-task baseline with thesame number of parameters. Our bestMTL model predicts potential suicide at-tempt, as well as the presence of atypicalmental health, with AUC > 0.8. We alsofind additional large improvements usingmulti-task learning on mental health taskswith limited training data.
1 Introduction
Suicide is one of the leading causes of deathworldwide, and over 90% of individuals who dieby suicide experience mental health conditions.1
However, detecting the risk of suicide, as wellas monitoring the effects of related mental healthconditions, is challenging. Traditional methodsrely on both self-reports and impressions formedduring short sessions with a clinical expert, but it isoften unclear when suicide is a risk in particular.2
Consequently, conditions leading to preventablesuicides are often not adequately addressed.
∗Now at Google Research.1https://www.nami.org/Learn-
More/Mental-Health-Conditions/Related-Conditions/Suicide#sthash.dMAhrKTU.dpuf
2Communication with clinicians at the 2016 JSALT work-shop (Hollingshead, 2016).
Automated monitoring and risk assessment ofpatients’ language has the potential to complementtraditional assessment methods, providing objec-tive measurements to motivate further care and ad-ditional support for people with difficulties relatedto mental health. This paves the way towards ver-ifying the need for additional care with insurancecoverage, for example, as well as offering directbenefits to clinicians and patients.
We explore some of the possibilities in the deeplearning and mental health space using written so-cial media text that people with different mentalhealth conditions are already producing. Uncov-ering methods that work with such text providesthe opportunity to help people with different men-tal health conditions by leveraging a task they arealready participating in.
Social media text carries implicit informationabout the author, which has been modeled in nat-ural language processing (NLP) to predict au-thor characteristics such as age (Goswami et al.,2009; Rosenthal and McKeown, 2011; Nguyenet al., 2014), gender (Sarawgi et al., 2011; Ciotet al., 2013; Liu and Ruths, 2013; Volkova etal., 2015; Hovy, 2015), personality (Schwartzet al., 2013; Volkova et al., 2014; Plank andHovy, 2015; Park et al., 2015; Preotiuc-Pietro etal., 2015), and occupation (Preotiuc-Pietro et al.,2015). Similar text signals have been effectivelyused to predict mental health conditions such asdepression (De Choudhury et al., 2013; Copper-smith et al., 2015b; Schwartz et al., 2014), suici-dal ideation (Coppersmith et al., 2016; Huang etal., 2015), schizophrenia (Mitchell et al., 2015) orpost-traumatic stress disorder (PTSD) (Pedersen,2015).
However, these studies typically model eachcondition in isolation, which misses the op-portunity to model coinciding influence factors.Tasks with underlying commonalities (e.g., part-
of-speech tagging, parsing, and NER) have beenshown to benefit from multi-task learning (MTL),as the learning implicitly leverages interactionsbetween them (Caruana, 1993; Sutton et al., 2007;Rush et al., 2010; Collobert et al., 2011; Søgaardand Goldberg, 2016). Suicide risk and relatedmental health conditions are therefore good can-didates for modeling in a multi-task framework.
In this paper, we propose multi-task learningfor detecting suicide risk and mental health condi-tions. The tasks of our model include neuroatypi-cality (i.e., atypical mental health) and suicide at-tempt, as well as the related mental health condi-tions of anxiety, depression, eating disorder, panicattacks, schizophrenia, bipolar disorder, and post-traumatic stress disorder (PTSD), and we explorethe effect of task selection on model performance.We additionally include the effect of modelinggender, which has been shown to improve accu-racy in tasks using social media text (Volkova etal., 2013; Hovy, 2015).
Predicting suicide risk and several mental healthconditions jointly opens the possibility for themodel to leverage a shared representation forconditions that frequently occur together, a phe-nomenon known as comorbidity. Further includ-ing gender reflects the fact that gender differencesare found in the patterns of mental health (WHO,2016), which may help to sharpen the model. TheMTL framework we propose allows such sharedinformation across predictions and enables the in-clusion of several loss functions with a commonshared underlying representation. This approachis flexible enough to extend to factors other thanthe ones shown here, provided suitable data.
We find that choosing tasks that are prerequi-sites or related to the main task is critical for learn-ing a strong model, similar to Caruana (1996). Wefurther find that modeling gender improves accu-racy across a variety of conditions, including sui-cide risk. The best-performing model from ourexperiments demonstrates that multi-task learningis a promising new direction in automated assess-ment of mental health and suicide risk, with possi-ble application to the clinical domain.
Our contributions
1. We demonstrate the utility of MTL in pre-dicting mental health conditions from socialuser text – a notoriously difficult task (Cop-persmith et al., 2015a; Coppersmith et al.,
2015b) – with potential application to detect-ing suicide risk.
2. We explore the influence of task selection onprediction performance, including the effectof gender.
3. We show how to model tasks with a largenumber of positive examples to improve theprediction accuracy of tasks with a smallnumber of positive examples.
4. We compare the MTL model against a single-task model with the same number of param-eters, which directly evaluates the multi-tasklearning approach.
5. The proposed MTL model increases the TruePositive Rate at 10% false alarms by up to9.7% absolute (for anxiety), a result with di-rect impact for clinical applications.
2 DisclaimerAs with any author-attribute detection, there is thedanger of abusing the model to single out people(overgeneralization, see Hovy and Spruit (2016)).We are aware of this danger, and sought to min-imize the risk. For this reason, we don’t pro-vide a selection of features or representative ex-amples. The experiments in this paper were per-formed with a clinical application in mind, anduse carefully matched (but anonymized) data, sothe distribution is not representative of the popu-lation as a whole. The results of this paper shouldtherefore not be interpreted as a means to assessmental health conditions in social media in gen-eral, but as a test for the applicability of MTL in awell-defined clinical setting.
3 Model ArchitectureA neural multi-task architecture opens the possi-bility of leveraging commonalities and differencesbetween mental conditions. Previous work (Col-lobert et al., 2011; Caruana, 1996; Caruana, 1993)has indicated that such an architecture allows forsharing parameters across tasks, and can be ben-eficial when there is varying degrees of annota-tion across tasks.3 This makes MTL particularlycompelling in light of mental health comorbidity,and given that different conditions have differentamounts of associated data.
Previous MTL approaches have shown consid-erable improvements over single task models, and
3We also experimented with a graphical model architec-ture, but found that it did not scale as well and provided lesstraction.
Yt
Wt
Wt+1
Wt+2
t 2 T
Yt
W0
Wt
WT+t
t 2 T
Single-task Multi-task
Figure 1: STL model in plate notation (left):weights trained independently for each task t (e.g.,anxiety, depression) of the T tasks. MTL model(right): shared weights trained jointly for all tasks,with task-specific hidden layers. Curves in ovalsrepresent the type of activation used at each layer(rectified linear unit or sigmoid). Hidden layersare shaded.
the arguments are convincing: Predicting multi-ple related tasks should allow us to exploit anycorrelations between the predictions. However, inmuch of this work, an MTL model is only onepossible explanation for improved accuracy. An-other more salient factor has frequently been over-looked: The difference in the expressivity of themodel class, i.e., neural architectures vs. discrim-inative or generative models, and critically, differ-ences in the number of parameters for comparablemodels. Some comparisons might therefore haveinadvertently compared apples to oranges.
In the interest of examining the effect of multi-task learning specifically, we compare the multi-task predictions to models with equal expressiv-ity. We evaluate the performance of a standardlogistic regression model (a standard approach totext-classification problems), a multilayer percep-tron single-task learning (STL) model, and a neu-ral MTL model, the latter two with equal numbersof parameters. This ensures a fair comparison byisolating the unique properties of MTL from thedimensionality-reduction aspects of deep architec-tures in general.
The neural models we evaluate come in twoforms. The first, depicted in plate notation onthe left in Figure 1, are the STL models. Theseare feedforward networks with two hidden layers,trained independently to predict each task. On theright in Figure 1 is the MTL model, where the firsthidden layer from the bottom is shared between
all tasks. An additional per-task hidden layer isused to give the model flexibility to map fromthe task-agnostic representation to a task-specificone. Each hidden layer uses a rectified linear unitas non-linearity. The output layer uses a logis-tic non-linearity, since all tasks are binary predic-tions. The MTL model can easily be extended to astack of shared hidden layers, allowing for a morecomplicated mapping from input to shared space.4
As noted in Collobert et al. (2011), MTL ben-efits from mini-batch training, which both allowsoptimization to jump out of poor local optima, andmore stochastic gradient steps in a fixed amountof time (Bottou, 2012). We create mini-batches bysampling from the users in our data, where eachuser has some subset of the conditions we are try-ing to predict, and may or may not be annotatedwith gender. At each mini-batch gradient step, weupdate weights for all tasks. This not only allowsfor randomization and faster convergence, it alsoprovides a speed-up over the individual selectionprocess reported in earlier work (Collobert et al.,2011).
Another advantage of this setup is that we donot need complete information for every instance:Learning can proceed with asynchronous updates,dependent on what the data in each batch hasbeen annotated for, while sharing representationsthroughout. This effectively learns a joint modelwith a common representation for several differenttasks, allowing the use of several “disjoint” datasets, some with limited annotated instances.
Optimization and Model Selection Even in arelatively simple neural model, there are a num-ber of hyperparameters that can (and have to) betuned to achieve good performance. We performa line search for every model we use, sweepingover L2 regularization and hidden layer width. Weselect the best model based on the developmentloss. Figure 4 shows the performance on the corre-sponding test sets (plot smoothed by rolling meanof 10 for visibility).
In our experiments, we sweep over the L2regularization constant applied to all weightsin {10−4, 10−3, 10−2, 0.1, 0.5, 1.0, 5.0, 10.0}, andhidden layer width (same for all layers in the net-work) in {16, 32, 64, 128, 256, 512, 1024, 2048}.We fix the mini-batch size to 256, and 0.05
4We tried training a 4-shared-layer MTL model to predicttargets on a separate dataset, but did not see any gains overthe standard 1-shared-layer MTL model in our application.
dropout on the input layer. Choosing a small mini-batch size and the model with lowest developmentloss helps to account for overfitting.
We train each model for 5,000 iterations, jointlyupdating all weights in our models. After this ini-tial joint training, we select each task separately,and only update the task-specific layers of weightsindependently for another 1,000 iterations (select-ing the set of weights achieving lowest develop-ment loss for each task individually). Weights areupdated using mini-batch Adagrad (Duchi et al.,2011) – this converges more quickly than otheroptimization schemes we experimented with. Weevaluate the tuning loss every 10 epochs, and se-lect the model with the lowest tuning loss.
4 DataWe train our models on a union of multiple Twitteruser datasets: 1) users identified as having anxiety,bipolar disorder, depression, panic disorder, eatingdisorder, PTSD, or schizophrenia (Coppersmith etal., 2015a), 2) those who had attempted suicide(Coppersmith et al., 2015c), and 3) those iden-tified as having either depression or PTSD fromthe 2015 Computational Linguistics and ClinicalPsychology Workshop shared task (Coppersmithet al., 2015b), along with neurotypical gender-matched controls (Twitter users not identified ashaving a mental condition). Users were identifiedas having one of these conditions if they stated ex-plicitly they were diagnosed with this conditionon Twitter (verified by a human annotator), andthe data was pre-processed to remove direction in-dications of the condition. For a subset of 1,101users, we also manually-annotate gender. The fi-nal dataset contains 9,611 users in total, with anaverage of 3521 tweets per user. The number ofusers with each condition is included in Table 1.Users in this joined dataset may be tagged withmultiple conditions, thus the counts in this tabledo not sum to the total number of users.
We use the entire Twitter history of each useras input to the model, and split it into character1-to-5-grams, which have been shown to capturemore information than words for many Twittertext classification tasks (Mcnamee and Mayfield,2004; Coppersmith et al., 2015a). We computethe relative frequency of the 5,000 most frequentn-gram features for n ∈ {1, 2, 3, 4, 5} in our data,and then feed this as input to all models. This inputrepresentation is common to all models, allowingfor fair comparison.
5 Experiments
Our task is to predict suicide attempt and men-tal conditions for each of the users in these data.We evaluate three classes of models: baseline lo-gistic regression over character n-gram features(LR), feed-forward multilayer perceptrons trainedto predict each task separately (STL), and feed-forward multi-task models trained to predict a setof conditions simultaneously (MTL). We experi-ment with a feed-forward network against inde-pendent logistic regression models as a way to di-rectly test the hypothesis that MTL may work wellin this domain.
We also perform ablation experiments to seewhich subsets of tasks help us learn an MTLmodel that predicts a particular mental conditionbest. For all experiments, data were divided intofive equal-sized folds, three for training, one fortuning, and one for testing (we report the perfor-mance on this).
All our models are implemented in Keras5 withTheano backend and GPU support. We train themodels for a total of up to 15,000 epochs, usingmini-batches of 256 instances. Training time onall five training folds ranged from one to eighthours on a machine with Tesla K40M.
Evaluation Setup We compare the accuracy ofeach model at predicting each task separately.
In clinical settings, we are interested in mini-mizing the number of false positives, i.e., incor-rect diagnoses, which can cause undue stress tothe patient. We are thus interested in bounding thisquantity. To evaluate the performance, we plot thefalse positive rate (FPR) against the true positiverate (TPR). This gives us a receiver operating char-acteristics (ROC) curve, allowing us to inspect theperformance of each model on a specific task atany level of FPR.
While the ROC gives us a sense of how wella model performs at a fixed true positive rate, itmakes it difficult to compare the individual tasksat a low false positive rate, which is also im-portant for clinical application. We therefore re-port two more measures: the area under the ROCcurve (AUC) and TPR performance at FPR=0.1(TPR@FPR=0.1). We do not compare our mod-els to a majority baseline model, since this modelwould achieve an expected AUC of 0.5 for alltasks, and F-score and TPR@FPR=0.1 of 0 for
5http://keras.io/
NEUROTYPICAL
ANXIETY
DEPRESSION
SUICID
EATTEM
PT
EATING
SCHIZOPHRENIA
PANIC
PTSDBIP
OLAR
LABELEDM
ALE
LABELEDFEM
ALE
NEUROTYPICAL 4820 - -ANXIETY 0 2407 47 184
DEPRESSION 0 1148 1400 54 158SUICIDE ATTEMPT 0 45 149 1208 186 532
EATING 0 64 133 45 749 6 85SCHIZOPHRENIA 0 18 41 2 8 349 2 4
PANIC 0 136 73 4 2 4 263 2 18PTSD 0 143 96 14 16 14 22 191 8 26
BIPOLAR 0 149 120 22 22 49 14 25 234 10 39
Table 1: Frequency and comorbidity across mental health conditions.
all mental conditions – users exhibiting a condi-tion are the minority, meaning a majority baselineclassifier would achieve zero recall.
6 Results
Figure 2 shows the AUC-score of each model foreach task separately, and Figure 3 the true positiverate at a low false positive rate of 0.1. Precision-recall curves for model/task are in Figure 5. STLis a multilayer perceptron with two hidden layers(with a similar number of parameters as the pro-posed MTL model). The MTL +gender and MTLmodels predict all tasks simultaneously, but areonly evaluated on the main respective task.
Both AUC and TPR (at FPR=0.1) demonstratethat single-task models models do not performnearly as well as multi-task models or logistic re-gression. This is likely because the neural net-works learned by STL cannot be guided by theinductive bias provided by MTL training. Note,however, that STL and MTL are often times com-parable in terms of F1-score, where false positivesand false negatives are equally weighted.
As shown Figure 2, multi-task suicide predic-tions reach an AUC of 0.848, and predictions foranxiety and schizophrenia are not far behind. In-terestingly however, schizophrenia stands out asbeing the only condition to be best predicted witha single-task model. MTL models show improve-ments over STL and LR models for predicting sui-cide, neuroatypicality, depression, anxiety, panic,bipolar disorder, and PTSD. The inclusion of gen-der in the MTL models leads to direct gains overan LR baseline in predicting anxiety disorders:anxiety, panic, and PTSD.
In Figure 3, we illustrate the true positive rate– that is, how many cases of mental health con-ditions that we correctly predict – given a lowfalse positive rate – that is, a low rate of predict-ing people have mental health conditions whenthey do not. This is particularly useful in clinicalsettings, where clinicians seek to minimize over-diagnosing. In this setting, MTL leads to the bestperformance across the board, for all tasks underconsideration: Neuroatypicality, suicide, depres-sion, anxiety, eating, panic, schizophrenia, bipo-lar disorder, and PTSD. Including gender in MTLfurther improves performance for neuroatypical-ity, suicide, anxiety, schizophrenia, bipolar disor-der, and PTSD.
Figure 2: AUC for different main tasks
MTL Leveraging Comorbid Conditions Im-proves Prediction Accuracy We find that theprediction of the conditions with the least amountof data – bipolar disorder and PTSD – are sig-
Figure 3: TPR at 0.10 FPR for different main tasks
nificantly improved by having the model alsopredict comorbid conditions with substantiallymore data: depression and anxiety. We areable to increase the AUC for predicting PTSDto 0.786 by MTL, from 0.770 by LR, whereasSTL fails to perform as well with an AUC of0.667. Similarly for predicting bipolar disorder(MTL:0.723, LR:0.752, STL:0.552) and panic at-tack (MTL:0.724, LR:0.713, STL:0.631).
These differences in AUC are significant at p =0.05 according to bootstrap sampling tests with5000 samples. The wide difference between MTLand STL can be explained in part by the increasedfeature set size – MTL training may, in this case,provide a form of regularization that STL cannotexploit. Further, modeling the common mentalhealth conditions with the most data (depression,anxiety) helps in pulling out more rare conditionscomorbid with these common health conditions.This provides evidence that an MTL model canhelp in predicting elusive conditions by using largedata for common conditions, and a small amountof data for more rare conditions.
Utility of Authorship Attributes Figures 2 and3 both suggest that adding gender as an auxiliarytask leads to more predictive models, even thoughthe difference is not statistically significant formost tasks. This is consistent with the findings inprevious work (Volkova et al., 2013; Hovy, 2015).Interestingly, though, the MTL model is worse atpredicting gender itself. While this could be adirect result of data sparsity (recall that we haveonly a small subset annotated for gender), whichcould be remedied by annotating additional users
for gender, this appears unlikely given the otherfindings of our experiments, where MTL helpedin specifically these sparse scenarios.
However, it has been pointed out by Caruana(1996) that not all tasks benefit from a MTL set-ting in the same way, and that some tasks servepurely auxiliary functions. Here, gender predic-tion does not benefit from including mental con-ditions, but helps vice versa. In other words, pre-dicting gender is qualitatively different from pre-dicting mental health conditions: it seems likelythat the signals for anxiety ares much more sim-ilar to the ones for depression than for, say, be-ing male, and can therefore add to detecting de-pression. However, the distinction between cer-tain conditions does not add information for thedistinction of gender. The effect may also be dueto the fact that these data were constructed withinferred gender (used to match controls), so theremight be a degree of noise in the data.
Choosing Tasks Although MTL tends to dom-inate STL in our experiments, it is not clearwhether modeling several tasks provide a benefi-cial bias in MTL models in general, or if there ex-ists specific subsets of auxiliary tasks that are mostbeneficial for predicting suicide risk and relatedmental health conditions. We perform ablation ex-periments by training MTL models on a subset ofauxiliary tasks, and prediction for a single maintask. We focus on four conditions to predict well:suicide attempt, anxiety, depression, and bipolardisorder. For each main task, we vary the auxil-iary tasks we train the MTL model with. Sinceconsidering all possible subsets of tasks is combi-natorily unfeasible, we choose the following tasksubsets as auxiliary:
• all: all mental conditions along with gender• all conds: all mental conditions, no gender• neuro: only neurotypicality• neuro+mood: neurotypicality, depression,
and bipolar disorder (mood disorders)• neuro+anx: neurotypicality, anxiety, and
panic attack (anxiety conditions)• neuro+targets: neurotypicality, anxiety, de-
pression, suicide attempt, bipolar disorder• none: no auxiliary tasks, equivalent to STL
Table 2 shows AUC for the four prediction taskswith different subsets of auxiliary tasks. Statisti-cally significant improvements over the respectiveLR baselines are denoted by superscript. Restrict-ing the auxiliary tasks to a small subset tends to
Figure 4: ROC curves for predicting each condition. The precision (diagnosed, correctly labeled) is onthe y-axis, while the proportion of false alarms (control users mislabeled as diagnosed) is on the x-axis.Chance performance is indicated by the dotted diagonal line.
hurt performance for most tasks, with exceptionto bipolar, which benefits from the prediction ofdepression and suicide attempt. All main tasksachieve their best performance using the full setof additional tasks as auxiliary. This suggests thatthe biases induced by predicting different kinds ofmental conditions are mutually beneficial – e.g.,multi-task models that predict suicide attempt mayalso be good at predicting anxiety.
Based on these results, we find it useful tothink of MTL as a framework to leverage auxiliarytasks as regularization to effectively combat datapaucity and less-than-trustworthy labels. As wehave demonstrated, this may be particularly use-ful when predicting mental health conditions andsuicide risk.
7 Discussion: Multi-task Learning
Our results indicate that an MTL framework canlead to significant gains over single-task mod-els for predicting suicide risk and several mentalhealth conditions. We find benefit from predict-
ing related mental conditions and demographic at-tributes simultaneously.
We experimented with all the optimizers thatKeras provides, and found that Adagrad seemsto converge fastest to a good optimum, althoughall the adaptive learning rate optimizers (such asAdam, etc.) tend to converge quickly. This indi-cates that the gradient is significantly steeper alongcertain parameters than others. Default stochasticgradient descent (SGD) was not able to convergeas quickly, since it is not able to adaptively scalethe learning rate for each parameter in the model –taking too small steps in directions where the gra-dient is shallow, and too large steps where the gra-dient is steep. We further note an interesting be-havior: all of the adaptive learning rate optimizersyield a strange “step-wise” training loss learningcurve, which hits a plateau, but then drops afterabout 900 iterations, only to hit another plateau,and so on. Obviously, we would prefer to have asmooth training loss curve. We can indeed achievethis using SGD, but it takes much longer to con-
Figure 5: Precision-recall curves for predicting each condition.
verge than, for example, Adagrad. This suggeststhat a well-tuned SGD would be the best optimizerfor this problem, a step that would require somemore experimentation and is left for future work.
We also found that feature counts have a pro-nounced effect on the loss curves: Relative featurefrequencies yield models that are much easier totrain than raw feature counts.
Feature representations are therefore anotherarea of optimization, e.g., different ranges of char-acter n-grams (e.g., n > 5) and unigrams. Weused character 1-to-5-grams, since we believe thatthese features generalize better to a new domain(e.g., Facebook) than word unigrams. However,there is no fundamental reason not to chooselonger character n-grams, other than time con-straints in regenerating the data, and accountingfor overfitting with proper regularization.
Initialization is a decisive factor in neural mod-els, and Goldberg (2015) recommends repeatedrestarts with differing initializations to find the op-timal model. In an earlier experiment, we tried ini-tializing a MTL model (without task-specific hid-den layers) with pretrained word2vec embeddings
of unigrams trained on the Google News n-gramcorpus. However, we did not notice an improve-ment in F-score. This could be due to the otherfactors, though, such as feature sparsity.
Table 3 shows parameters sweeps with hiddenlayer width 256, training the MTL model on thesocial media data with character trigrams as inputfeatures. The sweet spots in this table may be goodstarting points for training models in future work.
8 Related Work
MTL was introduced by Caruana (1993), based onthe observation that humans rarely learn things inisolation, and that it is the similarity between re-lated tasks that helps us get better.
Some of the first works on MTL were motivatedby medical risk prediction (Caruana et al., 1996),and it is now being rediscovered for this purpose(Lipton et al., 2016). The latter use a long short-term memory (LSTM) structure to provide severalmedical diagnoses from health care features (yetno textual or demographic information), and findsmall, but probably not significant improvementsover a structure similar to the STL we use here.
AuxiliaryTasks
Main Task
ANXIETY
BIPOLAR
DEPRESSION
SUICID
E
ATTEMPT
all 0.813∗† 0.752∗† 0.769† 0.835∗†
all conds 0.786 0.743† 0.772† 0.833∗†
neuro 0.763 0.740† 0.759 0.797neuro+mood 0.756 0.742† 0.761 0.804
neuro+anx 0.770 0.744† 0.746 0.792neuro+targets 0.750 0.747† 0.764 0.817
none (STL) 0.777 0.552 0.749 0.810LR 0.791 0.723† 0.763 0.817
Table 2: Test AUC when predicting Main Task af-ter training to predict a subset of auxiliary tasks.Significant improvement over LR baseline at p =0.05 is denoted by ∗, and over no auxiliary tasks(STL) by †.
Learning Loss L2 Loss Hidden LossRate Width10−4 5.1 10−3 2.8 32 3.0
5 ∗ 10−4 2.9 5 ∗ 10−3 2.8 64 3.010−3 2.9 10−2 2.9 128 2.9
5 ∗ 10−3 2.4 5 ∗ 10−2 3.1 256 2.910−2 2.3 0.1 3.4 512 3.0
5 ∗ 10−2 2.2 0.5 4.6 1024 3.00.1 20.2 1.0 4.9
Table 3: Average dev loss over epochs 990-1000of joint training on all tasks as a function of dif-ferent learning parameters. Optimized using Ada-grad with hidden layer width 256.
The target in previous work was medical con-ditions as detected in patient records, not mentalhealth conditions in social text. The focus in thiswork has been on the possibility of predicting sui-cide attempt and other mental health conditionsusing social media text that a patient may alreadybe writing, without requiring full diagnoses.
The framework proposed by Collobert et al.(2011) allows for predicting any number of NLPtasks from a convolutional neural network (CNN)representation of the input text. The model wepresent is much simpler: A feed-forward networkwith n-gram input layer, and we demonstrate howto constrain n-gram embeddings for clinical ap-plication. Comparing with further models is pos-sible, but distracts from the question of whetherMTL training can help in this domain. As we haveshown, it can.
9 Conclusion and Future Work
In this paper, we develop neural MTL models for10 prediction tasks (suicide, seven mental healthconditions, neurotypicality, and gender). We com-pare their performance with STL models trainedto predict each task independently.
Our results show that an MTL model with alltask predictions performs significantly better thanother models, reaching 0.846 TPR for neuroatyp-icality where FPR=0.1, and AUC of 0.848, TPRof 0.559 for suicide. Due to the nature of MTL,we find additional contributions that were not theoriginal goal of this work: Pronounced gains in de-tecting anxiety, PTSD, and bipolar disorder. MTLpredictions for anxiety, for example, reduce the er-ror rate from a single-task model by up to 11.9%.
We also investigate the influence of modeldepth, comparing to progressively deeper STLfeed-forward networks with the same number ofparameters. We find: (1) Most of the modelingpower stems from the expressivity conveyed bydeep architectures. (2) Choosing the right set ofauxiliary tasks for a given mental condition canyield a significantly better model. (3) The MTLmodel dramatically improves for conditions withthe smallest amount of data. (4) Gender predic-tion does not follow the two previous points, butimproves performance as an auxiliary task.
Accuracy of the MTL approach is not yet readyto be used in isolation in the clinical setting. How-ever, our experiments suggest this is a promisingdirection moving forward. There are strong gainsto be made in using multi-task learning to aid clin-icians in their evaluations, and with further part-nerships between the clinical and machine learn-ing community, we foresee improved suicide pre-vention efforts.
Acknowledgements
Thanks to Kristy Hollingshead Seitz, Glen Cop-persmith, and H. Andrew Schwartz, as well asthe organizers of the Johns Hopkins Jelinek Sum-mer School 2016. We are also grateful for the in-valuable feedback on MTL from Yoav Goldberg,Stephan Gouws, Ed Greffenstette, Karl MoritzHermann, and Anders Søgaard. The work re-ported here was started at JSALT 2016, andwas supported by JHU via grants from DARPA(LORELEI), Microsoft, Amazon, Google andFacebook.
ReferencesLeon Bottou. 2012. Stochastic gradient tricks. Neural
Networks, Tricks of the Trade, Reloaded, pages 430–445.
Rich Caruana, Shumeet Baluja, Tom Mitchell, et al.1996. Using the future to “sort out” the present:Rankprop and multitask learning for medical riskevaluation. Advances in neural information process-ing systems, pages 959–965.
Rich Caruana. 1993. Multitask learning: Aknowledge-based source of inductive bias. In Pro-ceedings of the Tenth International Conference onMachine Learning, pages 41–48.
Rich Caruana. 1996. Algorithms and applications formultitask learning. In ICML, pages 87–95.
Morgane Ciot, Morgan Sonderegger, and Derek Ruths.2013. Gender Inference of Twitter Users in Non-English Contexts. In Proceedings of the 2013 Con-ference on Empirical Methods in Natural LanguageProcessing, Seattle, Wash, pages 18–21.
Ronan Collobert, Jason Weston, Leon Bottou, MichaelKarlen, Koray Kavukcuoglu, and Pavel Kuksa.2011. Natural language processing (almost) fromscratch. Journal of Machine Learning Research,12(Aug):2493–2537.
Glen Coppersmith, Mark Dredze, Craig Harman, andKristy Hollingshead. 2015a. From ADHD to SAD:Analyzing the language of mental health on Twit-ter through self-reported diagnoses. In Proceedingsof the 2nd Workshop on Computational Linguisticsand Clinical Psychology: From Linguistic Signal toClinical Reality, pages 1–10.
Glen Coppersmith, Mark Dredze, Craig Harman,Kristy Hollingshead, and Margaret Mitchell. 2015b.Clpsych 2015 shared task: Depression and ptsd ontwitter. In Proceedings of the 2nd Workshop onComputational Linguistics and Clinical Psychology:From Linguistic Signal to Clinical Reality, pages31–39. Association for Computational Linguistics.
Glen Coppersmith, Ryan Leary, Eric Whyne, and TonyWood. 2015c. Quantifying suicidal ideation vialanguage usage on social media. In Joint StatisticsMeetings Proceedings, Statistical Computing Sec-tion, JSM.
Glen Coppersmith, Kim Ngo, Ryan Leary, and An-thony Wood. 2016. Exploratory analysis of socialmedia prior to a suicide attempt. In Proceedings ofthe Third Workshop on Computational Lingusiticsand Clinical Psychology, pages 106–117. Associa-tion for Computational Linguistics.
Munmun De Choudhury, Michael Gamon, ScottCounts, and Eric Horvitz. 2013. Predicting depres-sion via social media.
John Duchi, Elad Hazan, and Yoram Singer. 2011.Adaptive subgradient methods for online learningand stochastic optimization. Journal of MachineLearning Research, 12(Jul):2121–2159.
Yoav Goldberg. 2015. A primer on neural net-work models for natural language processing. arXivpreprint arXiv:1510.00726.
Sumit Goswami, Sudeshna Sarkar, and Mayur Rustagi.2009. Stylometric analysis of bloggers’ age andgender. In Third International AAAI Conference onWeblogs and Social Media.
Kristy Hollingshead. 2016. Detecting risk and pro-tective factors of mental health using social medialinked with electronic health records. In JSALT2016 Workshop. Johns Hopkins University.
Dirk Hovy and Shannon L. Spruit. 2016. The so-cial impact of natural language processing. In The54th Annual Meeting of the Association for Compu-tational Linguistics.
Dirk Hovy. 2015. Demographic factors improve clas-sification performance. In Proceedings of ACL,pages 752–762.
Xiaolei Huang, Xin Li, Tianli Liu, David Chiu, Ting-shao Zhu, and Lei Zhang. 2015. Topic model foridentifying suicidal ideation in chinese microblog.In Proceedings of the 29th Pacific Asia Conferenceon Language, Information and Computation, pages553–562.
Zachary C Lipton, David C Kale, Charles Elkan, andRandall Wetzell. 2016. Learning to diagnose withlstm recurrent neural networks. In Proceedings ofICLR.
Wendy Liu and Derek Ruths. 2013. What’s in a name?Using first names as features for gender inference inTwitter. In Analyzing Microtext: 2013 AAAI SpringSymposium.
Paul Mcnamee and James Mayfield. 2004. Charac-ter n-gram tokenization for european language textretrieval. Information retrieval, 7(1-2):73–97.
Margaret Mitchell, Kristy Hollingshead, and GlenCoppersmith. 2015. Quantifying the languageof schizophrenia in social media. In Proceedingsof the 2nd Workshop on Computational Linguisticsand Clinical Psychology: From Linguistic Signalto Clinical Reality, pages 11–20, Denver, Colorado,June 5. Association for Computational Linguistics.
Dong Nguyen, Dolf Trieschnigg, A. Seza Dogruoz, Ri-lana Gravel, Mariet Theune, Theo Meder, and Fran-ciska De Jong. 2014. Predicting Author Genderand Age from Tweets: Sociolinguistic Theories andCrowd Wisdom. In Proceedings of COLING 2014.
Greg Park, H Andrew Schwartz, Johannes C Eich-staedt, Margaret L Kern, David J Stillwell, MichalKosinski, Lyle H Ungar, and Martin EP Seligman.
2015. Automatic personality assessment throughsocial media language. Journal of Personality andSocial Psychology.
Ted Pedersen. 2015. Screening twitter users for de-pression and ptsd with lexical decision lists. InProceedings of the 2nd Workshop on ComputationalLinguistics and Clinical Psychology: From Linguis-tic Signal to Clinical Reality, pages 46–53. Associa-tion for Computational Linguistics.
Barbara Plank and Dirk Hovy. 2015. Personality traitson twitter—or—how to get 1,500 personality testsin a week. In Proceedings of the 6th Workshopon Computational Approaches to Subjectivity, Sen-timent and Social Media Analysis, pages 92–98.
Daniel Preotiuc-Pietro, Johannes Eichstaedt, GregoryPark, Maarten Sap, Laura Smith, Victoria Tobolsky,Hansen Andrew Schwartz, and Lyle H Ungar. 2015.The role of personality, age and gender in tweet-ing about mental illnesses. In Proceedings of theWorkshop on Computational Linguistics and Clini-cal Psychology: From Linguistic Signal to ClinicalReality, NAACL.
Daniel Preotiuc-Pietro, Vasileios Lampos, and Niko-laos Aletras. 2015. An analysis of the user occu-pational class through twitter content. In ACL.
Sara Rosenthal and Kathleen McKeown. 2011. Ageprediction in blogs: A study of style, content, andonline behavior in pre-and post-social media genera-tions. In Proceedings of the 49th Annual Meeting ofthe Association for Computational Linguistics: Hu-man Language Technologies-Volume 1, pages 763–772. Association for Computational Linguistics.
Alexander M Rush, David Sontag, Michael Collins,and Tommi Jaakkola. 2010. On dual decompositionand linear programming relaxations for natural lan-guage processing. In Proceedings of the 2010 Con-ference on Empirical Methods in Natural LanguageProcessing, pages 1–11. Association for Computa-tional Linguistics.
Ruchita Sarawgi, Kailash Gajulapalli, and Yejin Choi.2011. Gender attribution: tracing stylometric evi-dence beyond topic and genre. In Proceedings ofthe Fifteenth Conference on Computational NaturalLanguage Learning, pages 78–86. Association forComputational Linguistics.
Hansen Andrew Schwartz, Johannes C Eichstaedt,Lukasz Dziurzynski, Margaret L Kern, EduardoBlanco, Michal Kosinski, David Stillwell, Martin EPSeligman, and Lyle H Ungar. 2013. Toward per-sonality insights from language exploration in socialmedia. In AAAI Spring Symposium: Analyzing Mi-crotext.
Andrew H. Schwartz, Johannes Eichstaedt, L. Mar-garet Kern, Gregory Park, Maarten Sap, David Still-well, Michal Kosinski, and Lyle Ungar. 2014. To-wards assessing changes in degree of depression
through Facebook. In Proceedings of the Workshopon Computational Linguistics and Clinical Psychol-ogy: From Linguistic Signal to Clinical Reality,pages 118–125. Association for Computational Lin-guistics.
Anders Søgaard and Yoav Goldberg. 2016. Deepmulti-task learning with low level tasks supervisedat lower layers. In The 54th Annual Meeting ofthe Association for Computational Linguistics, page231. Association for Computational Linguistics.
Charles Sutton, Andrew McCallum, and KhashayarRohanimanesh. 2007. Dynamic conditional randomfields: Factorized probabilistic models for labelingand segmenting sequence data. Journal of MachineLearning Research, 8(Mar):693–723.
Svitlana Volkova, Theresa Wilson, and DavidYarowsky. 2013. Exploring demographic languagevariations to improve multilingual sentiment anal-ysis in social media. In Proceedings of EMNLP,pages 1815–1827.
Svitlana Volkova, Glen Coppersmith, and BenjaminVan Durme. 2014. Inferring user political prefer-ences from streaming communications. In Proceed-ings of the 52nd annual meeting of the ACL, pages186–196.
Svitlana Volkova, Yoram Bachrach, Michael Arm-strong, and Vijay Sharma. 2015. Inferring latentuser properties from texts published in social media(demo). In Proceedings of the Twenty-Ninth Confer-ence on Artificial Intelligence (AAAI), Austin, TX,January.
World Health Organization WHO. 2016.Gender and Women’s Mental Health.http://www.who.int/mental_health/prevention/genderwomen/en/. [Online;accessed 1-Feb-2016].
