I have implemented Maximum-entropy Markov model (MEMM) for the Named entity recognition (NER) problem. I have four classes: geographical, people, material (book titles etc) and other.  Class other is overrepresented in the training &amp; test datasets. It occurs 88% (about 40k samples) of the time.  I'm sure other authors must have had this problem. How do you suggest I go about solving this problem?  EDIT: A single data sample is a whole tagged sentence. Most of the sentence is tagged as other. I'm guessing the resampling techniques won't work here? Do you suggest I use single tokens for training dataset, instead of whole sentences?  EDIT 2: In response to my post being a duplicate: you do realize you've linked to a question that was asked 2 years after mine, right?
46805
This is a question in general, not specific to any method or data set. How do we deal with a class imbalance problem in Supervised Machine learning where the number of 0 is around 90% and number of 1 is around 10% in your dataset.How do we optimally train the classifier.  One of the ways which I follow is sampling to make the dataset balanced and then train the classifier and repeat this for multiple samples.   I feel this is random, Is there any framework to approach these kind of problems.
104906
I have a training set with about 3000 positive instances and 3000 negative instances. But my test data set is pretty much un-balanced. The positive set only has 50 instances and negative has 1500 instances.This causes the precision very low. Are there any approaches to solve this problem? I use SVM to build classifier.
121830
I want to compare the classification performance of two heuristics: $h_0$ and $h_1$. I have around 700 samples for which I know the output class $c$ and the results of the two heuristics $c_{\textit{h_0}}, c_{\textit{h_1}}$.  That is, for each sample I have $\text{sample}_{\textit{id}}, c, c_{\textit{h_0}}, c_{\textit{h_1}}$.   $h_0$ has around ~71% correct predictions of the output class and $h_1$ has around ~77% correct. What I want to calculate now is whether this performance difference is just by chance or if the prediction performance of $h_1$ is statistical significant better than that of $h_0$.  I have found , but I don't know whether I can just put my data in there like the following matrix to calculate a p-value.  $$   \begin{matrix}                                &amp; h_0 &amp; h_1 \\     \text{correct prediction} &amp; 497 &amp; 539 \\     \text{wrong prediction}   &amp; 203 &amp; 161   \end{matrix} $$  I don't know if I can calculate the p-value like that or if I'm completely on the wrong track. I would really appreciate it if someone could point me in the right direction.
41120
I have a list of email subjects like:  &lt;XYZ&gt; commented on &lt;ABC&gt; Weekly review for &lt;Company&gt; Your account is ready    And I want to find patterns in them so I can group them.  Is there a well known algorithm I can use?    Preferably with wide language implementations or easy re-implementation. The algorithm should be unsupervised. The number of different emails is not known.   Update:  I think I can break this down into two problems:   Group subjects by the similar words they use, resulting in the following. Each group should be very distinct from the rest (they should be almost perfectly exclusive) and the algorithm should give relatively small number of groups with good length of the common words.  [commented, on] [weekly, review] [your, account, is, ready]  Once grouped, it should be easy to find a state automaton that accepts only the group's subject and thus eliminates variable Then I can go back and check if there are any intersections and tweak the variables.   Having said that, is it better to use a completley different approach like neural nets maybe? I have zero experience with those, but if it makes more sense, I am open to learning.
79484
Related questions:     (This led to founding of )   The tag  has become cluttered with all kinds of questions ranging from sql tables (for all DBM systems, HTML tag &lt;table&gt;, outputting content in any language (PHP, C#, LaTeX to name a few) as a table and several plugins of various languages and/or IDEs (like jTable, tablesorters and the kind).  The current count stands at 18,315 questions having this ambiguous tag. A few of possible suggestions for cleaning process:    +  to  |   +  to  |   +  to  |   +  to  |   + other-specific-DBM systems to whatever_DBM_it_is-table  +  to  |    Since I've no idea about other languages and their specific features regarding table, I'd ask the community for their input on this (such as , ,  and ). The other questions are either tagged with arrays or sql and similar tags, and in my opinion, just removing  from such questions would be enough!
62906
I've taken an ML course previously, but now that I am working with ML related projects at my job, I am struggling quite a bit to actually apply it. I'm sure the stuff I'm doing has been researched/dealt with before, but I can't find specific topics.  All the machine learning examples I find online are very simple (e.g. how to use a KMeans model in Python and look at the predictions). I am looking for good resources on how to actually apply these, and maybe code examples of large scale machine learning implementations and model trainings. I want to learn about how to effectively process and create new data that can make the ML algorithms much more effective.
107096
There are Recurrent Neural Networks and Recursive Neural Networks. Both are usually denoted by the same acronym: RNN. According to , Recurrent NN are in fact Recursive NN, but I don't really understand the explanation.  Moreover, I don't seem to find which is better (with examples or so) for Natural Language Processing. The fact is that, although Socher uses Recursive NN for NLP in his , I can't find a good implementation of recursive neural networks, and when I search in Google, most of the answers are about Recurrent NN.  Besides that, is there another DNN which applies better for NLP, or it depends on the NLP task? Deep Belief Nets or Stacked Autoencoders? (I don't seem to find any particular util for ConvNets in NLP, and most of the implementations are with machine vision in mind).  Finally, I would really prefer DNN implementations for C++ (better yet if it has GPU support) or Scala (better if it has Spark support) rather than Python or Matlab/Octave.  I've tried Deeplearning4j, but it's under constant development and the documentation is a little outdated and I can't seem to make it work. Too bad because it has the "black box" like way of doing things, very much like scikit-learn or Weka, which is what I really want.
55894
I am performing a classification task, to try to detect an object. A picture of the environment is taken, candidates are generated of this possible object using vision algorithms, and once isolated, these candidates will be passed through a CNN for the final decision on whether the object has been detected or not. I am attempting to use transfer learning on InceptionV3 but am having difficulty training it, as I only have one set/class of images. The dilemma is that I only have one class of data and when I pass it through the network, I get a 100% accuracy (because there is nothing to compare it to). How should I overcome this? Should I find more categories online to add to my dataset? What should these categories be? Just to clarify, as an example, I have class &quot;cat&quot;. Not &quot;cat&quot; and &quot;dog&quot;. Not &quot;cat&quot; and &quot;no cat&quot;. Just &quot;cat&quot;. That is what my dataset consists of at the moment.
29751
I have categorized 800,000 documents into 500 categories using the Mahout topic modelling.  Instead of representing the topic using the top 5/10 words for each topics, I want to infer a generic name for the group using any existing algorithm. For the time being, I have used the following algorithm to arrive at the name for the topic:  For each topic   Take all the documents belonging to the topic (using the document-topic distribution output) Run python nltk to get the noun phrases Create the TF file from the output name for the topic is the phrase (limited towards max 5 words)   Please suggest a approach to arrive at more relevant name for the topics.
48866
I have a classification problem which is multi-label with N labels. I would like to know which method would be the better choice? Training N classifiers (1 for each label) or a single classifier which use K-hot encoding. I'm using a neural network for this with a softmax at the last layer. (Is there any chance that the softmax would cause a problem?) I'm using tensorflow for the model.
69914
So this question is about the BUGS modeling language. So you either know it or have no clue. I'm a newbie to this so it's been driving me mad. I want to define a simple two-state hidden Markov model (HMM) where the emission of each state follows a Normal distribution. I have an array data of Nxl dimension where each row is a subject and column is a continuous variable Time associated with the subject. So in WinBUGS, I would define the model as:   for(i in 1:N) { # for each subject     # Sample the initial observation     Time[i, 1] ~ dnorm(mu[State[1]], tau[State[1]])      for(j in 2:l[i]) { # for each observation         # Sample the observed variables         Time[i, j] ~ dnorm(mu[State[j]], tau[State[j]])         # Sample the hidden states         State[j] ~ dcat(P[State[j-1], ])     } }  # P is the transition matrix of the hidden Markov chain P[1, 1:2] ~ ddirch(Pinit) P[2, 1:2] ~ ddirch(Pinit)  # Set the initial states State[1] ~ dcat(Pinit)  # Sample the initial params mu[1] ~ dnorm(200, 1.0E-6) mu[2] ~ dnorm(400, 1.0E-6)  # The precision params tau[1] ~ dgamma(0.001, 0.001) tau[2] ~ dgamma(0.001, 0.001)   Initial values have also been provided via R2WinBUGS. The model is syntactically correct. But when I run, I got this error: "multiple definitions of node State[2]"  Can you please tell me why and how to solve this? I've searched around on the error but each has their own specific case and there's not a generic solution or explanation as to why this arises.
49678
I am working on a text clustering problem. The data contains several sentences. Is there a good algorithm which reaches high accuracy on short text?  Can you provide good references?   Algorithms such as KMeans, spectral clustering does not work well for this problem.
70523
Does anyone know of a Java library that handles finding sentence boundaries? I'm thinking that it would be a smart StringTokenizer implementation that knows about all of the sentence terminators that languages can use.  Here's my experience with BreakIterator:  Using the example : I have the following Japanese:  今日はパソコンを買った。高性能のマックは早い！とても快適です。   In ascii, it looks like this:  \ufeff\u4eca\u65e5\u306f\u30d1\u30bd\u30b3\u30f3\u3092\u8cb7\u3063\u305f\u3002\u9ad8\u6027\u80fd\u306e\u30de\u30c3\u30af\u306f\u65e9\u3044\uff01\u3068\u3066\u3082\u5feb\u9069\u3067\u3059\u3002   Here's the part of that sample that I changed:   static void sentenceExamples() {    Locale currentLocale = new Locale ("ja","JP");   BreakIterator sentenceIterator =       BreakIterator.getSentenceInstance(currentLocale);   String someText = "今日はパソコンを買った。高性能のマックは早い！とても快適です。";   When I look at the Boundary indices, I see this:  0|13|24|32   But those indices don't correspond to any sentence terminators.
93962
Is it possible to have test error lower than training error?  I have a classification problem with 2000 samples, 500 of which are positives, 1500 are negatives. I split my data into 70% training data, 30% test data.  Run random forest with 200 estimators and cv=10. I did this several times and compared the recall and precision score and notice the scores for my test set is significantly better. Is this possible?
6491
Okay, so I think I have a decent enough sample, taking into account the 20:1 rule of thumb: a fairly large sample (N=374) for a total of 7 candidate predictor variables.   My problem is the following: whatever set of predictor variables I use, the classifications never get better than a specificity of 100% and a sensitivity of 0%. However unsatisfactory, this could actually be the best possible result, given the set of candidate predictor variables (from which I can't deviate).   But, I couldn't help but think I could do better, so I noticed that the categories of the dependent variable were quite unevenly balanced, almost 4:1. Could a more balanced subsample improve classifications?
99437
It's common short-hand in neural networks literature to refer to categorical cross-entropy loss as simply "cross-entropy." However, this terminology is ambiguous because different probability distributions have different cross-entropy loss functions.  So, in general, how does one move from an assumed probability distribution for the target variable to defining a cross-entropy loss for your network? What does the function require as inputs? (For example, the categorical cross-entropy function for one-hot targets requires a one-hot binary vector and a probability vector as inputs.)  A good answer will discuss the general principles involved, as well as worked examples for   categorical cross-entropy loss for one-hot targets Gaussian-distributed target distribution and how how this reduces to usual MSE loss A less common example such as a gamma distributed target, or a heavy-tailed target Explain the relationship between minimizing cross entropy and maximizing log-likelihood.
58571
I have a data set with highly positively correlated features that I'm classifying with LR. AFAIK correlated weights are not a problem in the same way they are in Naive Bayes - overcounting will not occur with LR.   The strange things that I'm seeing is that some of the highly correlated features assume opposite weights: feature A might be highly positive and feature B highly negative, though not as much. Is this a symptom of something going wrong with optimization or is this expected (a priori I expect A and B to be positive class indicators)