texts: Output all 1 when try to use ensemble CNN on MNIST dataset

texts:
from tensorflow.keras import Sequential
from tensorflow.keras.layers import Conv2D, MaxPool2D, Dense, Flatten, Dropout, BatchNormalization
from tensorflow.keras.preprocessing.image import ImageDataGenerator
from collections import Counter

Generator = ImageDataGenerator(rotation_range = 40, shear_range = 0.15, zoom_range = 0.4)

models = []
eva_list = []
for i in range(5):

    X_train, X_test, Y_train, Y_test = train_test_split(train_image, label_image, test_size = 0.3)

    Train_generator = Generator.flow(X_train, Y_train, batch_size = 128)
    model = Sequential()
    #Conv 2D layer here( not important)

    model.add(Dense(10, activation="softmax"))
    model.compile(optimizer='adam',loss='CategoricalCrossentropy', metrics=['accuracy'])
    model.summary()

    model.fit(Train_generator, batch_size= 128, epochs= 50, verbose=2)
    eva_list.append(model.evaluate(X_test, Y_test, verbose=1))
    models.append(model)

test = test/ 255
test = test.to_numpy().reshape((-1, 28, 28, 1))

result_group = []
for i in range(len(models)):
#loop through the models to make prediction with each model, store the result in result group
    temp_result = models[i].predict(test)
    result_group.append(np.argmax(temp_result, axis = 1))

result = []
#loop through each test index, create a temporary list and find the most chosen number and use that as a final result
for i in range(len(result_group[0])):
    compare = []
    for z in range(len(result_group)):
        compare.append(result_group[z][i])
    common = Counter(compare)
    result.append(common.most_common(1)[0][0])
print(result)

I have spent multiple days on the MNIST dataset. I have trained a rather deep CNN model and get a good 99.1 percent result. However, after reading some discussion posts I figured it may be a good idea to do bagging to increase the accuracy. My approach is basically to create a list to store those newly trained CNN models. And in the end, using a for loop to do prediction on each of the models. However I don't know where I made a mistake, now all my model output predict 1 for all test data. Anyone can tell me what's going on?


texts: Converting the continuous numerical features into gaussian distribution and how to deal with NaN values after that?

texts:
I have a dataset in which there are few continuous numerical features that distribution over them is non gaussian and this means, skewness is nonzero (positive or negative).
I read that it is good to use a log transformation to bring them to a gaussian distribution before using the features. After doing this step, I see Nan values in the numerical data.
Is this expected?
If yes. How  do you fix the Nan values (with mean or median replacement)?
To add on - when do you remove outliers? After gaussian distribution conversion or before that??
Any immediate help on this is much appreciated. Thank you.


texts: Is there a standard data science epistemology?

texts:
In experimental science there is the 'scientific method', in mathematics there is the 'proof.' What are the standards for knowledge production in data science?

I've gathered that there a few different philosophies one can take; some of them are similar to the scientific method (introduce a falsifiable model and see if the data fits), others seem more hodge-podge (introduce some kind of learning machine and hope that the regularization prevents over fitting, and that cross validation catches it if not).

(There are some loose connections between those two, like how falsifiability fits into VC entropy and learning rates; this is explained in Vapnik's book on the Foundations of Statistical Learning.)

So, supposing that I have some data and I want to extract knowledge from it. What are the community standards for doing so? Where can I learn about the epistemological philosophy behind these standards, and their mathematical expression?


texts: Is it possible to train this image classifier?

texts:
I'm writing a mobile app that will enable a user to scan a Craft Beer Label from a bottle, tap, six pack, etc.

The scan will only work for my customers who are the Brewers themselves, so I will have access to all of the artwork used for their labels.

My concern is around training the model.  As you can imagine, there will be differences in lighting conditions at bars, amoung other challenges.

As I am just getting started in Machine Learning, is this task feasible?  How difficult will it be to train the model given the various conditions found at different pubs/bars.

Thanks.


texts: Bagging or Random Subspace Method for Random Forest?

texts: I am reading a lot about the Random Forest regressor. I read about bagging (Bootstrap and aggregation) and random subspace. But I am not sure if the random forest regressor is just using bagging or bagging and random subspace method. Because in some articles, the random subspace method seems like an alternative to the bagging method.


texts: Configuration of CNN model for recognition of sequential data - Architecture of the top of the CNN - Parallel Layers

texts:
I am trying to configure a network for character recognition of sequential data like license plates. Now I would like to use the architecture which is noted in Table 3 in Deep Automatic Licence Plate Recognition system (link: http://www.ee.iisc.ac.in/people/faculty/soma.biswas/Papers/jain_icgvip2016_alpr.pdf).
The architecture the authors presented is this one:

The first layers are very common, but where I was stumbling was the top (the part in the red frame) of the architecture. They mention 11 parallel layers and I am really unsure how to get this in Python. I coded this architecture but it does not seem to be right to me.
model = Sequential()
model.add(Conv2D(64, kernel_size=(5, 5), input_shape = (32, 96, 3), activation = "relu"))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))
model.add(Conv2D(128, kernel_size=(3, 3), activation = "relu"))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))
model.add(Conv2D(256, kernel_size=(3, 3), activation = "relu"))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))
model.add(Flatten())
model.add(Dense(1024, activation = "relu"))
model.add(Dense(11*37, activation="Softmax"))
model.add(keras.layers.Reshape((11, 37)))

Could someone help? How do I have to code the top to get an equal architecture like the authors?


texts: Date transformation for KNN to get distance

texts:
I have data set with date features like 01/01/2019 and I would like to use KNN. However, I cannot find a good transformation for dates that has a meaningful distance result for the last feature.

For example:

f1 | 1 | 2 | 3 | 4 | 01/01/2019

f2 | 10 | 3 | 12 | 1 | 14/01/2019

Does anyone have any recommendations?


texts: Image Segmentation with a challenging background

texts:
I'm working on an animal classification problem, with the data extracted from a video feed. The recording was made in a pen, so the problem is quite challenging with a dark background and many shadows:

Initially I tried scikit-image, but then someone helped me with an advanced tool called crf-rnn (http://crfasrnn.torr.vision/) that does a great job segmenting and labelling objects in an image. I did the following:

import caffe
net = caffe.Segmenter(MODEL_FILE, PRETRAINED)
IMAGE_FILE = '0045_crop2.png'
input_image = caffe.io.load_image(IMAGE_FILE)
from PIL import Image as PILImage
image = PILImage.fromarray(np.uint8(input_image))
image = np.array(image)
mean_vec = [np.mean(image[:,:,vals]) for vals in range(image.shape[2])]
im = image[:, :, ::-1]
im = im - reshaped_mean_vec
cur_h, cur_w, cur_c = im.shape
pad_h = 750 - cur_h
pad_w = 750 - cur_w
print(pad_h, pad_w, "999")
im = np.pad(im, pad_width=((0, max(pad_h,0)), (0, max(pad_w,0)), (0, 0)), mode = 'constant', constant_values = 255)
segmentation = net.predict([im])
segmentation2 = segmentation[0:cur_h, 0:cur_w]

The resulting image segmentation is rather poor (although two cows are recognized correctly):

I use a trained crf-rnn (MODEL_FILE, PRETRAINED), which works well for other problems, but this one is harder. I would appreciate any suggestions on how to pre-process this sort of image to extract the shape of most cows.


texts:
I'm using VGG16 pretrained architecture for classification and visualization of result using Guided Backpropagation technique.
I have used tensorflow code for calaulating the input gradientSharing code snippet below:
model =  VGG16(weights='imagenet')
result = model.predict(image_1, batch_size=1)
target_class = np.argmax(result[0])
imported_y = graph1.get_tensor_by_name(model.output.name)[0][target_class]
imported_x = graph1.get_tensor_by_name(model.input.name)
guided_grads_node = tf.gradients(imported_y, imported_x)

On above code snippet, I'm calculating the gradient with respect to the target class
imported_y = graph1.get_tensor_by_name(model.output.name)[0][target_class].
My question is what if I calculate the gradient with respect to softmax output i.e. imported_y = graph1.get_tensor_by_name(model.output.name).
The saliency maps in both the cases are very much same and can not be differentiated by naked eye.
I'm not able to understand the outcome, i.e. how the gradient is caclulated in both the cases i.e.
imported_y = graph1.get_tensor_by_name(model.output.name)[0][target_class],
imported_y = graph1.get_tensor_by_name(model.output.name).
Appreciate if you can share your explanation.


texts: Does K-Means' objective function imply distance metric is Euclidean

texts:
The objective/loss function of K-Means algorithm is to minimize the sum of squared distances, written in a math form, it looks like this:
$$J(X,Z) = min\ \sum_{z\in Clusters}\sum_{x \in data}||x-z||^2$$

If we have different distance metric, for instance, cosine (I realize there's a conversion between cosine and Euclidean but let's forget it for now), manhattan etc, does it mean we will have a different loss function?

That is, the traditional K-Means based on expectation maximization won't be working right? Because for every iteration we revise centroids, normally, by computing the average. However, for some of the metrics other than Euclidean, average might not be a legit representation for the center.


texts: Can some one explain how PCA is relevant in extracting parameters of Gaussian Mixture Models

texts: I am having some difficulty in seeing connection between PCA on second order moment matrix in estimating parameters of Gaussian Mixture Models. Can anyone connect the above??


texts: How to Classify an Image in a Class and a Subclass?

texts:
I am a beginner in Machine Learning and I have a situation where an Image needs to be classified into first a super class, then a subclass.
For Example,
I have a set of images of fruits, containing images of Mangoes, Banana, Peaches, Apples etc. Then each fruit can either be fresh or rotten. I have training set containing images of rotten and fresh fruits for each Fruit type.
So for a test image I need to classify which fruit it is (Mango, Banana, etc) and then whether it is fresh or rotten.
I want to know how to approach this kind of problem


texts: How to interpret hard negative mining curves while training a deep object detector?

texts:
I am training a single shot detector (SSD) in tensorflow object detection API. After having read the paper and some articles online, I understood that hard negative mining trains the network on 'hard examples', and that the number of 'mined' negatives shouldn't outnumber the positives to ensure that data stay balanced. But I am still unable to fully grasp the curves that I get throughout training:

This is my hard miner configuration:
hard_example_miner {
    num_hard_examples: 3000
    iou_threshold: 0.990000009537
    loss_type: CLASSIFICATION
    max_negatives_per_positive: 3
    min_negatives_per_image: 3
  }

SSD is using 3:1 as ratio between negatives and positives, and this shows in the figures.
I understand that the left figure is the number of negatives per step that the network is trained on.
Is the right figure there to only show the number of positives that the network has found for each step?
Isn't it a bad sign that the number of positives is decreasing?


texts: How is PACF analysis output related to LSTM?

texts:
I was going through a recent paper “A Novel Hybrid Data-Driven Model for Daily Land Surface Temperature ForecastingUsing Long Short-Term Memory Neural Network Based on Ensemble Empirical ModeDecomposition” found here.

The EEMD-LSTM scheme mentioned is indeed a very well designed model for predicting land surface temperature with more accuracy. I was reading and trying to implement the given steps (as mentioned in the publication) working with similar climatological time series data. I got the different IMFs and the residual.

For the next step, which is to feed all the IMFs and Residual to LSTM, as in Section 3.5 (Forecasting IMFs), the paper mentions that to fix input of the LSTM, a PACF (Partial Auto Correlation Function) analysis has to be done and the number of lags beyond 95% Confidence Interval(CI) is the number of inputs to LSTM.

I understand you are getting the PACF order for the decomposed results (for example: 4,6,5,5,6,6,1,1,1,1,1 for the max pooling station as shown in Fig 7 of the publication where 4 is the number of lags for original data series, 6 is the number of lags for IMF1 series,...., 1 is the number of lags for IMF9 series, and 1 is the number of lags for Residual series).

My question is: how exactly is this PACF order related to input of LSTM (does it determine the number of past data points to look back)?

It would be great if someone could provide some explanation on this.

Thanks.


texts: Calculate a ranking function from classification features

texts:
I am using 3 features (x1, x2, x3) for binary classification. All my feature values are in 0 to 1 range (unit range).

I obtained how important each feature was in classification as follows (i.e. feature importance)

x1 --> 0.1
x2 --> 0.5
x3 --> 0.7

It is clear that feature 3 (x3) contributes the most, x2 the second and x1 the least in classification.

I also performed correlation analysis to check if my features are positively or negatively correlated with the target (y) as follows.

x1 --> positively correlated
x2 --> positively correlated
x3 --> negatively correlated

I am wondering if it is possible to convert my classification features into a ranking function using feature importance and correlation.

For instance, my suggestion looks as follows.

ranking_score = 0.1*x1 + 0.5*x2 + 0.7*(1/x3)

The reason for using (1/x3) in the above equation is because it is negatively correlated with the target (y). Please let me know if my ranking_score equation is statistically correct? If not, please let me know your suggestions.

EDIT: Why ranking is important to me?

My features are related to employee details (x1, x2, x3). At first I used these 3 features to classify efficient and 'inefficient' employees. Now, I want to rank the efficient employees based on these 3 features. The above equation I proposed is to facilitate this task.

I am happy to provide more details if needed.


texts: Trouble finding the derivative of an optimization function

texts:
How can I derivate the following optimization function?
$$L=\sum_{u,i}(y_{u,i}-v_ix_u)^2+\lambda\left(\sum_i\|v_i\|_2^2+\sum_u\|x_u\|_2^2\right)$$
I just want to get the equations of the gradient descent method, so I want to get the partial derivative of $L$ with respect to $v$ and the partial derivative of $L$ with respect to $x$. This function is similar to ridge regression function.
I just know how to calculate the first of the term of the function, but I have trouble to derive the second addend of the function:
$$\delta v_i^k L=-2\sum_i(y_{u,i}-v_i x_u)x_u^k + ¿?$$
$$\delta x_i^k L=-2\sum_i(y_{u,i}-v_i x_u)v_u^k + ¿?$$


texts: Is there a good alternative to XGBoost for learn to rank?

texts:
My problem with XGBoost is that when I load the train dataset into the XGBoost DMatrix, there is a memory spike that is unavoidable, and I can't get my dataset loaded into RAM without crashing first.
I am trying to predict rankings over time, similar to a search engine query problem. XGBoost has produced good results using the pairwise objective but I have reached my limits due to the memory spike issue.


texts: A bump in CV score curve! What does it mean?

texts:
My learning curve is behaving strangely and I don't know if I'm doing anything wrong or it is because of the dataset nature.
I'm using a Neural network with (30,30,15,1) layers and 'ReLU' activation function for training and linear for the output layer.
My learning curve is showing weird behavior while learning. There is a bump like increase and decrease in the validation curve around 100 epochs which I don't have any idea about the source or how to resolve it(or even if it is needed to be resolved). Can anybody help me with this? (Any related comments are welcomed)


texts: Appropriate model metric for a truncated response variable?

texts:
Here's a straightforward question I can't seem to find a good answer to. Let's say you're using some variables to predict age. I'm assuming a regression model is the right approach. In this case, what would be a suitable metric to evaluate the model performance? My sense is that MAE and RMSE aren't appropriate because they assume your predictions can be equally wrong in both directions, but that's not true: predictions can be larger in the positive direction than in the negative direction. For example, if the true age is 5, the prediction can be no less than 0, but could be infinitely high.

Is RMSE/MAE an appropriate metric to evaluate the size of the error, and if not, what would be? Or if there isn't an appropriate "size" metric, should I use R^2 or some other "fit" metric?


texts: Audio Spectrum Normalization for NeuralNetwork Classification

texts:
in my free time I am working on a little machine learning project with focus on NeuralNetworks. It is all about Chord-Classification from an acoustic guitar through a microphone. So here is what I got so far:

I am using a simple FeedForward-Network (for first) and train it using the Backpropagation algorithm. First I record some sounds of me strumming simple chords (once, no complex strumming). This data is transformed in realtime with the FourierTransformation into a nice spectrum (this is done every 10ms) and saved in a file. After that I load these files containing the information about which chord was played and the spectrum data through time (every 10ms I got a full spectrum). This data then is normalized by the highest amplitude of all spectrum data (of all chord recordings). The now normalized data is used to train a simple neural network using batch training and backpropagation (with momentum). The activation function is the default sigmoid-function. After some epochs (about 30) the accuracy is mostly around 95%, enough for me! So then I export the network into a file, just to keep it stored. Now when I play some chords and want them to be classified, I load the NeuralNetwork. Before I can start playing, I have to calibrate the program for normalization reasons: I play a few chords and the program transforms it into a spectrum just like the training data. But this information is now used to normalize the real time data of me playing chords recorded through the mic. This basically works well. The program does what it should: Classifies the chords I play (mostly) correctly.

So whats my problem?

I don't like the way I am normalizing my data, as for now it depends on how loud I play the chords (highest amplitude). The problem is, if I train the network for example with chords played in a mid range volume it has problems classifing me playing very loud. Same problem would be with more quiet playing. Training the Network with loud and quiet data doesn't solve this either as the more quiet data (smaller amplitudes) is kinda getting lost by the loud data (high amplitudes).

This is where you come in: I want to know how I could optimize that. How can I normalize the spectrum data so that my network will classify loud data as well as quiet data correctly. (I know there may be better techniques like LSTM-ReccurentNetworks, but I want this to work properly, before I go onto that topic)

Thanks in advance!


texts: What is wrong with my Precision-Recall curve?

texts:
Hi,
I found this:
https://github.com/rafaelpadilla/Object-Detection-Metrics
I prepared my data: ground-truth and prediction files with bounding boxes.
but I got a very strange plot. What do you thing?


texts: Does Pooling remove spatial information of image in CNN?

texts: Pr. Geoffrey Hinton has pointed out that pooling-layers remove spatial feature information. But, essentially, does the process that last convolutional layer's features are flattened for FC layer makes the spatial information removed?


texts: Chunker/shallow parser for spoken language

texts:
I'm trying to extract NPs from transcribed spoken text, such as

  um it's the bl- it's the blue one in the right no left hand corner

which contains e.g. fillers (e.g. um) and disfluencies (e.g. bl-, right no left hand corner) that are not commonly seen in written text. Ideally, I'd like to get something like the three sequences it, the blue one and the left hand corner (or at the very least the right no left hand corner).

I'm currently using Stanford CoreNLP's pre-trained shift-reduce parser with a beam size of 4 (englishSR.beam.ser.gz) and bidirectional dependency network POS tagging (english-bidirectional-distsim.tagger) after filtering out fillers and duplicated tokens (e.g. uh it's it's that one → it's that one). This performs okay but seems to fail a lot more than I'd expect; Are there no chunkers or (shallow) parsers widely available which are tailored specifically to spoken English as opposed to written English? The language the chunker/parser is written in is irrelevant (i.e. it needn't have a Java API). I've also tried using Stanford CoreNLP's caseless models, which actually seem to perform a bit worse (however, I haven't done any rigorous comparisons).


texts: Is it possible to have a better performance with C4.5 than Bagged tree?

texts:
I am not sure but I have read that Bagged tree is used to improve the accuracy of a signle tree methods such as C4.5 but applying both of them over the same dataset I got better accuracy with C4.5, 100% while the Bagged tree gave me 97%
Is this result acceptable or not?


texts: Classes of neural nets and their applications

texts:
Would you say, you could design,  tune and/or train any DNN for any application, or do their designs inherently postulate some specialization?
Is there such a review?

For example, are CNNs better for the spatial domain (e.g image analysis), and RNN for temporal (e.g. time series data, or speech), or it does not matter?


texts: Classify text labels in to a similar category

texts: I'm trying to classify same kind of text labels in to one category. For example, if I have labels like qty, quantity, qty_no all of them should direct to Quantity. Since I'm new to data science, what's the best way to start this kind of thing?


texts: Performance of four GTX 1080 Ti's versus one Tesla V100 for Deep Neural Network training

texts:
While this may be slightly off-topic, this question does pertain to data science and machine learning. I want to train a VGG16 model on Imagenet from scratch. For this purpose, I am looking into either buying four to six GTX 1080 Ti's or one Tesla V100. I have a feeling that four GTX 1080 Ti's will perform far better than a single V100, but has there been any practical research into this?

EDIT - Each GTX 1080 Ti has 11.3 teraflops of FP32 performance (or 45.2 teraflops for four 1080 Ti's, or slightly less since the scaling isn't linear), whereas the V100 is 100 teraflops, or more than double the performance of four GTX 1080 Ti's. Anyone have any benchmarks for these two on imagenet performance?


texts: ``Hierarchial features extraction'' in Multilayer Perceptron models

texts:
I am referring to plain neural networks, MLPs.
I got to read the paper by Glorot and Bengio (2010), Understanding the difficulty of training deep feedforward neural networks.

Therein I read an interesting statement: ``Deep learning methods aim at learning feature hierarchies with features from higher levels of the hierarchy formed
by the composition of lower level features''.

What does that mean in the framework of MLPs? As far as I could understand, such a features extraction and abstraction process is made feasible by the use of convolutional layers, hence more advancer Deep Learning models.


texts: Extracting keywords from pdf file with python

texts:
I have a pdf file(link below).

I have to extract keywords from it and also need have there frequency in pdf file. E.g. 'Java':42, 'inheritance':3.

I have to store the keyword with their weights in an excel sheet.

Can you please tell me some code in python to do it.

https://drive.google.com/file/d/1gZCnlhwVMBIE0SugUUxDIgQrfVz-cDQR/view


texts: Single Layer Perceptron with three classes

texts:
I need some help with a single layered perceptron with multiple classes.

What I need to do is classify a dataset with three different classes, by now I just learnt how to do it with two classes, so I have no really a good clue how to do it with three.

The dataset have three different classes: Iris-setosa, Iris-versicolor and Iris-versicolor.

The url with the dataset and the information is in : http://ftp.ics.uci.edu/pub/machine-learning-databases/iris/iris.data.

I really appreciate any help anyone can give to me.

Thanks a lot!


texts: Pretrained CNN model on animal dataset ( turtles images if exist )

texts:
I was wondering if there is a pretrained CNN model on an animal dataset. I am participating in a turtle face detection competition and was wondering if there was a pretrained image model to fine tune that I could use for this competition.
And if someone could tell me, is there a website or something where I can find multiple pretrained models to use (I already know modelzoo, but is there another website?).


texts: Setting up network communication between R and Python

texts:
I want to share data between R and python via network sockets (I expect the solution to also work for R-R and python-python). I am interested in server being both R and Python side. For now I am interested in a simpler case of synchronous connections.

What are the best approaches, packages and formats to use for this task?


texts: Has this method of NLP processing with neural networks been done?

texts:
Take the sentence:

If you see a green light then you may cross the road.

I propose a neural network which produces as output from this sentence 3 masks  and a classifier as follows:

input   : If you see a green light then you may cross the road.
output 1: XX                       XXXX
output 2:    XXXXXXXXXXXXXXXXXXXXX
output 3:                               XXXXXXXXXXXXXXXXXXXXXXX
output 4: [if-then-statement]

So basically we train the neural network on a corpus to split the sentence into it's most general structure. Then we take the masks and use it on the input to produce a new phrase. e.g. let's take the mask from output 2:

input   : you see a green light
output 1:     xxx
output 2: xxx
output 3:         xxxxxxxxxxxxx
output 4:[event]

This time the neural network classifies the input as an event and creates masks for splitting the sentence into subject-object-verb.

So we repeat the process using the masks created to change the input until we have parsed the sentence into a tree-like representation. We use the same neural network each time.

If it fails to find a classification for an input it could go back up one level and try a different classification. (Much like a human might do if it doesn't quite understand a sentence).

The result would be an internal representation which might consist of a tree-like structure of masks and classifications. Which might then be more useful for reasoning about the meaning of the sentence.

This is kind of a top-down approach. I'm not sure how it would learn these masks from un-marked speech, but certainly I think it could learn them from a tagged corpus. I think maybe a similar approach might be used backwards to generate sentences.

Has this sort of thing been tried before?


texts: How can we use mean imputation without violating feature correlation?

texts:
Mean imputation is generally bad practice because it doesn’t take into account feature correlation. Imagine we have a table showing age and fitness score and imagine that an eighty-year-old has a missing fitness score. If we took the average fitness score from an age range of 15 to 80, then the eighty-year-old will appear to have a much higher fitness score that he actually should. Thus, I wonder if we have any way to use mean imputation without violating feature correlation.
For me, a straightforward solution is to replace mean of the whole population in data by mean of a group of similar subjects. For example, in the above example, we can fill missing score of a person by mean score of other persons of the same age, or for more flexibility, in the same range of age. This approach still works if we have more than 2 features and still want to take into account correlation among all features. We just need to define a similarity metrics between subjects based on all features, e.g. cosine similarity. Of course, similarity computation is expensive, that is a limitation of this approach. So I wonder if there is any better alternative.


texts: How to create a simple Autoencoder to select features from high dimensional dataset

texts: I am trying to create a simple Autoencoder to select features based on a high dimensional dataset. I can understand, how AE works and how AE will reduce the dimensionality but I am looking for a sample or tutorial for the AE feature selection which is implemented based on the Keras DL tool. I would really appreciate your help on this.


texts: What are some situations when normalizing input data to zero mean, unit variance is not appropriate or not beneficial?

texts: I have seen normalization of input data to zero mean, unit variance many times in machine learning. Is this a good practice to be done all the time or are there times when it is not appropriate or not beneficial?


texts: can you suggest best method for background / ambient noise removal from cough audio and keep only cough signal

texts: I'm performing background/ambient noise removal form cough signal and keep only cough audio signals using python.Can you suggest best noise filters to achieve this and keep only cough audio signal from .wav file.


texts: Data normalisation and recommendation based on skillset

texts:
Given the job title return the skillset required for the job
If a user is lacking some skills required for the job, we have to suggest the courses the user has to take to bridge the gap.

Problem with 1) is:

1) The provider is returning the skillset only if the keyword(tile of the job) is exact. For instance: For "Accountant" it gives the skillset whereas for "Accounting" it didn't give any result. One another example is: It gives result for "Developer" not for "Software Engineer" or "Software Developer"

Does NLP or machine learning solves this issue? If yes, can you give some recommendation on how this problem could be approached.

For 2): What is the way to handle recommendations?


texts: Can you apply PCA to part of your dataset?

texts: I am working with kaggle dataset that has over 130 features composed of 116 categorical and 14 continuous features. I plotted the heatmap for the 14 continuous variables and found that most of them are weakly correlated with the response variable but highly correlated with each other. I am trying to apply PCA to this part of the data and glue them back together as columns with the categorical variables. Is it ok to do so? Or should I one-hot-encoding / label encoding the categorical variables and do pca to the entire dataset?


texts: TensorFlow: how to print out all elements in a tensor

texts: I have a tensor that is of the size (19,19,5,80). I want to print out all the elements in it but if I use print(XXX.eval()), most of the elements will end up as ellipsis. How can I print all of them out?


texts: Clusters: how to improve results for text classification

texts:
I am trying to classify texts using kmeans, TfidfVectorizer, PCA. However, it seems that many texts are not correctly classified as you can see: I have texts in cluster2 that should be in Cluster 0 or 1.
My question is on how to improve the results, if increasing the number of clusters or adding more constraints (like specific words to look at for clustering texts).

Help and suggestions will be greatly appreciated.

Code:

def preprocessing(line):
    line = re.sub(r"[^a-zA-Z]", " ", line.lower())
    words = word_tokenize(line)
    words_lemmed = [WordNetLemmatizer().lemmatize(w) for w in words if w not in stop_words]
    return words_lemmed

vect =TfidfVectorizer(tokenizer=preprocessing)
vectorized_text=vect.fit_transform(df['Text'])
kmeans =KMeans(n_clusters=n).fit(vectorized_text)

cl=kmeans.predict(vectorized_text)
df['Predicted']=pd.Series(cl, index=df.index)

df.groupby("Predicted").count()

kmeans_labels =KMeans(n_clusters=n).fit(vectorized_text).labels_

pipeline = Pipeline([('tfidf', TfidfVectorizer())])
X = pipeline.fit_transform(df['Text']).todense()
pca = PCA(n_components=n).fit(X)
data2D = pca.transform(X)

kmeans.fit(X)
centers2D = pca.transform(kmeans.cluster_centers_)
labels=kmeans.labels_
cluster_name = ["Cluster"+str(i) for i in set(labels)]


texts: Feature Importance Scores from Gradient Boosting vs Random Forest

texts:
In sklearn, the feature_importances_ attribute exists for both RandomForestClassifier and GradientBoostingClassifier.

Would like to know what are the fundamental differences in how this attribute is calculated for both algorithms and the pros and cons between both approaches.


texts: Precision/recall from out-of-bag predictions in RandomForestClassifier

texts:
I would like to use out-of-bag training/validation with a classifier such as RandomForestClassifier.

Is it possible to get the out-of-bag predictions?

I want the OOB predictions so I can compute precision/recall on instances not used to train the classifier.

If there are alternative approaches to getting the precision/recall, please let me know.


texts: How to add more weight to certain features?

texts:
I have extracted features from two types of signals. Prior to merging them to create one feature vector, I have computed an importance score of every feature within that type of signal.
I would like to weight the features according to those scores. Would the best way to do this be by multiplying every feature with its score and then concatenate the features of both signals, and should the data be normalized again after multiplication? Or, is there a different way to assign feature weights based on some precomputed score. (I have used multihead self attention to compute scores that I want to use as weights, for every feature within the modality). The model that I will use after merging the features, will be GRU.


texts:
I am currently solving a classification problem for an imbalanced data set (approximately 17% of the minority class). I split the data using a stratified k-fold split from sklearn (Stratified shuffle split) after which I oversample the train data, using ADASYN, and fit the oversampled train data (approximately 250k+ instances after oversampling) to gradient boosting classifier. Oversampling has a huge effect on the performance, recall measure improves from 7% to 75%. Is this possible? If not, any ideas what could be going wrong?

My main question about this is could this improvement be possible?


texts: Advise on making predictions given collection of dimensions and corresponding probabilities

texts:
I am a CS graduate but am very new to data science. I could use some expert advise/insight on a problem I am trying to solve. I've been through the titanic tutorial on gaggle.com which I think was helpful but my problem is a bit different.

I am trying to predict diabetes risk based upon Age, Sex...and other factors given this data: http://www.healthindicators.gov/Indicators/Diabetes-new-cases-per-1000_555/Profile/ClassicData

The data gives new cases people per 1,000 people for each dimension (Age, Sex...etc). What I would like to do is devise a way to predict, given a list of dimensions (Age, Sex...etc) a probability factor for a new diagnosis.

So far my strategy is to load this data into R and use some package to create a decision tree, similar to what I saw in the titanic example on kaggle.com, then feed in a dimension list. However, I am a bit overwhelmed. Any direction on what I should be studying, packages/methods/examples would be helpful.


texts:
I am trying to see if my data is multimodal (in fact, I am more interested in bimodality of the data). I performed dip test and it does evidence against unmodal data. However, I want to see, in particular, if it is bimodal. I believe silver man's test can be used. However, I couldn't find the implementation of it in either r or in python. (The one in R is old and not working with the current version of R).
Also, assuming that I have a bimodal data and that I am able to get the two components (using mixtools in R), how do I figure out how to find the point of intersection of the two components. For example, here is the histogram (overlaid with its density estimation) of the entire data.

Here are the two components:

I want to get the the x value where the curves intersected. I could have uploaded the data, but the length of the vector is rather long. Any general thought and idea is welcome including the R and/or Python packages are welcome.

Thanks


texts: How do you know when you are using a multi gpu?

texts:
I have 2 gpus on my local machines, but i'm not sure that the model I am training is using both of them (the speed has not changed much).

My code:

def get_model():
    base_model = ResNet50(weights='imagenet', input_shape=(image_size,image_size,3), include_top=False)
    #base_model.trainable = False

    model = models.Sequential()
    model.add(base_model)
    model.add(layers.GlobalAveragePooling2D())
    model.add(layers.Dense(1024, activation='relu'))
    model.add(layers.Dropout(0.5))
    model.add(layers.Dense(196, activation='softmax'))
    model.summary()

    model = multi_gpu_model(model,gpus=2)

    #optimizer = optimizers.SGD(lr=1e-4, decay=1e-6, momentum=0.9, nesterov=True)
    optimizer = optimizers.RMSprop(lr=0.0001)
    model.compile(loss='categorical_crossentropy', optimizer=optimizer, metrics=['acc'])

    return model

I just added the multi_gpu_model setting, but am not sure that this is enough. I checked nvidia-smi every 0.5 seconds, but seems like only one gpu is working. How do I make sure that it uses the full 2 gpus?


texts: Determine Optimal CNN Complexity: Achieve ~100% Training Accuracy, Then Add Regularization?

texts: I was just wondering whether it is a decent idea to first try to make a CNN complex enough to achieve close to 100% accuracy on the training dataset, and then slowly add regularization, augmentation etc to make sure that the network doesn't overfit completely but instead is able to generalize better.


texts: How to balance time/effort with transformations, feature selection, and models efficacy in nlp?

texts:
Edit: Question has been edited for reopening (see comment section for justification)
Being to new text analytics, I haven't gotten the hang of navigating a typical workflow given the longer times associated with the larger feature sets. My question then is how does one navigate optimization decisions when the cost of exploring all are often so high?
To provide a specific example, in a single text analytics problem, I have prepared a few different transformations of my training set:

Stemmed vs lemmatized
Count vectorization vs TF-IDF vectorization
Full feature space vs 30% less features (identified by correlation analysis)
All cross-combinations of the above

In an effort to get a sense of which of the transformation sets above I should run further tuning on, I ran untuned RF, Logistic, Naive Bayes, SGD, and KNN models on (with cross validation). Unfortunately, it was clear no transformation combination really stood out as a likely "winner".
How does one proceed here? All decision points are of equal merit empirically, but exploring all seems strategically wasteful if not impractical.


texts: how do tree based methods deal with missing feature columns?

texts:
all,
i have trained a model using xgboost. Some of the features are one hot encoded e.g. currency where it is either gbp or usd. it seems that when i output the feature importance gbp and usd were in 7'th 8th place respectively.
now i would like to use the model to predict whether defaulter or not on australian countries, however the currency for these is in AUD. Therefore when i apply my feature engineering it will create a column AUD once one hot encoded.
since my model doesn't have AUD as a feature how does it handle features which have been unseen? i am not clear on this


texts:
Say I have a document and I want to assign all (or most) of the words to pre-assigned topics. So I could take a random selection of words and manually assign them to the appropriate topics, then I want an algorithm to assign the remaining words to the appropriate topics based on how closely they occur to the manually assigned phrases.

Is there an algorithm or technique that does this? I've looked into topic modelling and word2vec and they seem to assign 'topics' in some arbitrary space without any room for manual training and designation of topics.


texts: Standardization/Normalization test data in R

texts: I understand that one should standardize and normalize the test data (or any "unlabeled" data) with the training mean and sd. How can I implement this in R language? Is there a kind of "fitting" to the training set and a kind of applying to the test data?


texts: Feature importance over a subset of instance space instead of an entire instance space

texts:
I'm really curious if anyone has faced this problem before, or is it even widely studied at all.

Imagine we have a feature that isn't important (based on many widely available and textbook feature selection methods) over the entire instance space of training data. However, if we try to  divide the instance space into certain subsets, can the feature become important over that subset, and are there any methods available for arbitrary datasets?

When I say 'dividing the instance space', I mean for example if I have 3 features $X_0, X_1, X_2$ with a binary label $Y$, a subset of instance space will be a region where, for example, when $X_0 >5, X_1>5$. In particular, could it be possible that $X_2$ isn't significant over the entire instance space, but is significant over the restricted space mentioned above. If it's possible (my intuition says so), are there any methods in current literature that allow me to find such a space (if it exists)?

If anyone has any links/papers/resources which can direct me to a similar study, I would gladly accept it as well.


texts: How to tell which class is which with SHAP values in a summary plot?

texts:
I ran a random forest classifier to predict pollen color based on various environmental variables. The possible predictions are purple or yellow. I want to run a summary plot in shapely to get an understanding on the importance of those variables. I run the following code:
import shap
explainer = shap.TreeExplainer(rfc)
shap_values = explainer.shap_values(X_test)
shap.summary_plot(shap_values, X_test)

This correctly gives me the following graph:

Cool. This gives me a bunch of valuable information. But I don't understand what class 0 and class 1 are. Which is supposed to correspond to purple and which is supposed to correspond to yellow? I know its an unremarkable question for only 2 possibilities, but I plan on doing a classification with 3 classes and I need to know how the classes are assigned.


texts: How can I use the vectors of words in a sentence to get the vector of that sentence . I have used strategies like - Averaging the individual word vectors or a tf-idf weighted combination of the words . While these hacks work , there are obvious problems with these . Wanted to know what would be some other way to do this


texts: Possible to use different learning rate for different neuron in Keras/Tensorflow?

texts:
The simplest example is to have faster/slower learning rates in the upper/lower layers of a network. I found this post on tensorflow.

Is there a similar trick in Keras?
Going one step further, can we set different learning rates for specific range/set of neurons/weights in a particular layer?


texts: Example: red paprica and green paprica. The output I need is just paprica, should i label them the same or give them two labels "green paprica" and "red paprica" and leater on just treat them as one? Other example: 5 cent coin, "heads" and "tails". Output i need is just 5 cent coin. From machine learning point of course. Are there downsides and upsides of splitting labels or keeping them together in terms of accuracy?


texts: How to find optimal number of trees in random forest using Grid search in R?

texts:
From below code, I am getting optimal number of mtry. What is this mtry ? and How should I find the optimal number of tree that to be assigned to Random forest algorithm so that it will give High accuracy.

Any comment will be highly appreciated ! Thanks in Advance.

classifier = train(form = Survived ~ ., data = training_set_scaled, method = 'rf')
classifier
classifier$bestTune


texts: Model to predict based on frequency of occurrence

texts:
I have the following dataset

+-----------------------------------+
|  Passenger           |    Trip    |
+-----------------------------------+
| John                 | London     |
| Jack                 | Paris      |
| Joe                  | Sydney     |
| John                 | London     |
| John                 | London     |
| Jill                 | New york   |
| Jim                  | Sydney     |
| Jack                 | Paris      |
| James                | Sydney     |
+-----------------------------------+

And am trying to use scikit library to predict the likelihood of next possible trip of a passenger based on the frequency ( In this case John => London).
As a novice am unsure on which model / function to use.

Update 2:

If I have over 10 million records , how different should I approach this problem ?


texts: Common human-writable text format for specifying the architecture (untrained) of a neural network?

texts:
I am curious if there is a common human-writable text format for specifying the architecture (untrained) of a neural network.
There are informal notations such as described here.
I am aware of ONNX and NNEF, but these are for fully-trained models, and are not human-readable because they include all the parameters.
To explain my use-case more, which I believe is quite common-place for myself, and other researchers:

You learn about a new neural network architecture, that you want to try training but with new data or with a different training regmine (because use a different platform or GPU setup).
You google and find a github repo with the model defined in some Python file in your favorite framework---or a framework you don't understand. This is surrounded by a bunch of training and inference and preprocessing code that you can't use, and a pretrained model on a different data-set than you want.
What you want is just to be able to grab the model architecture into a JSON, transpile it into your framework, and then insert it into your typically pipeline orchestration + training code.


texts: How to utilize user feedback due to miss-classification when correct class label is unknown?

texts:
Suppose we are developing an app which is supposed to predict a dog's breed by it's picture. We trained a classifier (in my case an MLP) using some dataset and shipped the app to users. Now suppose some user comes and takes a picture of a friend's dog and the app tells her there is 90% chance that this dog is an X. The user knows that this is not true, but she doesn't know what is the dog's breed (if she knew, why would she use our app in the first place?). So we get a feedback which tells us "this is a picture of a dog which is not an X". This sample could be a sample of some other class or a new class or not a dog at all.

I'm looking for a way to use this feedback, to improve the precision of my MLP in class X without touching other classes.


texts: Multiclass ROC Curve using DecisionTreeClassifier

texts:
I built a DecisionTreeClassifier with custom parameters to try to understand what happens modifying them and how the final model classifies the instances of the iris dataset. Now My task is to create a ROC curve taking by turn each classes as positive (this means I need to create 3 curves in my final graph). To do this I need to instantiate a OnevsRestClassifier and passing the previous classifier as parameter, so it automatically recognize the parameters I modified (such as the weights of the class). This is my current code:
from sklearn.datasets import load_iris
from sklearn.metrics import accuracy_score
from sklearn.metrics import f1_score
from sklearn.metrics import roc_curve, auc
from sklearn.model_selection import cross_val_score
from sklearn.preprocessing import label_binarize
from sklearn.model_selection import train_test_split
from sklearn import tree
from sklearn.multiclass import OneVsRestClassifier
from sklearn.metrics import roc_auc_score
from sklearn.metrics import confusion_matrix
import numpy as np
import graphviz

iris = load_iris()
X = iris.data
y = iris.target

# Binarize the output
y = label_binarize(y, classes=[0, 1, 2])
n_classes = y.shape[1]

# My DecisionTreeClassifier
clf = tree.DecisionTreeClassifier(criterion="entropy",random_state=300,min_samples_leaf=5,
                                  class_weight={0:1,1:10,2:10})

np.random.seed(0)

indices = np.random.permutation(len(iris.data))
indices_training=indices[:-10]
indices_test=indices[-10:]

iris_X_train = iris.data[indices_training]
iris_y_train = iris.target[indices_training]
iris_X_test  = iris.data[indices_test]
iris_y_test  = iris.target[indices_test]

# Training
clf = clf.fit(iris_X_train, iris_y_train)

# Test
predicted_y_test = clf.predict(iris_X_test)

print(confusion_matrix(iris_y_test, predicted_y_test))

print("Predictions:")
print(predicted_y_test)
print("True classes:")
print(iris_y_test)

# Learn to predict each class against the other
classifier = OneVsRestClassifier(clf)

# Train
classifier = classifier.fit(iris_X_train, iris_y_train)

# Test
y_score = classifier.predict_proba(iris_X_test)

# Compute ROC curve and ROC area for each class
fpr = dict()
tpr = dict()
roc_auc = dict()
for i in range(n_classes):
    fpr[i], tpr[i], _ = roc_curve(iris_y_test, y_score)
    roc_auc[i] = auc(fpr[i], tpr[i])

My problem is that I get the error:

Class label 2 not present.

At the line: classifier = classifier.fit(iris_X_train, iris_y_train)
This is when I train the new classifier and I do not understand why. I checked on the iris dataset and there are three classes, so the label 2 should correspond to virginica, right?


texts: How to make a CNN predict a continuous value?

texts:
I understand the dimensionality of convolutions, max pooling and dense as function of stride and kernel size. But I'm having trouble wrapping my head around how to use these layers to end up with my final prediction, which should be one continuous variable.

Here is what I'm working with:

X training: n 2-channel 3D grids of size 46x46x46, shape: (n, 46, 46, 46, 2)
Y training: n-element vector of continuous values, shape: (n,)

I would imagine there will be some resizing and some concatenation involved. But there's no point doing that if I don't actually understand what it's doing.


texts: Design strategies for higher resolution images for Convolutional neural network?

texts:
By BVLC/Caffe examples cifar10, it has 3 convolution layers with pool/norm and its accuracy increases to around 82 easily. Now I want to increase my resolution by downloading the 10 classes of images from imagenet and make the image resolution to 55.

The scenario is that the images have higher resolution and less data: 5000 per-class while cifar-10 has 6000 per class. What are good design strategies for this new scenario? Is the performance going to drop a lot?


texts:
If I have 1000 words and want to reduce the dimension to 500, does that mean per hash there will be two words on average?
In the course NLP specialization on courser, 40 million unique word matrix was reduced to 4 million word matrix.I can only imagine how many collisions will be there, but it turns out that there would not be may collisions.
Can anyone please throw some light on this.
Thanks.


texts: Probability of the recipient to open the email

texts:
I am trying to build a model using logistic regression, where my dependent variable is y=1 if the mail was opened, y=0 if it was not.

I have data approximately 10 records (10 rows) for every recipient who got an email and want to calculate the probability of opening the email per user. However, do not know how to edit the dataset based on this requirement.

Because the results (probabilities) now are like for every row (every mail) and I want it for every recipient.

ID is an ID of the recipient(see the screenshot) If I counted the number of openings, I would lose binary operator, what is necessary for logistic regression.

Got any ideas?


texts: Type of model for space-characters recovery?

texts:
I have the string "Ihaveadog" and i need to recover space-characters in the string.I tried to use seq2seq model but failed.In my case i've got data like "Ihaveadog" -> "010000101000" encoded by one-hot method.What type of DL model is good for the task or the best way is the using classic ML methods?

Thanks for your time.


texts: laptops which are suitable for heavy Image and video processing

texts:
I am a graduate student and My thesis is based on deep learning, vision, video and Image processing. I am going to do heavy computing and processing. I am looking for a compatible laptop which has a fast GPU-enabled laptop. CUDA enabled on an NVIDIA GPU with good TFOPS would be a perfect choice. Also, it must have SSD. Anything by hp, Dell, Microsoft, Acer. do you have any recommendations?  my budget is $CAN 1800.
which GPU-Nvidia is suitable for this kind of programming?


texts: How to estimate probabilities of different classes for a Text

texts:
Suppose I have a piece of writing and I want to assign probabilities to different genres (classes) based on its contents. For example

  Text #1 : Comedy 10%, Drama 50%, Fiction 20%, Romance 1%, Mythology 5%, Adventure 10%

  Text #2 : Comedy 40%, Drama 3%, Fiction 2%, Romance 30%, Mythology 5%, Adventure 10%

We have given keywords possibly ngrams in each class through which we make a comparison

  Class 1 Comedy : k11, k12, ..., k1m

  Class 2 Drama : k21, k22, ..., k2n

  Class 3 Fiction : k31, k32, ..., k3o

  Class 4 Romance : k41, k42, ..., k4p

  Class 5 Mythology : k51, k52, ..., k5q

  Class 6 Adventure: k61, k62, ..., k6r

What can be the best probabilistic model that we can use for this task


texts: The distribution of dataset train and test are the differents, how to fix this?

texts:
I am new in data science and like some help to understand my problem. For instance, I have two signals non-stationary for the same condition (figure 1).

I acquisition them at different times(in the morning and in the afternoon), when applying the Kolmogorov-Smirnov test, the null hypothesis was rejected, I don't understand why distribution is different if I no change any parameters in my system of acquisition.

This is the main trouble in my analysis because of this no get model any algorithm of machine learning to classification (overfitting).

I read something about the Covariate Shift (*I saw this post), and Kullback_Leiber Importance Estimation Procedure, but I don't know iff will really work out.

*'Different Test Set and Training Set Distribution'


texts:
Let's pretend we have a series of events, which have binary results $ y_1, \ldots, y_n $.

For every event we have a number of people with more or less stable features $ f_1, \ldots, f_k $ (like sex for given client for every event) and a volatile feature $ g $ (for instance, the amount of spent money for the corresponding event).

We want to predict the target $ y_t $ using this data. Empirically we found that values like $$ \sum \{ g | f_1 > c \} $$ have some predictive power and want to generalize them (in both ways, using different conditions and different aggregations).

What kind of machine learning algorithm can be useful for this setting?

The problem is, essentially, that we want to find a cluster based on features $ f_1, \ldots, f_k $ that have predictive power in terms of feature $ g $.


texts: Cleaning data with two fields mixed in the same column?

texts:
I am working on a template for a dashboard that is giving me some serious trouble. The format the data comes in is highly unstructured. In the image below, circled items are "Site Names", data is Month-level.
The issue is that the "Site Names" are mixed in with the Month, with an inconsistent number of months for each site. Some sites will display one year worth of data, others will contain up to 3 years. There is no field indicating Year, only Month.

I have managed to extract a list of unique Site Names from this column, so I have those on their own. My issue now is that I have no idea how to match them to their corresponding Months. I was thinking through how to do so with an HLOOKUP, however after planning it out briefly I'm not sure this would work. The site names I would be using as a lookup are not all present in the top row. I am absolutely stumped on how to achieve this.
Provided this isn't asking too much in one question, I also am faced with the challenge of getting Excel to understand that the first instance of a month for each site is an older instance of a month. I.E. "Jul" would be 2018 the first time it appears, 2019 the second time, 2020 the third time. Any help/tips would be greatly appreciated.


texts: How many examples needed for named entity disambiguation?

texts:
If I want to build a named entity linking system for resumes using an ontology of occupations and skills about how many annotations would I need? The ontology has about 20,000 entities.

As a lower bound I'm guessing I would need about 10 examples per entity and maybe 3 different annotators to label each mention so ~600K annotations. Does that make sense?


texts: Is it better to have higher train accuracy with lower test accuracy or higher test accuracy with lower train accuracy?

texts:
The results from my RandomForest model with 5 max features are as follows:

84% train accuracy
76% test accuracy

The results with 10 max features:

79% train accuracy
77% test accuracy

Which result should I be favoring? Would I be incorrect in saying that the second result is better because the test accuracy is higher even though the training accuracy is lower? Ultimately, you want the model to perform best on the test (unseen) data?


texts:
I am building an app where I identify information from the SMS, something similar to expense management apps. I have a parser which reads all the SMS of user, identifies SMS of interest and parses useful information from it. This bit is working fine, however the problem is that I want to also score completeness and accuracy of the parser.
Completeness - Out of 100 SMS of interest, how many of them parser is able to identify
Accuracy - Out of SMS identified as relevant, for how many SMS parser is able to extract right information
I have a manual process to identify these. I take a random sample from the corpus along with parsing results and manually tag which ones are correct or wrong. I am able to identify gaps in parser and fix them by this process. However more gaps I fix more it becomes difficult to identify further gaps with this process. Also since sample is random, it has many SMS with same structure. Parsing success/failure for similar SMS is similar. Completeness and accuracy from the random sample does not correctly represents correctness and accuracy of whole corpus as well.
I was thinking that since financial SMS of interest have templates associated with them, if I can tag template of the SMS, I can keep track the templates which are known to be parsed correctly and only do manual checks on templates which are not yet tagged manually for correctness. This would also help me build tools to identify correctness, accuracy and unknown status for the whole corpus. It can also help me identify new templates as they are added to the corpus.
There are 1000s of templates. Examples of templates:
Template1 - Thanks for spending INR 100 at Amazon on 1st Jan, your available credit limit is 1,000
Template2 - You have spent INR 100 on your credit card ending in 1234 at Amazon on 1st Jan 2020
Template3 - You txn of INR 100 is credited/reversed by Amazon on 2nd Jan 2020, Avl Bal - 1,100
Template4 - You account xxxx1234 has been debited with INR 500 and bene A/C xxxx4321 has been credit on 3rd Jan

I have searched a lot on google and many results which seemed relevant were from this forum. Similarity based on n-gram seemed quite interesting, but I was not able to make much of how to apply it to my problem. One of the questions which seemed very similar: Identifying templates with parameters in text fragments. However, it differs slightly in the sense that it identifies sentences with similar templates out of given corpus. I want to tag template of a sentence without looking at the whole corpus. I am okay with training a model to tag the template, but I would want the model to identify new templates which are not part of original corpus and tag them as well.
I am a software engineer and have no prior experience with data science. Please pardon me if I am not making any sense or if this forum is not the right place for this question. I would really appreciate any help in identifying the right approach and some references to understand the concepts involved.


texts: Is there a C library for machine learning algorithms?

texts:
Are there any machine learning libraries for C.
Specifically interested in unsupervised learning.


texts: How to encode a time series as an image to feed it into CNN?

texts:
I want to try CNN in the task of stock chart pattern recognition. I suspect that feeding a line chart won't work because the image will have a lot of empty pixels.

What time series encoding options are available to get a lossless, dense image?


texts:
I have gone through this regarding handling missing values in categorical data.
Dataset has about 6 categorical columns with missing values. This would be for a binary classification problem
I see different approaches where one is to just leave the missing values in category column as such, other to impute using from sklearn.preprocessing import Imputer, but unsure which is better option.
In case if imputing is better option, which libraries could I use before applying the model like LR,Decision Tree, RandomForest.
Thanks!


texts: eta and learning_rate different in xgboost

texts:
I am creating a classification model using xgboost in python. I am using different eta values to check its effect on the model. My code is-

for eta in np.arange(0.2, 0.51, 0.03):
    xgb_model = xgboost.XGBClassifier(objective = 'multi:softmax', num_class = 5, eta = eta)
    xgb_model.fit(x_train, y_train)
    xgb_out = xgb_model.predict(x_test)
    print("For eta %f, accuracy is %2.3f" %(eta,metrics.accuracy_score(y_test, xgb_out)*100))

I expected different accuracies for some eta values, but to my surprise I got same accuracy for each eta. When I printed the model, I got this-

>>> print(xgb_model)
XGBClassifier(base_score=0.5, booster='gbtree', colsample_bylevel=1,
       colsample_bytree=1, eta=0.5, gamma=0, learning_rate=0.1,
       max_delta_step=0, max_depth=3, min_child_weight=1, missing=None,
       n_estimators=100, n_jobs=1, nthread=None, num_class=5,
       objective='multi:softprob', random_state=0, reg_alpha=0,
       reg_lambda=1, scale_pos_weight=1, seed=None, silent=True,
       subsample=1)

Here you can see eta = 0.5, but learning_rate = 0.1. While in xgboost docs, learning_rate is an alias for eta. So how is this possible that both have different values?


texts:
I have a timeseries data and I want to forecast it by applying ARIMA. After reading data, I decomposed it to analyze its components and get an idea whether it is stationary or not. It seems there is no seasonal component and a trend. Plots are as below:

I also applied Dickey-Fuller test to data and got 0.010293 p-value indicating that timeseries is stationary. Then I applied some transformations to improve test results and achived better stationarity. I applied log transoformation and after that I got 9.540976e-11 p-value. Then I applied differencing to log transformed data by 1 shift and got 2.253272e-27 p-value.
Because test results are not bad I moved to determine P, I, Q terms of the ARIMA model. I specified I term as 1, because shifting gave better test results. For P I used partial auto-correlation function and for Q I used auto-correlation function. Plots of them as below:

From these plots I pick P term of AR as 8 and Q term of MA 26. Simply I chose points in the x-axis where the y-axis is 0. I don't know well the theory behind this rule of thumb, but I saw it is used in different notebooks and probably there is a problem. Because when I tried to fit my ARIMA model as below, I got an error:
model = ARIMA(ts_log, order=(8,1,26))
results = model.fit(disp=-1)

ValueError: The computed initial AR coefficients are not stationary
You should induce stationarity, choose a different model order, or you can
pass your own start_params.


texts: Heuristic argument for Weight decay and regularization

texts:
I have been reading this book and I have no idea how to make an argument for this problem. http://neuralnetworksanddeeplearning.com/chap3.html

Connecting regularization and the improved method of weight initialization L2 regularization sometimes automatically gives us something similar to the new approach to weight initialization.

Suppose we are using the old approach to weight initialization. Sketch a heuristic argument that:

(1) supposing $λ$ is not too small, the first epochs of training will be dominated almost entirely by weight decay;

(2) provided $ηλ≪n$ the weights will decay by a factor of $e^{\frac{−ηλ}{m}}$ per epoch; and

(3) supposing λ is not too large, the weight decay will tail off when the weights are down to a size around $\frac{1}{\sqrt{n}}$, where n is the total number of weights in the network. Argue that these conditions are all satisfied in the examples graphed in this section.

Can anyone explain a little bit more and guide my direction? Thank you so much for reading it.

My argument is that:

(1) I am thinking about this formula $w' = w(1-\frac{ηλ}{n}) - \frac{n}{m} \sum \frac{dCx}{dw}$ is related. Since the weight is larger without using new weight initialization (the standard deviation of $w = \frac{1}{\sqrt{total number of weights in the network}}$, $\frac{ηλw}{n}$ will be significantly larger than sum of $\frac{dCx}{dw}$ （the average rate of change of $C$ in respect of weight for a minibatch).

(2) By substituting $ηλ+b = n$ into $w'$??

(3)

Thank you again.


texts: Should I include all dummy variables or N-1 dummy variables (keep one as reference) in neural networks

texts:
I have a categorical variable with N factor levels (e.g. gender has two levels) in classification problem. I have converted it into dummy variables (male and female).
I have to use neural network (nnet) to classify. I have two options -

Include any N-1 dummy variables in the input data (e.g. include either male or female). In statistical models, we use N-1 dummy variables.
Include all N dummy variables (e.g. include both male and female)

Can someone please highlight the pros and cons of both options in predictive power and interpretability


texts: How to get a (descriptive) overview of a large database?

texts:
I'm facing a data framework with

~ 20 k observations and
151 variables across
2078 subjects

At first I am primarily interested in how the data looks like related to a single parameter. But I cannot plot 2078 subjects on the x-axis and make a bar plot out of it or so.
What would be useful methods for such a situation? I prefer some visualizations but I think they won't be applicable. I'm afraid even non-visualization methods are not really helpful as well.


texts:
I have a data set that includes the individual performance of a number of "actors" over some period of time.

For a less-than-ideal contrived example (I'm not able to share the actual use case), you can imagine this is a set of stocks on the NYSE.  My goal is to use an LSTM model to predict T+1 value for a given actor

I have a dataframe which looks something like this:

+----------+---------+-----------+-----+-----------|-------+
|   Date   | ActorID | Feature 1 | ... | Feature X | Value |
+----------+---------+-----------+-----+-----------+-------+
| 1/1/2019 |       1 | 1.1       | ... | 0.123     | 1.111 |
| 1/2/2019 |       1 | 1.2       | ... | 0.456     | 1.222 |
| 1/3/2019 |       1 | 1.3       | ... | 0.789     | 1.333 |
| 1/1/2019 |       2 | 1.4       | ... | 0.123     | 1.444 |
| 1/2/2019 |       2 | 1.5       | ... | 0.456     | 1.555 |
| 1/3/2019 |       2 | 1.6       | ... | 0.789     | 1.666 |
+----------+---------+-----------+-----+-----------+-------+

My question is how to ideally feed this data into my LSTM model for training.  Do I flatten the dataframe and turn all the actors' data on a given day into a single array?  Do I feed them sequentially?

Any general direction on the approach here would be hugely helpful.

Thanks in advance!


texts:
Im currently writing an evolutionary program that has the task to optimize a weight set for a neural network for a specific and well defined environment. It is performing excellently but i am getting a sort of feeling that i might be stuck in a local minima.
Since i am mutating these weight sets based on an autoadapted range (calculated by a logistical function with the fitness of the solution as input), i was wondering if the shrinking of my mutation range and thus of my search space might be the reason for this possible local minima?


texts: How does a query into a huge database return with negligible latency?

texts:
For example, when searching something in Google, results return nigh-instantly.

I understand that Google sorts and indexes pages with algorithms etc., but I imagine it infeasible for the results of every single possible query to be indexed (and results are personalized, which renders this even more infeasible)?

Moreover, wouldn't the hardware latency in Google's hardware be huge? Even if the data in Google were all stored in TB/s SSDs, I imagine the hardware latency to be huge, given the sheer amount of data to process.

Does MapReduce help solve this problem?

EDIT: Okay, so I understand that popular searches can be cached in memory. But what about unpopular searches? Even for the most obscure search I have conducted, I don't think the search has ever been reported to be larger than 5 seconds. How is this possible?


texts:
Suppose I have data with two independent variable $X_1$, $X_2$ and one dependent variable say $y$, as follows:

$X_1$: $x_{1,1}$, $x_{1,2}$ , $x_{1,3}$

$X_2$: $x_{2,1}$, $x_{2,2}$, $x_{2,3}$

$y$: $y_1$, $y_2$, $y_3$

I built some Machine learning model which is good .

Now I want to generate predictions not just for test data but for all possible combinations of test data for example, if our test data looks like

$X_1$: $a$, $b$, $c$

$X_2$: $p$, $q$, $r$

then I want predictions for pairs $(a,p)$$(a,q)$,$(a,r)$,$(b,p)$,$(b,q)$....etc

I have tried np.ravel, Meshgrid kind of commands but find it difficult.


texts: Can I use Linear Regression to model a nonlinear function?

texts:
I have recently started studying the basics about regression, and as a beginner I started by Linear Regression.

I read this article that says that for this particular type of regression the relationship between independent and dependent variables has to be linear, which to me implies that I can only predict "lines" with Linear regression:
https://www.analyticsvidhya.com/blog/2015/08/comprehensive-guide-regression/

But then I started wondering about how to model functions like "y = log(x)" or "y= sqrt(x)" or "y=exp(x)" or "y=tan(x)" or other nonlinear functions by definition which are not "lines" but "curves".

Then I carried on doing research until I found this article that says that it is not the relationship between the independent and dependent variables that should be linear, but the final functional form passed to the model:
https://medium.freecodecamp.org/learn-how-to-improve-your-linear-models-8294bfa8a731

I want to know if that is really the case, and is it always possible to do this "change" in the functional form? Also if it is possible to use linear regression for nonlinear functions, is it still correct to measure the performance of the model using R_square metric?

Thank you.


texts: Suitable instance counting CNN for training on polygonal masks

texts:
I have a medical dataset labeled with polygonal masks (rather than rectangle boxes). It works well for pixel annotation with UNet to generate masks of healthy vs damaged skin. Now I need to do instance counting. Most of the CNNs like YOLOv4 consumes bounding boxes, not polygons.
Which CNN should I use for instance counting given that my dataset consists of labeled polygons?


texts: What do you mean by applying ant colony optimization (ACO) to clustering? What is the output one would get after it? Could you explain it using a two dimesional data set which is clustered into 3 groups?


texts: Suggestions for Open-Source Tool for Image Classifications (with Nesting)

texts:
I'm looking for an open source tool to assist my colleagues and I to label images for a machine learning application. We don't actually need bounding boxes or anything to pinpoint regions within each image, and instead need solely global image classifications (e.g. whether the image is one of a cityscape, rural setting etc).
The mission-critical functionality we're looking for is:

image classification (both radio boxes and checklists)
the ability to nest labels, e.g. if label1=cityscape then label2 is required etc

Ideally it would also have one or more of the following features:

managing users / assigning tasks etc
the ability to provide labellers with a qualification test to measure their labelling vs pre-labelled images
metrics such as time per label, total time etc
the labelling UI to allow for available label options to be images or icons rather than text (e.g. rather than have text labels of 'square' or 'round', we actually show a square icon or round icon)


texts:
Using sklearn I can consider sample weights in my model, like this:
from sklearn.linear_model import LogisticRegression
logreg = LogisticRegression(solver='liblinear')
logreg.fit(X_train, y_train, sample_weight=w_train)

Is there some clever way to consider sample weights also in the Logit method of statsmodel.api?
import statsmodels.api as sm
logit = sm.Logit(y, X)


texts: Why do UNET-like architectures perform better than sliding-window approaches?

texts:
I'm writing a thesis that heavily focuses on semantic segmentation of biomedical images.

I'm reviewing different segmentation approaches, identifying two main approach branches:

A sliding window-like approach: a classification network is used over different patches of original image to reconstruct a pixel-by-pixel estimates of the probability maps.
A full-image approach: like the FCNN and UNET approach, rely on fully convolutional architectures and the upscaling phase is incorporated in the network itself using transposed convolutions.
https://arxiv.org/abs/1505.04597

The second approach clearly outperforms the first one in every context. I have a vague hunch on why this happens: my hypothesis is that the transposed-convolution operations, being at their core local operations, force local criteria on the segmentation of close pixels so that pixel contiguity is heavily encouraged in the fully convolutional case.

I do not find this kind of explanation satisfying because of two reasons:

I do not have papers or real data to support this: I cannot seem to find any paper on the theme.
The sliding-window approach has a built-in form of local consistency as well: if overlapping windows share most of the pixels it's reasonable to think that - given the network is not totally chaotic and shows enough linearity - the outputs would be similar.

Do anyone have a bit of insight or source on any of this? All contributions will be largely appreciated.


texts: How to compute unseen bi-grams in a corpus (for Good-Turing Smoothing)

texts:
Consider a (somewhat nonsensical) sentence - "I see saw a see saw"

The observed bi-grams would be:
"I see""see saw""saw a"and,"a see".

My aim is to smoothen out the probability mass of the bi-gram probabilities by using Good-Turing smoothing. For this, I need to find the count of unseen bi-grams, i.e., bi-grams with a frequency count of 0.

How do I do this?

1) Would this be a list of all bi-grams formed by using 2 non-consecutive words? For example, "I saw", "saw saw", "a I", etc.?

2) Would repetitions of the same word be included as bi-grams? Eg. "I I", "see see", etc.?


texts:
I'm implementing prediction code for courses of computing fields using Naive Bayes classifier. The output is to predict whether the course is (management, design, database, analysis,…9 classes). I have only about 100 records of courses descriptions. I label 50 by search keyword and I will use them as 80% of them as train data and the 20% as test data where the other 50 courses to predict by the classifier. The classes are not balanced I can’t to change anything in data.

The accuracy is 75% how can I increase the accuracy?

Is it small for training dataset?


texts: Multi-Class Classification With Logistic Regression On Binary Data

texts:
I am trying to implement a multi-class classifier with using logistic regression. In my dataset, attributes are words, for example first attribute is 1 if the data instance includes word "x" and it is 0 if it does not include the word "x", so all of my attributes are binary.

I partially understood how to implement multi-class logistic regression for categorical data from here https://github.com/tpeng/logistic-regression/blob/master/src/Logistic.java. However I could not figured out how to decide the weights when the attributes are binary. I found some academical articles about it but it is too heavy for me. I think this is really simple and well-known thing. How can I do that?


texts: SPARK, ML: Naive Bayes classifier often assigns 1 as probability prediction

texts:
Hi I am using Spark ML to optimise a Naive Bayes multi-class classifier.

I have about 300 categories and I am classifying text documents. The training set is balanced enough and there is about 300 training examples for each category.

All looks good and the classifier is working with acceptable precision on unseen documents. But what I am noticing that often when classifying a new document, the classifier is often assigning a very high probability to one of the categories (the prediction probability is almost equal to 1), while the other categories receive very low probabilities (close to zero).

What are the possible reasons for this phenomenon?

I can think of one possible reason, which is all the words in some documents did not appear with some categories in the training dataset? but I am not totally convinced of that especially that for most of the results, there is always one category who has very high probability and all the others have very low probability.
Is there any other explanations or reasons?

I would like to add that in SPARK ML there is something called "raw prediction" and when I look at it, I can see negative numbers but they have more or less comparable magnitude, so even the category with the high probability has comparable raw prediction score, but I am finding difficulties in interpreting this scores.


texts: Evaluation methods for multi-class classification

texts: I am looking for single-number evaluation method that can be used in multi-class classification tasks that take into account imbalanced data-sets. For instance, ROC-AUC defined by binary classifiers, is a single-number and takes into account imbalanced data-sets. On the other hand, accuracy is single-number, defined for multi-class classifiers and does not take into account imbalanced data-sets. Finally, the confusion matrix is defined for multi-class, takes that into account but is not single-number. Is there any evaluation method that satisfies the three conditions?


texts: Project structure - many projects share same large dataset

texts:
I have a bunch of projects for my job that are largely unrelated except they use the same data, which is pretty big on disk in csv format.  I want these to exist separately from each other and I usually try to use the cookie cutter data science model for project structure, and keep all my data in a data folder in the root of the project.

But because this dataset is big, I don't want to have ten copies of it in the root of these ten projects.  I also don't want to put them in one big project sharing it because I feel like they don't belong together.

What's the best way to structure multiple different projects that all share the same large dataset?


texts: Suggestion for choosing (building) loss funciton

texts:
I would like to build a supervised learning model M satisfying the following conditions:

Training data {X, Y}, where $x \in R^m$  and $y \in R^n$
Assume: M(x) = p, then: 0 < p[k] <= y[k], for all k = 1:n. Here I mean p[k] as close as possible to y[k]

Could you please suggest what are the "best" loss function and optimization method that I can use to train this model M?

Thank you.


texts: How to understand features impact in a non linear case?

texts:
I give a simple example:
I have a set of houses with different features (# rooms, perimeter, # neighbours, etc...), almost 15, and a price value for each house. The features are also quite correlated (i.e. perimeter is often correlated with #rooms).
I want to establish what are the main features (or non-linear combination of them) that determine the price.

In a linear case, for instance, I can compute a Lasso regression and see the importance of each feature through the coefficients.
In my case, every feature (or maybe combination of them) has a non linear impact.
For example, the # of neighbours can have a quadratic impact (increase the price if #neighbours < 10, and decrease the price if > 10).

I want to identify the main important relationship among the features and the prices.  I don't need a predictor. For example, at the end I will discover that the price depends principally by #rooms/perimeter and #neighbours^2.

I was thinking to apply Kernel methods, in combination with regression or PCA. But I don't know a lot about kernel methods.

Thank you in advance.


texts: Scipy minimization failing with inequality constraints or bounds

texts:
I am trying to use scipy.optimize to solve a minimization problem but getting failures on using an inequality constraint or a bound. Looking for any suggestions regarding proper usage of constraints vs bounds, and if any other algorithm would be suitable in this case.
The problem is:

Here is a reproducible code (res['success'] and res1['success'] are both False- optimization fails):
import pandas as pd
import numpy as np
from scipy.optimize import minimize as scimin

def obj(wc, M, wb, S):
    return (wc.dot(M.T) - wb).dot(S).dot(wc.dot(M.T) - wb)

n=278
k= 16

c_labels = ['c'+ str(i) for i in range(k)]
r_labels_1 = ['r' + str(i) +s for i in range(k) for s in ['a', 'b']]
r_labels_2 = ['r' + str(i) for i in range(k, n-k)]
r_labels = r_labels_1 + r_labels_2

wb = pd.Series(index=r_labels, data=np.random.triangular(0, 1.0/n, 0.3, n))
wb = wb/wb.sum()
cc = pd.Series(index=c_labels, data=4 + 2*np.random.random(k))
cb = pd.Series(index=r_labels, data=3 + 10*np.random.random(n))
s_pre = np.random.rand(n, n)
S = pd.DataFrame(index=r_labels, columns= r_labels, data=s_pre.dot(s_pre.T))

M = pd.DataFrame(data=np.eye(k), index= ['r'+ str(i) +'a' for i in range(k)], columns = c_labels)
for i in range(k):
    M.loc['r' + str(i)+ 'b'] = M.loc['r' + str(i) + 'a']
M = M.loc[r_labels_1].applymap(lambda x: x* np.random.rand())
M = M / M.sum()
for i in r_labels_2:
    M.loc[i] = 0

one_k = pd.DataFrame(index=M.columns, data=np.ones(len(M.columns)))
con1 = {'type': 'eq', 'fun': lambda x: x.sum() - 1}
con2 = {'type': 'eq', 'fun': lambda x: cc.dot(x) - cb.dot(wb)}

# try 1 with inequality constraint
con3 = {'type': 'ineq', 'fun': lambda x: min(x)}
consts = [con1, con2, con3]
res = scimin(obj, x0=one_k, args=(M, wb, S), constraints=consts, method='SLSQP')
assert (res['success'] == True)
wc = res['x']
oj = obj(wc, M, wb, S)

# try 2 with bounds instead of inequality constraint
bounds = [(0, 1000)] *len(M.columns)
consts = [con1, con2]
res1 = scimin(obj, x0=one_k, args=(M, wb, S), constraints=consts, bounds= bounds, method='SLSQP')
assert (res1['success'] == True)
wc1 = res1['x']
oj1 = obj(wc1, M, wb, S)


texts: Doing Supervised Learning where output should be a probability that adds to 1

texts:
I'm trying to do some supervised learning on a dataset to surface which input is the most probable true input
Example Dataset
As you can see, each INPUT_ID only has one True IS_TRUE_INPUT. The goal is to obtain a probability of each input being the true input for each INPUT_ID.
I'm stuck as this does not seem like a normal classification problem and the probability of each input for each INPUT_ID should add up to 1.
Thank you for your help!


texts: crop part of image from image, change it and put it same place in original image

texts:
I have image (think of image of small plant) with bounding box. For data augmentation I want to rotate (45 degrees) the image (plant) in bounding box, and place it at same place in original image. so that I don't have to annotate again in image. Only difference is that image(plant) in box is rotated
by 45 degrees.   I am using cv2 and pil. I have below code writtend and got help up after that. Looking for inputs. Thanks for your time.
demo_test_img = 'repa_trn_23_4598958_2920949_png.rf.996ba731acd8b0d4b75c57acf85cc91b.jpg'
demo_test_img_path = os.path.join(f'{DIR_TRAIN}',demo_test_img)

# read image
demo_read_test_img = cv2.imread(demo_test_img_path, cv2.IMREAD_COLOR)

# colors for boxes.
colour_dict = {1:(220, 0, 0), 2:(0, 0, 220)}

# get number of boxes present in image.
demo_img_boxes_info = train_df[train_df["image_id"] == demo_test_img]
print('Number of boxes for demo image ', len(demo_img_boxes_info))

for index, row in demo_img_boxes_info.iterrows():
  if (row['class'] == 1 or row['class'] == 2):

    H = int(row['h'])
    W = int(row['w'])
    X = int(row['x'])
    Y = int(row['y'])
    cropped_image = demo_read_test_img[Y:Y+H, X:X+W]
    print('Box for  idx is ', index, 'is ',[X,Y,W,H])

    **# TODO: rotate cropped image and place it at same location of original demo test image.**

fig, ax = plt.subplots(1, 1, figsize=(16, 8))
ax.imshow(demo_read_test_img)


texts:
I am a newbie in this field. Developer since 20 years and more but never done anything (except tutorials) with ML, DL, and NLP. Though I've already read a bunch of articles and tutorials about this technology and I am starting to figure out the steps and the conditions required to make it work.

What I would like to achieve (the reason of my question) is this:

I have a file containing about 2 years of conversations between me and another person. I would like to extract sequences of messages that are related by the same topic. I mean messages that are sequential in time and that belong to the same conversation about a topic.

My goal is to extract the time we spent on every single topic.

There is any model already trained for this task? (This is of course the obvious question :-P )

Or, there is any model that I can use as a start training base?

Or, if not, what should be a good approach (steps, techniques, software) to train one by my self? (If possible).

Thanks

Update

Thanks to Erwan response I made some more research. I am not sure if Erwan's answer can be considered a resolutive answer but no doubts that it pointed me to a possible research direction "removing part of the fog in front of my eyes". Since, in my case I don't have a labeld dataset to train a supervised model I started to search for LDA solutions (as you implicitly suggested and also based on several tutorial I found) and found some: gensim's LDA model and parallelized LDA model; lda-project; I also found a Java implementation that is MALLET LDA class ecc...

I also started to browse arguments like Word2Vec, FastText (and family - I guess), even if I am not sure what are the purpose of these software compared to LDA, possibly because I have not a clear and complete view of the informations that LDA models can "capture" compared to the others.

Though the first step was to find a software that supports Italian language "natively" and I found spaCy a tool that provides lemmatization, tokenization, ecc... and seems to have the purpose to prepare text for other topic detection softwares (even if it seems it has its own class for this, guessing by name, TextCategorizer even if I am not able to figure out how to use it because of the lack of a complete documentation and examples).

So my guess is I could use spaCy to prepare the text to be feed to gensim's LDA (or any other implementation).

For data visualization I found pyLDAvis.


texts: How would you deal with inf. or NA for rate or ratio as a feature variable

texts:
I'm trying to create a feature for a churn model (binary classifier).
The feature is mean of sales growth rates for several months. But if I just take the mean of sales for several months, I often get NAN or inf. since sales are often zeros. I could impute some numbers like 0 or mean as the missing sales but I feel I'm modifying the pattern/underling distribution. How would you go by creating such a feature for a classification model?

Thanks!


texts: Is it possible to use TensorFlow inside a python script in Azure Machine Learning Studio?

texts:
I'm trying to get TensorFlow running inside a python script in Azure Machine Learning Studio. As TensorFlow is not part of Azure Machine Learning Studio, I needed to import it using a zip file. I followed the instructions here: https://stackoverflow.com/questions/44593469/how-can-certain-python-libraries-be-imported-in-azure-mllike-the-line-import-hu

However, when trying to import TensorFlow, I get:

ImportError: No module named _pywrap_tensorflow_internal
Failed to load the native TensorFlow runtime.

It seems like TensorFlow is much more than just a python library. It seems like it needs a native library to work. Sounds logically, but there's this video on YouTube showing that it should work after all: https://www.youtube.com/watch?v=SEvoZ6EnUpk

My question is: is it possible to run TensorFlow inside a python script in Azure Machine Learning Studio? If so, how?


texts: pandas groupby.count doesn't count zero occurrences

texts:
I am using groupby.count for 2 columns to get value occurrences under a class constraint. However, if value $x$ in feature never occurs with class $y$, then this pandas method returns only non-zero frequencies. Is there any solution or alternate method?
The script is like:
combine = np.vstack([X_train[:,-1], y_train]).T
combine_df = pd.DataFrame(combine, columns = ['feature','class'])
class_count_groupby = combine_df[combine_df['class'] == 2]['class'].groupby(combine_df['feature']).count()


texts: Machine learning to predict pollution at a single location

texts:
I have 2 months of air pollution measured data for a location of interest. I am highly interested to play with machine learning algorithms to predict pollution at the same location (let say, for the next two months).
Since air pollution is a geographic problem, does it make sense to apply and predict pollution at a single location using machine learning? Could someone shed a light on prons and cons of this?


texts: Approaching a multi-class classification problem but without labels

texts:
I am working on a business problem where I have a movie description dataset. In this dataset I've columns as - Movie title, Movie plot summary, Date of Release. Now based on this information and using machine learning I want to predict which category the movie falls into. For example The Conjuring should fall into Horror and Thriller i.e a multiclass classification problem. Now the problem is I don't have a label column besides the movie description and other info. Now I want my model to predict which categories a movie(unseen to model) should fall into. I have decided 5 labels that I want to consider - Horror, Thriller, Comedy, Romantic and Emotional. So, I want the dataset to look like this -

Conjuring| Description | Title | Horror,Thriller

The notebook| Description| Title | Romantic,Emotional

I believe if I want to proceed this problem as a classification problem then I have to think of some way to create labels to existing dataset by some script and logic. If not supervised then maybe if I can do clustering first and then based on where the data point lies I can do classification later on.

What I have tried ?

Once I decided what my 5 labels should be, I made 50 synonyms for each and then iterated the description of the movies and based on the number of occurrence of words I made frequency and based on majority of the occurrence I decided which category a movie should fall into. Very bad results from this approach.

I used K means clusters from the data and tried to extract information from the clusters. Could not get very meaningful information though.

To be very honest I am pretty clueless and just want a direction how to approach this problem.


texts: How can I build a self-attention model with tf.keras.layers.Attention?

texts:
I have completed an easy many-to-one LSTM model as following.
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense
from tensorflow.keras.layers import LSTM
from tensorflow.keras.layers import Dropout
model=Sequential()
model.add(LSTM(2**LSTM_units,input_length=data.shape[1],input_dim=data.shape[2],return_sequences=True))
model.add(Dropout(dropout))
model.add(Dense(1))
model.fit(data,res)
prediction=model.predict(test)

Where data.shape is (dates, time_interval, factors) and res.shape is (dates,1).
I would like to replace LSTM with Attention.
How can I get the similar result by using tf.keras.layers.Attention?


texts: Timeserieses Xi are noisy, and the noise depends on a timeseries Y. How do I remove that noise, to smooth the timeserieses Xi?

texts: The noises in my timeserieses Xi are highly correlated on the timeseries Y. How would you approach the problem of removing that noise? Ideally I would like all the Xi curves to be smooth, as well as when a new data point is ingested, I would like it to be directly "corrected". How would you approach this problem?


texts: I have been trying to understand reinforcement learning for quite sometime, but somehow I am not able to visualize how to write a program for reinforcement learning to solve a grid world problem. Can you suggest me some text books which would help me build a clear conception of Reinforcement Learning?


texts:
I have a dataset of this form:

chrX posX labelX

where chrX refers to the chromosome number, posX refers to location, and labelX is a categorical variable with 3 labels.
For example:

chr1 3223 1
chr1 3200 2
chr1 3100  1
chr1 1000 2
chr2 1000 1
chr1 3210 3

I want to be cluster these data with certain constraints (thus, this is no longer what 'clustering' typically is), such that, for each chromosome, the sites which are within, say $\pm 30$ positions are together, the constraint being they should have unique labels.

For example, an expected output would be:

chr1 : 3223(1) 3200(2) 3210(3) Notice how sites like chr1 3100 1 are not part of the 'cluster'.

One approach would be to simply calculate pairwise differences between all such sites, and grow the cluster taking care of unique labels.

Are there more elegant approaches to solve such problems?


texts: Python Keras NN, handling n_samples of float outputs

texts:
I am using Keras with Theanos backend in Python.

I have 2117 samples and each sample has an individual target (on purpose) ie. 2117 outputs.

As opposed to categories, the targets are ratings eg. (16.4714494876, 17.4129353234, 17.4476570289) the entirety of the number is important.

I am having problems/don't know where to start.

1) When i run the NN it only outputs the targets as whole integers as opposed to the format of the actual values. eg. 16 instead of 16.xxxxxx

2) Presumably i will only be able to gauge the accuracy of predictions based on how close the output is to the target since there are so many targets, does this type of classification problem have a name that i can research?

3) In 3 research papers i have read that apply NN to my specific classification problem they list the output layer as only having 1 neuron but provide no further explanation, how could this be?

Here is my model.

# fix random seed for reproducibility
seed = 7
np.random.seed(seed)

X = np.array(df[FEATURES].values)
Y = np.array(df["MTPS"].values)

# define baseline model
def baseline_model():
    # create model
    model = Sequential()
    model.add(Dense(10, input_dim=(len(FEATURES)), init='normal', activation='relu'))
    model.add(Dense(2117, init='normal', activation='softmax'))
    # Compile model
    model.compile(loss='sparse_categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
    return model
#build model
estimator = KerasClassifier(build_fn=baseline_model, nb_epoch=100, batch_size=5, verbose=2)

#cross validation
X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=0.2, random_state=seed)
estimator.fit(X_train, Y_train)

#print class predictions
print estimator.predict(X_test)
print Y_test

Thanks for any help.


texts:
What value does actually LSTM forget in a training phase?
for example, I do have a surface temperature data for 10 years. Then I made them as a training data for building my neural network using lSTM algorithm. Then what value or which time step in my data that LSTM forget?