texts:
I have a labelled dataset (0 and 1) with two types of short length time series data (from day1-day5) as follows.

Type1 (Temperature at location1):

sensor1, 38, 38, 35, 33, 32
sensor2, 33, 32, 35, 36, 32
and so on ....

Type2 (Temperature at location2):

sensor1, 18, 18, 12, 11, 09
sensor2, 13, 12, 15, 16, 12
and so on ....

In summary my dataset looks as follows.

ID, time-series1, time-series2, label
sensor1, [38, 38, 35, 33, 32], [18, 18, 12, 11, 09], 0
sensor2, [33, 32, 35, 36, 32], [13, 12, 15, 16, 12], 1
and so on ....

I would like to use my dataset in LSTM (or a equivalent deep learning model).

However, I am still not clear how I can use my dataset in a deep learning model such as LSTM.

I am happy to provide more details if needed.


texts: Applying machine learning algorithms to subset of attributes in dataframe

texts:
I have this huge mixed data set consisting of both numerical and categorical attributes which upon OneHotEncoding results into a data set with very high dimensionality.

Is it wise to apply machine learning algorithms like K-means clustering, dimensionality reduction and regression on subsets of data set? For example applying K-means clustering to numerical columns first and join the result with categorical data set later.


texts: Algorithm to predict the best time to recall a client

texts:
Let's immagine I have a dataset of calls from a call center to clients. Each call has a lot of information like at what time it was made, duration, if it was answered or not, if the client purchased something (the call center is selling stuff), and of course the client that has been called (I have an average of ten calls per client).
The calls are just made by the call center, it's never the client calling.
What would be the best way to do an algorithm that gives as an output the probability that the client buys something AND when to call the client, based on the previous called made to that client (Of course it's impossibile to make it time continuos, so it would be something that after the call says: the client will buy something with probability x%, call the client in "1 hours", or "2 hours", or "3 hours" (putting a limited amount of classes).
Any Ideas? Thanks


texts: Clustering algorithms for high dimensional binary sparse data

texts:
I have a dataset with 10,000 genes like below

person gene1 gene2 ... gene10000  ethnic
  1       0      1         1       asian
  2       1      0         1       European

Each row means, whether a person has a gene in their DNA or not. We are trying to classify different ethnic groups based on the data above. But first we want to use some clustering algorithms to visualize how the cluster would look like for different ethnic groups. We are not going to use this clustering algorithms to classify groups, it will be used just to visualize how it would look like, if we have nice separate clusters or overlapping clusters etc.

Please recommend some clustering algorithms for this type of dataset. Also, the dimension is 10000. Is this going to be a problem for clustering? Should I use some dimensionality reduction algorithms first? If so please give your recommendations. Thanks in advance.


texts:
I've been working through the tensorflow-2.0.0 beta tutorials. In the  advanced example a tensorflow.keras subclass is used. The presence of the @tf.function decorator on train_step and test_step means the model executes in graph mode (not sure if that's the correct terminology, I mean oposite to eager mode). If I remove these decorators I can single step right into the model call function and see the input/output tensor for each layer which is neat.

My question is, is there a programatic way to enable/disable the @tf.function decorators. Commenting them out to switch between eager and graph mode doesn't seem particularly scaleable but it's certainly useful for debugging/learning)


texts:
I have used tsne to visualise a large dataset and it has produced the following graph. I need help interpreting it, as I have never seen a tsne graph like this before!

I am aware that not much overlapping data is a good thing, but I have rarely seen more than a few clusters. Has something gone wrong here? Or are there just a lot of clusters in this dataset.

Thanks. P


texts: Loading a model with attention layer and custom metric

texts:
I have a neural network with SeqSelfAttention (https://pypi.org/project/keras-self-attention/). Also I implemented a custom metric for F1. Then I saved the model without problems, but when the model is loaded using keras.models.load_model(model_path, custom_objects=SeqSelfAttention.get_custom_objects())
the next error is presented:
ValueError: Unknown metric function: f1
If I coded with the F1 metric as:
saved_model = load_model(Modelfilename, custom_objects={"f1": f1})
the error say:
ValueError: Unknown layer: SeqSelfAttention
Are there a way of combine the two parameters in "custom_objects"?


texts: how does XGBoost's exact greedy split finding algorithm determine candidate split values for different feature types?

texts:
Based on the paper by Chen & Guestrin (2016)
"XGBoost: A Scalable Tree Boosting System", XGBoost's "exact split finding algorithm enumerates over all the possible splits on all the features to find the best split" (page 3). Thus, my understanding was that XGBoost enumerates over all features and uses unique values of each  feature as candidate split points to then choose the split value that maximizes the splitting criterion (gain).

Then, my question is why do the split values for float-type features chosen are often not one of the unique values for that feature? For example, for a certain feature in the data with floating point values, like so: 966.0, 1234.0, 2350.0, 4567.0 .... If Xgboost chooses to split on that feature, the split value can be, e.g. (feature < 1800.5), which is not one of the unique values for that feature, but is often the mid-point between two values.

Also, assuming no missing values in the data, for the ordinal type features, with the values like so: set(1, 2, 3, 4, 5, 6, 7, 8), declared as integers, how does XGBoost enumerate candidate split points? Does the algorithm treat that feature as categorical?

I've read a lot of good answers on the forum, like here
and here, have gone thru xgboost documentation and tutorials, but they don't seem to elucidate on the specific procedure of enumerating candidate split points for exact greedy algorithm.

Could someone, perhaps, familiar with the source code could shed light on these questions?


texts: Workaround for word embeddings that do not "see" antonyms

texts: Most word embeddings do not "see" antonyms. For instance, among many words they will place vectors for "dependent" and "independent" (as an example) quite close, - actually as close as with synonyms such as "independent" and "autonomous". So it is easy to identify synonyms as close vectors, but how to identify antonyms, or generally work with antonyms? There are some few rare papers that try to develop embedding algorithms "aware" of antonyms (just web-search word-embedding antonyms). But I am working with standard very powerful and already trained on massive data embedding libraries. Is there a workaround to somehow work with STANDARD embeddings but to make them not "blind" to antonyms? Thanks!


texts: Do we need to divide our gradients by batch size our we will use the sum (Mini batch GSD plus L2 Regularization)?

texts:
I am implementing L2 regularization in C++ and I used mini batch GSD. Without L2, I was using sum of gradients during back propagation and I was not dividing my cost function by batch size. I was using the sum of batch cost and my MNIST model was working fine.

Now, I am adding L2 regularization in my model and would like to know if it is necessary to take the average of the cost function and average of the gradients while updating weights?

Secondaly, my learning rate was constant 0.01/batchSize. Now with regularization, is it necessary to modify learning rate (like exponantial decay)?


texts: Factor Analysis with Mixed Data Concurrent Approach with PCAmixdata in R

texts: I am trying to perform Factor Analysis over Mixed Data using R with PCAmixdata package. My dataset is huge with almost 115000 records and almost 40 features of both categorical and continuous. When I tried to run PCAmixdata, I am getting memory issue that total memory allocation is reached and I am not able to proceed, I wanted to know if it is a right way to split the dataset row-wise like 30000 records at a time and combine the results of PCAmixdata. If not I want to know what are the other possible ways to overcome this issue.


texts: Find shared properties of a cluster samples

texts:
I have a dataset which contains ~15 features. With the elbow method, I found out that the optimal number of clusters is probably four. Therefore, I applied the K-means algorithm with four clusters. Now, I would like to understand why these clusters have been formed the way they are. In other words, I would like to know what are the shared properties of the points of a specific cluster.

My idea is the following:

Let's pretend that C1 are the coordinates of the centroid of the first cluster and that P1 and P2 are two points of this cluster.

$$
   C1 =
   \begin{pmatrix}
    5\\
    2\\
    4\\
    \end{pmatrix}
$$

$$
    P1 =
    \begin{pmatrix}
    8\\
    2\\
    6\\
    \end{pmatrix}
    P2 =
    \begin{pmatrix}
    9\\
    2\\
    0\\
    \end{pmatrix}
$$

If we compute the average distance of the different coordinates of P1 and P2 we obtain this:

$$
    DistAverage =
    \begin{pmatrix}
    ((8-5)+(9-5))/2\\
    ((2-2)+(2-2))/2\\
    ((6-4)+(4-0))/2\\
    \end{pmatrix}
    =
    \begin{pmatrix}
    3.5\\
    0\\
    3\\
    \end{pmatrix}
$$

Would this mean that the second feature is a "shared property" of the points of this cluster (since the average distance is 0) ?

I hope that the question was clear enough.


texts: How to use features with lags of different lengths in LSTM?

texts:
I'm trying to predict a time series, let's say I have 3 features and a target variable. I used the standard approach when feature lags have the same length, for example, 10. Then the size of my batch will be (32, 10, 3). But looking at the autocorrelation graphs, I want to use a lag length of 10 for the first feature, 5 for the second, and 7 for the third. How can I construct a batch in this case?
My architecture looks like this:
inp = Input(shape=(10, 3, batch_size=None)
x = LSTM(128, return_sequences=True)(inp)
x = Flatten()(x)
x = Dropout(0.2)(x)
out = Dense(2, activation='linear')(x)


texts: In text classification, how can I use a neural network on word embeddings?

texts:
I have read some papers on text classification but they are pretty abstract. I fail at understanding how to train a multi layer perceptron with data made of sentences -> label.

A perceptron takes a vector input of size m and outputs a vector.

As a first step I reduced the dimensions and I map each word to a vector of n features, where n is a constant.

Now how do I mix the words in a sentence to make an input of size m ? Each word maps to a vector of size n but then what ? Knowing that every sentence has a different number of words.

Thanks


texts:
I am reading about KNN

So I made another example to make things clearer

In this example (Image attached)

You can see there are in total 5 Greed Circles and 20 Blue Squares

by standard KNN (k=3) , X should be a Blue Square

This is obvious 2 Blue Squares vs 1 Green Circle.

But in weighted KNN things are difference

In this case we have to calculate the weight (Likelihood) for each instance

Each Green Circle likelihood is $\frac{1}{5} $ , we have 5 Green Circles

While for Blue Squares it is  $\frac{1}{20} $ , we have 20 Blue Squares

Therefore the weights around X will be $\frac{1}{5} $  Green Circle, and $\frac{2}{20} $ Blue Squares.

which means $\frac{1}{5} > \frac{2}{20} $

Then X is Green Circle

But if try to think of it logically then there are more Blue Squares than Green Circles which means X more likely to be Blue Square than Green Circle.

My question is :

Am I doing anything wrong here? Can someone explain why the equation is showing Green Circle while logic says Blue Square?


texts: I need sources of interrogative, exclamatory, and imperative sentences

texts:
I am working on accumulating a large database of labeled sentences for several projects/experiments. At present I am only using Wikipedia and Project Gutenberg as sources of data. Between these two sources I expect I will be able to extract millions of sentences. Labeling them may be another matter, but I'm less concerned about that right now.

My chief need is to find a broader variety of sentences. Questions, exclamations, invective, conversation, and so on. My project is several deep learning experiments to look at different types of sentences and structures. It would be helpful if the sentences were labeled.

The ultimate goal of my project is to be able to infer the type, structure, and coherence of a sentence (with separate models). Generating gibberish is pretty easy to do, so I'm not as concerned about finding sources of non-sense sentences.

My current concern is that the vast number of sentences are declarative (statements) rather than interrogatives, exclamatory or imperatives. This bias towards declarative sentences means that my sample sizes will be limited unless I can find sources of questions, exclamations and commands.

Alternatively, I would be open to suggestions about how to generate the data I need.


texts: Why can't my neural network learn how to predict the squares of natural numbers?

texts:
I want my neural network to learn to predict the square $n+1$ number having $n$ number.
I am considering a regression problem. That's what I'm doing:

from keras.preprocessing import sequence
from keras.models import Sequential
from keras.layers import Dense, Embedding, Dropout
import numpy as np

x = np.array([[int(i)] for i in range(1001)])
y = np.array([x*x for x in range(1001)])

model = Sequential()
model.add(Dense(100, activation = 'relu', input_dim = 1))
model.add(Dense(50, activation = 'relu'))
model.add(Dense(10, activation = 'relu'))
model.add(Dense(1))

model.compile(loss='mse',optimizer='adam', metrics=['mae'])
model.fit(x,y,epochs= 2500)
pred = model.predict([1001])
print(pred)

However, as a result, I get [[ 1000166.8125]]  instead 1002001.

Update:

x = np.array([[int(i)] for i in range(80001)])
y = np.array([x*x for x in range(80001)])
print(x)
print(y)
model = Sequential()
model.add(Dense(20, activation = 'relu', input_dim = 1))
model.add(Dense(20, activation = 'relu'))
model.add(Dense(1))

adam = optimizers.Adam(lr=0.0002,beta_1=0.9, beta_2=0.999, epsilon=None, decay=0.0, amsgrad=False,)
model.compile(loss='mse',optimizer=adam, metrics=['mae'])
model.fit(x,y,epochs= 3000)
pred = model.predict([80001])
print(pred)
model.save_weights("test.h5")
model_json = model.to_json()
json_file = open("test.json", "w")
json_file.write(model_json)
json_file.close()

result: [[  4.81360333e+09]]


texts:
I am running a multiple linear regression using backward elimination. Below is the code

import statsmodels.formula.api as sm
X = np.append(arr = np.ones((50, 1)).astype(int), values = X, axis =
1)
X_opt = X[:,[0,1,2,3,4,5]]
regressor_OLS = sm.OLS(endog= y, exog = X_opt).fit()
regressor_OLS.summary()
X_opt = X[:,[0,1,3,4,5]]
regressor_OLS = sm.OLS(endog= y, exog = X_opt).fit()
regressor_OLS.summary()
X_opt = X[:,[0,3,4,5]]
regressor_OLS = sm.OLS(endog= y, exog = X_opt).fit()
regressor_OLS.summary()
X_opt = X[:,[0,3,5]]
regressor_OLS = sm.OLS(endog= y, exog = X_opt).fit()
regressor_OLS.summary()
X_opt = X[:,[0,3]]
regressor_OLS = sm.OLS(endog= y, exog = X_opt).fit()
regressor_OLS.summary()

But when I am predicting using the above regressor_OLS model,

X_new = X[:, 3]
y_pred2 = regressor_OLS.predict(X_new)

I am getting the below error:

    y_pred2 = regressor_OLS.predict(X_new)
    Traceback (most recent call last):

  File "<ipython-input-18-263dee38fc26>", line 1, in <module>
    y_pred2 = regressor_OLS.predict(X_new)

  File "/Users/ritesh.satapathy/anaconda/lib/python3.6/site-packages/statsmodels/base/model.py", line 749, in predict
    return self.model.predict(self.params, exog, *args, **kwargs)

  File "/Users/ritesh.satapathy/anaconda/lib/python3.6/site-packages/statsmodels/regression/linear_model.py", line 359, in predict
    return np.dot(exog, params)

ValueError: shapes (1,50) and (2,) not aligned: 50 (dim 1) != 2 (dim 0)

I tried X_new = X_test[:,3] but still same error.

Traceback (most recent call last):

  File "<ipython-input-19-5020d55a4448>", line 1, in <module>
    y_pred2 = regressor_OLS.predict(X_ne1)

  File "/Users/ritesh.satapathy/anaconda/lib/python3.6/site-packages/statsmodels/base/model.py", line 749, in predict
    return self.model.predict(self.params, exog, *args, **kwargs)

  File "/Users/ritesh.satapathy/anaconda/lib/python3.6/site-packages/statsmodels/regression/linear_model.py", line 359, in predict
    return np.dot(exog, params)

ValueError: shapes (1,10) and (2,) not aligned: 10 (dim 1) != 2 (dim 0)


texts: I am working on binary classification problem, I try to evaluate the performance of some classification algorithms (LR,Decission Tree , Random forest ...). I am using a 10 fold cross-validation technique (to avoid over-fitting) with AUC ROC as scoring function to compare the performance of the algorithms, but I am getting a weird results with Random forest and AdbBoost, I have a perfect AUC_ROC score (i.e. =1) despite the fact that the Recall(TPR) and FPR of this algorithms are different from 1 and 0 respectively .


texts: Can latent factor model work for new users?

texts: I am studying latent factor model for recommendor system. It does matrix factorization(like SVD) on the user-item rating matrix. What I am not sure is, does a trained model work for a new user that is not included in the training dataset?


texts: numpy array quirks: does A.T return a view?

texts:
I am trying to understand the numpy array object, and a little mystified by the following:
A = np.array([[1,2,3],[4,5,6]]) # A is a 2x3 matrix
B = A

A = A.T # A is now the matrix [[1,4],[2,5],[3,6]]
A[0,1]= -1 # A is now the matrix [[1,-1],[2,5],[3,6]]

print(B)

I get for B:
[[1,2,3],[-1,5,6]]

So changing A using the assignment A[0,1]= -1 changed B (which is to be expected since numpy arrays are mutable objects, and python assignments are by reference). But what is mystifying is that A = A.T did not transpose B. Furthermore, if I ask (after the A = A.T statement):
A is B
I get False. What exactly is A.T returning? It seems to return a view of the data, but a copy of the alignment of data into a matrix.


texts:
Dataset
For an E-Commerce platform, I have a lot of products (Ad_Group_ID), for each product, I have the number of people that arrive on the landing page (Clicks) ; and the number of leads = the number of people that will buy the product (Conversions). Example:

Date,       Ad_group_ID,Clicks,Conversions
2017-08-17, product1,   10,    1
2017-08-18, product1,   100,   10
2017-08-17, product2,   5,     0
2017-08-18, product2,   50,    5

Goal
I want to predict the number of leads

Mathematical Model
I imagine this is going to follow a binomiale distribution like:

ML-Design
I wrote a class BinomialePredictor, that will delegate to scipy.binom. The results of the evaluation are acceptable IMHO (it has been wayyyy worse with try with Stochastic Gradient Descent and so on).

Question
Is there a module of scikit-learn that do this kind of "Binomial regression"? I think it's very strange. Or which kind of modeling would be better fit scikit-learn


texts: Significance of Object-Oriented Programming (OOP) in Data Science

texts: Can someone please explain to me the role of Object-Oriented Programming (OOP) and Object-Oriented Design (OOD) in Data Science? I am from a non-computer science background. Do I need to learn these as well to become a Data Scientist? Also, please tell me if I should be learning Python or R for the same.


texts: Best masters degree in data science and engineering in Europe

texts:
I have a BSc in Applied Mathematics and I was thinking about an MSc in Data Science or Data Engineering, but I have no ideia what are the best programs available in Europe.
Any suggestions of good universities with MSc's in this field?


texts: What is an appropriate approach to sampling for probability of default using a classification model?

texts:
If we have a loan book and want to train the data to predict the probability of default, what is an appropriate way to sample the historical data to train the model, given that each account is open over a period of time and not just at a single snapshot?

For example - for predicting defaults of new customers on a loan book, it is easy to select data to train a model at an equivalent point in time - i.e. when each customer is new, we may look at the FICO score, age of customer, home State etc at the point of loan application.

But once a loan has been made and we wish to build a model for likelihood of default given ongoing repayment history, what is the statistically appropriate sample to take?  A random sample within the life to date of the loan picking each record at a single random point in time, choosing all loans at a specified point in time (e.g. at exactly 6 months in from origination) or a sample of dates covering a period?

Or does it require something else - is a classification model actually appropriate for this type of scenario (assuming a binary outcome), or should some sort of time series or survival model be considered?


texts:
I have 2 scripts - A.py and B.py, and both are Flask apps.
A.py renders a web page and acts as my UI taking inputs from user.
B.py is hold the main logic and has a web service API being called by A.py.

Both run as flask app services in localhost from different ports - 5001 and 5002, respectively.

My doubts -

1) Every time I make any changes to my B.py (or A.py), do i need to stop and restart the services?

2) I'm currently running both the services from local machine for proof-of-concept purposes. If I use WSGI (thinking of Bottle), will it be accessible from other machines over intranet?

I'm on Windows 10 (64-bit), Python 3.6 and Flask 1.1.1

Please advise.

Edit 1) Added OS and Python version details.


texts:
How to construct the document-topic matrix using the word-topic and topic-word matrix calculated using Latent Dirichlet Allocation?
I can not seem to find it anywhere, even not from the author of LDA, M.Blei.

Gensim and sklearn just work, but I want to know how to use the two matrices to construct the document topic-matrix (Spark MLLIB LDA only gives me the 2 matrices and not the document-topic matrix).


texts: When to remove outlier in preparing features for machine learning algorithm

texts:
I have a numeric variable (price) and it has a long tail in both training and test data sets. I found that if you remove the highest 1% of the value in both train and test data set for this variable, then the histogram of this variable in train and test data set looks pretty much the same. See the following figure.

My question is: I still need to use the training data (with both features and labels) to make predictions on the test data (with features only). In this case, how should I deal with this feature variable? I was thinking about removing the top 1% data in both training and test data set, but as I still need to make predictions on that 1% test data, so this is not a good idea I guess. In this example, as the empirical distribution of this variable in both training and test data sets look the same before and after removing the "outlier", should we just leave this variable unchanged? Also, in general, how should we handle the outlier before we put the feature into the machine learning algorithm?


texts:
I am at the dimensionality reduction phase of my model.  I have a list of categorical columns and I want to find the correlation between each column and my continuous SalePrice column.  Below is the list of column names:

categorical_columns = ['MSSubClass', 'MSZoning', 'LotShape', 'LandContour', 'LotConfig', 'Neighborhood', 'Condition1',
                       'Condition2', 'BldgType', 'HouseStyle', 'RoofStyle', 'RoofMatl', 'Exterior1st', 'Exterior2nd',
                       'Foundation', 'Heating', 'Electrical', 'Functional', 'GarageType', 'PavedDrive', 'Fence',
                       'MiscFeature', 'SaleType', 'SaleCondition', 'Street', 'CentralAir']

Because its categorical vs continuous, I've read that ANOVA is the best way to go but I have never used it before and couldn't find a concise implementation of it in Python.  I want to loop through and output the correlation between each element in the list and the SalePrice column.


texts:
I'm attempting a rather simple exercise in machine learning and trying to classify samples of text as either plaintext or ciphertext (encrypted).

Here are two samples:

Plaintext: This is a sentence in plaintext which any human person can read
Ciphertext: 5oXbLiEZbMUgOOdYy+q4+rsDaqUngBrrUbpVeuu2ggvP6hHObC4GgTLhq

The specific encryption used doesn't have any special attributes I can use to classify (for example, the ciphertext isn't guaranteed to be significantly longer than plaintext), so the task is all about figuring out what text is indistinguishable from random characters and what text is readable plaintext.

My current heuristic involves counting whitespaces and assuming anything with a whitespace ratio above a certain threshold is plaintext, but I'm trying to find a better robust algorithm.


texts: Fine-tuning a model from an existing checkpoint with TensorFlow-Slim

texts:
I'm trying to retrain the final layer of a pretrained model with a new image dataset using TensorFlow-Slim.

Lets say I want to fine-tuning inception-v3 on flowers dataset. Inception_v3 was trained on ImageNet with 1000 class labels, but the flowers dataset only have 5 classes. Since the dataset is quite small we will only train the new layers.

An example at official tf github page shows how to do it:

$ DATASET_DIR=/tmp/flowers
$ TRAIN_DIR=/tmp/flowers-models/inception_v3
$ CHECKPOINT_PATH=/tmp/my_checkpoints/inception_v3.ckpt
$ python train_image_classifier.py \
    --train_dir=${TRAIN_DIR} \
    --dataset_dir=${DATASET_DIR} \
    --dataset_name=flowers \
    --dataset_split_name=train \
    --model_name=inception_v3 \
    --checkpoint_path=${CHECKPOINT_PATH} \
    --checkpoint_exclude_scopes=InceptionV3/Logits,InceptionV3/AuxLogits/Logits \
    --trainable_scopes=InceptionV3/Logits,InceptionV3/AuxLogits/Logits

I couldn't fully understand all the parameters in the above code:

train_dir = ?

dataset_dir = new dataset directory location

dataset_name = name of the dataset (but why ? )

dataset_split_name = ?

model_name = name of the model on which we want to train

checkpoint_path = path to the model checkpoint

checkpoint_exclude_scopes = ?

trainable_scopes = ?

Help me figure out what these parameter means and correct me if I'm wrong about any of the parameter?

Note: I'm aware that we can retrain inception_v3 with a method mentioned at official tensorflow website but I want to do the same with tensorflow-slim.


texts: What's the best way to plot a bar graph with large numeric difference in values?

texts:
I'm graphing the value of the Ruble against the US Dollar in the 1990s. There was hyper-inflation from 1992-1997 (where the "y" values expanded from 125 to 6000). On Jan 1st 1998, the government revalued the money to 5. Since then the number has trended upward but has only reached 70 or so.

This creates a problem in my bar or line graph. You can't understand the low values after the stabilization. Any suggestions to help with this discrepancy when viewing the whole dataset in one graph?

Plotting using matplotlib


texts: What's the best way to train a NER model?

texts:
I am trying to do a project using NLP. My goal is to process Cyber Threat Intelligence articles like this to extract information such as actor’s name, malwares and tools used…

To do that I want to use NER. However, there isn’t training data available on the web. So I was wondering if I should process manually 10-20 articles to make my training data or if I could do something like taking only interesting lines such as “Rancor conducted at least two rounds of attacks intending to install Derusbi or KHRat malware on victim systems” in multiples articles and replacing the group name by another actor. This way I could deduplicate my training data by the number of known actors. But doing that, only the actor name is changing. So, the context is always the same.

I am wondering what’s the best way to train my model considering the quantity of training data available?


texts: difference between supervised learning and imitation learning

texts:
I find these two concepts confusing because I feel that imitation learning is just a 'subset' of supervised learning. But after thinking hard enough, I could not think of any difference and feel that they are the same.
Are they the same? or different from each other?


texts: Optimizing Decision Model with incomplete testset

texts:
I have a table of 2 features (numbers) on which I define a simple binary classification "model" (i.e. a simple logical expression) which needs 2 parameters thresholds. The model tries to predict "events", i.e. whether an event has happened in the past or not. I also have s set of labels (observations of past events) which are known to be incomplete, meaning that if the label is 1 it has been observed, if its 0 it means we just don't know anything (e.g. event happened, but was not recorded)

feature A
feature B
label

0.1
1.2
1

-0.2
3
0

-0.5
2.0
0

My model would to something like
if (feature_A < tresh_A or feature_B < thesh_B) then 1 else 0

Due to the "sparseness" of the test data, I can only calculate recall, but not precision (as I don't know the true negatives, only true positives). Well I could calculate precision, but false-positives would be way too high because many of them would actually be true positives, but just missing in the obersvations....
Any advice on how to optimize the parameters in such a case? I could easily do a grid-search as it's computationally cheap, but I don't know what to optimize for. Intuitively, I would like to maximise recall and at the same time minimise overall positives positive prediction.


texts: Hypothesis about Telecom Data Analysis - a Q from new learner

texts:
in one of my online courses, I'm supposed to analyze and plot any problem of my choice, cool right!
Accordingly, I have the data from one Internet Service Provider (ISP) which is related to the Hotspot service in  Cafe Shops, this data showed the vouchers' sales and usage per each Service Site (i.e. Cafe shop).
Data variables are described below:
voucher_code, Site_ID, Start_Accounting_dateTime, End_Accounting_dateTime, Usage_Duration, Usage_Quota_MB.
My questions are,

I thought that by showing the correlation between the Total_Solled_Vouchers per month in a certain Service Site and the Avreage_Usage_Time of the internet in the same site per month, will demonstrate the Quality of other services provided in that certain Service Site (i.e. Cafe Shop). Is this hypothesis correct?

For such a dataset, what kind of analytics or analysis can be performed on it?

would highly welcome and appreciate your advice, comments, or opinions.
BR.


texts: Correcting ALL CAPS for human and algorithmic consumption

texts:
United States federal tax returns tend to be written in ALL CAPS to facilitate OCR. This practice has persisted even when returns are filed electronically. Thus, much of the text in the IRS 990 dataset is in all caps. This makes it hard to read, and limits the ability of algorithms such as Treebank to accurately tag part of speech.

I understand that the approach of the Stanford POS tagger may be more amenable to correction of capitalization, but in practice, I have not had much luck in using it to correct the text in the IRS 990 corpus, in which nearly every sentence contains one or more proper nouns.

Are there any "tricks of the trade" for improving the performance of an off-the-shelf POS tagger when using ALL CAPS text, and/or an algorithm that may do better at identifying the proper nouns therein?


texts:
I wrote in google-collab to get the model using keras, but I have to do predictions in Visual Studio using tensorflow

I've search for a method converting models from keras .h5 to tensorflow .pb, but none of them worked as it should.


texts: CNNs - Hyperparameter tuning with different training sizes of the same data set

texts:
I would like to compare how much the classification performance (test accuracy) of CNNs changes depending on the size of the data set.
For this I would like to use a data set like MNIST or Fashion MNIST.

I would like to start by first training a CNN with only a subset ob about 1000 images, then a CNN with a subset ob about 5000 images, then with a subset of about 10000 images etc.

For each ot these data sets an optimized CNN should be generated.

My current plan would be the following:

Create a simple base model with only 1 Conv Layer (with 32 filter), MaxPooling2D, Dropout (0.5) and a Dense Layer with about 100 Neurons.
Using this base model, I would search for the smallest dataset (with 1000 images) using GridSearchCV for batch size and optimizer und reuse both for all other datasets.
Then I would do the following for each data set:

Search for the optimal architecture of the network (Number of Conv
layer, Number of filter, Number of dense layer, Number of Neurons in Dense layer) using GridSearchCV (RandomizedSearchCV will propably be less suitable at this point)
Search for learning rate, weight decay, dropout rate etc. (using GridSearch or RandomizedSearch).

Maybe I could just use the default setting for one or the other not so important parameter and  exclude them from the search.

I am not sure if my approach is really recommendable.

I also tend to use RandomizedSearchCV to determine all hyperparameters simultaneously for each data set.

Many thanks in advance for your comments and suggestions.


texts: Both train and test scores high. Could my model be overfitting?

texts:
all. Currently I am working in a regressor to predict the median house value using 1990 census as a dataset (the same on Aurélien Géron used on his book).

My best result so far has been a score of ~0.96 for the training set and ~0.80 and for the test set with a bagging regressor. Here's how I had been doing:

precisao_treino = clf.score(dados_treino_feat,dados_treino_targ) #training score
precisao_valid = clf.score(dados_valid_feat,dados_valid_targ) #test score

(0.9698880465469455, 0.8087685894971024)

But here is where I get confused. That very high training score indicate that there is a potential overfit on my model. However, the test score is maintained a relative good score. So I don't know if my model is overfitting or if doing well at predicting. Could someone explain to me whether this a bad or good model?

Here's how my final dataset looks like, with one-hot-encoding:

['longitude',
 'latitude',
 'housing_median_age',
 'total_rooms',
 'total_bedrooms',
 'population',
 'households',
 'median_income',
 'median_house_value',
 '<1H OCEAN',
 'INLAND',
 'ISLAND',
 'NEAR BAY',
 'NEAR OCEAN']

My solution to removing outliers:

df = df[(np.abs(zscore(df)) < 6).all(axis=1)]

My model:

clf = ensemble.BaggingRegressor(n_estimators=20)


texts: Why multi-class SVM can't detect some classes?

texts:
I use PCA and multi-class SVM for classifying 4-class problem in the Python environment. But in results, I see some differences in detection rate (Unweighted Accuracy in this problem). For example:

FV  | cl.1  % |   cl.2  %  |   cl.3  % |    cl.4 % |

fv1  | 5.4 % | 47.4 % |  2.6 % |   78.6 % |

fv2 | 2.6 % |  41.6 % |  8.3 % |   78.4 % |

fv3 | 5.9 % |  49.3 % |  1.8 % |   76.5 % |

you can see that SVM couldn't detect classes 1 and 3 as well as classes 2 and 4. I would be appreciated if anybody helps me to solve this issue.


texts: Why isn't Maxout used in the state of the art models?

texts:
I have just read the paper from Ian Goodfellow et al. titled "Maxout Networks".

It seems that the Maxout activation should be quite powerful, as it can approximate any convex function, i.e. Relu, function that is used in many state-of-the-art models. Also, Maxout, in theory, should work better with dropout than other activation functions.

I am therefore wondering, what is the intuition for using Relu instead of Maxout.


texts:
The process of finding entity in a knowledge base (KB) that a given keyphrase in a text refers to is called entity linking. I have the
opposite problem. I have an entity in my knowledge base (KB) and I want to find all the ways people might refer to this entity. For instance, I have "Madonna" (singer) and I am looking for aliases like "Louise Ciccone", "Madonna Ritchie", "Queen of Pop", "Mo", etc.
Is it called aliasing? Or there is a better name in the literature?
I guess finding the right key words will help me find related research.


texts: ValueError: Error when checking input: expected embedding_1_input to have shape (4,) but got array with shape (1,) numpy related

texts:
I am trying to predict the pretrained model of keras Machine translation

but i get the error and i could not solve or find suitable answer

and it seems everyone gets this error

code

model = load_model('model.h5')
single_x_test = ['how are you']
model.predict(np.array(single_x_test, ndmin=0))

  ValueError: Error when checking input: expected embedding_1_input to
  have shape (4,) but got array with shape (1,)


texts: What's the difference between multi label classification and fuzzy classification?

texts:
Is it just the between academics and practitioners in term usage?

Or is theoretical difference of how we consider each sample: as belonging to multiple classes at once or to one fuzzy class?

Or this distinction has some practical meaning of how we build model for classification?


texts: Use a single Gaussian to represent a mixture of Gaussians

texts:
I want to merge a Gaussian mixture, $\sum_{i=1}^{K} w_i \exp(x; \mu_{i}, \Sigma_{i})$ into one single Gaussian, under the constraints that $w_1 >> w_2 \geq \dots \geq w_K$, i.e. we have a dominant Gaussian. Intuitively, the resulting Gaussian will be near the first Gaussian. I want to use a MLE estimate, seeking the optimal Gaussian $\exp(x; \mu, \Sigma)$. Obviously, formula for the expression of the mean is $$\mu = \sum_{i} w_i \mu_{i}$$
What about the covariance? Can any one help me with this problem?


texts:
I want to classify 13 classes, but 2 classes is not classified.
This is the test result.
As you can see in the image, class 3 and class 11 is not detected.
It's simple vector to class classification.
The dimension of input is 16 and I've added 2 dense layers in the model.
What do I need to add to model?


texts: Same validation accuracy, different train accuracy for two neural networks models

texts:
I'm performing emotion classification over FER2013 dataset. I'm trying to measure different models performance, and when I checked ImageDataGenerator with a model I had already used I came up with the following situation:

Model without data augmentation got:

train_accuracy = 0.76
val_accuracy = 0.70

Model with data augmentation got:

train_accuracy = 0.86
val_accuracy = 0.70

As you can see, validation accuracy is the same in both models, but train accuracy is significantly different. In this case:

Should I go with the model which uses data augmentation, as it's
train accuracy is higher?
Should I expect overfitting from it, and choose the model without
data augmentation as it's accuracy values are closer?
Third option. Should I perform more comprobations? If so, which ones?

Thanks for your time.


texts:
I am trying to import a image dataset in Google Colab in order to do image classification. The dataset is in my Google drive, so I do the following:

from google.colab import drive
drive.mount('/content/drive')

now, i want to import my images and divide them into trainig set and test set, so I do the following:

    trainingset = '/content/drive/My Drive/Colab Notebooks/Train'
testset = '/content/drive/My Drive/Colab Notebooks/Test'

batch_size = 32
train_datagen = ImageDataGenerator(
    rescale = 1. / 255,\
    zoom_range=0.1,\
    rotation_range=10,\
    width_shift_range=0.1,\
    height_shift_range=0.1,\
    horizontal_flip=True,\
    vertical_flip=False)

train_generator = train_datagen.flow_from_directory(
    directory=trainingset,
    target_size=(118, 224),
    color_mode="rgb",
    batch_size=batch_size,
    class_mode="categorical",
    shuffle=True
)

test_datagen = ImageDataGenerator(
    rescale = 1. / 255)

test_generator = test_datagen.flow_from_directory(
    directory=testset,
    target_size=(118, 224),
    color_mode="rgb",
    batch_size=batch_size,
    class_mode="categorical",
    shuffle=False
)

num_samples = train_generator.n
num_classes = train_generator.num_classes
input_shape = train_generator.image_shape

classnames = [k for k,v in train_generator.class_indices.items()]

print("Image input %s" %str(input_shape))
print("Classes: %r" %classnames)

    print('Loaded %d training samples from %d classes.' %
   (num_samples,num_classes))
     print('Loaded %d test samples from %d classes.' %
    (test_generator.n,test_generator.num_classes))

but as output I get:

Found 0 images belonging to 0 classes.
Found 0 images belonging to 4 classes.
Image input (118, 224, 3)
Classes: []
Loaded 0 training samples from 0 classes.
Loaded 0 test samples from 4 classes.

so, it seems the dataset has not been uploaded. I think the problem is that the dataset is composed by zip files, in particular I have a dataset composed by moder zip files, and each of these contains images. The images are of four classes: Haze, Sunny, Snowy, Rainy. So are images of these four weathers.

To be more clear I can show the dyrectory:

I am really having troubles understanding how to operate. Can somebody please help me?

Thanks in advance.


texts: How to control false positives in sequential A/B testing while keeping a low sample size?

texts:
I am working on a planned sequence of n independent A/B tests(=run a maximum of n tests or stop earlier if a good improvement is found) and in order to keep the significance level within an acceptable level(=0.05) I'm considering ways of controlling the false positives rate while keeping the sample size as low as I can(it's for a low traffic website)

I'm aware of the Bonferroni, Benjiamimi-Hochberg and related methods designed to control the proportion of false positives in the multiple comparisons situation. However the problem of computing the sample size still remains, the simplest approach seems the one of computing the required sample size starting from alpha = 0.05/n as dictated by the Bonferroni correction and then perhaps use Benjiamimi-Hochberg for the actual testing. Is there a better way of computing the sample size when Benjiamimi-Hochberg is used?

Further as the A/B tests are designed to test possible improvements I was thinking that if the sequential tests are based on independent samples then then probability of two consecutive false positives would be alpha^2 = 0.025. My understanding is that the sequential tests would be independent if the respective samples do not refer to the same users. These could for example be users that joined the website after the previous test.

If the above thinking makes some sense, I could use a "weak" Bonferroni (i.e. alpha' = alpha/(n*0.2) )correction in order to run tests with a reasonable power and lower sample size and once a positive is found(the null hypothesis is rejected), I could repeat the test and if it's positive again then the result is positive with p < 0.025 which means that I would accept the result. This approach would allow for some control over the sample size but... is it sound?

Any opinion would be greatly appreciated. Thanks in advance. AMAR


texts: How to load pre-processed .txt data and use it for training convolutional networks?


texts: How to define own model using Tensorflow object detection API

texts: I have 2000+ training images(of custom farm structures) and I want to define my own model using Tensorflow. I researched (https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/defining_your_own_model.md). This is a very high-level documentation. Can anyone help me on how to go about doing this?


texts: LSTM Multivariate time series forecasting with multiple inputs for each time step

texts:
I want to predict an output variable for the next day, for each of the users in my dataset. I was thinking of using LSTMs for achieving this.

The dataset

The dataset I am using has multiple inputs for each time step and it is dependent on the value from one of the inputs.

Each user_id has multiple features for each day and gives an output in the range of [1, 10]. Not all users have data for each day.

-----------------------------------------------------------------
|     day    |   user_id   | feature_1 |   feature_n  |  output |
-----------------------------------------------------------------
| 2020-02-11 |      1      |     20    |       9      |    8    |
| 2020-02-11 |      2      |     15    |       8      |    4    |
| 2020-02-12 |      1      |     10    |       2      |    6    |
| 2020-02-13 |      2      |     16    |       9      |    5    |
| 2020-02-13 |      3      |     19    |       1      |    7    |
| 2020-02-13 |      1      |     14    |       4      |    9    |

I want to forecast the output for each of the users for the next day.

I was thinking to split the dataset in multiple datasets for each of the users and use only the data for the user to predict the output, the problem is that my dataset isn't big.

Is there any way to use the whole dataset and after training the model to get as output the output value for each of the users for the next day?


texts: Data normalization in nonstationary data classification with Learn++.NSE based on MLP

texts:
I need to predict technical aggregate condition using vibration monitoring data. We consider this data to be nonstationary i.e. distribution parameters and descriptive statistics are not constant. I found that one of the best algorithms for such tasks in Learn++.NSE and we us it with MLP as a base classifier.

As I know, it's necessary no normalize data for operations with ANN. We decided to normalize using mean, stdev and sigmoidal function. We train networks of ensemble with sets with different values distribution parameters.

So, my questions are the following

How to normalize new trainig set during previous networks evaluation? Problem is in description statistics change
How to normalize input data while ensemble usage? Current statistics differ from the previous ones.


texts:
I am implementing Isolation Forest from scratch, I am in dilemma of how to set contamination value, and what if not used im the code. Is accuracy differ? What other parameters required to run correctly.
I am using only 5 fold cross validation, then drawing forest, then computing tye amomaly scores and if <=0.5 then Normal otherwise Anomaly. Is it correct?


texts: how does model.score(X_test,y_test)'s behave?

texts:
Related to model.score(X_test,y_test):

Why do we calculate score means r2score on X_test and y_test?
Should it be between $(y_{pred}, y_{test})$!


texts: Machine learning classification with time-domain signals how to ignore signal arrival time?

texts:
I am training a Tensorflow classifier model with signal data (converting signals to the spectrograms).
I want the model to be insensitive to the arrival time of the signal within the fixed window.
Moreover the model should classify both of these signals as the same.

Ideas

Train the model using the same signal at different arrival times
Preprocess the signal before providing it to the model

I want to avoid 1. because it will mean I have to generate a huge amount of signal data which will slow down training.
Are there any commonly applied preprocessing steps for this problem?


texts: How Calculate Effect (percentage) label of the input variables on the output variable by BernoulliNB

texts:
a description problem below. I have 10 words like X1 , X2 , X3 , ... , X10
and three Label like short , long , hold.
My problem is that how calculate Effect (percentage) label of the input variables
by BernoulliNB Algorithm.
NB = BernoulliNB()
NB.fit(X_train, y_train)

and how calculate Effect (percentage) label of the input variables


texts: How to create domain rules from raw unstructured text using NLP and deep learning?

texts:
How to create domain rules from raw unstructured text using NLP and deep learning techniques ?
For example for the below text on symptoms of Dengue, all three look pretty similar but if you want to make sure a person is having Dengue you want to exact definite common minimum rules out of these raw text , so as to confirm a person is having Dengue. Can someone give reference to some research or blogs where similar problem has been solved ?

1) Symptoms of dengue fever include severe joint and muscle pain, swollen lymph nodes, headache, fever, exhaustion, and rash.

2) High fever and at least two of the following:
Severe headache
Severe eye pain (behind eyes)
Joint pain
Muscle and/or bone pain
Rash
Mild bleeding manifestation (e.g., nose or gum bleed, petechiae, or easy bruising)
Low white cell count

3) Aching muscles and joints
Body rash that can disappear and then reappear
high fever
intense headache
pain behind the eyes
vomiting and feeling nauseous

For the above three extracts commons rule would look like

1) Fever
2) Joint and muscle pain
3) Headache
4) Rash


texts: Where to get the Datascience Use cases for practice

texts: I just started  learning data science. I have gone through some of the courses in coursera & udemy, now i want to practice what i have learned. What i want to know is from where can i get the Use cases (linear regression & multiple linear regression) so that i could practice


texts: I am confused about when to use which filter methods for feature selection. I tried to learn them through online resources and found methods like chi-square, variance threshold, F-test, Mutual information, etc. As I read through the methods, I got confused with which one I should use. How do I know if the method really selected the most relevant features and did not drop any features that were, in fact, contributing to the cause of the output?


texts: Relationship between log-odds and weighted sums in Logistic Regression

texts:
I've read several articles/tutorials on Logistic Regression and I've come across this idea of log-odds being equal to the weighted sum of features.

i.e. if $p$ is the probability of a sample belonging to positive class(target variable: 1), $(1-p)$ is the probability of it belonging to the negative(target variable: 0). Then, for input features $x_1$, $x_2$, $x_3$ ... $x_n$ and weights $\theta_0, \theta_1, \theta_2... \theta_n$, they write

\begin{equation}
log\bigg(\frac{p}{1-p}\bigg) = x_0 \theta_0 + x_1 \theta_1 + x_2 \theta_2 \dots x_n \theta_n
\end{equation}

where $x_0$ is bias

Now I get what features are, I get what weighted sum of feature is, I know about the odds, I know log, what I cannot comprehend is, how are those two quantities equal? I've gone through plenty of online articles but they always say that these two are equal, but they don't explain how.


texts: Difference between zero-padding and character-padding in Recurrent Neural Networks

texts:
For RNN's to work efficiently we vectorize the problem which results in an input matrix of shape
    (m, max_seq_len)

where m is the number of examples, e.g. sentences, and max_seq_len is the maximum length that a sentence can have. Some examples have a smaller lengths than this max_seq_len. A solution is to pad these sentences.
One method to pad the sentences is called "zero-padding". This means that each sequence is padded by zeros. For example, given a vocabulary where each word is related to some index number, we can represent a sentence with length 4,
    "I am very confused"

by
    [23, 455, 234, 90]

Padding it to achieve a max_seq_len=7, we obtain a sentence represented by:
    [23, 455, 234, 90, 0, 0, 0]

The index 0 is not part of the vocabulary.
Another method to pad is to add a padding character, e.g. "<<pad>>", in our sentence:
    "I am very confused <<pad>>> <<pad>> <<pad>>"

to achieve the max_seq_len=7. We also add "<<pad>>" in our vocabulary. Let's say it's index is 1000. Then the sentence is represented by
    [23, 455, 234, 90, 1000, 1000, 1000]

I have seen both methods used, but why is one used over the other? Are there any advantages or disadvantages comparing zero-padding with character-padding?


texts:
I am interested in accessing NLP models mentioned in scientific papers, to replicate some results and experiment.
But I only see waiting lists https://openai.com/blog/openai-api/ and licenses granted in large commercial deals https://www.theverge.com/2020/9/22/21451283/microsoft-openai-gpt-3-exclusive-license-ai-language-research .
How can a researcher not affiliated to a university or (large) tech company obtain access so to replicate experiments of scientific papers ?
Which alternatives would you suggest to leverage on pre-trained data sets ?


texts: How does SelectKBest order the best features?

texts:
I'm trying to run a quick univariate filtering on some data, using a t-test of independence, since my target is binary. However, when I run the filter using sklearn's SelectKBest, I get the same top features returned doing  a manual filter, but in different order. The only information about SelectKBest I could find is here and the documentation, but both seem like they should work like my manual method. My code is

import numpy as np
from sklearn.feature_selection import SelectKBest
from scipy.stats import ttest_ind

np.random.seed(0)

data = np.random.random((100,50))
target = np.random.randint(2, size = 100).reshape((100,1))

X = data
y = target.ravel()

k = 10
p_values = []
for i in range(data.shape[1]):

    t, p = ttest_ind(data[:,i], target)
    p_values.append([i,p[0],t[0]])

p_values = sorted(p_values, key = lambda x: x[1])
p_values = p_values[:k]

# Indices of the ranked p-values
idx = [i[0] for i in p_values]

# SelectKBest features
mdl = SelectKBest(ttest_ind, k = k)

X_new = mdl.fit_transform(X, y)
# Manually selected k best features
X_new2=X[:,idx]

# Print first row of sklearn features
print(X_new[0])
array([0.4236548 , 0.96366276, 0.38344152, 0.87001215, 0.63992102,
   0.52184832, 0.41466194, 0.06022547, 0.67063787, 0.31542835])

# Print first row of manually selected features
print(X_new2[0])
array([0.67063787, 0.4236548 , 0.31542835, 0.87001215, 0.38344152,
   0.63992102, 0.06022547, 0.52184832, 0.41466194, 0.96366276])

It seems SelectKBest is not ordering the features based solely on their p-values or their t-values. How does SelectKBest order the features then?


texts: Trying to increase the F1 score for an NLP data extraction problem

texts:
We have a problem reaching a decent F1 score when tackling this NLP data extraction problem.
When given a group (i.e. Female dogs, male dogs) we want to extract relevant numerical data from a paragraph:

Female dogs often have 5 big spots, whereas male dogs have 2 small
spots. Both dogs experience discrimination due to their spot count.
However, while female dogs are discriminated against 60% of the time,
they often receive 30% less harsh treatment, than male dogs. Male dogs
are discriminated against only 20% of the time but will receive harsh
treatment 70% of the time.

The female dogs grouping would extract:

5 big spots
Discriminated against 60%
30% less harsh treatment

Male dogs:

2 small spots
Discriminate against ___ 20%
Harsh treatment 70%

What we've tried:
Our current architecture embeds and vectorizes the words using BERT, while concurrently attaches POS and dependency tags as additional features for each token. We also attach the group words as additional tokens in front and at the end of the paragraph. The sequence of paragraph tokens is then fed through a bidirectional LSTM network, each token has a score (1 if it is relevant to the group, 0 if it is not relevant to the group) and we train the network to predict the relevant words in the paragraph. The training set is broken into 5 subsets for ensemble learning.
We have tried just the BERT embeddings without the POS and dependency tags and we lose ~2% in F1 score.
We score correctly positive predictions at a 1:3 negative weight (our loss function increases the loss by three-fold on positive tokens to punish missing positive tokens, while this inflates false positives it greatly improved recall for positive tokens at the correct threshold, with an overall boost to F1) to best accommodate the imbalance of negative tokens and paragraphs with no positive tokens (data is imbalanced and sparse).
We have approximately 2000 rows of data that we use (1000 rows of positive data and 1000 rows of negative data, at 95% negative tokens and 5% positive tokens), and 3000 rows of negative data we chose to exclude from the training We exclude a random 3000 rows each training iteration to swap the random samples to expose the models to as many negative cases as possible.
Our current F1 score is 45% at the optimal threshold.
Any suggestions would help!


texts: What is the point of using MissingIndicator in Scikit-learn?

texts:
I have recently discovered Sklearn's MissingIndicator but still wondering how could it improve the usual machine learning work. Clear that

from sklearn.impute import MissingIndicator
indicator = MissingIndicator()
X_mask = indicator.fit_transform(X)

could result in  X_mask as boolean indicator array of missing values in original X. But how could it be utilized actually beyond that it keeps track where imputed values are? I know that flagging NAs in a separate column per features could be useful for learners but here X_mask  is not combined with the imputed X so this is seemingly not the point.

Similar question arises with putting MissingIndicator in the ML pipeline:

transformer = FeatureUnion(
     transformer_list=[
         ('features', SimpleImputer(strategy='mean')),
         ('indicators', MissingIndicator())])

What is the point to add the pure mask array to the imputed array here?


texts: Bagging vs Boosting, Bias vs Variance, Depth of trees

texts:
I understand the main principle of bagging and boosting for classification and regression trees. My doubts are about the optimization of the hyperparameters, especially the depth of the trees

First question:  why we are supposed to use weak learners for boosting (high bias) whereas we have to use deep trees for bagging (high variance) ?
 - Honestly, I'm not sure about the second one, just heard it once and never seen any documentation about it.

Second question : why and how can it happen that we get better results in the grid searches for gradient boosting with deeper trees than weak learners (and similarly with weak learners than deeper trees in random forest)?


texts: Classify sentences containing typos into groups

texts:
My data is a list of sentences, where each sentence contains between 1 and 4 words. These sentences are typed in manually so some of them contain typos and some additional words such as GmbH, GER etc.

However, I do know the set of valid sentences. As an example we assume this valid set is given by {Hello human, Horse, Hello bird} and the data (where some sentences contain typos and extra words) is given by

Hello human
Horse
Hello human GmbH
Hello human GmbH, GER
Horse GmbH
Horse
Hello humn
Hell humn
Hello human
Hello bird

I would like to give each sentence above an ID 1, 2 or 3 where 1 is for Hello world, 2 is Horse and 3 is Hello bird. But due to the typos and extra words such as GmbH, GER I cannot make a simple comparison between strings.

Is there a numerical technique within NLP or a related field that I can use to achieve this task?


texts:
blue: training loss
black: validation loss

If modelling gives me such plot, what does it imply? Is it overfitting?
How should I make it better?

Code as below:

classifier = Sequential()
classifier.add(LSTM(
    input_dim=72,
    output_dim=50,
    return_sequences=True))
classifier.add(Dropout(0.20))

classifier.add(LSTM(
    100,
    return_sequences=False))
classifier.add(Dropout(0.20))

classifier.add(Dense(
    output_dim=1))
classifier.add(Activation('relu'))
classifier.compile(loss='mse', optimizer='rmsprop')

X_train = np.reshape(X_train, (X_train.shape[0], 1, X_train.shape[1]))
X_test = np.reshape(X_test, (X_test.shape[0], 1, X_test.shape[1]))

#classifier.fit(X_train, y_train, batch_size = 10, nb_epoch = 10)
cls_return = classifier.fit(X_train, y_train, batch_size=300, nb_epoch=500, validation_split=0.05)
history = cls_return.history
plt.plot(history['loss'], label='Training Loss', color='blue')
plt.plot(history['val_loss'], label='Validation Loss', color='black')
plt.show()


texts: Seeking Appropriate Clustering Algorithm

texts:
I'm analyzing the GDELT dataset and I want to determine thematic clusters. Simplifying considerably, GDELT parses news articles and extracts events. As part of that, it recognizes, let's say, 250 "themes" and tags each "event" it records in a column a semi-colon separated list of all themes identified in the article.

With that preamble, I've extracted, for 2016, a list of approximately 350,000 semi-colon separated theme lists, such as these two:

TAX_FNCACT;TAX_FNCACT_QUEEN;CRISISLEX_T11_UPDATESSYMPATHY;CRISISLEX_CRISISLEXREC;MILITARY;TAX_MILITARY_TITLE;TAX_MILITARY_TITLE_SOLDIER;TAX_FNCACT_SOLDIER;USPEC_POLITICS_GENERAL1;WB_1458_HEALTH_PROMOTION_AND_DISEASE_PREVENTION;WB_1462_WATER_SANITATION_AND_HYGIENE;WB_635_PUBLIC_HEALTH;WB_621_HEALTH_NUTRITION_AND_POPULATION;MARITIME_INCIDENT;MARITIME;MANMADE_DISASTER_IMPLIED;
CRISISLEX_CRISISLEXREC;EDUCATION;SOC_POINTSOFINTEREST;SOC_POINTSOFINTEREST_COLLEGE;TAX_FNCACT;TAX_FNCACT_MAN;TAX_ECON_PRICE;SOC_POINTSOFINTEREST_UNIVERSITY;TAX_FNCACT_JUDGES;TAX_FNCACT_CHILD;LEGISLATION;EPU_POLICY;EPU_POLICY_LAW;TAX_FNCACT_CHILDREN;WB_470_EDUCATION;

As you can see, both of these lists both contain "TAX_FNACT" and "CRISISLEX_CRISISLEXREC". Thus, "TAX_FNACT;CRISISLEX_CRISISLEXREC" is a 2-item cluster. A better understanding of GDELT informs us that it isn't a particularly useful cluster, but it is one nevertheless.

What I'd like to do, ideally, is compose a dictionary of lists. The key for the dictionary is the number of items in the cluster and value is a list of tuples of all theme clusters with that "key" number of elements paired with the number of times that cluster appeared. This ideal algorithm would run until it identified the largest cluster.

Does an algorithm already exist that I can use for this purpose and if so, what is it named? If I had to guess, I would imagine we've created something to extract x-item clusters and then I would just loop from 2->? until I don't get any results.


texts:
According to the PyTorch documentation for the Binary Cross Entropy Loss, we can write it as follows:
$$l_{n} = -w_{n}\cdot \left[y_{n}\cdot \log \left(x_{n}\right) + \left( 1-y_{n}\right)\cdot \log\left( 1-x_{n}\right) \right].$$
What do the $w_{n}$, $x_{n}$ and $y_{n}$ mean?


texts:
My problem is turning a string that looks like this.

"a OR (b AND c)"
into
  a OR bc

if the expression is like

"a AND (b OR c)"

 then gives

  ab OR ac

I can't able to design a correct set of loops using REGEX matching. The crux of the issue is that the code has to be completely general because I cannot assume how long the string pattern will be , nor exact places of OR AND in pattern will be.

If i input like below ,it will solve this type of expression also.

security OR ((internet OR online OR paperless) AND (bank*)) AND (mobile OR cell OR phone OR access) OR easy OR online WITHIN bank OR transaction OR mumbai OR delhi NEAR/10 agar OR (online OR internet) AND (bank) OR not OR (apple) EXCLUDE (mongo)


texts:
I'm trying to improve an NER Bert sequence tagger using LSTM layers in TensorFlow. I'm a bit unclear on the interface and how a LSTM layer should be set up.

Currently, I'm taking in 3-5 sentences and outputting a layer of shape
(?, 5, max_token_length,token_vector_output).
I'd like to take into consideration previous sentence output for the next sequence I'm tagging.

Reshaping the above to (?, 5, max_token_length*token_vector_output) we can then apply a tf.keras.layers.LSTM layer, but this will yield a tensor of shape (?, unit_size) This appears to kill all sequence data off. Is this the correct way to feed my data to a LSTM layer?

As an update, I think I have found a solution - let me know if this isn't a correct way to structure the network.

The LSTM will output a tensor (?, unit_size) regularly - but this is the current state of the LSTM layer. Using the return_sequences=True argument, it will instead return a tensor of shape (?,sequence_length, unit_size), which is the history of the internal states.

In my case, I will be structuring the input as (?, 5 * max_token_length, token_vector_output) to the LSTM layer, returning a (?, 5 * max_token_length, unit_size) tensor. Taking the last max_token_length on axis 1, I'll get the states during the sentence I'm tagging. My logits will a weighted sum of this state tensor and the regular sequential Bert Output of that sentence, and applying a softmax will give the final labels!


texts:
I need to do a cluster analysis for the following variables:

Trickquestion answer: Good/Wrong
count variable : range 0-9
time in minutes
count variable
Number of observations: 3300

Since I am new to cluster algorithms I'm struggling with choosing the best cluster algorithm. I have read about the following methods:

k prototypes
k means with Gower's distance
PAM algorithm.

For the cluster analysis I need to use R.

Can someone give advice about which methods suits the data best. Since I'm studying mathematics I need to give a full mathematical explanation, so a blackbox algorithm is not an option. Tips were I can find Mathematical information about the algorithms are also welcome.


texts: Should I use a cleaned labeled data for sentiment analysis?

texts: I wanted to work on twitter sentiment analysis.so before that I decided to collect some twitter data and label them on my own (pos,neg,neu). My doubt is that Should I clean the data before I label (i.e., removing RT,#,https,@ symbols) Or I can label them with out cleaning the Data? Does cleaning of data before labeling make any difference?


texts:
Context:
I am working on a classification project. where I recommend items to customers based on their past purchase history.

Question:
How will "time leakage" affect training?

Example:
Let's say that I am trying to predict today's purchases for some customer. I train only on the previous history, without any knowledge of what will be purchased today. My features consist of a set of binary variables that for any given day in history could be thought of as either recommending (1) or not recommending (0) an item (essentially acting as "dumb" classifiers themselves). For example, one of these features could be whether or not an item is in the customer's top 10 list, etc.

In this case, is it acceptable to compute top 10 over all of history and based on this list create a feature that is applied to each day in history? Of course, this will cause the features from the first day in history to have knowledge of the future.

Or would having a sliding window where feature creation that can only look into the past be more appropriate?


texts: Multiple Regression, Classification and Boundary Poins

texts:
I have two gangs which are doing crimes. And i want to classify them.
Lets say I'm looking for a regression function:
 M(x1, x2) = w1x1 + w2x2 + w3

Now I have found all three parameters w1, w2, w3.
Now I want to do classification. I get some boundary points which look like a line and they separate two classes from each other. Should I do another regression over that boundary points so that i have a exact line for my separation?
Because lets say i want Point(5,3). I want to know if its more likely that the crime is done from Gang A or B. But I have just some boundary points to separate. Should I use them for a regression?


texts: Are there any interesting application of linear regression

texts:
Linear regression is a widely used ML algorithm. So far I have only encountered 'boring' applications of it. (e.g predict sales for next quarter, predict housing prices for next year , predict population of a country by 2020 etc.)

What are some interesting/cool application of linear regression? Finding cool applications greatly helps in motivating myself to learn, hence the quest.

Cool as in: Application to stock trading, application to video games, application to astronomy, application to sports betting, application to airfare prediction


texts: How would you describe the trade-off between model interpretability and model prediction power in layman's terms?

texts:
I know it depends on the data and question asked but imagine a scenario that for a given dataset you could either go for a fairly complex nonlinear model (hard to interpret though) giving you a better prediction power perhaps because the model may see the nonlinearities present in the data, or have a simple model (perhaps a linear model or something) with less prediction power but easier to interpret. Here is a very good post discussing ideas on how to interpret machine learning models.

Industries, while being very cautious, are slowly becoming more interested in adopting more complex models! Still they want to know the trade-off clearly? A data scientist perhaps is the one sitting between data team and decision-makers, and often need to be able to explain these stuffs in layman's terms.

I am trying to brainstorm here to see what analogy you would come up with to describe such trade-off to a non-technical person?


texts: How to train a model to categorize user browser history?

texts:
Part-1: How do we constitute serialized training data for model training? For instance, if I were creating a prediction model using visited web pages data, what's the best way to render such data?
Example from training data: [<website 1, website 2, website 3>, entertainment], where entertainment is one of the labels (Summarizing the category of the web pages that user browsed in the last session)
Part-2: Is RNN the best method to train on such data for classification?


texts: How to get used to matricial/vectorial operations?

texts:
I came to data science/machine learning from another background in computer science and i feel that i'm lacking of experience with matricial/vectorial operations.
Python or Matlab, for instance, provide awesome features like numpy to easily manipulate tabular data. However, the first solution that comes to my mind when coding is for or while loops. I've faced situations where my code could be easily reduced to 1 line or  so with numpy.
I'd like to know if there are good lectures/books you can recommend to change this way of thinking when it comes to problem solving/coding approaches.
Thanks in advance.


texts: Neural Network for regression with one dependent and one independent variable

texts:
I am trying to make a simple neural network with one dependent and one independent variable. Could you maybe give me a tutorial or help me with the implementation of a neural network with one dependent and one independent variable. So far I have the following code, however my predictions are not good although the error is minimized. Should I scale X and Y or do I have some mistake?

Thank you in advance

import tensorflow as tf
import matplotlib.pyplot as plt
import numpy as np

x=[(i*i)+0.2 for i in range(1000)]
y=[i for i in range(1000)]

x_train=np.reshape(x,(-1,1))
y_train=np.reshape(y,(-1,1))
x_test=x_train[:,-10:]
y_test=y_train[:,-10:]
plt.scatter(x_train,y_train)
plt.show()

X=tf.placeholder(tf.float32,[None,1])
Y=tf.placeholder(tf.float32,[None,1])

n_inputs=1
n_hidden_1=20
n_hidden_2=20
n_outputs=1

weights={
    "h1": tf.Variable(tf.random_normal([n_inputs,n_hidden_1])),
    "h2": tf.Variable(tf.random_normal([n_hidden_1,n_hidden_2])),
    "out": tf.Variable(tf.random_normal([n_hidden_2,n_outputs]))
}

biases={
    "b1": tf.Variable(tf.random_normal([n_hidden_1])),
    "b2": tf.Variable(tf.random_normal([n_hidden_2])),
    "out": tf.Variable(tf.random_normal([n_outputs]))
}

def neural_net(x):
    layer_1=tf.add(tf.matmul(x,weights["h1"]),biases["b1"])
    layer_1=tf.nn.relu(layer_1)
    layer_2=tf.add(tf.matmul(layer_1,weights["h2"]),biases["b2"])
    layer_2=tf.nn.relu(layer_2)
    layer_3=tf.matmul(layer_2,weights["out"])+biases["out"]
    return layer_3

Y_pred=neural_net(X)

loss=tf.losses.mean_squared_error(X,Y_pred)
optimizer=tf.train.AdamOptimizer(learning_rate=0.01)
train_op=optimizer.minimize(loss)

epochs=1000
init=tf.global_variables_initializer()
with tf.Session() as sess:
    sess.run(init)
    for i in range(epochs):
        sess.run(train_op,feed_dict={X:x_train,Y:y_train})
        loss_op=sess.run(loss,feed_dict={X:x_train,Y:y_train})
        if i%10==0:
            print("Epoch "+str(i)+" loss "+str(loss_op))
    pred=sess.run(Y_pred,feed_dict={X:x_test})
    plt.plot(pred,color="red")
    plt.plot(y_test,color="blue")
    plt.show()
    plt.scatter(pred,y_test)
    plt.show()
    for i in range(len(pred)):
        print(str(pred[i])+" "+str(y_test[i]))


texts: Any idea about application of deep dream?

texts: Recently Google publicized interesting deep dream. Besides art generation such as http://deepdreamgenerator.com/, do you see any potential applications of deep dream in computer vision or machine learning?


texts:
I'm trying to create a regression model that predicts the duration of a task. The training data I have consists of roughly 40 thousand completed tasks with these variables:

Who performed the task (~250 different people)
What part (subproject) of the project the task was performed on (~20 different parts)
The type of task
The start date of the task (10 years worth of data)
How long the person who has to do the task estimates it will take
The actual duration this task took to finish

The duration can vary between half an hour to a couple of hundreds of hours, but is heavily right skewed (most tasks are completed within 10 hour). On log scale the distribution is still slightly right skewed.

The prediction doesn't have to be perfect, but I'm trying to improve the people's estimations. One question to ask is "What measure can we use the define beter?" I think the best measure would be the Mean Squared Error (MSE) since it weighs large errors much worse than small errors.

Before I turned to machine learning I tried some simple approaches such as adjusting the estimation by the average or median error, adjusting it by the average/median error grouped by person, grouped by subproject but each of these happened to perform worse.

With machine learning, one of the first problem I encountered was the number of categorical variables since for most models these have to be encoded someway (e.g. one-hot). Anyway, I tried to apply some linear models, for example with Stochastic Gradient Descent my approach would be:

One-hot encode the categorical features
The converted the date to unix timestamps
Normalize all the features that are not already between 0 and 1
Split the data in 80/20 learn and test sets.
With Grid Search cross validation and the learn set try to find the best hyper parameters and fit the model.
Predict with the test set
Calculate the error/score

Now one thing I noticed was that the results varied quite a bit: On one run the MSE was close to double of another run (150 and 280). Another thing is that the MSE of the people's estimate is about 80, so my model performs a bit worse.

During my efforts to improve the performance I stumbled across this question where someone suggests to use survival models. Now I'm unfamilliar with these kinds of models but it sounded promising but during my initial tests with this it turns out to be way too slow for my purposes (too large of a dataset).

In the same Datascience answer that suggested to use the survival models (and the Wikipedia page) they also mentioned Poisson regression, but I'm not sure how I would apply this to my case.

So a long story short: I have just two questions: 1. Was my approach of using SGD 'correct' and do you think I can improve the results with that? 2. Are other models better suited for this kind of prediction and if so, can you explain a bit how I would use them?


texts: Analysis for basic weight training analysis?

texts:
TL;DR: I'm doing a fairly basic project which involves exercise. It seems that descriptive statistics and basic data vis (ex: line graph) would be most appropriate for this project, but I wonder if you have any recommendations for analyses.

For this project, I am performing the same set of 15 single-joint exercises each week (we'll call these "Exercises"). Every 4 weeks, I'm performing 3 different multi-joint exercises (we'll call these "Lifts"). My goals are to:

Track my progress (strength gains) from the Exercises.
See if my rate of progress on the Exercises roughly matches my rate
of progress on the Lifts, which would possibly indicate some sort of
transfer effect.
Explore relationships between strength gains measured during each
workout and other factors (including sleep quality, stress)

My hypothesis is that my strength gains from the Exercises I perform will transfer well to the strength gains from the Lifts that I perform.

The design is a bit janky (and it's more of a correlational study design than an experimental design), but this project is mostly just to demonstrate data collection and basic data analysis concerns. However, I'd like to make the analysis as interesting as possible. So far, it seems that basic descriptives are the best way to analyze this data, with something like a line plot for the gains on the Exercises and the Lifts.

Do you have any suggestions for analyzes that could be performed?

The variables I'm tracking are:

Weight, repetitions, and sets for each Exercise (15 in total, each exercise performed weekly)
Weight, repetitions, and sets for each Lift (3 in total, each Lift
performed once every month)
Sleep quality for the night before each workout
Energy before the workout and after the workout
Stress levels before the workout and after the workout
Level of physical fatigue before the workout
Weight before the workout
Heart rate (before the workout and once during the workout)
Exertion of the workout (rated after the workout)

Thanks!


texts: Equation of a Multi-Layer Perceptron Network

texts:
I'm writing an article about business management of wine companies where I use a Multi-Layer Perceptron Network.
My teacher then asked me to write an equation that lets me calculate the output of the network. My answer was that due to the nature of multi-layer perceptron networks there is no single equation per se. What I have is a table of weights and bias. I can then use this formula:
$$f(x) = (\sum^{m}_{i=1} w_i * x_i) + b$$
Where:

m is the number of neurons in the previous layer,
w is a random weight,
x is the input value,
b is a random bias.

Doing this for each layer/neuron in the hidden layers and the output layer.
She showed me an example of another work she made (image on the bottom), telling me that it should be something like that. Looking at the chart, I suppose that it is a logistic regression.
So, my questions are the following:

Is there any equation to predict the output of a multi-layer perceptron network other than iterating over each neuron with $w*x+b$?
Should I just tell my teacher that a logistic regression is a different case and the same does not apply to this type of neural networks?
Is the first formula correct to show that a value of a neuron is the sum product of the previous layers plus the bias?

Edit 1: I didn't wrote the formula but I do also have activation functions (relu).


texts: Building an unsupervised learning model to detect suspicious transactions using DBSCAN

texts:
I am working on building a unsupervised learning model to detect suspicious transactions using DBSCAN. Do I train the model on all data columns (columns like account number, transaction date, transaction details) or only on the ones that I want to detect suspicion in (money withdrawal and deposit columns)?
This is the data set that I am working on.


texts: Is the forward chaining CV really suitable for time series?

texts: If the time series distribution is non-stationary, we have to retrain our model once in a while (because it must forget the old dependencies and learn new ones). In forward chaining CV we use longer and longer data sequences to train the model anew. Does it make sense if we don't drop old data ?


texts: Does RandomForest convergence imply I can solve a problem with a NN too?

texts:
I'm trying to perform a regression on a dataset, and I've been testing a few models, mostly for practice.
I was able to get good results with a RandomForestRegression model, as you can see in the scatter plot:

So, I tried to solve the same problem with a NN (again, I'm doing it mostly to practice), but the results are definitely bad.
As you can see in the top part of the following picture, the loss on the training and test set (red and green lines, respectively) seem to somewhat converge, but their value is still higher than the one obtained with the RF (the horizontal blue line). Also, the scatter plots are horrible, since the model is basically always predicting the mean of the data.

The model is a simple MLP, with one hidden layer. I'm using a tanh activation function for the hidden neurons. Since the losses seem to have converged, I don't know if using a smaller learning rate or training the model for more epochs would improve the results.
However, my question is more general, independently of this specific result.
I have the general understanding that NNs are a more flexible than a RandomForest, so I am wondering if the fact that I obtained a good result with a RandomForest, automatically implies that there should exist some NN configuration which could provide a result at least as good as the one obtained with the RF.
Is this true, and I just have to fix the NN model until I find the right configuration?
Or is it possible that this problem can be properly modelled by a RF, but not by a NN?


texts: How to define the number of features to select in RFECV?

texts:
I am trying to work on feature selection stage for my dataset.

I am a newbie to ML. I have around 60 columns and am trying to select top 15 features. I came to know about RFECV for which I wrote a code like as shown below. I am aware that n_features is present for RFE but it is missing for RFECV. Is there anyother way to assign the number of features to select?

model  = RandomForestClassifier(n_estimators=100, random_state=0)
# create the RFE model and select 15 attributes
rfe = RFECV(model,step=5, cv=5,min_features_to_select = 15,max_features_to_select = 15) # this doesn't work. `n_features=15` also doesn't work
rfe = rfe.fit(X_train_std, y_train)
# summarize the selection of the attributes
feat = rfe.support_
fret = rfe.ranking_
features = X.columns
print(features[feat].tolist())

Can someone help me to get top 15 features only? Where can I configure the n_features paramter?

Currently it displays more than 30 features. I don't really know how or from where does it get its number (30)?


texts: XGBoost: what to do when Kfold is not enough?

texts:
I have a dataset made of roughly 100 time-series and my final goal is to obtain a classification of each point (detection problem). To do so I have labels so I decided to use an XGB model to perform the detection over some features that I have created. The time-series are not sampled uniformly and the time-order looks not so important for this specific problem so far.

The problem is that when I perform the StratifiedKFold (as per Sklearn) the results looks promising and the standard deviation of the relevant metrics among the kfold is really small. Nevertheless, if I remove one time-series entirely from the training set and fitting the model over the other ones I am not able to replicate the same results.

This gap between Kfold performances and "real test" looks to me like the training is not really generalising the problem, despite the good results during the Kfold validation.

Do you have any idea to fix this problem? or any advice?


texts: Constructing graph of crypto financial instruments 2016-2017

texts:
Set of financial instruments represent the set of vertices of the graph. For any pair of vertices $i$ and $j$, an edge connecting them is added to the graph if the corresponding correlation coefficient  $C_{ij}$ based on the price  fluctuations of instruments $i$ and $j$ is greater than or equal to a specified threshold $\theta, (\theta ∈ [ − 1; 1])$.

(Reference)

Finding correlation
Degree of distribution
Analysis of cliques and independent sets in the market graph

I am wondering how to achieve the above? I am willing to understand the internal structure of the market.


texts: Improve Pandas dataframe filtering speed

texts:
I have a dataset with 19 columns and about 250k rows. I have worked with bigger datasets, but this time, Pandas decided to play with my nerves.

I tried to split the original dataset into 3 sub-dataframes based on some simple rules. However, it takes a long time to execute the code. About 15-20 seconds just for the filtering.

Any alternative way that will improve the performance of the code?

import pandas as pd

#read dataset
df = pd.read_csv('myData.csv')

#create a dataframe with col1 10 and col2 <= 15
df1 = df[(df.col1 == 10) & (df.col2 <= 15)]
df = df[~df.isin(df1)].dropna()

#create a dataframe with col3 7 and col4 >= 4
df2 = df[(df.col3 == 7) & (df.col4 >= 4)]
df = df[~df.isin(df2)].dropna()

In the end, I have the df1, df2, df dataframes with the filtered data.


texts: In what data science applications has the stack exchange dump been used?

texts: Anonymized dumps of the stack exchange data are available here. Do you know projects or article that have been using these data (for social network analysis or information retrieval) ? My little research on the subject on Google Scholar & co seems to indicate that this dataset has been very seldom used.


texts: Are there any image classification algorithms which are not neural networks?

texts:
Image classification is the task of assigning one of $n$ previously known labels to a given image. For example, you know that you will be given a couple of photos and each single image has exactly one of $\{cat, dog, car, stone\}$ in it. The algorithm should say what the photo shows.

The benchmark dataset for image classification is ImageNet; especiall thy large scale visual recognition challenge (LSVRC). It has exactly 1000 classes and a huge amount of training data (I think there is a down-sampled version with about 250px x 250px images, but many images seem to be from Flicker).

This challenge is typically solved with CNNs (or other neural networks).

Is there any paper which tries an approach which does not use neural networks in LSVRC?

To clarify the question: Of course, there are other classification algorithms like $k$ nearest neighbors or SVMs. However, I doubt they work at all for that many classes / that much data. At least for $k$-NNs I'm sure that prediction would be extremely slow; for SVMs I guess both fitting and prediction would be much to slow (?).


texts: Can a Box plot be used for finding the useful features from the dataset?

texts:
I am reading a book by professor Trevor Hastie and professor Robert Tibshirani called "Introduction to Statistical Learning". In the applied section of the chapter 4, there is a question 11(b) that says:

Explore the data graphically in order to investigate the association between "mpg01" and the other features. Which of the other features seem most likely to be useful in predicting "mpg01"? Scatterplots and boxplots may be useful tools to answer this question. Describe your findings.

Here I tried plotting those box plots but how to find the useful features by analyzing them.

I'm certain that from the correlation matrix and the scatter plots I can judge what all features could be useful, but how come a box-plot be used for that matter, could someone please share their opinion.


texts: What is the effect of changing the seed value on the classification results?

texts:
I used weka program  to make a classification, first I used a dataset using the explorer and the seed value was equal to one, then I used the experimenter for the same dataset and the seed value was also equal to one and I got the same classification results.
But the second time I used the same data set and the value of the seed is equal to two through the explorer, but the result obtained by the explorer is different from the result obtained by the experimenter, knowing that the value of the seed is also equal to two in the experimenter window.

What is the reason for this ?


texts:
We are using random forest algorithm but having some trouble understanding the scoring method it uses.
take for example the following CM of the test set:
Threshold 45 cm is:
[[67969 48031]
 [ 3321 11120]] and the prescion is: 0.18799344051632602
Threshold 50 cm is:
[[77642 38358]
 [ 4785  9656]] and the prescion is: 0.2011080101632834
Threshold 55 cm is:
[[88825 27175]
 [ 6796  7645]] and the prescion is: 0.2195577254445159
Threshold 60 cm is:
[[100411  15589]
 [  9629   4812]] and the prescion is: 0.2358707906463611
Threshold 65 cm is:
[[112421   3579]
 [ 13098   1343]] and the prescion is: 0.2728565623674755
Threshold 70 cm is:
[[115895    105]
 [ 14371     70]] and the prescion is: 0.3999999997714286
Threshold 75 cm is:
[[115998      2]
 [ 14440      1]] and the prescion is: 0.3333333222222226
Threshold 80 cm is:
[[116000      0]
 [ 14441      0]] and the prescion is: 0.0
Threshold 85 cm is:
[[116000      0]
 [ 14441      0]] and the prescion is: 0.0
Threshold 90 cm is:
[[116000      0]
 [ 14441      0]] and the prescion is: 0.0

This is how we used the RF and printed it's score:
grid_clf = RandomizedSearchCV(clf, param_grid, cv=tscv, verbose=10,n_iter=20,n_jobs=-1,scoring='roc_auc')
grid_clf.fit(X_train, y_train)
print(grid_clf.score(X_test,y_test))

The score we got for this model is 0.7350173458471928
As far as I understand the scoring when using roc_auc is between 0.5 and 1.
How can such a bad model received such a good score?
How is this scoring calculated?
Provided we predicted enough True Positives, we don't mind missing "1"'s and predicting False Positives.
we of course do mind predicting True Negatives
Can I change the scoring to fit what I believe are better results?
Thanks


texts: using unsupervised learning algorithms on images

texts:
I am working on a project to classify images of types of cloth (shirt, tshirt, pant etc). While this is a standard supervised classification problem, the accuracy of the neural network is not good. This is because of the close similarity of the types of cloth that I am trying to classify.

I am working with 9 classes with around 10,000 images per class. For the classification problem I tried using CNN to classify the images. But over fitting took place with a good training accuracy (around 95%), but not so great validation accuracy (around 77%).

I wanted to know if there was any way I could create clusters based on the type of cloth using some unsupervised learning algorithm like K Means or DBScan.


texts: Confusion Matrix of LSTM using Sklear getting error

texts:
I want to calculate the confusion Matrix of my LSTM model. Shape of y_test= (17799,1) y_Pred= (17799,1) I used thefollowing code:from sklearn.metrics import confusion_matrix
cm = confusion_matrix(y_test, y_pred)getting following error: raise ValueError("{0} is not supported".format(y_type))

ValueError: continuous is not supported
both y_test and y_predict are normalized using minmaxscaler(0,1)
first few rows of both varaibles looks like:
y_test={0,0,0,0.0216212}
y_predict={0.000433061,0.000433061,0.000433061,0.000450924}

Please suggest how to calculated confusion matrix, tp,fp,tn,fn .


texts: How to update weights in a neural network using gradient descent with mini-batches?

texts:
[I've cross-posted it to cross.validated because I'm not sure where it fits best]

How does gradient descent work for training a neural network if I choose mini-batch (i.e., sample a subset of the training set)? I have thought of three different possibilities:

Epoch starts. We sample and feedforward one minibatch only, get the error and backprop it, i.e. update the weights. Epoch over.

Epoch starts. We sample and feedforward a minibatch, get the error and backprop it, i.e. update the weights. We repeat this until we have sampled the full data set. Epoch over.

Epoch starts. We sample and feedforward a minibatch, get the error and store it. We repeat this until we have sampled the full data set. We somehow average the errors and backprop them by updating the weights. Epoch over.


texts: Gradient of NN output with respect to inputs

texts:
I've trained a neural network (NN) on a problem where multiple inputs can be mapped to the same output. I'd like to use this NN to go from an output to an input i.e. given an output vector $y$, I want to find an input $x$ such that the NN returns some $z$ close to the given output $y$ when fed an input of $x$.

I was thinking of using gradient descent to do this. Do any of the common deep learning APIs let you take gradients of NNs with respect to their inputs?

I've looked around and haven't found anything, but figured I'd check here before moving forward.


texts: Mapping “event” series, with segments of variable length, to time series for loss calculation

texts:
I have a time series of data which I map to a time series of non-exclusive binary features (in my case, short-time audio spectrograms to features of the audio). I will refer to a collection of these classes that remains roughly constant over some time as “event”.

For example, my audio stream might contain the word “leave”, so my time series of classes might contain

l-sound   00000000000011111111100000000000000000000000000000000000000000000
vowel     00000000000000000000011111111111111111110000000000000000000000000
fricative 00000000000000000000000000000000000000011111111111000000000000000
high      00000000000000000001111111111111111111000000000000000000000000000
noise     00011111010000000000000000000000000000000000000111111100000000000

which correspond to – taking into account that transitions beween events may be noisy – the events sequence

l-sound   00010000
vowel     00001000
fricative 00000100
high      00001000
noise     01000010

All of these features are binary and not mutually exclusive, and every (realized) combination of features corresponds to one label, so if # stood for exactly the feature set {noise}, i for {high, vowel} and ʎ for {l-sound, high} etc., the columns of this event sequence might be given as the string _#_liv#_.

I have vastly more target data in this compressed, event-centered string format, a lot more than I have time-series of binary feature vectors. Because the mapping from event sequence to time series is just a somewhat noisy stretching of the data, probably even with characteristic lengths for each event, my intuition suggests that there should be a better, more deterministic way to use this data than making a seq2seq learn its internal structure.

I have tried to build a separate calculation graph that maps a set of lengths to differences of sigmoid functions, in order to obtain differentiable functions that approximate interval characteristic functions (0011110000 etc.), hoping that an optimizer might be able to shift the boundaries towards the actual ranges. It does not look promising.

Given that my model currently produces time-series data, but I have a lot of event-sequence data, how can I best use my event-sequence data to train the model?


texts: Can we use Stop Words while using multinomial navies theorem?

texts:
I'm collecting Twitter tweets for sentiment analysis. I chose to use Multinomial Navies theorem for finding the sentiment.

I found some examples of SVM theorem making use of stop words. My question is: can we apply stop words in Multinomial Cavies theorem or not?


texts:
I want to evaluate the performance of my linear regression model. I have the true values of y (y-true). I am thinking of two way for evaluation but not sure which one is correct.

Let's assume that we have 2 samples and each sample has two outputs as following:

y_true = [[0.5, 0.5],[0.6, 0.3]]
y_pred = [[0.3, 0.7],[0.9, 0.1]]

- Approach#1 :

One way to calculate the sum of the difference between the actual and predicted for each vector and then average all, as follows:

sum_diff_Vector(1) = abs( 0.5 - 0.3 ) + abs( 0.5 - 0.7 ) =  0.4

sum_diff_Vector(2) = abs( 0.6 - 0.9 ) + abs( 0.3 - 0.1 ) = 0.5

Then avg ( sum_diff_Vector(1) , sum_diff_Vector(2) ) = 0.45

- Approach#2 :

Another way to use the mean absolute error provided by sklearn.metrics in python.  The thing with this metric, as opposed to the previous method, it calculates the mean absolute error for each output over all samples independently and then average all of them, as follows:

MAE_OUTPUT(1) = abs(( 0.5 - 0.3 ) + ( 0.6 - 0.9 )) / 2 = 0.25

MAE_OUTPUT(1) = abs(( 0.5 - 0.7 ) + ( 0.3 - 0.1 )) /2 = 0.2

Then avg ( MAE_OUTPUT(1) , MAE_OUTPUT(1) ) =  0.225

Which way is correct and I should use ? please advise?


texts: How to explain a Calibration Plot for many models?

texts:
I have a heavy imbalanced dataset with a classification problem. I try to plot the Calibration Curve from the sklearn.calibration package. In specific, I try the following models:

rft = RandomForestClassifier(n_estimators=1000)
svc = SVC(probability = True, gamma = "auto")
gnb = MultinomialNB(alpha=0.5)
xgb = xgboost.XGBClassifier(n_estimators=1000, learning_rate=0.08, gamma=0, subsample=0.75, colsample_bytree=1, max_depth=7)

The calibration curve is the following

As you can see on the plot, Random Forest and XGBoost are close to the perfectly calibrated model. However, Naive Bayes and SVM perform terribly.

How can I explain/describe the behaviour of those two models?


texts:
I have some observations belonging to groups and I would like to compute the similarity of them within different groups in order to tell which observations, within specific groups, have similar characteristics or not.

Which metrics can be useful to do such things and under which conditions?

I know that there are many metrics to compute the similarities between individuals in a group, but there is no threshold to tell whether individuals in a cluster share similar patterns or not.