texts: Accuracy reduces drastically when using TruncatedSVD with hashingvector

texts:
I have around 0.8 million product description with categories. There are around 280 categories. I want to train a model with given dataset so that in future I can predict Category for the given product description. Since the dataset was large I was not able to able TF-IDF on that data it throws MemoryError.

I found that Hashingvector was desirable when dealing with large data. But when Hashingvector was applied I found it produced data with 1048576 features. It took around 1 hour to train and SGD model and produced 78% accuracy.

Code:

import pandas as pd
from sklearn.feature_extraction.text import HashingVectorizer
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
from sklearn.decomposition import TruncatedSVD
from sklearn.linear_model import SGDClassifier
from sklearn.calibration import CalibratedClassifierCV

data_file = "Product_description.txt"
#Reading the input/ dataset
data = pd.read_csv(data_file, header = 0, delimiter= "\t", quoting = 3, encoding = "utf8")
data = data.dropna()

train_data, test_data, train_label,  test_label = train_test_split(data.Product_description, data.Category,
                                                                   test_size=0.3, random_state=100, stratify=data.Category)

sgd_model = SGDClassifier(loss='hinge', n_iter=20, class_weight="balanced", n_jobs=-1,
                          random_state=42, alpha=1e-06, verbose=1)
vectorizer = HashingVectorizer(ngram_range=(1,3))
data_features = vectorizer.fit_transform(train_data.Product_description)
sgd_model.fit(data_features, train_label)
test_data_feature = vectorizer.transform(test_data.Product_Combined_Cleansed)
Output_predict = sgd_model.predict(test_data_feature)
print(accuracy_score(test_label, Output_predict))

Output:

Accuracy 77.01%

Since the dimension was high I thought reducing the dimension would increase the accuracy and reduce the training time. I used TrancatedSVD to reduce the dimension but this drastically reduced the prediction accuracy but reduced the training time to 10 minutes.

Code2:

from sklearn.decomposition import TruncatedSVD
clf = TruncatedSVD(100)
clf.fit(data_features)

Output:

Accuracy 14%

Edit:

When I tried TruncatedSVD with 1000 as limit it throws memory error so only I opted to use 100 as the limit.

It is stated that reducing n_features on HashingVector will cause collisions there can be collisions: distinct tokens can be mapped to the same feature index. However, in practice, this is rarely an issue if n_features is large enough (e.g. 2 ** 18 for text classification problems) in scikit site.

I got optimum accuracy when I used ngram between 1 to 3 so only used that.


texts: Why is a general/original softmax loss not preferred in FR (face recognition)?

texts: In some papers I've read that softmax loss is not preferred in FR since it does not give a good inter-class and intra-class margins, but could not understand 'why?'. So can someone explain, why softmax loss is not preferred in FR, in both mathematically and theoretically.


texts: How to interpret Logistic regression coefficients using scikit learn


texts: How to make a Label Encoder trained on a training dataset transform an unseen value of a test dataset?

texts:
During the data preprocessing stage, I decided to apply the Label Encoding on one of the columns because it contained data points in string format. Suppose the column contains the following distinct values : -
pandas_dataframe["types"].values = ["a", "b", "c", "d"]

So I fit label encoder on types column values and the label encoder assigned it values like ("a", 0), ("b", 1), ("c", 2), ("d", 3)
All the above is done with respect to the training dataset.
But while preprocessing the test dataset, when I use the previously trained label encoder I observe that a new value (suppose e) is introduced in the types column on which the label encoder was never trained. In this situation how should I handle this new value e?


texts: How to create a pipeline out of this kaggle notebook and automate it with Luigi?

texts:
I am new to ML and I was wondering if there's anyway to convert this notebook into an ML pipeline and automate it with a tool like luigi?
https://www.kaggle.com/olivierg13/wine-ratings-analysis-w-supervised-ml
Please help out if you can instead of downvoting/reporting/closing.


texts: How do convolutional layers get connected to each other?

texts:
I'm trying to analyze the architecture of a CNN and I was expecting it to be shaped:

Input Image: 28 x 28
1st conv.: 28 x 28 x nb_filters
2nd conv.: 28 x 28 x nb_filters x nb_filters

But apparently it goes:

Input Image: 28 x 28
1st conv.: 28 x 28 x nb_filters
2nd conv.: 28 x 28 x nb_filters

I thought that every filter in the second layer would be applied to every activation map in the first layer resulting in:

Activation maps after second layer = nb_filters_1st_layer x nb_filters_2nd_layer

If it's not working that way, how do I know what filters are applied to what activation maps?


texts:
The model of linear regression is linear in parameters.

What does this actually mean?


texts: How to train a xgboost model on data that is too big for the memory?

texts: What are the best practices to train xgboost (eXtreme gradient boosting) models on data that is to big to hold it in memory at once? Splitting the data and train multiple models? Are there more elegant solutions?


texts:
I give to keras an input of shape input_shape=(500,).
For some reasons, I would like to decompose the input vector into to vectors of respective shapes input_shape_1=(300,) and input_shape_2=(200,)
I want to do this within the definition of the model, using the Functional API. In a way, I would like to perform slicing on a tf.Tensor object.
Help is welcome!


texts:
Should there be a flat layer in between the conv layers and dense layer in YOLO?

It's something not specified in the paper, but I see most implementations of YOLO on github do this. In my implementation, I do not flatten the 7*7*1024 feature map and directly add a Dense(4096) layer after it (I'm using keras with tensorflow backend).

Code in question:

x = Conv2D(filters=1024, kernel_size=(3,3), padding='same', activation=leaky_relu)(x)
# x = Flatten()(x) # not using this right now
x = Dense(4096, activation=leaky_relu)(x)
x = Dense(30)(x)

My output of model.summary is as follows:

conv2d_294 (Conv2D)          (None, 14, 14, 1024)      9438208
_________________________________________________________________
conv2d_295 (Conv2D)          (None, 7, 7, 1024)        9438208
_________________________________________________________________
conv2d_296 (Conv2D)          (None, 7, 7, 1024)        9438208
_________________________________________________________________
conv2d_297 (Conv2D)          (None, 7, 7, 1024)        9438208
_________________________________________________________________
dense_14 (Dense)             (None, 7, 7, 4096)        4198400
_________________________________________________________________
dense_15 (Dense)             (None, 7, 7, 30)          122910

I feel that this implementation is more correct than the flat one (unless otherwise specified by the author) because on flattening the output we lose the spatial information needed to predict bounding boxes and confidence scores for each element in the grid.


texts: DBSCAN - Best way to find the Eps and MinPts for geospatial data (coordinates)

texts:
Question: The best way to find out the Eps and MinPts parameters for DBSCAN algorithm?
Problem: The goal is to find the locations (clusters) based on coordinates (input data). The algorithm calculates the most visited areas and retrieves these clusters.
Approach:
I defined the epsilon (EPS) parameter as 1.5 km - converted to radians to be used by the DBSCAN algorithm: epsilon = 1.5 / 6371.0088 (ref to this 1.5 km: https://geoffboeing.com/2014/08/clustering-to-reduce-spatial-data-set-size/).
If I define the MinPts to a low value (e.g. MinPts = 5, it will produce 2000 clusters), the DBSCAN will produce too many clusters and I want to limit the relevance/size of the clusters to an acceptable value.
I use the haversine metric and ball tree algorithm to calculate great-circle distances between points.
Suggestions:

knn approach to find EPS;
domain knowledge and to decide the best values for EPS and MinPts.

Data: I'm using 160k coordinates but the program should be capable to handle different data inputs.


texts: How can I detect patterns and/or keywords or phrases?

texts:
I am collecting data in a database via php from apache.

I am interested in detecting patterns in each column for now.

For example manual examination of the data shows the pattern phpmyadmin is various forms and capitalization and at different positions in the text.  Also to detect any other patterns.

How would I detect that programmatically using the computer instead of my brain?

I am going to need a detailed explanation as I am brand new to doing this kind of thing.

A pattern is any sequence of characters that repeat together. Like phpmyadmin is a sequence of 9 letters that repeat together. I detected "phpmyadmin" manually, I would like to detect it programmatically. Excluding single digit matches of common characters which is ascii 32-127


texts: Split unprocessed dataset into train and test sets

texts:
I have two files namely - log.csv and label.csv. My aim is to to split these files into training and testing datasets before preprocessing the log_train file further.

The log.csv file has a no. of columns including id. For one value of id, the file has 'z' no. of rows (this 'z' is different for different value of id) . Whereas, in the label.csv file there is just one row for each value of id.  I wish to split the files into - log_train.csv, log_test.csv, label_train.csv and label_test.csv obviously such that all rows corresponding to one value of id goes either to train or test file with corresponding values in label_train or label_test file.

I have already tried doing:
X_train,X_test,y_train,y_test=train_test_split(df,df_label,test_size = 0.2, random_state = 0)

but this gives me the error:
Traceback (most recent call last):
File "tmp.py", line 35, in <module>
    X_train, X_test, y_train, y_test = train_test_split(df,df_label,test_size = 0.2, random_state = 0)
  File "/home/yamini/virtual/home/yamini/virtual/lib/python3.6/site-packages/sklearn/model_selection/_split.py", line 2184, in train_test_split
    arrays = indexable(*arrays)
  File "/home/yamini/virtual/home/yamini/virtual/lib/python3.6/site-packages/sklearn/utils/validation.py", line 260, in indexable
    check_consistent_length(*result)
  File "/home/yamini/virtual/home/yamini/virtual/lib/python3.6/site-packages/sklearn/utils/validation.py", line 235, in check_consistent_length
    " samples: %r" % [int(l) for l in lengths])
ValueError: Found input variables with inconsistent numbers of samples: [608, 3]

How can I achieve this in python?


texts: Training object detection models from the scratch without using pre-trained models and weights

texts:
I have a data set of 25 images. I wish to run Faster RCNN or yolov3 object detection models on this images.I want to create my custom trained model and get weights after running say 10 epochs. Later I can save these weights and use that for prediction. Build a model, train on my image data set and get weights.
Is it possible?


texts: Regression using gradiant boosting - smoother predictions

texts:
I have practical machine learning problem. I have trained a LightGBM model to predict house prices. Compared with other models I have tried, the loss (RMSE) is quite low and overall I'm quite happy with the predictions.

When investigating the predictions by tweaking features, I have noticed one problem. The trend is correct thus that when I increase size of house, the price generally goes up. However, in many cases a small change in a feature can result in prediction change in the wrong direction. Let's say all features remain the same but I increase house size a little and the predicted price goes down for example. If I increase it further it goes up again.

Are there any some techniques to handle this problem, and get "smoother", more continuous predictions?


texts: Extract relevant vocabulary from a document

texts:
I am training a DSSM model for QnA. I have 200 queries and their correspondent answers - the answer is answering what kind of information should an article related to the query contained. E.g:

Title:
literacy rates Africa

Description:
What are literacy rates in African countries

I have trained my model with the whole vocabulary but validation has no lead to great results. By the whole vocabulary I mean a list containing all the words used as I thought that eliminating prepositions, conjuctions and so on may lead to a loss in semantic meaning.
Now I am trying to find a way to extract the more relevant vocabulary of my documents. I have done some research and I have thought of n-grams. In fact there is a similar example in CNKT where the vocabulary they use for the answers is formed both for single words and n-grams but I couldn't find the way to do it by myself yet.
So far I have found a way to do the n-grams but this is not what I want as for example in the sentence:

the cow jumps over the moon

I get the following code:
the_cow_jumps
cow_jumps_over
jumps_over_the
over_the_moon

When I would be interested in the 4-gram:
cow_jumps_over_moon

Keep in mind that even eliminating the articles (or stopwords) I would still be getting more than one n-gram which is not what I want as my main objective is get the final vocabulary to train my model.
As an example of what I want could be the words:
book
book_character
book_editions_published
book_subject
books_published

Instead of:
book
character
editions
published
subject


texts: Why does only Adam optimiser perform well in my case?

texts:
I am trying to classify texts into categories (one text can have multiple categories), to do this I used one-hot encoded labels (with sklearn.preprocessing.MultiLbaleBinarizer()). My texts are also one-hot encoded but with keras.preprocessing.text.one_hot.

This is my code :

csv_data_name = 'learning_base'
df = pd.read_csv('../../datasets/csv/' + csv_data_name + '.csv ')
df['label_list'] = np.array(df.labels.str.split('|'))
df = df.drop(columns=['labels'])

# Remove incomplete lines (lines without labels)
df = df.dropna()

encoder = MultiLabelBinarizer()
labels_binary = encoder.fit_transform(df.label_list)
output_dim = labels_binary.shape[1]

vocab_size = 200000
encoded_docs = [one_hot(d, vocab_size) for d in df.text]
max_length = 200
padded_docs = pad_sequences(encoded_docs, maxlen=max_length, padding='post')

# Data division
training_set_data, testing_set_data_whole, training_set_target, testing_set_target_whole = train_test_split(
    padded_docs,
    labels_binary,
    test_size=0.1,
    random_state=0,
    shuffle=False)

model = Sequential()
model.add(Embedding(vocab_size, 200, input_length=max_length))
model.add(Flatten())
model.add(Dense(64, activation='relu'))
model.add(Dense(output_dim, activation='sigmoid'))

model.compile(optimizer=TFOptimizer(tf.train.AdamOptimizer()), loss='mean_squared_error',
              metrics=['acc'])

model.fit(training_set_data, training_set_target, epochs=50, verbose=True, batch_size=200,
          callbacks=[EarlyStopping(monitor='acc', patience=3, verbose=True), tensorboard])

Everything works fine except when I try another optimizer the learning seems incorrect, only Adam seems to work as intended.

This is my results :

|Optimizer|Time by epoch|Strict classification succes|Precision|Recall|epoch|
|-------- | ----------- |--------------------------- | ------- |------|---|
|GradientDescent                          |00:21|0%|12.5%|0%|3 (ES)|
|GradientDescent (`learning_rate = 0.01`) |00:21|0%|3.9%|16.1%|7 (ES)|
|Adadelta                                 |00:24|0%|3.1%|25.7%|3 (ES)|
|Adagrad                                  |00:26|0%|/|/|13 (ES)|
|Ftrl                                     |00:27|0%|/|/|8 (ES)|
|Momentum (`momentum = 0.01`)             |00:22|0%|/|/|18 (ES)|
|Adam (`learning_rate = 0.01` : default)  |02:37|53%|80%|60.54%|13 (ES)|

And here is the tensorboard for accuracy :


texts: Suspiciously good accuracy using neural network

texts:
I have a dataset from EEG data that is 24 features (24 electrodes) and 88000 samples with 3 classes, it is normalised and everything and had some noise filtered out via bandpassing.
When I classify with anything but a neural network the accuracy is pretty bad and I am using a 60/40 for training/test set just to make sure I can trust the result.
For example:

Gaussian naive bayes: 42%
Logistic Regression: 52%
Linear Discriminant Analysis: 51%

However I played around with a neural network achieving 95%+ averaged with:

3 hidden layers: 100, 200, 100
Activation: relu
Learning rate: adaptive

I think this is super fishy so I did a PCA analysis

And plotted it with dimension reduction to two dimensions

As you can see, there is nothing significantly separable, which really confuses me.
I am definitely using the test set to run the cross validation which is 40% of the sample data.
Can someone please advise on what's happening and whether I can trust this result? And what further steps I can do to make this result more concrete?
I don't want to celebrate too early!!!


texts: Metrics to determine K in K-cross fold validation

texts:
Consider a scenario where the dataset in hand is quite large, let's assume 50000 samples (quite well balanced between two classes).  What metrics can be used to decide the K value in a K-fold cross-validation? In other words, can a 5-fold CV be enough or should I go for a 10-fold CV?

The rule of thumb is the higher K, the better. But, putting aside the computational costs, what can be used to decide the value of K? Should we look at the overall performance, e.g. average accuracy? That is, if accuracy (5CV) ~ accuracy(10CV), we can opt for 5-fold CV?. Is the standard deviation between the performance of different folds important? That is, the lower the better?


texts:
I wanted to ask I am usually using R when I do time series but I am trying out python now.

When doing the Arima model I use ARIMA function
But Can I also use SARIMAX function for just Arima models?

When I do SARIMAX order $= (2, 0, 4)$ (noseasonal parametrs) and then when I do ARIMA order $= (2, 0, 4) $

Outputs are not the same even thou the model should be the same?

What am I missing?


texts: Getting low/very uncalibrated predictions on new data for regression using xgbtree method in train() function in R's caret package

texts:
I recently tried creating a regression model (imagine a model with the target variable being an integer with values 10-110) using xgboost (method = 'xgbTree') with caret. The model trains successfully, but when applied to new data, it looks like the predictions using the model are much lower than actuals. To clarify, while my maximum value in my new dataset is well over 100, the predictions only go up to about 47. It seems the model is still directionally accurate though, as those marked as "47" have on average, the highest actual values of the group (say, an average of 80), and those with the lowest predicted values have the lowest average actual values (15 or so).
I have seen this problem with xgbtree and classification models as well, where the predicted probabilities are not well-calibrated with the actual probabilities of classes in new data. In those cases, I simply ran a logistic regression of the predicted probability on whether a binary class flag, to come up with new well-calibrated probabilities. But it seems bizarre and concerning for this to happen during regression as well. Is there something I should be doing differently in the train function? Should I be using the "maximize" or "metric" arguments? I unfortunately cannot share an example, but maybe I will try to replicate with a different public dataset. I can find nothing about this online and after going through Max Kuhn's writings/textbook (he only talks about calibration with classification models), so I came here.
tc <- trainControl(method = "cv",
                   number = 5,
                   # repeats = 3,
                   search = "grid")

tg <- expand.grid(eta = c(0.01),
                  nrounds = c(100,200),
                  max_depth = c(3,6,9,11,15),
                  min_child_weight = c(25,50),
                  colsample_bytree = c(0.3,0.6,0.9),
                  gamma = c(0.1,0.5),
                  subsample = c(1))

mod <- train(age ~.,
             data = train,
             method = "xgbTree",
             na.action = na.pass,
             tuneGrid = tg,
             trControl = tc)
```


texts: Which plot to use for data spanned on multiple years?

texts:
I've a dataset of - Number of accidents that happens in each state. The dataset spans from 2001 to 2014.

Which plot should I use to represent the information - number of accidents in each state per year,  in 1 plot.


texts: How can NoSQL databases like MongoDB be used for data analysis? What are the features in them that can make data analysis faster and powerful?


texts: Is there a way to print SVM margin obtained for each kernel used in sklearn?

texts: It is suggested in SVM to experiment with different classifiers using the various kernels available.Tuning the parameters is advised too.I am experimenting with various gamma and regularization parameters. Is there a way to print 'margin' for each experiment?


texts:
I am trying to fit a regression model to predict the revenue generated from the sales of a particular product. I have found out seasonality both in the outcome(sales revenue) as well as in the input variables.
Sales Revenue

The 'Seasonally Adjusted Annual Rate' and the 'Non Seasonally Adjusted' version of each of the input variables are available.

What version should be used to training the model ?
Should 'month' be included as a categorical variable as a means to capture
seasonality ?
Should the seasonality be removed from revenue if the Seasonally Adjusted version of inputs are used?


texts: Categorical, nominal or continuous variable?

texts:
I'm trying to understand what level of measurement is best for describing the 'number of rooms in a flat' feature.

First of all, I think it's not a continuous feature, because rational values like 1.42, for example, don't make sense.

To decide whether a feature is categorical or nominal, we should try to find an ordering between values. And here is my question: should we look for an order with respect to the response feature (in my case 'Price of a property')?

We can say a '1-room flat' is cheaper than a '2-room flat', and so on. But that is not always the case. In general, maybe it's true, but there are cases when a '1-room flat' in the city centre is way more expensive than the one further away.

So, a can't decide which representation to choose - categorical or nominal.


texts: Face detection for different poses more robust than MTCNN?

texts:
I am using the MTCNN model described on machinelearningmastery here:  MTCNN ipazc

But it won't detect certain orientations, ie. somebody lying on the ground so the top of the head points to the right of the frame and their chin to the left.  Thus, I am going to cv2.flip on the y-axis and also rotate using the method described on pyimagesearch here:  rotation of images to try the detect() method on at least 8 different orientations of the same frame and then output the one with the most detections.

This may or may not work very well on video when faces and objects need to be detected on every frame.  Thus, I am looking for links to pretrained model zoos for MTCNN and other face detection, object detection and face recognition algorithms.


texts: Why cant RNN learn long term dependencies=?

texts:
In Colah's blog, he explain this.

  In theory, RNNs are absolutely capable of handling such “long-term
  dependencies.” A human could carefully pick parameters for them to
  solve toy problems of this form. Sadly, in practice, RNNs don’t seem
  to be able to learn them. The problem was explored in depth by
  Hochreiter (1991) [German] and Bengio, et al. (1994), who found some
  pretty fundamental reasons why it might be difficult.

In quick explanation what is the fundamental reasons why it might be difficult?


texts: Predicting probability from scikit-learn SVC decision_function with decision_function_shape='ovo'

texts:
I have a multiclass SVM classifier with labels 'A', 'B', 'C', 'D'.

This is the code I'm running:

>>>print clf.predict([predict_this])
['A']
>>>print clf.decision_function([predict_this])
[[ 185.23220833   43.62763596  180.83305074  -93.58628288   62.51448055  173.43335293]]

How can I use the output of decision function to predict the class (A/B/C/D) with the highest probability and if possible, it's value? I have visited https://stackoverflow.com/a/20114601/7760998 but it is for binary classifiers and could not find a good resource which explains the output of decision_function for multiclass classifiers with shape ovo (one-vs-one).

Edit:

The above example is for class 'A'. For another input the classifier predicted 'C' and gave the following result in decision_function

[[ 96.42193513 -11.13296606 111.47424538 -88.5356536 44.29272494 141.0069203 ]]

For another different input which the classifier predicted as 'C' gave the following result from decision_function,

[[ 290.54180354 -133.93467605  116.37068951 -392.32251314 -130.84421412   284.87653043]]

Had it been ovr (one-vs-rest), it would become easier by selecting the one with higher value, but in ovo (one-vs-one) there are (n * (n - 1)) / 2 values in the resulting list.

How to deduce which class would be selected based on the decision function?


texts: how to apply similarity algorithm(or comparision) of over one million vectors with other one million vectors?

texts:
How can I apply similarity algorithm (or comparison) of over one million vectors with another one million vectors?

I am following this pyimage search tutorial but don't know how to scale up the algorithm when I have millions of images to compare. Like comparing similarity of over 1 million images with other 1 million images. It stops with an out of memory error at 100000.


texts: Corelation between overtime and sick leave

texts:
I have a scenario where I have to identify employees, who when take sick or any other paid leave his/her colleague (any other employee) gets overtime.

My data set is as follows:

employee_code----period---Leave_hours--- Overtime hours

How to identify the relation between overtime hours and leave_hours between employees?

Which machine learning model can be used to get employee wise relation...?


texts: Questions about Neural Network training (back propagation) in the book PRML (Pattern Recognition and Machine Learning)

texts:
I am reading Chapter 5 of PRML. Some symbols don't seem to be clear to me.

In page 243, for the chain rule for partial derivative $\dfrac{\partial E_n}{\partial w_{ji}}=\dfrac{\partial E_n}{\partial a_j}\dfrac{\partial a_j}{\partial w_{ji}}$ (equation (5.50)), a notation is defined as $\delta_j\equiv\dfrac{\partial E_n}{\partial a_j}$ (equation (5.51)). In my understanding $\delta_j$ is the first part in the chain rule. However in equation (5.54), the book mentioned

  As we have seen already, for the output units, we have
  $$\delta_k=y_k-t_k$$

Question 1:
$y_k-t_k$ is the error on output unit $k$, which is simply the difference between the $k$th output unit value and the corresponding target value. But, from the definition of the notation $\delta_k$, we should have $$\delta_k=\dfrac{\partial \frac{1}{2}(y_k-t_k)^2}{\partial a_k}=(y_k-t_k)\dfrac{\partial y_k}{\partial a_k}=(y_k-t_k)\dfrac{\partial h(a_k)}{\partial a_k}$$ where $h(a_k)$ is the activation function.
So why in the book $\delta_k=y_k-t_k$??

In page 242, Section 5.3. Error Backpropagation,

  Consider a simple linear model where the outputs $y_k$ are linear
  combinations of the input variables $x_i$ so that
  $y_k=\sum_iw_{ki}x_i$. For a particular input pattern $n$, the error
  function is $E_n=\dfrac{1}{2}\sum_k(y_{nk}-t_{nk})^2$, where
  $y_{nk}=y_k(\boldsymbol{x_n},\boldsymbol{w})$. So the gradient of this
  error function with respect to a weight $w_{ij}$ is given by
  $$\frac{\partial E_n}{\partial w_{ji}}=(y_{nj}-t_{nj})x_{ni}$$ which
  can be interpreted as a ‘local’ computation involving the product of
  an ‘error signal’ $y_{nj} − t_{nj}$ associated with the output end of
  the link $w_{ji}$ and the variable $x_{ni}$ associated with the input
  end of the link.

Question 2:
I am not clear with the structure of this neural network. The one in the book is a two-layer neural network with linear activation, is it?


texts: How to get windspeed when temperature is maximum for each city?

texts:
Here is the sample dataset, I have Weather_Data,
I want to calculate
"What is the windspeed when temperature is maximum for each city"
I have tried following code:

df = pd.read_csv('weather_by_cities.csv')
g = df.groupby('city')
g.max()

But Instead of getting corresponding Wind Speed, when temperature is maximum, it's giving maximum wind speed along with maximum temperature, which is little annoying.


texts: R code that gives results like Wolfram Alpha for the expectation of a function of a random variable?

texts:
When I ask Wolfram Alpha to calculate $E[f(X)]$ where $f(x) = e^{-x^2}$ and $X \sim \mathcal{N}(1,4)$, it gives the result
$$
E[f(X)] = \frac{1}{3\sqrt[9]{e}} \approx 0.29828,
$$
and the following plot which appears to be based on taking a number of simulations:

How can I generate the same type of plot using R?


texts: Solving Feature Distribution variance between Training and Prediction for Ranking models

texts:
I am building a linear regression model to improve ranking of documents. And trying to identify problems due to which model performance estimates don't match actual impact

One major problem is feature distribution for training and prediction.

Eg:

For training data, features distribution is computed on top of seen documents.

But during online prediction, the model is applied to all matching documents. This introduces variance between what the model believed to be actual feature value distribution and has the potential to impact results negatively.

I am assuming this is a standard problem, but haven't come across good articles/papers which discuss this.

Few possible solutions in my head -

Introduce more randomness in training data but that has limited scope without impacting actual users or corrupting data significantly
During prediction, apply the model only on top K documents (which were being seen previously). This increases parity between training and prediction but limits effectiveness of model to only rerank a limited set.

Is there a better way to handle this difference in training and prediction ?


texts: Best anomaly detection algorithm based on two conditions

texts:
Choosing the right anomaly detection algorithm seems quite hard at the moment.
It might be because I am bombarded with so many alternatives likes clustering, K-Means DBSCAN and so many others.

On my side I have a csv files with thousands of lines, the columns have header that either show the name of the file and the features.

I’m my case this is how the file would look like, keep in mind the file is way larger.

Values columns is the one I want to check about anomalies, in this case I would get a lot of anomalies because the numbers belong to different units-

So first I would need to filter Unit to lets say meters and then check values column data for any anomalies

I would appreciate some advice on what would be the best approach to tackle this problem


texts: Monotonicity of Jaccard and Dice in multilabel datasets

texts: I understand that Jaccard and Dice follow a monotonic relation on binary datasets because the two are related as $J = {S \over {(2 - S)}}$, and I guess this would be the case when micro-average is used with multi-label datasets. However, would the two metrics follow a monotonic relation when macro-average is used?


texts:
Does anyone know some good tutorials on online machine learning technics?
I.e. how it can be used in real-time environments, what are key differences compared to normal machine learning methods etc.

UPD: Thank you everyone for answers, by "online" I mean methods which can be trained in a real-time mode, based on a new inputs one by one.


texts:
I am currently searching a neural network that can classify if there is a human in an image or not.

I checked the ImageNet dataset, but the 1000 classes there contain nothing like human or person or such, mostly animals and things.

Segmentation CNNs usually have a lot longer runtime, and all I really need is basically a binary NN, just "Yes = Human in Image", "No = No human in image". Can someone point me to a training set or a pre-trained network?


texts: Image to image translation from image to sketch using GANs, do I need to annotated the face at all?

texts: So far I have read 2 papers on GANs and it seems like they are unsupervised networks that only uses supervision for the discriminator. If I want to translate an image from a picture to line art all I need to do is to prepare my ground truth "the unmodified picture" and the corresponding line art correct? Do I have to pre-classify any facial feature for the generator to learn the corresponding line art or is that what Conditional GANs are for? I am trying to brainstorm my own project but the data preprocessing stage is a little hazy.


texts: When to use deep learning for java as opposed to python

texts:
I have been asked to explore options to build deep learning based applications using java, so i happend to browse a website called dl4j (https://deeplearning4j.org) which has got implemantations of different neural networks starting from MLP to RNN/LSTM.

But I couldn't understand the rationale of using dl4j over python based implemenation. So, could someone please clarify on following items please.

ETL
Data pre-processing
Making use of pre-trained models / transfer learning
Distributed computing
Processing large voulme of data (images,time series data,sensor data..etc)
Production implementation (batch or real time / online)
Post production training (bacth or real time / online)
Mobile app development
IoT edge device computing


texts: Q learning for blackjack, reward function?

texts:
I am currently learning reinforcement learning and am have built a blackjack game.

There is an obvious reward at the end of the game (payout), however some actions do not directly lead to rewards (hitting on a count of 5), which should be encouraged, even if the end result is negative (loosing the hand).

My question is what should the reward be for those actions ?

I could hard code a positive reward (fraction of the reward for winning the hand) for hits which do not lead to busting, but it feels like I am not approaching the problem correctly.

Also, when I assign a reward for a win (after the hand is over), I update the q-value corresponding to the last action/state pair, which seems suboptimal, as this action may not have directly lead to the win.

Another option I thought is to assign the same end reward to all of the action/state pairs in the sequence, however, some actions (like hitting on count <10) should be encouraged even if it leads to a lost hand.

Note: My end goal is to use deep-RL with an LSTM, but I am starting with q-learning.


texts: Why do machine learning engineers insist on training with more data than validation set?

texts:
Among my colleagues I have noticed a curious insistence on training with, say, 70% or 80% of data and validating on the remainder.  The reason it is curious to me is the lack of any theoretical reasoning, and it smacks of influence from a five-fold cross-validation habit.
Is there any reason for choosing a larger training set when attempting to detect overfitting during training?  In other words, why not use $n^{0.75}$ for training and $n - n^{0.75}$ for validation if the influence really is from cross-validation practices carried over from linear modeling theory as I suggest in this answer?
I posted a similar question on stats.stackexchange.com but based on the response thought I might have a more interesting discussion here.  The concept of training for multiple epochs is, in my opinion, inherently Bayesian and thus the concept of cross-validation may be ill-suited at worst, unnecessary at best, for reasons I suggest in that post.


texts:
The following plot shows coefficients obtained with linear regression (with mpg as the target variable and all others as predictors).

For mtcars dataset (here and here) both with and without scaling the data:

How do I interpret these results? The variables hp and disp are significant only if the data is scaled. Are am and qsec equally important or is am more important than qsec? Which variable should one say are important determinants of mpg?

Thanks for your insight.


texts: Best image PPI for computer vision deep learning algorithms

texts:
For computer vision tasks using deep learning, should I worry about image size (e.g. 256 x 256) or PPI (pixels per inch)?

I find that PPI is not discussed in the computer vision/deep learning literature.

What's the best PPI for images of size 256 x 256?


texts: Macro- or micro-average for imbalanced class problems

texts:
The question of whether to use macro- or micro-averages when the data is imbalanced comes up all the time.

Some googling shows that many bloggers tend to say that micro-average is the preferred way to go, e.g.:

Micro-average is preferable if there is a class imbalance problem.
On the other hand, micro-average can be a useful measure when your dataset varies in size.

A similar question in this forum suggests a similar answer.

However, this seems quite counter-intuitive. For example if we have a data set with 90%-10% class distribution then a baseline classifier can achieve 90% mico-averaged accuracy by assigning the majority class label.

This is corroborated by books, e.g. An Introduction to Information Retrieval says (page 282) "Microaveraged results are therefore really a measure of effectiveness on the large classes in a test collection.  To get a sense of effectiveness on small classes, you should compute macroaveraged results."

In the end the real decision about which measure to use should be based on the relative mis-classification costs for the classes. But a quick look at the internet seems to suggest use of micro-averaging. Is this correct or misleading?


texts: Is ML a good solution for identifying what the user wants to do from a sentence?

texts:
I am learning machine learning and I'm trying to implement a solution for a real problem: predict from a human sentence what programming function he/she is trying to do.

I have a series of programming functions related with a series of descriptions (there can be n > 0 descriptions for each unique function).

I created a neural network and a bag of words model trying to convert a human sentence "we get the data from the database" to a programming function. So far it works with very easy examples but not with my real data.

Something like this works:

"description"     | programming function
lala lolo lulu ka | function1
lala lolo lulu ke | function1
lala lolo lulu ko | function1
lala lele lili ka | function2
lala lele lili ki | function2
lala lele lili ko | function2

Every word in the description is converted to an neuron-input (with value of 1 if present and 0 if not present) and every possible function is converted to a neuron-output.

I'm using pyBrain with back propagation and an error threshold of 0.005. The neural network has three layers, and the middle one has lenght: number of possible words + number of possible programming functions (this is kind of arbitrary).

I know full text search or auto-complete is probably a better alternative for this task, but I'm just experimenting with machine learning and I'd like this to work if possible. In my real data I have 1000 descriptions related with ~500 functions.

So my question is:

Is bag of words + neural networks a good approach for solving this?
Maybe Word2vec is a better option?
If neither is good, is there any known machine learning approach that could work with something like that?


texts: Multi-Feature One-Hot-Encoder with varying amount of feature instances

texts:
Let's assume we have data instances like this:
[
    [15, 20, ("banana","apple","cucumber"), ...],
    [91, 12, ("orange","banana"), ...],
    ...
]

I am wondering how I can encode the third element of these datapoints. For multiple features values we could use sklearn's OneHotEncoder, but as far as I could find out, it cannot handle inputs of different length.
Here is what I've tried out:
X = [[15, 20, ("banana","apple","cucumber")], [91, 12, ("orange","banana")]]

ct = ColumnTransformer(
    [
        ("genre_encoder", OneHotEncoder(), [2])
    ],
    remainder='passthrough'
)
print(ct.fit_transform(X))

This will only output
[[1.0 0.0 15 20]
 [0.0 1.0 91 12]]

as expected, because the tuples are handled as the possible values this feature can be represented with.
We can't embed our features directly (like [15, 12, "banana", "apple", "cucumber"]), because

we don't know how many instances of this feature we will have (two? three?)
each position would be interpreted as an own feature and thus if we had banana in the first nominal slot in one datapoint and in the second one in our second nominal slot, they would not count to the same "pool of values" a feature can embody

Example:
X = [["banana","apple","cucumber"], ["orange","banana", "cucumber"]]
enc = OneHotEncoder()
print(enc.fit_transform(X).toarray())

[[1. 0. 1. 0. 1.]
 [0. 1. 0. 1. 1.]]

This representation contains 5 slots instead of 4, because the first slot is interpreted as using banana or orange, the second one as apple or banana and the last one only has the option cucumber.
(This would also not solve the problem of having different amounts of feature values per datapoint. And replacing empty ones with None does not solve the problem either, because then None faces this positional ambiguity.)
Any idea how to encode those "Multi-Muliti-"features, that can take multiple values and consist of a varying amount of elements? Thank you in advance!


texts:
I just finished working on my first machine learning algorithm i.e Linear regression. I want to reduce the rmse by optimising the model. I found out that gradient decent does the same job. But i dont know how to do it in python. I refered to some videos on youtube but every video explains y=MX+c . But i have about 50 variables in my model.

Is there any library for it. Please help me out.


texts: Need help understanding the structure of this convoluted neural network

texts:
I'm trying to grasp the structure of this convoluted neural network.

(Source)
I understand the first layer is a 6x6 conv with stride 2 followed by 3x3 max pool and then 6 5x5 convs and another 3x3 max pool. After this, however, outputs from a fully connected layer of 64 neurons are "tiled over the special dimensions of the response map of pool2".
I don't understand what this means. The output of pool2 should be 64 (because of 64 filters) 18x18 arrays. In the first 18x18 array do I add output1 to each of the 18*18=324 values, and in the second array add output2 to each of the 324 values, etc.?
TLDR
What do I do with the 64 outputs (each is a 18x18 array) and the 64 outputs from the fully connected layer?


texts:
Apology if this sounds a bit lame.

I am trying to use Allennlp for my NLP tasks and would like to use regularization to reduce overfitting. However from all the online tutorials, all the regularizers are set as None, and I still couldn't find out how to use the regularizer after many many attempts.

If I use the example in the official tutorial (https://github.com/titipata/allennlp-tutorial) , what if I want to add in regularizer for LSTM and feedforward layer?

class AcademicPaperClassifier(Model):
    """
    Model to classify venue based on input title and abstract
    """
    def __init__(self,
                 vocab: Vocabulary,
                 text_field_embedder: TextFieldEmbedder,
                 title_encoder: Seq2VecEncoder,
                 abstract_encoder: Seq2VecEncoder,
                 classifier_feedforward: FeedForward,
                 initializer: InitializerApplicator = InitializerApplicator(),
                 regularizer: Optional[RegularizerApplicator] = None) -> None:
        super(AcademicPaperClassifier, self).__init__(vocab, regularizer)
        self.text_field_embedder = text_field_embedder
        self.num_classes = self.vocab.get_vocab_size("labels")
        self.title_encoder = title_encoder
        self.abstract_encoder = abstract_encoder
        self.classifier_feedforward = classifier_feedforward
        self.metrics = {
                "accuracy": CategoricalAccuracy(),
                "accuracy3": CategoricalAccuracy(top_k=3)
        }
        self.loss = torch.nn.CrossEntropyLoss()
        initializer(self)

    def forward(self,
                title: Dict[str, torch.LongTensor],
                abstract: Dict[str, torch.LongTensor],
                label: torch.LongTensor = None) -> Dict[str, torch.Tensor]:

        embedded_title = self.text_field_embedder(title)
        title_mask = get_text_field_mask(title)
        encoded_title = self.title_encoder(embedded_title, title_mask)

        embedded_abstract = self.text_field_embedder(abstract)
        abstract_mask = get_text_field_mask(abstract)
        encoded_abstract = self.abstract_encoder(embedded_abstract, abstract_mask)

        logits = self.classifier_feedforward(torch.cat([encoded_title, encoded_abstract], dim=-1))
        class_probabilities = F.softmax(logits, dim=-1)
        argmax_indices = np.argmax(class_probabilities.cpu().data.numpy(), axis=-1)
        labels = [self.vocab.get_token_from_index(x, namespace="labels") for x in argmax_indices]
        output_dict = {
            'logits': logits,
            'class_probabilities': class_probabilities,
            'predicted_label': labels
        }
        if label is not None:
            loss = self.loss(logits, label)
            for metric in self.metrics.values():
                metric(logits, label)
            output_dict["loss"] = loss

        return output_dict


texts: Does Feature Normalization affect Gradient Descent | Linear Regression

texts:
am new to datascience and i want to learn linear regression so i coded linear regression from scratch and performed gradient descent to find the best $w_\theta$ and $b_\theta$ values using a tutorial. And it went just fine i was able to find the best $w_\theta$ , $b_\theta$ values and i ploted the line-of-best-fit (below).

And the gradient descent code i used to find the $w_\theta$ , $b_\theta$ values is below .

def step_gradient_descent(data,m,b,learning_rate=0.0001):
        b_gradient= m_gradient = 0
        N=float(len(data))
        for i in range(len(data)):
                [x,y]=data[i]
                y_=(m*x)+b
                m_gradient+= - (2/N)*(x*(y-y_))
                b_gradient+= -(2/N)*(y-y_)
        #print("m ={}, b ={}".format(m_gradient,b_gradient))
        m_new = m-(learning_rate*m_gradient)
        b_new = b-(learning_rate*b_gradient)
        return (m_new,b_new)

def perform_gradient_descent(data,m,b,lr=0.0001,epochs=1000):
        m_array=b_array=[]
        for i in range(epochs):
                if(i % 100 == 0 ):
                        print("Running {}/{}".format(i,epochs))
                (m,b) = step_gradient_descent(data,m,b,lr)
        return (m,b,m_array,b_array)

and then i performed features normalization / standization on $x$ , $y$ below

def featureNormalize(data):
    mean = np.mean(data , axis=0 )
    std = np.std(data , axis=0 )
    norm = ( data - mean ) / std
    return norm

and then after that i plotted the line-of-best-fit , which was different from the above plotted one.

So things i tried to get back the previous line-of-best-fit as before are

changing learning rate ( $r$ )
changing epochs ( $n$ )

which didnot work out. So my understanding is that feature normalization / standization should not have any effect on the gradient descent but that didnot happen in this case. Just curious to figure out what's happening in my case.

Link to my notebook

Thanks in Advance.


texts:
I am following this example to read an image (from my disk) with tensorflow:

img_raw = tf.io.read_file(image_path)
img_tensor = tf.image.decode_image(img_raw)

print(repr(img_raw))
print(img_tensor.shape)
print(img_tensor.dtype)

but the shape is coming as unknown:

<tf.Tensor 'ReadFile_4:0' shape=() dtype=string>
<unknown>
<dtype: 'uint8'>

Any idea what I am doing wrong?


texts: Neural Net Backprop Weight updating Pseudo code help please

texts:
Here is my code for Backpropagation weight updating. It's a simple network with 1 hidden layer and 1 output neuron. The activation function of both hidden and output layer uses tanh. I propagate the error using the gamma variables in the code.
The problem is I get very small change in the average error after the first iteration. I'm using initial weights randomly between -0.1 and 0.1 and learning rate of 0.03. Inputs are roughly between -0.001 and 0.001. Target output is between -0.5 and 0.5. I just want to check if the code is correct. I've tried higher values for learning rate and initial weights and the error actually goes up and settles around 1.0. Now is the code wrong or do I just need to fine tune the initial weights and learning rate? Many thanks for your replies!

ERROR = 0.5 * MathPow( (TARGET - output), 2 );

double gamma = ERROR * ( 1 - ( output * output ) );

// HIDDEN-OUTPUT LAYER UPDATE...
for ( int i=0; i<HIDDEN_LAYER_SIZE; i++ )
{
       double gamma_hi = gamma  * HIDDEN_OUTPUT_WEIGHTS[ i ];

       HIDDEN_GAMMAS[ i ] = gamma_hi;

       double hi_weight_delta = LEARNING_RATE * gamma_hi  * HIDDEN_OUTPUTS[ i ];

       HIDDEN_OUTPUT_WEIGHTS[ i ] += hi_weight_delta;
}

// INPUT-HIDDEN LAYER UPDATE

for ( int i=0; i<HIDDEN_LAYER_SIZE; i++ )
{
   for ( int j=0; j<INPUT_LAYER_SIZE; j++ )
   {
        double gamma_ij = HIDDEN_GAMMAS[ i ] * INPUT_HIDDEN_WEIGHTS[ i ][ j ];

        double ij_weight_delta = LEARNING_RATE * gamma_ij * INPUT_LAYER_INPUTS[ j ];

        INPUT_HIDDEN_WEIGHTS[ i ][ j ] += ij_weight_delta;
    }
}

// OUTPUT CODE:

// Calculate value of each hidden...
   for ( int i=0; i<HIDDEN_LAYER_SIZE; i++ )
   {
      double hi = 0.0;

      for ( int j=0; j<INPUT_LAYER_SIZE; j++ )
      {
         hi += INPUT_LAYER_INPUTS[ i ]* HIDDEN_OUTPUT_WEIGHTS[ i ][ j ];
      }

      HIDDEN_OUTPUTS[ i ] = TANH( hi );
   }

   double output = 0.0;

   // Calculate value of the ouput
   for ( int i=0; i<HIDDEN_LAYER_SIZE; i++ )
   {

      output += HIDDEN_OUTPUTS[ i ] * HIDDEN_OUTPUT_WEIGHTS[ i ];
   }

   output = TANH( output );


texts: How to Keep Missing Values in Ordinal Logistic Regression

texts:
I’m using mord package in python to do ordinal logit regression (predict response to movie rating 1-5 stars).

One of my predictor variables is also ordinal but there are some missing values where the viewer skipped a question because it wasn’t applicable due to skip logic from a prior question or because they missed it.
What’s the best way to indicate a value is “missing” and/or “not applicable” while also retaining the ordinal nature of this predictor variable for everyone else? I don’t think I should delete this viewer or try to impute the value.

I get an error if I leave the NaN. I thought about dummy coding so I have something like question5_never, question5_sometimes, question5_always, question5_na, question5_missing, but I not sure.


texts: How do I calculate distance of test data point from centroids in KMeans scikit-learn?

texts:
I am using Kmeans clustering on my data
After training the model, I want to calculate the distance between Test Data points and CLuster centers.
How do I do it?
My code is like below:

model = KMeans(clusters=2, random_state=42)

model.fit(X_train)

# get centroids
centroids = model.cluster_centers_

But I'm not sure how to use those centroids to calculate the distances for new data point


texts: How can i add weights in a bag of words model in text analysis?

texts: I have a twitter sentiment analysis using bag of word approach from the training set. Now i want to add weights to certain words so that they are considered more important than others.


texts: How to convert my tensorflow model to pytorch model?

texts: I performed transfer learning using ssd + mobilenet as my base model in tensorflow and freezed a new model. Now I want to convert that model into pytorch. Is there any way how I can achieve it? Any help would be really helpful..


texts: How to find 'pre-requisite' relationships between sentences

texts:
I have two arrays. Each of them is consists of some sentences indicating some action. For example:
array1 = ['Driving the car', 'Testing the function', 'Checking the variable']
array2 = ['Fueling the car', 'Assigning value in the variable', 'Writing the function']

For each sentence of array1, there is a pre-requisite sentence in array2. I want to find out which sentence of array2 is pre-requisite sentence for each of the sentence in array1.
I am not a data science guy. I just fell into this problem in my software engineering domains. Apology in advance if the question is not up to the mark of this forum. I have explored some of the questions like here or here, but seems for those I need deep knowledge in domains like RNN or ML.
I have also tried to use simple word2vec and than by calculating minimum cosine distance to identify the relationship. But that did not also resulted in good categorization. I also dont have much data to train a model.
So is there any way to find this type of relationship?


texts: Sklearn and PCA. Why is max n_row == max n_components?

texts:
I posted my question on stack overflow, but there someone suggested that I should try it here. What I'm doing now :)

OK, first to my data. I have a word-bi-gram frequency matrix (1100 x 100658, dtype=int), where the first 5 columns contain information about the document. So every row is a document and every column a word-bi-gram like (of-the, on-the, and-that,...). I want to visualize the data, but before I do that, I want to reduce the dimension. So I thought I do that with PCA from sklearn. First I set the column labels with

myPandaDataFrame.columns = word-bi-grams

then I deleted some doc-columns, because I want to see what kind of information I can get if I only look at the proficiency.

del existing_df['SUBSET']
del existing_df['PROMPT']
del existing_df['L1']
del existing_df['ESSAYID']

then I set the proficiency column to be the index with

myPandaDataFrame.columns.set_index(['PROFICIENCY'], inplace=True, drop=True)

and then I did this

from sklearn.decomposition import PCA
x = 500
pcax = PCA(n_components=x)
pcax.fit(myPandaDataFrame)
PCA(copy=True, n_components=x, whiten=False)
existing_2dx = pcax.transform(myPandaDataFrame)
existing_df_2dx = pandas.DataFrame(existing_2dx)
existing_df_2dx.index = myPandaDataFrame.index
existing_df_2dx.columns = ['PC{0}'.format(i) for i in range(x)]

But with this implementation I can only set 1100 n_components as a maximum. This is the number of documents (rows). This makes me suspicious. I tried a couple of examples / tutorials, but I can't get it right. So I hope someone can help me find out what I'm doing wrong? If would also be very happy about a good example / tutorial that can help me with my problem. Thank you.

With best regards.


texts: Is training in Biostats sufficient to get into Data Science?

texts: I'm considering going back to graduate school after a few years working as a market analyst.  I prefer to stay in my current city, where there are two statistics programs I'm interested in.  The first is a more traditional stats program at a private university and the other is a biostats program at a public university.  The biostats program is significantly cheaper, which isn't the sole factor but it is a big one.  I would like to do consulting after grad school (a company like Accenture, for example).  The core curriculum for both is similar - regression analysis, applied modeling, etc - but obviously the Biostats program has classes like epidemiology and survival analysis whereas the traditional program has electives in data mining and visualization.  Would the biostats program be enough to get a solid data science job or should I stick to a traditional program?


texts:
I am looking for books/tutorials that help you gain the insight into the thought process behind data analysis.

Most of the books I've read were mostly documentation - the author shows you a function and some data that he applies that function to. They also show how to use graphs and how histogram/boxplots etc. work. They go through popular libraries like numpy, pandas and how to use them.

I am interested in how people on (for example) Kaggle in their kernels on Titanic get their ideas from, on what to apply to datasets. Those people know which columns to plot as a function of each other, when to plot histograms, when to plot density functions and so on.

I am somewhat experienced in Machine Learning. It's pretty obvious which algorithm can be applied to what situation. Data exploring seems like a very ambiguous exercise with many solutions/ideas.

Another way to put is: Where to get the ideas for data exploration from?


texts: Difference between convolution structures

texts:
I am having a hard time understanding the difference what is a multichannel CNN:
In the paper titled, "A Multichannel 2D Convolutional Neural Network Model for Task-Evoked fMRI Data Classification" (https://www.hindawi.com/journals/cin/2019/5065214/#sec2.1) Figure 1 is a multichannel CNN where the authors are using a channel to represent an input image type. Image is a 2D grayscale matrix. According to the diagram, it appears the number of images = number of channels = number of different CNNs. Is that true? I was under the impression that irrespective of the number of channels, we have one CNN that applies filters to each input kind separately. However, this is not the same as having multiple CNNs for each input type.
I am confused -- does multi channel CNN mean that there is a separate CNN for each input data Or a single CNN that applies filters to each input separately.


texts: How to create a social network like IBM's Watson News Explorer?

texts:
I was much impressed by the news-explorer view of IBM.It shows a network view with on searching a keyword and it explores the relational edges with a graph.

Here it is

I think it uses the k-means clustering in background.I too want to do the same for twitter data.I have tweets and users and mentions i want show same a network relation between them.can any one suggest me any graphical data tool which shows the network just like IBM by taking data.

using clustering algorithms like k-means or any other make my work done?
or do i need to go for any other approach ?


texts: Word Frequency Analysis of Document Sets

texts:
I'm doing some work trying to extract commonly occurring words from a set of human classified documents and had a couple questions for anyone who might know something about NLP or statistical analysis of text.

We have a set of a bunch of documents, and users have classified them as either good or bad. What I'd like to do is figure out what words are common to the good documents, but not necessarily the other ones.

I could, for example, use the (frequency within good documents / total frequency) which would essentially normalize the effect of a word being generally common. This, unfortunately, gives very high precedence to words that occur in only a few good documents & not at all in the other documents. I could add some kind of minimum threshold for # of occurrences in good docs before evaluating the total frequency, but it seems kind of hacky.

Does anyone know what the best practice equation or model to use in this case is? I've done a lot of searching and found a lot of references to TF-IDF but that seems more applicable for assessing the value of a term on a single document against the whole set of docs. Here I'm dealing with a set of docs that is a subset of the larger collection.

In other words, I'd like to identify which words are uniquely or more important to the class of good documents.


texts: What is the use of using width/height shift in data augmentation?

texts:
I'm not sure to understand the use of augmentation data using width shift and height shift.
Say I have limited image data, and I want to create new data using Keras' ImageDataGenerator. To classify between images, I use CNNs. Since CNNs are translation invariant, aren't the translational shifts from keras useless as those shifts will not result in new images per say? (I know new images will be created from the generator, but the CNNs will not learn new features from the pictures, and might instead cause overfitting?)


texts: Initialize a recommender system with no dataset

texts: Consider a platform for content recommendation based on the user history. The contents are books and articles and by history I mean what the user has read, what he has shared and so on. I know that there are many studies on how to train a recommendation system, however I don't know what to do to "start" (the platform has not been launched yet and the dataset is empty). Are there techniques that consider an initialization phase where no data is available?


texts: Predicting future airfare using past data


texts: Should I remove a feature that is a non-linear function of another feature if I use it with a non-linear machine learning model?

texts:
Three of my features look as follows on the plot

I can fit analytic functions to express $x_1$, $x_2$ and $x_3$ as non-linear functions of a parameter $x_0$. Does this mean that $x_1$, $x_2$ and $x_3$ are redundant features if I want to use them in a non-linear algorithm?
I know that I should not throw these features away if I use linear regression. For example, adding polynomials of the features may improve the result. But what if I use a non-linear model such as a neural network, or kernel-ridge regression? Shouldn't they learn these non-linear relationships thus making a feature that is a non-linear function of another feature redundant?


texts: Predicting high frequency sparse time series data in python

texts:
I have a dataset of a couple of EV charging stations (10 min frequency) over 1 year. This data consists of lots of 0's, since there is no continuous flow of cars coming to charge but rather reoccurring charging events as peaks(for example from 7-9 am seems to be a frequent charging timeframe when people are coming to the office) I have also aggregated weather and weekday/holiday data to be used as features.

I now wish to predict the energy demand for a timeframe of 6h in the future. So far I have tried SARIMA with terrible results, since the algorithm seems to be obscured by the sparse data.
I have tried different transformations (Box Cox, Normalization, Standardization), differencing, auto-arima for optimal parameters, so far no luck.
I am willing to try different machine learning as well as statistical algorithms. Does anyone have some recommendations as of what I can do to generate a moderatly accurate prediction with a sparse dataset? (Python)


texts: Cluster documents based on topic similarity

texts:
I have set of documents where I have assigned topics per each document.

E.g., Topics of document 1 -> 1.0 Science, 1.0 politics, 0.8 History, 0. 8 Information and Technology

Now I want to cluster these documents and find what are the documents that share similar like topics.

Can you please suggest/recommend suitable clustering technique for me to start with

PS: I am interested in something like this -> https://www.quora.com/How-do-I-cluster-documents-using-topic-models
However, the topics assigned for my documents varies


texts: why does not CBOW of word2vec include center word in context

texts: In CBOW, each word has input vector and output vector, or context vector and self vector,  the context aggregate the surrounding words, what if the context also contain the input vector of the center word?


texts:
I was reading this paper and came across the below paragraph. Can you please help me understand what does the highlighted term noise-tolerant learning or noisy-labeled training data mean with a simple example and how is it useful when we don't have labels in our dataset etc? I am learning ML and your inputs would be helpful

To address the scarcity of labeled training data, Chen et al used
active learning to intelligently select training samples for labeling,
demonstrating that classifier performance could be preserved with
fewer samples.16 Another trend is the use of “silver standard training
sets,” a semisupervised approach where training samples are labeled
using an imperfect heuristic rather than by manual review.17–22 The
intuition is that noise-tolerant classifiers trained on imperfectly
labeled data will abstract higher order properties of the phenotype
beyond the original labeling heuristic (so-called “noise-tolerant
learning”23).


texts: Dot products are pretty simple for 1- or 2-dimensional arrays, but anything beyond that is incomprehensible to me. I tried looking into numpy‘s dot function but the C code is incomprehensible to me. What is the formula numpy is using to calculate the inner product of multidimensional arrays?


texts: How exactly dependent variable is expressed in terms of independent variables using Partial Least Square Regression Method?

texts:
I understand the working of NIPALS algorithm but while doing the regression using PLS how exactly the relation between known and unknown is established using Principle Component Analysis.
The idea is clear up to the points both dependent and independent are expressed in form of their principle components. After I am not able to understand.

PS: I have read text and some papers about it but they are very difficult to understand. This Question may sound a trivial one, but please bear with me. Correct me if I am wrong.


texts: Optimum minimum number of instances in weka's j48

texts: There is a parameter named minnumobj in the options of the j48 tree algorithm in weka. This parameter indicates the minimum number of participants to be in a leaf. Does this parameter have an optimum value regarding the instances or attributes? Or should I set the value that achieves the highest classification success? For example, I have 6500 instances and 9 attributes and I got highest success when minnumobj=100.


texts: Creating a loss function with two minima?

texts:
I'm using keras to build a model. The model has one output, and I want a loss similar to mse loss, but there are two values to predict, and I'm fine if the model predicts either one of them, so I want the loss function to look something like this

(p.s. the graph doesn't have to be symmetrical)

Does any function like this exist? If not, how do I make one


texts:
I have a big table (150 Million rows and ~ 70 columns).
In three of the columns in the table I have text input (3-20 words/column), which I need to use for a classification algorithm.

For smaller datasets, I have used the tm R package and created a DocumentTermMatrix, where I used the frequency of word (or word parts) as predictors in a SVM & Decision Forest algorithm.

However, now my dataset is much bigger, which is why I converted it to a xdf file and used RevoScaleR packages for merging, joining etc. so far.

However, I am unsure how to do text mining with very large datasets (e.g., create a document term matrix which I can use for classification). I did not find any pre-build functions. It may be possible to create a document term matrix in a chunk of data, and then somehow sum the frequencies. I am not sure if that is the best way, could you help me with that?

Thank you,
Lisa


texts: Is it advisable to merge similar datasets to improve model accuracy?

texts:
I'm trying to build a classifier that would help me classify whether a statement collected from Reddit is bullish, bearish or neutral.
To this end, I have hand-labelled a fairly small dataset of 2500 entries, each with max 280 characters. Unfortunately, the data is unbalanced (60% neutral, 25% bullish and 15% bearish) and my initial attempts are returning poor results for the bullish and bearish classes.
I've managed to obtain 10,000 entries of similar data from StockTwits, each labelled bullish or bearish (no neutral...) and 280 characters max.
I tested supplementing my Reddit dataset with bullish/bearish data from this dataset to balance out the classes. In other words, I added data from the StockTwits dataset until the number of bullish and bearish entries in matched the neutral. With this model, I'm getting much better results.
Is merging similar datasets in this way advisable? My gut instinct says "no" but I haven't found anything suggesting this is not a good idea.


texts: Performing Binary classification using binary dataset

texts: Can a dataset with only  binary digits 0's and 1's perform good classification such as decision tree and naive Bayes


texts: Not able to get a good accuracy score for the classification problem

texts:
I have taken a music popularity dataset which has five classes based on the popularity of the songs.I have made a Random forest model to predict the popularity of a given song(given its features).I have done feature engineering like feature scaling and feature selection. Without doing feature selection, the accuracy that I got was around 0.625
But even after doing feature selection and feature scaling the accuracy did not improve even a bit.It was still around 0.62.
What have I done wrong?
Is there any other way by which I can increase my model's performance?


texts: Best structure for a LSTM Bert sentence classifier

texts:
I'm interested in classifying sentences using BERT. Finetuning on a single sentence had very poor results. I'd like to add a forward and backward LSTM layer to try to improve results. I'm having trouble figuring out the best structure for an efficient implementation.

My first idea is to implement it as normal, but use a tf while loop to get all the BERT embeddings, and then feed them into the output. This seems really inefficient because I'll be calculating way more embeddings than I need.

The next would be to pre calculate all BERT embeddings and then feed those as input to my LSTM and classifier layer. Using this I think it's clear how to train the LSTM and classifier model, but how would I train the BERT Model as well? Is this a common method of constructing / training models? Any references?

Finally the last idea was to use the 2 sentence BERT embedding. It appears it would be limited to a 3 sentence window - previous sentence, current, and next. So scaling up might not work if the model still does not perform that well. Another issue is that it's unclear, at least to me, if the pooled output of a sentence

[cls] sentenceA [sep] sentenceB [sep]

gives a representation of sentenceA, of both of them, or of sentenceB 'given' sentenceA. The pooled output is the representation of [cls] token, but in the paper / github there was no reference to if the [sep] representation encodes anything at all.

Any help, insights, or similar work would be appreciated!


texts: How to get intuitive understanding which deep learning architecture suits for my problem

texts:
I'm working on a research problem where I need to perform classification for coarse prediction in a feature space and then fine grained regression for getting more precise values. I know that this way of regression should work. I also will essentially deal with feature maps.

I am thinking of using a 'stacked hourglass network'. Do I need to identify this by sheer experimentation or can someone intuitively remove some possibilities saying a particular architecture may not be suitable for my problem.

I found stacked hourglass network to upscale and downscale essentially the heatmaps but now am confused with changing the model for sequential classification and regression task. Any clues would be welcomed.

Stacked Hourglass Networks for Human Pose Estimation


texts:
What are the pros / cons of removing stop words from text in the context of a text classification problem, I'm wondering what the best approach is (i.e. to remove or not to remove)?

I've read somewhere (but can't locate the reference) that it may be detrimental the the performance of a model in the case of sentiment analysis to remove stop words.


texts: Difference between MultiDiGraph and DiGraph

texts:
This post was originally posted to stackoverflow (https://stackoverflow.com/questions/57315017/best-way-to-include-weights-in-a-directed-acyclic-graph) sorry for long link, goo.gl was unable to shorten it.

Someone recommended I post it in this forum as it seemed more suitable.

I am working on my thesis, which consists of writing a Tabu Search algorithm to solve a Flexible Job Shop Problem. To solve this problem, I am using graphs as a problem/solution representation (the problem is an Undirected Graph, which must be turned into a Directed Acyclic Graph).

In short, I have a start (0) and end (*) node, which are 'dummy' operations whose weight/duration is 0. Nodes represent operations and paths represent the machines on which operations are processed, so the length of an edge (u,v) is the setup time incurred when swapping between operations u and v on a given machine. The weight of a node should be the processing time of that operation on the machine it's assigned. In a Directed Acyclic Graph, the length of the longest path from 0 to * represents the makespan (maximum production time) of the solution.

To implement this, I am using the networkx library. It's been going well, but I have come across some issues, namely when computing the makespan.
First of all, since the functions I found (dag_longest_path() and dag_longest_path_length()) don't account for node weights, I transfered node weights to the edges preceding them. That seemed to solve the problem.
However, I just noticed that these functions have different outputs in case the graph is either a DiGraph or a MultiDiGraph, as shown below.

Before presenting the code, forgive my noobness here, first time poster. The instance I am using consists of 30 operations and 8 machines, but for simplicity purposes I am presenting a simple example with 5 operations and 3 machines.

import networkx as nx

operations = (1,2,3,4,5)
G=nx.DiGraph()
G.add_node(0,pos=(0,0)) #starting node
G.add_node('*',pos=(1,0)) #end node

G.add_nodes_from(operations)

G.add_edge(0,1, weight = 500)
G.add_edge(1,2, weight = 200)
#these edges are for one machine

G.add_edge(0,3, weight = 150)
G.add_edge(3,4, weight = 177)

#these are for a second machine

G.add_edge(0,5, weight = 999)
#this edge is for the last machine

G.add_edge(2, '*', weight = 0)
G.add_edge(4, '*', weight = 0)
G.add_edge(5, '*', weight = 0)
#edges connecting final operation of each machine to the end node

print(nx.dag_longest_path(G))
print(nx.dag_longest_path_length(G))
#printing the longest path and corresponding length

The previous code outputs the following:

[0, 5]
999

This is sort of expected, as the path going through node 5 is indeed the longest path. However, it does not show the end node '*'.

If instead of G=nx.DiGraph() I do G=nx.MultiDiGraph(), the output is the following:

[0, 1, 2, '*']
3

Now the '*' node is included, rightfully so, but the path returned is the one with most nodes instead of the one with the longest edges.

What am I missing here? MultiDiGraph() seems to be the correct graph type here, as nodes 0 and '*' both have multiple edges leaving/entering them.

I need a way for the output to be:

[0, 5, '*']
999

Thank you in advance!

PS: feel free to correct any errors in formatting and presentation as this is my first post


texts:
I am working on developing a data dashboard / app for a Microsoft SQL database. Currently, I am developing the dashboard using Shiny and R. The app is mostly for exploratory analysis to allow people to filter out some subset of data, build some plots, and export data / plots.

My question is what are some other options for creating this type of dashboard? I am aware of Tableau.


texts: What is the best approach for classifying non-English text

texts:
What would be the best approach for classifying non-English (Sinhala / Tamil) text? Currently I use Fasttext. Are there any better options?

I want to classify user questions into chatbot intents. Therefore, there may be many target classes.


texts: How to annotate the object more efficiently

texts:
I want to perform object detection and object counting on the given below image using neural networks.
My first step will be to annotate and label each object present in the training set of images. In my case, I want to detect the black metal objects present in the image. So as you can see there are many instances of an object in a single image and hence labeling tasks would be very tedious and time-consuming for me.
I want to know whether I need to draw a bounding box around each object while labeling the images or labeling a few of them will do my job?
Also is there any better method to annotate data or is there any better way to solve this type of problem?


texts: Understanding Bernoulli Trials, Bayesian Setting

texts:
I am required to complete a project on ML applications. I guess there is a lot of statistics in ML, not helpful for a non-maths background.

I am getting too bogged down by notations. There are too many notations.

I am trying to read about what Bernoulli trials are and I can't relate to it.

What is a Bayesian setting and why is Bayesian thing everywhere? What makes it such an omnipotent distribution?

Are there any theorems/theories that I must know ? Any book/notes/resources where I could learn about these stuff in a relatable way (I have very little maths background, but I can learn)?


texts: Which other algorithms fit residuals like XGBoost?

texts:
XGBoost and standard gradient boosting train learners to fit the residuals rather than the observations themselves. I understand that this aspect of the algorithm matches the boosting mechanism which allows it to iteratively fit errors made by previous learners.

Which other algorithms or also train single or multiple learners to fit residuals?
Does this method only make sense for learners built in a sequence? Or also for any ensemble methods?
Is there a deep significance to fitting residuals or is this a mathematical convenience that allows for iterative improvement from learners built in succession?


texts: Response variable with very high number of class levels in R

texts:
I've got a dependent variable in R with about 11,000 class levels. There's ample data for modeling this many classes, however some statistical packages like h2o only support up to 1,000 classes or so.

What are the best options for classification models in R with this high number of class levels?


texts: Resume Parsing - extracting skills from resume using Machine Learning

texts:
I am trying to extract a skill set of an employee from his/her resume. I have resumes stored as plain text in Database. I do not have predefined skills in this case. How should I approach this problem?

I can think of two ways:

Using unsupervised approach as I do not have predefined skillset with me.
I will extract the skills from the resume using topic modelling but if I'm not wrong Topic Modelling uses BOW approach which may not be useful in this case as those skills will appear hardly one or two times. (I would appreciate if you enlighten me more about topic modelling).
Another approach is manually labeling the skills for resume and making it supervised learning problem. But I have around 500 resumes, manual labeling will be very tedious and very time consuming.

Any suggestions are welcome.

Thanks.


texts:
For the logistic regression below, how can I manually update the coefficients a and b manually?
EDIT
y = 1.0 / (1.0 + exp(-ax - b))

after observing the following data examples (x, y), the parameters a and b need to be updated using gradient descent with a learning rate = 1.
1.
input: x = 1, y = 1, a = 1, b = 1
expected output: a = 0.881, b = 0.881
2.
input: x = 2.2, y = 0, a = 5.1, b = 5.7
expected output: a = 7.3, b = 6.7
UPDATE
def update(x, y, a, b):
    t = 1 + np.exp(-a * x - b)
    da = (y - 1 / t) * x * (t - 1) * t**-2
    db = (y - 1 / t) * (t - 1) * t**-2
    new_a = a - da
    new_b = b - db
    return new_a, new_b

This got me
update(1, 1, 1, 1)
(0.9874844578263232, 0.9874844578263232)

Did I miss something here?
FINAL
def manual_update(x, y, a, b):
  p = 1 / (1 + np.exp(-a*x-b))
  pdpda = x*np.exp(-a*x-b) / (1+np.exp(-a*x-b))**2
  pdpdb = np.exp(-a*x-b) / (1+np.exp(-a*x-b))**2
  pdLda = -y*pdpda/p+(1-y)*pdpda/(1-p)
  pdLdb = -y*pdpdb/p+(1-y)*pdpdb/(1-p)
  return round(a + pdLda, 3), round(b + pdLdb, 3)

this worked perfectly.


texts: Using StyleGAN for generative design of aircraft

texts:
I haven't worked on a machine learning project in a year and now that I am in university I am trying again :)
Anyway, the end goal is to create a GAN that can design aircraft. I was inspired by the Czinger21c because it was said that an AI generated the car. To me, this sounds like a GAN at work, but I personally have no experience working with GANs and I'm looking for a bit of guidance as well.
I've looked for a bit, and I saw a video about StyleGAN for faces. I was wondering if I can modify this for use with a parametric study; I want to have the GAN learn optimized aircraft and create its own.
I have experience working with CNNs, so I know I can do this, except I'm just working with two neural networks.
Thank you


texts: number of neurons for mnist dataset using mlp?

texts:
I am trying to find out what is optimum number of neurons that can be used in MNIST dataset(60,000 training and 10,000 testing data). I build a single hidden layer model using keras,with relu activation but when I increase the neurons in hidden layer the accuracy improves slightly nearly .4% per 100 neurons ,even if I use 2000 neurons instead of 784 neurons it doesn't has any considerable change.I thought that increasing neurons above certain limit will decrease the classification accuracy because of overfitting.but It doesn't seem to look happening I even tried using 2000 neurons but the accuracy doesn't decrease on the test data.exceeding this limit I think is not possible for me because it can crash my computer.

Is hit and trial the only way to find the optimum number of neurons in a MLP hidden layer. How can I further tweak the parameters , can it be some other hyper parameters that I can play with like intialzing hidden layer weights initially.or adding more layers.(but adding more layer doesn't help much) I tried it.


texts: Explain output of a given classifier w.r.t features

texts:
Given a binary classifier, is it always possible to explain why it has classified some input as a positive class ? And by that I mean, if we have a big set of features, is there a tool that says : 'For this output, these are the features that were the most responsible for labeling it as a positive' ?

Thanks  !


texts: How to get summary statistics in Orange?

texts:
I just started using Orange, and am having trouble finding how to get basic summary statistics, like the n (count), average, and standard deviation.

Is there a widget that does this and I'm simply overlooking it?


texts: What is the difference between CountVectorizer() and Tokenizer() or are they the same?

texts:
from sklearn.feature_extraction.text import CountVectorizer

from keras.preprocessing.text import Tokenizer

I am going through some NLP tutorials and realised that some tutorials use CountVectrizer and some use Tokenizer. From my understanding, I thought that they both use one-hot encoding but someone please clarify this.
What I don't understand is why CountVectorizer is not used on Deep Learning models such as RNN and Tokenizer() is not used on ML Classifiers such as SVM, Naive Bayes.
Thank you


texts: Why is it important to have sufficient number of instances in your dataset for each stratum?

texts:
As per the figure 1, most of the median-income values are clustered around \$20,000-\$50,000, but some median incomes go far beyond \$60,000.

I didn't understand the explanation behind why housing['median_income'] has to be divide by 1.5

 housing['income_cat'] = np.ceil(housing['median_income'] / 1.5)

  Explanation - It is important to have a sufficient number of instances in your dataset for each stratum, or else the estimate of the stratum's importance may be biased.

Can someone help me to understand the explanation and also why is it only 1.5 ?. As per the explanation, why the estimate of stratum's importance will be biased when there is no adequate instances for each category ?


texts: When should you use learning rate scheduling over adaptive learning rate optimization algorithm?

texts:
In order to converge to the optimum properly, there have been invented different algorithms that use adaptive learning rate, such as AdaGrad, Adam, and RMSProp. On the other hand, there is a learning rate scheduler such as power scheduling and exponential scheduling.

However, I don't understand at what kind of situations you should use one over the other. I feel that using adaptive learning rate optimization algorithm such as Adam is simpler and easier to implement than using learning rate scheduler.

So how can you use it apart properly, depending on what kind of problems?


texts: Help interpreting GAN output, and how to fix it?

texts:
After a few tries, I had trained a GAN to produce semi-sensible output. In this model, it almost instantly found a solution and got stuck there. The loss for both the discriminator and generator were 0.68 (I have used a BCE loss), and the accuracies for both went to around 50%. The output of the generator looked at first glance good enough to be real data, but after analysing it I could see it was still not very good.
My solution here was to increase the power of the discriminator (increased the size of it) and re-train. I hoped by making it larger it would force the generator to create better samples. I got the following output.

It seems that as the GAN loss increases, and is producing worse samples, the discriminator can pick it out more easily.
When I check my output from the trained generator I see it follows some basic rules the real data is following, but again under closer scrutiny, they fail more complex tests the real data would pass.
My questions are:

Is my above interpretation of the plots correct?
For this run, have I made the discriminator to powerful? Should I increase the power of the generator?
Is there another technique I should investigate to stop this form of mode collapse?

Thank you
EDIT:
The architecture I am using is a form of Graph GAN. The generator is just a series of linear layers. The discriminator is 3 Graph Conv Layers, then some linear layers. Slightly similar to this paper.
Two potentially unconventional things I am doing:

There is no batch normalisation, I have found this has a very negative effect on the training. Though I could try and persevere with it.
I am using StandardScaler to scale my data. This choice was made as it easily allows you to unscale data. This is useful as I can take the output of the generator and easily transform it into an original scale. However, StandardScaler does not scale things between 1 and -1, so I cannot use tanh as the final activation function of my generator, instead the final layer of the generator is just Linear.

The outputs of the GAN (once rescaled and the shape has been changed) are similar to:
[[ 46.09169   -25.462175   20.705683  -31.696495 ]
 [ 35.10637   -18.956036   15.20579   -24.803787 ]
 [ 10.253135   -5.759581    5.9068713  -6.3003526]]

An example of the truth is:
[[ 45.6         30.294546   -17.218746   -29.41284   ]
 [  1.8186008    1.7064333    0.5984112    0.19312467]
 [ 44.31433     28.234058   -17.615921   -29.262213  ]]

Notably, the top-left value in the matrix will always be 45.6. My Generator does not even consistently produce this.


texts: How is the modulo number selected to build the hash table in DHP algorithm?

texts:
I'm trying to understand the DHP(Direct Hashing and Pruning) algorithm and I got stuck at explaining the selection of modulo number.
The paper shows an example of using the hash function at page 7: h{{x y}) = ((order of x)*10 + (order of y)) mod 7

My questions are:

What are the basis to define the function this way?
How is the modulo number selected(7, in this example)?


texts: Evaluating performance of classifier on lopsided dataset

texts:
I have a binary classifier that I would like to evaluate the performance of. It's been both trained and tested on a data set where the ratio of true to false labels is lopsided. This means that while it's quite poor at correctly guessing true, its overall performance on the test set looks very good when using a metric such as right_guesses/total.

What is a better metric to use? Preferably, one where the true false labels account for the same percentage of the score although their numbers are unequal.


texts: Does gensim use Negative sampling in Word2vec?

texts:
When I train a word2vec model in Gensim on a huge amount of words/data (let's say hundreds of thousands of word vectors), is gensim using negative sampling automatically?

Alternatively, is there a way to make it use it?


texts: Does the Bishop book imply that a neuron feeds to itself in chapter 5.3?

texts:
I just read Bishop's book Pattern Recognition and Machine Learning. I read the chapter 5.3 about backpropagation, and it said that, in a general feed-forward network, each unit computes a weighted sum of its inputs of the form $$a_j=\sum\limits_{i}w_{ji}z_i$$

Then the book says that the sum in the above equation transformed by the non-linear activation function $h(.)$ to give the activation $z_j$ of unit $j$ in the form $$z_j=h(a_j)$$

I think the notation is somehow akward: suppose I want to compute $a_2$, then
 $$a_2=w_{21}z_1+w_{22}z_2+\dots$$

Then does  $$a_2=w_{21}z_1+w_{22}h(a_2)+\dots$$ mean that the neuron $a_2$ is connected to itself?


texts: Assigning scalar values for PID for order in Neural Network

texts:
I have built a neural network using Windows Process's I started off with only two features, the file path with parent process, and the file path with child process.  I am slowly adding features for instance I just added the destination user name independent variable so my dataset looks like this before it goes through label encoding and dummy variables:

    ParentProcess                           ChildProcess           Suspicous
0   C:\Program Files\Splunk\bin\splunkd.exe C:\Program Files\Splunk\bin\splunk-optimize.exe 0
1   C:\Windows\cmd.exe  C:\Windows\System32\backgroundTaskHost.exe  1
2   C:\Program Files\Splunk\bin\splunkd.exe C:\Program Files\Splunk\bin\splunk-optimize.exe 0
3   C:\Program Files (x86)\Citrix\ICA Client\SelfS...   C:\Windows\System32\book.pdf ...    1
4   C:\Windows\System32\svchost.exe C:\Windows\System32\gpupdate.exe    0
7   C:\Windows\System32\cmd.exe C:\Windows\System32\cscript.exe 0

I would like to add process ID's available to me as features or in pre-processing somehow so that the model can learn whether the order kicks off normally or is anomalous.  It would sort of look like a tree.  Where smss.exe kicks off smss.exe, csrss.exe, wininit.exe, and winlogon.exe.  The smss.exe and csrss.exe processes die but winitit.exe and winlogon.exe that were children become parents of other processes, all tracked by process id's.  I would like to assign smss.exe a 1 then crss.exe, wininit.exe, and winlogon.exe a 2 and so on.  Some processes will be 3 and so on and some processes will die at 2.

Example:

smss.exe (1) PID = 1050 -> winit.exe (2) PID = 18888, winlogon.exe (2) PID = 9522
winit.exe (2) PID = 18888 -> services.exe (3) PID = 20000, lsass.exe (3) PID = 20001
winlogon.exe (2) PID = 9522 -> userinit.exe (3) PID 11224
services.exe (3) PID = 20000 -> svchost.exe (4) PID 25001

Is there a way to use something like this in a neural network kind of like you would in a time series?  Or have the neural net learn these numbers?  Or have seen something like this being done like an m-tree style or scalar value for depth?


texts: We have a data set of n variables (profile attributes) and want to feed through a model, and classify into M buckets (functionally signifying some action to be performed) .  Which MLmodel/ algorithm is best suited for this. Can someone point me some code.


texts: NLP algorithm: sentiment with specific guidelines

texts:
So I have this situation, I have filtered a bunch of single independent sentences that I filtered because they contain the word X (in my case, X = "budget"). if the meaning of the sentence is "budget increases/goes up/etc" my result is 1 and, if not, 0. all sentences hold such meaning (the word budget is not used in different ways)

this is not a traditional "sentiment" problem and I do not know how to approach this. I could run into a variety of cases ("budget did not go down" // "A budget cut was in order" // "A budget cut was not in order")

if required, I can categorize by hand up to 400/500 sentences and say with certainty the result (0, 1). could you point me to any model that would solve this? would the amount of hand-labeled samples be enough?


texts: How to plan a model analysis that avoids overfitting?

texts:
Coming from statistics, I'm freshly trying to learn machine learning. I've read a lot of tutorials about ML, but have no real training.

I'm working on a little project where my dataset have 6k lines and around 300 features.

As I've read in my tutorials, I split my dataset into a training sample (80%) and a testing sample (20%), and then train my algorithm on the training sample with cross-validation (5 folds).

As I re-ran my program twice (I've only tested KNN which I now know is quite not appropriate), I got really different results, with different sensitivity, specificity and precision.

I guess that if I re-run the program until metrics are good, my algorithm will be overfitted, and I also guess it would be because of the resample of test/training samples, but please correct me if I'm wrong.

If I'm going to try a lot of algorithms to see what I can get, should I fix my samples somewhere? Is it even OK to do try several algorithms? (it would not always be in statistics)

In case it matters, I'm working with python's scikit-learn module.

*PS: my outcome is binary and my features are mostly binary, with few categorial and few numeric. I'm thinking about logistic, but which algorithm would be the best one ?


texts: What can we understand from max-activation generated images?

texts:
There are several approaches to generate psychedelic images, providing maximum activations for individual neirons in convolutional neural networks. For example there is a lot of them there https://app.slack.com/client/T040HKJE3/threads/thread/C04655480-1581762147.250800?cdn_fallback=2 or a bit in https://arxiv.org/abs/1311.2901 (M. Zeiler) or in https://www.youtube.com/watch?v=ghEmQSxT6tw But what can we get from them?

E.g. here
does presence of 6 fish’es pieces means there are mainly 6 different positions of fishes which the net saw on 1000 train images? Or this is rather caused by some corner effects or patter size? Like 224 source pixels/100 = 2.2 fishes horizontally.

I also don't understand well how they are obtained, so can't conjecture what aspects they can depict.

There is approach when we find parts of train/validation images which gives strongest activations. This is perfectly understandable for me.


texts:
I performed Linear Regression on the data set with two attributes (Salary and YearsExperience) using orange data mining tool.

Salary is a dependent variable and YearsExperience is an independent variable.

Can I find the equation : Salary = b0+YearsExperience*b1.


texts: CNN for classification giving extreme result probabilities

texts:
I'm having issues with my CNN, using Keras with Theano backend.
Basically, I need to classify 340x340 grayscale images into 6 categories. The problem is my CNN gives too "hard" probabilities, for instance it will rarely give predictions with some uncertainty, and always tries to push for a 90%+ for one class. The problem is that for my coursework, the penalty used is very harsh for complete miss classification, and uncertainty is much preferred. ( so having a prediction like [0.6, 0.3, 0.2, ...] is much better than having [0.9,0.03,0.02,..].

I'm unsure why this is happening. My dataset consists of 2400 images,
which are from different CCTV, and task is about recognising possible objects. Only 800 of the samples are actually from the data, the other 1600 have been generated through data augmentation.
Note that it is therefore extremely likely a that some pictures are either identical, or extremely similar (e.g. the same scene, one second later)

model = Sequential()
#1
# Few filter to take big stuff out
# Also, first layer is not conv so that I can reuse that layer separately
model.add(Dropout(0.1, input_shape=(1,340,340)))
model.add(Convolution2D(64, 4, 4, border_mode='same'))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2), dim_ordering="th"))

#2
model.add(Dropout(0.1))
model.add(Convolution2D(128, 4, 4, border_mode='same'))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2), dim_ordering="th"))

#3
model.add(Dropout(0.1))
model.add(Convolution2D(256, 4, 4, border_mode='same'))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2), dim_ordering="th"))

#4
model.add(Flatten())
model.add(Dense(1024))
model.add(Activation('relu'))
#5
model.add(Dense(512))
model.add(Activation('relu'))
#6
model.add(Dense(6))
model.add(Activation('softmax'))

opt = SGD(lr=0.001)
model.compile(loss='categorical_crossentropy', optimizer=opt, metrics=['accuracy'])

print "Training.."

filepath = "log/weights-improvement-{epoch:02d}---{val_acc:.2f}.hdf5"
checkpoint = ModelCheckpoint(filepath, monitor='val_acc', verbose=1, save_best_only=True, mode='max')
callbacks_list = [checkpoint]
model.fit(X_t, y_t, validation_split=0.1, nb_epoch=500, batch_size=32, callbacks=callbacks_list)

How do you suggest I fix this?
Thank you in advance!