texts: Which Amazon EC2 instance for Deep Learning tasks?

texts:
I have discovered that Amazon has a dedicated Deep Learning AMI with TensorFlow, Keras etc. preinstalled (not to mention other prebuilt custom AMIs). I tried out this with a typical job on several GPU-based instances to see the performances. There are five such in the Ireland region (maybe in other regions exist even more, I don't know, this variance is a bit confusing):

g2.2xlarge
g2.8xlarge
p2.xlarge
p2.8xlarge
p2.16xlarge

My first question is, what is the difference between the two groups (g-something and p-something)? Both group mentions "GPUs along with CPU", but no further clue for Deep Learning usability.

My second problem is that I have been running my job on g2.2 and g2.8 as well, and while the task processing toke quite long time to run, the workload of the GPUs was relatively low (20-40%). Why didn't the framework increase the workload if there is spare processor capacity? Is is necessary/possible to paramter/set anything to optimize the work?


texts: i got the following error : 'numpy.ndarray' object has no attribute 'detectMultiScale' to show image in python?

texts:
import cv2

videoCam = cv2.VideoCapture(0)

face = cv2.CascadeClassifier('haarcascade_frontalface_default.xml')
eye = cv2.CascadeClassifier('haarcascade_eye.xml')
nose = cv2.CascadeClassifier('haarcascade_mcs_nose.xml')
mouth = cv2.CascadeClassifier('haarcascade_mcs_mouth.xml')

while True:
    cond, frame = videoCam.read()
    frame = cv2.resize(frame, (154, 154))

    gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
    muka = face.detectMultiScale(gray, 1.3, 5)
    for (x,y,w,h) in muka:
        cv2.rectangle(gray, (x,y), (x+w, y+h), (0, 255, 0), 5)

        roi_warna = frame[y:y+h, x:x+w]
        roi_gray = gray[y:y+h, x:x+w]
        mata = eye.detectMultiScale(roi_gray)
        nose = nose.detectMultiScale(roi_gray)
        mouth = mouth.detectMultiScale(roi_gray)
        for (mx,my,mw,mh)in mata:
            cv2.rectangle(roi_warna, (mx,my), (mx+mw, my+mh), (255,255,0), 2)
        for (nx, ny, nw, nh)in nose:
            cv2.rectangle(roi_warna, (nx, ny), (nx + nw, ny + nh), (0, 0, 255), 2)
        for (bx, by, bw, bh)in mouth:
            cv2.rectangle(roi_warna,(bx,by),(bx+bw,by+bh),(0,0,0),2)

    cv2.imshow('Face dan Eye detection', frame)

    k = cv2.waitKey(1) & 0xff
    if k == ord('q'):
        break

videoCam.release()
cv2.destroyAllWindows()


texts: Is there room to improve the model? if the train data accuracy is 99.8% but test data accuracy is 90%?

texts:
I understand this is a wide question.

But there can be some suggestions.

I can try some methods which I do not know.

I think the model is already prefect on train data. But the test accuracy is a little low. So do you think there still can be some improvement on the deep model?


texts: Word Embedding for Item Names(integer, one-hot encoding)

texts:
I am looking for the way to get the similarity between two item names using integer encoding or one-hot encoding.
For example, "lane connector" vs. "a truck crane".
I have 100,000 item names consisting of 2~3 words as above.
also, items have its size(36mm, 12M, 2400*1200...) and unit(ea, m2, m3, hr...)
I wanna make (item name, size, unit) as a vector. To do this, I need to change texts to numbers using some way. All I found is only word2vec things, but my case has no context corpus. So I don't think it is possible to learn some context from my data.


texts: I am looking for a dataset, where I could use NLP techniques to estimate target value using regression. For example, I could be give a few sentences that describe an accident, and the target value would be the cost of an accident. Kaggle has quite a few datasets for NLP, but they  all, as far as I can see, for classification.


texts: How to calculate size and offset of YOLO grid in a fully convolutional network with zero padding?

texts:
Fully convolutional network with zero padding:

I have a fully convolutional network which does not have any padding in convolutional layers. This implies that, after each convolution operation, the output feature map is smaller than the input feature map. No padding convolutional layers are crucial for my application since I wish to have strict translation invariance.

Following are some important points about my CNN:

It only consists of max-pool and convolutional layers.
The stride of my CNN is 8 which is calculated by multiplying the strides of max-pool and convolution layers.
If I pass a 255x255 image, I get an output map of 17x17x5 (4 bounding box values + confidence score).

Comparison with YOLO:

YOLO fits the output grid simply on the whole image. I guess they are able to do so because they maintain feature size in convolutional layers with padding. Since I don't have any padding in my convolutional layers, I know that the output corresponds to some part of the image with border pixels getting washed away.

Key questions:

I wish to fit the output grid (17x17) in the original image space (255x255) but I can't figure out the convolutional arithmetic. I have two questions here:

What should be the offset of the grid in the original image from top left?
What are the dimensions (width and height) of each grid cell?

I guess the answers to above two questions should be functions of network stride $s$, input image size $d_i$ and output size $d_o$ (feel free to prove me wrong). Just to be clear, in my fully convolutional network, $s$ = 8, $d_i$ = 255 and $d_o$ = 17.


texts:
I am working with a machine learning approach that counts cars in images. I have a predicted dataset, which is the predicted output from the machine learning approach and a paired "true" dataset, which is the result of a human going through each image and counting the number of cars.

The following is a sample of what the datasets look like (note that the actual dataset has 2500 paired samples):

import pandas as pd

d = {'true': [0,0,0,1,1,0,1,0,0,0,0,0,0,0,4,2,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1],
     'predicted': [0,0,0,0,0,0,1,0,0,0,0,0,0,0,2,2,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,1]}
df = pd.DataFrame(data=d)

    true  predicted
0      0          0
1      0          0
2      0          0
3      1          0
4      1          0
5      0          0
6      1          1
7      0          0
8      0          0
9      0          0
10     0          0
11     0          0
12     0          0
13     0          0
14     4          2
15     2          2
16     0          0
17     0          0
18     0          0
19     0          0
20     0          0
21     0          0
22     0          0
23     0          0
24     0          1
25     0          0
26     0          0
27     0          0
28     0          0
29     0          0
30     0          0
31     0          0
32     1          1

I am looking for a way to present the predicted approach to an audience so that they see if the predictions are statistically the same as the true observations and visualize any trends in the data (e.g. the predicted approach has a tendency to over or under predict). If these were categorical data, I would use a confusion matrix, however, I am not sure how to deal with these paired, discrete datasets that are heavily weighted with 0's.

What approach can I take to statistically compare the predicted vs true datasets?


texts: How to cache GridSearchCV optimizer result in Google Colab?

texts:
I have to execute GridSearchCV() cell every time I reload the page and it takes a lot of time. Is there a caching mechanism which stores the GridSearchCV result so that I can use it without executing the cell on the reload?

Code:

 neigh_clf = KNeighborsClassifier()
 grid_search = GridSearchCV(neigh_clf, param_grid, cv=5,verbose=3,n_jobs=-1)
 grid_search.fit(X_train, y_train)
 grid_search.best_params_

I am trying to save grid_search object here so that I can retrieve best_params_ value after reload


texts: Remove columns with a certain number of consecutive zeros

texts:
I'm using pandas and I am dealing with time series of sales.

What I would like to do is to remove the columns where a certain number of consecutive zeros appear, since forecasting for sparse series or repeated zero values tend to be unreliable. My dataframe has a bit more than 4400 time series for a time span of 5 years.

I have tried applying this strategy from a previous SO question but I can't drop the columns fullfilling the condition.

Below is an example where the threshold would be 12. Any column having 12 or more consecutive zeros would be discarded.

input :

      A   B   C

0     1   1   8
1     2   0  14
2     3   0  20
3     0   0  15
4     0   0  23
5     0   0  25
6     0   0  22
7     0   0  18
8     0   0  16
9     0   0  14
10    0   0   8
11    1   0  10
12   18   0   7
13   34   0   4
14  110   1  14
15   11  30   2
16    0  24   8
17    0  11   7
18    1  22  11
19    0  33   3
20    0  90  12
21   12  32  19
22   11  90  17
23   77  13   2

desired output:

      A   C

0     1   8
1     2  14
2     3  20
3     0  15
4     0  23
5     0  25
6     0  22
7     0  18
8     0  16
9     0  14
10    0   8
11    1  10
12   18   7
13   34   4
14  110  14
15   11   2
16    0   8
17    0   7
18    1  11
19    0   3
20    0  12
21   12  19
22   11  17
23   77   2

Thanks.


texts: Is it necessary to convert labels in string to integer for scikit_learn and xgboost?

texts:
I have a tabular data with labels that are in string. I will feed the data to decision trees in scikit_learn and XGBoost classifier.

Is it necessary to convert the labels in string to integers for these algorithms? Will the algorithms work less effectively when the labels are not in numerical format?

I am using python v3.7


texts:
I'm struggling to write my algorithm in a concise and correct way.
The following is an explanation for an optimizer's update step of part of a vector of weights (not a matrix in my case).

I have a vector $\alpha \in \mathbb{R}^d$, and a set $S$ that includes some indices $1\leq i \leq d$ ($S \subseteq \{1,\dots, d\}$).
Now, I want to denote that $\alpha$ is 0 for every index $i\in S$, and otherwise it's the value as in $\alpha_i$.
At first I denoted it $\alpha_S$, but I'm not sure it is properly defined or understandable.

I could use the following notation:

$\alpha_S =
  \begin{cases}
    \alpha_j        & j \in S\\
    0               & j \notin S
  \end{cases}$

But its line height is twice the size, and I want to avoid that.

Is there any other formal, simplistic way to notate this correctly? Maybe some kind of a masking vector to be multiplied with $\alpha$?

Thanks!


texts: Reduce number of vectors in dataset to achieve the "same average dimensions result"?

texts:
Edit for re-opening the question, I'll try to answer questions made by @user2974951:
I have a large user preference statistics for trichotomic data sets. You can visualize each data trio as a 3D vector with X, Y and Z values. All vectors complies to X + Y + Z = 1 because of the trichotomous shape of the data I'm using. It can also be visualized as a points in an equilateral triangle.
I have many tests, each with a large set of 3D vectors (features).
Simply averaging all components from X1 to Xn:
X = (X1+X2+X3+...Xn)/n
Y = (Y1+Y2+Y3+...Yn)/n
Z = (Z1+Z2+Z3+...Zn)/n

This gives me some averaged X,Y,Z for each test
           feature1     feature2     feature3               Average
test1->   X11,Y11,Z11  X12,Y12,z12  X13,Y13,Z13  ...  ->    X1,Y1,Z1
test2->   X21,Y21,Z21  X22,Y22,Z22  X23,Y23,Z23  ...  ->    X2,Y2,Z2
test3->   X31,Y31,Z31  X32,Y32,Z32  X33,Y33,Z33  ...  ->    X3,Y3,Z3

What I want now is to reduce the number features to get a similar result to some defined extent. I want reduce, for example, the features by half, only picking those features that would get the most similar average.
(It would be too much to ask the same if I would want to also keep the most similar standard deviation possible or a combination of both?)
So, how could I select best features?
Some PYTHON EXAMPLE would be amazing
Sorry if I'm asking using the wrong words. I'm new to machine learning.
UPDATE: Some sample data
      V0                   V1                   V2                   V3                   V4                   [...]
T1    0.5,0,0.5            1,0,0                0.5,0.5,0            0.16,0,0.84          0,0,1                [...]
T2    0.57,0.11,0.32       0.53,0.15,0.32       0.24,0.51,0.24       0.18,0.15,0.67       0.54,0.15,0.31       [...]
T3    0,0.17,0.83          0.57,0.03,0.4        0.31,0.4,0.29        0.04,0.3,0.66        0.07,0.05,0.87       [...]
T4    0.1,0.43,0.47        0.81,0,0.19          0.25,0,0.75          0,0.21,0.79          0.43,0.19,0.38       [...]
T5    0,1,0                0.99,0.01,0          0.21,0.58,0.21       0,0.61,0.39          0.5,0,0.5            [...]
T7    0.29,0.37,0.34       0.53,0.36,0.11       0.27,0.48,0.25       0.13,0.47,0.4        0.28,0.56,0.16       [...]
T8    0.82,0.15,0.03       0.43,0.38,0.19       0.47,0.31,0.22       0.2,0.22,0.58        0.35,0.33,0.33       [...]
T9    0.29,0.22,0.49       0.35,0.32,0.33       0.3,0.4,0.3          0.28,0.36,0.36       0.33,0.34,0.34       [...]

Each Tn averages all Vn of each dimension X, Y or Z.
I want to reduce the number or Vn to calculate the average, selecting the most relevant Vn according to my data to get similar average of each T. Each vector V comes from a sensor and I want to reduce the number of sensors to get the same average to some approximation or to some limited number of sensors and calculate the difference. I have like +2000 tests T each with +100 vectors V.
So the Target of the Feature Selection is the average of all Features


texts: How to create a prediction interval with the fact that the residuals follow a specific distribution (in python)

texts:
I am looking at a software development pipeline where I am predicting the lead time of different products flowing through the pipeline.

After applying a boxcox transformation on the lead time (target variable) and creating a XGBoost regressor model I can see that the residuals follow a t-locationScale distribution.

So now I looked at this guide which describes a method to create a prediction interval for any regression model assuming that the residuals are normally distributed. https://qucit.com/a-simple-technique-to-estimate-prediction-intervals-for-any-regression-model_en/

But I tried to tweak it to my distribution.

So a t-locationScale distribution has a $\sigma$, $\mu$ and $\nu$ parameter. The variance is only defined for $\nu>2$. My specific distribution has $\nu = 2.56$ and $\mu = 0.04$, $\sigma = 0.97$ So I could take the 95% interval of this distribution and say that for any $\hat{y}$, the prediction interval is the 95% interval of the residual distribution.

But I want to take into consideration that the prediction interval should change with different inputs. I created a regressor model, which I trained and then made predictions using the validation set. I then took the square of the error and trained an additional error model on this data. Such that the error model could predict the variance of the residuals distribution.

  xgb = XGBoostRegressor()
  xgb.fit(X_train,y_train)
  y_hat = xgb.predict(X_val)
  val_error = (y_hat-y_val)**2

  xgb_error = XGBoostRegressor()
  xgb_error.fit(X_val, val_error)

  variance_hat_residuals = xgb_error.predict(X_test)

The relationship between variance and $\sigma$ and $\nu$ for a t-locationScale distribution is

var = $\sigma^2 *\frac{\nu}{\nu-2}$

Now here is where I make an assumption which I am not sure makes sense.

I assume that the degrees of freedom $\nu$ is the same as for all residuals, $\nu = 2.56$ and then I solve for $\sigma$ through the following.

$\hat{\sigma} = \sqrt{\frac{\hat{var}*(\nu-2)}{\nu}}$

And estimate the lower and upper quantiles from this distribution.

 residual_distribution_lower_quantile = scipy.stats.t.ppf(q = 0.025, df = 2.56, scale = sigma)
 residual_distribution_upper_quantile = scipy.stats.t.ppf(q = 0.0975, df = 2.56, scale = sigma)

I then predict the lead time $\hat{y}$ and say that the mean of the distribution is $\hat{y}$

   pred = xgb.prediction(X_test)
   lower_interval = pred + residual_distribution_lower_quantile
   upper_interval = pred + residual_distribution_upper_quantile

Does it make sense to make the claim of $\nu$ is static? My score for the prediction interval is now $81\%$ since I am clearly simplifying the problem.

Any suggestions for improving my method?


texts: What is the advantage of a tensorflow.data.Dataset over a tensorflow.Tensor?

texts: I have my own input data class. It has x and y as well as test and train values (1 Tensor for each combination). I noticed there is a Dataset class built in to TensorFlow. What is the advantage of this class over a regular Tensor? Is it mainly around handling large datasets / laziness? It doesn't appear to have features tailored for x vs y data, or test vs train. All my data fits into memory so I am not clear it would be beneficial to use the built in class over my current one. Of course, the first assumption is it would be foolish not to use the built in class.


texts: Updating One-Hot Encoding to account for new categories

texts:
My question is focused around how to appropriately update an encoded feature set when a new category is introduced by the test data. I use the data in logistic regression and I know it is not a 'live' model (i.e. gradient descent is performed whenever new data is introduced) but do I have to retrain the model to account for added features or do I just add it to subsequent test set values.

To exemplify the problem consider a TV Show training set where each show has a 'networks' feature set that includes one or more of the following:

["abc","cbs","nbc"]

Then, in the testing set there is a TV Show with the feature set:

["abc", "hulu"]

Would I have to add the new feature retroactively to the training data and retrain the model eventhough it will never occur? Wouldn't this introduce 'look-ahead-bias'?

How do I account for the added feature in the encoder going forward?


texts: When should ordinal data be represented catigorically and when as integer?

texts:
I am doing the Kaggle competition House Prices: Advanced Regression Techniques to learn more about data analysis. I would like to apply multiple models to the data(Regularized LR, Random Forests, Neural Networks, and ensemble methods).

When inspecting the data, I found many fields were ordinal data represented as categorical data. Two examples:

HeatingQC: Heating quality and condition

   Ex   Excellent
   Gd   Good
   TA   Average/Typical
   Fa   Fair
   Po   Poor
LotShape: General shape of property

   Reg  Regular
   IR1  Slightly irregular
   IR2  Moderately Irregular
   IR3  Irregular

I was wondering whether I should keep the fields like this, or whether I should turn encode them as integers(i.e. give each class in the category a number like 1,2,3 or 4) . Since the question probably is 'it depends', I hope you could give me some more general insight in when I should keep this data ordinal, or when to transform it into integers.


texts: Spark Scala concatenate 2 different data frames

texts:
I have 2 different Spark data frames and I want to concatenate them together by columns with no join operations.
How can I do it using Scala?


texts:
Going through the presentation and material of Summingbird by Twitter, one of the reasons that is mentioned for using Storm and Hadoop clusters together in Summingbird is that processing through Storm results in cascading of error. In order to avoid this cascading of error and accumulation of it, Hadoop cluster is used to batch process the data and discard the Storm results after the same data is processed by Hadoop.

What is the reasons for generation of this accumulation of error? and why is it not present in Hadoop? Since I have not worked with Storm, I do not know the reasons for it. Is it because Storm uses some approximate algorithm to process the data in order to process them in real time? or is the cause something else?


texts: Remove unnecessary Neurons from a Neural Network regarding a particular output

texts:
Suppose we have a Neural Network with a binary output (0 or 1). What I am trying to do is to remove neurons or layers from the NN while maintaining a correct classification for all the instances that were classified as 1 in the original NN, same thing for the output 0. Said differently, is there any way to spot neurons that are paramount to the correct classification of the instances of a particular class ? The aim is to remove all the unnecessary neurons regarding that output. Currently, I am trying to use the back propagation phase to try to attribute a fitness to each neuron regarding its contribution to a certain class.
In the case of Binary Neural Networks (binary weights and activations), a research track could be compiling the NN to a Boolean Formula and reasoning on it to spot the neurons that does not contribute to the chosen output, but it is not always obvious to carry out this compilation.


texts:
I have a student performance data, where I have marks of various subjects for the students and I want to find similar students with good marks in a particular subject using machine learning.

How do I go about it, Collaborative filtering?

The end Goal is to recommend jobs based on a student's top performance in a particular Subject.

Suggestions are open. I have attached the head of my data.


texts: Python Machine Learning to analyse test-case lab data and identify new test cases from real-world random data

texts:
I am currently working on a project where I have predefined test-cases I need to run/test and then analyse the data. My aim is to be able to identify system failures (against pre-defined criteria). Eventually I would like to be able to present statistics of the failures vs test conditions and ideally get to the point where I can anticipate failure modes, even before these happen.

I have quasi-infinite amounts of random data generated by systems identical to the one under test, but being used in a random manner "in the real world" (i.e. not staged scenarios). From this random data I would like to be able to identify scenarios which are similar to the staged ones as well as new ones which have not yet been identified as use-cases.

The system under test is a control system. The ultimate goal is to verify that the control system behaves and reacts as expected/specified when subject to various conditions. In general I am dealing with flags and sequences thereof. (ex: temperature goes up,  electric motor gets derated. Load increases, current to motor increases to maintain speed.)

In essence therefore I am talking about 4 tasks within the same project:

Identify failing scenarios (Pass/Fail identification for the test cases is already being taken care of by other means, more or less)
Identify trends/patterns in causes of failure, therefore anticipate failure modes (ex: the motor is more likely to fail under hot conditions, and the failures are more likely to happen during transient modes.)
Identify Use cases from the real world which match the test-case scenario and therefore identify frequency of occurrence of each test-case scenario
Identify common use-cases from real world to create new test cases which may not have been thought of

I'm guessing this would qualify as machine learning(?)

I would mostly want to use Python for this as:

I am keen to become more proficient with the language (my experience so far has been limited to following tutorials).
It is pretty much the only thing I have at my disposal.

So, my main question at this point is (assuming this is even realistically possible, and I am going down the right path):

Are there any existing examples of something similar which I can look at?


texts:
So I'm learning RNN, and tried to do a prediction LSTM, but I do not understand how the output works. I have this LSTM RNN:

lstm_model = Sequential()
lstm_model.add(LSTM(100, batch_input_shape=(BATCH_SIZE, TIME_STEPS, x_t.shape[2]), dropout=0.0, recurrent_dropout=0.0, stateful=False,   kernel_initializer='random_uniform'))
lstm_model.add(Dropout(0.5))
lstm_model.add(Dense(25,activation='relu'))
lstm_model.add(Dense(1,activation='sigmoid'))
optimizer = optimizers.RMSprop(lr=0.003)
lstm_model.compile(loss='mean_squared_error', optimizer=optimizer)

and fit with:

history = lstm_model.fit(x_t, y_t, epochs=300, verbose=2, batch_size=BATCH_SIZE,
                    shuffle=False, validation_data=(trim_dataset(x_val, BATCH_SIZE),
                    trim_dataset(y_val, BATCH_SIZE)), callbacks=[csv_logger])

When I then try to predict using

gotten = lstm_model.predict(x_test_t[-500:],batch_size=BATCH_SIZE)

Will I get only one new output? I'm very confused with the way you run and get predictions

Thanks for the help


texts:
I really do not understand what does this code do
M = sparse.coo_matrix(([1]*n, (Y, range(n))), shape=(k,n)).toarray()

The code is related to calculating the sparse function in this equation, but I am really confused and I do not know how it iterates through it and what is:
1- sparse.coo_matrix
2- (Y, range(n)))
3-shape=(k,n)).toarray()
??
Also, What exactly does this term means in the equation and how to interpret it into code:
Thank you , and please forgive my poor English.


texts: Clustering sets based on common elements

texts:
I am looking for a clustering algorithm for the purposes of combining routing information.
Suppose we have the following sets:
A={1,2,3,4,5}
B={2,3,4,6}
C={3,4,7,8}
D={8,9,10,11}
If we want 3 groups, you could have arranged a variety of combinations ({1A;2B;3CD},{1A;2BC;3D}, etc.).  If we consider the overlap in elements as a scoring metric, something like 1A, 2B, and 3CD would have a score of 2 since only one element (8) overlaps between C and D.  However, if we group 1AB, 2C, 3D then the score would be 6 since three elements (2,3,4) overlap between A and B.  If we specified 2 groups, you might get even higher: 1AB, 2CD with a score of 7.  I should also mention that the initial sets can also be weighted, representing multiple occurences of the set equal to its weight.  The output of the algorithm would provide a table of set assignments to a group index.  Is there any algorithm that can accomplish this efficiently (there are hundreds of million records to cluster).
I have looked into the Apriori algorithm, which appears to be doing something similar but doesn't provide final clusters (just counts of instances for overlaps, it doesn't do the assignment; though I could be misunderstanding).  Other similar threads have taken an approach of conducting k-means clustering based on a distance metric of overlap--however this doesn't seem quite right either because it doesn't tell you which elements are overlapping (again I could be missing something here).  It seems like hierarchical (agglomerative) clustering would be a good approach in principle, but again it is unclear what metric would be clustered on (again since overlaps don't actually contain any information about the elements in the sets).


texts: How to compare Factor and Principal Component Analysis results?

texts:
I am currently working on an assignment where I am to perform a comparison of different Dimension Reduction techniques in Python. I am using the Scikit-learn functions to perform PCA and FA. However I am not sure what is the best way to compare the two in terms of "success". I was thinking I could compare how many "new features" (principal components vs factors) is needed to explain a certain % of the variance in the observations. However, I do not know how to calculate this for the Factor Analysis results? The PCA has the explained_variance-attribute, but I don't see an equivalent for FA.

Also, if someone has a better suggestion for comparing the two techniques in terms of success - Pit would be great :)


texts: Orange 3 - Feature selection / importance

texts:
I am using (and loving) Orange 3 for some projects at my school and have a question:

When using Python and e.g. doing a RandomForest Classification, I can easily access the feature importances by feature_importances_

In Orange 3 there seems to be no feature to access that in the visual programming interface (would be a great add-on), so I tried to write my own python script in orange...

I linked the RandomForest Classifier to the Python script widget's in_classifier and have the following code:

print(in_classifier)
importances = in_classifier.feature_importances_

The result is however:

Random Forest Classification {'random_state': 0, 'n_estimators': 500, 'n_jobs': 1, 'min_samples_leaf': 1, 'max_leaf_nodes': None, 'max_features': 'auto', 'bootstrap': True, 'verbose': 0, 'max_depth': None, 'oob_score': False, 'min_samples_split': 2, 'class_weight': None, 'min_weight_fraction_leaf': 0.0, 'criterion': 'gini'}
    Traceback (most recent call last):
      File "", line 1, in
      File "", line 2, in
    AttributeError: 'RandomForestClassifier' object has no attribute 'feature_importances_'

=> I don't understand whey the RandomForestClassifier Object doesn't provide this array?

Anyone any idea?

many thanks!!


texts: How to do parameter estimation on relatively few samples?

texts:
Context:
I have a time series on which I've trained an autoencoder for anomaly detection (AD) purposes.
My time-series is 6 years worth of daily data, but for AD purposes, I am looking at one week window. This means I've only got 330 sub-sequences for my purposes.
I used 10% of those sub-sequences as my validation set, so 33 samples. It's not much, but it my model generalizes well to these few samples.
Problem:
For AD, I want to estimate the parameters of the residual distribution. I need these estimates in order to calculate an anomaly score using Mahalanobis distance. However, I don't think 33 samples is enough to estimate such parameters. Using the entire dataset seems like a bad idea, a sort of data leak.
Question:
Is it possible to synthesize new data in order to expand my validation set?
I thought of adding "random" noise to some samples, but I'm pretty sure that would make the parameter estimates skewed as it might change the distribution in the wrong direction.


texts: Alternatives to Heat Maps for Communicating Daily Data

texts: I am currently using the heat map below to communicate the number of messages sent on a messing platform over time. I don't think this is the most effective way to communicate the information as it's difficult to read and see trends. Although, I really like the ability to compare by week day (most of the messages are not send on weekends and I'd like to communicate this). What is a good alternative to heat maps that maintain the ability to communicate the weekday? Thanks for your help!


texts: Getting results of folds in orange canvas

texts:
I am using Orange Data mining 3.8 to classify a dataset using leave one out cross validation.

I know that the results ($AUC$, $CA$, $F1$) are averaged; How can I get the results ($AUC$, $CA$, $F1$) of every fold?


texts: How to persist data scaler for predictions

texts:
I have a Support Vector Machine in Scikit-learn (Python) that gets trained once in a while when enough new data has accumulated (user help train the model by submitting new data).

I store the model in .pkl format for persistence. However, the SVM needs scaled data and I'm wondering what would be a good way to persist the scaler, since it was fitted on the train data that changes over time (as users submit more data).

Is there a common solution for this? Or is there a value that I can store with the model that works as some sort of seed to recreate the scaler?

EDIT (added implementation):

I have a database with a dataset. users periodically add data points to this dataset. I also have an estimator database which keeps track of all the trained models and their .pkl file locations. When enough new data points are added the system will start to train a new estimator. It extracts all the features and scales the data, then trains a model. After that it compare the existing best model with the newly trained one and if the new one is better, it sets the new one to 'active'. There is only one active model a a time and that is the one that will be used for predictions on new data. (I hope this helps)


texts: How can you recommend songs based on a user's past listening history by genre (content filtering)?

texts:
I'm interested in getting a user's past listening history from Spotify (API call to recently played) and being able to suggest songs from the Charts (another API call for current chat listings) that a user may be interested in listening to.

It's clear that you could use Genre to support this type of algorithm, the more times a user listenes to a genere, it has a higher likelihood of being suggested from the Charts (weight?).

I'd like to focus on the activity of a single user and not necessarily other users using the system (content based filtering and not collaborative filtering)

How can this be done in practice? Are there any examples of algorithms or tutorials anywhere?


texts: How to access substrings in pandas column and store it into new columns?

texts:
I'm working on a dataset for building permits. In the dataset there is a column that gives the location (lattitude and longitude) for the building permit. The data in the location column look like this:

0    (37.785719256680785, -122.40852313194863)
1     (37.78733980600732, -122.41063199757738)
2      (37.7946573324287, -122.42232562979227)
3     (37.79595867909168, -122.41557405519474)
4     (37.78315261897309, -122.40950883997789)
Name: location, dtype: object

As you can see, the data is stored as strings. I wanted to store the lattitude and longitude in two separate columns, so I wrote the following code to accomplish this:

df.location = df.location.str.replace('(','')
df.location = df.location.str.replace(')','')

for i in range(len(df)):
    if df.location[i] == np.nan:
        df['lattitude'] = np.nan
    else:
        df['lattitude'] = df.location[i][0:df['location'][i].index(',')]

for i in range(len(df)):
    if df.location[i] == np.nan:
        df['longitude'] = np.nan
    else:
        df['longitude'] = df.location[i][0:df['location'][i].index(',')]

There is some missing data in the column, 1700 entries to be exact. So in order to avoid a key error, I wrote the if-else statement to fill in the new columns with np.nan anytime the loop would iterate to a missing entry.

When I ran the code, I got the following error:

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-72-81826748e81c> in <module>()
      7         df['lattitude'] = np.nan
      8     else:
----> 9         df['lattitude'] = df.location[i][0:df['location'][i].index(',')]
     10
     11 for i in range(len(df)):

AttributeError: 'float' object has no attribute 'index'

Can anybody spot the error I'm making?


texts: Canny edge detection not working on Gaussian blurred images

texts:
I am trying to detect edges on this lane image. First blurred the image using Gaussian filter and applied Canny edge detection but it gives only blank image without detecting edges.

I have done like this:
#imports
import matplotlib.pyplot as plt
import numpy as np
import cv2
import matplotlib.image as mpimg
image= mpimg.imread("Screenshot from Lane Detection Test Video 01.mp4.png")
image = image[:,:,:3]
image_g = cv2.cvtColor(image, cv2.COLOR_RGB2GRAY)

image_blurred = cv2.GaussianBlur(image_g, (3, 3), 0)
threshold_low = 50
threshold_high = 100
image_blurred = image_blurred.astype(np.uint8)
image_canny = cv2.Canny(image_blurred, threshold_low, threshold_high)
plt.imshow(image_canny,cmap='gray')


texts: Which GPU instance to opt for in Google Cloud?

texts:
I am looking to rent a GPU instance in Google Cloud for casual deep learning model training purposes and wondered about the differences between the available Nvidia Tesla versions which are

Nvidia Tesla T4
Nvidia Tesla P4
Nvidia Tesla V100
Nvidia Tesla P100
Nvidia Tesla K80

Here is the GPU Pricing page.

If anyone has used it or currently doing or know the best GPU to rent it'll be appreciated if you can share your experience or knowledge about it.


texts: How to create vectors from text for address matching using binary classification?

texts:
I want to compare two addresses for their similarity (purely similarity in terms of textual occurrences rather than semantic similarity). I have a labelled dataset having "Address1", "Address2" and "Similar", where "Similar" takes a value of 0 (not similar) or 1 (similar).

Now, how would I convert the text in "Address1" and "Address2" into numerical vectors, so as to treat this as a binary classification problem?


texts: Why does xgboost give this unexpected result?

texts:
This is a really simple example where my training data has a single feature vector (1,2,3) and an equivalent target vector (1,2,3).  I can get xgboost to build a regression tree that perfectly fits the data, but when I use it to make predictions, the predictions against the training data, the predictions don't match the target.  What gives?

train <- matrix(c(1,2,3))
target <- matrix(c(1,2,3))
bst <- xgboost(data=train, nrounds=1, label=target, eval_metric="rmse", max.depth=3, min_child_weight=1, gamma=0, lambda=0, alpha=0)
predict(bst, train)  # returns 0.65 0.95 1.25

xgb.plot.tree("f1", model=bst)

xgb.model.dt.tree("f1", model=bst)
    ID Feature Split Yes  No Missing Quality Cover Tree Yes.Feature Yes.Cover Yes.Quality No.Feature No.Cover No.Quality
1: 0-0      f1   2.5 0-1 0-2     0-1    1.50     3    0          f1         2        0.50       Leaf        1       0.75
2: 0-1      f1   1.5 0-3 0-4     0-3    0.50     2    0        Leaf         1        0.15       Leaf        1       0.45
3: 0-2    Leaf    NA  NA  NA      NA    0.75     1    0          NA        NA          NA         NA       NA         NA
4: 0-3    Leaf    NA  NA  NA      NA    0.15     1    0          NA        NA          NA         NA       NA         NA
5: 0-4    Leaf    NA  NA  NA      NA    0.45     1    0          NA        NA          NA         NA       NA         NA


texts: Why do classes with fewer words have higher probability?

texts:
Given that I have a word that does not occur in any of my documents: newword, and given that I have two classes: class1 and class2. For instance, the total number of words in class1 is 3, the total number of words in class2 is 6, and the number of unique words in all documents is 8.

Also, given the Naive Bayes formula, the word newword will have a higher probability of belonging to class1 (because the denominator will be lower while the numerator stays the same). Is there any statistical/logical theoretical explanation (that drives the algorithm) about this behavior?

In a nutshell, I just want to know what is the motivation of giving higher probability to classes with fewer words.

Ps.: I am using Laplace Smoothing. Therefore:
 P(newword|class1) = 0 + 1 / 3 + 8 = 0.09 >
P(newword|class2) = 0 + 1 / 6 + 8 = 0.07


texts: What is the significance of model merging in Keras?

texts:
I have learned that Keras has a functionality to "merge" two models according to the following:

from keras.layers import Merge

left_branch = Sequential()
left_branch.add(Dense(32, input_dim=784))

right_branch = Sequential()
right_branch.add(Dense(32, input_dim=784))

merged = Merge([left_branch, right_branch], mode='concat')

What is the point in mergint NNs, in which situations is it useful? Is it a kind of ensemble modelling? What is the difference between the several "modes" (concat, avg, dot etc...) in the sense of performance?


texts: How to handle like meaning sentences when working on text summarization

texts:
Suppose we have a text like Today is a very bad day. Very bad day is today. I wont come to play.
What kind of technique should I use to summarize similar texts like above? From articles, I found over the web till now I think that extractive summarization will give importance to the first two sentences because major key points of the texts(according to the frequency of words) are present in the first two sentences. Similarly, the abstractive summarization technique will also make a summary certainly considering the first two sentences. But, in the ideal case, the third sentence is a must.
What should be my approach considering my text may have many sentences with similar meanings??
I am very new to this any kind of suggestion or help will be great.


texts: Mean Absolute Error increasing with more correlated factors

texts:
I am using Microsoft Azure Machine Learning Studio to predict stock market prices. We have the variables- Index price(target-to be predicted),Low price,High price,dates and days. We use split of 0.7 and run Linear regression. We get Mean absolute error of 109. We then try to add more variables(macroeconomic factors which positively effect the index prices) which are correlated with the target variable and should improve the predictions- we find that the Mean Absolute error increases to 110.I have attached the pics for your reference. Are we interpreting wrong or what's wrong in we are doing?
PS:We tried Boosted Tree regression as well-but the same problem as described above is observed.
Errors


texts: Q-Learning experience replay: how to feed the neural network?

texts:
I'm trying to replicate the DQN Atari experiment. Actually my DQN isn't performing well; checking another one's codes, I saw something about experience replay which I don't understand. First, when you define your CNN, in the first layer you have to specify the size (I'm using Keras + Tensorflow so in my case it's something like (105, 80, 4), which corresponds to height, width and number of images I feed my CNN.). In the codes I revisited, when they get the minibatch from the memory, I see they usually fed the CNN without "packing" it on 4 batches. How it is possible? I mean for example if you get 32 random sampled experiences, don't you need to make batches of 4 before feeding it?
Here are an example of what I'm saying: https://github.com/yilundu/DQN-DDQN-on-Space-Invaders/blob/master/replay_buffer.py
https://github.com/yilundu/DQN-DDQN-on-Space-Invaders/blob/master/deep_Q.py
In this code, that's how he/she stores the experiences:

def add(self, s, a, r, d, s2):
        """Add an experience to the buffer"""
        # S represents current state, a is action,
        # r is reward, d is whether it is the end,
        # and s2 is next state
        experience = (s, a, r, d, s2)
        if self.count < self.buffer_size:
            self.buffer.append(experience)
            self.count += 1
        else:
            self.buffer.popleft()
            self.buffer.append(experience)

Then when you need to use them:

 def sample(self, batch_size):
        """Samples a total of elements equal to batch_size from buffer
        if buffer contains enough elements. Otherwise return all elements"""

        batch = []

        if self.count < batch_size:
            batch = random.sample(self.buffer, self.count)
        else:
            batch = random.sample(self.buffer, batch_size)

        # Maps each experience in batch in batches of states, actions, rewards
        # and new states
        s_batch, a_batch, r_batch, d_batch, s2_batch = list(map(np.array, list(zip(*batch))))

        return s_batch, a_batch, r_batch, d_batch, s2_batch

Ok, so now you have a batch of 32 states, actions, rewards, done and next states.

This is how you feed the state batch (s_batch) and next state batch (s2_batch) to the CNN:

def train(self, s_batch, a_batch, r_batch, d_batch, s2_batch, observation_num):
        """Trains network to fit given parameters"""
        batch_size = s_batch.shape[0]
        targets = np.zeros((batch_size, NUM_ACTIONS))

        for i in range(batch_size):
            targets[i] = self.model.predict(s_batch[i].reshape(1, 84, 84, NUM_FRAMES), batch_size = 1)
            fut_action = self.target_model.predict(s2_batch[i].reshape(1, 84, 84, NUM_FRAMES), batch_size = 1)
            targets[i, a_batch[i]] = r_batch[i]
            if d_batch[i] == False:
                targets[i, a_batch[i]] += DECAY_RATE * np.max(fut_action)

        loss = self.model.train_on_batch(s_batch, targets)

        # Print the loss every 10 iterations.
        if observation_num % 10 == 0:
            print("We had a loss equal to ", loss)

In my code (https://bitbucket.org/jocapal/dqn_public/src/master/Deimos_v2_13.py) I get a batch of 32 experiences; then make small batches of 4 experiences and feed the CNN. My question is: am I doing it wrong? And if so, how can I feed 32 experiences when my CNN is waiting for 4 experiences?

Another example of what I'm saying: https://yanpanlau.github.io/2016/07/10/FlappyBird-Keras.html


texts: Self-supervised learning for automatic labeling of data using LDA and Word2Vec

texts:
I am trying to implement this paper A Brand-New Look at You: Predicting Brand Personality in Social Media Networks with Machine Learning for labeling Twitter data of brands with a corresponding brand personality. Basically a brand can exhibit certain traits based on Aaker's Personality Traits for brands as per the image below.

On the top most level there are personality dimensions and below them there are traits associated with these dimensions. I am using the aforementioned paper as I do not have access to any labeled data just data from Twitter of brands.
In the paper they used LDA Topic Clusters and Word2Vec word representations along with dictionaries of the brand personality traits to accomplish this task. I am having issues visualizing the exact steps according to the paper.
The steps based on the paper (page 14-15):

Create refined clusters of posts utilizing LDA and Word2Vec(these clusters describe the various topics being discussed).
Take each of the Twitter posts and use Word2Vec distance similarities between the Tweets and each of the refined topics(to see which Tweet most accurately represents each of the refined topic clusters)

Then the Tweets were only included in only one refined topic cluster and the Tweets with a 50% similarity in Word2Vec distance moved on to the next step(more steps such as using the brand personality dictionary with this clusters but I need a better understanding of what I am doing here first).
I am stuck at how I am supposed to use LDA and Word2Vec together. So if I cluster the tweets using LDA and Word2Vec then I would need to compare the distance similarity of the topic clusters from the Word2Vec clusters and the Tweets and if the Tweets are similar(threshold of 50%) they would be used in the next step. I am confused at how LDA plays a role here as I do not know where the topic clusters for it are being used in this step.


texts: sklearn serialize label encoder for multiple categorical columns

texts:
I have a model with several categorical features that need to be converted to numeric format. I am using a combination of LabelEncoder and OneHotEncoder to achieve this.
Once in production, I need to apply the same encoding to new incoming data before the model can be used. I've saved on disk the model and the encoders using pickle. The problem here is that the LabelEncoder keeps only the last set of classes (for the last feature it has encoded), thus it can't be used to encode all the categorical features for the new data. To face this issue I am saving on disk a different LabelEncoder for each one of the categorical features, but this does not seem to scale very well to me, especially when you have a large number of categorical features.

What is the common practice for this situation? Is it possible to serialize and save just one encoder for all the categorical features to be used in production?


texts: How to run only part of the code in PyCharm, having some data in operating memory

texts:
In Jupyter Notebook I have blocks of Python code and I can run them arbitrarily. What is super cool about that is I can spend 10 minutes training some ML model and then play around with it, without the need to run the training process again and again.
How can I achieve something similar using PyCharm? While I like this feature in Jupyter it is so less convenient to write code.


texts: Basic Time Series Classification Examples

texts:
I've been using matlab until now to classify a large number of labelled time series I have. This has been relatively successful but I'd like to try using Tensorflow to apply a Deep Learning paradigm instead.

I'm a complete noob at this and so I'm a bit overwhelmed with the literature as I'm struggling to generalise examples such as the 0-9 digits classification examples to my problem.

My current code reads in the 1064 time series (length 3125), reads in the labels, converts the labels to onehot_encoding and extracts training and validation sets.

#Not all code included
def read_data():
    #Get labels from the labels.txt file
    labels = pd.read_csv('labels.txt', header = None)
    labels = labels.values

    data = pd.read_csv('ts.txt',header = None)

    return data, labels

data, labels = read_data()

# Default split is 75% train 25% test
ts_train, ts_test, labels_train, labels_test = train_test_split(data,labels)

onehot_encoder = OneHotEncoder(sparse=False)
labels_train = onehot_encoder.fit_transform(labels_train)
labels_test = onehot_encoder.fit_transform(labels_test)

#Construct the graph
batch_size = 100
seq_len = 3125
learning_rate = 0.0001
epochs = 1000

But now I need to construct the graph and I'm a bit lost. If someone could link me some useful examples I'd be very grateful thank you.


texts: Is there an open source implementation for bag-of-visual words?

texts:
I'm not quite sure I understand the bag-of-visual-words representation, so I may misformulate my question.

What I'm currently looking for is an open source library (possibly with python API). I give it pictures as input, and its output is a set of (sparse) features, so that I can perform my stuff base on this features. Idealy, I would like this piece of software to work without internet connection (so that I can work with it while in the plane).

EDIT: I just learnt that facebook recently (summer 2016) released some of its image recognition code (namely multipathnet, deepmask and sharpmask)


texts: Model with 2 datasets: combine time series data and statistics

texts:
I am new to data science modelling so apologies if using wrong terminology in advance.
I have a standard time series dataset of historical prices which is used to train/test a simple Random Forest classifier model which predicts the returns direction (+/-).
I also have a few general statistics for 'day of the week direction' eg. frequency counts: Monday UP=120, Monday DOWN=90, Tuesday UP=67, Tuesday DOWN=50, Friday UP=55, Friday DOWN=181.
How can I combine the results from the time series RF model with the bias from the statistics into a new model? eg. give more weight to positive/up predictions on Monday, and more weight to negative/down predictions on Friday?


texts: How to know which loss function is suitable for image classification?

texts: I am implementing a CNN model for image classification where I am learning about loss functions. There are several types of loss functions to determine error. However, how do we find out which function is suitable for a image multi-classification model?


texts: ROC-AUC curve as metric for binary classifier without machine learning algorithm

texts:
Maybe it's a trivial question, but I'm a bit confused right now... I'll explain: I've some elements in my data, each with a value between 0 and 1 and an associated label (1, 0).
I need to test some thresholds and then compute the ROC-AUC curve.
For example with a threshold = 0.4, all the values greater than 0.4 will be predicted as true (1) and all the values under 0.4 will be predicted as false (0), then I'll compare the result with the actual labels to calculate the True Positive Rate and the False Positive Rate and finally build the ROC-AUC curve.

So, my question is, considering that, based on the threshold chosen, I already know under which prediction an element will fall, I don't need a machine learning classifier to do the default training and testing stages, right?


texts: How to check for "statistical significance" of categorical feature in black box models

texts:
Let's say we have a categorical feature $X_i$ and we have build a black-box classification model like xgboost with $X_i$ as one of many predictors. We'd like to ask a question: does $X_i$ affects the overall prediction and, if so, how much?

In particular $X_i$ could be:

a dichotomous variable
a n-level variable where we are interested in the potential difference between two particular levels

In white-box models like linear regression we have tests to obtain statistical significance. But can we obtain statistical-significance-alike with black box models? Does any tool from explainable artifficial intelligence field is applicable to that? Or would it be better to just perform standard t-test on the output probabilities predictions?


texts:
I have a text dataset which I vectorize using a tfidf technique and now in order to make a cluster analysis I am measuring distances between these vector representations. I have found that a common technique is to measure distance using cosine similarity, and when I ask why euclidean distance is not used, the common answer is that cosine similarity works better when vectors have different magnitude.

Since my text vectorized representation is normalized I wonder which is the advantage of using cosine similarity over euclidean distance in order to cluster my data?


texts: Forecasting Multiple (few hundreds) uni-variate time series with inflated zeros

texts:
Hello Practitioners,
Being a newbie seeking help to gain experience in Data Science.
Lets take a scenario where a big company wants to forecast its sales (a specific product) across different stores in different geographic locations.
As an Analyst, a task is given to forecast few hundreds of series (sales) for next 3 months. Since, we are forecasting sales across different geographic
locations, the nature of the series would not be same for all. There would hundreds of models to check with.
What are the suggested approaches for this scenario with your experience in this field? Also, how important it is know the nature of each series in this scenario?


texts: Recover a integer valued function with *-learning

texts:
I have the following problem. From a technical model we have a function $f(n,p)$ approximating its runtime. The function $f$ which maps
$$
 f: \mathbb{N} \times \mathbb{P} \to \mathbb{R}_{+}
$$
where $\mathbb{P} = \{1,\ldots,50\} \subset\mathbb{N}$. $n$ defines the amount of input and $p$ is a parameter of the process, which has a continuous influence on the runtime. We are interested in the value of $p$ such that $f(n,p)$ for a given $n$. At the run the experiment  some $n$, like 500, 1000, 5000, and for all possible values of $p$ and then select the optimal $p$ by hand. Like for everything smaller than 750 use the optimal $p$ of the test run with $n=500$ and so on. Typically for a fixed $n$ the runtime over $p$ has the following style but with noise added:

For increasing $n$ the bottom moves to the right and due to noise the bottom is not flat. So in the example the optimal $p$ is somewhere between $20$ and $30$.
We are now searching for a function $g$:
$$
g: \mathbb{N} \to \mathbb{P}
$$
which is defined by
$$
 g(n) = \operatorname{argmin}\limits_{p\in\mathbb{P}} f(n,p)
$$
Since sampling $f$ for fixed $n$ is not a problem and I already have a large set of samples I think this will be a good problem to start with for looking into the machine learning area for myself.
My basic question is where to start? Search for machine learning tutorial I mostly end up with the image recognition examples and did see any good connections between those and my problem.
Any suggestions about stuff to read for this problem? Any idea which software to use? And finally how to extract $g$ such that we can provide this function to the users or implement somewhere? At the moment I know that $g$ is non-decreasing for increasing $n$.
Edit
The sampling results can also look like this
Thereby the spikes in the "mininum" point area are no noise. They are caused by technical internals of the regarded process. So while processing the data and removing the noise these points need to kept. And they make finding $g$ so hard because in one sampling run the minimal $p$ lies left, in the middle or right of them. For this reason I though this would be a good point to use machine learning to get some insights.


texts: Is a large number of epochs good or bad idea in CNN

texts: In my CNN model, by using large number of epochs like 400 or above, the validations accuracy and some times test accuracy gets better, but I think this large number of epochs is not good idea? I am right or not? why?


texts: how to use word embedding to do document classification etc?

texts:
I just start learning NLP technology, such as GPT, Bert, XLnet, word2vec, Glove etc. I try my best to read papers and check source code. But I still cannot understand very well.

When we use word2vec or Glove to transfer a word into a vector, it is like:

[0.1,0.1,0.2...]

So, one document should be like:

[0.1,0.1,0.2...]
[0.1,0.05,0.1...]
[0.1,0.1,0.3...]
[0.1,0.15,0.1...]
.......

So, one document is a matrix. If I want to use some traditional method like random forest to classify documents, how to use such data? I was told that Bert or other NLP models can do this. But I am really curious about how the word embedding are applied in the traditional methods?


texts: Algorithm for finding best juice combinations

texts:
I am making a fun experiment in which a machine will mix different percentages of three juices – orange, apple and grape. After each mix is dispensed, a participant will taste the juice and rate it on a numeric scale, a score from 1 to 7.

Using the data collected, I would like to try and find the optimal juice mix programmatically.

For this I wish to implement a machine learning algorithm, that will both generate new ratios of juices to try, and using the response will try and find to the optimal juice mixture percentages.

What algorithm would you recommend me to use?

Notes:

I am planning on displaying this at an event, and estimate around 200 people to taste, at least.
I am aware that due to the fact that different people have different taste preferences, I aim for the best tasting mixture that will fit most people.


texts: How to use LSTM to make prediction with both feature from the past and the current ones?

texts: Suppose I have a data frame with 2 columns, which are sales and promotions. I want to predict the next day sales based on the past sales and promotion info of 3 days, plus the promotion to be applied at the next day? How do I process and reshape the dataframe? I mean if only previous promotions need to be considered, then, after some shifting, the data frame could be reshaped into (sample size, 3, 2), but it becomes a problem if I also need to consider the promotion at the next day. It is a pretty common issue, does anyone has any thought about this?


texts: Is it always better to using stacked LSTM than single LSTM?

texts:
I am currently studying LSTM and RNNs.

I came across several concepts like Multidimensional LSTM and Stacked LSTM.

I have used Stacked LSTM and it gives me a better performance than single LSTM. As per my understanding, if I increase the depth of LSTM, the number of hidden units also increases. It means overfitting, right? Then why am I getting better results?

[Note: I have used BatchNorm and Dropout after every stack of LSTM ]


texts:
Following Andrew Ng's machine learning course, he explains SVM kernels by manually selecting 3 landmarks and defining 3 gaussian function based on them. Then he says that we are actually defining 3 new features which are $f_1$, $f_2$ and $f_3$.

$\hskip0.9in$

And by applying these 3 gaussian functions on every input data:
$$x=(x_1,x_2)\to \hat{x}=(f_1(x), f_2(x), f_3(x))$$

it seems that we are mapping our data from $\mathbb R^2$ space to a $\mathbb R^3$ space. Now our goal is to find a hyperplane in the 3 dimensional space, where our transformed data is linearly separable. Is my understanding correct? If not, how these 3 new features should be interpreted?

$\hskip1in$

In some blog posts, i have read that by using a gaussian kernel, we are mapping our data to an infinite dimensional space (where gaussian kernel computes the dot product of transformed input data) which contradicts with my above understandings.


texts:
Is there an efficient way to handle pruning in Decision Tree with Python ?
Currently I'm doing that:
def do_best_tree(Xtrain, ytrain, Xtest, ytest):
    clf = DecisionTreeClassifier()
    clf.fit(Xtrain, ytrain)
    path = clf.cost_complexity_pruning_path(Xtrain, ytrain)
    ccp_alphas = path.ccp_alphas
    clfs = []
    for ccp_alpha in tqdm(ccp_alphas):
        clf = DecisionTreeClassifier(ccp_alpha=ccp_alpha)
        clf.fit(Xtrain, ytrain)
        clfs.append(clf)
    return max(clfs, key=lambda x:x.score(Xtest, ytest))

But it's super slow (as I create and fit a lot of trees).
Is there a more efficient way to do this with scikit-learn, or another library that handle this ?


texts: GAN Loss Function Notation Clarification

texts: In the Generative Adversarial Network loss function, what do these mean?: $E_{x~p_{data}(x)}$ and $E_{z~p_{z}(z)}$ and how are they used in this context?


texts:
I have been getting poor results on my time series predictions with a LSTM network. I'm looking for any ideas to improve the model.

The above graph shows the True Data vs. Predictions. The True Data is smooth zig zag shaped, from 0 to 1. However the predictions rarely reach 0 or 1.

The distribution in the prediction data-set rarely reaches 0 or 1 and it's centered around 0.5.

However the distributions in the True Data set is evenly distributed.

Here is the LSTM model built in keras:

model = Sequential()
model.add(Dropout(0.4, input_shape=(train_input_data_NN.shape[1], train_input_data_NN.shape[2])))
model.add(Bidirectional(LSTM(30, dropout=0.4, return_sequences=False, recurrent_dropout=0.4), input_shape=(train_input_data_NN.shape[1], train_input_data_NN.shape[2])))
model.add(Dense(1))
model.compile(loss='mae', optimizer='adam')

How do I get the predictions to be more similar to the true data?


texts: What's the difference between probabilistic programming such as pyro and belief networks?

texts:
I heard about ubers pyro and stumbled upon this Wikipedia article.

As I understand, a bayesian network is the same as a belief network according to this post.

Does someone know how these are related?


texts: How do I merge two data frames in Python Pandas?

texts: I have two data frames df1 and df2 and I would like to merge them into a single data frame. It is as if df1 and df2 were created by splitting a single data frame down the center vertically, like tearing a piece of paper that contains a list in half so that half the columns go on one paper and half the columns go on the other. I would like to merge them back together. How do I do it?


texts: Feature Selection on Aggregated Targetdata

texts:
I have a question about feature selection on a dataset where the target variable is aggregated by the sum of different data points. I want to predict the number of sales depending on a variety of features like:

week
price per unit
store country
store city
2-3 other categorical meta-data
other features

I am aware that this data should be interpreted as time series but because of the lack of available historical data, no model can compete with the naive approach.

The problem I am facing is, that the target variable is the sum of grouped features like month, price, country and city. If I add or remove one of the grouped features I will get some identical data points and the dataset will be inconsistent, so I have to perform a grouped aggregation (sum) to get consistency back. This operation will change the target variable and the number of data points. I have no idea how to validate regression models trained on different subsets of the features because the underlying dataset is not equal. I know there are other feature selection techniques than wrapper methods like filter or embedded which provides workarounds but I would like to know if there are techniques to resolve this issue.

Example:

    week  price_per_unit store_country  sales
0      1             3.0            C1     30
1      1             3.0            C2     32
2      1             4.0            C1     23
3      2             3.5            C1     19
4      2             3.5            C2     27
5      2             6.5            C1     35
6      3             2.0            C1     17
7      3             3.0            C1     15
8      3             4.0            C2      7
9      4             2.0            C1     19
10     4             5.0            C1     41
11     4             5.0            C2     21

After dropping the column store_country:

    week  price_per_unit  sales
0      1             3.0     30
1      1             4.0     23
2      1             3.0     32
3      2             6.5     35
4      2             3.5     19
5      2             3.5     27
6      3             2.0     17
7      3             3.0     15
8      3             4.0      7
9      4             5.0     41
10     4             2.0     19
11     4             5.0     21

Now there are duplicate data points and the sale column is wrong because I need the sum so after aggrgation I have:

   week  price_per_unit  sales
0     1             3.0     62
1     1             4.0     23
2     2             3.5     46
3     2             6.5     35
4     3             2.0     17
5     3             3.0     15
6     3             4.0      7
7     4             2.0     19
8     4             5.0     62

Let's assume I want to perform forward or backward selection with linear regression. In every step a column will be added or removed. So the number of rows depends on the feature which is selected. I can't think of a metric to compare these regression models.


texts: Mathematics: Can the result of a derivative for the Gradient Descent consist of only one value?

texts:
I have a problem of a task using the formula of the Gradient Descent:

Perform two steps of the gradient descent towards a local minimum for the function given below, using a step size of 0.1 and an initial value of [1, 1]

I only get as result of the derivative 0.4x1, because x2 does not exist.
Is this correct or should the result for the derivative be (0.4, 0)?
Note: Sorry, if my equation of the derivative is bad. I'm not a mathematician. Please, correct me, if my equation is absolutely wrong.


texts: How to understand backpropagation using derivative

texts: Before I was learning about gradient descent, but now I understand this. Now, I have a problem with the backpropagation algorithm. I know the idea - minimalize error in multilayer neural network using chain rule. However, I don't understand the role of the derivative of the sigmoid function. This derivative is described in the algorithm. What is the point of this? Can you explain this step by step using simple language?


texts: Array of categorical variables vs one-hot encoding

texts:
I have some JSON data to be transformed to a machine-learning friendly format. Every object in my data, which will eventually become an instance in my dataset, has the exact same fields (in this case, foo, bar and array). The array field contains a variable number of sub-objects (from 0 to 10). Each of these sub-objects has one categorical field with a massive range and some other fields which can be safely ignored. All categorical fields in these sub-objects belong to the same range.

Example (massively simplified object):

{
"foo": 1
"bar": 0.5
"array" : [
    {
        "categorical": "Lorem"
        "other": 34
        "stuff": 56
    },
    {
        "categorical": "Ipsum"
        "other": 53
        "stuff": 12
    },
    {
        "categorical": "Dolore"
        "other": 6
        "stuff": 101
    }
]
}

Obviously foo and bar are easily represented as numeric attributes. I would now like to represent this array (array) as one large one-hot (or several-hot) vector. Assuming I don't care about any other field of the sub-objects except for the category field categorical, here is my question:

Question

Is it valid and possible to set more than one bit in a one-hot vector (which would make it, I assume, a several-hot vector) to represent all categories present in this instance? If not, how could it be done?


texts: Compare the MAE Vs R^2 output for Machine learning 10-fold CV

texts:
I am comparing two 10-fold cv machine learning outputs. I got:
For model 1:
MAE= 0.252
R^2= 0.447
RMSE= 0.37

For Model 2:
MAE= 0.434
R^2= 0.768
RMSE= 1.164

My question is: Is it possible to get higher $R^2$ for higher MAE? What does this mean?


texts: Error while writing perceptron algorithm binary classifier

texts:
I am a beginner and I am designing an binary classifier using Perceptron algorithm using FASHION-MNIST dataset. While designing the same I have written the following code:

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import matplotlib.image as mpimg
import seaborn as sns
np.random.seed(2)
from sklearn.model_selection import train_test_split
from sklearn.metrics import confusion_matrix
import itertools
from keras.utils.np_utils import to_categorical
from keras.models import Sequential
from keras.optimizers import RMSprop,Adam
from keras.layers import Dense, Dropout, Flatten, Conv2D, MaxPool2D
from keras.optimizers import RMSprop
from keras.preprocessing.image import ImageDataGenerator
from keras.callbacks import ReduceLROnPlateau
import warnings
warnings.filterwarnings('ignore')
import os

def read_dataset(train_path, test_path):

    train = pd.read_csv(train_path)
    test = pd.read_csv(test_path)

    y_train = train["label"]
    x_train = train.drop(labels = ["label"],axis = 1)  #throw the labels column

    y_test = test["label"]
    x_test = test.drop(labels = ["label"], axis = 1)

    x_train = x_train / 255.0
    x_test = x_test / 255.0

    x_train = x_train.values.reshape(-1,28,28,1)
    x_test = x_test.values.reshape(-1,28,28,1)

    y_train = to_categorical(y_train, num_classes = 10)
    #x_train = to_categorical(x_train, num_classes = 10)

    y_train = y_train.astype(int)
    y_test = y_test.astype(int)

    #y_train = y_train.reshape((y_train.shape[0]))
    #y_test = y_test.reshape((y_test.shape[0]))

    #x_train = x_train.iloc[:,:].values
    #x_test = x_test.iloc[:,:].values

    #y_train = y_train.iloc[:,:].values
    #y_test = y_test.iloc[:,:].values

    return (x_train, y_train, x_test, y_test)

def peptrain(x, y, x_var_test, y_var_test):
    w = np.zeros((x.shape[1]))
    maxiter = []
    errors = []
    train_acc = []
    test_acc = []

    for i in range(5):
        error = 0
        for t in range(x.shape[0]):
            y_cap = np.sign(np.dot(w,x[t]))
            print(y_cap)

            if y_cap == 0:
                y_cap = -1

            if y_cap != y[t]:
                error = error + 1
                w = w + y[t] * x[t]

        errors.append(error)
        maxiter.append(i + 1)

        trainacc = 1 - ( error / x.shape[0] )
        testacc = peptest(x_var_test, y_var_test, w)

        train_acc.append(trainacc)
        test_acc.append(testacc)

    return (train_acc, test_acc, maxiter, errors)

def peptest(x, y, w):
    mistakes = 0
    for t in range(x.shape[0]):
        y_cap = np.sign(np.dot(w,x[t]))
        print(y_cap)

        if y_cap == 0:
            y_cap = -1

        if y_cap != y[t]:
            mistakes = mistakes + 1

    return (1 - (mistakes / x.shape[0]))

if __name__ == '__main__':

    train_location = "/Users/coraljain/Downloads/fashionmnist/fashion-mnist_train.csv"
    test_location = "/Users/coraljain/Downloads/fashionmnist/fashion-mnist_test.csv"

    x_train, y_train, x_test, y_test = read_dataset(train_location, test_location)
    train_acc, test_acc, maxiter, errors = peptrain(x_train, y_train, x_test, y_test)

Here I am getting the following error:

ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

for the following part.

if y_cap == 0:

What option do I have here? How am I supposed to use all and any function?


texts: Generating the right target for an LSTM model

texts:
Trying to explain my question on a simplified data set.
Having the following dataset:
   day   f1    f2
0    0   10  1000
1    1   45  2000
2    2  120  3400
3    3   90  5000

I'm trying two approaches to generates a score based on the data observations:
Approach 1:
I've scaled the features so the max value is 1.0 by dividing each feature by it's max value to get:
   day        f1    f2
0    0  0.083333  0.20
1    1  0.375000  0.40
2    2  1.000000  0.68
3    3  0.750000  1.00

I created a score where score = (f1,f2) so now the data looks like so:
   day        f1    f2     score
0    0  0.083333  0.20  0.141667
1    1  0.375000  0.40  0.387500
2    2  1.000000  0.68  0.840000
3    3  0.750000  1.00  0.875000

Approach 2:
I did a similar score calculation, however, the normalization of the features were done with CDF like so:
from scipy import stats
df['f1'] = df.f1.apply(stats.norm.cdf, args=(df.f1.mean(),df.f1.std()))
df['f2'] = df.f2.apply(stats.norm.cdf, args=(df.f2.mean(),df.f2.std()))

   day        f1        f2
0    0  0.123267  0.143672
1    1  0.330776  0.312474
2    2  0.865919  0.624118
3    3  0.687676  0.891864

And the final score:
   day        f1        f2     score
0    0  0.123267  0.143672  0.133469
1    1  0.330776  0.312474  0.321625
2    2  0.865919  0.624118  0.745019
3    3  0.687676  0.891864  0.789770

Looking at the larger picture.
My actual data set is composed of 1280 sequences of 30 days each with 10 features (shape is (1280,30,10)) and I'm trying to predict day 30 score based on the first week of data.
Looking at the histogram for day 30 score of my first approach it looks like so:

while my other approach ac produce the following:

Question: When building an LSTM regression model, what scoring method would better reflect the overall score? Is that a product question, or is there a better statistical method to calculate the score?


texts: Imbalanced dataset: how to deal with test data?

texts:
I plan to use many methods to solve the imbalanced dataset problem on the training set. But I couldn't find any paper that describes how they dealt with the test dataset? I assume that they just tested on the original dataset without any adjustments? Will I need to adjust the threshold on the test set with original imbalanced ratio?

update. Thanks for everyone's response!
I've found a paper that discusses how we should adjust the posterior probability and threshold.
Calibrating Probability with Undersampling
for Unbalanced Classification
However, I also believe that in practice we can just fit test set directly.


texts: Hyperparameter Tuning in Machine Learning

texts: What is the difference between Hyper-parameter Tuning and k-NN algorithm? Is k-NN also a type of Hyper-parameter tuning?


texts:
I'm storing a set of house data in my Elasticsearch database with various attributes for each entity (such as price, number of bathrooms, sqft, etc..) I want to create a basic ranking engine that takes a house and returns me the score based on similar houses. Ideally in a way that I can put weight on specific attributes.
Here's a basic example of what I'm trying to achieve:
House A gets a 0.75 (out of 1) score from the system based on relevant houses with very similar attributes (similarly priced with size) however some of these houses have private pools hence House A is not the best value among this list of houses.
What would be a good approach/technology to achieve this?


texts: Should we denormalize our data after normalization?

texts:
If we use sklearn library's preprocessing.normalize() function to normalize our data before learning, like this:

preprocessing.normalize(training_set)
model.add(LSTM())

Should we do a denormalization to the result of LSTM to get predicted result in a true scale? If yes, how to denormalize?


texts: Training EfficientNet always throws CUDA exhausted error

texts: I have been trying a hands on the EfficientNet-B6 architecture but evertime I try to train the same, It simply throws up CUDA resource exhausted error. I have been training it on 16GB Tesla P100


texts:
What are the best known Data Science Methodologies today? By methodology I mean a step-by-step phased process that can be used for framing guidance, although I will be grateful for something close too.

To help clarify, there are methodologies in the programming world, like Extreme Programming, Feature Driven Development, Unified Process, and many more. I am looking for their equivalents, if they exist.

A google search did not turn up much, but I find it hard to believe there is nothing out there. Any ideas?


texts: How to train a Text Based data for a Machine Learning problem?

texts:
I am working on a project for displaying products to customer by context, based on a search query. For example, I don't want customers to have to enter a specific product name, instead searching based on functionality (e.g., "walls do not heat much" would return product names such as "Whirlpool NEO IC355 ROY 3S 340 L Double Door Refrigerator")
I have a training set comprised of the functionality associated products. I am planning to use Logistic Regression to train a model on these data. How do I process this data in Python or extract features to feed into logistic regression? I have heard of "Bag of words model", but not sure how to use this, or is it even applicable here?

I know there are plenty of NLTK libraries available. But, I want to implement it from the scratch or using minimum external libraries possible.

Please help or are there any resources to refer to?


texts: Time series forecasting produce same values with different training data

texts:
I'm developing a python program which predict daily timeseries values.
Each daily timeseries contains 288 values (a record every 5 minutes).

The main idea is to train a LSTM model with 7 days data and predict the next day. After the prediction, train again the model with the last 7 days (so the one just predicted and discarding the first one).

In order to do this, I have implemented the following code:

for idx in range(len(self.test)):
    graph = tf.Graph()
    with tf.Session(graph=graph):

        history = [x for x in self.train]

        # prepare data
        train_x, train_y = to_supervised(self.train, self.n_input, self.n_output)
        self.fit_model(train_x, train_y)

        # make the prediction
        yhat_sequence = self.forecast(history, self.n_input)

        real_sequence = self.test[idx, :].reshape(-1, len(self.test[idx, :]))
        real_sequence = self.scaler_test.inverse_transform(real_sequence)

        yhat_sequence = yhat_sequence.reshape(-1, len(yhat_sequence))
        yhat_sequence = self.scaler_train.inverse_transform(yhat_sequence)

        # add real sequence to history
        history.append(self.test[idx, :])

        # add real sequence to train
        last_day_scaled = np.array([self.test[idx, :]])

        self.train = np.concatenate((self.train, last_day_scaled))

        # remove first day from train in order to have always one week of training
        self.train = self.train[1:, :]

        # reset model in order to train it again
        self.model.reset_states()
        self.model = None

Here the fit_model function:

def fit_model(self, train_x, train_y):
    # define parameters
    verbose, epochs, batch_size = 1, 100, 128

    # reshape trains
    ...

    n_timesteps, n_features, n_outputs = train_x.shape[1], train_x.shape[2], train_y.shape[1]

    if self.model is None:
        print("Create model")

        self.model = Sequential()
        self.model.add(LSTM(100, input_shape=(n_timesteps, n_features)))
        self.model.add(LeakyReLU(alpha=0.1))
        self.model.add(RepeatVector(n_outputs))
        self.model.add(LSTM(100, return_sequences=True))
        self.model.add(LeakyReLU(alpha=0.1))
        self.model.add(TimeDistributed(Dense(100)))
        self.model.add(LeakyReLU(alpha=0.1))
        self.model.add(TimeDistributed(Dense(1)))
        self.model.compile(loss='mse', optimizer='adam')

    # fit network
    self.model.fit(train_x, train_y, epochs=epochs, batch_size=batch_size, verbose=verbose)

This codes works without errors, but predictions done training after training are very similar to each other even if training data change.

Pictures below are three different days (blu real data and green prediction)

I've try to increase epochs, neurons, add DroupsOut and to increase number of training days

Is there something wrong with my code? How can I improve my prediction training after training?


texts: Time series stationarize vs normalization

texts:
I have multiple time series coming from sensor measurements of an industrial machine. The industrial machine runs different 'Recipes'. Every recipe has different set of parameters which are set before a recipe begins. So this means across time my sensor measurements will be affected when a new recipe begins. So lets say for the first 1 hour I run my recipe which gives me temperature measurments of 80 degree celcius, then i change to another recipe and run it for 2nd hour which gives me temperature as 90 degree celcius.
In total I have a lot of recipes running. So eventually my timeseries has varying mean and variance across time which comes from running different recipes. My end goal is to predict when the machine will breakdown. So eventually the features go into some machine learning model.
Before feeding my sensor measurements to the machine learning model i must do some sort of normalization or maybe also stationarize my timeseries.

Do you recommend normalizing every part of my timeseries ( by every part i mean every recipe window of my timeseries ) or stationarize the timeseries ?

For normalization I can simply put in a standard scaler for every recipe but to stationarize a timeseries i find it bit difficult especially because I dont see any kind of seasonality or trend in my timeseries. But still the timeseries is non stationary ( because mean and variance varies over time due to changes in recipes ).

So how should I stationarize my timeseries ?


texts: Clustering or classifing n-gram-based text categories

texts:
I have large set of data records looking like this:

"text", "category"

I extract n-grams from text (2-, 3- and 4-grams) and store count of each n-gram per category, like so:

"ngram1", "category1", 1000
"ngram1", "category2", 20
"ngram1", "category3", 15
"ngram2", "category1", 25
"ngram2", "category2", 550
"ngram2", "category3", 600

Is there a clustering or classification algorithm that could help with finding groups within categories, based on counts of same n-grams within categories?

In the above example that algorithm should be able to show that "category2" and "category3" are part of the same group.


texts:
How to fix TypeError: ufunc 'isnan' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe''?

from sklearn.preprocessing import Imputer
imputer=Imputer(missing_values="np.nan",strategy="mean",axis=0)
imputer=imputer.fit(x[:,1:3])
x[:,1:3]=imputer.transform(x[:,1:3])

  Traceback (most recent call last):

  File "", line 3, in
      imputer=imputer.fit(x[:,1:3])

  File
  "C:\Users\Nibir\Anaconda3\lib\site-packages\sklearn\preprocessing\imputation.py",
  line 167, in fit
      self.axis)

  File
  "C:\Users\Nibir\Anaconda3\lib\site-packages\sklearn\preprocessing\imputation.py",
  line 254, in _dense_fit
      mask = _get_mask(X, missing_values)

  File
  "C:\Users\Nibir\Anaconda3\lib\site-packages\sklearn\preprocessing\imputation.py",
  line 30, in _get_mask
      if value_to_mask == "NaN" or np.isnan(value_to_mask):

  TypeError: ufunc 'isnan' not supported for the input types, and the
  inputs could not be safely coerced to any supported types according to
  the casting rule ''safe''


texts: Clustering mixed data types - numeric, categorical, arrays, and text

texts:
I have a dataset with 4 types of data columns:

              numeric  categorical            tags                     text
id
1               51585           27  [A, B, C, ...]  "Some text bla bla bla"
2               53596           27  [B, D, E]               "Other text..."
3             1176345           27  [D, A, F, ...]                    "..."
4                 168           24             NaN                    "..."
5               88564           22             NaN                    "..."

numeric - continuous numeric values.
categorical - discrete categories, either numbers or strings (the type doesn't really matter because I can convert it to whatever works)
tags - array containing discrete values. Each row can have a different array length.
text - a string of text.

I am new to data science so maybe this is a "beginner" question.

How can I use all of these different data types in a clustering algorithm?

Here is what I learned so far:

K-means is good for numerical data. I successfully applied it to a subset of my data with only numerical columns. I also used some evaluation metrics (such as silhouette coefficient) to help me choose the number of clusters. So this works in principal but since it's not using most of my data the results are not good.
Then I read about clustering categorical data. I found the Gower Distance which is a distance between categorical data. So far I've used it with K-means (I passed the distance matrix generated by Gower into K-means). From here it should be easy to join the Gower distance matrix with the numeric columns from my original dataset and pass all of them to K-means.

I am aware there are other clustering algorithms besides K-means, and I plan to check some others as well. But before I do, I want to find some way to utilize all of my data in a single algorithm.

The tags and text columns stump me. I can't find a way to use them for clustering. I found some articles about clustering words from a text document - this is not what I want to do. I want to use a text column as one (or more) "feature" among others for clustering.
I am aware of the "bag of words" method for converting text into a vector of numbers. I can also easily imagine how to use this same method for converting the tags into a vector. However that seems like a bit of an overkill because it will increase the dimentionality of my data by a lot. Are there other ways to tackle this?

Bottom line - I am looking for a way to use all these data types together for clustering. I summarized what I know so far, but I am open to any solution, even if it's completely different from what I've listed above.

Thanks!


texts: Unable to transform the variable to normal distribution

texts:
I have a variable called 'list_price' in my data ,which is not normally distributed.
Here is the screenshot of how the distribution looks like before transformation

Tried transforming it with the following transformations:
1.log
2.Sqrt
3.reciprocal
4.boxcox
None has resulted in normal distribution.Most of the transformations gave a skewed distribution after transformation.
Would like to use this variable in modelling.
How would we proceed further in such scenarios?
Should we go with original distribution itself?
Need some guidance on this


texts:
We would like to implement rule engine in one of our products, but when I read on the Internet, many are suggesting that rule engines are an old way of doing things and it will be very hard to manage as we add more rules into it. To be successful in implementing a rule based system we need to know all the rules beforehand.

Instead, everyone seems to be suggesting that we use Machine Learning algorithms to predict the outcome, but I am new to Machine Learning.

So my Question is, should I learn Machine Learning or go on implementing Drools as rule engine in my application.

Or can you suggest any online resources achieving the same using Machine Learning?


texts: May the training sets of two predictive modeling cases (with an overlap in features) be combined?

texts:
Say we have dataset D1 (columns A, B, C) and D2 (columns A, B, D) with target variable E. As both datasets are rather small, their respective predictive models do not perform really well. To improve predictive performance, may/should I apply the steps below?

Remove the not-in-common columns (so column A, B and E remain in both datasets)
Split each dataset in a train and a test set (cross-validation to evaluate, so no validation set)
Train our predictive model on the combination of the training sets of D1 and D2
Assess test set performance of D1 and D2 on their respective test sets (using 1 model rather than 2)

I assume this is allowed, but am not fully sure if there are any (bad) repercussions.


texts: Applying machine learning to cache algorithm

texts:
I have an application that receives ~10k of requests per day; each request has multiple parameters and goes through a pipeline with multiple steps and they finish in about 1 hour.
Let's say the requests are like bake_cake [flavor=chocolate] [topping1=strawberry] [topping2=cream]

Given a history of requests from previous days, is there a ML framework that can help me predict the "100 most like requests to arrive today" so I can cache their results? Or some other similar strategy that can help me delivering part of those requests faster?

Ideally it would be based on how much I gain by delivering faster, how much I lose by processing a request that might not come, etc.; but for now even simpler algorithms could be of much help since currently there's no optimization.


texts: How to get confidence score from a PyTorch based BiLSTM-CRF model

texts:
I have created a BiLSTM-CRF based NER model referring the following link: https://pytorch.org/tutorials/beginner/nlp/advanced_tutorial.html
Now, the ask is to get the confidence score for each classification. How can I achieve that?


texts: Which method to use to remove trend from time series?

texts:
From what I understand, differencing is necessary to remove the trend and seasonality of a time series. So I assumed it basically does the same thing as signal.detrend from the scipy library.

But I tried differencing and then, separately, used signal.detrend and my time series looked completely different.

Original:

Differencing:

Imported libraries:

The x axis represents months and the y axis is sales. The colours on the first two charts just represent three different years.


texts: what is the main difference between GAN and autoencoder?

texts: what is the main difference between GAN and other older generative models? what were the characteristics of GAN that made it more successful than other generative models?


texts: Database missing extensions:tym_system_time,quantile orange widget error

texts:
I'm working with the orange data mining tool. In most of the examples data  taken from files or google sheets links but in the blog link they given that we can take data from postgresql database tables also but when follow the steps of blog I'm getting error.I attached the screen shots of error.

I need help in solving this error.


texts: EM clustering with missing and misspelling data

texts:
I am currently working on a project that requires me to cluster the unlabeled input. The records contain personal information such as name, DOB, height, sex, etc. We need to cluster the same person in one group, here is the sample data:

+------------------------------------+
|        Record1     Record2         |
+------------------------------------+
| First Name   'Harry'     'Harry'   |
| Middle Name  'Jay'       'J'       |
| Last Name    'Potter'    'Potter'  |
| DOB Month    1           1         |
| DOB Day      1           1         |
| DOB Year     1993        1993      |
| Dr License   'A1234567'  Null      |
| Sex          1           1         |
| Address      Hogwarts'  'Hagwarts' |
+------------------------------------+

Those two records should be clustered in the same group.
I am still in the beginning phase of the project. I want to try with EM clustering using Gaussian Mixture model. But I do not know how I should preprocess the string, or should I do it and how to deal with the Null value.


texts: SVD based method for predicting missing values in future experiments

texts:
I would like to pose my question in terms of a hypothetical example, where I have a matrix:

11 12 13...1m
21 22 23 ..2m
.  .  .    3m
.  .  .  .  .
11 n2 n3 .  nm
The matrix has no missing values. Each column corresponds to one experiment and each row corresponds to a particular measurement parameter ( eg: temperature @ point 1, temperature @ point 2... Pressure....)
next, I conduct a New experiment, and I measure most parameters but not all n parameters. I want to estimate the missing parameter(s) in my New experiment.
Is there a formula based on SVD/PCA to calculate the missing value in my New experiment based on my 'training' matrix?
Are there any better machine learning / statistical methods to perform this calculation ?
Thanks in advance!


texts: Account for unknown error in time series data

texts:
Given:
Time series data collected from sensors.

There is an unexpected gradual drop in the initial data when sensors are idle.
However, this drop is not so visible when sensors are active because the drop is masked by the actual measurements.

Main question: How do I eliminate this drop from the sensor output?

Not sure if this is an accurate assumption, but let's assume that the drop is a convolution of an unknown system X (leak, sensor drift etc.) with the system being measured G.

I am feeding this into a neural network.

How do I get an approximation to the unknown influence X (from initial data)?

My goal: Apply (-X) to all other data collected from G before training the neural net and expect an increase in performance.
Should I expect an increase in performance?


texts: Predict class having only class proportions for every attribute (non labeled data)

texts:
I am working with a big data set (millions of observations) where for each observation I am trying to predict a probability (or score it) of being of a class. I haven't any labeled training data and it is impossible to collect it.

So far I found a service that allows to query several times per hour with min amount of allowed observations starting from several thousands and with no upper boundary, ultimately service returns the proportion of the class for each tested group of observations. I assume here that an observation with a target class is more likely to appear in corresponding groups, and I would like to preserve this assumtion here, as without it the idea doesn't make any sense.

Data can be non-randomly grouped into multiple slices by attributes where some of attributes has quite good positive or negative relationship with the class.

way to think about it:

| observation_id  | attr1 | attr2 | attr3 | … | attrN |
|-----------------|-------|-------|-------|---|-------|
| _1              | 1     | 0     | 1     | … | 0     |
| _2              | 0     | 0     | 1     | … | 1     |
| _3              | 0     | 0     | 1     | … | 0     |
| _4              | 1     | 0     | 1     | … | 1     |
| _5              | 1     | 0     | 0     | … | 0     |
| _6              | 1     | 0     | 0     | … | 1     |
| …               | …     | …     | …     | … | …     |
| M               | 1     | 0     | 1     | … | 0     |

If I group observations by attrs I receive sets of a different cardinality of observation and I can gather such info:

attr1 = {_1,_4,_5,_6,...,M} and has 73% proportion of a target class

attr3 = {_1,_2,_3,_4,...,M} and has 46% proportion of a target class

...

attrN = {has some observations} and has some proportion of a class

So it turns out that any observation is a member of a group with a known probability (proportion) to have a target class if you pick one observation at random. The objective is to find a way to infer which observation relates to a target class from from class proportion for every observation given knowledge of the class proportion for every group.

Is there any model type or scoring approach to optimise this process to infer the likelihood of observation being of the class from knowing the proportion of the class in multiple groups where this observation lives? By optimisation I mostly mean minimisation of testing iterations as it takes quite a lot of time, and if possible gaining more information from lower amount of features (in this case - affinity to a group by attribute).

Any hint or reference to an article/book/page will be super helpful :) Thanks a lot!


texts:
I am wanting to make print statements "showing my working out" of Binary Cross Entropy loss function, that works with scalar inputs, not arrays.
import math as m
import numpy as np

y_ans = 1
y_pred = 1

print("Cost(y,y^) = -yln(y^) - (1-y)-y^ln(1-y^)")
print('Cost('+str(y_ans)+', '+str(y_pred)+') = -'+str(y_ans)+'ln('+str(y_pred)+') -('+str(1-y_ans)+')ln('+str(1-y_pred)+')')
#ans = round((-y_ans*m.log(y_pred)) - ((1-y_ans)*m.log(1-y_pred)), 2)
ans = round((-y_ans*np.log(y_pred)) - ((1-y_ans)*np.log(1-y_pred)), 2)
print('Cost('+str(y_ans)+', '+str(y_pred)+') = '+str(ans))

The answer should be 0, but I get NaN
Cost(y,y^) = -yln(y^) - (1-y)-y^ln(1-y^)
Cost(1, 1) = -1ln(1) -(0)ln(0)
/home/runner/MalwareAnalysis/BinaryCrossEntropy.py:10: RuntimeWarning: divide by zero encountered in log
  ans = round((-y_ans*np.log(y_pred)) - ((1-y_ans)*np.log(1-y_pred)), 2)
/home/runner/MalwareAnalysis/BinaryCrossEntropy.py:10: RuntimeWarning: invalid value encountered in double_scalars
  ans = round((-y_ans*np.log(y_pred)) - ((1-y_ans)*np.log(1-y_pred)), 2)
Cost(1, 1) = nan

print((-y_ans*np.log(y_pred)))
-0.0
print((1-y_ans)*np.log(1-y_pred))
nan

I'm fairly new to Python.


texts: Interpretation of accuracy score on subset of data points

texts:
I have a multi-class problem that I am building a classifier for. I have N total data points I would like to predict on. If I instead predict on n < N data points and get an accuracy score, is there a way I can say (with some degree of confidence) what I think the same model's accuracy score would be on the remaining data points?

Can somebody point me to an article discussing this, or suggest a formula to research?


texts: I'm trying to determine number of clusters for k-means using sklearn.metrics.silhouette_score. I have computed it for range(2,50) clusters. How to interpret this? What number of clusters should I choose?


texts:
What is the human performance on ImageNet, the top 1 error? (not top 5 error).
The best of the machine learning algorithm top 1 error is 11.6% ,according to thie website https://paperswithcode.com/sota/image-classification-on-imagenet

.