texts: Why can't we feed datetime to Linear Regression and how does toordinal() different from any other integer datatype?

texts:
I'm working with dates for the first time. First I knew I had to convert it to timestamps which gave me the values in "datetime64" values. But then I came to know that Linear Regression from sklearn does not accept datetime as dtype for regression. Why is that?
Also, after executing df.date = df.date.map(datetime.toordinal) I saw that the datatype for the column is "int64". How does the sklearn LinearRegression know it is not any random int64 dataand a datetime data (considering the fact that I got the needed model after plotting)?


texts: What can be done with highly correlated variables (>.95 and <-.95)

texts:
I hope we can remove the highly correlated variables based on the feature importance may be with PCA etc.

Is there anything we can do with highly correlated variables/

Thanks in advance !


texts: Viterbi-like algorithm suggesting top-N probable state sequences implementation

texts:
Traditional Viterbi algorithm (say, for hidden Markov models) provides the most probable hidden state sequence given a sequence of observations.

There probably is an algorithm for decoding top-N probable hidden states sequences (k-shortest paths or the like).

But is there a good implementation anywhere?

Thanks.

UPD (copy-pasted from comments): Viterbi algorithm decodes the MOST probable sequence given observations + model. That is, argmax_x p(state_1,...,state_n|obs_1,...,obs_n). What I am asking for is an implementation of how to get the 'next argmax-es' with possibly slightly smaller probabilities GIVEN the observed sequence.


texts: Any good Implementations of Bi-LSTM bahdanau attention in Keras?

texts:
From past few weeks I'm trying to learn sequence to sequence machine translation modelling but I couldn't find any good examples/tutorials with bahdanau attention implemented. I did come across a ton of examples where people have implemented attention but it's mostly on gru (or) luong attention (or) outdated code.

Has anyone come across any good implementations of bahdanau attention model in keras which you have implemented (or) tried? I really want to learn the coding part but no good material regarding implementation is found?

Please help.


texts:
given a vocabulary with $|V|=4$ and V = {I, want, this, cat} for example.
How does the bag-of-words representation with this vocabulary and one-hot encoding look like regarding example sentences:

You are the dog here
I am fifty
Cat cat cat

I suppose it would look like this

$V_1 = \begin{pmatrix}
0 \\
0 \\
0 \\
0 \\
\end{pmatrix}$

$V_2 = \begin{pmatrix}
1 \\
0 \\
0 \\
0 \\
\end{pmatrix}$

$V_3=\begin{pmatrix}
0 \\
0 \\
0 \\
1 \\
\end{pmatrix}$

But what exactly is the point of this representation? Does is show the weakness of one-hot encoding with a fixed vocabulary or did I miss something?


texts:
I have been practicing with the following dataset:
http://archive.ics.uci.edu/ml/datasets/Concrete+Compressive+Strength
for building a prediction model based on a MLP, but I have some doubts if the approach followed is the correct. I wanted to tune the activation function based on the following models: identity, logistic, tanh and relu. So what I did is the following:
First I divided my dataset in 80/20/20 for training, validation and test; and for what I know the hyperparameter tuning is in the validation set. So my pseudocode for the validation part is like the following:
for each item in activation function list
    model=MLP(activation=item,solver="adam") with 1000 iterations
    fit(Xtrain,ytrain)
    plot(lossCurve) with the training data
    fit(Xval,yval)
    plot(lossCurve) with the validation data
end for

with this loop I found that the "best" activation function was "relu". I am putting two graphs as an example:

After that I got "adam" and "relu" as hyperparameters and the I tried them with the training and testing set, so roughly I did this:
model=MLP(activation="relu",solver="adam")
fit(Xtrain,ytrain)
plot(lossCurve) with the training data
fit(Xtest,ytest)
plot(lossCurve) with the test data

and the curve I get was the following:

What I wanted to know is if my approach is the correct. I ask this because it is not so easy to find examples of loss curves using scikit. I think because in many Internet tutorials the hyperparameter tuning is made by GridSearch or using CV, and the ones that use loss curves are implemented en Keras or TensorFlow.
I wanted to force my model to obtain a curve like this:

which is an overfitted model and just for the sake of learning. So I was wondering, do all models overfit? or what is happening in my tests? Maybe I made something wrong.
Any help would be greatly appreciated.
Thanks


texts: Negative learning implementation in pytorch

texts:
I have read a paper on Negative Learning: https://arxiv.org/abs/1908.07387.
The idea is that you can train a network not only by telling what label of the sample is, but by telling what it surely is not.
Let's call the latter a "negative" label.
An excerpt from the paper says (top formula is for usual "positive" label loss (PL), bottom - for "negative" label loss (NL):

I have a problem where "negative" labels collection is much easier than labeling each sample. So it is tempting to use it.
Is there some implementation of such a loss function in pytorch? Or should I write a custom loss layer code? If so, how should I do it?


texts: Adapting ZFNet on 2244x224 image using a filter 7X7

texts:
I am building a model based on ZFNet in Tensorflow 2.0. I am using the Petal images dataset. The images are of size 224x244x3.
So my question is when implementing the first layer (conv2d) with filter size = 7 and a stride of 3 and padding of 0. I am getting the output dimension of 109.5 using formula (n+2p-f/S + 1). So if I use the above-mentioned values what will be the dimension returned by TensorFlow in the first layer.
and secondly, how can I adjust the parameter values so it returns a whole number.
reference formula : (n+2p-f)/2 +1
reference calculations:
224+0-7/2 +1  = 109.5
Thanks.


texts: Outlier detection: Should the metric used in kNN take into account variance explained by each coordinate?

texts:
After applying PCA and working with the reduced dataset, I want to delete the outliers. To do this my idea is to compute the kNN-graph and delete those vertices (points) that have an inner degree of 0. That is that they are not in the neighborhood of any other point besides itself.

My question is if it is a good idea to instead of working with the $l_2$ distance, I should add weights that take into account the variance explained by each coordinate.

The idea is that if the variance explained by coord 1 is 0.93, and by coordinate 2 only 0.05, then the points are expected to be "close together" on the projection over coord 2. It seems natural that the distance on the second coordinate should be much more important than the first one for outlier detection.

First approach was to simply use the inverse of the variance explained as weights. Does that idea make sense? If so, what is the proper way to implement it?


texts: Suppose I write a program in Keras for MNIST data set. I used TensorFlow as my backend in Keras. Is it possible in any way that I can extract the backend code used for TensorFlow that has been used by Keras. if yes, please provide an example.


texts: Having trouble installing and loading tidyverse- No DIB package

texts:
after a fresh (Control panel, Windows 7) uninstall, then re-install of Rstudio, I have tried to install and load tidyverse.  I get the following message, seemingly because the package DBI is missing:

  install.packages("tidyverse")
  Installing package into ‘C:/Users/jshea/Documents/R/win-library/3.3’
  (as ‘lib’ is unspecified)
  trying URL 'https://cran.rstudio.com/bin/windows/contrib/3.3/tidyverse_1.1.1.zip'
  Content type 'application/zip' length 41994 bytes (41 KB)
  downloaded 41 KB
  package ‘tidyverse’ successfully unpacked and MD5 sums checked

The downloaded binary packages are in
    C:\Users\jshea\AppData\Local\Temp\1\RtmpmoPd5V\downloaded_packages

  library(tidyverse)
  Error in loadNamespace(i, c(lib.loc, .libPaths()), versionCheck = vI[[i]]) :
    there is no package called ‘DBI’
  Error: package or namespace load failed for ‘tidyverse’

Earlier this morning, when I did this the first time, the system seemed to download all the packages except there was some kind of error associated with DBI- I think a database package.  Even uninstalling and deleting obvious remnants of Rstudio does not seem to give me a clean base to fix this.  What can I do?


texts: Ensure class balanced batches while hyperparameter tuning keras models with grid search

texts:
Ensuring class balanced batches while training keras models is possible using fit_generator method. I used imblearn.keras.BalancedBatchGenerator for that and it works fine!
But i wanted to do that with GridSearchCV to fine-tune my model, so i created an inherited class KerasBatchClassifier(KerasClassifier) as a Keras Wrapper for the Scikit-Learn API. (Inspired by this StackOverflow question)
Ensuring class balanced batches to fit the model is working, but i haven't figured out how to do it with validation data ! Any ideas to do that ?


texts: How to predict a group of events using machine learning?

texts: Whether any machine learning model can dynamically predict different types of stages it would go through for completing a process.


texts: I would like to know I how you represent a feature vector like this dataset wise. The vector length is dynamic but the each element has a fixed length (9). For xgboost implementation, do I just create columns for each of these features? Or is there any way to encode this vectores so they're readable for this kind of model?


texts: Why don't we find the analytical function of the cost function?

texts: Then we could derive it and find minimum(s). e.g. in small networks the cost function has not so many variables.


texts: How to convert a mel spectrogram to log-scaled mel spectrogram

texts:
I was reading this paper on environmental noise discrimination using Convolution Neural Networks and wanted to reproduce their results. They convert WAV files into log-scaled mel spectrograms. How do you do this? I am able to convert a WAV file to a mel spectrogram

y, sr = librosa.load('audio/100263-2-0-117.wav',duration=3)
ps = librosa.feature.melspectrogram(y=y, sr=sr)
librosa.display.specshow(ps, y_axis='mel', x_axis='time')

I am also able to display it as a log scaled spectrogram:

librosa.display.specshow(ps, y_axis='log', x_axis='time')

Clearly, they look different, but the actual spectrogram ps is the same. Using librosa, how can I convert this melspectrogram into a log scaled melspectrogram? Furthermore, what is the use of a log scaled spectrogram over the original? Is it just to reduce the variance of the Frequency domain to make it comparable to the time axis, or something else?


texts: Evaluate document similarity / content-based recommender system

texts:
I'm planning on building a basic content-based recommender system with word2vec and cosine similarity.
The data consists of 300k documents in varying length.
How do I evaluate my model if I have no labels / categories whatsoever?


texts:
I need to decide between SVM (One-Class Support Vector Machine) and PCA (PCA-Based Anomaly Detection) as anomaly detection methods. Azure ML is used and provides SVM and PCA as methods - hence the choice of 2 possible methods.

Does anyone have suggestions or a defined process for method selection? (Similar to cheat sheets you get for selecting a regression method).

The use case is to detect anomalies in high frequency network traffic data (from firewalls, routers & switches)?


texts: How does it work if we have different "time-steps" and "LSTM-units"?

texts:
As I know we can create a LSTM layer like:
model.add(LSTM(units), input_shape = (time-steps, feature_number))
And the units number and time-steps number can be different numbers! So how does it work? For ex, if I specify my LSTM like this:
model.add(LSTM(10), input_shape = (1, feature_number))

How will it feed the network? Shouldn't each unit of LSTM be 1 time-step? So how is it possible when I have 10 units but 1 time-step?


texts:
I am trying to understand the loss function of autoencoders.
I think the loss function of the decoder is to compare the input image with the decoded image, tell me if I am correct.
but I don't understand what is being compared to find the encoder error.
Someone explains to me that it compares to find the encoder error. and decoder.


texts: Vader vs TextBlob opposite outcome: why?

texts:
I've been studying for a Data Science course and yesterday I was challenged with a sentiment analysis, for which tons of material can be found online. So bear with me, ad I'm trying to get to the bottom of some different questions (I hope).
After a quick research, I've identified three possible models as candidate for my analysis:

FastText from Facebook,
TextBlob,
Vader from NLTK

I had to desist in using FastText as couldn't find anywhere a "model" to use. The online examples were pointing to tweets, cooking dataset and so on, but I need something a bit more generic.
I found some posts online where an nl761 something was mentioned but no trace on their resource.
Perhaps you can feed me in here with the link?
So I moved forward with the last two, and got my sets of customer comments analysed. I was expecting some difference, but with surprise I got results kind of diametrically opposed.
With Vader I got 68% of my comments being Neutral, whereas TextBlob marked 75% of the overall set as Positive.
I understand the big limitations of these algorithm is the lexicon that may be not updated, but somehow that outcome is suggesting a different kind of problem, don't you reckon?
I'm tempted to create my "source of truth" and ran past the comments with a Multinomial Bayes as next step. But although this could improve my outcome, it will probably not solve the discrepancy matter at the base of this thread.
Do you think there is something else I should have done other than pre-processing the comments to remove urls, stopwords, contractions and so on?
Thanks for your help.


texts: Can somebody please suggest what is the correct stage to remove correlated variables before feature engineering or after feature engineering ?


texts: Do I need to worry about not known positive examples in my negative examples when training a machine learning model?

texts:
Scenario
Let's say I want to predict whether people buy certain items together in a shop. I have features computed for every item. I have a list of items I know are bought together frequently (like a gold standart). My training inputs are the vector difference between two items. The negative examples are all the item pairs other than the gold standart (maybe a subset of this, due to class imbalance).
Question
Do I have to worry about having potential positve examples in my negative example data set? (those are the item pairs I would like to predict)
Is it better to try and have negative examples which I'm certain people are not buying together and train a model based on that?
I feel like the small number of potential positive examples are not relevant in the final accuracy of the model, but I'm not sure about it.


texts: Spectral clustering with heat kernel weight matrix

texts:
I am studying normalized graph cuts, and one of the way to define weight matrix is using heat kernel, which is $W_{ij} = e^{\frac{−∥x_i − x_j∥^2}{σ^2}}$.

I want to ask: what's the meaning of sigma? Does it affect on the partition of the data? How do we pick sigmas? And what happens if its too large or too small?

What happens to the Laplacian matrix L as $\sigma → 0$ and $\sigma → \infty$? What are the eigenvectors and eigenvalues of this matrix?


texts: Extract most informative parts of text from documents

texts:
Are there any articles or discussions about extracting part of text that holds the most of information about current document.

For example, I have a large corpus of documents from the same domain. There are parts of text that hold the key information what single document talks about. I want to extract some of those parts and use them as kind of a summary of the text. Is there any useful documentation about how to achieve something like this.

It would be really helpful if someone could point me into the right direction what I should search for or read to get some insight in work that might have already been done in this field of Natural language processing.


texts:
How can we extract information from time-stamp variable for modelling? I have a variable with format mm-dd-yyyy hh:mm:ss I want to predict an outcome variable using time-stamp as input variable. I do not think i can directly use this column for modelling and will need to do some transformation before i can use this in model. Not sure what type of transformation i need to do.

For example, if you have date field, you can create dummy variables of day o week and use that as input variable in model. Not sure how to proceed in the case of time-stamp.

Any help?


texts:
I am currently reading:

  Stephen Jose Hanson: Meiosis Networks, 1990.

and I stumbled about this:

  It is possible to precisely characterize the search problem in terms of the resources or degress of freedom in the learning model. If the task the learning system is to perform is classification then the system can be analyzed in terms of its ability to dichotomize stimulus points in feature space.

  Dichotomization Capability: Network Capacity Using a linear fan-in or hyperplane type neuron we can characterize the degrees of freedom inherent in a network of units with thresholded output. For example, with linear boundaries, consider 4 points, well distributed in a 2-dimensional feature space. There are exactly 14 linearly separable dichotomies that can be formed with the 4 target points. However, there are actually 16 ($2^4$) possible dichotomies of 4 points in 2 dimensions consequently, the number of possible dichotomies or arbitrary categories that are linearly implementable can be thought of as a capacity of the linear network in $k$ dimensions with $n$ examples.

What is a "dichonomy" in this case?

(Side questions: what is a fan-in type neuron?)


texts: Keras - no prediction probability for multiple output models?

texts:
I have built the following model:

def create_model(conv_kernels = 32, dense_nodes = 512):

    model_input=Input(shape=(img_channels, img_rows, img_cols))
    x = Convolution2D(conv_kernels, (3, 3), padding ='same', kernel_initializer='he_normal')(model_input)
    x = Activation('relu')(x)
    x = MaxPooling2D(pool_size=(2, 2))(x)
    x = Convolution2D(conv_kernels, (3, 3), kernel_initializer='he_normal')(x)
    x = Activation('relu')(x)
    x = MaxPooling2D(pool_size=(2, 2))(x)
    x = Dropout(0.25)(x)
    x = Flatten()(x)

    conv_out = (Dense(dense_nodes, activation='relu', kernel_constraint=maxnorm(3)))(x)

    x1 = Dense(nb_classes, activation='softmax')(conv_out)
    x2 = Dense(nb_classes, activation='softmax')(conv_out)
    x3 = Dense(nb_classes, activation='softmax')(conv_out)
    x4 = Dense(nb_classes, activation='softmax')(conv_out)

    lst = [x1, x2, x3, x4]

    model = Model(inputs=model_input, outputs=lst)
    sgd = SGD(lr=lrate, momentum=0.9, decay=lrate/nb_epoch, nesterov=False)
    model.compile(loss='categorical_crossentropy', optimizer=sgd, metrics=['accuracy'])

    return model

When I do a prediction:

model.predict(X_test)

it works properly. However, when I want to get prediction probability like this:

model.predict_proba(X_test)

my model has no predict_proba function. Why not? Is it because of the multiple-output nature of the model?


texts: Keras loaded model output is different from the training model output

texts:
When I train my model it has a two-dimension output - it is (none, 1) - corresponding to the time series I'm trying to predict. But whenever I load the saved model in order to make predictions, it has a three-dimensional output - (none, 40, 1) - where 40 corresponds to the n_steps required to fit the conv1D network. What is wrong?

Here is the code:

df = np.load('Principal.npy')

    # Conv1D
#model = load_model('ModeloConv1D.h5')
model = autoencoder_conv1D((2, 20, 17), n_steps=40)

model.load_weights('weights_35067.hdf5')

# summarize model.
model.summary()

    # load dataset
df = df

    # split into input (X) and output (Y) variables
X = f.separar_interface(df, n_steps=40)
# THE X INPUT SHAPE (59891, 17) length and attributes, respectively ##

# conv1D input format
X = X.reshape(X.shape[0], 2, 20, X.shape[2])

# Make predictions

test_predictions = model.predict(X)
## test_predictions.shape =  (59891, 40, 1)

test_predictions = model.predict(X).flatten()
##test_predictions.shape = (2395640, 1)

plt.figure(3)
plt.plot(test_predictions)
plt.legend('Prediction')
plt.show()

In the plot below you can see that it is plotting the input format.

Here is the network architecture:

_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
time_distributed_70 (TimeDis (None, 1, 31, 24)         4104
_________________________________________________________________
time_distributed_71 (TimeDis (None, 1, 4, 24)          0
_________________________________________________________________
time_distributed_72 (TimeDis (None, 1, 4, 48)          9264
_________________________________________________________________
time_distributed_73 (TimeDis (None, 1, 1, 48)          0
_________________________________________________________________
time_distributed_74 (TimeDis (None, 1, 1, 64)          12352
_________________________________________________________________
time_distributed_75 (TimeDis (None, 1, 1, 64)          0
_________________________________________________________________
time_distributed_76 (TimeDis (None, 1, 64)             0
_________________________________________________________________
lstm_17 (LSTM)               (None, 100)               66000
_________________________________________________________________
repeat_vector_9 (RepeatVecto (None, 40, 100)           0
_________________________________________________________________
lstm_18 (LSTM)               (None, 40, 100)           80400
_________________________________________________________________
time_distributed_77 (TimeDis (None, 40, 1024)          103424
_________________________________________________________________
dropout_9 (Dropout)          (None, 40, 1024)          0
_________________________________________________________________
dense_18 (Dense)             (None, 40, 1)             1025
=================================================================


texts: Steps for back propagation of convolutional layer in CNN

texts:
Imagine I have the following layers in a CNN:

Conv-layer 1 (without reLU) [3 filters @ 1x3x3] =>
ReLU-layer 1 =>
Maxpooling-layer 1 [2x2]

Conv-layer 2 (without reLU) [10 filters @ 3x4x4] =>
ReLU-layer 2 =>
Maxpooling-layer 2 [2x2]

What I know is that, when "deriving" the Maxpooling-layer, the maximum value of that pooling has the value of 1 (because that value is the only one that is affecting the result of the CNN), and the other values take the value of 0.

After this step, I have the reLU-layer, where the derivative of the reLU-layer is 0 when x <= 0, and 1 when x > 0.

And in the last step, I have the Conv-layer, but I don't know how to calculate the derivative. How do I calculate the whole delta for that specific weight?

If there is a way to calculate the gradients of those weights using matrix notation, please let me know.

Notation

The matrix notation that I use is the following: [number of filters - depth - rows - columns]. For example, when I say [3 filters - 1 x 3 x 3] I mean that I have 3 filters with depth 1, 3 rows and 3 columns (depth is "equivalent" to "channels").

Steps in Conv layer, using my notation

Imagine I have an input X with shape [depth:1, rows: 28, columns: 28]

Then, I have the conv-layer, where I have 3 filters (you can call the filter as W) with shape [depth:1, rows:3, columns:3]

So, what I do is, using 1 filter a time, convolve input X , so the resulting matrix has the following shape => [depth:1, rows: 26, columns: 26] (this is the result of convolving 1 filter). So, when I do the convolution using the 3 filters, the resulting matrix is shaped [depth:3, rows: 26, columns: 26].

After this, I apply ReLU to each value of the resulting matrix (the 3x26x26 matrix), and then apply non-overlapping maxpooling with a window of 2 rows x 2 columns.


texts: What classifier is best to use for multi-class number outputs?

texts:
I have 50 features which are real numeric values, and the outputs are 5 numbers. How can I best construct a classifier for this using sci kit learn?
I think Random Forest would be good but what else could I use?


texts: Minimum Error Rate Training using Powell Search for Machine Translation

texts:
From the tutorial slides: http://mt-class.org/jhu/slides/lecture-tuning.pdf, (slide 37) the powell search algorithm goes as such:

Input: sentences with n-best list of translations, initial parameter values
1: repeat
2:   for all parameter do
3:     set of threshold points T = {}
4:     for all sentence do
5:       for all translation do
6:         compute line l: parameter value → score
7:       end for
8:       find line l with steepest descent
9:       while find line l2 that intersects with l first do
10:        add parameter value at intersection to set of threshold points T
11:        l = l2
12:      end while
13:    end for
14:    sort set of threshold points T by parameter value
15:    compute score for value before first threshold point
16:    for all threshold point t ∈ T do
17:      compute score for value after threshold point t
18:      if highest do record max score and threshold point t
19:    end for
20:    if max score is higher than current do update parameter value
21:    end for
22: until no changes to parameter values applied

And on slide 31, there's a formula to compute the line l which is used for line 6 of the pseudo code above:

p(ei|f) = λai + bi

However, it's unclear what should the values be for ai and bi.

Given a sentence from the nbest list as such:

0 ||| including options , чтобы buy 20 больше planes , соотношении volume - в том 26 миллиардов долларов .  ||| LexicalReordering0= -3.10978 0 0 -6.72232 0 0 Distortion0= 0 LM0= -111.207 WordPenalty0= -18 PhrasePenalty0= 16 TranslationModel0= -13.1426 -8.45991 -12.0762 -11.3076 ||| -746.351

When computing the score for this line for the first parameter of LexicalReordering0, is the p(ei|f) this?:

p(ei|f) = λ*-3.10978 + bi

But what is the value of bi in this case?

Is p(ei|f) = -746.351 for this sentence ?

Does the → score on line 6 of the pseudo code mean -746.351 from the nbest list or does it mean the MT evaluation metric score for this translation with respect to the reference translation (e.g. BLEU)?

What should line 6 of the pseudo code do?


texts:
I'm looking for some method/model to help me with my current problem:
I have a geometry, consisting of points, and eges. For each point I take information about itself and its neighbours. For now I put these information inside a vector, which works as feature set. all of the info are continuous values.
Then, my target can intuitively be seen as another type of information of geometry regarding the same node for which we extract the info. Also the target is represented as a vector.
For now I had my Deep NN working on this kind of training set, where each example consist of (info on some node_i, relative target). Thus, the NN performs a multi-target regression
The problem now is that this kind of model can work if I have only the same amount of information both for features and target. Seeing it again in the geometry, it only works with nodes which have the same fixed number of neighbors.
Now, I would like to generalize this model, accepting also nodes with less neighbors, which would give rise to less features and also less target values.
To stay inside the geometry example, if you think about a chessboard, my model works for internal nodes, having always 4 neighbors, but if I am on the boundary this cannot work anymore.
For these reason, I started to study Graph neural networks, since we can see our geometry as a graph with nodes and edges.
But it is not clear to me if I can use a training set where each example is itself a graph. Furthermore, I'm not sure it could solve the problem of the fixed-dimension constraint.
Is there any model or method or preprocessing which can suit my acutal problem?


texts: how to create a searchable tree on Persian text?

texts:
I wanna clean a Persian text from stop-words. I already have stop-word data that is provided on the below link. It seems to me, if I have a pre-built tree on stop-words, I could save lots of time. I want to search each word of text in this pre-built tree, if the word is in the tree I delete it from the text, if not I hold it.

O(n * l) to O(n*log(l)).

This is my stop-words

If you have better suggestions than the pre-built tree search, I would be grateful to share it with me.


texts: Group datetime64 values per week in dataframe

texts:
I want to plot a bar of my data grouped per week. I made the following dataframe, with nonsense data, but it should do the trick.

What I want to do is plot per week a bar chart (%)

Does anyone know how I can do this? I'm using datetime64 so can this be done easily?

I want to make one graph which contains 52x3 bars.

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import random

date_expected = np.arange('2005-06', '2006-07', dtype= 'datetime64[D]')
cat = ['True','False', 'Maybe']

value = [random.choice(cat) for i in range(len(date_expected))]

data = {'Date_expected': date_expected,  'Value': value }
df = pd.DataFrame(data)
df = pd.DataFrame(data)

print(df.head())


texts: Label A records B times or label A*B records

texts:
This question concerns pre-training data sourcing.

Suppose you have a human workforce of B individuals and a potentially unlimited source of data.

The task is labeling images with classes. These classes are somewhat subjective (emotions). This means one individual might label the same image with a different class than another individual.

For then using these labeled records as training data on a neural network that predicts classes on images, is it better to

1) have a number of records (A) labeled redundantly by all B individuals.

2) have every individual label A different records each, yielding A x B labeled records.

Intuition behind 1) is that the mean of subjective labeling would be somewhat objective. Thus training data would be mostly objective.
In addition, probabilities (50% happy, 50% surprised) could be used as input.

Intuition behind 2) is that subjectiveness in labeling of individuals is natural and the NN is trained on that, becoming somewhat "general"/"objective" in it's predictions. Also, more data is always better.

Please excuse the use of subjective and objective in combination with Machine Learning. I know this might not be correct at all.


texts:
I have a dataset of vectors representing movement with various characteristics. Some vectors represents the movement that was stopped by external factor and therefore, observed value for length of such a vector  (v_length) is incomplete (marked as incomplete == 1). The data looks like below:

# A tibble: 10 x 9
   v_length incomplete v_angle    x0    y0    x1    y1 type    vap
      <dbl>      <dbl>   <dbl> <dbl> <dbl> <dbl> <dbl> <chr> <dbl>
 1     1.70          1   0.869  66.6   0.5  67.7   1.8 A         0
 2     1.82          1  -0.165  37.4  65.6  39.2  65.3 B         0
 3     2.57          1   0.236  61.3  49.7  58.8  49.1 A         0
 4     3.14          1   1.18   57.8  40.5  59    43.4 A         0
 5    12.6           0   0.119  52.5  33.7  65    35.2 A         0
 6    20.5           0  -0.847  65.3  32.3  78.9  16.9 A         0
 7    33.0           0  -0.180  77.5  13.7  45    19.6 A         0
 8    14.1           0  -0.780  45    19.6  35    29.5 B         0
 9     2.97          0   1.00   35    29.5  33.4  27   B         0
10     6.59          0   0.732  33.4  27    38.3  31.4 A         0

I want to impute a v_length for incomplete observations (incomplete==1). My first idea was to use some parametric survival model (e.g. Weibull). But as I'm not experienced in Survival analysis I have been struggling with a good setup. My first doubt is if it is correct to use v_length as one of the predictors as well? It doesn't make sense at first sight, but the predictions for the model without v_length as one of predictors looks very strange:

The idea behind inclusion is to help the model know what was the observed vector length, so it can predict a value higher than that. After inclusion of v_length in predictors the output looks like below:

However, we still have plenty of values lower than actual vector length, while I obviously don't want a model to predict a lower value than observed.

So here's my question: is Weibull survival model suitable for this task? What's the correct setup if so? What are the other methods suitable for imputation of right-censored data?


texts:
I'm working on Housing Price dataset, where target is to predict the housing price.
Price of the house will always be positive and according to me, it's possible that model can predict a negative outcome for some of the samples.

If it's correct, is there any way to control the training such that
model always predict at least the positive value.

As in case of classification case we use Sigmoid/Softmax activation
function to normalized the outcome in probability. Can we have some
activation function for positive value?

Can I use Poisson loss?


texts: Is there an analog to SQL's STRING_AGG (or FOR XML PATH) function in Python?

texts:
Asked this in SE but maybe this is too data-oriented so trying to post it here. I am trying to find the analog to the SQL function STRING_AGG so that I can concatenate across columns and rows of a table (or dataframe).

Here is an example of what I am trying to achieve:

input:

output:

With SQL I can easily group by the ID_No and also specify the order by via the RUN_No. The syntax for achieving what I want would be:

SELECT ID_NO,
STRING_AGG(CONCAT('(', RUN_No, ') ', Start, ' to ', Stop))
WITHIN GROUP (ORDER BY RUN_No ASC) AS "Sequence"
FROM X_TBL GROUP BY ID_NO

So what would be the way to achieve the same grouping, concatenating and ordering in Python? I do have my data stored as a dataframe. I am able to concatenate across columns using the following code, but then wasn't sure how to group by the "ID_No", or concatenate across the rows within each ID_No.

sample['Merge'] = sample['Start'].map(str) + ", " + sample['Stop']


texts: Should I try to predict the probability of being 1 or zero based on recall score?

texts:
I built a random forest model to classify if a men's NCAA basketball game would go over or under the total. The model was given training data that specified over as 1 and under as 0. I then used a classification report to calculate the precision, recall, f1-score, and support. These were my results:

for a model like this when my goal is to accurately predict whether the test data will have an outcome of 1 or 0 should I use recall in my decision for choosing whether to try to predict if the outcome will be a 0 or 1?
Also I had my model using predict_proba and using the second value from the array return which would give me the probability of the test data outcome being a 1, but from this classification report it looks like I should be trying to predict the probability of the outcome being 0 as it is more descriptive when it comes to recall.
# Example:
# less accurate way

ynew = model.predict(Xnew)

results = ynew[0][1] # predict probability of Xnew being 1

# Example 2:
# more accurate way

ynew = model.predict(Xnew)

results = ynew[0][0] # predict probability of Xnew being 0

Is my assumption correct?


texts: Convert graph database relationships into node properties

texts:
I have a neo4j database that stores forum posts. Replies to specific posts are encoded using a replyto relationship. I am trying to retrieve all the information from the database, convert the relationships into node properties, then output all the nodes with the new replyto property in JSON format. I think the replyto property in the JSON object should be an array of all the post IDs that are replies to the given post, but please suggest something better if you think of it.

What cypher query should I use to retrieve this data and how should I convert the relationships into node properties using javascript?


texts: How to apply my deep learning model to a new dataset?

texts:
I am doing semantic segmentation (multi-class classification of image pixels) using convolutional neural networks (CNN) in Keras.

In particular, I am applying this to aerial images of crops (vegetation). In Keras, I successfully developed a workflow to segment/classify different crops for one specific dataset (let's call this dataset rural area #1).

Can I apply my Keras weights trained on rural area #1 for initializing the training of another dataset rural area #2? Such as:

model = load_model("weights_ruralarea1.hdf5")

Then I will proceed to model.fit.

The rural area #2 dataset has little training images for training the CNN. So would using my weights for rural area #1 be a form of transfer learning? or will it be a form of fine-tuning?


texts: Can we use sentence transformers to embed sentences without labels?

texts:
I was trying to use this project :
https://github.com/UKPLab/sentence-transformers
for embedding non english sentences, the language is not a human speaking language, its machine language (x86)
but the problem is i cannot find a simple example where it shows how can i embed sentences using a custom dataset without any labels or similarity values of the sentences.
basically i have an array of sentences lists without any labels for sentences or similarity values for them, and i want to embed them into vectors in a way that it preserves the semantic of the sentence the best way possible, so far i have used word2vec and doc2vec using gensim library so i wanted to try this method to see if its any better?


texts: Is it possible to use unlabeled text articles for summarization when fine tuning BERT?

texts:
I know that unlabeled data could be used in pre-training but if I want to do a fine tuning of unlabeled articles for summarization, is it mandatory that the articles are labeled with existing summaries (for example CNN daily mail dataset). The reason I want to do this is because I'm using a non-english pre-trained BERT model. Do you otherwise think it's possible to use some sort of statistical technique to get a saliency score which can be used together with the pre-trained BERT model?
PS. I'm new to this domain, so I'm sorry if there's any misunderstanding.


texts: Is there a dataset that can predict duration to complete a task using task description?

texts:
I am working on a problem where the task is to predict how long will it take to complete a given task using just the description as a feature set. The description is something like "pick up X and place at y".
Currently what I have done is vectorized the descriptions using tensorflow feature embedding: a categorical column with vocabulary list, and an embedding column with dimensions set to 100.
Since the dataset I have only consists of 6 rows, and my client wants to see if the same model can be trained on a larger dataset I am looking for something similar. Therefore, please advise, also am I doing this the right way ?.


texts: Good mathematically explained algorithm for Hyperparameter Optimization (Bayesian) for implementing in Java

texts:
I have implemented Random Forest, Bagging, Gradient Boosting, etc... in java myself. It took a long time to complete these machine learning algorithm codes. But at last all of them are well running. Now I am facing problem in tuning all the hyperparameters corresponding to aforementioned ML algorithms to get the approx. best result. In python or R, there are already some packages (like random search, grid search or Bayesian hyperparameter optimization) which gives best approx. hyperparameters for the corresponding ML algorithm. Now for Gradient Descent or $k$-means Clustering or KNN I have already implemented mathematical codes in Java for best choosing $k$ and learning rate $\alpha$, which will give best approx. result. But for Random forest, Bagging, Gradient Boosting, etc... like algorithms, I want to implement Bayesian hyperparameter optimization in Java. I have go through many books and papers like, enter link description here, enter link description here. But I am not able to understand their algorithms so that I can implement them in Java. These are not much simply I know its may be difficult, but I have already implemented Random Forest and blah blah, so I think I can code this in Java if I get some simple algorithms which will help me to read and think.
If anyone has already implemented this model (Bayesian Optimization for Hyperparameters) in Java or elsewhere thoroughly, please refer me some books or paper from where I can read and can successfully implement in Java, like you.
Please help me to solve this coding problem. Thanks in Advance.


texts: Non-monotone missing data, and inverse probability weighting

texts:
I'm having difficulty identifying whether or not my missing data pattern is 'monotone'.

I have two variables with missing data, and the missing data patterns in each variable do not completely overlap. I.e., all the below combinations are possible:

A  B
1  0
0  1
1  1
0  0
0  NA
NA 0
NA 1
1  NA
NA NA

Does this technically make my missing data pattern 'non-monotone'? Note that this is not repeated measures data - the two variables in question are independent of each other.

The reason I'm concerned about this is because I want to adjust for missingness using inverse probability weighting, and apparently it is debated whether or not this strategy can be used with non-monotone missing data.

Many thanks,
Alice


texts: Sklearn train_test_split() error: Found input variables with inconsistent numbers of samples

texts:
I am fitting a regression model on randomly generated X1,x2 and Y be the sum of x1, x2 but I am getting this error

  ValueError: Found input variables with inconsistent numbers of samples: [2, 10000000]

Note:- I am doing this only for learning purposes

My code:-

X = np.random.random_integers(100000000,size=(2,10000000))
X=(X-(100000000/2))/(100000000/2) # Scaling [-1,1]
Y = X[0]+X[1]

regr = linear_model.LinearRegression()
X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.2,
random_state=0)
regr.fit(X_train,Y_train)


texts: what actually word embedding dimensions values represent?


texts: Perform Pearson's correlation and chi-squared test for feature selection in a dataset with a mixed type of features

texts:
I've a dataset of about 200 features and 5000 instances. These features comprise of different data types like percent (string like 4.50%), dollar amount (value between $0 - $1,000,000), discrete distribution as string (colors like red, blue), etc.

There are lots of different format in the data. I can't use a regex pattern of a feature in another feature.

So how can I convert these features into float (for correlation feature selection), or into some categories (for chi-squared test and one way ANOVA test)?


texts: how can i sum first day value of each id together?


texts:
Why don't the gym environments come with "valid actions"? The normal gym environment accepts as input any action, even if it's not even possible.

Is this a normal thing in reinforcement learning? Do the models really have to learn what valid actions are all the time? Would it not be much nicer to have a env.get_valid_actions() functions so that the model knows what actions are doable? Or is this somehow possible and I'm missing it?


texts: Best tool for text pre-processing, involving tokenization, lemmatization, stop-word removal, feature vector extraction?

texts:
For text processing there are plenty of tools out there like CoreNLP, SpaCy, NLTK, textblob etc. each offering different suites of pre-processing functions and people recommend different tool for different tasks like NLTK for tokenization etc.

Which tool would you recommend which performs all these tasks?


texts: Perplexity increasing on Test DataSet in LDA (Topic Modelling)

texts:
I was plotting the perplexity values on LDA models (R) by varying topic numbers. Already train and test corpus was created.

Unfortunately, perplexity is increasing with increased number of topics on test corpus. I am not sure whether it is natural, but i have read perplexity value should decrease as we increase the number of topics.

Also, i plotted perplexity on train corpus and it is decreasing as topic number is increased. I am not sure whether it represent over-fitting of my model.

Please help me understand

Perplexity on Train Data

Perplexity on Test Data


texts: using word embedding features with linear prediction models

texts: I have been seeing that word embedding features (e.g. here or there) are used on classification or regression tasks where the classifier/regressor is a linear one: e.g. Linear/Logistic Regressor or GBM. However, I cannot really understand how these regressors/classifiers should interpret the word embedding dimensions and be able to capture the cosine relationship between these dimensions. Can anyone recommend a paper/blog post that can answer my question?


texts: Are Naive Bayes algorithms affected by outliers in the data? Suppose there is a data set, does one need to remove outliers before applying Naive Bayes?


texts: Is it good practice to transform some variables and not others?

texts:
I have a dataset with categorical variables encoded into numeric values, other variables that are continuous and have many outliers, and other continuous variable with a fairly normal distribution.
I was planning to use the sklearn preprocessing method .PowerTransformer in order to transform all of them, but maybe it might make more sense to just use it for those columns that have not normal distribution at all and many outliers?
It's for a classification problem (the Titanic machine learning one).


texts: Can we custom / teach machine learning to put-attention at a feature or make a feature the most importance in model?

texts:
I have real dataset that contains place, address, and category of two places. I will match those name and address but I will give exception based the category column.
This the example of dataset stucture
id     is_park     place_name    address                                         places_name_candidate_match   address_candidate_match                            match
1      False      Liberty Bell  143 S. 3rd Street, Philadelphia, Pennsylvania   Liberty Bell                  S. 3rd Street, Pennsylvania                        True
2      False      Liberty Bell  143 S. 3rd Street, Philadelphia, Pennsylvania   Liberty Bell                  100 Hersheypark Drive, Hershey, Pennsylvania       True
3      True       Hershey Park  100 Hersheypark Drive, Hershey, Pennsylvania    Hershey Park                  S. 3rd Street, Pennsylvania                        False
4      True       Hershey Park  100 Hersheypark Drive, Hershey, Pennsylvania    Hershey Park                  100 Hersheypark Drive, Hershey, Pennsylvania       True

The scenario is that if the place is not a park, it will match if the place name is the same even though the address is different.
but if the place is park, then place and address must have high similarity in order to be true.
it would seem easy to teach the machine if there was an is_park feature. but the problem is, the dataset that I have to teach the scenario engine 'if park then place and address must match to have a true value' is still very few, so the model doesn't catch the pattern. (the dataset I have is the original dataset, so it only has limited data).
so, can I modify / order / teach the machine to put-attention to a feature in this is_park?
Addition: I changed all text values to numeric by calculating similarity, length, and others to be processed when made into a model.
So far, I use Random Forest to create a model, is it possible for that? or have to use another algorithm? let me know, thank you!


texts: is this grid search methodology correct?

texts:
I have some data say 1 million rows, I then put 200k aside (to validate against) and call this remaining 800,000 the training set (X) as you see below, so it is not the entire data and the remaining 200k is the validation set.
My question is, is the below grid search ok? My roc curves were extremely high say 0.96. I suspect it is because of the content below, should I instead use grid search on x_train and y_train instead then further split x_train and y_train again to use on pipeline.fit()??

X=pd.read_csv('data_subset.csv')

X=X.dropna()
Y=X['status'].values
X=X.drop(columns=['status'])

x_train, x_test, y_train, y_test = train_test_split(
    X, Y, stratify=Y, random_state=42, test_size=0.2)

model = lgb.LGBMClassifier(silent=True, subsample=0.8,colsample_bytree=0.2,objective='binary')

parameters={
    'learning_rate_grid_lgbm': [0.015, 0.001, 0.1],
}

i then run grid search on this data:

clf = GridSearchCV(model, parameters, cv=3, scoring='roc_auc')
clf.fit(X, y) ### i.e. instead clf.fit(x_train,y_train)

model = clf.best_estimator_

pipeline = make_pipeline(model)

pipeline.fit(x_train, y_train)
predictions = pipeline.predict_proba(x_test)[:, 1]
roc_auc_score(y_test, predictions)

my confusion is, if I have to use x_train on gridsearch, how should I proceed with the steps after?


texts:
I have a insurance dataset as given below. For which I need to build a model to calculate the charges.

   age     sex     bmi  children smoker     region      charges
0   19  female  27.900         0    yes  southwest  16884.92400
1   18    male  33.770         1     no  southeast   1725.55230
2   28    male  33.000         3     no  southeast   4449.46200
3   33    male  22.705         0     no  northwest  21984.47061
4   32    male  28.880         0     no  northwest   3866.85520

But I am not sure whether the column "region" can be dropped. Is there any test can be performed to consider only significant variables?


texts: How can we define missing rating in recommender system?

texts:
I was reading about collaborative filtering where we need to pass (user, item and rating) in case of matrix factorisation (SVD). Now, my question is given data of following form

User |   Item   | Rating
A    |   X1     | 1
A    |   X2     | 3
B    |   X2     | 4
C    |   X1     | 3
C    |   X3     | 2

we need to convert the it into

U/I       | X1  | X2  | X3
A         |  1  | 3   | -
B         |  -  | 4   | -
C         |  3  | -   | 3

So, we need to replace all the - with the predicted value prior to applying svd on it. Now I would like to understand what are most practical or mostly adopted way to predict such missing rating:
I am aware about following crude ways, but they are not prediction of value but they are mere replacement of missing value

Replace all missing values by neutral rating . (Say either 2 or 3 if rating are from 1-5).
Replace with mean rating of the movie.
Replace with average rating of that user.
etc...


texts:
I am using a library called MFE to generate meta-features. However, I am working right now with several files and I notice that I am using only 1 core of my machine and taking too much time.

I have been trying to implement some libraries as I saw in another question:
library(iterators)
library(foreach)
library(doParallel)

This one, but me being dumb could not implement it ='(.

I just would like to put this snippet running in all my cores so I can make it faster:

## Extract general, statistical and model based measures
metafeatures(Species ~ ., iris, groups=c("general", "statistical", "model.based"))

So how would I do this? MFE library

[EDIT]:

What I am doing right now:

library(iterators)
library(foreach)
library(doParallel)
library(mfe)

# foreach
split = detectCores()
eachStart = 25
# set up iterators
iters = iter(rep(eachStart, split))
# set up combine function
cl = makeCluster(split)
registerDoParallel(cl)
result = foreach(nstart=iters) %do%
  metafeatures(Species ~ ., iris, groups=c("general", "statistical", "model.based"))
stopCluster(cl)

result

SAMPLE OF EXPECTED RESULT
What I am expecting is this following output and using all cores from my CPU.


texts: Transform a multiclass dataset into a multi-label one

texts:
I have a dataset of feature/label pairs. My labels are probabilities of each feature vector to belong to the K classes. Here is an example for K = 3:

D1 = { (V0, [0.33,0.33,0.33]), (V1, [0.9,0.07,0.03]), (V2, [0.5,0.25,0.25])... }

The probabilities are normalized for a given data point. Yet the task is more a multilabel one, and it would make more sense to have independent Bernoulli distributions e.g.

D2 = { (V0, [0.9,0.9,0.9]), (V1, [0.99,0.0,0.0]), (V2, [0.9,0.2,0.5])... }

Is there a trick (smart heuristic) out there which would allow me to transform D1 into D2 based on the way the probability weights are distributed in D1?


texts: How to use class_weight parameter for validation set?

texts:
I am using Keras' class_weight parameter to deal with an imbalanced class problem. I am doing this to define the weights :
weights = class_weight.compute_class_weight('balanced',np.unique(trainY),trainY)

then, in my network:
model.add(LSTM(..., class_weight=weights,...,callbacks=callbacks_list))

However, I also use a callbacks list to prevent overfitting, by imposing EarlyStopping based on validation_accuracy ...:
earlystop = EarlyStopping(monitor='val_acc', min_delta=0.001, patience=5, verbose=1, mode='auto')
callbacks_list = [earlystop]

However, my weights are only defined based on my training set... But my validation set also contains imbalanced data, in different proportions than the training set, and I would also like to give appropriate weights to have a fair evaluation of val_loss parameter...
So my question is

With which weights is the val_loss computed ? With the weights given in class_weightparameter ? Or is this parameter only used for training loss ?

How could I define weights that could be used for the validation set, to have a more accurate value of val_loss ?

Thanks


texts: How to order a python dataframe by adding the row values?

texts:
I have the following dataframe:

    M1  M2  M4   M5
N1  45  46  54   57
N2  32  36  29   56
N3  56  44  40   55
N4  57  43  42   54

How is it possible to order it according to the sum of its lines, from the largest sum to the lowest? That is, N1 in the first line, because the sum of the line values is the largest, then N4, N3 and finally N2 (because the sum of the values is the smallest). Thus:

    M1  M2  M4   M5
N1  45  46  54   57
N4  57  43  42   54
N3  56  44  40   55
N2  32  36  29   56


texts: When using an auto encoder to create non-linear dimensional reduced featires, is it more common to use the output of the network (the prediction of the input features) or to use the weights from the (or 1 of the if there are multiple) hidden layers? If the hidden layer is used, do you use the hidden layer activation as features or weights from the hidden layer to the output?


texts:
As there are numerous tools available for data science tasks, and it's cumbersome to install everything and build up a perfect system.

Is there a Linux/Mac OS image with Python, R and other open-source data science tools installed and available for people to use right away? An Ubuntu or a light weight OS with latest version of Python, R (including IDEs), and other open source data visualization tools installed will be ideal. I haven't come across one in my quick search on Google.

Please let me know if there are any or if someone of you have created one for yourself? I assume some universities might have their own VM images. Please share such links.


texts: How do I get the name of Sagemaker Estimator's job

texts:
I'm having a stumbling block with SageMaker. How do I know what my job name is?

For example:

mnist_estimator = MXNet(entry_point='/home/ec2-user/sample-notebooks/sagemaker-python-sdk/mxnet_mnist/mnist.py',
                        role=role,
                        output_path=model_artifacts_location,
                        code_location=custom_code_upload_location,
                        train_instance_count=1,
                        train_instance_type='ml.m4.xlarge',
                        base_job_name=’foo’,
                        hyperparameters={'learning_rate': 0.1})

and then when I call fit() it prints out:

  INFO: Creating training job with name: foo-2018-01-10-20-13-57-893

and when I look in my S3 bucket I see:

  2018-01-10 15:20:45     411784 artifacts/foo-2018-01-10-20-13-57-893/output/model.tar.gz

So my job name is "foo-2018-01-10-20-13-57-893" and not "foo", but how can I get this from python? I guess I'm looking for a way of extracting that from the Estimator itself, but I'm just blanking on it. I'm reading the python source but that's not helping me, probably because I'm just learning python also.


texts: Results Random Forest Regression Optimization

texts:
I'm going to optimize the hyperparameters of a Random Forest Regressor using scikit-learn and GridSearchCV, with the following code:
from sklearn.model_selection import GridSearchCV
param_grid = {
    'bootstrap': [True],
    'max_depth': [30,40,50,60,79],
    'min_samples_leaf': [5,10,15,20],
    'min_samples_split': [10,12,14,16,20],
    'n_estimators': [80,100,110,120]
}

rf = RandomForestRegressor()

grid_search = GridSearchCV(estimator = rf, param_grid = param_grid,
                          cv = 10, n_jobs = -1, verbose = 2)
grid_search.fit(X_train, y_train)
def evaluate(model, test_features, test_labels):
    predictions = model.predict(test_features)
    errors = abs(predictions - test_labels)
    mape = 100 * np.mean(errors / test_labels)
    accuracy = 100 - mape
    print('Model Performance')
    print('Average Error: {:0.4f} degrees.'.format(np.mean(errors)))
    print('Accuracy = {:0.2f}%.'.format(accuracy))

    return accuracy

print (grid_search.best_params_)

best_grid = grid_search.best_estimator_
grid_accuracy = evaluate(best_grid, X_test, y_test)
print (grid_accuracy)

The performances after the optimization decrease in terms of RMSE. Basically, just using the standard library call the RMSE is lower than the RMSE with hyperparameters optimization. is it possible?


texts:
Introduction

I've implemented linear regression using sklearn and after all calculations I've got results like this:

Feature: 0, coef: -9985335237.46533
Feature: 1, coef: 417387013140.39661
Feature: 2, coef: -2.85809
Feature: 3, coef: 1.50522
Feature: 4, coef: -1.07076

Data

My data is based on user visits in gym. All data normalized 0 <= x <= 1. Data set has 10k observations.

X:

feature_0: gym's rating
feature_1: gym's review(rating) count
feature_2: gym's one visit price
feature_3: gym's unlimited subscription price
feature_4: distance to gym from user's home | calculated min(x / 30, 1.0), because mean is 15.17

Y: user's visit count to that gym

Data sample

Code

from sklearn.datasets import make_regression
from sklearn.linear_model import LinearRegression
from matplotlib import pyplot
from numpy import loadtxt

# define dataset
x = loadtxt('formatted_data_x.txt')
y = loadtxt('formatted_data_y.txt')
# define the model
model = LinearRegression()
# fit the model
model.fit(x, y)
# get importance
importance = model.coef_
# summarize feature importance
for i,v in enumerate(importance):
    print('Feature: %0d, coef: %.5f' % (i,v))

Question

Why linear regression feature coefficients become super large? Is it okay?

Feature: 0, coef: -9985335237.46533
Feature: 1, coef: 417387013140.39661
...

P.S: I'm new to this "part" of StackExchange and ML\DS at all, so please if I do something wrong or I have to provide more information, let me know! Any help would be appreciated. Thanks in advance!


texts: Which algorithm to use to identify clusters with a similar value?

texts:
Here, an example of my problem:
10000 observations of people with several features [age, gender, region, number of sons, ...] and a value to predict "income".
There is not a general relationship between features and income, therefore a normal regression has poor results.
Nevertheless, I want to identify specific patterns where this relationship exists. For instance:
[young, woman, 2 son] -> high income
[young, man] -> small income
...

Maybe doing a clustering on the features, and then a regression on each cluster?
Or pattern recognition? topic modeling?

Thank you in advance


texts:
Bit of a noob in this stats world, so apologies in advance for any naiveté. I did a fair bit of stats long ago in college but it's a distant memory, so please assume little knowledge!

The dataset I've got is of the amount of money spent by medical practices in a given month. There already exists a formula that weighs the population (produced long ago using different data) to account for ages (old patients cost more, 20-somethings less etc.), however there are lots of other factors, that we have data for, it isn't weighing. I'd like to try and do that.

Provisional regressions demonstrate a reasonable correlation between, for example, '% of patients with a long term condition' and the existing weighted spend per head of population.

So my question is: what approach should I take to improve the population weighting to account for these other factors, for which there is a correlation.

Ideally I'd like to generate a formula such as:

Improved weighted population = Existing weighted population * (x * %long term condition) * (y * population deprivation levels)...

The aim would be to improve the error between the current prediction (spend__per_astroPU) and the observed true result (spend_per_raw_head).

I've put a sample of the dataset below, and a link to a python notebook with the full dataset imported into dataframes.

drug_spend  Total Population    astroPU_weight  ASTROPU2013_Population  spend_per_raw_head  spend__per_astroPU  % with a long-standing health condition
0   804443.050      4150    1.146   4757    193.842 169.107 58.230
1   17534209.330    19886   1.130   22478   881.736 780.061 60.218
2   3593560.340     9471    1.033   9786    379.428 367.214 61.756
3   3043412.970     7929    1.272   10089   383.833 301.657 57.046
4   11163851.800    13733   1.033   14189   812.922 786.796 54.217

Python Notebook


texts: How to decide which method to use TFIDF, or BOW

texts:
In a huge dataset for NLP it is taking very long time to classify my dataset
therefore, trying each feature extraction method separetly is time consuming and not effecient.
Is there a way that can tell me which method (TFIDF or Bag Of Words) is more likely to give the highest F1 score.
I tried test them on smaller subset (1000 records) it was fast but best method in smaller subset does not mean it is the best in complete dataset.
any other way to decide which method to use?


texts: Using Neural Networks on images to predict successful vs unsuccessful trades

texts:
The context for the problem is day trading: identifying the moment when one can buy units of a stock and then wait for them to grow in value before selling them at the new price. Currently I have a system already developped to identify such moments, but it yields a high number of false positives. I have previously tried some more classical Statistics to come up with estimates of the error rate depending on factors such as the behavior of Bollinger bands, and whether the tendency changes when analyzed at different time scales, among others. But these have been ad-hoc follow the nose attempts. I am trying to leverage neural networks to see about automating this process.

The current approach: for each moment we identify as having potential, create a snapshot of the candlesticks (prices) leading up to the moment, and then see if the moment turned out to be successful or not and use that as label. Feed the images and labels to a convolutional neural network to perform classification in order to weed out false positives. Currently the training data has about 3000 pictures, almost equally split between the successes and failure cases.

Problem: The neural network's performance is very poor, about 50% accuracy, and not getting better even after 100's of epochs, even when only evaluated on the training data. Changing optimizers and other parameters (steps per epoch, learning rate) has not helped so far.

I am trying to diagnose if this is a badly thought out use of neural networks, a matter of not feeding it the right data or not enough of it, or just a matter of not using the correct parameters for the neural network (and how I can find out which of these it is). Does anyone have any advice on how to proceed?


texts: What are the possible ways to handle class unbalance in a large scale image recognition problem with Deep Neural Nets?

texts:
I have 22 classes of objects but they have very skewed distributions where max class has 100.000 images and the min class has 1600 images. In that setting I would like to hear some possible solutions to this balance problem.

I have tried followings so far;

multiply number of instances in the lower classes up to the max class by replicating the instances, possibly adding some noise as well.
changing the learning regarding the class distribution in the given minibatch of the next epoch. (no implemented but in my mind)

What are your suggestions?


texts:
I am trying to implement the following idea.

For a daily newsletter I would like to generate an engaging and funny intro text, such as:

  Good morning. Sorry if there are beer stains and buffalo sauce smeared
  on the Brew this morning. Yesterday was the sports equinox, a very
  special day in which all four major U.S. sports leagues (NBA, NHL,
  MLB, and NFL) were in action, plus Tiger and the Premier League.

  In related news...our Nats in 7 prediction? It's in the realm of
  possibility.

As these texts are custom written I find it extremely hard to generate the text.

I was thinking of using an NLG template or on the other hand creating a template that adds daily news articles.

Any suggestions how you would approach this idea?

I appreciate your replies!


texts: Most important part of feature standardization and how is standardization affected by sparsity?

texts:
I am thinking of preprocessing techniques for the input data to a convolutional neural network (CNN) using sparse datasets and trained with SGD. In Andrew Ng's coursera course, Machine Learning, he states that it is important to preprocess the data so it fits into the interval $ \left[ 3, 3 \right] $ when using SGD. However, the most common preprocessing technique is to standardize each feature so $ \mu = 0 $ and $ \sigma = 1 $. When standardizing a highly sparse dataset many of the values will not end up in the interval.

I am therefore curious - would it be better to aim for e.g. $ \mu = 0 $ and $ \sigma = 0.5 $ in order for the values be closer to the interval $ \left[ 3, 3 \right] $? Could anyone argue based on a knowledge of SGD on whether it is most important to aim for $ \mu = 0 $ and $ \sigma = 1 $ or $ \left[ 3, 3 \right] $?


texts: Can't understand Output shape of a Dense layer - keras

texts:
I am following an online tutorial to classify images and started off with dense layers as a starting point to classify cifar10 data.

# Create a model and add layers
model = Sequential()
model.add(Dense(512, activation='relu', input_shape=(32, 32, 3)))
model.add(Dense(10, activation='softmax'))

# Print summary
model.summary()

dense_1 output shape (None, 32, 32, 512). 'None' represents the batch size, but what does '32,32' represent? Why isn't the shape (None, 512)? Same happens with the dense_2 layer.

Can someone explain it here or point me to a resource that explains this?


texts: Should we sample z in VAE encoder during inference, when used in RL pipeline like World Models?

texts:
My question has been motivated by reading World Models by Ha and Schmidhuber. In shortcut, they introduce a RL framework where the current state (an image) is encoded via VAE into a latent vector $z$, which is then used as a state representation for the agent. Specifically, they first train the VAE on the images of their simulator, then freeze the model and use only the encoder for abstracting the image input for training the rest of their pipeline.
Now during training they of course sample the latent vector $z$ from $\mu$ and $\sigma$ that the encoder predicts, but I am wondering now: Should this still be the case when the whole pipeline finished training and is used for inference? Wouldn't the sampling process introduce stochasticity into the pipeline which at this point should be (at least in my eyes) undesirable in use-cases like the ones in the paper (e.g. the self-driving car simulation)? I feel like just taking the predicted mean as our $z$, thus removing the stochasticity, would maximize the exploitative behaviour of the agent, or am I missing something?
Thanks in advance for your thoughts.


texts: How to give breakdown of text prediction sentences in Shiny

texts:
I have created a text prediction app that displays 10 sentences based on the word searched in the toolbar. However, all the sentences are being displayed in a single toolbar horizontally. I would like the sentences to appear sequentially i.e. vertically, one after another
    ui<- shinyUI(fluidPage(

  # Application title
  mainPanel(
   img(src='image.jpg', align = "right"),

  #titlePanel(title=div(img(src="spsimage.jpg"))),

  fluidRow(HTML(" Search Bar")),
  #fluidRow(HTML(" Date: 06-29-2020") ),

  fluidRow(
    br(),
    p("Text predictior app ")),
  br(),
  br(),

  fluidRow(HTML("Enter a word. Press \"Next words\" button to predict the following words") ),
  fluidRow( p("\n") ),

  # Sidebar layout
  sidebarLayout(

    sidebarPanel(
      textInput("inputString", "Enter a word here",value = " "),
      submitButton("Next words")
    ),

    mainPanel(
      h4("Predicted Next Word"),
      verbatimTextOutput("prediction"),
      textOutput('text1'),

    )
  )
)))

server <- function(input, output, session) {
  sentences <- reactive({
    make_sentences(input$inputString)
  })
  output$prediction <- renderText({ sentences() })
}

the output is coming as - Angry look for some time for Angry glance, and the duc d’enghien Angry look was a tone or Angry tears. “Lise!” Was that? The Angry tears. “Lise!” Said princess anna Angry look as much as she Angry look for three of the Angry tears. “Lise!” Said to the Angry look fixed on with an Angry glance, and fireworks are not
I would prefer -

1- Angry look for some time for
2- Angry glance, and the duc d’enghien
3- Angry look was a tone or Angry tears. “Lise!” Was that?
4-The Angry tears. “Lise!” Said princess anna
5-Angry look as much as she
'''''


texts: Finding out why your model is doing better?

texts:
I fitted a logistic regression model on a data set and got an AUC score of .70.
I added some additional out-hot encoded categorical features to the model and the AUC improved slightly to .74.
How do I assess how the model improved? What plots/other analyses are used to to assess the performance gain?
I understand that the model improved, but I want to be able to explain why adding those features improved the model.
This is just a general data science question.
Thanks!


texts: Residual Deviance in GLM Output in Python vs. R

texts:
I am coming from R to Python for econometrics. In R, in case of fitting a logistic regression with glm, the output summary would include the result of test of the proposed model against the saturated model in the 'Residual Deviance' section. In Python, Statsmodels' glm output, however, I cannot see this particular test. Can somebody point me toward the right direction that does not involve fitting two models for every model I test?
Note that I am specifically asking for the following:
letting LL be log likelihood, where do I find 2(LL(Saturated_model) - LL(Proposed_model)) and the df = df(Saturated_model) - df(Proposed_model)?


texts: What is the Intuition behind weight vector W which is normal to the plane? Is the weight vector W same as the W which is normal to the plane π?

texts:
In an interview, I was asked the intuition behind the weight vector. I told the weight vector is a vector which we try to minimize to a local minima with the help of regulariser so we don't overfit. Weights tells us the influence of a feature on the model.
Although I am not sure if my intuition is correct. Is Weight vector W always normal to the plane? Say we have 5 features and after training say logistic reg we get weight vector of 5 values. Does this mean i have 5 W's each with its own plane? Because W is always normal to the plane π.
First I was asked when we say that there is a plane π, and the normal to the plane π is W. What does it mean? Later I was asked what can we interpret from weight vector? Is Weight vector W same as W which is normal to the plane
It would really help if someone could help me clear this understanding visually and intuitively.


texts: Creating new columns based on 3 column and create new data frame

texts:
Heading

Consider my data frame
rs123   T   C   0   0   1   1   0   0   1   0   0   1   0   0
rs124   T   C   0   0   1   0   0   1   0   0   1   0   0   1
rs125   A   A   1   0   0   1   0   0   1   0   0   1   0   0

Similarity, i have total 93 columns excluding first three

I want to create my data as

And then transform into new data frame as below

For first row if 1 is present in column 1 then output should be TT
For first row if 1 is present in column 2 then output should be TC
For first row if 1 is present in column 3 then output should be CC

For more detail you can refer below snip

Kindly help me to find solution using python, Its very urgent

Thanks in Advance.


texts:
Most videos in benchmark datasets like UCF101 are short (<40 sec) and monotonous, as in they 'focus' on a person performing a specific action (jumping, running, etc)   . The whole video can be run through the ConvNet+LSTM with a logit output predicting the class of the video.

What if these conditions are very different? The video (or videos) are

long, e.g. 10-20 minutes,
'diverse', as in switch between different, seemingly unrelated scenes, e.g. full-screen subtitles,
there are only two classes, but the difference between them is subtle, as in sentiment type.

With regards to the first condition, there's a problem with the GPU use - if I extract, for example, 1000 frames from the video to represent it, I don't know if there's any way to put them all to CUDA. The way I'm thinking of solving this problem is by splitting the video into consecutive segments (e.g. 10) and building a batch from these segments, e.g. if I take 100 frames in each segment (data point), it will be size (10, 100, 3, H, W). But the label is more like (1,1) rather than (10,1), because there a label per whole video.

So it will be a sequence split into minibatches, that right?

I want to employ a ConvNet+LSTM for predicting the label of the video. The way I see it, LSTM can output a logit value per minibatch, in total 10 values, that will be inputs in a single-layer classifier with 1 output.

I haven't implemented it yet, but in general my question is this:

HOW CAN AN LSTM HANDLE A VERY LONG VIDEO WITH DIVERSE INFORMATION IN IT?

CAN I EMPLOY A SEQUENCE OF LSTMS?


texts: Problem understanding probabilistic generative models for classification

texts:
I am a student and I am studying machine learning. I am focusing on probabilistic generative models for classification and I am having some troubles understanding this topic.
In the slide of my professor it is written the following:

which I don't understand.
So far, I have understood that in the generative probailistic models, we ant to estimate $P(C_i|x)$, which is the probability of having class $i$ given a data $x$, using the likelihood and the Bayes theorem.
So, it starts by writing the Bayes rule, but the the slides says that we can write this as a sigmoid, but why?
If I have to try to give an answer to it, I would say because the sigmoid gives a number from $0$ to $1$, and so a probability, but it is just a guess I am doing.
Moreover, it continues by saying that we can use a gaussian distribution for $P(x|C_i)$, and so $P(x|C_i)=N(\mu ,\sigma )$, and so :

I don't understand what it is doing, can somebody please help me?
I don't know if my question is clear so sorry if it is not but I am really confused. If it is not lcear please tell me I will try to edit it. Thanks in advance.
Note: if it can be useful, this has been taken from the Bishop book at page 197


texts: ValueError: Failed to convert a NumPy array to a Tensor (Unsupported object type int) in Python

texts:
I have written the following code for a neural network to perform regression on a dataset, but I am getting a ValueError. I have looked up to different answers and they suggested to use df = df.values to get a numpy array. I tried it but it still produced the same error. How to fix this?
CODE
from keras import Sequential
from keras.layers import Dense, Dropout, Flatten
from keras.optimizers import Adam
from sklearn.model_selection import train_test_split

#Define Features and Label
features = ['posted_by', 'under_construction', 'rera', 'bhk_no.', 'bhk_or_rk',
            'square_ft', 'ready_to_move', 'resale', 'longitude',
            'latitude']

X=train[features].values
y=train['target(price_in_lacs)'].values

#Train Test Split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state = 23, shuffle = True)

#Model
model = Sequential()
model.add(Dense(10, activation='relu', kernel_initializer='random_normal', input_dim = 10))
model.add(Dense(1, activation = 'relu', kernel_initializer='random_normal'))

#Compiling the neural network
model.compile(optimizer = Adam(learning_rate=0.1) ,loss='mean_squared_logarithmic_error', metrics =['mse'])

#Fitting the data to the training dataset
model.fit(X_train,y_train, batch_size=256, epochs=100, verbose=0)

ERROR
ValueError: Failed to convert a NumPy array to a Tensor (Unsupported object type int).


texts: K-Nearest Neighbor Classifier Best K Value

texts:
I created a KNeighborsClassifier for my dataset adjusting the k hyper-parameter (the number of neighbors) in a for loop. The k value was between 1 and 20. The result was the graph below:

How do I interpret this graph? Which would be my best k value?


texts: Clustering Analysis with Categorical Variables of Limited Range

texts:
I've been recently been implementing some k-modes/k-medoids type procedures in R, such as k-prototypes, and the daisy package.
The output is fine. Yet, I wondered if anyone was aware of clustering packages specifically for categorical data where the range of data is small such as [1,2,3].
My suspicion with such limited range is that the mode/medoid might not discriminate as accurately as possible. However, this is just a speculative question.
Would love to hear some thoughts.


texts: Should the times series be made stationary before doing a clustering analysis?

texts: For times series analysis and forecasting, we try to make the times series stationary before proceeding with the experiment. I would like to know if such a procedure is necessary if one is working on an unsupervised learning task and needs to do times series clustering? Thanks.


texts: Extracting layer output from Classification model of SimpleTransformer

texts:
I have fine tuned a bert base model for text classification task. Now, I want to extract hidden layer output so as to combine this output with other features to train a random forest model. Problem lies in the fact that I don't know how to extract hidden layer output. It would be really great if someone can help me out in this regard.
from simpletransformers.classification import ClassificationModel

model_xlm = ClassificationModel('bert', 'bert-base-uncased')
model_xlm.train_model(df_train)


texts:
Is there any way to convert the detected object from an object detection algorithm (Mask RCNN in my case) to a white silhouette (while the background has been set to black)?
I have commented on the code below that I am working on (so it doesn't look complicated). If anyone could provide a direction or a solution, it will be highly appreciated.
# Load video and save codec

video = 'video.mp4'  # put the name of your video file HERE
save_fps = 30
weightsfile = "/mydrive/mask_rcnn/Mask_RCNN/mask_rcnn_coco.h5" # pre-trained weights
fourcc = cv2.VideoWriter_fourcc(*'MP4V')

###This applies a black background to any pixel that is NOT A PART OF THE DETECTED OBJECT###
def apply_mask(image, mask):
#black background creator
 get_range = cv2.inRange(image, (0,0,0), (255,255,255))
 bb_image= cv2.bitwise_not(get_range)
 #black background creator
 image[:, :, 0] = np.where(
     mask == 0,
     bb_image[:, :],
     image[:, :, 0]
 )
 image[:, :, 1] = np.where(
     mask == 0,
     bb_image[:, :],
     image[:, :, 1]
 )
 image[:, :, 2] = np.where(
     mask == 0,
     bb_image[:, :],
     image[:, :, 2]
 )
 return image

This function is used to show the object detection result in the original image.
def display_instances(image, boxes, masks, ids, names, scores):
    # max_area will save the largest object for all the detection results
max_area = 0

# n_instances saves the amount of all objects
n_instances = boxes.shape[0]

if not n_instances:
    print('NO INSTANCES TO DISPLAY')
else:
    assert boxes.shape[0] == masks.shape[-1] == ids.shape[0]

for i in range(n_instances):
    if not np.any(boxes[i]):
        continue

    # compute the square of each object
    y1, x1, y2, x2 = boxes[i]
    square = (y2 - y1) * (x2 - x1)

    # use label to select person object from all the 80 classes in COCO dataset
    label = names[ids[i]]
    if label == 'person':
        # save the largest object in the image as main character
        # other people will be regarded as background
        if square > max_area:
            max_area = square
            mask = masks[:, :, i]
            #w_image = cv2.inRange(mask, (1,1,1), (255,255,255))
            #mask = np.where(
            #    w_image[:,:]

        else:
            continue
    else:
        continue

    # apply mask for the image
    image = apply_mask(image, mask)

return image

This function runs the code given above
from google.colab.patches import cv2_imshow
#from visualize_cv import display_instances
import time

start = time.time()
videofile = video

cap = cv2.VideoCapture(videofile)
height = int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT))
width = int(cap.get(cv2.CAP_PROP_FRAME_WIDTH))

print("Video resolution : ", height, width)
assert cap.isOpened(), 'Cannot capture source'

#Add extension mask_rcnn to video name
video_mask_rcnn =  video.split(".")[0] + "_mask_rcnn_new_background." + video.split(".")[1]

#Video save
videoSaved = cv2.VideoWriter(video_mask_rcnn, apiPreference=cv2.CAP_FFMPEG,
                             fourcc = fourcc,fps =save_fps, frameSize=(width, height),
                             isColor= True)

frames = 0

while cap.isOpened():

    ret, frame = cap.read()
    if ret:

        #Inference
        #
        results = model.detect([frame], verbose=0)
        r = results[0]
        frame = display_instances(frame, r['rois'], r['masks'], r['class_ids'], class_names, r['scores'])

        # video save line
        #
        videoSaved.write(frame)

        frames += 1
        if frames%50 == 0: print("{} processed frames ok".format(frames))
    else:
        break
videoSaved.release()
end = time.time()
print("Video processed with mask rcnn and saved as " + video_mask_rcnn)
print('Total time elapsed: {:.3f} seconds'.format((end - start)))


texts: Timeseries VAR vs VARMA model: issue in time to fit model

texts:
I want to use VARMA model on a data of about 80000 samples with 10 features. I tried using VARMA model from statsmodels with p=50 and q=10 but it is taking too much time to build the model. I tested the model was running even after 12 hours. Then I tested VARMA using p=50 and q=0, this also was running even after an hour with maxiter=1. The code I am using is:

from statsmodels.tsa.statespace.varmax import VARMAX
modelVARMA = VARMAX(dff, order=(50,0))
resultsVARMA=modelVARMA.fit(maxiter=1)

I also tested VAR model with p=50 and it gave the model quickly.
Is there any way to run VARMA model fast as it is in case of VAR.


texts: How do you calculate the number coefficients in polynomial regression?

texts:
So I can't seem to find much on this by searching so I came here. Let's say I  had 3 variables $x_1,x_2,x_3$ and the let's say the degree of the polynomial was $d=2$, I can define the length of a vector of variables as $l$ such that $l=3$ in this case. What I'm looking for is the length of a vector of coefficients that would be needed to create a polynomial regression on these variables.
Would I be right in saying that the length of the vectors of coefficients needed for polynomial regression would be $d\times l$?


texts:
I'm doing some classification with a text corpus [professional emails], I've already done all the basic
pre processing steps (stemming, remove stop words, punctuation, low frequency words, words length ...)  which gives me a F1-score = 0.6.

I was wondering what could be the next step(s) to improve my results, I thought of "stacking" (use multiple classifiers) and maybe removing words which are common to the different classes (4 differents types of texts) [ not sure about this solution ]

Edit:

I tried with 15 differents classifier the best one is gradient boosting (gbm )with an F1-score=0.6002159.
The features have been extracted through a bag of words ( Document Term Matrix ) with Tf-Idf [I don't think taking into account the length of a text, punctuation or others is relevant] - [It is mainly professional emails]


texts: Getting a ValueError from train_test_split

texts:
I'm working on this dataset.  I'm trying to select features using Random Forest.  This is the relevant code:

x_train, y_train, x_test, y_test = train_test_split(train, test_size = 0.3)

sel = SelectFromModel(RandomForestClassifier(n_estimators=100))
sel.fit(x_train, y_train)
sel.get_support()
selected_feat = x_train.columns[sel.get_support()]

The code x_train, y_train, x_test, y_test = train_test_split(train, test_size = 0.3) throws the following ValueError:

Traceback (most recent call last):
  File "C:/Users/security/Downloads/AP/Boston-Kaggle/Boston.py", line 25, in <module>
    x_train, y_train, x_test, y_test = train_test_split(train, test_size = 0.3)
ValueError: not enough values to unpack (expected 4, got 2)

I don't know what it means by not enough values.  The dataset has a big dimension.


texts: Develop a “Credit Risk Scoring System” with R

texts:
I want to achieve during my internship a credit risk scoring system based on "credit german dataset".

I develop a model with R using logistic regression and i validate it. Now i must use this model to calculate a new applicant score and deduce whether good or bad.

I don't know how to proceed?


texts:
Using Linguistic Cues for the Automatic Recognition of Personality in Conversation and Text
www.aaai.org › JAIR-3012PDF
Web results
Using Linguistic Cues for the Automatic Recognition of Personality in ...
While going through the paper I found out the scoring have veen done via regression model. However I can t find the data containing the assessment scores. I can see the essays dataset containing 1/0 (classification) of text.
Please let me know if I am missing something here.
Thanks,


texts: i have trained a model using fer2013 dataset using CNN for Emotion detection. Now i want to use it in a image

texts:
I have a trained model and saved the weights in fer.h5. Now i want to use the pre trained model in another set of images and save it to a excel file
import sys, os
import pandas as pd
import numpy as np

from keras.models import Sequential
from keras.layers import Dense, Dropout, Activation, Flatten
from keras.layers import Conv2D, MaxPooling2D,
BatchNormalization,AveragePooling2D
from keras.losses import categorical_crossentropy
from keras.optimizers import Adam
from keras.regularizers import l2
from keras.utils import np_utils
# pd.set_option('display.max_rows', 500)
# pd.set_option('display.max_columns', 500)
# pd.set_option('display.width', 1000)

df=pd.read_csv('fer2013/fer2013.csv')

# print(df.info())
# print(df["Usage"].value_counts())

# print(df.head())
X_train,train_y,X_test,test_y=[],[],[],[]

for index, row in df.iterrows():
    val=row['pixels'].split(" ")
    try:
    if 'Training' in row['Usage']:
        X_train.append(np.array(val,'float32'))
        train_y.append(row['emotion'])
    elif 'PublicTest' in row['Usage']:
           X_test.append(np.array(val,'float32'))
           test_y.append(row['emotion'])
except:
    print(f"error occured at index :{index} and row:{row}")

num_features = 64
num_labels = 7
batch_size = 64
epochs = 40
width, height = 48, 48

X_train = np.array(X_train,'float32')
train_y = np.array(train_y,'float32')
X_test = np.array(X_test,'float32')
test_y = np.array(test_y,'float32')

train_y=np_utils.to_categorical(train_y, num_classes=num_labels)
test_y=np_utils.to_categorical(test_y, num_classes=num_labels)

#cannot produce
#normalizing data between oand 1
X_train -= np.mean(X_train, axis=0)
X_train /= np.std(X_train, axis=0)

X_test -= np.mean(X_test, axis=0)
X_test /= np.std(X_test, axis=0)

X_train = X_train.reshape(X_train.shape[0], 48, 48, 1)

X_test = X_test.reshape(X_test.shape[0], 48, 48, 1)

# print(f"shape:{X_train.shape}")
##designing the cnn
#1st convolution layer
model = Sequential()

model.add(Conv2D(64, kernel_size=(3, 3), activation='relu', input_shape=
(X_train.shape[1:])))
model.add(Conv2D(64,kernel_size= (3, 3), activation='relu'))
# model.add(BatchNormalization())
model.add(MaxPooling2D(pool_size=(2,2), strides=(2, 2)))
model.add(Dropout(0.5))

#2nd convolution layer
model.add(Conv2D(64, (3, 3), activation='relu'))
model.add(Conv2D(64, (3, 3), activation='relu'))
# model.add(BatchNormalization())
model.add(MaxPooling2D(pool_size=(2,2), strides=(2, 2)))
model.add(Dropout(0.5))

#3rd convolution layer
model.add(Conv2D(128, (3, 3), activation='relu'))
model.add(Conv2D(128, (3, 3), activation='relu'))
# model.add(BatchNormalization())
model.add(MaxPooling2D(pool_size=(2,2), strides=(2, 2)))

model.add(Flatten())

#fully connected neural networks
model.add(Dense(1024, activation='relu'))
model.add(Dropout(0.2))
model.add(Dense(1024, activation='relu'))
model.add(Dropout(0.2))

model.add(Dense(num_labels, activation='softmax'))

# model.summary()

#Compliling the model
model.compile(loss=categorical_crossentropy,
              optimizer=Adam(),
              metrics=['accuracy'])

    #Training the model
    model.fit(X_train, train_y,
          batch_size=batch_size,
          epochs=epochs,
          verbose=1,
          validation_data=(X_test, test_y),
          shuffle=True)

#Saving the  model to  use it later on
fer_json = model.to_json()
with open("fer.json", "w") as json_file:
    json_file.write(fer_json)
model.save_weights("fer.h5")


texts: Looking for a rough explanation of additive hidden nodes and radial basis functions

texts: I'm working on a neural networks project right now and for that I'm reading a bunch of scientific papers, in a few of those the terms additive hidden nodes and radial basis functions are thrown around, but I seem to have trouble to get a clear explanation of the terms there and anywhere else on the internet. I seem to have gathered that these are classifications for neuron types but I would love to get a more clear intuitive explanation of the terms. Preferably one that doesn't require me to be very mathy to understand. I'm fairly new to neural networks so beyond sigmoid neurons and backpropagation algorithms I'm still a bit lost when it comes to the common termonology.


texts: how to disable query from beeline results

texts:
I am occuring a strange hive-client beeline behavior.
In the outputed file with query results there is also a queary at the beggining and at the end. Is there any option to disable such behavior? I can't see such option in the beeline -help

-bash-4.2$ beeline -help
Usage: java org.apache.hive.cli.beeline.BeeLine
   -u <database url>               the JDBC URL to connect to
   -n <username>                   the username to connect as
   -p <password>                   the password to connect as
   -d <driver class>               the driver class to use
   -i <init file>                  script file for initialization
   -e <query>                      query that should be executed
   -f <exec file>                  script file that should be executed
   --hiveconf property=value       Use value for given property
   --hivevar name=value            hive variable name and value
                                   This is Hive specific settings in which variables
                                   can be set at session level and referenced in Hive
                                   commands or queries.
   --color=[true/false]            control whether color is used for display
   --showHeader=[true/false]       show column names in query results
   --headerInterval=ROWS;          the interval between which heades are displayed
   --fastConnect=[true/false]      skip building table/column list for tab-completion
   --autoCommit=[true/false]       enable/disable automatic transaction commit
   --verbose=[true/false]          show verbose error messages and debug info
   --showWarnings=[true/false]     display connection warnings
   --showNestedErrs=[true/false]   display nested errors
   --numberFormat=[pattern]        format numbers using DecimalFormat pattern
   --force=[true/false]            continue running script even after errors
   --maxWidth=MAXWIDTH             the maximum width of the terminal
   --maxColumnWidth=MAXCOLWIDTH    the maximum width to use when displaying columns
   --silent=[true/false]           be more silent
   --autosave=[true/false]         automatically save preferences
   --outputformat=[table/vertical/csv2/tsv2/dsv/csv/tsv]  format mode for result display
                                   Note that csv, and tsv are deprecated - use csv2, tsv2 instead
  --truncateTable=[true/false]    truncate table column when it exceeds length
   --delimiterForDSV=DELIMITER     specify the delimiter for delimiter-separated values output format (default: |)
   --isolation=LEVEL               set the transaction isolation level
   --nullemptystring=[true/false]  set to true to get historic behavior of printing null as empty string
   --help                          display this message


texts: Improve NER label results on Non-English text

texts:
I am working on some Medieval Latin text and was using various methods of NER such as CLTK (Latin Model), Spacy (Multilingual, Italian, Spanish Model) and StanfordNER (Spanish Model). When I used the non-Latin models I used the original Latin text as the translated one was not making any sense.

Fortunately Spacy Multilingual model managed to extract all Persons and Places of the sample documents but with additional words that I am not considering them as Entities. Moreover, the labels are incorrect.

Here is an example output:

{'LOC': ['Artali', 'Artalis', 'Bruges', 'Unde'],
'MISC': ['Marianum lu Tignusu'],
 'PER': ['Simone de Mazara',
  'Artalem de Alagona',
  'Apoca',
  'Coram',
  'Pero de Naso',
  'Pero Caruana',
  'Bartholomeo Xacara',
  'Testamur',
  'Artalis de Alagona',
  'Melite',
  'Simonis de Mazara',
  'Simonem',
  'Simone',
  'Mariano',
  'Artalis',
  'Artalem',
  'Simoni',
  'Panormi',
  'Renunciando']}

where the LOCATIONS should be: Panormi, Bruges, Melite and PERSONAL names should be all others except Unde, Apoca, Coram, Testamur, Renunciando which are neither locations nor personal names.

I was thinking of ignoring the labels and do some classification ML algorithm. The problem is that I do not have any training data available and the only possible usable corpus that I think it might be useful is Proiel treebank which labels proper nouns as NE. How would you go with such a problem?


texts: SVM/Naive Bayesian text classification on multiple features

texts:
I was building a text classifier which takes into account certain features of the text and classifies them into two - "Yes" or "No".

I have trimmed the text, removed stopwords and have applied TFIDF on it and now I'm performing SVM classification on it and getting around 81% accuracy on a dataset of size 25,000 with 80:20 training and test split.

I'd like to include some features in the text and have my classifier predict on that.

Example of how my data might look like:-

The code I was using for predicting the label just using text is:-

labeldf=meatydf['Label]

review_train, review_test, label_train, label_test = train_test_split(meatydf['text'],labeldf, test_size=0.2,random_state=13, shuffle=True)

X_train_tf = tf_vect.fit_transform(review_train)
X_test_tf = tf_vect.transform(review_test)

clf = svm.SVC(C=10,gamma=0.001,kernel='linear')
clf.fit(X_train_tf,label_train)
pred = clf.predict(X_test_tf)

I want to put all the columns of meatydf(text, feature1, feature2 and feature3) as features to help the classifier predict the labels and so I passed the entire dataframe(meatydf) to train_test_split:-

review_train, review_test, label_train, label_test = train_test_split(meatydf,labeldf, test_size=0.2,random_state=13, shuffle=True)

but I get the ValueError: Found input variables with inconsistent numbers of samples: [4, 21876]
which I understand is because of the difference in the shape of X_train_tf and label_train.

I'm not sure how I go about fixing this problem and the internet scouring hasn't helped me very much either, probably because I don't know what I should be looking for being very recently introduced to the topic.

Thanks a ton for your time.