texts: Multi Model data set Visualization Python

texts:
I have a data-set with two variable frequency values as given below

col[0],col[1],col[2],col[3]
val1, 23, 34, key1
val1, 23, 56, key2
val1, 23, 54, kay3
val2, 34, 76, kay1
val2, 34, 87, key4
val3, 90, 23, key2

Real Datasets

Reduction_Paris 957 228 #promos
Reduction_Paris 957 214 #promos
Reduction_Paris 957 181 #paris
Reduction_Paris 957 175 #promos
CNNsWorld   361 156 #breakingnews
CNNsWorld   361 156 #gamedev
CNNsWorld   361 156 #paris
CNNsWorld   361 156 #parisattack
FollowParisNews 705 131 #france
FollowParisNews 705 131 #news
FollowParisNews 705 131 #paris
Reduction_Paris 957 128 #promos
Reduction_Paris 957 128 #paris
FollowParisNews 705 128 #france
FollowParisNews 705 128 #news
FollowParisNews 705 128 #paris
FollowParisNews 705 125 #france
FollowParisNews 705 125 #news
FollowParisNews 705 125 #paris
Reduction_Paris 957 123 #paris
FollowParisNews 705 119 #france

The data set has more than 1000s of rows val is a string variable and key is also string. where as the col[1] and col[2] are the frequencies respectively of val and key. The val and key are connected to each other several values are connected to several keys.

I am trying to plot two kinds of graphs.

A graph to show pair inverse cumulative distribution function (ccdf). col[1] and col[2] are to be considered in this case. When it is only one value we can very well plot ccdf as individual but as a pair I have no idea how to get the distribution plots.
A connected graph to show the connection relationship between the val col[0] and key col[3]. I tried using Cytoscape and chose key as node and val as edge but I do not get the graph that I am looking for. Any solution on how I can show the relationship between the two ?

May be something like this kind of solution I am looking for, Cluster of Keywords col[3] and the col[0] connected to them.Either of the one type of graph.

I am comfortable with python so any suggestion in that area will be helpful. Thank you.


texts: Any tool that can help on manually label a time series data please?

texts:
i am working with a 10 years weekly financial time series data set, which has the standard format date open high low close volume.

i would like to manually label (classify) the data set with label/class uptrend,downtrend,buy,sell n etc in a new column.

my question is whether there is any tool to help on this data processing work.

For example, maybe the data can be loaded and plotted. i can then select a range (sequence)of data from the graph and give a label to it.


texts: Pattern Recognition, Bishop - (Linear) Discriminant Functions 4.1

texts:
Please refer "Pattern Recognition and Machine Learning" - Bishop, page 182.

I am struggling to visualize the intuition behind equations 4.6 & 4.7. I am presenting my understanding of section 4.1.1 using the diagram:

Pls. Note: I have used $x_{\perp}$ and $x_{p}$ interchangeably.

Equations 4.6, 4.7 from book: $$\mathbf{x} = \mathbf{x_{\perp}} + \textit{r}\mathbf{\frac{w}{\Vert{w}\Vert}} \tag {4.6}$$ Muiltiplying both sides of this result by $\mathbf{w^{T}}$ and adding $w_{0}$, and making use of $y(\mathbf{x}) = \mathbf{w^{T}x} + w_{0}$ and $y(\mathbf{x_{\perp}}) = \mathbf{w^{T}x_{\perp}} + w_{0} = 0$, we have $$r = \frac{y(\mathbf{x})}{\Vert{\mathbf{w}}\Vert} \tag{4.7}$$

  Questions:

  Is $y(\mathbf{x})$ the (orthogonal) projection of $(\mathbf{w^{T}x} + w_{0})$ along $\mathbf{w}$, the weight vector?
  Are the lengths normalized to express them as multiples of unit vector $\mathbf{\frac{{w}}{\Vert{w}\Vert}}$. If so, can the distance
  $r = $$\frac{y(\mathbf{x})}{\Vert\mathbf{w}\Vert}$ exceed 1.
  Given that, $$ y(\mathbf{x}) = \mathbf{w^{T}x} + w_{0}$$ i.e. $y(\cdot)$ has two parts: $\textit{orthogonal component above/ below}$
  decision boundary, and the $\textit{bias}$. And so, I'm calculating
  $y(\cdot)$ as:

  $$y(\mathbf{x}) = \mathbf{\frac{w^{T}x}{\Vert{w}\Vert}} + \frac{w_{0}}{\Vert\mathbf{w}\Vert}$$ while the book gets it as:
  $$y(\mathbf{x}) = {\frac{y(\mathbf{x})}{\Vert\mathbf{w}\Vert}} + \frac{w_{0}}{\Vert\mathbf{w}\Vert}$$
  I am struggling to visualize how do we get the first term in the equation above (in book, eqn. 4.7)

Alternatively, presenting my doubt/argument w.r.t. to book eqns 4.6 & 4.7; by substituting  $r$ (eq. 4.7) into eq. 4.6 we get:
$$\mathbf{x} = \mathbf{x_{p}} + y(\mathbf{x}) \qquad {(\Vert{\mathbf{w}}\Vert^{2} = \mathbf{w})}$$

which again seems to be incorrect - by triangle rule of vector addition.

  Given the context, where am I losing track? Request your inputs.


texts: Decision tree where identical set of features results in different outcomes

texts:
I am following the example described in this page to test my decision tree program.

The initial data set is

age            astigmatism, tp-rate, contact-lenses
---------------------------------------------------
young,          no,  normal,  soft
young,          yes, reduced, none
young,          yes, normal,  hard
pre-presbyopic, no,  reduced, none
pre-presbyopic, no,  normal,  soft
pre-presbyopic, yes, normal,  hard
pre-presbyopic, yes, normal,  none
pre-presbyopic, yes, normal,  none
presbyopic,     no,  reduced, none
presbyopic,     no,  normal,  none
presbyopic,     yes, reduced, none
presbyopic,     yes, normal,  hard

age, astigmatism, tp-rate are features and the type of contact lenses is the outcome.

When I trace the example (where features with lowest entropies are used as the decision nodes), we reach the following situation a little further down the page:

The blue circles are obviously mine to make tracing easier. As we see in the constructed tree up to this point, the path

[Astigmatism = yes] -> [TP-Rate = normal] -> [Age = PP (pre-presbyopic)]

will lead to the outcome of [Contact-lenses = None].

But when you look at the data, we see that this path can lead to either a hard or a none type of contact lens.

So has the example made the right decision?

And if not, what should we do in situations where the split data-set ends up, under an identical set of features, with more than one outcome?


texts: Variational Autoencoder Latent Space size

texts:
Variational Autoencoder: Imagine we use a batch size of e.g. 32. Furthermore we got 2 Linear Layers (mu, sigma) which are 300 long. The output dimension of the encoder (conv2d layer) is (32, 64 , 64, 64) and is then connected to the linear layers.
so there are 262.144 (=64*64*64) neurons connected to 300 which gives a total of ~78 000 000 parameters.

So my question is if there is any estimation possible to choose a proper latent space size ?
Is it in my case even realistic to go from 262.144 neurons to 300 and get a proper reconstruction?
The reason I ask is, that I really struggle on picking a right latent space plus I sometimes don't even see big differences between 300 or e.g. 500....

Thanks!


texts: One scaler for all features or one scaler per feature?

texts: I have a time series with more than 30 features. For preprocessing with scikit learn do you usually use one scaler per feature or one scaler for all features that should be standardized/normalized?


texts: Confused about false positive and false negative in confusion matrix?

texts:
I am working on binary classification for classifying cancer=1 and no-cancer=0, I use confusion matrix from sklearn, this is my confusion matrix on test set:

 # confusion matrix
 [[18  0]
 [ 7 15]]

 # in my reading the order is:
   TN=18
   FP=0
   FN=7
   TP=15

but in some tutorials, I see different ordering for FP and FN, some said same as my reading,See here, but other said reverse of FP and FN, see here.
my question is which one is true in my case? please give me a reference to be sure about the answer.


texts: Remove Outliers from Dataframe using pandas in Python

texts:
I would like to remove outliers from my dataset. It looks like this:

                  time       Gbps
0  2018-11-20 00:00:00  29.821748
1  2018-11-20 01:00:00  38.620987
2  2018-11-20 02:00:00   0.010000
3  2018-11-20 00:00:00  29.821748
4  2018-11-20 01:00:00  38.620987
5  2018-11-20 02:00:00   0.010000

As you take a look at this table, you can see that number 5 and 2 are the outliers. I wrote a interquartile range (IQR) method to remove them. However, it does not work. I don't know if I do something wrong in Pandas/Python, or it's the fact I do something wrong in statistics. Any ideas? The result from this function is the same frame as I had at the beginning.

def IQR(data):
    q1 = data['Gbps'].quantile(0.25)
    q3 = data['Gbps'].quantile(0.75)
    iqr = q3 - q1
    fence_low = q1 - 1.5 * iqr
    fence_high = q3 + 1.5 * iqr
    cleaned_data = data.loc[(data['Gbps'] > fence_low) & (data['Gbps'] < fence_high)]
    return cleaned_data

data = {
    'time': ['2018-11-20 00:00:00', '2018-11-20 01:00:00', '2018-11-20 02:00:00', '2018-11-20 00:00:00', '2018-11-20 01:00:00', '2018-11-20 02:00:00'],
  'Gbps': [29.8217476333333333, 38.6209872666666667, 0.01, 29.8217476333333333, 38.6209872666666667, 0.01]
}

df1 = pd.DataFrame(data, columns = ['time', 'Gbps']
cleaned1 = IQR(df1)
print(cleaned1)


texts: How to interpret training and testing accuracy which are almost the same?

texts:
Note - I have read this post but still don't understand
I have a Naive Bayes classifier, when I input my training data to test the accuracy, I get 63.05%. When I input my test data, the accuracy is 65.00%.
Why are the training and test accuracy almost identical? For information, my data is split in 70/30. Does this mean that there is no overfitting?


texts: Advantages of one shot learning over image classification

texts:
This is a rather conceptual question. From what I've read I gather that one shot learning is useful for use cases in which you don't have datasets of millions of images of employees etc. By a one shot operation, the network or networks will do a similarity measure to determine whether the given image belongs to a bunch of images or not.

However, some of these siamese networks are trained on millions or hundreds of millions of images. So doesn't that defeat the purpose of one shot learning ? Why can't plain image classification + transfer learning be used instead ? For instance, in face recognition systems, for every image - classify, doesn't fall into any of the classes - reject. If this is a legitimate new employee image, add the image to the database and retrain the network using transfer learning.

Does one shot learning remove the need for this retraining ? Can someone help me with these basic intuitions please


texts: LSTM fot text classification always returns the same results

texts:
Hello fellow Data Scientists,
I'm trying to make a classifier that was to classify sequences of text into some predefined classes, but i always get the same output, can anyone help me understand why?
The training of the model:

   # The maximum number of words to be used. (most frequent)
MAX_NB_WORDS = 100
#2155
# Max number of words in each complaint.
MAX_SEQUENCE_LENGTH = 100
# This is fixed.
EMBEDDING_DIM = 20

cf.go_offline()
cf.set_config_file(offline=False, world_readable=True)

def treina(model_name):

    df = pd.read_csv("divididos.csv",sep='§',header=0)
    df.info()

    max_len = 0
    for value in df.Perguntas:
        if(len(value)>max_len):
            max_len = len(value)
    max_words = 0
    for value in df.Perguntas:
        word_count = len(value.split(" "))
        if(word_count>max_words):
            max_words = word_count

    tokenizer = Tokenizer(num_words=MAX_NB_WORDS, filters='!"#$%&()*+,-./:;<=>?@[\]^_`{|}~', lower=True)
    tokenizer.fit_on_texts(df['Perguntas'].values)
    word_index = tokenizer.word_index
    X = tokenizer.texts_to_sequences(df['Perguntas'].values)
    X = pad_sequences(X, maxlen=MAX_SEQUENCE_LENGTH)
    Y = pd.get_dummies(df['Class']).values

    X_train, X_test, Y_train, Y_test = train_test_split(X,Y, test_size = 0.05, random_state = 42)
    print(X_train.shape,Y_train.shape)
    print(X_test.shape,Y_test.shape)

    #Balance data
    sm = SMOTE(random_state=12)
    X_train, Y_train = sm.fit_sample(X_train, Y_train)
    print(X_train.shape,Y_train.shape)

    #LSTM net
    model = Sequential()
    model.add(Embedding(MAX_NB_WORDS, EMBEDDING_DIM, input_length=X.shape[1]))
    model.add(LSTM(20, dropout=0.2, recurrent_dropout=0.2,activation="relu",return_sequences=True))
    model.add(LSTM(10, dropout=0.2, recurrent_dropout=0.2,activation="relu"))
    model.add(Dropout(0.2))
    model.add(Dense(11, activation='softmax'))
    opt = adam(lr=0.3)
    model.compile(loss='categorical_crossentropy', optimizer='rmsprop', metrics=['accuracy'])

    epochs = 100
    batch_size = 20

    history = model.fit(X_train, Y_train, epochs=epochs, batch_size=batch_size,validation_split=0.1)
    accr = model.evaluate(X_test,Y_test)
    print('Test set\n  Loss: {:0.3f}\n  Accuracy: {:0.3f}'.format(accr[0],accr[1]))
    model.save(model_name)
    return model

and the testing:

def corre(modelo):
    labels = ["a","b","c","d","e","f","g","h","i","j","k"]
    model = load_model(modelo)
    a = 0
    tokenizer = Tokenizer(num_words=MAX_NB_WORDS, filters='!"#$%&()*+,-./:;<=>?@[\]^_`{|}~', lower=True)
    while (a==0):
        new_complaint = input()
        new_complaint = [new_complaint]
        seq = tokenizer.texts_to_sequences(new_complaint)
        padded = pad_sequences(seq, maxlen=MAX_SEQUENCE_LENGTH)
        pred = model.predict(padded)
        print(pred, labels[np.argmax(pred)])

Thank you for your time


texts: How to properly set up neural network training for stable accuracy and loss

texts:
I have a DenseNet121 implemented in Pytorch for image classification. For now, the training set-up is pretty straightforward:

the data is loaded. An important characteristic here is that the validation data is fixed from the outset and can never change. The rest of the data I split into training and testing. None of the data sets overlap!
for every epoch iterate through the training data loader, calculating loss, optimizing etc.
every 100 batches evaluate the loss using the validation data loader
at the end of an epoch compare the current state’s loss on validation data with the state that previously had the best validation loss (for the first epoch just compare this with a random high number like 1e5) and save the current state if it better or keep the older state.
after all epochs are finished save the state with the lowest validation error as the best model
the best model is then applied to test data. Accuracies are calculated and ROC curves drawn

I was wondering how to extend my set-up to make it a proper statistical experiment set-up, with the goal of getting stable accuracy and loss results ie I want to be able to say that my model gives consistently the same results. I was thinking along the lines of running the testing step say 10 times and averaging the error? Or do you see some deficiency in the training that I could improve to improve stability?
Thanks in advance


texts: Bayesian network for classification using PyMc or PyMc3

texts:
I am seraching for a  while  an example  on how to use  PyMc/PyMc3 to do  classification task, but have not  found  an concludent   example  regarding  on how to do   the  predicton on a  new data point.
I am currious if some   could   give  me some references.


texts: how to log transformation works, In what case we need to do log transformation


texts: How do I predict a set of frequently bought items?

texts:
I have a dataset of retail transactions wherein different users buy certain items together. For example, a user A buys a toothpaste, a toothbrush and a floss at the same time, and a user B buys a toothpaste, a shampoo and a soap together.

I'm required to find out for each item in the dataset a list of items that are frequently bought together with it. For instance, one may expect that whenever a toothbrush is purchased, a toothpaste can be one of the items that are frequently bought together with it.

I tried to make use of the Apriori Algorithm but that would only give me sets of items that are bought together or, I'm unaware if it can somehow be modified to predict frequent purchases for a list of items in the dataset.

Which other algorithm can I put to use here?


texts: XGBoost in final model what does "yes=3,no=4" mean?

texts:
XGBoost v0.6 Binary Classification

$ cat predin.txt
0 1:10 2:10 3:10
1 1:10 2:10 3:100
1 1:10 2:100 3:10
1 1:100 2:10 3:10

In other words, the negative instance label 0 requires low values for all three features.

./xgboost/xgboost m.conf
./xgboost/xgboost m.conf task=dump model_in=0002.model name_dump=dump.raw.txt
$ cat dump.raw.txt
booster[0]:
0:[f1<55] yes=1,no=2,missing=1
    1:[f2<55] yes=3,no=4,missing=3
        3:[f3<55] yes=5,no=6,missing=5
            5:leaf=-1.5
            6:leaf=1.5
        4:leaf=1.5
    2:leaf=1.5
booster[1]:
0:[f1<55] yes=1,no=2,missing=1
    1:[f2<55] yes=3,no=4,missing=3
        3:[f3<55] yes=5,no=6,missing=5
            5:leaf=-0.784694
            6:leaf=0.784694
        4:leaf=0.784694
    2:leaf=0.784694

What does "yes=3,no=4" mean?


texts: How do local minima occur in the equation of loss function?

texts:
In gradient descent, I know that local minima occur when the derivative of a function is zero, but when the loss function is used, the derivative is equal to zero only when the output and the predicted output are the same (according to the equation below).
So, when the predicted output equals the output, that means the global minima is reached! So, my question is: How can a local minima occur, if zero gradient occurs only for the "perfect" fit?
$$\theta_j := \theta_j - {\alpha \over m} \sum_{i=1}^M (\hat y^i-y^i)x_j^i$$


texts: Rephrase Neural Network: Where to Start?

texts:
I'd like to solve the problem that denoted in the below picture.

With which network could I try and start to solve this problem?


texts: Identify outliers for annotation in text data

texts:
I read the book "Human-in-the-Loop Machine Learning" by Robert (Munro) Monarch about Active Learning. I don't understand the following approach to get a diverse set of items for humans to label:

Take each item in the unlabeled data and count the average number of word matches it has with items already in the training data
Rank the items by their average match
Sample the item with the lowest average number of matches
Add that item to the ‘labeled’ data and repeat 1-3 until we have sampled enough for one iteration of human review

It's not clear how to calculate the average number of word matches.


texts: Formulate Confusion Matrix from Precision Score

texts:
I have the precision scores from 5 classes. Is it possible to create the confusion matrix from that scores. While running the code, I only saved those precision values by mistake. So now I want to calculate the confusion matrix from those precision values. Is it possible ? For testing 1400 samples were used.
The precision score of 5 classes:
A > 86.78
B > 85.92
C > 83.21
D > 83.5
E > 82.73


texts: Sequence forecasting in keras not possible for variable-length sequence forecasts

texts:
I'm using keras for multiple-step ahead time series forecasting of a univariate time series of type float. Judging by the results I got, the approach works
works perfectly well. There is, however, an important detail in the training process that baffles me:

keras requires the sequence length of the input sequences (X matrix) to be equal to the forecasting horizon (y matrix). That means, for example,
that keras needs input sequences of length 20 in order to forecast the next 20 time steps. My goal is to be able to forecast as many time steps as I specify, given the last 20 time steps. With the below code, this is not possible:

    import numpy as np
    from keras.layers.core import Dense, Activation, Dropout, TimeDistributedDense
    from keras.layers.recurrent import LSTM
    from keras.models import Sequential
    from keras.optimizers import RMSprop
    np.random.seed(1234)

    model = Sequential()
    layers = [1, 20, 40, 1]

    model.add(LSTM(
        input_dim=layers[0],
        output_dim=layers[1],
        return_sequences=True))
    model.add(Dropout(0.3))

    model.add(LSTM(
        layers[2],
        return_sequences=True))
    model.add(Dropout(0.3))

    model.add(TimeDistributedDense(
        output_dim=layers[3]))
    model.add(Activation("linear"))

    rms = RMSprop(lr=0.001)
    model.compile(loss="mse", optimizer=rms)
    model.fit(
        X_train, y_train,
        batch_size=512, nb_epoch=100, validation_split=0.1)

I get a dimension error whenever the sequence lengths of the sequences in X_train and in y_train differ from one another.

What am I doing wrong and how can I fix it? Why are my results still pretty good?

The question is related to this.


texts: Cross validation Vs. Train Validate Test cont'd

texts:
Cross validation Vs. Train Validate Test

I have one further question relating to @louic answer in the post above:

"Training happens k times, each time leaving out a different part of the training set. Typically, the error of these k-models is averaged. This is done for each of the model parameters to be tested, and the model with the lowest error is chosen."

When we say "the model with the lowest error is chosen" which of the following does this mean:

we choose the parameters which has the lowest averaged error and then find the the model with the absolute lowest error from within the K models
we choose the parameters which has the lowest averaged error and then train a new model - on the entire training set, using these parameters?
we choose the parameters which has the lowest averaged error and then take the average weights of the K-models trained
Thanks

Iain


texts: Predicting topics for customer reviews based on topics mapped to n-grams?

texts:
I have a large number of unlabelled customer review data(text column) and my objective is to classify each review to a particular topic.

Also I have a list of unigrams,bigrams and trigrams(not a part of the customer review data) where each n-gram(word column) is tagged with a particular topic(topic column).

Is there a way to classify the customer reviews to the topics associated with the n-grams?
I am mostly thinking in terms of Word2Vec and Doc2Vec approaches.


texts:
I have been reading about Snowball Stemmer.
I wonder how does it work?
Does it use rules to stem word
or does it use machine learning to do that?
I checked snowballstem.org but could not find the answer!


texts: How do I handle division by 0 in Kaggle?

texts: I have written a RNN algorithm in Python/IDLE and ran it. It works and is successful. My dataset is large so I wanted to run it using Kaggle's computers, rather than my laptop. However, when I copied and pasted the code, the Kaggle console reported that there had been a division by 0. I'm guessing that IDLE has a built in method of handling division by 0's whereas Kaggle doesn't, hence why it was never reported in the IDLE console, and had been reported in the Kaggle console. How can I overcome this?


texts: Caps_Net. searching for example and library to use

texts: Which library is most recommended and easy to use?


texts: Multi-class classification with only one feature

texts: I am studying the efficacy of using a single feature for predicting a set of events (which is a multi-class classification problem). I was wondering if it makes any sense to use only one feature for multi-class classification. Thanks.


texts: How to limit a function's cosine value as it approaches 0

texts:
How do I get a cosine to peak up to a certain value as it hits 0? With the above code, the y-value peaks to infinity as it approaches 0.
import matplotlib.pyplot as plt
import numpy as np

amp = 1
time = np.linspace(-5, 5, 1000)

signal = np.cos(1 * np.pi * time) * abs(1/time)

plt.figure(figsize=(10, 10))
plt.ylim(-4, 4)
plt.plot(time, signal, color="black")
plt.savefig('output.png', bbox_inches='tight', pad_inches=0,
            transparent=True)

Put another way, based on my google-fu, how can I create a symmetrical exponentially decaying sinusoid with a maximum y-value as it approaches 0?


texts: LightGBM vs Sklearn LightGBM- Mistake in Implementation- Exact same parameters giving different results

texts:
While passing the exact same parameters to LightGBM and sklearn's implementation of LightGBM, I am getting different results. Initially, I was getting the exact same results on doing this, however, I made some changes to my code and now I can't find out why they're not coming the same. This means that the performance metrics and feature importance are coming differently. Please help me figure it out, I can't figure out the mistake I am making. It could either be a mistake in the way I am implementing LightGBM using the original library or in sklearn's implementation. Link for explanation on why we should get identical results.
x_train, x_test, y_train, y_test = train_test_split(df_dummy[df_merge.columns], labels, test_size=0.25,random_state=42)

n_folds = 5

lgb_train = lgb.Dataset(x_train, y_train)

def objective(params, n_folds = n_folds):
    """Objective function for Gradient Boosting Machine Hyperparameter Tuning"""

    print(params)

    params['max_depth'] = int(params['max_depth'])
    params['num_leaves'] = int(params['num_leaves'])

    params['min_child_samples'] = int(params['min_child_samples'])
    params['subsample_freq'] = int(params['subsample_freq'])

    # Perform n_fold cross validation with hyperparameters

    # Use early stopping and evalute based on ROC AUC
    cv_results = lgb.cv(params, lgb_train, nfold=n_folds, num_boost_round=10000,
                        early_stopping_rounds=100, metrics='auc')

    # Extract the best score
    best_score = max(cv_results['auc-mean'])

    # Loss must be minimized
    loss = 1 - best_score
    num_iteration = int(np.argmax(cv_results['auc-mean']) + 1)

    of_connection = open(out_file, 'a')
    writer = csv.writer(of_connection)
    writer.writerow([loss, params, num_iteration])

    # Dictionary with information for evaluation
    return {'loss': loss, 'params': params, 'status': STATUS_OK, 'estimators': num_iteration}

space = {
    'min_child_samples': hp.quniform('min_child_samples', 5, 100, 5),
    'reg_alpha': hp.uniform('reg_alpha', 0.0, 1.0),
    'reg_lambda': hp.uniform('reg_lambda', 0.0, 1.0),
    'colsample_bytree': hp.uniform('colsample_bytree', 0.5, 1.0),
    'max_depth' : hp.quniform('max_depth', 3, 10, 1),
    'subsample' : hp.quniform('subsample', 0.6, 1, 0.05),
    'num_leaves': hp.quniform('num_leaves', 20, 150, 1),
    'subsample_freq': hp.quniform('subsample_freq',0,10,1),
    'min_gain_to_split': hp.quniform('min_gain_to_split', 0.01, 0.1, 0.01),

    'learning_rate' : 0.05,
    'objective' : 'binary',

}

out_file = 'results/gbm_trials.csv'
of_connection = open(out_file, 'w')
writer = csv.writer(of_connection)

writer.writerow(['loss', 'params', 'estimators'])
of_connection.close()

trials = Trials()
best = fmin(objective, space, algo=tpe.suggest, trials=trials, max_evals=10)
bayes_trials_results = sorted(trials.results, key = lambda x: x['loss'])

results = pd.read_csv('results/gbm_trials.csv')

# Sort with best scores on top and reset index for slicing
results.sort_values('loss', ascending = True, inplace = True)
results.reset_index(inplace = True, drop = True)
results.head()
best_bayes_estimators = int(results.loc[0, 'estimators'])

best['max_depth'] = int(best['max_depth'])
best['num_leaves'] = int(best['num_leaves'])

best['min_child_samples'] = int(best['min_child_samples'])

num_boost_round=int(best_bayes_estimators * 1.1)
best['objective'] = 'binary'
best['boosting_type'] = 'gbdt'

best['subsample_freq'] = int(best['subsample_freq'])

#Actual LightGBM

best_gbm = lgb.train(params=best, train_set=lgb_train, num_boost_round=num_boost_round)

#Sklearn's Implementation of LightGBM

best_sk = dict(best)
del best_sk['min_gain_to_split']
sk_best_gbm = lgb.LGBMClassifier(**best_sk, n_estimators=num_boost_round, learning_rate=0.05, min_split_gain=best['min_gain_to_split'])
sk_best_gbm.fit(x_train, y_train)

sk_best_gbm.get_params()


texts:
I am using gensim library to find most similar words to some words that I have. Using 10000 data samples (short text mainly 1-2 sentences) to train, I get really bad results! Why this is so? Also by repeating train and test it get different results. Should not it be deterministic?

model = gensim.models.Word2Vec(senforword2vec)
print(model.most_similar(positive=['bad'], topn=1))

  [('amount', 0.9987443089485168)]

In another train and

  [('yet', 0.9984885454177856)]


texts: Learning character sequences and predicting sequences

texts:
I'm novice to deep learning and sorry if these questions may look very basic.

******************* First Question *******************

I need to predict the next k-characters given some initial text input, where the algorithm is learnt on a piece of text (character sequence).
I've done some work on predicting only 1-character at a time (using LSTM and Keras wrapper), but my question is how to generate k-characters and not only one. Of course I can generate k-characters recursively using only 1-character prediction algorithm, but I'm wondering if this problem (input text to k-characters) is well known and there exist a state-of-the-art solution.

What I learnt from reading relevant topics, is that this problem is (similar) to "sequence-to-sequence" learning and of course there are many solutions and implementations available (theano, tensorflow, etc).
Am I correct?

For example, given a string of letters:

seq_1 = 'aaaaabababcbcbbababbbdcbbbaaabbbaad...'

the algorithm should predict (given a randomly picked input sequence):

aaabbb -> aad

(the last 9 characters from the aforementioned example  )

******************* Second Question *******************

I know, that neural network algorithms are very "data hungry" and in order to make meaningful learning and prediction accuracy, one has to provide a lot of data (compared to shallow learning methods)
If I have only relatively short strings of letters (50-100 characters), does it make any sense to train seq2seq algorithm (LSTMs network) on such short sequences, but with many bootstrapped random instances?

Are there any theoretical (or some rules of thumb) bounds on the size of the training set and the reliability of the trained algorithm? (overfitting)

I'm quite motivated by the example of learning temporal patterns in alphabet with LSTM RNNs:

The author got very nice results, by random sampling of sequences of varied length and showing them to LSTM RNN. The idea was to randomly generate a relatively large (~1000 instances) data set and then train the network.

Can anyone comment on this way (bootstrapping) of creating a training set from a small sequence? Is it a standard approach? Are there any statistical flaws?
Many thanks!!


texts: Unsupervised Learning - Using the Outcome of Learning

texts:
My understanding in Unsupervised Learning is that -- when you want a computer to learn on its own by examining a large dataset. The goal is to establish some form of cluster or association-based grouping of data from the given input aka "Find patterns - known/unknown"
So my question is - "Now what do we do with it?" - e.g. I have now identified a pattern using Unsupervised Learning. Do I now look for Supervised Learning to empower my Data Insights work? Or, do I make decision as a human/team/committee to do certain things based on those?


texts:
So i'm very new to this, forgive my silly questions.

I've got some data I need to analyse, I would usually just use excel it doesnt seem to be able to do the job.

I have .csv files containing data for every seconds over a year period.
The data consists simlpy of  timestamp and valueX.

I need to analyse these values to see if values X goes below a certain value, and if it does I have various calculations to carry out. Value A is proportional to X and just instantaneous, then value B is a counter proportional to A.

Could you suggest a method/ language / software that would be the best and most accessible to do this? I don't have much experience in data analytics/big data apart from using excel and a bit of matlab and python.

thanks in advance

EDIT:
Thanks for the replies, my data is like this...

I have a text file for January values.

01/02/2016 00:00:00,49.972332
01/02/2016 00:00:01,49.9690056
01/02/2016 00:00:02,49.9600029
01/02/2016 00:00:03,49.9490013
01/02/2016 00:00:04,49.9430046

Different text files for each month, some files have text or notes at the top which I'd need to clean up.

For some context the second value is grid frequency, If it goes below a certain threshold, it constitutes an grid fault. I need to know how long these faults occur for, how often, how far the f value falls etc.

EDIT 2:
Thanks for the tips folks, I've been able to read in my data, clean it up and combine it into one dataframe.
I'm now try to add some extra column to the dataframe that are dependent of the col 1 value but having some trouble.

    "Read in month csv files and create dataframes"
df_jan_short = pd.read_csv('2016_01_short.csv', header = None ,parse_dates = [0], index_col = 0, names = ['timestamp','Freq'],squeeze= True)
df_feb_short = pd.read_csv('2016_02_short.csv', header = None ,parse_dates = [0], index_col = 0, names = ['timestamp','Freq'],squeeze= True)

    "Combine dataframes into one dataframe for year" months_short = [df_jan_short, df_feb_short] year_short = pd.concat(months_short)

    "Repalce NaN values in df with 50.00 Hz"
year_short.fillna(50.00,inplace = True)

    "Check what range freq is in"

year_short['Case'] = 'No Case'
year_short['Case'](year_short['Freq'] > Fc & year_short['Freq'] < Ff ) = 'A' year_short['Case'](year_short['Freq'] > Fa & year_short['Freq'] < Fc ) = 'B'
year_short['Case'](year_short['Freq'] < Fa ) = 'C'
year_short['Case'](year_short['Freq'] > Ff & year_short['Freq'] < Fh ) = 'G'
year_short['Case'](year_short['Freq'] > Fh ) = 'H'

But I get the following error...

runfile('C:/Users/ShaneOKeeffe/Documents/Grid Freq/Py/4.py', wdir='C:/Users/ShaneOKeeffe/Documents/Grid Freq/Py')
Traceback (most recent call last):

  File "C:\ProgramData\Anaconda3\lib\site-packages\IPython\core\interactiveshell.py", line 2862, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)

  File "<ipython-input-74-3a0fbc1f23f6>", line 1, in <module>
    runfile('C:/Users/ShaneOKeeffe/Documents/Grid Freq/Py/4.py', wdir='C:/Users/ShaneOKeeffe/Documents/Grid Freq/Py')

  File "C:\ProgramData\Anaconda3\lib\site-packages\spyder\utils\site\sitecustomize.py", line 710, in runfile
    execfile(filename, namespace)

  File "C:\ProgramData\Anaconda3\lib\site-packages\spyder\utils\site\sitecustomize.py", line 101, in execfile
    exec(compile(f.read(), filename, 'exec'), namespace)

  File "C:/Users/ShaneOKeeffe/Documents/Grid Freq/Py/4.py", line 48
    year_short['Case'](year_short['Freq'] > Fc & year_short['Freq'] < Ff ) = 'A'
                                                                                ^
SyntaxError: can't assign to function call


texts: How to classify whether text answer is relevant to an initial text question

texts:
I have a text classification problem in which i need to classify an answer to a message as either relevant or not.

In the first phase of my calculations, I have already used a SVM to determine if the original message was relevant or not, deciding whether a message contains a hint or question if somebody's twitter account has been hacked.

example:
"Hey @foobar, have you been hacked?"   <-- relevant
"My bank account has just been hacked" <-- not relevant

However, when I want to classify whether the answer is relevant, I would want to have both the original message and the answer as input, right? An answer is relevant in my case if it, in any way, responds to the original message. Is this approach possible using a SVM or any other machine learning tool? I'm using python with the scikit-learn library.

example:
"Hey @foobar, have you been hacked?"
"@barfoo it seems so, thx for suggesting" <-- relevant

"Hey @foobar, have you been hacked?"
"Lose 20 pounds quickly! http://blabla.com" <-- not relevant

I'm not very experienced in this field, so any input would be very appreciated.


texts: Cannot get the prediction right using Stochastic Gradient Descent: Always predicts 1

texts:
I have a CSV file with 20 columns and 785 rows. The 785th row for each column is a label describing the encoded image. The encoded image is either 3 or 5. So 1-784 row is the encoded image and 785th row is the label that names the image.

I loaded the CSV file 3_5_small.csv and segregated the labels from the encoded data.

which as you see is the image of number 3.

Now, I decided to use logistic regression to predict the images from the encoded data. I used Stochastic Gradient Descent as explained in the machine learning course by Andrew NG. But I do not think, I got it right. Before following the code, here are the steps I did:

Transformed train_labels_3_5 which contained only 3 and 5 to 1 and 0 respectively. So I want to predict the image of 3. If the output probability is < 0.5, it will be 5 and > 0.5 will be 3.
Randomly shuffled the train_data_3_5 and train_labels_3_5 to the same degree.
Randomly generated the theta vector
Passed the theta vector and X vector into the hypothesis function
Updated the theta vector.

This is all I did. Here is the code to what I have done.

train <- function(data, labels, alpha = 0.001) {
  #browser()

  #Initialize the theta vector
  theta <- seq(from = 0, to = 1, length.out = nrow(data))

  number_of_iterations = 10
  for(noi in 1:number_of_iterations) {
    for(i in seq(1:ncol(data))) {
      x = as.vector(data[,i]) #Create a x vector
      h = hypothesis(x, theta) #Call the hypothesis function to get the probability
      y = labels[1,i]
      theta <- theta - (alpha * ((h - y) * x))
    }
  }
  return(theta)
}

But on the test data and even on the training data, this does not predict correct at all. I do not know where have I gone wrong. I have revisited the algorithm, the lecture but cannot figure out, what am I doing incorrectly. It always predicts 1 no matter I pass the vector for 3 or 5!


texts: What are machine learning/deep learning models for generating contextually related words and synonyms?

texts:
I have a task to work on models for finding synonyms and contextually related words.
For example, if I enter:

'car' it should generate -> 'vehicle'
'sun' and 'sea' could generate 'beach', or some other related word to the first two.

So I used so far word2vec and nltk to generate examples.
But since I am not an expert on NLP I really find it difficult to use other algorithms or to build my neural network architecture.
I would appreciate it if someone can give me other suggestions and some explanations, that could be useful.


texts: How to train a model on top of a transformer to output a sequence?

texts:
I am using huggingface to build a model that is capable of identifying mistakes in a given sentence.
Say I have a given sentence and a corresponding label as follows ->
correct_sentence = "we used to play together."
correct_label = [1, 1, 1, 1, 1]

changed_sentence = "we use play to together."
changed_label = [1, 2, 2, 2, 1]

These labels are further padded with 0s to an equal length of 512. The sentences are also tokenized and are padded up(or down) to this length.
The model is as follows:
class Camembert(torch.nn.Module):
    """
    The definition of the custom model, last 15 layers of Camembert will be retrained
    and then a fcn to 512 (the size of every label).
    """
    def __init__(self, cam_model):
        super(Camembert, self).__init__()
        self.l1 = cam_model
        total_layers = 199
        for i, param in enumerate(cam_model.parameters()):
            if total_layers - i > hparams["retrain_layers"]:
                param.requires_grad = False
            else:
                pass
        self.l2 = torch.nn.Dropout(hparams["dropout_rate"])
        self.l3 = torch.nn.Linear(768, 512)

    def forward(self, ids, mask):
        _, output = self.l1(ids, attention_mask=mask)
        output = self.l2(output)
        output = self.l3(output)
        return output

Say, batch_size=2, the output layer will therefore be (2, 512) which is same as the target_label.
To the best of my knowledge, this method is like saying there are 512 classes that are to be classified which is not what I want, the problem arises when I try to calculate loss using torch.nn.CrossEntropyLoss() which gives me the following error (truncated):
 File "D:\Anaconda\lib\site-packages\torch\nn\functional.py", line 1838, in nll_loss
    ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), igno
re_index)
RuntimeError: multi-target not supported at C:/w/1/s/tmp_conda_3.7_100118/conda/conda-bld/p
ytorch_1579082551706/work/aten/src\THCUNN/generic/ClassNLLCriterion.cu:15

How am I supposed to solve this issue, are there any tutorials for similar kinds of models?


texts: RandomForest - Reasons for memory usage / consumption?

texts:
Which factors influence the memory consumption?

Is it the number of trees (n_estimators) or rather the number of data records of the training data or something other?


texts: Why is DDPG a Policy Gradient Method even though it's actor does not output probability?


texts: Do larger numbers of hidden layers have a bigger effect on a classification model's accuarcy?

texts:
I trained different classification models using Keras with different numbers of hidden layers and the same number of neurons in each layer. What I found was the accuracy of the models decreased as the number of hidden layers increased, however, the decrease was more significant in larger numbers of hidden layers. The following graph shows the accuracy of different models where the number of hidden layers changed while the rest of the parameters stay the same (each model has 64 neurons in each hidden layer):

My question is why is the drop in accuracy between 8 hidden layers and 16 hidden layers much greater than the drop between 1 hidden layer and 8 hidden layers, even though the difference in the number of hidden layers is the same (8).


texts: SGDClassifier accuracy MUCH worse than LinearSVC

texts:
with LinearSVC, I get an accuracy off 0.89:

linearSVC = LinearSVC().fit(X_train, y_train)
y_pred = linearSVC.predict(X_test)
print('accuracy', np.sum(y_pred == y_test) / y_test.shape[0])

Because I need online training, I tried the same with SGDClassifier:

sgd = SGDClassifier().fit(X_train, y_train)
y_pred = sgd.predict(X_test)
print('accuracy', np.sum(y_pred == y_test) / y_test.shape[0])

but now I get 0.004 - what am I missing here? The shape of X_train is (21519, 8255). Of course I do not expect the exact same accuracy, but SGDClassifier basically gets nothing right...

EDIT

I just found the reason - although I do not understand why there is SUCH a difference: I had an amount feature which includes numbers (= currency amount). With it included, the SGDClassifier performs horrible (= 0.3%), when transforming it to positive/negative, I get around 90% accuracy.


texts: Functions to do MLE and MSE calculations in R

texts: Are there libraries available or does one have to write these functions from scratch.


texts:
So I have a CNN task at hand. Let's say I have a dataset containing pictures of a number of classes, their attributes and sub attributes and so on.

What is the state of art/How can we design a system which first predicts the class, then based on that class the attributes which are specific to that class e.g claws present in birds but not in humans and so on.

How is such a system designed with minimum resource utilization?

Note: My question is not about prediction of classes and attributes but it is the design of a system which will predict the class then go predict the specific attributes belonging to the class and so on.


texts: how to use machine learning or network analysis to prevent contagious disease

texts:
I would like to share a question about my research. I have data about networking of a contagious disease.

I have 60 pigs, and I have the contact time with others pigs of each individual in 28 study day. So the structure of data would be

Day         ID     Network_friend (s)     Network_time (s)
1            1      200                       20000
1            2      300                       3000
.
.
.

Those 60 pigs are in similar age and body condition so there is not too much variety among the sample.

I am wondering if there is a way to predict the individuals who will get sick in 2 days to prevent the spread of the disease. Should I use any network analysis for it?

I appreciate any replies!

Thank you!


texts: How to perform topic modelling on query search results

texts:
How can I model topics in the results returned by a search engine with higher weightage to documents ranked higher in the result set?

The use case that I am looking at involves extracting the most significant topics returned in the search results.

Eg. If the user searches for a query q1 which returns documents D1...Dn with scores S1...Sn (in descending order) then I propose the notion that the theme of such a set of documents is represented better by documents scored higher in the result set.

Is it possible to incorporate this information in to topic modelling algorithms like LDA?


texts: Why the validation loss is 0 even if accuracy not 100%?

texts: I am training a CNN to perform a binary classification and at some point I get validation loss 0, however the validation accuracy is 84.5%. Why does this happen? I use binary_crossentropy as loss function and sigmoid as activation in the last layer.


texts: If I had an automated system that pays my bills, but the website where I pay them won't tell me when the next bill will be available. What is a good approach of predicting the date of the next bill? I don't see a regression model here as a good approach. I believe it's better to get a simple statistical mean or median between bill dates in days, and then guess the next bill date as: nextBill = lastBillDate + mean. Is that the right approach?


texts: How to improve the neural art algorithm?

texts:
I have been playing around the algorithm with tensorflow in this paper.
I tried to convert a photo to a Chinese ink and wash painring, but I got some strange patterns in the output picture(those in the top left corner).

I totally followed the algorithm in the paper except that I add another loss term for total variation denoising.
I have some confusions about this:

In the paper it used the 16-layer vgg net. I am going to try some
other networks, such as googlenet or alexnet. But I was wondering
whether it would avoid these patterns if I use some other networks.

In the paper, it used the gram matrix to represent the style
loss. But I don't know the reason here. I was also wondering if
there's any other way to stand for the style loss.

Since I have used total variation denoising here, and the
result is not satisfactory. I was wondering if there're some other
ways of denoising to avoid these patterns.

(I'm sorry that I can't
upload my content input image because I don't have enough reputation
in stack overflow as a new user. If you're interested, you can
search the google image with"Shanghai Jiaotong university" and find
the original image there.)


texts: R Recommender System for very sparse matrices required

texts:
I am trying to build a recommender system based on a large and very sparse matrix. Dimensions of that matrix would approximately be 12000 x 37000, possibly even more rows up to 100000. However, this matrix is extremely sparse. With the 12000x37000 version, about 0.053% of the matrix is non-NA. I've tried SVD, but alas, to no avail. To ensure I have not caused any error during my proceedings:

I created a data.table with unique triplets of "User" - "Item" - "Rating". I should mention "Rating" can stretch anywhere from 0 to about 150.
Now, I applied dcast.data.table to the triplet table, corrected the problem with the first value becoming a column, converted to a matrix. Now, I had a matrix with users as rows, items as columns and the rating as cell content.

Split into Test and Validation set, replaced "NA" with 0, subtracted row means for every row, applied propack.svd from the "svd" package to that matrix, multiplied the three matrices delivered by propack and added the row means to it. (user-means).

After that, I compared the values from the validation set to the corresponding values in my prediction matrix.. and no surprise, the Root mean square error was horribly high, around 6-7.(Mean of non-NA values is around 4.5). I've tried multiple variants of normalisation too, but I just could not get the RMSE below 5.8, ever.

Is there any way to build a viable item recommender system for this dataset? Possibly via arules or clustering?


texts: What is a reasonable standard deviation in accuracy between different train-test splits?

texts: My model's performance depends on the train-test-split performed. I did 1000 train-test splits and had an average accuracy of 75.4 % and the accuracy had a standard deviation of 2.4 % over those 1000 splits. Is that a reasonble standard deviation to have or not?


texts: Maximize the margin formula in support vector machines algorithm

texts:
I was recently reading about support vector machines and how they work and I stumbled on an article and came across Maximize the distance margin.

Can anyone tell me what do we have to minimize here? I wasn't able to understand this part I pasted below. What is $w$ and $m$ here in the formulas given below?

Maximize the margin

For the sake of simplicity, we will skip the derivation of the formula for calculating the margin, m which is
$$m = \frac{2}{\|\vec{w}\|}$$

The only variable in this formula is w, which is indirectly proportional to m, hence to maximize the margin we will have to minimize ||w||. This leads to the following optimization problem:

$$\min_{(\vec{w}, b)} \frac{\|\vec{w}\|^2}{2}$$

subject to $$y_i(\vec{w}\cdot\vec{x}+b) \ge 1, \forall i = 1, \ldots, n$$

The above is the case when our data is linearly separable. There are many cases where the data can not be perfectly classified through linear separation. In such cases, Support Vector Machine looks for the hyperplane that maximizes the margin and minimizes the misclassifications.

Link to the article : Hackerearth


texts: Algorithm for generating rules for classifying documents

texts:
Im looking for an algorithm that can deduct a set of rules based on a dataset of "training documents" that can be applied to classify a new unseen document. The problem is that I need these rules to be viewable by the user in the form of some string representation. For example, the algorithm found that documents have a minimum word count of 1000 and that there are 4 citations in each document. The key is that these rules must be deducted by a algorithm. An example of this in practice would be:

Document 1 contains 890 words and only 2 citations

I need it to return something like:

- You should add more words to make it better
- Add more citations to prove your point


texts: How to build a predictive model on normalized data

texts:
I have a dataset of accounts that contain a number of users that share a subscription to a product (think Netflix family account or something like that). In order to predict whether an account will be cancelled or not I want to use the traffic patterns of each one of the users.

Is there an algorithm out there that can take the normalised data as is to generate predictions on whether the account will be cancelled?

I'm aware of the several techniques of feature generation that could be used to denormalize the data, but it would be extremely useful if a model (or models) could be built with the data as is.

Thanks!


texts: How to extract Question/s from document with NLTK?

texts:
How to extract Only Question/s from document with NLTK ?

Can we categorise this Question into Y/N and details type answerable ?

Note: I am one week old in NLTK ;-)


texts: Multi-Source Time Series Data Prediction

texts:
I was wondering if anyone has experience with time series prediction for data from multiple sources. So for instance, time series $a,b,..,z$ each have their own shape, some may be correlated with others. The ultimate goal is to have a model trained such that the value at time $t+1$ for any given data source can be predicted.

I personally have two solutions that in theory could work, but was wondering if anyone knew of other frequently used methods.

Multi-task learning with LSTM
Use feature engineering to model properties of each time series source as features along with usual features and use these with LSTM


texts:
I was recently reading the Knowledge Distillation paper, and encountered the term smooth probabilities. The term was used to denote when the logits were divided a temperature.

  Neural networks typically produce class probabilities by using a
  softmax output layer that converts the logit, zi, computed for
  each class into a probability, qi, by comparing zi with the other
  logits where T is a temperature that is normally set to 1. Using a
  higher value for T produces a softer probability distribution over
  classes.

What does that mean intuitively?


texts: Where does the "deep learning needs big data" rule come from

texts:
When reading about deep learning I often come across the rule that deep learning is only effective when you have large amounts of data at your disposal. These statements are generally accompanied by a figure such as this:

The example (taken from https://hackernoon.com/%EF%B8%8F-big-challenge-in-deep-learning-training-data-31a88b97b282 ) is attributed to a 'famous slide from Andrew Ng'. Does anyone know what this figure is actually based upon? Is there any research that backs up this claim?


texts:
I implemented a Keras model for my all-integer dataset with values greater than or equal to 0. The train data has dimensions of (393, 108) and prediction data has (1821, 108). Code is as follows.
import keras
from keras.models import Sequential
from keras.layers import Dense

X = data.iloc[:, :-1]
y = data.iloc[:, -1]

model = Sequential()
model.add(Dense(X.shape[1]-1, input_dim=X.shape[1], activation='tanh'))

for i in range(X.shape[1] - 2, 2, -100):
    model.add(Dense(i, activation='tanh'))

model.add(Dense(1, activation='tanh'))

opt = keras.optimizers.Adam(learning_rate=100)
model.compile(loss='categorical_crossentropy', optimizer=opt)
model.fit(X, y)
model.predict(X0)

I am getting all nan values as results.
array([[nan],
       [nan],
       [nan],
       ...,
       [nan],
       [nan],
       [nan]], dtype=float32)


texts: is there any python code already available to do this? - I would appreciate if you can elaborate on coding of this problem as I am new to NN.


texts: How do I use a model after it's fitted to predict the class of a single string?

texts:
After a model is built, how can I use it to predict the class of a single string?

model.predict() is returning something like [[0.41100174 0.5889983 ]] instead of it's predicted class (0 or 1).

Say I just built model like so:

hist = model.fit(data.x_train,
                 data.y_train,
                 validation_data=(data.x_test, data.y_test),
                 epochs=500,
                 batch_size=50,
                 shuffle=False,
                 verbose=2,
                 callbacks=[checkpoint, estopping, tensorboard])

I'm looking to predict a string's class using model.predict(), but it returns something like [[0.41100174 0.5889983 ]] instead of it's predicted class (0 or 1).

The shape of data.x_test (used for validation data) is the same shape as data.x_data (reformatted string to predict): (1, 250, 70) (except the number of rows, obviously)

Here's how I'm trying to use the model to predict the class of a string.

def predict_string(model,s):
    df = pd.DataFrame([s], columns=['text'])
    df = df.reset_index(drop=True)
    df['label'] = [0]

    df.label = pd.to_numeric(df.label, errors='coerce') # Convert to integer
    df = df.dropna()
    df = df[df.label.apply(lambda x: x !="")]
    df = df[df.text.apply(lambda x: x !="")]

    vocab_len = 70
    data = char_preproc(df.text, df.label, vocab_len, True, None)
    y_pred = model.predict(data.x_data)
    return y_pred

s = "Best movie ever" # Out: [[0.41100174 0.5889983 ]]

# s = "Worst movie ever" # Out: [[0.5436389  0.45636114]]

y_pred = predict_string(model, s)
print("Review: {}\"\nPredict: {}".format(s, y_pred))

I'm not sure it matters, but for testing, I'm classifying movie reviews as good (1) or bad (0) using a Character-level CNN trained on the Rotten Tomatoes Movie Review dataset, running on GPU via Google Colab.


texts: SVR is giving same prediction for all features

texts:
I'm creating a basic application to predict the 'Closing' value of a stock for day n+1, given features of stock n using Python and Scikit-learn
A sample row in my dataframe looks like this (2000 rows)
       Open     Close    High     Low      Volume
0      537.40   537.10   541.55   530.47   52877.98

Similar to this video, where he uses 'Dates' and 'Open Price'. In this example, Dates are the features and Open price is the target.
Now in my example, I don't have a 'Dates' value in my dataset, but instead want to use Open, High, Low, Volume data as the features because I thought that would make it more accurate
I was defining my features and targets as so
features = df.loc[:,df.columns != 'Closing']
targets = df.loc[:,df.columns  == 'Closing']

Which would return a  df looking like this
features:
       Open      High      Low      Vol from
29     670.02    685.11    661.09   92227.36

targets:
       Close
29     674.57

However I realised that the data needs to be in a numpy array, so I now get my features and targets like this
features = df.loc[:,df.columns != 'Closing'].values
targets = df.loc[:,df.columns  == 'Closing'].values

So now my features look like this
[6.70020000e+02 6.85110000e+02 6.61090000e+02 9.22273600e+04
  6.23944806e+07]
 [7.78102000e+03 8.10087000e+03 7.67541000e+03 6.86188500e+04
  5.41391322e+08]

and my targets look like this
[  674.57]
[ 8042.64]

I then split up my data using
X_training, X_testing, y_training, y_testing = train_test_split(features, targets, test_size=0.8)

I tried to follow the Scikit-Learn documentation, which resulted in the following
svr_rbf = svm.SVR(kernel='rbf', C=100.0, gamma=0.0004, epsilon= 0.01 )
svr_rbf.fit(X_training, y_training)
predictions = svr_rbf.predict(X_testing)
print(predictions)

I assumed that this would predict the Y values given the testing features, which I could then plot against the actual y_testing values to see how similar they are. However, the predictions is printing out the same value for each X_testing feature.
[3763.84681818 3763.84681818 3763.84681818 3763.84681818 3763.84681818

I've tried changing the value of epsilon, c and gamma but that doesnt seem to change the fact that the predictions always gives the same value
I know that it might not be accurate to predict stock prices, but I must have done something wrong to get the same value when applying the model to various different test data


texts: Structure of Convolutional Neural Network to analyze a sequence of frames

texts:
I want to apply a CNN to a series of image sequences to classify that sequences of frames/images in two groups/categories. We are in a binary classification problem. My dataset is composed by a lot of 'batches' of frames. For example, each batch of frames could be compose of 20 frames of 64x64 pixels. One important thing is that the order of that 20 frames is important. If you shuffle the order of that 20 frames the output could change.

For all this, I want to create a CNN to solve this binary classification problem. I'm usig Keras and TensorFlow.

What is my question? Well, I'm not sure if I have to use a TimeDistributed layer or not. The input shape of the neural network is the following one: (20, 64, 64, 1), whose meaning is: 20 frames with a 64x64 size (1 channel - grayscale). Should I use a TimeDistributed layer?


texts: how to calculate recency(RFM) from a given date attribute in R

texts:
custid  tot_prod       tot_rev          Last_tran   total_trans

1   1002          13       1465.25         11/23/2011       4
2   1003          2         353.25          1/1/2011        1
3   1004          12        2466.5          11/25/2011      12

Above is the Retail dataset of customer. I need to calculate recency(a new attribute when the customer is last seen) from last transaction(last_tran)  in terms of month.i.e. he purchased in last 3 months or last 6 months or may be one year in R. please suggest.


texts: What is the reasonable max number of features for LSTM?

texts:
I've been trying to find information on how many features I can use in LSTM.
I found that LSTM can handle up to 500-1000 timesteps, but what about features (sequences)?

Can I use 1000 features per timestep?

What is the working maximum and what affects that number? Thank you.


texts:
I am trying to interpret the CNN model from the below settings. AS I am new to deep learning and I am not able to fully comprehend the layer structure . Could someone please tell me is these two settings are similar because I read in one research paper they used first setting and wrote that is similar to the second setting

A CNN for MNIST with two 5x5 convolution layers (the first with 32 channels, the second with 64, each followed with 2x2 max pooling), a fully connected layer with 512 units and ReLu activation, and a final softmax output layer (1,663,370 total parameters).

The CNN for MNIST has 8 layers with the following structure: 3×3×32 Convolutional → 3×3×64 Convolutional → 2×2 MaxPool → Dropout → Flatten → 1 × 128 Full connected → Dropout → 128 × 10 Fully connected → Softmax.


texts: R langauge how to create xgb.DMatrix object from data frame (newbe)

texts: In R, how does one create an xgb.DMatrix object from an R data frame?


texts: Help to choose algorithm for computing difference between 2 texts?

texts:
I have a task to create a tool, which will be able to find articles-duplicates of a given reference article.
I know word vectorization (tf-idf,word2vec), RNN methods, but i can not choose something suitable for my situation.

My requirments:

data are being collected on the fly  (program parses articles from web sites, so i don't have regular DB with collection of texts)
there is a reference text, whose copies need to be found
copies could be copypasted, partially copypasted (by paragraphs) or paraphrased
reference-vs-copy comparison algorithm is preferable, but not required (instead of reference-vs-corps)
algorithm shouldn't do deep semantic analyzis, only kind of word counting, word vectorization, substring search
instead one algorithm, i can use a set of herurisitcs
algorithms can do false positive dicisions

I come up with such ideas:

download pretrained word2vec and compare means of word-vectors
Build a dictionary word->count from every text and compare it to reference dictionary
collect about 100 texts, vectorize them according to tf-idf and find closest to the reference

I will apreciate, if you will point specific algorithms, libs, examples based on key-word extractions, dummy substring search, line difference comparison for python or CLI.


texts: Technical question: how to build exe from python code?

texts:
I have a predictive model in IPython notebook. How can I create dll or exe from it? And is it possible to save in executable file already trained model?

I want a program with very simple syntax(X - file with objects, out -- file with answers):

./model X out

I don't know if is appropriate community for this question. If not, I'll delete it.


texts: Keras data generator for images x and continuous y

texts:
My use case is doing a regression on images, which is different than most examples.

Is it possible to use one of the canned data generators to yield images from a directory for x and a single integer for y? How would I do so?

I'd also like to shuffle the dataset and do an 80 : 20 validation split.


texts: why does ID3 Decision tree algorithm not give the best decision tree?

texts: I was going through ID3 algorithm, and what I believe is it incorporates Greedy Search rule to get come up with the decision tree. If it gives the best split possible at every stage, how does it not give the best decision tree possible?


texts: How to create and format an image dataset from scratch for machine learning?

texts:
I've only worked with ML with .csv formats. I've worked with image formats too but only premade imagesets (MNIST,etc). If I were to create an imageset from scratch, how are the class labels typically formated? Would I have to manually title the image of a jpeg?

Best,
Jeremy


texts: Advanced frequent pattern identification

texts:
I have some dictionary data like this:

Tag | Definition
Noun | a man who works in a restaurant, serving people food and drink
Noun | a person who does a specified type of work or who works in a specified way
Noun | a person who habitually seeks to harm or intimidate those whom they perceive as vulnerable.
Adj | relating to meaning in language or logic.
Adj | of or like snow, especially in being pure white
Adj | of or according to syntax.

My task is to find an appropriate 'Tag' based on a given 'Definition'.

There are obviously some patterns that we use when describing a noun and other patterns for describing an adjective. I would like some kind of algorithm or method for finding these patterns. I tried machine learning (e.g., Naive Bayes, SVM, Logistic Regression, MLP) but due to the fact that most of my samples are 'Noun', I could not reach a good accuracy. (5000 Nouns, 2000 Adjectives, 900 Verbs, 900 Other)

I am thinking of using another algorithm that does not require a lot of data or can handle imbalanced classes. I've seen FP-Growth and Apriori algorithms. But I am not sure if they are capable of handling advanced patterns like this:

A person who does X to Y.

By mentioning this pattern, I wanted to point out that there are some keywords in a pattern ('A', 'person', 'who', 'does', 'to') and some other words that should be ignored. Is there an algorithm for handling this situation?

Note: My objective is to find frequent ordered-sets.


texts: Why I would use TF-IDF after Bag-of-Words (CountVectorizer)?

texts:
In my recent studies over Machine Learning NLP tasks I found this very nice tutorial teaching how to build your first text classifier:
https://towardsdatascience.com/machine-learning-nlp-text-classification-using-scikit-learn-python-and-nltk-c52b92a7c73a
The point is that I always believed that you have to choose between using Bag-of-Words or WordEmbeddings or TF-IDF, but in this tutorial the author uses Bag-of-Words (CountVectorizer) and then uses TF-IDF over the features generated by Bag-of-Words.
text_clf = Pipeline([('vect', CountVectorizer()),
...                      ('tfidf', TfidfTransformer()),
...                      ('clf', MultinomialNB()),
... ])

Is that a valid technique? Why would I do it?


texts: Differentiate between positive and negative clusters

texts:
I have applied k-means clustering on my dataset of Amazon Alexa reviews.
model = KMeans(n_clusters=2, max_iter=1000, random_state=True, n_init=50).fit(X=word_vectors.vectors.astype('double'))

Now I want to check which cluster is positive and which is negative, can anyone suggest me some way to do that?
Also, is there any way to check is a particular word belongs to which cluster. E.g, the word 'bad' belongs to which cluster - 0 or 1


texts: Having trouble getting TensorFlow to do something trivial

texts:
I have a vector x and want to compute a vector y such that y[j] = x[j]**2 using a neural network specified by TensorFlow, below. It doesn't work so well, the error is high.
Am I doing something wrong?
Any help will be appreciated.

The way it works is it first generates data in Xtrain, Ytrain, Xtest, and Ytest and then creates placeholder variables to get TensorFlow going.
Then it specifies three hidden layers and one output layer. Then it trains, and Ypred, the prediction for Ytest, is created using a feed dictionary.

import numpy as np
import tensorflow as tf

n = 10
k = 1000
n_hidden = 10
learning_rate = .01
training_epochs = 100000

Xtrain = []
Ytrain = []
Xtest = []
Ytest = []

for i in range(0,k,1):
    X = np.random.randn(1,n)[0]
    Xtrain += [X]
    Ytrain += [Xtrain[-1]**2]
    X = np.random.randn(1,n)[0]
    Xtest += [X]
    Ytest += [Xtest[-1]**2]

x = tf.placeholder(tf.float64,shape = (k,n))
y = tf.placeholder(tf.float64,shape = (k,n))

W1 = tf.Variable(tf.random_normal((n,n_hidden),dtype = tf.float64))
b1 = tf.Variable(tf.random_normal((n_hidden,),dtype = tf.float64))
x_hidden1 = tf.nn.sigmoid(tf.matmul(x,W1) + b1)

W2 = tf.Variable(tf.random_normal((n,n_hidden),dtype = tf.float64))
b2 = tf.Variable(tf.random_normal((n_hidden,),dtype = tf.float64))
x_hidden2 = tf.nn.sigmoid(tf.matmul(x_hidden1,W2) + b2)

W3 = tf.Variable(tf.random_normal((n,n_hidden),dtype = tf.float64))
b3 = tf.Variable(tf.random_normal((n_hidden,),dtype = tf.float64))
x_hidden3 = tf.nn.sigmoid(tf.matmul(x_hidden1,W3) + b3)

W4 = tf.Variable(tf.random_normal((n,n_hidden),dtype = tf.float64))
b4 = tf.Variable(tf.random_normal((n_hidden,),dtype = tf.float64))
y_pred = tf.matmul(x_hidden3,W4) + b4

penalty = tf.reduce_sum(tf.abs((y - y_pred)))
train_op = tf.train.AdamOptimizer(learning_rate).minimize(penalty)

model = tf.global_variables_initializer()
with tf.Session() as sess:
    sess.run(model)
    for i in range(0,training_epochs):
        sess.run(train_op,{x: Xtrain,y: Ytrain})

    Ypred = y_pred.eval(feed_dict = {x: Xtest})


texts:
I have been trying to find the substitute of numpy and perform some linear algebra using C++. Here's a list of the libraries I have encountered:

Eigen
Armadillo
Dlib
GNU Scientific library

Please guide which is the best library of all and is most widely and easily used.
Also explain different uses of different libraries if possible.
If I wish to perform some linear regression using C++, which one will come handy?


texts: Generalization error problem on training set

texts:
Training data:
$\mathcal {T} =\{(2,1),(3,2),(4,6),(0,0),(1,1)\}$
you already computed a predictor for the output using linear regression by least squares, where you used the first 3 samples as training samples:
$f(X) = -4.5  +  2.5X$
Approximate the generalization error using the validation set approach, i.e. on the remaining validation set.
How I started:
$\text{Error = Irreducible Error + Bias$^2$ + Variance .}$
$\text{$EGE(f, x_0) $=$σ^2_ε$ + $[E_T (f_T (x_0)) − f_{exact}(x_0)]^2$ + $E_T(f_T (x_0) − E_T (f_T (x_0)))^2$
}$
How to compute the term $E_T (f_T (x_0))$ and $E_T(f_T (x_0) − E_T (f_T (x_0)))^2$


texts:
I need to design and implement my 3rd year project which is a real-time alert system for parents to keep their kids safe with their smartphones.

My data is time-based and unsupervised, which means I don't have score for any events, it contains events from :

mobile browsing ( http requests )
gps coordinates
calls ( outgoing & incoming )

Should I choose cluster based / density based approach ?
How do I find rare & suspicious events that parents should be notified for with the help of machine learning ?

Any suggestions on how to present the results visually ?

Is a week of data enough for training ?

Are there any good open sources in Java to detect anomaly ?


texts: Which is the best Machine Learning appoach/model when all features are not available for prediction?

texts:
I have 50 Features in a Dataset to predict 1 Variable "Units Sold". I am currently using XGBoost model (Supervised Learning) to train all these 50 Features and the accuracy of the model on the test dataset (after dividing the entire dataset in train/test ratio) is very good.
But in reality, I can ask for only 3 of the Features (out of 50) from the user. And based on these 3 inputs from the user, I need to Forecast the "Units Sold".
Ideally, if I use only these 3 features for training the model, the accuracy falls very low which is why I want to include all these 50 features while training.
I am thinking that since number of features Available while doing the forecast are less than the ones used for Training, I should be using Unsupervised Learning or some other approach.
That is what I need the guidance on.
The list for all the features would be very long but I will mention a few important ones:-

Week_End
ArticleID
Ads_Used (this is a factor of 5 unique values and are used as
one-hot-encoding)
Units_Lag (Units sold in the previous Week. There are 4 Lags to cover
whole month)
Units_Percentage_With_Ads (which Ad sold how many Units that week)

etc.
From the user, I can input only the "Week_End", "ArticleID" and "Ad_Type".


texts: Categorical variables with multiple entries transformed to entity embedding

texts:
I have structured data with lots (tens of thousads) of categories organized into columns. The goal is to enter the data into gradient boosting machine algorithm for a specific prediction.

Some columns have more than one entry for the same sample, i.e., sample1 for column1 has entry1 for line1 and entry2 for line2. Here's an example:

df = pd.DataFrame({'pat': [1, 2, 3, 3, 3, 3], 'diag_type': ['D', 'OP', 'D', 'D', 'D', 'OP'],
                  'diag': ['D_1', 'OP_1', 'D_1', 'D_3', 'D_4', 'OP_2']})

    pat     diag_type   diag
0   1   D   D_1
1   2   OP  OP_1
2   3   D   D_1
3   3   D   D_3
4   3   D   D_4
5   3   OP  OP_2

Since I have so many categories I would need to do some restructuring, and I decided that entity embedding is the best way to do it. However, I haven't figure out a good way to restructure the dataframe. I thought about doing some sort of dummy coding as exemplified in this stackoverflow post, e.g.:

d.get_dummies(df.set_index('pat')).sum(level=0)
    diag_type_D     diag_type_OP    diag_D_1    diag_D_3    diag_D_4    diag_OP_1   diag_OP_2
pat
1   1   0   1   0   0   0   0
2   0   1   0   0   0   1   0
3   3   1   1   1   1   0   1

But then I still get a lot of sparsity and it takes forever. Is there a better way of doing it?


texts: Can parallel computing be utilized for boosting?

texts: Since boosting is sequential, does that mean we cannot use multi-processing or multi-threading to speed it up? If my computer has multiple CPU cores, is there anyway to utilized these extra resources in boosting?


texts: How to train a model to predict a time window than an event will occur on a website

texts:
I have a list of historical timestamps of when a specific event occurred on a website. Currently the timestamp represents a 30 minute window that the event happened within. I am looking to train a model that will predict a time window that the given event is most likely to happen within.

The different events I am trying to predict can have various frequencies such as events that occur every day, week, month, quarter, year. To start off I am interested in looking at events that happen on a daily basis since I can have the most immediate feedback.

For example, I would like to predict when a website will update a given report on their page, I know that this typically happens once per business day. I have a process that will go to the website every 30 minutes to check and see if the report has been posted. Therefore my dataset is a list of timestamps that represent the end time of a 30 minute window the report was published in.

I would like to speed up my process in for a time window the event is most likely to occur so I can receive the report as quickly as possible, without speeding up the process 24 hours per day.

What would be the best way to attack this problem? The prediction will need to adjust every day based upon a new time from the day before.


texts: How word2vec understands the relationship between numbers?

texts: I am using word2vec to train a big corpus of textual information. While analyzing the scatter plot of words I found that it has not only grouped numbers together but also ordered them based on their value. I am able to understand that it groups together words sharing similar context but not able to understand how it got the values also correctly. Can someone please give an intuitive explanation of how this happens? I am attaching the image where I have zoomed in on the numbers.


texts:
I'm trying to classify about 5000 time series for patient admissions into two groups. which consists of large time gaps on nothing (months or years) followed by a short period of a significant number of data points, an admission. Each data point consists of a code (out of 6000 codes) or a code and a numerical value.

Not every time series contains every code, and the codes are quite sparse. I've tried to look at the distribution of values in each code, but got nothing. I've tried a naive Baysian approach of looking at the probabilities of codes in each set, after ignoring any values associated, and did a little better. I'm now out of ideas. I looked at RNNs, but thought the long gaps in data would affect their performance.

I looked at GLM's but the missing data and high dimensionality worried me.

If anyone could suggest a technique that might offer some traction, I would be very grateful.


texts:
Suppose you have a population of N individuals with fitness 1, 2, . . . , N (i.e., all individuals have a
unique fitness value). Suppose you repeatedly apply tournament selection without replacement with
tournament size s = 2 to this population, without doing crossover, mutation, and replacement.

In other words, you run a genetic algorithm with selection alone.
After a certain number of generations you will end up with a population consisting of N copies of
the same individual.
Can you give an estimate of the number of generations needed to achieve that?

Is the answer to this just log(n) ?


texts:
Let's say that I have image data with shape $(32, 32, 3)$ and $50000$

If I would like to reshape it to $(50000, 3, 32, 32)$ what should I do?

I tried np.transpose(0, 3, 1, 2) but it failed.

If I would like to print the number $3$ from $(50000, 3, 32, 32)$ what should I do?


texts: Principal Component Analysis and abnormal data

texts:
I know that PCA is good in differentiating between anomalies and normal data and it helps to differentiate between them when it tries to transfer the data to another dimension. I mean it can somehow maximize the separation between the regular and irregular data points. I also have seen this in my codes. However, someone has asked me to prove it! by a simple detailed example (e.g, a simple 2*2 matrix)? Can anyone help?

Thanks in advance.


texts: Finding multiple histogram graphs with Seaborn

texts:
When graphing with matplotlib I get this 4 histograms model: 4 Histograms

Using Seaborn I am getting the exact graph I need but I cannot replicate it to get 4 at a time:

I want to get 4 of the seaborn graphs (image 2) in the format of the image 1 (4 at a time with the calculations I made with seaborn).

My seaborn code is the following:

import os
import re
import time
import ipdb
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
BASE_DIR = os.path.dirname(os.path.abspath(__file__))
path_file = os.path.join(BASE_DIR, 'camel_product_list.csv')

gapminder = pd.read_csv(path_file)
print(gapminder.head())
df = gapminder
sns.distplot(df['average_histogram_ssim'], hist=True, kde = False, label='All values')
df = gapminder[gapminder.color == 'green']
# sns.distplot(df['lifeExp'], hist = True, kde = True, label='Only Matches')
sns.distplot(df['average_histogram_ssim'], hist_kws={"histtype": "step",
                            "linewidth": 3,
                            "alpha": 1, "color": "b"} ,
                            kde = False, label='Only Matches')
# Plot formatting
plt.legend(prop={'size': 12})
plt.title('ratio_image SSIM')
plt.xlabel('Data Range')
plt.ylabel('Density')
plt.show()

The names of the columns of the dataframe are:

'ratio_text','ratio_image', 'ratio_hist', 'ratio_sub', 'color'

I'm using the color column as a filter.

How can I get the 4 seaborn plots for ratio_text','ratio_image', 'ratio_hist', 'ratio_sub', filtered by all colors and green color?


texts: Activation function vs Squashing function

texts:
This may seem like a very simple and obvious question, but I haven't actually been able to find a direct answer.
Today, in a video explaining deep neural networks, I came across the term Squashing function. This is a term that I have never heard or used. Our professor always used the term Activation function instead. Given the definitions I've been able to find, the two seem to be interchangeable terms.
Are they really synonymous or is there a difference?


texts: I just want to clear one doubt - we use gradient descent to optimize the weights and biases of the neural network, and we use backpropagation for the step that requires calculating partial derivatives of the loss function, or am I misinterpreting something?


texts:
I am new to machine learning and I am struck at one thing. Please help.

I am developing an app to sort images based on user preferences. So initially I have n data points(images) with m features each. Now when user create an album and add some images in it then my algorithm should find similar images from the gallery and add it to the same album. This should work even if user creates more albums. And not all images are needed to be sorted, only the relevant one.

What algorithms should I use to achieve this?


texts:
I am using Scikit-Learn for this classification problem. The dataset has 3 features and 600 data points with labels.

First I used Nearest Neighbor classifier. Instead of using cross-validation, I manually run the fit 5 times and everytime resplit the dataset (80-20) to training set  and test set. The average score turns out to be 0.61

clf = KNeighborsClassifier(4)
score = 0
for i in range(5):
    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
        clf.fit(X_train, y_train)
        score += clf.score(X_test, y_test)
print(scores / 5.0)

However when I ran cross-validation, the average score is merely 0.45.

clf =  KNeighborsClassifier(4)
scores = cross_val_score(clf, X, y, cv=5)
scores.mean()

Why does cross-validation produce significantly lower score than manual resampling?

I also tried Random Forest classifier. This time using Grid Search to tune the parameters:

param_grid = {
    'bootstrap': [True],
    'max_depth': [8, 10, 12],
    'min_samples_leaf': [3, 4, 5],
    'min_samples_split': [8, 10, 12],
    'n_estimators': [62, 64, 66, 68, 70]
}
clf_ = RandomForestClassifier()
grid_search = GridSearchCV(estimator = clf, param_grid = param_grid,
                          cv = 5, n_jobs = -1, verbose = 2)
grid_search.fit(X, y)
grid_search.best_params_, grid_search.best_score_

The best score turned out to be 0.508 with the following parameters

({'bootstrap': True,
  'max_depth': 10,
  'min_samples_leaf': 4,
  'min_samples_split': 10,
  'n_estimators': 64},
 0.5081967213114754)

I went ahead make prediction on the whole 600 data points and accuracy is quite high 0.7688.

best_grid = grid_search.best_estimator_
y_pred = best_grid.predict(X)
accuracy_score(y, y_pred)

I know .best_score_ is the 'Mean cross-validated score of the best_estimator.` But I don't understand why it seems so much lower than the prediction accuracy on the whole set.


texts:
I've built a binary classification model based on Keras and I am getting about 70% accuracy, and about 72% precision and 88% recall, making up to 79% F1-Score. I've tried different data models (deriving new features (columns) from the original ones), but the same model (same hyperparameters, same number of hidden layers (1), and same number of nodes in each layer).

For the different models created, after evaluating, the values of accuracy, precision, recall and F1-Score are almost the same as above. However, the Recall was always (for all models) high for all of the models tested, ranging from 85% to 100%.

What does that say about my model? Is it good enough? Why is the Recall for all models so high, but precision around 61% and 73%?

PS: When I mean "all models", as I said above, it is the same keras model, with same layers and nr of nodes and hyperparameters, just trying different dataframe as input. The first one is the original one, and the others have for example 1 feature less, one that I think is not relevant. Or another model, with input data same as original one, but with adding a derived feature, for example adding a feature "is_weekend", denoted with 1 or 0, but this feature is derived from the date feature already available in the original data. And for me, it's another model, as the input data is not the same.

Here are the evaluation results for my first model, the basic dataframe, with original features, no derived columns in it:

basic_df
accuracy:  0.7068636796381271
f1-score:  0.7915721938539619
precison:  0.6974898160810807
recall:  0.9234265146423681

My dataset in not strongly imbalanced. I have 6992 data points, with 4303 class 1 (61.54%) and 2689 class 0 (38.46%).


texts: Why is stochastic gradient descent so much worse than batch GD for MNIST task?

texts:
Here the code from Tensorflow tutorial:
A Multilayer Perceptron implementation example

With batch size = 100 we quickly got Accuracy: 94.59%.

If I set the batch size to be one, the training takes ten times more time but the accuracy is only nine percent.

I have tested different learning rates with no luck. SGD performance is terrible for small batch sizes. We can expect that SGD performance will be lower, but not ten times less!  What is the reason for this performance loss?


texts: Are there algorithms for clustering objects with pairwise distances, without computing all pairwise distances?

texts:
I'm looking for a clustering algorithm that clusters objects, by using their pairwise distances, without needing to calculate all pairwise distances.

Normally pairwise clustering is done like this: (see here)

Compute full distance matrix between all pairwise combination of objects
Assuming that the distances there are non-euclidean, one might use Spectral Clustering or Affinity propagation on the distance matrix and retrieve the clustering results.

Here comes the however:

Computing the full distance matrix for all pairwise combination of objects is computationally very expensive. So my though was, whether there are some clustering algorithms that only do lookups on a subset of the pairwise distances, so it is not necessary to compute the full matrix?

I know Spectral Clustering works also on sparse matrices, but since it is theoretically possible to compute all pairwise distances, which ones should be left out?

Exited to hear your ideas, thanks!


texts: Google Cloud's vision API dominant colors

texts:
I was trying Google Cloud's Vision API. I uploaded a sample image, and here is the results for the dominant colors. I realized it doesn't simply count pixel colors. The background has many gray pixels which are not included.
How does it perform dominant colors? Any thoughts?


texts: Display images after augmentation in Keras

texts:
How can I display all images after augmentation?

How can I get the number of the trained data after augmentation?

Thank you


texts: What does the "Loss" value given by Keras mean?

texts:
I setup my neural net to use mean square error as shown below. To my understanding (and from reading the documentation) this means that if the correct result of a row is 0.7 and the net predicts 0.8 the contribution to the loss by this entry is (0.8 - 0.7) squared = 0.01

from keras.models import Sequential
from keras.layers import Dense

#...build up neural network layers here...

net.compile(optimizer = 'adam', loss = 'mean_squared_error')
net.fit(training_data, training_results, batch_size = 4, epochs = 100)

I get the following output.

Epoch 100/100
1190/1190 [==============================] - 0s 133us/step - loss: 0.0082

Wow the the loss is tiny, my little neural network is doing so well! However if I validate the result on my original training data

prediction = net.predict(training_data)
prediction_delta = (prediction - training_results)

Although some of the values in the prediction_delta are small overall the loss is way higher than 0.0082 with single values as high as 0.44. Note that this is for the same training data used to fit the net and is not the test data (which also shows similar results) so I would expect to get the value 0.082 back. How does Keras calculate this loss number?


texts: Can machine learning models treat a vector as a whole feature to learn

texts:
We know a ML model naturally takes a feature vector with real valued elements as input and learn to predict. But can it treat a fixed-size vector as a whole feature to learn?
For example, when using a 128-bit string as a feature, we can calculate the total number of different strings occurred in datasets and apply one-hot encoding to them. However, the dimension of the encoding can be very large（thousands）in this way. Or we can use binary to represent the 128-bit string, which only contributes to 128 dimensions. I don't know if the traditional ML models (not CNN) such as random forests are able to learn the dependence of the 128 features. Or is there any ways they can treat the 128 features as a whole?


texts: Rolling window features for multiclass classification

texts:
I'm doing a multiclass classification and data is considered as not being a time-series. Working on a feature engineering and trying to solve the problem with classic KNN, RF, boosting etc. I'm creating new features based on rolling window and found out that people usually do mean aggregation.

Does the mean characteristic of rolling distribution is the most informative?
Are there any other characteristics (like std, quantilies, IQR, etc.) that may be used?
Is it worth to perform some kind of data transformation (like scaling, quantile transfomation or Box-Cox)?


texts: Q table creation and update for dynamic action space

texts:
I am trying to implement a Q-learning algorithm for energy optimization. It is a finite MDP with states represented as 6 dimensional vectors of integers. The number of discrete values in each index of the state vector varies from 24 to 90.

The action space varies from state to state and goes up to 300 possible actions in some states, and below 15 possible actions in some states.

If I could make some assumptions (just for the purpose of testing the model), I could reduce the states to about 400 and actions to less than 200.

How can I construct a Q-table for such an environment? I am not sure how to approach this in Python, how to prevent the table containing lots of impossible state/action combinations, or prevent the agent trying to take those unwanted actions.


texts:
Sorry for the wrong terminology I might use, since I’m a noob. For my supervised learning project for the university I have a dataset (features and labels) which has to evaluated in several ways and also RBF (Radial basis function kernel) should be used to train the classifier. In one of the subtasks I need to manually plot 2 learning curves for training and test data (the original dataset is split 75% for training data and 25% for test data). Using methods from scikit-learn to plot it is not allowed. So, I’m doing it the following way:

Iteratively take random chunks of samples (with replacement) from the training set and train the data model using fit(). I have 525 training samples in total and in iterations I take chunks of [10, 25, 100, 300, 525] samples
Get the score() of each chunk and append to the array of training scores (this array will be used for plotting the curve of the training data)
Get the score() of the whole test set (all 175 samples) in each iteration and append it to the array of test scores (this array will be used for plotting the curve of the test data)

Ultimately plot both curves. On y axis I have the scores, on x axis size of the chunks from the training set. I have doubts that my approach is correct, because some times when I run it, the resulting plots looks suspiciously different from each other.
Sometimes it looks similar to this, which to me looks correct

But sometimes like this
Is it a normal behaviour and if yes, what is the reason of them being so different. If not, then what is wrong in my approach? Thanks


texts: Is it possible to implement a classifier according to quarters? What about missing data?

texts:
I have a dataset with several individuals and features. I'm studying behavior over the year (for instance, averages or iterations of money gained, jobs, etc.).

My ultimate goal is to implement a classifier since I have a specific feature for every person (which is equal to 0, 1, or 2). When I first tried to implement a SVM, I ended up with bad results because I did not have enough data / features: I have too many number 1 individuals and not enough 0 and 2's, so my classifier almost always put people into category 1. Therefore, I tried to increase my number of lines by separating my data into quarters (i.e JAN, FEV, MAR, then APR, MAY, JUN, then JUL, AUG, SEPT, and finally OCT, NOV, DEC)

I was wondering two things:

Would that be a good idea? Do I have to be cautious of a particular hypothesis that could impact my results?
In case it is a good idea, I have some data available for some quarters of the year but sometimes it is missing (for instance let's imagine I don't have "Age" available for my last quarter) ; do I have to drop the feature ? Or would it be wiser to abandon the last quarter ? Or is it possible to make the classifier work despite that lack of information without actually deleting anything ?


texts: Is it possible to train the neural networks for programming

texts: I want to train my neural networks for the task of programming in a different language. So, I want to know whether is it possible or not to train a neural net for programming?


texts: What algorithm should I use to find which sub-component is associated with a attribute?

texts:
I have a n x m one hot encoded matrix of a number of units, where n corresponds to the number of units, and m is the number of total sub-components. This corresponds to a n x 5  matrix which has the 5 attribute values for each of these units.

If I want to create clusters of these sub-components corresponding to these 5 attributes (basically see which sub-component is associated with which attribute), which technique should I use? Would something like PCA or t-SNE work or does this problem require a different approach and why should I use what I use?

I am quite new to this, I would appreciate any help on this matter.


texts: Analyzing A/B test results which are not normally distributed, using independent t-test

texts:
I have a set of results from an A/B test (one control group, one feature group) which do not fit a Normal Distribution.
In fact the distribution resembles more closely the Landau Distribution.

I believe the independent t-test requires that the samples be at least approximately normally distributed, which discourages me using the t-test as a valid method of significance testing.

But my question is:
At what point can one say that the t-test is not a good method of significance testing?

Or put another way, how can one qualify how reliable the p-values of a t-test are, given only the data set?


texts: Web page data extraction using machine learning

texts:
I would like to extract some specific information from web pages. Web pages contain person profiles, and I want to extract information such as name, email, research interested-areas.
Structure of each page is different from one another. How can I extract such information using machine learning? What kind of a method, features I can use?

Or can I use NLP for such task?


texts: Getting answers to bullets (numbered items) from text via NLP

texts:
This is related to information extraction. In real world data, documents are written in bullets/numbered items form. For example,

How to create a website:

 - Get A DNS
 - Get a Hosting
 - Deploy wordpress or some site ...

above is sample of a structured data. Take another example where content is semi structured,

While sandeep was going to home there was a road on the way he saw a

 - Car
 - 2 wheeler
 - cart

and he carefully crossed the road

If I have to find out the "steps to create a website" or "what is on the road"  .. is there an established method using NLP? As some data is semi structured, so simple classification may not work in this case.
Maybe deep learning network or some pretrained model? I have seen google is able to return such results when we search something like "what are steps create a website".


texts: Feature selection: Information leaking if done before CV-split?

texts:
Currently I'm doing some simple feature selection based on correlation between features and variance within one feature. I'm applying this on the whole dataset used for model building before creating the cross-validation.

My question now is if this is acceptable workflow or can significantly affect the CV stats suggesting a better model than it actually is?

Is it technically better to do the CV-split and only then select features on the training set for that split to not leak information?


texts: How to do time series regression without scikit and numpy in Python?

texts:
On a recent Hackerrank interview I was faced with the following problem:

Given a set of timestamps (format 2019-11-26 11:00) and their corresponding stock prices (single float value), approximate the missing stock prices for a set of timestamps.

I tried solving it with an SVR model at first, but I had to realize that none of the usual data science libraries were available for this test.

How would you solve this problem without data science libraries in a limited amount of time?

I ended up doing the following: Took the nearest available measurements and solved the y = mx + b equation based on them, so my prediction would be the corresponding y value for my x.

This seemed to solve the problem quite well for the given test case. My solution in Python


texts: How to fine-tune BERT for Question Answering?

texts:
I wish to train two domain-specific models:

Domain 1: Constitution and related Legal Documents
Domain 2: Technical and related documents.

For Domain 1, I've access to a text-corpus with texts from the constitution and no question-context-answer tuples.
For Domain 2, I've access to Question-Answer pairs.
Is it possible to fine-tune a light-weight BERT model for Question-Answering using just the data mentioned above?
If yes, what are the resources to achieve this task?
Some examples, from the huggingface/models library would be mrm8488/bert-tiny-5-finetuned-squadv2, sshleifer/tiny-distilbert-base-cased-distilled-squad, /twmkn9/albert-base-v2-squad2.